A deploy killed my OTP processes but not the containers running my AI agents. That
mismatch kept a kanban card “executing” for eighteen hours after the work — and the
merged PR — was already done. It seems to be a lesson many Elixir projects eventually
need: OTP supervision stops at the edge of your VM.

The setup: each agent runs in its own Docker Swarm container, and a TaskRunner
GenServer per task watches it over an HTTP socket proxy. When I deploy, every
TaskRunner dies. The containers keep running, happily compiling somebody’s project,
with nobody left listening. The database still says status: :running. The world says
something else.

What I ended up writing is a small reconciler GenServer that ticks every sixty seconds
and asks, for each session that claims to be running: is this still true? A few things I
learned the hard way:

• The row outlives the process. Anything that must survive a deploy has to be a
  Postgres row, not GenServer state. The GenServer is a fast index over the rows,
  rebuilt on boot.
• Adopt, don’t fail. If the container is still alive, a fresh TaskRunner
  re-attaches to the run instead of throwing away thirty minutes of agent work because
  of a CSS deploy.
• Every wait needs a deadline. My first adoption loop polled for an exit-code
  marker file inside the container — but a Swarm reschedule replaces the container, and
  the marker dies with the old one’s /tmp. The loop polled forever, and every other
  recovery sweep politely stepped around the “running” session. That was my
  eighteen-hour card.
• Pull the decision out of the I/O. The give-up decision is now a pure function of
  four values — decide(container_started_at, session_started_at, elapsed_ms, budget_ms). The incident that cost me a day is a unit test with the real timestamps,
  no mocks, no Docker, milliseconds to run. If the live container started after the
  session’s exec, the marker is provably gone — give up now, don’t poll for it.

Full write-up with the code:
https://t0ha.dev/en/elixir/devops/2026/09/04/supervising-what-you-dont-own.html

Have you ended up writing your own restart strategy for work that lives outside the
BEAM? I’m curious how you bounded the waits.