T0ha666
A deploy killed my OTP processes but not my containers — writing a reconciler GenServer
A deploy killed my OTP processes but not the containers running my AI agents. That
mismatch kept a kanban card “executing” for eighteen hours after the work — and the
merged PR — was already done. It seems to be a lesson many Elixir projects eventually
need: OTP supervision stops at the edge of your VM.
The setup: each agent runs in its own Docker Swarm container, and a TaskRunner
GenServer per task watches it over an HTTP socket proxy. When I deploy, every
TaskRunner dies. The containers keep running, happily compiling somebody’s project,
with nobody left listening. The database still says status: :running. The world says
something else.
What I ended up writing is a small reconciler GenServer that ticks every sixty seconds
and asks, for each session that claims to be running: is this still true? A few things I
learned the hard way:
-
The row outlives the process. Anything that must survive a deploy has to be a
Postgres row, not GenServer state. The GenServer is a fast index over the rows,
rebuilt on boot. -
Adopt, don’t fail. If the container is still alive, a fresh
TaskRunner
re-attaches to the run instead of throwing away thirty minutes of agent work because
of a CSS deploy. -
Every wait needs a deadline. My first adoption loop polled for an exit-code
marker file inside the container — but a Swarm reschedule replaces the container, and
the marker dies with the old one’s/tmp. The loop polled forever, and every other
recovery sweep politely stepped around the “running” session. That was my
eighteen-hour card. -
Pull the decision out of the I/O. The give-up decision is now a pure function of
four values —decide(container_started_at, session_started_at, elapsed_ms, budget_ms). The incident that cost me a day is a unit test with the real timestamps,
no mocks, no Docker, milliseconds to run. If the live container started after the
session’s exec, the marker is provably gone — give up now, don’t poll for it.
Full write-up with the code:
https://t0ha.dev/en/elixir/devops/2026/09/04/supervising-what-you-dont-own.html
Have you ended up writing your own restart strategy for work that lives outside the
BEAM? I’m curious how you bounded the waits.
Most Liked
Mrlaura
One edge case I’d be interested in is what happens when the container is gone, the exit marker is gone, but the agent has already completed its external side effects.
Your eighteen-hour incident is actually a good example: the PR had already been created and merged, even though the session was still marked as running.
If reconciliation eventually marks that session as failed and requeues it, how do you prevent the next attempt from doing the same work again?
I’d consider keeping a stable task ID across retries, with a separate ID for each execution attempt. That would let the reconciler check for externally visible results, such as an existing branch or PR associated with the task, before starting another container.
I’d also be careful about two reconcilers adopting the same session during overlapping deploys. A conditional database update with an ownership version could make the adoption exclusive, while a fencing token could prevent an old runner from writing a final status after a newer one takes ownership.
The distinction I’d make is between “we lost the execution result” and “the execution failed.” They’re not necessarily the same state.
For an AI agent that can push commits or open PRs, I’d rather leave an uncertain result for reconciliation than automatically retry and risk duplicate side effects.
Does Camelot currently check Git or PR state before requeuing a session whose container was replaced?
aicost_tools
From the aicost.tools side, what would worry me most is that an orphaned agent container isn’t just a stale status row: it can keep calling the model for 18 hours with nobody listening. On top of the reconciler I’d give each container a lease that the TaskRunner renews every N seconds, and have the agent stop itself (or at least stop making LLM calls) once the lease expires, so a deploy can never leave paid work running unattended. Do you cap spend per task as well, or only wall-clock time?
Popular Backend topics
Other popular topics
Categories:
Sub Categories:
Popular Portals
- /elixir
- /rust
- /wasm
- /ruby
- /erlang
- /phoenix
- /keyboards
- /python
- /js
- /rails
- /security
- /go
- /swift
- /vim
- /java
- /clojure
- /typescript
- /emacs
- /haskell
- /svelte
- /onivim
- /kotlin
- /react
- /c-plus-plus
- /crystal
- /tailwind
- /gleam
- /ocaml
- /elm
- /vscode
- /flutter
- /html
- /ash
- /deepseek
- /zig
- /opensuse
- /php
- /centos
- /react-native
- /scala
- /lisp
- /sublime-text
- /textmate
- /nixos
- /debian
- /deno
- /agda
- /django
- /nodejs
- /kubuntu
- /arch-linux
- /ubuntu
- /spring
- /revery
- /manjaro
- /quarkus
- /julia
- /diversity
- /lua
- /laravel









