T0ha666

T0ha666

A deploy killed my OTP processes but not my containers — writing a reconciler GenServer

A deploy killed my OTP processes but not the containers running my AI agents. That
mismatch kept a kanban card “executing” for eighteen hours after the work — and the
merged PR — was already done. It seems to be a lesson many Elixir projects eventually
need: OTP supervision stops at the edge of your VM.

The setup: each agent runs in its own Docker Swarm container, and a TaskRunner
GenServer per task watches it over an HTTP socket proxy. When I deploy, every
TaskRunner dies. The containers keep running, happily compiling somebody’s project,
with nobody left listening. The database still says status: :running. The world says
something else.

What I ended up writing is a small reconciler GenServer that ticks every sixty seconds
and asks, for each session that claims to be running: is this still true? A few things I
learned the hard way:

  • The row outlives the process. Anything that must survive a deploy has to be a
    Postgres row, not GenServer state. The GenServer is a fast index over the rows,
    rebuilt on boot.
  • Adopt, don’t fail. If the container is still alive, a fresh TaskRunner
    re-attaches to the run instead of throwing away thirty minutes of agent work because
    of a CSS deploy.
  • Every wait needs a deadline. My first adoption loop polled for an exit-code
    marker file inside the container — but a Swarm reschedule replaces the container, and
    the marker dies with the old one’s /tmp. The loop polled forever, and every other
    recovery sweep politely stepped around the “running” session. That was my
    eighteen-hour card.
  • Pull the decision out of the I/O. The give-up decision is now a pure function of
    four values — decide(container_started_at, session_started_at, elapsed_ms, budget_ms). The incident that cost me a day is a unit test with the real timestamps,
    no mocks, no Docker, milliseconds to run. If the live container started after the
    session’s exec, the marker is provably gone — give up now, don’t poll for it.

Full write-up with the code:
https://t0ha.dev/en/elixir/devops/2026/09/04/supervising-what-you-dont-own.html

Have you ended up writing your own restart strategy for work that lives outside the
BEAM? I’m curious how you bounded the waits.

Most Liked

Mrlaura

Mrlaura

One edge case I’d be interested in is what happens when the container is gone, the exit marker is gone, but the agent has already completed its external side effects.

Your eighteen-hour incident is actually a good example: the PR had already been created and merged, even though the session was still marked as running.

If reconciliation eventually marks that session as failed and requeues it, how do you prevent the next attempt from doing the same work again?

I’d consider keeping a stable task ID across retries, with a separate ID for each execution attempt. That would let the reconciler check for externally visible results, such as an existing branch or PR associated with the task, before starting another container.

I’d also be careful about two reconcilers adopting the same session during overlapping deploys. A conditional database update with an ownership version could make the adoption exclusive, while a fencing token could prevent an old runner from writing a final status after a newer one takes ownership.

The distinction I’d make is between “we lost the execution result” and “the execution failed.” They’re not necessarily the same state.

For an AI agent that can push commits or open PRs, I’d rather leave an uncertain result for reconciliation than automatically retry and risk duplicate side effects.

Does Camelot currently check Git or PR state before requeuing a session whose container was replaced?

aicost_tools

aicost_tools

From the aicost.tools side, what would worry me most is that an orphaned agent container isn’t just a stale status row: it can keep calling the model for 18 hours with nobody listening. On top of the reconciler I’d give each container a lease that the TaskRunner renews every N seconds, and have the agent stop itself (or at least stop making LLM calls) once the lease expires, so a deploy can never leave paid work running unattended. Do you cap spend per task as well, or only wall-clock time?

Where Next?

Popular Backend topics Top

New
AstonJ
This article was written by @rvirding …over a decade ago! Posting here in case anyone else finds it of interest and adding it to our Erla...
New
First poster: bot
Rust vs Go — Bitfield Consulting. Which is better, Rust or Go? Which language should you choose for your next project, and why? How do t...
New
First poster: bot
Taking PHP Seriously - Slack Engineering. Slack uses PHP for most of its server-side application logic, which is an unusual choice these...
New
First poster: bot
Julia is a scientific programming language that is free and open source.1 It is a relatively new language that borrows inspiration from l...
New
paulanthonywilson
So you’re enjoying using WebSockets with Elixir’s Phoenix Framework, and you want to send some binary messages. Maybe it’s an audio clip,...
New
First poster: bot
What’s Next for Teal, the typed dialect of Lua - FOSDEM 2021. This is my talk about the latest updates on the Teal programming language,...
New
First poster: bot
Just a small test with lists in cython. Considering echosystem, multithreading and ease of use, Julia is a clear winner here.
New
First poster: bot
Our blog has had a long standing interest in novel uses of the BEAM, or Erlang virtual machine, as shown by the many articles we have pub...
New
Johannes-Werbrouck
In this blog post over on Fly.io I take a look at PHP 8.1’s new Enum class, and show you how it can be used to set up a level structure f...
New

Other popular topics Top

PragmaticBookshelf
Machine learning can be intimidating, with its reliance on math and algorithms that most programmers don't encounter in their regular wor...
New
PragmaticBookshelf
Brace yourself for a fun challenge: build a photorealistic 3D renderer from scratch! In just a couple of weeks, build a ray tracer that r...
New
New
DevotionGeo
The V Programming Language Simple language for building maintainable programs V is already mentioned couple of times in the forum, but I...
New
PragmaticBookshelf
Use WebRTC to build web applications that stream media and data in real time directly from one user to another, all in the browser. ...
New
AstonJ
If you get Can't find emacs in your PATH when trying to install Doom Emacs on your Mac you… just… need to install Emacs first! :lol: bre...
New
New
New
PragmaticBookshelf
Explore the power of Ash Framework by modeling and building the domain for a real-world web application. Rebecca Le @sevenseacat and ...
New
PragmaticBookshelf
Escape callback hell and ship fast, clean code that reads as smoothly as it runs. Squash bugs and stamp out memory leaks with an intuitiv...
New