T0ha666
A $63 Mistake in Production: How an AI Agent's Loop Escaped Without Guards
An automated trigger fired every two minutes for 8 hours. 243 agent runs. 31M tokens. $63 wasted. The problem: nobody noticed until morning.
This is a case study in production safety when AI agents are on the loop — and a cautionary tale about human approval gates.
The technical breakdown:
- An Ash/Oban cron trigger polling GitHub every 2 minutes — seemed fine on review
- The check was level-triggered (state-based) instead of event-triggered
- GitHub review feedback is immutable history, not a dismissable flag
- So the agent kept trying to fix the same feedback, over and over
- No cost counter, no alert, no visibility — just a morning invoice
Why the existing guards failed:
- Freshness check existed but lived on an unreachable code path
- Attempt counter got reset to 0 on every pass (accidentally disarmed)
- Branch order made recovery impossible once one reviewer approved
The real lesson: when you put an expensive actuator (LLM tokens) on a loop, “harmless redundancy” stops being a category. Every pass has to earn itself.
Full postmortem:
Built with Elixir/Phoenix/Ash, and the hardest part wasn’t the code — it was the code review.
Most Liked
aicost_tools
$63 / 31M tokens is about $2.03 per million blended across those 243 runs, so each pass averaged roughly 127k tokens — exactly the kind of loop where a level-triggered poll quietly becomes a meter. Two cheap guards that would have caught this before morning: a hard spend or token ceiling on the actuator itself (kill the Oban job when the cumulative prompt+completion for that trigger crosses a few dollars), and an alert on “same review thread / same feedback hash seen N times,” because immutable GitHub review history will never clear a level check. Seat plans hide this until the invoice; an API key at least makes every retry visible as line items, but only if something is watching the running total rather than a single-request freshness flag that can be skipped. The attempt counter resetting to 0 is the scarier bug than the missing alert — once the safety latch disarms itself, redundancy stops being free.
Popular Ai topics
Other popular topics
Categories:
Sub Categories:
Popular Portals
- /elixir
- /rust
- /wasm
- /ruby
- /erlang
- /phoenix
- /keyboards
- /python
- /js
- /rails
- /security
- /go
- /swift
- /vim
- /java
- /clojure
- /typescript
- /emacs
- /haskell
- /svelte
- /onivim
- /kotlin
- /react
- /c-plus-plus
- /crystal
- /tailwind
- /gleam
- /ocaml
- /elm
- /vscode
- /flutter
- /html
- /ash
- /deepseek
- /zig
- /opensuse
- /php
- /centos
- /react-native
- /scala
- /lisp
- /sublime-text
- /textmate
- /nixos
- /debian
- /deno
- /agda
- /django
- /kubuntu
- /nodejs
- /arch-linux
- /ubuntu
- /spring
- /revery
- /manjaro
- /julia
- /lua
- /quarkus
- /diversity
- /laravel









