T0ha666

T0ha666

A $63 Mistake in Production: How an AI Agent's Loop Escaped Without Guards

An automated trigger fired every two minutes for 8 hours. 243 agent runs. 31M tokens. $63 wasted. The problem: nobody noticed until morning.

This is a case study in production safety when AI agents are on the loop — and a cautionary tale about human approval gates.

The technical breakdown:

  1. An Ash/Oban cron trigger polling GitHub every 2 minutes — seemed fine on review
  2. The check was level-triggered (state-based) instead of event-triggered
  3. GitHub review feedback is immutable history, not a dismissable flag
  4. So the agent kept trying to fix the same feedback, over and over
  5. No cost counter, no alert, no visibility — just a morning invoice

Why the existing guards failed:

  • Freshness check existed but lived on an unreachable code path
  • Attempt counter got reset to 0 on every pass (accidentally disarmed)
  • Branch order made recovery impossible once one reviewer approved

The real lesson: when you put an expensive actuator (LLM tokens) on a loop, “harmless redundancy” stops being a category. Every pass has to earn itself.

Full postmortem:

Built with Elixir/Phoenix/Ash, and the hardest part wasn’t the code — it was the code review.

Most Liked

aicost_tools

aicost_tools

$63 / 31M tokens is about $2.03 per million blended across those 243 runs, so each pass averaged roughly 127k tokens — exactly the kind of loop where a level-triggered poll quietly becomes a meter. Two cheap guards that would have caught this before morning: a hard spend or token ceiling on the actuator itself (kill the Oban job when the cumulative prompt+completion for that trigger crosses a few dollars), and an alert on “same review thread / same feedback hash seen N times,” because immutable GitHub review history will never clear a level check. Seat plans hide this until the invoice; an API key at least makes every retry visible as line items, but only if something is watching the running total rather than a single-request freshness flag that can be skipped. The attempt counter resetting to 0 is the scarier bug than the missing alert — once the safety latch disarms itself, redundancy stops being free.

Where Next?

Popular Ai topics Top

RobertRichards
In the early days, online gaming was limited to the screens of smartphones, PCs, tablets, and other devices. However, with the advent of ...
New
masterhood13
[Project Update] Part 2 of My Dota 2 Match Outcome Predictor – Now Available! Hey DevTalk community! I just published the second part o...
New
ozornin
I recently wrote an essay on effects of AI, how it all can become real bad, and how we can avoid it. I started writing the post intend...
New
New
vipulbhj
Agents execute at scale. Accountability doesn’t transfer. The founder who delegates everything to AI doesn’t become a CEO with thousands ...
New
xiji2646-netizen
You guys aren’t gonna believe this. Anthropic‘s engineers just dropped a goldmine — a deep dive into how they’re actually using Claude C...
New
aralroca
I wrote about a different approach to AI agents. Instead of building bots that screenshot your screen and click buttons (like OpenAI Oper...
New
xiji2646-netizen
Anthropic announced Claude Mythos Preview this week – and then said it will not release it to the public. Their reasoning: the model’s cy...
New
wolf4earth
The structural argument, briefly: literacy installs a cognitive substrate in human brains; LLMs are built entirely on the same kind of su...
New
xiji2646-netizen
I have been looking through Seedance 2.0 prompt examples and noticed a pattern: The prompts that work best are usually not just longer. ...
New

Other popular topics Top

AstonJ
I ended up cancelling my Moonlander order as I think it’s just going to be a bit too bulky for me. I think the Planck and the Preonic (o...
New
AstonJ
Do the test and post your score :nerd_face: :keyboard: If possible, please add info such as the keyboard you’re using, the layout (Qw...
New
dimitarvp
Small essay with thoughts on macOS vs. Linux: I know @Exadra37 is just waiting around the corner to scream at me “I TOLD YOU SO!!!” but I...
New
Exadra37
I am asking for any distro that only has the bare-bones to be able to get a shell in the server and then just install the packages as we ...
New
PragmaticBookshelf
Build highly interactive applications without ever leaving Elixir, the way the experts do. Let LiveView take care of performance, scalabi...
New
DevotionGeo
The V Programming Language Simple language for building maintainable programs V is already mentioned couple of times in the forum, but I...
New
PragmaticBookshelf
Build efficient applications that exploit the unique benefits of a pure functional language, learning from an engineer who uses Haskell t...
New
PragmaticBookshelf
Develop, deploy, and debug BEAM applications using BEAMOps: a new paradigm that focuses on scalability, fault tolerance, and owning each ...
New
PragmaticBookshelf
Build modern server-driven web applications using htmx. Whatever programming language you use, you’ll write less (and cleaner) code. ...
New
PragmaticBookshelf
A concise guide to MySQL 9 database administration, covering fundamental concepts, techniques, and best practices. Neil Smyth MySQL...
New