T0ha666

T0ha666

A $63 Mistake in Production: How an AI Agent's Loop Escaped Without Guards

An automated trigger fired every two minutes for 8 hours. 243 agent runs. 31M tokens. $63 wasted. The problem: nobody noticed until morning.

This is a case study in production safety when AI agents are on the loop — and a cautionary tale about human approval gates.

The technical breakdown:

  1. An Ash/Oban cron trigger polling GitHub every 2 minutes — seemed fine on review
  2. The check was level-triggered (state-based) instead of event-triggered
  3. GitHub review feedback is immutable history, not a dismissable flag
  4. So the agent kept trying to fix the same feedback, over and over
  5. No cost counter, no alert, no visibility — just a morning invoice

Why the existing guards failed:

  • Freshness check existed but lived on an unreachable code path
  • Attempt counter got reset to 0 on every pass (accidentally disarmed)
  • Branch order made recovery impossible once one reviewer approved

The real lesson: when you put an expensive actuator (LLM tokens) on a loop, “harmless redundancy” stops being a category. Every pass has to earn itself.

Full postmortem:

Built with Elixir/Phoenix/Ash, and the hardest part wasn’t the code — it was the code review.

Most Liked

aicost_tools

aicost_tools

$63 / 31M tokens is about $2.03 per million blended across those 243 runs, so each pass averaged roughly 127k tokens — exactly the kind of loop where a level-triggered poll quietly becomes a meter. Two cheap guards that would have caught this before morning: a hard spend or token ceiling on the actuator itself (kill the Oban job when the cumulative prompt+completion for that trigger crosses a few dollars), and an alert on “same review thread / same feedback hash seen N times,” because immutable GitHub review history will never clear a level check. Seat plans hide this until the invoice; an API key at least makes every retry visible as line items, but only if something is watching the running total rather than a single-request freshness flag that can be skipped. The attempt counter resetting to 0 is the scarier bug than the missing alert — once the safety latch disarms itself, redundancy stops being free.

Where Next?

Popular Ai topics Top

tonyxrandall
The rapid development of AI (artificial intelligence) has opened up new ethical frontiers at a startling pace. As the impact of AI is so ...
New
RobertRichards
In the early days, online gaming was limited to the screens of smartphones, PCs, tablets, and other devices. However, with the advent of ...
New
masterhood13
[Project Update] Part 2 of My Dota 2 Match Outcome Predictor – Now Available! Hey DevTalk community! I just published the second part o...
New
ozornin
I recently wrote an essay on effects of AI, how it all can become real bad, and how we can avoid it. I started writing the post intend...
New
waseigo
Top-tier LLMs, Rust and Erlang NIFs; nifty, and night and day vs. C, but let me tell you about vibe coding… After I submitted my blog po...
New
vipulbhj
Agents execute at scale. Accountability doesn’t transfer. The founder who delegates everything to AI doesn’t become a CEO with thousands ...
New
aralroca
I wrote about a different approach to AI agents. Instead of building bots that screenshot your screen and click buttons (like OpenAI Oper...
New
xiji2646-netizen
I have been evaluating the three major AI video generation APIs for a project and figured this might save others some research time. Curi...
New
xiji2646-netizen
Anthropic announced Claude Mythos Preview this week – and then said it will not release it to the public. Their reasoning: the model’s cy...
New
wolf4earth
The structural argument, briefly: literacy installs a cognitive substrate in human brains; LLMs are built entirely on the same kind of su...
New

Other popular topics Top

PragmaticBookshelf
Machine learning can be intimidating, with its reliance on math and algorithms that most programmers don't encounter in their regular wor...
New
AstonJ
poll poll Be sure to check out @Dusty’s article posted here: An Introduction to Alternative Keyboard Layouts It’s one of the best write-...
New
New
PragmaticBookshelf
Author Spotlight Mike Riley @mriley This month, we turn the spotlight on Mike Riley, author of Portable Python Projects. Mike’s book ...
New
PragmaticBookshelf
Programming Ruby is the most complete book on Ruby, covering both the language itself and the standard library as well as commonly used t...
New
PragmaticBookshelf
A masterclass in the fundamentals and principles of functional programming. Minh Quang Tran The Art of Functional Programming is a ma...
New
PragmaticBookshelf
Fight complexity and reclaim the original spirit of agility by learning to simplify how you develop software. The result: a more humane a...
New
PragmaticBookshelf
Escape callback hell and ship fast, clean code that reads as smoothly as it runs. Squash bugs and stamp out memory leaks with an intuitiv...
New
mindriot
Ok, well here are some thoughts and opinions on some of the ergonomic keyboards I have, I guess like mini review of each that I use enoug...
New
PragmaticBookshelf
This book forgoes the abstract and instead provides concrete examples to help you better leverage the unique properties of Elixir, Erlang...
New