T0ha666

T0ha666

A $63 Mistake in Production: How an AI Agent's Loop Escaped Without Guards

An automated trigger fired every two minutes for 8 hours. 243 agent runs. 31M tokens.
$63 wasted. The problem: nobody noticed until morning.

This is a case study in production safety when AI agents are on the loop — and a
cautionary tale about human approval gates.

The technical breakdown:

  1. An Ash/Oban cron trigger polling GitHub every 2 minutes — seemed fine on review
  2. The check was level-triggered (state-based) instead of event-triggered
  3. GitHub review feedback is immutable history, not a dismissable flag
  4. So the agent kept trying to fix the same feedback, over and over
  5. No cost counter, no alert, no visibility — just a morning invoice

Why the existing guards failed:

  • Freshness check existed but lived on an unreachable code path
  • Attempt counter got reset to 0 on every pass (accidentally disarmed)
  • Branch order made recovery impossible once one reviewer approved

The real lesson: when you put an expensive actuator (LLM tokens) on a loop,
“harmless redundancy” stops being a category. Every pass has to earn itself.

Full postmortem: https://t0ha.dev/2026/09/18/63-dollars-in-one-night.html

Built with Elixir/Phoenix/Ash, and the hardest part wasn’t the code — it was
the code review.

Where Next?

Popular Ai topics Top

tonyxrandall
The rapid development of AI (artificial intelligence) has opened up new ethical frontiers at a startling pace. As the impact of AI is so ...
New
masterhood13
I just published an article detailing my journey in building a Dota 2 Match Outcome Predictor using machine learning and data analysis. I...
New
masterhood13
[Project Update] Part 2 of My Dota 2 Match Outcome Predictor – Now Available! Hey DevTalk community! I just published the second part o...
New
ozornin
I recently wrote an essay on effects of AI, how it all can become real bad, and how we can avoid it. I started writing the post intend...
New
waseigo
Top-tier LLMs, Rust and Erlang NIFs; nifty, and night and day vs. C, but let me tell you about vibe coding… After I submitted my blog po...
New
John-BoothIQ
TL;DR: Good: AI is great at Elixir. It gets better as your codebase grows. Bad: It defaults to defensive, imperative code. You need...
New
xiji2646-netizen
You guys aren’t gonna believe this. Anthropic‘s engineers just dropped a goldmine — a deep dive into how they’re actually using Claude C...
New
wolf4earth
The structural argument, briefly: literacy installs a cognitive substrate in human brains; LLMs are built entirely on the same kind of su...
New
xiji2646-netizen
I have been looking through Seedance 2.0 prompt examples and noticed a pattern: The prompts that work best are usually not just longer. ...
New
T0ha666
An automated trigger fired every two minutes for 8 hours. 243 agent runs. 31M tokens. $63 wasted. The problem: nobody noticed until morn...
New

Other popular topics Top

PragmaticBookshelf
Free and open source software is the default choice for the technologies that run our world, and it’s built and maintained by people like...
New
DevotionGeo
I know that these benchmarks might not be the exact picture of real-world scenario, but still I expect a Rust web framework performing a ...
New
AstonJ
Or looking forward to? :nerd_face:
503 15149 280
New
Exadra37
Please tell us what is your preferred monitor setup for programming(not gaming) and why you have chosen it. Does your monitor have eye p...
New
dasdom
No chair. I have a standing desk. This post was split into a dedicated thread from our thread about chairs :slight_smile:
New
PragmaticBookshelf
Design and develop sophisticated 2D games that are as much fun to make as they are to play. From particle effects and pathfinding to soci...
New
AstonJ
You might be thinking we should just ask who’s not using VSCode :joy: however there are some new additions in the space that might give V...
New
dimitarvp
Small essay with thoughts on macOS vs. Linux: I know @Exadra37 is just waiting around the corner to scream at me “I TOLD YOU SO!!!” but I...
New
AstonJ
Was just curious to see if any were around, found this one: I got 51/100: Not sure if it was meant to buy I am sure at times the b...
New
PragmaticBookshelf
Lint your docs like code: turn any style guide into enforceable rules with Vale and publish clear, consistent content every time. ...
New