for engineers running agents in prod
What breaks when you ship an agent.
A weekly trace of one real incident, one runtime fix, and the line of code that would have stopped it. No framework tutorials. No demo screenshots. Read the log, not the pitch.
4,812 engineers on-call · issue 47 out thursday
- 01logs.query { svc: "api-gateway", range: "15m", level: "error" } → 412 rows · top: upstream timeout (389) [200] 890tok · 640ms
- 02metrics.fetch { query: "rate(http_5xx[5m])" } → spike at 03:12:04 UTC, +38x baseline [retry] 305tok · 180ms
- 03↳ metrics.fetch.retry { attempt: 2, backoff: "240ms" } → ok on retry · upstream was cold [retry] 61tok · 240ms
- 04k8s.describe_pod { ns: "prod", pod: "checkout-7f9d4" } → OOMKilled · restarts: 6 [err] 220tok · 310ms
- 05git.blame { file: "checkout/pool.go", line: 88 } → commit a3f9c2 · raised pool size 3d ago [200] 176tok · 260ms
- 06memory.write { key: "PD-58213.hypothesis" } → stored · confidence 0.81 [200] 94tok · 90ms
- 07k8s.scale { deploy: "checkout", replicas: 8 } → rolled out · 4/8 ready [200] 130tok · 420ms
- 08metrics.fetch ▍
recent issues
archive →- #472026-08-06the retry storm that took down its own fallbackincident
- #462026-07-30context windows are not memory — a taxonomyarchitecture
- #452026-07-23we cut escalations 34% by deleting a toolcase study
- #442026-07-16eval suites lie to you the same way unit tests dotesting
- #432026-07-09the 12-line budget guard we now ship by defaultruntime
one incident, worked start to finish
One issue. One incident, worked start to finish. Five minutes, timed against the actual word count.
Issue 47 — the retry storm that took down its own fallback
Issue 47 · 5 min read
Morning.
An agent at a mid-size fintech spent four hours last week retrying a payments call that was never going to succeed — because its own circuit breaker was implemented as a tool the model had to remember to call. It forgot. Nobody paged anyone, because from the outside, retries look like progress.
“The agent wasn’t stuck. It was doing exactly what we asked, four thousand times.”
This issue:
- ·Why circuit breakers belong in the runtime, not the tool schema
- ·A 12-line budget guard we now ship by default
- ·Reader trace: a support-triage agent that cut escalations 34%
Full breakdown, with the trace, below the fold.
— Priya
editor, Signal AI
from the rotation
- Marcus Ude
Platform eng, Fenwick Logistics“The budget-guard snippet from issue 41 is now a required review comment on every agent PR at our shop. Caught two runaway loops before they hit the API bill.”
- Ren Okafor
Staff SRE, Lumen Health“Forwarded the retry-storm issue to our whole on-call rotation. First newsletter that's actually changed how we write tool schemas.”
- Dana Sorensen
Eng lead, Northbeam“Most agent content is either hype or a LangChain tutorial. This is the only one that reads like it was written by someone who's been paged at 3am.”