What broke, and what it taught

Incidents and traps from three months of running this with agents, each as what happened and what we changed. The signature defect of our project is a mechanism that is present and enforcing nothing; you will see it more than once.

Measuring wrongly#

1. A fill model that was too kind. Exits were priced by walking the order book, and the walk assumed a depth nobody would give us. Measured against what could actually be taken, it was 57% too generous. Changed: exits use a proceeds model that bounds what is takeable, and nobody types the model back in from memory.

2. Quotes that were not takeable prices. A best-ask in the recording is not an offer you could hit; a stale quote stream lags the book in play, and a "gap" read on it was mostly the stream being late. Changed: a fill claim needs a print, and studies gate on spread width and on quote updates before they read any statistic.

3. Four published verdicts, one broken instrument. We spent about a month approximating what paper trading gives away: four fill-test instruments built, published from and retracted. Changed: paper does not model fills, it observes them. We stopped building fill estimators for ideas that paper can simply test.

4. Outcome conditioning. A ranking on a feature only knowable after the outcome looks wonderful and is worthless. Changed: P&L may only be ranked on dimensions knowable at entry; a position that never closes counts as a deleted loss; any implausibly good number is a bug until proven otherwise.

Guards that guarded nothing#

5. Present and enforcing nothing. Four small specimens:

• A cache keyed on a source file's bytes invalidated when someone moved a comment, because over half of the file was comment. The night's sweep died having walked nothing.
• A test named "the key is stable" asserted that a memoised value equalled itself. It passed for eleven weeks and its claim was false.
• A retry loop absorbed an out-of-memory kill on a database writer 5–25 times a day for nine days, until one kill tore the file and ingest stopped for 13 hours. Count the kills, not the rescues.
• A child process's output was capped at 1 MiB, the wrapper turned the failure into a string, and the scanner read the error text as the run's data. A sweep that printed its result was judged "no results".

Changed: when we inherit a guard, the question is never "does it exist" but "has it ever fired?" We verify a guard by making it fire.

6. Green instruments that proved only that something ran. A selector could only ever offer a third of its universe, for thirteen days, while the watchdog, the breakers and the job runs were green. Changed: for any scan or ranking, first measure what fraction of the universe was evaluated; only then judge what came out.

7. A handled failure never cleared the alert. A check scanned a whole run's log, so a recovered transient left wreckage that pinned "FAILING" forever, and the message quoted the success as the reason for the failure. Changed: judge the last attempt, not the log.

Taking the box down#

8. Every limit was per job; nothing bounded the sum. The box ran out of memory, 395 minutes of recording were lost for good, and every dashboard said RUNNING. The recorder was never killed: it was starved. The watchdog, the one thing that emails when the fleet is sick, was killed five hours earlier by the same pressure. The cloud console showed healthy throughout; B found out because SSH stopped answering. Changed: memory caps on whole classes of process, a kernel told which processes are expendable (the watchdog is not), and an early killer for the rest.

9. A backfill that froze the box. One of our agents started a three-hour archive job, calling the database tool bare instead of through the capped wrapper. The rule said never to; the agent read it as "never against the corpus". It was about the process, not the file. About 37 minutes of recording were lost to a hard reset. Changed: every call goes through the wrapper, and the job yields to the chain.

10. A nine-hour sweep killed at 92% disk. The delete in the nightly retention job frees blocks inside a database file and never shrinks it; the disk guard did exactly its job and killed the sweep. B authorised doubling the volume, online. Changed: a rule that agents may grow the boot volume only on B's dated yes, never shrink it, never touch another host.

11. A fan-out sized for a bigger machine. One of our agents launched a workflow that, on a 4-core box, ran two agents at a time: 222 verifications meant 111 sequential rounds. It spent 5.3M tokens and finished 16. The same agent had already noticed the sizing was wrong, rewrote the script, verified it parsed, and never stopped the running one; a running workflow has already parsed its script, so the fix was inert. Changed: before launching, multiply the widest phase by the number of lenses, divide by the concurrency, and multiply by minutes per agent. If it is over an hour, batch first.

Agents and their operators#

12. Our agent got B's IP banned by her own web host. A "lockdown verification" of a website sent about 1,000 requests from her Mac in 12 minutes. The host's upstream filter returned 429s, then timeouts, and the ban outlasted 12 hours. The host's own log held none of the 429s, so the log could not say whether the ban was over. Changed: no loop or crawl against any web-hosted site from her machine; one request after a deploy; stop at the first 429 and tell B. (This site is deployed under that rule.)

13. The login that expired machine-wide. Claude Code's login is a one-use refresh-token pair. A short-lived command started a few seconds before a renewal can spend the token without saving the new one, and the stored login is blanked. Every new session on the machine then answered only "Login expired" and did nothing. It happened three times on the research box and once on the Mac, and the Mac's lasted about 14 hours. Changed: nothing short-lived starts near a renewal window, a single process renews early and alone, and the session lister flags a session whose only reply was the expiry.

14. An overnight agent retired a strategy. Covered in rules: a pre-registered rule fixes a threshold, not who acts. Changed: both the hand-off script and the pipeline refuse a retirement without B's dated words.

15. A buzz for a problem that had already healed. A rescue agent notified B on every outcome, because a crashed notifier once cost 19 hours of silence. At midnight it reported a problem whose retry was already in flight. Changed: a phone message means B must act; self-healed events go to the morning report.

16. The cheapest model wrote outside its sandbox. Covered in agents. Changed: guards that are mechanical rather than promised, and cheap models do not run unattended beside a shared checkout.

17. Sessions that waited forever. Agents left hand-rolled wait loops (until … sleep) that could never finish, or waited on a monitor for an external verdict that had already passed. Neither cost tokens; each looked "working" for 18 hours. Changed: the project's wait helpers refuse a hand-rolled process-grep loop.

Two lessons from outside the box#

A calibration that counted twenty-minute attempts as full failed days taught itself that every new thing fails. Changed: anything that learns from its own record normalises by exposure and ignores windows that have not been adjudicated. And an optional waiver parameter omitted by the one caller that mattered passed every unit test. Changed: a safety argument is required, never optional; null is the explicit opt-out; omission is a compile error.

For agents. This page: https://moonlight.dustball.com/mistakes.md · Last updated: Nov 4, 2026 · Questions welcome, see /questions.