The tests exist but check the happy path. Bugs concentrate in code that already needed fixes and in a few large PRs that cut across the system, and they surface weeks later in Canvas and Cloud.
Whole repository, all authors: 905 fix commits traced to the PR that introduced the bug, 602 bugs read in full, 988 PRs tested statistically, and line coverage from CI on main.
| Hypothesis | Holds? | Evidence |
|---|---|---|
| We don't test PRs well | Partly | 87% of code PRs change tests and line coverage is 90%, but the tests check the happy path. Adding a cross-package test lowers the odds of a later fix (0.61×). |
| We don't write regression tests | No | 551 of 602 fixes (92%) add a test for the bug, but only for the reported symptom. |
| The changes themselves are risky | Yes | Changing code that already needed fixes is the strongest bug-specific factor, fix PRs need fixes more often, and 36 large cross-cutting PRs produced 41% of confirmed bugs. |
| It's the architecture | Partly | Several code paths for one behavior, downstream contracts not tested here, unwrapped dependencies, and objects shared across conversations. |
| It's the coding agents | Unclear | Agent co-authorship: 1.34× odds, p = 0.10. The visible effect is indirect: agents make large PRs cheap and write tests that mirror their own code. |
Agent-server HTTP API. 110 of 602 bugs, the most of any part, at 93% coverage. Mainly wrong status codes or unchecked input (27) and dropped settings (21).
Agent-server runtime. Highest density, 12.5 fix commits per 1,000 lines; mainly concurrency and lifecycle.
SDK conversation. One API, two implementations (local and remote); 94% covered, 109 fix commits.
SDK LLM. 117 fix commits, the most of any part; 50 of its 103 bugs came from provider or LiteLLM changes.
Consumer boundary. Canvas, Cloud, the CLI and automations found 30% of all bugs. No test in this repo checks what they expect.
Tools. Lowest coverage (44%), yet fewer bugs per line than the SDK core.
External systems. LLM providers are mocked on every PR; real-LLM tests run only with a label.
Only 57 bugs (9%) were a logic error inside one function. The tests checked what the author intended, not what they didn't think of; in 37 PRs the test asserted the buggy behavior. Fixes repeat the pattern: secret masking was fixed four times (#4788, #5227, #6155, #6259), each with a test for one symptom. A property test, "masking changes only secret values, for every event type", would have covered the last three.
One logistic regression over the 988 code PRs merged before July 2026, with month controls and errors clustered by author. A cause also had to predict later fixes more than later ordinary edits to the same lines, to rule out code that is merely busy. Odds ratios count all fix commits; in brackets, confirmed defect fixes only.
| Suspected cause | What the data says | Verdict |
|---|---|---|
| Changing code that already needed fixes | 1.20× odds per doubling of past fixes on the touched files, CI 1.07–1.35 (1.29×). | Strong |
| The PR is itself a fix | 1.62×, CI 1.08–2.44 (1.87×). Read in full, about 1 fix in 10 needed a real follow-up. | Supported |
| No cross-package or integration test | PRs that add one: 0.61×, CI 0.39–0.94 (0.52×). | Supported |
| Skipping a file that usually changes alongside | 1.37×, p = 0.058 (1.33×, p = 0.12). | Suggestive |
| Large PRs | 1.42× per doubling of size, but not worse per line, and size predicts ordinary edits just as much. | Predicts, not a cause |
| Agent co-authorship | 1.34×, CI 0.94–1.90, p = 0.10 (1.23×). | Inconclusive |
| Feature PRs, multi-package PRs | Not significant once size is held constant. | Not supported |
| No tests, mock-heavy tests, low coverage | Changing tests 1.11× (CI 0.68–1.82); coverage ranks buggy PRs no better than chance (AUC 0.50). | No effect found |
| Complex functions | Nothing beyond function length (AUC 0.735 with length, 0.734 adding complexity). | No effect |
To flag PRs when they open, size is the best single signal (AUC 0.72). "Over 300 lines or 2+ packages" flags 23% of PRs and catches 42% of the buggy ones.
Measured on the 423 bugs with a confirmed introducing PR, after reading 45 of them at random. Bugs enter at the design level, mostly through a few large PRs. How a PR is reviewed or merged made no measurable difference.
| Pattern | Example | Evidence |
|---|---|---|
| A cross-cutting PR creates a family of bugs | Deferred-init mode (#3287) gated only /api, not /sockets or /v1; VSCode and the trajectory route kept launch-time values. 6 dropped-setting bugs from one gap. | 36 PRs carry 175 of 423 bugs; same-cause pairs 2× chance. |
| Old code from the original port | The agent-server port (#317): 13 bugs. | 28% of bugs from 18% of PRs; median 170 days to fix vs 43. Older code has had longer to show bugs. |
| Objects assumed per conversation are shared | agent.llm mutated in place (#2907); one condenser shared by all sub-agents; process-wide litellm.modify_params. | About 30 bugs, by keyword. |
| A central change applied to everything | Masking rewrote base64 images and timestamps (#4788, #5182); truncation hit every text block (#124). | Recurs in the sample; not counted. |
| Silent defaults and fallbacks | A bare httpx.Client() fell back to a 5-second timeout (#735); a route kept enable_browser=True (#5151). | 1.38× odds per e-fold of fallback lines (p = 0.008); not significant after correcting for 16 tests. |
| Replacing an implementation | A lookup started raising KeyError, turning 404s into 500s (#610). | Weak: 10% of introducing PRs vs 8% of all PRs. |
No effect, with size, type, past fixes and month held constant: reviews, comments, time open, weekend merges, merging within 48 hours of a release, linking an issue, other PRs editing the same files meanwhile, adding async code, adding type: ignore or Any.
Share of the 602 bugs each strategy could have caught before merge, judged from the case labels. These are upper bounds: they assume the strategy would have thought of the right case.
| Strategy | Counts a bug when |
|---|---|
| Tests written by a second agent from the issue | The tests checked only the happy path, asserted the bug, did not exist, or the behavior was never specified; and a unit, contract or property test would have caught it. |
| Mutation testing on the diff | The introducing PR had tests that checked only the happy path or asserted the bug; a unit or contract test would have caught it; it was not a case the feature never handled. |
| Pre-merge agent audit | Wrong status codes or unchecked input, or found by an agent audit and catchable by a unit or contract test. |
| Real agent-server and cross-package tests | The cheapest catching test was a cross-package test, or the tests mocked the boundary or sat on the wrong layer. |
| Invariant tests where code was fixed before | The introducing PR was itself a fix. |
| Wrapped external dependencies | The root cause was an external dependency, unless no practical test exists. |
| Human review | The case review judged an attentive reviewer could clearly have caught it. |
| Concurrency and lifecycle stress tests | The root cause was concurrency or lifecycle. |
| Consumer contract tests | The root cause was a contract between components, or a downstream product found it and a contract or cross-package test would have caught it. |
| One settings resolver and parity tests | The root cause was a setting dropped on one code path. |
Combined, tests from the issue reach 70% of bugs; adding real agent-server tests reaches 87%, and wrapped dependencies 94%. The pre-merge audit reaches 24% of bugs but only 2% of escaped ones; human review reaches 15%.
| Change | What it means | Evidence |
|---|---|---|
| A second agent writes the tests | On PRs over 300 lines or 2+ packages, a session that sees only the issue and public API writes tests that must fail on main. | 196 happy-path misses, 37 tests asserted the bug; reaches 70% of bugs. |
| Test against the real agent-server | The real FastAPI app instead of mocked responses; one suite run against local and remote conversations (map B, C). | Cross-package tests 0.61×; 31 tests mocked the boundary that broke. |
| Put downstream expectations in SDK CI | Behavior Canvas, Cloud and automations rely on becomes a test here (map E). | 178 of 602 bugs found by downstream products. |
| One implementation per behavior | Resolve settings once for every launch path; a PR that adds a mode lists every route and client it must reach, and tests them. | 70 dropped-setting bugs; 6 from #3287 alone. |
| Wrap external dependencies | Upper bounds and adapter tests for LiteLLM and fastmcp; a nightly install without the lockfile (map D, G). | 93 bugs (15%), 50 in the LLM layer. |
| Fix the rule and the family | On a repeat fix, test the rule, not the symptom. After the first bug from a PR, check the rest of that PR for the same gap. | Past fixes 1.20× per doubling; same-cause pairs 2× chance. |
| Keep conversation state per conversation | Freeze or copy the LLM and agent when a conversation starts; test two conversations at once. | About 30 shared-object bugs. |
Won't help: more review rounds (PRs that later needed a fix had more reviews), a blanket "add tests" rule, or a coverage target. Track: PRs with a confirmed fix within 120 days (24% today), bugs found downstream or by users (46%), and median days from introduction to fix (64).
Published methods aimed at the same causes. The SDK uses pyright and FastAPI's test client, but none of these. Counts marked ≈ come from keyword searches of the case explanations.
| Method | What it does here, and the evidence | Targets |
|---|---|---|
| Fault injection | Rerun tests with each LLM, MCP and agent-server call failing in turn: timeout, 429, 5xx, malformed response. (Filibuster, SoCC 2021; most catastrophic failures were mishandled errors, OSDI 2014.) | ≈123 bugs |
| Stateful property tests | Hypothesis drives start, run, pause, resume and restart in random order and checks invariants after each step. (Each property test killed about 50× more mutants than a unit test, OOPSLA 2025.) | 124 bugs |
| Differential testing | Run one scenario through local vs remote, profile vs inline, run vs arun, and locked vs newest LiteLLM; diff the outputs. (Twitter Diffy.) | 163 bugs |
| Tests under real configurations | Run existing tests under the profiles and settings Canvas, Cloud and automations use. (ctest, OSDI 2020.) | 70 bugs |
| Upgrade testing | Start the new release on data written by the previous one; run an old client against a new server. (SOSP 2021: 123 real upgrade failures studied.) | ≈52 bugs |
| API fuzzing | Schemathesis generates requests from the agent-server's OpenAPI schema and flags 5xx and off-schema responses. (ICSE 2022.) Only 6 of its targets escaped. | 37 bugs |
| Mutation testing on the diff | Surviving mutants on changed lines go to the test agent to kill. (Google, ICSE-SEIP 2018; Meta ACH, FSE 2025: 73% of its tests accepted.) | 182 bugs |
main at 46b1fa5 (28 Oct 2026): 2,732 squash commits, each one PR; metadata for 2,762 merged PRs and 742 issues labeled bug from the GitHub API.fix, hotfix or bugfix; 664 change Python source in the four packages (tests excluded).git blame -w on the parent commit for every line the fix removed or changed. 523 fixes change existing lines and point to 601 introducing commits; 141 only add lines and cannot be traced this way.bug issue with a linked fix PR that changed package code (275), plus the 390 fix commits without an issue, of which 327 were real defects (63 refactors, features, docs or dependency bumps were dropped). Agents read each issue, fix and introducing PR and filled one rubric: root cause, who found it, the introducing PR (confirmed for 423), whether it had tests, why they missed, and the cheapest test that would have caught it. Blind re-coding of 24 cases agreed with κ = 0.86 on root cause, 0.84 on why missed and 0.82 on who found it.Co-authored-by trailer, test, mock and real-server lines added, tests under tests/cross or tests/integration, and usually co-changed files left out. Section 05 adds 16 factors from PR metadata and diffs (980 PRs with metadata).main.