A Passing Test Can Hide a Dying Repository

Most coding-agent demos end at the same cinematic moment: the test suite turns green, the patch is displayed, and the machine is invited to take a bow.

That moment is a checkpoint, not a career.

A test-passing patch proves something valuable. It does not prove that the agent can live inside a codebase. Real repositories do not reset after each victory. Requirements change. Old assumptions become traps. A fix in one subsystem quietly narrows the options in another. The next engineer inherits not just the behavior of the software, but the shape of the decisions that produced it.

That is the blind spot behind much of the current coding-agent spectacle. SWE-bench moved evaluation beyond toy functions by asking models to resolve real GitHub issues in real repositories. Its unit of progress is still an isolated issue and a generated patch judged primarily through tests. SWE-bench Verified improves the task set by filtering 500 instances for clearer descriptions, tests, and solvability. Both are useful. Neither asks what happens after the patch becomes the starting point for the next ten changes.

The missing variable is trajectory degradation.

SWE-EVO makes the time horizon longer. Its tasks are drawn from release notes of mature Python projects and require multi-step changes across a repository while preserving existing behavior. That design matters because preservation is harder than addition. An agent can satisfy a new request while quietly breaking the contracts nobody remembered to write down.

SlopCodeBench sharpens the point. It follows agents through repeated extensions under evolving specifications, with dozens of checkpoints rather than one final verdict. The reported failure mode is uncomfortable: an agent can continue passing local checkpoints while its code becomes more bloated and less maintainable. The monster is not always a red test. Sometimes it is a green test surrounded by an architecture that has begun to calcify.

This is why the next generation of evaluations needs to score the path, not only the destination. Long-Horizon-Terminal-Bench treats inspection, command execution, partial results, revision, and recovery from mistakes as part of the task. RE-Bench offers another useful idea: compare agents with human experts over realistic eight-hour attempts, rather than compressing open-ended work into a short contest.

Even these evaluations leave a hole. They do not yet give us a mature measure of reviewability: how much effort a human maintainer needs to understand a change, whether the explanation matches the code, or whether the patch leaves the system easier to modify. A benchmark can expose degradation without telling us the full price of cleaning it up.

That should change how we read coding scores. A benchmark result is evidence that an agent can navigate one carefully defined dungeon. It is not evidence that the agent can keep the kingdom standing after the map changes.

The serious question is no longer only, “Can it solve the issue?” It is, “After this solution, is the next issue easier or harder to solve?” Until evaluations measure that distance, a green suite remains a bright signal with a shadow behind it.

Sources

• SWE-bench, Jiménez et al., ICLR 2024.
• SWE-bench Verified, official benchmark-maintainer page.
• SWE-EVO, long-horizon software evolution evaluation.
• SlopCodeBench, iterative degradation evaluation.
• Long-Horizon-Terminal-Bench, terminal-task trajectory evaluation.
• RE-Bench, ICML 2025.