acceptodds
Withdrawn from ICLR 2027

Terminal Self-Degradation: Code Agents End Worse Than They Reached

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Code agents are scored on the patch they finally submit. That score hides a failure we can only see by executing what came before it: in a failed trajectory, an earlier executable state can pass a strict superset of the tests the submitted patch passes. We call this terminal self-degradation (TSD). Its strongest form, solved-then-broke, is a trajectory that reached a state passing the entire suite and then submitted one that does not. To measure it we replay every reconstructable intermediate state of fresh full-benchmark SWE-bench Verified rollouts through the official harness. A scaffold is admitted only if its reconstructed states pass a submission-time parity check, and a bash-only scaffold that fails it is excluded. On our primary cell (two frontier GPT-5.6 models on SWE-Agent), at least 4.9% of all failures are TSD. Nine are solved-then-broke: a harness-certified resolving patch was on disk and something else was submitted, and in seven of the nine the agent had itself evaluated that state before editing it away. It is more pronounced on six open-weight model lines (5.7–14.6% of all failures, every one above the frontier floor), and also holds on a second scaffold and eight non-Python language groups, surviving triple re-execution. Degradation proceeds mostly through PASS_TO_PASS regressions that a stable or rising pass count masks. The agents' own tests never separated the lost state from the submitted one. Most primary-cell losses are clean un-fixes that differ from the final only on the held-out target, so a reference-free selector built on repository tests recovers almost none of them. We establish TSD's existence, generality, and structure, report lower bounds rather than prevalences, and argue that the gap between failing to generate a fix and failing to retain one is a diagnostic axis that final-patch scoring, and any signal derived from it, cannot currently report.

Primary area:
interpretability and explainable AI
Keywords:
code agents, trajectory analysis, evaluation, software engineering agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2tQ2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper deals with investigating the quality of agent-generated code over its whole generation duration until the provided final patch. It is indeed interesting to see that the final outcome of agent-based code generation might be wrong whereas in intermediate steps this is not the case. The paper is recent because it focuses on agentic programming, which is a still under-investigated but hot topic of software engineering.

The paper also comes with an initial experimental evaluation, which comprises some agents and some programs from two (maybe connected) repositories. This is fine for an initial evaluation but maybe not enough for a final analysis. Anyhow, the outcome of the paper itself is fine and the final conclusions are justified with its content. The practical implications, however, remain somehow vague. If an agent does not provide a passing outcome, the programming is not done. Even when knowing that there might be an outcome at an intermediate step that fulfills all test cases. When saying this, obviously for providers of agentic coding tools that might make a difference.

To improve the paper, the authors should be more precise when explaining concepts. Everything should be defined in sufficient detail. The experiments could also be improved considering more and more diverse benchmarks.

Critical strengths
  • The paper deals with a problem occurring when using agentic-coding tools
  • The paper comes with an initial experimental evaluation, showing the existence of the problem
  • There might be implications of the paper for agentic-coding tools provider
Critical weaknesses
  • The paper is hard to read and understand.
  • Some concepts and explanations need more precision and better (formal) definition
  • The experimental evaluation is only an initial one.
  • The consequences for practice could be discussed much more.
Questions for authors

There are no further questions.

Reviewer_NiRXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies cases where a coding agent reaches a better executable state but submits a worse one. Terminal self-degradation (TSD) requires an intermediate checkpoint to pass a strict superset of the tests passed by the final state. Solved-then-broke (STB) is the special case where the intermediate state passes every test and the final state does not. The authors replay intermediate states through the official harness, check reconstruction fidelity, exclude unreliable bash-only reconstruction, and identify a sphinx patch-extraction artifact in OpenHands. In the main SWE-Agent experiment, with two models and 1,000 runs, they detect TSD in 15 of 305 failures, including one single-checkpoint case. Nine cases are STB. Further experiments find the same phenomenon across models and languages. Existing tests often cannot distinguish the lost state from the final state, and a simple checkpoint selector recovers few cases.

Critical strengths
  1. The finding is supported by execution results and careful artifact checks. The authors run intermediate checkpoints through the official harness and remove unreliable reconstructions and patch-extraction artifacts. They also reproduce the phenomenon across models, scaffolds, and languages.

  2. The paper explains why recovery is difficult. In 12 of 15 artifact-free STB cases across the two main scaffolds, the lost and final states have the same P2P pass vector. This shows why existing test results may not identify the better checkpoint.

Critical weaknesses
  1. OpenHands intermediate states are not fully verified. Matching the final state does not rule out errors in earlier reconstructed states. The full bash-write audit covers only SOL. The reported OpenHands TSD cases need individual checks of tracked-file changes.

  2. Passing the harness does not prove the task was fully solved. The agent's own tests were still red at the winning state in six of nine primary STB cases. Independent review of those tests and patches is needed to distinguish a lost correct fix from a gap in the benchmark tests.

Questions for authors
  1. Could you audit tracked-file writes in every artifact-free OpenHands TSD case and report the counts after excluding unverified intermediate states? For sympy-22080, can the winning checkpoint be checked directly from the original runtime state or full logs?

  2. Could you independently review the winning and final patches for the nine primary STB cases? Please include the issue requirements, why the agent's own test was red, whether that test checked a valid requirement, and whether the final state improved any real part of the task. A structured review of at least the six red-at-winning cases would help establish what “losing a solution” means here.

  3. Detected-event lower bounds over all failures can be listed together, but different missing-data patterns and checkpoint visibility prevent them from ranking the true rates across systems. Could the main table consistently use confirmed artifact-free counts and separately list unassessed cases, single-checkpoint cases, and flaky positives? For checkpoint selection, please distinguish “the same P2P information cannot reliably separate these states” from “any method using repository tests cannot recover them.” The second claim also depends on whether additional tests, test outputs, or other tie-breaking signals are allowed.

Reviewer_aa6jOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies cases where a coding agent's earlier patch passes every benchmark test passed by its final patch, plus additional tests. The authors reconstruct intermediate patches from SWE-Agent and OpenHands runs and evaluate them with the benchmark's original eval harness. They call this behavior "terminal self-degradation." In the main SWE-Agent experiment, they find 15 such cases among 1,000 runs, including nine where an earlier patch passes all benchmark tests but the final patch fails.

Critical strengths
  • Terminal self-degradation is an interesting problem. Studying intermediate patches can reveal useful repairs that an agent later loses, which final pass/fail scores alone do not show.

  • Beyond identifying the problem, the paper examines over-editing, regressions missed by pass-to-pass tests, and disagreements between agent-written tests and benchmark tests. It also evaluates ways to recover earlier patches, rather than assuming that detecting the problem is enough to solve it.

Critical weaknesses
  • My main concern is the practical significance. Only 9/1,000 primary runs have an observed earlier patch that passes all benchmark tests while the final patch fails. And even then, perfectly selecting these intermediate patches is very heard. The tested selector using pass-to-pass tests recovers only one of the 15 genuine solved-then-broke cases. Also, the paper does not establish whether checkpoint selection is worth its execution cost compared with spending the same budget on other test-time methods.

  • The evaluation is not comprehensive and only tests on one of the SWE benchmark suites. Comparing short, local fixes with long, multi-file repairs across different software-repair benchmarks would help show whether terminal self-degradation is concentrated in particular settings. This would make the findings more useful for deciding which agent workflows need checkpoint recovery.

  • Checkpoint quality is measured only through benchmark tests. Passing more benchmark tests does not necessarily make the earlier patch a better solution to the issue. In 6 out of 9 primary solved-then-broke cases, the agent's own test was failing at the earlier patch. That test could be wrong, but it could also capture a valid requirement missing from the benchmark. The benchmark tests could themselves be incomplete or overly restrictive. Independent review of the issue, both patches, and the disputed tests is needed to distinguish a genuinely lost repair from a disagreement between test suites.

Questions for authors
  • For the six cases with failing self-tests at the earlier benchmark-passing patch, which tests reflect valid issue requirements? Could independent review show that the final patch is actually preferable in some of these cases?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.