acceptodds
Under review as a conference paper at ICLR 2027

SWE-Residual: Evaluating Starting-State Robustness in Coding Agents

Abstract

Software engineering (SWE) benchmarks are the primary way to measure the capabilities of coding agents: an agent is placed in a code repository or machine environment, given a request, and judged by hidden tests. These benchmarks almost always start the agent from a clean state that contains no earlier work on the task. In practice, however, agents often continue work that a developer or another agent has already done and reported as complete, with no indication of what remains incorrect or missing. We introduce SWE-Residual, a benchmark for Starting-State Robustness: the ability to maintain performance on a task when the starting state already contains earlier work on that task. SWE-Residual contains 198 Residual States drawn from 111 tasks in DeepSWE and Terminal-Bench, where the earlier work modifies a code repository or a whole machine environment. Each Residual State is the state a predecessor agent left behind after submitting its work as complete, although the hidden tests still fail. The evaluated agent receives only the original request and this state, without the predecessor's trajectory or any sign that the attempt failed; because the request and hidden tests are unchanged, the starting state is the only difference from the source benchmark. Across five frontier agents, success drops by 20.6–53.5 percentage points on DeepSWE and 20.6–34.9 points on Terminal-Bench relative to a Clean Start, and agent rankings on DeepSWE reverse. Failed continuations show a pattern we call acceptance bias: agents treat the inherited work as the definition of what remains to be done instead of checking it against the request. On DeepSWE, in 58.9–88.2% of failed runs, they submit with exactly the same tests failing, although they pass most of these tests from a Clean Start. SWE-Residual thus exposes a capability gap that clean-start evaluation does not capture.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Coding Agents, Software Engineering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_brmDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

There currently lacks a benchmark that investigates the ability of agents to continue previous work (of either an agent or developer) that looks complete but is actually incorrect. SWE-Residual creates a benchmark (along with a framework for generating similar such evals from other existing completion-style benchmarks) in which predecessor agents (DeepSeek, Qwen, GLM) attempt the original DeepSWE and Terminal-Bench tasks, and only attempts that were submitted as complete but still fail the hidden tests are kept as starting states (residual states) for the evaluated agent, for a total of 198 states from 111 tasks. The evaluated agent is provided only the original request and inherited state without evidence of earlier attempts and their failures. Its performance is compared against the same agent’s performance starting from clean starting states with no prior work. The authors find significant performance drops in the former in five frontier agents (20.6 to 53.5 point drops on DeepSWE and 20.6 to 34.9 points on Terminal-Bench). Additionally, they find that rankings reverse on DeepSWE, and identify a prevalent failure mode (acceptance bias) associated with this drop.

Critical strengths

S1: The paper tackles a novel and salient problem with a clean and reusable experimental design. It is common to encounter incorrect code that looks complete in practice, and purely completion-style benchmarks could not measure them. Instead, the pair design in the paper creates a controlled comparison between the clean starting state and residual state, allowing for precise measurement of the Residual Gap. This methodology could be applied to any completion-style benchmark with hidden tests, making it particularly future-proof.

S2: The paper reveals a large performance drop across evaluated agents (20.6 to 53.5 points). This drop exists regardless of the predecessor model (19.0 to 51.8 points). For the three analyzed agents, most tasks solvable from a clean start fail when starting from a residual state (61.9 to 81.3%). Furthermore, the paper characterizes this phenomena as acceptance bias, finding that agents often open but seldom change files left out by the predecessor.

Critical weaknesses

W1: The paper would benefit from a diagnostic baseline that determines whether documented failures are due to an inability to notice problems, a tendency to trust prior work, or lack of capability to fix them. My recommendation is to run a baseline, maybe on a subset, where the agent is prompted along the lines of “the prior work may be flawed,” alongside a control set of correct inherited states. Given that acceptance bias is a main claim for the origins of these errors, understanding how mitigatable it is would be helpful.

W2: The paper’s rigor would benefit from more statistical support for headline claims. Figure 2 could include denominators and confidence intervals. The ranking reversal phenomenon should be statistically supported (potentially via bootstrap confidence intervals) if the authors decide to mention it.

Questions for authors

Q1: What was the degree of human review agreement in stage III? What was the criterion by which candidate options are accepted or rejected? Would it be possible to clarify the feasibility or lack thereof of conducting Stage III entirely with a LLM judge?

Q2: To what extent does it matter whether the evaluated agent is of the same family as predecessors? A quick analysis could be helpful in understanding whether this matters and whether it could introduce confounds.

Q3: How should the different benchmarks be pooled by default? As figure 1 averages across benchmarks while table 8 averages across tasks.

3 comments
AuthorsOpenReview ↗

Rebuttal(1/2)

We thank the reviewer for the thoughtful and constructive feedback. Our detailed responses are provided below.

W1: Add diagnostic experiments to distinguish acceptance bias, failure to notice problems, and inability to fix them. Evaluate how much acceptance bias can be mitigated through explicit warnings and correct-state controls.

We added a 2×2 diagnostic experiment to test whether the failures come from acceptance bias, and how far a simple warning mitigates it. We vary two factors: the inherited state (a Residual State or a correct inherited state) and the prompt (Blind: the original request only; May Be Flawed: the request plus one sentence). The warning does not say that the earlier attempt failed or where the problems are. Clean Start on the same tasks is the reference.

Correct inherited states. For each task, we rerun the same predecessor agent from the clean repository and keep runs that are submitted as complete and pass all hidden tests, reconstructed in the same way as Residual States (Stage IV). We only include tasks that have both a Residual State and a correct inherited state, so all conditions are compared on the same tasks. This subset is easier than the full track, because every task in it is solved by the predecessor agent in at least one run. Clean Start success is therefore higher than in Table 3.

Inherited statePromptGLM-5.3-FlashKimi-K2.8-Preview
Clean StartOriginal request89.283.4
Residual StateBlind25.623.1
Residual StateMay Be Flawed53.447.8
Correct inherited stateBlind97.396.4
Correct inherited stateMay Be Flawed95.494.1

On correct inherited states, success means the final repository still passes all hidden tests. On Residual States, the warning raises success from 25.6% to 53.4% for GLM-5.3-Flash (+27.8 points) and from 23.1% to 47.8% for Kimi-K2.8-Preview (+24.7 points). The warning gives no information about the defects and adds no ability, so these gains do not come from an inability to notice or to fix the problems: the same agents found and fixed the defects once warned. The tasks are also within the agents' ability: Clean Start success on the same tasks is 89.2% and 83.4%. On correct inherited states, the same warning lowers success only from 97.3% to 95.4% and from 96.4% to 94.1%.

Together, these results show that acceptance bias causes a large part of the gap and that a simple prompt partly mitigates it.

W2: Strengthen statistical support by adding sample sizes and confidence intervals to Figure 2, and validating ranking reversals through bootstrap analysis.

We have strengthened the statistical support for Figure 2 and the ranking reversal claim by adding sample counts, 95% confidence intervals, and a paired bootstrap analysis.

  1. Figure 2: Sample counts and confidence intervals.

We now report the number of tasks and residual states used for each result in Figure 2. Since each task may have multiple Residual States, we first average the outcomes within each task and then average across tasks, giving each task equal weight. We therefore use task-level cluster bootstrap rather than treating individual Residual States as independent samples. Specifically, we resample tasks with replacement, keep all Residual States associated with each sampled task, and repeat this process 10,000 times to obtain 95% confidence intervals.

BenchmarkAgentFailure Rate (%)Sample Size (Tasks / States)95% CI
DeepSWEGemini-3.8-Flash75.465 / 119[64.1–85.7]
DeepSWEQwen-3.8-Flash63.446 / 84[49.6–76.4]
DeepSWEGLM-5.3-Flash79.062 / 114[68.4–88.2]
Terminal-BenchGemini-3.8-Flash80.010 / 16[50.5–98.3]
Terminal-BenchQwen-3.8-Flash61.97 / 11[28.2–90.4]
Terminal-BenchGLM-5.3-Flash81.38 / 13[56.2–99.1]
  1. Ranking reversal: Paired bootstrap analysis.

We further test the ranking reversal between GLM-5.3-Flash and Qwen-3.8-Flash on DeepSWE. We use a paired task-level bootstrap over the 90 tasks, keeping the Clean Start and Residual Start outcomes of both models together in each resampled task. This preserves the pairing between models and starting conditions. The results are:

ComparisonPerformance Difference95% Bootstrap CI
Clean Start: GLM - Qwen+17.8 pp[+8.0, +27.6]
Residual Start: Qwen - GLM+15.2 pp[+4.1, +26.3]

Both confidence intervals exclude zero. This provides statistical support for the observed ranking reversal: GLM performs significantly better than Qwen under Clean Start, while Qwen performs significantly better under Residual Start.

We have added the sample counts and confidence intervals to Figure 2 and included the bootstrap results in Section 5.2. We will also report task-level bootstrap 95% confidence intervals for the success rates and Residual Gaps in Table 3.

AuthorsOpenReview ↗

Rebuttal(2/2)

Q1: Clarify human-review agreement, acceptance criteria, and the feasibility of using an LLM judge in Stage III.

  1. Human review agreement and acceptance criteria.

Stage III uses LLM-assisted human review. GLM-5.3 is used only to summarize each predecessor trajectory, it does not make any inclusion decisions. Two human reviewers independently assess each candidate using the original request, the changes from the clean state, and the trajectory summary. The two reviewers agreed on 207 out of 208 candidates, giving a raw agreement of 99.5%. There was only one disagreement. After discussing the candidate, both reviewers agreed to exclude it.

A candidate is retained only if it satisfies both of the following criteria:

  • Task-directed: The predecessor's actions reflect a coherent attempt to satisfy the original requirements, rather than unrelated or incidental changes.
  • Task-verifier consistent: The original request and hidden tests specify consistent requirements for task completion.

Thus, both criteria must be met for a candidate to be retained.

Of the 207 candidates that passed human review, 9 were subsequently excluded during state reconstruction and validation, leaving 198 validated Residual States in the final benchmark. We have clarified this distinction in the revised manuscript.

  1. Feasibility of an LLM-only judge.

An LLM-only review is technically feasible in principle, but we have not established its reliability. In particular, checking whether hidden tests are consistent with the original request requires more than summarizing an execution trajectory. An LLM may overlook subtle mismatches between the requested behavior and what the tests actually verify.

We therefore use GLM-5.3 to reduce the effort required to inspect long trajectories, while keeping final inclusion decisions with human reviewers. We do not rule out fully automated review, but it would require a separate evaluation against human judgments before replacing this process.

Q2: Analyze whether model-family similarity between evaluated agents and predecessors affects performance and introduces potential confounds.

We performed a subgroup analysis using our existing results, grouping Residual States by predecessor family and comparing all evaluated agents within each group. For each group, we use the matched Clean Start results on the same tasks to account for differences in task difficulty.

A family-level match is available for two predecessor groups: Qwen-3.8-Flash evaluated on states produced by Qwen-3.8-27B, and GLM-5.3-Flash evaluated on states produced by GLM-5.3. These are models from the same families, but not identical checkpoints. Our evaluated set does not include a DeepSeek-family model. The results show no consistent advantage for same-family successors:

  • Qwen: Qwen-3.8-Flash has a Residual Gap of 19.3 points on Qwen-generated states, compared with 27.8–51.8 points for cross-family agents. This suggests a possible family-specific advantage.
  • GLM: GLM-5.3-Flash has a Residual Gap of 30.5 points on GLM-generated states, which is within the range of cross-family agents (27.8–38.8 points). Its Residual Start success rate is also the lowest in this group, at 2.8%.

Importantly, all five evaluated agents perform worse under Residual Start than under their matched Clean Start reference in every predecessor group. Thus, the performance drop persists regardless of predecessor family.

These results suggest that family similarity may benefit individual model comparisons, but does not consistently improve continuation performance or explain the overall performance degradation. We have added this analysis and clarified its scope in Appendix E.1.

Q3: Clarify the default aggregation strategy for pooling results across different benchmarks.

We report the two benchmarks separately in our main results, since DeepSWE and Terminal-Bench evaluate agents in two distinct execution settings. When a single cross-benchmark summary is needed, we use an equal-weighted average of the two benchmark scores, giving each benchmark 50% weight. Within each benchmark, tasks are weighted equally, with multiple Residual States averaged within each task.

Figure 1 follows this rule and uses a 50/50 average of the two benchmark scores to provide a compact summary.

Table 8 serves a different purpose: it analyzes predecessor-family effects over the combined task set. For this analysis, we give each source task equal weight, averaging the relevant Residual States within each task before aggregating across tasks. This is a task-level aggregation rather than a benchmark-level aggregation, and is intended for the predecessor-family subgroup analysis rather than as the default cross-benchmark summary.

We have clarified these aggregation principles in the revised manuscript.

I would like to thank the authors for taking the time to address the weaknesses and questions I raised. The 2x2 diagnostic experiment largely addresses my concerns about the source of model failures on inherited states. The bootstrap CIs also addressed my concerns about statistical significance; it would be good to revise the reported numbers in section 5.3 (i.e., 60% on both tracks). The performance drop regardless of predecessor family is a reassuring finding. I would like to raise my soundness rating to a 3.

Reviewer_KXZfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SWE-Residual, a benchmark that evaluates coding agents when they inherit an apparently completed but incorrect implementation from another agent, rather than starting from scratch. It contains 198 residual states from 111 tasks across DeepSWE and Terminal-Bench.

Critical strengths

Relevant problem with strong empirical findings. I find Agent handoffs are practically important, and the substantial performance drops and ranking reversals was surprising to me and suggest a limitation of conventional coding benchmarks. Furthermore, the authors performed a rigorous and detailed evaluation of multiple frontier agents across two environments and provide detailed analysis beyond aggregate success rates.

Critical weaknesses

Novelty is questionable: For me, the main concern I have about this paper is novelty, particularly relative to Recovery-Bench and Handoff Debt. I read what the authors wrote about it in related work section. But in my understanding, Recovery-Bench already studies agents recovering from environments modified by failed predecessor attempts. In fact, already evaluates agents on environments modified by failed predecessors, including an Env Replay Only setting (Section 4.2, Figure 2), where agents receive no previous trajectory. It also demonstrates performance degradation and ranking changes relative to clean starts. However, the recovery agent is explicitly informed that the previous attempt failed. Similarly, Handoff Debt already evaluates repository-only handoffs without predecessor history. Thus, in my perspective, the main distinction of SWE-Residual seems to be selecting final states that predecessors explicitly submitted as complete despite failing hidden tests, and evaluating them without revealing the failure. While I think this is a potentially interesting distinction, the paper does not establish that these specific differences lead to a distinct failure mode beyond what existing recovery and handoff benchmarks have already demonstrated. I am therefore unsure whether this incremental contribution is sufficiently novel for ICLR.

The acceptance-bias explanation is not convincingly established: My second minor concern is that retaining inherited code and failing to repair it could reflect general debugging difficulty, anchoring, or lack of failure awareness. The paper provides mostly correlational evidence without experiments isolating the proposed mechanism.

Questions for authors

1- Recovery-Bench and Handoff Debt already study continuation from modified states. Beyond using submitted-but-failing final states, what fundamentally new evaluation question does SWE-Residual answer?

2- Can you compare against intermediate handoff checkpoints, ideally from the same predecessor trajectories, to demonstrate that apparently completed but incorrect states produce a distinct failure mode?

3- How do you distinguish acceptance bias from the general difficulty of understanding and repairing another agent's implementation?

4- What happens if the agent is instructed to discard the inherited changes and solve the task from scratch, using the same evaluation budget?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for the thoughtful and constructive feedback. Our detailed responses are provided below.

W1&Q1: Clarify what fundamentally new evaluation question SWE-Residual addresses beyond Recovery-Bench and Handoff Debt

SWE-Residual studies a different question from Recovery-Bench and Handoff Debt. In real use, agents often pick up work that was reported as done, and nobody tells them whether it still contains errors. SWE-Residual asks whether an agent in this situation can still complete the task, finding and fixing on its own whatever the earlier work got wrong or left out. Neither Recovery-Bench nor Handoff Debt asks this.

Specifically, the three benchmarks ask different questions:

  • Recovery-Bench: Can an agent recover from a known failed attempt? Its prompt informs the agent that the previous attempt failed [1], so the need for further work is given.
  • Handoff Debt: What information should a handoff pass on so that a successor can resume interrupted work cheaply? This is a different line of work. It studies handoff design: it varies what the predecessor passes on (repository only, raw trace, or notes) and measures the successor's effort. The work is interrupted mid-task and never submitted as complete, so it does not test whether an agent can recognize and repair flawed work that was presented as finished.
  • SWE-Residual: Can an agent identify and close the remaining gaps in work that appears complete? Every state was submitted as complete, the verdict is withheld, and on DeepSWE the median state already passes 98.4% of the hidden tests.

Only in SWE-Residual must the agent decide on its own whether work that looks finished is actually finished. This leads to a different kind of failure, as shown in Table 4. Across the five agents, 58.9–88.2% of failed continuations end with exactly the same tests failing as at the start. The agents also rarely touch the files where work is still missing: files that the reference solution changes but the predecessor never touched are edited in 40.5–47.9% of Clean Start runs, but in only 6.7–15.0% of Residual Start runs. For most agents, the dominant failure pattern is not unsuccessful repair, but failure to initiate independent verification in the first place.

To test the difference from Recovery-Bench directly, we additionally ran a prompt ablation on SWE-Residual (DeepSWE), comparing three prompts on the same Residual States:

  1. Blind: the original task prompt (the setting used in the paper).
  2. Requirement Audit: asks the agent to check the inherited work against the original requirements, without revealing any failure.
  3. Failure-Aware: the same audit instruction, plus a statement that the predecessor's submission failed, without saying what failed. This provides the same key failure signal that Recovery-Bench gives its agents.
SettingGLM-5.3-FlashKimi-K2.8-Preview
Clean Start68.957.8
Blind (Residual Start)15.413.0
Requirement Audit32.427.1
Failure-Aware54.846.4

Most of the drop comes from the agent not looking for errors. Once the agent is told that the predecessor failed, success rises from 15.4% to 54.8% (GLM-5.3-Flash) and from 13.0% to 46.4% (Kimi-K2.8-Preview), which recovers about three quarters of the drop from Clean Start. The agent gets no hint about where the error is, yet it finds and fixes it far more often. This suggests that many errors are within the agents' debugging capabilities once they are prompted to look for them.

By default, the agent does not look for one: it treats the inherited work as correct. Simply asking it to check helps only partly (32.4% and 27.1%), which suggests the agent searches much harder when it knows an error exists. Recovery-Bench cannot measure this behavior, because it tells the agent up front that the previous attempt failed.

We will add this ablation to the paper.

[1] https://github.com/letta-ai/recovery-bench/blob/c5f83f2ba4f882a9b544c7bf0fa9be1bc3859c78/recovery_bench/prompts.py#L22-L26

AuthorsOpenReview ↗

Rebuttal (2/2)

W2&Q3: Distinguishing acceptance bias from general debugging difficulty

The key distinction is whether the agent independently verifies the inherited implementation.

  • Under debugging difficulty, the agent attempts to identify and fix errors but fails.
  • Under acceptance bias, the agent assumes the inherited work is correct and does not adequately verify it against the original requirements.

Our prompt ablation experiment, presented in response to W1 and Q1, directly examines this distinction. With the task and repository state unchanged, the Requirement Audit prompt explicitly instructs the agent to verify the implementation against the original requirements, improving success from 15.4% to 32.4% (GLM-5.3-Flash) and from 13.0% to 27.1% (Kimi-K2.8-Preview). The Failure-Aware prompt adds only the information that the previous attempt failed, without revealing what or where the errors are. Success further increases to 54.8% and 46.4%, recovering approximately 74% of the performance gap to Clean Start.

These results suggest that many inherited errors are within the agents' debugging capabilities once they are prompted to question the correctness of prior work. The substantially greater improvement under Failure-Aware also indicates that an explicit verification instruction alone does not fully overcome the tendency to trust inherited implementations.

Behavioral evidence further supports this interpretation. Across failed continuations, 58.9–88.2% end with exactly the same failing tests as at the start. Moreover, agents modify files omitted by the predecessor in only 6.7–15.0% of Residual Start runs, compared with 40.5–47.9% under Clean Start. These patterns are consistent with insufficient independent verification, rather than debugging difficulty alone.

Together, these findings identify the failure to independently question inherited correctness as a major source of performance degradation: agents do not initiate the necessary verification in the first place. We will incorporate this ablation into Section 5.4.

Q2: Compare completed-but-failing states with intermediate checkpoints.

  1. Setup.

For each of the 165 DeepSWE Residual States, we replay the same predecessor trajectory and extract an intermediate state at 50% of edit progress, before the predecessor declares the task complete. We evaluate GLM-5.3-Flash and Kimi-K2.8-Preview from these states under the same blind-continuation protocol, using identical requests, hidden tests, harnesses, and budgets. We follow the metrics in Tables 3 and 4, defining left-out files based on the final Residual State across all conditions.

  1. Results
AgentStartSuccess (%)Left-out files (%)Steps (× Clean)
GLM-5.3-FlashClean68.940.5–
GLM-5.3-FlashIntermediate47.231.60.87
GLM-5.3-FlashResidual15.46.70.57
Kimi-K2.8-PreviewClean57.845.7–
Kimi-K2.8-PreviewIntermediate41.837.20.89
Kimi-K2.8-PreviewResidual13.08.10.56

Agents succeed in 41.8% to 47.2% of runs from the intermediate checkpoint, against 13.0% to 15.4% from Residual States. The two kinds of state also change what agents do. On states where the reference solution changes files that the predecessor never touched, agents edit at least one such file in 31.6% to 37.2% of runs from the intermediate checkpoint. This is within 9 points of their Clean Start rate and more than four times their rate from Residual States (6.7% to 8.1%). They also take 0.87 to 0.89 times as many steps as from a Clean Start, against 0.56 to 0.57 from Residual States. From unfinished work, agents keep working on the parts of the request that the predecessor's work does not cover. From apparently finished work, they stop early and leave those parts untouched, which is the acceptance bias of Section 5.4. We will add this ablation to the paper.

Q4: Test whether discarding inherited changes improves performance under the same budget.

We ran this experiment on SWE-Residual (DeepSWE), where the inherited work is a set of uncommitted repository changes that git can discard. Each agent starts from the same Residual States with the same budget. The request is extended with an instruction to discard any changes from previous work, restore the repository to the task's base revision, and solve the task from scratch.

The results are in the table below. Once the agents discard the earlier work, their success rises sharply and comes close to their Clean Start rate.

AgentClean Start (%)Residual Start (%)Residual Start + discard instruction (%)
GLM-5.3-Flash68.915.464.8
Kimi-K2.8-Preview57.813.052.1

The instruction removes the need to decide which parts of the inherited work to trust, which is precisely what blind continuation evaluates. However, applying this rule to every handoff would also discard correct inherited work.

AuthorsOpenReview ↗

A Kindly Reminder

Dear Reviewer KXZf,

Thank you again for your thoughtful review and constructive feedback.

We have carefully addressed your concerns in our rebuttal and provided additional clarifications and experimental results.

As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our responses and let us know whether they have adequately addressed your concerns. If so, we would be grateful if you could consider updating your assessment accordingly.

We are also happy to discuss any remaining questions or concerns.

Thank you for your time and consideration.

Reviewer_JJQXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper presents SWE-Residual, a benchmark that tests whether a coding agent can finish a task when it starts from earlier work that was claimed as done but still fails the hidden tests. The states are built from failed-but-submitted runs of three open-weight models on DeepSWE and Terminal-Bench, giving 198 states from 111 tasks. The new agent only sees the original request and the leftover state. Across five frontier agents, success drops by about 20 to 50 points compared with starting clean, and the agent ranking on DeepSWE changes. The authors also describe "acceptance bias": agents tend to keep the earlier work as it is and often submit with the same tests still failing.

Critical strengths
  1. The problem is new and matters in practice. Handing work between sessions or between agents is very common now, and I have not seen a benchmark that tests the case where the earlier work was claimed as done but is actually broken. Table 1 does a good job of showing how this differs from Recovery-Bench and the handoff papers.
  2. The experiment design is simple and fair. The request, the tests, the harness and the budget stay the same, and only the starting state changes, so the drop can be read directly. The drop also shows up for every agent, on both tracks and for states from all three predecessors.
  3. I found the "left-out files" analysis in Section 5.4 the most convincing part. From a clean start, agents edit the files the earlier attempt missed in more than 40% of runs. From the residual state they only do it in 7–15% of runs, even though they often open those files. That is a good concrete sign that the earlier work narrows what the agent looks at.
Critical weaknesses
  1. I am not fully convinced that the signs of "acceptance bias" are really a bias. The median state already passes 98.4% of the tests, so keeping most of the earlier code and finishing in fewer steps could just be sensible behavior. The comparison between runs that keep more and runs that keep less is also mixed up with how good each state is to begin with. It would help to compare how much of the earlier code survives in a correct solution, or to compare runs on the same state.
  2. The paper does not test any change to the setup that could explain why the gap happens. For example, what if the agent is told the earlier attempt failed, or is simply asked to check the existing work against the request? If performance comes back, that supports acceptance bias. If it doesn't, the problem may be that the remaining bugs are just hard to find. Right now the mechanism is not clear, and the paper does not suggest how agents could be improved.
  3. The Terminal-Bench part is small, with only 21 tasks and 33 states, and all of the behavior analysis is done on DeepSWE only. So the claim that the effect holds on both tracks rests mostly on one table.
Questions for authors
  1. Why does Section 5.3 only include three of the five agents? GPT-6-Sol and Kimi are left out, and Kimi is also missing from the comparison of failed runs in Section 5.4. I could not find a reason for this.
  2. In Section 5.4, who decided whether an agent "named the defect" in the 30 sampled runs, and how was this done? Was there more than one annotator?
  3. For the human review in Stage III, how were disagreements between the two reviewers handled, and how often did they agree? Also, how many states were removed because the request and tests did not match, and could this filter remove some of the harder cases?
  4. Did you try telling the agent that the earlier attempt failed, or prompting it to check the existing work first? I think this result would make the paper much stronger.
  5. In the case study, the agent runs the package tests and they all pass. Were these tests written by the predecessor? If the earlier agent's own tests often give a false "all good" signal, that might explain part of the drop.
  6. The states here come from weaker open-weight models. It would be interesting to see states from stronger agents, from humans, or from the same agent continuing its own work.
4 comments
AuthorsOpenReview ↗

Clarification Regarding Review–Submission Alignment

Thank you for taking the time to review our submission.

We may be misunderstanding some of the comments, but we noticed that several major aspects of the review appear to describe a different work, SWE-CLIMB, rather than our submission, SWE-Residual. In particular, the review discusses its Exploration/Operation/Evolution tasks, GPT-based grading, and experiments that we cannot find in our paper.

Could you kindly clarify whether these comments were intended for our submission? If so, we would greatly appreciate any pointers to the relevant parts of our paper so that we can better understand and address your concerns.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for the thoughtful and constructive feedback. Our detailed responses are provided below. We have revised the manuscript accordingly and uploaded the updated version, with all changes highlighted in blue. Please kindly note that the original Sections 5.3 and 5.4 are now Sections 5.2 and 5.3 in the revised manuscript.

W1: Verify whether "acceptance bias" reflects genuine bias rather than sensible retention of already high-performing initial code, using controlled comparisons on identical starting states or against correct solutions.

We agree that preserving correct code and finishing in fewer steps can be entirely sensible. If an agent makes one small edit and satisfies the remaining requirement, that is successful reuse, not acceptance bias.

More generally, the correct way to continue is to first check the inherited work against the original request, keep what is correct, and complete what is missing. However, the agents skip this check: they accept the inherited work as correct and complete, and do not verify it against the request, even when it does not meet the requirements. As a result, they stop before the task is complete, although they are able to complete it. This is what we call acceptance bias.

To test this interpretation, we compared three prompts on the same Residual States. Blind uses the original task prompt as in our paper. Requirement Audit asks the agent to check the inherited implementation against the original requirements, without telling it whether anything is wrong. Failure-Aware adds one fact to that instruction: the predecessor’s submission failed. It gives no indication of which requirement was missed or how to fix it.

SettingGLM-5.3-FlashKimi-K2.8-Preview
Clean Start68.957.8
Blind (Residual Start)15.413.0
Requirement Audit32.427.1
Failure-Aware54.846.4

The most informative comparison is Blind versus Requirement Audit. The inherited code is identical, and the agent receives no failure feedback, yet asking it to check the implementation against the request roughly doubles task success for both models. Explicitly telling the agent that the prior submission failed raises success further, recovering 74% and 75% of the respective gaps between Blind and Clean Start. These results suggest that, under the original prompt, agents often settle on the inherited solution before adequately checking whether it fulfills the request.

We also compared code retention in the Blind and Requirement Audit runs from the experiment above. For each model, we paired runs starting from exactly the same residual state and measured the percentage of lines added or replaced by the predecessor that survived in each final submission. We averaged the paired results within each task and then across tasks.

Inherited lines retained (%)GLM-5.3-FlashKimi-K2.8-Preview
Blind94.897.1
Requirement Audit87.189.6

Both models retained most inherited code, but explicit requirement checks led to more revisions and higher success, supporting our interpretation that Blind agents tend to accept inherited implementations without sufficient verification.

We have clarified this distinction in Section 5.3 and added the same-state comparisons and code-retention analysis in Section 5.4 (Table 5).

W2&Q4: Test whether telling the agent that the earlier attempt failed, or asking it to check the existing work, recovers performance

Thank you for this suggestion. We tested both suggestions, as reported in our response to W1.

Prompting a requirements audit improved success for both models. Adding the predecessor’s failure verdict to that same audit prompt yielded a further improvement. Together, the audit instruction and failure disclosure recovered approximately three quarters of the observed Blind-to-Clean gap.

These results also suggest that many of the remaining bugs are within the agents' ability to find and fix once they look for them, which is consistent with the acceptance bias described in Section 5.3.

To explore mitigation, we ran a 2×2 experiment adding a one-sentence warning about potentially flawed prior work, evaluated on both Residual and correct inherited states. We use tasks with both state types, which have higher Clean Start success than Table 3 because each was solved by the predecessor at least once.

Inherited statePromptGLM-5.3-FlashKimi-K2.8-Preview
CleanOriginal request89.283.4
ResidualBlind25.623.1
ResidualMay Be Flawed53.447.8
Correct inherited stateBlind97.396.4
Correct inherited stateMay Be Flawed95.494.1

A one-sentence warning improves Residual State success by 27.8/24.7 points (GLM/Kimi), while reducing success on correct inherited states by only 1.9/2.3 points, demonstrating substantial recovery at minimal cost.

These results are added in Section 5.4 (Tables 5–6), with prompts in Appendix G.

AuthorsOpenReview ↗

Rebuttal (2/3)

W3: The Terminal-Bench track is small, and the behavior analysis uses DeepSWE only

Thank you for raising this. The Terminal-Bench track is smaller because we keep only runs that were submitted as complete, still failed, and passed all filters, and we did not relax these criteria to add states. The drop on this track is supported by two results, and we have newly computed one behavior measure for it.

  • The drop on Terminal-Bench. All five agents lose 20.6 to 34.9 points (Table 3), on the same tasks and with the same agent in both conditions. With Figure 2(b) now extended to all five agents, 52.2% to 81.3% of each agent's Clean Start successes fail from a Residual Start. Both results match DeepSWE.
  • Why the behavior analysis uses DeepSWE. The measures in Table 4 need a patch, a reference solution, and many per-test outcomes. On Terminal-Bench the inherited work is a machine state, and every state has only one or two failing tests, so the line-level and file-level measures are not defined there.
  • Behavior on Terminal-Bench. Step counts are available on both tracks, so we further computed the same step ratio as in Table 4 for Terminal-Bench. Continuations take 0.62 to 0.91 times as many steps as Clean Start runs, against 0.56 to 1.02 on DeepSWE. Normally submitted continuations used fewer steps than matched Clean Start runs on this track.

We have expanded the Terminal-Bench analysis in Sections 5.2, 5.3, and clarified the limitations of the behavior analysis in Appendix F.

Q1: Extend the analyses in Sections 5.2 and 5.3 to all five agents

Thank you for pointing this out. We have extended these analyses as follows.

  1. Section 5.2.

Residual Start failure rate (%) on the tasks each agent completes from a Clean Start:

AgentDeepSWETerminal-Bench
Gemini-3.8-Flash75.480.0
Qwen-3.8-Flash63.461.9
GLM-5.3-Flash79.081.3
GPT-6-Sol54.052.2
Kimi-K2.8-Preview80.457.4

For every agent on both tracks, more than half of its Clean Start successes do not carry over to a Residual Start.

  1. Section 5.3.

We sampled 30 failed continuations of Kimi-K2.8-Preview and annotated them with the same procedure. It names the defect in 1 of 30 runs, against 11 of 30 for GPT-6-Sol and at most 2 of 30 for the other three agents.

We also extended the median split by code retention to all five agents. The higher-retention group had lower task-weighted success for every agent, including GPT-6-Sol (18.8% in the higher-retention group vs. 43.5% in the lower-retention group) and Kimi-K2.8-Preview (3.5% vs. 16.9%)

We have updated Figure 2 and Section 5.2 to cover all five agents, and extended the acceptance-bias analysis in Section 5.3 accordingly.

Q2: Clarify who judged whether an agent "named the defect", how this was done, and whether there was more than one annotator

Thank you for the question. We used the same procedure as the human review in Stage III, with two annotators.

  • Sampling. For each agent, we randomly sampled 30 failed continuations.
  • Annotation. As in Stage III, GLM-5.3 summarized each trajectory, and two human reviewers independently judged each run. GLM-5.3 made no labeling decision.
  • Criterion. A run "names the defect" when the agent explicitly points out a specific problem in the inherited work and then tries to fix it.

We have added the annotation procedure and the criterion for identifying defects to Appendix D.3

Q3: Clarify how reviewer disagreements were handled, the agreement rate, and the effect of the task–verifier consistency filter

Thank you for these questions. The two human reviewers agreed on 207 of 208 candidates, and Stage III removed only one candidate in total, so the filter has very little effect on which cases enter the benchmark.

  • Agreement. Two reviewers independently assessed each of the 208 candidates and agreed on 207 of them (99.5% raw agreement).
  • Disagreements. There was one disagreement. The two reviewers discussed that candidate and both agreed to exclude it.
  • States removed for a request–test mismatch. One. This is the candidate above: it was excluded under the task–verifier consistency criterion, because its failing tests came from an environment issue in the verifier and not from anything the request asks for.
  • Effect on harder cases. The filter removes only cases where the verifier's verdict is not a reliable measure of the request. It does not consider how much work remains or how any agent performs on the state. The single removed candidate was a faulty case, not a hard one.

For completeness, 9 of the 207 retained candidates were later excluded during state reconstruction and validation, which gives the 198 Residual States in the benchmark.

We have clarified the human review process in Section 3.2 and the subsequent validation results in Appendix B.3 (Table 7).

AuthorsOpenReview ↗

Rebuttal (3/3)

Q5: In the case study, the agent runs the package tests and they all pass. Were these tests written by the predecessor? If the earlier agent's own tests often give a false "all good" signal, that might explain part of the drop.

Thank you for this helpful observation. In the onedump case, the package tests were indeed written by the predecessor agent. We agree that passing these tests may have given the successor a misleading signal that the task was complete.

We would like to clarify that acceptance bias is not limited to trusting the predecessor's implementation code; it also includes uncritically trusting the tests written by the predecessor. When a successor assumes that passing inherited tests means the original requirements are fully satisfied, without independently checking whether those tests provide sufficient coverage, this is also a form of acceptance bias.

To further investigate this pattern, we analyzed normally submitted failed continuations:

MetricShare of Failed Runs
Ran predecessor-written tests98.4%
All executed local tests passed50.5%
Submitted without further requirement verification after passing tests43.9%

Here, the metrics measure whether agents executed predecessor-written tests, whether their executed local tests passed despite hidden-test failures, and whether they submitted after passing local tests without further checking the original requirements, respectively.

These findings support the observation that passing inherited tests may contribute to premature completion. If the predecessor overlooks part of the original requirements, its tests may overlook the same part. The successor may then accept passing tests as sufficient evidence of task completion without independently verifying the original requirements. This highlights an important aspect of acceptance bias: agents should independently verify not only inherited implementations, but also whether inherited tests adequately cover the original requirements.

We have expanded the analysis of inherited tests in Section 5.3 and updated the onedump case study in Appendix E.2.

Q6: The states here come from weaker open-weight models. It would be interesting to see states from stronger agents, from humans, or from the same agent continuing its own work.

Thank you for this suggestion. We selected three open-weight models from different families to generate diverse, realistic, and reproducible Residual States. Our current evaluation already covers several settings: stronger agents continuing weaker agents' work, predecessor and successor models from the same family (Qwen and GLM), and a stronger predecessor followed by a smaller successor (GLM-5.3 → GLM-5.3-Flash).

We agree that extending the benchmark to include stronger SOTA agents and human developers as predecessors, as well as same-agent continuation, would be valuable. These settings would help us better understand how the source of inherited work affects the Residual Gap. SWE-Residual provides a reusable evaluation framework for studying these scenarios.

Due to time constraints, we could not complete these additional experiments during the rebuttal period. We plan to continue this investigation and include additional results in the camera-ready version. We have clarified predecessor selection in Section 3.2, reported comparisons by predecessor family in Section 5.2 and Appendix E.1, and discussed the remaining limitations in Appendix F.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.