- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
There currently lacks a benchmark that investigates the ability of agents to continue previous work (of either an agent or developer) that looks complete but is actually incorrect. SWE-Residual creates a benchmark (along with a framework for generating similar such evals from other existing completion-style benchmarks) in which predecessor agents (DeepSeek, Qwen, GLM) attempt the original DeepSWE and Terminal-Bench tasks, and only attempts that were submitted as complete but still fail the hidden tests are kept as starting states (residual states) for the evaluated agent, for a total of 198 states from 111 tasks. The evaluated agent is provided only the original request and inherited state without evidence of earlier attempts and their failures. Its performance is compared against the same agent’s performance starting from clean starting states with no prior work. The authors find significant performance drops in the former in five frontier agents (20.6 to 53.5 point drops on DeepSWE and 20.6 to 34.9 points on Terminal-Bench). Additionally, they find that rankings reverse on DeepSWE, and identify a prevalent failure mode (acceptance bias) associated with this drop.
Critical strengths
S1: The paper tackles a novel and salient problem with a clean and reusable experimental design. It is common to encounter incorrect code that looks complete in practice, and purely completion-style benchmarks could not measure them. Instead, the pair design in the paper creates a controlled comparison between the clean starting state and residual state, allowing for precise measurement of the Residual Gap. This methodology could be applied to any completion-style benchmark with hidden tests, making it particularly future-proof.
S2: The paper reveals a large performance drop across evaluated agents (20.6 to 53.5 points). This drop exists regardless of the predecessor model (19.0 to 51.8 points). For the three analyzed agents, most tasks solvable from a clean start fail when starting from a residual state (61.9 to 81.3%). Furthermore, the paper characterizes this phenomena as acceptance bias, finding that agents often open but seldom change files left out by the predecessor.
Critical weaknesses
W1: The paper would benefit from a diagnostic baseline that determines whether documented failures are due to an inability to notice problems, a tendency to trust prior work, or lack of capability to fix them. My recommendation is to run a baseline, maybe on a subset, where the agent is prompted along the lines of “the prior work may be flawed,” alongside a control set of correct inherited states. Given that acceptance bias is a main claim for the origins of these errors, understanding how mitigatable it is would be helpful.
W2: The paper’s rigor would benefit from more statistical support for headline claims. Figure 2 could include denominators and confidence intervals. The ranking reversal phenomenon should be statistically supported (potentially via bootstrap confidence intervals) if the authors decide to mention it.
Questions for authors
Q1: What was the degree of human review agreement in stage III? What was the criterion by which candidate options are accepted or rejected? Would it be possible to clarify the feasibility or lack thereof of conducting Stage III entirely with a LLM judge?
Q2: To what extent does it matter whether the evaluated agent is of the same family as predecessors? A quick analysis could be helpful in understanding whether this matters and whether it could introduce confounds.
Q3: How should the different benchmarks be pooled by default? As figure 1 averages across benchmarks while table 8 averages across tasks.
3 comments
Rebuttal(2/2)
Q1: Clarify human-review agreement, acceptance criteria, and the feasibility of using an LLM judge in Stage III.
- Human review agreement and acceptance criteria.
Stage III uses LLM-assisted human review. GLM-5.3 is used only to summarize each predecessor trajectory, it does not make any inclusion decisions. Two human reviewers independently assess each candidate using the original request, the changes from the clean state, and the trajectory summary. The two reviewers agreed on 207 out of 208 candidates, giving a raw agreement of 99.5%. There was only one disagreement. After discussing the candidate, both reviewers agreed to exclude it.
A candidate is retained only if it satisfies both of the following criteria:
- Task-directed: The predecessor's actions reflect a coherent attempt to satisfy the original requirements, rather than unrelated or incidental changes.
- Task-verifier consistent: The original request and hidden tests specify consistent requirements for task completion.
Thus, both criteria must be met for a candidate to be retained.
Of the 207 candidates that passed human review, 9 were subsequently excluded during state reconstruction and validation, leaving 198 validated Residual States in the final benchmark. We have clarified this distinction in the revised manuscript.
- Feasibility of an LLM-only judge.
An LLM-only review is technically feasible in principle, but we have not established its reliability. In particular, checking whether hidden tests are consistent with the original request requires more than summarizing an execution trajectory. An LLM may overlook subtle mismatches between the requested behavior and what the tests actually verify.
We therefore use GLM-5.3 to reduce the effort required to inspect long trajectories, while keeping final inclusion decisions with human reviewers. We do not rule out fully automated review, but it would require a separate evaluation against human judgments before replacing this process.
Q2: Analyze whether model-family similarity between evaluated agents and predecessors affects performance and introduces potential confounds.
We performed a subgroup analysis using our existing results, grouping Residual States by predecessor family and comparing all evaluated agents within each group. For each group, we use the matched Clean Start results on the same tasks to account for differences in task difficulty.
A family-level match is available for two predecessor groups: Qwen-3.8-Flash evaluated on states produced by Qwen-3.8-27B, and GLM-5.3-Flash evaluated on states produced by GLM-5.3. These are models from the same families, but not identical checkpoints. Our evaluated set does not include a DeepSeek-family model. The results show no consistent advantage for same-family successors:
- Qwen: Qwen-3.8-Flash has a Residual Gap of 19.3 points on Qwen-generated states, compared with 27.8–51.8 points for cross-family agents. This suggests a possible family-specific advantage.
- GLM: GLM-5.3-Flash has a Residual Gap of 30.5 points on GLM-generated states, which is within the range of cross-family agents (27.8–38.8 points). Its Residual Start success rate is also the lowest in this group, at 2.8%.
Importantly, all five evaluated agents perform worse under Residual Start than under their matched Clean Start reference in every predecessor group. Thus, the performance drop persists regardless of predecessor family.
These results suggest that family similarity may benefit individual model comparisons, but does not consistently improve continuation performance or explain the overall performance degradation. We have added this analysis and clarified its scope in Appendix E.1.
Q3: Clarify the default aggregation strategy for pooling results across different benchmarks.
We report the two benchmarks separately in our main results, since DeepSWE and Terminal-Bench evaluate agents in two distinct execution settings. When a single cross-benchmark summary is needed, we use an equal-weighted average of the two benchmark scores, giving each benchmark 50% weight. Within each benchmark, tasks are weighted equally, with multiple Residual States averaged within each task.
Figure 1 follows this rule and uses a 50/50 average of the two benchmark scores to provide a compact summary.
Table 8 serves a different purpose: it analyzes predecessor-family effects over the combined task set. For this analysis, we give each source task equal weight, averaging the relevant Residual States within each task before aggregating across tasks. This is a task-level aggregation rather than a benchmark-level aggregation, and is intended for the predecessor-family subgroup analysis rather than as the default cross-benchmark summary.
We have clarified these aggregation principles in the revised manuscript.
I would like to thank the authors for taking the time to address the weaknesses and questions I raised. The 2x2 diagnostic experiment largely addresses my concerns about the source of model failures on inherited states. The bootstrap CIs also addressed my concerns about statistical significance; it would be good to revise the reported numbers in section 5.3 (i.e., 60% on both tracks). The performance drop regardless of predecessor family is a reassuring finding. I would like to raise my soundness rating to a 3.
Rebuttal(1/2)
We thank the reviewer for the thoughtful and constructive feedback. Our detailed responses are provided below.
We added a 2×2 diagnostic experiment to test whether the failures come from acceptance bias, and how far a simple warning mitigates it. We vary two factors: the inherited state (a Residual State or a correct inherited state) and the prompt (Blind: the original request only; May Be Flawed: the request plus one sentence). The warning does not say that the earlier attempt failed or where the problems are. Clean Start on the same tasks is the reference.
Correct inherited states. For each task, we rerun the same predecessor agent from the clean repository and keep runs that are submitted as complete and pass all hidden tests, reconstructed in the same way as Residual States (Stage IV). We only include tasks that have both a Residual State and a correct inherited state, so all conditions are compared on the same tasks. This subset is easier than the full track, because every task in it is solved by the predecessor agent in at least one run. Clean Start success is therefore higher than in Table 3.
On correct inherited states, success means the final repository still passes all hidden tests. On Residual States, the warning raises success from 25.6% to 53.4% for GLM-5.3-Flash (+27.8 points) and from 23.1% to 47.8% for Kimi-K2.8-Preview (+24.7 points). The warning gives no information about the defects and adds no ability, so these gains do not come from an inability to notice or to fix the problems: the same agents found and fixed the defects once warned. The tasks are also within the agents' ability: Clean Start success on the same tasks is 89.2% and 83.4%. On correct inherited states, the same warning lowers success only from 97.3% to 95.4% and from 96.4% to 94.1%.
Together, these results show that acceptance bias causes a large part of the gap and that a simple prompt partly mitigates it.
We have strengthened the statistical support for Figure 2 and the ranking reversal claim by adding sample counts, 95% confidence intervals, and a paired bootstrap analysis.
We now report the number of tasks and residual states used for each result in Figure 2. Since each task may have multiple Residual States, we first average the outcomes within each task and then average across tasks, giving each task equal weight. We therefore use task-level cluster bootstrap rather than treating individual Residual States as independent samples. Specifically, we resample tasks with replacement, keep all Residual States associated with each sampled task, and repeat this process 10,000 times to obtain 95% confidence intervals.
We further test the ranking reversal between GLM-5.3-Flash and Qwen-3.8-Flash on DeepSWE. We use a paired task-level bootstrap over the 90 tasks, keeping the Clean Start and Residual Start outcomes of both models together in each resampled task. This preserves the pairing between models and starting conditions. The results are:
Both confidence intervals exclude zero. This provides statistical support for the observed ranking reversal: GLM performs significantly better than Qwen under Clean Start, while Qwen performs significantly better under Residual Start.
We have added the sample counts and confidence intervals to Figure 2 and included the bootstrap results in Section 5.2. We will also report task-level bootstrap 95% confidence intervals for the success rates and Residual Gaps in Table 3.