- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
- This paper proposes a way to test the pre-implementation abilities of LLM coding agents rather than traditional benchmarks which test final implementation correctness by evaluating model patches. The setup is realistic with a user simulator which has the knowledge about withheld information about the task to assist the agent in any clarifications. The plan produced by the agent is evaluated with an agentic plan judge with task specific checklists.
- Tasks contain blockers to test clarification abilities of planning agents. Blockers are designed such that they ensure multiple plausible paths and lead to different final implementations without clarification.
- The evaluation captures both axes: clarifications posed to the user and planning evaluation through checklists. Both the evaluations for the plan and simulated user have a high agreement rate with human experts.
Critical strengths
- The benchmark evaluates the pre-implementation abilities of coding agents, clarification and planning, which are understudied in recent literature but very timely given that major coding tools now have a native plan mode. The design decisions are grounded in software engineering and requirements engineering literature. Running agents in the native Plan Mode of real harnesses also makes the evaluation realistic, so the results reflect how these tools are actually used.
- The paper makes a solid effort to validate its metrics. Both judges are checked against human labels and the execution study also shows the plan scores matter for implementation: adding a plan raises execution success by 8–12 points, and Plan Coverage predicts which plans lead to success better than plan length or an overall 1–10 rating does.
Critical weaknesses
- There seems to be limited headroom for frontier models, which already reach 89–93 Plan Coverage. Most of their remaining gap is in verification obligations, which look over-specific (appendix analysis case 1). I suspect this partly reflects the task source, since the tasks involve implementations of moderate size. Harder tasks, like those in existing benchmarks, might stress planning more. It would help to see a reference ceiling (e.g., expert-written plans) and some discussion of how the benchmark can stay challenging and continue to distinguish frontier models as they improve.
- The User Simulator design is a bit simple. It returns a stored answer when a question matches a withheld decision and the same template reply for everything else. From what I understand, it always returns the full stored answer even when the question covers only part of a decision. It also can’t handle confirmation-style or multiple-choice questions, or respond realistically to reasonable questions outside the registry. The paper doesn't tests how much this simplification affects the clarification scores. A more realistic simulator or a small study with real developers would make the interactive part of the benchmark much more aligned with real world usage.
Questions for authors
- I'm curious if you analyzed the types of information withheld across tasks? Do they cover a diverse range of decision types, e.g., failure behavior, scope, or interactions with existing options? It would be great to see a breakdown by type, along with how often agents recover each type through clarification. That would show which kinds of hidden information agents reliably ask about and which they tend to miss.
- The benchmark tasks currently have a cutoff of June 1, 2026. What are your thoughts on the saturation of the benchmark with new model releases in the future? How did this influence your choice to use SWE-Factory over sampling tasks from existing benchmarks which are less prone to contamination?
3 comments
Rebuttal to Reviewer YZy6 (2/2): Blocker Categories and Saturation
We continue below with your two questions.
Q1. Breakdown of Blocker Categories and Clarification Recall
Thank you for this valuable suggestion. Yes, withheld decisions span diverse functional requirement types. Building upon the blocker criteria in Sec 3.2, App. D, and the task instances in App. F, and informed by your three examples, we manually annotated all 929 blockers into four core decision categories (three from your examples and one that emerged during annotation) plus an Others category. Two authors independently audited 100 blockers with an inter-annotator agreement of Cohen's κ = 0.84. Blocker Recall below is pooled across blockers in each category over all 24 configurations (session-macro recall is 86.3% in Obs. 3):
| Decision category | Blockers | Share (%) | Mean Blocker Recall (%) | Range across 24 configurations (%) |
|---|---|---|---|---|
| Failure and error behavior | 186 | 20.0 | 89.2 | 71.5–97.8 |
| Scope and boundary conditions | 349 | 37.6 | 86.5 | 67.3–96.3 |
| Interaction and precedence across options | 32 | 3.4 | 78.1 | 56.3–90.6 |
| Output and interface contract | 135 | 14.5 | 90.4 | 74.1–98.5 |
| Others | 227 | 24.4 | 84.6 | 64.8–95.6 |
Agents most reliably clarify Output and interface contract decisions and most frequently miss Interaction and precedence across options decisions. We have added this category breakdown to Sec 6 (Obs. 3), the annotation taxonomy to App. B, the audit methodology to App. D, and the complete per-configuration recall table to App. M.4.
Q2. Benchmark Saturation and Contamination Control
Thank you for raising saturation and contamination. We expect SWE-Plan to stay informative as models improve because tasks can be rebuilt after later cutoffs. We chose SWE-Factory over existing benchmarks for two reasons. First, established benchmarks like SWE-bench predate current model training cutoffs and face documented pretraining memorization and contamination risks [1, 2]. Because blockers in SWE-Plan are decisions reconstructed from developer patches and tests (Sec 3.2), an agent that has memorized an older patch could bypass clarification altogether. Furthermore, existing benchmark issues can leak solutions [3], and evaluating clarification and planning would still require constructing new rewritten requests, blocker registries, and obligation checklists from scratch. Second, SWE-Factory let us pin every repository after June 1, 2026, placing patches and commits after all published cutoffs, while the newly written requests, registries, and checklists are not public before release (App. C.3).
To keep the benchmark challenging and uncontaminated as new models appear, we take two concrete measures:
- Continuous expansion via our construction pipeline. Following automated pipeline practices for refreshing live benchmarks [4, 5], our SWE-Factory-based construction pipeline can rebuild tasks from repository commits after newer cutoff dates, and its active use inside our company to evaluate new models provides sustained engineering resources for this rolling expansion.
- Versioned tracking with a private split. We commit to maintaining the benchmark across future versions, keeping newly added tasks as an internal private split, and publishing full results.
We would value any further suggestions on keeping SWE-Plan informative as models improve. We have added the task-source discussion to Sec 3.2 and App. C.3 and updated the Reproducibility Statement.
We hope these revisions strengthen the paper, and we are grateful for your time and constructive feedback. Thank you once again for your careful reading and expert suggestions. We hope our comments can address your concerns and look forward to your further thoughts.
References
[1] Liang et al. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason.
[2] Prathifkumar et al. Does SWE-Bench-Verified Test Agent Ability or Model Memory?
[3] Aleithan et al. SWE-Bench+: Enhanced Coding Benchmark for LLMs.
[4] Zhang et al. SWE-bench Goes Live!
[5] Badertdinov et al. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents.
Thank you for the clarifications. I will keep my score.
Rebuttal to Reviewer YZy6 (1/2): Expert Ceiling and User Simulator
We sincerely thank you for your careful and constructive review. We address your weaknesses first and your questions in the next comment. Revisions in the updated PDF are marked in blue.
W1. Expert Reference Ceiling and Verification Specificity
Thank you for suggesting an expert reference ceiling, which lets us locate where headroom remains. Plan Mode ends with a reviewable plan that the developer checks before implementation (
Sec 3.1), so we follow the same workflow. On 30 stratified tasks, two independent experts with repository access but without the obligation checklists, developer patches, or tests reviewed and refined the candidate plans ofGPT-5.6-S, Codex. We scored the refined plans with the Plan Judge and executed them across the four executors ofSec 5.2on the same 30 tasks:Expert review raises verification support by 13.6 points while non-verification support rises by 2.4 points, indicating that remaining headroom concentrates in verification. This lifts Plan Coverage from 87.3 to 95.0 and accompanies an 8.3-point rise in execution success (82.5% to 90.8% across 120 executor runs). Across all 392 tasks and 24 configurations, verification support averages 61.6% vs. 90.0%–95.3% for non-verification roles (
Tab. 8). We address how SWE-Plan stays challenging as models improve inQ2.Regarding verification specificity, experts without checklist access added tests that bring verification support to 88.4%, so most verification obligations can be met at planning time, and the 13.6-point gap of the agent plans lies on content an expert planner reaches. Each verification obligation states a property separating a valid implementation from a near miss (
App. I.1) and cites the request, patch, or test lines that establish it (App. I.2). In Case 1 (App. Q.1), the registered answer requires stopping the drain immediately, so a test that the drain reads exactly once is the direct observable check of non-retrying behavior, yet 14 of the 16 Claude Code and Codex configurations omitted it, and all 16 omitted the partial-drain check. In an external audit of 50 sampled verification obligations (App. I.6), developers confirmed 86.0% as necessary tests, and replacing verification content with equal-lengthWeak testson 75 tasks lowers execution success by 4.0 to 8.0 points (Sec 5.2,App. L.6). We have added these analyses toSec 6(Obs. 1) andApp. P.W2. User Simulator Matching and Live Developer Study
Thank you for these careful observations. Under
Sec 4.1andApp. H.1, a question matches only when a direct answer to the complete question determines the decision, so a partial question does not match and receives the template reply without leaking the answer. We also clarified inSec 4.1andApp. H.1that the User Simulator evaluates questions together with their options, matching multiple-choice questions when options map to distinct decision values and confirmation questions when a direct answer settles the decision (as in Fig. 2).The remaining difference is that real developers can give free-form guidance for out-of-registry questions. To test whether this shifts scores, we recruited 5 developers as live CLI users across a stratified subset of 75 of the 100 tasks in
Sec 5.2on three models in OpenCode, paired against simulator runs on the same tasks (Agreement is the question share where the simulator's offline verdict matches whether the developer's reply settles a registered decision):Although 18.7% of turns in the live sessions had open-ended replies, Blocker Recall and Plan Coverage differ from simulator runs by only 0.9 and 0.5 points on average across the three models, with 91.1% agreement. We have added these clarifications to
Sec 4.1and the live study toSec 5.1andApp. L.3.