acceptodds
Under review as a conference paper at ICLR 2027

SWE-Plan: Benchmarking Coding Agents Before They Code

Abstract

Before implementation, current coding agents typically clarify requirements, ask for critical decisions, and form a plan, a process that mainstream products have formalized as Plan Mode. However, existing benchmarks largely entangle this pre-implementation phase with code execution and overlook requirement clarification, leaving this phase unanalyzed and the effects of models and harnesses unattributed. To bridge this gap, we introduce SWE-Plan, the first benchmark that isolates the complete pre-implementation phase and evaluates it directly. Grounded in requirements engineering, SWE-Plan comprises 392 repository-level tasks with 929 withheld critical decisions and 4,937 atomic planning obligations, assessing whether agents recover these decisions and produce complete, repository-consistent plans, with metric reliability validated by human judgment and downstream execution. Evaluating 8 models under 3 mainstream harnesses, we demonstrate that (I) clarification and planning are correlated within a system at r = 0.62 yet not interchangeable; (II) failures in both dimensions concentrate on requirements and verification obligations that are never explicitly stated; and (III) planning rankings correlate at ρ = 0.63 across harnesses, while a 5.5 pp change in ask rate accompanies only a 0.8 pp change in question precision, indicating that the base model mainly determines planning and questioning ability, while the harness mainly shifts the willingness to ask. SWE-Plan provides a diagnostic and systematic evaluation foundation for optimizing coding agents and dedicated planners.

Primary area:
datasets and benchmarks
Keywords:
coding agents, benchmark, planning, requirement clarification, software engineering

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_YZy6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • This paper proposes a way to test the pre-implementation abilities of LLM coding agents rather than traditional benchmarks which test final implementation correctness by evaluating model patches. The setup is realistic with a user simulator which has the knowledge about withheld information about the task to assist the agent in any clarifications. The plan produced by the agent is evaluated with an agentic plan judge with task specific checklists.
  • Tasks contain blockers to test clarification abilities of planning agents. Blockers are designed such that they ensure multiple plausible paths and lead to different final implementations without clarification.
  • The evaluation captures both axes: clarifications posed to the user and planning evaluation through checklists. Both the evaluations for the plan and simulated user have a high agreement rate with human experts.
Critical strengths
  1. The benchmark evaluates the pre-implementation abilities of coding agents, clarification and planning, which are understudied in recent literature but very timely given that major coding tools now have a native plan mode. The design decisions are grounded in software engineering and requirements engineering literature. Running agents in the native Plan Mode of real harnesses also makes the evaluation realistic, so the results reflect how these tools are actually used.
  2. The paper makes a solid effort to validate its metrics. Both judges are checked against human labels and the execution study also shows the plan scores matter for implementation: adding a plan raises execution success by 8–12 points, and Plan Coverage predicts which plans lead to success better than plan length or an overall 1–10 rating does.
Critical weaknesses
  1. There seems to be limited headroom for frontier models, which already reach 89–93 Plan Coverage. Most of their remaining gap is in verification obligations, which look over-specific (appendix analysis case 1). I suspect this partly reflects the task source, since the tasks involve implementations of moderate size. Harder tasks, like those in existing benchmarks, might stress planning more. It would help to see a reference ceiling (e.g., expert-written plans) and some discussion of how the benchmark can stay challenging and continue to distinguish frontier models as they improve.
  2. The User Simulator design is a bit simple. It returns a stored answer when a question matches a withheld decision and the same template reply for everything else. From what I understand, it always returns the full stored answer even when the question covers only part of a decision. It also can’t handle confirmation-style or multiple-choice questions, or respond realistically to reasonable questions outside the registry. The paper doesn't tests how much this simplification affects the clarification scores. A more realistic simulator or a small study with real developers would make the interactive part of the benchmark much more aligned with real world usage.
Questions for authors
  1. I'm curious if you analyzed the types of information withheld across tasks? Do they cover a diverse range of decision types, e.g., failure behavior, scope, or interactions with existing options? It would be great to see a breakdown by type, along with how often agents recover each type through clarification. That would show which kinds of hidden information agents reliably ask about and which they tend to miss.
  2. The benchmark tasks currently have a cutoff of June 1, 2026. What are your thoughts on the saturation of the benchmark with new model releases in the future? How did this influence your choice to use SWE-Factory over sampling tasks from existing benchmarks which are less prone to contamination?
3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer YZy6 (1/2): Expert Ceiling and User Simulator

We sincerely thank you for your careful and constructive review. We address your weaknesses first and your questions in the next comment. Revisions in the updated PDF are marked in blue.


W1. Expert Reference Ceiling and Verification Specificity

Thank you for suggesting an expert reference ceiling, which lets us locate where headroom remains. Plan Mode ends with a reviewable plan that the developer checks before implementation (Sec 3.1), so we follow the same workflow. On 30 stratified tasks, two independent experts with repository access but without the obligation checklists, developer patches, or tests reviewed and refined the candidate plans of GPT-5.6-S, Codex. We scored the refined plans with the Plan Judge and executed them across the four executors of Sec 5.2 on the same 30 tasks:

PlannerPlan CoverageNon-verification (%)Verification (%)Execution success (%)
Expert-reviewed plans95.097.688.490.8
GPT-5.6-S, Claude Code88.895.677.285.0
GPT-5.6-S, Codex87.395.274.882.5
Opus-5, Claude Code87.694.876.081.7
Gem-3.8, OpenCode82.491.468.876.7
Mean of 24 configurations77.788.561.571.7

Expert review raises verification support by 13.6 points while non-verification support rises by 2.4 points, indicating that remaining headroom concentrates in verification. This lifts Plan Coverage from 87.3 to 95.0 and accompanies an 8.3-point rise in execution success (82.5% to 90.8% across 120 executor runs). Across all 392 tasks and 24 configurations, verification support averages 61.6% vs. 90.0%–95.3% for non-verification roles (Tab. 8). We address how SWE-Plan stays challenging as models improve in Q2.

Regarding verification specificity, experts without checklist access added tests that bring verification support to 88.4%, so most verification obligations can be met at planning time, and the 13.6-point gap of the agent plans lies on content an expert planner reaches. Each verification obligation states a property separating a valid implementation from a near miss (App. I.1) and cites the request, patch, or test lines that establish it (App. I.2). In Case 1 (App. Q.1), the registered answer requires stopping the drain immediately, so a test that the drain reads exactly once is the direct observable check of non-retrying behavior, yet 14 of the 16 Claude Code and Codex configurations omitted it, and all 16 omitted the partial-drain check. In an external audit of 50 sampled verification obligations (App. I.6), developers confirmed 86.0% as necessary tests, and replacing verification content with equal-length Weak tests on 75 tasks lowers execution success by 4.0 to 8.0 points (Sec 5.2, App. L.6). We have added these analyses to Sec 6 (Obs. 1) and App. P.


W2. User Simulator Matching and Live Developer Study

Thank you for these careful observations. Under Sec 4.1 and App. H.1, a question matches only when a direct answer to the complete question determines the decision, so a partial question does not match and receives the template reply without leaking the answer. We also clarified in Sec 4.1 and App. H.1 that the User Simulator evaluates questions together with their options, matching multiple-choice questions when options map to distinct decision values and confirmation questions when a direct answer settles the decision (as in Fig. 2).

The remaining difference is that real developers can give free-form guidance for out-of-registry questions. To test whether this shifts scores, we recruited 5 developers as live CLI users across a stratified subset of 75 of the 100 tasks in Sec 5.2 on three models in OpenCode, paired against simulator runs on the same tasks (Agreement is the question share where the simulator's offline verdict matches whether the developer's reply settles a registered decision):

ModelModeAsk-F1Blocker RecallPlan CoverageAgreement (%)
GPT-5.6-SSimulator81.978.389.3—
Developers82.579.190.291.8
Opus-5Simulator82.493.489.6—
Developers82.094.189.890.5
Gem-3.8Simulator85.479.280.6—
Developers86.180.481.191.0
MeanSimulator83.283.686.5—
Developers83.584.587.091.1

Although 18.7% of turns in the live sessions had open-ended replies, Blocker Recall and Plan Coverage differ from simulator runs by only 0.9 and 0.5 points on average across the three models, with 91.1% agreement. We have added these clarifications to Sec 4.1 and the live study to Sec 5.1 and App. L.3.

AuthorsOpenReview ↗

Rebuttal to Reviewer YZy6 (2/2): Blocker Categories and Saturation

We continue below with your two questions.


Q1. Breakdown of Blocker Categories and Clarification Recall

Thank you for this valuable suggestion. Yes, withheld decisions span diverse functional requirement types. Building upon the blocker criteria in Sec 3.2, App. D, and the task instances in App. F, and informed by your three examples, we manually annotated all 929 blockers into four core decision categories (three from your examples and one that emerged during annotation) plus an Others category. Two authors independently audited 100 blockers with an inter-annotator agreement of Cohen's κ = 0.84. Blocker Recall below is pooled across blockers in each category over all 24 configurations (session-macro recall is 86.3% in Obs. 3):

Decision categoryBlockersShare (%)Mean Blocker Recall (%)Range across 24 configurations (%)
Failure and error behavior18620.089.271.5–97.8
Scope and boundary conditions34937.686.567.3–96.3
Interaction and precedence across options323.478.156.3–90.6
Output and interface contract13514.590.474.1–98.5
Others22724.484.664.8–95.6

Agents most reliably clarify Output and interface contract decisions and most frequently miss Interaction and precedence across options decisions. We have added this category breakdown to Sec 6 (Obs. 3), the annotation taxonomy to App. B, the audit methodology to App. D, and the complete per-configuration recall table to App. M.4.


Q2. Benchmark Saturation and Contamination Control

Thank you for raising saturation and contamination. We expect SWE-Plan to stay informative as models improve because tasks can be rebuilt after later cutoffs. We chose SWE-Factory over existing benchmarks for two reasons. First, established benchmarks like SWE-bench predate current model training cutoffs and face documented pretraining memorization and contamination risks [1, 2]. Because blockers in SWE-Plan are decisions reconstructed from developer patches and tests (Sec 3.2), an agent that has memorized an older patch could bypass clarification altogether. Furthermore, existing benchmark issues can leak solutions [3], and evaluating clarification and planning would still require constructing new rewritten requests, blocker registries, and obligation checklists from scratch. Second, SWE-Factory let us pin every repository after June 1, 2026, placing patches and commits after all published cutoffs, while the newly written requests, registries, and checklists are not public before release (App. C.3).

To keep the benchmark challenging and uncontaminated as new models appear, we take two concrete measures:

  • Continuous expansion via our construction pipeline. Following automated pipeline practices for refreshing live benchmarks [4, 5], our SWE-Factory-based construction pipeline can rebuild tasks from repository commits after newer cutoff dates, and its active use inside our company to evaluate new models provides sustained engineering resources for this rolling expansion.
  • Versioned tracking with a private split. We commit to maintaining the benchmark across future versions, keeping newly added tasks as an internal private split, and publishing full results.

We would value any further suggestions on keeping SWE-Plan informative as models improve. We have added the task-source discussion to Sec 3.2 and App. C.3 and updated the Reproducibility Statement.


We hope these revisions strengthen the paper, and we are grateful for your time and constructive feedback. Thank you once again for your careful reading and expert suggestions. We hope our comments can address your concerns and look forward to your further thoughts.


References

[1] Liang et al. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason.

[2] Prathifkumar et al. Does SWE-Bench-Verified Test Agent Ability or Model Memory?

[3] Aleithan et al. SWE-Bench+: Enhanced Coding Benchmark for LLMs.

[4] Zhang et al. SWE-bench Goes Live!

[5] Badertdinov et al. SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agents.

Thank you for the clarifications. I will keep my score.

Reviewer_1DtVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Coding agents can produce plans that appear perfectly reasonable which in reality don't meet the actual requirements. SWE-PLAN is used to determine whether the agents are in fact asking for the right bits of information and are taking the constraints into account prior to them going off to code. It's a mechanism for catching planning failures early and thus reducing the number of incomplete or wrong fixes.

The authors found three main gaps within existing benchmarks:

(a) End-to-end coding benchmarks assess the final code so they can't tell you whether the failure was due to bad planning or bad implementation. (b)Planning benchmarks will typically provide the agent with the full specification so the agent won't need to ask any clarifying questions. (c) Clarification benchmarks test the questions in isolation and don't test whether the final plan actually makes use of the answers.

Critical strengths
  1. The paper is about a significant gap in the assessment of software engineering. The current coding benchmarks only assess the final patches and so it's extremely difficult to work out whether a failure was due to bad requirement definition, bad planning or bad implementation. SWE-PLAN addresses this by taking the pre-implementation phase into account as a capability that can be directly measured.

  2. The benchmark assesses both clarification and planning within a single, realistic workflow. The agent looks at the repository in read only, asks about the withheld decisions and then produces a plan. The two capabilities are scored separately so it's possible for the paper to demonstrate that they are related but not interchangeable.

  3. The task set is quite large and drawn from real repos. It contains 392 tasks from 154 repos, 929 withheld decisions and 4,937 planning obligations. This enables analysis of both whether agents are asking for the right information and whether their plans include the necessary actions.

  4. The evaluation is validated against human judgement and coding results. Agreement with human experts is reported for both evaluators (Cohen's κ = 0.89 for clarification and 0.76 for planning). There is also a finding that higher plan scores are associated with greater downstream execution success, confirming the practical value of the measures.

  5. The experiments produce useful diagnostic results. Of the eight models and three harnesses considered the authors identify weaknesses around requirements that are implicit (in particular verification obligations). The analysis also helps to separate out model capability from harness effect, which in turn provides some guidance for improving agents and planning systems.

Critical weaknesses
  1. SWE-PLAN uses actual repositories and pre-selected blockers to test clarification. It has a simulator which returns fixed answers. Real users can be unsure, can change their requests, can behave in all sorts of unexpected ways. The benchmark is testing one plan only and doesn't assess whether the agent is capable of adapting or replanning in the face of changing requirements. The authors are very aware of this limitation.

  2. The blockers and planning requirements are derived from the issue, the repository, the developer patch and the tests. Patch specific detail is stripped out and other valid approaches are allowed. The checklists may still contain detail that only becomes apparent upon implementation. External reviewers should be assessing whether the checklists represent what a reasonably competent planner would know at the outset.

  3. For the execution study we pick three plans per task to give a good spread of Plan Coverage scores and test 100 tasks. This is to see whether the plans with the higher scores are more likely to result in successful execution. The plans we select may not be the plans that agents would actually produce. Testing with randomly selected plans or with a separate held out set would be a much stronger test.

  4. The paper validates its scoring tools via human experts and provides no scores for plans produced by experienced developers. A comparison of agent plans with human plans would be useful in terms of demonstrating whether the benchmark is in fact measuring competent planning and how agent performance stacks up against human performance.

  5. The task set is predominantly Python (364 of 392). The repos are of moderate size and the harnesses are using their defaults. The results are amenable to comparison across the different configurations tested. More work would be required to establish whether they would be applicable to other languages, larger repos or with custom harnesses.

Questions for authors
  1. What makes a plan executable in SWE-PLAN? Does the score look at whether the steps are in the correct order, whether there are any dependencies and whether they can actually be done in the repo, or does it just check that the plan references the required items?

  2. The checklists are based on the original code changes and tests. How do you ensure they're capturing the requirements of the task rather than just the decisions that were made in the original implementation?

  3. The simulated user gives a fixed answer to each recognised question. How would the results be different if the users were uncertain, gave partial answers or added requirements?

  4. The execution study selects plans to cover a wide range of scores and tests 100 tasks. Do the results generalise to randomly selected plans and tasks across repeated coding runs?

  5. Can you get plans written by experienced developers and then test whether those plans result in good code? That would be a useful way of understanding what a high SWE-PLAN score actually means.

3 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 1DtV (1/3): Realistic Users

We sincerely thank you for the detailed and constructive review. Your ten points group into six topics, which we answer in three comments, following the order of your weaknesses. This comment covers the User Simulator (W1 & Q3), the second covers the checklists (W2 & Q2) and the execution study (W3 & Q4), and the third covers expert plans (W4 & Q5), generalization (W5), and what makes a plan executable (Q1). Revisions in the updated PDF are marked in blue.


W1 & Q3: Uncertain, Partial, and Evolving Users

Thank you for these questions about realistic users. We agree that requirements that are added or change during a session, and the replanning they call for, lie beyond what SWE-Plan measures, since it evaluates the plan that follows an initial underspecified request (App. A). Within this setting, the User Simulator returns consistent registered answers so that every configuration faces the same user and results stay comparable and reproducible. To test whether real users change the scores, we recruited 5 developers as live CLI users on three models in OpenCode, across a stratified subset of 75 of the 100 execution-study tasks in Sec 5.2, paired against simulator runs on the same tasks (Agreement is the question share where the simulator's offline verdict matches whether the developer's reply settles a registered decision):

ModelModeAsk-F1Blocker RecallPlan CoverageAgreement (%)
GPT-5.6-SSimulator81.978.389.3—
Developers82.579.190.291.8
Opus-5Simulator82.493.489.6—
Developers82.094.189.890.5
Gem-3.8Simulator85.479.280.6—
Developers86.180.481.191.0
MeanSimulator83.283.686.5—
Developers83.584.587.091.1

Although 18.7% of turns in the live sessions had open-ended replies, Blocker Recall and Plan Coverage differ from simulator runs by only 0.9 and 0.5 points on average across the three models, with 91.1% agreement.

In a second study, 5 developers played two instructed users on the same 75 tasks across the three models in OpenCode. For one randomly assigned blocker per task, the Uncertain user states no preference and leaves the choice to the agent, while the Partial user provides half of the registered specification. For instance, when an agent asks whether an exported report should default to JSON or CSV and whether an existing file should be overwritten, an Uncertain user replies that they have no preference and the agent should decide, whereas a Partial user specifies JSON but leaves the conflict behavior unstated. Native Plan Judge rules evaluate whether the resulting plans support the unchanged obligations:

ModelFull answersPartial userUncertain user
GPT-5.6-S89.385.282.4
Opus-589.683.480.6
Gem-3.880.672.867.4
Mean86.580.576.8

When users provide incomplete specifications (Partial) or leave decisions unconstrained (Uncertain), mean Plan Coverage drops by 6.0 and 9.7 points respectively. We have added the live study to Sec 5.1, the instructed-user study to Sec 6, and both protocols to App. L.3 & L.4, and updated App. A.


AuthorsOpenReview ↗

Rebuttal to Reviewer 1DtV (2/3): Checklist Validity and Execution Robustness

W2 & Q2: Blockers, Obligations, and Planning-Time Knowledge

Thank you for this central question. A decision of the original implementation that the repository cannot determine, and whose plausible answers lead to materially different correct plans, becomes a blocker (Sec 3.2). Its answer is removed from the request and the container, so a competent planner is not expected to know it at the outset and must ask for it, and the User Simulator then returns the user's answer. Obligations are the properties that every valid route must satisfy once the request and these answers are given, and choices on which valid routes differ become accepted freedoms that earn no credit (App. I.2). The checklists therefore state what a planner can determine before implementation from the request, the answers obtained by asking, and the repository.

The developer patch does not define the checklist. Three coding agent configurations from different model families each enumerate at least 10 routes under a prompt stating that the patch is one valid route and neither the only nor the preferred one (App. I.3 & I.4). Two experts reject infeasible routes, merge equivalent ones, and keep only the properties held by every retained route, and patch-specific names and helper structures are excluded (App. I.2 & I.5). Checklists are frozen before any plan is scored (Sec 4.2).

In an external audit of 50 sampled verification obligations, developers confirmed 86.0% as necessary tests (App. I.6). To cover all five roles, 4 external developers who took no part in task construction re-judged these 50 for planning-time determinability and audited 147 new obligations from the other four roles for necessity and determinability. For each obligation they saw the request, its registered answers, the repository at the pinned pre-fix commit, and the obligation, without the developer patch or tests. They judged whether it is a necessary property of a correct solution and, for necessary obligations, whether a competent planner can determine it before implementation from these inputs, directly or through engineering judgment (Determinable), or whether it only becomes apparent upon implementation (Implementation-only):

RoleObligationsNecessary (%)Determinable (%)Implementation-only (%)
Verification5086.088.411.6
Other four roles14793.994.95.1
All19791.993.46.6

Auditors judged 93.4% of the necessary obligations determinable before implementation. We have added the audit to App. I.6 and a summary to Sec 4.2.


W3 & Q4: Natural Plans and Repeated Runs

Thank you for asking whether the execution study holds beyond the plans we selected. The three plans of each task are unmodified outputs of the evaluated configurations. We chose them from three different Plan Coverage bins so that scores vary within each task and span the full range across tasks (Sec 5.2, App. L.5), and execution outcomes play no role in this choice. Because the choice depends only on Plan Coverage, it leaves the chance of success at a given Plan Coverage unchanged, and on the 225 plans of the 75 tasks used for the plan variants in Q1, a within-task conditional logit controlling for log plan length gives an odds ratio of 1.42 per 10 points of Plan Coverage [95% CI 1.24, 1.63]. Since the original study executes each condition once, we ran two further tests with the four executors, each giving the executor the request, the answers, and the plan.

For repeated runs, we reran all 300 plans 2 more times. Same is the share of plans with the same outcome in all three runs, Range is the spread of the success rate across the three runs, and AUC gives the within-task AUC of each run:

ExecutorSame (%)Range (points)AUC (runs 1, 2, 3)
GPT-5.6-S87.72.00.81, 0.82, 0.80
GPT-5.6-L82.02.70.79, 0.78, 0.80
Opus-588.31.70.84, 0.85, 0.85
Sonnet-584.32.70.86, 0.85, 0.84

For random plans, we drew 100 held-out tasks from the 292 tasks outside the original 100, drew 3 plans per task at random from the 24 evaluated configurations without using Plan Coverage, and executed each plan once. Tasks whose three plans all succeed or all fail carry no within-task information, so AUC uses the remaining tasks:

ExecutorSuccess (%)Tasks used for AUCAUC
GPT-5.6-S83.7410.75
GPT-5.6-L73.0480.73
Opus-584.7380.77
Sonnet-576.7450.76

On the random plans from held-out tasks, Plan Coverage continues to predict execution success, as a within-task conditional logit controlling for log plan length yields an odds ratio of 1.34 per 10 points [95% CI 1.07, 1.68] (compared with 1.42 on the 225 selected plans). We have added a sentence to Sec 5.2 and the full protocol and tables to App. L.6.


AuthorsOpenReview ↗

Rebuttal to Reviewer 1DtV (3/3): Expert Plans, Generalization, and Scoring

W4 & Q5: Expert-Reviewed Plans and Their Execution

Thank you for suggesting this comparison. In Plan Mode, the developer reviews the agent's plan before implementation (Sec 3.1), so we measured how far experienced developers raise a strong agent plan in this workflow. On 30 stratified tasks, two independent experts with repository access but without the obligation checklists, developer patches, or tests reviewed and refined the candidate plans of GPT-5.6-S, Codex. We scored the refined plans with the Plan Judge and executed them across the four executors of Sec 5.2 on the same 30 tasks:

PlannerPlan CoverageNon-verification (%)Verification (%)Execution success (%)
Expert-reviewed plans95.097.688.490.8
GPT-5.6-S, Claude Code88.895.677.285.0
GPT-5.6-S, Codex87.395.274.882.5
Opus-5, Claude Code87.694.876.081.7
Gem-3.8, OpenCode82.491.468.876.7
Mean of 24 configurations77.788.561.571.7

Expert review raises verification support by 13.6 points while non-verification support rises by 2.4 points. This lifts Plan Coverage from 87.3 to 95.0 and accompanies an 8.3-point rise in execution success (82.5% to 90.8% across 120 executor runs). On these 30 tasks, the expert-reviewed plans have the highest value on all four measures. In this comparison, the largest gain is in the verification content that experienced developers add in review. We have added this comparison to Sec 6 (Obs. 1) and the protocol and per-role results to App. P.


W5: Languages, Repository Scale, and Harnesses

Thank you for this observation. We agree that the concentration on Python, the moderate repository sizes, and the default harness settings are real limitations of the current benchmark, as App. A states, and that results on other languages, larger repositories, or customized harnesses need further work. The table reports the 24-configuration mean on each subset:

SubsetTasksAsk-F1Plan Coverage
All tasks39284.183.5
Python36484.383.7
Non-Python2882.180.8
Smallest third by snapshot files13085.385.1
Largest third by snapshot files13182.681.7

The non-Python tasks and the largest third of repositories stay within 2.7 points of all tasks on both measures, so the overall results remain close on these subsets. We have added the subset results to App. M.5.


Q1: What Makes a Plan Executable

Thank you for this question. Mentioning a required item earns no credit by itself. Plan Coverage credits an obligation only when the plan explicitly supports it or an equivalent valid route, and a plan that names the right component but commits to a mechanism incompatible with the obligation is labeled a conflict and penalized (Sec 4.2, App. J.1). The Plan Judge is a read-only agent that inspects the repository at its pinned revision before labeling each obligation. Fit with the repository is checked through two roles, ownership (the change sits where the existing decomposition places it) and preservation (existing behavior and interfaces stay intact). The Judge does not score step order separately, but if a different order breaks a precondition, the obligations that depend on it are judged absent or conflicting.

To test whether the score follows what a plan commits to, we used the same 75 tasks as in W1 & Q3. One expert rewrote the highest-scoring plan of each task against the frozen checklist and the repository within 4% of its word count, and a second expert checked that only the targeted obligations changed. Near miss ×1 and Near miss ×2 rewrite one or two supported obligations into their frozen near misses, Weak tests replaces the verification content with equal-length tests that cannot separate a correct implementation from its near misses, and Shuffled steps shuffles the plan steps with their text unchanged. The Plan Judge rescored every variant, and the table reports Plan Coverage and the success change from the original plan, as a range over the four executors of Sec 5.2:

VariantPlan CoverageSuccess change (points)
Original94.8—
Near miss ×181.5−5.3 to −9.3
Near miss ×269.4−9.3 to −16.0
Weak tests80.7−4.0 to −8.0
Shuffled steps93.20.0 to −4.0

At equal length, near misses lower Plan Coverage by 13.3 and 25.4 points and success by 5.3–16.0 points, so the score follows the correctness of what a plan commits to. Weak tests lower them by 14.1 and 4.0–8.0 points. Shuffling lowers Plan Coverage by only 1.6 points (obligations whose preconditions the new order breaks) and success by at most 4.0 points. We have added the Judge's checks to Sec 4.2 and the variants to App. L.6.


Thank you again for your careful reading and constructive suggestions. We hope our comments can address your concerns and look forward to your further thoughts.

Reviewer_Wi3iOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SWE-Plan is a benchmark for evaluating coding agents before they write code. Existing benchmarks only check whether the final patch passes tests, folding requirement clarification and planning into one end-to-end outcome; SWE-Plan isolates this pre-implementation phase and measures it directly. It splits the phase into two dimensions, clarification and planning, builds 392 tasks from 154 real repositories, and withholds 2 to 4 key decisions per task so that agents cannot proceed correctly without asking. Both dimensions are scored by evaluators calibrated against human experts. Testing 8 models across 3 harnesses, the authors find that clarification and planning are coupled but not interchangeable across models, that failures in both concentrate where nothing is explicitly stated, and that planning ability is mainly model-driven while willingness to ask is more harness-driven.

Critical strengths
  1. Clear problem formulation and timely benchmark. The paper identifies a genuine gap: existing coding benchmarks fold requirement interpretation, clarification, and planning into end-to-end execution outcomes. It isolates the pre-implementation phase as a measurable object, and grounding the two dimensions in requirements engineering gives the benchmark a principled, non-arbitrary foundation.

  2. Rigorous construction and validation. The benchmark is built from real repositories with expert-verified blockers and obligation checklists, and both evaluators are calibrated against human experts with strong agreement (κ = 0.89 and 0.76). The execution-decoupled evaluation, the predictive-validity analysis, and the multi-model multi-harness experiments together make the findings credible and practically informative.

Critical weaknesses
  1. The predictive-validity gain from adding a plan is attributed to plan quality, but a plan is itself a specification: a high-Plan-Coverage plan may simply provide more explicit, copyable information and thereby lower task difficulty. Since Plan Coverage rewards explicit coverage, higher-scoring plans are mechanically more “copyable,” and the experiment does not separate information quantity from quality. The verification ablation isolates only one component. A cleaner test would hold information constant while manipulating quality (e.g., inject concrete conflicts, scramble dependencies, or use an equally detailed but invalid route), or withhold the registered answers, to see whether success still tracks Plan Coverage.
Questions for authors

See my concern in Critical Weaknesses. If this is resolved, I would keep my rating at 4.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Wi3i: Test of Plan Quality & Information Quantity

We are sincerely grateful for your generous assessment and for this insightful comment. Your distinction between how much a plan spells out and how good it is sharpens what predictive validity should mean. We agree that this separation is essential, both for the predictive-validity result and for whether the benchmark could be gamed through verbose plans.

Your point that a plan is itself a specification is well taken. An executor relies on what a plan states explicitly, so explicitness is part of what makes a plan useful, and Plan Coverage credits an obligation only when the plan explicitly supports it. As you note, explicitness and correctness can then move together, and we agree that they should be tested separately. Text that addresses no obligation earns no credit, a concrete incompatible mechanism is labeled a conflict, and each obligation is paired with a near miss frozen in the checklist before any plan was scored. The full rules are in App. I.2 & J.1. In Tab. 3, plan length reaches a within-task AUC of 0.53–0.58 vs. 0.79–0.86 for Plan Coverage.

To test your point directly, we ran the tests you proposed, plus a mirror padding test, on a stratified subset of 75 of the 100 tasks in Sec 5.2. One expert manually rewrote each plan against the frozen checklist and repository, and a second checked that lengths matched and only the targeted obligations changed. The Plan Judge rescored every variant, and execution follows Sec 5.2. In the first two tests, success follows plan quality at a fixed length and changes little when text is added without quality.

1. Same Information, Lower Quality

For each task, we built four variants of its highest-scoring plan, with word counts within 4% of the original:

  • Near miss ×1 and Near miss ×2 rewrite one or two supported obligations into their near misses in the frozen checklist, giving an equally detailed route with concrete conflicts. Obligations that restate a registered answer are excluded.
  • Weak tests replaces the verification content with tests of equal length that cannot distinguish a correct implementation from its near misses.
  • Shuffled steps shuffles the plan steps and keeps their text unchanged.

Plan Coverage and success (%) are as follows:

ExecutorOriginalNear miss ×1Near miss ×2Weak testsShuffled steps
Plan Coverage94.881.569.480.793.2
GPT-5.6-S94.788.084.089.393.3
GPT-5.6-L84.077.368.076.080.0
Opus-596.090.786.792.096.0
Sonnet-590.781.377.384.088.0

At the same length, one near miss lowers success by 5.3 to 9.3 points and two lower it by 9.3 to 16.0 points, with the drop comparable to or exceeding the 8.0 to 11.7 point gain from adding a plan in Tab. 3. Weak tests lower success by 4.0 to 8.0 points across all executors and Plan Coverage by 14.1. Shuffling the steps lowers success by at most 4.0 points (0.0 to 4.0 points) and Plan Coverage by 1.6.

2. More Information, Same Quality

For the padding test, we padded each lowest-scoring plan with accurate repository descriptions that address no obligation, up to the length of the highest-scoring plan:

ExecutorLowestLowest + paddingHighest
Plan Coverage54.654.994.8
GPT-5.6-S64.065.394.7
GPT-5.6-L53.352.084.0
Opus-566.768.096.0
Sonnet-552.053.390.7

Padding shifts success by −1.3 to +1.3 points and Plan Coverage by +0.3, whereas the highest-scoring plan of the same length succeeds 28.0 to 37.3 points more often.

3. Withholding the Registered Answers

Following your second suggestion, we reran all 225 plans without the registered answers. On the runs with answers, we also compute Plan Coverage restricted to ownership, preservation, and verification obligations, whose content lies beyond the copyable answers:

ExecutorSuccess, withSuccess, withoutAUC, withAUC, withoutAUC, three roles
GPT-5.6-S83.177.30.810.790.76
GPT-5.6-L72.966.20.780.770.74
Opus-585.380.40.840.810.78
Sonnet-576.470.20.850.830.79

Without answers, success drops by 4.9 to 6.7 points, while Plan Coverage keeps an AUC of 0.77 to 0.83. Since plans carry the copyable answers, the drop is small, and the three-role Plan Coverage still reaches an AUC of 0.74 to 0.79, and in a within-task conditional logit controlling for log plan length, the full Plan Coverage has an odds ratio of 1.42 per 10 points [95% CI 1.24, 1.63].


We have added a pointer in Sec 4.2, these results in Sec 5.2, and App. L.6 with the full protocol, per-executor tables, and before/after examples, with all revisions marked in blue. Thank you again for this valuable suggestion, and we hope these tests resolve your concern.

Reviewer_Wi3iOpenReview ↗

Reply to Rebuttal

I'll keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.