acceptodds
Under review as a conference paper at ICLR 2027

COSWE: Evaluating Interactive Planning in Collaborative Software Engineering

Abstract

Coding agents increasingly handle repository exploration, implementation, and testing, but real development often begins with only a partial description of what the user wants. Early benchmarks assessed code against predefined specifications, while recent work has brought requirement clarification and user feedback into the evaluation. This shift calls for measuring interactive planning ability: the speed and quality with which agents identify and formulate the task’s target technical solution through interaction. The target reflects the configured user’s requirements and preferences, without assuming a universally optimal design. We introduce PlanScore as the primary metric, combining discovery timing, resolution delay, and completion to track progress toward accepted decisions. To support this evaluation, we build COSWE, a benchmark based on a dataset of real repository changes. Each task pairs an underspecified request with private reference requirements and a stateful user simulator, so agents must clarify choices, provide behavioral evidence, and explain technical trade-offs as they develop a plan. Report fidelity complements the interaction-based score by assessing whether final reports preserve the reference requirements. Together, the dataset, interaction protocol, and evaluation framework make both the process and outcome of planning observable. Experiments reveal differences in agents’ discovery and resolution performance and show that report omissions consistently exceed contradictions.

Primary area:
datasets and benchmarks
Keywords:
Agentic coding, Software engineering, Interactive benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XsigOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces COSWE, a benchmark for evaluating interactive planning in collaborative software engineering. Built from real-world GitHub pull requests, COSWE requires coding agents to clarify incomplete requirements, provide evidence, and explain technical trade-offs through interactions with a simulated user. The authors propose PlanScore to evaluate how efficiently agents discover and resolve missing requirements, along with additional metrics for token efficiency, report fidelity, and implementation correctness.

Critical strengths
  1. The idea is practical and meaningful.
  2. The Search, Experience, and Credence categorization is well motivated and provides a useful way to capture different types of user interaction in software engineering tasks.
Critical weaknesses
  1. The design of PlanScore is not sufficiently justified. The use of reciprocal functions and a geometric mean appears largely heuristic and lacks empirical validation. Moreover, PlanScore conflates completion with interaction efficiency, making it difficult to determine whether score differences stem from higher completion rates or faster resolution. Reporting efficiency separately would help clarify the added value of PlanScore.

  2. The benchmark statistics are insufficiently reported. The paper does not provide basic dataset statistics, such as the number of tasks and repositories, the distribution of S/E/C facets, or the number of hidden requirements and tests per task. Statistics on task difficulty and complexity are also missing.

  3. The paper lacks controlled comparisons between agents operating without user interaction, with interactive clarification, and with access to complete requirements. This makes it difficult to disentangle the contribution of interactive planning from the agents’ inherent coding capabilities.

  4. The experimental analysis could be more comprehensive. While the paper reports overall planning scores and their correlations with implementation performance, it provides limited analysis of how different facet types, task complexity, and interaction failure modes affect the results. A more fine-grained analysis would help explain where agents struggle and provide deeper insights into their interactive planning capabilities.

Questions for authors
  1. Although COSWE defines three types of interaction facets (Search, Experience, and Credence), the evaluation only includes Experience and Credence facets (written in Section 4.1). Why are Search facets excluded from PlanScore and Clarification Completion, given that direct requirement clarification is also an important part of interactive planning?

  2. What reasoning effort settings were used for each model? These settings are not reported in the paper.

Reviewer_tTjrOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This work aims to capture settings where agents receive incomplete requests and must clarify user requirements.
  • The authors introduce COSWE, a benchmark built from real repository changes with hidden requirements and a simulated user that reveals information under predefined conditions.
Critical strengths
  • This work includes metrics that are orthogonal but complementary to performance.
Critical weaknesses
  • The distinction from prior work is very unclear (e.g., [1] and [2]).
  • Insufficient detail about the user simulator provided around experimental configuration and validation
  • Since PlanScore and pass rate are positively correlated, it is difficult to glean new insights for new development that were not already highlighted in prior work.
  • Lack of detail about how many examples were considered and filtered out in the “filter and select” process, along with human validation.

[1] https://arxiv.org/abs/2606.29957 [2] https://arxiv.org/abs/2502.13069

Questions for authors

Please address above weaknesses.

Reviewer_j14MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

COSWE introduces a benchmark for testing how coding agents recover hidden requirements in repository-change tasks. A stateful user simulator reveals requirements in response to questions, evidence, and explanations. The paper compares nine coding-agent configurations using interaction scores, final-report fidelity, and hidden tests.

Critical strengths
  • The benchmark uses real repository changes and links hidden requirements to a reference specification and executable tests. Separating the user’s target from disclosure rules makes controlled comparisons possible.
  • The study evaluates requirement recovery, final documentation, and code behavior separately. This helps distinguish agreement with the simulated user from successful implementation.
  • The authors examine sensitivity to token calibration and task selection. They also report missing data and acknowledge limits on simulator acceptance and causal interpretation. These qualifications help readers interpret the results.
Critical weaknesses
  • PlanScore can favor later discovery without earlier resolution (C.1). For a decision resolved in round 8, discovery in round 4 scores 22.36; discovery in round 8 scores 35.36. It also counts messages rather than individual questions, so it does not measure question burden. These choices weaken its interpretation as a measure of planning efficiency. The study does not establish what timing adds beyond the fraction of requirements resolved.
  • The evidence for planning quality has a narrower scope than the term suggests. PlanScore records trigger and unlock events for Experience and Credence requirements; direct-answer Search requirements are excluded (C.1). Gates do not independently verify that experiments ran or that causal explanations are correct (A.3). Final reports describe decisions after implementation, rather than plans recorded beforehand. Incomplete matching between reports and interaction records also limits diagnosis of omissions: a requirement may never have been obtained, or may have been clarified but left undocumented.
  • I could not find task and repository counts, runs per model, agent prompts, or budget settings in the manuscript. The paper also does not summarize how many evaluated tasks completed each required validation check (A.1–A.2). These omissions make it difficult to assess benchmark coverage and whether model comparisons use comparable conditions.
  • Report fidelity depends on GLM 5.3 without independent human validation (C.3). The numerical archive omits the report and reference text and the exact judge prompt, which prevents independent checks of label correctness. A single uncertain label (needs_review) makes an entire report unscored. Models can therefore retain different repetitions even when evaluated on the same tasks. Label errors or selective exclusions could affect rankings and the finding that omissions exceed contradictions.
Questions for authors
  • Can you provide a protocol table with task and repository counts, runs per model, prompts, agent frameworks, budgets, and completed validation checks?
  • Using existing traces, do rankings hold under a score that cannot reward later discovery when the resolution round stays fixed? Please compare against completion alone and report individual-question counts. Can a small audit check whether gate decisions correctly apply the evidence and explanation requirements?
  • Can blinded human reviewers check whether the reference requirements cover the full specification and whether report labels match the text? Please include review-pending reports and distinguish missing reports from uncertain judgments by model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.