- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces COSWE, a benchmark for evaluating interactive planning in collaborative software engineering. Built from real-world GitHub pull requests, COSWE requires coding agents to clarify incomplete requirements, provide evidence, and explain technical trade-offs through interactions with a simulated user. The authors propose PlanScore to evaluate how efficiently agents discover and resolve missing requirements, along with additional metrics for token efficiency, report fidelity, and implementation correctness.
Critical strengths
- The idea is practical and meaningful.
- The Search, Experience, and Credence categorization is well motivated and provides a useful way to capture different types of user interaction in software engineering tasks.
Critical weaknesses
-
The design of PlanScore is not sufficiently justified. The use of reciprocal functions and a geometric mean appears largely heuristic and lacks empirical validation. Moreover, PlanScore conflates completion with interaction efficiency, making it difficult to determine whether score differences stem from higher completion rates or faster resolution. Reporting efficiency separately would help clarify the added value of PlanScore.
-
The benchmark statistics are insufficiently reported. The paper does not provide basic dataset statistics, such as the number of tasks and repositories, the distribution of S/E/C facets, or the number of hidden requirements and tests per task. Statistics on task difficulty and complexity are also missing.
-
The paper lacks controlled comparisons between agents operating without user interaction, with interactive clarification, and with access to complete requirements. This makes it difficult to disentangle the contribution of interactive planning from the agents’ inherent coding capabilities.
-
The experimental analysis could be more comprehensive. While the paper reports overall planning scores and their correlations with implementation performance, it provides limited analysis of how different facet types, task complexity, and interaction failure modes affect the results. A more fine-grained analysis would help explain where agents struggle and provide deeper insights into their interactive planning capabilities.
Questions for authors
-
Although COSWE defines three types of interaction facets (Search, Experience, and Credence), the evaluation only includes Experience and Credence facets (written in Section 4.1). Why are Search facets excluded from PlanScore and Clarification Completion, given that direct requirement clarification is also an important part of interactive planning?
-
What reasoning effort settings were used for each model? These settings are not reported in the paper.