- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper transforms the 500 SWE-bench Verified tasks into interactive coding tasks. An agent initially receives a shortened query and must obtain further information from a persona-conditioned user simulator that knows the issue text but cannot inspect or execute code. A self-revision step checks simulated replies for violations of the intended setting. The paper also proposes a schema-guided prompting strategy and evaluates task resolution, dialogue naturalness, and coherence. Across four backbones, the schema-guided agent achieves 46.9% average resolution, versus 44.1% for OH Interactive and 32.9% for the basic OpenHands adaptation. Human annotation evaluates both simulator rule adherence and agreement with dialogue-quality judges.
Critical strengths
- Evaluating interaction at repository scale is worthwhile. The benchmark retains executable patch evaluation while adding a controllable dialogue channel, allowing task success and communication quality to be studied separately.
- Restricting simulator knowledge to the original issue text, without providing reference patches or evaluation tests, is a sensible design choice. The simulator prompts and self-revision checks explicitly address fabricated execution, unavailable information, and excessive disclosure.
Critical weaknesses
-
The simulator validation establishes compliance with authored rules more directly than realism as a user model. Table 4's 97.5% violation-free dialogues is encouraging, but the criteria evaluate fidelity to issue knowledge, goal adherence, and the imposed environment. They do not show that real users disclose information, answer multi-part questions, or become impatient in the same way. Appendix B explicitly allows the simulator to ignore some questions, forbids volunteering unrequested information, and removes critical details from the initial query. These choices plausibly favor particular dialogue policies. Figure 10 illustrates why this matters: the simulator answers only selected questions, and the resulting missing information changes the patch. This is a valid interaction failure under the benchmark rules, but its frequency and ranking implications in real use remain unknown. A small human-interaction study or ranking sensitivity across simulator models, disclosure policies, and persona assignments would materially strengthen the benchmark's external validity.
-
The experiments do not isolate adaptive dialogue planning from recovering deliberately withheld issue information. The initial-query construction intentionally removes useful details, and the unavailable-user ablation removes access to that information entirely. Its performance drop therefore demonstrates the value of the information channel, but does not by itself establish that sophisticated multi-turn dialogue planning is required. A fixed sequence of focused questions about symptoms, expected behavior, reproduction, and errors would be a useful control, followed by the same coding agent and matched interaction budget. A replay condition supplying information obtained in a prior dialogue could further separate elicitation from coding. Figure 2's association between information-seeking and success is descriptive, not causal evidence that asking more questions improves a given agent.
Questions for authors
- How stable are agent rankings under a different simulator, disclosure policy, or persona seed, and how do simulated exchanges compare with a small set of real user interactions?
- How does schema guidance compare with a fixed clarification checklist under the same coding and dialogue budgets?
- What run counts and aggregation produce Table 1, and what are the paired confidence intervals for the gains over OH Interactive?
- Are human-rating samples paired by task/persona, and can the exact agent prompt, annotation protocol, and fixed benchmark artifacts be provided?
4 comments
Response to W1-2
Response to W1
We agree that our simulator validation establishes adherence to the behaviors our benchmark is designed to model, rather than realism as a model of real users. We do not claim our simulator faithfully reproduces real user behavior, and have scoped the claims in our paper accordingly (see our response to Reviewer 5NBG, W1). We argue that perfect faithfulness to the behavior of real users is out of scope for this paper and a challenging, moving target for the field generally.Instead, we propose a well-designed benchmark for evaluating key goal-oriented dialogue behaviors in coding agents that we believe are necessary but not sufficient for strong, interactive coding agents. We have made language edits to claims in this paper to reflect this.
On the disclosure rules: we do not believe there is any favoring of dialogue policies by our disclosure rules invalidates their use. Using the example in Figure 10 you reference, in which an agent overwhelms a user with questions: a user which does not answer a complete and lengthy list of questions is realistic: answering each questions costs the user time and effort. Importantly though, an agent which asks questions in this way is not necessarily penalized for it unfairly in terms of task resolution: if it happens that a critical detail is implied in the answer to a question the user ignored, the agent can and should just ask again (our user is not designed to refuse questions). A policy which confuses an unanswered question with a refused one is not a correct policy and thus not unfairly penalized. Figure 10 illustrates exactly this: GPT-5 asks many questions, the user answers only two, and GPT-5 declines to follow up on the unanswered question that determines the correct patch. We agree the frequency and ranking implications of such failures with real users remain unknown, however measuring the behavior of real coding agent users is out of scope for this paper, and this user behavior is changing rapidly as agents evolve.
On sensitivity to simulator models, persona assignments, and a human study: please see our responses to Reviewer 5NBG. In Q4, we report resolve rates by persona for each agent, finding only minor differences across personas relative to differences across agents, and discuss our choice of a single, human-validated user simulator. In W1, we discuss the limitations of what a human study could contribute in this setting.
Response to W2
Thank you for the feedback, we have updated the paper to clarify the contribution of the schema-guided agent and its structured dialogue planning instructions. We do not claim that structured dialogue planning in the explicit form we instruct for is necessary for strong coding agents. Instead, our core contribution is a benchmark which carefully constructs problems such that dialogue moves are required to resolve them. As you note, our user-unavailable ablation supports this construction, demonstrating that far fewer problems are solvable without knowledge gained from the user through dialogue moves.
The contribution of our schema-guided agent is not intended as next-generation coding agent, only a targeted modification to agents that demonstrates the dialogue management capabilities our benchmark evaluates for are steerable in LLM agents. The benchmark remains our core contribution. As our method simply provides instructions to OpenHands for conducting structured dialogue planning, we argue its improvement over existing agent suggests existing agents are not adequately reasoning through dialogues.
On a fixed sequence of focused questions: while this could be an informative baseline, we believe the question it would answer is whether it is better to allow an LLM agent to dynamically plan its dialogue (our self-generated schemas), or to follow a prescriptive one. While an interesting question, we do not believe it is critical to the scope of our paper. Our schema-guided agent instead answers a different question more related to the core contribution of our benchmark: if we design a benchmark for evaluating the ability of agents to conduct goal-oriented dialogues, do methods grounded in the theory of goal-oriented dialogues improve performance?
On a replay condition: we agree this would help separate elicitation from coding ability, and note it as a direction for future analysis.
On Figure 2: we agree this association is correlational. We have revised the paper to describe it as such, retitling the paragraph "Information-Seeking is Associated with Task Resolution" and ensuring the text does not describe the relationship as causal.
Thank you!
Thank you again for the very detailed review! We apologize for our late rebuttal, and do not anticipate a response in time for us to reply again. If time permits during the AC-Reviewer discussion period, we would still really appreciate your consideration of our responses and how they might affect your scores, and would be grateful for any comments on our responses you might have. Thank you again for your feedback on our paper!
Thank you for the detailed response. Given the benchmark contribution and the new analysis, I am updating my recommendation to weak acceptance.
Response to Q1-4
Q1: Ranking stability across simulators, disclosure policies, and persona seeds; comparison with real users
Personas: each persona is assigned to a given problem, so all agents face the same (persona, task) pairs. Generally, we find only minor differences in task-performance across personas (below, resolve rates averaged over the 4 models; n is the number of tasks assigned to each persona):
While re-runs assigning different personas to each task might more explicitly analyze this sensitivity, we find performance correlates more strongly with agent choice than with assigned persona, and believe the additional precision provided by a persona re-assigned experiment would not justify its cost.
Simulator models: we aim to provide a single open-weight user simulator validated with human annotation (Section 8.3, Table 4). While the benchmark might behave differently under a different simulator LLM, all evaluated systems must use the same simulator for fair comparison. Gollowing prior work, we fix a single user simulator across all compared agents (Vijayvargiya et al., 2026; Yao et al., 2024) and instead validate its reliability (Sec 8.3, Table 4).
Real users: please see our response to Reviewer 5NBG (W1), where we discuss the limitations of what a human study could contribute in this setting, and how we have scoped our claims accordingly.
Q2: Schema guidance vs. a fixed clarification checklist under matched budgets
Please see our response to W2.
Q3: Run counts, aggregation, and paired confidence intervals
To quantify uncertainty in the performance differences of Table 1, we conduct a paired bootstrap analysis with 10,000 resamples of the 500 benchmark tasks. For the average, the same resampled tasks are used for all models at once. We report 95% percentile confidence intervals and two-sided p-values.
Ours vs. OH Interactive (% Resolved, n = 500 tasks, paired bootstrap B = 10,000)
As shown above: two differences between model x agent pairs do not exclude zero in the 95% CI, suggesting one cannot conclude from our current experiments that the schema-guided agent performs better than OH Interactive for GPT-5, nor that the OH Interactive agent performs better than the schema-guided with Devstral 2. The two remaining advantages for the schema-guided agent are significant. For this averaged group of 4 models, we find the averaged performance advantage of the schema-guided agent over OH Interactive is also significant.
As shown below: all improvements of the schema-guided variant over the OpenHands agent hold:
Ours vs. OpenHands (% Resolved, n = 500 tasks, paired bootstrap B = 10,000)
Q4: Pairing of human-rating samples; benchmark artifacts
Pairing: Yes, all human-annotated dialogues are paired by task: 30 tasks x 12 (model, agent) settings for a total of 360 annotated dialogues.
We will release the full set of annotation rubrics and other benchmark artifacts in a code release paired with the paper.