acceptodds
Under review as a conference paper at ICLR 2027

Dialogue-SWEBench: A Benchmark for Dialogue-Driven Coding Agents

Abstract

AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue-SWEBench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems through dialogue with a user. We design a novel, persona-grounded user simulator to support our task evaluation, and augment our task evaluation with automatic evaluations of dialogue quality. We also propose a new schema-guided agent, aimed at improving the dialogue planning capabilities of off-the-shelf coding agents, which improves over strong baselines by 3-14%. Our results indicate that better coding models do not always correspond to better dialogue models, suggesting that dialogue capability is a distinct and currently understudied dimension of coding agent performance.

Primary area:
datasets and benchmarks
Keywords:
dialogue, benchmark, coding agents, agent, SWE Agent, SWE

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4NPSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper transforms the 500 SWE-bench Verified tasks into interactive coding tasks. An agent initially receives a shortened query and must obtain further information from a persona-conditioned user simulator that knows the issue text but cannot inspect or execute code. A self-revision step checks simulated replies for violations of the intended setting. The paper also proposes a schema-guided prompting strategy and evaluates task resolution, dialogue naturalness, and coherence. Across four backbones, the schema-guided agent achieves 46.9% average resolution, versus 44.1% for OH Interactive and 32.9% for the basic OpenHands adaptation. Human annotation evaluates both simulator rule adherence and agreement with dialogue-quality judges.

Critical strengths
  1. Evaluating interaction at repository scale is worthwhile. The benchmark retains executable patch evaluation while adding a controllable dialogue channel, allowing task success and communication quality to be studied separately.
  2. Restricting simulator knowledge to the original issue text, without providing reference patches or evaluation tests, is a sensible design choice. The simulator prompts and self-revision checks explicitly address fabricated execution, unavailable information, and excessive disclosure.
Critical weaknesses
  1. The simulator validation establishes compliance with authored rules more directly than realism as a user model. Table 4's 97.5% violation-free dialogues is encouraging, but the criteria evaluate fidelity to issue knowledge, goal adherence, and the imposed environment. They do not show that real users disclose information, answer multi-part questions, or become impatient in the same way. Appendix B explicitly allows the simulator to ignore some questions, forbids volunteering unrequested information, and removes critical details from the initial query. These choices plausibly favor particular dialogue policies. Figure 10 illustrates why this matters: the simulator answers only selected questions, and the resulting missing information changes the patch. This is a valid interaction failure under the benchmark rules, but its frequency and ranking implications in real use remain unknown. A small human-interaction study or ranking sensitivity across simulator models, disclosure policies, and persona assignments would materially strengthen the benchmark's external validity.

  2. The experiments do not isolate adaptive dialogue planning from recovering deliberately withheld issue information. The initial-query construction intentionally removes useful details, and the unavailable-user ablation removes access to that information entirely. Its performance drop therefore demonstrates the value of the information channel, but does not by itself establish that sophisticated multi-turn dialogue planning is required. A fixed sequence of focused questions about symptoms, expected behavior, reproduction, and errors would be a useful control, followed by the same coding agent and matched interaction budget. A replay condition supplying information obtained in a prior dialogue could further separate elicitation from coding. Figure 2's association between information-seeking and success is descriptive, not causal evidence that asking more questions improves a given agent.

Questions for authors
  1. How stable are agent rankings under a different simulator, disclosure policy, or persona seed, and how do simulated exchanges compare with a small set of real user interactions?
  2. How does schema guidance compare with a fixed clarification checklist under the same coding and dialogue budgets?
  3. What run counts and aggregation produce Table 1, and what are the paired confidence intervals for the gains over OH Interactive?
  4. Are human-rating samples paired by task/persona, and can the exact agent prompt, annotation protocol, and fixed benchmark artifacts be provided?
4 comments
AuthorsOpenReview ↗

Response to Q1-4

Q1: Ranking stability across simulators, disclosure policies, and persona seeds; comparison with real users

Personas: each persona is assigned to a given problem, so all agents face the same (persona, task) pairs. Generally, we find only minor differences in task-performance across personas (below, resolve rates averaged over the 4 models; n is the number of tasks assigned to each persona):

PersonanOpenHands (%)OH Interactive (%)Ours (%)
Pragmatic11235.543.147.8
Cautious10133.745.350.0
Perfectionist9830.444.145.2
Lazy engineer9532.948.246.6
Introverted9431.439.644.1
All50032.944.146.8

While re-runs assigning different personas to each task might more explicitly analyze this sensitivity, we find performance correlates more strongly with agent choice than with assigned persona, and believe the additional precision provided by a persona re-assigned experiment would not justify its cost.

Simulator models: we aim to provide a single open-weight user simulator validated with human annotation (Section 8.3, Table 4). While the benchmark might behave differently under a different simulator LLM, all evaluated systems must use the same simulator for fair comparison. Gollowing prior work, we fix a single user simulator across all compared agents (Vijayvargiya et al., 2026; Yao et al., 2024) and instead validate its reliability (Sec 8.3, Table 4).

Real users: please see our response to Reviewer 5NBG (W1), where we discuss the limitations of what a human study could contribute in this setting, and how we have scoped our claims accordingly.

Q2: Schema guidance vs. a fixed clarification checklist under matched budgets

Please see our response to W2.

Q3: Run counts, aggregation, and paired confidence intervals

To quantify uncertainty in the performance differences of Table 1, we conduct a paired bootstrap analysis with 10,000 resamples of the 500 benchmark tasks. For the average, the same resampled tasks are used for all models at once. We report 95% percentile confidence intervals and two-sided p-values.

Ours vs. OH Interactive (% Resolved, n = 500 tasks, paired bootstrap B = 10,000)

ModelOH Interactive (%)Ours (%)Δ (pp)95% CIp-value
GPT-5-mini54.258.8+4.6[+0.8, +8.6]0.024
GPT-554.658.0+3.4[+0.0, +6.8]0.064
Devstral 2 Small (24B)42.038.2-3.8[-7.8, +0.2]0.073
Qwen 3 Coder (30-A-3B)25.432.2+6.8[+3.0, +10.6]< 0.001
Average (4 models)44.146.8+2.8[+0.8, +4.8]0.006

As shown above: two differences between model x agent pairs do not exclude zero in the 95% CI, suggesting one cannot conclude from our current experiments that the schema-guided agent performs better than OH Interactive for GPT-5, nor that the OH Interactive agent performs better than the schema-guided with Devstral 2. The two remaining advantages for the schema-guided agent are significant. For this averaged group of 4 models, we find the averaged performance advantage of the schema-guided agent over OH Interactive is also significant.

As shown below: all improvements of the schema-guided variant over the OpenHands agent hold:

Ours vs. OpenHands (% Resolved, n = 500 tasks, paired bootstrap B = 10,000)

ModelOpenHands (%)Ours (%)Δ (pp)95% CIp-value
GPT-5-mini34.258.8+24.6[+20.0, +29.2]< 1e-4
GPT-547.258.0+10.8[+6.6, +15.0]< 1e-4
Devstral 2 Small (24B)26.838.2+11.4[+7.0, +15.8]< 1e-4
Qwen 3 Coder (30-A-3B)23.232.2+9.0[+5.2, +13.0]< 1e-4
Average (4 models)32.946.8+14.0[+11.4, +16.5]< 1e-4

Q4: Pairing of human-rating samples; benchmark artifacts

Pairing: Yes, all human-annotated dialogues are paired by task: 30 tasks x 12 (model, agent) settings for a total of 360 annotated dialogues.

We will release the full set of annotation rubrics and other benchmark artifacts in a code release paired with the paper.

AuthorsOpenReview ↗

Response to W1-2

Response to W1

We agree that our simulator validation establishes adherence to the behaviors our benchmark is designed to model, rather than realism as a model of real users. We do not claim our simulator faithfully reproduces real user behavior, and have scoped the claims in our paper accordingly (see our response to Reviewer 5NBG, W1). We argue that perfect faithfulness to the behavior of real users is out of scope for this paper and a challenging, moving target for the field generally.Instead, we propose a well-designed benchmark for evaluating key goal-oriented dialogue behaviors in coding agents that we believe are necessary but not sufficient for strong, interactive coding agents. We have made language edits to claims in this paper to reflect this.

On the disclosure rules: we do not believe there is any favoring of dialogue policies by our disclosure rules invalidates their use. Using the example in Figure 10 you reference, in which an agent overwhelms a user with questions: a user which does not answer a complete and lengthy list of questions is realistic: answering each questions costs the user time and effort. Importantly though, an agent which asks questions in this way is not necessarily penalized for it unfairly in terms of task resolution: if it happens that a critical detail is implied in the answer to a question the user ignored, the agent can and should just ask again (our user is not designed to refuse questions). A policy which confuses an unanswered question with a refused one is not a correct policy and thus not unfairly penalized. Figure 10 illustrates exactly this: GPT-5 asks many questions, the user answers only two, and GPT-5 declines to follow up on the unanswered question that determines the correct patch. We agree the frequency and ranking implications of such failures with real users remain unknown, however measuring the behavior of real coding agent users is out of scope for this paper, and this user behavior is changing rapidly as agents evolve.

On sensitivity to simulator models, persona assignments, and a human study: please see our responses to Reviewer 5NBG. In Q4, we report resolve rates by persona for each agent, finding only minor differences across personas relative to differences across agents, and discuss our choice of a single, human-validated user simulator. In W1, we discuss the limitations of what a human study could contribute in this setting.

Response to W2

Thank you for the feedback, we have updated the paper to clarify the contribution of the schema-guided agent and its structured dialogue planning instructions. We do not claim that structured dialogue planning in the explicit form we instruct for is necessary for strong coding agents. Instead, our core contribution is a benchmark which carefully constructs problems such that dialogue moves are required to resolve them. As you note, our user-unavailable ablation supports this construction, demonstrating that far fewer problems are solvable without knowledge gained from the user through dialogue moves.

The contribution of our schema-guided agent is not intended as next-generation coding agent, only a targeted modification to agents that demonstrates the dialogue management capabilities our benchmark evaluates for are steerable in LLM agents. The benchmark remains our core contribution. As our method simply provides instructions to OpenHands for conducting structured dialogue planning, we argue its improvement over existing agent suggests existing agents are not adequately reasoning through dialogues.

On a fixed sequence of focused questions: while this could be an informative baseline, we believe the question it would answer is whether it is better to allow an LLM agent to dynamically plan its dialogue (our self-generated schemas), or to follow a prescriptive one. While an interesting question, we do not believe it is critical to the scope of our paper. Our schema-guided agent instead answers a different question more related to the core contribution of our benchmark: if we design a benchmark for evaluating the ability of agents to conduct goal-oriented dialogues, do methods grounded in the theory of goal-oriented dialogues improve performance?

On a replay condition: we agree this would help separate elicitation from coding ability, and note it as a direction for future analysis.

On Figure 2: we agree this association is correlational. We have revised the paper to describe it as such, retitling the paragraph "Information-Seeking is Associated with Task Resolution" and ensuring the text does not describe the relationship as causal.

AuthorsOpenReview ↗

Thank you!

Thank you again for the very detailed review! We apologize for our late rebuttal, and do not anticipate a response in time for us to reply again. If time permits during the AC-Reviewer discussion period, we would still really appreciate your consideration of our responses and how they might affect your scores, and would be grateful for any comments on our responses you might have. Thank you again for your feedback on our paper!

Thank you for the detailed response. Given the benchmark contribution and the new analysis, I am updating my recommendation to weak acceptance.

Reviewer_o9X3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper turns SWE-bench Verified (including 500 tasks) into a dialogue benchmark. Each task starts with a short, underspecified query derived from the issue title.

Critical strengths
  1. This is a practical problem with a clean design. Real users start vague, yet coding benchmarks assume a full specification. Because the simulator knows only what the issue says, the benchmark measures elicitation rather than the simulator's imagination.

  2. The simulator and judges are validated rather than assumed.

  3. Self-revision raises the share of defect-free dialogues from 82.5% to 97.5%.

Critical weaknesses
  1. The proposed agent's gains are within single-run noise. Each configuration is run once. On 500 tasks, that gives roughly ±4.4 points of binomial uncertainty. Against OH Interactive, the schema-guided agent's gains are +4.6, +3.4, −3.6 and +6.9 points: it is worse on Devstral, and only the Qwen gain clearly exceeds the noise.

  2. The abstract's "3–14%" compares averages and hides that loss. The claim that "better coding models are not better dialogue models" partly rests on GPT-5 scoring 58.0% against GPT-5-mini's 58.8%, which is too small a gap to support it. The agent is one of the paper's two contributions, so this weighs heavily.

  3. There is no simple baseline that asks one open question before starting. Only OpenHands has full-specification results. The schema-guided agent should be run on the original SWE-bench Verified too.

  4. Important details missed: how does the schema-guided agent perform on the original, fully specified SWE-bench Verified? How often does the base Claude Code agent call message_user? What fraction of tasks does each agent solve with no user turns? The second number also bears on contamination. A per-persona breakdown, a second simulator model, or a small study with real users would strengthen the claim that the benchmark predicts real interactive performance.

Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

Response to W1

Thank you for the thoughtful review! We detail our responses to each weakness below:

W1: Significance of Results

The 4.4 point binomial uncertainty treats each agent's resolve rate independently. Since all agents are evaluated on the same 500 tasks, a paired comparison gives a more precise estimate of the difference between them. To quantify uncertainty in these differences, we conduct a paired bootstrap analysis with 10,000 resamples of the 500 benchmark tasks: each resample draws 500 tasks with replacement and recomputes the difference in resolve rate between Ours and the baseline on the same drawn tasks. For the average, the same resampled tasks are used for all models at once. We report 95% percentile confidence intervals and two-sided p-values.

Ours vs. OH Interactive (% Resolved, n = 500 tasks, paired bootstrap B = 10,000)

ModelOH Interactive (%)Ours (%)Δ (pp)95% CIp-value
GPT-5-mini54.258.8+4.6[+0.8, +8.6]0.024
GPT-554.658.0+3.4[+0.0, +6.8]0.064
Devstral 2 Small (24B)42.038.2-3.8[-7.8, +0.2]0.073
Qwen 3 Coder (30-A-3B)25.432.2+6.8[+3.0, +10.6]< 0.001
Average (4 models)44.146.8+2.8[+0.8, +4.8]0.006

As shown above: two differences between model x agent pairs do not exclude zero in the 95% CI, suggesting one cannot conclude from our current experiments that the schema-guided agent performs better than OH Interactive for GPT-5, nor that the OH Interactive agent performs better than the schema-guided with Devstral 2. The two remaining advantages for the schema-guided agent are significant. For this averaged group of 4 models, we find the performance advantage of the schema-guided agent over OH Interactive holds.

As shown below: all improvements of the schema-guided variant over the plain OpenHands agent hold:

Ours vs. OpenHands (% Resolved, n = 500 tasks, paired bootstrap B = 10,000)

ModelOpenHands (%)Ours (%)Δ (pp)95% CIp-value
GPT-5-mini34.258.8+24.6[+20.0, +29.2]< 1e-4
GPT-547.258.0+10.8[+6.6, +15.0]< 1e-4
Devstral 2 Small (24B)26.838.2+11.4[+7.0, +15.8]< 1e-4
Qwen 3 Coder (30-A-3B)23.232.2+9.0[+5.2, +13.0]< 1e-4
Average (4 models)32.946.8+14.0[+11.4, +16.5]< 1e-4

Thus in the first table, we find the improvement on 2/4 models for the schema-guided agent over OH Interactive to be statistically significant (GPT 5 mini, Qwen), with one improvement narrowly not significant (GPT 5). Importantly, the disadvantage we observe on Devstral is also not significant, and the average improvement is. In the second table, we find all advantages over OpenHands are both sizable and significant.

AuthorsOpenReview ↗

Response to W2

As we find in our response to W1, our average advantage is significant, and we report all advantages and disadvantages clearly in Table 1, with the lone disadvantage (OH Interactive > Schema-Guided on Devstral) not being statistically significant. We also find the difference between GPT 5 and GPT 5 mini to not be significant as the small differences in resolve rate suggests and re-frame our claims accordingly. Importantly, we think this comparison is still worthy of discussion in the paper: GPT 5 is widely understood to be a stronger model and better coding model than its smaller counterpart GPT 5 Mini. The lack of statistical significance only shows that we cannot conclude that GPT 5 mini is strictly better than GPT 5 at this task. We believe the prevailing assumption would be that the performance of GPT 5 would be significantly better than GPT 5 mini, and discuss this counterintuitive observation without concluding GPT 5 mini is greater than GPT 5 in this setting (instead using words like "GPT 5 mini rivals GPT 5").

Concerning the schema-guided agent as contribution: we believe the core contribution of our work is a dialogue-driven benchmark for coding agents. As discussed in our response to another review, we have re-framed the contribution of the schema-guided agent around the purpose of what it demonstrates with respect to our benchmark. We believe our evaluation models a key capability required for successful task-oriented dialogue: refining a user's open-ended dialogue intent into a well-formed solvable goal through conversation, then solving that problem. Our schema-guided agent demonstrates that coding agents can be steered through prompting to better solve these dialogue problems through structured dialogue planning, an approach which draws on classic theories of goal-oriented dialogue. We believe our bootstrap analysis is sufficient to support the claims we make about this agent in the paper.

AuthorsOpenReview ↗

Response to W3

On a simple baseline that asks questions before starting: we believe the OH Interactive baseline of (Vijayvargiya et al., 2026) serves this role. It uses the same OpenHands agent and tools as our other systems, modifying only the prompt to request that the agent gather requirements from the user before starting:

I have not provided all the necessary details about the issue and I have some hidden details that are helpful. Please ask me specif[i]c questions using non-code commands to gather the relevant information that I have to help you solve the issue. Ensure you have all the details you require to solve the issue.

[...]

  1. As a first step, look at the issue and ask me questions to get all the necessary details about the issue. You can also ask me questions if you run into a problem in later steps.

We believe this is baseline is instructed to ask such questions, and its improvement over the off-the-shelf OpenHands reflects this. Our schema-guided agent further instructs an agent with the same toolset to plan its dialogue moves using self-generated schemas.

On full-specification results: we include the full-specification OpenHands results (Appendix C) only as a reference point, it does not contribute to the claims about agents evaluated in our benchmark setting. Our schema-guided agent is designed for the dialogue setting: its prompt instructs the agent to establish common ground with the user using self-generated dialogue schemas before solving the problem, and in the full-specification setting there is no user to converse with. We therefore believe its full-specification performance would not inform the comparisons our benchmark is designed to make, which are between agents in the dialogue setting, nor would it contribute meaningfully to any of the claims we make in the paper.

Reviewer_5NBGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Dialogue-SWEBench, adapting 500 SWE-Bench Verified tasks to evaluate coding agents through dialogue with simulated users. It proposes a grounded user simulator with self-revision and a schema-guided agent that tracks missing requirements. Experiments assess task resolution and dialogue quality, with schema guidance improving average resolution from 44.1% to 46.9% over the interactive baseline.

Critical strengths
  • The paper addresses an important evaluation gap. It makes understanding incomplete requests and eliciting requirements explicit parts of coding-agent evaluation. Adapting 500 existing software engineering tasks from SWE-Bench Verified provides a substantial testbed for studying these capabilities in the context of real repositories. This addresses an important aspect of how coding assistants are used.

  • The agent must produce code changes that pass execution tests. This connects dialogue behavior to task completion and gives the benchmark a concrete success criterion. The simulator’s knowledge comes from the issue description, without access to the reference patch or evaluation tests, which is a useful design choice for limiting solution leakage.

Critical weaknesses
  • The benchmark validation does not establish transfer to real users. The evaluation provides a credible initial demonstration through execution-based task scoring, comparisons across models and agent frameworks, and human checks of the dialogue judge and simulator. These checks provide evidence about performance and consistency within the designed environment. However, the validation does not establish that the simulator reproduces real users’ behavior or that agent scores and rankings predict effectiveness in actual human-agent collaboration. The authors acknowledge the limitation of simulated users, but this transfer question remains unresolved. A focused evaluation with real users, comparing agent performance and rankings against those obtained with the simulator, would address the most important gap. Sensitivity tests using different simulator models, personas, and amounts of missing information would further establish the robustness of the benchmark’s conclusions. These evaluations would help justify broader claims about real-world interactive coding; the current evidence is strongest as an initial demonstration of the proposed simulated setting.

  • The initial-query pipeline deliberately removes important details, and the simulator is instructed to withhold information unless asked. The resulting task primarily tests whether an agent can recover information already present in an existing issue description. This is a useful capability, but it covers a narrower form of collaboration than jointly discovering requirements, correcting mistaken assumptions, or asking users to investigate and test behavior. The user-unavailable ablation demonstrates the value of access to additional information; it does not isolate which conversational strategies are effective. Similarly, the association between question frequency and resolution is correlational. Naturally written initial requests, controlled variation in the amount of missing information, and a baseline that makes a standard request for relevant details would help clarify what the benchmark rewards.

  • The experiments do not isolate what makes schema guidance effective. The gains could come from clearer instructions to gather requirements or from maintaining structured dialogue state. A matched checklist baseline and an ablation removing state updates would clarify the schema’s specific contribution.

  • The average improvement over the interactive baseline is 2.8 percentage points, but uncertainty estimates are missing and benefits vary across models. Repeated runs and paired comparisons on the same tasks would help establish whether the improvement is stable.

  • The efficiency evaluation captures only part of the interaction cost. Monetary costs exclude simulator inference, while turn counts and dialogue ratings only indirectly measure human effort. Reporting total costs and measuring answering time, unnecessary questions, and repeated requests would strengthen the efficiency claims.

Questions for authors
  • The simulator evaluation establishes adherence to the specified rules. What evidence supports the transfer of agent performance and rankings to interactions with real users? If such evidence is unavailable, how should the benchmark’s claims about real-world collaboration be scoped?

  • Can you isolate the benefit of maintaining a structured dialogue state from the benefit of clearer prompting? A matched baseline that requests the same requirement details without maintaining a schema would help clarify the contribution.

  • The average improvement over OH Interactive is 2.8 percentage points, with variation across models. Can you provide paired uncertainty estimates and, where feasible, repeated-run results to establish the stability of these gains?

  • How sensitive are agent rankings to the simulator model, assigned persona, and amount of information removed from the initial request? What evidence supports robustness beyond the particular simulation configuration used?

  • Can you report total evaluation costs, including simulator generation and self-revision, and clarify which efficiency conclusions are supported by the current measurements? How adequately do dialogue turns and quality ratings capture the effort required from a human user?

  • Some relevant related works are omitted from the paper. Example paper is CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding (Luo et al., ICML 2026). Please discuss CentaurEval and clarify the distinct contributions of Dialogue-SWEBench, particularly its simulated-user protocol, repository-level dialogue evaluation, and dialogue-quality assessment.

8 comments
AuthorsOpenReview ↗

Response to W1

Thank you for the thorough review! We detail our responses to each weakness below:

On comparisons to real human users: We first argue that perfect alignment with real human users is not directly measurable for an automatically evaluated benchmark such as ours: the distribution of real human user behavior with coding agents is strongly dependent on that user's experience with coding agents, the qualities of the agent itself, and the context in which they are working. We therefore believe a real user's behavior can only be understood in that context, and their behavior with respect to a prospective/future agent cannot be simulated from their experience with a prior one (e.g, how a user asks Claude Code a question or otherwise converses with it depends on their current understanding of its capabilities and beliefs in what it can accomplish). For this reason, we argue the value of our user simulator is not in perfectly replicating the behavior of some real user, but instead modeling realistic behaviors for evaluating a set of general dialogue capabilities/desiderata our benchmark measures: when faced with an open-ended dialogue request which is not yet a well-formed problem, we would like agents to be able to infer what details they do not yet know that are needed to solve the problem, identify which of those the user should be responsible for knowing and providing, and conduct a natural and coherent dialogue that results in a solvable problem and its solution.

On the unreliability of a human study for this setting: We further argue that the conclusions one could draw from a human study are limited. An automatic benchmark such as ours relies on well-defined problems whose answers can be evaluated with execution tests. This creates a gap between feasible human studies and what is evaluated. We consider a few possible studies below and their limitations:

  • Real humans assigned existing, evaluable problems: assigning SWE-Bench problems to human dialogue users who may not understand their context only evaluates a human's ability to simulate another's dialogue, not what a real conversation with the issue submitter might look like. A faithful human study for these problems would require recruiting the original issue submitters before they posted the GitHub issue or solved the problem on their own.
  • Real humans with their own problems: while more realistic, this setting leads to a substantial distribution shift from what can be evaluated automatically. Further, the execution tests needed to evaluate correctness of these real user problems would not be available.

While a small scale human study might allow one to rank agents we evaluate by human preference, this would apply only to the agents we evaluate in this paper, not the future agents we wish to evaluate with our benchmark. We therefore position our contribution as evaluating a useful and important set of dialogue capabilities, not necessarily all the factors which go into evaluating an interactive coding agent.

Scope of our user simulator's behavior: Importantly, we do not claim to perfectly model real users, only to model an important set of realistic behaviors that allow us to produce a useful evaluation of agents in a dialogue setting:

  • Provides a natural language open-ended intent which does not yet fully communicate a solvable problem
  • Responds faithfully to questions using real knowledge and adhering to real evaluable requirements
  • Stays on task, and does not accept or deviate to un-evaluated tasks
  • Respects the environmental limitations of interacting with a coding agent only through chat

We follow the first point by construction in Section 3.3 ("Crafting initial queries"; details in Appendix B) and evaluate our simulator's behavior against the last 3 in Table 4 (Section 8.3).

Paper Improvements

To address this weakness in our paper, we've identified any places in the paper that violated the scope of this claim and improved the writing to better align with this aim. In the intro, we add:

Our evaluation models a key capability required for successful task-oriented dialogue: refining a user's open-ended dialogue intent into a well-formed solvable goal through conversation, then solving that problem. We evaluate this process with a controlled user simulator and the naturalness and coherence of an agent's dialogue contributions with an annotation-validated LLM Judge.

And replace We evaluate whether our user simulator can effectively serve as a proxy for real-world users through human annotation. with We evaluate whether our user simulator adheres to the set of realistic behaviors our benchmark is designed to model: faithfulness to the issue, goal adherence, and environment adherence.

AuthorsOpenReview ↗

Response to W2

We agree our benchmark evaluates a "narrower form of collaboration than jointly discovering requirements, correcting mistaken assumptions, or asking users to investigate and test behavior." In response to this weakness and W1, we edit the paper to ensure our claims are scoped accordingly. We argue the capability we do evaluate is useful and a central requirement for effective task- or goal-oriented dialogue, as one might have with an interactive coding agent (see also our response to W1). In the Limitations, we note the additional skills a strong interactive agent should possess that our evaluation does not test:

Our benchmark also targets a specific form of collaboration: recovering task-relevant information that the user already knows but omitted from their initial request. Effective interactive agents likely require additional collaborative skills we do not evaluate, such as jointly discovering requirements the user has not yet formed, correcting a user's mistaken assumptions, or asking the user to investigate and test behavior in their own environment.

On naturally written initial requests: each initial query is manually reviewed during benchmark construction to verify that it is clear and faithful to the user's original intent (§3.3, Appendix B). Only 13% of queries required any edits, and these were minor revisions to give the query an issue-resolution framing rather than an open-ended question. As discussed in our response to W1, there are significant challenges in faithfully pairing an uninformed, naturally occurring initial query with a solvable, evaluable task for automatic evaluation.

On controlled variation in the amount of missing information: we argue the amount of removed information from a given task cannot be readily quantified for control: any given detail which may be in the issue text may also be discoverable by the agent in the repository, and the value of each detail to task resolution may vary wildly. Instead, we simply validate that our tasks do actually require engagement with the user in order to solve in our ablation which removes the user simulator after the first turn (Section 8.2, Figure 4). We observe a substantial performance drop in this setting: on a difficulty-stratified sample of 50 problems with our schema-guided agent, resolve rate falls by 10–28 percentage points across all four models (16.0 points on average), e.g. from 68.0% to 40.0% for GPT-5-mini and from 58.0% to 44.0% for GPT-5.

On a baseline that makes a standard request for relevant details: the OH Interactive baseline (Vijayvargiya et al., 2026) serves this role. Its prompt instructs the agent to first ask the user specific questions to gather all necessary details before solving the issue (more prompt details quoted in our response to W3), and it otherwise uses the same OpenHands agent and tools as each other evaluated system.

On the association between question frequency and resolution: we agree this analysis is correlational, and does not isolate which conversational strategies are effective. We revise §6 to describe it as such. Nevertheless, when combined with our ablation which removes useful user simulator responses (Section 8.2, Figure 4), we believe there is evidence to suggest that the information sought in dialogue contributes to task resolution, but we will be careful not to describe this relationship as causal.

AuthorsOpenReview ↗

Response to W3

We address this weakness in two ways:

  1. by clarifying the relationship between baselines (below)
  2. By re-scoping the contribution of the schema-guided agent (below, and updated in the paper)

Clarifying Baselines

First, we argue the existing baselines already sufficiently isolate the contribution of each variant. Each uses the OpenHands agent with only minor differences in the prompt. The main baseline uses the OpenHands agent off-the-shelf, with its default prompts and our additional message_user tool.

The OH Interactive variant of (Vijayvargiya et al., 2026) represents a baseline with clear instructions for gathering requirements, as it includes:

I have not provided all the necessary details about the issue and I have some hidden details that are helpful. Please ask me specif[i]c questions using non-code commands to gather the relevant information that I have to help you solve the issue. Ensure you have all the details you require to solve the issue.

[...]

  1. As a first step, look at the issue and ask me questions to get all the necessary details about the issue. You can also ask me questions if you run into a problem in later steps.

It uses the same set of tools as the other two baselines: its modification is the requested inclusion of instructions for requirement gathering.

Our schema-guided variant instead modifies the off-the-shelf OpenHands prompts to include instructions for a structured process to dialogue-planning through the maintenance of the structured dialogue state:

Phase 1. PROBLEM FORMULATION: Always start by establishing common ground with me on the details of the issue by continuing our dialogue. Based on our early discussions, draft a schema of the issue in your thoughts or reasoning to capture the key details of the issue:

SCHEMA: <ISSUE_TYPE>
 <SCHEMA KEY 1>: VALUE | UNKNOWN | KNOWN-UNKNOWN
 ...

 THOUGHT:

For example, if the <initial_query> informed you they are encountering a bug with a null pointer exception, your schema could look like:

SCHEMA: bug
  actual behavior: the user stated they are experiencing a null pointer exception
  expected behavior: UNKNOWN
  reproduction steps: UNKNOWN
  ... etc.

  THOUGHT: since we don't know how to reproduce the NPE yet, we need to ask the user for reproduction steps. We also need to confirm the expected behavior.

Use UNKNOWN for values you do not have enough information about yet. Use KNOWN-UNKNOWN for values you have confirmed the user does not know (i.e. the user explicitly stated they do not know this). ... Leave any unknown values as UNKNOWN, and ask me about these if it is plausible I would know them. If you ask many questions but only receive partial answers, don't assume the unanswered questions are KNOWN-UNKNOWN; instead, ask fewer, more targeted questions each turn.

We thus believe any difference between the description of the planning process and the structure requested are not separable, and the existing OH Interactive baseline suitably covers the requested method for 'clear instructions for requirement gathering.'

Re-scoping the contribution of the schema-guided agent

We wish to note that the core contribution of our paper is a benchmark approach for evaluating key dialogue capabilities in interactive SWE agents, not a method for agentic coding. While we propose a modification to existing agents and report it in our results: its purpose is to demonstrate that the dialogue-planning capabilities of coding agents is both steerable and measurable in our benchmark: our modifications to the OpenHands baseline are associated with improved performance (Table 1), more information seeking from the agent (Figure 2), and more coherent dialogues (Figure 3b). We have updated the paper to reflect this.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.