acceptodds
Under review as a conference paper at ICLR 2027

UserSWE: Evaluating Coding Agents in Multi-Turn Conversations with Dynamic Users

Abstract

As coding agents become more capable, developers increasingly use them to fix bugs through multi-turn interactions. Existing benchmarks do not adequately cover this scenario. Most present a complete issue report in a single turn without interaction. Their evaluation can produce both false positives and false negatives, accepting incorrect patches or rejecting correct ones. We introduce UserSWE for evaluating coding agents’ ability to fix repository-level bugs through multi-turn interactions with dynamic users and execution-grounded verdicts at every turn. Its construction pipeline turns repository history and existing single-turn benchmarks into multi-turn tasks. At each turn, information pools constrain what the simulated user can reveal, while a persona chain sampled from a transition model calibrated on 20,859 development sessions governs how it speaks. Tests run after every turn, and a pair of agents from different model families writes and gates each verdict, using test results as primary evidence and model judgment only for aspects the tests cannot resolve. We evaluate eleven agents on 72 tasks and find that agent rankings change with the interaction budget. Claude Fable 5.1 resolves the most tasks in one turn, GPT-6-Astra resolves the most by the eighth turn, and Claude Opus 5 leads at most intermediate budgets, which indicates that single-turn performance is insufficient to characterize interactive coding ability.

Primary area:
datasets and benchmarks
Keywords:
coding agents, benchmark, multi-turn interaction, software engineering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4sbBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UserSWE is a 72-task multi-turn bug-fixing benchmark: 22 two-issue "cascades" mined from linked issues/PRs and 50 tasks converted from single-turn benchmarks. A simulated user releases information turn by turn, in a voice that follows a persona chain calibrated on sessions from an internal coding assistant. Tests run after every turn, and a cross-family Judge/Reviewer pair writes and checks each verdict. Over 11 agents, the paper argues that rankings change with the turn budget, so single-turn evaluation is insufficient.

Critical strengths
  • A real problem, and separating what the user reveals (information pool plus release policy) from how they say it (persona) is a reusable design.
  • Calibrating personas on real sessions is a good instinct. Showing that the "clear and correct" user assumed by prior work opens only 19% of code-changing sessions is a useful finding.
  • Careful engineering: tests after every turn, canonical test results overriding model verdicts, and blocked access to source issue/PR records and evaluation assets.
Critical weaknesses

Main concerns

  1. The robustness of the headline ranking changes is not established. GPT-6-Astra's final lead is a single task, and the paper's own tests separate only 5 of 55 agent pairs on final resolution. No repeated runs are reported. Turn 1 is also not a single-turn baseline, since agents get a staged partial request, and the separate single-turn comparison uses different tasks. Single-issue tasks stop at five turns, so later reorderings, including GPT-6-Astra's lead, are driven by the cascades; curves should be split by task type. The missing control is the same converted tasks with the full original issue given in one turn.

  2. The judges overlap with the leaderboard and have no measured accuracy. Three of the top four agents (Opus 5, Fable 5.1, GPT-6-Astra) are either a judge model or from the same family as one, and the Judge also writes the simulated user's next message. Self-preference is never tested, for example by swapping roles on the same tasks. Humans maintain an audit rubric, but no human-labeled accuracy is reported for verdicts or EqualPass test rewrites (about 65 grants).

  3. The simulator lacks held-out validation, and its ablation does not isolate persona style. The persona's "cause" axis gives hints constructed from the gold patch, so changing the persona changes information, not just voice, and the persona ablation (Table 4) mixes the two. The release policy also uses canonical test results to decide what the user reveals next, a privileged signal whose realism needs justification or an ablation. Persona frequencies are calibrated on real sessions, but beyond a human plausibility check (B.4) there is no held-out comparison of simulated turns with real user turns. Related work that grounds simulated users in real maintainer and user exchanges, such as CodeAssistBench (Kim et al., NeurIPS 2025 D&B), is not discussed.

  4. The metric's design choices are unexamined. LDPR pools tasks with 5- and 8-turn budgets and counts any turn without a code edit as half a turn, with no sensitivity analysis.

Smaller issues

  • Results belong to model-scaffold pairs, not models (Claude Code also runs four non-Anthropic models).
  • Counts of refusals, judge or test errors, and unscorable runs are not reported, and it is unclear how the latter three enter Table 3.
  • The persona model is built from real user sessions; consent, review, and de-identification should be stated.
Questions for authors
  1. How many runs per agent underlie Table 3, and are any of the Figure 3 reorderings significant across reruns or resampled persona chains?
  2. On the 50 converted tasks, how do rankings compare when agents get the full original issue in one turn?
  3. How does per-agent LDPR change when Judge/Reviewer roles are swapped on the same tasks, and how accurate are verdicts and EqualPass grants on a human-labeled sample?
  4. How do results change if the release policy cannot see test results, or with a persona ablation that holds the released information fixed?
  5. How sensitive are rankings to the half-turn weight and to scoring the 5- and 8-turn subsets separately?
  6. How were the 72 tasks selected, and what are their dates relative to model cutoffs?
Reviewer_AcY6OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces UserSWE, a 72-task benchmark for coding agents that fix repository-level bugs over several turns with a simulated user. Eleven model–scaffold agents are evaluated. The main claim is that rankings change with the interaction budget. Fable 5.1 leads at one turn, Opus 5 leads on LDPR and at intermediate budgets, and GPT-6-Astra leads at turn 8. The authors conclude that single-turn performance is insufficient. Further analyses show that a clear-and-correct user makes the benchmark easier than the dynamic persona chain, and that the Reviewer rewrites 40.9% of Judge decisions.

Critical strengths
  1. The cascade route is a good idea. Mining maintainer-linked issue pairs and checking mechanically that the second phase fails with only the first fix applied (A.4) gives natural, verifiable multi-stage tasks, and the process is repeatable.

  2. The simulator is anchored to real usage data. Persona frequencies and transitions come from about 13.5k real code-changing sessions. The finding that only 19% of sessions open with a "clear and correct" user is a useful, concrete statistic, and the persona grid is cleanly factored.

Critical weaknesses
  1. The central claim, that single-turn performance is insufficient, is never tested on the same tasks. The only single-turn comparison uses a different set of 42 tasks and only four agents (Fig. 3). The right experiment gives all 11 agents the same 72 tasks as full-issue single-turn prompts. It then reports the rank correlation between that ordering and the UserSWE ordering. Without it, I can't tell whether UserSWE adds information beyond single-turn ability or is largely single-turn ability plus a turn discount.

  2. The LLM Judge and Reviewer are the weakest part of an "execution-grounded" framework, and there is no ground-truth audit. The Reviewer rewrites 40.9% of Judge decisions, 62.4% under the Codex configuration and 25.4% under Claude Code. That is a large gap between the two configurations' behavior. The authors present this as proof the Reviewer matters. I read it as the Judge being wrong or non-compliant in a large share of turns. Resolution is decided by all-pass tests, so most rewrites concern evidence text and the next user message. Appendix E.4 says this directly: the rewrite "does not move the score". What I need is how often the Reviewer changes an outcome (resolved or not, or k). Nothing reports it. There is also no human check of Judge-only, Judge+Reviewer and tests-only verdicts on a sample. Nothing shows the Reviewer's rewrites are more correct, not just different. The Judge also writes the next user message, so the simulator's behavior depends on which model family holds that role.

Questions for authors
  1. Run all 11 agents on single-turn full-issue versions of the same 72 tasks. What is the rank correlation with LDPR and with resolve@K?

  2. Where does UserSWE disagree, and is that disagreement explained by interaction behavior?

  3. How many Fable 5.1 refusals occurred, and what are the headline numbers excluding them?

  4. How does the ranking change if you evaluate only the 50 single-issue tasks (K=5) and the 22 cascades (K=8) separately?

  5. How sensitive is LDPR to the discount shape and to the half-turn rule? Can an agent gain by asking many cheap clarifying questions, and how much of the pool can it extract that way?

Reviewer_dKGUOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a new interactive evaluation for SWE agents, in which problems are solved through a multi-turn interaction with a user. The evaluation includes 72 problems: 50 from existing benchmarks, and 22 from a pipeline which finds linked github issues and combines them to form a single problem. A novel dynamic user simulator engages with a coding agent which is evaluated against each task. The user simulator reveals information about the problem each turn, using a chain of persona-labels to behave dynamically (for example presenting information clearly-and-correctly in turn 1 but then clear-but-off-target on the next turn).

Overall, I believe the paper contributes novel and interesting methods for interactive SWE evaluation, but the presentation is very unclear, the evaluation set is too small to draw conclusions from, and key pieces of the evaluation are not validated through experiments.

Critical strengths

This paper approaches a timely and important problem, and brings several valuable method contributions:

  • The persona-transition matrix used to inform the dialogue moves of the user is particularly compelling, as it models changes in user behavior rather than just style.
  • While I think it needs to be presented more clearly, the turn-by-turn evaluation and LDPR metric is a novel and useful method for evaluating conversational SWE agents.
  • the pipeline which finds multi-commit problems for inclusion in the benchmark is a valuable contribution

With a larger evaluation set, careful presentation, and careful validation of the components of the evaluation, I think this could be a strong paper.

Critical weaknesses
  1. Substantial components in the evaluation are not clearly defined or motivated:
    • What does it mean for reviewer/judge to ‘rotate’ (L344), and why is this necessary?
    • The exact mechanism of the judge and reviewer isn’t clear, nor what a single item in the symptom or theory information pools looks like/where it comes from, other than stating it comes from the issue/gold patch. Are they LLM extracted, or paraphrased? How?
    • The mechanism of the user simulator isn't clear: is it just a prompted LLM? What is the prompt?
  2. Similarly, many components are not meaningfully validated:
    • Why have a reviewer step? Table 5 presents evidence about when it is used but not that it is necessary.
    • The EqualPass method modifies tests at evaluation time. While I think its motivation in preventing false negatives is reasonable, this approach could easily also contribute to false positives, but it doesn’t seem like this is checked for in an experiment (there are again some statistics about its usage in L455-457 but not any experiment evaluating that it works as expected and doesn’t introduce errors).
  3. The evaluation set is quite small (n=72), and thus many of the presented results are unlikely to be statistically significant:
    • The finding that GPT-6-Astra is best after 8 turns is referenced in the abstract, intro (L90) and results (L361), but is based on a comparison of 59/72 resolved vs. 58/72 resolved (Grok, Opus 5), differing by only 1 problem.
  4. Some portions of the text are unclear or possibly even untrue, and should be checked carefully. There are several, but a couple examples:
    • “The few that do run several turns stay static, releasing the information in a written order or phrasing every turn in one fixed conversational style, which leaves the agent talking to an unresponsive mechanism and not to a simulation of a developer.” (L43-45)
      • What is the meaning of unresponsive here? In what way does this work simulate a developers that others do not? The existing conversational SWE benchmarks cited later in the paper (SWE-Together, SWE-Interact) both simulate a user’s responses to questions from an agent. While this paper does make novel contributions to user simulation and interactive evaluation, I don’t think this sentence accurately describes the differences between this and prior work.
    • It’s unclear how L46-49 motivates this work: are unrestricted LLM judges commonly used in prior interactive SWE evals, in a way this work corrects?
    • L377 SWE Chat is cited as a benchmark but it is only a real-world collection of coding agent dialogues.
    • Overall, I think this paper needs a careful re-write to present the evaluation methods more clearly.
Questions for authors

The biggest questions I have concern clarity in the evaluation approach, discussed above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.