- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
UserSWE is a 72-task multi-turn bug-fixing benchmark: 22 two-issue "cascades" mined from linked issues/PRs and 50 tasks converted from single-turn benchmarks. A simulated user releases information turn by turn, in a voice that follows a persona chain calibrated on sessions from an internal coding assistant. Tests run after every turn, and a cross-family Judge/Reviewer pair writes and checks each verdict. Over 11 agents, the paper argues that rankings change with the turn budget, so single-turn evaluation is insufficient.
Critical strengths
- A real problem, and separating what the user reveals (information pool plus release policy) from how they say it (persona) is a reusable design.
- Calibrating personas on real sessions is a good instinct. Showing that the "clear and correct" user assumed by prior work opens only 19% of code-changing sessions is a useful finding.
- Careful engineering: tests after every turn, canonical test results overriding model verdicts, and blocked access to source issue/PR records and evaluation assets.
Critical weaknesses
Main concerns
-
The robustness of the headline ranking changes is not established. GPT-6-Astra's final lead is a single task, and the paper's own tests separate only 5 of 55 agent pairs on final resolution. No repeated runs are reported. Turn 1 is also not a single-turn baseline, since agents get a staged partial request, and the separate single-turn comparison uses different tasks. Single-issue tasks stop at five turns, so later reorderings, including GPT-6-Astra's lead, are driven by the cascades; curves should be split by task type. The missing control is the same converted tasks with the full original issue given in one turn.
-
The judges overlap with the leaderboard and have no measured accuracy. Three of the top four agents (Opus 5, Fable 5.1, GPT-6-Astra) are either a judge model or from the same family as one, and the Judge also writes the simulated user's next message. Self-preference is never tested, for example by swapping roles on the same tasks. Humans maintain an audit rubric, but no human-labeled accuracy is reported for verdicts or EqualPass test rewrites (about 65 grants).
-
The simulator lacks held-out validation, and its ablation does not isolate persona style. The persona's "cause" axis gives hints constructed from the gold patch, so changing the persona changes information, not just voice, and the persona ablation (Table 4) mixes the two. The release policy also uses canonical test results to decide what the user reveals next, a privileged signal whose realism needs justification or an ablation. Persona frequencies are calibrated on real sessions, but beyond a human plausibility check (B.4) there is no held-out comparison of simulated turns with real user turns. Related work that grounds simulated users in real maintainer and user exchanges, such as CodeAssistBench (Kim et al., NeurIPS 2025 D&B), is not discussed.
-
The metric's design choices are unexamined. LDPR pools tasks with 5- and 8-turn budgets and counts any turn without a code edit as half a turn, with no sensitivity analysis.
Smaller issues
- Results belong to model-scaffold pairs, not models (Claude Code also runs four non-Anthropic models).
- Counts of refusals, judge or test errors, and unscorable runs are not reported, and it is unclear how the latter three enter Table 3.
- The persona model is built from real user sessions; consent, review, and de-identification should be stated.
Questions for authors
- How many runs per agent underlie Table 3, and are any of the Figure 3 reorderings significant across reruns or resampled persona chains?
- On the 50 converted tasks, how do rankings compare when agents get the full original issue in one turn?
- How does per-agent LDPR change when Judge/Reviewer roles are swapped on the same tasks, and how accurate are verdicts and EqualPass grants on a human-labeled sample?
- How do results change if the release policy cannot see test results, or with a persona ablation that holds the released information fixed?
- How sensitive are rankings to the half-turn weight and to scoring the 5- and 8-turn subsets separately?
- How were the 72 tasks selected, and what are their dates relative to model cutoffs?