- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SWE-Interact evaluates coding agents in multi-turn sessions where a simulated user gradually reveals requirements and reviews the agent’s work. It adapts 75 tasks from three existing benchmarks while keeping their original goals and final tests. The results show that agents perform much worse in this interactive setting.
Critical strengths
-
The paper studies an important problem. In practice, users often start with incomplete requests and add details after reviewing the code. This benchmark captures several challenges in that workflow.
-
Using the same tasks and final tests makes the comparison useful. GPT 5.5 drops from 48.0% to 24.7%, and Opus 4.8 from 50.7% to 26.7%, despite spending more resources. This is a clear and interesting finding.
-
The analysis goes beyond success rates. Tracking plans and code revisions helps explain where agents struggle. The finding that agents can cover most requirements yet still fail the final tests is particularly useful.
Critical weaknesses
-
The user simulator sometimes fails to reveal requirements checked by the final tests. Figure 4 attributes around 15–16% of audited failures for two models to this issue. The authors acknowledge this limitation, but should show how much it affects the results and model comparisons.
-
The claim of greater realism needs more support. Longer sessions and lower success rates show increased difficulty, but do not directly establish similarity to real users. The neutral-user comparison also changes both persona and disclosure rules. A more focused comparison or a small human study would help.
-
The failure analysis needs clearer validation. The labeling prompt was adjusted using 20 manually reviewed trajectories. Agreement on a separate set would provide stronger evidence that the labels are reliable.
Questions for authors
- How do the results change when simulator-omission cases are separately accounted for?
- Are the performance drops consistent across the three source benchmarks?
- The simulator is instructed to request one change at a time. How sensitive are the results to this rule? Would answering early clarification questions more fully reduce unnecessary revisions?
- The simulator is prompted to start with a vague request, but how is the amount of information in that request controlled? Do all agents receive the same initial request for each task? It would be useful to check how the results change when the initial request includes more or fewer requirements.
2 comments
[2/2]
The failure analysis needs clearer validation. [W3]
The 20 manually audited failed trajectories (randomly sampled across models and tasks) were used both to define the failure categories and to iterate on the labeling prompt until agreement exceeded 85% (Section 3.3). The 85% figure is agreement on that development set, not a held-out estimate, and we will label it that way.
We now drew a new sample of 15 failed trials, disjoint from the original 20, with 5 from each of the three categories as labelled by the audit agent. We labeled each trial without seeing the LLM label and agreed with it in 12 of 15 cases (80%, close to the 85% on the development set).
| LLM label | Agreed | Our label when we disagreed |
|---|---|---|
| Simulator Omission | 5 / 5 | – |
| Requirements Forgotten | 4 / 5 | Simulator Omission (1) |
| Revision Degradation | 3 / 5 | Requirements Forgotten (2) |
Are the performance drops consistent across the three source benchmarks? [Q2]
The performance drop was not uniform across the benchmarks. For illustration, for GPT 5.5, the relative score drop was highest on DeepSWE (62.5%), followed by SWE Atlas (50%) and SWE-bench Pro (33%). The numbers were similar across Opus trials as well.
This follows from the fact that DeepSWE tasks are harder, instructions more detailed and execution oriented. This makes trajectories longer with more back-and-forth between the solver agent and the user, leading to more opportunities for errors.
We will update the writing of the paper with a more detailed per-benchmark analysis.
The simulator is instructed to request one change at a time. How sensitive are the results to this rule? [Q3]
How is the amount of information in the initial request controlled? Do all agents receive the same initial request for each task? [Q4]
The brief opening and the one-change-at-a-time rule follow the most common real-user pattern in SWE-chat: Expert Nitpicker users in vibe coding mode, who start brief and add requirements as they review (Section 2.3, Appendix A.5).
The simulator writes its own opening message in each trial from the "brief, vague overview" instruction, so the amount of initial detail is set by the prompt rather than by fixed text. However, this means that the opening message is not identical across trials for a task, but we inspected them and they were semantically equivalent in all cases. We experimented with having a fixed "first turn" per task, but we found that today's frontier models understand the idea behind it, and are able to start with a vague opening without needing such handholding.
We thank the reviewer for the positive assessment. We are glad that they found the problem to be important, as well as our design and analysis to be useful. We address each of their constructive suggestions and comments below:
Our failure audit (Section 3.3, Figure 4a) labeled every failed trial, so we can check this directly. Each model has 150 multi-turn trials: 75 tasks, each run once with each simulator. The following table recomputes the multi-turn resolve rate in two ways. Omission excluded drops the trials labeled Simulator Omission. Upper bound counts all of them as "False negatives" and gives them a pass. This obviously overstates the effect, but the idea is to show it as a ceiling under a perfect user simulator:
We agree that longer, harder sessions show difficulty, not realism. Our claim is narrower: the simulator is modeled on the most common real-user pattern in large scale studies like SWE-chat (Expert Nitpicker users in vibe coding mode, drawn from thousands of real user messages; Section 2.3), and it reproduces that pattern's interaction shape, a brief opening followed by layered corrections / revisions / feedback. These are characteristics of realistic user design, but not a strict reproduction of it.
Since the actual tasks done in SWE-Chat are quite different from the benchmarks we used, a direct comparison (ex: in terms of number of user-solver back and forth, message length, and workspace inspection) is uninformative. This is why we limited our comparison to trajectory shape, message realness and disclosure patterns. We will rephrase our writing to describe an interaction regime modeled on real-user data, and list the lack of a human or blinded comparison as future work.
On the neutral control: the reviewer is right that it changes the persona instructions, examples and disclosure rules together (Section 3.4). It shows the combined effect of our simulator design, not the effect of the persona alone, and we will say this explicitly.