acceptodds
Under review as a conference paper at ICLR 2027

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

15 views

Abstract

We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they remain susceptible to implementation mistakes and revision degradation, particularly as interactions become longer and requirements accumulate. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact captures a complementary dimension of coding-agent capability: interactive goal discovery and iterative refinement with a user in the loop.

Primary area:
datasets and benchmarks
Keywords:
software engineering evaluation, coding agents, agents evaluation, evaluation protocols, user-simulation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_SydHOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SWE-Interact evaluates coding agents in multi-turn sessions where a simulated user gradually reveals requirements and reviews the agent’s work. It adapts 75 tasks from three existing benchmarks while keeping their original goals and final tests. The results show that agents perform much worse in this interactive setting.

Critical strengths
  • The paper studies an important problem. In practice, users often start with incomplete requests and add details after reviewing the code. This benchmark captures several challenges in that workflow.

  • Using the same tasks and final tests makes the comparison useful. GPT 5.5 drops from 48.0% to 24.7%, and Opus 4.8 from 50.7% to 26.7%, despite spending more resources. This is a clear and interesting finding.

  • The analysis goes beyond success rates. Tracking plans and code revisions helps explain where agents struggle. The finding that agents can cover most requirements yet still fail the final tests is particularly useful.

Critical weaknesses
  • The user simulator sometimes fails to reveal requirements checked by the final tests. Figure 4 attributes around 15–16% of audited failures for two models to this issue. The authors acknowledge this limitation, but should show how much it affects the results and model comparisons.

  • The claim of greater realism needs more support. Longer sessions and lower success rates show increased difficulty, but do not directly establish similarity to real users. The neutral-user comparison also changes both persona and disclosure rules. A more focused comparison or a small human study would help.

  • The failure analysis needs clearer validation. The labeling prompt was adjusted using 20 manually reviewed trajectories. Agreement on a separate set would provide stronger evidence that the labels are reliable.

Questions for authors
  1. How do the results change when simulator-omission cases are separately accounted for?
  2. Are the performance drops consistent across the three source benchmarks?
  3. The simulator is instructed to request one change at a time. How sensitive are the results to this rule? Would answering early clarification questions more fully reduce unnecessary revisions?
  4. The simulator is prompted to start with a vague request, but how is the amount of information in that request controlled? Do all agents receive the same initial request for each task? It would be useful to check how the results change when the initial request includes more or fewer requirements.
2 comments

We thank the reviewer for the positive assessment. We are glad that they found the problem to be important, as well as our design and analysis to be useful. We address each of their constructive suggestions and comments below:

How much does the simulator omission explain the drop from single-turn to multi-turn tasks? [W1, Q1]

Our failure audit (Section 3.3, Figure 4a) labeled every failed trial, so we can check this directly. Each model has 150 multi-turn trials: 75 tasks, each run once with each simulator. The following table recomputes the multi-turn resolve rate in two ways. Omission excluded drops the trials labeled Simulator Omission. Upper bound counts all of them as "False negatives" and gives them a pass. This obviously overstates the effect, but the idea is to show it as a ceiling under a perfect user simulator:

ModelSingle-turnMulti-turnOmission / audited failuresMulti-turn, omission excludedMulti-turn, upper bound
Opus 4.850.7%26.7% (40/150)16 / 10129.9% (40/134)37.3% (56/150)
GPT 5.548.0%24.7% (37/150)5 / 10225.5% (37/145)28.0% (42/150)
Sonnet 4.621.3%18.7% (28/150)16 / 10520.9% (28/134)29.3% (44/150)
Gemini 3.5 Flash29.3%17.3% (26/150)7 / 10718.2% (26/143)22.0% (33/150)
Kimi K2.625.3%14.7% (22/150)0 / 11614.7% (22/150)14.7% (22/150)
  • Even with all the omission trials excluded, Opus 4.8 drops 20.8 points and GPT 5.5 drops 22.5 points from single-turn. Even under the upper bound, the drops are 13.4 and 20.0 points.
  • The failure-mode findings. Simulator Omission is its own category in Figure 4a, so the shift from instruction-following, assumption and forgotten-requirement failures in weaker agents to implementation mistakes and revision degradation in stronger agents is measured on the remaining failures.
  • The model ordering is unchanged when omission trials are excluded. However, we retain these tasks because future adaptations with an even stronger user-sim model can handle the requirements faithfully.

The claim of greater realism needs more support. [W2]

We agree that longer, harder sessions show difficulty, not realism. Our claim is narrower: the simulator is modeled on the most common real-user pattern in large scale studies like SWE-chat (Expert Nitpicker users in vibe coding mode, drawn from thousands of real user messages; Section 2.3), and it reproduces that pattern's interaction shape, a brief opening followed by layered corrections / revisions / feedback. These are characteristics of realistic user design, but not a strict reproduction of it.

  • Interaction shape: user interactions and user tool calls per trial (Table 1), with and without the persona (Table 3), and per simulator (Table 4).
  • How requirements surface over a session: goal-discovery coverage from the initial plan through each revision (Figure 3a).
  • A side-by-side excerpt of a real SWE-chat session, our simulated user, and the neutral simulator (Appendix A.2). This is illustrative only.

Since the actual tasks done in SWE-Chat are quite different from the benchmarks we used, a direct comparison (ex: in terms of number of user-solver back and forth, message length, and workspace inspection) is uninformative. This is why we limited our comparison to trajectory shape, message realness and disclosure patterns. We will rephrase our writing to describe an interaction regime modeled on real-user data, and list the lack of a human or blinded comparison as future work.

On the neutral control: the reviewer is right that it changes the persona instructions, examples and disclosure rules together (Section 3.4). It shows the combined effect of our simulator design, not the effect of the persona alone, and we will say this explicitly.

[2/2]

The failure analysis needs clearer validation. [W3]

The 20 manually audited failed trajectories (randomly sampled across models and tasks) were used both to define the failure categories and to iterate on the labeling prompt until agreement exceeded 85% (Section 3.3). The 85% figure is agreement on that development set, not a held-out estimate, and we will label it that way.

We now drew a new sample of 15 failed trials, disjoint from the original 20, with 5 from each of the three categories as labelled by the audit agent. We labeled each trial without seeing the LLM label and agreed with it in 12 of 15 cases (80%, close to the 85% on the development set).

LLM labelAgreedOur label when we disagreed
Simulator Omission5 / 5–
Requirements Forgotten4 / 5Simulator Omission (1)
Revision Degradation3 / 5Requirements Forgotten (2)

Are the performance drops consistent across the three source benchmarks? [Q2]

The performance drop was not uniform across the benchmarks. For illustration, for GPT 5.5, the relative score drop was highest on DeepSWE (62.5%), followed by SWE Atlas (50%) and SWE-bench Pro (33%). The numbers were similar across Opus trials as well.

This follows from the fact that DeepSWE tasks are harder, instructions more detailed and execution oriented. This makes trajectories longer with more back-and-forth between the solver agent and the user, leading to more opportunities for errors.

We will update the writing of the paper with a more detailed per-benchmark analysis.

The simulator is instructed to request one change at a time. How sensitive are the results to this rule? [Q3]

How is the amount of information in the initial request controlled? Do all agents receive the same initial request for each task? [Q4]

The brief opening and the one-change-at-a-time rule follow the most common real-user pattern in SWE-chat: Expert Nitpicker users in vibe coding mode, who start brief and add requirements as they review (Section 2.3, Appendix A.5).

The simulator writes its own opening message in each trial from the "brief, vague overview" instruction, so the amount of initial detail is set by the prompt rather than by fixed text. However, this means that the opening message is not identical across trials for a task, but we inspected them and they were semantically equivalent in all cases. We experimented with having a fixed "first turn" per task, but we found that today's frontier models understand the idea behind it, and are able to start with a vague opening without needing such handholding.

Reviewer_8T27OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SWE-Interact, an interactive evaluation framework for realistic software engineering workflows. Unlike existing SWE benchmarks, which typically provide agents with a complete task description upfront, SWE-Interact gradually reveals task requirements through interactions between the interactions between coding agent and user simulator. Experimental results show a substantial performance gap between conventional single-turn SWE evaluation and interactive development, and the paper further analyzes failure modes, agents' ability to uncover full goals, and their interactive refinement capabilities through detailed case studies.

Overall, I find the problem setting practically relevant and potentially valuable for the SWE community. However, I have substantial concerns regarding the evaluation scope, reliability of the user simulator, and validity of the evaluation methodology. In particular, the relatively small task set and limited analysis of simulator variability make it difficult to assess the generality and robustness of the reported findings.

Critical strengths
  1. Practical relevance: Existing SWE benchmarks typically provide agents with relatively explicit task specifications (e.g., the problem description, expected behavior, and reproduction steps). In real world, however, requirements are often incomplete or ambiguous initially and are refined through interactions with users. SWE-Interact addresses this by evaluating agents under an iterative, user-driven process. This provides a practically meaningful complement to existing SWE benchmarks.

  2. Interactive user-simulation design: The proposed user simulation harness converts a conventional static LLM-based evaluation into an interactive process in which the user agent can inspect the modifications and provide subsequent feedback. This design provides certain novelty.

Critical weaknesses
  1. Limited evaluation scope and insufficient justification of task selection: The evaluation selects only 25 tasks from each of three source benchmarks, resulting in a relatively small evaluation set compared with existing SWE benchmarks, which often contain hundreds tasks. The paper does not provide sufficient justification for either the choice of the three source benchmarks or the fixed number of 25 tasks per benchmark. It is unclear why other relevant benchmarks, such as SWE-bench-Verified, SWE-Multilingual and BeyondSWE are excluded. More details are also needed regarding the task filtering process, including the number and proportion of excluded tasks and the reasons for exclusion.

  2. Insufficient analysis of user-simulator stability: The responses generated by a user simulator can vary across runs and may depend strongly on the underlying LLM. The paper uses GPT-5.5 and Claude Opus 4.7 as user simulators and averages only the two execution results, but this limited number of simulator models does not sufficiently establish the robustness of the conclusions. I would encourage the authors to evaluate a broader range of simulator models and conduct more independent runs, reporting statistics such as mean and standard deviation.

  3. Unclear validity and correctness of simulated user feedback: A fundamental concern is how the framework ensures that the user simulator provides accurate and appropriate feedback. For example, how does the simulator determine whether the agent's current modification satisfies the user's implicit or partially revealed requirements? More importantly, how is it ensured that subsequent feedback remains consistent with the underlying original task rather than drifting toward an unintended objective? The discussion of Simulator Omission Error in Section 3.3 further highlights this concern: in some interactions, the simulator may fail to reveal an important task requirement. Such behavior may reflect limitations of the simulator rather than limitations of the coding agent, potentially confounding the evaluation.

  4. Potential inconsistency between the original requirement and the final interactive objective: Relatedly, the paper does not sufficiently explain how it guarantees that the requirements revealed across multiple turns remain faithful to the original task specification. This issue is particularly important because the interaction process itself determines which parts of the original requirement become observable to the agent. The observation in Section 3.1 that Kimi submits after a single round of user correction for most tasks further suggests that some interactions terminate before sufficient requirements can be revealed.

  5. Insufficient validation of the LLM-as-a-Judge evaluation: Section 3.2 uses an LLM-as-a-Judge pipeline to determine whether agents successfully uncover the underlying task objectives. In particular, GPT-5.5 decomposes the original task into sub-goals, while Claude Sonnet 4.6 evaluates whether these sub-goals have been achieved. However, the paper does not provide sufficient details about the reliability of this evaluation pipeline. It is unclear whether the judging process is repeated across multiple runs, how accurate the automatically generated checkpoints are, or how well the judge's decisions agree with human evaluation. This concern is particularly relevant given the paper's own observation that the grading model tends to assign higher scores to plans than to concrete implementations. Without human validation or inter-rater agreement analysis, it is difficult to determine whether the reported results genuinely measure the agents' ability to discover and implement the underlying requirements.

Questions for authors
  1. How do you ensure that the requirements revealed by the user simulator throughout the interaction remain consistent with the original task specification? How do you validate the correctness of the simulator's responses based on the agent's intermediate modifications?

  2. How accurate are the task decompositions and checkpoints generated by GPT-5.5, and how reliable are the subsequent LLM-based scoring? Could the authors provide more details about the judging procedure, variance analysis, and agreement with human judgments?

  3. Have you evaluated additional LLMs as user simulators? Given that the simulator directly influences the interaction trajectory, broader simulator-model coverage and repeated runs would help establish whether the observed conclusions are robust across different simulator choices.

  4. Could the authors provide more details about Task Design, including why the selected three SWE benchmarks were preferred over alternatives such as SWE-bench-Verified or SWE-Multilingual, the number and proportion of tasks removed during manual filtering, the reasons for exclusion, and why exactly 25 tasks were selected from each benchmark?

4 comments

We thank the reviewer for the careful and constructive review.

First, a note on what SWE-Interact is intended for. We do not intend it as a leaderboard for robustly ranking models. It highlights a gap in current SWE benchmarks, and we lay out several foundational primitives that would be a core component of such a leaderboard, such as an agentic user model, real world data-driven user-persona design and study of goal discovery and partial progress. As a result, it's the first work that studies how coding agents discover goals, keep track of requirements and revise their work.

Our biggest contribution is our analysis about these behavior patterns across models and both simulators, not about small differences between individual scores. A user simulator built on an LLM is stochastic and produces some false negatives, as at the time of the study, the models weren’t perfect at acting as user simulators. We acknowledge and describe this in the paper (Section 3.3), and we have thoroughly iterated on the design that reduced this as much as possible, so future benchmarks can build on top of our learnings. We will make this discussion more prominent. We thank all reviewers for adding thoughtful comments regarding this, and we will state this framing directly in the introduction of the revision.

The paper does not provide sufficient justification for either the choice of the three source benchmarks or the fixed number of 25 tasks per benchmark. It is unclear why other relevant benchmarks, such as SWE-bench-Verified, SWE-Multilingual and BeyondSWE are excluded. [W1, Q4]

We chose benchmarks which balanced two considerations:

  1. Tasks had large multi-part goals that challenged the best models at the time of study, which enables long multiturn development. This ruled out simpler benchmarks like SWE-bench Verified and SWE-Multilingual which had small focused changes in most tasks.
  2. There were a subset of solved and unsolved tasks, which would help us study whether scores improved or degraded on multiturn versions of the tasks. This ruled out extremely hard benchmarks like FrontierSWE and ProgramBench. (We do believe that these are excellent sources of tasks for a future version of SWE-Interact, since our initial experiments on these ended up with 1000+ turns. However, due to near-zero pass rates across models and extremely high costs, we didn't include them in this initial version).

SWE Bench Pro, DeepSWE and SWE Atlas (Refactoring) all balanced these considerations. Because of cost considerations involved in running such multiturn long-running sessions, we limited it to 75 tasks (25 each to maintain uniformity), in line with other contemporary SWE Benchmarks like TerminalBench (60-80 tasks), DeepSWE (113 tasks) and SWE Atlas Refactoring (70 tasks).

The 25 tasks from each were chosen with a combination of manual + programmatic approach. We first ran pass@3 trials on Opus 4.8 and GPT 5.5 on all tasks from DeepSWE, SWE Atlas and 100 random tasks from SWE Bench Pro, split the tasks into full-pass (all 3 solved trials) and partial-pass (1 or more failed trials), chose 15+15 each across all 3 benchmarks (90 in total), and manually discarded 5 from each benchmark that didn’t have significant multiturn instructions, while targeting a 50% single-turn solve rate for Opus 4.8 and GPT 5.5.

limited number of simulator models does not sufficiently establish the robustness of the conclusions. Have you evaluated additional LLMs as user simulators? [W2, Q3]

Per-simulator results Table 4 reports each agent under each simulator separately. For the two strongest agents the drop holds under both simulators. While there is some variation in scores under the two user simulators, they don’t differ drastically.

The two simulators differ in how closely they follow the persona: GPT 5.5 reveals requirements over more turns and inspects the workspace 4 to 5 times more (Section 3.4). We report each simulator separately in Table 4 alongside the averages so this difference stays visible. While we did some initial experiments with other models (cheaper API based models or cheaper open models), they all suffered from too many Omissions leading to False Negatives.

Why two simulators and two runs. Multi-turn trials are long (144 to 425 agent steps on average) and cost USD 6 to USD 12 per trial in agent inference. The evaluation currently covers 750 multi-turn and 375 single-turn trials. Therefore, the cost overrun is significantly high when compared to a general SWE benchmark, and therefore limits the number of experiments we could do.

We agree that more user-simulator model focused research is valuable, and we hope our testbed inspires this for future research. We will describe possible extensions in the paper.

(continued in part 2/3)

How do you ensure that the requirements revealed by the user simulator throughout the interaction remain consistent with the original task specification? How do you validate the correctness of the simulator's responses based on the agent's intermediate modifications? [W3, Q1]

We iterated over multiple rounds to ensure the simulator is as faithful to the task as possible, across the user’s persona, information disclosed, and tools provided.

What the user knows: It has the original task's full instructions and interface specification (Section 2.3) and never sees the verifier or its tests (Section 4). Its prompt (Appendix A.5) makes the task block the only source of truth: anything the task does not specify is answered with "your call", instead of making up a detail. Because the verifier checks a specific interface, the simulator must eventually disclose every requirement the verifier checks. Effectively, this setup changes how requirements are revealed, not the end goal or the verifier (Section 2.2).

User inspecting changes : After each agent message, the harness gives the simulator a snapshot of the agent's changes and a read-only shell into the agent's workspace (git, grep, sed, find; Section 2.3). It compares what it finds against the task block and raises the next mismatch. It does not run tests or grade: final correctness comes only from the original verifier.

Since the simulator can only draw requirements from the task block, its feedback doesn’t affect the overall goal of the task. The simulator-induced error our audit found is omission (a requirement never disclosed), not drift toward a different objective. The simulator also never edits the code directly, and only providing verbal feedback through messages.

Such behavior may reflect limitations of the simulator rather than limitations of the coding agent, potentially confounding the evaluation. [W3]

Our failure audit (Section 3.3, Figure 4a) labeled every failed trial, so we can check this directly. Each model has 150 multi-turn trials: 75 tasks, each run once with each simulator. The following table recomputes the multi-turn resolve rate in two ways. Omission excluded drops the trials labeled Simulator Omission. Upper bound counts all of them as passes; this overstates the effect, so we use it only as a ceiling, not as a corrected score.

ModelSingle-turnMulti-turnOmission / audited failuresMulti-turn, omission excludedMulti-turn, upper bound
Opus 4.850.7%26.7% (40/150)16 / 10129.9% (40/134)37.3% (56/150)
GPT 5.548.0%24.7% (37/150)5 / 10225.5% (37/145)28.0% (42/150)
Sonnet 4.621.3%18.7% (28/150)16 / 10520.9% (28/134)29.3% (44/150)
Gemini 3.5 Flash29.3%17.3% (26/150)7 / 10718.2% (26/143)22.0% (33/150)
Kimi K2.625.3%14.7% (22/150)0 / 11614.7% (22/150)14.7% (22/150)
  • Even with the omission trials excluded, Opus 4.8 drops 20.8 points and GPT 5.5 drops 22.5 points from single-turn. Even under the upper bound, the drops are 13.4 and 20.0 points.
  • The failure-mode findings. Simulator Omission is its own category in Figure 4a, so the shift from instruction-following, assumption and forgotten-requirement failures in weaker agents to implementation mistakes and revision degradation in stronger agents is measured on the remaining failures.
  • The model ordering is unchanged when omission trials are excluded. However, we retain these tasks because future adaptations with an even stronger user-sim model can handle the requirements faithfully.

The observation in Section 3.1 that Kimi submits after a single round of user correction for most tasks further suggests that some interactions terminate before sufficient requirements can be revealed. [W4]

To match real world usage, the agent is specifically instructed to implement user’s instruction, and wait for the user’s final confirmation before “submitting”. In turn, the user is instructed to only ask the solver agent to submit after it has disclosed all the requirements.

In the case of Kimi, it submits right after it implements the first turn, before the user could add requirements or ask it to submit, in most of its trials (unlike all the other agents). This is an instruction following failure on the solver agent’s part, and is recorded as such in the failure mode analysis.

(continued in Part 3/3)

Section 3.2 uses an LLM-as-a-Judge pipeline to determine whether agents successfully uncover the underlying task objectives…It is unclear whether the judging process is repeated across multiple runs, how accurate the automatically generated checkpoints are…

The goal-discovery analysis has two LLM steps (Appendix A.3), and we agree with the reviewer that they differ in reliability.

Rubric generation. GPT 5.5 turns the full task instruction into 6 to 18 binary rubric items, with weights that sum to 100 and a concrete "achieved when" condition for each. This step is already well constrained: most tasks in these benchmarks already list their requirements (SWE-bench Pro tasks, for example, state them in stages), so the rubric mostly restates them as checkable items. We manually checked 10 rubrics before running the analysis and found them to be fully consistent with the task’s instructions.

Checkpoint scoring. The checkpoints are the agent's own commits. Sonnet 4.6 scores the plan and each commit against the rubric, seeing only the patch at that revision, not later revisions or test results. This is the less reliable step, and Section 3.2 already notes one bias: plans are graded more generously than code, which we transparently disclose. As a check against ground truth, Figure 3b compares the final score with the original verifier: almost all verifier-passing solutions score 90% or higher. Each checkpoint was scored twice.

This analysis is meant for descriptive post-hoc analysis, and doesn't affect the actual task mechanism: the success on a task (i.e. resolve rates) and model comparisons come from the original task verifiers.

Dear reviewer,

We hope we have addressed all your concerns. Since the rebuttal deadline is approaching, we would appreciate it if you could examine our rebuttal and re-assess the scores.

Thanks again for your thoughtful and detailed comments.

Reviewer_EsGvOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SWE-INTERACT adapts 75 tasks from SWE-bench Pro, SWE Atlas Refactoring, and DeepSWE into interactive coding sessions. A persona-conditioned user simulator progressively reveals requirements and inspects the agent workspace, while final solutions are scored by the original task verifiers. Experiments compare fully specified single-turn tasks with multi-turn sessions across five coding agents, two simulator models, and a neutral-persona control. The paper further analyzes goal discovery, failure modes, and revision behavior through checkpointed trajectories.

Critical strengths
  1. The benchmark targets a practically important capability: incorporating requirements through interaction rather than merely implementing a complete initial specification. Reusing the same underlying tasks and final verifiers provides a useful comparison, while workspace-grounded user inspection is a meaningful extension over a simulator that only produces conversational replies. The separation between simulator and agent containers also makes the intended information boundary explicit.

  2. The study offers substantial diagnostic material for a first 75-task testbed. The strongest agents drop from approximately 48-51% single-turn resolve rate to approximately 25-27% in the interactive setting, and the paper examines plans, intermediate commits, and final correctness rather than reporting only a leaderboard. The neutral-persona and two-simulator analyses are particularly helpful: they expose how interaction design affects results instead of assuming that one simulator is an interchangeable human proxy. Explicitly identifying Simulator Omission as an evaluation failure is also a valuable and candid finding.

Critical weaknesses
  1. The benchmark has an unresolved measurement-validity issue: a final verifier can penalize requirements that the simulated user never disclosed, so an observed failure need not reflect an agent's ability to understand or implement communicated requirements. The paper candidly identifies this problem, but Figure 4 attributes approximately 15-16% of audited failed trials to Simulator Omission for Sonnet and Opus. That is material for a benchmark intended to measure interactive capability. The appendix example of an undisclosed default argument shows that even an otherwise reasonable implementation can fail an uncommunicated interface requirement. The large single-to-multi-turn performance drop may persist after accounting for these cases, but the current version does not establish how much of the reported comparison is robust to simulator-induced penalties. This affects confidence in the benchmark's main measurement, beyond the interpretation of close rankings.

  2. The validation of failure attribution is not yet strong enough to resolve that measurement concern. The failure-label prompt was iterated against a 20-trajectory manual subset, so the reported >85% overlap does not clearly provide independent evidence that simulator omissions, forgotten requirements, and implementation regressions are reliably distinguished. The two simulator models also produce meaningfully different interaction patterns and outcomes, while pooled Wilson intervals over two runs of the same 75 tasks do not capture task pairing or simulator-conditioned variation. Real-user-inspired persona design and qualitative examples are useful motivation, but longer and harder sessions do not independently validate the resulting capability measurement. Taken together, these leave insufficient evidence that the central failure decomposition and performance comparisons are stable under the simulator's own errors and variability.

Questions for authors
  1. Could you report omission counts and denominators per agent, simulator, and source benchmark, distinguishing simulator non-disclosure from agent termination before clarification? Using the existing trajectories, please show how the single-to-multi-turn drop and between-agent comparisons change when confirmed simulator-error cases are flagged or excluded, with exclusions and denominators explicit. Please state which conclusions remain supported. This should be a sensitivity analysis, rather than automatically counting omitted-requirement cases as agent successes.

  2. Could you provide task-paired uncertainty for the single-turn versus multi-turn change and report the two simulator conditions separately, keeping repeated observations of a task grouped? The existing Wilson intervals are useful descriptive summaries, but a paired/grouped analysis would help establish the reliability of the main performance comparison under simulator variation. This can use the current task-level outcomes without a complete rerun.

  3. Could you manually validate a small disjoint sample of failure-mode assignments, focusing on Simulator Omission, Requirements Forgotten, and Revision Degradation? Please specify whether the original 20 trajectories were used for prompt tuning or also for final validation, and provide a short disagreement breakdown. A bounded independent audit would help determine whether errors in the failure classifier materially change the conclusions; a large annotation campaign is not required.

  4. Please make the scope of the realism claim more precise. A small blinded comparison of already collected simulator and real-user excerpts, or quantitative summaries of requirement disclosure, message length, and workspace inspection, would be useful if feasible. Otherwise, framing the persona as a real-data-inspired interaction regime would match the evidence better than treating increased difficulty as proof of realism.

  5. Could you state execution/interaction limits, crash or infrastructure-error handling, and whether Table 1's dollar costs include the user simulator? The simulator makes substantial workspace tool calls, so separating agent cost from simulator/evaluation cost would help others reproduce the benchmark and interpret its overhead.

3 comments

We thank the reviewer for the careful and constructive review.

First, a note on what SWE-Interact is intended for. We do not intend it as a leaderboard for robustly ranking models. It highlights a gap in current SWE benchmarks, and we lay out several foundational primitives that would be a core component of such a leaderboard, such as an agentic user model, real world data-driven user-persona design and study of goal discovery and partial progress. As a result, it's the first work that studies how coding agents discover goals, keep track of requirements and revise their work.

Our biggest contribution is our analysis about these behavior patterns across models and both simulators, not about small differences between individual scores. A user simulator built on an LLM is stochastic and produces some false negatives, as at the time of the study, the models weren’t perfect at acting as user simulators. We acknowledge and describe this in the paper (Section 3.3), and we have thoroughly iterated on the design that reduced this as much as possible, so future benchmarks can build on top of our learnings. We will make this discussion more prominent. We thank all reviewers for adding thoughtful comments regarding this, and we will state this framing directly in the introduction of the revision.

The benchmark has an unresolved measurement-validity issue: a final verifier can penalize requirements that the simulated user never disclosed, so an observed failure need not reflect an agent's ability to understand or implement communicated requirements. [W1, Q1]

Our failure audit (Section 3.3, Figure 4a) labeled every failed trial, so we can check this directly. Each model has 150 multi-turn trials: 75 tasks, each run once with each simulator. The following table recomputes the multi-turn resolve rate in two ways. Omission excluded drops the trials labeled Simulator Omission. Upper bound counts all of them as passes; this overstates the effect, so we use it only as a ceiling, not as a corrected score.

ModelSingle-turnMulti-turnOmission / audited failuresMulti-turn, omission excludedMulti-turn, upper bound
Opus 4.850.7%26.7% (40/150)16 / 10129.9% (40/134)37.3% (56/150)
GPT 5.548.0%24.7% (37/150)5 / 10225.5% (37/145)28.0% (42/150)
Sonnet 4.621.3%18.7% (28/150)16 / 10520.9% (28/134)29.3% (44/150)
Gemini 3.5 Flash29.3%17.3% (26/150)7 / 10718.2% (26/143)22.0% (33/150)
Kimi K2.625.3%14.7% (22/150)0 / 11614.7% (22/150)14.7% (22/150)
  • Even with the omission trials excluded, Opus 4.8 drops 20.8 points and GPT 5.5 drops 22.5 points from single-turn. Even under the upper bound, the drops are 13.4 and 20.0 points.
  • The failure-mode findings. Simulator Omission is its own category in Figure 4a, so the shift from instruction-following, assumption and forgotten-requirement failures in weaker agents to implementation mistakes and revision degradation in stronger agents is measured on the remaining failures.
  • The model ordering is unchanged when omission trials are excluded. However, we retain these tasks because future adaptations with an even stronger user-sim model can handle the requirements faithfully.

We will also update the paper with statistics regarding omission counts and denominators per agent, simulator, and source benchmark.

The failure-label prompt was iterated against a 20-trajectory manual subset, so the reported >85% overlap does not clearly provide independent evidence that simulator omissions, forgotten requirements, and implementation regressions are reliably distinguished. [W2, Q3]

The 20 manually audited failed trajectories (randomly sampled across models and tasks) were used both to define the failure categories and to iterate on the labeling prompt until agreement exceeded 85% (Section 3.3). The 85% figure is agreement on that development set, not a held-out estimate, and we will label it that way.

We drew a new sample of 15 failed trials, disjoint from the original 20, with 5 from each of the three categories as labelled by the audit agent. We labeled each trial without seeing the LLM label and agreed with it in 12 of 15 cases (80%, close to the 85% on the development set).

LLM labelAgreedOur label when we disagreed
Simulator Omission5 / 5–
Requirements Forgotten4 / 5Simulator Omission (1)
Revision Degradation3 / 5Requirements Forgotten (2)

(continued in part 2/2)

The two simulator models also produce meaningfully different interaction patterns and outcomes, while pooled Wilson intervals over two runs of the same 75 tasks do not capture task pairing or simulator-conditioned variation. [W3, Q2]

Per-simulator results. Table 4 reports each agent under each simulator separately. For the two strongest agents the drop holds under both simulators. While there is some variation in scores under the two user simulators, they don’t differ drastically.

The two simulators differ in how closely they follow the persona: GPT 5.5 reveals requirements over more turns and inspects the workspace 4 to 5 times more (Section 3.4). We report each simulator separately in Table 4 alongside the averages so this difference stays visible. While we did some initial experiments with other models (cheaper API based models or cheaper open models), they all suffered from too many Omissions leading to False Negatives.

Why two simulators and two runs. Multi-turn trials are long (144 to 425 agent steps on average) and cost USD 6 to USD 12 per trial in agent inference. The evaluation currently covers 750 multi-turn and 375 single-turn trials. Therefore, the cost overrun is significantly high when compared to a general SWE benchmark, and therefore limits the number of experiments we could do.

We agree that more simulators and repeated runs would be more interesting, and we hope our testbed inspires this for future research.

Paired test. Figure 5 (Appendix A.1) contains each model's paired single-turn to multi-turn transitions. For Opus 4.8, 28% of tasks pass single-turn and fail multi-turn, against 4% the other way; for GPT 5.5 it is 28% against 7%. Counting each task once, which treats the two simulator runs of a task as fully correlated and is therefore conservative, an exact McNemar test gives p < 0.001 for Opus 4.8 (21 vs 3 tasks) and p = 0.002 for GPT 5.5 (21 vs 5). We will add this paired analysis and per-simulator confidence intervals to Appendix A.1.

Real-user-inspired persona design and qualitative examples are useful motivation, but longer and harder sessions do not independently validate the resulting capability measurement. [W4, Q4]

We agree that longer, harder sessions show difficulty, not realism. The claim we want to hold in this testbed is more narrow: the simulator is modeled on the most common real-user pattern in SWE-chat (Expert Nitpicker users in vibe coding mode, drawn from thousands of real user messages; Section 2.3), and it reproduces that pattern's interaction shape, a brief opening followed by layered corrections / revisions / feedback. These are characteristics of realistic user design, but not a strict reproduction of it.

  • Interaction shape: user interactions and user tool calls per trial (Table 1), with and without the persona (Table 3), and per simulator (Table 4).
  • How requirements surface over a session: goal-discovery coverage from the initial plan through each revision (Figure 3a).
  • A side-by-side excerpt of a real SWE-chat session, our simulated user, and the neutral simulator (Appendix A.2). This is illustrative only.

Since the actual tasks done in SWE-Chat are quite different from the benchmarks we used, a direct comparison (in terms of the number of user-solver back-and-forths, message length, and workspace inspection) is uninformative. This is why we chose to limit our comparison to trajectory shape, message realness and disclosure patterns. We will rephrase our writing to describe an interaction regime modeled on real-user data, and list the lack of a human or blinded comparison as future work.

Could you state execution/interaction limits, crash or infrastructure-error handling, and whether Table 1's dollar costs include the user simulator? [Q5]

The solver-agents were run with their native harnesses (like Claude Code, Codex CLI), so it runs with the default settings that come with it (compaction, context limit, timeouts, etc) to mimic the default user experience. We don’t set any additional limits. Table 1's dollar cost doesn’t include the user simulator, but it is about 5-8% of the total cost in most cases, so it doesn’t affect the results much.

Thank you for the additional analysis. Even after excluding cases where the simulated user omitted requirements, the two strongest agents still performed substantially worse on multi-turn tasks than on their single-turn counterparts. This addresses my main concern, and I will raise my overall rating

Comments on the paper

1 comment
Area_Chair_Zn1xOpenReview ↗

Please engage in the discussion

Dear Reviewers,

As the ddl of author-reviewer discussion is approaching, could you please read the authors' responses and update your reviews if necessary?

As you know, we have only three reviewers in total. So each of your assessments may have a crucial impact on the final decision. Please be active. Thanks!

Regards,

Your AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.