acceptodds
Under review as a conference paper at ICLR 2027

CUA-SWE: When Computer-Use Agents Meet Visual Software Engineering

Abstract

Software development requires more than editing code: developers repeatedly run software, interact with its interfaces, visually inspect its behavior, and use these observations to decide what to change next and whether a change works. Existing coding agents and computer-use agents are largely studied in isolation, leaving this integrated development process underexplored. Diagnosing a runtime interaction failure requires agents to connect visual observations with the responsible code, then use the application again to verify the repair. We introduce CUA-SWE, a benchmark, a scalable and verifiable environment, and evaluation pipeline for software engineering with computer use. Beyond studying how GUI feedback supports diagnosis and repair, we ask whether agents can complete software engineering tasks when required specification or operational information is available only through the running application's visual interface. CUA-SWE spans four software engineering domains and requires agents to modify code and configuration, execute commands, interact with running software, and inspect visual feedback within the same task. Each task includes deterministic, task-specific tests that verify whether the resulting software satisfies the requirements and preserves specified behavior. Our evaluation characterizes how frontier agents combine source-level execution with application screenshots and graphical interaction to produce verified software changes. We examine performance across domains and task information requirements, alongside the development behaviors associated with successful repairs. CUA-SWE provides a unified testbed for studying how agents use visual feedback and interaction to guide software engineering, with executable correctness criteria for the resulting software.

Primary area:
datasets and benchmarks
Keywords:
computer-use agents, coding agents, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_v4fiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CUA-SWE contains 105 software-engineering tasks across four domains, combining code editing, screenshot-based GUI interaction, and protected behavioral tests. Hybrid access mainly improves performance on tasks requiring application materials, while source-specified tasks show little average improvement.

Critical strengths
  1. Four-domain coverage and protected functionality, regression, and permitted-change checks provide a useful evaluation resource (§3.2; Appendix B.3).
  2. Separating specification sources and patch correctness from access-rule compliance helps interpret performance (Appendices I, N).
Critical weaknesses
  1. I find the benchmark's distinctive contribution insufficiently established. As §5 and Appendix O.5 discuss, Programming with Pixels already combines visual development, coding tools, and programmatic verification. CUA-SWE broadens domain coverage and organizes tasks around application interaction, but the paper does not convincingly identify what new capability distinction or general insight it reveals.
  2. The main comparison offers limited additional insight into visual software engineering. Of 105 tasks, 63 depend on application materials; source-specified subsets show average gains of 0.0 points on Web and −0.6 on Game (Table 9). The authors acknowledge this pattern. I therefore read the aggregate gains as evidence that access to otherwise unavailable requirements matters. The repair examples are useful illustrations, but do not sufficiently develop a broader account of when visual feedback changes diagnosis or verification.
Questions for authors
  1. Which specific capability distinction or finding most clearly differentiates CUA-SWE from the closely related settings discussed in §5 and Appendix O.5?
  2. What conclusion about visual diagnosis or verification, beyond acquiring missing requirements, is supported by the existing source-specified results and trajectories?
3 comments

We thank the reviewer for the feedback. We believe the review contains several factual errors and misunderstands the main contribution of CUA-SWE. In this comment, we'd like to address these issues and the points where we disagree with the reviewer's assessment, so that we hope the reviewer can get a better understanding of our contribution.

Some Misunderstandings

"The repair examples are useful illustrations, but do not sufficiently develop a broader account of when visual feedback changes diagnosis or verification."

The paper's account is quantitative, not built on examples. App. P analyzes 214 reviewed trajectories across all domains and both conditions (Tables 12–18). It measures when agents first observe the faulty behavior relative to their first edit, whether they re-check the changed application in the GUI before submitting, how many screenshots and GUI actions successful and unsuccessful repairs use, and at which stage failed attempts break down. Re-checking the final edit through the GUI is associated with task success of 45.8% vs. 5.7% on Web, 68.1% vs. 5.9% on DevOps and 83.7% vs. 10.6% on Mobile. The examples in Sec. 4.3 illustrate these measured patterns.

Points where we respectfully disagree

1. CUA-SWE's distinction from Programming with Pixels is clear and fundamental.

"Programming with Pixels already combines visual development, coding tools, and programmatic verification… the paper does not convincingly identify what new capability distinction or general insight it reveals."

Programming with Pixels studies how an agent operates its development tools: the GUI it controls is an IDE, and its central finding is that agents do better when pixel-level IDE control is replaced by file and shell tools. CUA-SWE studies whether an agent can use the software it is building as a source of evidence. Code access is held fixed, with full shell access in both conditions, and the only variable is whether the agent can operate the running application. That application is where defects appear and, for 63 of 105 tasks, where the requirement exists. Programming with Pixels cannot measure this, because its GUI replaces the coding interface rather than adding access to the application under development, and most of its tasks come from existing benchmarks whose requirements are stated up front.

2. Access to requirements that live only in the running application is our central finding, not a limitation.

"I therefore read the aggregate gains as evidence that access to otherwise unavailable requirements matters."

That is precisely the problem CUA-SWE is built to study. In real development, the material needed to finish a task often can be observed only by running the software: using the app, playing the game, working with a live service. None of it is in the repository. We construct 63 of the 105 tasks this way on purpose, and on every task the defect itself appears in the running application (Sec. 3.2). Showing that current agents depend on, and differ sharply in, their ability to obtain such material is the insight the benchmark delivers.

This is a capability, not an access privilege. The agent has to recognize that information is missing, find it by operating the application, interpret it, and turn it into code. With identical GUI access, hybrid success on these 63 tasks ranges from 20.6% to 71.4% across models.

The comparison also shows visual diagnosis and verification beyond missing requirements. At the patch level, GUI access improves source-specified Web tasks by +7.4 pp on average and by +13.5 pp on the tasks any model solves, even though code-only agents have the full specification and can run tests and scripts. GPT-6-Astra gains +27.3 pp, twice the gain of any other model.

Given these clarifications, if our answers successfully address the concerns, we would appreciate if the reviewer could reconsider our overall score. We will also address the other questions below.

Below we answer the reviewer's two questions with the supporting numbers.

Q1. "Which specific capability distinction or finding most clearly differentiates CUA-SWE from the closely related settings discussed in §5 and Appendix O.5?"

CUA-SWE holds code access fixed and varies only whether the agent can operate the application it is building. Related settings vary something else:

What the GUI operatesCode accessWhat is variedRequirement can exist only at runtime
Programming with PixelsThe IDEThrough the IDE, optionally with file/shell toolsHow the agent edits codeNo
SWE-bench MultimodalNothing (static images)Repository tools—No
CUA-SWEThe running applicationFull CLI and shell in both conditionsAccess to the running applicationYes (63 of 105 tasks)

The resulting finding: access to the running application helps most where requirements live at runtime, and on source-specified tasks it improves repairs most for the strongest model (Q2).

Q2. "What conclusion about visual diagnosis or verification, beyond acquiring missing requirements, is supported by the existing source-specified results and trajectories?"

Three conclusions, each from the existing runs.

(a) GUI access improves repairs even when the specification is in the source. Patch correctness on source-specified Web tasks, code-only → hybrid (computed from the same runs; added to Table 9):

ModelΔ (pp)
GPT-6-Astra+27.3
Claude-Opus-4.8+13.6
GPT-5.6-Sol+9.1
Claude-Sonnet-5+9.1
Claude-Fable-5+4.5
GPT-5.6-Luna+0.0
Grok-4.6+0.0
GPT-5.6-Terra−4.5

The average is +7.4 pp, or +13.5 pp on the 12 Web S tasks that any model solves. On Game S tasks, 10 of 20 are solved by no model in either condition, which caps any measurable difference.

(b) Verifying in the GUI predicts success. Among edited hybrid attempts, re-checking the final edit through the GUI is associated with task success of 45.8% vs. 5.7% on Web, 68.1% vs. 5.9% on DevOps and 83.7% vs. 10.6% on Mobile (App. P, Table 15).

(c) GUI evidence adds to what execution alone provides. Code-only agents do run programs: 47 of 85 edited code-only trajectories obtain relevant runtime information through non-GUI channels such as tests and scripts (App. P, Table 14). Hybrid agents nonetheless repair more source-specified Web tasks, so the added gain comes from observing the assembled application, not from execution alone.

Conclusion

We thank the reviewer again for your time and feedback! Although there are several points which we disagree in your original assessment, we agree that our writing and presentation could have been better to avoid confusion, and your feedback can certainly help us improve in that regard. If our answers successfully address your concerns, we would appreciate if you could reconsider your assessment and improve our ratings. Thanks!

Dear reviewer, as we are close to the discussion deadline, would you let us know if our answers have addressed your concerns? If yes, would you consider improving our ratings? We will be ready to address any additional questions you may have. Thanks!

Reviewer_57mWOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CUA-SWE, a benchmark for software engineering agents that combine code modification with visual application interaction. It contains 105 tasks across web development, game development, mobile applications, and DevOps, with deterministic tests for task correctness and regression detection. The paper compares code-only agents against hybrid computer-use agents that additionally access screenshots and graphical interactions, reporting substantial improvements on tasks requiring information from running applications.

Critical strengths
  1. The benchmark could address a practical gap between coding agents and computer-use agents. Real software development often requires running applications, observing failures, and connecting visual behavior to source-code modifications. CUA-SWE evaluates these capabilities within a unified task.
  2. The evaluation uses executable correctness criteria rather than subjective visual judgments. Task-specific tests verify requested functionality and detect regressions, allowing different valid implementations without requiring exact reference-patch matching.
  3. The paired code-only and hybrid evaluations provide useful insights. The experiments distinguish tasks with source-specified requirements from those requiring application-material access, helping characterize when graphical interaction contributes to software development.
Critical weaknesses
  1. SWE-bench Multimodal already evaluates code changes involving visual information, while GameDevBench studies multimodal game development and visual feedback. CUA-SWE expands the domain coverage and combines runtime interaction with source editing, but the paper does not sufficiently demonstrate which new capabilities or failure modes cannot be evaluated using these existing benchmarks.
  2. Many application-material-dependent tasks require information unavailable to code-only agents. The paper's own analysis suggests that the hybrid advantage is concentrated in these tasks, while source-specified tasks show much smaller differences. This makes it difficult to isolate improvements in visual debugging from the simpler benefit of accessing otherwise missing specifications.
  3. The headline task-success metric mixes software correctness with interface compliance. A patch can pass the independent verifier but still be marked unsuccessful because of trajectory-validity constraints. While enforcing the benchmark protocol is reasonable, combining these factors obscures whether failures reflect software engineering limitations or harness compliance. Separate reporting of patch correctness would improve interpretability.
Questions for authors
  1. How does CUA-SWE differ from SWE-bench Multimodal and GameDevBench in the capabilities and failure modes it evaluates?

  2. How much of the hybrid improvement remains when code-only agents receive equivalent task specifications?

  3. How do the main conclusions change when reporting patch correctness separately from trajectory validity?

3 comments

We thank the reviewer for the feedback. We believe the review contains several factual errors and misunderstands the main contribution of CUA-SWE. In this comment, we'd like to address these issues and the points where we disagree with the reviewer's assessment, so that we hope the reviewer can get a better understanding of our contribution.

Some Misunderstandings

"The headline task-success metric mixes software correctness with interface compliance… Separate reporting of patch correctness would improve interpretability."

Patch correctness is already reported separately. The paper defines patch correctness P and task success S = P·U as distinct measures (Sec. 2, Sec. 4.1) and reports both for every model, domain and access condition, together with the rate of correct patches that fail trajectory validity (App. N, Table 11). Such cases never occur on DevOps or Mobile and are concentrated on Web. No main conclusion changes under patch correctness; in fact, the hybrid advantage grows (next comment). We keep U in the headline metric because it enforces each condition's access rules, which is what keeps the code-only versus hybrid comparison controlled.

Points where we respectfully disagree

1. Running the application to obtain essential materials is the problem we study, not a confound.

"This makes it difficult to isolate improvements in visual debugging from the simpler benefit of accessing otherwise missing specifications."

CUA-SWE is built around a situation that is routine in real development: the material needed to finish the task can be observed only by running the software. A developer has to use the app to see how a layout behaves, play the game to see what an in-game reference card requires, or work with a live service to learn how it actually behaves. None of this is in the repository. We construct 63 of the 105 tasks this way on purpose, and on every task the defect itself appears in the running application (Sec. 3.2). Treating this as a confound to be removed would remove the very problem the benchmark exists to measure.

Obtaining these materials is not a "simpler benefit" either. The agent has to recognize that information is missing, find it by operating the application, interpret it, and turn it into code. Access does not make this easy: with identical GUI access, hybrid success on these 63 tasks ranges from 20.6% to 71.4% across models. That spread measures a capability, not an access privilege.

Where the specification is in the source, the paper isolates visual diagnosis and verification as well. Every task is labeled by where its specification lives (App. A.3), and Tables 9–10 compare both conditions within each type. On source-specified Web tasks, GUI access improves patch correctness by +7.4 pp on average and by +13.5 pp on the tasks any model solves, with GPT-6-Astra gaining +27.3 pp.

2. CUA-SWE evaluates capabilities and failure modes that SWE-bench Multimodal and GameDevBench cannot.

"…the paper does not sufficiently demonstrate which new capabilities or failure modes cannot be evaluated using these existing benchmarks."

In SWE-bench Multimodal, visual information is a static image in the issue or tests; the agent never needs to operate the running application. GameDevBench builds features in one game engine from tutorials; its visual feedback consists of editor screenshots and recorded gameplay video, and its authors note that all tasks can be verified through code, with visual feedback as a supplement. In neither benchmark does the agent operate the running software with mouse and keyboard while editing code through the shell, and in neither does the requirement exist only in the running application.

CUA-SWE evaluates exactly these capabilities, across web, game, mobile and DevOps development, under a controlled code-only versus hybrid comparison:

  • reproducing a defect by operating the running software;
  • acquiring requirements that exist only in the running application;
  • verifying a repair in the running application before submitting.

It also exposes failure modes the other benchmarks cannot observe (App. P): agents that edit before observing the faulty behavior (32 of 116 edited hybrid trajectories), agents that never re-check their final edit in the GUI and succeed far less often (5.7% vs. 45.8% on Web), and agents that bypass the access rules to obtain runtime information (App. N).

Given these clarifications, if our answers successfully address the concerns, we would appreciate if the reviewer can reconsider the soundness score. We will also address the other questions below.

Below we answer the reviewer's three questions with the supporting numbers.

Q1. "How does CUA-SWE differ from SWE-bench Multimodal and GameDevBench in the capabilities and failure modes it evaluates?"

SWE-bench MultimodalGameDevBenchCUA-SWE
Agent operates the running application (mouse, keyboard)NoNo (editor screenshots, recorded video)Yes
Requirement can exist only in the running applicationNoNoYes (63 of 105 tasks)
DomainsJavaScript librariesGodot gamesWeb, game, mobile, DevOps
Controlled code-only vs. GUI comparison under identical testsNoWith and without screenshot/video add-onsYes, broken down by task type

The failure modes CUA-SWE exposes are listed in our first comment: editing before observing, not re-verifying the final edit, and bypassing access rules.

Q2. "How much of the hybrid improvement remains when code-only agents receive equivalent task specifications?"

Source-specified (S) tasks are exactly this condition: the full specification is in the instruction and source, available to both conditions. Averaged over eight models on identical tasks (Tables 9–10; patch-level values computed from the same runs and added to Table 9):

DomainTypeTasksTask success Δ (pp)Patch correctness Δ (pp)
WebS22+0.0+7.4
GameS20−0.6+0.6
WebM14+42.0+60.7
GameM9+23.6+27.8
DevOpsM20+48.1+48.1
MobileM20+38.8+39.1

On Web S tasks, five of eight models improve at the patch level (GPT-6-Astra +27.3 pp, Claude-Opus-4.8 +13.6, GPT-5.6-Sol +9.1, Claude-Sonnet-5 +9.1, Claude-Fable-5 +4.5); restricted to the 12 Web S tasks that any model solves, the average gain is +13.5 pp. On Game S tasks, 10 of 20 are solved by no model in either condition, which limits any measurable difference. For M tasks, supplying the material as text would require transcribing a live service's behavior or a drawing's relationships, which is part of what the agent is asked to do.

Q3. "How do the main conclusions change when reporting patch correctness separately from trajectory validity?"

They do not change, and the access effect becomes stronger (App. N, Table 11):

  • GPT-6-Astra remains first at 59.9% under both measures.
  • The average four-domain hybrid gain over code-only rises from +27.5 pp (task success) to +31.1 pp (patch correctness).
  • Correct patches that fail trajectory validity occur 0 times on DevOps and Mobile, 5 times on Game and 35 times on Web across nine models; they mainly affect Claude-Sonnet-5, whose four-domain score rises from 19.6% to 33.8%.

Conclusion

We thank the reviewer again for your time and feedback! Although there are several points which we disagree in your original assessment, we agree that our writing and presentation could have been better to avoid confusion, and your feedback can certainly help us improve in that regard. If our answers successfully address your concerns, we would appreciate if you could reconsider your assessment and improve our ratings. Thanks!

Dear reviewer, as we are close to the discussion deadline, would you let us know if our answers have addressed your concerns? If yes, would you consider improving our ratings? We will be ready to address any additional questions you may have. Thanks!

Reviewer_LG2dOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark with 105 tasks which require fixes in code. For each of the tasks, an agent can run the application and observe faults through computer use (e.g., by screenshotting). The fixes themselves are checked by hidden deterministic checks. The authors compare agent harnesses that only see code vs harnesses that see code and also have GUI access.

Critical strengths
  • Having agents debug software by running it interactively through CUA while also having access to code is still underexplored compared to pure coding (or pure computer-use) benchmarks
  • Deterministic checks
Critical weaknesses
  1. Fig 1 seems very misleading, as it is not established that most tasks are even solvable without access to the web interface. In fact, according to A.3, tasks are labeled either Source-specified or Application-material-dependent (M), and according to l 772, there's 58 M tasks. In one example in A.2/A.3 for example a task is said to only show an imported drawing that is central to the task in the GUI.

  2. The paper is not the first paper to look at agents in a scenario where they can both look at code and also start browsers. For example SWE-bench multimodal also equips its agent with a browser, and Programming with Pixels also seems very similar. So the novelty of this paper is somewhat incremental.

  3. I see substantial ways that the benchmark selection can be heavily biased becaused of its construction: First, all tasks are synthetically created by LMs (not specified which). it seems reasonable that the model that created the tasks might have an advantage in solving them. Second, there is a nontrivial difficulty selection/promotion, which again makes use of unspecified models. This even more clearly might bias the benchmark. This problem of potentially introduced biases is not discussed in the paper. Furthermore, the question of how seeds have been drawn is also not addressed.

  4. minor points:

    • the formalization in section 2 seems to be very vanilla and seems to be largely superfluous, the POMDP formalization is never even used.
    • Section 4.3, half of the paragraphs are mostly anecdotal. It's often not clear what is being talked about "three-attempt Game study", "Core Ball" etc. (only introduced in appendix)
Questions for authors
  1. Can we break down comparisons between Code and CUA agents based on the tasks that are actually achievable by both of them?

  2. How is difficulty calibrated exactly? What models were being used for this? How can we make sure this doesn't bias the benchmark?

  3. Which models are being used for task construction? How can we make sure that this doesn't bias the benchmark?

  4. How many task variants are there on average? Isn't there high correlation between task variants? Task count should ideally exclude very similar task variants

  5. I don't fully understand the explanation of "We begin with a concrete software behavior" that is used to seed the synthetic task generation

  6. For the SFT experiment, how closely related are the tasks between the training and test set? The 11% -> 33% improvement is only from 1->3 tasks

  7. Since deterministic checks/hidden tests are being used, have these been annotated for whether or not they might be overly specific? I.e., how was it made sure that every fix to the problem passes the tests?

3 comments

We thank the reviewer for the feedback. We believe the review contains several factual errors and misunderstands the main contribution of CUA-SWE. In this comment, we'd like to address these issues and the points where we respectfully disagree with the reviewer's assessment, so that we hope the reviewer can get a better understanding of our contribution.

  1. Misunderstanding of the main contribution "Fig 1 seems very misleading, as it is not established that most tasks are even solvable without access to the web interface."

This is not a flaw; it is the problem CUA-SWE is built to study. In real development, many issues surface only when the software runs: a selection lost after resizing, a game state that breaks after a jump, a service that misbehaves under live traffic. Developers find these issues by using the application, and the requirement often lives outside the repository, in a design drawing, a deployed service, or the product itself. CUA-SWE reproduces this setting. For 63 of 105 tasks, the requirement exists only in the running application, and for every task, human review confirms that the defect is observable through it before the task is admitted (Sec. 3.2). Fig. 1 measures what this setting is about: how much access to the running application adds to source-level development under identical tests. Code-only access is the reference that shows how far source access alone goes, and the paper reports it per task type (Tables 9–10).

  1. Novelty relative to SWE-bench Multimodal and Programming with Pixels "SWE-bench multimodal also equips its agent with a browser, and Programming with Pixels also seems very similar."

CUA-SWE is not "code plus a browser." It is an environment in which a computer-use agent develops software the way a developer does: full CLI and shell access to the repository, where it reads, edits, builds, tests and runs scripts, together with computer-use access through screenshots and mouse and keyboard actions to the running application it is modifying, in the same episode. Tasks are built so that the running application carries information the agent cannot finish without.

Neither cited work provides this setting. In SWE-bench Multimodal, visual information is a static image in the issue or the tests; solving a task does not require operating the running application, and the benchmark does not test whether doing so matters. Programming with Pixels gives a computer-use agent an IDE: the GUI it operates is the development tool, not the application under development, and most of its tasks are drawn from existing datasets. Its central finding, that agents do better when pixel-level IDE control is replaced by file and shell tools, is our starting point: agents keep full shell access, and computer use is directed at the software being built. Neither work offers a controlled comparison of code-only and hybrid access under identical tests, or behavioral verifiers validated against negative repairs.

  1. Task construction and potential bias "All tasks are synthetically created by LMs (not specified which)… the question of how seeds have been drawn is also not addressed."

This is factually incorrect. Tasks are adapted from open-source applications and documented software defects, such as the clear-on-blur bug reported in React Datepicker and the gauge semantics of StatsD, or built as purpose-designed applications when controlled behavior is needed (App. A.7). The LLM author writes the task package from a seed drawn from these sources; it does not invent the software or the defect. Seeds and difficulty calibration are specified in Sec. 3.2–3.3, App. A.8 and Fig. 6.

The construction process also leaves no room for model-specific bias in scoring. Correctness is defined by each task's verifier, which must fail on the original program, pass on the reference repair, and reject plausible partial repairs; no model's behavior enters it. Construction trials only guide which variant to build next, and all tasks were fixed before the evaluations in Sec. 4. The results confirm this: rankings do not follow model provider, with models from the same provider at both the top and the bottom of the table.

  1. Task variants "Task count should ideally exclude very similar task variants."

Each variant has its own requirement, codebase and verifier, just as SWE-bench counts each issue in a repository as a separate task. The paper already accounts for related tasks: results are reported per family (App. H), and the uncertainty analysis resamples whole families (App. F), so the reported intervals already reflect correlation within families.

Given these corrections, we ask the reviewer to reconsider the soundness and informativeness scores. We will now address your other questions in the comment below.

We will also address the other questions in the comment below.

Q1. "Can we break down comparisons between Code and CUA agents based on the tasks that are actually achievable by both of them?"

This breakdown is in the paper. Every task is labeled before evaluation by where its specification lives (App. A.3, Table 2): 42 source-specified (S) tasks and 63 application-material-dependent (M) tasks. Tables 9–10 compare both conditions on identical tasks within each type, averaged over eight models:

DomainTypeTasksCode-only (%)Hybrid (%)Δ (pp)
WebS2223.923.9+0.0
WebM148.950.9+42.0
GameS2010.610.0−0.6
GameM92.826.4+23.6
DevOpsM200.648.8+48.1
MobileM200.639.4+38.8

On S tasks, the strongest agent still gains substantially (GPT-6-Astra: +27.3 pp on Web S tasks), while weaker agents do not yet convert GUI evidence into repairs.

Q2/Q3. "What models were being used… How can we make sure this doesn't bias the benchmark?"

Models like GPT-6-astra and those that come after were used during the construction. Again, the results show no provider advantage as we explained in point 3 of the above comment. Four-domain task success ranges from 59.9% to 15.6% among OpenAI models and from 39.9% to 19.6% among Anthropic models, and Grok-4.6 (41.9%) is within 0.3 pp of second place.

Q4. "How many task variants are there on average?"

The 105 tasks span 43 families: Web 9, Game 12, DevOps 10, Mobile 12 (2.4 tasks per family on average). Resampling whole families, GPT-6-Astra's Game pass@3 lead over GPT-5.6-Sol is 31.0 pp with a 95% interval of [11.5, 50.0] (App. F).

Q5. "I don't fully understand the explanation of 'We begin with a concrete software behavior'."

A seed fixes an application, a development need, and an interaction that exposes the behavior. For example, in Captain Callisto, collecting one item moves the inventory counter from 0/2 to 2/2. From this seed, the LLM author builds the runnable task, reference and negative repairs, and tests; human reviewers then confirm that the defect reproduces and is observable through the agent's interface (Sec. 3.2, App. A.8).

Q6. "For the SFT experiment, how closely related are the tasks between the training and test set?"

The SFT study uses a 27/9 train/test split of Web tasks, and the held-out pass rate rises from 1/9 to 3/9 (Sec. 4.4). The paper presents it as an initial study giving early positive indications of a learning signal and states that its core contribution is the benchmark and environment. No conclusion of the current manuscript depends on this experiment and we are leaving the post-training experiments for the future version of this manuscript.

Q7. "How was it made sure that every fix to the problem passes the tests?"

Verifiers check the behavior of the submitted program under controlled interactions, not patch content: correctness "depends on the behavior of c_T, rather than on matching the reference patch" (Sec. 3.2; examples in App. B.3). Accepted repairs come from 11 different agents writing their own patches; for example, GPT-6-Astra and Grok-4.6 each pass the same 18 Web tasks with independently generated repairs.

Minor: formalization. "The POMDP formalization is never even used." Sec. 2 defines the agent's policy over its interaction history h_t, and Sec. 3.2 uses h_t to separate what the agent can observe from the privileged state the verifier checks.

Minor: Section 4.3. "Half of the paragraphs are mostly anecdotal." The cases illustrate a quantitative analysis of 214 reviewed trajectories (App. P, Tables 12–18). Among edited hybrid attempts, re-checking the final edit through the GUI is associated with task success of 45.8% vs. 5.7% on Web, 68.1% vs. 5.9% on DevOps, and 83.7% vs. 10.6% on Mobile. The three-attempt study is introduced in Sec. 4.1 (Table 4).

Closing Remark

We thank the reviewer again for your time and feedback! Although there are several points which we disagree in your original assessment, we agree that our writing and presentation could have been better to avoid confusion, and your feedback can certainly help us improve in that regard. If our answers successfully address your concerns, we would appreciate if you could reconsider your assessment and improve our ratings. Thanks!

Dear reviewer, as we are close to the discussion deadline, would you let us know if our answers have addressed your concerns? If yes, would you consider improving our ratings? We will be ready to address any additional questions you may have. Thanks!

Comments on the paper

1 comment

Dear reviewers,

as the discussion ddl is approaching, could you please both see whether your concerns are addressed by author rebuttal and follow-up if you have additional concerns?

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.