- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CUA-SWE contains 105 software-engineering tasks across four domains, combining code editing, screenshot-based GUI interaction, and protected behavioral tests. Hybrid access mainly improves performance on tasks requiring application materials, while source-specified tasks show little average improvement.
Critical strengths
- Four-domain coverage and protected functionality, regression, and permitted-change checks provide a useful evaluation resource (§3.2; Appendix B.3).
- Separating specification sources and patch correctness from access-rule compliance helps interpret performance (Appendices I, N).
Critical weaknesses
- I find the benchmark's distinctive contribution insufficiently established. As §5 and Appendix O.5 discuss, Programming with Pixels already combines visual development, coding tools, and programmatic verification. CUA-SWE broadens domain coverage and organizes tasks around application interaction, but the paper does not convincingly identify what new capability distinction or general insight it reveals.
- The main comparison offers limited additional insight into visual software engineering. Of 105 tasks, 63 depend on application materials; source-specified subsets show average gains of 0.0 points on Web and −0.6 on Game (Table 9). The authors acknowledge this pattern. I therefore read the aggregate gains as evidence that access to otherwise unavailable requirements matters. The repair examples are useful illustrations, but do not sufficiently develop a broader account of when visual feedback changes diagnosis or verification.
Questions for authors
- Which specific capability distinction or finding most clearly differentiates CUA-SWE from the closely related settings discussed in §5 and Appendix O.5?
- What conclusion about visual diagnosis or verification, beyond acquiring missing requirements, is supported by the existing source-specified results and trajectories?
3 comments
Below we answer the reviewer's two questions with the supporting numbers.
Q1. "Which specific capability distinction or finding most clearly differentiates CUA-SWE from the closely related settings discussed in §5 and Appendix O.5?"
CUA-SWE holds code access fixed and varies only whether the agent can operate the application it is building. Related settings vary something else:
| What the GUI operates | Code access | What is varied | Requirement can exist only at runtime | |
|---|---|---|---|---|
| Programming with Pixels | The IDE | Through the IDE, optionally with file/shell tools | How the agent edits code | No |
| SWE-bench Multimodal | Nothing (static images) | Repository tools | — | No |
| CUA-SWE | The running application | Full CLI and shell in both conditions | Access to the running application | Yes (63 of 105 tasks) |
The resulting finding: access to the running application helps most where requirements live at runtime, and on source-specified tasks it improves repairs most for the strongest model (Q2).
Q2. "What conclusion about visual diagnosis or verification, beyond acquiring missing requirements, is supported by the existing source-specified results and trajectories?"
Three conclusions, each from the existing runs.
(a) GUI access improves repairs even when the specification is in the source. Patch correctness on source-specified Web tasks, code-only → hybrid (computed from the same runs; added to Table 9):
| Model | Δ (pp) |
|---|---|
| GPT-6-Astra | +27.3 |
| Claude-Opus-4.8 | +13.6 |
| GPT-5.6-Sol | +9.1 |
| Claude-Sonnet-5 | +9.1 |
| Claude-Fable-5 | +4.5 |
| GPT-5.6-Luna | +0.0 |
| Grok-4.6 | +0.0 |
| GPT-5.6-Terra | −4.5 |
The average is +7.4 pp, or +13.5 pp on the 12 Web S tasks that any model solves. On Game S tasks, 10 of 20 are solved by no model in either condition, which caps any measurable difference.
(b) Verifying in the GUI predicts success. Among edited hybrid attempts, re-checking the final edit through the GUI is associated with task success of 45.8% vs. 5.7% on Web, 68.1% vs. 5.9% on DevOps and 83.7% vs. 10.6% on Mobile (App. P, Table 15).
(c) GUI evidence adds to what execution alone provides. Code-only agents do run programs: 47 of 85 edited code-only trajectories obtain relevant runtime information through non-GUI channels such as tests and scripts (App. P, Table 14). Hybrid agents nonetheless repair more source-specified Web tasks, so the added gain comes from observing the assembled application, not from execution alone.
Conclusion
We thank the reviewer again for your time and feedback! Although there are several points which we disagree in your original assessment, we agree that our writing and presentation could have been better to avoid confusion, and your feedback can certainly help us improve in that regard. If our answers successfully address your concerns, we would appreciate if you could reconsider your assessment and improve our ratings. Thanks!
Dear reviewer, as we are close to the discussion deadline, would you let us know if our answers have addressed your concerns? If yes, would you consider improving our ratings? We will be ready to address any additional questions you may have. Thanks!
We thank the reviewer for the feedback. We believe the review contains several factual errors and misunderstands the main contribution of CUA-SWE. In this comment, we'd like to address these issues and the points where we disagree with the reviewer's assessment, so that we hope the reviewer can get a better understanding of our contribution.
Some Misunderstandings
"The repair examples are useful illustrations, but do not sufficiently develop a broader account of when visual feedback changes diagnosis or verification."
The paper's account is quantitative, not built on examples. App. P analyzes 214 reviewed trajectories across all domains and both conditions (Tables 12–18). It measures when agents first observe the faulty behavior relative to their first edit, whether they re-check the changed application in the GUI before submitting, how many screenshots and GUI actions successful and unsuccessful repairs use, and at which stage failed attempts break down. Re-checking the final edit through the GUI is associated with task success of 45.8% vs. 5.7% on Web, 68.1% vs. 5.9% on DevOps and 83.7% vs. 10.6% on Mobile. The examples in Sec. 4.3 illustrate these measured patterns.
Points where we respectfully disagree
1. CUA-SWE's distinction from Programming with Pixels is clear and fundamental.
"Programming with Pixels already combines visual development, coding tools, and programmatic verification… the paper does not convincingly identify what new capability distinction or general insight it reveals."
Programming with Pixels studies how an agent operates its development tools: the GUI it controls is an IDE, and its central finding is that agents do better when pixel-level IDE control is replaced by file and shell tools. CUA-SWE studies whether an agent can use the software it is building as a source of evidence. Code access is held fixed, with full shell access in both conditions, and the only variable is whether the agent can operate the running application. That application is where defects appear and, for 63 of 105 tasks, where the requirement exists. Programming with Pixels cannot measure this, because its GUI replaces the coding interface rather than adding access to the application under development, and most of its tasks come from existing benchmarks whose requirements are stated up front.
2. Access to requirements that live only in the running application is our central finding, not a limitation.
"I therefore read the aggregate gains as evidence that access to otherwise unavailable requirements matters."
That is precisely the problem CUA-SWE is built to study. In real development, the material needed to finish a task often can be observed only by running the software: using the app, playing the game, working with a live service. None of it is in the repository. We construct 63 of the 105 tasks this way on purpose, and on every task the defect itself appears in the running application (Sec. 3.2). Showing that current agents depend on, and differ sharply in, their ability to obtain such material is the insight the benchmark delivers.
This is a capability, not an access privilege. The agent has to recognize that information is missing, find it by operating the application, interpret it, and turn it into code. With identical GUI access, hybrid success on these 63 tasks ranges from 20.6% to 71.4% across models.
The comparison also shows visual diagnosis and verification beyond missing requirements. At the patch level, GUI access improves source-specified Web tasks by +7.4 pp on average and by +13.5 pp on the tasks any model solves, even though code-only agents have the full specification and can run tests and scripts. GPT-6-Astra gains +27.3 pp, twice the gain of any other model.
Given these clarifications, if our answers successfully address the concerns, we would appreciate if the reviewer could reconsider our overall score. We will also address the other questions below.