- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PhotoArena evaluates GUI-only agents on 39 Photoshop workflows. It separates final delivery, assessed through PSD structure and visual quality, from process adherence, assessed by aligning agent actions with expert milestones. Experiments compare four model configurations and two interaction harnesses, reporting low delivery success despite higher scores on evaluable operations.
Critical strengths
- PSD checks catch missing masks and clipping relationships that a flattened image can conceal (Appendix A.5). Structured access is restricted to the evaluator, and failed exports remain in the denominator.
- The process track distinguishes executing an operation from executing it correctly. Comparing local changes also helps avoid repeatedly penalizing inherited errors, Figure 5 illustrates this distinction.
- Figure 3 shows that harness effects depend on the model. Peekaboo raises Opus's mean delivery score from 0.026 to 0.274, but changes Codex from 0.094 to 0.098 while runtime rises from 37.2 to 83.1 minutes.
Critical weaknesses
- GPT-4o judges completeness, visual quality, anchors, and individual beats, yet no quantitative human validation is reported (Appendix A.2). Repeated votes from one model do not establish accuracy. This matters especially when one completeness decision can zero the delivery score.
- Process scores exclude unlocalized beats and uncertain judgments, potentially using different task subsets across dimensions. Table 1 omits the coverage counts promised in Appendix A.1. Without them, action scores of 1.000 cannot establish broad action competence.
- Binary-search anchoring assumes that a visible effect persists. Undo, visibility changes, and cancelled previews violate this assumption. Appendix A.8 even counts open-dialog previews as execution. Algorithm 1's boundary check does not resolve non-monotone windows.
- With one run per task, 7.7% versus 5.1% success would mean three successes versus two. Repeat-run results and uncertainty estimates are not reported. Nor are counts of tasks receiving the topology-gate waivers permitted in Appendix A.5.
Questions for authors
- How accurate are outcome gates and milestone anchors against expert labels? Do model rankings survive human adjudication?
- How many tasks and beats contribute to each A/S/D score? Do conclusions hold on common coverage, separating unattempted operations from localization failures?
- How often are visible effects non-monotone? Please compare binary-search anchors with expert labels or exhaustive scans on affected trajectories.
Reply to all reviewers
We sincerely thank all three reviewers (Reviewer y1yy, Reviewer RXd2, and Reviewer YpVf) for their evaluations of our work.
We are glad that all three reviewers collectively recognized the core contribution of our work: the pure-GUI isolation, the structural verification through PSD instead of flat images, and the decouple of the end-to-end evaluation and the process evaluation. We are also deeply grateful for the reviewers' critical and constructive feedback points. Below, we provide our response addressing the shared parts raised across all reviews.
Dedicated GUI Baselines
[Directly addressing: Reviewer y1yy, Reviewer RXd2]
Both reviewers noted that while we evaluated four frontier agent systems, evaluating dedicated GUI agent frameworks under identical conditions would better contextualize the results. In response, we evaluated two representative, dedicated computer-use frameworks under the exact same pure-GUI constraints:
• Agent S3
• Standalone UI-TARS
The results under the Table 1 protocol are shown below:
We find that the dedicated GUI agents in fact perform worse compared with the general frontier agent systems. For the dedicated GUI agents, the predominant failure modes were artifact and structure contract violations, such as failure to export the required deliverable files, navigating complex Photoshop sub-menus, and maintaining layer topology. More execution results and breakdowns are detailed in Section 5.3 & Appendix C.1.
Human Calibration
[Directly addressing: Reviewer y1yy, Reviewer RXd2, Reviewer YpVf]
Reviewers questioned the validity of relying on GPT-4o for visual judgements without human calibration data, noting that our fail-closed gates could produce systematic errors. In response, we conducted a blind human calibration study with professional image editors on a sampled set of trajectories and deliverables. We find an ~80% criterion-level agreement between human experts and the VLM judge. Furthermore, our evaluation gate is explicitly conservative: human adjudication confirmed that no failed attempts were falsely marked as successful by the GPT-4o evaluation pipeline. More details are in Section 4.3 & Appendix A.2.
Process Track Denominators & Coverage
[Directly addressing: Reviewer y1yy, Reviewer RXd2, Reviewer YpVf]
Reviewers pointed out that reporting high Action Execution scores (Ā ≈ 1.0) without explicit denominators can mask task coverage, as unlocalized steps were excluded. In response, we clarify the metric definitions: Ā was originally designed as conditional control precision (measuring whether the agent took the correct action when a milestone was localized), rather than whole-workflow completion. We have further updated Table 1 and Appendix C to explicitly report planned beats, attempted steps, localized anchors, and unlocalized steps for each system. We also added an unconditional macro score that assigns zero to unlocalized steps, detailed in Table 1 & Appendix A.1.
Besides the main parts, we acknowledge that the non-monotonicity issue in our binary search algorithm exists; however, the existing process-based evaluation still holds. Our process-based evaluation mainly serves as a heuristic for analyzing and possibly improving the model, while the end-to-end signal remains the primary metric for the benchmark. We also recognize that the benchmark scale of ~40 tasks is modest, and we are actively working to expand it in future works.