acceptodds
Under review as a conference paper at ICLR 2027

PhotoArena: Benchmarking Pure-GUI Computer-Use Agents in Adobe Photoshop via Dual-Track Verification

Abstract

Professional creative software offers multiple routes to task completion, including visual GUI interaction and application scripting. When programmatic shortcuts are available, successful task completion alone does not establish an agent's ability to perform the corresponding visual interactions. We therefore focus on a specific question: can computer-use agents complete professional editing workflows using only screenshots, mouse actions, and keyboard inputs? Studying this capability raises two evaluation challenges. First, the interaction protocol needs to distinguish pure GUI performance from task completion through scripts or application APIs. Second, assessing these workflows calls for evidence of both execution and delivery: a final rendered image can conceal missing task-required editable structures, while final artifacts alone do not reveal whether required intermediate operations were performed. To address these challenges, we introduce **PhotoArena**, a benchmark for professional creative tasks in Adobe Photoshop under a strict **GUI-only interaction protocol**, paired with a **dual-track evaluation framework**. The Outcome Delivery Track checks prompt-required PSD structure and visual completeness before assessing visual consistency, instruction following, and visual quality. The Process Adherence Track aligns execution trajectories with semantic operations using an LLM-based forward anchor and evaluates step-level execution without requiring agent frames to visually match the expert during alignment. We report the two tracks separately to distinguish procedural adherence from final deliverable quality. Evaluated frontier agents execute at least 60% of task-relevant operations, yet end-to-end task success remains at or below 10% under the GUI-only protocol. These findings highlight the difficulty of translating local interaction successes into complete, editable deliverables under the GUI-only protocol.

Primary area:
datasets and benchmarks
Keywords:
Computer-use agents, GUI automation, Adobe Photoshop

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YpVfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhotoArena evaluates GUI-only agents on 39 Photoshop workflows. It separates final delivery, assessed through PSD structure and visual quality, from process adherence, assessed by aligning agent actions with expert milestones. Experiments compare four model configurations and two interaction harnesses, reporting low delivery success despite higher scores on evaluable operations.

Critical strengths
  • PSD checks catch missing masks and clipping relationships that a flattened image can conceal (Appendix A.5). Structured access is restricted to the evaluator, and failed exports remain in the denominator.
  • The process track distinguishes executing an operation from executing it correctly. Comparing local changes also helps avoid repeatedly penalizing inherited errors, Figure 5 illustrates this distinction.
  • Figure 3 shows that harness effects depend on the model. Peekaboo raises Opus's mean delivery score from 0.026 to 0.274, but changes Codex from 0.094 to 0.098 while runtime rises from 37.2 to 83.1 minutes.
Critical weaknesses
  • GPT-4o judges completeness, visual quality, anchors, and individual beats, yet no quantitative human validation is reported (Appendix A.2). Repeated votes from one model do not establish accuracy. This matters especially when one completeness decision can zero the delivery score.
  • Process scores exclude unlocalized beats and uncertain judgments, potentially using different task subsets across dimensions. Table 1 omits the coverage counts promised in Appendix A.1. Without them, action scores of 1.000 cannot establish broad action competence.
  • Binary-search anchoring assumes that a visible effect persists. Undo, visibility changes, and cancelled previews violate this assumption. Appendix A.8 even counts open-dialog previews as execution. Algorithm 1's boundary check does not resolve non-monotone windows.
  • With one run per task, 7.7% versus 5.1% success would mean three successes versus two. Repeat-run results and uncertainty estimates are not reported. Nor are counts of tasks receiving the topology-gate waivers permitted in Appendix A.5.
Questions for authors
  1. How accurate are outcome gates and milestone anchors against expert labels? Do model rankings survive human adjudication?
  2. How many tasks and beats contribute to each A/S/D score? Do conclusions hold on common coverage, separating unattempted operations from localization failures?
  3. How often are visible effects non-monotone? Please compare binary-search anchors with expert labels or exhaustive scans on affected trajectories.
Reviewer_RXd2OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces PHOTOARENA, a benchmark for evaluating computer-use agents on professional Adobe Photoshop workflows under a strict GUI-only interaction protocol. Agents receive only screenshots and must act through mouse and keyboard inputs, while scripting interfaces, accessibility trees, and document object models are withheld. The benchmark contains 39 long-horizon tasks and combines two complementary evaluation tracks: an Outcome Delivery Track that verifies PSD structure and visual completeness before scoring consistency, instruction following, and craftsmanship, and a Process Adherence Track that localizes and scores task-relevant operations along execution trajectories. Experiments on four frontier model systems reveal a substantial gap between localized interface control and end-to-end professional delivery: action-level scores are often high, whereas the best strict delivery success rate is only 7.7%. A harness comparison further shows that visual re-verification can substantially improve one model but provide little benefit to another. Overall, the paper offers a useful diagnostic benchmark and empirical baseline for professional visual software agents, although the strength of its conclusions is constrained by the modest benchmark scale, reliance on VLM-based judgments, limited statistical reporting, and the narrow focus on Photoshop.

Critical strengths
  • The strict separation between agent-facing GUI interactions and evaluator-side structural access cleanly isolates visual computer-use ability from scripting or API-based automation.
  • The combination of programmatic PSD inspection, visual completeness checks, multi-facet outcome scoring, and step-level trajectory evaluation captures failure modes that final-image-only benchmarks would miss.
  • Semantic forward anchoring avoids rigid pixel-level synchronization with expert demonstrations, while change-based Step Match scoring prevents early cascading errors from being penalized repeatedly.
  • The paper provides detailed evaluator prompts, scoring schemas, information-isolation rules, qualitative trajectory decompositions, and a useful harness ablation demonstrating when visual re-verification helps.
Critical weaknesses
  • The benchmark contains only 39 tasks, focuses on a single proprietary application and software version, and evaluates a small number of model systems. The paper also does not report confidence intervals, repeated-trial variance, or significance testing, making the robustness of the model rankings uncertain.
  • Many evaluation components depend on GPT-4o judgments, but the paper does not provide human agreement studies, judge calibration, error rates, or systematic bias analysis. Three-vote binary consensus improves stability but does not establish that the resulting labels are correct.
  • Unlocalized beats, unscorable trajectories, and low-certainty judgments are excluded rather than counted as failures, and different process dimensions may cover different task subsets. Consequently, high Action Execution scores should not be interpreted as whole-workflow completion and may be optimistic.
Questions for authors

See weakness

Reviewer_y1yyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a benchmark of 39 Photoshop tasks under a GUI-only interaction protocol, evaluating final deliverables and execution processes separately. It combines PSD structure checks, visual judgments, and trajectory alignment to distinguish successful local operations from complete task delivery.

Critical strengths
  1. Evaluating editable PSD structure can reveal failures that are invisible in the final image. This reflects a practical requirement of professional editing workflows.
  2. Reporting process performance separately from final outcomes, while counting crashes and export failures in the outcome denominator, helps identify where agents fail.
Critical weaknesses
  1. Baseline coverage is narrow. Sections 5.1–5.3 evaluate four GPT/Claude models and compare two harnesses for two of them, but omit dedicated GUI agents and agent frameworks such as UI-TARS and Agent S3. Their performance under the same GUI-only restrictions and budget remains unknown. This limits how broadly the low success rates characterize current computer-use systems; OSWorld scores obtained with different tools or multiple attempts are not directly comparable.
  2. The main concern is evaluator validity. Completeness checks, visual scoring, alignment, and per-beat scoring rely on GPT-4o, but quantitative agreement with human experts is not reported (Appendix A.2). Voting in some stages cannot rule out systematic errors. The paper explicitly excludes some unjudgeable steps, yet Table 1 lacks the coverage counts promised in A.1. The process scores therefore need both human validation and their task/beat denominators before they can support the stated conclusions.
Questions for authors
  1. Can the authors evaluate a dedicated GUI agent or agent framework available at submission time, such as UI-TARS or Agent S3, under the same screenshot-only interface, allowed actions, and execution budget? If adaptation is impractical, what specifically prevents it?
  2. On an independently human-labeled sample, what are the agreement rates for each judgment and the false acceptance and false rejection rates for binary decisions? Do the main findings survive human adjudication?
  3. How does the binary-search anchor handle undo, hidden layers, or canceled previews, which can violate the monotonicity assumption in A.6? Please report alignment accuracy and per-model task/beat coverage, including reasons for exclusions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reply to all reviewers

We sincerely thank all three reviewers (Reviewer y1yy, Reviewer RXd2, and Reviewer YpVf) for their evaluations of our work.

We are glad that all three reviewers collectively recognized the core contribution of our work: the pure-GUI isolation, the structural verification through PSD instead of flat images, and the decouple of the end-to-end evaluation and the process evaluation. We are also deeply grateful for the reviewers' critical and constructive feedback points. Below, we provide our response addressing the shared parts raised across all reviews.

Dedicated GUI Baselines

[Directly addressing: Reviewer y1yy, Reviewer RXd2]

Both reviewers noted that while we evaluated four frontier agent systems, evaluating dedicated GUI agent frameworks under identical conditions would better contextualize the results. In response, we evaluated two representative, dedicated computer-use frameworks under the exact same pure-GUI constraints:

• Agent S3

• Standalone UI-TARS

The results under the Table 1 protocol are shown below:

Model FamilyModel SystemG_structG_compMean RStrict (R=1)Runtime
Dedicated GUI AgentsAgent S310.3%10.3%0.09410.3%92.6 min
Standalone UI-TARS0.0%0.0% 0.0000.0%118.6 min

We find that the dedicated GUI agents in fact perform worse compared with the general frontier agent systems. For the dedicated GUI agents, the predominant failure modes were artifact and structure contract violations, such as failure to export the required deliverable files, navigating complex Photoshop sub-menus, and maintaining layer topology. More execution results and breakdowns are detailed in Section 5.3 & Appendix C.1.

Human Calibration

[Directly addressing: Reviewer y1yy, Reviewer RXd2, Reviewer YpVf]

Reviewers questioned the validity of relying on GPT-4o for visual judgements without human calibration data, noting that our fail-closed gates could produce systematic errors. In response, we conducted a blind human calibration study with professional image editors on a sampled set of trajectories and deliverables. We find an ~80% criterion-level agreement between human experts and the VLM judge. Furthermore, our evaluation gate is explicitly conservative: human adjudication confirmed that no failed attempts were falsely marked as successful by the GPT-4o evaluation pipeline. More details are in Section 4.3 & Appendix A.2.

Process Track Denominators & Coverage

[Directly addressing: Reviewer y1yy, Reviewer RXd2, Reviewer YpVf]

Reviewers pointed out that reporting high Action Execution scores (Ā ≈ 1.0) without explicit denominators can mask task coverage, as unlocalized steps were excluded. In response, we clarify the metric definitions: Ā was originally designed as conditional control precision (measuring whether the agent took the correct action when a milestone was localized), rather than whole-workflow completion. We have further updated Table 1 and Appendix C to explicitly report planned beats, attempted steps, localized anchors, and unlocalized steps for each system. We also added an unconditional macro score that assigns zero to unlocalized steps, detailed in Table 1 & Appendix A.1.


Besides the main parts, we acknowledge that the non-monotonicity issue in our binary search algorithm exists; however, the existing process-based evaluation still holds. Our process-based evaluation mainly serves as a heuristic for analyzing and possibly improving the model, while the end-to-end signal remains the primary metric for the benchmark. We also recognize that the benchmark scale of ~40 tasks is modest, and we are actively working to expand it in future works.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.