acceptodds
Under review as a conference paper at ICLR 2027

How GUI Agents Fail: GraceBench for Execution Stability Diagnosis

Abstract

As GUI agents are increasingly deployed for mobile task automation, their reliability depends not only on final task outcomes, but also on whether they reach these outcomes through stable interaction trajectories. Whether in daily automation scenarios or testing workflows, unnecessary actions, detours, and execution errors can add noise to collected data, introduce irreversible side effects, or undermine trust, even when the agent eventually completes the task. It is therefore important to systematically diagnose the execution stability, including whether an agent deviates from a feasible path, enters a loop, becomes stagnant, or recovers from unstable actions. Existing benchmarks, however, primarily assess final task success or a small set of intermediate milestones, leaving step-level execution behavior unclear. To bridge this gap, we introduce GraceBench, an online evaluation framework that represents the reachable interaction space of mobile applications as UI Transition Graphs (UTGs), providing a structural reference for step-level execution diagnosis. To support reliable alignment between dynamic UI observations and graph states, GraceBench uses video-grounded trajectory evaluation over this graph-aligned representation, assessing process stability beyond final success by measuring metrics such as deviation rate, loop burden rate, and recovery ability. GraceBench contains 169 tasks spanning 10 applications and constructs 10 UTGs with 2,909 canonical GUI states and 3,116 observed transitions. Across 3,718 trajectories, GraceBench shows that outcome scores hide execution instability: GPT-5.6 Sol achieves the highest success rate (SR) but the lowest recovery ability, 51.24% of successful runs still contain a deviation or loop, and trajectories with the same success rate (SR) and completion rate (CR) can differ substantially in recovery behavior. Code will be made publicly available.

Primary area:
datasets and benchmarks
Keywords:
GUI Agent, Benchmark, Stability Diagnosis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X76jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GraceBench for evaluating the execution stability of mobile GUI agents. The authors construct UI Transition Graphs (UTGs) through app exploration and manual review, extract task-specific UTG subgraphs containing reference paths and alternative valid routes, and align action-synchronized execution videos with the graphs to diagnose stagnation, deviation, looping, and recovery. The benchmark contains 169 tasks across 10 applications and evaluates 11 models under detailed and concise instructions, yielding 3,718 trajectories. The experiments show that 51.24% of successful trajectories still contain a deviation or loop, and that identical task outcomes and completion rates can correspond to different process metrics. Overall, GraceBench provides graph-aligned video evidence and process-level diagnostics for deviations, loops, and recovery weaknesses, supporting future research on state tracking, loop detection, and recovery policies.

Critical strengths
  1. The research question has practical significance. Final success rate alone cannot fully characterize unnecessary actions, repeated loops, or recovery after errors during execution. These behaviors increase interaction costs and may introduce noise into performance measurements and logs collected along the trajectory. Evaluating task outcomes separately from execution process quality is a valuable research direction.

  2. The benchmark construction and data collection are substantial, and the associated procedures are described in detail. The paper details app exploration, state normalization, task-subgraph extraction, execution recording, and metric calculation. The tasks and trajectories collected from real applications, together with paired detailed and concise instructions for the same task, provide a resource for analyzing behavior under different execution conditions. The authors also provide substantive human auditing and statistical analyses. Outcome-controlled comparisons, statistical analyses that cluster by task, and cross-task ranking consistency analyses support the view that process metrics contain information beyond final success rate.

Critical weaknesses
  1. The measurement validity of the graph reference and stability metrics has not been sufficiently established. Section 3.3 labels steps that leave the task subgraph or enter a state outside the application-level UTG as deviations, and treats a return to the task subgraph as recovery. However, graphs obtained through finite exploration do not cover all valid task paths. Although the authors manually verify the validity of the collected states and transitions, this does not establish that actions outside the graph are necessarily erroneous or unstable. The paper does not quantify what proportion of deviation labels actually correspond to valid routes not captured by the graph, reasonable exploration, or transient environment states, nor whether these cases affect the main metrics and conclusions. The graph sizes and one-hop neighborhood statistics in Appendix D.1 also do not directly establish adequate coverage of valid paths.

    This issue also affects the interpretation of metrics such as completion rate (CR). Appendix B defines CR as the fraction of required transitions completed, while Appendix C.1 explicitly distinguishes final-state success from prescribed-path coverage. It therefore remains necessary to clarify how the required transitions are constructed for multi-path tasks and whether CR should be interpreted as task progress or reference-path coverage. If equal CR does not imply equal semantic task progress, the interpretation of outcome-controlled comparisons based on CR matching is also limited.

  2. The video-based evaluator is insufficiently specified, and the benefits of its components are not adequately validated. Section 3 outlines synchronized screen recording, video segmentation based on action timestamps, and graph-aligned diagnosis, but details of the video-based evaluation model remain missing. These details determine whether the core evaluation method can be reproduced and audited. The human audit in Appendix E provides meaningful support for reliability, but uses five-frame ordered storyboards and does not directly validate the additional benefit of full video over screenshots. Moreover, the audit excludes 676 GUI-specific model trajectories for which media files are unavailable; whether its reliability findings extend to these records requires further verification. The paper also presents video-grounded evaluation as a core contribution of the benchmark, yet does not directly compare diagnostic performance using pre-action and post-action screenshots, multiple frames, and video, or isolate the respective contributions of the graph reference and video evidence.

Questions for authors
  1. Section 4.3 and Appendices C.4-C.5 provide evidence of process differences and associations beyond SR/CR. Despite this, could the authors further use direct comparisons or representative real trajectories to show which specific agent weaknesses GraceBench additionally identifies relative to existing process evaluation methods or simple trajectory statistics, such as the fraction of repeated states or excess steps?

  2. The conclusion suggests that these diagnostics can support research on state tracking, loop detection, and recovery policies. Could the authors use a real trajectory example to explain how the behavioral labels help localize a specific system weakness or inform an improvement? Which failure causes can be inferred from the diagnostic evidence, and which require additional evidence?

8 comments
AuthorsOpenReview ↗

Response to Reviewer X76j [1/4]

All authors are grateful for your time devoted to reviewing this paper and your constructive suggestions. Please see our detailed responses to your comments and suggestions below.

Q1 "Could the authors further use direct comparisons or representative real trajectories to show which specific agent weaknesses GraceBench additionally identifies relative to existing process evaluation methods or simple trajectory statistics, such as the fraction of repeated states or excess steps?"

A: For the requested real trajectory example, we add a case to Appendix G showing how GraceBench detects unstable behavior that simple trajectory statistics miss and helps localize specific agent weaknesses. See the anonymous execution video.

In this trajectory, the agent completed all required transitions and returned to the task path, but an extra visual effect added during the deviation remained in the final state. Semantic state completion, excess steps, and the fraction of repeated states do not show this side-effect risk. GraceBench flags the risk through its deviation labels.

Evaluation informationWhat it showsWhat it does not show
Completion rate and semantic state completionWhether all required transitions were completedWhich off-task actions occurred and what side-effect risks they introduced
Excess stepsIncreased execution costWhich actions added cost, and whether they supported recovery or caused deviation
Fraction of repeated statesWhether states were revisitedWhether the agent entered an unexpected state and introduced deviation risk
GraceBench labels and linked visual evidenceWhere deviations and instability risks occurred, whether the agent returned, and how the task progressed afterwardNone

Q2 "Could the authors use a real trajectory example to explain how the behavioral labels help localize a specific system weakness or inform an improvement? Which failure causes can be inferred from diagnostic evidence, and which require additional evidence?"

A: In the example above, GraceBench shows where the agent chose the wrong UI entry and introduced a side-effect risk, and whether it later returned to the correct page.

This helps localize problems in UI entry selection and instruction following. The example also reveals incomplete recovery after a deviation introduced a side effect. The labels point to the failing steps; whether the side effect persisted to the final state and whether the recovery was complete still requires a manual check of the execution video.

AuthorsOpenReview ↗

Response to Reviewer X76j [2/4]

W1(a) "Graphs obtained through finite exploration do not cover all valid task paths. Although the authors manually verify the validity of the collected states and transitions, this does not establish that actions outside the graph are necessarily erroneous or unstable."

A: For valid routes not captured by the graph, we clarified the review rules for unmatched states and valid alternative routes, which are detailed in Appendix F and summarized below:

Valid alternative routes.

  1. Identify routes that leave the task subgraph and follow them until their first return. Keep a candidate only if its first return is to a different, later reference node, both endpoints have unique reference positions, and its action count does not exceed the directed shortest-path distance between them in the task subgraph.
  2. Review candidates against the task requirements. Reject routes with unexpected side effects or unrelated pages.
  3. Change deviation and recovery labels to on_path only after both checks pass. Skipped reference transitions are not counted as completed.

Unmatched states.

  1. Match states first to the task subgraph, then to the application-level UI Transition Graphs(UTG).
  2. Keep the visible state description if no match is found.
  3. Finally, we use the task goal, constraints, and full trajectory to review unmatched steps ,and label these steps as on_path, stagnation, or deviation. An unmatched state alone does not mean deviation.

We have further clarified this in section 3.3 of the revised paper:

  1. "A step is labeled deviation when it enters or remains in a task-irrelevant branch confirmed by independent validation (Appendix F), or violates task constraints."
  2. "GraceBench uses T′T' to identify missing required transitions, persistent deviation, repeated no-progress actions, loops, successful recovery after deviation, and to check the behavior labels of unmatched voutv_{\mathrm{out}} states. Appendix F gives the corresponding review rules."

W1(b) "The paper does not quantify what proportion of deviation labels actually correspond to valid routes not captured by the graph, reasonable exploration, or transient environment states, nor whether these cases affect the main metrics and conclusions. "

A:

Following your suggestion, we manually audited 226 sampled deviation labels from 100 trajectories against the task requirements and full execution evidence. Only 2.65% were valid routes absent from the subgraph.

Audit categoryLabel count (approx.)PercentageInterpretation
Unnecessary detours or redundant actions17979.20%Actual deviations; The automatic labels are correct
Valid alternative routes62.65%Valid off-graph routes labeled as deviations
Insufficient evidence177.52%Available evidence does not allow a definitive judgment on whether the flagged actions constitute deviations.
Other labeling errors2410.62%Labeling errors during the audit
Total226100%Across the 100 sampled trajectories

Unnecessary detours or redundant actions accounted for 79.20%, valid alternatives wrongly labeled as deviations for 2.65%, insufficient evidence for 7.52% and other labeling errors for 10.62%. These errors do not change our main conclusions.

AuthorsOpenReview ↗

Response to Reviewer X76j [3/4]

W1(c) "It therefore remains necessary to clarify how the required transitions are constructed for multi-path tasks and whether CR should be interpreted as task progress or reference-path coverage. "

A: We clarified the completion rate (CR) calculation in Appendices A and F.

CR is the fraction of required transitions completed along a valid path in the task subgraph that is compatible with the actual execution. It measures task progress along that path.

We calculate CR as follows:

  1. Map the execution to the task subgraph and select the transition sequence for the branch taken. Exclude mutually exclusive branches that were not taken.
  2. Divide the number of completed required transitions by the total required transitions in that sequence.
  3. If only a common prefix is completed and the execution does not yet distinguish two valid branches, use the average required-transition count of the two compatible complete paths as the denominator.

CR therefore measures normalized completion for the task graph and route. This follows the same basic idea as ColorBench's multi-solution milestone evaluation [1] and Mobile-Agent-v2's operation completion ratio [2]: measuring partial progress through completed task elements. We added the CR calculation procedure to Appendix A and the corresponding rules to Appendix F.

Equal CR does not necessarily mean equal semantic task progress, but CR still measures normalized completion of required transitions. Our ablation shows that runs with the same SR and CR can differ in detours, repeated actions, and recovery after errors. Task completion alone therefore does not capture process stability.

References

[1] Yuanyi Song et al. ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon Tasks. arXiv:2510.14621, 2025.

[2] Junyang Wang et al. Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration. arXiv:2406.01014, 2024.

Reviewer_25CNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GraceBench, a benchmark for diagnosing the execution stability of mobile GUI agents. It aligns action-level video evidence from real-device executions with task-specific UI Transition Graphs to identify deviation, stagnation, looping, and recovery. The evaluation covers 169 tasks across 10 applications, 11 models, and two instruction conditions, yielding 3,718 trajectories. The authors report that 51.24% of successful runs contain a deviation or loop, and that trajectories matched on task, instruction condition, success, and reference-transition completion can still differ in process metrics. The validation includes comparisons controlling for task outcomes, cross-task consistency analyses, and a sampled human audit of generated transition labels.

Critical strengths
  1. Useful diagnosis beyond task outcomes. Combining task-specific graphs with action-aligned video provides a structured way to locate deviations, repeated behavior, and recovery events. Comparisons between trajectories with the same task, instruction condition, success label, and completion rate show that the process metrics distinguish executions that receive identical outcome scores. This offers a potentially useful basis for identifying specific execution problems and guiding agent improvements.

  2. Empirical breadth and checks on consistency. The study evaluates 11 models under two instruction conditions and reports task-cluster uncertainty estimates, matching-sensitivity analyses, and cross-task ranking consistency. Deviation and loop-burden rankings are particularly consistent across task splits. A blinded human audit of 3,487 transitions from the nine-model subset with available video reports Cohen's kappa of 0.7663-0.8656. These checks provide evidence that the observed patterns persist across tasks and some support for annotation reliability within the audited population.

Critical weaknesses
  1. The validity of graph-defined deviation is insufficiently established. Section 3.3 labels actions that leave the task subgraph or enter an unmatched state as deviations. Although the graphs are manually reviewed and include alternative routes, the paper does not quantify how often agents encounter valid routes absent from the reference or explain how such cases avoid being penalized. The average of 41.09 supporting states measures a one-hop neighborhood, not coverage of valid completion paths, as Appendix D.1 acknowledges. Human agreement under the supplied reference also does not establish its coverage. Particularly under concise instructions, an admissible route could therefore generate both deviation and subsequent recovery events. Without an assessment of these potential false positives, the extent to which the metrics reflect execution problems rather than graph incompleteness remains unclear. This directly weakens the interpretation of the headline finding that 51.24% of successful runs exhibit instability.

  2. The core evaluator is insufficiently specified and validated for independent verification. The diagnostic evaluators are described as expert agents, without specifying their models, prompts, or operational rules for evaluation-time state matching and loop detection. Appendix B defines success through frozen evaluation summaries, leaving the procedure that originally produced these labels insufficiently explained. The human audit is valuable, but covers diagnostic fields rather than final success labels and excludes the 676 records from the two GUI-specific models because their media are absent from the analysis package. Its agreement results therefore do not establish reliability across the full leaderboard. Statistical consistency of the frozen scores cannot resolve uncertainty about their generation. Since the measurement procedure is the benchmark's central contribution, these omissions materially limit reproducibility and confidence in the model comparisons.

Questions for authors
  1. Clarification and targeted validation of deviation labels. How are routes that satisfy the original task constraints but are absent from the task subgraph handled? On a sample of deviation-labeled transitions, can you assess how often the actions still constitute legitimate task progress and report the effect of any resulting corrections on the main instability findings?

  2. Clarification of the evaluation protocol. Which models and prompts generate the diagnostic and final success labels? Please specify the evaluation-time state-matching criteria, loop-detection rules and required trajectory history, and the procedure for verifying task success against the goal and its constraints.

  3. Clarification and sensitivity analysis for the unaudited records. Were the 676 GUI-specific-model records produced with the same evaluator and labeling protocol as the remaining records? Does restricting the analysis to the nine-model subset with available media preserve the main findings about instability in successful runs and differences between trajectories with matched outcomes?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 25CN [1/2]

We sincerely thank you for your time devoted to reviewing this paper and your constructive comments. Please see our detailed responses to your comments and suggestions below.

W1(a) "Although the graphs are manually reviewed and include alternative routes, the paper does not quantify how often agents encounter valid routes absent from the reference or explain how such cases avoid being penalized. "

A: To avoid penalizing the valid routes absent from the reference, our method includes rules for dealing with valid alternative routes and unmatched states, which are detailed in Appendix F and clarified below:

Valid alternative routes.

  1. Identify routes that leave the task subgraph and follow them until their first return. Keep a candidate only if its first return is to a different, later reference node, both endpoints have unique reference positions, and its action count does not exceed the directed shortest-path distance between them in the task subgraph.
  2. Review candidates against the task requirements. Reject routes with unexpected side effects or unrelated pages.
  3. Change deviation and recovery labels to on_path only after both checks pass. Skipped reference transitions are not counted as completed.

Unmatched states.

  1. Match states first to the task subgraph, then to the application-level UI Transition Graphs(UTG).
  2. Keep the visible state description if no match is found.
  3. Finally, we use the task goal, constraints, and full trajectory to review unmatched steps ,and label these steps as on_path, stagnation, or deviation. An unmatched state alone does not mean deviation.

We have further clarified this in section 3.3 of the revised paper:

  1. "A step is labeled deviation when it enters or remains in a task-irrelevant branch confirmed by independent validation (Appendix F), or violates task constraints."
  2. "GraceBench uses T′T' to identify missing required transitions, persistent deviation, repeated no-progress actions, loops, successful recovery after deviation, and to check the behavior labels of unmatched voutv_{\mathrm{out}} states. Appendix F gives the corresponding review rules."

These rules reflect the principle behind our evaluation: using the graph as a reference to track agent behavior, while checking other routes against the task requirements.

W1(b) & Q1 "Particularly under concise instructions, an admissible route could therefore generate both deviation and subsequent recovery events. Without an assessment of these potential false positives, the extent to which the metrics reflect execution problems rather than graph incompleteness remains unclear."&"Clarification and targeted validation of deviation labels. "

A: For the assessment of these potential positives ,we audited 226 deviation labels from 100 trajectories and found that only 2.65% had been wrongly assigned to valid alternative routes, with limited impact on our main conclusions.

Audit categoryLabel count (approx.)PercentageInterpretation
Unnecessary detours or redundant actions17979.20%Actual deviations; The automatic labels are correct
Valid alternative routes62.65%Valid off-graph routes wrongly labeled as deviations
Insufficient evidence177.52%Available evidence does not allow a definitive judgment on whether the flagged actions constitute deviations.
Other labeling errors2410.62%Labeling errors during the audit
Total226100%Across the 100 sampled trajectories

The audit of 226 deviation labels from 100 trajectories found 6 valid local alternatives labeled as deviations. Unnecessary detours or redundant actions accounted for 79.20%, valid alternatives wrongly labeled as deviations for 2.65%. These errors do not change our main conclusions.

The point of this work is using graph-aligned trajectories to locate and compare process differences that final success alone cannot show. Even when a valid route is absent from the task subgraph, the points where it leaves and returns to the known path still help explain how the agent completes the task.

AuthorsOpenReview ↗

Response to Reviewer 25CN [2/2]:

W2(a) & Q2 "The core evaluator is insufficiently specified and validated for independent verification. "&" Clarification of the evaluation protocol. "

A: For the insufficient clarification of evaluation protocol, we now give the complete evaluator configuration, decision rules, and evaluation procedure in Appendix F.

Appendix F describes the evaluator configuration, input format, state matching, labeling rules, review of alternative routes, and the full evaluation procedure. It also explains how we check task success, apply fixed scoring rules, and replay saved evaluation records so that the evaluation can be checked and reproduced. We clarified your main concern below:

1) Models and prompts generating diagnostic and final success labels. We use doubao-seed-2-0-lite-260428 for video mapping and review, with code-based rule correction. Success is reviewed separately. Prompts will be released with the code.

2) Evaluation-time state-matching criteria. States are matched by visible page identity and task-relevant context, options, and parameters, first against the task subgraph, then the application UTG. Unmatched states do not automatically imply deviation.

3) Loop-detection rules. A state/action pattern of length ≤3 must repeat ≥2 times without new task progress or useful evidence. The code checks revisited state identity.

4) Required trajectory history. Mapping uses the current action video, preceding mapped record, and the last known state. Behavior and loop review use the full mapped trajectory.

More details are in Appendix F.

W2(b) & Q3 "Clarification and sensitivity analysis for the unaudited records. Were the 676 GUI-specific-model records produced with the same evaluator and labeling protocol as the remaining records? "

A: The 676 GUI-specific-model records were produced with the same evaluator and labeling protocol as the remaining records, so our main conclusions remain unchanged.

For clarification and sensitivity analysis for the unaudited records, we have now collected all execution videos and expanded the human video audit to 372 trajectories, sampled from all the 3718 trajectories of 11 models, including runs from GUI-specific models.

FieldCohen’s κ
State match0.8041
Deviation0.7928
Recovery0.7751
Loop0.8563

To ensure the coverage of our human audit, we used stratified sampling by 11 execution models and 10 domains (i.e. across the 11 × 15 model–scenario strata) to select from 3,718 trajectories with MP4 recordings. We used random seed 20261108 and kept at least one trajectory per nonempty stratum. The sample includes 372 trajectories, covering about 10% of the auditable transitions. For each diagnostic field, we compared the automatic labels with the final human reference labels to compute Cohen’s κ.We have now corrected the description of audit coverage in the paper.

We hope that our responses could address your concerns. Please let us know if you have further questions.

  1. The audit identifies errors beyond the 2.65% attributed to valid alternative routes. It remains unclear how correcting these errors affects the finding that 51.24% of successful trajectories contain deviations or loops.
  2. The shortest-path-length restriction may still classify valid but longer routes as deviations, potentially conflating inefficiency with incorrect behavior
Reviewer_ihKLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GraceBench evaluates mobile GUI agents by mapping their actions and video observations onto task-specific UI transition graphs. It reports deviation, looping, and recovery measures alongside task success for 3,718 trajectories across 169 tasks and 10 applications.

Critical strengths
  • The benchmark asks a useful practical question: did an agent complete the task through a reliable path? The answer can differ from the success label, which is an important and interesting insight. (e.g., 663 of 1,294 successful trajectories contain a labeled deviation or loop.)

  • The authors also audit a 10% sample of transitions with blinded human judgments, reporting substantial agreement for deviation, recovery, and loop labels.

Critical weaknesses
  • The deviation measure depends heavily on the curated task subgraph.

    • A route can be valid for the user yet absent from that graph, in which case the metric may penalize useful exploration or recovery.
    • This matters especially for the paper’s claim about successful but unstable runs.
  • Also, The paper should separate what is directly video-validated from what is inherited from those records.

    • For example, the human label audit covers 3,042 trajectories with video (Appendix C);
    • Another 676 aggregate-only records lack media and could not enter that audit, although they contribute to the reported comparisons.
Questions for authors

In a human audit of successful runs labeled as deviations, what fraction are unnecessary detours rather than valid alternative routes? Do the main model rankings change if the 676 records without video are excluded?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ihKL [1/2]

All authors are grateful for your time devoted to reviewing this paper and your constructive suggestions. Please see our detailed responses to your comments and suggestions below.

W1 "A route can be valid for the user yet absent from that graph, in which case the metric may penalize useful exploration or recovery."

A: For routes that are valid yet absent from the graph (i.e., valid alternative routes), our method includes rules dealing with valid alternative routes and unmatched states (described in Appendix F).Useful exploration or recovery is therefore not penalized, and a state outside the graph is not automatically labeled as a deviation. Appendix F now describes the two sets of rules:

Valid alternative routes.

  1. Identify routes that leave the task subgraph and follow them until their first return. Keep a candidate only if its first return is to a different, later reference node, both endpoints have unique reference positions, and its action count does not exceed the directed shortest-path distance between them in the task subgraph.
  2. Review candidates against the task requirements. Reject routes with unexpected side effects or unrelated pages.
  3. Change deviation and recovery labels to on_path only after both checks pass. Skipped reference transitions are not counted as completed.

Unmatched states.

  1. Match states first to the task subgraph, then to the application-level UI Transition Graphs (UTG).
  2. Keep the visible state description if no match is found.
  3. Finally, we use the task goal, constraints, and full trajectory to review unmatched steps ,and label these steps as on_path, stagnation, or deviation. An unmatched state alone does not mean deviation.

These rules reflect the principle behind our evaluation: using the graph as a reference to track agent behavior, while checking other routes against the task requirements.

We have further clarified this in section 3.3 of the revised paper:

  1. "A step is labeled deviation when it enters or remains in a task-irrelevant branch confirmed by independent validation (Appendix F), or violates task constraints."
  2. "GraceBench uses T′T' to identify missing required transitions, persistent deviation, repeated no-progress actions, loops, successful recovery after deviation, and to check the behavior labels of unmatched voutv_{\mathrm{out}} states. Appendix F gives the corresponding review rules."

Q1 " In a human audit of successful runs labeled as deviations, what fraction are unnecessary detours rather than valid alternative routes?"

A: For the human audit of deviation labels, we audited 226 deviation labels from 100 sampled trajectories, and only 2.65% of them were valid alternative routes, with limited impact on our conclusions.

The results are shown below.

Audit categoryLabel count (approx.)PercentageInterpretation
Unnecessary detours or redundant actions17979.20%Actual deviations; The automatic labels are correct
Valid alternative routes62.65%Valid off-graph routes incorrectly labeled as deviations
Insufficient evidence177.52%Available evidence does not allow a definitive judgment on whether the flagged actions constitute deviations.
Other labeling errors2410.62%Labeling errors during the audit
Total226100%Across the 100 sampled trajectories

The low rate of errors involving valid alternative routes (2.65%) supports our method's ability to handle off-graph routes.

AuthorsOpenReview ↗

Response to Reviewer ihKL [2/2]

W2 &Q2 "The paper should separate what is directly video-validated from what is inherited from those records."&" Do the main model rankings change if the 676 records without video are excluded?"

A: On the one hand, the 676 trajectories of two GUI-specific-models were evaluated using exactly the same evaluator and labeling protocol as the others,so the main model rankings remain unchanged. On the other hand, their execution videos were not included in human audit originally. We have collected their execution videos and expanded the human audit to cover all 11 evaluated models and further demonstrated the validity of our main conclusion.Below we report human audit results:

FieldCohen’s κ
State match0.8041
Deviation0.7928
Recovery0.7751
Loop0.8563

To ensure the coverage of our human audit, we used stratified sampling by 11 execution models and 10 domains (i.e. across the 11 × 15 model–scenario strata) to select from 3,718 trajectories with MP4 recordings. We used random seed 20261108 and kept at least one trajectory per nonempty stratum. The sample includes 372 trajectories, covering about 10% of the auditable transitions. For each diagnostic field, we compared the automatic labels with the final human reference labels to compute Cohen’s κ.We have now corrected the description of audit coverage in the paper.

We hope that our responses could address your concerns. Please let us know if you have further questions.

I thank the authors for the clear response. I keep my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and helpful comments. We have revised the paper and added clarifications and additional results in response to their feedback. Revisions in the paper are marked in blue. The main updates are summarized below.

On the one hand, we have revised the main paper to improve its clarity:

  • In Section 3.3, we clarify the definition of deviation, the handling of valid alternative routes, and the scope of the human label audit. (ihKL, 25CN, X76j)

On the other hand, we have revised the Appendix to provide additional details and analyses:

  • In our responses, we report an audit of deviation labels from 100 trajectories. This demonstrates that valid alternative routes have limited impact on our main conclusion. (ihKL, 25CN, X76j)
  • In Appendix B, we clarify completion rate (CR) calculation for multi-path tasks, including branch selection and the denominator used when only a common prefix is completed. (X76j)
  • In Appendix E, we expand the human video audit to 372 sampled trajectories, including runs from GUI-specific models, and report Cohen’s κ of 0.7751–0.8563 across the four audit fields. This demonstrates the reliability of our evaluator and the validity of our main conclusion (ihKL, 25CN, X76j)
  • In Appendix F, we highlight rules for valid alternative routes and unmatched states, including checks for task constraints and unexpected side effects. (ihKL, 25CN, X76j)
  • In Appendix F, we specify the evaluator configuration, state matching, labeling rules, loop detection, task-success checks, and scoring procedure. (25CN, X76j)
  • In Appendix F, we add a comparison on 92 actions: diagnostic-label accuracy is 84.8% with video and 71.7% with pre-/post-action screenshots. (X76j)
  • In Appendix G, we add a real trajectory example to show what the behavioral labels reveal beyond simple trajectory statistics and which judgments require further video inspection. (X76j)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.