acceptodds
Under review as a conference paper at ICLR 2027

Traceverifier: Evidence-Linked Claim Graphs For Gui Agent Trajectory Verification

Abstract

Reliable verification is essential for evaluating GUI agents, curating training trajectories, and producing learning signals. Yet observable-trajectory verification remains difficult: plausible final states can conceal instruction-violating execution, while long trajectories can obscure decisive evidence. A single task label further hides the evidence behind a decision and whether downstream requirements were blocked by earlier errors. We formulate a new task, Structured Trajectory Verification, which maps a task instruction and an observed GUI trajectory to a structured verification report rather than a single outcome label. To address this task, we present TraceVerifier, which resolves instruction-derived claims using localized evidence, semantics-matched operators, and assessment dependencies, producing an evidence-grounded report that distinguishes direct failures from dependency-blocked requirements. We also construct TraceVerifierBench, comprising 384 mobile GUI trajectories from 85 applications, with outcome labels and two fine-grained diagnostic pilots. Across three task-outcome evaluations, TraceVerifier obtains the highest accuracy and pass precision among the evaluated verifier methods, exceeding the strongest method baseline on each benchmark by 2.85–12.71 accuracy points. Beyond final-label performance, its structured reports achieve 82.27% evidence-localization and 81.59% immediate-blocker , demonstrating measurable evidence grounding and failure attribution. Finally, TraceVerifier's structured outputs serve as fine-grained supervision: Process-Trace Distillation yields a best observed Qwen3.5-9B accuracy of 85.66%, within 0.82 points of the Qwen3.5-27B teacher.

Primary area:
applications to robotics, autonomy, planning
Keywords:
GUI agent, GUI agent verifier, trajectory verification, VLM, evidence grounding, dependency-aware reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dyVdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

TraceVerifier proposes a structured design for verifying GUI agent trajectories, addressing the limitations of outcome-focused evaluation in locating key evidence and tracing the sources of errors. It decomposes task instructions into claims, locates relevant evidence and assigns specialized verification operators to each claim, and uses a dependency graph to distinguish direct failures from downstream blockages. The evaluation comprises 384 mobile GUI trajectories with an average length of 10.8 steps. On General Apps, the method achieves 86.48% accuracy, outperforming the strongest evaluated baseline by 12.71 percentage points; the paper also reports an evidence localization F1 score of 82.27%. Although the method achieves high accuracy on the authors' benchmark, I have several major concerns about the evaluation and benchmark design as well as whether the experimental evidence sufficiently supports the paper's claims.

Critical strengths
  1. The limitation summary of existing trajectory judgement is good.
Critical weaknesses
  1. The paper motivates its approach by arguing that critical evidence can become buried in long trajectories, yet the benchmark contains trajectories averaging only 10.8 steps. This is considerably shorter than some contemporary GUI tasks, such as certain OSWorld2 tasks involving more than 100 GUI actions. The current evaluation therefore provides limited support for the method’s applicability to genuinely long-horizon tasks and raises questions about whether the benchmark adequately captures the practical challenges motivating the work.

  2. The evaluation does not include strong proprietary models used directly as trajectory judges. It therefore remains unclear whether a sufficiently capable model could achieve comparable performance without the proposed structured framework. This omission limits the evidence for the framework’s additional value over a strong direct-judging baseline.

  3. Table 4 does not include a 9B baseline trained only on the final Pass/Fail labels. Although the results demonstrate an improvement over the model before distillation, they do not establish that structured process supervision outperforms outcome-only supervision under a matched training budget. Consequently, the reported gains cannot be clearly attributed to the process supervision itself.

  4. The framework relies on the Contract Generator and Claim Compiler to convert task instructions into claims, but the paper does not separately evaluate the completeness, correctness, or granularity of these claims. It also provides insufficient analysis of how decomposition errors affect subsequent verification and final judgments. More broadly, it remains unclear how strongly the framework’s effectiveness depends on the capabilities of the models used to construct its verification structure.

Questions for authors
  1. Could you clarify the example in Figure 1? In the case involving "psychology" and opening the first article, what specifically makes the resulting state incorrect? Which task requirement is violated. The final state should be satisified in current benchmarks.

  2. How does TraceVerifier perform on substantially longer trajectories?

  3. How does the framework compare with a strong proprietary model judging trajectories directly? For example, could a model such as GPT-6 Astra achieve comparable performance without the proposed structure? Such a comparison would help establish the practical benefit of the framework.

  4. How does process distillation compare with outcome-only supervision under controlled conditions? Specifically, how does the distilled 9B model compare with a 9B model trained on the same examples using only final Pass/Fail labels under a matched training budget?

  5. How sensitive is the framework to the models used for contract generation and claim compilation? How are the completeness, correctness, and granularity of the generated claims assessed? When claims are incorrect or incomplete, how do these errors affect the final verdict, and can the downstream verification process detect or compensate for them?

Reviewer_yKrUOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies trajectory verification for GUI agents and introduces TraceVerifier, which decomposes task instructions into execution and output requirements. The authors construct TraceVerifierBench, comprising 384 trajectories across 85 applications, and validate the method through task-level evaluation, fine-grained diagnosis, and process-trajectory distillation experiments.

Critical strengths
  • The problem has practical value. Judging task success solely from the final interface may overlook instruction violations during execution. By examining the execution process, supporting evidence, and causes of failure, this work helps make GUI agent evaluation more interpretable.
  • The experiments provide fairly comprehensive validation across multiple levels.
Critical weaknesses
  • Limited evaluation of long trajectories. The benchmark averages only 10.8 steps, with a median of 7; just 51 trajectories (13.3%) exceed 20 steps. Appendix C.4 groups all trajectories above 20 steps together, and their higher failure rate may confound the accuracy results. Finer length breakdowns and experiments controlling for label distribution, task difficulty, or irrelevant steps would strengthen the claims.
  • The pipeline’s costs are not adequately justified. The full system improves accuracy over the simplified architectures in Appendix C.1 by 3.69–4.92 percentage points, but its additional computational cost is unreported. The same backbone does not imply the same inference budget. Model call counts, token usage, latency, and comparisons under matched compute budgets are needed.
  • Key implementation details are missing. Section 3.2 does not clarify whether contract generation and claim compilation use separate LLM calls, a joint call, or deterministic rules. Section 3.3 also leaves key-step selection, window sizing, evidence selection, and insufficient-evidence handling unclear, limiting reproducibility.
  • Operator execution and overhead are unclear. Appendix C.2 reports operator assignments rather than actual model calls. It remains unclear whether operators require separate prompts and calls, whether claims can be batched, and how routing and retries work. A worked example with intermediate outputs and call counts would help.
Questions for authors

see Weaknesses

Reviewer_3riKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TraceVerifier. Rather than only making a final pass/fail judgment, it decomposes a GUI task into typed claims, links each claim to local trajectory evidence, and uses a dependency graph to distinguish direct failures from downstream blocked claims. This produces a structured, trace-level verification report.

Critical strengths
  • The structured report is more informative than a single pass/fail label.

  • The paper considers evidence localization and failure attribution in addition to the final decision.

Critical weaknesses
  1. The motivation is not strong enough.

The paper gives an example where a final state hides an incorrect execution, but does not show how often this happens or how much it affects evaluation. The example in Figure 1 may also be handled by checking the final content more carefully. The paper should clarify which failures truly require trajectory evidence and cannot be detected by a stronger outcome verifier.

The paper also motivates trajectory curation and policy learning, but does not use TraceVerifier to train or improve a GUI agent. It only trains a smaller verifier. The practical value for the downstream use cases is unclear.

  1. The three challenges and the proposed techniques are not connected clearly.

My understanding is that typed claims and specialized operators check whether a required interaction happened, evidence linking and compact memory locate the evidence, and the dependency graph handles downstream failure attribution. The paper should state and evaluate this mapping more clearly. Evidence localization and dependency resolution have dedicated diagnostics, while the first and most important challenge i.e., detecting invalid execution behind a plausible outcome, does not have specific evaluation.

  1. It is unclear how a user goal becomes execution claims.

Users often specify only a final goal, not a detailed interaction path. The paper says that the Contract Generator converts an instruction into "atomic requirements," but does not define atomicity or explain how required steps are separated from one possible path. Many GUI tasks allow several valid paths. If the generated claims encode a canonical path, the verifier may reject a valid execution. This also weakens the motivating example: opening an article from the wrong section may be a wrong final target rather than a failure that requires trace-level verification.

  1. The benchmark and baseline choices need more justification.

Most results rely on TraceVerifierBench, which contains 384 trajectories generated by one GUI agent. The ordinary pass/fail labels mainly evaluate final verifier accuracy and do not by themselves show the value of trace-based verification.

The appendix discusses several closely related evidence-selective and task-decomposed verifiers, but only some are included in the experiments. The paper should explain the baseline selection and compare more directly with the closest trace- or task-decomposition methods.

  1. Some experiments are hard to interpret.

Table 2 evaluates the components only through the final pass/fail result. Removing evidence scoping and removing specialized operators give almost the same result, so it is difficult to see what each component contributes. More direct evaluation would help, such as evidence quality for evidence scoping and claim-type accuracy for specialized operators.

The motivation for distilling the verifier from 27B to 9B is also unclear. Is the goal on-device use, lower evaluation cost, or only model compression? The paper should state the intended setting and explain why this experiment demonstrates the value of TraceVerifier.

Questions for authors
  1. How common are outcome-equivalent but trace-invalid trajectories in existing GUI benchmarks?

  2. Can the authors show that using TraceVerifier for data filtering or supervision improves an actual GUI agent?

  3. How is an "atomic requirement" defined, and how does the Contract Generator avoid rejecting valid alternative execution paths?

  4. What is the intended use case for the 9B distilled verifier, and why is 9B the relevant model size?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.