acceptodds
Under review as a conference paper at ICLR 2027

GUIVista: Training GUI Agents from Tutorial Videos with Offline Capability Diagnosis

Abstract

Public software tutorial videos offer scalable supervision for GUI-agent mid-training, but their visual transitions often underdetermine the action that caused them. Even when these interactions are used for training, a failed action does not reveal whether the model chose incorrectly or struggled to express and execute the right action reliably. We present GUIVista, a framework that combines reliable trajectory construction with offline capability diagnosis. GUIVista revisits uncertain action proposals using transition evidence and action-family verification before filtering them into grounded, multi-turn training trajectories. It also evaluates checkpoints by asking them to select a successful action from plausible alternatives under the same GUI context, separating action understanding from open-ended execution. On an action-recovery evaluation, GUIVista improves overall action-type and full-action accuracy over prior video-based systems from 24.75% / 8.80% to 50.42% / 31.29%. Under the shared downstream SFT protocol, GUIVista mid-training improves end-to-end OSWorld AVG@4 from 18.40% for SFT-only to 21.75%. The offline diagnostic tracks post-SFT OSWorld performance across matched checkpoints, with Pearson and Spearman . These results show that GUIVista can turn noisy tutorial videos into useful training supervision and provide an efficient signal for comparing the capabilities acquired during mid-training. We will release all code for data construction and evaluation.

Primary area:
datasets and benchmarks
Keywords:
CUA, Mid-train Dataset, Offlline evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QztnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This work proposes GUIVista, a framework that converts software tutorial videos into data for GUI-agent mid-training. The pipeline combines inverse-dynamics action prediction with action reconstruction and action-specific verification to improve the recovery of long-tail interactions. Under the same downstream SFT protocol, adding GUIVista data to mid-training improves OSWorld performance from 18.40% to 21.75% and also improves performance on ScreenSpot-Pro. The paper additionally introduces the GUIVista diagnostic, referred to as PACE in the appendix, an offline action-selection evaluation whose scores are reported to correlate with OSWorld performance.

Critical strengths

This paper proposes a method to convert unsupervised software-using to GUI-agent training data. Experiments show that mid-training with the resulting data improves downstream performance on OSWorld and ScreenSpot-Pro under a shared SFT protocol.

Critical weaknesses
  1. The main idea is similar to VideoAgentTrek: turning software tutorial videos into GUI-agent training data. The added reconstruction and verification steps have ablations, but their benefit over simpler annotation methods with a similar budget remains unclear. A direct-annotation baseline using another strong VLM, such as Gemini, would help assess whether this complexity is necessary.
  2. The OSWorld gain is modest (18.40% to 21.75%). There is also limited analysis of how the model’s actual task-solving behavior changes and what accounts for this improvement.
  3. The paper does not clearly show whether the broader training data help the model understand software better or handle unseen applications and tasks.
Questions for authors
  1. Are there any comparisons with direct VLM annotation or simple self-correction under similar inference budgets, including both annotation accuracy and cost?
  2. Which evaluation applications or task types are unseen in the mid-training data, and how does the model perform on these subsets compared with SFT-only?
  3. Did mid-training on the generated data introduce any helpful or harmful patterns in the model’s behavior?
5 comments

Dear Reviewer Qztn, thank you for your valuable comments.

Response to W1&Q1

1. Updated Evaluation and Cost Analysis. As detailed in our response to Reviewer otSZ, Q3, we have incorporated the original IDM evaluation with a more reliable CUA-Suite-based benchmark, covering 1,000 samples across 82 applications. Here, we focus on the trade-off between annotation accuracy and inference cost.

2. API and Local GPU Costs. Gemini 3.5 Flash-Lite achieve 49.10% overall action type accuracy under our direct-annotation configuration, requiring one call per sample at approximately $0.0011. According to Gemini's official pricing, the respective input/output prices are $0.30/$2.50 per million tokens. Our local annotation pipeline runs on 8×A100 GPUs without API fees. Using a Cloud reference price of $1.80/GPU-hour and assuming a sustained throughput of 32 calls/second across eight GPUs, the estimated compute cost is $0.000124 per call, or $0.000868 per sample for the full pipeline ≈69˙5calls≈6\.95 calls. Actual costs depend on deployment throughput.

3. Annotation Accuracy and Cost Comparison. We further compare direct VLM annotation, confidence-based filtered, reconstruction, and expert verification on the updated CUA-Suite evaluation set:

MethodCoverageTypeType+ArgCalls / SampleCost / Sample
Gemini-3.5-Flash-Lite (Trek Harness wo Wait&Terminate)100.00%49.10%17.33%1$0.001087 (Measured)
Annotated-only93.70%40.60%21.40%1≈$0.000124 (Estimated)
+ [email protected]36.00%23.90%13.19%2≈$0.000249 (Estimated)
+ Rebuilder76.90%49.60%22.18%≈4≈$0.000500 (Estimated)
+ Experts81.50%45.20%23.83%≈6.95≈$0.000868 (Estimated)
+ Rebuilder + Experts85.00%47.20%24.89%≈6.95≈$0.000868 (Estimated)

4. Effectiveness and Necessity. Under the same one-call budget, our direct annotation achieves 21.40% Type+Arg accuracy, outperforming Gemini's 17.33% at a lower estimated cost. Reconstruction-only further improves accuracy to 22.18%, while expert-only achieves 23.83%. Combining both yields the highest accuracy ∗∗248˙9**24\.89%**, improving upon direct annotation by 3.49 percentage points and Gemini by 7.56 percentage points, with a lower estimated per-sample compute cost than Gemini's measured API cost.

These results demonstrate that reconstruction and expert verification provide complementary improvements beyond direct annotation and simple filtered, while quantifying their additional inference overhead. We have incorporated the updated evaluation and cost comparisons into the revised manuscript.

Response to W2&Q3 Part1

1. Fine-Grained Capability and Behavioral Analysis. We thank the reviewer for this suggestion. We have added a detailed trajectory-level behavioral analysis to the revised manuscript, examining action diversity, execution effectiveness, repetition, trajectory efficiency, and reasoning-action consistency across the Base, 400K, and 800K checkpoints.

2. Capability Improvements. Beyond the OSWorld improvement from 18.40% to 21.75%, our offline diagnostic shows that the 400K checkpoint improves overall performance from 22.96% to 34.54% +115˙8pp+11\.58 pp, with particularly large gains in long-horizon execution ∗∗+170˙8pp∗∗**+17\.08 pp**, rare-action recognition ∗∗+150˙4pp∗∗**+15\.04 pp**, complex tasks ∗∗+130˙8pp∗∗**+13\.08 pp**, and fine-grained grounding ∗∗+82˙6pp∗∗**+8\.26 pp**.

3. Behavioral Changes from Base to 400K: More Diverse and Effective Actions. We analyze the action predictions recorded in OSWorld execution trajectories. For each checkpoint, action-type shares and behavioral rates effectiveactions,repeatedactions,andno−opseffective actions, repeated actions, and no-ops are calculated by dividing the number of occurrences in each category by the total number of recorded actions. The Base model is heavily dominated by clicking 906˙90\.6% of all actions, whereas the 400K checkpoint exhibits a more diverse action distribution, including keyboard operations, text entry, and scrolling. Meanwhile, the effective-action rate increases from 43.3% to 84.3%, while repeated-action and no-op rates decrease from 82.6% to 19.8% and 56.6% to 15.2%, respectively, indicating a substantial shift toward more diverse and effective execution.

Behavioral MetricBase400K800K
Click proportion90.60%47.90%51.10%
Keyboard proportion1.70%18.50%16.60%
Scroll proportion0.25%7.80%7.30%
Effective actions43.30%84.30%87.40%
Repeated actions82.60%19.80%19.70%
No-op operations56.60%15.20%12.00%
Invalid negative coordinates0.25%6.27%5.65%

These results demonstrate that mid-training significantly improves action diversity and execution effectiveness, rather than merely increasing the number of actions. However, the higher frequency of invalid coordinates indicates remaining challenges in action grounding.

Response to W2&Q3 Part2

4. Behavioral Changes from 400K to 800K: More Efficient Successes but Persistent Failures. Continued mid-training primarily improves successful execution efficiency, rather than extending planning horizons. As OSWorld AVG@4 increases from 20.55% to 21.75%, the model gains success on 17 tasks while losing success on another 17, with higher scores across repeated rollouts on newly successful and consistently passing tasks outweighing the losses. We analyze execution trajectories under a maximum budget of 50 steps per task, with the step-limit rate indicating the proportion of trajectories reaching this limit. We further analyze emerging action-sequence patterns in agent trajectories, finding that mid-training substantially expands multi-action combinations (e.g., click→type→enter: 3.6% → 30.0%), reflecting a shift from repetitive clicking toward more diverse and structured task execution.

Execution Metric400K800K
Overall average steps34.05%33.31%
Successful trajectory steps22.34%19.46%
Failed trajectory steps39.40%39.64%
Newly successful tasks: average steps29.21%19.24%
Newly successful tasks: 50-step limit rate44.10%19.10%
Newly failed tasks: average steps28.24%33.41%
Newly failed tasks: 50-step limit rate39.70%55.90%

5. Beneficial Pattern: More Efficient Successful Execution. From 400K to 800K, successful trajectories become 12.9% shorter 223˙4→194˙6steps22\.34 → 19\.46 steps, while newly successful tasks require 34.1% fewer steps 292˙1→192˙429\.21 → 19\.24. These results indicate that continued mid-training improves execution efficiency rather than extending planning horizons.

6. Harmful Pattern: Persistent Execution Failures. Failed trajectories remain approximately 39.5 steps, while the 50-step limit rate for the 17 newly failed tasks increases from 39.7% to 55.9%. This indicates that some tasks remain prone to prolonged unsuccessful execution despite the overall improvements.

7. Reasoning-Action Consistency. From 400K to 800K, average rationale length decreases from 371.7 to 350.3 characters, while rationale-action consistency improves from 45.9% to 48.2%. This suggests more concise reasoning with better alignment to executed actions.

8. Summary and Manuscript Updates. Our analysis reveals two main benefits of mid-training: 11 improved action diversity and effectiveness from Base to 400K, and 22 more efficient successful execution from 400K to 800K. Increased coordinate errors and persistent failure trajectories remain limitations, potentially reflecting stage-specific training fluctuations similar to the temporary performance decline observed at 25K–50K. Further scaling may help recover these capabilities, although the underlying mechanisms require additional investigation. We have incorporated these quantitative findings and behavioral analyses into the revised manuscript.

Reviewer_otSZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GUIVista reconstructs GUI training trajectories from public tutorial videos by correcting and verifying uncertain action proposals before filtering. It also introduces a four-option offline diagnostic intended to assess action understanding before SFT. Experiments report improved action recovery, downstream gains on ScreenSpot-Pro and OSWorld, and correlations between diagnostic scores and online performance.

Critical strengths

S1. The action-recovery ablation provides useful evidence for the proposed pipeline. With the same base annotator, reconstruction and expert verification improve action-type accuracy from 40.58% to 50.42% and full-action accuracy from 25.50% to 31.29%.

S2. The additional mid-training stage produces measurable downstream gains. Under the shared Qwen2.5-VL-3B backbone and downstream SFT recipe, ScreenSpot-Pro improves from 15.31% to 18.98%, and the reported OSWorld AVG@4 improves from 18.40% to 21.75%.

Critical weaknesses

W1. The evidence for pre-SFT checkpoint selection is not yet convincing. Appendix F states that the 19-model panel also participated in diagnostic item selection, making its reported correlation an in-sample result.

W2. The downstream experiments do not isolate the benefit of correction-before-rejection from additional mid-training. There is no matched downstream comparison using base or filtering-only annotations. The auxiliary training source is also inconsistent: Section 4.2 specifies 220K ProCUA prefixes, while Appendix E specifies AgentNet and explicitly excludes ProCUA, making the training gains harder to interpret.

Questions for authors

Q1: How did the 19-model panel influence item selection, and what correlation holds for models or checkpoints excluded from that process? (with W1)

Q2: Which dataset actually supplies the 220K auxiliary prefixes, and are the OSWorld results AVG@4 or pass@4, given the conflicting description in Tables 2 and 6 (The reviewer asks this because the score of the Qwen baseline is the same, which is highly unlikely to be a simple coincidence)? Do reconstructed trajectories outperform base or filtering-only annotations when the mid-training budget, auxiliary mixture, and downstream SFT recipe are held fixed? (with W2)

Q3: It would be good if the authors could position their tutorial-derived corpus relative to expert video-demonstration resources such as CUA-Suite[1], particularly in action-label quality and difficult-action coverage.

[1] CUA-Suite: Massive Human-Annotated Video Demonstrations for Computer-Use Agents.

4 comments

Dear Reviewer otSZ, thank you for your valuable comments.

Response to W1&Q1

1. Unweighted Diagnostic Correlation. The 19-model evaluation panel participated in the selection of diagnostic items, but the GUIVista diagnostic score (%) is computed without model-specific weighting in this analysis. Across the 19 models, the unweighted GUIVista diagnostic score still exhibits a strong correlation with OSWorld (Pearson r = 0.9005, Spearman ρ = 0.9193).

2. Independent Checkpoint Validation. More importantly, we further evaluated 7 mid-training checkpoints that were not involved in diagnostic item selection. Their unweighted GUIVista diagnostic scores exhibit a strong correlation with OSWorld AVG@4 (Pearson r = 0.9252), providing checkpoint-level validation evidence independent of the item-selection process and further supporting the effectiveness of the GUIVista diagnostic score for checkpoint selection during training. We have added the unweighted diagnostic scores of these 7 checkpoints and their corresponding OSWorld AVG@4 results to the appendix to support the verifiability of these findings.

Response to W2&Q2

1. Auxiliary Data Source. The auxiliary prefixes come from ProCUA. We corrected the inconsistent data-source description in Appendix E of the revised manuscript and updated the experimental weights to make the conclusions clearer.

2. OSWorld Evaluation Metrics. All OSWorld results reported in the main text are AVG@4 scores obtained from actual measurements, whereas the Qwen baseline in the appendix uses results published on the official OSWorld leaderboard rather than our local evaluation results.

Specifically, the local evaluation results for Qwen2.5-VL-3B Base are AVG@4 = 1.94% and pass@4 = 4.18%. A typographical error in the metric labeling in the appendix led to misunderstanding.

We will correct the error in the appendix and explicitly distinguish AVG@4 from pass@4, as well as official leaderboard results from local evaluation results, to ensure consistency in the definitions and sources of all reported metrics.

3. Matched Training Baselines. We added two baselines, original annotations Annotated−onlyAnnotated-only and filtered-only annotations Filtered−onlyFiltered-only, and conducted a full epoch of mid-training and downstream SFT using the same training recipe. The only difference was the processing pipeline applied to data from the same source. The results are as follows:

Evaluation BenchmarkAnnotated-onlyFiltered-onlyRebuilder+Experts
ScreenSpot-Pro AfterMid−trainingAfter Mid-training21.82%20.94%29.22%
ScreenSpot-Pro (After SFT)10.50%18.15%18.98%
OSWorld (AVG@4, After SFT)4.27%19.53%21.75%

4. Downstream Improvements. On ScreenSpot-Pro, reconstructed trajectories achieve higher performance than both the original-annotation and filtered-only baselines, and this advantage is maintained after downstream SFT. Meanwhile, OSWorld AVG@4 improves from 18.40% to 21.75% +33˙5percentagepoints+3\.35 percentage points, further demonstrating the benefits of the method for real-world GUI tasks.

Response to W2&Q3

1. Updated IDM Evaluation. We thank the reviewer for recommending CUA-Suite [1], a valuable expert-annotated dataset. Following this suggestion, we have incorporated a more reliable CUA-Suite-based evaluation set into our existing IDM evaluation experiments and added CUA-Suite to the Related Work section. Specifically, we processed all 87 applications, extracting annotated actions and corresponding pre/post-action keyframes. After normalizing actions into 8 categories, we obtained 37,721 valid action windows and constructed a category-balanced evaluation set of 1,000 samples spanning 82 applications and 926 tasks. Each sample contains two frames and a task instruction, with ground-truth actions and arguments excluded from the input. Detailed construction procedures have been added to the revised appendix.

2. Models and Comparison Settings. Consistent with our original IDM experiments, we use Holo-3.1-35B-A3B as the annotator, deployed on 8×A100 GPUs. We evaluate direct annotation, confidence-based filtered Meaningful@08˙Meaningful@0\.8, reconstruction-only, expert-only, and the full reconstruction-and-verification pipeline. For a strong VLM baseline, we use Gemini-3.5-Flash-Lite with the official open-source VideoAgentTrek Harness.

3. Action-Space Alignment. Since CUA-Suite's human annotations do not include wait or terminate, their inclusion in the default VideoAgentTrek Harness leads to out-of-scope predictions, reducing coverage by 21.1 percentage points 100100% → 78\.9%. To ensure a fair comparison, we remove only these two actions from the Harness action space and prompt, keeping all other settings unchanged.

4. Annotation Accuracy and Cost. Coverage measures valid executable outputs, Type measures primary action-type accuracy, and Type+Arg evaluates both action type and arguments under the AgentNetBench evaluation standard coordinatetolerance:100/1000coordinate tolerance: 100/1000, with distance-based partial credit assigned to coordinate localization. Results are shown below:

MethodCoverageTypeType+Arg
Gemini-3.5-Flash-Lite (Trek Harness wi Wait&Terminate)78.90%40.70%15.11%
Gemini-3.5-Flash-Lite (Trek Harness wo Wait&Terminate)100.00%49.10%17.33%
Annotated-only93.70%40.60%21.40%
+ [email protected]36.00%23.90%13.19%
+ Rebuilder76.90%49.60%22.18%
+ Experts81.50%45.20%23.83%
+ Rebuilder + Experts85.00%47.20%24.89%

Our full pipeline achieves the highest Type+Arg accuracy 248˙924\.89%, outperforming Gemini 173˙317\.33% by 7.56 percentage points, while requiring a lower estimated per-sample cost. These results demonstrate the value of reconstruction and expert verification for improving action-label quality.

5. Fine-Grained Error Analysis. After removing wait and terminate, Gemini's remaining errors are primarily associated with action-type confusion and argument grounding, particularly coordinate localization. Among 1,000 predictions, 509 have incorrect action types, while 335 additional predictions have correct action types but incorrect arguments or action sequences (238 argument errors and 97 sequence errors). We further examine representative difficult actions:

ActionGemini TypeGemini Type+ArgOurs Rebuilder+Experts Type+Arg
Left Click83.97%38.17%73.28%
Right Click69.80%34.23%69.13%
Text Entry63.89%9.03%15.97%

6. Argument Grounding Improvements. Although Gemini achieves relatively high action-type accuracy, its Type+Arg accuracy decreases substantially when argument correctness is considered. For left and right clicks, our pipeline improves Type+Arg accuracy by 35.11 and 34.90 percentage points, respectively, indicating more accurate coordinate grounding. For text entry, our method also improves Type+Arg accuracy from 9.03% to 15.97%, demonstrating benefits beyond coordinate-based actions. These results highlight the importance of reconstruction and action-specific verification for improving action-label quality.

[1] CUA-Suite: Massive Human-Annotated Video Demonstrations for Computer-Use Agents.

The work's setting is more solid and the its position relavative to related works is stronger now after the rebuttal, so the reviewer has increased score to 3.

Reviewer_MpMDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work tackles the problem of constructing a GUI agent training dataset by processing public tutorial videos (such as YouTube videos), where its framework is called GUIVista. For data construction, its inverse dynamics pipeline first performs action proposal with Holo-3.1-35B-A3B using keyframes, transcripts, and OCR/UI parsing as input, where uncertain proposals go through action-family-specific experts for text, pointer, and scroll actions and their voting. As a result, the authors construct a dataset named GUIVista-1.1M, which contains ~1.27M steps from 25,450 videos across 198 applications. Additionally, the authors use GUIVista-1.1M for a mid-training stage starting from Qwen2.5-VL-3B. To address the issue that fully end-to-end benchmark evaluation of such models requires a subsequent downstream fine-tuning stage, the authors propose to turn successful trajectories on the benchmark (OSWorld) into multiple-choice questions, enabling offline evaluation as a proxy measure. They suggest that the scores obtained with this protocol for multiple models correlate with OSWorld scores.

Critical strengths

None

Critical weaknesses
  • Overall, this work needs clearer positioning regarding its contributions and novelty. Here are examples. (a) Gathering public tutorial videos (including YouTube videos) and turning them into training data is a problem that has been tackled by prior work in the past. This includes not only the papers that are already cited by this work but also uncited yet closely related papers: Video2GUI ("Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining", Xiong et al., ICML 2026), MONDAY ("Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents", Jang et al., CVPR 2025). (b) Leveraging OCR, UI parsers, and narrations/transcripts and multi-stage action annotation have been explored by MONDAY (Jang et al., CVPR 2025) and Video2GUI (Xiong et al., ICML 2026). The details of the multi-stage action annotation (correction-before-rejection) can differ, but these still need to be clarified. Also, the MONDAY dataset was used for their mid-training. (c) Other than RuDE (which is already cited by this work), converting trajectories for agentic tasks (e.g., they employ Mind2Web) into multiple-choice questions for evaluation seems quite closely related to AgentVQA ("AgentVQA: A Unified Benchmark for Agentic Visual Understanding", Gupta et al., ICLR 2026).
  • I have concerns regarding the offline evaluation protocol. Firstly, it leverages successful trajectories for the same target benchmark, which means its generality is yet to be shown. Also, the "weak models" used for collecting the successful trajectories are Qwen3-VL-8B-Instruct, Qwen2.5-VL-7B-Instruct, UI-TARS-1.5-7B, and EvoCUA-8B-20260105, which are all Qwen or Qwen-fine-tuned models. At the same time, most of the models used for validating this evaluation protocol (in Table 6) are also Qwen or Qwen-fine-tuned models except for Claude-Sonnet-4.5, Kimi-K2.6, and GPT-5.4. Since these models share the same base model family, they likely share the behavioral biases and tendencies on these tasks, as well, which can weaken the practical usefulness of the protocol. In addition, since these models are directly evaluated on the downstream tasks (OSWorld tasks) without further fine-tuning, such correlation with the offline evaluation is less interesting and may not be as useful. Focusing on predicting the downstream performance of mid-trained checkpoints without the subsequent downstream SFT stage, I believe the correlation likely gets meaningfully weaker if we exclude the 100K point (Figure 4), where the offline evaluation predicts the 800K checkpoint, which scores the highest on the downstream tasks, to be worse than the 300K and 400K checkpoints. This suggests that while the protocol may be useful for determining under-trained checkpoints (such as the 100K checkpoint), it may not be effective beyond that purpose.
Questions for authors

Please refer to the Critical Weaknesses section.

5 comments

Dear reviewer, please respond to the author's comment to further facilitate the discussion as the author-reviewer discussion period ends at November 18 at 11:59pm AoE

Best, AC

Response to Q1

I acknowledge that the four models are used only to generate the negative alternatives, not the original successful trajectories themselves.

The issue is not whether the reviewer acknowledges the error, but why such a fundamental factual error occurred in the first place.

Section 3.3 explicitly states:

The evaluation contexts come from successful OSWorld-Verified Ubuntu trajectories.

Appendix E further clarifies:

Gold is the current decision from a successful trajectory, not a vote among generators.

The reviewer failed to understand these basic methodological distinctions and then used this demonstrably incorrect premise to challenge our diagnostic and justify an extremely negative assessment.

Regarding model diversity, the reviewer again overlooks our explicitly documented evidence:

1. Successful trajectory diversity: As already clarified in our earlier response (refer to Reviewer MpMD), successful trajectories originate from multiple model families, including Holo3, Claude-Sonnet-4.5, Kimi-K2.6, Qwen3.7-Plus, and UiPath-GPT-5. The four models listed in Section 4.3 are exclusively negative candidate generators. Moreover, three of these generators have undergone substantial GUI-specific SFT, with distinct fine-tuning backgrounds and markedly different capability levels. Simply categorizing them as "Qwen-based" is an inaccurate and overly simplistic characterization that ignores their actual capability diversity.

2. Cross-model and checkpoint validation: Using unweighted diagnostic scores, we validate the diagnostic across 19 established GUI models from different model families Pearsonr=09˙005Pearson r = 0\.9005 and seven independently held-out mid-training checkpoints Pearsonr=09˙252;refertoReviewerotSZ,Q1,A1–A2Pearson r = 0\.9252; refer to Reviewer otSZ, Q1, A1–A2.

3. Diagnostic diversity: The frozen diagnostic contains 2,849 items spanning 10 application groups and six action families, including Point, Keyboard, Scroll, Drag, and Multi-action (Appendix E, Table 7).

4. Independent IDM evaluation: Our CUA-Suite evaluation further covers 1,000 samples across 82 applications and 926 tasks, providing broad cross-application validation of action annotation (refer to Reviewer otSZ, Q3, A4–A6; Reviewer Qztn, Q1, A3–A4).

Response to Q2

ICLR asks reviewers to be concise and to "only discuss points that are actually likely to influence your final accept / reject decision."

"None" here does not mean that I think this work has no positive aspects.

We are astonished that the reviewer invokes "conciseness" and a formatting option to justify assigning "Critical Strengths: None". This explanation is academically irresponsible and fundamentally unfair.

Because my view on the significance of long-tail action recovery differs from the authors'. Even when focusing only on it as the central contribution, my view remains the same.

The reviewer's personal view, without any technical justification or supporting evidence, is not a valid basis for dismissing the significance of long-tail action recovery. Recent CUA research explicitly recognizes complex, low-frequency interactions as a critical bottleneck. FedGUI [1] incorporates 17 action types across heterogeneous platforms, while CUActSpot [2] extends evaluation beyond click-centric interactions to dragging, drawing, and complex operations across five modalities. As CUA mid-training advances toward diverse devices, professional workflows, and richer action spaces, recovering reliable supervision for these difficult actions is essential, not optional.

The OSWorld gain cited in the author response reflects the resulting training data as a whole where its causal link to this component is not strong.

This criticism overlooks our controlled experiments and incorrectly reduces the entire CUA domain to a single benchmark. OSWorld provides a standard end-to-end evaluation, but its task distribution cannot comprehensively capture action diversity, fine-grained grounding, or heterogeneous computer-use capabilities. OSWorld 2.0 [3] itself highlights the limited task horizons of its predecessor.

Our experimental evidence directly contradicts this dismissal:

1. Matched training controls: Under identical data sources and training recipes, Rebuilder+Experts improves OSWorld AVG@4 from 19.53% to 21.75% and mid-training ScreenSpot-Pro accuracy from 20.94% to 29.22%, demonstrating benefits beyond a single benchmark (refer to Reviewer otSZ, Q2, A3–A4).

2. Independent rare-action evaluation: On our CUA-Suite-based evaluation [4], right-click Type+Arg accuracy reaches 69.13%, compared with only 34.23% for Gemini, directly demonstrating the value of action-specific reconstruction and verification (refer to Reviewer otSZ, Q3, A5–A6; Reviewer Qztn, Q1, A3–A4).

The reviewer nevertheless disregards these findings while invoking hypothetical distributional trade-offs without providing any contradictory evidence.

Our pipeline explicitly addresses the quality–cost trade-off through improved annotation accuracy and efficient verification. On CUA-Suite, our pipeline substantially outperforms Gemini in Type+Arg accuracy for both left-click 732˙873\.28% vs\. 38\.17% and right-click 691˙369\.13% vs\. 34\.23%, while achieving a lower estimated per-sample cost $0.000868 vs. Gemini's measured API cost of $0.001087; refer to Reviewer Qztn, Q1, A2–A4; Reviewer otSZ, Q3, A5–A6. These results demonstrate substantial improvements in action grounding accuracy alongside competitive annotation costs, providing concrete evidence of the pipeline's practical value.

We ask the reviewer to explicitly clarify which concerns raised in the original assessment have been resolved by our evidence, and which remain unaddressed and require further justification, rather than introducing new speculative criticisms.

[1] FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems. Findings of ACL 2026.

[2] Covering Human Action Space for Computer Use: Data Synthesis and Benchmark. 2026.

[3] OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks. 2026.

[4] CUA-Suite: Massive Human-Annotated Video Demonstrations for Computer-Use Agents. 2026.

Comments on the paper

2 comments

We have submitted a second-round response addressing Reviewer MpMD's four points, supported by additional experiments and evidence from our revised manuscript:

1. Diagnostic Construction and Diversity: We reiterated the fundamental distinction between successful trajectory sources and negative candidate generators, and provided evidence of diversity across model families, applications, action types, and independently evaluated checkpoints.

2. Main Contribution and Experimental Evidence: We demonstrated the importance of long-tail action recovery through matched training controls, cross-benchmark results, and independent CUA-Suite evaluations. We also cited contemporary studies, FedGUI [1] and CUActSpot [2], which independently highlight the importance of diverse, complex, and low-frequency GUI actions.

3. Related Work and Novelty: We addressed the incorrect characterization of AgentVQA and clarified the distinct technical contribution of Correction-before-Rejection.

4. Checkpoint Diagnostic Validity: We provided complete diagnostic scores, additional checkpoints, quantitative behavioral analyses, and complementary benchmark evidence demonstrating the diagnostic's practical value.

Importantly, Reviewer MpMD has explicitly acknowledged two factual errors in the original review: confusing successful trajectory sources with negative candidate generators, and incorrectly identifying AgentVQA as an accepted ICLR 2026 paper.

However, the second response also introduces several concerns not explicitly raised in the original review, including:

  • Dismissing the significance of long-tail action recovery based on personal judgment:

"because my view on the significance of long-tail action recovery differs from the authors'"

  • Introducing new causal-attribution and distributional trade-off objections:

"its causal link to this component is not strong"

"such changes are not necessarily Pareto improvements in general and practical performance"

Our central concern is that, despite acknowledging fundamental factual errors and receiving substantial new experimental evidence, the reviewer has not clarified which original concerns have been resolved or how these corrections affect the exceptionally negative assessment. Instead, new justifications and speculative objections continue to emerge.

[1] FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systems. Findings of ACL 2026.

[2] Covering Human Action Space for Computer Use: Data Synthesis and Benchmark. 2026.

AuthorsOpenReview ↗

General Response: Summary of Clarifications and Revisions

Dear Reviewers and Area Chair,

We sincerely thank the reviewers for their constructive feedback, which has helped us strengthen the experimental rigor and analysis of GUIVista.

We are encouraged that reviewers recognized the "useful evidence" provided by our action-recovery ablations (Reviewer otSZ) and the "measurable downstream gains" achieved through mid-training (Reviewer otSZ), also acknowledged by Reviewer Qztn.

To address the shared concerns regarding the necessity of reconstruction and verification, the reliability of offline diagnostics, and the sources of downstream improvements, we have conducted additional experiments and provided detailed clarifications. These include matched training baselines, strong VLM comparisons on CUA-Suite, independent checkpoint validation, and fine-grained analyses of task-solving behavior and cross-application generalization.

We have supplemented our rebuttal with these results and will incorporate the corresponding clarifications into the revised manuscript, with updates highlighted in blue\color{blue}{\text{blue}}.

Summary of Key Updates and Responses

  • Diagnostic Reliability and Diversity (Reviewers otSZ, MpMD): We clarified that successful trajectories originate from diverse model families, not the negative candidate generators. Our 2,849-item diagnostic spans 10 application groups and six action families. Unweighted diagnostic scores strongly correlate with OSWorld performance across 19 models (Pearson r = 0.9005) and seven independently held-out checkpoints (r = 0.9252).

  • Long-tail Action Recovery and Downstream Effectiveness (Reviewers otSZ, MpMD): Matched training experiments demonstrate improvements in OSWorld AVG@4 (19.53% → 21.75%) and ScreenSpot-Pro (20.94% → 29.22%), directly validating the benefits of Correction-before-Rejection and action-specific verification.

  • Independent Annotation Quality and Cost Validation (Reviewers otSZ, Qztn): On CUA-Suite (1,000 samples across 82 applications), our pipeline substantially outperforms Gemini in Type+Arg accuracy for left-click (73.28% vs. 38.17%) and right-click (69.13% vs. 34.23%), at a lower estimated per-sample cost ($0.000868 vs. $0.001087 for Gemini's measured API cost).

  • Checkpoint Behavior and Generalization (Reviewers Qztn, MpMD): New trajectory-level analyses show effective-action rates increasing from 43.3% to 84.3% (Base → 400K), while successful trajectories become 12.9% shorter (400K → 800K). Additional ScreenSpot-Pro, OSWorld 2.0, and unseen-application evaluations further examine transferability and diagnostic validity.

  • Related Work and Manuscript Corrections (Reviewers otSZ, MpMD): We clarified GUIVista's distinct Correction-before-Rejection contribution relative to MONDAY, Video2GUI, and AgentVQA, expanded related-work discussions, and corrected data-source and evaluation-metric descriptions.

We have incorporated these experiments, analyses, and clarifications into the revised manuscript. We believe the additional evidence substantially strengthens the methodological rigor, experimental validity, and practical significance of GUIVista.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.