- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Cobra, a framework that jointly trains observation routing and action policies for GUI agents. A router decides whether to supplement screenshots with a pruned accessibility tree, while modality-specific LoRA adapters train the actor for the two observation formats. The method introduces Counterfactual Modality Advantage to provide step-level routing credit from terminal rewards. Experiments on three GUI benchmarks show improved task success, with controlled comparisons and ablations on OSWorld-Verified.
Critical strengths
-
The paper addresses an important problem for GUI agents: deciding when structural information is useful and which parts should reach the actor. Learning this decision from task outcomes is a practical direction for improving observation use.
-
The evaluation spans Linux, Android, and Windows environments. The fixed-observation comparisons and component ablations on OSWorld-Verified provide useful evidence for the proposed design.
-
The paper is generally clear and well organized, with theoretical analysis and implementation details that help explain the method.
Critical weaknesses
-
The performance degradation from combining vision and accessibility trees needs further analysis. In Table 2, Always-fused Online RL achieves 39.6% success, below the 40.8% of Visual-only Online RL. This is somewhat surprising and directly relevant to the motivation for dynamic routing. The paper discusses redundant information, stale nodes, and modality conflicts, but does not establish which factors cause the degradation. Controlling tree length, filtering irrelevant nodes, and examining failures would help distinguish information quality from context overhead and optimization issues. The shared-adapter gradient analysis does not directly explain why always-fused input underperforms.
-
The lack of training-free comparisons makes it difficult to isolate the router's value. I would like to see the same frozen actor evaluated under four inference-time settings without additional training: vision only, accessibility tree only, both inputs, and router-selected inputs. Tasks, interaction budgets, and decoding settings should be identical, and the router's training status should be specified. The existing untrained-router ablation still trains the actor and therefore does not replace this comparison. This experiment would clarify whether directly adding structural information is already effective and how much additional benefit routing provides.
-
The cost analysis excludes router overhead and does not establish an overall efficiency advantage. The reported token savings concern the actor, but the router also processes screenshots, rule-compressed accessibility trees, and history at every step. Some costs may simply be transferred to the router. A fair system-level comparison should include both models' input, output, and computational costs, together with end-to-end latency and average cost per successful task. The current results demonstrate shorter actor context, but whether these savings offset the additional routing overhead remains unclear.
-
The main experiments lack uncertainty estimates. Several differences are relatively small: the 1.2 percentage-point decrease from always-fused input, the 2.1-point gain from CMA, and the 1.1-point gain over always-fused input on tasks excluded from updates. Although the appendix reports three evaluations per checkpoint, this does not capture training variability. Results across independent training seeds and paired confidence intervals for performance differences would help establish whether these findings are stable enough to support the paper's explanations and generalization claims.
Questions for authors
Refer to the weaknesses.