- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FocusGUI addresses Sequential Distraction in omni-modal GUI agents whose relevant evidence changes at each step. It keeps the full screenshot, audio, video, history, and device state while scoring channels against the current observation. The prioritized features are compressed into a 64-dimensional latent and injected as a residual into four prompt positions. The backbone is frozen; only the lightweight adapter is trained from action supervision.
Critical strengths
-
The adapter is small and simple, and the zero-initialized residual is a sensible way to avoid hurting the frozen model at the start of training.
-
Using the same split, prompt, and decoding for all four backbones is good practice.
-
The paired correction analysis (153 corrections vs 13 regressions) and the bootstrap intervals in the depth plots are welcome.
-
The paper tries to explain why cropping methods hurt Exact Match, which is a fair point about region-based methods in omni-modal settings.
Critical weaknesses
-
The closed-loop evaluation protocol is underspecified. The paper's Algorithm 2 explicitly says that the predicted action is executed and that it re-enters the next history, so the method is intended to use the model's own past predictions rather than always using ground-truth action history. However, the paper does not clearly explain whether executing a predicted action also changes the subsequent screenshot, video, audio, and device state, or whether the evaluation replays a fixed recorded trajectory while changing only the textual history. If observations remain fixed, the depth analysis does not establish environment-level trajectory drift; if the evaluation is genuinely closed-loop, the observation-transition procedure should be described in detail.
-
Three of the four baselines (DRS-GUI, OmniParser, RegionFocus) are training-free, while FocusGUI is trained on 1,807 in-domain steps. GUI-Actor is trained, but it is a different kind of head. The natural baselines are missing: (a) prompt tuning or prefix tuning with K learned soft tokens that do not depend on the observation, (b) LoRA with a similar parameter count, and (c) a simple linear residual from the pooled hidden state. Baseline (a) is important. It would show how much of the gain is just learning the action format and the action-type distribution of OmniGUI (for example how often NONE or Wait is correct).
-
The cropping and parsing methods lower EM in 12 of 12 cases. This might be correct, but it could also come from coordinate mapping errors after cropping. Please give more detail on how these were adapted to omni-modal models and how coordinates were mapped back.
-
Phrases like "exactly what our framing predicts" and "Nothing in Eq. (2) carries state across steps: drift is produced one step at a time, so removing the corruption at each step removes its accumulation" are stronger than the evidence. The theory in Appendix G shows how a residual can change attention, but it does not show that this is what happens or that it helps.
Questions for authors
-
Is evaluation done with ground-truth history or with the model's own past predictions?
-
How is Goal Progress defined exactly? It is described as "the fraction of correctly executed steps per episode," but it is much lower than EM, so I think it might count steps until the first error. Please clarify.
-
What do you get with a learned soft prompt of K=4 tokens that does not depend on the observation?
-
Can you report mean and standard deviation over at least 3 seeds for Table 1 and Table 2?
-
What is the extra inference cost?
4 comments
Responses to Reviewer EzHc (2/2)
For Weakness 4: Some claims are stronger than the evidence
We have revised the discussion around Eq. (2), the baseline results, and Appendix G; the new experiments support the role of the priorities.
Empirically, replacing α at test time reduces TM by 2.8–4.1 points, α correlates positively with the ablation-based modality importance of Fig. 1 (mean Spearman 0.45, 90% episode-level bootstrap interval [0.22, 0.63]), and the priorities of the five non-screen channels vary significantly across interaction phases.
For Question 2: How exactly is Goal Progress defined?
GP is computed for each episode as the number of consecutive exact-match steps from the start until the first error, divided by the episode length, and then averaged across episodes. Correct steps after the first error are not counted, which is why GP is lower than per-step EM.
We clarified the GP definition in Sec. 3.1.
For Question 4: Mean and std over at least 3 seeds
The TM and EM gains exceed the seed variation on every backbone.
| Backbone | Frozen TM / EM / GP | FocusGUI TM | FocusGUI EM | FocusGUI GP |
|---|---|---|---|---|
| Qwen2.5-Omni | 49.61 / 12.60 / 5.50 | 70.21 ± 2.80 | 39.63 ± 2.51 | 16.43 ± 3.23 |
| MiniCPM-o 4.5 | 36.02 / 13.58 / 7.55 | 68.57 ± 0.89 | 39.30 ± 2.37 | 14.55 ± 2.22 |
| VITA-1.5 | 13.98 / 9.65 / 6.09 | 57.55 ± 1.31 | 30.18 ± 3.50 | 6.24 ± 0.79 |
| Baichuan-Omni-1.5 | 24.41 / 8.86 / 4.32 | 62.27 ± 3.55 | 32.15 ± 5.36 | 9.27 ± 1.89 |
Each run is evaluated on the same 508 test steps (mean ± std). On Qwen2.5-Omni, the submitted run (71.65 / 40.16 / 17.49) is within one standard deviation of the 3-seed mean. The 3-seed soft-prompt control is reported under W2 & Q3.
We added the three-seed results to Appendix C.5 (Table 7).
For Question 5: What is the extra inference cost?
The adapter takes 0.40 ms per step. End-to-end latency is 2.94 s with FocusGUI and 1.58 s for the frozen backbone; the increase comes from an additional prefill pass for channel extraction.
| Component (one NVIDIA L20, 32 steps after 2 warm-up steps) | Time per step |
|---|---|
| FocusGUI adapter (1.445M params) | 0.40 ms |
| Frozen backbone, end to end | 1.58 s |
| FocusGUI, end to end | 2.94 s |
Decoding, the LM head and the output length are unchanged.
We added inference timings to Appendix D.7 (Table 14).
Thank you for your thoughtful feedback; we remain open to further discussion and will use your additional suggestions to improve the clarity and rigor of the paper.
Response by reviewer EzHc
I thank the authors for their thoughtful response. It resolves most of my concerns hence I raise my score. Thanks.
We sincerely appreciate your time, thoughtful feedback, and willingness to reconsider our work. Thank you for acknowledging our clarifications and raising your score.
Responses to Reviewer EzHc (1/2)
Dear Reviewer EzHc,
Thank you for identifying these ambiguities. We clarify the evaluation protocol and report the additional controls below. Revisions in the updated manuscript are highlighted in purple.
For Weakness 1 & Question 1: Ground-truth or self-predicted history, and do observations change?
The main results use the ground-truth action history provided by the dataset, with fixed recorded observations; predicted actions are not executed. We additionally evaluated with each model's own predicted history, and FocusGUI's advantage remains.
OmniGUI is built from recorded expert episodes, so the screenshot, video, audio and device state at each step are fixed recordings. Algorithm 2 describes deployment rather than evaluation.
We additionally evaluate self-conditioned history on the same 508 steps from 142 episodes. Each method receives its own previous predictions, while observations remain fixed. Both use SDPA attention and generate at most 128 tokens:
FocusGUI leads at every depth, with EM of 29–48% versus 8–27% for the frozen model across steps 0 to 6+.
We clarified the protocols in Sec. 3.1 and added the self-conditioned results to Appendix C.7.
For Weakness 2 & Question 3: Soft prompt, LoRA and linear residual baselines
All three trained controls improve over the frozen backbone. FocusGUI improves further on every metric, particularly GP, indicating that learning the action format and action-type distribution does not account for the full gain.
(a) Observation-independent soft prompt with K = 4 learned tokens, 3 seeds:
(b, c) LoRA with a matched parameter count (q_proj and v_proj in all 28 layers, 1,441,792 vs. 1,444,992 parameters), and a linear residual from the pooled hidden state at the same K = 4 positions:
FocusGUI exceeds the soft prompt by 5.12 TM, 4.92 EM and 5.46 GP, with about 50% higher Goal Progress. This comparison supports the value of observation-conditioned residuals beyond an input-independent prompt. All controls use the same data, schedule and decoding.
We added these controls to Table 4 and their configurations to Appendix C.6.
For Weakness 3: Could the EM drop of cropping baselines come from coordinate mapping?
No coordinate remapping is performed in these adaptations: every method predicts directly in the full-screen frame.
We clarified the baseline adaptations in Appendix D.6.