acceptodds
Under review as a conference paper at ICLR 2027

Focused Omni-Modal GUI Agents

Abstract

Graphical User Interface (GUI) agents increasingly act over streaming omni-modal observations spanning screenshots, audio, video, interaction history, and device states. However, exposing this complete context uniformly at every step ignores that relevance is transient and sparse: the signal determining one action can become clutter at the next. Because executed actions feed back into the context, these errors compound into Sequential Distraction, driving the agent into irreversible trajectory drift. We formulate the underlying requirement as Focused Perception: a representation that is step-adaptive, action-sufficient, and non-intrusive. We realize this in , a lightweight adapter that dynamically prioritizes multimodal channels against a step-specific decision reference. Instead of uniformly processing all inputs, it compresses the most relevant evidence into a compact latent and subtly injects it into the frozen decoder's embeddings, guiding the model's attention without altering its native parameter space. FocusGUI trains from action supervision alone, requiring neither modality-relevance labels nor backbone fine-tuning. Across four omni-modal backbones, it raises overall Type Match by and Exact Match by over the frozen backbone on average, and Type Match by over the strongest existing enhancement method.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
GUI Agent, omni-modal

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_EzHcOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FocusGUI addresses Sequential Distraction in omni-modal GUI agents whose relevant evidence changes at each step. It keeps the full screenshot, audio, video, history, and device state while scoring channels against the current observation. The prioritized features are compressed into a 64-dimensional latent and injected as a residual into four prompt positions. The backbone is frozen; only the lightweight adapter is trained from action supervision.

Critical strengths
  1. The adapter is small and simple, and the zero-initialized residual is a sensible way to avoid hurting the frozen model at the start of training.

  2. Using the same split, prompt, and decoding for all four backbones is good practice.

  3. The paired correction analysis (153 corrections vs 13 regressions) and the bootstrap intervals in the depth plots are welcome.

  4. The paper tries to explain why cropping methods hurt Exact Match, which is a fair point about region-based methods in omni-modal settings.

Critical weaknesses
  1. The closed-loop evaluation protocol is underspecified. The paper's Algorithm 2 explicitly says that the predicted action is executed and that it re-enters the next history, so the method is intended to use the model's own past predictions rather than always using ground-truth action history. However, the paper does not clearly explain whether executing a predicted action also changes the subsequent screenshot, video, audio, and device state, or whether the evaluation replays a fixed recorded trajectory while changing only the textual history. If observations remain fixed, the depth analysis does not establish environment-level trajectory drift; if the evaluation is genuinely closed-loop, the observation-transition procedure should be described in detail.

  2. Three of the four baselines (DRS-GUI, OmniParser, RegionFocus) are training-free, while FocusGUI is trained on 1,807 in-domain steps. GUI-Actor is trained, but it is a different kind of head. The natural baselines are missing: (a) prompt tuning or prefix tuning with K learned soft tokens that do not depend on the observation, (b) LoRA with a similar parameter count, and (c) a simple linear residual from the pooled hidden state. Baseline (a) is important. It would show how much of the gain is just learning the action format and the action-type distribution of OmniGUI (for example how often NONE or Wait is correct).

  3. The cropping and parsing methods lower EM in 12 of 12 cases. This might be correct, but it could also come from coordinate mapping errors after cropping. Please give more detail on how these were adapted to omni-modal models and how coordinates were mapped back.

  4. Phrases like "exactly what our framing predicts" and "Nothing in Eq. (2) carries state across steps: drift is produced one step at a time, so removing the corruption at each step removes its accumulation" are stronger than the evidence. The theory in Appendix G shows how a residual can change attention, but it does not show that this is what happens or that it helps.

Questions for authors
  1. Is evaluation done with ground-truth history or with the model's own past predictions?

  2. How is Goal Progress defined exactly? It is described as "the fraction of correctly executed steps per episode," but it is much lower than EM, so I think it might count steps until the first error. Please clarify.

  3. What do you get with a learned soft prompt of K=4 tokens that does not depend on the observation?

  4. Can you report mean and standard deviation over at least 3 seeds for Table 1 and Table 2?

  5. What is the extra inference cost?

4 comments
AuthorsOpenReview ↗

Responses to Reviewer EzHc (1/2)

Dear Reviewer EzHc,

Thank you for identifying these ambiguities. We clarify the evaluation protocol and report the additional controls below. Revisions in the updated manuscript are highlighted in purple.


For Weakness 1 & Question 1: Ground-truth or self-predicted history, and do observations change?

The main results use the ground-truth action history provided by the dataset, with fixed recorded observations; predicted actions are not executed. We additionally evaluated with each model's own predicted history, and FocusGUI's advantage remains.

  1. OmniGUI is built from recorded expert episodes, so the screenshot, video, audio and device state at each step are fixed recordings. Algorithm 2 describes deployment rather than evaluation.

  2. We additionally evaluate self-conditioned history on the same 508 steps from 142 episodes. Each method receives its own previous predictions, while observations remain fixed. Both use SDPA attention and generate at most 128 tokens:

Method (own predicted history)TMEMGP
Frozen Qwen2.5-Omni49.8013.586.50
FocusGUI66.5436.8117.66

FocusGUI leads at every depth, with EM of 29–48% versus 8–27% for the frozen model across steps 0 to 6+.

We clarified the protocols in Sec. 3.1 and added the self-conditioned results to Appendix C.7.

For Weakness 2 & Question 3: Soft prompt, LoRA and linear residual baselines

All three trained controls improve over the frozen backbone. FocusGUI improves further on every metric, particularly GP, indicating that learning the action format and action-type distribution does not account for the full gain.

(a) Observation-independent soft prompt with K = 4 learned tokens, 3 seeds:

MethodTMEMGP
Frozen backbone49.6112.605.50
Soft prompt, K = 465.09 ± 2.4134.71 ± 5.6310.98 ± 3.75
FocusGUI70.21 ± 2.8039.63 ± 2.5116.43 ± 3.23

(b, c) LoRA with a matched parameter count (q_proj and v_proj in all 28 layers, 1,441,792 vs. 1,444,992 parameters), and a linear residual from the pooled hidden state at the same K = 4 positions:

MethodTMEMGP
Linear residual (3.68M params)57.2830.129.77
LoRA64.7737.2812.23
FocusGUI71.6540.1617.49

FocusGUI exceeds the soft prompt by 5.12 TM, 4.92 EM and 5.46 GP, with about 50% higher Goal Progress. This comparison supports the value of observation-conditioned residuals beyond an input-independent prompt. All controls use the same data, schedule and decoding.

We added these controls to Table 4 and their configurations to Appendix C.6.

For Weakness 3: Could the EM drop of cropping baselines come from coordinate mapping?

No coordinate remapping is performed in these adaptations: every method predicts directly in the full-screen frame.

  1. The three methods supply candidate boxes, which are appended to the prompt as text, while the model receives the full, uncropped screenshot and all other channels: up to 12 element boxes (OmniParser-style), up to 10 region candidates (RegionFocus-style), and up to 12 boxes from a three-level quadrant search (DRS-style).
  2. The model predicts coordinates in the full-screen frame, as with every other method.
  3. The lower EM is consistent with screenshot-derived candidates biasing grounding toward visually salient regions, even when the required evidence lies elsewhere on the screen or in the audio and video.

We clarified the baseline adaptations in Appendix D.6.

AuthorsOpenReview ↗

Responses to Reviewer EzHc (2/2)

For Weakness 4: Some claims are stronger than the evidence

We have revised the discussion around Eq. (2), the baseline results, and Appendix G; the new experiments support the role of the priorities.

Empirically, replacing α at test time reduces TM by 2.8–4.1 points, α correlates positively with the ablation-based modality importance of Fig. 1 (mean Spearman 0.45, 90% episode-level bootstrap interval [0.22, 0.63]), and the priorities of the five non-screen channels vary significantly across interaction phases.

For Question 2: How exactly is Goal Progress defined?

GP is computed for each episode as the number of consecutive exact-match steps from the start until the first error, divided by the episode length, and then averaged across episodes. Correct steps after the first error are not counted, which is why GP is lower than per-step EM.

We clarified the GP definition in Sec. 3.1.

For Question 4: Mean and std over at least 3 seeds

The TM and EM gains exceed the seed variation on every backbone.

BackboneFrozen TM / EM / GPFocusGUI TMFocusGUI EMFocusGUI GP
Qwen2.5-Omni49.61 / 12.60 / 5.5070.21 ± 2.8039.63 ± 2.5116.43 ± 3.23
MiniCPM-o 4.536.02 / 13.58 / 7.5568.57 ± 0.8939.30 ± 2.3714.55 ± 2.22
VITA-1.513.98 / 9.65 / 6.0957.55 ± 1.3130.18 ± 3.506.24 ± 0.79
Baichuan-Omni-1.524.41 / 8.86 / 4.3262.27 ± 3.5532.15 ± 5.369.27 ± 1.89

Each run is evaluated on the same 508 test steps (mean ± std). On Qwen2.5-Omni, the submitted run (71.65 / 40.16 / 17.49) is within one standard deviation of the 3-seed mean. The 3-seed soft-prompt control is reported under W2 & Q3.

We added the three-seed results to Appendix C.5 (Table 7).

For Question 5: What is the extra inference cost?

The adapter takes 0.40 ms per step. End-to-end latency is 2.94 s with FocusGUI and 1.58 s for the frozen backbone; the increase comes from an additional prefill pass for channel extraction.

Component (one NVIDIA L20, 32 steps after 2 warm-up steps)Time per step
FocusGUI adapter (1.445M params)0.40 ms
Frozen backbone, end to end1.58 s
FocusGUI, end to end2.94 s

Decoding, the LM head and the output length are unchanged.

We added inference timings to Appendix D.7 (Table 14).


Thank you for your thoughtful feedback; we remain open to further discussion and will use your additional suggestions to improve the clarity and rigor of the paper.

Reviewer_EzHcOpenReview ↗

Response by reviewer EzHc

I thank the authors for their thoughtful response. It resolves most of my concerns hence I raise my score. Thanks.

We sincerely appreciate your time, thoughtful feedback, and willingness to reconsider our work. Thank you for acknowledging our clarifications and raising your score.

Reviewer_cCg2OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes FocusGUI, a small adapter on a frozen backbone. The adapter scores perceptual channels against a step-specific reference, compresses them into a compact latent and adds it as a zero-initialized residual to the last K prompt embeddings. It is trained only with action-token loss. On OmniGUI (708 episodes) with four backbones, it improves Type Match, Exact Match and Goal Progress over the frozen model and over four screen-refinement or action-head baselines.

Critical strengths

The method is simple (about 1.2M to 1.5M parameters) and applied to four different backbones with the same hyperparameters, with consistent gains. The framing of relevance as sparse across modalities is useful, and the interaction-depth analysis and negative control on Instant tasks are a good attempt at testing the mechanism.

Critical weaknesses

The main claim is not isolated from a simpler explanation. In Table 2 the variant without channel decomposition and prioritization already reaches 66.26 TM and 35.00 EM, against 49.61 and 12.60 for the frozen model and 71.65 and 40.16 for the full model.

The test set is small (some categories have about 80 to 120 steps), results come from one benchmark with no seeds or variance in the tables, and the sensitivity in Table 3 is non-monotonic and noisy (Temporal GP ranges from 12.5 to 26.4).

Questions for authors

Does a capacity-matched adapter without prioritization (same positions, same parameter budget, trained on the same data) recover most of the gain? Table 2 suggests yes.

4 comments
AuthorsOpenReview ↗

Responses to Reviewer cCg2 (1/2)

Dear Reviewer cCg2,

Thank you for the detailed review. Revisions in the updated manuscript are highlighted in purple.


For Weakness 1 & Question 1: Does a capacity-matched adapter without prioritization recover the gain?

No. None of the trained controls closes the gap: budget-matched LoRA, a larger linear residual, and an observation-independent soft prompt all score lower on TM, EM, and GP.

MethodParamsTMEMGP
Frozen backbone049.6112.605.50
Pooled features → linear residual3.68M57.2830.129.77
LoRA1.44M64.7737.2812.23
FocusGUI1.44M71.6540.1617.49
  1. LoRA is applied to q_proj and v_proj in all 28 decoder layers (rank 5 in 16 layers, rank 4 in 12), giving 1,441,792 parameters against 1,444,992 for FocusGUI. FocusGUI exceeds it by 6.88 TM, 2.88 EM and 5.26 GP.
  2. The linear residual mean-pools the frozen full-modal hidden state, maps it to 256 dimensions with a fixed random projection, standardizes it with training-set statistics, and produces the residual at the same K = 4 positions with a single linear layer. With 2.5× more parameters, it trails FocusGUI by 14.37 TM and 7.72 GP.
  3. An observation-independent soft prompt at the same positions, trained with 3 seeds, reaches 65.09 ± 2.41 / 34.71 ± 5.63 / 10.98 ± 3.75 (TM / EM / GP), against 70.21 ± 2.80 / 39.63 ± 2.51 / 16.43 ± 3.23 for FocusGUI.

All controls use the same data, schedule and decoding.

We added these controls to Table 4 and their configurations to Appendix C.6.

For Weakness 2 & Question 3: Small test set, no seeds, noisy Table 3; transfer to unseen apps?

The three-seed results support the TM and EM improvements across backbones. We also find limited sensitivity to K and gains on an app-disjoint test set.

  1. We ran FocusGUI with 3 seeds on all four backbones, each evaluated on the same 508 test steps (mean ± std):
BackboneFrozen TM / EM / GPFocusGUI TMFocusGUI EMFocusGUI GP
Qwen2.5-Omni49.61 / 12.60 / 5.5070.21 ± 2.8039.63 ± 2.5116.43 ± 3.23
MiniCPM-o 4.536.02 / 13.58 / 7.5568.57 ± 0.8939.30 ± 2.3714.55 ± 2.22
VITA-1.513.98 / 9.65 / 6.0957.55 ± 1.3130.18 ± 3.506.24 ± 0.79
Baichuan-Omni-1.524.41 / 8.86 / 4.3262.27 ± 3.5532.15 ± 5.369.27 ± 1.89

The TM gains (20.6–43.6 points) and EM gains (20.5–27.0 points) far exceed the seed standard deviations on every backbone.

  1. On the paired Qwen2.5-Omni test steps, FocusGUI corrects 153 errors of the frozen model and introduces 13 (exact McNemar test, p ≈ 1.7 × 10⁻³¹).

  2. In Table 3, the overall metrics vary by at most 3.0 points across a sixteenfold range of K, which is within the seed standard deviation. The wider range of Temporal GP reflects the metric on a small subset: the Temporal test subset has 124 steps in 32 episodes, and since GP counts only the correct prefix of each episode, a single early error in one episode can shift the subset's GP by about 3 points. The default (K = 4, dz = 64) is used for all backbones without per-benchmark tuning.

  3. To test transfer, we built an app-disjoint split: training uses 20 apps, validation 3 apps (meituan, bilibili, wechat) and testing 6 apps (spotify, weibo, amazon, imdb, aiqiyi, taobao), with no app shared between splits (495 test steps, 133 episodes):

MethodTMEMGP
Qwen2.5-Omni (frozen)46.0613.337.21
FocusGUI64.6528.699.01

FocusGUI improves TM by 18.59 points and more than doubles EM on unseen apps, with TM and EM improving in all five task categories.

We added the seed and transfer results to Appendices C.5 and C.9.

AuthorsOpenReview ↗

Responses to Reviewer cCg2 (2/2)

For Weakness 3 & Question 4: How were the baselines adapted to omni-modal backbones?

All baseline adaptations retain the complete observation and predict coordinates in the full-screen frame. GUI-Actor improves EM under the shared pipeline, while the three region-based methods reduce it. This contrast is consistent with a limitation of screenshot-derived candidates rather than a loss of input context.

  1. We implemented OmniParser-style (up to 12 parsed element boxes), RegionFocus-style (up to 10 region candidates) and DRS-style (three-level quadrant search, up to 12 boxes) adaptations for the omni-modal backbones. The candidates are appended to the prompt as text. The system prompt, the uncropped screenshot, audio, video, history and decoding are the same as for FocusGUI, and coordinates are predicted in the full-screen frame, so no coordinate remapping is needed.

  2. Under the same setup, the trained GUI-Actor head improves EM on all four backbones:

BackboneFrozen EMGUI-Actor EM
Qwen2.5-Omni12.6013.78
VITA-1.59.6511.02
MiniCPM-o 4.513.5814.17
Baichuan-Omni-1.58.8611.02
  1. The region candidates depend only on the screenshot and may direct grounding toward salient visual regions even when the required evidence lies elsewhere on the screen or in the audio and video.

We detailed the baseline adaptations in Appendix D.6.

For Question 2: Results when the model's own predictions re-enter the history?

Yes. FocusGUI improves all three metrics when each method receives its own previous predictions, and its EM remains higher at every depth.

Method (own predicted history)TMEMGP
Frozen Qwen2.5-Omni49.8013.586.50
FocusGUI66.5436.8117.66
StepNFrozen EMFocusGUI EM
014219.7238.73
114112.7729.08
21037.7733.01
3629.6848.39
43312.1248.48
5128.3341.67
6+1526.6740.00

At each step, the recorded action history is replaced by the method's own previous predictions, while the observations remain the recorded ones (508 steps, 142 episodes). Both methods use the same inference configuration (SDPA attention, up to 128 generated tokens).

We added these results to Sec. 3.4 and Appendix C.7.


We look forward to your further feedback and would be happy to address any additional questions or suggestions as we continue to strengthen this work.

Thank you for taking the time to consider our response and for raising your score. We appreciate your constructive feedback.

Thanks for the comments. I have updated my review.

Reviewer_4t4UOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper presents FocusGUI, an adapter that is trained to prioritize multimodal channels during each step of GUI agents. FocusGUI-learned representations are injected into the decoder embedding, and the model's parameters are kept frozen.

Critical strengths
  • The paper clearly explains the methodology. It first explains the intuition behind the proposed adapter, then describes each adapter component individually. In doing so, it motivates and describes each transformation objective. The proposed approach is overall clearly explained and well motivated.
  • The proposed methodology is demonstrated to be empirically sound by the results computed on four different backbone agents and 4 baselines. Across different metrics and configurations, the proposed approach leads almost always the best results, with large gains with respect to baselines as well.
Critical weaknesses
  • While the explanation of the architecture is detailed and well-reasoned, some of the results from the ablation study may require further attention. In particular, the improvements reported in Table 2 compared to the base model are already quite significant when training is performed using only action-sufficient projection (AP) and residual alignment (RA), and without channel decomposition (CD) and step-adaptive prioritization (SP). This configuration already achieves 66.26 TM and 35.00 EM, compared to 49.61 and 12.60 for the frozen model. When the modules constituting the prioritization are integrated, the relative improvements are only about 5% on TM and 5% on EM. Therefore, most of the improvement appears to stem from a soft prompt trained and fine-tuned for each phase. It would be useful to further explore the role of prioritization, which is central to the rationale behind the proposed approach. For detailed questions regarding the prioritization phases, see also Q1.
  • The Sequential Distraction is central in the paper's framing: this concept introduces the idea that the agents' own action, fed back to the model, may compound errors. However, the empirical validation of this concept is not clearly presented. In particular, the evidence offered seems quite indirect: Figure 3 shows that accuracy declines with step index for the frozen backbone and holds for FocusGUI. This is compatible with compounding, but also with simpler explanations like longer context and history length, different step compositions at different depths (e.g., more wait or termination actions in later steps), or harder episodes being longer. The "3+" bin pools all later steps, and the paper does not control for these factors or report the action-type distribution by depth.
Questions for authors

Q1) The paper presents the priorities (alpha) as evidence that FocusGUI performs adaptive focus by step (Fig. 4, Fig. 5), but also states that alpha reflects the internal organization of the adapter rather than the model's use of different modalities. To reinforce the soundness of the analysis of the proposed approach, it would be useful to determine whether the assigned priorities are interpretable and accurate. Specifically:

  • Do they agree with the dominant/interfering mode labels derived from ablation and shown in Fig. 1, for example, through rank correlation for each step?
  • How do the model outputs change if alpha is replaced by uniform weights or shuffled across the various steps during testing, without retraining?
  • Can the authors provide the variance or confidence intervals for the phase means shown in Fig. 4? Are the observed differences statistically significant?
4 comments
AuthorsOpenReview ↗

Responses to Reviewer 4t4U (1/2)

Dear Reviewer 4t4U,

Thank you for the careful review. We address the role of prioritization and the interpretation of the depth analysis below. Revisions in the updated manuscript are highlighted in purple.

For Weakness 1: Does the gain come from prioritization or from a trained soft prompt?

The gain is not explained by training an additional module: FocusGUI outperforms all three trained controls, including one with 2.5× more parameters.

  1. The Table 2 variant without CD and SP is not a static soft prompt. It computes its residual from the backbone's integrated hidden state at every step and is therefore conditioned on the observation.

  2. We trained the following controls with the same data, schedule and decoding; the residual-based controls use the same K = 4 boundary positions.

MethodTMEMGP
Frozen backbone49.6112.605.50
Linear residual from pooled hidden state (3.68M params)57.2830.129.77
LoRA (1.44M params)64.7737.2812.23
FocusGUI (1.44M params)71.6540.1617.49
  1. An observation-independent soft prompt at the same positions, trained with 3 seeds, reaches 65.09 ± 2.41 TM, 34.71 ± 5.63 EM and 10.98 ± 3.75 GP, compared with 70.21 ± 2.80, 39.63 ± 2.51 and 16.43 ± 3.23 for FocusGUI under the same seeds.

  2. Replacing the learned priorities at test time, without retraining, reduces TM by 2.8–4.1 points (Q1(b)), so the model relies on them at inference.

We added the trained controls to Sec. 3.3 and their configurations to Appendix C.6.

For Weakness 2: Is the depth trend due to compounding, or to action composition or episode difficulty?

FocusGUI's advantage persists under all of the controls below: when the model's own predictions feed back into the history, at every depth, within the most frequent action types, and on a fixed episode set.

  1. In a self-conditioned evaluation, each method's previous predictions replace the recorded action history, while the observations remain the recorded ones. Both methods use the same inference configuration (SDPA attention, up to 128 generated tokens). The former 3+ bin is separated by step below:
StepNFrozen EMFocusGUI EM
014219.7238.73
114112.7729.08
21037.7733.01
3629.6848.39
43312.1248.48
5128.3341.67
6+1526.6740.00
Overall (TM / EM / GP)50849.80 / 13.58 / 6.5066.54 / 36.81 / 17.66
  1. Within the same ground-truth action type, FocusGUI still improves. For tap and wait, which together account for 66% of the test steps, EM rises from 6.39 to 15.04 on tap (266 steps) and from 47.14 to 70.00 on wait (70 steps).

  2. Restricting the analysis to episodes with at least four steps keeps the episode set fixed across depths, and the margin remains.

  3. On Instant tasks, the TM difference is not statistically distinguishable from zero at the first two steps; gains appear from depth 2 onward (+28.57 and +46.67 TM). The benefit therefore depends on depth rather than appearing uniformly from the first step.

We added the self-conditioned evaluation to Sec. 3.4 and Appendix C.7.

AuthorsOpenReview ↗

Responses to Reviewer 4t4U (2/2)

For Question 1(a): Do the priorities agree with the modality labels of Fig. 1?

Yes. The priorities correlate positively with the ablation-based modality importance used in Fig. 1. The highest-priority modality agrees with the ablation-dominant modality about twice as often as chance.

StatisticResult
Mean Spearman correlation+0.45
90% episode-level bootstrap interval[+0.22, +0.63]
Top-1 agreement (44 steps with a positive dominant modality)55.18%
Chance level26.52%

We map α to the four modalities ablated in Fig. 1: screen (screen and screen-spatial channels), audio, video and history (text/history channel). The reference importance of a modality is the drop in the frozen model's log-likelihood of the ground-truth action when that modality is removed. We use all 59 held-out test steps covered by the Fig. 1 probe (16 episodes). Ties in α are split evenly, and the chance level accounts for the number of available modalities at each step. Interaction state and full-modal hidden have no counterpart among the ablated modalities and are not included. FocusGUI receives no modality-relevance labels, so this alignment emerges from action supervision alone.

We added this analysis and its confidence interval to Appendix C.8.

For Question 1(b): What happens if α is made uniform or shuffled at test time?

Replacing α at test time consistently lowers TM by 2.8–4.1 points without retraining, so the step-specific priorities directly shape the model's predictions.

α at test timeTMDrop vs. learned
Learned71.65—
Uniform67.524.13
Swapped across episodes (seed 1)68.313.35
Swapped across episodes (seed 2)68.902.76
Swapped across episodes (seed 3)67.913.74

We use the submitted checkpoint and regenerate all 508 test steps under each condition. In the swapped condition, each step receives the full priority vector of a step at the same depth from a different episode, which preserves the distribution of α but removes its link to the current observation.

We added the priority interventions to Appendix C.8.

For Question 1(c): Are the phase differences in Fig. 4 significant?

Yes. All five non-screen channels show phase differences that remain significant after BH correction at q < 0.05. Neither screen channel passes this threshold.

Channelq
Text / History0.00028
Audio0.00028
Video0.00028
Interaction State0.00028
Full-modal Hidden0.00028
Screen0.575
Screen Spatial0.066

We use the weighted between-phase sum of squares as the test statistic, permute phase labels within each episode (5,000 permutations), and apply BH correction across the seven channels.

We added the phase tests to Appendix C.8 and clarified the Fig. 4 caption.


We welcome further discussion of any remaining concerns and would be grateful for additional suggestions to help us refine the analysis and improve the paper.

I thank the author for their additional experiments and explanations. All my concerns have been addressed; I will raise my score.

We appreciate your careful review and your openness to our clarifications. Thank you for raising your score; your feedback has improved both the evaluation and the presentation of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.