- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper uses attention analysis and attention knockout to study how audiovisual information moves through decoder layers. It then combines conservative compression before the LLM with within-LLM compression that gradually shifts from intra-modal guidance to text guidance.
Critical strengths
(1) The attention-knockout analysis provides a useful empirical basis for the layer-dependent compression schedule. (2) Evaluation across three models, five benchmarks, and several retention levels is fairly broad. The component ablations support the complementary roles of the two stages.
Critical weaknesses
(1) SEATS already combines compression before and within the LLM with late removal of modality tokens. OmniPack also combines early coverage with later task guidance, while prior information-flow work studies late token removal. This paper's more specific contribution is the gradual shift from intra-modal to text guidance, but its ablations do not compare that shift with a fixed switch layer under the same layer-wise token budgets. (2) At 25% retention on DailyOmni, OmniRelay's measured prefill time is 0.344 seconds versus 0.305 seconds for SEATS. Accuracy at matched measured latency would make the efficiency trade-off clearer.
Questions for authors
Under identical layer-wise token budgets, could the authors compare gradual guidance, a fixed switch layer, and single-source guidance, then report accuracy at matched measured latency?
3 comments
We also compare OmniRelay with OmniPack and SEATS on Qwen2.5-Omni-3B/7B. We omit SEATS results for Qwen2.5-Omni-3B because its 36-layer architecture is incompatible with the 28-layer pruning schedule in the SEATS implementation used in our evaluation.
Table 1: Results on AVUT, WorldSense and DailyOmni with Qwen2.5-Omni-7B.
| Method | AVUT (%) ↑ | WorldSense (%) ↑ | DailyOmni (%) ↑ | Avg. relative perf. (%) ↑ |
|---|---|---|---|---|
| Full Tokens | 64.76 | 46.63 | 62.91 | 100.00 |
| 25% Retention | ||||
| OmniPack | 62.34 | 45.74 | 58.23 | 95.64 |
| SEATS | 63.09 | 45.30 | 59.82 | 96.55 |
| OmniRelay (Ours) | 64.65 | 46.00 | 60.99 | 98.48 |
| 15% Retention | ||||
| OmniPack | 60.50 | 45.27 | 56.89 | 93.65 |
| SEATS | 59.75 | 44.33 | 57.14 | 92.72 |
| OmniRelay (Ours) | 60.84 | 45.68 | 59.48 | 95.49 |
| 10% Retention | ||||
| OmniPack | 58.77 | 44.01 | 55.97 | 91.37 |
| SEATS | 58.65 | 43.03 | 56.06 | 90.65 |
| OmniRelay (Ours) | 59.86 | 44.23 | 57.31 | 92.80 |
Table 2: Results on AVUT, WorldSense and DailyOmni with Qwen2.5-Omni-3B.
| Method | AVUT (%) ↑ | WorldSense (%) ↑ | DailyOmni (%) ↑ | Avg. relative perf. (%) ↑ |
|---|---|---|---|---|
| Full Tokens | 61.88 | 46.44 | 61.24 | 100.00 |
| 25% Retention | ||||
| OmniPack | 58.71 | 45.21 | 54.30 | 93.63 |
| SEATS | — | — | — | — |
| OmniRelay (Ours) | 60.61 | 45.49 | 58.90 | 97.36 |
| 15% Retention | ||||
| OmniPack | 57.79 | 43.66 | 53.13 | 91.39 |
| SEATS | — | — | — | — |
| OmniRelay (Ours) | 58.71 | 44.58 | 57.39 | 94.86 |
| 10% Retention | ||||
| OmniPack | 56.69 | 42.02 | 51.71 | 88.84 |
| SEATS | — | — | — | — |
| OmniRelay (Ours) | 57.84 | 43.69 | 56.73 | 93.40 |
As shown in Tables 1 and 2, our OmniRelay outperforms OmniPack on both backbones and SEATS on Qwen2.5-Omni-7B across all three benchmarks at every tested retention rate, further demonstrating the effectiveness of our design.
W2.At 25% retention on DailyOmni, OmniRelay's measured prefill time is 0.344 seconds versus 0.305 seconds for SEATS. Accuracy at matched measured latency would make the efficiency trade-off clearer.
A2. Thank you for the suggestion. Different compression methods can introduce different prefill costs at the same layer-averaged token retention ratio. Figure 5(a) in Section 4.2 of the revised manuscript therefore plots accuracy against measured prefill latency across retention ratios to show the accuracy–efficiency trade-off.
As shown in Table 3 and Figure 5(a), OmniRelay offers a favorable accuracy–latency trade-off relative to the evaluated baselines. On DailyOmni, SEATS achieves 59.82% accuracy in 305 ms at 25% retention. Our OmniRelay provides two measured operating points around this latency. At 15% retention, it achieves 59.48% accuracy in 235 ms, saving 70 ms (23%) with a 0.34% in accuracy. At 25% retention, it achieves 60.99% accuracy in 344 ms, gaining 1.17% with 39 ms of additional prefill latency.
To directly address your request for a comparison at matched measured latency, we additionally evaluate OmniRelay at 20% token retention on DailyOmni. OmniRelay achieves 60.15% accuracy with a prefill latency of 301 ms, compared with 59.82% at 305 ms for SEATS at 25% retention. Thus, OmniRelay improves accuracy by 0.33% while being 4 ms faster at closely matched measured latency.
Q1.Under identical layer-wise token budgets, could the authors compare gradual guidance, a fixed switch layer, and single-source guidance, then report accuracy at matched measured latency?
A3. We compare gradual transitions, hard switches at fixed layers and single-source guidance on Qwen2.5-Omni-7B at 25% token retention. We keep the per-layer token budgets and pruning locations fixed across these comparisons. The two single-source controls correspond to Table 5(b) of the revised manuscript.
| Guidance schedule | AVUT (%) ↑ | WorldSense (%) ↑ | DailyOmni (%) ↑ | Avg. relative perf. (%) ↑ | DailyOmni prefill (ms) ↓ |
|---|---|---|---|---|---|
| Intra-modal guidance only | 62.28 | 45.78 | 60.65 | 96.92 | 339 |
| Text guidance only | 61.36 | 44.77 | 59.57 | 95.15 | 333 |
| Hard switch at L10 | 63.78 | 45.78 | 59.90 | 97.29 | 334 |
| Hard switch at L13 | 62.17 | 45.37 | 60.32 | 96.39 | 338 |
| Hard switch at L16 | 61.13 | 45.15 | 59.65 | 95.35 | 335 |
| Best reversed hard switch (L16) | 64.01 | 44.51 | 60.23 | 96.68 | 334 |
| Gradual cosine transition | 64.19 | 45.93 | 60.65 | 98.01 | 344 |
| Gradual transition (Ours) | 64.65 | 46.00 | 60.99 | 98.48 | 344 |
As shown in the table above, our gradual transition improves average relative performance by 1.19% over the best forward hard switch, by 1.80% over the best reversed hard switch, and by 1.56% and 3.33% over intra-modal-only and text-only guidance respectively. At the same reported prefill latency of 344 ms, it also achieves 60.99% vs. 60.65% accuracy on DailyOmni with the cosine transition.
References
[1] From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs. NeurIPS 2026.
[2] Stage-adaptive Token Selection for Efficient Omni-modal LLMs. arXiv:2605.20035.
[3] OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models. arXiv:2608.03812.
We hope that the above response will fully address your concerns, and we eagerly anticipate your further suggestions and thoughts.
We express our sincere gratitude for your thorough assessment and recognition of our analysis, evaluation and presentation. We appreciate the opportunity to address your concerns.
A1.Thank you for your concerns about distinguishing our work with prior information-flow studies and baselines. Our work is distinct from an incremental extension of SEATS and OmniPack. Our main novelty lies in identifying a depth-dependent pattern of audiovisual information flow and designing a compression strategy around that observation. Specifically, we observe a shift from intra-modal interactions in early layers to audiovisual-to-text information transfer in middle layers. This shift directly motivates both the choice of guidance signals in DARC and the order in which we use them across layers. Below, we compare OmniRelay with prior information-flow work and related baselines.Comparison with prior information-flow work. From Senses to Decisions [1] observed that information flows through a modality-to-text and then a text-to-final-token pathway, which is verified in our experiments on Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6. However, a key question underexplored in their work is how information flows directly between audio and video. Using attention knockout, we observe that in early layers: (1) intra-modal interactions have a substantial effect on prediction, while direct audio–video interactions have a minor effect. (2) both intra- and inter-window interactions contribute to prediction. These insights motivate the design of DARC.
Comparison with related baselines. Motivated by the observed information relay, OmniRelay uses DARC to gradually shift from intra-modal guidance to text guidance as audiovisual information accumulates in text representations. DARC then removes the remaining audiovisual tokens in late layers. Table 1 summarizes the key differences in within-LLM compression among OmniRelay, SEATS [2] and OmniPack [3].
We explain the reason for our design choices below.
For compression guidance, all three methods use text guidance to retain task-relevant information. Our method additionally follows the observed information relay across layers, gradually shifting from intra-modal anchor guidance to text guidance. This allows it to exploit global redundancy across windows. Besides, audio and video transfer little information, suggesting that these interactions may offer a limited signal in OmniPack.
For Metrics, OmniPack method uses cosine similarity within the LLM. Our DARC uses attention weights to select important tokens directly along the pathways examined in our knockout analysis.
For Attention query tokens, SEATS selects the final token because it can attend to all preceding tokens. However, our information relay shows that the final token receives information mainly through text tokens, making it a less suitable query for scoring audiovisual tokens directly. Our ablation study below also supports this conclusion.
Conclusion. So our OmniRelay makes three main contributions. (1) We identify a depth-dependent information relay and characterize how intra-modal and cross-modal interactions contribute to prediction across layers. (2) We translate this observation into DARC, an attention-based compression method that gradually shifts from intra-modal anchor guidance to text guidance and removes all remaining audiovisual tokens in late layers. (3) We demonstrate OmniRelay's effectiveness across three models and five benchmarks.