acceptodds
Under review as a conference paper at ICLR 2027

OmniRelay: Information-Flow-Guided Token Compression for Omni-Modal LLMs

Abstract

Omni-modal large language models (Omni-LLMs) achieve strong performance on audiovisual understanding tasks, but processing long audiovisual token sequences incurs substantial computational and memory costs. To reduce these costs, existing token compression methods exploit local redundancy at the pre-LLM stage and task relevance at the within-LLM stage, but make limited use of global modality dependencies. To determine when these dependencies should guide compression, we analyze attention patterns and perform attention knockout across layers. We observe a depth-varying relay: audiovisual information is integrated through intra-modal interactions within and across windows in early layers, transferred to text in middle layers, and finally relayed to the final input token for prediction. We therefore propose OmniRelay, a training-free pre&within-LLM framework that adapts compression guidance across layers. Its Depth-Adaptive Relay Compression (DARC) progressively shifts from intra-modal to text guidance before removing the remaining audiovisual tokens at late layers. Complementing DARC, our proposed Saliency–Diversity and Coverage-Aware Compression (SDCC) reduces local redundancy before the LLM while preserving salient and diverse tokens with broad cross-window coverage. We evaluate OmniRelay on 3 Omni-LLMs and 5 audiovisual benchmarks. It retains 99.51% of full-token performance at 25% layer-averaged audiovisual token retention, with 79% fewer FLOPs and a 3.3× prefill speedup.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal, omni, token reduction, model efficiency

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_aqD7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses attention analysis and attention knockout to study how audiovisual information moves through decoder layers. It then combines conservative compression before the LLM with within-LLM compression that gradually shifts from intra-modal guidance to text guidance.

Critical strengths

(1) The attention-knockout analysis provides a useful empirical basis for the layer-dependent compression schedule. (2) Evaluation across three models, five benchmarks, and several retention levels is fairly broad. The component ablations support the complementary roles of the two stages.

Critical weaknesses

(1) SEATS already combines compression before and within the LLM with late removal of modality tokens. OmniPack also combines early coverage with later task guidance, while prior information-flow work studies late token removal. This paper's more specific contribution is the gradual shift from intra-modal to text guidance, but its ablations do not compare that shift with a fixed switch layer under the same layer-wise token budgets. (2) At 25% retention on DailyOmni, OmniRelay's measured prefill time is 0.344 seconds versus 0.305 seconds for SEATS. Accuracy at matched measured latency would make the efficiency trade-off clearer.

Questions for authors

Under identical layer-wise token budgets, could the authors compare gradual guidance, a fixed switch layer, and single-source guidance, then report accuracy at matched measured latency?

3 comments

We express our sincere gratitude for your thorough assessment and recognition of our analysis, evaluation and presentation. We appreciate the opportunity to address your concerns.


W1a. SEATS already combines compression before and within the LLM with late removal of modality tokens. OmniPack also combines early coverage with later task guidance, while prior information-flow work studies late token removal. This paper's more specific contribution is the gradual shift from intra-modal to text guidance.

A1. Thank you for your concerns about distinguishing our work with prior information-flow studies and baselines. Our work is distinct from an incremental extension of SEATS and OmniPack. Our main novelty lies in identifying a depth-dependent pattern of audiovisual information flow and designing a compression strategy around that observation. Specifically, we observe a shift from intra-modal interactions in early layers to audiovisual-to-text information transfer in middle layers. This shift directly motivates both the choice of guidance signals in DARC and the order in which we use them across layers. Below, we compare OmniRelay with prior information-flow work and related baselines.

Comparison with prior information-flow work. From Senses to Decisions [1] observed that information flows through a modality-to-text and then a text-to-final-token pathway, which is verified in our experiments on Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6. However, a key question underexplored in their work is how information flows directly between audio and video. Using attention knockout, we observe that in early layers: (1) intra-modal interactions have a substantial effect on prediction, while direct audio–video interactions have a minor effect. (2) both intra- and inter-window interactions contribute to prediction. These insights motivate the design of DARC.

Comparison with related baselines. Motivated by the observed information relay, OmniRelay uses DARC to gradually shift from intra-modal guidance to text guidance as audiovisual information accumulates in text representations. DARC then removes the remaining audiovisual tokens in late layers. Table 1 summarizes the key differences in within-LLM compression among OmniRelay, SEATS [2] and OmniPack [3].

MethodSEATSOmniPackOmniRelay (Ours)
Compression guidancetext querytext query with av interactions1.intra-modal query   2.text query
Metricsattentioncosine similarityattention
Attention query tokensfinal tokenN/Atext tokens
Assumptionfinal token attends to all tokens.similarity denotes semantic relevance.Observed depth-adaptive information relay.

We explain the reason for our design choices below.

  1. For compression guidance, all three methods use text guidance to retain task-relevant information. Our method additionally follows the observed information relay across layers, gradually shifting from intra-modal anchor guidance to text guidance. This allows it to exploit global redundancy across windows. Besides, audio and video transfer little information, suggesting that these interactions may offer a limited signal in OmniPack.

  2. For Metrics, OmniPack method uses cosine similarity within the LLM. Our DARC uses attention weights to select important tokens directly along the pathways examined in our knockout analysis.

  3. For Attention query tokens, SEATS selects the final token because it can attend to all preceding tokens. However, our information relay shows that the final token receives information mainly through text tokens, making it a less suitable query for scoring audiovisual tokens directly. Our ablation study below also supports this conclusion.

    Query tokens for text guidance   AVUT (%) ↑   WorldSense (%) ↑   DailyOmni (%) ↑   Avg. relative perf. (%) ↑
    Final input token64.0145.7159.3197.05
    Text tokens (Ours)64.6546.0060.9998.48

Conclusion. So our OmniRelay makes three main contributions. (1) We identify a depth-dependent information relay and characterize how intra-modal and cross-modal interactions contribute to prediction across layers. (2) We translate this observation into DARC, an attention-based compression method that gradually shifts from intra-modal anchor guidance to text guidance and removes all remaining audiovisual tokens in late layers. (3) We demonstrate OmniRelay's effectiveness across three models and five benchmarks.

We also compare OmniRelay with OmniPack and SEATS on Qwen2.5-Omni-3B/7B. We omit SEATS results for Qwen2.5-Omni-3B because its 36-layer architecture is incompatible with the 28-layer pruning schedule in the SEATS implementation used in our evaluation.

Table 1: Results on AVUT, WorldSense and DailyOmni with Qwen2.5-Omni-7B.

MethodAVUT (%) ↑WorldSense (%) ↑DailyOmni (%) ↑Avg. relative perf. (%) ↑
Full Tokens64.7646.6362.91100.00
25% Retention
OmniPack62.3445.7458.2395.64
SEATS63.0945.3059.8296.55
OmniRelay (Ours)64.6546.0060.9998.48
15% Retention
OmniPack60.5045.2756.8993.65
SEATS59.7544.3357.1492.72
OmniRelay (Ours)60.8445.6859.4895.49
10% Retention
OmniPack58.7744.0155.9791.37
SEATS58.6543.0356.0690.65
OmniRelay (Ours)59.8644.2357.3192.80

Table 2: Results on AVUT, WorldSense and DailyOmni with Qwen2.5-Omni-3B.

MethodAVUT (%) ↑WorldSense (%) ↑DailyOmni (%) ↑Avg. relative perf. (%) ↑
Full Tokens61.8846.4461.24100.00
25% Retention
OmniPack58.7145.2154.3093.63
SEATS————
OmniRelay (Ours)60.6145.4958.9097.36
15% Retention
OmniPack57.7943.6653.1391.39
SEATS————
OmniRelay (Ours)58.7144.5857.3994.86
10% Retention
OmniPack56.6942.0251.7188.84
SEATS————
OmniRelay (Ours)57.8443.6956.7393.40

As shown in Tables 1 and 2, our OmniRelay outperforms OmniPack on both backbones and SEATS on Qwen2.5-Omni-7B across all three benchmarks at every tested retention rate, further demonstrating the effectiveness of our design.

W2. At 25% retention on DailyOmni, OmniRelay's measured prefill time is 0.344 seconds versus 0.305 seconds for SEATS. Accuracy at matched measured latency would make the efficiency trade-off clearer.

A2. Thank you for the suggestion. Different compression methods can introduce different prefill costs at the same layer-averaged token retention ratio. Figure 5(a) in Section 4.2 of the revised manuscript therefore plots accuracy against measured prefill latency across retention ratios to show the accuracy–efficiency trade-off.

As shown in Table 3 and Figure 5(a), OmniRelay offers a favorable accuracy–latency trade-off relative to the evaluated baselines. On DailyOmni, SEATS achieves 59.82% accuracy in 305 ms at 25% retention. Our OmniRelay provides two measured operating points around this latency. At 15% retention, it achieves 59.48% accuracy in 235 ms, saving 70 ms (23%) with a 0.34% in accuracy. At 25% retention, it achieves 60.99% accuracy in 344 ms, gaining 1.17% with 39 ms of additional prefill latency.

To directly address your request for a comparison at matched measured latency, we additionally evaluate OmniRelay at 20% token retention on DailyOmni. OmniRelay achieves 60.15% accuracy with a prefill latency of 301 ms, compared with 59.82% at 305 ms for SEATS at 25% retention. Thus, OmniRelay improves accuracy by 0.33% while being 4 ms faster at closely matched measured latency.


Q1. Under identical layer-wise token budgets, could the authors compare gradual guidance, a fixed switch layer, and single-source guidance, then report accuracy at matched measured latency?

A3. We compare gradual transitions, hard switches at fixed layers and single-source guidance on Qwen2.5-Omni-7B at 25% token retention. We keep the per-layer token budgets and pruning locations fixed across these comparisons. The two single-source controls correspond to Table 5(b) of the revised manuscript.

Guidance scheduleAVUT (%) ↑WorldSense (%) ↑DailyOmni (%) ↑Avg. relative perf. (%) ↑DailyOmni prefill (ms) ↓
Intra-modal guidance only62.2845.7860.6596.92339
Text guidance only61.3644.7759.5795.15333
Hard switch at L1063.7845.7859.9097.29334
Hard switch at L1362.1745.3760.3296.39338
Hard switch at L1661.1345.1559.6595.35335
Best reversed hard switch (L16)64.0144.5160.2396.68334
Gradual cosine transition64.1945.9360.6598.01344
Gradual transition (Ours)64.6546.0060.9998.48344

As shown in the table above, our gradual transition improves average relative performance by 1.19% over the best forward hard switch, by 1.80% over the best reversed hard switch, and by 1.56% and 3.33% over intra-modal-only and text-only guidance respectively. At the same reported prefill latency of 344 ms, it also achieves 60.99% vs. 60.65% accuracy on DailyOmni with the cosine transition.

References

[1] From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs. NeurIPS 2026.

[2] Stage-adaptive Token Selection for Efficient Omni-modal LLMs. arXiv:2605.20035.

[3] OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models. arXiv:2608.03812.


We hope that the above response will fully address your concerns, and we eagerly anticipate your further suggestions and thoughts.

Reviewer_ViRHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OmniRelay, a training-free framework for compressing audio and video tokens in omni-modal LLMs. Motivated by an analysis of information flow across decoder depth, it combines pre-LLM saliency, diversity, and coverage-based selection (SDCC) with within-LLM compression that progressively shifts from intra-modal to text-based guidance (DARC), followed by late removal of the remaining audiovisual tokens. Experiments on Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6 across five benchmarks report favorable accuracy–prefill-latency trade-offs under substantial token reduction.

Critical strengths
  1. The information-flow analysis provides a coherent motivation for the compression design. The paper uses attention analysis and attention knockout to motivate different guidance signals at different depths. This connects the proposed design to observed model behavior and provides insight beyond reporting a compression heuristic alone.

  2. The evaluation provides useful evidence of practical efficiency. The paper evaluates several compression budgets, includes relevant baselines such as OmniZip and SEATS at their supported settings, and reports prefill latency and memory alongside task performance. The component and strategy-combination ablations further help assess the contributions of the two compression stages.

Critical weaknesses
  1. Two discrepancies between the paper and the public Qwen implementation leave the evaluated method insufficiently specified. On page 6, the paper defines video anchors as the retained tokens from the medoid windows and the last window, and audio anchors as all retained audio tokens. However, the public Qwen implementation snapshot inspected on 21 October 2026 in omnirelay/models/inner_hooks.py uses last_token_per_window for both modalities, selecting the last surviving token in each window. The paper also describes jointly ranking audio and video tokens during text-guided compression under a shared total budget, whereas the implementation computes separate video_keep and audio_keep values and applies modality-specific top-k selection through topk_keep_mask. These differences affect the evidence used for importance scoring and whether the budget can move between modalities. The authors should identify the exact implementation corresponding to the reported results and reconcile these differences. The current evidence leaves uncertainty about which algorithm the main experiments validate.

  2. The evidence for transfer across different omni-modal architectures could be stronger. The three evaluated models comprise two sizes of Qwen2.5-Omni and MiniCPM-o-2.6. These results are useful, but they do not fully establish how the depth-dependent guidance and token-organization assumptions transfer to other multimodal designs. Appendix G acknowledges that adapting the method requires backbone-specific heuristic parameters and extraction of spatial positions. A clearer account of what transfers unchanged, what requires adaptation, and how those choices are made would strengthen the generalization claim. Evaluation on another audio-video understanding system with a different fusion design, such as JavisGPT [1], would provide additional evidence.

Overall assessment. I lean toward weak rejection. The information-flow motivation and empirical compression results are useful, but the unresolved differences between the described algorithm and the public implementation prevent a clear assessment of which method the experiments validate. The limited explanation of architecture-specific adaptation further weakens the generalization claim. A precise mapping between the paper, code revision, and reported experiments, together with a clearer account of transfer across backbones, could change my assessment.

Reference

[1] Kai Liu et al. JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation. NeurIPS 2025. Paper; official implementation.

Questions for authors
  1. Clarification of the evaluated algorithm: Which implementation and configuration were used for Tables 2–5? Please explain both the anchor-set discrepancy and the joint-versus-separate top-k discrepancy above, provide the corresponding code revision, and clarify whether reconciling the description and implementation changes the reported results.

  2. Clarification of architectural transfer: Which components and hyperparameters were kept fixed when moving between Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6, and which required backbone-specific choices? What evidence supports transferring the depth-dependent information-flow assumptions to other audio-video fusion designs?

Suggestion for additional evaluation: Testing the audio-video understanding setting of JavisGPT, or another suitable omni-modal model, would strengthen the transfer evidence. This is a suggested extension of the empirical coverage; the clarification questions above can be addressed separately from a new model evaluation.

5 comments

We are deeply grateful for your helpful feedback. We particularly appreciate your recognition of the motivation, evaluation and paper presentation. Hereafter, we provide detailed responses to your concerns and questions

Q1. Clarification of the evaluated algorithm: Which implementation and configuration were used for Tables 2–5? Please explain both the anchor-set discrepancy and the joint-versus-separate top-k discrepancy above, provide the corresponding code revision, and clarify whether reconciling the description and implementation changes the reported results.

A1. We thank the reviewer for carefully checking our code and pointing out these discrepancies. Due to a miscommunication among the co-authors, we inadvertently uploaded an earlier version. We have updated the anonymous repository to the correct revision in the 11 November snapshot and verified that it matches the algorithm described in Section 3.2 and corresponds to the implementation used to obtain the results in Tables 2–5. The reported results in Tables 2–5 therefore remain unchanged.

We illustrate the differences and provide additional ablations on anchor selection and audio–video budget allocation below.

Differences. The old and revised versions differ in anchor selection and audio–video budget allocation.

  1. Anchor selection. The old version used last token per window to select one anchor per modality in each temporal window. In the revised version, DARC uses the retained video tokens from the SDCC medoid windows and the final temporal window as video anchors, and all retained audio tokens as audio anchors (medoid and final video windows + all audio windows).
  2. Audio–video budget allocation. The old version used separate budgets and applied Topka(l)\mathrm{Topk}_a^{(l)} to audio tokens and Topkv(l)\mathrm{Topk}_v^{(l)} to video tokens independently. The revised version jointly ranks audio and video tokens by their combined scores and retains the top K(l)K^{(l)} tokens under a shared budget.

Ablations. We ablated the two choices individually and together on Qwen2.5-Omni-7B at 25% retention. As shown in the table below, both medoid and final video windows + all audio windows and shared budget improve performance. The last token per window is a less effective anchor, even though it can attend to all preceding tokens in that window. Joint selection under a shared budget outperforms selection under separate budgets and allows the audio–video split to adapt to the relative importance of each modality for each input.

Audio–video budget allocationIntra-modal anchorsAVUTWorldSenseDailyOmniAvg. Relative Perf.
Separate budgetslast token per window62.2845.0858.4895.27%
Separate budgetsOurs64.0145.8759.5797.30%
Ourslast token per window62.8645.5558.9096.13%
OursOurs64.6546.0060.9998.48%

W2. The evidence for transfer across different omni-modal architectures could be stronger. The three evaluated models comprise two sizes of Qwen2.5-Omni and MiniCPM-o-2.6. These results are useful, but they do not fully establish how the depth-dependent guidance and token-organization assumptions transfer to other multimodal designs. Appendix G acknowledges that adapting the method requires backbone-specific heuristic parameters and extraction of spatial positions. A clearer account of what transfers unchanged, what requires adaptation, and how those choices are made would strengthen the generalization claim.

Q2a. Clarification of architectural transfer: Which components and hyperparameters were kept fixed when moving between Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6, and which required backbone-specific choices?

A2. Thank you for suggesting that we clarify how OmniRelay transfers across architectures. Below, we describe which components and hyperparameters are shared across Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6 and how we choose the backbone-specific settings.

What transfers unchanged. We use SDCC to reduce local redundancy within temporal windows, with the same audio temporal-distance weight λa=0.20\lambda_a = 0.20. We observe the same qualitative information relay across all three backbones and apply the same DARC procedure. DARC gradually shifts from intra-modal anchor guidance to text guidance, jointly ranks audio and video tokens under a shared budget, and removes the remaining audiovisual tokens in late layers.

What requires adaptation. The two Qwen backbones use a 2D spatial-distance term for visual token selection (λv=0.20\lambda_v = 0.20). For MiniCPM-o-2.6, the Resampler outputs lack an explicit mapping to 2D coordinates, so we disable the spatial term (λv=0\lambda_v = 0) and use feature distance in the visual diversity criterion. We also adjust the pre-LLM audio and video retention ratios to account for differences in token counts across model families (Appendix C, Table 7). Backbones with the same LLM depth share the same pruning boundaries (ls,le)(l_s, l_e). We use the layer-wise attention knockout profiles as qualitative guidance for choosing the Gaussian centers (μ1,μ2)(\mu_1, \mu_2) and shared width σ\sigma. For each backbone, the distance and guidance parameters are fixed across benchmarks and retention ratios. The table below summarizes the DARC settings for the three backbones.

BackbonePruning boundaries (ls,le)(l_s, l_e) (zero-based)Guidance parameters (μ1,μ2,σ)(\mu_1, \mu_2, \sigma)
Qwen2.5-Omni-7B(5, 21)(10, 15, 5)
Qwen2.5-Omni-3B(7, 27)(13, 19, 6)
MiniCPM-o-2.6(5, 21)(12, 17, 5)

Qwen2.5-Omni uses an endpoint-normalized logistic schedule, and MiniCPM-o-2.6 uses a power-law schedule to better align pruning with its information-flow pattern (Appendix C.1).

Q2b. Evaluation on another audio-video understanding system with a different fusion design, such as JavisGPT [1], would provide additional evidence.

A3. Thank you for suggesting JavisGPT [1] as an additional omni-LLM to evaluate. Its motivation to improve audio–video alignment is consistent with our observation that blocking direct audio–video attention inside the LLM has a relatively small effect on predictions in models with separate audio and video tokens. We are also particularly interested in whether the same depth-dependent information relay holds when audio and video are explicitly aligned and fused before entering the LLM. We also included OmniVinci [2] to test a complementary design that aligns audio and video within temporal windows and retains separate modality tokens. We evaluated both models with attention knockout and token compression. The knockout results are presented in Appendix F of the revised manuscript, and the compression results are reported below.

Information flow across fusion designs. On AVUT and WorldSense, the layer-wise attention knockout results for OmniVinci and JavisGPT show the same ordering as in Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6. Modality-to-text attention has its strongest effect in middle layers, and text-to-final-token attention has its strongest effect in later layers (Appendix F, Figures 7–8). This ordering holds for OmniVinci, which aligns audio and video representations within temporal windows, and for JavisGPT, which fuses audio and video into unified SyncAV tokens. For MiniCPM-o-2.6 and the other evaluated backbones with separate audio and video tokens, blocking direct audio–video attention generally has a smaller effect on the probability assigned to the baseline-predicted answer token than blocking intra-modal attention. The depths and magnitudes of these effects vary across backbones.

Compression results on OmniVinci and JavisGPT. We evaluate both backbones at 25% and 10% layer-averaged token retention. For SyncAV tokens in JavisGPT, we use DARC’s video-anchor rule and set the token budget to the sum of the audio and video budgets. As shown in tables 1 and 2, our OmniRelay achieves the highest accuracy among the evaluated compression methods on all three benchmarks at both retention ratios on each backbone, supporting its applicability across different omni-modal architectures.

Table 1. Results on AVUT, WorldSense and DailyOmni with JavisGPT.

Method   AVUT (%) ↑   WorldSense (%) ↑   DailyOmni (%) ↑   Avg. relative perf. (%) ↑
Full Tokens (100%)50.6337.0447.79100.00
25% Retention
Random47.4035.4445.5394.86
FastV44.5834.6544.1991.36
VisionZip48.2735.9445.7896.05
SEATS49.0236.1946.1297.01
OmniRelay (Ours)50.0036.3547.2098.55
10% Retention
Random44.9835.0642.7791.00
FastV37.6031.8738.5180.30
VisionZip44.6933.8942.7789.75
SEATS46.1434.8445.0393.14
OmniRelay (Ours)46.6635.5645.6194.53

Table 2. Results on AVUT, WorldSense and DailyOmni with OmniVinci.

Method   AVUT (%) ↑   WorldSense (%) ↑   DailyOmni (%) ↑   Avg. relative perf. (%) ↑
Full Tokens (100%)64.9943.8264.66100.00
25% Retention
Random57.2141.5858.5691.16
VisionZip-om59.7540.7963.2494.28
SEATS63.2142.6264.0897.88
OmniFocus60.2139.7260.7492.41
OmniRelay (Ours)63.8442.7565.0898.81
10% Retention
Random52.0839.4750.7182.88
VisionZip-om49.7138.6254.3082.87
SEATS58.1340.2360.6591.68
OmniFocus51.4437.5854.9783.31
OmniRelay (Ours)58.7140.3261.0792.27

References

[1] JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation. NeurIPS 2025.

[2] OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM. ICLR 2026.


Thank you for taking the time to consider our response. We hope the clarifications and additional experiments are helpful, and we would sincerely appreciate your reconsideration of the overall rating.

Reviewer_pePeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes OmniRelay, a training-free pre- and within-LLM token compression framework for Omni-LLMs. Through layer-wise attention knockout on Qwen2.5-Omni-7B, the authors reveal a three-stage information relay: audiovisual tokens interact intra-modally in early layers, transfer information to text in middle layers, and relay from text to the final token in late layers. Guided by this insight, OmniRelay combines (1) pre-LLM Saliency-Diversity and Coverage-Aware Compression (SDCC) for local redundancy reduction and medoid-based inter-window video budget allocation, and (2) within-LLM Depth-Adaptive Relay Compression (DARC), which transitions from intra-modal anchor guidance to text guidance before dropping remaining audiovisual tokens at late layers. Across 3 Omni-LLMs and 5 benchmarks, OmniRelay retains 99.51% of full-token performance at 25% retention with 79% fewer FLOPs and a 3.3x prefill speedup.

Critical strengths
  1. Mechanistically Grounded Design: Layer-wise attention knockout (Sec. 2.2) convincingly shows that direct audio-video knockout (A <-> V) has negligible impact on accuracy, whereas intra-modal (intra-/inter-window) and modality-to-text pathways dominate sequentially across depth, strongly justifying DARC's depth-adaptive transition.
  2. Strong Empirical Performance: Across 3 Omni-LLMs and 5 benchmarks, OmniRelay consistently outperforms recent baselines (OmniZip, OmniFocus, SEATS), especially under aggressive 10%-15% retention ratios.
Critical weaknesses
  1. Insufficient Review of Recent Literature:
  • Zhu et al., "HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models"
  • Du et al., "Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention"
  • Ma et al., "ApET: Approximation-Error Guided Token Compression for Efficient VLMs"
  • Ju et al., "ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models Via Spatial-Temporal Forest Modeling"
  1. Missing Formulations in Main Text: Sec. 3.2 omits the explicit weighting formula combining intra-modal and text guidance (whose Gaussian parameters \mu_1, \mu_2, \sigma only appear in Appendix E, Table 12) and the exact layer schedule (l_s, l_t, l_e). Sec. 3.1 also lacks formal definitions for window grouping similarity and coverage gain discounting.
  2. Prefill Overhead & Knockout Generalization: In Table 3 (25% retention), OmniRelay's prefill latency (0.344s) is slightly slower than SEATS (0.305s) and OmniFocus (0.282s) despite similar FLOPs; please provide a latency breakdown of SDCC vs. DARC overhead. Also, does the knockout trend in Sec. 2.2 hold on MiniCPM-o-2.6?
Questions for authors
  1. Insufficient Review of Recent Literature.

  2. Missing Mathematical Formulations and Layer Schedules in Section 3: Sec. 3.2 qualitatively describes a weighted sum transitioning from intra-modal to text guidance, while Appendix E (Table 12) introduces Gaussian parameters (\mu_1, \mu_2) = (10, 15) and \sigma = 5. Could the authors provide: (a) the exact mathematical formula for combining the two normalized scores via (\mu_1, \mu_2, \sigma), (b) the specific layer indices (l_s, l_t, l_e) and layer-wise budget schedule K^{(l)} for each backbone, and (c) the explicit formulas for consecutive window grouping similarity and non-medoid coverage gain discounting in Sec. 3.1?

  3. Latency Breakdown of Algorithmic Overhead at 25% Retention: In Table 3 and Table 9. OmniRelay has nearly identical FLOPs to VisionZip-om, OmniFocus, and SEATS, yet its prefill latency is slower than VisionZip-om, OmniFocus, and SEATS. Could the authors provide a wall-clock latency breakdown quantifying the overhead of pre-LLM SDCC (pairwise distance + dynamic programming) versus within-LLM DARC?

  4. Generalizability of Attention Knockout to MiniCPM-o-2.6: The knockout analysis in Sec. 2.2 is conducted solely on Qwen2.5-Omni-7B (temporally interleaved with TMRoPE). Does the negligible impact of direct audio-video knockout (A <-> V) and the three-stage relay pattern also hold on MiniCPM-o-2.6, which uses time-division multiplexing and Resampler tokens (where the 2D spatial distance term had to be disabled in Appendix C)?

4 comments

Thank you for your detailed assessment and recognition of our motivation, method design, presentation and soundness. We hope the responses below could address your concerns.

W1 & Q1. Insufficient Review of Recent Literature.

A1. Thank you for suggesting these relevant references. They helped us broaden our discussion of related work and clarify how these studies relate to OmniRelay.

Zhu et al. [1] incorporate attention-head importance into visual token pruning. Du et al. [2] combine unified spatiotemporal compression with text-aware token merging. Ma et al. [3] guide token compression using linear approximation errors, and Ju et al. [4] model spatiotemporal token relationships using a forest structure for pruning. These studies offer complementary perspectives on visual token compression. Our OmniRelay addresses audiovisual token compression in omni-modal LLMs and uses layer-wise information-flow analysis to guide a gradual transition from intra-modal to instruction-text guidance. We have added these references and discussion in Appendix G (Additional Related Work) of the revised manuscript.

Reference

[1] Zhu et al. HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models.

[2] Du et al. Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention.

[3] Ma et al. ApET: Approximation-Error Guided Token Compression for Efficient VLMs.

[4] Ju et al. ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models Via Spatial-Temporal Forest Modeling.


W2. Missing Formulations in Main Text: Sec. 3.2 omits the explicit weighting formula combining intra-modal and text guidance (whose Gaussian parameters μ1,μ2,σ\mu_1, \mu_2, \sigma only appear in Appendix E, Table 12) and the exact layer schedule (ls,lt,le)(l_s, l_t, l_e). Sec. 3.1 also lacks formal definitions for window grouping similarity and coverage gain discounting.

Q2. Missing Mathematical Formulations and Layer Schedules in Section 3: Sec. 3.2 qualitatively describes a weighted sum transitioning from intra-modal to text guidance, while Appendix E (Table 12) introduces Gaussian parameters (μ1,μ2)=(10,15)(\mu_1, \mu_2) = (10, 15) and σ=5\sigma = 5. Could the authors provide: (a) the exact mathematical formula for combining the two normalized scores via (μ1,μ2,σ)(\mu_1, \mu_2, \sigma), (b) the specific layer indices (ls,lt,le)(l_s, l_t, l_e) and layer-wise budget schedule K(l)K^{(l)} for each backbone, and (c) the explicit formulas for consecutive window grouping similarity and non-medoid coverage gain discounting in Sec. 3.1?

A2. We sincerely appreciate your suggestion to improve the mathematical formulation of our method. We have updated Section 3.1 (Saliency–Diversity and Coverage-Aware Compression) and Section 3.2 (Depth-Adaptive Relay Compression) in the revised manuscript to make the overall procedure clearer and more precise.

  • (a) Mathematical formula for combining the two normalized scores. Let s~m,i(l)\tilde{s}_{m,i}^{(l)} and s~text,i(l)\tilde{s}_{\mathrm{text},i}^{(l)} denote the min–max normalized intra-modal and instruction-text-guided scores. We compute the combined score Sm,i(l)S_{m,i}^{(l)} as

    gj(l)=exp⁡(−(l−μj)22σ2),j∈1,2,αl=g2(l)g1(l)+g2(l),Sm,i(l)=(1−αl)s~m,i(l)+αls~text,i(l).g_j(l) = \exp\left(-\frac{(l - \mu_j)^2}{2\sigma^2}\right), \quad j \in \\{1,2\\}, \qquad \alpha_l = \frac{g_2(l)}{g_1(l) + g_2(l)}, \qquad S_{m,i}^{(l)} = (1 - \alpha_l)\tilde{s}_{m,i}^{(l)} + \alpha_l\tilde{s}_{\mathrm{text},i}^{(l)}.

    Here, μ1\mu_1 and μ2\mu_2 are the Gaussian centers for intra-modal and text guidance respectively and σ\sigma is the shared standard deviation. We have revised this in Section 3.2, Eqs. (10)–(11).

  • (b) Compression layer indices and token budgets. We have reported the compression boundaries (ls,le)(l_s, l_e) for each backbone in Table 10. The Layer-Wise Token Budgets paragraph specifies K(l)K^{(l)} including the logistic schedule for Qwen2.5-Omni-3B/7B and the power-law schedule for MiniCPM-o-2.6. We omit the explicit transition layer ltl_t because the layer-dependent weight αl\alpha_l already defines a smooth transition from intra-modal to text guidance. We have revised this in Appendix C.1.

  • (c) Formulas for grouping similarity and non-medoid coverage gain discounting. We have updated the similarity formula for grouping consecutive video windows and the formula for discounting the marginal coverage gain of non-medoid windows in Section 3.1, Eqs. (5) and (7).

W3a. Prefill Overhead & Knockout Generalization: In Table 3 (25% retention), OmniRelay's prefill latency (0.344s) is slightly slower than SEATS (0.305s) and OmniFocus (0.282s) despite similar FLOPs; please provide a latency breakdown of SDCC vs. DARC overhead.

Q3. Latency Breakdown of Algorithmic Overhead at 25% Retention: In Table 3 and Table 9. OmniRelay has nearly identical FLOPs to VisionZip-om, OmniFocus, and SEATS, yet its prefill latency is slower than VisionZip-om, OmniFocus, and SEATS. Could the authors provide a wall-clock latency breakdown quantifying the overhead of pre-LLM SDCC (pairwise distance + dynamic programming) versus within-LLM DARC?

A3. Thank you for your helpful comment. We now report the wall-clock latency breakdown of SDCC and DARC. On DailyOmni with Qwen2.5-Omni-7B at 25% retention, pre-LLM SDCC takes 140 ms and within-LLM DARC accounts for 62 ms (18.0%) of the 344 ms measured prefill latency. The tables below show the wall-clock time of each component.

Pre-LLM SDCC Latency Breakdown. On DailyOmni at 25% retention, SDCC takes 140 ms before LLM prefill. Intra-window selection and inter-window budget allocation account for 130 ms of this time.

OperationLatency
Intra-window S–D selection60 ms
Inter-window budget allocation70 ms
Other operations10 ms
Total SDCC140 ms

Within-LLM DARC Latency Breakdown. In the same setting, DARC takes 62 ms, accounting for 18.0% of the 344 ms measured prefill latency.

OperationLatency
Attention scoring25 ms
Score fusion and token selection14 ms
Other operations23 ms
Total DARC62 ms

Attention scoring, score fusion and token selection together account for 39 ms of the DARC overhead. This overhead helps explain why our OmniRelay's measured prefill latency (344 ms) is higher than that of the SEATS (305 ms) at similar reported FLOPs. We will focus on the strategies for reducing this overhead of DARC in the future.


W3b. Does the knockout trend in Sec. 2.2 hold on MiniCPM-o-2.6?

Q4. Generalizability of Attention Knockout to MiniCPM-o-2.6: The knockout analysis in Sec. 2.2 is conducted solely on Qwen2.5-Omni-7B (temporally interleaved with TMRoPE). Does the negligible impact of direct audio-video knockout (A <-> V) and the three-stage relay pattern also hold on MiniCPM-o-2.6, which uses time-division multiplexing and Resampler tokens (where the 2D spatial distance term had to be disabled in Appendix C)?

A4. Thank you for this important question. We have added layer-wise attention knockout results for MiniCPM-o-2.6 and other backbones on AVUT and WorldSense in Appendix F, Figures 7–8 of the revised manuscript.

Attention knockout results for MiniCPM-o-2.6. We use the same attention knockout protocol as for Qwen2.5-Omni-3B/7B. On MiniCPM-o-2.6, we observe (1) the same three-stage relay pattern and (2) a generally smaller effect when blocking direct audio–video attention than when blocking intra-modal attention. Besides, we also observe that the layers at which these effects peak and the magnitudes of the effects vary slightly across backbones. This variation motivates the use of backbone-specific Gaussian parameters for DARC's transition from intra-modal to text guidance different as reported in Appendix C.1, Table 10.

We look forward to your feedback and will be available for any of your inquiries at your earliest convenience.

Thank you for the clarifications! The rebuttal addresses my concerns well, and I will keep my original score, as it already reflects my positive assessment.

Thank you again for your positive assessment of our work. We are encouraged by your feedback and appreciate the time and care you devoted to reviewing our paper.

Reviewer_Di12OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OmniRelay, a training-free framework for audiovisual token compression in omni-modal LLMs. SDCC combines saliency–diversity selection with coverage-aware budget allocation before the LLM. DARC progressively shifts within-LLM selection from intra-modal anchor guidance toward text guidance, followed by complete removal of audiovisual tokens in late layers. Attention analysis and pathway knockout motivate the design, while experiments across multiple backbones and benchmarks evaluate the performance–computation trade-off.

Critical strengths
  1. The compression results are practically useful. The evaluation covers three model configurations and five audiovisual understanding benchmarks. On Qwen2.5-Omni-7B, the method preserves 99.51% of the average relative performance at 25% layer-averaged audiovisual token retention, demonstrating strong performance preservation.

  2. The information-flow analysis provides a clear motivation. The analysis distinguishes intra-modal, modality-to-text, and text-to-final-input-token pathways, and examines both intra-window and inter-window interactions. These observations establish a clear connection to the staged use of different guidance signals.

Critical weaknesses
  1. The novelty over SEATS is limited. SEATS already combines pre-LLM selection, progressively stronger pruning with depth, query-guided budget allocation, and complete late-layer modality removal. The added anchor guidance and coverage-aware allocation are useful refinements, but the manuscript does not sufficiently establish substantial new knowledge beyond the existing framework.
  2. The contribution of the central design is insufficiently isolated. Table 5 supports combining intra-modal and text guidance, but does not establish that the proposed depth ordering is essential. Without fixed-mixture and reversed-transition controls, the gains cannot be clearly attributed to information-flow-aligned scheduling rather than simply combining complementary signals.
  3. The scope and overhead accounting of the efficiency evaluation are insufficiently specified. Measured latency analysis primarily focuses on prefill on DailyOmni. The timing boundaries and cost breakdown for encoding, pre-LLM compression, and within-LLM scoring are unclear. End-to-end latency is also insufficiently evaluated, limiting assessment of the practical deployment benefits.
Questions for authors
  1. Discussion and comparison with OmniFit are missing. OmniFit (ICML 2026) also studies training-free layer-adaptive budget allocation and alignment-guided token selection. The distinction in analytical findings and methodological design should be clarified, with comparisons under compatible settings or an explanation of the comparison limitations.
  2. Controlled guidance-transition ablations are needed. Compare the proposed transition against a fixed mixture and a reversed transition while keeping per-layer budgets, pruning locations, text-query aggregation, and the final removal layer identical. This would establish the independent contribution of the depth ordering.
  3. The scope of the efficiency measurements should be clarified. Specify the operations included in prefill latency, report the main compression and scoring overheads, and provide end-to-end latency or clearly delimit the scope of the existing measurements.

[1] Wang et al. OmniFit: Bridging Modalities via Layer-Adaptive Token Compression for Omnimodal Large Language Models. ICML 2026.

3 comments

We sincerely thank the reviewer for the thoughtful and encouraging feedback on our method's motivation, performance and paper presentation. We appreciate the opportunity to further clarify your concerned questions.

W1. The novelty over SEATS is limited. SEATS already combines pre-LLM selection, progressively stronger pruning with depth, query-guided budget allocation, and complete late-layer modality removal. The added anchor guidance and coverage-aware allocation are useful refinements, but the manuscript does not sufficiently establish substantial new knowledge beyond the existing framework.

Q1. Discussion and comparison with OmniFit are missing. OmniFit (ICML 2026) also studies training-free layer-adaptive budget allocation and alignment-guided token selection. The distinction in analytical findings and methodological design should be clarified, with comparisons under compatible settings or an explanation of the comparison limitations.

A1. We thank the reviewer for asking us to clarify the contribution of our work and how it differs from SEATS [1] and OmniFit [2]. Our work is distinct from an incremental extension of SEATS. Our main novelty lies in identifying a depth-dependent pattern of audiovisual information flow and designing a compression strategy around that observation. Specifically, we observe a shift from intra-modal interactions in early layers to audiovisual-to-text information transfer in middle layers. This shift directly motivates both the choice of guidance signals in DARC and the order in which we use them across layers. Below, we explain how OmniRelay differs from SEATS and OmniFit in its motivating analysis and method design.

Motivation analysis.

  1. Both SEATS and OmniFit observe that the effect of compression varies across layers, while leaving the underlying information flow unexplained. Through attention knockout, we identify a depth-adaptive information relay within the LLM that audiovisual information is primarily propagated through intra-modal interactions in early layers, progressively transferred to text in middle layers, and finally relayed to the final token. This observation informs which signals should guide compression at different depths. For instance, this observation indicates that the final input token which SEATS uses as a query is unsuitable for scoring audiovisual tokens directly because it primarily aggregates information from text tokens. Following the information flow described above, we therefore use text tokens as guidance. The ablation results below also support this choice.

    Query tokens for text guidance   AVUT (%) ↑   WorldSense (%) ↑   DailyOmni (%) ↑   Avg. relative perf. (%) ↑
    Final input token64.0145.7159.3197.05
    Text tokens (Ours)64.6546.0060.9998.48
  2. OmniFit finds inter-modal guidance more effective than intra-modal guidance, and SEATS assumes that cross-modal fusion occurs in early layers. However, we observe that knocking out direct audio–video attention has little effect on accuracy, whereas knocking out intra-modal attention has a much larger effect across various models and benchmarks (Figures 3, 7 and 8) . These results suggest that information transfer in early layers relies much more heavily on intra-modal while little on inter-modal interactions.

Method design.

  1. Although SEATS, OmniFit and our OmniRelay all adopt layer-adaptive budget allocation within the LLM, we progressively shift from intra-modal guidance to text guidance to follow the observed depth-adaptive information relay. Intra-modal guidance removes global redundancy across windows, and text guidance helps retain task-relevant information.

  2. Our observation that intra-modal attention has a much larger effect than direct audio–video attention motivates us to use intra-modal anchors to select important tokens, instead of the cross-modal signals used in OmniFit.

Besides, we evaluate OmniFit, SEATS and our OmniRelay on Qwen2.5-Omni-3B/7B. As the official implementation of OmniFit is unavailable up to now, we used our reimplementation.

Table 1: Results on AVUT, WorldSense and DailyOmni with Qwen2.5-Omni-7B.

Method   AVUT (%) ↑   WorldSense (%) ↑   DailyOmni (%) ↑   Avg. relative perf. (%) ↑
Full Tokens64.7646.6362.91100.00
25% Retention
OmniFit62.2844.7756.8194.16
SEATS63.0945.3059.8296.55
OmniRelay (Ours)64.6546.0060.9998.48
15% Retention
OmniFit58.1344.1455.7291.00
SEATS59.7544.3357.1492.72
OmniRelay (Ours)60.8445.6859.4895.49
10% Retention
OmniFit57.3843.4154.3089.34
SEATS58.6543.0356.0690.65
OmniRelay (Ours)59.8644.2357.3192.80

Table 2: Results on AVUT, WorldSense and DailyOmni with Qwen2.5-Omni-3B.

Method   AVUT (%) ↑   WorldSense (%) ↑   DailyOmni (%) ↑   Avg. relative perf. (%) ↑
Full Tokens61.8846.4461.24100.00
25% Retention
OmniFit58.8244.1452.8892.15
OmniPack58.7145.2154.3093.63
SEATS————
OmniRelay (Ours)60.6145.4958.9097.36
15% Retention
OmniFit53.6342.8151.8887.86
OmniPack57.7943.6653.1391.39
SEATS————
OmniRelay (Ours)58.7144.5857.3994.86
10% Retention
OmniFit55.5441.0550.1386.67
OmniPack56.6942.0251.7188.84
SEATS————
OmniRelay (Ours)57.8443.6956.7393.40

As shown in the table 1 and 2 above, OmniRelay achieves the highest accuracy among the evaluated compression methods on all three benchmarks at every tested retention ratio, further demonstrating the effectiveness of our approach.

References

[1] SEATS: Stage-adaptive Token Selection for Efficient Omni-modal LLMs. arXiv:2605.20035.

[2] OmniFit: Bridging Modalities via Layer-Adaptive Token Compression for Omnimodal Large Language Models. ICML 2026.


W2. The contribution of the central design is insufficiently isolated. Table 5 supports combining intra-modal and text guidance, but does not establish that the proposed depth ordering is essential. Without fixed-mixture and reversed-transition controls, the gains cannot be clearly attributed to information-flow-aligned scheduling rather than simply combining complementary signals.

Q2. Controlled guidance-transition ablations are needed. Compare the proposed transition against a fixed mixture and a reversed transition while keeping per-layer budgets, pruning locations, text-query aggregation, and the final removal layer identical. This would establish the independent contribution of the depth ordering.

A2. Thank you for suggesting these ablations. We have compared our forward transition from our intra-modal to text guidance with three fixed mixtures and the reversed smooth transition. We vary only the text-guidance weight schedule αl\alpha_l while keeping the per-layer token budgets, pruning locations, text-query aggregation and final removal layer identical.

Table 3. Ablation of guidance schedules on Qwen2.5-Omni-7B.

Guidance scheduleAVUT (%) ↑\uparrowWorldSense (%) ↑\uparrowDailyOmni (%) ↑\uparrowAvg. relative perf. (%) ↑\uparrow
Ours64.6546.0060.9998.48
Fixed mixture (αl=0.25\alpha_l=0.25)63.7845.6260.5797.53
Fixed mixture (αl=0.50\alpha_l=0.50)63.9645.7160.2397.51
Fixed mixture (αl=0.75\alpha_l=0.75)64.3645.7160.4897.85
Reversed transition64.0744.8660.3297.01

As shown in the table 3 above, Our forward transition achieves 98.48% average relative performance, outperforming the best fixed mixture αl=0.75\alpha_l=0.75 by 0.63% and the reversed smooth transition by 1.47%. It also achieves the highest accuracy on all three benchmarks in this guidance-schedule ablation, supporting the effectiveness of the proposed guidance ordering.

W3. The scope and overhead accounting of the efficiency evaluation are insufficiently specified. Measured latency analysis primarily focuses on prefill on DailyOmni. The timing boundaries and cost breakdown for encoding, pre-LLM compression, and within-LLM scoring are unclear. End-to-end latency is also insufficiently evaluated, limiting assessment of the practical deployment benefits.

Q3. The scope of the efficiency measurements should be clarified. Specify the operations included in prefill latency, report the main compression and scoring overheads, and provide end-to-end latency or clearly delimit the scope of the existing measurements.

A3. Thank you for pointing out that the timing boundaries and compression overheads needed clarification. We illustrate them from end-to-end latency and cost breakdown of SDCC and DARC.

End-to-end latency. We first describe the timing boundaries and report the latency of each stage in order.

  1. Encoding time measures the time spent encoding the inputs into tokens.
  2. Pre-LLM compression time measure the time SDCC takes to compress the encoded tokens before they enter the LLM.
  3. Prefill time measures the time from when the LLM receives the compressed tokens to when it generates the first output token. This includes the time spent on within-LLM compression (DARC) in the middle layers.
  4. Decoding time measures the time taken to generate the remaining answer tokens.

We report end-to-end latency and its breakdown by stage on DailyOmni with Qwen2.5-Omni-7B. As shown in the table, audiovisual encoding takes 0.627 s in all configurations because token compression occurs after encoding. It accounts for 28.8% of end-to-end latency with full tokens and 42.0–49.6% with OmniRelay. As the retention ratio decreases, shorter sequences substantially reduce prefill latency by lowering attention costs. Decoding time also decreases from 0.410s to 0.360s at 10% retention. The savings in prefill and decoding more than offset the additional cost of pre-LLM compression, yielding 1.46×–1.72× end-to-end speedups over full-token inference.

MethodRetentionEncodingPre-LLM compression (SDCC)LLM prefill (incl. DARC)DecodingEnd-to-end latency
Full Tokens100%0.6270.0001.1410.4102.178
Ours25%0.6270.1400.3440.3831.494
Ours15%0.6270.1130.2350.3721.347
Ours10%0.6270.0960.1820.3601.265

Latency breakdown of SDCC and DARC. We also report the latency and breakdown of SDCC and DARC on DailyOmni with Qwen2.5-Omni-7B at 25% retention.

  1. As shown below, SDCC perform compression between the encoding and prefilling by 140 ms overhead. Intra-window selection and inter-window budget allocation account for the most of the time.

    OperationLatency
    Intra-window S–D selection60 ms
    Inter-window budget allocation70 ms
    Other operations10 ms
    Total SDCC140 ms
  2. As shown below, DARC takes 62 ms during LLM prefill, accounting for 18.0% of the total prefill latency of 344 ms. DARC reduces prefill latency from 1,141 ms to 344 ms relative to the full-token inference, achieving a substantial time saving of 797 ms with little overhead. Attention scoring accounts for 25 ms of DARC's runtime, while score fusion, token selection, and other operations account for the remaining 37 ms.

    OperationLatency
    Attention scoring25 ms
    Score fusion and token selection14 ms
    Other operations23 ms
    Total DARC62 ms

We sincerely thank you for your thoughtful feedback and welcome any following questions. We appreciate your guidance and will improve the manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Rebuttal Summary

Dear Reviewers and Area Chairs,

We sincerely express our gratitude to the reviewers for their time and thoughtful feedback. As the discussion period concludes, we would like to summarize our discusssion with the reviewers as below:


Strengths

  • The information-flow analysis provides an empirical motivation for depth-dependent compression. [pePe, ViRH, Di12, aqD7]
  • Experiments across multiple backbones and benchmarks demonstrate favorable performance–efficiency trade-offs. [pePe, ViRH, Di12, aqD7]
  • Ablations support the complementary roles of SDCC and DARC. [ViRH, aqD7]

Responses to Main Concerns

  • We clarified differences from prior work, added baseline comparisons and validated the guidance ordering through controlled same-budget ablations. [Di12, aqD7]
  • We clarified the code revision corresponding to the reported results, provided design ablations and added knockout and compression experiments on JavisGPT and OmniVinci. [ViRH]
  • We improved the formulations, specified backbone-dependent settings, and reported compression overheads and end-to-end latency. At closely matched prefill latency, OmniRelay achieves 60.15% accuracy in 301 ms versus SEATS’s 59.82% in 305 ms. [pePe, Di12, aqD7]

Discussion Outcome

Reviewer pePe confirmed their concerns were addressed and maintained their positive rating. Reviewer ViRH confirmed their concern was resolved and stated they would raise their score. Reviewers Di12 and aqD7 have not yet provided follow-up feedback.


We thank the AC and all reviewers again for their collective efforts throughout the review and discussion process.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.