acceptodds
Under review as a conference paper at ICLR 2027

MoCo: Modality-Aware Decoder-Side Token Compression for Video–Audio OmniLLMs

Abstract

Long audio–video inputs make inference costly for omnimodal large language models (OmniLLMs), motivating token compression that preserves question-relevant evidence. Question-conditioned attention offers a natural signal for selecting which tokens to retain. However, we find that shallow question-conditioned attention favors audio even for visual questions, causing direct cross-modal ranking to allocate less capacity to visual evidence. Allocating audio and video budgets separately improves accuracy without changing token scores or within-modality rankings. Based on these findings, we propose MoCo, a training-free framework for decoder-side token compression that separates modality-budget allocation from question-conditioned token selection. It assigns audio and video shares of a joint token budget, then uses question-conditioned attention to rank tokens within each modality. Within a single prefill pass, MoCo prunes audio early to reduce computation and delays video selection to exploit more informative attention scores at later shallow layers. Across four OmniLLMs, MoCo achieves higher average performance than OmniZip at all three evaluated retention ratios. MoCo preserves 98.7% and 97.3% of full-token performance on Qwen2.5-Omni-7B and 3B, respectively, while retaining only 25% of multimodal tokens. At matched FLOPs, MoCo preserves up to 3.1 percentage points more of full-token performance than OmniZip on Qwen2.5-Omni-7B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
OmniLLMs, Efficient Inference, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zpzDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper finds that shallow question-conditioned attention favors audio even for visual questions. MoCo therefore separates audio-video budget allocation from token ranking within each modality, and prunes audio and video at different shallow decoder layers.

Critical strengths

(1) The paper identifies a concrete failure mode of shared cross-modal top-K selection. Its controlled budget-allocation comparison keeps the scores, total budget, and pruning step fixed. (2) Results across several models, together with matched-FLOPs comparisons, provide useful evidence for the proposed design.

Critical weaknesses

(1) OmniFocus and OmniScope also estimate importance or allocate capacity separately by modality. MoCo's decoder-layer readouts and staged pruning are distinct, but direct comparisons under the same model, budget, and compute would better establish their added value. (2) At 25% token retention, MoCo uses more estimated FLOPs than OmniZip. The matched-FLOPs results help, but measured prefill latency and peak memory, including scoring and pruning overhead, are still needed to assess deployment efficiency.

Questions for authors

Could the authors provide matched-condition comparisons with recent modality-balanced methods and report measured latency and memory across compression levels?

2 comments
AuthorsOpenReview ↗

RESPONSE TO REVIEWER zpzD

Thank you very much for your positive assessment and helpful suggestions. We are glad the reviewer found that the paper identifies a concrete failure mode of shared cross-modal Top-K selection. We address each point below.

W1: Comparison with OmniFocus and OmniScope

Q: OmniFocus and OmniScope also estimate importance or allocate capacity separately by modality. MoCo's decoder-layer readouts and staged pruning are distinct, but direct comparisons under the same model, budget, and compute would better establish their added value.

A: Thank you for pointing us to these works. OmniFocus (Cao et al., arXiv:2607.03050, July 2026) and OmniScope (Su et al., arXiv:2607.23193, July 2026) are concurrent works. Both compress tokens before the LLM: OmniFocus allocates the budget over temporal chunks by query similarity and then selects tokens by inter-modal association and intra-modal saliency, while OmniScope scores video frames with an external CLIP model and audio tokens by query similarity. MoCo is the only one of the three that selects tokens inside the decoder, after audio-video fusion, from question-conditioned attention at modality-specific depths (L3 for audio, L5 for video). Token survival is therefore decided by the decoder's own attention after the question has interacted with both modalities, with no external scorer and no pre-LLM similarity heuristic. The paper already includes matched-retention comparisons with DASH, a recent OmniLLM compression method (Table 8): MoCo is higher in five of the six comparisons, by up to 3.13 points. We plan to comprehensively compare MoCo with OmniFocus, OmniScope and other concurrent methods in future work. However, as these are concurrent works, this comparison is beyond the scope of this rebuttal.

W2: FLOPs at 25% retention; measured prefill latency and peak memory including scoring and pruning overhead

Q: At 25% token retention, MoCo uses more estimated FLOPs than OmniZip. The matched-FLOPs results help, but measured prefill latency and peak memory, including scoring and pruning overhead, are still needed to assess deployment efficiency.

A: Following your suggestion, we measured prefill latency and peak memory, including scoring and pruning overhead. Since the two methods differ in accuracy at equal retention, we compare them at matched accuracy (Table 5). Setup: Qwen2.5-Omni-7B on WorldSense, on a single A100 (40 GB) GPU, FlashAttention-2.

Matched accuracyMethod (retention)Prefill (ms)TTFT (ms)Peak memory, max
Full Tokens1,475 (1x)3,384 (1x)37.9 GiB
~95%MoCo (15%)459 (3.21x)2,191 (1.54x)26.4 GiB
OmniZip (36.7%)480 (3.07x)2,542 (1.33x)28.3 GiB
~96.5%MoCo (20%)509 (2.90x)2,224 (1.52x)26.4 GiB
OmniZip (48.1%)638 (2.31x)2,682 (1.26x)30.4 GiB
~98%MoCo (25%)569 (2.59x)2,329 (1.45x)26.6 GiB
OmniZip (55.8%)750 (1.97x)2,830 (1.20x)31.7 GiB

At every matched-accuracy level, MoCo needs less than half the tokens of OmniZip and has the larger measured prefill and TTFT speedups and the lower peak memory. The pruning steps themselves take only about 4% of prefill.

Q1: Latency and memory across compression levels

Q: Could the authors provide matched-condition comparisons with recent modality-balanced methods and report measured latency and memory across compression levels?

A: The setup is the same as in W2.

RetentionFull Tokens35%25%20%15%10%
Prefill (ms)1,475683569509459414
Prefill speedup1x2.16x2.59x2.90x3.21x3.56x
TTFT (ms)3,3842,3852,3292,2242,1912,129
TTFT speedup1x1.42x1.45x1.52x1.54x1.59x
Peak memory, max (GiB)37.926.826.626.426.426.4

MoCo gives a 2.2x-3.6x measured prefill speedup and a 1.4x-1.6x TTFT speedup across retention levels, and lowers peak memory from 37.9 GiB to below 27 GiB. Peak memory stays low at every retention level because it is set in the early decoder layers, before video tokens are pruned.

We hope the measured results address your concerns, and we would be grateful if you would consider raising your score. Please let us know if you have any further questions. Thank you again for your support.

AuthorsOpenReview ↗

A Gentle Reminder Before the Discussion Period Ends

Dear Reviewer zpzD,

We hope all is well with you. We would like to gently follow up on our responses to your comments and the additional experiments we have provided. Since the discussion period will end in less than 24 hours, we would greatly appreciate it if you could take a moment to look over our responses and let us know whether any questions or concerns remain, should your schedule permit. We are more than happy to clarify anything further.

We fully understand how busy your schedule may be, and we are truly grateful for your time and consideration. Thank you again for your attention to our work!

Reviewer_bYNaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MoCo, a training-free token compression framework for audio-video OmniLLMs. Motivated by the observation that shallow attention favors audio even for visual questions, MoCo separates modality budget allocation from within-modality token selection, using fixed per-modality budgets and attention-based pruning at different decoder layers. Experiments on four OmniLLM backbones and multiple benchmarks show better accuracy preservation than existing compression baselines.

Critical strengths
  1. The paper identifies an interesting phenomenon that shallow question-conditioned attention tends to favor audio tokens even for visually grounded questions. This observation provides useful insights.

  2. The method is evaluated across multiple OmniLLM backbones, multiple benchmarks, and different token retention ratios. The authors also provide comparisons under matched token retention and estimated FLOPs.

Critical weaknesses
  1. The paper presents an interesting observation that shallow decoder attention can exhibit a strong audio preference even for visually grounded questions. However, the proposed solution primarily relies on a fixed modality-budget split selected on a small development set rather than an adaptive allocation strategy. Since the observed attention bias is most pronounced in shallow layers and may change at greater depths, it remains unclear whether a static budget split is the most appropriate solution. Moreover, different questions and model architectures may have substantially different modality requirements.
  2. The efficiency evaluation relies on analytic decoder FLOPs, without wall-clock latency, memory, or throughput measurements. This is particularly concerning because MoCo’s token selection depends on extracting decoder attention scores, which may conflict with the design of fused kernels such as FlashAttention that avoid materializing attention maps.
Questions for authors

You use shallow-layer question-conditioned attention for scoring, but the conclusion that audio selection is insensitive to depth is only an average trend. Are there particular question types (e.g., those involving vocal tone, music, or ambient sound cues) for which shallow-layer ranking is poor and audio tokens become valuable only at deeper layers?

3 comments
AuthorsOpenReview ↗

RESPONSE TO REVIEWER bYNa (Part 1/2)

Thank you very much for your constructive comments. We are glad the reviewer found the observation on shallow audio preference insightful. We have tried to address every point carefully.

W1: Separate modality budgets instead of an adaptive allocation

Q: The paper presents an interesting observation that shallow decoder attention can exhibit a strong audio preference even for visually grounded questions. However, the proposed solution primarily relies on a fixed modality-budget split selected on a small development set rather than an adaptive allocation strategy. Since the observed attention bias is most pronounced in shallow layers and may change at greater depths, it remains unclear whether a static budget split is the most appropriate solution. Moreover, different questions and model architectures may have substantially different modality requirements.

A: Thank you for raising this point. MoCo separates audio-video budget allocation from question-conditioned token selection: which tokens are kept within each modality is still decided per sample. An adaptive allocation needs a per-sample signal, and the natural signal is the attention score itself. This is what shared Top-K does, and Obs. 1 shows that this signal favors audio regardless of the question: with the same scores, separate modality budgets improve accuracy over shared Top-K by 2.11 points on 6,103 examples, with gains on all three benchmarks (Table 4a). An allocation read from attention must also be read before pruning: on Qwen2.5-Omni-7B, MoCo prunes by L5 of the 28 decoder layers, and reading it at greater depths would keep all tokens through those layers and give up most of the compute saving. The audio preference appears in the shallow layers of all four models (first table in our response to Reviewer x4Qh, W1.1), so an allocation read from shallow attention would inherit the same bias. The same audio budget share is reused across benchmarks, question types and retention ratios, and on Qwen3-Omni and OmniVinci-9B (Sec. 5.2, Appendix B). A single share therefore transfers across models and benchmarks without any per-benchmark or per-sample tuning. Performance is also stable around the default share: for an audio budget share between 0.20 and 0.40, normalized performance stays between 97.9% and 98.8% (Table 10). Per-sample adaptive allocation is a promising direction that we leave for future work.

W2.1: Compatibility of attention-score extraction with FlashAttention

Q: This is particularly concerning because MoCo's token selection depends on extracting decoder attention scores, which may conflict with the design of fused kernels such as FlashAttention that avoid materializing attention maps.

A: MoCo is compatible with FlashAttention: all results in the paper and in this response were obtained with FlashAttention-2 enabled. MoCo does not need the full attention map; it only needs the attention rows of the question tokens at the two scoring layers. These few rows are computed separately, and the full sequence still goes through the fused kernel. The pruning steps themselves take only about 4% of prefill. We apologize for any confusion on this point.

(Continued in Part 2/2.)

AuthorsOpenReview ↗

RESPONSE TO REVIEWER bYNa (Part 2/2)

W2.2: Wall-clock time, memory, and throughput

Q: The efficiency evaluation relies on analytic decoder FLOPs, without wall-clock latency, memory, or throughput measurements.

A: Following your suggestion, we measured Qwen2.5-Omni-7B on WorldSense on a single A100 (40 GB) GPU, batch size 1, FlashAttention-2. Since the methods differ in accuracy at a given retention, we compare them at matched accuracy (Norm. Avg. over three benchmarks, Table 5). Speedups relative to full-token inference are given in parentheses.

Matched accuracyMethod (retention)Prefill (ms)TTFT (ms)Throughput (requests/s)Peak memory, max
Full Tokens1,475 (1x)3,384 (1x)0.1937.9 GiB
~95%MoCo (15%)459 (3.21x)2,191 (1.54x)0.2826.4 GiB
OmniZip (36.7%)480 (3.07x)2,542 (1.33x)0.2628.3 GiB
~96.5%MoCo (20%)509 (2.90x)2,224 (1.52x)0.2826.4 GiB
OmniZip (48.1%)638 (2.31x)2,682 (1.26x)0.2530.4 GiB
~98%MoCo (25%)569 (2.59x)2,329 (1.45x)0.2726.6 GiB
OmniZip (55.8%)750 (1.97x)2,830 (1.20x)0.2431.7 GiB

At every matched-accuracy level, MoCo needs less than half the tokens of OmniZip (15% vs 36.7%, 20% vs 48.1%, 25% vs 55.8%) and has the larger prefill speedup (2.59x vs 1.97x at ~98%), the larger TTFT speedup (1.45x vs 1.20x), the higher throughput (0.27 vs 0.24 requests/s) and the lower peak memory (26.6 vs 31.7 GiB).

Q1: Question types where shallow-layer audio ranking is poor

Q: You use shallow-layer question-conditioned attention for scoring, but the conclusion that audio selection is insensitive to depth is only an average trend. Are there particular question types (e.g., those involving vocal tone, music, or ambient sound cues) for which shallow-layer ranking is poor and audio tokens become valuable only at deeper layers?

A: Thank you for this insightful question. We broke down the full WorldSense grid (3,172 questions, Qwen2.5-Omni-7B) by the benchmark's audio-type and task-type labels, and compared accuracy (%) when audio is scored at L3 and at L5, averaged over the five video scoring layers, with the 95% CI of the difference:

Question groupQuestionsAudio scored at L3Audio scored at L5Difference
Audio: speech only29342.3942.59+0.20 [-0.68, 1.02]
Audio: music only28146.8347.05+0.21 [-0.57, 1.07]
Audio contains ambient events2,08542.9342.91-0.02 [-0.48, 0.43]
Audio-centric tasks (recognition, counting, change, source localization)40944.6044.40-0.20 [-1.17, 0.78]
Emotion tasks24251.9052.40+0.50 [-0.25, 1.40]

No audio type or task type benefits significantly from deeper audio scoring. This is consistent with Obs. 3: audio selection is less sensitive to scoring depth.

We hope the measured results and the FlashAttention clarification address your concerns, and we would appreciate it if you would reconsider your rating. Thank you again for your time.

AuthorsOpenReview ↗

、A Gentle Reminder Before the Discussion Period Ends

Dear Reviewer bYNa,

We hope this message finds you well. We are writing to kindly follow up on our responses to your review and the additional experiments we have provided. With less than 24 hours remaining in the discussion period, we would be very grateful if you could take a moment to read our responses and share any remaining questions or concerns, should your schedule permit. We would be glad to provide any further clarification.

We understand that this is a busy time for you, and we sincerely appreciate your time and consideration. Thank you again for your attention to our work!

Reviewer_x4QhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MoCo, a training-free decoder-side token compression framework for video–audio OmniLLMs. Building upon OmniZip, MoCo introduces a series of modifications aimed at addressing specific "bad cases." At its core, the method still relies on question-conditioned top-k token selection to achieve compression. One key insight is that audio attention weights typically exceed those of video; consequently, rather than performing a joint top-k selection, the authors propose allocating separate budgets for audio and video tokens. Furthermore, they filter these tokens across different network depths based on their respective sensitivity to audio or video attention, and incorporate a "temporal coverage bonus" to prevent the excessive loss of information that might otherwise occur if only a single video chunk were retained.

While these design choices are intuitively sound and well-motivated, the supporting experimental analysis is not particularly robust. I do not believe this work offers any robust, generalizable insights. The performance gains appear to stem from hyperparameter tuning rather than a cohesive, principled methodology. If the authors could provide additional experiments—such as a solid mechanistic analysis—to contribute new knowledge to the community, and subsequently derive a method from this analysis, I would consider raising my score.

Critical strengths
  1. Relevant problem, competent engineering. Long audio–video inputs are genuinely expensive for OmniLLMs, and a training-free, single-prefill method has practical appeal. The reported numbers on Qwen2.5-Omni-7B (25% tokens → 98.7% of full-token performance) are strong on their face.
  2. Clear narrative. The "three observations → three fixes" structure makes the motivation legible, with each component tied to a stated failure mode.
  3. Obs. 2 is a genuinely well-designed controlled experiment. Holding scores, candidates, budget, and pruning step fixed while only varying budget separation cleanly isolates a +2.11 pp effect. Reframing "audio attention is higher" as a potential bias rather than a signal is a reasonable perspective, and has some value independent of whether the proposed fix generalizes.
Critical weaknesses

W1. Diagnostics are narrow and do not support generalizable knowledge (primary concern) "Shallow audio preference" may well be a Qwen–dataset-specific artifact rather than a universal property. "The method works on four backbones" is not the same as "the knowledge holds on four backbones" — and the paper conflates the two. The diagnostic samples are also thin and homogeneous: Obs. 1 uses 100 paired clips and 30 prompts; Obs. 3 uses 100 examples per benchmark per grid cell; the evidence is insufficiently solid. I recommend validating the core analysis on all four models using the full multi-dataset suite, so as to provide the community with solid knowledge and a solid basis for the method, and to dispel my concern that the performance gains stem merely from tuning choices such as layer selection. W2. Staged pruning is a grid-search choice post-hoc rationalized The choice (ℓa,ℓv)=(3,5)(\ell_a, \ell_v) = (3, 5) rests on Obs. 3, but no mechanism is offered for why video is depth-sensitive and audio is not; a 5×5 sensitivity grid is equally consistent with "video needs later layers" and "L3/L5 is the argmax of a noisy grid." The "plateau" is not quantified — no significance test against neighboring cells (L2/L5, L3/L4). The missing control ablation is also critical: Table 4b compares single-layer vs. staged readout but omits the control where audio is also pruned at L5 with separate budgets. Without it, the +2.47 pp gain cannot be cleanly attributed to staging rather than to early audio pruning freeing context/compute for video scoring. The "early audio pruning saves FLOPs" argument is also circular: any depth-insensitive modality would benefit from early pruning, so this does not require Obs. 3 as knowledge. W3. Minor Benchmark coverage is inconsistent across tables (some exclude Video-MME).

Questions for authors

see weakness,if w1 is resolved, I will consider increasing the rating.

5 comments
AuthorsOpenReview ↗

RESPONSE TO REVIEWER x4Qh (Part 1/4)

Thank you very much for your careful and insightful review. We are glad the reviewer found Obs. 2 a well-designed controlled experiment and the narrative clear. We have tried to address every point carefully.

W1: Diagnostics are narrow and do not support generalizable knowledge (primary concern)

A: Following your recommendation, we re-measured both observations on all four models with the full four-benchmark suite (8,803 questions per model). We answer the four points of W1 in turn.

W1.1

Q: "Shallow audio preference" may well be a Qwen–dataset-specific artifact rather than a universal property.

A: The shallow audio preference is not specific to Qwen2.5-Omni or to one dataset: on the full benchmarks (all 8,803 questions per model), it appears on all four models and all four benchmarks. For each of decoder layers 1 to 5 (L1 to L5), the table gives the percentage of questions with higher per-token attention to audio than video:

ModelBenchmarkL1L2L3L4L5
Qwen2.5-Omni-7BAVUT100100100100100
DailyOmni100100100100100
WorldSense100100100100100
Video-MME10010010010096
Qwen2.5-Omni-3BAVUT100100100100100
DailyOmni100100100100100
WorldSense100100100100100
Video-MME10010010097100
Qwen3-Omni-30B-A3BAVUT100100100100100
DailyOmni100100100100100
WorldSense100100100100100
Video-MME6810010010099
OmniVinci-9BAVUT100100889442
DailyOmni100100919452
WorldSense100100767622
Video-MME1009428242

Three findings follow.

  1. The preference appears in all 16 model-benchmark pairs at L1 and L2. On Qwen2.5-Omni-7B it holds for every one of the 8,803 questions from L1 to L4.
  2. It persists through L5 on Qwen2.5-Omni-7B, 3B and Qwen3-Omni, and is still present on OmniVinci-9B, where it is strongest at L1 and L2 and remains the majority on three benchmarks through L4.
  3. The preference persists across question types on all four models. With paired audio and visual questions on 3,939 clips, as in Fig. 1(a), the two question types differ by at most 4.1 points at every layer from L1 to L5, on every model and benchmark. At L1 and L2, attention favors audio even under visual questions on all four models, as stated in Obs. 1, which is why MoCo sets the audio and video budgets separately from the attention ranking.

The query-key geometry of Fig. 1 holds as well: at layer 2, at least 99% of query positions align more closely with the audio-key centroid than with the video-key centroid (positive cosine gap) on all four models and all four benchmarks (all 16 model-benchmark pairs).

(Continued in Part 2/4.)

AuthorsOpenReview ↗

RESPONSE TO REVIEWER x4Qh (Part 2/4)

W1.2

Q: "The method works on four backbones" is not the same as "the knowledge holds on four backbones" — and the paper conflates the two.

A: The knowledge, not only the method, is verified on each of the four backbones. The table below reports the observation itself on each backbone, without using accuracy (column 1: all 8,803 benchmark questions; columns 2 and 3: the 3,939 paired clips, taking the largest difference and the lowest percentage over the four benchmarks):

BackboneQuestions favoring audio at L2 (%)Largest difference between audio and visual questions, L1 to L5 (points)Query positions aligned more closely with the audio-key centroid at L2 (%)
Qwen2.5-Omni-7B100.00.4100.0
Qwen2.5-Omni-3B100.00.399.9
Qwen3-Omni-30B-A3B99.90.399.8
OmniVinci-9B98.14.199.6

Obs. 3 is reported for each backbone in W1.3. The four backbones are Qwen2.5-Omni-7B and 3B, Qwen3-Omni-30B-A3B, whose audio encoder is trained from scratch, and OmniVinci-9B, which NVIDIA trained independently with different audio and vision encoders. Qwen2.5-Omni is the most widely used testbed for OmniLLM token compression, and most related studies evaluate on it alone.

W1.3

Q: The diagnostic samples are also thin and homogeneous: Obs. 1 uses 100 paired clips and 30 prompts; Obs. 3 uses 100 examples per benchmark per grid cell; the evidence is insufficiently solid. I recommend validating the core analysis on all four models using the full multi-dataset suite, so as to provide the community with solid knowledge and a solid basis for the method, ...

A: Both observations are now measured on the full benchmarks, on all four models: 8,803 of the benchmarks' own questions per model, from four benchmarks, for Obs. 1, together with 3,939 paired clips for the comparison of audio and visual questions, and 8,803 questions per model on all 25 cells of the scoring-layer grid for Obs. 3.

Obs. 3 also holds at full scale: video selection is more sensitive to scoring depth than audio selection in 13 of the 16 model-benchmark pairs, including all eight on Qwen2.5-Omni. In the remaining three, video accuracy varies by less than 1 point across scoring layers. Accuracy range (max - min, points) across scoring layers, as in Fig. 3(a) (bold: video is the more sensitive modality):

ModelBenchmarkVideo scoring layerAudio scoring layer
Qwen2.5-Omni-7BAVUT3.212.40
DailyOmni2.631.20
WorldSense1.821.38
Video-MME1.640.40
Qwen2.5-Omni-3BAVUT3.622.81
DailyOmni2.091.94
WorldSense2.411.35
Video-MME2.070.42
Qwen3-Omni-30B-A3BAVUT1.761.23
DailyOmni1.490.79
WorldSense0.571.42
Video-MME0.912.56
OmniVinci-9BAVUT2.662.51
DailyOmni1.670.67
WorldSense1.391.37
Video-MME0.361.18

On Qwen2.5-Omni the effect is statistically significant. With the pruning step unchanged, moving the video scoring layer from L3 to L5 raises accuracy by 0.91 points on 7B (95% CI [0.41, 1.41]) and by 2.31 points on 3B ([1.66, 2.89]) over the four benchmarks combined, while moving the audio scoring layer from L3 to L5 changes it by -0.05 ([-0.32, 0.23]) and +0.25 ([-0.06, 0.59]).

W1.4

Q: ... and to dispel my concern that the performance gains stem merely from tuning choices such as layer selection.

A: A direct test of this concern is to remove the layer selection. Without any layer selection, MoCo is 2.5 points above OmniZip on Qwen2.5-Omni-7B and 1.9 points above on Qwen3-Omni. The table compares OmniZip with MoCo at all 25 combinations of audio and video readout layers (the 25 cells of the scoring-layer grid), using mean accuracy at 25% retention over the four benchmarks of Table 1 (Qwen2.5-Omni-7B) and the three benchmarks of Table 2 (Qwen3-Omni):

BackboneOmniZipMoCo, average of all 25 cells (no layer selection)MoCo, best of the 25 cellsCells above OmniZip
Qwen2.5-Omni-7B55.2957.7859.0224 of 25
Qwen3-Omni62.2464.1565.2425 of 25

On both models, the gain over OmniZip therefore does not hinge on a particular choice of readout layers. The readout layers were configured once on a development split disjoint from evaluation and reused across all benchmarks and retention ratios (Sec. 5.2). On Qwen3-Omni, MoCo reuses the 7B configuration without re-tuning and still achieves higher average performance than OmniZip at all three retention ratios (Table 2). These results confirm that MoCo's gains do not require per-benchmark layer tuning. The control and the neighboring-cell tests are given in W2.2.

(Continued in Part 3/4.)

AuthorsOpenReview ↗

RESPONSE TO REVIEWER x4Qh (Part 3/4)

W2.1

Q: The choice (ℓa, ℓv) = (3, 5) rests on Obs. 3, but no mechanism is offered for why video is depth-sensitive and audio is not; a 5×5 sensitivity grid is equally consistent with "video needs later layers" and "L3/L5 is the argmax of a noisy grid."

A: Thank you for raising this question. The configuration was set on a development split disjoint from evaluation (Sec. 5.2) before any full-scale grid was run, so the full-scale grid is a test of that choice, not its source. The two readings make different predictions on the full benchmarks, and the data support "video needs later layers."

  1. The argmax of a noisy grid would fall back to the grid average on new data. (3,5) does not. On the 8,803 questions, it is above the average of the 25 cells on all four benchmarks, by 2.55, 1.16, 0.85 and 0.37 points on AVUT, DailyOmni, WorldSense and Video-MME, with paired-bootstrap 95% CIs excluding zero on AVUT, DailyOmni and WorldSense.
  2. A noisy grid would show no consistent trend with depth. The full-scale grid shows one. Marginal accuracy profile on Qwen2.5-Omni-7B (averaged over the four benchmarks and over the other modality's scoring layers):
Scoring layerL1L2L3L4L5
Video56.4557.4257.8958.5058.66
Audio56.9257.8858.1458.0057.97

Video accuracy rises at every step through L5, while audio accuracy changes by less than 0.2 points after L3. The paired tests in W1.3 confirm both with the pruning step unchanged.

Mechanism. It follows from the query-key geometry of Obs. 1 (Fig. 1, Eq. 2), which we traced layer by layer on all 3,939 clips. Mean cosine gap of the query positions between the audio-key and video-key centroids (positive means closer audio-centroid alignment):

ModelL1L2L3L4L5L6
Qwen2.5-Omni-7B0.0110.0590.0740.069-0.008-0.001
Qwen2.5-Omni-3B0.0140.0610.0800.0460.0090.000

The query positions align more closely with the audio-key centroid from L1 to L4, most strongly at L3, and the cosine gap falls to 0.03 or below at L5 on every benchmark, under both audio and visual questions. This transition is measured from the attention geometry alone, with no accuracy involved, and it points to the same two layers as the accuracy tests: L3, where the audio-centroid alignment peaks, and L5, where it drops to near zero. Beyond the geometry, Appendix C (Fig. 5) reports two interventions on this preference: replacing the audio with silence or noise shifts attention toward video, and matching token distances balances attention at L7 and L10 while an audio-favoring residual remains at L14 and L19.

W2.2

Q: The "plateau" is not quantified — no significance test against neighboring cells (L2/L5, L3/L4). The missing control ablation is also critical: Table 4b compares single-layer vs. staged readout but omits the control where audio is also pruned at L5 with separate budgets. Without it, the +2.47 pp gain cannot be cleanly attributed to staging rather than to early audio pruning freeing context/compute for video scoring.

A: Thank you for suggesting these tests. The full grid on Qwen2.5-Omni-7B contains them. In the control, audio is scored at L3 but pruned at L5 together with video, under the same separate budgets.

ConfigurationAVUTDailyOmniWorldSenseVideo-MME
MoCo: audio pruned at L3 (Table 1)63.6762.0744.8066.2
Control: audio pruned at L563.5061.9644.6765.93
Neighboring cell (2,5)62.1261.4044.7065.93
Neighboring cell (3,4)62.2862.2444.6165.96

The control matches MoCo within 0.3 points on every benchmark, so MoCo's accuracy does not come from early audio pruning freeing context for video scoring. Audio is pruned at L3 to reduce computation (Sec. 4).

Significance tests against the neighboring cells (paired bootstrap on the same questions; (3,5) is the control row above; difference in points with 95% CI):

ComparisonAVUTDailyOmniWorldSenseVideo-MME
(3,5) minus (2,5)+1.38 [0.41, 2.33]+0.55 [-0.25, 1.39]-0.03 [-0.57, 0.50]0.00 [-0.48, 0.48]
(3,5) minus (3,4)+1.22 [-0.07, 2.55]-0.28 [-1.50, 1.09]+0.06 [-0.69, 0.82]-0.04 [-0.81, 0.78]

(3,5) lies on a stable plateau: it is statistically on par with or above both neighboring cells on every benchmark, and no neighboring cell is significantly above it. Scoring video at L5 instead of L1 improves accuracy by 1.4 to 3.2 points on the four benchmarks, and every paired-bootstrap 95% CI excludes zero, so the depth effect is not grid noise.

(Continued in Part 4/4.)

Comments on the paper

2 comments
AuthorsOpenReview ↗

GENERAL RESPONSE

We sincerely thank all reviewers [x4Qh, bYNa, zpzD] for their valuable time and insightful reviews. We are glad that the reviewers found the problem relevant and the method practically appealing [x4Qh], the shallow audio preference an interesting phenomenon that provides useful insights [bYNa], the controlled budget-allocation comparison of Obs. 2 well designed [x4Qh, zpzD], and the evaluation across models, benchmarks and retention ratios useful evidence for the design [bYNa, zpzD]. We summarize the new experiments below; the full tables are in our response to each reviewer.

  • Obs. 1 at full scale [x4Qh]: on the full benchmarks (all 8,803 questions per model) and 3,939 paired clips, the shallow audio preference appears on all four models and all four benchmarks, at decoder layers 1 and 2 (L1 and L2) in every model-benchmark pair, and it persists across question types on all four models. At layer 2, at least 99% of query positions align more closely with the audio-key centroid than with the video-key centroid (positive cosine gap) on all four models and all four benchmarks (all 16 model-benchmark pairs). (Response to x4Qh, W1.1 and W1.2.)
  • Obs. 3 at full scale [x4Qh]: on 8,803 questions per model, video selection is more sensitive to scoring depth than audio selection in 13 of the 16 model-benchmark pairs, including all eight on Qwen2.5-Omni, where the effect is statistically significant. (Response to x4Qh, W1.3.)
  • Control and significance tests [x4Qh]: with audio also pruned at L5, accuracy stays within 0.3 points of MoCo, and the paper's configuration is statistically on par with or above its neighboring cells. Averaged over all 25 cells of the scoring-layer grid on Qwen2.5-Omni-7B, MoCo is still 2.5 points above OmniZip. (Response to x4Qh, W1.4 and W2.2.)
  • Measured efficiency [bYNa, zpzD]: at matched accuracy, MoCo needs less than half the tokens of OmniZip and has the larger prefill speedup (2.59x vs 1.97x at ~98% accuracy), the larger TTFT speedup (1.45x vs 1.20x), the higher throughput and the lower peak memory (26.6 vs 31.7 GiB), all measured with FlashAttention-2. (Response to bYNa, W2.2; response to zpzD, W2 and Q1.)
  • Question-type breakdown [bYNa]: on 3,172 WorldSense questions, no audio type or task type benefits significantly from deeper audio scoring. (Response to bYNa, Q1.)

We hope these results address the reviewers' concerns, and we are happy to answer any further questions.

AuthorsOpenReview ↗

Revised Paper Uploaded

Dear Reviewers,

We have uploaded the revised version of our paper, which incorporates the additional results from our responses. All changes are highlighted in blue.

Thank you again for your time and feedback.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.