- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper finds that shallow question-conditioned attention favors audio even for visual questions. MoCo therefore separates audio-video budget allocation from token ranking within each modality, and prunes audio and video at different shallow decoder layers.
Critical strengths
(1) The paper identifies a concrete failure mode of shared cross-modal top-K selection. Its controlled budget-allocation comparison keeps the scores, total budget, and pruning step fixed. (2) Results across several models, together with matched-FLOPs comparisons, provide useful evidence for the proposed design.
Critical weaknesses
(1) OmniFocus and OmniScope also estimate importance or allocate capacity separately by modality. MoCo's decoder-layer readouts and staged pruning are distinct, but direct comparisons under the same model, budget, and compute would better establish their added value. (2) At 25% token retention, MoCo uses more estimated FLOPs than OmniZip. The matched-FLOPs results help, but measured prefill latency and peak memory, including scoring and pruning overhead, are still needed to assess deployment efficiency.
Questions for authors
Could the authors provide matched-condition comparisons with recent modality-balanced methods and report measured latency and memory across compression levels?
2 comments
A Gentle Reminder Before the Discussion Period Ends
Dear Reviewer zpzD,
We hope all is well with you. We would like to gently follow up on our responses to your comments and the additional experiments we have provided. Since the discussion period will end in less than 24 hours, we would greatly appreciate it if you could take a moment to look over our responses and let us know whether any questions or concerns remain, should your schedule permit. We are more than happy to clarify anything further.
We fully understand how busy your schedule may be, and we are truly grateful for your time and consideration. Thank you again for your attention to our work!
RESPONSE TO REVIEWER zpzD
Thank you very much for your positive assessment and helpful suggestions. We are glad the reviewer found that the paper identifies a concrete failure mode of shared cross-modal Top-K selection. We address each point below.
W1: Comparison with OmniFocus and OmniScope
A: Thank you for pointing us to these works. OmniFocus (Cao et al., arXiv:2607.03050, July 2026) and OmniScope (Su et al., arXiv:2607.23193, July 2026) are concurrent works. Both compress tokens before the LLM: OmniFocus allocates the budget over temporal chunks by query similarity and then selects tokens by inter-modal association and intra-modal saliency, while OmniScope scores video frames with an external CLIP model and audio tokens by query similarity. MoCo is the only one of the three that selects tokens inside the decoder, after audio-video fusion, from question-conditioned attention at modality-specific depths (L3 for audio, L5 for video). Token survival is therefore decided by the decoder's own attention after the question has interacted with both modalities, with no external scorer and no pre-LLM similarity heuristic. The paper already includes matched-retention comparisons with DASH, a recent OmniLLM compression method (Table 8): MoCo is higher in five of the six comparisons, by up to 3.13 points. We plan to comprehensively compare MoCo with OmniFocus, OmniScope and other concurrent methods in future work. However, as these are concurrent works, this comparison is beyond the scope of this rebuttal.
W2: FLOPs at 25% retention; measured prefill latency and peak memory including scoring and pruning overhead
A: Following your suggestion, we measured prefill latency and peak memory, including scoring and pruning overhead. Since the two methods differ in accuracy at equal retention, we compare them at matched accuracy (Table 5). Setup: Qwen2.5-Omni-7B on WorldSense, on a single A100 (40 GB) GPU, FlashAttention-2.
At every matched-accuracy level, MoCo needs less than half the tokens of OmniZip and has the larger measured prefill and TTFT speedups and the lower peak memory. The pruning steps themselves take only about 4% of prefill.
Q1: Latency and memory across compression levels
A: The setup is the same as in W2.
MoCo gives a 2.2x-3.6x measured prefill speedup and a 1.4x-1.6x TTFT speedup across retention levels, and lowers peak memory from 37.9 GiB to below 27 GiB. Peak memory stays low at every retention level because it is set in the early decoder layers, before video tokens are pruned.
We hope the measured results address your concerns, and we would be grateful if you would consider raising your score. Please let us know if you have any further questions. Thank you again for your support.