acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Audio-Visual Correlation-Aware Token Compression for Omnimodal Large Language Models

Abstract

Omni-modal large language models (OmniLLMs) support unified audio-visual understanding. However, processing large numbers of audio and video tokens causes substantial computational and memory costs, making efficient token compression essential. Audio and video can provide unique or repeated information at different moments, while existing methods typically apply one fixed strategy throughout a sequence. Guiding one modality with the other may discard information unique to each modality when they are complementary, while compressing them separately may preserve redundant information when they overlap. Such fixed strategies fail to adapt to this changing correlation. To address this mismatch, we propose OmniTrim, a training-free framework for adaptive audio-visual token compression. Specifically, Information-Aware Budgeting allocates the token budget across temporal chunks according to their audio-visual activity and temporal variation. Correlation-Aware Selection then adaptively chooses between separate selection and joint token merging based on the local audio-visual correlation, preserving complementary information and removing redundancy. Experiments on six audio-visual benchmarks demonstrate that OmniTrim outperforms existing OmniLLM compression baselines. At 25% token retention, it preserves 98.8% of the full-token performance on Qwen2.5-Omni-7B, while delivering a 4.35× prefill speedup, increasing decoding throughput by 55.3%, and maintaining the lowest peak GPU memory usage and end-to-end latency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omnimodal Large Language Models, Audio-Visual Token Compression, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tJs2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets token compression for omnimodal LLMs (Qwen2.5-Omni). Its argument is that audio and video alternate between complementary chunks, where each modality carries unique information, and redundant chunks, where they overlap. It claims existing methods use one fixed strategy throughout a sequence. OmniZip-style methods let one modality guide the other. OmniScope-style methods compress each modality independently.

Critical strengths
  1. The method is training-free and plug-and-play. It runs before the LLM forward pass and its core operations are cheap (norms, mean-pooling, cosine). There is no retraining and no dependence on a specific attention implementation.

  2. The two-level framing is reasonable. Separating "where to spend tokens" (temporal budget) from "how to spend them" (selection mode) is clean, and the appendix gives enough detail on smoothing and merging to be reproducible in principle.

Critical weaknesses
  1. The central premise, complementary versus redundant chunks, is never measured. \rho is not validated. Contribution 1 says "we reveal how audio-visual correlation changes over time". The evidence is Figure 2, which is explicitly a schematic with one hand-picked video. There is no histogram of \rho, no per-video trajectory and no annotation of what a "complementary" or "redundant" chunk is. \rho is the cosine of two mean-pooled embeddings from different encoders. In Qwen2.5-Omni, the audio and vision towers are not contrastively aligned with each other. Their projected embeddings need not be geometrically comparable at all. Cosine between mean-pooled vectors is usually dominated by shared anisotropy or mean-offset components, not semantic overlap.

  2. The "adaptive" part is mostly Separate-only, and the gain may come from merging, not cross-modal redundancy removal. Over Separate-only, the full model gains 0.3 avg points at 45% (49.2 vs 48.9), 0.5 at 35% and 1.0 at 25% (Table 2). Per benchmark, much of the 45% gain comes from ShortVid and LVOmniBench. The joint branch bundles three things: cross-modal pooled ranking, cross-modal assignment, and similarity-weighted feature merging. Merging is a known trick (ToMe-style) that could help a separate path too.

  3. Baseline coverage is thin, and the comparison may not be level. The related work lists OmniSIFT, OmniFit, OmniSelect, Omni-Prune and PRIM, which are the closest prior work. None appear in the experiments. The claim "outperforms existing OmniLLM compression baselines" rests on only two omni baselines, OmniZip and OmniScope. FastVID and DyCoke are video methods applied "naively to both audio and video". Audio is a 1-D sequence with no spatial structure, so these are weak straw baselines. They mainly inflate the margin.

Questions for authors
  1. Show the real distribution of \rho: a histogram across all chunks, and per-video trajectories. How bimodal is it, and what is the within-video versus across-video variance? Does ρ track anything beyond a global offset between the two embedding spaces?

  2. Report Separate+merging and Joint-without-merging ablations, plus a joint branch with within-modality-only merging. Which component produces the gain over Separate-only?

  3. Add OmniSIFT, OmniFit, OmniSelect and Omni-Prune, or explain why they cannot be run. Add random and uniform-temporal baselines and a "keep all audio, prune only video" baseline. Match all methods on actual tokens entering the LLM or on FLOPs.

  4. How are position IDs and TMRoPE handled after pruning and merging? Does cross-modal merging hurt speech-heavy questions?

  5. How does OmniTrim relate to a query-aware variant? Would adding query relevance to the IAB or CAS signals help, and how does the method fare against query-aware baselines when they are given their best settings?

Reviewer_DcjbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OmniTrim is a training-free audio-visual token compression method for OmniLLMs. It redistributes a shared token budget across temporal chunks using feature activity and temporal variation, then routes each chunk to separate or joint audio-visual selection according to pooled cross-modal similarity. The joint branch also merges discarded features into retained representatives. Experiments use Qwen2.5-Omni at two model sizes and six benchmarks, with component ablations and efficiency measurements.

Critical strengths

The method addresses a relevant limitation of fixed audio-visual compression policies: modalities can contribute distinct or overlapping information at different times.

Critical weaknesses
  1. The routing ablations do not isolate the value of correlation. Table 2 changes branch operations; Figure 5 changes both routing assignments and joint-route frequency. Joint processing also adds merging. Shuffle routing assignments while fixing route fractions, budgets, and branch operations to establish whether correlation actually selects the right chunks.

  2. The joint-ranking scores lack a defined common scale. Eq. (6) combines visual feature norms and audio attention scores without specifying token-level calibration. Rescaling one modality can change the audio/video retention balance while preserving each modality's internal ranking. Specify the calibration and report retained token counts per modality; chunk-level normalization does not resolve this issue.

  3. QA-only evaluation leaves broader applicability unverified. Extend evaluation to temporal grounding, event localization, and audio-visual captioning on benchmarks such as LongVALE [1], AVSCapBench [2], or other suitable datasets. Report task quality and compression-inclusive costs across context lengths to test temporal precision, descriptive coverage, and long-context evidence retention.

  4. Model and baseline coverage remain limited. Experiments use only Qwen2.5-Omni-3B/7B. Extend full-method evaluation to Qwen3-Omni-30B-A3B, and assess transfer to the larger dense Qwen3.5-27B and MoE Qwen3.5-35B-A3B under explicitly defined, compatible settings. Beyond OmniZip [3], include OmniSelect [4] and other recent, directly relevant baselines. Compare accuracy, compression overhead, end-to-end latency, and peak memory under comparable budgets, accounting for auxiliary inference costs.

[1] Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos, CVPR, 2025.

[2] Yanghai Wang, Jiahao Wang, Jiafu Tang, Yuanxing Zhang, Zhe Cao, Hanyan Bian, Zijie Zhang, Weiliang Luo, Zhiyu Pan, Zixuan Dong, Jiaheng Liu, and Zhaoxiang Zhang. AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning, arXiv:2607.12820, 2026.

[3] Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Jian Liu, and Huan Wang. OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models, CVPR, 2026.

[4] Morunliu Yang, Ruotao Xu, Le Li, Yue Wang, Jianxin Zhang, Juntao Li, Yihang Lou, Siwei Feng, and Peifeng Li. OmniSelect: Dynamic Modality-Aware Token Compression for Efficient Omni-modal Large Language Models, NeurIPS, 2026.

Questions for authors

See weakness.

Reviewer_RUTPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniTrim is a training-free audio-visual token compression framework. It allocates temporal budgets using feature activity and variation, then switches between separate selection and joint selection/merging using chunk-level audio-visual similarity. Experiments on six benchmarks and two Qwen2.5-Omni sizes report strong average accuracy retention and reduced inference cost.

Critical strengths
  1. The proposed OmniTrim is a training-free audio-visual token compression framework, which is somehow interesting. Besides, it allocates temporal budgets using feature activity and variation, then switches between separate selection and joint selection/merging using chunk-level audio-visual similarity.
  2. Experiments on six benchmarks and two Qwen2.5-Omni sizes report strong average accuracy retention and reduced inference cost.
  3. The presentation is good and it is easy to follow.
Critical weaknesses
  1. The routing signal is not directly validated as a measure of redundancy. Equation 4 assumes that high cosine similarity between mean-pooled audio and video features favors joint compression. However, semantically related modalities may still contain different details needed to answer a question. The routing ablations show that switching policies helps, but not that the benefit comes from correctly identifying redundant versus complementary information. Chunk examples linking ρ\rho, routing decisions, and preserved or lost evidence would make this central explanation more convincing.

  2. Equation 6 uses encoder attention-derived importance ha,ih_{a,i}, while Section 4.1 states that all experiments use FlashAttention. The standard FlashAttention integration does not expose these weights, requiring a fallback, separate score computation, or a modified implementation. Please specify the audio encoder, visual encoder, and LLM attention backends for OmniTrim and every baseline, including the full-token model. Score-extraction overhead must be included in latency and peak memory to establish a fair efficiency comparison.

  3. Baseline coverage and hyperparameter selection leave comparison fairness unresolved. Section 2.2 discusses OmniSIFT, OmniFit, OmniSelect, and Omni-Prune, yet Table 1 includes only OmniZip, OmniScope, and two visual compression methods adapted to audio. Also several related work are ignored, e.g., FastAV: Efficient Token Pruning for Audio-Visual Large Language Model Inference, ICASSP 2026, and Gong et al., EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs, ICML 2026. Comparisons with these relevant methods would better support the state-of-the-art claim. If they cannot be compared, please specify the reason.

Questions for authors
  1. Since Equation 6 relies on encoder attention-derived importance scores, while the paper states that FlashAttention is used in all experiments, how are these scores extracted in practice?
  2. Are the associated computational overheads included in the efficiency comparisons?
  3. Why are relevant methods such as OmniSIFT, OmniFit, OmniSelect, FastAV, and EchoingPixels excluded from the comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.