acceptodds
Under review as a conference paper at ICLR 2027

JEC-Omni: Token Compression for Efficient Omnimodal Large Language Models via Joint Entropy Optimization

Abstract

Omnimodal Large Language Models (OmniLLMs) have demonstrated remarkable capabilities in audio-video understanding, but encoding continuous signals yields massive token sequences that impose enormous computational overhead and memory footprints. While training-free token compression offers a practical solution, existing omnimodal methods fail to precisely identify both intra- and cross-modal redundancies, struggling to stably maintain the full-token baseline accuracy under target token budgets. In this paper, we propose JEC-Omni, a novel training-free token compression framework that explicitly preserves high joint entropy for OmniLLMs. Our approach systematically eliminates redundancies through two core modules: MAST-P, which removes intra-frame spatial and inter-frame motion redundancies in the visual stream, and SICM-P, which leverages intra-modal temporal consistency and cross-modal semantic overlap to prune the audio stream. Extensive evaluations on the Qwen2.5-Omni and Qwen3-Omni architectures demonstrate that JEC-Omni achieves a superior trade-off between inference efficiency and accuracy. Remarkably, our method maintains 97% of the original model accuracy while retaining merely 25% of the raw audio-video tokens, reducing the end-to-end inference memory footprint by 1.1x to 1.8x and decreasing the Time-to-First-Token (TTFT) by 1.5x to 2.5x.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omnimodal Large Language Models, Token Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vyz2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies audio-visual token compression in omni-modal large language models and proposes JEC-Omni, a training-free method that compresses tokens before they enter the LLM, motivated by the joint entropy of audio and video. For video, the method groups every two adjacent frames: the first frame removes spatial redundancy, and the second frame searches nearby positions in the previous frame for similar content to remove redundancy caused by motion-induced displacement. For audio, the method combines temporal redundancy with the similarity to the retained visual content and removes the parts deemed redundant. The retention ratios of the two modalities are preset. The authors evaluate the method on several audio-visual understanding benchmarks with two families of omni-modal models. The results show that it approaches the full model on some benchmarks and reduces time-to-first-token latency.

Critical strengths
  1. Simple, training-free and easy to deploy. Compression is performed before the LLM using only local feature similarity, without attention maps or training, and works directly with FlashAttention. The paper measures end-to-end time-to-first-token latency including the encoders, which is closer to real deployment than reporting token counts or FLOPs alone.
  2. Good performance retention on WorldSense. Under moderate compression ratios, the results are close to the full model and better than existing audio-visual compression methods.
  3. Thorough module-replacement experiments. The paper fixes the compression method of one modality while replacing that of the other, and compares different redundancy signals, which explains the role of each module better than comparing whole pipelines only.
Critical weaknesses
  1. Limited novelty: both modules have very close published work. MAST-P handles redundant content across frames at different positions, which video methods such as FastVID [1] already do, yet the paper avoids comparing with them by arguing that "audio and video redundancy are distributed differently", and only compares with DyCoke [2] and OmniZip [3], which match tokens at the same position. The paper also claims that existing methods do not use cross-modal signals to decide what to remove, but OmniSIFT [4] removes audio conditioned on the retained video, the same idea as SICM-P with a learned scorer instead of cosine similarity. Excluding these parts, the new local maximum-similarity matching and cosine criterion are exactly the parts with the smallest gains (W2, W5). I suggest redefining the contributions and comparing with the methods above.
  2. MAST-P is only marginally better than random pruning, within experimental variance. In Table 4, MAST-P outperforms random video pruning by only 0.6 (3B) and 1.0 (7B) points; in Table 3, both variants without motion search underperform random pruning, and the table itself is non-monotonic (with spatial-only pruning, the 35% budget result is lower than the 25% one). Experimental variance is thus comparable to the 0.5–0.7 point gain from motion search, and the claim that MAST-P "significantly outperforms all baselines" (l.373) is not supported.
  3. Hyperparameters and ablations are all determined on WorldSense, where the largest advantage also appears. Tables 3–6 and the sensitivity analysis use only WorldSense, and the default hyperparameters are its optimum; the largest advantage over DASH [5] is also on WorldSense, while results on other benchmarks are mixed (Table 1). This amounts to tuning on the test set. I suggest selecting hyperparameters on held-out data and reporting ablations with confidence intervals on other benchmarks.
  4. The joint-entropy framework does not lead to the algorithm used. When selecting a subset, all terms in H(V,A)=H(V)+H(A)−I(V;A) change together: removing tokens lowers I(V;A) but also lowers H(A), so "minimizing mutual information to maximize joint entropy" is not a valid selection objective, and the paper does not explain how the cosine scores relate to entropy. I suggest narrowing this title-level claim to "an information-theory-inspired empirical rule".
  5. The cross-modal criterion of SICM-P is not validated, and its scope is small. Eq. (9) directly compares outputs of two independent encoders by cosine similarity without evidence that they are aligned, so the score may mainly reflect a systematic offset between modalities. In Table 5, using only the cross-modal term performs on par with using only the intra-modal term, and fusing them adds only 0.2–0.3 points. Moreover, from §4.1, audio accounts for only about 15% of all tokens, so SICM-P affects only 5–7% of the tokens, and "eliminating cross-modal redundancy" concerns only a small part of the budget. References [1] Leqi Shen, Guoqiang Gong, Tao He, et al. FastVID: Dynamic Density Pruning for Fast Video Large Language Models. In NeurIPS, 2025. [2] Keda Tao, Can Qin, Haoxuan You, Yang Sui, Huan Wang. DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models. In CVPR, 2025. [3] Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Jian Liu, Huan Wang. OmniZip: Audio-Guided Dynamic Token Compression for Fast Omnimodal Large Language Models. In CVPR, 2026. [4] Yue Ding, Yiyan Ji, Jungang Li, et al. OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-Modal Large Language Models. In ICML, 2026. [5] Bingzhou Li, Tao Huang. DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omni-Modal Token Compression. arXiv:2603.15685, 2026.
Questions for authors
  1. Compared with OmniSIFT [4], which removes audio conditioned on the retained video, and video methods such as FastVID [1], which handle redundant content at different positions, what are the key differences and additional benefits of MAST-P and SICM-P? Could you compare with them under the same budgets, or explain why such a comparison is not needed?
  2. Could you compare MAST-P, random video pruning and same-position temporal matching on at least one benchmark other than WorldSense? Please select hyperparameters on held-out data and report paired confidence intervals.
  3. Keeping the retained video tokens, the audio budget and the intra-modal scores fixed, if the video is replaced by a temporally shuffled version or by the video of another sample, does the cross-modal term still bring gains? In dialogue-heavy tasks, is the information in the audio removed due to high similarity actually contained in the visual content?
  4. The entropy of which random variable does the method actually optimize, and how do the cosine scores relate to it? If entropy is only a motivation, would you be willing to tone down the "joint entropy optimization" claim?
  5. For each model, including Qwen3-Omni, what are the actual numbers of video and audio tokens and the overall retention ratio?
Reviewer_geiGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JEC-Omni is a training-free audio-video token compressor motivated by joint entropy. Its video module processes frames in pairs, using spatial similarity within the first frame and local matching across frames to account for motion. The audio module removes tokens with high similarity to neighbouring audio tokens and to the retained visual tokens.

Experiments on Qwen2.5-Omni and Qwen3-Omni cover four video-understanding benchmarks. The method retains roughly 97–98% of full-token performance under substantial compression and is competitive with OmniZip and DASH. End-to-end measurements show larger memory and TTFT savings for longer videos.

Critical strengths

The compressor is simple, training-free and does not need attention logits, which is a practical advantage for deployment with existing kernels. I also appreciate the end-to-end efficiency measurements, including the encoders, rather than LLM-only FLOPs. The module-swap experiments in Tables 4 and 6 help isolate the components, and both the motion window and cross-modal audio score show modest benefits.

Critical weaknesses

I am not convinced that the method amounts to joint entropy optimisation. The implemented scores are cosine similarities; the paper does not show that these estimate mutual information or preserve information relevant to the answer. An audio token similar to a visual token may provide precisely the correspondence needed for audio-visual reasoning, rather than dispensable redundancy. The entropy identity alone does not justify discarding it. The pipeline is also structurally close to OmniSIFT, although the training-free formulation and pruning direction differ. A direct comparison is missing, as are comparisons with relevant training-free methods such as OmniRefine. This leaves the contribution beyond existing compression heuristics less clear than the framing suggests.

The empirical advantage is modest and not consistent across settings. At 25% retention on 7B, DASH scores 64.9 versus 64.7 for JEC-Omni. Other gains are often a few tenths of a point without uncertainty estimates. Table 2 mixes retention ratios, and Qwen3-Omni has no compression baseline. The efficiency gain over DASH is also much smaller than the gain over full tokens: about 1198 versus 1235 ms TTFT in the 7B AVUT experiment. Finally, WorldSense is used for sensitivity analysis as well as evaluation, and the audio/video budget split is hand-set. Parameter selection needs clarification before small performance differences can be interpreted confidently.

Questions for authors

Can you provide matched-budget comparisons with relevant prior methods and an OmniZip or DASH baseline on Qwen3-Omni? Please include uncertainty estimates and explain whether hyperparameters and modality budgets were selected on held-out data or tuned per benchmark.

What evidence supports the entropy interpretation? An analysis of questions helped or hurt by removing visually similar audio tokens would be particularly useful for checking whether the cross-modal score removes redundancy or necessary correspondence.

Reviewer_iRpgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes JEC-Omni, a novel training-free token compression framework tailored for efficient OmniLLMs. To elminate both intra-modal and cross-modal redundancies, the authors propose MAST-P and SICM-P for visual and audio token compression, respectively. MAST-P removes spatially and temporally similar video tokens, whereas SICM-P uses intra-modal and cross-modal similarity to prune audio tokens. Experiment show that with JEC-Omni, OmniLLMs can maintain competitive performance with only 25% of the raw audio-video tokens.

Critical strengths
  1. Comprehensive Experiment: The proposed method is evaluated on multiple benchmarks, LLM backbones and token budgets, confirming its effectiveness. The authors reported not only accuracy scores but also efficiency metrics, demonstrating advantage in real-world applications.
  2. Compatibility: The method is training-free and is compatible with acceleration infrastructures like FlashAttention, which facilitatates industrial deployment.
Critical weaknesses
  1. Theory-Implementation Gap: The joint entropy optimization theory remains a motivation rather than a measured property of the retained tokens. Therefore, it is unclear how MAST-P and SICM-P affects H(V), H(A) and I(V;A) quantitatively.
  2. Marginal Improvement: On audio–video QA benchmarks, the advantage of JEC-Omni over DASH is often less than 1 pp.
Questions for authors
  1. The authors emphasizes token reduction and accuracy retention performance with a token retention budget of 25%. However, most ablation studies are conducted under a 35% token retention budget. Have you validated the effectiveness of MAST-P and SICM-P under the 25% token retention budget?
  2. The number of audio tokens is much more that that of the visual tokens in a unit-length video segment. However, the selected budgets retain a much larger fraction of audio than video tokens (64% versus 30% at the 35% total budget; 54% versus 20% at 25%). Why?
  3. Is there any quantitative method to compare H(V), H(A) and I(V;A) of the proposed method and the baselines? Or is the entropy formulation only a theoretical guide for method design?

Other minor suggenstion:

  1. It would be better if you include a limitation section in the paper.
  2. You may consider using \citep rather than \cite to include citations with parentheses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.