acceptodds
Under review as a conference paper at ICLR 2027

-Omni: Reducing Structural and Precision Redundancy in Omni-LLMs for Long-Form Generation

Abstract

Omni-modal large language models (Omni-LLMs) typically process large numbers of multimodal tokens from audio-visual inputs, incurring substantial computational costs and a large memory footprint for the key–value (KV) cache. Aggressive token pruning can alleviate this problem while maintaining competitive performance on multiple-choice tasks, which require only a single option as output. However, we observe substantial quality degradation when generating long-form, open-ended responses. We attribute this gap to query-guided selection, which can retain sufficient evidence for answer selection while discarding details needed for comprehensive generation. This motivates reducing overlapping information across tokens, termed structural redundancy, while preserving diverse content. However, the retained representations may still contain precision redundancy due to unnecessarily high numerical precision in their cached values. To address these two levels of redundancy in token representations, we introduce a training-free framework named R-Omni. First, Structure-Aware Token Consolidation (STC) consolidates redundant representations using spatiotemporal similarity in video and local continuity in audio. Then, Semantic-Sensitivity Guided Precision Allocation (SSPA) allocates value precision to the retained representations according to the estimated reduction in attention-output error under quantized keys, with query attention and audio-visual alignment providing a semantic prior. Extensive experiments on long-form generation benchmarks demonstrate an average 5.0 reduction in the memory footprint while preserving 97.98% of baseline generation quality and achieving a 1.48 end-to-end speedup, highlighting the benefits of jointly reducing structural and precision redundancy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Omni-modal Large Language Models, Token pruning, KV Cache Compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_T776OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets the KV-cache and prefill costs of omni-modal LLMs when the output is long-form generation (dense captioning, event narration) rather than multiple choice. The authors' starting observation is that existing query-guided token pruning keeps multiple-choice accuracy high but degrades open-ended generation, because tokens that the initial query ignores are still needed later in generation. They split the redundancy in the input into “structural” (overlapping information across tokens) and “precision” (unnecessarily high bit-width of cached values), and attack both with a training-free two-stage framework. Stage one, Structure-Aware Token Consolidation (STC), merges video tokens within and across adjacent frames using cosine-similarity regions (union-find over spatial neighbors, centroid prototype per region, frame-mean test to inherit regions) and merges locally similar adjacent audio token pairs. Stage two, Semantic-Sensitivity Guided Precision Allocation (SSPA), keeps keys at channel-wise INT4 and assigns value precision (INT2/INT4/16-bit) per 32-token page by greedy rounds that maximize the attention-output error removed per extra bit, using attention from the final prefill queries and an audio–video alignment score as a semantic prior on top.

Critical strengths

The evaluation is complete for such a compression paper: three Omni-LLMs with quite different audio-visual front-ends, three long-form benchmarks, compression-ratio sweeps up to 9×, naive prune-then-quantize compositions as baselines (Table 2), and a long-context efficiency study (16K–256K) whose latency numbers include the calibration and allocation overhead rather than hiding it.

Critical weaknesses
  1. The headline quality number rests on an aggregation that needs justification. EER improves as compression gets more aggressive (85.66 for the 3B baseline vs. 95.65 under OmniSelect) while Correct drops by nine points, so “average relative quality” mixes metrics that move in opposite directions, and the “97.98% of baseline quality” claim is hard to interpret. More importantly, for a paper whose central claim is about generation quality, there is no human evaluation and no LLM-as-judge study of the actual generated long-form text anywhere in the paper or appendix.

  2. The advantage over the strongest KV-quantization-only baseline is thin on two of the three backbones. On Qwen2.5-Omni-7B the method reaches 100.11 average relative quality versus 99.19 for OTT, and on MiniCPM-o-4.5 93.86 versus 93.03 for OScaR. On both backbones the pruning-based baselines collapse for long-form generation, so the headline “outperforms all baselines” largely reduces to “KV quantization beats token pruning on this task” — which Table 1 already shows for KIVI/OTT/OScaR by themselves. What the structural consolidation stage specifically contributes beyond a larger compression ratio on a strong backbone is not demonstrated; the ablation in Table 3 suggests its main effect is ratio, not quality.

Questions for authors
  1. On Qwen2.5-Omni-7B, is the gap to OTT (100.11 vs. 99.19 average relative quality) statistically meaningful? Per-benchmark numbers with variance would clarify whether STC contributes quality on this backbone or only compression ratio.

  2. How does STC behave on short inputs (e.g., a few dozen frames)? The frame-pair persistence mechanism and spatial region construction seem designed for long videos; does consolidation still help when prefill is already cheap?

3 comments
AuthorsOpenReview ↗

Response to Reviewer T776 (1/3) : W1

W1: The headline quality number rests on an aggregation that needs justification. Moreover, there is no human evaluation or LLM-as-judge study of the actual generated long-form text.

Thank you for highlighting the need for clearer quality evidence. We would like to clarify the evaluation metrics, aggregation procedure, and existing LLM-based evaluation protocol, while highlighting the qualitative evidence already provided in our appendix.

1. Metric clarification. First, we clarify that EER (Event Error Rate) is a lower-is-better metric. Therefore, its increase from 85.66% to 95.65% under OmniSelect indicates deterioration rather than improvement, consistent with the decrease in Correct from 24.45% to 15.18%.

Following the Video-SALMONN 2 evaluation protocol, let C, M, I, and H denote correctly described, missed, incorrect, and hallucinated events, respectively. With N = C + M + I, the metrics satisfy:

\mathrm{EER}=\frac{M+I+H}{N} =1-\mathrm{Correct}+\frac{H}{N}.$$ **2. Aggregation clarification.** The average relative quality reported in Table 1 is computed by normalizing each metric against its corresponding BF16 baseline. For higher-is-better metrics (Correct, SODA-C, and SODA-M), we use the ratio of the compressed method to BF16; for the lower-is-better EER, we use the inverse ratio. The four normalized metrics are then averaged with equal weighting across the three model configurations.We acknowledge that this aggregate is intended as a summary of performance across heterogeneous benchmarks, rather than a standalone measure of generation quality. **Excluding EER from the aggregation still yields approximately 97.86% average relative quality**, compared with the originally reported 97.98%. **3. LLM-as-Judge evaluation and case studies.** We would like to clarify that **Video-SALMONN 2 already employs an official LLM-based event evaluator**, as described in Appendix C. This evaluator directly assesses the generated long-form descriptions against annotated audiovisual events, identifying correctly described, missed, incorrect, and hallucinated content. Therefore, our evaluation already includes LLM-based assessment of actual generated text, rather than relying solely on conventional similarity metrics. Furthermore, **Appendix E (Figures 7–8)** presents qualitative case studies comparing the generated descriptions across methods. These examples illustrate how R²-Omni preserves fine-grained visual details, dialogue-related information, and audiovisual event consistency under compression. In particular, Figure 8 provides an event-level comparison of unsupported descriptions, illustrating the preservation of factual consistency. **Conclusion.** We will clarify the direction and aggregation of evaluation metrics, explicitly highlight the existing LLM-as-Judge protocol in the main text, and better connect the quantitative results with the qualitative evidence in Appendix E. These clarifications provide a more transparent assessment of long-form generation quality without relying solely on the aggregate relative-quality score. The metric definitions and aggregation check are summarized in **Appendix F.5** of the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer T776 (2/3) : W2

W2: The advantage over the strongest KV-quantization-only baseline is thin on two of the three backbones. What does the structural consolidation stage specifically contribute beyond a larger compression ratio?

Thank you for this insightful observation. We would like to clarify that STC primarily contributes by reducing structural redundancy, complementing the precision reduction achieved by KV quantization. Its benefits extend beyond KV-cache compression to reduced prefill computation and a potentially more efficient precision allocation space.

1. Additional memory reduction. To better characterize the contribution of STC, we conducted an additional comparison on a fixed subset of 100 Video-SALMONN 2 videos using Qwen2.5-Omni-7B. SSPA-only retains the original input tokens while applying mixed-precision allocation, whereas R²-Omni enables both STC and SSPA.

MethodCorrect ↑EER ↓Hallucination ↓Allocated KV ↓
BF1639.10%78.76%17.86%1344.6 MiB
SSPA-only39.78%69.73%9.51%453.2 MiB
R²-Omni38.47%78.93%17.40%274.5 MiB

Analysis. Compared with SSPA-only, R²-Omni further reduces allocated KV storage by 39.43%, achieving approximately 4.90× compression relative to BF16, compared with 2.97× for SSPA-only. We acknowledge that SSPA-only achieves better quality at its larger memory budget. Therefore, this experiment demonstrates the additional compression benefit of STC and the resulting memory–quality trade-off, rather than an unconditional quality improvement.

2. Coupling between structural and precision redundancy. Importantly, structural and precision redundancy are not entirely independent. Without STC, structurally redundant tokens still occupy KV-cache capacity and participate in page-wise precision allocation. By consolidating similar tokens, STC reduces the number of representations competing for a limited precision budget, potentially allowing SSPA to allocate higher precision to retained, information-distinct tokens under the same memory constraint. Thus, STC not only reduces sequence length but also reshapes the precision allocation space. This interaction provides an additional motivation for jointly addressing structural and precision redundancy, although the present experiment does not independently establish a quality gain from this coupling at matched memory budgets.

3. Prefill efficiency. Beyond cache storage, STC reduces the number of tokens processed by the LLM, a benefit that KV quantization alone cannot directly provide. As shown in Table 3, the original 256K-context efficiency ablation reports:

MethodTTFT (s) ↓E2E (s) ↓
SSPA-only (w/o STC)58.5478.86
R²-Omni27.0340.34

Under this 512-token generation setting, R²-Omni achieves a 2.17× TTFT speedup and 1.95× E2E speedup relative to SSPA-only. These results demonstrate that STC provides substantial inference efficiency benefits by shortening the sequence processed during prefill, beyond what KV-cache quantization alone can achieve.

Conclusion. We appreciate the reviewer's suggestion and clarify that STC is not intended to universally improve generation quality over KV quantization alone. Rather, STC complements SSPA in three aspects: (1) reducing sequence length and prefill computation, (2) enabling additional KV-cache compression, and (3) reducing structural redundancy in the representations subject to precision allocation, potentially improving the utilization of a fixed precision budget. Together, these complementary roles motivate our joint design for balancing generation quality, memory consumption, and long-context inference efficiency.

The STC-versus-SSPA-only comparison is included in Appendix F.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer T776 (3/3) : Q1 & Q2

Q1: On Qwen2.5-Omni-7B, is the gap to OTT (100.11 vs. 99.19 average relative quality) statistically meaningful? Per-benchmark numbers with variance would clarify whether STC contributes quality on this backbone or only compression ratio.

Thank you for this important question. We agree that the relatively small improvement over OTT requires statistical validation beyond the reported point estimates.

Setting. We consider three paired evaluation runs on Qwen2.5-Omni-7B under identical evaluation configurations and comparable compression budgets. We report the mean and standard deviation of average relative quality and assess the differences using a two-sided paired t-test.

Results. The three-run comparison is summarized below.

MethodSeed 1Seed 2Seed 3Mean ± SD
OTT99.5399.1998.7699.16 ± 0.39
R²-Omni100.07100.1199.89100.02 ± 0.12

Analysis. In this comparison, R²-Omni achieves a mean improvement of +0.86 percentage points over OTT. The paired t-test yields t(2)=5.00t(2)=5.00 with a two-sided p=0.038p=0.038, and the 95% confidence interval for the paired difference is approximately [+0.12,+1.61][+0.12,+1.61] percentage points. These results support a statistically significant advantage over OTT at the 0.05 level.

The paired-run comparison and statistical test are included in Appendix F.5 of the revised manuscript.

Q2: How does STC behave on short inputs (e.g., a few dozen frames)? The frame-pair persistence mechanism and spatial region construction seem designed for long videos; does consolidation still help when prefill is already cheap?

Thank you for this valuable suggestion. We conducted additional experiments with 16 and 32 sampled frames to examine the effectiveness of structural consolidation on short visual inputs.

Setting. We evaluate Qwen2.5-Omni-3B on Video-SALMONN 2 , preserving the complete audio inputs and identical decoding configurations while varying only the sampled frame count. We compare full R²-Omni against BF16 for generation quality and KV-cache usage, and separately evaluate STC-only against BF16 to isolate its latency contribution. All compression configurations follow the default settings specified in Appendix B, without additional tuning for different frame counts.

1. Generation Quality and Memory.

FramesMethodCorrect ↑EER ↓Hall. ↓KV Memory ↓
16BF1624.93%85.66%10.59%162.1 MiB
R²-Omni24.99%84.27%9.26%42.1 MiB
32BF1626.77%82.68%9.45%262.0 MiB
R²-Omni27.83%80.81%8.63%57.3 MiB

At 16 and 32 frames, R²-Omni achieves approximately 3.85× and 4.57× KV-cache compression, respectively. Meanwhile, Correct increases slightly by +0.06 and +1.06 percentage points, with lower EER and hallucination rates at both settings. These results indicate that the full framework can achieve substantial memory savings without an observed quality degradation on the tested few-frame inputs.

2. Inference Efficiency. To isolate the runtime contribution of STC, we additionally compare STC-only against BF16.

FramesBF16 TTFT ↓STC-only TTFT ↓BF16 E2E ↓STC-only E2E ↓Speedup ↑
16457.3 ms430.8 ms2580.0 ms2476.5 ms1.042×
32804.5 ms733.4 ms2845.0 ms2769.4 ms1.027×

STC reduces time to first token by 5.8% and 8.8% at 16 and 32 frames, respectively, while achieving modest end-to-end speedups of 1.042× and 1.027×. This demonstrates that structural consolidation can provide measurable inference benefits even when prefill is relatively inexpensive.

Conclusion. STC remains effective with only a few dozen sampled frames, contributing to substantial KV-cache reduction and modest inference acceleration while maintaining competitive generation quality. These experiments vary the frame-sampling density of the same videos rather than video duration, and therefore specifically establish effectiveness under sparse visual sampling.

The few-frame quality, memory, and latency results are included in Appendix F.3 of the revised manuscript.

Reviewer_omiQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

R²-Omni addresses training-free compression for long-form audio-visual generation. It combines structure-aware token consolidation with mixed-precision KV caching, allocating value precision using attention-output sensitivity and multimodal semantic priors. The approach targets complementary structural and numerical redundancy. Experiments across three model configurations and three long-form benchmarks report approximately 5x final-cache compression while retaining 97.98% average relative quality. A separate long-context efficiency evaluation reports a 1.48x end-to-end speedup at 256K input tokens.

Critical strengths
  1. The paper targets a meaningful evaluation gap: retaining enough evidence for a short answer is different from retaining the diverse details needed for long-form generation. Table 6 documents stronger degradation on detailed captioning than multiple-choice answering under aggressive pruning. The main experiments consequently evaluate three long-form tasks across Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o-4.5. Table 1 shows the strongest average relative quality among the listed compressed methods for each model configuration at approximately matched final-cache budgets. This is a useful empirical contribution beyond conventional multiple-choice compression evaluations.

  2. The framework combines evidence-preserving structural reduction with precision reduction in an operational inference system. The allocation objective evaluates value upgrades in attention-output space while accounting for quantized keys, rather than using semantic importance alone. The appendix specifies packed mixed-precision decoding and includes scoring/allocation/packing overhead in TTFT. Table 7 also distinguishes final cache size from peak GPU memory and shows an actual 256K latency reduction from 118.910s to 80.347s. These measurements make the contribution practically more convincing than a storage-only analysis.

Critical weaknesses
  1. The specific contribution of the proposed output-sensitive allocation is not yet isolated. Table 4 removes alignment or the semantic prior, but retains the output-space allocation machinery. Table 3 removes an entire stage and does not consistently match final compression across tasks. Consequently, the experiments do not establish how much of the gain requires the residual-based allocation and outer refinement, rather than STC followed by a simpler value-precision policy. A comparison using the same retained tokens, key format, protected layers, and actual byte budget is important because the method combines several known compression ingredients with a comparatively elaborate allocator.

  2. The joint quality-efficiency operating point is insufficiently documented. Table 1's quality results and Table 7's 16K-256K efficiency experiment use different settings, and the latter does not report corresponding generation quality. Table 3 additionally reports E2E=40.34s with TTFT=27.03s and TPOT=25.90ms, whereas Table 7 reports approximately the same TTFT/TPOT but E2E=80.347s for 2,048 output tokens; the ablation's output length is not stated. The appendix also shows slowdowns at 16K-64K. These do not negate the long-context result, but prevent interpreting the headline retention and speedup as a single generally applicable operating point.

Questions for authors
  1. At matched actual cache bytes, how does SSPA compare with a simpler fixed or attention-ranked value-precision policy using exactly the same STC output, INT4 key format, protected layers, and quantization metadata? What is the incremental quality and calibration cost of the output-residual objective and outer refinement? This would clarify whether the main algorithmic complexity is necessary.

  2. What input/output lengths and model configuration are used for Table 3's efficiency columns, and why does its E2E differ from Table 7 despite nearly identical TTFT/TPOT? Please identify the exact compression configuration used at each long-context point and report quality at representative 128K/256K settings. This would establish the quality-efficiency trade-off supporting the headline speedup.

  3. How were the dataset-specific precision ceilings, protected layers, and the pruning/quantization split for composed baselines selected? Were these choices fixed using separate validation data, and are the direct compositions competitive after tuning that split at the same byte budget? A documented selection procedure would strengthen the comparison with simpler combinations.

5 comments
AuthorsOpenReview ↗

Response to Reviewer omiQ (1/3) : W1

W1: The specific contribution of the proposed output-sensitive allocation is not yet isolated. Table 4 removes alignment or the semantic prior, but retains the output-space allocation machinery. Table 3 removes an entire stage and does not consistently match final compression across tasks.

Thank you for this constructive observation. We agree that the original ablations mainly isolate individual compression stages and semantic signals, rather than the incremental contribution of output-sensitive allocation. To address this concern, we conducted additional comparisons against simpler value-precision allocation policies.

1. Controlled allocator comparison. We evaluate Qwen2.5-Omni-3B on the same Video-SALMONN 2 testset, using 128 sampled frames and complete audio. We compare Uniform, Attention-ranked, Attention + A/V semantic prior, and the full SSPA allocator. All policies share identical STC outputs, INT4 Keys, protected-layer settings, and quantization formats.

Allocation PolicyCorrect ↑EER ↓Hall. ↓KV (MiB) ↓
Uniform27.94%84.30%12.24%146.8
Attention-ranked26.83%81.91%8.75%146.8
Attention + A/V prior27.46%81.20%8.66%146.8
Full SSPA29.24%79.26%8.50%147.9

2. Quality analysis. Full SSPA achieves the highest reference-event coverage and the lowest hallucination rate among the evaluated policies, while maintaining competitive EER. Compared with Attention-ranked allocation, Full improves Correct by +2.41 percentage points , reduces EER by 2.65 points, and lowers Hallucination by 0.25 points. Compared with Attention + A/V prior, Full improves Correct by +1.78 points and reduces Hallucination by 0.16 points.

These results suggest that output-sensitive allocation can improve event coverage without an accompanying increase in hallucination at the evaluated operating point.

3. Role of output-sensitive allocation. Unlike policies based solely on attention importance or semantic relevance, SSPA evaluates the reduction in attention-output distortion produced by each precision upgrade, accounting for the current residual under quantized Keys. Its outer refinement further updates Key reconstruction under the Value-induced cache layout.

This enables precision decisions to consider the effect of quantization on the attention readout rather than relying exclusively on token-level importance scores. The additional results provide empirical evidence for the benefit of this design, particularly in preserving reference-event coverage while maintaining factual consistency.

Conclusion. The additional comparison supports the contribution of output-sensitive allocation: Full SSPA achieves the highest Correct and lowest Hallucination, with EER comparable to the strongest simpler policy, at similar final KV-cache footprints.

The allocator-policy comparison is included in Appendix F.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer omiQ (2/3) : W2 & Q1

W2: The joint quality-efficiency operating point is insufficiently documented. Table 1's quality results and Table 7's 16K–256K efficiency experiment use different settings, and the latter does not report corresponding generation quality.

Thank you for this observation. We clarify that the reported 97.98% average relative quality and 1.48× speedup come from separate evaluation protocols, rather than a single jointly measured operating point.

1. Quality versus efficiency. Table 1 evaluates generation quality across three Omni-LLMs and three long-form benchmarks at approximately 5× KV-cache compression. In contrast, Table 7 measures inference efficiency on Qwen2.5-Omni-7B with 16K–256K raw input tokens. We will explicitly distinguish benchmark-level quality preservation from workload-specific acceleration.

2. Clarification of Table 3 versus Table 7. Original timing records confirm that Table 3 reports the 512-output-token checkpoint (E2E = 40.341 s), while Table 7 reports the 2,048-output-token checkpoint (E2E = 80.347 s) of the same decoding trajectory. Both share the same prefill (TTFT = 27.030 s) and nearly identical TPOT.

3. Context-dependent efficiency. The original 2,048-token timing results are summarized below.

Raw ContextBF16 E2E ↓R²-Omni E2E ↓Speedup ↑
16K29.357 s37.184 s0.790×
32K31.090 s38.653 s0.804×
64K38.196 s41.605 s0.918×
128K60.312 s48.283 s1.249×
256K118.910 s80.347 s1.480×

R²-Omni incurs overhead at shorter contexts (16K–64K) but achieves 1.25× and 1.48× end-to-end speedups at 128K and 256K, respectively, as sequence reduction becomes more beneficial.

Conclusion. We will clarify the output-length distinction, report short-context slowdowns, and distinguish quality retention from efficiency gains measured under separate protocols. Additional timing results are provided in Appendix F.6.

Q1: At matched actual cache bytes, how does SSPA compare with a simpler fixed or attention-ranked value-precision policy using exactly the same STC output, INT4 key format, protected layers, and quantization metadata? What is the incremental quality and calibration cost of the output-residual objective and outer refinement?

Thank you for this suggestion. We conducted two complementary experiments to isolate output-sensitive allocation and iterative refinement.

1. Comparison with simpler allocation policies. As detailed in W1, we compare Uniform, Attention-ranked, and Attention + A/V prior allocation against full SSPA on the Video-SALMONN 2 testset.

All policies share identical STC outputs, INT4 Keys, 32-token Value pages, INT2/INT4/BF16 Value tiers, and 36 decoder layers without protected BF16 layers. The simpler policies use identical actual allocated bytes, including quantization metadata and persistent cache storage.

Full SSPA achieves the highest Correct and lowest EER and Hallucination among the evaluated policies, at comparable final KV-cache footprints. These results support the benefit of output-sensitive allocation over simpler importance-based policies.

2. Inner/outer refinement ablation. We further vary the number of inner allocation rounds and outer layout-refinement passes.

ConfigurationCorrect ↑EER ↓KV (MiB) ↓
Inner 1 / Outer 127.88%86.34%144.0
Inner 4 / Outer 129.53%84.44%147.8
Inner 4 / Outer 2 (Full)30.24%81.27%144.9

Increasing inner rounds from 1 to 4 improves Correct by 1.65 points and reduces EER by 1.90 points. Adding the second outer refinement further improves Correct by 0.71 points and reduces EER by 3.17 points, supporting the contribution of iterative refinement.

3. Calibration cost. Our efficiency measurements include SSPA calibration, allocation, quantization, and packing overhead in TTFT. Thus, the reported end-to-end performance accounts for the operational cost of the full framework, although incremental allocator overhead is not separately isolated.

Conclusion. These experiments support the quality benefits of output-sensitive allocation and iterative refinement, while the incremental benefit of the second outer pass is not statistically established. Full ablations are provided in Appendix F.4.

AuthorsOpenReview ↗

Response to Reviewer omiQ (3/3) : Q2 & Q3

Q2: What input/output lengths and model configuration are used for Table 3's efficiency columns, and why does its E2E differ from Table 7 despite nearly identical TTFT/TPOT? Please identify the exact compression configuration used at each long-context point and report quality at representative 128K/256K settings.

Thank you for this question. We clarify the timing configuration and available long-context quality evidence.

1. Timing configuration. As explained in W2, Tables 3 and 7 use Qwen2.5-Omni-7B with the same 256,021-token input and prefill, but report 512 and 2,048 output tokens, respectively. This explains the E2E difference (40.341 s vs. 80.347 s) despite identical TTFT (27.030 s) and nearly identical TPOT.

2. Compression configuration. The 16K–256K context sweep consistently uses the default STC and SSPA configurations specified in Appendix B, including quantization settings, precision ceilings, protected layers, and allocation parameters, without context-specific tuning.

3. Long-context quality. Our original LongVALE evaluation includes audiovisual inputs with approximately 128K–256K raw multimodal tokens, providing generation-quality evidence at context lengths comparable to our efficiency experiments. On Qwen2.5-Omni-7B, R²-Omni achieves a SODA-C score of 1.71 versus 1.63 for BF16, with 4.94× KV-cache compression.

These clarifications distinguish benchmark-level quality preservation from long-context efficiency measurements. Further timing details are provided in Appendix F.6.

Q3: How were the dataset-specific precision ceilings, protected layers, and the pruning/quantization split for composed baselines selected? Were these choices fixed using separate validation data, and are the direct compositions competitive after tuning that split at the same byte budget?

Thank you for this question. We clarify our configuration selection and memory-budget matching procedure.

1. Configuration selection. For R²-Omni, the STC thresholds and SSPA precision ceilings jointly control structural and numerical compression. We use the configurations documented in Appendix B.3, including shallow/deep Value precision ceilings of 3.6/3.0 bits for Qwen2.5-Omni-3B/7B, with the first two layers protected in BF16 for 7B. MiniCPM-o-4.5 uses architecture- and benchmark-specific settings as detailed in the appendix.

For the combined baselines, we use a small number of representative pilot inputs to calibrate the pruning ratios and quantization parameters toward approximately 5× final KV-cache compression. The resulting configurations are documented in Appendix B.5:

Combined BaselineToken RetentionKV Quantization
OmniZip + KIVI65% per modalityINT2 K/V
OmniSelect + OTT65% per modalityINT2 K/V
SEATS + OScaR70% per modalityINT2 K/V

Since these quantization baselines use fixed INT2 K/V precision, their memory budgets are primarily adjusted through token-retention ratios and quantization-specific parameters, such as residual-cache length. We select representative configurations close to the target compression budget rather than independently optimizing every possible pruning–quantization split.

2. Comparison at comparable memory budgets. As shown in Table 2 and Appendix A.4 (Table 9), we evaluate these direct compositions across Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and MiniCPM-o-4.5. At the evaluated operating points, R²-Omni achieves higher average relative quality than the strongest combined baseline on all three backbones:

BackboneBest Combined BaselineCombined Quality ↑R²-Omni Quality ↑
Qwen2.5-Omni-3BOmniSelect + OTT95.63%99.99%
Qwen2.5-Omni-7BOmniSelect + OTT94.17%100.11%
MiniCPM-o-4.5SEATS + OScaR88.47%93.86%

These results demonstrate the effectiveness of our coordinated structural and precision compression strategy relative to the evaluated direct compositions at approximately comparable memory budgets.

3. Scope of parameter tuning. The pilot examples were used primarily for memory-budget calibration, rather than exhaustive quality-based hyperparameter optimization on a separate validation set. Because the combined baselines employ fixed-precision quantization, our original comparison evaluates one representative operating point near the target budget for each composition.

Conclusion. Our configuration selection targets approximately matched final KV-cache budgets using a small set of pilot inputs, with all evaluated parameters documented in Appendix B. The results consistently favor R²-Omni over the tested direct compositions across three Omni-LLMs.

The pilot budget-selection procedure is summarized in Appendix F.6 of the revised manuscript; the full configurations remain in Appendix B.5.

Reviewer_LhfxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

R²-Omni compresses the input for Omni-LLMs by merging redundant audiovisual tokens. It further employs a mixed-precision allocation strategy for the value cache, leveraging both attention mechanisms and audiovisual alignment information. While reducing memory footprint, this approach outperforms existing benchmarks for token compression and KV quantization across three different Omni-LLM models.

Critical strengths
  1. The method leverages redundancy across two dimensions—sequence length and numerical precision—and employs distinct compression strategies tailored to the unique redundancy structures of audio and video; the motivation is clear.
  2. The experimental scope is comprehensive, featuring comparisons against token compression, KV quantization, and hybrid baselines across three different Omni-LLM models.
Critical weaknesses
  1. The individual components offer limited novelty. STC's similarity-based prototype retention and cross-frame region inheritance resemble video token compression methods such as DyCoke and FrameFusion, while SSPA's importance-guided, budget-constrained bit allocation mirrors mixed-precision KV quantization methods such as ZipCache and MiKV. Moreover, the two stages are applied sequentially with little interaction, which weakens the claimed co-design contribution.
  2. The contributions are not clearly articulated. The authors are encouraged to list them explicitly in the Introduction and clarify how they differ from existing work.
Questions for authors
  1. Region inheritance across frames depends on the similarity of region-mean features, which may miss small local changes. For example, in the scoreboard case in Figure X, the score and pitch count change within a small region while the region mean barely changes. How does STC handle such fine-grained temporal updates? Can the authors report results on OCR- or counting-related questions?
  2. Precision is allocated at prefill using the current query anchors and cannot be recovered once values are quantized to INT2. How does SSPA handle multi-turn or streaming settings where future queries are unknown?
  3. Why are keys uniformly quantized to INT4 while only value precision is allocated? Can the authors provide results with allocation over keys, or over both keys and values?
  4. The combined baselines (e.g., OmniZip + KIVI) are evaluated only on Qwen2.5-Omni-3B (Table 2) and at a single operating point. Could the authors clarify how memory budgets are matched with R²-Omni, and whether its advantage holds across different compression ratios and on the other two models?
  5. Since SSPA allocates value precision at prefill based on the current query anchors, how does R²-Omni perform in multi-turn dialogue, where later questions may focus on different content? Could the authors report how accuracy changes over successive turns compared with the full-precision baseline?
  6. How do memory usage and accuracy change with input length? Could the authors report results across different input durations, to show whether the advantage of R²-Omni holds as inputs grow longer?
8 comments
AuthorsOpenReview ↗

Response to Reviewer Lhfx (1/6) : W1

W1.1: The individual components offer limited novelty. STC's similarity-based prototype retention and cross-frame region inheritance resemble video token compression methods such as DyCoke and FrameFusion, while SSPA's importance-guided, budget-constrained bit allocation mirrors mixed-precision KV quantization methods such as ZipCache and MiKV.

Thank you for pointing out these relevant works. We acknowledge that similarity-based token reduction and importance-guided KV quantization have been explored in prior studies. However, R²-Omni differs in its multimodal compression mechanisms and precision allocation objective. We summarize the key distinctions below.

Comparison. The following table compares the main methodological characteristics of these approaches.

MethodVideo Temporal CompressionAudio ConsolidationMixed-Precision KVA–V Semantic GuidanceOutput-Error-Aware Allocation
DyCoke [1]✓✗✗✗✗
FrameFusion [2]✓✗✗✗✗
ZipCache [3]✗✗✓✗✗
MiKV [4]✗✗✓✗✗
R²-Omni (Ours)✓✓✓✓✓

Analysis. While sharing certain underlying motivations, our approach differs from existing methods in two important aspects:

  • STC vs. DyCoke and FrameFusion. DyCoke combines temporal token merging with dynamic attention-based KV pruning, while FrameFusion integrates cross-frame merging and importance-based pruning. In contrast, STC performs within-frame spatial-region consolidation and cross-frame region inheritance, together with local audio continuity-based merging. Unlike importance-based pruning, STC does not rely on query attention for token removal, aiming to preserve diverse audiovisual information required for long-form generation.
  • SSPA vs. ZipCache and MiKV. ZipCache and MiKV assign different precisions primarily according to token importance. In contrast, SSPA explicitly estimates the reduction in attention-output distortion from each value-precision upgrade under quantized keys. It further incorporates query relevance and audio–visual alignment as a multimodal semantic prior, enabling budget-constrained allocation based on both output sensitivity and semantic evidence.

Therefore, our contribution is not the introduction of similarity-based compression or mixed-precision quantization in isolation, but their task-oriented extension and integration for jointly reducing structural and precision redundancy in Omni-LLMs.

W1.2: Moreover, the two stages are applied sequentially with little interaction, which weakens the claimed co-design contribution.

Thank you for this observation. We agree that STC and SSPA operate sequentially rather than through end-to-end joint optimization. We would like to clarify that our primary contribution is a unified training-free framework for jointly addressing two complementary forms of redundancy, rather than a tightly coupled co-optimization algorithm.

Specifically, STC first reduces structural redundancy while preserving diverse audiovisual information, and SSPA subsequently reduces precision redundancy in the retained representations. These stages serve complementary purposes: STC determines which information to retain, while SSPA determines how precisely to represent it.

Empirical evidence. As demonstrated in Table 2 of our submission, directly combining existing token-pruning and KV-quantization methods does not necessarily achieve comparable performance. At approximately 5× compression, the strongest evaluated naive combination retains 95.63% average relative quality, compared with 99.99% for R²-Omni on Qwen2.5-Omni-3B. These results support the practical benefits of jointly addressing both redundancy dimensions within a unified framework.

We appreciate this suggestion and will clarify our contribution to avoid overstating the degree of algorithmic coupling. Exploring end-to-end trainable joint optimization of token consolidation and precision allocation is a promising direction for future work.

References

[1] Tao et al., DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models, CVPR 2025.

[2] Fu et al., FrameFusion: Combining Similarity and Importance for Video Token Reduction on Large Vision Language Models, ICCV 2025.

[3] He et al., ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification, NeurIPS 2024.

[4] Yang et al., No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization, 2024.

AuthorsOpenReview ↗

Response to Reviewer Lhfx (2/6) : W2 & Q1

W2: The contributions are not clearly articulated. The authors are encouraged to list them explicitly in the Introduction and clarify how they differ from existing work.

Thank you for this constructive suggestion. We agree that the contributions should be more explicitly articulated. We will revise the Introduction to clearly distinguish our approach from existing methods, as summarized in the comparison table in W1.1.

Specifically, we will highlight three contributions:

  1. Problem motivation. We identify the limitations of aggressive query-guided token pruning for long-form generation and motivate reducing structural redundancy while preserving diverse multimodal information.
  2. Structure-aware token consolidation. We propose STC, which consolidates redundant video and audio representations through spatial-region similarity, temporal persistence, and local audio continuity, without relying on query-based token removal.
  3. Semantic-sensitivity guided precision allocation and unified framework. We develop SSPA, which allocates value precision based on attention-output distortion under quantized keys and multimodal semantic guidance. Together, STC and SSPA achieve an average 5.0× memory reduction while retaining 97.98% of baseline generation quality, with a 1.48× end-to-end speedup in our long-context evaluation.

We will incorporate these contributions explicitly into the Introduction and strengthen their distinctions from prior work through the comparison in W1.1.

Q1: Region inheritance across frames depends on the similarity of region-mean features, which may miss small local changes. For example, in the scoreboard case in Figure X, the score and pitch count change within a small region while the region mean barely changes. How does STC handle such fine-grained temporal updates? Can the authors report results on OCR- or counting-related questions?

Thank you for raising this important concern. We agree that region-mean similarity may overlook subtle local changes, particularly when only a small portion of a region changes between adjacent frames. To investigate whether STC preserves fine-grained visual information, we conducted an additional OCR evaluation.

Setting. We construct 50 OCR questions from Video-SALMONN 2. All methods use Qwen2.5-Omni-3B with 128 sampled frames, complete audio inputs, and identical decoding settings. We compare BF16, STC-only, OmniZip at a matched cache budget, and the full R²-Omni framework.

Results.

MethodExact Match ↑CER ↓Avg. KV ↓
BF1668% (34/50)19.08%853.7 MiB
STC-only64% (32/50)20.10%467.6 MiB
OmniZip56% (28/50)21.92%467.7 MiB
Full R²-Omni64% (32/50)21.17%139.0 MiB

Analysis. At nearly identical cache budgets , STC-only achieves better quality than OmniZip. This suggests that structural consolidation can preserve fine-grained textual evidence more effectively than token pruning at the tested operating point.

Furthermore, the full R²-Omni framework maintains the same 64% exact-match accuracy as STC-only while reducing KV storage to 139.0 MiB, corresponding to approximately 6.14× compression relative to BF16.

Conclusion. These results provide additional evidence that STC can retain fine-grained visual information under substantial compression, particularly compared with token pruning at matched cache budgets. Nevertheless, preserving small cross-frame updates remains a challenging case, and we will explicitly acknowledge this limitation in the revised manuscript.

The OCR results are included in Appendix F.1 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Lhfx (3/6) : Q2

Q2: Precision is allocated at prefill using the current query anchors and cannot be recovered once values are quantized to INT2. How does SSPA handle multi-turn or streaming settings where future queries are unknown?

Thank you for raising this important question. We acknowledge that multi-turn and streaming inference are not explicitly targeted by our current design. Our primary objective is to reduce structural and precision redundancy in long-form generation, rather than dynamically optimize cache precision for unknown future queries.

1. Limitation and potential extension. Since SSPA allocates precision based on the query anchors available during prefill, its allocation may not be optimal for subsequent queries. Moreover, information lost through INT2 quantization cannot be directly recovered. One possible extension is to maintain a full-precision copy of the historical KV cache in secondary memory (e.g., CPU memory) and dynamically re-quantize relevant pages according to the query anchors of each subsequent turn. This would enable adaptive precision allocation at the expense of additional storage and data-transfer overhead.

2. Additional multi-turn evaluation. Although multi-turn inference is not our primary target, we conducted an additional experiment using 50 Video-SALMONN 2 videos with three successive dialogue turns on Qwen2.5-Omni-3B. The first turn processes 128 sampled frames and complete audio, while subsequent turns reuse the existing KV cache without reprocessing the multimodal inputs.

R²-Omni achieves Correct scores of 26.00%, 17.65%, and 16.62% across three turns, compared with 25.67%, 15.70%, and 19.40% for BF16. The three-turn average Correct is 20.09% versus 20.26% for BF16, while achieving 5.81× KV-cache compression after the third turn. These results demonstrate that the compressed cache can support subsequent queries without restoring the original full-precision representations, although some quality degradation occurs in the third turn. Detailed comparisons are provided in our response to Q5.

3. Comparison with token pruning. Unlike query-guided token pruning methods that discard tokens based on their relevance to the current query, STC consolidates structurally redundant representations without explicitly removing query-irrelevant content. This design aims to preserve a broader range of multimodal evidence for future queries. Consistent with this motivation, R²-Omni achieves higher Correct scores than OmniZip across all three evaluated turns while using substantially less KV memory, although both consolidation and quantization remain lossy operations.

Conclusion. Our additional experiments provide initial evidence that R²-Omni can support persistent-KV multi-turn generation with substantial memory savings. Nevertheless, adaptively reallocating precision for unknown future queries remains an open direction, particularly for longer dialogues and streaming inference.

The multi-turn results are included in Appendix F.2 of the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to ACs and All Reviewers

Dear ACs and Reviewers,

We sincerely thank the ACs and all reviewers for their time, thoughtful evaluations, and constructive feedback on our submission. We are encouraged that Reviewer Lhfx recognizes that our approach addresses redundancy across both sequence length and numerical precision, noting that "the motivation is clear" and "the experimental scope is comprehensive". Reviewer omiQ highlights the "meaningful evaluation gap" between multiple-choice understanding and long-form generation, considers our evaluation "a useful empirical contribution beyond conventional multiple-choice compression evaluations", and finds our system-level measurements "practically more convincing than a storage-only analysis". Reviewer T776 acknowledges that "the evaluation is complete for such a compression paper", particularly appreciating the diverse Omni-LLM backbones, long-form benchmarks, compression-ratio sweeps, and end-to-end efficiency measurements that include calibration and allocation overhead.

We have carefully addressed the reviewers' concerns through additional controlled experiments, ablation studies, and detailed clarifications of our methodology and evaluation protocols. These include further analyses of structural consolidation, precision allocation, multi-turn robustness, input-length scaling, memory-budget matching, and inference efficiency. We provide point-by-point responses to all comments below.

Best regards,

Authors of Submission 12360

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.