- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper targets the KV-cache and prefill costs of omni-modal LLMs when the output is long-form generation (dense captioning, event narration) rather than multiple choice. The authors' starting observation is that existing query-guided token pruning keeps multiple-choice accuracy high but degrades open-ended generation, because tokens that the initial query ignores are still needed later in generation. They split the redundancy in the input into “structural” (overlapping information across tokens) and “precision” (unnecessarily high bit-width of cached values), and attack both with a training-free two-stage framework. Stage one, Structure-Aware Token Consolidation (STC), merges video tokens within and across adjacent frames using cosine-similarity regions (union-find over spatial neighbors, centroid prototype per region, frame-mean test to inherit regions) and merges locally similar adjacent audio token pairs. Stage two, Semantic-Sensitivity Guided Precision Allocation (SSPA), keeps keys at channel-wise INT4 and assigns value precision (INT2/INT4/16-bit) per 32-token page by greedy rounds that maximize the attention-output error removed per extra bit, using attention from the final prefill queries and an audio–video alignment score as a semantic prior on top.
Critical strengths
The evaluation is complete for such a compression paper: three Omni-LLMs with quite different audio-visual front-ends, three long-form benchmarks, compression-ratio sweeps up to 9×, naive prune-then-quantize compositions as baselines (Table 2), and a long-context efficiency study (16K–256K) whose latency numbers include the calibration and allocation overhead rather than hiding it.
Critical weaknesses
-
The headline quality number rests on an aggregation that needs justification. EER improves as compression gets more aggressive (85.66 for the 3B baseline vs. 95.65 under OmniSelect) while Correct drops by nine points, so “average relative quality” mixes metrics that move in opposite directions, and the “97.98% of baseline quality” claim is hard to interpret. More importantly, for a paper whose central claim is about generation quality, there is no human evaluation and no LLM-as-judge study of the actual generated long-form text anywhere in the paper or appendix.
-
The advantage over the strongest KV-quantization-only baseline is thin on two of the three backbones. On Qwen2.5-Omni-7B the method reaches 100.11 average relative quality versus 99.19 for OTT, and on MiniCPM-o-4.5 93.86 versus 93.03 for OScaR. On both backbones the pruning-based baselines collapse for long-form generation, so the headline “outperforms all baselines” largely reduces to “KV quantization beats token pruning on this task” — which Table 1 already shows for KIVI/OTT/OScaR by themselves. What the structural consolidation stage specifically contributes beyond a larger compression ratio on a strong backbone is not demonstrated; the ablation in Table 3 suggests its main effect is ratio, not quality.
Questions for authors
-
On Qwen2.5-Omni-7B, is the gap to OTT (100.11 vs. 99.19 average relative quality) statistically meaningful? Per-benchmark numbers with variance would clarify whether STC contributes quality on this backbone or only compression ratio.
-
How does STC behave on short inputs (e.g., a few dozen frames)? The frame-pair persistence mechanism and spatial region construction seem designed for long videos; does consolidation still help when prefill is already cheap?
3 comments
Response to Reviewer T776 (2/3) : W2
W2: The advantage over the strongest KV-quantization-only baseline is thin on two of the three backbones. What does the structural consolidation stage specifically contribute beyond a larger compression ratio?
Thank you for this insightful observation. We would like to clarify that STC primarily contributes by reducing structural redundancy, complementing the precision reduction achieved by KV quantization. Its benefits extend beyond KV-cache compression to reduced prefill computation and a potentially more efficient precision allocation space.
1. Additional memory reduction. To better characterize the contribution of STC, we conducted an additional comparison on a fixed subset of 100 Video-SALMONN 2 videos using Qwen2.5-Omni-7B. SSPA-only retains the original input tokens while applying mixed-precision allocation, whereas R²-Omni enables both STC and SSPA.
| Method | Correct ↑ | EER ↓ | Hallucination ↓ | Allocated KV ↓ |
|---|---|---|---|---|
| BF16 | 39.10% | 78.76% | 17.86% | 1344.6 MiB |
| SSPA-only | 39.78% | 69.73% | 9.51% | 453.2 MiB |
| R²-Omni | 38.47% | 78.93% | 17.40% | 274.5 MiB |
Analysis. Compared with SSPA-only, R²-Omni further reduces allocated KV storage by 39.43%, achieving approximately 4.90× compression relative to BF16, compared with 2.97× for SSPA-only. We acknowledge that SSPA-only achieves better quality at its larger memory budget. Therefore, this experiment demonstrates the additional compression benefit of STC and the resulting memory–quality trade-off, rather than an unconditional quality improvement.
2. Coupling between structural and precision redundancy. Importantly, structural and precision redundancy are not entirely independent. Without STC, structurally redundant tokens still occupy KV-cache capacity and participate in page-wise precision allocation. By consolidating similar tokens, STC reduces the number of representations competing for a limited precision budget, potentially allowing SSPA to allocate higher precision to retained, information-distinct tokens under the same memory constraint. Thus, STC not only reduces sequence length but also reshapes the precision allocation space. This interaction provides an additional motivation for jointly addressing structural and precision redundancy, although the present experiment does not independently establish a quality gain from this coupling at matched memory budgets.
3. Prefill efficiency. Beyond cache storage, STC reduces the number of tokens processed by the LLM, a benefit that KV quantization alone cannot directly provide. As shown in Table 3, the original 256K-context efficiency ablation reports:
| Method | TTFT (s) ↓ | E2E (s) ↓ |
|---|---|---|
| SSPA-only (w/o STC) | 58.54 | 78.86 |
| R²-Omni | 27.03 | 40.34 |
Under this 512-token generation setting, R²-Omni achieves a 2.17× TTFT speedup and 1.95× E2E speedup relative to SSPA-only. These results demonstrate that STC provides substantial inference efficiency benefits by shortening the sequence processed during prefill, beyond what KV-cache quantization alone can achieve.
Conclusion. We appreciate the reviewer's suggestion and clarify that STC is not intended to universally improve generation quality over KV quantization alone. Rather, STC complements SSPA in three aspects: (1) reducing sequence length and prefill computation, (2) enabling additional KV-cache compression, and (3) reducing structural redundancy in the representations subject to precision allocation, potentially improving the utilization of a fixed precision budget. Together, these complementary roles motivate our joint design for balancing generation quality, memory consumption, and long-context inference efficiency.
The STC-versus-SSPA-only comparison is included in Appendix F.3 of the revised manuscript.
Response to Reviewer T776 (3/3) : Q1 & Q2
Q1: On Qwen2.5-Omni-7B, is the gap to OTT (100.11 vs. 99.19 average relative quality) statistically meaningful? Per-benchmark numbers with variance would clarify whether STC contributes quality on this backbone or only compression ratio.
Thank you for this important question. We agree that the relatively small improvement over OTT requires statistical validation beyond the reported point estimates.
Setting. We consider three paired evaluation runs on Qwen2.5-Omni-7B under identical evaluation configurations and comparable compression budgets. We report the mean and standard deviation of average relative quality and assess the differences using a two-sided paired t-test.
Results. The three-run comparison is summarized below.
| Method | Seed 1 | Seed 2 | Seed 3 | Mean ± SD |
|---|---|---|---|---|
| OTT | 99.53 | 99.19 | 98.76 | 99.16 ± 0.39 |
| R²-Omni | 100.07 | 100.11 | 99.89 | 100.02 ± 0.12 |
Analysis. In this comparison, R²-Omni achieves a mean improvement of +0.86 percentage points over OTT. The paired t-test yields with a two-sided , and the 95% confidence interval for the paired difference is approximately percentage points. These results support a statistically significant advantage over OTT at the 0.05 level.
The paired-run comparison and statistical test are included in Appendix F.5 of the revised manuscript.
Q2: How does STC behave on short inputs (e.g., a few dozen frames)? The frame-pair persistence mechanism and spatial region construction seem designed for long videos; does consolidation still help when prefill is already cheap?
Thank you for this valuable suggestion. We conducted additional experiments with 16 and 32 sampled frames to examine the effectiveness of structural consolidation on short visual inputs.
Setting. We evaluate Qwen2.5-Omni-3B on Video-SALMONN 2 , preserving the complete audio inputs and identical decoding configurations while varying only the sampled frame count. We compare full R²-Omni against BF16 for generation quality and KV-cache usage, and separately evaluate STC-only against BF16 to isolate its latency contribution. All compression configurations follow the default settings specified in Appendix B, without additional tuning for different frame counts.
1. Generation Quality and Memory.
| Frames | Method | Correct ↑ | EER ↓ | Hall. ↓ | KV Memory ↓ |
|---|---|---|---|---|---|
| 16 | BF16 | 24.93% | 85.66% | 10.59% | 162.1 MiB |
| R²-Omni | 24.99% | 84.27% | 9.26% | 42.1 MiB | |
| 32 | BF16 | 26.77% | 82.68% | 9.45% | 262.0 MiB |
| R²-Omni | 27.83% | 80.81% | 8.63% | 57.3 MiB |
At 16 and 32 frames, R²-Omni achieves approximately 3.85× and 4.57× KV-cache compression, respectively. Meanwhile, Correct increases slightly by +0.06 and +1.06 percentage points, with lower EER and hallucination rates at both settings. These results indicate that the full framework can achieve substantial memory savings without an observed quality degradation on the tested few-frame inputs.
2. Inference Efficiency. To isolate the runtime contribution of STC, we additionally compare STC-only against BF16.
| Frames | BF16 TTFT ↓ | STC-only TTFT ↓ | BF16 E2E ↓ | STC-only E2E ↓ | Speedup ↑ |
|---|---|---|---|---|---|
| 16 | 457.3 ms | 430.8 ms | 2580.0 ms | 2476.5 ms | 1.042× |
| 32 | 804.5 ms | 733.4 ms | 2845.0 ms | 2769.4 ms | 1.027× |
STC reduces time to first token by 5.8% and 8.8% at 16 and 32 frames, respectively, while achieving modest end-to-end speedups of 1.042× and 1.027×. This demonstrates that structural consolidation can provide measurable inference benefits even when prefill is relatively inexpensive.
Conclusion. STC remains effective with only a few dozen sampled frames, contributing to substantial KV-cache reduction and modest inference acceleration while maintaining competitive generation quality. These experiments vary the frame-sampling density of the same videos rather than video duration, and therefore specifically establish effectiveness under sparse visual sampling.
The few-frame quality, memory, and latency results are included in Appendix F.3 of the revised manuscript.
Response to Reviewer T776 (1/3) : W1
Thank you for highlighting the need for clearer quality evidence. We would like to clarify the evaluation metrics, aggregation procedure, and existing LLM-based evaluation protocol, while highlighting the qualitative evidence already provided in our appendix.
1. Metric clarification. First, we clarify that EER (Event Error Rate) is a lower-is-better metric. Therefore, its increase from 85.66% to 95.65% under OmniSelect indicates deterioration rather than improvement, consistent with the decrease in Correct from 24.45% to 15.18%.
Following the Video-SALMONN 2 evaluation protocol, let C, M, I, and H denote correctly described, missed, incorrect, and hallucinated events, respectively. With N = C + M + I, the metrics satisfy:
\mathrm{EER}=\frac{M+I+H}{N} =1-\mathrm{Correct}+\frac{H}{N}.$$ **2. Aggregation clarification.** The average relative quality reported in Table 1 is computed by normalizing each metric against its corresponding BF16 baseline. For higher-is-better metrics (Correct, SODA-C, and SODA-M), we use the ratio of the compressed method to BF16; for the lower-is-better EER, we use the inverse ratio. The four normalized metrics are then averaged with equal weighting across the three model configurations.We acknowledge that this aggregate is intended as a summary of performance across heterogeneous benchmarks, rather than a standalone measure of generation quality. **Excluding EER from the aggregation still yields approximately 97.86% average relative quality**, compared with the originally reported 97.98%. **3. LLM-as-Judge evaluation and case studies.** We would like to clarify that **Video-SALMONN 2 already employs an official LLM-based event evaluator**, as described in Appendix C. This evaluator directly assesses the generated long-form descriptions against annotated audiovisual events, identifying correctly described, missed, incorrect, and hallucinated content. Therefore, our evaluation already includes LLM-based assessment of actual generated text, rather than relying solely on conventional similarity metrics. Furthermore, **Appendix E (Figures 7–8)** presents qualitative case studies comparing the generated descriptions across methods. These examples illustrate how R²-Omni preserves fine-grained visual details, dialogue-related information, and audiovisual event consistency under compression. In particular, Figure 8 provides an event-level comparison of unsupported descriptions, illustrating the preservation of factual consistency. **Conclusion.** We will clarify the direction and aggregation of evaluation metrics, explicitly highlight the existing LLM-as-Judge protocol in the main text, and better connect the quantitative results with the qualitative evidence in Appendix E. These clarifications provide a more transparent assessment of long-form generation quality without relying solely on the aggregate relative-quality score. The metric definitions and aggregation check are summarized in **Appendix F.5** of the revised manuscript.