acceptodds
Under review as a conference paper at ICLR 2027

RABBIT: Dynamic Budget Allocation Token Compression for Omni-modal LLMs

Abstract

Omnimodal large models integrate video, audio, and text modalities and have shown strong cross modal capabilities for real world interactive applications. However, long audio and video inputs introduce a large number of tokens, making inference expensive and hindering the deployment of omni modal large models in practical applications. Existing compression methods mainly rely on heuristic token importance metrics and fixed audio and video compression ratios, which often fail under aggressive compression and overlook sample level modality differences. We propose a reinforcement learning based audio and video token compression method RABBIT (Reinforcement Learning for Adaptive Budget Balancing in Token Compression) that learns a unified ranking of task relevant audio and video tokens for each sample. Our method uses modality specific scoring modules and joint audio and video rollout sampling to adaptively allocate the compression budget between audio and video while preserving full token model behavior through teacher logit distillation. Experiments on multiple audio and video understanding benchmarks show that our method outperforms the evaluated baselines under different compression ratios. On WorldSense, it preserves of full token performance with only audio and video tokens and outperforms DivPrune and SEATS by and percentage points, respectively. Further analysis shows that the learned sample level audio and video budget allocation is effective, improving SEATS by points on WorldSense and points on VideoMME at the retain ratio.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Omni models, token compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nXipOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

When processing long‑video and long‑audio inputs, multimodal large‑language models suffer from an explosion of audio‑visual tokens, which leads to prohibitively high inference costs. To address the issue that existing compression methods adopt fixed audio‑video retention ratios and fail to adapt to the modality‑wise information distribution across different samples, resulting in rapid performance degradation under heavy compression, this work proposes a reinforcement‑learning‑based adaptive budget‑balancing framework. It designs two independent MLP scoring modules for audio and video respectively, which score audio‑visual tokens with global difference features of each modality. A reward function weighted by compression reward and distillation reward is constructed via rollout sampling, and within‑sample reward normalization is adopted to stabilize policy‑gradient training. The framework learns a sample‑specific unified ranking list for audio and visual tokens. During inference, deterministic Top‑k truncation is directly performed, and a single learned ranking supports multiple compression ratios. Extensive experiments are conducted on multiple benchmarks, showing prominent advantages under high compression ratios such as 10%. Ablation experiments verify the effectiveness of the dual‑branch MLP and global difference features.

Critical strengths
  1. Long audio‑video inputs incur a large token count and substantial inference cost. Sample‑level adjustment of audio and video budgets constitutes a reasonable improvement over methods with fixed modality ratios.

  2. The dual‑branch MLP combined with modality‑mean difference features avoids explicit attention‑map computation or pairwise token similarity calculation. Only scoring and top‑k selection are required at inference time.

Critical weaknesses
  1. This method combines reinforcement learning with dynamic audio‑video budgets and compares itself with existing token pruning and modality‑adaptive approaches. It is necessary to better situate its relationship with recent works on learned pruning and dynamic modality routing.

  2. There exists potential distribution and objective misalignment between training and evaluation. The policy is trained via teacher logit distillation on caption‑based datasets, while the downstream target tasks cover world‑knowledge reasoning, video question answering and inference. Teacher‑logit fidelity is not equivalent to downstream task accuracy, and stronger evidence is required to verify cross‑task generalization.

Questions for authors
  1. Report the net cost of RABBIT’s policy training relative to inference‑side savings, including training FLOPs, wall‑clock training time, GPU memory footprint, and checkpoint size.

  2. Add state‑of‑the‑art baselines of learned pruning and dynamic modality routing under the same backbone, identical input token budget and unified evaluation protocol.

  3. Illustrate the impact on performance when disabling SEATS’s intra‑layer / cross‑layer compression modules.

  4. Investigate whether the ranking learned via caption‑based distillation remains effective on unseen tasks, datasets, or different audio‑video lengths. Furthermore, compare it against models trained directly with downstream‑task rewards.

  5. Analyze the discrepancy between deterministic top‑k truncation at inference and Bernoulli rollout used in training, and verify whether a single stable ranking is truly shared across different target retention rates.

5 comments

Part 1/2

We thank the reviewer for the constructive comments. We address the requested comparisons, training cost, transfer evidence, and train–inference consistency below.

W1 / Q2. Relation to learned pruning and dynamic modality routing

To address the reviewer's concern about learned pruning and dynamic modality routing methods, we add OmniSIFT, EchoingPixels, and VisionSelector as baselines to show the advantage of our method.

VisionSelectorOmniSIFTEchoingPixelsRABBIT
Modalities (original method)VA+VA+VA+V
Dynamic A/V allocationNo (10% each)NoYesYes
TrainingSFTSFTSFTRL
Updates LLMNoYesYesNo
DailyOmni Acc. (%)51.2949.7151.4653.38
WorldSense Acc. (%)41.5537.7039.1642.69
Prefill TFLOPs (DailyOmni)7.076.8516.906.86

RABBIT is 1.92–3.67 points higher on DailyOmni and 1.14–4.99 points higher on WorldSense than all three, at comparable or lower prefill computation. It also outperforms EchoingPixels, the only baseline with dynamic allocation, which fine-tunes the LLM and needs 2.5× more prefill computation. This shows that RL-trained joint selection on a frozen backbone outperforms these learned pruning and dynamic-allocation methods, whether or not they fine-tune the LLM.

W2 / Q4. Caption supervision and downstream generalization

To justify the need for caption supervision and to show that our method generalizes, we argue from both a heuristic and an empirical perspective.

Our reason for training on captions is that a caption covers the entire audio-video content, whereas a QA reward only covers the part relevant to one question, so caption distillation should better teach the policy to preserve all multimodal information.

To verify this, we trained a policy with a direct task reward on 8,000 multiple-choice questions from OmniVideo-100K and compared it with caption-trained RABBIT. We compare these two method with DailyOmni and LVOmniBench under 10% retention. The results are listed below.

Training dataRewardDailyOmniLVOmniBench
OmniVideo-100K QACompression + KL50.4633.63
AVoCaDO captionsCompression + KL53.3834.32

Although it never sees the QA task during training, caption-trained RABBIT is 2.92 points higher on DailyOmni and 0.69 points higher on LVOmniBench, whose inputs are up to four times longer than the 64 frames used in training. This supports our design: caption distillation teaches the policy to preserve the overall audio-video information, and in our setting it transfers to unseen tasks, datasets and longer inputs better than direct QA supervision, without needing task labels.

Q1. Training cost and inference savings

To quantify the net cost, we measured the training of the Qwen2.5-Omni-3B policy used in all experiments (8K samples, 1,000 updates) and its inference savings at 10% retention.

MetricValue
Training FLOPs5.38 × 10¹⁸
Wall-clock time / GPU-hours4.73 h / 37.88
Peak GPU memory per GPU18.13 GiB
Trainable parameters / checkpoint size4.20M / 8.00 MiB
Prefill TFLOPs per query, full → 10% (DailyOmni)73.79 → 6.86
Latency per query, full → 10% (WorldSense)2.477 → 1.504 s

Dividing the training cost by the per-query saving gives a break-even point of about 80K queries by FLOPs and 140K by GPU time. The one-time cost is thus recovered after roughly 80K–140K queries, after which every query saves 90.7% of prefill computation and 39.3% of latency.

Part 2/2

Q3. Disabling SEATS's intra- and cross-layer modules

To isolate the effect of SEATS's intra-layer pruning (Stage II) and cross-layer removal (Stage III), we ran the official SEATS code on Qwen2.5-Omni-3B with the stages enabled incrementally, and also attached Stages II+III to RABBIT. All runs use identical DailyOmni inputs.

MethodInput AV retentionPrefill TFLOPsAcc. (%)
SEATS-I10.02%6.8545.95
SEATS I+II10.02%5.3645.45
SEATS I+II+III10.02%5.1945.45
Full SEATS (official 10% layer-mean target)14.04%6.9448.54
RABBIT10.00%6.8653.38
RABBIT + SEATS II+III10.00%5.2053.47

Disabling Stages II and III does not reduce accuracy (45.95 vs. 45.45 with both enabled). At similar computation, full SEATS is 4.84 points below RABBIT, and adding Stages II+III to RABBIT cuts prefill computation by about 24% without accuracy loss. RABBIT's gains therefore come from better input selection, which is complementary to SEATS's in-LLM stages.

Q5. Bernoulli training and deterministic top-k

The reviewer asks whether replacing Bernoulli rollouts with deterministic top-k at inference creates a mismatch, and whether one ranking is truly shared across retention rates. We therefore (i) compared deterministic top-k with Gumbel-top-k, which samples exactly k tokens from the same keep probabilities, at 10% retention, and (ii) ran RABBIT separately at 10%, 15% and 25% retention and computed the Spearman correlation between the scores of all audio-video tokens before selection.

SelectionDailyOmni Acc. (%)
Gumbel-top-k (seeds 42 / 43 / 44)52.13 / 52.21 / 52.05
Deterministic top-k53.38

Top-k outperforms every sampled run. This is expected: for a fixed k, the Bernoulli log-probability of a mask is a constant plus the sum of the selected tokens' log-odds, so top-k is the most probable k-token mask under the trained policy.

Retention rates compared10% vs. 15%10% vs. 25%15% vs. 25%
Spearman ρ (all AV tokens, before selection)1.001.001.00

The token rankings at the three retention rates are identical on every question (ρ = 1.00), because the scorer does not take the budget as input.

Hence, (i) deterministic top-k deploys the most probable mask of the Bernoulli-trained policy and loses no accuracy relative to sampling from it, and (ii) a single stable ranking is computed once and shared by all target retention rates, which only change where it is cut.

We believe these results address the reviewer's concerns and would be grateful if the reviewer would consider raising the score.

Reviewer_nXipOpenReview ↗

In summary, the authors have addressed my requests by adding experiments, but the empirical evidence is not yet strong enough to fully convince me on the advantages of their design. I will keep my original score.

  1. For W1/Q2: While the authors have added OmniSIFT, EchoingPixels and VisionSelector for comparison, the accuracy gaps against these baselines are relatively modest. The comparison only covers one fixed retention ratio (10%). It is still unclear whether the observed behaviour holds across different retention budgets.
  2. For Q1: The break-even analysis is informative, but it relies heavily on the assumed production query volume. The practical value of this trade-off remains application-dependent and is not sufficiently discussed.
Reviewer_tXc7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RABBIT, a reinforcement-learning-based method for audio and video token compression in omni-modal large language models. It uses modality-specific MLPs to score tokens and jointly ranks them to obtain sample-dependent audio and video budgets. Training combines a compression reward with a teacher-logit distillation reward while keeping the backbone frozen. At inference time, deterministic top-k selection supports different token retention ratios. Experiments on Qwen2.5-Omni-3B and 7B report improved accuracy over the evaluated baselines across four audio-video QA benchmarks.

Critical strengths
  1. The paper is readable and generally easy to follow. The overall framework and training procedure are clearly described.

  2. The motivation is reasonably clear. Different samples require different amounts of audio and visual information, making adaptive modality-budget allocation a sensible direction.

Critical weaknesses
  1. The backbone coverage is too limited. Experiments are restricted to Qwen2.5-Omni-3B and 7B, which belong to the same model generation and architecture. For an ICLR 2027 submission, experiments on Qwen3-Omni-30B-A3B are needed to establish applicability to a newer Omni model with a different architecture.

  2. The baseline comparison is insufficiently comprehensive. Missing methods include general visual token compression approaches such as VisionZip [1] and SparseVLM [2], video-oriented methods such as DyCoke [3], VidCom2 [4], and HoliTom [5], and the omni-modal methods DASH [6] and OmniSIFT [7]. Comparisons should cover both accuracy and efficiency under clearly specified, comparable compression budgets.

  3. The availability of the training data raises reproducibility concerns. RABBIT trains on 8K caption examples from AVoCaDO [8], but I could not find a public release of the corresponding training dataset. The authors should explain this and the detailed data selection procedure.

  4. Evaluation on long audio-video understanding is insufficient. The authors should include LVOmniBench [9] and VideoOdyssey-AV [10], which would provide more direct evidence of whether compression preserves the information needed for reasoning across extended audio-visual contexts.

  5. The task coverage is restricted to multiple-choice QA. Temporal grounding on LongVALE [11] and omni-modal captioning on AVSCapBench [12] would test whether the method preserves temporal evidence and detailed audio-visual information. These evaluations are needed to substantiate broader applicability.

[1] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. VisionZip: Longer is Better but Not Necessary in Vision Language Models. CVPR, 2025.

[2] Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A. Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference. ICML, 2025.

[3] Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models. CVPR, 2025.

[4] Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models. EMNLP, 2025.

[5] Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. HoliTom: Holistic Token Merging for Fast Video Large Language Models. NeurIPS, 2025.

[6] Bingzhou Li and Tao Huang. DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression. ECCV, 2026.

[7] Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, Yuanxing Zhang, Jiaheng Liu, Qiang Liu, Pengfei Wan, and Liang Wang. OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models. ICML, 2026.

[8] Xinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua, Linli Yao, Yang Shi, Bozhou Li, Yuanxing Zhang, Qiang Liu, Pengfei Wan, Liang Wang, and Tieniu Tan. AVoCaDO: An Audiovisual Video Captioner Driven by Temporal Orchestration. ICLR, 2026.

[9] Keda Tao, Yuhua Zheng, Jia Xu, Wenjie Du, Kele Shao, Hesong Wang, Xueyi Chen, Xin Jin, Junhan Zhu, Bohan Yu, Weiqiang Wang, Jian Liu, Can Qin, Yulun Zhang, Ming-Hsuan Yang, and Huan Wang. LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs. arXiv:2603.19217, 2026.

[10] Haichen He, Jiayi Zhou, Sifeng Shang, Yihan Hu, Yuanhan Zhang, and Kaiyang Zhou. VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding. arXiv:2605.22907, 2026.

[11] Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng. LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware Omni-Modal Perception of Long Videos. CVPR, 2025.

[12] Yanghai Wang, Jiahao Wang, Jiafu Tang, Yuanxing Zhang, Zhe Cao, Hanyan Bian, Zijie Zhang, Weiliang Luo, Zhiyu Pan, Zixuan Dong, Jiaheng Liu, and Zhaoxiang Zhang. AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning. arXiv:2607.12820, 2026.

Questions for authors

Could the authors demonstrate the general applicability of the proposed method through additional experiments across different model architectures, long audio-video benchmarks, and tasks beyond multiple-choice QA?

4 comments

Part 1/2

We thank the reviewer for the constructive comments. We have run new experiments for all concerns to support our claims.

W1. Applicability to Qwen3-Omni-30B-A3B

To show that our method generalizes across backbones, we followed the reviewer's suggestion and retrained it on Qwen3-Omni-30B-A3B, then evaluated it on DailyOmni at 10% retention. The results are as follows:

MethodAV retentionDailyOmni Acc. (%)
Full-token100%72.85
SEATS-I10%57.73
RABBIT10%58.65

RABBIT again outperforms SEATS-I at the same 10% budget, by 0.92 points. This shows that RABBIT's training recipe carries over to a newer MoE omni-modal backbone without modifying the model.

W2. Additional baselines under comparable budgets

To broaden the accuracy–efficiency comparison, we evaluated the requested methods with RABBIT on Qwen2.5-Omni-3B and all 1,197 DailyOmni questions.

MethodUpdates LLMAV retentionAccuracy (%)Prefill TFLOPs*
VisionZip (V + random A / official A+V)No9.89%44.11 / 43.616.74
VidCom²†No10.01%42.11 / 42.276.81
HoliTom†No10.15%42.61 / 43.366.91
HoliTom, two-stage†No10.15% (8.51% in-LLM)42.27 / 44.115.94
SparseVLM v1.5 + SEATS-I audio§No90.48% input; 10.00% final; 22.19% layer-mean48.6215.32¶
DASH, strict-budget adaptation§No10.01%42.696.82¶
DyCoke (TTM)‡No26.69%43.11—
OmniSIFTYes10.05%49.716.85
EchoingPixelsYes10.00%51.4616.90
RABBITNo10.00%53.386.86

*Analytical FLOPs of decoder prefill and token selection; encoders and decoding excluded. †Video-only method, paired with random / SEATS-I audio selection. ‡DyCoke's temporal merging always keeps the first frame of each four-frame group, so it cannot go below about 25% retention. ¶Partial estimates: decoder prefill plus counted selection matrix multiplications

RABBIT achieves the highest accuracy, 1.92–11.27 points above the evaluated baselines, including 4.76 points above SparseVLM and 10.69 above the DASH adaptation. These results support RABBIT's competitive accuracy–efficiency trade-off on this benchmark while keeping the backbone frozen, with the adaptation and budget differences disclosed above.

W3. Training-data availability and subset selection

The training data come from the public AVoCaDO-Captioner/training_set, and the 8K samples we used are included in the supplementary material.

W4. Long audio-video understanding

To demonstrate the effectiveness of our method on long audio-visual understanding, we run experiments on LVOmniBench and VideoOdyssey-AV at 10% retention; the results are shown below.

MethodLVOmniBenchVideoOdyssey-AV
SEATS-I32.8422.22
RABBIT34.3223.73

RABBIT is 1.48 points above SEATS-I on LVOmniBench and 1.51 points on VideoOdyssey-AV. On LVOmniBench, the learned policy keeps its advantage on inputs four times longer than those seen in training, showing that it preserves the evidence needed for long-context reasoning.

Part 2/2

W5. Temporal grounding and captioning

To test whether compression preserves detailed audio-visual information beyond multiple-choice QA, We evaluated LongVALE-TVG and AVSCapBench under 10% retention.

The results of LongVALE-TVG are listed below.

MethodmIoU[email protected][email protected][email protected]
Full6.408.584.091.83
SEATS-I 10%5 116.583.201.41
RABBIT 10%6.178.133.951.78

At 10% retention, RABBIT keeps 96% of the full-token mIoU versus 80% for SEATS-I and is higher across all three recall thresholds.

The results of AVSCapBench are listed below.

MethodAV retentionVisualAudioSynergyOverall recall
Full-token100%26.6911.174.4015.87
SEATS-I10%16.035.111.238.65
RABBIT10%18.9010.153.5312.05

At 10% retention, RABBIT keeps 76% of the full-token overall recall versus 55% for SEATS-I and is higher on all three event types.

This shows that the learned allocation preserves detailed audio-visual content in temporal grounding and open-ended captioning, not only the evidence needed for multiple-choice answers.

We believe these results, together with the clarified data provenance, support reconsideration of the score.

Thank you to the authors for the thorough rebuttal, additional experiments, and detailed clarifications. Most of my concerns have been satisfactorily addressed. Please make sure to incorporate these new results and discussions into the revised manuscript. Accordingly, I have raised my rating.

Dear Reviewer,

Thank you for your thoughtful feedback and for raising your rating. We will include all the new results and discussions from the rebuttal in the revised manuscript.

Best regards, The Authors

Reviewer_9LCCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RABBIT, a reinforcement learning method that dynamically allocates audio and video token budgets to improve the efficiency of omni-modal large language models.

Critical strengths
  1. The paper is well-structured and easy to follow.
  2. This paper is the first to introduce reinforcement learning into audio-video token compression and demonstrates its effectiveness.
Critical weaknesses
  1. The paper lacks comparisons with relevant baselines. EchoingPixels[1], OmniSIFT[2] and VisionSelector[3] are both highly related works, but neither is included in the experiments.
  2. In my view, reward design is a core component of reinforcement learning. However, this work does not ablate the two reward terms or analyze the choice of their relative weights.
  3. The paper does not provide reward curves over training, making it difficult to assess the stability and convergence of the learned policy. [1] Gong C, Wang D, Wei Z, et al. EchoingPixels: Aliasing-Resistant Joint Token Reduction for Audio-Visual LLMs[C]//Forty-third International Conference on Machine Learning. [2] Ding Y, Ji Y, Li J, et al. Omnisift: Modality-asymmetric token compression for efficient omni-modal large language models[J]. arXiv preprint arXiv:2602.04804, 2026. [3] Zhu J, Zhu Y, et al. VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs[C]//ACM International Conference on Multimedia, 2026.
Questions for authors

See the Weakness.

3 comments

Part 1/2

We thank the reviewer for the constructive comments. We have run new experiments for all concerns to support our claims.

W1. Comparison with OmniSIFT, EchoingPixels, and VisionSelector

To address the reviewer's concern about related baselines, we show the advantages of our method through both qualitative analysis and quantitative experiments.

(a) Controlled comparison with learned omni-modal compressors. To isolate the method from the training data, we retrained the baselines with their official code on exactly the same 8,000 AVoCaDO samples used by RABBIT. All methods use Qwen2.5-Omni-3B and are evaluated on all 1,197 DailyOmni questions at a 10% AV-token budget.

MethodUpdates LLMTrainable paramsDailyOmniWorldSense
OmniSIFTYes3.4B49.7137.70
EchoingPixelsYes3.7B51.4639.16
VisionSelectorNo8.4M51.2941.55
RABBITNo4.2M53.3842.69

RABBIT achieves the highest accuracy on both benchmarks, 1.92–3.67 points above the three baselines on DailyOmni and 1.14–4.99 points on WorldSense. Together, these results support our claim that RL-trained joint audio-video selection on a frozen backbone preserves the information needed under a tight budget better than these learned compressors, whether they fine-tune the LLM or keep it frozen.

(b) Method comparison. Following the reviewer's suggestion, we compare RABBIT with OmniSIFT, EchoingPixels, and VisionSelector below:

EchoingPixelsOmniSIFTVisionSelectorRABBIT
ModalitiesA+VA+VV onlyA+V
Sample-adaptive A/V budget✓ (joint top-k)No (separate prescribed drop rates)—✓ (joint top-k)
Training signalSFT, straight-through top-kSFTSFT, differentiable top-kRL (teacher KL + compression reward)
Updates LLM weights✓ (full + 4 bidirectional layers, modified RoPE)✓ (full)✗✗
One model for all retain ratios✗ (one model per ratio)Configurable inference drop rates✓✓

RABBIT combines a frozen backbone, a single policy across retention ratios, and policy-gradient training that requires only LLM forward passes.

W2. Ablation of the two reward terms and their weights

To show that each reward term in RABBIT is necessary and that the chosen weights are reasonable, we add an ablation study on the reward function. We trained five RABBIT policies on Qwen2.5-Omni-3B that differ only in the reward weights with the same 8K samples, 1,000 updates, K=16, seed 42, and hardware. We evaluated them on all 1,197 DailyOmni questions at 10% retention.

Rewardλ_c : λ_dTrain-time retention*Teacher KLDailyOmni Acc. (10%)
R_distill only0 : 1100.0%0.00036.01
Both0.20 : 0.8020.9%0.08853.47
Both (default)0.35 : 0.6510.7%0.12853.38
Both0.50 : 0.506.7%0.16353.22
R_comp only1 : 02.0%0.46849.46
*Averaged over the last 100 training updates.

Each term alone is insufficient: without R_comp the policy keeps all tokens (36.01), and without R_distill retention collapses to 2.0% (KL 0.468) and accuracy drops to 49.46. With both terms, the ratio shifts training retention from 20.9% to 6.7% as expected, while accuracy stays within 53.22–53.47. Since inference applies top-k at the target budget, the ratio only needs to place training retention near that budget. This confirms that both terms are needed and that the weight ratio is easy to set (match training retention to the target budget) without careful tuning.

Part 2/2

W3. Reward curves and training stability

To demonstrate that the RL training is effective and stable, the revised paper now includes the reward curves from training the 3B model.

Figure 7 shows fast convergence followed by a stable plateau. In the first ~120 updates, the compression reward rises from near zero, expected AV retention falls from 99% to below 15%, and teacher KL peaks at about 0.16. Afterwards all curves stay flat without divergence: over the last 100 updates, total reward is 0.885 (0.65 at initialization), KL 0.128 and retention 10.7%. The checkpoints at updates 200, 400, 600, 800 and 1000 reach 52.21, 51.88, 52.63, 53.72 and 53.38% accuracy. This shows that RL training converges stably and that RABBIT's accuracy does not depend on picking a favorable checkpoint.

We hope these new results address the reviewer's concerns. If they do, we would be grateful if the reviewer would consider raising the score.

Dear Reviewer,

Thank you again for your valuable comments. We have posted detailed responses and new experiments to address your concerns. As the discussion period ends soon, we would greatly appreciate it if you could let us know whether our responses resolve your questions, or whether there is anything else we can clarify.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Post-rebuttal Discussion Summary

We sincerely thank the reviewers for their valuable feedback and constructive discussion during the rebuttal period. As the discussion period draws to a close, we briefly summarize the main concerns raised and the additional experiments and evidence provided in response

  • Baseline coverage and methodological positioning. We added learned and training-free compression baselines and clarified RABBIT’s combination of joint audio-video selection, a frozen backbone, and forward-only LLM evaluation during policy training.
  • Comparison fairness across retention budgets. We compared the same checkpoints at 10% and 25% retention under consistent evaluation protocols, demonstrating competitive accuracy–efficiency trade-offs while reporting budget-dependent differences.
  • Reward design and training stability. We added ablations of both reward terms and their relative weights, together with training curves and intermediate-checkpoint evaluations supporting stable convergence.
  • Training–inference consistency and ranking stability. We compared deterministic and stochastic selection and reported identical token-score rankings across the tested retention budgets, supporting reuse of one learned ranking.
  • Applicability to newer backbones. We trained RABBIT on Qwen3-Omni-30B-A3B with the backbone frozen and obtained higher DailyOmni accuracy than SEATS-I at the same 10% retention budget.
  • Task coverage and cross-task transfer. We added captioning, temporal-grounding, and long-video evaluations, and compared caption-based distillation with direct QA-reward training, clarifying both transfer performance and task generalization.
  • Training-data transparency and reproducibility. We identified the public training-data source and clarified the exact procedure used to select the 8K training examples, correcting the manuscript’s description.
  • Computational efficiency and practical value. We quantified policy-training costs, measured current-checkpoint latency and memory on standard and long inputs, and provided workload-specific compute-amortization estimates.

We are encouraged that both reviewers who participated in the rebuttal discussion confirmed that the additional experiments and evidence addressed their main concerns and raised their ratings. In particular, Reviewer tXc7 stated: “Most of my concerns have been satisfactorily addressed. Please make sure to incorporate these new results and discussions into the revised manuscript. Accordingly, I have raised my rating.” After two rounds of discussion, Reviewer nXip similarly stated: “The authors have addressed my main concerns with new experiments, so I have raised my score.”

We thank all reviewers again for their time and valuable feedback. We will incorporate the additional experiments, analyses, and clarifications presented during the rebuttal into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.