- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
When processing long‑video and long‑audio inputs, multimodal large‑language models suffer from an explosion of audio‑visual tokens, which leads to prohibitively high inference costs. To address the issue that existing compression methods adopt fixed audio‑video retention ratios and fail to adapt to the modality‑wise information distribution across different samples, resulting in rapid performance degradation under heavy compression, this work proposes a reinforcement‑learning‑based adaptive budget‑balancing framework. It designs two independent MLP scoring modules for audio and video respectively, which score audio‑visual tokens with global difference features of each modality. A reward function weighted by compression reward and distillation reward is constructed via rollout sampling, and within‑sample reward normalization is adopted to stabilize policy‑gradient training. The framework learns a sample‑specific unified ranking list for audio and visual tokens. During inference, deterministic Top‑k truncation is directly performed, and a single learned ranking supports multiple compression ratios. Extensive experiments are conducted on multiple benchmarks, showing prominent advantages under high compression ratios such as 10%. Ablation experiments verify the effectiveness of the dual‑branch MLP and global difference features.
Critical strengths
-
Long audio‑video inputs incur a large token count and substantial inference cost. Sample‑level adjustment of audio and video budgets constitutes a reasonable improvement over methods with fixed modality ratios.
-
The dual‑branch MLP combined with modality‑mean difference features avoids explicit attention‑map computation or pairwise token similarity calculation. Only scoring and top‑k selection are required at inference time.
Critical weaknesses
-
This method combines reinforcement learning with dynamic audio‑video budgets and compares itself with existing token pruning and modality‑adaptive approaches. It is necessary to better situate its relationship with recent works on learned pruning and dynamic modality routing.
-
There exists potential distribution and objective misalignment between training and evaluation. The policy is trained via teacher logit distillation on caption‑based datasets, while the downstream target tasks cover world‑knowledge reasoning, video question answering and inference. Teacher‑logit fidelity is not equivalent to downstream task accuracy, and stronger evidence is required to verify cross‑task generalization.
Questions for authors
-
Report the net cost of RABBIT’s policy training relative to inference‑side savings, including training FLOPs, wall‑clock training time, GPU memory footprint, and checkpoint size.
-
Add state‑of‑the‑art baselines of learned pruning and dynamic modality routing under the same backbone, identical input token budget and unified evaluation protocol.
-
Illustrate the impact on performance when disabling SEATS’s intra‑layer / cross‑layer compression modules.
-
Investigate whether the ranking learned via caption‑based distillation remains effective on unseen tasks, datasets, or different audio‑video lengths. Furthermore, compare it against models trained directly with downstream‑task rewards.
-
Analyze the discrepancy between deterministic top‑k truncation at inference and Bernoulli rollout used in training, and verify whether a single stable ranking is truly shared across different target retention rates.
5 comments
Part 2/2
Q3. Disabling SEATS's intra- and cross-layer modules
To isolate the effect of SEATS's intra-layer pruning (Stage II) and cross-layer removal (Stage III), we ran the official SEATS code on Qwen2.5-Omni-3B with the stages enabled incrementally, and also attached Stages II+III to RABBIT. All runs use identical DailyOmni inputs.
| Method | Input AV retention | Prefill TFLOPs | Acc. (%) |
|---|---|---|---|
| SEATS-I | 10.02% | 6.85 | 45.95 |
| SEATS I+II | 10.02% | 5.36 | 45.45 |
| SEATS I+II+III | 10.02% | 5.19 | 45.45 |
| Full SEATS (official 10% layer-mean target) | 14.04% | 6.94 | 48.54 |
| RABBIT | 10.00% | 6.86 | 53.38 |
| RABBIT + SEATS II+III | 10.00% | 5.20 | 53.47 |
Disabling Stages II and III does not reduce accuracy (45.95 vs. 45.45 with both enabled). At similar computation, full SEATS is 4.84 points below RABBIT, and adding Stages II+III to RABBIT cuts prefill computation by about 24% without accuracy loss. RABBIT's gains therefore come from better input selection, which is complementary to SEATS's in-LLM stages.
Q5. Bernoulli training and deterministic top-k
The reviewer asks whether replacing Bernoulli rollouts with deterministic top-k at inference creates a mismatch, and whether one ranking is truly shared across retention rates. We therefore (i) compared deterministic top-k with Gumbel-top-k, which samples exactly k tokens from the same keep probabilities, at 10% retention, and (ii) ran RABBIT separately at 10%, 15% and 25% retention and computed the Spearman correlation between the scores of all audio-video tokens before selection.
| Selection | DailyOmni Acc. (%) |
|---|---|
| Gumbel-top-k (seeds 42 / 43 / 44) | 52.13 / 52.21 / 52.05 |
| Deterministic top-k | 53.38 |
Top-k outperforms every sampled run. This is expected: for a fixed k, the Bernoulli log-probability of a mask is a constant plus the sum of the selected tokens' log-odds, so top-k is the most probable k-token mask under the trained policy.
| Retention rates compared | 10% vs. 15% | 10% vs. 25% | 15% vs. 25% |
|---|---|---|---|
| Spearman ρ (all AV tokens, before selection) | 1.00 | 1.00 | 1.00 |
The token rankings at the three retention rates are identical on every question (ρ = 1.00), because the scorer does not take the budget as input.
Hence, (i) deterministic top-k deploys the most probable mask of the Bernoulli-trained policy and loses no accuracy relative to sampling from it, and (ii) a single stable ranking is computed once and shared by all target retention rates, which only change where it is cut.
We believe these results address the reviewer's concerns and would be grateful if the reviewer would consider raising the score.
In summary, the authors have addressed my requests by adding experiments, but the empirical evidence is not yet strong enough to fully convince me on the advantages of their design. I will keep my original score.
- For W1/Q2: While the authors have added OmniSIFT, EchoingPixels and VisionSelector for comparison, the accuracy gaps against these baselines are relatively modest. The comparison only covers one fixed retention ratio (10%). It is still unclear whether the observed behaviour holds across different retention budgets.
- For Q1: The break-even analysis is informative, but it relies heavily on the assumed production query volume. The practical value of this trade-off remains application-dependent and is not sufficiently discussed.
Part 1/2
We thank the reviewer for the constructive comments. We address the requested comparisons, training cost, transfer evidence, and train–inference consistency below.
To address the reviewer's concern about learned pruning and dynamic modality routing methods, we add OmniSIFT, EchoingPixels, and VisionSelector as baselines to show the advantage of our method.
RABBIT is 1.92–3.67 points higher on DailyOmni and 1.14–4.99 points higher on WorldSense than all three, at comparable or lower prefill computation. It also outperforms EchoingPixels, the only baseline with dynamic allocation, which fine-tunes the LLM and needs 2.5× more prefill computation. This shows that RL-trained joint selection on a frozen backbone outperforms these learned pruning and dynamic-allocation methods, whether or not they fine-tune the LLM.
To justify the need for caption supervision and to show that our method generalizes, we argue from both a heuristic and an empirical perspective.
Our reason for training on captions is that a caption covers the entire audio-video content, whereas a QA reward only covers the part relevant to one question, so caption distillation should better teach the policy to preserve all multimodal information.
To verify this, we trained a policy with a direct task reward on 8,000 multiple-choice questions from OmniVideo-100K and compared it with caption-trained RABBIT. We compare these two method with DailyOmni and LVOmniBench under 10% retention. The results are listed below.
Although it never sees the QA task during training, caption-trained RABBIT is 2.92 points higher on DailyOmni and 0.69 points higher on LVOmniBench, whose inputs are up to four times longer than the 64 frames used in training. This supports our design: caption distillation teaches the policy to preserve the overall audio-video information, and in our setting it transfers to unseen tasks, datasets and longer inputs better than direct QA supervision, without needing task labels.
To quantify the net cost, we measured the training of the Qwen2.5-Omni-3B policy used in all experiments (8K samples, 1,000 updates) and its inference savings at 10% retention.
Dividing the training cost by the per-query saving gives a break-even point of about 80K queries by FLOPs and 140K by GPU time. The one-time cost is thus recovered after roughly 80K–140K queries, after which every query saves 90.7% of prefill computation and 39.3% of latency.