- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes CAFD, a framework for adapting OmniLLMs to fixed audio-video token compression pipelines without reference answers, rationales, or correctness rewards. A compressed-context student generates responses, and a full-context EMA self-teacher provides distributional supervision along the student’s own response prefixes. Adaptation uses LoRA and a clipped Jensen–Shannon divergence objective. On Qwen2.5-Omni-7B, the authors train one model per compressor at nominal 5% retention and evaluate across five compressors, five benchmarks, and five deployment budgets. CAFD improves 120 of 125 conditions, with a mean accuracy gain of 1.44 percentage points and 26.9% mean accuracy-gap recovery across compressors.
Critical strengths
- Practical adaptation without reference supervision. CAFD uses the original audio-video input to provide teacher supervision while retaining the existing compression pipeline. The generic-prompt and video-disjoint adaptation experiments further support its usefulness when per-example questions and reference responses are unavailable. This makes the approach relevant to deployment settings with accessible media but limited annotations.
- Consistent improvements across compression settings. The evaluation covers token selection, merging, and inner-LLM pruning, with compression configurations held fixed within each before/after comparison. Improvements in 120 of 125 conditions provide evidence that adaptation complements multiple compression pipelines. Reusing each compressor-specific checkpoint across five deployment budgets also strengthens the practical value of the results.
Critical weaknesses
The contribution of the central training choices is insufficiently isolated. The paper compares divergence objectives and supervision sources, but does not compare EMA against frozen or synchronized teachers, or student-generated prefixes against alternative trajectory sources. Given closely related approaches such as NOPD, these controls are important for establishing what CAFD’s specific training design adds beyond applying existing on-policy self-distillation to audio-video compression.
Questions for authors
- Teacher updates. How does CAFD compare with a frozen initial teacher and a teacher synchronized with the student at every update, while keeping the compression setting, training data, LoRA configuration, and JSD objective fixed? These controls would clarify the contribution of EMA.
- Trajectory source. What is the benefit of using student-generated prefixes over teacher-generated prefixes under otherwise matched distillation settings? The SFT comparison changes both the trajectory source and the supervision objective, so it does not isolate the contribution of on-policy sampling.
- Full-context capability retention. How do the adapted checkpoints perform at 100% token retention compared with the original full-token model? This would clarify whether adaptation preserves full-context performance or introduces a trade-off between compressed and uncompressed inference.
5 comments
Q3: Full-context capability after compression adaptation.
To examine whether compression adaptation preserves full-context capability, we evaluated the original model and the DivPrune-, OmniZip*-, and SEATS-adapted train-5 CAFD checkpoints at 100% retention on WorldSense, DailyOmni, and AVUT. Compression was disabled, and the evaluation questions and prompt template were held fixed. Table R3 reports this comparison.
Table R3. Full-token accuracy (%). Adapted checkpoints were trained at 5% retention.
| Weights | WorldSense | DailyOmni | AVUT | Avg. |
|---|---|---|---|---|
| Unadapted | 46.85 | 62.91 | 64.65 | 58.13 |
| CAFD-DivPrune-adapted | 47.89 | 63.16 | 66.49 | 59.18 |
| CAFD-OmniZip*-adapted | 47.89 | 63.41 | 66.38 | 59.22 |
| CAFD-SEATS-adapted | 47.54 | 63.83 | 66.44 | 59.27 |
All three adapted checkpoints improve full-token accuracy on all three benchmarks, showing no observed accuracy trade-off between compressed and uncompressed inference.
We sincerely thank you for these helpful suggestions. We will consider incorporating the additional experimental results and clarifications into the revised manuscript.
[1] Idan Shenfeld et al. (2026). Self-Distillation Enables Continual Learning. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 111542-111558). PMLR.
[2] Hao Wang et al. "Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents." arXiv preprint arXiv:2604.10674 (2026).
[3] Shayan Talaei et al. "Speculative Self-Distillation enables Efficient Knowledge Internalization." arXiv preprint arXiv:2611.12016 (2026).
[4] Siyan Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 162433-162448). PMLR.
Response to rebuttal
Thank you for running the requested controls (Tables R1–R3). Table R3 addresses Q3, confirming that adaptation preserves and slightly improves full-context accuracy (59.18%–59.27% vs. 58.13% unadapted). On Q1 and Q2, Tables R1–R2 show that EMA (53.59% vs. 53.62% frozen) and student-generated trajectories (53.59% vs. 53.79% teacher trajectories) offer no accuracy gain over simpler baselines. The methodological novelty beyond existing asymmetric-view distillation is therefore modest; the paper's main value lies in demonstrating ground-truth-free compression adaptation across diverse compressors and budgets. In the revision, please include Tables R1–R3 and revise the Abstract, Sections 1–2, and Section 3.3 so they do not overstate the necessity of EMA or student on-policy rollouts. I maintain my rating of 3.
We sincerely thank the reviewer for the constructive review and helpful recommendations. We appreciate your recognition of the reference-free adaptation setting and consistent improvements across compression methods. Your proposed controls help us separate the effects of Teacher updates and trajectory source, while your question about full-context capability motivates checking whether adaptation preserves performance with uncompressed inputs. We address the weaknesses W# and questions Q# below.
W1: Teacher-update and trajectory-source comparisons.
Thank you for suggesting these controls; we examined Teacher updates and trajectory source separately, keeping all other settings unchanged.
1. Teacher updates.
To examine EMA's contribution, we compared EMA, frozen, and per-update synchronized Teachers while keeping all other training settings unchanged (Table R1).
Table R1. Teacher-update comparison. DivPrune is used for training (5% retention) and evaluation. Accuracy (%) is averaged over all five deployment budgets.
All three Teacher-update rules achieve 53.55–53.62% mean accuracy, compared with the 51.97% Unadapted baseline, with only a 0.07-point spread. This consistency supports the full-context-to-compressed-context adaptation framework across the tested Teacher-update choices.
2. Trajectory source.
To address your question about trajectory source, we compared trajectories generated by the compressed Student and the full-context EMA Teacher, keeping the Teacher-update rule and all other CAFD training settings unchanged. Table R2 reports the results.
Table R2. Trajectory-source comparison with JSD fixed. DivPrune is used for training (5% retention) and evaluation. Accuracy (%) is averaged over all five deployment budgets.
Student- and Teacher-generated trajectories achieve similar mean accuracies (53.59% and 53.79%, respectively; a 0.20-point difference), both improving over the 51.97% Unadapted baseline. Prior studies report advantages of on-policy over off-policy self-distillation in continual learning [1], multi-turn agent training [2], and knowledge internalization [3]. Together, these findings suggest that the relative advantage depends on the domain and task, while either trajectory source supports effective full-context-to-compressed-context adaptation.
Our contribution is mainly the full-context-to-compressed-context adaptation framework. Our choice of on-policy self-distillation is motivated by its effectiveness in prior studies [1, 2, 4]. The framework is not restricted to this specific recipe: these ablations show that adaptation remains effective across alternative Teacher-update rules and trajectory sources.
Q1: EMA versus frozen and synchronized Teachers.
Following your suggestion, we compared frozen, synchronized, and EMA Teachers while keeping all other CAFD training settings unchanged. Please see W1 above and Table R1 for the comparison.
Q2: Student- versus Teacher-generated trajectories.
Thank you for asking about the choice of trajectory source. We compared Student-generated and Teacher-generated trajectories while keeping all other CAFD training settings unchanged. Please see W1 and Table R2 for the results.