acceptodds
Under review as a conference paper at ICLR 2027

Learning to Reason with Compressed Context: Ground-Truth-Free Adaptation of OmniLLMs via Self-Distillation

Abstract

Omni-modal large language models (OmniLLMs) enable unified audio-video understanding, but their long multimodal token sequences make deployment computationally expensive. Token compression reduces this cost, yet aggressive compression often lowers accuracy. Existing works predominantly focus on designing better compression mechanisms; however, adapting the underlying language model to reason effectively over the remaining compressed context remains under-explored. To address this, we propose **CAFD** (Compressed-Context Adaptation via Full-Context Distillation), a ground-truth-free self-distillation framework that adapts OmniLLMs to fixed compression pipelines without requiring reference answers, rationales, or correctness rewards. CAFD leverages the full-token view of the same multimodal sample as a source of privileged information: a full-context self-teacher provides soft target supervision to a compressed-context student along the student's on-policy trajectory. Evaluated on Qwen2.5-Omni-7B across five audio-video benchmarks, five compression pipelines, and five deployment budgets, CAFD demonstrates consistent gains, improving 120 out of 125 conditions with an average accuracy boost of 1.44 points and recovering 26.9% of the accuracy gap on average. These results demonstrate that the proposed ground-truth-free adaptation offers an effective and practical route to improving the accuracy–efficiency trade-off in deployed OmniLLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
omni-modal large language models, multimodal token compression, compressed-context adaptation, ground-truth-free adaptation, on-policy self-distillation, efficient multimodal inference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_rauVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes CAFD, a framework for adapting OmniLLMs to fixed audio-video token compression pipelines without reference answers, rationales, or correctness rewards. A compressed-context student generates responses, and a full-context EMA self-teacher provides distributional supervision along the student’s own response prefixes. Adaptation uses LoRA and a clipped Jensen–Shannon divergence objective. On Qwen2.5-Omni-7B, the authors train one model per compressor at nominal 5% retention and evaluate across five compressors, five benchmarks, and five deployment budgets. CAFD improves 120 of 125 conditions, with a mean accuracy gain of 1.44 percentage points and 26.9% mean accuracy-gap recovery across compressors.

Critical strengths
  1. Practical adaptation without reference supervision. CAFD uses the original audio-video input to provide teacher supervision while retaining the existing compression pipeline. The generic-prompt and video-disjoint adaptation experiments further support its usefulness when per-example questions and reference responses are unavailable. This makes the approach relevant to deployment settings with accessible media but limited annotations.
  2. Consistent improvements across compression settings. The evaluation covers token selection, merging, and inner-LLM pruning, with compression configurations held fixed within each before/after comparison. Improvements in 120 of 125 conditions provide evidence that adaptation complements multiple compression pipelines. Reusing each compressor-specific checkpoint across five deployment budgets also strengthens the practical value of the results.
Critical weaknesses

The contribution of the central training choices is insufficiently isolated. The paper compares divergence objectives and supervision sources, but does not compare EMA against frozen or synchronized teachers, or student-generated prefixes against alternative trajectory sources. Given closely related approaches such as NOPD, these controls are important for establishing what CAFD’s specific training design adds beyond applying existing on-policy self-distillation to audio-video compression.

Questions for authors
  1. Teacher updates. How does CAFD compare with a frozen initial teacher and a teacher synchronized with the student at every update, while keeping the compression setting, training data, LoRA configuration, and JSD objective fixed? These controls would clarify the contribution of EMA.
  2. Trajectory source. What is the benefit of using student-generated prefixes over teacher-generated prefixes under otherwise matched distillation settings? The SFT comparison changes both the trajectory source and the supervision objective, so it does not isolate the contribution of on-policy sampling.
  3. Full-context capability retention. How do the adapted checkpoints perform at 100% token retention compared with the original full-token model? This would clarify whether adaptation preserves full-context performance or introduces a trade-off between compressed and uncompressed inference.
5 comments

We sincerely thank the reviewer for the constructive review and helpful recommendations. We appreciate your recognition of the reference-free adaptation setting and consistent improvements across compression methods. Your proposed controls help us separate the effects of Teacher updates and trajectory source, while your question about full-context capability motivates checking whether adaptation preserves performance with uncompressed inputs. We address the weaknesses W# and questions Q# below.

W1: Teacher-update and trajectory-source comparisons.

Thank you for suggesting these controls; we examined Teacher updates and trajectory source separately, keeping all other settings unchanged.

1. Teacher updates.

To examine EMA's contribution, we compared EMA, frozen, and per-update synchronized Teachers while keeping all other training settings unchanged (Table R1).

Table R1. Teacher-update comparison. DivPrune is used for training (5% retention) and evaluation. Accuracy (%) is averaged over all five deployment budgets.

WeightsTeacher updateWorldSenseDailyOmniAVUTAvg.
Unadapted-42.9954.7458.1951.97
CAFD-adaptedEMA, decay 0.99944.1657.0459.5653.59
CAFD-adaptedFrozen initial Teacher44.0457.1859.6353.62
CAFD-adaptedCurrent Student weights at each optimizer update43.9656.9859.7153.55

All three Teacher-update rules achieve 53.55–53.62% mean accuracy, compared with the 51.97% Unadapted baseline, with only a 0.07-point spread. This consistency supports the full-context-to-compressed-context adaptation framework across the tested Teacher-update choices.

2. Trajectory source.

To address your question about trajectory source, we compared trajectories generated by the compressed Student and the full-context EMA Teacher, keeping the Teacher-update rule and all other CAFD training settings unchanged. Table R2 reports the results.

Table R2. Trajectory-source comparison with JSD fixed. DivPrune is used for training (5% retention) and evaluation. Accuracy (%) is averaged over all five deployment budgets.

WeightsTrajectory sourceWorldSenseDailyOmniAVUTAvg.
Unadapted-42.9954.7458.1951.97
CAFD-adaptedCompressed Student44.1657.0459.5653.59
CAFD-adaptedFull-context EMA Teacher44.2157.4459.7253.79

Student- and Teacher-generated trajectories achieve similar mean accuracies (53.59% and 53.79%, respectively; a 0.20-point difference), both improving over the 51.97% Unadapted baseline. Prior studies report advantages of on-policy over off-policy self-distillation in continual learning [1], multi-turn agent training [2], and knowledge internalization [3]. Together, these findings suggest that the relative advantage depends on the domain and task, while either trajectory source supports effective full-context-to-compressed-context adaptation.

Our contribution is mainly the full-context-to-compressed-context adaptation framework. Our choice of on-policy self-distillation is motivated by its effectiveness in prior studies [1, 2, 4]. The framework is not restricted to this specific recipe: these ablations show that adaptation remains effective across alternative Teacher-update rules and trajectory sources.

Q1: EMA versus frozen and synchronized Teachers.

Following your suggestion, we compared frozen, synchronized, and EMA Teachers while keeping all other CAFD training settings unchanged. Please see W1 above and Table R1 for the comparison.

Q2: Student- versus Teacher-generated trajectories.

Thank you for asking about the choice of trajectory source. We compared Student-generated and Teacher-generated trajectories while keeping all other CAFD training settings unchanged. Please see W1 and Table R2 for the results.

Q3: Full-context capability after compression adaptation.

To examine whether compression adaptation preserves full-context capability, we evaluated the original model and the DivPrune-, OmniZip*-, and SEATS-adapted train-5 CAFD checkpoints at 100% retention on WorldSense, DailyOmni, and AVUT. Compression was disabled, and the evaluation questions and prompt template were held fixed. Table R3 reports this comparison.

Table R3. Full-token accuracy (%). Adapted checkpoints were trained at 5% retention.

WeightsWorldSenseDailyOmniAVUTAvg.
Unadapted46.8562.9164.6558.13
CAFD-DivPrune-adapted47.8963.1666.4959.18
CAFD-OmniZip*-adapted47.8963.4166.3859.22
CAFD-SEATS-adapted47.5463.8366.4459.27

All three adapted checkpoints improve full-token accuracy on all three benchmarks, showing no observed accuracy trade-off between compressed and uncompressed inference.

We sincerely thank you for these helpful suggestions. We will consider incorporating the additional experimental results and clarifications into the revised manuscript.


[1] Idan Shenfeld et al. (2026). Self-Distillation Enables Continual Learning. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 111542-111558). PMLR.

[2] Hao Wang et al. "Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents." arXiv preprint arXiv:2604.10674 (2026).

[3] Shayan Talaei et al. "Speculative Self-Distillation enables Efficient Knowledge Internalization." arXiv preprint arXiv:2611.12016 (2026).

[4] Siyan Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 162433-162448). PMLR.

Reviewer_rauVOpenReview ↗

Response to rebuttal

Thank you for running the requested controls (Tables R1–R3). Table R3 addresses Q3, confirming that adaptation preserves and slightly improves full-context accuracy (59.18%–59.27% vs. 58.13% unadapted). On Q1 and Q2, Tables R1–R2 show that EMA (53.59% vs. 53.62% frozen) and student-generated trajectories (53.59% vs. 53.79% teacher trajectories) offer no accuracy gain over simpler baselines. The methodological novelty beyond existing asymmetric-view distillation is therefore modest; the paper's main value lies in demonstrating ground-truth-free compression adaptation across diverse compressors and budgets. In the revision, please include Tables R1–R3 and revise the Abstract, Sections 1–2, and Section 3.3 so they do not overstate the necessity of EMA or student on-policy rollouts. I maintain my rating of 3.

Reviewer_HNeiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets the high prefill cost of long audio-video reasoning in omni-modal LLMs and proposes CAFD, a post-training adaptation method. Prior token-compression work concentrates on designing better pruning or merging operators and pays little attention to the distribution shift a model faces when its input is reduced to a sparse token sequence. The few methods that fine-tune the model depend on supervised answer labels or reinforcement-learning rewards. CAFD takes a different route. It treats the uncompressed full token sequence of the same video as privileged information and feeds that view to an EMA teacher. The student, trained on a 5% compressed view, first generates its own analysis and answer through an on-policy rollout. A clipped Jensen–Shannon divergence then aligns the student’s full-vocabulary distribution along that trajectory with the teacher’s.

The authors test this recipe on Qwen2.5-Omni-7B across five audio-video multiple-choice benchmarks, five compression methods (pre-LLM selection, token merging, and progressive inner-LLM pruning), and five retention budgets. Without increasing inference FLOPs and without using human answer labels, CAFD improves 120 of the 125 conditions, by 1.44 percentage points on average.

Critical strengths

The method is practical because it requires neither answer labels nor task-specific prompts. Tables 3 and 4 make this concrete: CAFD matches distillation that provides ground-truth answers and rationales to the teacher (three-benchmark averages of 53.59 and 53.44). Table 5 is equally convincing. Training on raw unlabeled video with one fixed generic instruction still raises held-out WorldSense accuracy, from 43.08% to 44.00% under the video-disjoint split.

The experimental effort is substantial, and the controls are clean. The authors do not restrict the study to one or two simple pruning methods. They evaluate Uniform, DivPrune, a token-merging variant, the cross-modal OmniZip* variant, and the inner-LLM method SEATS, covering 125 conditions. The appendix checks train–test leakage with video hashes and frame fingerprints, reports an analytical prefill-cost calculation, and analyzes which predictions flip after adaptation.

Critical weaknesses

The method has limited originality and overlaps heavily with recent on-policy self-distillation work. Its training framework — on-policy student rollouts, an EMA self-teacher, symmetric JSD alignment, and supervision without ground-truth answers — is nearly identical, in both formulation and mechanism, to the recent Clue-OPSD (arXiv:2608.25356) and to the earlier OPSD (Zhao et al., 2026). Using an undegraded full input to supervise a degraded input is also very close to RP-OPSD, which uses a high-resolution view to supervise a low-resolution view. The paper essentially transfers an existing OPSD recipe to multimodal token compression. It introduces no mechanism specific to the fact that features have been pruned or merged, such as reweighting the distillation loss by the attention assigned to discarded tokens.

The training setup is also inflexible. A model trained at a 5% budget can be evaluated directly at 10–35%, but each compression algorithm still requires its own LoRA. Locking the training retention at 5% creates two problems. At the relatively generous 35% budget, three compressors drop slightly on OmniVideoBench (Table 1): Uniform (33.80 to 33.70), OmniZip* (35.00 to 34.70), and SEATS (35.70 to 35.50). In the task breakdown of Table 18, WorldSense Hallucination falls by 4.00 percentage points. A plausible cause is that the student sees only a severely incomplete 5% context but is forced to match a teacher that sees the full token sequence, and therefore learns to guess and to fill in evidence it does not have.

Questions for authors

1.The optimization objective is very close to OPSD and Clue-OPSD. When this recipe is moved to extreme multimodal token compression, were any algorithm changes introduced to keep training stable? Or does the contribution lie mainly in the empirical study of full-context self-distillation for compression adaptation?

  1. Why does the WorldSense Hallucination category in Table 18 drop by 4.00 percentage points? If training did not fix (rtrainr_{\text{train}}=5%), and instead sampled the retention ratio at each step from ([5%, 35%]), would that reduce both the hallucination regression and the slight drops at the 35% evaluation budget?

  2. How well does a LoRA trained on one compressor, such as DivPrune, transfer zero-shot to another, such as SEATS or OmniZip*? The current experiments use only Qwen2.5-Omni. Could the authors add even one other OmniLLM backbone to test whether the gain generalizes?

4 comments

We thank the reviewer for the thoughtful assessment and detailed suggestions. We appreciate your recognition of CAFD's practical value and controlled evaluation. Your suggestions on compressor reuse, mixed-retention training, and an additional backbone help us examine the flexibility and scope of adaptation more directly. We address the weaknesses W# and questions Q# below.

W1: Contribution relative to OPSD and related self-distillation methods.

Thank you for asking us to clarify CAFD's contribution relative to OPSD and related work. As discussed in Sections 2.2–2.3, CAFD addresses model adaptation in OmniLLM token compression. Most work in this area focuses on designing token-selection, pruning, or merging operators; CAFD complements these methods by adapting the model to their compressed contexts, without reference answers or rationales.

The full-context-to-compressed-context adaptation framework is our central contribution, spanning pre-LLM selection, token merging, and inner-LLM pruning. We adopted an OPSD-based training recipe, motivated by the effectiveness of this family of techniques in prior work [1–3]. The framework is not restricted to the specific Teacher-update and trajectory-source choices in this default recipe.

Table 4 further demonstrates CAFD's practical flexibility: adaptation remains effective using only media and a shared instruction, without per-example questions, options, reference answers, or rationales during training.

Your suggestion of a loss that accounts for discarded tokens offers a useful extension of the framework through compression-aware distillation objectives.

W2: Cross-compressor transfer and addressing performance regressions.

1. Cross-compressor transfer.

To address your concern about compressor-specific adaptation, we investigated whether a checkpoint trained with DivPrune can be reused with other compressors. We evaluated the DivPrune-adapted train-5 checkpoint with all five deployment compressors on WorldSense, DailyOmni, and AVUT at all five budgets, without further adaptation. Table R1 reports this cross-compressor comparison alongside the Unadapted baseline.

Table R1. Cross-compressor transfer from DivPrune. The CAFD checkpoint was trained at 5% retention. Accuracy (%) is averaged equally across WorldSense, DailyOmni, and AVUT and evaluation retentions of 35%, 25%, 15%, 10%, and 5%.

WeightsUniformDivPruneDivPrune-MergeOmniZip*SEATS
Unadapted47.1651.9752.3949.7653.04
CAFD-DivPrune-adapted48.9453.5954.0751.2454.23

The DivPrune-adapted checkpoint improves the three-benchmark, five-budget mean for all five deployment compressors, including the four not used during adaptation. These results demonstrate useful zero-shot reuse without a separate adaptation run for each deployment compressor.

2. Performance regressions.

As you note, training exclusively at 5% retention may contribute to the occasional accuracy drops at higher retention ratios on OmniVideoBench and the WorldSense Hallucination regression. To investigate this possibility, we followed your suggestion in Q2 and compared fixed-5% training with balanced mixed-retention training. Please see our response to Q2 below for the results.

Q1: Algorithmic changes and training stability under extreme compression.

Thank you for asking whether extreme compression required changes to the optimization recipe. We used OPSD as an established instantiation of the compression-adaptation framework described in W1, without adding a compression-specific stabilization mechanism.

Appendix A.3 and Table 6 of our paper report consistent performance across three independent DivPrune train-5 seeds. Table R2 summarizes these existing results.

Table R2. Seed sensitivity, summarized from Appendix A.3 and Table 6 of our paper. DivPrune is used for training (5% retention) and evaluation. Accuracy (%) is averaged equally across WorldSense, DailyOmni, and AVUT and the five evaluation budgets.

Training seedMean accuracy (%)
4253.59
14253.65
24253.47
Mean ± sample SD53.57 ± 0.09

The main run completes 358 updates with finite loss, and the three seeds achieve 53.57 ± 0.09% accuracy. These results support stable training with the existing recipe at 5% retention.

Q2: Hallucination regression and balanced mixed-retention training.

Thank you for the insightful hypothesis in W2 and the constructive suggestion here. We agree that training exclusively at 5% retention may contribute to these regressions: the Student is asked to match a full-context Teacher while seeing severely compressed evidence. Following your suggestion, we compared fixed-5% training with balanced mixed-retention training to test whether broader budget exposure alleviates this issue.

For mixed-retention training, we used a seed-42 shuffled, balanced schedule over {5%, 10%, 15%, 25%, 35%}, assigning each budget to approximately 20% of optimizer updates. The data, LoRA configuration, and optimizer-update count matched fixed-5% training.

1. WorldSense Hallucination.

We first examine the WorldSense Hallucination category you highlighted. Table R3 compares fixed-5% and balanced mixed-retention DivPrune, reporting the five-budget mean accuracy change relative to Unadapted.

Table R3. Fixed-5% versus balanced mixed-retention CAFD training on WorldSense Hallucination. DivPrune is used for both training and evaluation. Accuracy changes are relative to Unadapted and averaged over all five evaluation budgets.

Training retentionHallucination delta (pp)
5%−4.00
Balanced mix−1.33

Balanced mixed-retention training reduces the mean deficit from 4.00 to 1.33 percentage points, recovering approximately two-thirds of the loss. This supports your suggestion that broader exposure to training budgets can mitigate the Hallucination-category regression, while leaving a smaller residual gap.

2. OmniVideoBench regressions.

We also examine the OmniVideoBench drops discussed in W2 and referred to in the second part of Q2. Table R4 compares the three configurations at all five regression points in Table 1.

Table R4. OmniVideoBench regression points. Accuracy (%) is reported at the five operating points where fixed-5% CAFD underperforms Unadapted in Table 1. For adapted models, training and evaluation use the same compressor.

CompressorEval. retentionUnadaptedCAFD, fixed-5% trainingCAFD, balanced mixed-retention training
Uniform35%33.8033.7034.10
OmniZip*35%35.0034.7034.70
SEATS35%35.7035.5036.10
25%35.4035.0036.40
15%36.0035.7036.80

Mixed-retention training brings four of the five affected OmniVideoBench operating points above their Unadapted baselines, including all three SEATS points; OmniZip* at 35% retains its 0.30-point deficit. Together with the Hallucination improvement in Table R3, these results show that mixed-retention training alleviates both issues raised in your question.

Q3: Cross-compressor transfer and an additional OmniLLM backbone.

Thank you for suggesting checks of both compressor reuse and backbone generalization.

1. Cross-compressor transfer.

The DivPrune-adapted checkpoint transfers to other compressors without further training; please see W2 above and Table R1 for the comparison.

2. Additional backbone.

To address your concern about backbone generalization within the rebuttal time constraints, we added Qwen3-Omni-30B-A3B-Instruct [4], using DivPrune for both training and evaluation. We used rank-64 LoRA only on its Thinker text-decoder attention projections (q/k/v/o), keeping MoE experts frozen. We trained one CAFD checkpoint at 5% retention and compared it with its Unadapted DivPrune model at all five budgets on WorldSense, DailyOmni, and AVUT, alongside its full-token reference.

Table R5. Additional OmniLLM backbone. DivPrune is used for both training (5% retention) and evaluation. Compressed accuracies (%) average the five budgets.

BackboneWeightsCompressorWorldSenseDailyOmniAVUTAvg.
Qwen3-Omni-30B-A3B-InstructUnadaptedFull tokens55.0173.9376.9968.65
UnadaptedDivPrune49.4366.1067.1760.90
CAFD-adaptedDivPrune50.6967.9968.0962.26

CAFD improves accuracy across all 15 benchmark–budget combinations on Qwen3-Omni. Mean accuracy rises from 60.90% to 62.26%, supporting adaptation beyond the original Qwen2.5-Omni-7B.

We sincerely thank you for these helpful suggestions. We will consider incorporating the additional experimental results and clarifications into the revised manuscript.


[1] Siyan Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 162433-162448). PMLR.

[2] Kaishen Wang et al. "Where to Look Matters: On-Policy Self-Distillation for Long-Video Understanding." arXiv preprint arXiv:2608.25356 (2026).

[3] Qihui Zhu et al. "RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models." arXiv preprint arXiv:2607.24447 (2026).

[4] Jin Xu et al. "Qwen3-Omni Technical Report." arXiv preprint arXiv:2509.17765 (2025).

Thank you again for your careful review and constructive suggestions. We have incorporated the supplementary experiments addressing your questions into the revised manuscript:

  • Cross-compressor transfer: Appendix A.6 (Tables 13–14).
  • Balanced mixed-retention training, including WorldSense Hallucination and OmniVideoBench regressions: Appendix A.8 (Tables 16–18).
  • Additional Qwen3-Omni backbone: Appendix A.5 (Tables 11–12).
  • Compression-aware distillation objectives: included as a future direction in Section 5.

To address your concern about the relation to OPSD [1], our contribution is mainly the adaptation framework for OmniLLM token compression, complementing compressor design by adapting the model to the context it retains.

We have revised the manuscript throughout to clarify that our contribution is mainly the full-context-to-compressed-context adaptation framework. Changes are marked in blue in the revised manuscript. Section 3 describes one implementation using on-policy Student rollouts, an EMA Teacher, and clipped JSD. The revised presentation treats these as implementation choices and avoids overstating their necessity or superiority. The controls demonstrate effective adaptation across alternative choices, with their relative advantages varying across tasks. We appreciate your suggestions, which have helped improve the experimental coverage and presentation.


[1] Siyan Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 162433-162448). PMLR.

We sincerely thank you for the time and effort you have dedicated to reviewing our work. Your careful assessment and valuable suggestions have helped us improve the manuscript. We greatly appreciate your contribution to the review process.

Reviewer_jTpFOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Compressed-Context Adaptation via Full-Context Distillation, a reference-answer-free method for adapting an omni-modal language model to a fixed audio-video token-compression pipeline where a compressed-context student generates responses, and a full-context teacher supplies next-token distributions along those student-generated prefixes. Training optimizes by minimizing a clipped Jensen-Shannon divergence, with the teacher updated through an exponential moving average of the student's LoRA parameters and the deployment uses only the adapted student and the original compressor.

Critical strengths
  • The paper tackles a genuinely useful problem of speeding up a chosen compression pipeline without needing a heavy teacher model at inference time. The evaluations are robust and across various scenarios like test token selection, merging, and inner-model pruning while keeping settings identical between the baseline and compressed models.

  • The consistent gains across all five pipelines, along with being able to reuse a single checkpoint across different budgets, show that this works well beyond just one specific setup.

  • The evaluations are extensive and helpful in understanding the results and back up the main claims of the paper. The ablation using a fixed, generic prompt still improves average performance from 51.97% to 53.47% across three benchmarks without needing test-time questions or choices.

  • The authors also do a good job ruling out data leakage through video-disjoint splits and overlap checks. Training across three random seeds on DivPrune shows steady results (53.57% +/- 0.09%), and performance holds up on held-out benchmarks not used during retention tuning, with an average bump of around 1.21.

Critical weaknesses
  • Although the experimentation and evaluations are extensive the experiments do not clearly isolate why the specific on-policy, soft-target distillation recipe is necessary over simpler baselines. The ablations look at divergence losses, prompt conditioning, and ground-truth supervision, but they miss a straightforward alternative: standard supervised fine-tuning on responses generated by a full-context teacher using the same unlabeled media. A baseline like that would also be reference-free and would clarify whether online student rollouts and matching the full output distribution actually justify their overhead. Testing a simple frozen teacher alongside off-policy distillation would also pinpoint what the EMA updates and student-generated trajectories really contribute. This is an important distinction because the core idea of pairing student trajectories with privileged context comes directly from OPSD, leaving its application to compressed inputs as the main contribution here.

  • The headline numbers suggest broader empirical validation than what the independent experiments actually support. Everything is tested on a single backbone, and Appendix B.6 notes that model retention during training was tuned using scores from WorldSense, DailyOmni, and AVUT. Because of that, the 125 evaluated conditions are not 125 independent tests of generalization; they rely on the same questions, reused checkpoints, and benchmarks that shaped the training setup in the first place. While evaluations on OmniVideoBench and Video-MME give helpful independent validation, the true scope is narrower than the condition count implies. In addition, Table 11 only estimates decoder-prefill FLOPs while leaving out vision encoders, compression runtime, and token decoding, so it leaves actual wall-clock speedups, peak memory drops, and the cost of training unmeasured.

Questions for authors
  • How does CAFD compare with SFT on full-context teacher-generated responses, using the same media, LoRA scope, and a clearly reported training-cost budget? Does online soft distillation provide a meaningful advantage?

  • What happens with a frozen teacher, off-policy prefixes, and unclipped JSD? Which choices are necessary for the reported gains?

  • What are the training GPU-hours and measured inference latency/peak memory at representative operating points? Approximately how many inference requests are needed to amortize adaptation under the reported deployment gains?

8 comments

We sincerely thank the reviewer for the careful reading and constructive feedback. We appreciate your recognition of CAFD's practical motivation and controlled evaluation. Your suggestions help us examine what the distillation recipe adds beyond simpler alternatives and clarify the scope and practical cost of adaptation. We address each weakness and question below, using W# for weaknesses and Q# for questions.

W1: Comparison with Teacher-generated SFT and training-design controls.

Thank you for these suggestions; we address the SFT baseline, adaptation cost, Teacher updates, and trajectory source in turn.

1. Teacher-generated SFT.

To test the simpler reference-free alternative you propose, the initial full-context Teacher generated responses for the same 5,700 examples, without reference answers or correctness filtering. We trained a compressed Student for each of DivPrune and SEATS on these responses using standard cross-entropy, matching the data, Student prompts, train-5 compression, LoRA scope, and optimizer-update count of the corresponding CAFD runs. Table R1 compares this Teacher-generated SFT baseline with CAFD.

Table R1. Teacher-generated SFT versus CAFD. Accuracy (%) is averaged over the five retention budgets: 35%, 25%, 15%, 10%, and 5%. For adapted models, training and evaluation use the same compressor.

CompressorTraining methodWorldSenseDailyOmniAVUTAvg.
DivPruneUnadapted42.9954.7458.1951.97
Teacher-generated SFT43.9756.5459.3153.27
CAFD44.1657.0459.5653.59
SEATSUnadapted43.6156.5758.9453.04
Teacher-generated SFT44.4557.1359.7353.77
CAFD44.7757.7860.1554.23

CAFD's accuracy gains over Unadapted are 24.0% larger than those of Teacher-generated SFT for DivPrune and 63.5% larger for SEATS, computed from unrounded three-benchmark mean accuracies. CAFD also achieves higher accuracy on each of the three benchmarks. These consistent benefits support full-context soft distillation for both pre-LLM selection and inner-LLM pruning.

Prior studies of on-policy self-distillation report benefits over SFT on model-generated responses in tool use [1], visual reasoning [2], and mathematics [3]. Together with our consistent cross-benchmark gains, these findings suggest greater generalization potential for CAFD than for Teacher-generated SFT.

2. Adaptation cost.

To assess the accuracy–cost trade-off raised in your review, Table R2 reports offline response-generation and Student-training costs separately. Each SFT recipe's standalone cost includes both stages; both experiments shared one generated response corpus.

We merge LoRA before deployment to avoid adapter operations. Reported costs include startup, checkpoint finalization, and merging; finalization and merging take approximately 90 seconds.

Table R2. Approximate training cost on RTX 6000D GPUs, in allocated GPU-hours.

CompressorTraining methodOffline response generationTrainingTotal
DivPruneTeacher-generated SFT11.7217.8529.58
CAFD-37.0137.01
SEATSTeacher-generated SFT11.7218.6430.36
CAFD-35.5335.53

Teacher-generated SFT has a modestly lower total adaptation cost, while CAFD delivers higher mean accuracy for both compressors.

3. Teacher updates.

To examine EMA's contribution, we compared EMA, frozen, and per-update synchronized Teachers while keeping all other training settings unchanged (Table R3).

Table R3. Teacher-update comparison. DivPrune is used for training (5% retention) and evaluation. Accuracy (%) is averaged over all five deployment budgets.

WeightsTeacher updateWorldSenseDailyOmniAVUTAvg.
Unadapted-42.9954.7458.1951.97
CAFD-adaptedEMA, decay 0.99944.1657.0459.5653.59
CAFD-adaptedFrozen initial Teacher44.0457.1859.6353.62
CAFD-adaptedCurrent Student weights at each optimizer update43.9656.9859.7153.55

All three Teacher-update rules achieve 53.55–53.62% mean accuracy, compared with the 51.97% Unadapted baseline, with only a 0.07-point spread. This consistency supports the full-context-to-compressed-context adaptation framework across the tested Teacher-update choices.

4. Trajectory source.

To address your question about trajectory source, we compared trajectories generated by the compressed Student and the full-context EMA Teacher, keeping the Teacher-update rule and all other CAFD training settings unchanged. Table R4 reports the results.

Table R4. Trajectory-source comparison with JSD fixed. DivPrune is used for training (5% retention) and evaluation. Accuracy (%) is averaged over all five deployment budgets.

WeightsTrajectory sourceWorldSenseDailyOmniAVUTAvg.
Unadapted-42.9954.7458.1951.97
CAFD-adaptedCompressed Student44.1657.0459.5653.59
CAFD-adaptedFull-context EMA Teacher44.2157.4459.7253.79

Student- and Teacher-generated trajectories achieve similar mean accuracies (53.59% and 53.79%, respectively; a 0.20-point difference), both improving over the 51.97% Unadapted baseline. Prior studies report advantages of on-policy over off-policy self-distillation in continual learning [1], multi-turn agent training [4], and knowledge internalization [5]. Together, these findings suggest that the relative advantage depends on the domain and task, while either trajectory source supports effective full-context-to-compressed-context adaptation.

Our contribution is mainly the full-context-to-compressed-context adaptation framework. Our choice of on-policy self-distillation is motivated by its effectiveness in prior studies [1, 3, 4]. The framework is not restricted to this specific recipe: these ablations show that adaptation remains effective across alternative Teacher-update rules and trajectory sources.

W2: An additional backbone and practical efficiency.

1. Additional backbone.

To address your concern about backbone generalization within the rebuttal time constraints, we added Qwen3-Omni-30B-A3B-Instruct [6], using DivPrune for both training and evaluation. We used rank-64 LoRA only on its Thinker text-decoder attention projections (q/k/v/o), keeping MoE experts frozen. We trained one CAFD checkpoint at 5% retention and compared it with its Unadapted DivPrune model at all five budgets on WorldSense, DailyOmni, and AVUT, alongside its full-token reference.

Table R5. Additional OmniLLM backbone. DivPrune is used for both training (5% retention) and evaluation. Compressed accuracies (%) average the five budgets.

BackboneWeightsCompressorWorldSenseDailyOmniAVUTAvg.
Qwen3-Omni-30B-A3B-InstructUnadaptedFull tokens55.0173.9376.9968.65
UnadaptedDivPrune49.4366.1067.1760.90
CAFD-adaptedDivPrune50.6967.9968.0962.26

CAFD improves accuracy across all 15 benchmark–budget combinations on Qwen3-Omni. Mean accuracy rises from 60.90% to 62.26%, supporting adaptation beyond the original Qwen2.5-Omni-7B.

2. Adaptation cost.

To quantify the adaptation cost, Table R6 reports the recorded eight-GPU DivPrune training costs across five training retentions.

Table R6. DivPrune adaptation on eight RTX 6000D GPUs. Allocated GPU-hours are computed as eight times the unrounded elapsed time; both columns are rounded to two decimal places.

Training retentionElapsed time (h)Allocated GPU-hours
35%5.5244.16
25%5.2541.96
15%4.8538.82
10%4.8338.68
5%4.6337.01

The main train-5 run takes 4.63 hours (37.01 allocated GPU-hours).

3. Deployment overhead.

After LoRA merging, CAFD changes only model weights and adds no inference-time computation or memory overhead for pre-LLM compressors at the same retention budget. Runtime and memory savings over full-token inference come from the compressor. For SEATS, adapted weights may change inner-LLM pruning decisions and thus runtime and peak memory.

4. Cost amortization.

To examine whether the adaptation cost can be recovered during deployment, we compared Unadapted-35% with CAFD-25% and the full-token model with CAFD-SEATS-25%. These pairs test whether adaptation permits a lower retention budget while maintaining accuracy; we estimate cost amortization from training allocation and measured per-request latency. Table R7 summarizes the comparison.

The break-even request count is calculated from the measured latency difference:

= \frac{3600 H_{\mathrm{adapt}}} {t_{\mathrm{reference}}-t_{\mathrm{CAFD}}},$$ **Table R7.** Cost amortization on WorldSense. All CAFD checkpoints were trained at 5% retention. Each adapted row is paired with the immediately preceding Unadapted reference. $t_{\mathrm{reference}}$ and $t_{\mathrm{CAFD}}$ denote the corresponding mean E2E latencies per request. For adapted models, training and evaluation use the same compressor. | Compressor | Weights | Eval. retention (%) | Accuracy (%) | Training GPU-h | $t_{\mathrm{reference}}$ − $t_{\mathrm{CAFD}}$ (s/request) | Break-even requests | | --- | --- | --- | --- | --- | --- | --- | | DivPrune | Unadapted | 35 | 45.49 | - | - | - | | | CAFD-adapted | 25 | **46.15** | 37.01 | 0.0912 | 1,461,880 | | SEATS | Unadapted | 35 | 46.44 | - | - | - | | | CAFD-adapted | 25 | **47.32** | 35.53 | 0.1377 | 928,838 | | Full tokens | Unadapted | 100 | 46.85 | - | - | - | | SEATS | CAFD-adapted | 25 | **47.32** | 35.53 | 1.1832 | 108,098 | At 25% retention, both adapted checkpoints exceed their Unadapted-35% reference accuracies and amortize adaptation after approximately 0.93–1.46 million requests. CAFD-SEATS-25% also exceeds full-token accuracy by 0.47 points and amortizes adaptation after approximately 108,000 requests. ## **Q1: Teacher-generated SFT: accuracy and training cost.** Following your suggestion, we compare CAFD with SFT on full-context Teacher responses using matched media, LoRA scope, and training settings. The accuracy comparison and generation-plus-training costs are reported in W1 above, Tables R1–R2. ## **Q2: Frozen Teacher, off-policy trajectories, and unclipped JSD.** Following your suggestion, we examined the three factors separately; W1 and Tables R3–R4 report the frozen-Teacher and off-policy controls. For clipping, we tested whether the adaptation gains depend on this part of the objective by comparing clipped and unclipped JSD. Both runs used on-policy Student rollouts and the same remaining training settings. Table R8 reports the results. **Table R8.** Clipped versus unclipped JSD. DivPrune is used for training (5% retention) and evaluation. Accuracy (%) is averaged over all five deployment budgets. | Weights | JSD clipping | WorldSense | DailyOmni | AVUT | Avg. | | --- | --- | ---: | ---: | ---: | ---: | | CAFD-adapted | Enabled | 44.16 | 57.04 | 59.56 | 53.59 | | CAFD-adapted | Disabled | **44.21** | **57.23** | **59.67** | **53.70** | CAFD improves over Unadapted on all three benchmarks with either clipping setting. The clipped default and unclipped variant achieve 53.59% and 53.70% mean accuracy, respectively, retaining gains of 1.62 and 1.73 points over the 51.97% baseline. Removing clipping therefore preserves the adaptation benefit, with only a 0.11-point difference from the default. Together, these ablations support the adaptation framework across multiple training-recipe choices. ## **Q3: Training cost, inference latency, peak memory, and amortization.** For inference latency and peak memory, W2 provides an architectural explanation based on the merged deployment model and accounts for the SEATS exception. Tables R6–R7 report adaptation GPU-hours and measured per-request latency savings, from which we calculate the break-even request counts. We sincerely thank you for these helpful suggestions. We will consider incorporating the additional experimental results and clarifications into the revised manuscript. --- [1] Idan Shenfeld et al. (2026). Self-Distillation Enables Continual Learning. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 111542-111558). PMLR. [2] Shuai Wang et al. "Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation." arXiv preprint arXiv:2607.23125 (2026). [3] Siyan Zhao et al. (2026). Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models. In Proceedings of the 43rd International Conference on Machine Learning (Vol. 306, pp. 162433-162448). PMLR. [4] Hao Wang et al. "Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents." arXiv preprint arXiv:2604.10674 (2026). [5] Shayan Talaei et al. "Speculative Self-Distillation enables Efficient Knowledge Internalization." arXiv preprint arXiv:2611.12016 (2026). [6] Jin Xu et al. "Qwen3-Omni Technical Report." arXiv preprint arXiv:2509.17765 (2025).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

Dear Reviewers, ACs, SACs, and PCs,

We sincerely thank you all for your time, effort, and dedication throughout the review process. We greatly appreciate the reviewers’ careful evaluations and constructive feedback, which have helped us revise the manuscript, strengthen the experimental support, and clarify the presentation of our work. We are also grateful to the chairs for coordinating the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.