- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the in-domain gains vs. cross-domain degradations in skill internalization through context distillation in language models. They decompose the state-share drift into document- and model-induced components, and show that it can explain the trade-offs between skill gains and cross-domain retention. They propose an algorithm that tracks the drift and corrects the updates to better balance the two objectives. Their experiments with Qwen models on SRA-Bench show that DCSD can provide a better balance between gains on the target skill and retention of the other skills.
Critical strengths
-
The paper studies an important topic of context internalization, specifically as the overhead of multiple skill files compounds.
-
The paper has a decent presentation and clarifies the sources of drift through the distillation process.
-
The experimental setup covers a good set of cross-domain evals for degradation checks and many important context distillation baselines.
Critical weaknesses
-
The key findings of the paper regarding "heterogeneous distillation" and "self-distillation" are drawn based on only one pair of models (Qwen3-8B -> Qwen3-1.7B) and one model (Qwen3-4B), respectively; and in particular one training dataset.. This would question the generality of the claims and whether the observations can be attributed to the heterogeneous vs. self setup, or artifacts of the model/experimental setup. To solidify the findings, I'd suggest testing the findings under different model families and for larger sizes.
-
The main results of the paper (Table 1) are reported with only one seed, with somewhat mixed gains. In both cases, heterogeneous and self-distillation, the delta with respect to the OPCD seems small. Moreover, in both cases, the in-domain performance is lower than some alternating baselines, which somewhat defeats the purpose. As a stronger result, I'd like to see if DCSD can match the baselines in-domain while having lower degradation across other domains.
Questions for authors
-
Having a larger teacher can cause a larger drift, but presumably the teacher should be stronger both in terms of raw capabilities and in-context learning from the skill files. Under this assumption, why wouldn't the student actually learn more from the teacher as opposed to losing its capabilities?
-
Have you compared against ICL/RAG-based baseline of having the skill in the student's context? Should we consider that to be an upper bound for skill internalization, or can the model, after training, outperform those baselines?
-
Please see my other two points above.
5 comments
Response to Reviewer bXCe (2/3): Weakness 2, Question 1
Weakness 2: Results use a single seed, the gains over OPCD are small, and in-domain performance is below some baselines.
Following your suggestion, we retrained DCSD with a different training seed (Qwen3-8B → Qwen3-1.7B):
| DCSD training run | SRA-Bench | Cross-domain Avg. |
|---|---|---|
| Manuscript (Table 1) | 53.5 | 40.8 |
| Different training seed | 53.2 | 40.7 |
The difference between the two runs is at most 0.3 points, much smaller than the cross-domain gain over OPCD (53.1 / 37.6), which is about 3 points.
On the small difference from OPCD: in the newly added Qwen3-14B → Qwen3-4B setting (the table under Weakness 1), DCSD is again higher than OPCD on both SRA-Bench and the cross-domain average (OPCD 61.7 / 54.4, DCSD 64.0 / 56.1), and keeps the cross-domain average above the initial student (55.4). Together with the 8B → 1.7B setting of the manuscript (OPCD 53.1 / 37.6, DCSD 53.5 / 40.8), the improvement is in the same direction in both heterogeneous pairs. In self-distillation (4B → 4B), the teacher and the student start from the same model, so the component that comes from the difference between the two models (the model-induced component) is absent, and the state-shared drift is much smaller than in heterogeneous distillation (Figure 3(a)). OPCD therefore loses almost no cross-domain performance (initial student 55.4 → 55.3), which leaves little loss for DCSD to recover, and the two methods give similar results (OPCD 61.9 / 55.3, DCSD 61.0 / 55.5).
It is true that the in-domain performance is lower than some baselines. In both heterogeneous pairs, the method with the highest SRA-Bench is Off-policy CD (54.6 on 8B → 1.7B, 65.0 on 14B → 4B). However, this high in-domain score comes with a large cross-domain loss. Off-policy CD is a little over 1 point higher than DCSD on SRA-Bench, but its cross-domain average is 8.6 points lower on 8B → 1.7B (Off-policy CD 32.2, DCSD 40.8) and 3.5 points lower on 14B → 4B (Off-policy CD 52.6, DCSD 56.1). That is, at a similar in-domain level, DCSD keeps the degradation on other domains much smaller.
Question 1: If the teacher is stronger, why does the student lose capabilities rather than learn more from it?
In heterogeneous distillation, the student learns the teacher's ability to better interpret and use skill documents. The rise of SRA-Bench from 30.6 to 53.1 (OPCD) and 53.5 (DCSD) is the result. However, the only signal the student receives is the distribution given by the document-conditioned teacher on rollouts generated from training-domain queries. It is therefore hard to say that the teacher's general ability or its in-context learning ability itself is transferred, and an improvement on domains that were not trained on is also hard to expect.
The cross-domain loss arises from this training structure. Because the student follows the teacher distribution on its own rollouts, it also learns the tokens that are repeatedly preferred across states because of the format of the skill documents or the domain of the training tasks, and its responses also become longer. As these preferences carry over to other domains, the instruction-following and rule-following abilities that each domain requires are degraded. DCSD estimates and removes such preferences that recur across states, and thereby reduces this degradation.
Response to Reviewer bXCe (3/3): Question 2
Question 2: Comparison with ICL/RAG baselines, and whether they are an upper bound for skill internalization.
In Figure 1 and Table 3 of the appendix, we compared with giving the initial student documents retrieved from the full SRA-Bench skill pool (26,065 documents). The table below adds the oracle, which assumes that the gold skill is known and provides the gold document directly, and the results with the 14B teacher. Because providing the gold document assumes perfect retrieval, it is an upper bound for retrieval-based skill use, but not for internalization. As in Table 3, the last column reports the prompt tokens added over document-free inference (402 tokens per query).
| Student | Method | SRA-Bench | Additional tokens per query |
|---|---|---|---|
| Qwen3-1.7B | Initial student, no document | 30.6 | 0 |
| + gold document — oracle (assumes the gold skill is known) | 45.3 | 990.9 | |
| + retrieval, BM25 top-1 | 30.0 | 4,385.5 | |
| + retrieval, BM25 top-3 | 31.1 | 13,581.7 | |
| + retrieval, LLM Selection | 32.1 | 6,469.1 | |
| DCSD (8B teacher), no document | 53.5 | 0 | |
| DCSD (8B teacher) + gold document (oracle) | 61.1 | 990.9 | |
| Qwen3-4B | Initial student, no document | 51.6 | 0 |
| + gold document — oracle (assumes the gold skill is known) | 75.2 | 990.9 | |
| + retrieval, BM25 top-1 | 56.2 | 4,524.0 | |
| + retrieval, BM25 top-3 | 56.6 | 13,701.1 | |
| + retrieval, LLM Selection | 61.8 | 5,713.3 | |
| DCSD (self-distillation), no document | 61.0 | 0 | |
| DCSD (self-distillation) + gold document (oracle) | 73.1 | 990.9 | |
| DCSD (14B teacher), no document | 64.0 | 0 | |
| DCSD (14B teacher) + gold document (oracle) | 75.5 | 990.9 |
For the 1.7B student, DCSD without any document (53.5) is higher than the oracle (45.3): the 1.7B student cannot fully use the gold document even when it is given in context, whereas distillation lets it learn how the larger teacher solves these training-domain tasks with the document. For the 4B student, the oracle (75.2) is still higher, but giving the gold document to DCSD with the 14B teacher reaches the same level (75.5). Retrieval, in contrast, adds about 4,400–13,700 prompt tokens per query, and yet the best retrieval for the 4B student (LLM Selection, 61.8) is lower than DCSD with the 14B teacher (64.0), which answers with no additional tokens.
Response to Reviewer bXCe (1/3): Weakness 1
Thank you for the thoughtful questions. We added an experiment with a larger teacher (Qwen3-14B → Qwen3-4B), retrained DCSD with a different training seed, and organized the comparison with in-context skill use.
Weakness 1: The findings rest on a single model pair, a single self-distillation model, and a single training dataset; validate with other model families and larger models.
We added heterogeneous distillation with a larger teacher, Qwen3-14B → Qwen3-4B. In the manuscript, the students of heterogeneous distillation (8B → 1.7B) and self-distillation (4B → 4B) were different, so it was not possible to tell whether the difference between the two settings came from heterogeneous vs. self-distillation or from the student model. The student in the new setting is Qwen3-4B, the same as in the self-distillation of the manuscript, so this is a comparison that keeps the student fixed and changes only the teacher (4B itself → 14B). The training data, schedule, evaluation (full test sets), and table format are the same as in Table 1 (bold = best among the distillation methods in each column):
DCSD is higher than OPCD on both SRA-Bench and the cross-domain average (+2.3, +1.7), and is the only method that keeps the cross-domain average above the initial student (56.1 vs 55.4). In contrast, Off-policy CD, which has the highest in-domain score (65.0), falls 2.8 points below the initial student on cross-domain (52.6); in the 8B → 1.7B setting of Table 1 in the manuscript, it also had the highest SRA-Bench (54.6) but the largest cross-domain loss (43.8 → 32.2). On the same 4B student, changing the teacher from the student itself to Qwen3-14B increases the cross-domain gain of DCSD over OPCD from +0.2 to +1.7. In heterogeneous distillation, the model-induced shift is added, so the cross-domain degradation becomes larger, and the recovery by DCSD is correspondingly larger.
On the training data: the only benchmark used for internalization training is SRA-Bench, but we trained on its four task sets of different nature combined (MedCalc-Bench, TheoremQA, LogicBench, CHAMP). The length of the skill documents (median 652–1,088 tokens) and their notation (whether code blocks and LaTeX are used) differ across tasks, which alleviates to some extent the concern that the training data is biased toward one type. We were not able to add other model families or other training datasets within the rebuttal period, and we will state this as a limitation of the paper.