acceptodds
Under review as a conference paper at ICLR 2027

Context Distillation Distills More Than Skills: Navigating Skill Gains and Cross-Domain Retention with State-Shared Drift

Abstract

Skill documents provide language model agents with reusable procedural knowledge, but retrieving and processing them at inference time incurs computational cost. On-policy context distillation can internalize such knowledge by training a document-free student to match a document-conditioned teacher. However, this process can improve the target skill while degrading instruction and rule following in unrelated domains. We identify state-shared drift: systematic token-score shifts that recur across prediction contexts with context-dependent magnitudes. Under heterogeneous distillation, we decompose this drift into document-induced and model-induced sources whose directions and scales differ substantially. Larger state-shared drift coincides with greater cross-domain degradation, while progressively attenuating it traces a trade-off between document-free skill gains and cross-domain retention. Based on this, we propose Drift-Corrected Skill Distillation (DCSD), which separately estimates the two source-specific drift components and uses them as correction directions. Control documents and selective masking help preserve skill guidance, while correction strength and correcting one or both sources adjust the balance between skill gains and cross-domain retention. Experiments with Qwen3 models on SRA-Bench and six cross-domain instruction- and rule-following benchmarks show that DCSD substantially improves cross-domain retention while preserving useful document-free skill gains.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
context distillation, on-policy distillation, skill internalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bXCeOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the in-domain gains vs. cross-domain degradations in skill internalization through context distillation in language models. They decompose the state-share drift into document- and model-induced components, and show that it can explain the trade-offs between skill gains and cross-domain retention. They propose an algorithm that tracks the drift and corrects the updates to better balance the two objectives. Their experiments with Qwen models on SRA-Bench show that DCSD can provide a better balance between gains on the target skill and retention of the other skills.

Critical strengths
  • The paper studies an important topic of context internalization, specifically as the overhead of multiple skill files compounds.

  • The paper has a decent presentation and clarifies the sources of drift through the distillation process.

  • The experimental setup covers a good set of cross-domain evals for degradation checks and many important context distillation baselines.

Critical weaknesses
  • The key findings of the paper regarding "heterogeneous distillation" and "self-distillation" are drawn based on only one pair of models (Qwen3-8B -> Qwen3-1.7B) and one model (Qwen3-4B), respectively; and in particular one training dataset.. This would question the generality of the claims and whether the observations can be attributed to the heterogeneous vs. self setup, or artifacts of the model/experimental setup. To solidify the findings, I'd suggest testing the findings under different model families and for larger sizes.

  • The main results of the paper (Table 1) are reported with only one seed, with somewhat mixed gains. In both cases, heterogeneous and self-distillation, the delta with respect to the OPCD seems small. Moreover, in both cases, the in-domain performance is lower than some alternating baselines, which somewhat defeats the purpose. As a stronger result, I'd like to see if DCSD can match the baselines in-domain while having lower degradation across other domains.

Questions for authors
  • Having a larger teacher can cause a larger drift, but presumably the teacher should be stronger both in terms of raw capabilities and in-context learning from the skill files. Under this assumption, why wouldn't the student actually learn more from the teacher as opposed to losing its capabilities?

  • Have you compared against ICL/RAG-based baseline of having the skill in the student's context? Should we consider that to be an upper bound for skill internalization, or can the model, after training, outperform those baselines?

  • Please see my other two points above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bXCe (1/3): Weakness 1

Thank you for the thoughtful questions. We added an experiment with a larger teacher (Qwen3-14B → Qwen3-4B), retrained DCSD with a different training seed, and organized the comparison with in-context skill use.

Weakness 1: The findings rest on a single model pair, a single self-distillation model, and a single training dataset; validate with other model families and larger models.

We added heterogeneous distillation with a larger teacher, Qwen3-14B → Qwen3-4B. In the manuscript, the students of heterogeneous distillation (8B → 1.7B) and self-distillation (4B → 4B) were different, so it was not possible to tell whether the difference between the two settings came from heterogeneous vs. self-distillation or from the student model. The student in the new setting is Qwen3-4B, the same as in the self-distillation of the manuscript, so this is a comparison that keeps the student fixed and changes only the teacher (4B itself → 14B). The training data, schedule, evaluation (full test sets), and table format are the same as in Table 1 (bold = best among the distillation methods in each column):

MethodSRA-BenchACPBench MCQACPBench BoolCOLLIEManyIFEvalMulti-IF t1Multi-IF t2Multi-IF t3KOR-BenchIFEval PromptIFEval Inst.Avg.
Initial student (Qwen3-4B)51.655.661.935.344.980.163.848.248.478.284.555.4
Off-policy CD65.056.266.628.040.078.354.535.448.678.284.852.6
OPD²50.650.068.934.632.770.448.629.141.471.077.948.7
OPCD61.752.261.033.144.681.463.846.745.080.486.054.4
DCSD (ours)64.053.768.634.946.081.863.747.146.681.086.656.1

DCSD is higher than OPCD on both SRA-Bench and the cross-domain average (+2.3, +1.7), and is the only method that keeps the cross-domain average above the initial student (56.1 vs 55.4). In contrast, Off-policy CD, which has the highest in-domain score (65.0), falls 2.8 points below the initial student on cross-domain (52.6); in the 8B → 1.7B setting of Table 1 in the manuscript, it also had the highest SRA-Bench (54.6) but the largest cross-domain loss (43.8 → 32.2). On the same 4B student, changing the teacher from the student itself to Qwen3-14B increases the cross-domain gain of DCSD over OPCD from +0.2 to +1.7. In heterogeneous distillation, the model-induced shift is added, so the cross-domain degradation becomes larger, and the recovery by DCSD is correspondingly larger.

On the training data: the only benchmark used for internalization training is SRA-Bench, but we trained on its four task sets of different nature combined (MedCalc-Bench, TheoremQA, LogicBench, CHAMP). The length of the skill documents (median 652–1,088 tokens) and their notation (whether code blocks and LaTeX are used) differ across tasks, which alleviates to some extent the concern that the training data is biased toward one type. We were not able to add other model families or other training datasets within the rebuttal period, and we will state this as a limitation of the paper.

AuthorsOpenReview ↗

Response to Reviewer bXCe (2/3): Weakness 2, Question 1

Weakness 2: Results use a single seed, the gains over OPCD are small, and in-domain performance is below some baselines.

Following your suggestion, we retrained DCSD with a different training seed (Qwen3-8B → Qwen3-1.7B):

DCSD training runSRA-BenchCross-domain Avg.
Manuscript (Table 1)53.540.8
Different training seed53.240.7

The difference between the two runs is at most 0.3 points, much smaller than the cross-domain gain over OPCD (53.1 / 37.6), which is about 3 points.

On the small difference from OPCD: in the newly added Qwen3-14B → Qwen3-4B setting (the table under Weakness 1), DCSD is again higher than OPCD on both SRA-Bench and the cross-domain average (OPCD 61.7 / 54.4, DCSD 64.0 / 56.1), and keeps the cross-domain average above the initial student (55.4). Together with the 8B → 1.7B setting of the manuscript (OPCD 53.1 / 37.6, DCSD 53.5 / 40.8), the improvement is in the same direction in both heterogeneous pairs. In self-distillation (4B → 4B), the teacher and the student start from the same model, so the component that comes from the difference between the two models (the model-induced component) is absent, and the state-shared drift is much smaller than in heterogeneous distillation (Figure 3(a)). OPCD therefore loses almost no cross-domain performance (initial student 55.4 → 55.3), which leaves little loss for DCSD to recover, and the two methods give similar results (OPCD 61.9 / 55.3, DCSD 61.0 / 55.5).

It is true that the in-domain performance is lower than some baselines. In both heterogeneous pairs, the method with the highest SRA-Bench is Off-policy CD (54.6 on 8B → 1.7B, 65.0 on 14B → 4B). However, this high in-domain score comes with a large cross-domain loss. Off-policy CD is a little over 1 point higher than DCSD on SRA-Bench, but its cross-domain average is 8.6 points lower on 8B → 1.7B (Off-policy CD 32.2, DCSD 40.8) and 3.5 points lower on 14B → 4B (Off-policy CD 52.6, DCSD 56.1). That is, at a similar in-domain level, DCSD keeps the degradation on other domains much smaller.

Question 1: If the teacher is stronger, why does the student lose capabilities rather than learn more from it?

In heterogeneous distillation, the student learns the teacher's ability to better interpret and use skill documents. The rise of SRA-Bench from 30.6 to 53.1 (OPCD) and 53.5 (DCSD) is the result. However, the only signal the student receives is the distribution given by the document-conditioned teacher on rollouts generated from training-domain queries. It is therefore hard to say that the teacher's general ability or its in-context learning ability itself is transferred, and an improvement on domains that were not trained on is also hard to expect.

The cross-domain loss arises from this training structure. Because the student follows the teacher distribution on its own rollouts, it also learns the tokens that are repeatedly preferred across states because of the format of the skill documents or the domain of the training tasks, and its responses also become longer. As these preferences carry over to other domains, the instruction-following and rule-following abilities that each domain requires are degraded. DCSD estimates and removes such preferences that recur across states, and thereby reduces this degradation.

AuthorsOpenReview ↗

Response to Reviewer bXCe (3/3): Question 2

Question 2: Comparison with ICL/RAG baselines, and whether they are an upper bound for skill internalization.

In Figure 1 and Table 3 of the appendix, we compared with giving the initial student documents retrieved from the full SRA-Bench skill pool (26,065 documents). The table below adds the oracle, which assumes that the gold skill is known and provides the gold document directly, and the results with the 14B teacher. Because providing the gold document assumes perfect retrieval, it is an upper bound for retrieval-based skill use, but not for internalization. As in Table 3, the last column reports the prompt tokens added over document-free inference (402 tokens per query).

StudentMethodSRA-BenchAdditional tokens per query
Qwen3-1.7BInitial student, no document30.60
+ gold document — oracle (assumes the gold skill is known)45.3990.9
+ retrieval, BM25 top-130.04,385.5
+ retrieval, BM25 top-331.113,581.7
+ retrieval, LLM Selection32.16,469.1
DCSD (8B teacher), no document53.50
DCSD (8B teacher) + gold document (oracle)61.1990.9
Qwen3-4BInitial student, no document51.60
+ gold document — oracle (assumes the gold skill is known)75.2990.9
+ retrieval, BM25 top-156.24,524.0
+ retrieval, BM25 top-356.613,701.1
+ retrieval, LLM Selection61.85,713.3
DCSD (self-distillation), no document61.00
DCSD (self-distillation) + gold document (oracle)73.1990.9
DCSD (14B teacher), no document64.00
DCSD (14B teacher) + gold document (oracle)75.5990.9

For the 1.7B student, DCSD without any document (53.5) is higher than the oracle (45.3): the 1.7B student cannot fully use the gold document even when it is given in context, whereas distillation lets it learn how the larger teacher solves these training-domain tasks with the document. For the 4B student, the oracle (75.2) is still higher, but giving the gold document to DCSD with the 14B teacher reaches the same level (75.5). Retrieval, in contrast, adds about 4,400–13,700 prompt tokens per query, and yet the best retrieval for the 4B student (LLM Selection, 61.8) is lower than DCSD with the 14B teacher (64.0), which answers with no additional tokens.

Reviewer_7wAuOpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Paper makes the following contributions: (1) it frames context-distillation supervision as capturing two kinds of drifts: model-induced and context-induced. It shows that these drifts have different magnitude and directionality. (2) it proposes a new approach DCSD which separately estimates the two source-specific drifts and uses them as correction directions during training. Experiments show that DCSD can induce the same in-domain improvements as baselines on-policy context distillation, with slightly lower out-of-domain degradation.

Critical strengths
  • The decomposition of supervision into model- and document-induced drifts is elegant. This frames a known limitation of context-distillation in a new light. The experiments related to figure 3 show interesting insights related to drift magnitude of heterogenous vs self-distillation, and the directionality and magnitude of the two kinds of drifts.
Critical weaknesses
  • The performance improvements over OPCD, the most relevant baseline, is only marginal. For heterogenous setting, it reduces forgetting by ~3 points, and only by .2 points in the self-distillation setting. At the same time, Appendix D.4 reports that it takes more than twice the number of secs/ step, more than 4 times the time for teacher scoring. Overall, the proposed approach does not show sufficient gains over the simple baseline to justify its added complexity over the simple OPCD baseline.

  • The paper aims to remove the state-shared drift, measured by eq (6). However, the drift can also capture useful guidance like new reasoning strategies. The strategy to account for this (decribed in lines 328) is quite adhoc. Furthermore, can we establish that larger drift actually corresponds to degradation? What happens if we perform context-distillation on only the lower 80 percentile of tokens by drift? What is the in-domain and out-of-domain accruacy/degradation trade-off?

  • The main results are only shown for one heterogenous setting (Qwen3-8B → Qwen3-1.7B) and one self-distillation setting (Qwen3-4B → Qwen3-4B). This is insufficient to establish the efficacy of the method. Furthermore, it is confusing why the experiments do not fix the student model (to say Qwen3-4B) for richer analysis. This would show an apples-to-apples comparison where model-drift is, by construction, eliminated for one of the settings.

  • The paper is written quite confusingly. The notation and explanations is dense. It requires significant re-writing to improve clarity.

Questions for authors

Please address weaknesses 2 and 3 above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 7wAu (1/3): Weakness 1

Thank you for the detailed comments. We ran the experiments you suggested (distilling only the tokens in the lower 80% of drift, and a comparison with the student fixed to Qwen3-4B), and answer the weaknesses in order below.

Weakness 1: The gain over OPCD is marginal relative to the added training cost.

In self-distillation, the teacher and the initial student are the same model, so there is no model-induced drift, and DCSD corrects only the document-induced component. This is why the difference from OPCD is small. In heterogeneous distillation, model-induced drift is added, so DCSD has more to remove and the gain is clear. In the 8B → 1.7B setting of the manuscript, OPCD lowers the cross-domain average by 6.2 points relative to the initial student, whereas DCSD keeps SRA-Bench at the same level (53.5 vs 53.1) and reduces this loss by about half (3.2 points). DCSD costs more to train than OPCD (Table 9), but this cost is paid only once at training time, and the inference cost is the same.

The results of the newly added Qwen3-14B → Qwen3-4B setting with a larger teacher are shown below (same format as Table 1 of the manuscript; bold = best among the distillation methods in each column). DCSD is higher than OPCD on both SRA-Bench (+2.3) and the cross-domain average (+1.7), and is the only method whose cross-domain average remains above the initial student (56.1 vs 55.4).

MethodSRA-BenchACPBench MCQACPBench BoolCOLLIEManyIFEvalMulti-IF t1Multi-IF t2Multi-IF t3KOR-BenchIFEval PromptIFEval Inst.Avg.
Initial student (Qwen3-4B)51.655.661.935.344.980.163.848.248.478.284.555.4
Off-policy CD65.056.266.628.040.078.354.535.448.678.284.852.6
OPD²50.650.068.934.632.770.448.629.141.471.077.948.7
OPCD61.752.261.033.144.681.463.846.745.080.486.054.4
DCSD (ours)64.053.768.634.946.081.863.747.146.681.086.656.1
AuthorsOpenReview ↗

Response to Reviewer 7wAu (2/3): Weakness 2

Weakness 2: The drift may also capture useful guidance, the rule that handles this seems ad hoc, and does larger drift actually lead to degradation (e.g., if only low-drift tokens are distilled)?

DCSD does not subtract the entire drift; it subtracts only the component that recurs across generation states. The correction direction at each state is determined by the average of the changes at the same tokens in the other states (Eq. 11), so a reasoning strategy used only at a specific state hardly enters this direction. However, a preference that recurs across states can still overlap with useful guidance at the current state, and the rule at line 328 is for this case. We do not apply the document-induced correction to the two tokens most directly related to the current prediction: the token the student actually sampled and the token the document-conditioned teacher ranks highest. This reduces the risk of erasing the guidance that the document provides at the current state while subtracting the recurring component. Removing this rule lowers both SRA-Bench (53.5 → 48.9) and the cross-domain average (40.8 → 38.6) (Table 8), which shows that this rule is needed to preserve the signal required for skill learning.

Whether larger drift leads to larger degradation can be seen in the correction-strength experiment of the manuscript (Figure 4). As more of the estimated drift is subtracted from the teacher distribution (α = 0 → 0.5 → 1 → 1.5), that is, as the drift the student learns becomes smaller, the cross-domain loss relative to the initial student decreases steadily, from 6.2 → 4.5 → 3.0 → 2.1 points.

To also examine whether the degradation is concentrated in a few generation states with large drift, we ran the requested experiment that distills only the tokens in the lower 80% of drift. The training procedure is the same as OPCD; only the generation states included in the loss differ. At each generation state s_t (the t-th position of the response), we compute the magnitude of the state-shared drift in the same way as DCSD, exclude the states in the top 20% from the loss, and train only on the remaining 80% of states. The drift is used only to choose which states to exclude; it is not subtracted from the teacher distribution as in DCSD. The results are as follows (Qwen3-8B → Qwen3-1.7B):

MethodSRA-BenchCross-domain Avg.
Initial student (Qwen3-1.7B)30.643.8
OPCD53.137.6
OPCD, trained only on the lower-80%-drift states52.937.3
DCSD53.540.8

Even when the top-20%-drift states are excluded, both SRA-Bench and the cross-domain average stay at the level of OPCD (52.9 / 37.3 vs 53.1 / 37.6). We attribute this to the fact that state-shared drift is not concentrated in particular states but recurs across many generation states, so the same preferences are still learned from the remaining 80% of states. That is, what determines the degradation is not a few states with large drift but the total amount of the recurring component that the student learns; this is why DCSD, which subtracts this component at all states, raises the cross-domain average to 40.8 while maintaining SRA-Bench.

AuthorsOpenReview ↗

Response to Reviewer 7wAu (3/3): Weakness 3, Weakness 4

Weakness 3: Only one heterogeneous and one self-distillation setting; why not fix the student for an apples-to-apples comparison?

In the heterogeneous-distillation experiments of the manuscript, we did not fix the student to 4B because, on SRA-Bench, which we use as the in-domain benchmark, Qwen3-8B is not better than Qwen3-4B. Even with the gold skill document, 8B is lower than 4B on TheoremQA and CHAMP and lower on average (70.4 vs 75.2), so it was not suitable as a teacher for the 4B student (SRA-Bench test accuracy):

ModelContextMedCalcTheoremQALogicBenchCHAMPAvg.
Qwen3-4B (student)With gold document69.569.185.376.975.2
Qwen3-4B (student)No document26.749.272.957.551.6
Qwen3-8BWith gold document75.362.086.857.570.4
Qwen3-8BNo document32.438.473.440.346.1

Following your suggestion, we added a comparison that fixes the student to 4B, using Qwen3-14B, which is stronger than the 4B student, as the teacher; the results are in the table under Weakness 1. Together with the self-distillation setting of the manuscript (4B → 4B, Table 1), this gives a comparison on the same student between the cases without and with model-induced shift: the cross-domain gain of DCSD over OPCD is +0.2 under self-distillation and +1.7 under 14B → 4B, so it grows when model-induced shift is present.

Weakness 4: The writing is hard to follow, and the notation and explanations are dense.

In the revised version, we rewrote the paragraph after Eq. 6 in Section 5 in simpler terms. In the final version, we will further revise the method section as a whole so that it is easier to follow.

As the discussion period draws to a close, we would be grateful if you could take another look at our rebuttal and revised manuscript. We would appreciate your feedback on whether our response addresses your concerns and would be happy to clarify any remaining points. Thank you for your time and consideration.

Reviewer_bUY3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to internalize skill documents so a model can use their procedures without reading them at inference. It finds that on-policy context distillation can improve skill performance while shifting token preferences in ways that reduce performance on unrelated instruction- and rule-following tasks. The proposed method, DCSD, estimates recurring shifts caused by the document and by teacher–student differences, then corrects the teacher’s training signal. Experiments with Qwen3 models show that DCSD preserves substantial document-free skill gains while improving cross-domain retention.

Critical strengths
  • Insightful analysis. The paper identifies recurring shifts in the supervision signal and shows that, in heterogeneous distillation, the model-induced shift is larger than the document-induced shift. This finding motivates correcting the two sources separately.

  • Important problem and targeted method. The paper addresses cross-domain performance loss during skill distillation and proposes a correction informed by its analysis of recurring shifts.

  • Strong central result. DCSD improves cross-domain retention over OPCD while maintaining nearly the same skill performance. The correction sweeps help show the trade-off between the two.

Critical weaknesses
  • The comparison does not fully establish the benefit over simpler tuning. The paper sweeps DCSD’s correction strength, but does not show whether varying OPCD’s learning rate or choosing an earlier checkpoint could achieve similar cross-domain retention at matched skill performance. This matters because DCSD roughly doubles OPCD’s training cost.

  • The mechanism is not fully isolated. The drift analysis and correction sweeps are suggestive, but do not rule out the possibility that a general weakening of distillation would yield similar gains. The paper also does not directly validate whether noisy control documents isolate the intended recurring signal.

Questions for authors
  1. The paper sweeps DCSD’s correction strength but uses a fixed learning rate for OPCD. Could the authors compare DCSD with OPCD across learning rates and earlier checkpoints, reporting cross-domain retention at matched SRA-Bench performance? This would clarify whether DCSD improves the trade-off beyond tuning ordinary distillation, as learning-rate choice can affect capability retention in SFT (Lin et al., ICLR 2026).

  2. Figure 3 associates stronger state-shared drift with greater cross-domain degradation, and Figure 4 shows improved retention when the estimated drift is attenuated. Can the authors provide evidence that the identified direction is responsible for the degradation, rather than the benefit coming from a more general weakening of the distillation signal? For example, how would an equal-strength correction in a random or orthogonal direction compare at a matched skill gain?

  3. What evidence supports the assumption that adding noise to the skill documents suppresses procedure-specific guidance while preserving the recurring document-induced preferences that DCSD aims to estimate?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bUY3 (1/2): Weakness 1 & Question 1, Weakness 2 & Question 2

Thank you for the careful reading and for suggesting concrete experiments. We ran both experiments (a comparison against OPCD with different learning rates and checkpoints, and a correction of the same strength along an orthogonal direction), and answer your questions in order below.

Weakness 1 & Question 1: Can OPCD with a different learning rate or an earlier checkpoint match DCSD's cross-domain retention at the same SRA-Bench performance?

As suggested, we retrained OPCD on Qwen3-8B → Qwen3-1.7B with four learning rates (5e-7, 1e-6, 2e-6, 5e-6) and evaluated checkpoints from step 100 at 200-step intervals up to convergence. The OPCD run in the manuscript (lr 1e-6) did not keep intermediate checkpoints, so the 1e-6 row is also a retraining under the same setting:

SRA-Bench / Cross-domain Avg.step 100step 300step 500step 700Converged
OPCD, lr 5e-740.4 / 41.142.6 / 40.049.4 / 40.051.6 / 39.651.6 / 39.6 (step 700)
OPCD, lr 1e-6 (manuscript setting)43.3 / 39.850.5 / 39.051.5 / 38.253.3 / 38.053.3 / 38.0 (step 700)
OPCD, lr 2e-646.3 / 38.753.9 / 37.054.3 / 36.352.3 / 36.151.8 / 35.8 (step 800)
OPCD, lr 5e-649.9 / 35.3–––52.0 / 33.6 (step 200)
OPCD (manuscript, Table 1)53.1 / 37.6 (step 600)
DCSD (manuscript, Table 1)53.5 / 40.8 (step 500)

For all four learning rates, the cross-domain average of OPCD decreases as training proceeds. DCSD has a cross-domain average of 40.8 at SRA-Bench 53.5, whereas OPCD checkpoints with a similar SRA-Bench (53–54) have 36.3–38.0.

As you noted, a lower learning rate reduces the cross-domain loss (cross-domain average at convergence: 5e-7 39.6, 1e-6 38.0, 2e-6 35.8, 5e-6 33.6). However, SRA-Bench also drops: with the lowest learning rate (5e-7), the converged checkpoint is at 51.6 / 39.6, lower than DCSD (53.5 / 40.8) on both metrics. The only checkpoint whose cross-domain average is at the DCSD level is step 100 of lr 5e-7 (40.4 / 41.1), and there SRA-Bench is 13 points lower than DCSD.

In short, neither adjusting the learning rate of OPCD nor selecting an earlier checkpoint achieves the in-domain performance and the cross-domain performance of DCSD together.

Weakness 2 & Question 2: Does the gain come from the estimated drift direction, or from a generally weaker distillation signal?

Following your suggestion, we ran a control experiment that applies a correction of the same strength along a different direction. At each generation state, we compute the DCSD correction vector, keep its magnitude, replace only its direction with a random direction orthogonal to it, and subtract the result from the teacher distribution. The values at the two masked tokens are kept the same as in DCSD, so the two training runs differ only in the direction of the correction. Because the vocabulary space in which the correction lies is high-dimensional, a random direction is also nearly orthogonal to the estimated direction, so this single experiment answers both the random-direction and the orthogonal-direction cases (Qwen3-8B → Qwen3-1.7B):

MethodSRA-BenchCross-domain Avg.
Initial student (Qwen3-1.7B)30.643.8
OPCD53.137.6
DCSD (α = 1)53.540.8
DCSD (α = 1.5)50.041.7
Random orthogonal correction (same strength as DCSD)49.941.0

The orthogonal correction of the same strength also raises the cross-domain average to a level similar to DCSD (41.0), but SRA-Bench drops to 49.9. That is, changing the teacher distribution by the same strength regardless of direction weakens the signal overall and loses skill gain as well, whereas DCSD, which corrects along the estimated direction, obtains the same level of retention while maintaining SRA-Bench (53.5). Comparing at the same skill gain, at SRA-Bench of about 50, DCSD (α = 1.5) is slightly higher than the orthogonal correction on both metrics (50.0 / 41.7 vs 49.9 / 41.0).

AuthorsOpenReview ↗

Response to Reviewer bUY3 (2/2): Weakness 2 & Question 3

Weakness 2 & Question 3: Do noisy control documents suppress procedure-specific guidance while preserving the recurring document-induced preferences?

In DCSD, control documents are used only to estimate the direction of preferences that recur across states. Below is an example of a control document actually used in training (part of the TheoremQA skill "Linear Systems").

Original:

**Key cases:**
- Unique solution: rank(A) = rank([A|b]) = n (full column rank).
- No solution: rank(A) < rank([A|b]) (inconsistent).
- Infinitely many solutions: rank(A) = rank([A|b]) < n (underdetermined).

Control document:

###Solution points:**
- Square solution when n(A) = n([A bb]) = n but rank rank)
- Infinite solution ( rank(A) ≠ rank([A|b]). (inconsistent system
- Infiniteinitely many solutions: rank(A) = rank([A|b]) < n (underdetermined system

The list structure, the formatting, and domain expressions such as rank and augmented matrix remain, but the rules that carry the exact knowledge are broken or replaced with incorrect content (the unique-solution condition collapses, and the no-solution case becomes "infinite solution"). The remaining domain expressions and formatting push the teacher distribution in the same direction across many states of the rollout, as the original document does. Therefore, by collecting only the component that recurs across generation states from the change induced by the control documents (Eq. 11), we obtain a direction similar to the recurring component induced by the original document, while the exact procedural knowledge used only at specific steps hardly enters this direction. We will add examples of control documents to the appendix of the revised version.

As the discussion period draws to a close, we would be grateful if you could take another look at our rebuttal and revised manuscript. We would appreciate your feedback on whether our response addresses your concerns and would be happy to clarify any remaining points. Thank you for your time and consideration.

Thank you for the new experiments; Q1 is resolved, but the orthogonal control matches DCSD’s cross-domain retention (41.0 vs. 40.8), so I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.