acceptodds
Under review as a conference paper at ICLR 2027

Learning from Interaction Experience via Localized Context Distillation

Abstract

Learning from interaction experience is essential for enabling large language models to continually improve during deployment. Context distillation offers a promising path: privileged information extracted from experience is provided to a teacher, whose behavior is then distilled into a student model that has no access to it. However, both existing forms of context distillation have limitations. Off-policy context distillation trains the student on teacher-generated trajectories and thus suffers from exposure bias; whereas on-policy context distillation trains on the student's own rollouts, whose states are often uninformative and fail to convey behaviors the student cannot yet produce. To resolve this dilemma, we introduce Localized Context Distillation (LoCD). LoCD first localizes the critical spans of a student trajectory that require revision. The teacher then regenerates only these spans from the student's own prefix, and the student is trained on the regenerated segments. LoCD thereby concentrates learning on the most informative states while keeping exposure bias low. Extensive experiments across mathematical reasoning, code generation, and agent tasks demonstrate that LoCD consistently outperforms strong baselines and achieves more stable training with smaller and sparser parameter updates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, On-Policy Learning, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BjXKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Localized Context Distillation, or LoCD, to address limitations of on-policy and off-policy distillation. LoCD identifies critical spans in student trajectories and regenerates only those spans using privileged teacher context. The method is evaluated on mathematical reasoning, coding, and agent tasks, with improvements over several distillation baselines.

Critical strengths
  1. The method is simple and well motivated. Localizing teacher intervention to critical spans is a reasonable way to combine on-policy student trajectories with off-policy correction.

  2. The reported improvements are consistent across mathematical reasoning, coding, and agent tasks. Some gains are substantial, particularly on LiveCodeBench and ALFWorld.

  3. The initial-accuracy analysis provides a useful explanation for why on-policy distillation struggles when the student rarely generates the desired behavior.

Critical weaknesses
  1. The main comparisons lack sufficient training details, uncertainty reporting, and compute matching. It is difficult to determine whether the reported gains would hold under comparable budgets and independent runs.

  2. Important baselines are missing from the main evaluation, particularly reference-solution SFT and a simple mixture of on-policy and off-policy distillation (e.g., GKD; Agarwal et al., 2024). Figure 10 includes SFT results but does not explain them adequately.

  3. The paper does not sufficiently isolate the contribution of localization. LoCD is absent from the diagnostic experiments in Figures 1–2, and localization accuracy, exposure bias, and forgetting are not directly evaluated.

  4. Some results are difficult to reconcile. Table 1 is inconsistent about the model checkpoint, Table 2 contains identical entries for two baselines, and the WebShop results in Figure 6 appear different from Table 3.

Questions for authors
  1. Can the authors compare LoCD against reference-solution SFT and an on-policy/off-policy mixture under matched compute budgets, and report results across independent seeds? This would help establish whether the improvements are robust.

  2. Can the authors evaluate LoCD in the initial-accuracy experiments of Figures 1–2, including held-out problems, and provide ablations that isolate localization from other changes in supervision? This would directly test the paper's central motivation.

  3. Can the authors provide the full localization procedure and clarify the model checkpoint, duplicated baseline results in Table 2, and the discrepancy between Figure 6 and Table 3? These details are important for interpreting and reproducing the results.

8 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer BjXK [1/5]

We appreciate the reviewer's careful reading and constructive suggestions. We group related Weaknesses and Questions and respond to each below. The revisions made in response to these comments are highlighted in blue in the revised manuscript.

[W1] The main comparisons lack sufficient training details, uncertainty reporting, and compute matching. It is difficult to determine whether the reported gains would hold under comparable budgets and independent runs.

[Response] We thank the reviewer for raising these points on experimental rigor. In summary:

  • Training details: the revised manuscript adds a complete appendix of experimental details (Appendix C).
  • Uncertainty: math and code results are reported over three independent runs, and LoCD's main gains are statistically significant.
  • Compute matching: all methods, including the baselines added in this rebuttal, are trained under the same budget.

The details are as follows.

(1) The revised manuscript adds a complete appendix of experimental details (Appendix C). It covers datasets and splits (C.1), feedback construction for each task (C.2), training configurations (C.3 and Table 4), span statistics (C.3 and Table 5), and evaluation and checkpoint-selection protocols (C.4 and Table 6). All methods use full-parameter fine-tuning with AdamW and a constant learning rate of 1e-6, with task-specific batch sizes and numbers of training steps listed in Table 4.

(2) Results are reported over three independent runs, and the main gains are statistically significant. Math and code results come from three independent training runs, each selecting its own checkpoint, and are reported as mean ± std (Tables 1–2). LoCD achieves the best result on all 20 math and code benchmark columns. The improvements over the strongest baseline are statistically significant (one-sided Welch's t-test, p < 0.05):

Table R1: Significance of LoCD's gains over the strongest baseline on Qwen3-8B (three runs each).

BenchmarkStrongest baselineLoCDGainp-value
AIME2538.98 ± 2.447.87 ± 4.6+8.890.029
HMMT25-Nov29.17 ± 4.338.43 ± 3.0+9.260.022
MATH-50091.07 ± 1.193.80 ± 1.5+2.730.035
LiveCodeBench18.19 ± 0.824.14 ± 2.1+5.950.014

For agent tasks, LoCD achieves the best results on all four tasks (WebShop, ALFWorld, SWE-Gym, and SWE-bench Verified) and stays above the baselines throughout training (Figure 6). Since agent training requires extensive environment interaction, we will include multiple independent runs in a future version.

(3) All methods are compared under the same budget. To ensure a fair comparison, LoCD and all baselines, including the reference-solution SFT and GKD baselines added in this rebuttal, share the same model, task data, feedback sources, batch size, and number of optimization steps within each task (Section 5.1 and Appendix C.3 in the revised manuscript). The newly added baselines are further matched with LoCD in training tokens.

AuthorsOpenReview ↗

Rebuttal to Reviewer BjXK [2/5]

[W2 & Q1] Important baselines are missing from the main evaluation, particularly reference-solution SFT and a simple mixture of on-policy and off-policy distillation (e.g., GKD; Agarwal et al., 2024). Figure 10 includes SFT results but does not explain them adequately.

Can the authors compare LoCD against reference-solution SFT and an on-policy/off-policy mixture under matched compute budgets, and report results across independent seeds? This would help establish whether the improvements are robust.

[Response] We thank the reviewer for the helpful suggestion. Following it, we add two baselines on Qwen3-8B for math, code, and WebShop:

  • Reference-solution SFT [1]: supervised fine-tuning directly on reference solutions. Since WebShop has no reference solutions, we use trajectories generated by Claude Opus 5.5.
  • GKD [2] (on/off-policy mixture): we follow the original setting and official implementation of GKD, which mixes student-generated and teacher-generated trajectories, both supervised by the same context-conditioned teacher.

Both baselines share the same data, feedback sources, batch size, and optimization steps as LoCD, with matched training tokens.

Table R2: Comparison with reference-solution SFT and GKD on Qwen3-8B (%).

MethodAIME24AIME25LiveCodeBenchWebShop
Reference-solution SFT52.7836.9418.7145.80
GKD45.8337.5018.0526.60
LoCD61.3047.8724.1452.40

LoCD consistently outperforms both baselines by large margins, improving over reference-solution SFT by 5.43–10.93 points and over GKD by 6.09–25.80 points. Reference-solution SFT trains the student entirely on text it would not generate itself; GKD still trains on whole teacher-generated trajectories in its off-policy batches, while its on-policy batches rarely contain the behaviors the student is missing. In contrast, LoCD keeps the student's own trajectory and introduces teacher supervision only on short spans that need revision, which provides the missing behaviors while keeping training close to the student's own distribution.

Regarding Figure 10, we apologize for the confusion. "SFT" in this figure denotes TRD [3], the off-policy context-distillation baseline in our main experiments, rather than reference-solution SFT. In the revised manuscript (Figure 14), we relabel it as "Off-policy" and explain in the caption that Off-policy and On-policy denote TRD and OPCD, respectively.

[1] Ouyang, Long, et al. "Training Language Models to Follow Instructions with Human Feedback." NeurIPS, 2022.

[2] Agarwal, Rishabh, et al. "On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes." ICLR, 2024.

[3] Jiang, Li, et al. "Trajectory-Refined Distillation." arXiv:2606.08432, 2026.

AuthorsOpenReview ↗

Rebuttal to Reviewer BjXK [3/5]

[W3 & Q2] The paper does not sufficiently isolate the contribution of localization. LoCD is absent from the diagnostic experiments in Figures 1–2, and localization accuracy, exposure bias, and forgetting are not directly evaluated.

Can the authors evaluate LoCD in the initial-accuracy experiments of Figures 1–2, including held-out problems, and provide ablations that isolate localization from other changes in supervision? This would directly test the paper's central motivation.

[Response] We thank the reviewer for the suggestions on directly testing our central motivation. In summary:

  • LoCD in Figures 1–2: LoCD substantially outperforms on-policy context distillation on low- and zero-initial-accuracy problems, including held-out problems.
  • Ablations: each of LoCD's three components contributes to its performance, and the best performance requires all three.
  • Localization accuracy, exposure bias, and forgetting: we evaluate each of them with controlled variants, span statistics, and parameter-update analysis.

The details are as follows.

(1) LoCD overcomes the low-initial-accuracy failure of on-policy context distillation. We add LoCD to Figures 1–2 of the revised manuscript, under the same settings as the original diagnostic experiments.

Low-initial-accuracy problems (Figure 1). Problems are binned by each model's own initial avg@128. Table R3 aggregates the bins with [0, 40]% initial accuracy; full per-bin results are shown in Figure 1 of the revised manuscript.

Table R3: avg@128 (%) on problems with initial accuracy in [0, 40]%.

ModelOn-policy (OPCD [4])Off-policy (TRD [3])LoCD
Qwen3-4B21.0838.2943.94
Qwen3-8B29.0962.6567.13
Falcon3-10B-Instruct13.5533.3334.24

Zero-initial-accuracy problems (Figure 2). We collect problems on which the initial student fails in all 128 rollouts, and split them into training and held-out problems. All methods are trained on the training problems, and Table R4 reports the results on the held-out problems, which are never seen during training.

Table R4: avg@128 (%) on held-out problems with zero initial accuracy, after 100 training steps (Figure 2 in the revised manuscript).

ModelOn-policyOff-policyLoCD
Qwen3-4B2.4728.1232.44
Qwen3-8B3.0043.2356.23

On these problems, On-policy barely improves, whereas LoCD learns to solve them, and the gains carry over to problems never seen in training. Moreover, when OlympiadBench from Table 1 is stratified by initial accuracy, LoCD outperforms On-policy in every bin up to 60% initial accuracy, by up to 19.06 points, and achieves the best overall avg@128 (66.01 vs. 62.08 for Off-policy and 61.75 for On-policy).

[3] Jiang, Li, et al. "Trajectory-Refined Distillation." arXiv:2606.08432, 2026.

[4] Ye, Tianzhu, et al. "On-Policy Context Distillation for Language Models." arXiv:2602.12275, 2026.

Reviewer_LaAsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a hybrid approach for self-distillation by combining on-policy prefixes and off-policy corrections. The teacher sees hints extracted from the feedback as privileged information. Based on that feedback, the same model tries to identify errors in the reasoning trace. The authors diagnose that on-policy distillation is less effective when correct reasoning rarely appears in the student's samples. Motivated by this, off-policy distillation is applied only on "critical" spans where errors were identified.

The largest contribution, in my view, is the mentioned diagnosis, which is validated across three models, tasks with zero initial accuracy, and synthetic knowledge-injection tasks. Other contributions are the proposed method, demonstrated gains on different domains, and more stable training with smaller (more local) updates.

Critical strengths
  • The authors show that on-policy self-distillation becomes much less effective when the student rarely (or never) produces the behavior induced by privileged context. This diagnosis is valuable in general, regardless of the proposed method, and it is well supported with evidence (Figures 1, 2, and 3).

  • The improvements are consistent across a broad range of domains, including code, math, interactive agents, and SWE tasks.

Critical weaknesses
  • The proposed mechanism is not directly quantified and verified. Exposure bias and localization accuracy are never measured, and LoCD is not tested on the zero-accuracy and length-controlled experiments. Additionally, there is no control for incorrect hints. This brings into question where the gains come from, and whether the suggested reasons are empirically supported.

  • Although the experiments are broad, there are some issues with empirical rigor. If I understood correctly, there are five checkpoints per training run on math and code datasets. The authors report the highest score on the test set, among those five checkpoints. This introduces a selection bias, which is especially problematic on small datasets like AIME (only 30 problems). Some of the reported gains in Tables 1 and 2 have small margins and fall within one std. deviation. Additionally, agent results come from a single training run, and are hard to assess since there is no variance estimate.

Questions for authors
  1. Does LoCD fix/mitigate the failures it was designed for (e.g., experiments used in Figs. 1, 2, and 3)?
  2. Do the gains for math and code hold under proper checkpoint selection (e.g., using held-out problems for validation)?
6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer LaAs [1/3]

We thank the reviewer for the valuable feedback on our analysis and empirical rigor. Since the Weaknesses and Questions are closely related, we respond to them jointly. For ease of reference, the key revisions are marked in blue in the revised manuscript.

[W1 & Q1] The proposed mechanism is not directly quantified and verified. Exposure bias and localization accuracy are never measured, and LoCD is not tested on the zero-accuracy and length-controlled experiments. Additionally, there is no control for incorrect hints. This brings into question where the gains come from, and whether the suggested reasons are empirically supported.

Does LoCD fix/mitigate the failures it was designed for (e.g., experiments used in Figs. 1, 2, and 3)?

[Response] We address this concern in three parts: (1) whether LoCD mitigates the failures it is designed for; (2) where the gains come from, including a control for incorrect hints; and (3) how LoCD limits exposure bias.

(1) LoCD mitigates the low-initial-accuracy failure of on-policy context distillation. Section 3 shows that on-policy context distillation struggles when correct behavior rarely appears in the student's own rollouts. We add LoCD to Figures 1–2 of the revised manuscript to test this directly under the same settings.

Low-initial-accuracy problems (Figure 1). Problems are binned by each model's own initial avg@128. Table R1 summarizes the problems with initial accuracy in [0, 40]%.

Table R1: avg@128 (%) on problems with initial accuracy in [0, 40]% (Figure 1 in the revised manuscript).

ModelOn-policy (OPCD [1])Off-policy (TRD [2])LoCD
Qwen3-4B21.0838.2943.94
Qwen3-8B29.0962.6567.13
Falcon3-10B-Instruct13.5533.3334.24

LoCD achieves the highest accuracy on all three models, improving over On-policy by 22.86, 38.04, and 20.69 points.

Zero-initial-accuracy problems (Figure 2). These problems are never solved in 128 initial rollouts. Table R2 reports the held-out problems that are not used for training.

Table R2: avg@128 (%) on held-out problems with zero initial accuracy, after 100 training steps (Figure 2 in the revised manuscript).

ModelOn-policyOff-policyLoCD
Qwen3-4B2.4728.1232.44
Qwen3-8B3.0043.2356.23

On these problems, the student never produces a correct answer on its own, so On-policy barely improves. LoCD learns to solve them, and the gains carry over to held-out problems that are never used in training.

Gains in Table 1 also concentrate on low-initial-accuracy problems. We stratify OlympiadBench from Table 1 by the base model's per-problem initial avg@128, and evaluate all methods on the same groups.

Table R3: avg@128 (%) of Qwen3-8B on OlympiadBench by initial accuracy.

Initial acc. (%)0(0,20](20,40](40,60](60,80](80,100]All
#Problems105117534861290674
Base0.006.4529.6450.6769.8896.7054.99
Off-policy7.6530.5963.4166.2373.9091.0762.08
On-policy2.3214.0758.1472.4879.5997.6361.75
LoCD7.3333.1371.5875.8879.3295.0766.01

LoCD outperforms On-policy in every bin up to 60% initial accuracy, by 3.40–19.06 points, and achieves the best overall accuracy.

Knowledge injection (Figure 3). This experiment isolates the length dependence of on-policy context distillation, so each sample contains only a query and a direct answer. Since the answer is a synthetic fact the student has never seen, the student is wrong from the first token, leaving no correct prefix to keep. The span localized by LoCD is therefore the entire answer, making LoCD identical to off-policy context distillation, which Figure 3 already shows learns answers of all lengths quickly. LoCD's advantage of correcting only the erroneous part arises in multi-step trajectories, as validated in Figures 1–2 and Tables 1–3.

[1] Ye, Tianzhu, et al. "On-Policy Context Distillation for Language Models." arXiv:2602.12275, 2026.

[2] Jiang, Li, et al. "Trajectory-Refined Distillation." arXiv:2606.08432, 2026.

AuthorsOpenReview ↗

Rebuttal to Reviewer LaAs [2/3]

[W1 & Q1] (continued)

(2) The gains come from correcting the right position with the right hint, not from simply adding teacher-generated text. Since there is no ground-truth annotation of where a trajectory goes wrong, localization accuracy cannot be measured directly. We therefore test it with controlled variants, each replacing one component of LoCD while keeping everything else unchanged:

  • Random-point: replaces the feedback-guided correction point with a randomly selected one.
  • Raw Experience: uses the complete interaction experience as privileged context, instead of a targeted hint.
  • Full-train: extends teacher generation and distillation from the localized span to the entire remaining trajectory.
  • Shuffled hint (incorrect-hint control, new in this rebuttal): keeps the correction point, but replaces the hint with one from a randomly selected other problem.

Table R4: Ablation of LoCD on Qwen3-8B (%).

MethodAIME24LiveCodeBenchWebShop
Base28.0616.7118.00
Random-point51.0519.2138.20
Raw Experience59.2021.2946.20
Shuffled hint42.2316.5828.40
Full-train58.1317.2148.40
LoCD61.3024.1452.40
  • Incorrect hints substantially hurt performance. Shuffled hint drops by 19.07, 7.56, and 24.00 points, falling below Base on LiveCodeBench. The amount of teacher-generated text is unchanged, so if it alone drove the gains, changing only the hint would not cause such a drop.
  • The correction point also matters. Random-point drops by 10.25, 4.93, and 14.20 points, showing that correcting at the feedback-localized position is clearly better than correcting at an arbitrary one.

The revised manuscript defines the first three variants in Section 5.4, and their numbers are labeled in Figure 9.

(3) LoCD limits exposure bias by supervising only a small fraction of tokens with teacher-generated text. Exposure bias arises when the model is trained on text it would not generate itself at inference. In LoCD, the teacher only rewrites short spans; the rest of each trajectory is generated by the student itself.

Table R5: Span statistics of LoCD (Table 5 in Appendix C.3 of the revised manuscript).

TaskAvg. span length (tokens)Supervised tokens (%)
Math43.69.0
ALFWorld30.516.8
WebShop18.37.3
  • The teacher-generated part is small. Corrected spans contain only 18–44 tokens on average, covering only 7–17% of trajectory tokens, so most of what the student is trained on is its own text.
  • Making the teacher-generated part longer hurts. Full-train, which lets the teacher rewrite the entire remaining trajectory, drops by 3.17, 6.93, and 4.00 points on AIME24, LiveCodeBench, and WebShop (Figure 9 in the manuscript). Correcting only a short span therefore fixes the error without training the student on long teacher-generated text.
AuthorsOpenReview ↗

Rebuttal to Reviewer LaAs [3/3]

[W2 & Q2] Although the experiments are broad, there are some issues with empirical rigor. If I understood correctly, there are five checkpoints per training run on math and code datasets. The authors report the highest score on the test set, among those five checkpoints. This introduces a selection bias, which is especially problematic on small datasets like AIME (only 30 problems). Some of the reported gains in Tables 1 and 2 have small margins and fall within one std. deviation. Additionally, agent results come from a single training run, and are hard to assess since there is no variance estimate.

Do the gains for math and code hold under proper checkpoint selection (e.g., using held-out problems for validation)?

[Response] We address this concern in three parts: (1) checkpoint selection, (2) the margins of the gains, and (3) the reliability of the agent results.

(1) Checkpoints are selected on validation sets that are separate from the reported test sets. All methods follow the same rule:

  • Math: AIME24 serves as the validation set, as in Waypoint-OPD [3]; the other seven benchmarks are not used for selection.
  • Code: 100 problems with ratings in [800, 1600) from open-r1/codeforces serve as the validation set, disjoint from both the 1,793 training problems and CF-422. No benchmark reported in Table 2 is used for checkpoint selection.
  • WebShop: following the WebShop benchmark [4], a 150-task subset of the official development split is used for selection, and results are reported on the full 500-task test set.
  • ALFWorld: since valid_seen and valid_unseen both serve as evaluation sets, 150 tasks held out from the training directory are used for selection.

For math and code, each of the three independent runs selects its own checkpoint, and we report mean ± std on the test sets. The revised manuscript adds a full description of the data splits (Appendix C.1) and the evaluation and checkpoint-selection protocol (Appendix C.4).

Since AIME24 is used for validation, we also report the average over the remaining seven math benchmarks, where LoCD still outperforms the strongest baseline:

Table R6: Average accuracy (%) on the seven math benchmarks excluding AIME24.

ModelStrongest baselineLoCDGain
Qwen3-8B40.69 (OPCD)46.14+5.45
Falcon3-10B-Instruct23.46 (TRD)24.80+1.34

The advantage of LoCD therefore does not depend on checkpoint selection.

(2) The gains are consistent, and the main gains clearly exceed one standard deviation. LoCD achieves the best result on all 20 benchmark columns in Tables 1–2 (eight math benchmarks for each of the two models, plus four code benchmarks). Table R7 lists the averages, which are the primary metrics in Tables 1–2, together with the benchmarks where LoCD shows the largest gains. These gains exceed the standard deviations of both LoCD and the strongest baseline.

Table R7: Main gains of LoCD over the strongest baseline on Qwen3-8B (%).

TaskBenchmarkStrongest baselineLoCDGain
MathAverage42.39 (OPCD)48.04+5.65
MathAIME2538.98 ± 2.4 (OP²SD [5])47.87 ± 4.6+8.89
MathHMMT25-Nov29.17 ± 4.3 (OPCD)38.43 ± 3.0+9.26
CodeAverage14.55 (TRD)17.20+2.65
CodeLiveCodeBench18.19 ± 0.8 (TRD)24.14 ± 2.1+5.95

The smaller margins occur mainly on the hardest benchmarks, where every method scores very low (e.g., below 1% on CF$_{1600+}$), leaving little room for any difference.

(3) The agent results are consistent across training and tasks. LoCD stays above the on-policy and off-policy baselines throughout training in Figure 6, and achieves the best results on all four agent tasks: WebShop, ALFWorld, SWE-Gym, and SWE-bench Verified. This consistency makes it unlikely that the advantage comes from a lucky single run. Since agent training and evaluation require extensive environment interaction, we will include results from multiple independent runs in a future version, given the limited compute during the rebuttal.

[3] Hou, Bingxi, et al. "Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation." arXiv:2610.21601, 2026.

[4] Yao, Shunyu, et al. "WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents." NeurIPS, 2022.

[5] Ichihara, Yuki, et al. "Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation." arXiv:2608.09228, 2026.

Reviewer_bzVGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how an LLM can internalize “interaction experience” (its own trajectory plus feedback) into parameters via context distillation, where a teacher that sees experience-derived privileged context supervises a student that does not. It first presents a diagnostic study arguing that on-policy context distillation (OPCD) struggles when the target behavior is rare in the student’s own rollouts: on OpenThoughts problems binned by initial accuracy, OPCD’s gain over off-policy distillation vanishes at low accuracy and is near zero on zero-accuracy problems, and a synthetic knowledge-injection task shows on-policy learning time grows with answer length. The authors attribute this to a sequential dependency: later tokens only become learnable once earlier tokens appear in the student’s rollouts. They then propose Localized Context Distillation (LoCD): a prompt-based localizer identifies critical spans in the student trajectory and the matching privileged context, the teacher regenerates only that span from the student’s prefix, and the student is trained token-wise on just that span. Experiments on math, code, WebShop, ALFWorld, and SWE-Gym/SWE-bench Verified with Qwen3-8B, Falcon3-10B, and Qwen3-Coder-30B-A3B show LoCD beating six distillation baselines on average, retaining entropy, and producing smaller and sparser weight updates.

Critical strengths
  1. The paper provides a systematic empirical diagnosis of when on-policy context distillation struggles, combining initial-accuracy analysis with controlled knowledge-injection experiments.
  2. LoCD has a simple, intuitive design and an explicit training objective that connects corrective teacher supervision to student-generated prefixes.
  3. Evaluation across four task settings and multiple model backbones shows consistent aggregate improvements, supported by additional analyses of training dynamics and parameter changes.
Critical weaknesses

1.The paper does not directly test whether LoCD overcomes the low-initial-accuracy and length-dependent learning failures used to motivate it: LoCD is absent from the Section 3 experiments, and Table 1 gains are not stratified by initial accuracy. By the paper’s own analysis, within-span supervision is off-policy and post-span continuation is unsupervised, so span length governs both coverage of absent behavior and exposure bias at once. This coverage–exposure tradeoff remains uncharacterized because span lengths, supervised-token fractions, and performance across initial-accuracy bins are not reported. 2. The experiments support the effectiveness of the combined LoCD procedure, but not the attribution to localization and attribution specifically. The Figure 9 variants are undefined, and as drawn the random-point variant already approaches the strongest baseline on AIME24. The closest prior method, backtracking distillation (Wang et al. 2026a), is absent. LoCD’s teacher receives span-specific privileged context while baseline teachers receive a single global context, and training budgets per interaction and across rounds are unreported.

Questions for authors
  1. Can you add LoCD to the Section 3 experiments (the zero-initial-accuracy subset in Figure 2 and the knowledge-injection task in Figure 3), and stratify the Table 1 gains by initial-accuracy bin? Please also report the distribution of span lengths and the fraction of trajectory tokens that receive supervision, so the coverage–exposure tradeoff can be assessed directly.
  2. Please define the “Random-point”, “Raw trajectory”, and “Full-train” variants in Figure 9 and report their numbers. Can you add backtracking distillation (Wang et al. 2026a) as a baseline on math, code, and at least one agent task, under matched training tokens?
  3. Do the baseline teachers receive the same span-specific privileged context as LoCD’s teacher, or a single global context? How many experience-collection rounds are used, how many training instances does each interaction yield for LoCD versus the baselines, and which divergence is used in the main tables?
7 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer bzVG [1/4]

We sincerely thank the reviewer for the insightful comments on our mechanism and baselines. As several Weaknesses overlap with the Questions, we address them together below. The main revisions in the revised manuscript are highlighted in blue.

[W1 & Q1] The paper does not directly test whether LoCD overcomes the low-initial-accuracy and length-dependent learning failures used to motivate it: LoCD is absent from the Section 3 experiments, and Table 1 gains are not stratified by initial accuracy. By the paper’s own analysis, within-span supervision is off-policy and post-span continuation is unsupervised, so span length governs both coverage of absent behavior and exposure bias at once. This coverage–exposure tradeoff remains uncharacterized because span lengths, supervised-token fractions, and performance across initial-accuracy bins are not reported.

Can you add LoCD to the Section 3 experiments (the zero-initial-accuracy subset in Figure 2 and the knowledge-injection task in Figure 3), and stratify the Table 1 gains by initial-accuracy bin? Please also report the distribution of span lengths and the fraction of trajectory tokens that receive supervision, so the coverage–exposure tradeoff can be assessed directly.

[Response] We thank the reviewer for the constructive suggestions on directly testing our motivation. We conduct additional experiments and analyses to address the three concerns:

  • LoCD overcomes the low-initial-accuracy failure of on-policy context distillation. We add LoCD to the experiments in Figures 1–2. On problems with low or zero initial accuracy, LoCD substantially outperforms on-policy context distillation, and its gains carry over to held-out problems. On the knowledge-injection task (Figure 3), LoCD is identical to off-policy context distillation, which already learns without delay.
  • The gains in Table 1 concentrate on low-initial-accuracy problems. We stratify OlympiadBench by initial accuracy. LoCD's improvement over on-policy context distillation comes mainly from the low-accuracy bins, reaching up to 19.06 points.
  • Short spans are enough to correct errors, while keeping most of the trajectory generated by the student. We report span lengths and supervised-token fractions. LoCD supervises only short spans covering 7–17% of trajectory tokens, which is enough for the student to learn the missing behaviors while keeping exposure bias small.

The details are as follows.

(1) LoCD overcomes the low-initial-accuracy failure of on-policy context distillation. We add LoCD to Figures 1–2 of the revised manuscript, under the same settings as the original experiments.

Low-initial-accuracy problems (Figure 1). Problems are binned by each model's own initial avg@128. Table R1 aggregates the bins with [0, 40]% initial accuracy; full per-bin results are shown in Figure 1 of the revised manuscript.

Table R1: avg@128 (%) on problems with initial accuracy in [0, 40]%.

ModelOn-policy (OPCD [1])Off-policy (TRD [2])LoCD
Qwen3-4B21.0838.2943.94
Qwen3-8B29.0962.6567.13
Falcon3-10B-Instruct13.5533.3334.24

LoCD achieves the highest accuracy on all three models, improving over On-policy by 22.86, 38.04, and 20.69 points. These results show that LoCD remains effective precisely where on-policy context distillation struggles, i.e., when correct behavior rarely appears in the student's own rollouts.

Zero-initial-accuracy problems (Figure 2). We collect problems on which the initial student fails in all 128 rollouts, and split them into training and held-out problems (64/64 for Qwen3-4B and 24/24 for Qwen3-8B). All methods are trained on the training problems, and Table R2 reports the results on the held-out problems, which are never seen during training.

Table R2: avg@128 (%) on held-out problems with zero initial accuracy, after 100 training steps (Figure 2 in the revised manuscript).

ModelOn-policyOff-policyLoCD
Qwen3-4B2.4728.1232.44
Qwen3-8B3.0043.2356.23

Since the student never produces a correct answer on its own, On-policy barely improves. In contrast, LoCD learns to solve these problems, and the gains carry over to problems never seen in training.

[1] Ye, Tianzhu, et al. "On-Policy Context Distillation for Language Models." arXiv:2602.12275, 2026.

[2] Jiang, Li, et al. "Trajectory-Refined Distillation." arXiv:2606.08432, 2026.

AuthorsOpenReview ↗

Rebuttal to Reviewer bzVG [2/4]

[W1 & Q1] (continued)

Knowledge injection (Figure 3). This experiment isolates the length dependence of on-policy context distillation, so each sample contains only a query and a direct answer. Since the answer is a synthetic fact the student has never seen, the student is wrong from the first token, leaving no correct prefix to keep. The span localized by LoCD is therefore the entire answer, making LoCD identical to off-policy context distillation, which Figure 3 already shows learns answers of all lengths quickly. LoCD's advantage of correcting only the erroneous part arises in multi-step trajectories, as validated in Figures 1–2 and Tables 1–3.

(2) The gains in Table 1 concentrate on low-initial-accuracy problems. We use OlympiadBench, the largest benchmark in Table 1 (674 problems), and stratify it by the base model's per-problem initial avg@128. All methods are evaluated on the same groups.

Table R3: avg@128 (%) of Qwen3-8B on OlympiadBench by initial accuracy.

Initial acc. (%)0(0,20](20,40](40,60](60,80](80,100]All
#Problems105117534861290674
Base0.006.4529.6450.6769.8896.7054.99
Off-policy7.6530.5963.4166.2373.9091.0762.08
On-policy2.3214.0758.1472.4879.5997.6361.75
LoCD7.3333.1371.5875.8879.3295.0766.01

The improvement of LoCD primarily stems from the low-accuracy regime. In the [0, 60]% bins, LoCD achieves consistent and substantial gains over On-policy, by up to 19.06 points. In the high-accuracy regime, LoCD performs comparably to On-policy while remaining clearly better than Off-policy. Overall, LoCD achieves the best performance (66.01 vs. 62.08 for Off-policy and 61.75 for On-policy).

(3) Short spans are enough to correct errors, while keeping most of the trajectory generated by the student. Span length controls a trade-off: a longer span lets the teacher demonstrate more of the missing behavior, but also trains the student on more text it would not generate itself (exposure bias). Table R4 reports the span statistics of LoCD (Table 5 in Appendix C.3 of the revised manuscript).

Table R4: Span statistics of LoCD.

TaskAvg. span length (tokens)Supervised tokens (%)
Math43.69.0
ALFWorld30.516.8
WebShop18.37.3

The two ends of this trade-off can be assessed with the results in the manuscript. At the short end, LoCD's spans contain only 18–44 tokens and cover 7–17% of trajectory tokens, yet they are enough for the student to solve problems that On-policy cannot (Tables R1–R2). At the long end, the Full-train variant (Figure 9 in the manuscript) extends teacher supervision from the span to the entire remaining trajectory, and performance drops by 3.17, 6.93, and 4.00 points on AIME24, LiveCodeBench, and WebShop. A small number of corrected tokens is therefore sufficient for the student to learn the teacher's behavior, while keeping exposure bias small.

[W2 (Part 1) & Q2] The experiments support the effectiveness of the combined LoCD procedure, but not the attribution to localization and attribution specifically. The Figure 9 variants are undefined, and as drawn the random-point variant already approaches the strongest baseline on AIME24. The closest prior method, backtracking distillation (Wang et al. 2026a), is absent.

Please define the “Random-point”, “Raw trajectory”, and “Full-train” variants in Figure 9 and report their numbers. Can you add backtracking distillation (Wang et al. 2026a) as a baseline on math, code, and at least one agent task, under matched training tokens?

[Response] We thank the reviewer for the helpful suggestions on isolating each component of LoCD and comparing with the closest prior method. We address the two concerns as follows:

  • Each of LoCD's three components contributes to its performance. We define the three ablation variants in Figure 9 and report their numbers. Replacing any single component degrades performance on all three benchmarks.
  • LoCD outperforms Backtracking Distillation (BD). We add BD as a baseline on math, code, and WebShop under matched training tokens. LoCD outperforms BD on all four benchmarks, by 6.02–24.00 points.

The details are as follows.

AuthorsOpenReview ↗

Rebuttal to Reviewer bzVG [3/4]

[W2 (Part 1) & Q2] (continued)

(1) Each of LoCD's three components contributes to its performance. LoCD consists of three components: it first uses the feedback to find where the trajectory goes wrong (localization), then writes a targeted hint explaining what to fix there (attribution), and finally distills only the corrected span (localized distillation). Each variant in Figure 9 in the manuscript replaces one component while keeping everything else unchanged:

  • Random-point (tests localization): replaces the feedback-guided correction point with a randomly selected one.
  • Raw Experience (tests attribution; named "Raw trajectory" in the manuscript): uses the complete interaction experience as privileged context, instead of the targeted hint extracted from it.
  • Full-train (tests localized distillation): keeps the student prefix, but extends teacher generation and distillation from the localized span to the entire remaining trajectory.

Table R5: Ablation of LoCD on Qwen3-8B (%).

MethodAIME24LiveCodeBenchWebShop
Base28.0616.7118.00
Random-point51.0519.2138.20
Raw Experience59.2021.2946.20
Full-train58.1317.2148.40
LoCD61.3024.1452.40
  • Localization: Random-point drops by 10.25, 4.93, and 14.20 points. On AIME24, Random-point (51.05) remains clearly below the strongest baseline OP²SD [3] (56.85), while LoCD with feedback-guided localization exceeds it by 4.45 points.
  • Attribution: Raw Experience drops by 2.10, 2.85, and 6.20 points, showing that a targeted hint is more effective than the full experience.
  • Localized distillation: Full-train drops by 3.17, 6.93, and 4.00 points, even though it supervises more tokens.

Each component contributes to the performance of LoCD, and the best performance requires all three. The revised manuscript defines these variants in Section 5.4, and their numbers are labeled in Figure 9.

(2) LoCD outperforms Backtracking Distillation (BD) [4]. We thank the reviewer for pointing out this closely related work, and we further compare LoCD with it. Notably, BD and LoCD address different problems: BD was originally designed for distillation from a stronger teacher, whereas LoCD learns from the model's own interaction feedback without a stronger teacher. We therefore adapt BD to our self-distillation setting: following its official implementation, the teacher is the same Qwen3-8B model with the same privileged context as LoCD, and training tokens are matched.

Table R6: Comparison with BD on Qwen3-8B (%).

MethodAIME24AIME25LiveCodeBenchWebShop
BD55.2839.7216.9028.40
LoCD61.3047.8724.1452.40

LoCD consistently outperforms BD, by 6.02, 8.15, 7.24, and 24.00 points on AIME24, AIME25, LiveCodeBench, and WebShop. We attribute this gap to how the two methods decide where to correct. BD flags positions where the student's tokens have low likelihood under the teacher, but a low-likelihood token is not necessarily the cause of failure. LoCD instead uses the feedback to first identify why the trajectory fails, and then localizes the span responsible for it. The gap is largest on WebShop, where failures are revealed only by environment feedback over multiple steps, so teacher likelihood alone provides little guidance on which action to correct.

[3] Ichihara, Yuki, et al. "Privileged Solutions or Context-Induced Teacher Behavior? Dissecting On-Policy Self-Distillation." arXiv:2608.09228, 2026.

[4] Wang, Bing, et al. "Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation." arXiv:2605.19433, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.