acceptodds
Under review as a conference paper at ICLR 2027

Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation

Abstract

On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, standard OPD requires a live teacher inference server throughout training, resulting in substantial infrastructure overhead. In this work, we investigate whether on-policy distillation can be performed offline. A natural approach is to precompute teacher log-probabilities once over SFT rollouts and reuse them during training. In practice, however, this offline variant fails to reliably match the performance of standard OPD. To understand this discrepancy, we identify an important design choice, which we term teacher consistency. This choice uses the same teacher model for both supervised fine-tuning and OPD. Across the evaluated teacher combinations, consistent pairings achieve higher distillation scores, with a larger difference for offline than online OPD. Building on this insight, we propose Lightning OPD, an offline on-policy distillation framework that precomputes teacher log-probabilities over student rollouts and adopts teacher consistency as a practical design guideline. This design eliminates the need for a live teacher server entirely. Our theoretical analysis bounds the online–offline gap under teacher consistency and shows that teacher mismatch can lead to a non-vanishing gap. Extensive experiments on mathematical reasoning and code generation demonstrate that Lightning OPD achieves competitive performance with significantly improved efficiency. Starting from an Qwen3-8B-SFT model, Lightning OPD reaches 69.9% on AIME 2024 in just 30 GPU hours, achieving a 4.0x reduction in distillation GPU cost over standard OPD and substantially lowering the barrier to entry for academic research on LLM post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On Policy Distillation, Post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XaKxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the cost of keeping a teacher model running throughout online distillation. Lightning OPD collects student responses and caches teacher scores once, then trains the student offline. The authors advocate using the same teacher for SFT and distillation to reduce distribution shift when reusing fixed trajectories, supported by theoretical analysis and teacher pairing experiments. On math and coding tasks, the method achieves performance close to online OPD in the evaluated settings while reducing the distillation GPU cost for Qwen3-8B to one quarter.

Critical strengths
  1. The method is simple and practical, removing the overhead of keeping a teacher running during student training. In Table 2, the distillation cost for the 8B model drops from 120 to 30 GPU hours, including offline preprocessing.

  2. The experiments cover multiple models and both math and coding tasks, with overall performance close to online OPD. The teacher pairing experiments also offer a useful observation that teacher choices during SFT and distillation affect offline training performance.

Critical weaknesses
  1. Table 3 uses different SFT checkpoints and distillation durations, making it difficult to attribute the performance differences to teacher consistency. Figure 3(b) controls the budget but reports only score fitting loss, without the corresponding task performance.

  2. Table 1 lacks comparisons with existing offline distillation methods under the same settings, while results for GKD and AOPD come from other papers. It remains unclear how much additional performance or cost benefit the objective based on a single sampled token score provides over existing offline approaches.

Questions for authors
  1. With matched SFT steps and distillation budgets in Table 3, does teacher consistency still reduce the task performance gap between offline and online methods?

  2. With the same initialization, teacher, and data, how does the proposed method compare with existing offline KD objectives in accuracy and GPU cost?

3 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer XaKx

W1/Q1. Teacher consistency with matched SFT steps and distillation budgets.

Thank you for this question, and we apologize that the caption of Table 3 was unclear. The SFT steps and distillation budgets in Table 3 are already matched.

  • Different SFT checkpoints. Each row uses the checkpoint fine-tuned on its own SFT teacher's demonstrations, all trained with the same settings (Table 4, 3,000 steps). Within a row, all four runs start from the same checkpoint.
  • Different distillation durations. These are wall-clock times, which differ because the OPD teachers differ in inference cost (e.g., QwQ-32B vs. Qwen3-8B). Every run takes 150 updates with global batch 256, learning rate 2×10⁻⁶ and a 4,096-token response cap (Table 5).

Within each row, the OPD teacher is the only factor that varies. We have rewritten the caption to state this.

Teacher consistency reduces the offline–online task gap. Lightning OPD minus standard OPD on AIME 2024 (points), from Table 3:

Student / SFT teacherConsistent OPD teacherSwitched OPD teacherDifference
4B / Qwen3-8B+2.7+0.12.6
4B / QwQ-32B+0.3−1.92.2
8B / Qwen3-32B+1.4−1.73.1
8B / QwQ-32B+2.2−2.95.1

With the consistent teacher, Lightning OPD matches or exceeds standard OPD in every row (+0.3 to +2.7). With a switched teacher, it is level in one row and behind in the other three (−1.7 to −2.9).

Link to Figure 3(b). Figure 3(b) is a short budget-matched diagnostic (20 updates) on the same two 4B initializations and two OPD teachers as the 4B block of Table 3. The two agree: a teacher change raises the score-loss gap G=Loff−LonG=L_{\mathrm{off}}-L_{\mathrm{on}} by 0.0067 and 0.53, and lowers Lightning OPD's AIME 2024 margin over standard OPD by 2.6 and 2.2 points.

We have added the offline–online differences above to Appendix C.3, which details Figure 3(b).

W2/Q2. Comparison with existing offline KD objectives under the same settings.

Thank you for this suggestion. We ran the requested comparison. All methods start from the same 4B SFT student and use the same Qwen3-8B teacher, 17,398 prompts, 150 updates (batch 256, learning rate 2×10⁻⁶) and evaluation. The top-10 objectives train on the same cached student trajectories as Lightning OPD.

  • Rang-style KD. Our implementation of Rang et al. (2025): (1−λ)·CE + λ·KL(teacher ‖ student) over the renormalized teacher top-10 logits at every position, with their default λ = 0.9.
  • Top-10 forward KL. The same objective with λ = 1.
  • SeqKD. Continued SFT on Qwen3-8B responses to the same prompts.

Accuracy.

MethodAIME 2024AIME 2025HMMT 2025Avg.
SFT (Table 1)56.752.134.047.6
SeqKD56.952.534.748.0
Top-10 forward KL62.356.736.951.9
Rang-style KD (CE + KL)62.157.737.752.5
Lightning OPD (Table 1)68.158.439.855.4

Cost. H100 GPU hours for the distillation stage, with the same accounting as Table 2.

MethodRolloutScoringTraining (150 updates)TotalStored cache (incl. responses)
Lightning OPD1028200.48 GB
Rang-style KD1028203.6 GB
Top-10 forward KL1028203.6 GB
SeqKD1008180.48 GB
  • The objective accounts for the gain. Lightning OPD's gain over SFT is 1.6–1.8× that of the top-10 objectives. SeqKD barely improves on SFT (48.0 vs. 47.6), so the gain comes from teacher feedback on the student's own trajectories.
  • Same cost, 7.6× smaller cache. The top-10 objectives cost the same 20 GPU hours, and the optimizer step takes the same time (12.3 vs. 12.0 s per update). The 3.6× (4B) and 4.0× (8B) savings over online OPD in Table 2 come from removing rollouts and live teacher serving during training, which all offline objectives share. Among them, Lightning OPD is the most accurate and has the smallest cache.

We have added these controlled baselines and their costs to a new Appendix C.5.

I appreciate the authors' efforts in addressing my concerns through the additional results. After considering the rebuttal, my final rating is 2.

Dear Reviewer XaKx,

Thanks for your response. We sincerely hope you will take the opportunity to reconsider our work. We are also open and eager to engage in further discussions with you if you have any further concerns. Thank you once again for your time and consideration.

Reviewer_WnTHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose Lightening OPD, an efficient variant of OPD that eliminates the need for a online-teacher during training. The method precomputes teacher log-probability on student-generated trajectories and reuses them throughout optimization.

Critical strengths
  1. Practical efficiency and strong empirical performance: The paper present a simple and computationally efficient alternatives to standard OPD. Eliminating online teacher inference substantially reduces training costs while achieving comparable or even better performance on several reasoning and coding benchmarks.

  2. Theoretical analysis of teacher consistency: The author make a meaningful effort to mathematically characterize why using different teachers for SFT and OPD can be problematic.

Critical weaknesses
  1. Insufficient comparison with existing offline distillation methods: The most closely related work (Rang et al. (2025) in Appendix D) already investigates offline distillation using precomputed teacher supervision on student-generated trajectories. As discussed in Appendix D, the primary methodological difference are the training objective (cross-entropy combined with KL divergence versus log-probability differences) and the amount of teacher information stored (top-k logits versus a single log-probability per sampled token). However, the paper provides neither a direct experimental comparison nor a theoretical explanation of why the proposed objective should be preferable. Without a controlled comparison using the same student, teacher, and offline trajectories, it is difficult to determine whether the reported improvements arise from the proposed objective, teacher consistency, or other experimental choices.

  2. Limited novelty and generality of the teacher consistency analysis: The observation that teacher consistency improves distillation appears somewhat expected. Since the student is already fine-tuned on trajectories generated by a particular teacher, switching to a different teacher naturally introduces a distribution mismatch and requires the student to adapt to different supervision targets. While the theoretical analysis is mathematically interesting, it primarily formalizes this intuition under specific assumptions. In particular, the analysis establishes that the offline–online gap becomes small when the student is already close to the SFT teacher, whereas a teacher switch can introduce a non-vanishing gap under certain conditions. It remains unclear how much these results extend beyond the analyzed settings or provide new practical guidance beyond the intuitive recommendation to maintain consistency between SFT and distillation teachers.

  3. Insufficient explanation of why offline OPD performs well: The paper devotes considerable theoretical attention to explaining why teacher mismatch can harm offline distillation. However, a more fundamental question remains insufficiently addressed: why does offline OPD with fixed trajectories achieve performance comparable to, or even better than, standard online OPD? The analysis in Appendix A.1 establishes that the proposed updates correspond to minimizing squared log-probability errors on cached trajectories. While this explains how learning can proceed with fixed supervision, it does not explain why this approach should achieve comparable or superior downstream performance relative to online OPD. Furthermore, Appendix C.1 shows that periodically refreshing trajectories actually degrades performance. This is an interesting and somewhat counterintuitive observation, yet the paper provides little explanation of why fixed trajectories are more effective. Understanding this phenomenon would be more valuable for explaining the effectiveness of the proposed method and identifying when offline distillation should be preferred over online OPD.

Questions for authors

see weakness.

4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer WnTH

W1. Controlled comparison with Rang et al. (2025).

Thank you for this suggestion. We ran the requested comparison. All methods start from the same 4B SFT student and use the same Qwen3-8B teacher, prompts, 150 updates (batch 256, learning rate 2×10⁻⁶) and evaluation. The KD objectives train on the same cached student trajectories as Lightning OPD, so only the objective and the stored teacher information vary.

  • Rang-style KD. Our implementation of Rang et al.: (1−λ)·CE + λ·KL(teacher ‖ student) over the renormalized teacher top-10 logits, with their default λ = 0.9.
  • Top-10 forward KL. The same objective with λ = 1.
  • SeqKD. Cross-entropy on Qwen3-8B responses to the same prompts.
MethodTeacher information storedAIME 2024AIME 2025HMMT 2025Avg.
SFT (Table 1)–56.752.134.047.6
SeqKDresponses56.952.534.748.0
Top-10 forward KLtop-10 logits per token62.356.736.951.9
Rang-style KD (CE + KL)top-10 logits per token62.157.737.752.5
Lightning OPD (Table 1)one log-probability per token68.158.439.855.4
  • The objective accounts for the gain. Lightning OPD's gain over SFT is 1.6–1.8× that of the top-10 objectives, with 7.6× less stored teacher information (0.48 vs. 3.6 GB). SeqKD barely improves on SFT, so the gain comes from teacher feedback on the student's own trajectories.
  • Why the sampled-token objective is preferable. On student-sampled tokens, its expected update Ea∼πθ[(log⁡πT(a∣s)−log⁡πθ(a∣s))∇θlog⁡πθ(a∣s)]\mathbb E_{a\sim\pi_\theta}\big[(\log\pi_T(a\mid s)-\log\pi_\theta(a\mid s))\nabla_\theta\log\pi_\theta(a\mid s)\big] equals −∇θKL(πθ ∥ πT)-\nabla_\theta \mathrm{KL}(\pi_\theta\,\|\,\pi_T), the reverse-KL objective of online OPD, and Theorem 3.1 bounds the cost of caching by O(h3)O(h^3) under teacher consistency. Reverse KL corrects the tokens the student actually produces, which is where the large errors are: over 99.8% of tokens with initial error above 2 nats are over-weighted by the student. The top-10 objective instead fits a truncated forward KL, gives no teacher score to sampled tokens outside the teacher's top-10, and its CE term also reinforces the student's sampled tokens, including mistakes.
  • The objective and teacher consistency are complementary. With the OPD teacher switched to QwQ-32B, Lightning OPD still leads Rang-style KD on AIME 2024 (62.5 vs. 60.1), but by less than with the consistent teacher (68.1 vs. 62.1).

We have added this comparison to a new Appendix C.5, and the discussion above to Appendix D.

W2. Novelty and generality of the teacher-consistency analysis.

We respectfully disagree that the contribution lacks novelty. That a consistent teacher helps may seem natural in hindsight, but it had not been treated as a design principle for OPD. For example, the Thinking Machines recipe (Lu, 2025) initializes a Qwen3-8B student on QwQ-32B demonstrations and then distills it from a Qwen3 teacher. We turn this choice into a principle with a theoretical account, and build Lightning OPD on it, which matches standard OPD at 4.0× lower distillation GPU cost.

The theory predicts more than the mismatch intuition. That intuition applies to online and offline OPD alike, whereas the theory predicts a penalty specific to fixed rollouts. In Table 3, switching the OPD teacher lowers AIME 2024 by 5.6, 3.8, 6.8 and 6.6 points for Lightning OPD, vs. 3.0, 1.6, 3.7 and 1.5 for standard OPD, a 1.8–4.4× larger penalty. The penalty is an interaction: from an exact fit, changing only how often a context is visited, or only its target, gives zero gap, and the gap arises when both change. In the budget-matched factorial (App. C.3), a teacher change increases the offline–online gap from both initializations, by 0.0067 and 0.53.

Generality. Theorem 3.1 holds for any finite vocabulary and length with parameters shared across tokens, so it covers transformer students under standard smoothness conditions. Appendix A.4 extends the mechanism of Theorem 3.3 to shared-parameter score SGD, where the leading gap after a teacher change is η2κ\eta^2\kappa with κ=−r⊤Cgˉ\kappa=-r^\top C\bar g. Empirically, the predictions hold at two scales, with three teachers and from both SFT initializations.

Practical guidance. The initial score error comes free with the cache and flags the offline penalty before training (0.23 with the consistent teacher vs. 0.49 with a switched one at 4B, App. C.2). If the OPD teacher must change, the SFT data should come from the new teacher (W3).

We have added the switch-penalty comparison and the interaction estimates to Appendix C.3, the general mechanism to Appendix A.4, and this guidance to Appendix D.

AuthorsOpenReview ↗

Rebuttal for Reviewer WnTH, continued

W3. Why fixed trajectories match online OPD, and when to prefer them.

Thank you for this question. Theorem 3.1 answers it. With the OPD teacher equal to the SFT teacher UU, offline and online learners that take the same updates with the same feedback budget differ in final response-level KL to UU by at most CNh3C_N h^3, where hh bounds the initial per-token error. Both apply the same update, which in expectation is the reverse-KL gradient of online OPD (W1). Residuals of size O(h)O(h) move the student by only O(h)O(h), so online and cached responses differ with probability O(h)O(h), and where they differ, both learners already fit the teacher and their risks differ by O(h2)O(h^2). In practice, the median initial error at 4B is 0.001 nats, and the per-token KL from the SFT reference after 150 updates is 0.048 (App. C.2). Accordingly, Lightning OPD matches standard OPD in Table 1 (55.4 vs. 54.4 at 4B, 57.0 vs. 55.6 at 8B).

Fixed trajectories can come out ahead. The sign of the gap follows coverage (App. A.3). If training reduces the student's visits to a context that still needs correction, online collection supervises it less, while the cache keeps it. In the factorial (App. C.3), the QwQ-initialized student with a consistent teacher reaches lower score loss offline than online (G=−0.069G=-0.069).

Refreshing does not help. By the same argument (App. A.2), refreshed responses come from a student within O(h)O(h) of the initial one, so refreshing changes the final KL by at most O(h3)O(h^3). In Appendix C.1, fixed (68.1), refreshed every 50 or 30 updates (64.7, 63.9) and standard OPD (65.4) are comparable on AIME 2024. The largest difference, 4.2 points, is about one of the 30 problems, and standard OPD, which refreshes at every update, scores above both periodic schedules.

When to prefer offline OPD. With a consistent teacher, offline OPD matches standard OPD at 4.0× lower cost. After a teacher change, fixed rollouts pay a larger penalty (W2), and the remedy is to restore consistency: after SFT on QwQ-32B data, Lightning OPD distilled from QwQ-32B reaches 63.1 (4B) and 68.7 (8B), matching or exceeding standard OPD (62.8 and 66.5). When SFT cannot be repeated, online collection is safer.

We have added this explanation and the refresh corollary to Appendix A.2, and their connection to the refresh experiment to Appendix C.1.

Reviewer_WnTHOpenReview ↗

Dear authors

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the controlled comparison with existing offline KD methods and the additional theoretical analysis. These results have substantially addressed my concerns and strengthened the paper. Given the demonstrated empirical effectiveness and computational efficiency of Lightning OPD, I have decided to raise my score.

Thank you very much for your encouraging feedback. We are glad to see the additional results can address your concerns. Your suggestions helped us strengthen our work. Thank you again for your time and support!

Reviewer_oKWGOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Lightning OPD, a novel distillation framework that precomputes teacher log-probabilities on student-generated trajectories and reuses them during training, instead of continuously scoring newly generated trajectories. The authors also find that using the same teacher for SFT data generation and distillation is a useful design principle for achieving strong performance in offline distillation. They further provide a theoretical analysis and conduct experiments on mathematical reasoning and code generation tasks with Qwen3 and OLMo models. The results show performance comparable to standard online OPD at substantially lower computational cost.

Critical strengths
  1. The paper addresses a meaningful computational bottleneck in OPD. The proposed method removes the need for a live teacher server during training thereby significantly reducing distillation cost and making the approach appealing for researchers with limited resources.
  2. The investigation of teacher consistency offers insight about the relationship between SFT and distillation. The teacher-pairing experiments suggest that this choice matters for both online and offline OPD, providing inspiration for the design of post-training pipelines.
Critical weaknesses
  1. Theorem 3.1 assumes that the initial student is close to the SFT teacher across all states and tokens, which is a strong condition.
  2. The paper provides results with a fixed cache and examines periodic refresh. However, it is less clear how sensitive the method is to the cache size, the number of responses per prompt, or longer training.
Questions for authors
  1. Could the authors provide more information about the distribution of initial student–teacher score errors?
  2. Could the authors provide analysis on performance difference when the number of cached responses per prompt changes?
  3. Could the authors further discuss how their choice of teacher supervision relates to other work on distillation objectives and student-generated responses in the paper, including the following works:
    • Wenda X et al. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap through Interleaved Sampling. ICLR, 2025.
    • Guanghui W et al. The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works. ICML, 2026.
    • Makoto S et al. TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models. ICLR, 2025.
    • Guanghui W et al. ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence. ICML, 2025.
    • Changyi H et al. DA-KD: Difficulty-Aware Knowledge Distillation for Efficient Large Language Models. ICML, 2025.
    • Daniil T et al. On Teacher Hacking in Language Model Distillation. ICML, 2025.
4 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer oKWG

W1. Theorem 3.1 assumes the initial student is uniformly close to the SFT teacher.

Thank you for the careful reading. The uniform bound is a sufficient condition that makes the local analysis rigorous, not a requirement of the method. It keeps the score residual and the parameter movement at O(h)O(h) along training, which yields the comparative prediction we use: under teacher consistency, the extra error from freezing the rollouts satisfies ∣DB(U)∣≤CNh3|D_B(U)|\le C_N h^3, whereas a changed teacher can leave an offline–online gap that does not vanish as h→0h\to0 (Theorem 3.3).

  • Teacher-consistent SFT targets this regime. The student is first fine-tuned on outputs of the same teacher that later supplies the log-probabilities, so SFT directly minimizes the discrepancy the theorem starts from. The resulting fit is tight on most tokens, and larger errors are rare (Q1).
  • The prediction holds where we measure it. In a factorial with matched updates and teacher-feedback budgets (App. C.3), the offline–online gap is at most 0.0034 under consistent teachers and rises to 0.010 and 0.46 after a teacher change. On benchmarks, the consistent pairing scores highest for every student, scale and method (Table 3).

We have added a sentence after Theorem 3.1 stating that the uniform bound is a sufficient condition for the local analysis, and pointing to these tests.

Q1. Distribution of the initial student–teacher score errors.

We measured the per-token error et=log⁡πθ0(yt∣st)−log⁡U(yt∣st)e_t=\log\pi_{\theta_0}(y_t\mid s_t)-\log U(y_t\mid s_t) (nats) on 2,048 cached student responses (8.25M tokens at 4B, 8.20M at 8B), i.e., on the states the cache trains on.

Student / OPD teachermedian ∣e∣\lvert e\rvertp90p99tokens with ∣e∣>1\lvert e\rvert>1
4B / Qwen3-8B (consistent)0.0010.371.902.8%
4B / QwQ-32B (switched)0.0360.942.829.1%
8B / Qwen3-32B (consistent)0.0040.512.634.6%
8B / QwQ-32B (switched)0.0260.812.797.5%
  • The bulk of tokens is fit tightly. Under the consistent teacher, half of all tokens are within 0.004 nats and 90% within about 0.5 nats.
  • Teacher consistency improves the fit throughout. The median error is 7–36× smaller than with the switched teacher, and the fraction of tokens above 1 nat is 1.6–3.2× lower.
  • Larger errors are rare and one-sided. Over 99.8% of tokens with ∣e∣>2\lvert e\rvert>2 are tokens the student over-weights relative to the teacher. They lie in the student's own rollouts, so the cache contains them, and Lightning OPD's advantage At=log⁡V−log⁡πθA_t=\log V-\log\pi_\theta gives them the largest negative weight.

We have added these statistics to Appendix C.2.

W2 / Q2. Sensitivity to cache size, responses per prompt, and longer training.

Thank you for the suggestion. We ran all three ablations at 4B. Everything else follows Table 1: the same SFT student and Qwen3-8B teacher, 256 cached responses per update, 150 updates unless stated, the same learning rate and evaluation. Smaller caches use nested random prompt subsets. Extra responses per prompt are sampled from the same SFT student with the original settings and scored by the same teacher.

VariantPrompts × responses per promptAIME 2024AIME 2025HMMT 2025Avg.
SFT (initialization, Table 1)–56.752.134.047.6
Lightning OPD (Table 1)17,398 × 168.158.439.855.4
Cache size
50% of prompts8,699 × 167.159.340.455.6
25% of prompts4,350 × 164.760.439.254.8
Responses per prompt, all prompts
2 per prompt17,398 × 267.759.037.054.5
4 per prompt17,398 × 467.261.841.356.7
Responses per prompt, same total responses
2 per prompt8,699 × 266.860.738.655.4
4 per prompt4,350 × 467.257.839.854.9
Longer training
300 updates17,398 × 166.860.640.355.9
  • The method is robust to all three factors. Every variant's average is within −0.9 to +1.3 points of Lightning OPD (55.4) and about 7 to 9 points above SFT (47.6).
  • Cache size. Half the prompts match the full cache (55.6 vs. 55.4). With 25%, the average is 54.8, keeping over 90% of the gain over SFT although each cached response is reused about 9 times. Rollout, scoring and storage scale linearly with the cache, so halving it halves these costs.
  • Responses per prompt (Q2). With all prompts, 1, 2 and 4 responses per prompt give 55.4, 54.5 and 56.7. At a fixed total number of responses, trading prompts for responses per prompt changes the average by at most 0.5 points. One response per prompt is therefore sufficient.
  • Longer training. At 300 updates each cached response is used 4.4 times on average (vs. 2.2 at 150), and the average is 55.9 vs. 55.4.

We have added this table to a new Appendix C.4.

AuthorsOpenReview ↗

Rebuttal for Reviewer oKWG, continued

Q3. Relation to work on distillation objectives and student-generated responses.

Thank you for these references. Lightning OPD's supervision is one cached teacher log-probability per sampled token, on responses sampled once from the SFT student. Its expected update is the reverse-KL gradient, and teacher consistency makes the SFT teacher also the OPD teacher.

  • ABKD, TAID and Bridge-Garden redesign the target, while Lightning OPD changes how supervision is collected. Reverse KL is the α=0, β=1 case of ABKD (Wang et al., 2025). TAID (Shing et al., 2025) moves an interpolated target from the student toward the teacher, whereas teacher consistency starts the student near the teacher (Q1). Bridge-Garden (Wang et al., 2026) mixes hard and soft labels on teacher-generated responses, while Lightning OPD trains on the student's own responses. On the same cache, our sampled-token objective outperforms top-10 logit targets: the three-benchmark average is 55.4, vs. 52.5 for CE + top-10 KL (λ = 0.9, following Rang et al., 2025) and 51.9 for top-10 forward KL, from 47.6 after SFT, while storing 7.6× less teacher information.
  • Speculative KD (Xu et al., 2025) uses a live teacher to replace student tokens outside its top-K, because teacher feedback can be unreliable on unfamiliar student samples. Teacher-consistent SFT addresses this before training: the student's responses are already well fit by the teacher (Q1), so they can be sampled and scored once. Speculative KD itself reports that the teacher rejects only 4.7% of an SFT-initialized student's tokens at the start.
  • Teacher hacking (Tiapkin et al., 2025) arises when a fixed set of teacher-generated responses is reused over many epochs, and online student data prevents it. Lightning OPD's cache holds the student's own responses, and benchmark accuracy, a ground-truth metric, holds when training twice as long on it (55.9 vs. 55.4, W2/Q2).
  • DA-KD (He et al., 2025) selects difficult samples from a fixed dataset, which is orthogonal to how supervision is computed. Lightning OPD is also robust to the data budget: a random 50% of prompts matches the full cache (W2/Q2).

We have discussed these six works in Appendix D, including a paragraph on student-generated caches and teacher hacking.

Thank you for the detailed response and the additional experiments. The new results and analyses have addressed my concerns and further strengthened the paper. I will raise my score accordingly and support the acceptance of this work.

Thank you for your thoughtful feedback and constructive suggestions. We are pleased that the additional results have helped address your concerns. Your comments have improved the paper, and we sincerely appreciate the time and care you devoted to reviewing our work.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and effort in reviewing this paper. Please take a moment to read the authors’ rebuttal and the other reviewers’ comments, and participate in the discussion. It would be helpful to indicate which concerns have been addressed and which remain unresolved, and to follow up with the authors where further clarification is needed. Please also update your review and score if warranted, or briefly explain why your assessment remains unchanged. Your engagement will be valuable in reaching a well-informed decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.