Lightning OPD: Efficient Post-Training for Large Reasoning Models with Offline On-Policy Distillation

On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, standard OPD requires a live teacher inference server throughout training, resulting in substantial infrastructure overhead. In this work, we investigate whether on-policy distillation can be performed offline. A natural approach is to precompute teacher log-probabilities once over SFT rollouts and reuse them during training. In practice, however, this offline variant fails to reliably match the performance of standard OPD. To understand this discrepancy, we identify an important design choice, which we term teacher consistency. This choice uses the same teacher model for both supervised fine-tuning and OPD. Across the evaluated teacher combinations, consistent pairings achieve higher distillation scores, with a larger difference for offline than online OPD. Building on this insight, we propose Lightning OPD, an offline on-policy distillation framework that precomputes teacher log-probabilities over student rollouts and adopts teacher consistency as a practical design guideline. This design eliminates the need for a live teacher server entirely. Our theoretical analysis bounds the online–offline gap under teacher consistency and shows that teacher mismatch can lead to a non-vanishing gap. Extensive experiments on mathematical reasoning and code generation demonstrate that Lightning OPD achieves competitive performance with significantly improved efficiency. Starting from an Qwen3-8B-SFT model, Lightning OPD reaches 69.9% on AIME 2024 in just 30 GPU hours, achieving a 4.0x reduction in distillation GPU cost over standard OPD and substantially lowering the barrier to entry for academic research on LLM post-training.