- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes DIR, which reuses the policy shift induced by a single RL run as an implicit reward signal. It constructs an extrapolated teacher from the logit difference between two frozen pre- and post-RL checkpoints, then continues to push the student forward via reverse-KL distillation on its own on-policy trajectories. On mathematical reasoning and multi-turn agentic search problems, DIR not only improves beyond the performance ceiling reached by converged RLVR, but also approaches later-stage RL performance at substantially lower training cost. Its key appeal is that it provides dense token-level supervision using only a pair of checkpoints, with no need for a separate reward model, verifier, or preference data.
Critical strengths
- DIR is simple and easy to reproduce: it requires only a single pair of frozen checkpoints to run on-policy training, with no need for a separately trained reward model, verifier, external teacher, or additional preference data, which substantially lowers the infrastructure overhead compared to continued RL or DPO-style pipelines.
- The method is well motivated theoretically. Starting from the KL-regularized policy optimum, the authors cleanly derive a closed-form reward-reuse teacher, and the overall theoretical storyline is both coherent and appropriately scoped.
- DIR provides dense token-level supervision, which is orders of magnitude denser than the sparse outcome rewards used in RLVR, leading to more stable and sample-efficient training.
Critical weaknesses
- The extrapolation coefficient α lacks theoretical or empirical guidance. The main experiments fix α=1, and the ablation covers only three values {0.5, 1, 2}, leaving the method's robustness with respect to this key hyperparameter largely unclear.
- The stability of DIR under longer-horizon distillation is unknown. The teacher distribution is constructed from two frozen checkpoints π_pre and π_post, so the further the student drifts from π_post, the more out-of-support π_T becomes. The paper only trains for 1–2 epochs, leaving open the question of whether longer training or iterative application of DIR would suffer from distributional drift.
- DIR requires maintaining three models simultaneously at every training step — the student π_θ and two frozen checkpoints π_pre and π_post, and performs three forward passes per prefix. The tripled forward-pass cost is the dominant source of memory overhead during training, and the two frozen checkpoints must be strictly aligned in version and quantization precision; any mismatch directly contaminates the logit difference used as the implicit reward signal.
Questions for authors
Please see details in my comments on weaknesses.
4 comments
Response to Reviewer MSds (Part 2)
Response to W2
Very interesting question! According to your suggestion, we extended training to 600 steps and evaluated iterative applications of DIR.
All three runs resume from the same Qwen3-4B DIR step-100 checkpoint (derived from RL279).
Fixed teacher: Continues for 500 additional DIR updates using the original Base/RL pair, maintaining throughout.
Refresh every 50: Updates 101–150 use RL279/DIR100 as the Base/RL pair, updates 151–200 use DIR100/DIR150, and subsequent phases analogously use the two most recent checkpoints. Each teacher pair remains frozen within its respective phase, completing 10 times DIR training.
Alternating RL–DIR: Consists of 10 cycles, each performing 10 GRPO updates followed by 40 DIR updates. In each cycle, the DIR teacher is constructed from that cycle's pre- and post-RL checkpoints, while decays linearly from 0.75 to 0.075 across the ten cycles. This continuation comprises 100 RL and 400 DIR updates in total.
Evaluation are conducted on AIME24/25, AMC23, HMMT25, MATH-500, Minerva, and Olympiad-Bench.
| Total updates | Fixed teacher | Refresh every 50 steps | Alternating RL–DIR |
|---|---|---|---|
| 100 | 44.19 | 44.19 | 44.19 |
| 300 | 43.92 | 44.06 | 44.46 |
| 600 | 44.32 | 44.43 | 44.23 |
Across all three strategies, DIR maintains stable performance through 600 total updates (44.23%–44.43% versus 44.19% at step 100), without observable performance collapse. In particular, the fixed-teacher results demonstrate that DIR can sustain long-horizon distillation without refreshing the original teacher, despite continued student updates.
Response to Reviewer MSds (Part 3)
Response to W3
We agree that the frozen checkpoints introduce additional storage and scoring overhead. In our Math implementation, they are scored sequentially under torch.no_grad() with FSDP CPU offloading. Only the student model requires backpropagation on GPU. Consequently, forward-pass counts alone do not determine peak memory consumption.
Overall training cost is also governed by rollout generation and the number of required updates. Whereas our GRPO baseline samples 8 rollouts per prompt, DIR requires only 1 rollout. RQ2 (Fig. 2) accounts for the combined cost of both the initial RL stage and subsequent DIR training. On Math, 20 additional DIR updates reach 40.72% accuracy in 102.43 GPU-hours, compared to 41.02% in 213.77 GPU-hours for RL-250 (52.08% less compute). On Search, DIR achieves 35.42% in 971.01 GPU-hours, versus 33.73% in 969.41 GPU-hours for RL-224 (+1.69 percentage points at nearly identical compute).
Robustness regarding mismatch. Thanks for raising such practical concerns. We evaluated DIR under different forward precision. The results are shown as follow,
| / precision | Seven-task average accuracy | Difference from FP32/FP32 setting |
|---|---|---|
| BF16 / BF16 | 43.72 | −0.47 |
| FP32 / BF16 | 43.94 | −0.25 |
| BF16 / FP32 | 43.72 | −0.47 |
| FP32 / FP32 | 44.19 | Reference |
Precision differences can perturb logits, but the mixed configurations completed training without a clear aggregate accuracy degradation. Therefore, DIR is robust to the quantization precision of two frozen checkpoints.
Dear Reviewer MSds,
Thank you again for your positive and constructive feedback. As the discussion phase will end in less than 24 hours, we wanted to gently follow up to see if our rebuttal have addressed your concerns. Please let us know if there is anything else we can clarify before the deadline.
Best regards,
Authors
Response to Reviewer MSds (Part 1)
Response to W1
Thank you for requesting clearer guidance on α. Section 3.2, Eq. (4), and Appendix B.2 identify 1/α as the KL regularization weight relative to πpost: increasing α permits stronger extrapolation. This provides a theoretical interpretation, although it does not prescribe a universally optimal value. The main results use α=0.75 for mathematical reasoning and α=1 for agentic search.
Empirically, Section 4.4 (RQ3) and Table 2 evaluate four coefficients, α∈{0.5,0.75,1,2}, in both domains. For mathematical reasoning, the reported average accuracy across seven benchmarks remains between 43.83% and 44.17%, a spread of only 0.34 percentage points. All four settings outperform GRPO (40.56%) by 3.27–3.61 percentage points.
For agentic search, α∈{0.5,0.75,1} yields closely matched average accuracy across HotpotQA, 2Wiki, MuSiQue, and Bamboogle: 57.80%–58.18%, compared with 56.80% for GRPO. These three settings also consistently improve both accuracy and recall on BCP+ and FRAMES; for example, BCP+ accuracy is 37.59%–38.80%, versus 36.14% for GRPO.
Together, these results support empirical robustness across the tested moderate coefficients {0.5,0.75,1} in both domains, with gains preserved across multiple choices. The stronger setting α=2 remains effective in the reported math ablation but causes instability and eventual collapse in search, as disclosed in Table 2's footnote. We therefore recommend {0.5,0.75,1} as a practical initial search grid and will clarify this guidance and the observed limitation of stronger extrapolation.