acceptodds
Under review as a conference paper at ICLR 2027

Look Back to Move Forward: Reusing RL-Induced Policy Shifts for Policy Improvement

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves language model reasoning, but obtaining these gains often requires costly trajectory sampling under sparse outcome feedback. This raises the question: can the policy shift learned during an RL run be reused to further improve the same model? We propose Distillation with Implicit Reward (DIR), a framework that turns RL-induced policy shifts into reusable supervision for further policy improvement. Under idealized KL-regularized policy optimization, the log-policy ratio before and after RL is equivalent to the inducing reward up to scale and an additive constant. Motivated by this correspondence, DIR extracts a token-level implicit reward from a fixed checkpoint pair and reuses it in a second regularized optimization anchored at the post-RL policy, yielding an extrapolated distillation target. DIR then distills this target on-policy, providing dense token-level supervision on states visited by the evolving student without a separately trained reward model or an external teacher. Experiments on mathematical reasoning and agentic search demonstrate two complementary benefits. DIR improves performance beyond RLVR. When initialized from intermediate RL checkpoints, it also approaches later-stage RL performance at substantially lower training cost. These results suggest that RL-induced policy shifts can serve as reusable supervision, supporting both continued performance improvement and more compute-efficient post-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Post-Training, On-Policy Distillation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MSdsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes DIR, which reuses the policy shift induced by a single RL run as an implicit reward signal. It constructs an extrapolated teacher from the logit difference between two frozen pre- and post-RL checkpoints, then continues to push the student forward via reverse-KL distillation on its own on-policy trajectories. On mathematical reasoning and multi-turn agentic search problems, DIR not only improves beyond the performance ceiling reached by converged RLVR, but also approaches later-stage RL performance at substantially lower training cost. Its key appeal is that it provides dense token-level supervision using only a pair of checkpoints, with no need for a separate reward model, verifier, or preference data.

Critical strengths
  1. DIR is simple and easy to reproduce: it requires only a single pair of frozen checkpoints to run on-policy training, with no need for a separately trained reward model, verifier, external teacher, or additional preference data, which substantially lowers the infrastructure overhead compared to continued RL or DPO-style pipelines.
  2. The method is well motivated theoretically. Starting from the KL-regularized policy optimum, the authors cleanly derive a closed-form reward-reuse teacher, and the overall theoretical storyline is both coherent and appropriately scoped.
  3. DIR provides dense token-level supervision, which is orders of magnitude denser than the sparse outcome rewards used in RLVR, leading to more stable and sample-efficient training.
Critical weaknesses
  1. The extrapolation coefficient α lacks theoretical or empirical guidance. The main experiments fix α=1, and the ablation covers only three values {0.5, 1, 2}, leaving the method's robustness with respect to this key hyperparameter largely unclear.
  2. The stability of DIR under longer-horizon distillation is unknown. The teacher distribution is constructed from two frozen checkpoints π_pre and π_post, so the further the student drifts from π_post, the more out-of-support π_T becomes. The paper only trains for 1–2 epochs, leaving open the question of whether longer training or iterative application of DIR would suffer from distributional drift.
  3. DIR requires maintaining three models simultaneously at every training step — the student π_θ and two frozen checkpoints π_pre and π_post, and performs three forward passes per prefix. The tripled forward-pass cost is the dominant source of memory overhead during training, and the two frozen checkpoints must be strictly aligned in version and quantization precision; any mismatch directly contaminates the logit difference used as the implicit reward signal.
Questions for authors

Please see details in my comments on weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer MSds (Part 1)

Response to W1

Thank you for requesting clearer guidance on α\alpha. Section 3.2, Eq. (4), and Appendix B.2 identify 1/α1/\alpha as the KL regularization weight relative to πpost\pi_{\mathrm{post}}: increasing α\alpha permits stronger extrapolation. This provides a theoretical interpretation, although it does not prescribe a universally optimal value. The main results use α=0.75\alpha=0.75 for mathematical reasoning and α=1\alpha=1 for agentic search.

Empirically, Section 4.4 (RQ3) and Table 2 evaluate four coefficients, α∈{0.5,0.75,1,2}\alpha \in \lbrace 0.5,0.75,1,2 \rbrace, in both domains. For mathematical reasoning, the reported average accuracy across seven benchmarks remains between 43.83% and 44.17%, a spread of only 0.34 percentage points. All four settings outperform GRPO (40.56%) by 3.27–3.61 percentage points.

For agentic search, α∈{0.5,0.75,1}\alpha \in \lbrace 0.5,0.75,1 \rbrace yields closely matched average accuracy across HotpotQA, 2Wiki, MuSiQue, and Bamboogle: 57.80%–58.18%, compared with 56.80% for GRPO. These three settings also consistently improve both accuracy and recall on BCP+ and FRAMES; for example, BCP+ accuracy is 37.59%–38.80%, versus 36.14% for GRPO.

Together, these results support empirical robustness across the tested moderate coefficients {0.5,0.75,1}\lbrace 0.5,0.75,1 \rbrace in both domains, with gains preserved across multiple choices. The stronger setting α=2\alpha=2 remains effective in the reported math ablation but causes instability and eventual collapse in search, as disclosed in Table 2's footnote. We therefore recommend {0.5,0.75,1}\lbrace 0.5,0.75,1 \rbrace as a practical initial search grid and will clarify this guidance and the observed limitation of stronger extrapolation.

AuthorsOpenReview ↗

Response to Reviewer MSds (Part 2)

Response to W2

Very interesting question! According to your suggestion, we extended training to 600 steps and evaluated iterative applications of DIR.

All three runs resume from the same Qwen3-4B DIR step-100 checkpoint (derived from RL279).

Fixed teacher: Continues for 500 additional DIR updates using the original Base/RL pair, maintaining πT∝πpost1.75/πpre0.75\pi_T\propto\pi_{\mathrm{post}}^{1.75}/\pi_{\mathrm{pre}}^{0.75} throughout.

Refresh every 50: Updates 101–150 use RL279/DIR100 as the Base/RL pair, updates 151–200 use DIR100/DIR150, and subsequent phases analogously use the two most recent checkpoints. Each teacher pair remains frozen within its respective phase, completing 10 times DIR training.

Alternating RL–DIR: Consists of 10 cycles, each performing 10 GRPO updates followed by 40 DIR updates. In each cycle, the DIR teacher is constructed from that cycle's pre- and post-RL checkpoints, while α\alpha decays linearly from 0.75 to 0.075 across the ten cycles. This continuation comprises 100 RL and 400 DIR updates in total.

Evaluation are conducted on AIME24/25, AMC23, HMMT25, MATH-500, Minerva, and Olympiad-Bench.

Total updatesFixed teacherRefresh every 50 stepsAlternating RL–DIR
10044.1944.1944.19
30043.9244.0644.46
60044.3244.4344.23

Across all three strategies, DIR maintains stable performance through 600 total updates (44.23%–44.43% versus 44.19% at step 100), without observable performance collapse. In particular, the fixed-teacher results demonstrate that DIR can sustain long-horizon distillation without refreshing the original teacher, despite continued student updates.

AuthorsOpenReview ↗

Response to Reviewer MSds (Part 3)

Response to W3

We agree that the frozen checkpoints introduce additional storage and scoring overhead. In our Math implementation, they are scored sequentially under torch.no_grad() with FSDP CPU offloading. Only the student model requires backpropagation on GPU. Consequently, forward-pass counts alone do not determine peak memory consumption.

Overall training cost is also governed by rollout generation and the number of required updates. Whereas our GRPO baseline samples 8 rollouts per prompt, DIR requires only 1 rollout. RQ2 (Fig. 2) accounts for the combined cost of both the initial RL stage and subsequent DIR training. On Math, 20 additional DIR updates reach 40.72% accuracy in 102.43 GPU-hours, compared to 41.02% in 213.77 GPU-hours for RL-250 (52.08% less compute). On Search, DIR achieves 35.42% in 971.01 GPU-hours, versus 33.73% in 969.41 GPU-hours for RL-224 (+1.69 percentage points at nearly identical compute).

Robustness regarding mismatch. Thanks for raising such practical concerns. We evaluated DIR under different forward precision. The results are shown as follow,

πpre\pi_{\mathrm{pre}} / πpost\pi_{\mathrm{post}} precisionSeven-task average accuracyDifference from FP32/FP32 setting
BF16 / BF1643.72−0.47
FP32 / BF1643.94−0.25
BF16 / FP3243.72−0.47
FP32 / FP3244.19Reference

Precision differences can perturb logits, but the mixed configurations completed training without a clear aggregate accuracy degradation. Therefore, DIR is robust to the quantization precision of two frozen checkpoints.

Dear Reviewer MSds,

Thank you again for your positive and constructive feedback. As the discussion phase will end in less than 24 hours, we wanted to gently follow up to see if our rebuttal have addressed your concerns. Please let us know if there is anything else we can clarify before the deadline.

Best regards,

Authors

Reviewer_Ds8tOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Distillation with Implicit Reward (DIR), a method that extracts an implicit reward signal from the contrast between pre- and post-RL checkpoints to construct an on-policy distillation target. By converting this learned policy shift into dense supervision, DIR improves model capabilities without relying on a stronger external teacher. Evaluated on mathematical reasoning and agentic search, DIR consistently outperforms its source RL checkpoints and allows models trained on intermediate checkpoints to reach late-stage RL performance with significantly reduced compute.

Critical strengths

The paper introduces Distillation with Implicit Reward (DIR) and validates its efficacy on both mathematical reasoning and agentic search; the idea is quite understandable.

Critical weaknesses
  1. The proposed approach would benefit from a clearer motivation and a more convincing demonstration of genuine novelty. DIR is quite similar to W2S-OPD [1].
  2. Tables 11–14; Appendix H.3 shows that DIR often searches longer rather than using evidence more efficiently
  3. evaluation is limited to Qwen3-4B for mathematics and Qwen3-8B for search. This does not establish robustness across model families, scales, or RL objectives.

[1] Weak-to-Strong On-Policy Distillation

Questions for authors

I would appreciate it if the authors could kindly address the concerns raised in the Critical Weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ds8t (Part 1)

Response to W1: Novelty and Relationship to W2S-OPD

We thank the reviewer for bringing up the concurrent Arxiv preprint, i.e., W2S-OPD. W2S-OPD is a concurrent work released late July 2026, which we have already cited and discussed in Sections 1–2. Despite sharing a related logit-extrapolation form, the two works address totally different research questions: W2S-OPD studies weak-to-strong transfer to a separate student, whereas DIR studies post-RL self-improvement. This nontrivial setting motivates our distinct experiments (Section 4), comparisons against matched-compute Continued RL, and evaluations on multi-turn agentic search and cumulative RL compute savings (Table 1, Figure 2).

Response to W2

We thank the reviewer for this insightful observation. While DIR tends to produce longer search trajectories, its improvements are not limited to increased retrieval coverage, but also extend to evidence retention and answer conversion.

Specifically, on BrowseComp-ZH (Appendix H.2, Table 15), DIR increases the reference-answer retrieval hit rate from 61.25% to 67.47%. Importantly, conditional on a retrieval hit, summary retention improves from 49.15% to 57.95%, and final-answer conversion improves from 30.51% to 33.85%. These results provide complementary evidence that DIR improves how retrieved information is preserved and translated into correct answers, rather than merely increasing the likelihood of retrieving relevant evidence. We will clarify this distinction in the discussion.

AuthorsOpenReview ↗

Response to Reviewer Ds8t (Part 2)

Response to W3

We thank the reviewer for the suggestion. To address this concern, we have completed three additional mathematical reasoning experiments: Qwen3-8B with GRPO, Qwen3-8B with PPO, and OLMo3-7B-Instruct-SFT followed by GRPO. Together with the original Qwen3-4B experiments, these cover two model families, 4B/7B/8B scales, and two RL optimization methods.

All three settings use DIR with α = 0.75 and 20 additional updates. We evaluate the same seven benchmarks as Table 1, with the same problem and sampling counts: 1,576 problems and 7,864 responses per evaluated checkpoint. Within each setting, the starting RL model, continued-RL control, and DIR use the same evaluation protocol. Following the format of Table 1, we report all seven benchmark accuracies (%) and their unweighted average (Avg.). Bold denotes the best result within each setting.

  • Qwen3-8B: GRPO
MethodAIME24AIME25AMCHMMT25MATH500MinervaOlympiadAvg.
Qwen3-8B27.9221.6769.0612.9283.8528.7749.8142.00
GRPO70.2160.8394.6932.9294.7538.4268.4765.76
Continued RL69.1759.7995.3132.0894.7038.6067.4065.29
DIR76.8866.8896.4139.3895.2542.5668.5569.41
  • Qwen3-8B: PPO
MethodAIME24AIME25AMCHMMT25MATH500MinervaOlympiadAvg.
Qwen3-8B27.9221.6769.0612.9283.8528.7749.8142.00
PPO72.9260.8394.8434.7994.4538.4268.8166.44
Continued RL72.2961.4694.5334.1794.4038.8867.9966.25
DIR77.7166.2596.8840.6294.8042.0069.1069.62
  • OLMo3-7B-Instruct-SFT: GRPO
MethodAIME24AIME25AMCHMMT25MATH500MinervaOlympiadAvg.
OLMo3-7B-Instruct-SFT3.545.0030.310.6258.5019.3023.8520.16
GRPO11.0412.7140.001.8862.1517.0029.8224.94
Continued RL8.7512.0840.622.7162.8016.8230.3424.88
DIR15.2121.4648.126.2565.8516.6434.4629.71

Using unrounded scores, DIR improves the seven-benchmark average over the starting RL checkpoint by 3.66, 3.19, and 4.77 percentage points, respectively. These gains occur on all seven benchmarks in both Qwen settings and six of seven benchmarks for OLMo. Compared with continued RL, the average gains are 4.12, 3.38, and 4.84 percentage points, respectively. Thus, the improvement extends to a larger mathematical reasoning model, a distinct model family, and PPO in addition to GRPO.

Continued RL starts from the same post-RL weights as DIR. Before inspecting evaluation scores, we fixed its endpoint as the first complete update reaching or exceeding the DIR20 training budget. Continued-RL/DIR costs are 7.860/7.225, 11.417/7.324, and 6.480/5.472 GPU-hours, respectively, with budget deviations of +8.80%, +55.88%, and +18.41%.

The results provide additional evidence of transfer across the tested mathematical model families, scales, and RL optimization methods.

Dear Reviewer Ds8t,

Thank you very much for taking the time to read our responses and for letting us know that your concerns have been largely addressed! As the discussion period will end in less than 24 hours, we would be grateful if you could consider reflecting this in your final assessment. Please let us know if there is anything else we can clarify before the deadline.

Best regards,

Authors

Reviewer_asTBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether the policy shift from an RLVR run can be reused as supervision for the same model. The authors propose DIR, which builds a frozen teacher from the pre- and post-RL checkpoints: the teacher is the post-RL policy tilted by the implicit reward ratio between the two. A student initialized from the post-RL policy is distilled toward this teacher on its own rollouts, with no reward model and no outcome reward. The experiments cover mathematical reasoning and agentic search. DIR improves math accuracy beyond an RL plateau, and from intermediate checkpoints it approaches later RL checkpoints at a fraction of the GPU hours; the search results are more mixed, and the paper reports them candidly.

Critical strengths
  1. The main strength is the math result. GRPO plateaus around 40-41 macro accuracy, and DIR reaches 44.17 after 20 updates, above both the best reported RL checkpoint (Table 7) and a compute-matched continued-RL control (Table 1). The gain holds for all four alpha values, so it is not an alpha choice. It also survives the obvious selection check: the periodic checkpoint evaluation used AIME24, AIME25 and 83 problems from the 2022-2023 AMC12 exams, which likely overlap with AMC23; on the four benchmarks never used for checkpoint evaluation (HMMT25, MATH-500, Minerva, Olympiad) the average gain is still 2.0-2.4 points. That is a strong, cheap result.

  2. The theory is correct and honestly scoped. Eqs. (13)-(16) check out, Appendix B.3 is explicit that the local teacher is not the sequence-level tilt, and Section 5.1 says the implicit reward is not a calibrated predictor of task success. The paper also reports evidence against itself: continued RL beats DIR on BCP+ accuracy, completion drops from 90% to 73%, DIR keeps searching after the answer evidence appears, and alpha = 2 collapses on search. Few papers report this much against their own headline.

Critical weaknesses
  1. The paper's thesis is that the RL-induced shift is the reusable supervision, but no experiment isolates the pre-RL contrast. p_T = p_post * (p_post/p_pre)^alpha has two ingredients: the anchor and the contrast with p_pre. Nothing separates them. The decisive control is the same pipeline with p_pre replaced by a uniform reference; alpha = 0 is not that control, because the student starts at p_post and the gradient vanishes. Nor is the teacher ever evaluated directly: p_T is a usable two-model decoder, so decoding from it would show what the on-policy distillation adds beyond the extrapolation it imitates. Both controls cost about as much as a DIR run.

The teacher family is also not new. The per-token geometric mixture is decoding-time realignment (DeRa, Liu et al., ICML 2024), the reward up-scaling view is EFT (Mitchell et al., ICLR 2024), and neither is cited. The paper presents the identity as a derivation and credits the reward correspondence to Rafailov et al., so this is a positioning fix, not a novelty indictment. But it sharpens the attribution question: once the teacher is known, the distinctive claim is exactly the one the missing control would test.

  1. The search half is weaker. At matched compute, continued RL beats DIR on BCP+ accuracy (39.52 vs 38.80) and completion (81.33% vs 73.37%), so the paper's own control wins on the headline search metric; RL was also still improving at its last step (Table 12). The multi-hop gain is driven by Bamboogle, and on the inferred 1000/1000/1000/125 split the micro-average is flat. The efficiency claim is the part I find least convincing: DIR-48 is compared with RL288, which sits below earlier checkpoints on a non-monotone curve, and the text gives no tolerance for "comparable". Against the earliest checkpoint that reaches DIR's accuracy (RL224), the saving is about 38%, not the 54% in the figure. The math panel is fine; RL250 is the earliest checkpoint at or above DIR-20, so 47.9% is conservative. On variance, the paper reports none and does not describe multiple runs; the per-question deltas are small, and the recall results are more consistent, which is the better support for the search claim.
Questions for authors
  1. From step 279, what does the same pipeline give with p_pre replaced by a uniform reference, i.e. teacher softmax((1+alpha) z_post)? And what does p_T itself score when decoded directly? Please also position the method against DeRa and EFT.

  2. Table 1 does not state its alpha or DIR step, and Table 3 uses alpha = 2, step 90. How were alpha = .75 and step 20 chosen, and did the AIME24/25 checkpoint evaluation inform that choice? A macro-accuracy vs step curve for each alpha would settle it.

  3. RL starts from Qwen3-8B, but p_pre is described as "Qwen3-8B Base" and "the corresponding Qwen3 base model". Are these the same checkpoint? If p_pre is the Base model, r_delta includes Qwen's post-training shift as well as the RL-induced one.

  4. The KL coefficient, reference policy and clipping settings for the RL runs are never reported. If beta = 0, the correspondence in Appendix B.1 has no counterpart in the runs that produced these checkpoints. What were they?

  5. Please give the per-checkpoint GPU-hour values behind Fig. 2 and state the accuracy tolerance, reporting the saving against the earliest RL checkpoint reaching DIR's accuracy as well as against RL288. For RQ2, is there a continued-RL control from step 96 at matched GPU hours?

I have one future-work suggestion (does not affect my assessment): the alpha = 2 collapse and the post-evidence over-searching suggest gating the extrapolation with outcome validation, as Section 5.2 proposes.

5 comments
AuthorsOpenReview ↗

Response to Reviewer asTB (Part 1)

Response to W1 / Q1: Pre-RL Contrast, Direct Teacher Evaluation, and Relation to DeRa/EFT

We thank the reviewer for these constructive suggestions on more ablation and relevant prior work. We conducted additional experiments to ablate the contribution of the pre-RL contrast, directly evaluate the extrapolated teacher, and clarify the relationship with DeRa and EFT.

1. Uniform-reference ablation: Is the pre-RL contrast necessary?

Following the reviewer's suggestion, we replace (πpre\pi_{\mathrm{pre}}) with a uniform reference, i.e.,

πTUniform=softmax((1+α)zpost),\pi_T^{\mathrm{Uniform}}=\mathrm{softmax}((1+\alpha)z_{\mathrm{post}}),

compared with the original DIR teacher:

πTDIR=softmax((1+α)zpost−αzpre).\pi_T^{\mathrm{DIR}}=\mathrm{softmax}((1+\alpha)z_{\mathrm{post}}-\alpha z_{\mathrm{pre}}).

We use Qwen3-4B at RL step 279 with (α=0.75\alpha=0.75). Both methods initialize the student from the same post-RL checkpoint and use the same on-policy reverse-KL distillation pipeline, teacher top-32 selection, and training configuration. We evaluate at 20 and 100 training steps using identical evaluation protocols across seven mathematical reasoning benchmarks.

Method20 Steps100 Steps
Uniform-reference OPD41.69%41.91%
DIR44.17%44.19%
Improvement+2.48 pp+2.28 pp

DIR achieves higher accuracy on all seven benchmarks at both checkpoints. These results demonstrate that the improvement cannot be explained by merely sharpening the post-RL distribution. The actual pre/post-RL contrast provides additional, useful supervision beyond the post-RL anchor alone.

2. Direct teacher evaluation: What does OPD contribute?

Thanks for this insightful suggestion, we additionally evaluate the extrapolated teacher directly without OPD (direct teacher).

For math reasoning task, at (\alpha=0.75), the direct teacher achieves 43.25%, compared with 40.56% for the original GRPO checkpoint and 41.37% for direct uniform sharpening. DIR further improves accuracy to 44.17% / 44.19% after 20/100 steps, outperforms direct teacher decoding by 0.92 / 0.94 percentage points.

We further compare direct teacher with DIR across different extrapolation strengths (macro-average accuracy):

αDeRa λDirect TeacherDIR (OPD)Improvement
0.51.543.36%43.83%+0.47 pp
0.751.7543.25%44.17%+0.92 pp
1.02.043.21%43.92%+0.71 pp
2.03.042.32%44.13%+1.81 pp

While direct teacher decoding already outperforms the original GRPO checkpoint, DIR consistently achieves higher accuracy than direct decoding at every α, with improvements ranging from 0.47 to 1.81 percentage points.

For search agent task, DIR consistently achieves higher accuracy than the direct teacher, reaching a macro-average accuracy of 53.01 compared to 50.91.

BenchmarkDirect TeacherDIR (OPD)
BCP+36.6338.80
FRAMES46.0448.08
HotpotQA67.9071.00
2Wiki55.5056.60
MuSiQue33.0034.00
Bamboogle66.4069.60

These results demonstrate that OPD provides additional performance gains beyond decoding-time extrapolation.

Importantly, DIR also provides a clear deployment advantage by distilling the two-model teacher into a single student. On the same H20 GPU, using 16 prompts with 512 generated tokens each, direct teacher decoding achieves 588.2 tokens/s, whereas DIR at steps 20/100 achieves 1164.1 / 1085.7 tokens/s, corresponding to 1.98× / 1.85× higher throughput.

3. Relation to DeRa and EFT.

We thank the reviewer for mentioning the relevant prior work DeRa and EFT.

DeRa (Liu et al., ICML 2024) performs decoding-time realignment by interpolating the logits of a reference and an aligned model:

πDeRa=softmax(λzaligned+(1−λ)zref).\pi_{\mathrm{DeRa}}=\mathrm{softmax}(\lambda z_{\mathrm{aligned}}+(1-\lambda)z_{\mathrm{ref}}).

Setting zaligned=zpost,zref=zprez_{\mathrm{aligned}}=z_{\mathrm{post}}, z_{\mathrm{ref}}=z_{\mathrm{pre}}, and λ=1+α\lambda=1+\alpha recovers exactly our extrapolated teacher.

EFT (Mitchell et al., ICLR 2024) combines a base policy with the behavioral change encoded by a fine-tuned/reference policy ratio:

πEFT∝πbaseπftπref.\pi_{\mathrm{EFT}}\propto\pi_{\mathrm{base}}\frac{\pi_{\mathrm{ft}}}{\pi_{\mathrm{ref}}}.

Taking πbase=πpost,πft=πpost\pi_{\mathrm{base}}=\pi_{\mathrm{post}}, \pi_{\mathrm{ft}}=\pi_{\mathrm{post}}, and πref=πpre\pi_{\mathrm{ref}}=\pi_{\mathrm{pre}}, and scaling the log-probability ratio by α\alpha, yields exactly our teacher:

πT∝πpost(πpostπpre)α.\pi_T\propto\pi_{\mathrm{post}}\left(\frac{\pi_{\mathrm{post}}}{\pi_{\mathrm{pre}}}\right)^\alpha.

We acknowledge these connections and will explicitly cite and discuss both works. Please also kindly remind that unlike decoding-time extrapolation, DIR improves beyond the evaluated RL checkpoint and direct teacher, and produces a deployable single-model policy with higher inference throughput.

AuthorsOpenReview ↗

Response to Reviewer asTB (Part 2)

Response to W2: Search Performance, Compute Efficiency, and Statistical Evidence

We thank the reviewer for the careful and constructive review. According to your suggestion, we conducted more comprehensive comparison with Continued RL, and detailed the aggregation of multi-hop results, compute efficiency, and evaluation variance as below.

(1) Comparison with Continued RL on Search Benchmarks.

We acknowledge that DIR does not outperform Continued RL on every search metric. Its clearest benefit is improved retrieval recall, along with accuracy gains over the original RL checkpoint.

On BCP+, DIR improves accuracy from 36.14 to 38.80 and recall from 53.78 to 60.77. At matched additional compute, Continued RL achieves higher accuracy (39.52 vs. 38.80), while DIR achieves higher recall (60.77 vs. 58.48). DIR also exceeds Continued RL in both accuracy and recall on BrowseComp-ZH (22.84/67.47 vs. 19.72/63.54) and FRAMES (48.85/90.46 vs. 46.04/86.96).

We will make this more transparent in our revision.

(2) Macro- vs. Micro-Averaged Multi-Hop Accuracy.

We will report both macro- and micro-averaged accuracy using the exact evaluation-set sizes, and clarify that the evidence supports a benchmark-dependent improvement rather than a uniform gain across multi-hop QA tasks.

(3) Compute Efficiency and RL Checkpoint Selection.

We thank the reviewer for pointing out the ambiguity in our efficiency comparison. We will revise the analysis to compare each DIR checkpoint with the earliest evaluated RL checkpoint whose accuracy reaches or exceeds it.

Under this criterion, DIR48 (33.01%) is compared with RL224 (33.73%). Their cumulative computational costs are 651.7 and 969.4 GPU-hours, respectively, corresponding to a 32.8% saving. This comparison includes both the initial RL training and the subsequent DIR stage.

We will also report the final-checkpoint comparison separately: DIR96 achieves 35.42% accuracy versus 36.14% for RL320, with a 0.72 percentage-point gap while using 59.5% of the cumulative GPU-hours.

(4) Evaluation Variance and Statistical Evidence.

We agree that reporting evaluation variance is important. To address this concern, we conducted 3 independent evaluation runs of the DIR model. The resulting accuracy statistics (mean and standard deviation, in %) are summarized below:

BenchmarkDIR Acc MeanDIR Acc Std
HotpotQA70.730.40
2WikiMultiHopQA56.630.29
MuSiQue33.930.45
Bamboogle68.800.80
BrowseComp-Plus (BCP+)38.470.42
BrowseComp-ZH22.720.20
FRAMES48.761.03

These results provide additional evidence of the evaluation variability of DIR. We also agree that the retrieval recall results provide more consistent support for DIR's benefits in agentic search than the relatively small accuracy differences.

We will revise our manuscript to more precisely characterize DIR's benefits and limitations: DIR consistently improves retrieval recall across all long-horizon search benchmarks, generally achieves higher accuracy and reduces training cost.

AuthorsOpenReview ↗

Response to Reviewer asTB (Part 3)

Response to Q2: Clarification of Extrapolation Strength and Checkpoint Selection

We thank the reviewer for the suggestion. Table 1 uses α=0.75 at step 20, achieving 44.17% macro-average accuracy (+3.61 points over GRPO), while Table 3 uses α=2 at step 90 to analyze behavioral changes under stronger extrapolation and longer training. The four α settings in Table 2 correspond to steps 100, 20, 60, and 90, respectively. We will clarify these configurations in the revision.

As described in Appendix D.2, AIME24/25 were included in periodic checkpoint evaluations and informed checkpoint selection for math reasoning. Following the reviewer's suggestion, we provide the complete macro-average accuracy across seven benchmarks for all four α values, evaluated every 10 steps.

Table R1: Macro-average accuracy (%) across DIR training steps and extrapolation strengths. GRPO baseline: 40.56%.

α / DIR Step102030405060708090100
0.542.2343.2643.2143.5443.6843.6243.6543.7343.8043.83
0.7542.2644.1743.7843.6543.7843.8044.1243.9943.9044.19
1.042.5843.6543.7843.8943.8643.9243.7143.6443.5443.56
2.042.0843.4543.7443.5444.0643.7943.7043.8044.1343.73

These results highlight DIR's rapid improvement and robustness. With α=0.75, DIR reaches 44.17% within just 20 steps, only 0.02 percentage points below the highest observed accuracy (44.19%). Moreover, all 40 configurations from steps 10–100 outperform GRPO, demonstrating that the gains are not dependent on a particular α or checkpoint. We will include these results and clarify the checkpoint-selection procedure in the revised manuscript.

Response to Q3

We thank the reviewer for pointing out this typo. The Qwen3-8B model used to initialize GRPO and the pre-RL reference used by DIR are exactly the same checkpoint. Therefore, the extracted policy shift reflects the GRPO-induced changes, without introducing an additional shift from earlier Qwen post-training. We will revise the naming to avoid ambiguity.

Response to Q4

Thank you for this insightful comment. KL-regularized policy optimization is a theoretical assumption underlying our analysis, rather than an explicit penalty used in our RL training. This assumption is motivated by the implicit inductive bias of on-policy RL. As shown by previous work [1], policy-gradient optimization favors minimum-KL solutions even without explicit regularization. The same assumption is also widely used to derive the relationship between policies and rewards [2,3]. We will clarify this assumption and distinguish the idealized reward correspondence from the policy shift observed in practice in our revision.

[1] RL's Razor: Why Online Reinforcement Learning Forgets Less, ICLR’26

[2] From r to Q∗: Your Language Model is Secretly a Q-Function, COLM’24

[3] Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolation

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank the AC and all reviewers for their constructive feedback and discussion. We are encouraged that the reviewers recognized DIR as a simple, understandable, and easy-to-reproduce framework (MSds, Ds8t) that provides dense token-level supervision from a single pair of frozen checkpoints without requiring a separate reward model, verifier, or external teacher (MSds, asTB).

During the rebuttal, we conducted additional experiments and clarified the details to address the following concerns:

  • Generalization Across Models, Scales, and RL Optimizers (Reviewer Ds8t): On Qwen3-8B (GRPO), Qwen3-8B (PPO), and OLMo3-7B-Instruct-SFT (GRPO), DIR consistently improves the 7-benchmark math average by +3.66, +3.19, and +4.77 points over the starting RL checkpoints. Following these results, Reviewer Ds8t confirmed that his/her concerns have been largely addressed.
  • Pre-RL Contrast & Direct Teacher Comparison (Reviewer asTB): We validated that DIR outperforms both a uniform-reference sharpening baseline (+2.48 pp) and direct decoding from the two-model extrapolated teacher, while achieving 1.98$\times$ higher inference throughput as a single distilled model. We also reported multi-run evaluation variance and refined the compute-efficiency comparison.
  • Hyperparameter & Long-Horizon Stability (Reviewer MSds): Evaluating the full (α×step)(\alpha \times \text{step}) grid shows that all 40 configurations outperform GRPO, and extending training to 600 updates (under fixed-teacher, periodic-refresh, and alternating RL–DIR schedules) as well as mixed FP32/BF16 precision confirms strong long-horizon and numerical stability.

We thank the reviewers again for their valuable insights, which have further strengthened our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.