acceptodds
Under review as a conference paper at ICLR 2027

An RL View of OPD: Least Square Policy Distillation for Sample-Efficient LLM Reasoning

Abstract

We study on-policy distillation (OPD) through the lens of reinforcement learning, establishing a connection between the reverse-KL objective in OPD and KL-regularized policy optimization. Building on this connection, we introduce Least-Square Policy Distillation (LSPD), an RL-inspired framework that brings optimistic exploration and off-policy data reuse from value-based RL into policy distillation. LSPD preserves policy diversity through exploration while improving rollout efficiency by repeatedly learning from previously collected trajectories. Our theoretical analysis connects LSPD to optimistic value-based learning and shows that its idealized formulation achieves a sharp regret bound under online exploration. Empirically, LSPD consistently outperforms existing distillation baselines across six mathematical reasoning benchmarks and diverse teacher–student settings, with average gains of points in Avg@16. Remarkably, through Pass@ evaluations up to , we found that LSPD better preserves policy diversity by achieving stronger performance as grows. Its fully off-policy variant achieves comparable performance to vanilla OPD using only the first of rollout batches. Together, these results provide an RL perspective on OPD that offers both a principled interpretation and a practical route toward more effective and rollout-efficient language model distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy Distillation, Reinforcement Learning, LLM Reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zrnfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper approaches the technique of On-Policy Distillation through a KL-Regularized RL perspective. It start from the OPD objective and introduces this idea of token-level reward fitting and learns an optimist estimate of the reward to encourage exploration. By then employing the analytical solution of KL-RL with this optimistic reward, it arrives at an objective that distills the teacher into the student while controlling the entropy of the student (i.e., “imitate this teacher but keeping your options open”). The paper provides empirical evidence in math reasoning benchmarks, claiming improvements over other OPD baselines, and demonstrating better entropy control and off-policy learning.

Critical strengths
  • The paper is well written and very clear. The experimental design is clear, as well as the mathematical derivations and proofs.

  • The proposed method is well developed, assumptions are clear, and the connections between OPD and KL-control justified.

  • The empirical evidence is supportive to the claims: they not only show the improvements over prior methods, but also shows the main two strengths of the method: the capacity of controlling the entropy (preventing collapse and encouraging exploration) and enabling off-policy learning.

  • The paper also provides theoretical analysis on the Regret, which complements well with the empirical evidence.

Critical weaknesses
  • My main concern is the statistical significance of the results. I did not find any references to how many experimental seeds were used, so I am assuming a single one. The paper brings results in benchmarks which usually contain few datapoints (like AIME) and do not report confidence intervals. Given how close the methods’ performance are (sometimes reporting < 1pp gains), it is unclear if these gaps are due to other factors like observation noise, checkpoint selection, etc. I would strongly recommend reporting at least confidence intervals (on a few training/eval seeds) to assess the statistical significance of the results.

  • The paper does not discuss any limitations of the method, and I think this is important. Particularly, I am curious to know whether implicitly learning this optimistic reward function might lead to reward hacking if you train for long enough.

  • I believe it is important to clarify that the connection between OPD and KL-RL/MaxEnt-RL is not a novel contribution of this work (section 4.1) - in fact, it was already presented in prior works (e.g., see Appendix C.1 in [1]), which also points to the direction of defining of OPD defining an implicit reward function defined by the teacher. The text, particularly in the beginning of sec 4.2, and partially the paper title, gives a misleading impression that this paper is introducing this as a novel idea.

  • The related work does a good job on listing related papers, but needs to better contextualize and contrast this work. It now works as a list of what previous papers introduced, but does not clarify what they share in common and what they differ from the current work.

[1] Hubotter et. al. Reinforcement Learning via Self-Distillation, 2026.

Questions for authors

See my question about limitations and reward hacking in the weaknesses.

5 comments

We thank the reviewer for the constructive feedback and support for acceptance. We address the reviewer's concerns below:

W1. Can you provide the confidence intervals and more seeds for LSPD?

Response. We evaluated the checkpoints from our proposed methods and all baselines for the Qwen3-8B → Qwen3-4B-Base distillation experiment across all six benchmarks using three random seeds. The table below reports the mean performance and standard deviation across the three seeds. The results demonstrate that our proposed methods achieve improved overall performance, with both LSPD-RB (43.23 ± 0.99) and LSPD (42.89 ± 0.67) outperforming the strongest baseline (41.91 ± 0.75) in terms of Avg@16 across 6 benchmarks, which demonstrates statistically significant improvement.

MethodMATH-500MinervaOlympiadAMC23AIME24AIME25
KD79.68 ± 0.21 / 94.40 ± 0.5338.15 ± 0.79 / 58.58±0.85‾\underline{58.58 \pm 0.85}45.97 ± 0.68 / 67.85 ± 1.0351.20 ± 0.60 / 77.11 ± 1.2017.71 ± 0.42 / 33.33 ± 3.3317.08 ± 0.63 / 36.67±5.77‾\underline{36.67 \pm 5.77}
OPD79.41 ± 0.32 / 95.00 ± 0.3535.95 ± 1.20 / 58.33 ± 0.2144.64 ± 0.80 / 67.85 ± 0.7448.49 ± 0.30 / 79.92 ± 1.8417.92 ± 1.04 / 30.00 ± 5.7715.83 ± 1.46 / 40.00 ± 3.33
EOPD80.41 ± 0.86 / 94.87 ± 1.2137.78 ± 0.64 / 59.56 ± 0.6446.36 ± 0.69 / 68.00 ± 0.5951.05 ± 1.25 / 80.72 ± 2.4117.92 ± 0.63 / 36.67±6.67‾\underline{36.67 \pm 6.67}17.92 ± 0.42 / 40.00 ± 6.67
LSPD81.54 ± 0.45 / 95.33±0.31‾\underline{95.33 \pm 0.31}38.99 ± 0.69 / 59.56 ± 0.7447.22±0.30‾\underline{47.22 \pm 0.30} / 69.04±1.03‾\underline{69.04 \pm 1.03}52.48 ± 0.89 / 81.93±3.61‾\underline{81.93 \pm 3.61}18.96±0.83‾\underline{18.96 \pm 0.83} / 36.67±5.77‾\underline{36.67 \pm 5.77}18.13±0.83‾\underline{18.13 \pm 0.83} / 36.67±3.33‾\underline{36.67 \pm 3.33}
LSPD-RB81.37±0.62‾\underline{81.37 \pm 0.62} / 95.40 ± 0.5338.49±0.99‾\underline{38.49 \pm 0.99} / 58.09 ± 0.3747.41 ± 0.57 / 69.33 ± 0.4452.11±1.41‾\underline{52.11 \pm 1.41} / 84.34 ± 4.3421.25 ± 1.25 / 50.00 ± 6.6718.75 ± 1.10 / 36.67±6.67‾\underline{36.67 \pm 6.67}

W2. The paper does not discuss any limitations of the method, and I think this is important. Particularly, I am curious to know whether implicitly learning this optimistic reward function might lead to reward hacking if you train for long enough.

Response. We thank the reviewer for highlighting this point and have added a discussion of the limitations. The entropy bonus promotes diversity without directly accounting for response correctness or uncertainty in the implicit reward. An excessively strong bonus may encourage diverse but incorrect continuations, creating a tradeoff between exploration and fidelity to teacher supervision. Our practical algorithm uses a fixed entropy coefficient; adapting this coefficient to the student’s learning progress or the coverage of collected data remains a direction for improvement. Moreover, LSPD inherits the support and coverage limitations of standard OPD: the teacher must assign nonzero probability to tokens sampled by the student. Even when this support condition holds, a substantial mismatch between the teacher and student distributions can limit the effectiveness of distillation.

Regarding reward hacking, we trained LSPD for a full 200 training steps and did not observe significant performance drops, suggesting that reward hacking is not a significant issue within the evaluated training horizon. The practical objective also retains an explicit log-probability matching term (Eq. 4.6), which continually anchors the student to teacher supervision on sampled trajectories and penalizes deviations from these targets. This provides a constraint on the implicit reward throughout optimization.

We have added a concise version of this discussion in our revised manuscripts.


W3. Can you clarify the contribution of this paper beyound MaxEnt-RL?

Response. We agree that the connection between OPD and KL-regularized/maximum-entropy RL presented in Section 4.1 has been established in prior work. We cited [2] in the original manuscript and have now added [1] to Section 4.1 alongside it.

Our contribution goes beyond the objective reformulation in Eq. (4.1). We build on this connection to incorporate two key ideas from reinforcement learning for improving sample efficiency: off-policy trajectory reuse through a replay buffer and exploration. Specifically, our framework shows how reverse-KL-based, OPD-style distillation can benefit from these ideas to reuse historical trajectories and conduct distributional exploration to improve sample efficiency. This integration is the key novelty of our work. We further provide a theoretical analysis of the optimistic formulation in Theorem 6.2, establishing its sample-efficiency guarantees.

W4. The related work does a good job on listing related papers, but needs to better contextualize and contrast this work. It now works as a list of what previous papers introduced, but does not clarify what they share in common and what they differ from the current work.

Response. We thank the reviewer for pointing this out. We have expanded the related work discussion to clarify the similarities and differences between these studies and our work.

[1] Hubotter et. al. Reinforcement Learning via Self-Distillation, 2026.

[2] Yang et. al. Learning beyond teacher: Generalized on-policy distillation with reward extrapolation, 2026.

Dear Reviewer zrnf,

Thank you again for your detailed and constructive review and your support for our work. As we are about halfway through the discussion period, we would like to kindly follow up.

We would like to briefly summarize our main clarifications and additional analyses:

  • Multi-seed evaluation (W1). We additionally evaluate our methods and all baselines using three random seeds across six benchmarks for Qwen3-8B → Qwen3-4B-Base distillation. LSPD-RB achieves an average Avg@16 of 43.23 ± 0.99, and LSPD achieves 42.89 ± 0.67, compared with 41.91 ± 0.75 for the strongest baseline, demonstrating improved mean performance.

  • Limitations and reward hacking (W2). We add an explicit limitations discussion addressing the exploration–fidelity tradeoff, fixed entropy coefficient, and teacher–student support mismatch. We also extend LSPD training to 200 steps without observing significant performance degradation. The explicit teacher log-probability matching term further constrains deviations from teacher supervision.

  • Contributions beyond MaxEnt-RL (W3). We clarify that our main contribution is not the existing equivalence between OPD and KL-regularized RL, but the integration of off-policy trajectory reuse and exploration into a sample-efficient distillation framework, supported by the theoretical guarantees in Theorem 6.2.

  • Related work (W3 & W4). We add relevant references to Section 4.1 and expand the related work discussion to better distinguish our contributions from existing language model distillation and RL methods.

We would greatly appreciate it if you could let us know whether our responses address your concerns, and we would be happy to clarify any remaining questions.

Reviewer_XMSFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Standard on-policy distillation (OPD) needs fresh student rollouts for every update, which is costly in the training process. To this end, this work proposes Least-Square Policy Distillation (LSPD), which preserves policy diversity through exploration while improving rollout efficiency by repeatedly learning from previously collected trajectories. This work first rewrites OPD as a KL-regularized RL problem, with the teacher’s log-probability ratio as the reward, and then use ideas from optimistic RL to derive a new training objective. In practice, LSPD minimizes the squared difference between the student’s and teacher’s log-probabilities on sampled tokens, plus an entropy item to keep the student’s outputs diverse. This new loss can also be computed on older trajectories, the student can be trained several times on each batch, or on a replay buffer of past trajectories. Across three Qwen3 teacher–student pairs and six math benchmarks, LSPD outperforms OPD and EOPD on average, and LSPD-RB achieves performance comparable to OPD using only about a quarter of the rollout batches.

Critical strengths

This work provides a clear view of OPD through KL-regularized RL, and the final method is simple and easy to implement while allowing rollouts to be reused. The LSPD with replay-buffer reaches OPD-level performance with far fewer rollout batches, which is indeed useful in practice. Also, I think the experiments are fairly comprehensive shown in this manuscript, like covering three teacher–student pairs, six benchmarks, and Pass@k scaling. The entropy analysis and ablations are also convincing.

Critical weaknesses

(1) First, I think the novelty of LSPD is overstated. The gradient of the squared loss (log⁡π−log⁡πE)2(\log\pi - \log\pi^E)^2 is exactly twice the gradient of the standard OPD loss, and this holds true on any data. So applying the OPD loss to stored rollouts already gives the same update. In this case, I have to say that LSPD can be directly viewed as OPD plus an entropy term. Therefore, I cannot appreciate the lengthy theoretical analysis in Section 4. It is largely disconnected from the method that is actually used in the experiments. In my view, this part over-packages a simple method with heavy theory, rather than providing real insight into why it works.

(2) Based on Table 1 and Table 3, LSPD without the entropy term still beats OPD on average Avg@16, even though their gradients should be the same, while the entropy term itself adds not too much. This suggests the gains may just come from reusing past rollouts, rather than from the proposed formulation. Therefore, I believe that to support the contribution of the new formulation, the authors should compare LSPD with OPD under exactly the same training setup like the same number of updates per rollout batch, the same replay data for LSPD-RB, and also include OPD with an entropy term as a baseline.

Questions for authors

Please see weaknesses

4 comments

We thank the reviewer for the constructive feedback. We address the reviewer's concerns below:

W1. What's the difference and contribution of LSPD despite their similar gradient format with OPD?

Response. We thank the reviewer for the comment. We clarify the methodological contributions of LSPD and their connections to the theoretical analysis.

First, regarding the gradient equivalence, we would like to highlight that the key distinction lies in the optimization framework and the data distribution. Standard OPD takes expectations over trajectories generated by the current student policy (Ey∼π(⋅∣q)E_{\color{red}{ \mathbf{y}\sim\pi(\cdot|\mathbf{q})}}), whereas LSPD (Eq. 4.6) optimizes over accumulated trajectories generated by different historical policies in a replay buffer (E(q,y)∼B{E}_{\color{blue}{(\mathbf{q},\mathbf{y})\sim\mathcal{B}}}).

This enables repeated updates on stored rollouts and historical data reuse through LSPD-RB and also makes the exploration (by maximizing the entropy) necessary. Furthermore, existing OPD implementations commonly employ PPO-based optimization [1,2,3] with additional components for on-policy optimization such as clipping. LSPD instead directly optimizes the proposed objective, without requiring any on-policy components introduced in PPO. These components together establish a practical framework for off-policy distillation.

Second, regarding the theoretical analysis, Section 4 presents a perspective based on optimistic KL-regularized reinforcement learning, highlighting the potential of leveraging off-policy data for more sample-efficient learning. Our theoretical analysis is closely connected to the practical algorithm rather than merely adding theoretical complexity. Starting from the equivalent KL-regularized formulation of OPD (Eq. 4.1), we introduce least-squares reward estimation over historical trajectories (Eq. 4.2) and construct a confidence set for optimistic policy optimization (Eq. 4.3). The corresponding Lagrangian relaxation (Eqs. 4.4–4.5) motivates the squared log-probability matching objective used in practice. Our practical algorithm (Eq. 4.6) directly builds upon this formulation. The theoretical analysis in Section 6 is based on the closed-form update in Line 198 and the confidence set in Eq. 4.3, which are closely aligned with the algorithm used in practice. This analysis provides theoretical support for leveraging historical trajectories and optimistic optimization in policy distillation.


W2. What's the performance of OPD with replay data and with entropy term compared with LSPD?

Response. We would first like to clarify that the gradients used in our actual implementations are not identical. Our OPD baseline uses a PPO-style implementation, including on-policy components such as clipping, which changes the practical update relative to the idealized OPD objective, strictly inherited from the existing implementations [1,2,3]. LSPD, by contrast, directly optimizes an objective compatible with off-policy training. Therefore, the equivalence of the idealized gradients format does not imply identical updates in the implementations used in our experiments with different on-policy components. Moreover, comparing between LSPD-RB and OPD, the operating data distribution is different as well, as stated in the response to W1.

To further assess whether the gains can be explained by rollout reuse alone, we additionally evaluate OPD with a replay buffer and 256 updates per rollout batch, matching LSPD-RB. Using Qwen3-4B as the teacher and Qwen3-1.7B-Base as the student, both methods use the same rollout budget of 2,560 rollouts over 10 rollout collection steps. The Avg@16 scores are reported below:

BenchmarkOPD-RBLSPD-RB
AMC2334.1136.60
AIME249.1711.67
AIME257.928.33
MATH-50068.0870.69
Minerva27.0929.57
Olympiad30.7233.50

LSPD-RB achieves higher scores on all six benchmarks under matched update counts and rollout budgets. These results show that simply adding replay to OPD does not recover LSPD-RB’s performance, supporting the benefit of our proposed formulation for off-policy distillation.

[1] Li, Y., Zuo, Y., He, B., Zhang, J., Xiao, C., Qian, C., ... & Ding, N. (2026). Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016.

[2] Yang, W., Liu, W., Xie, R., Yang, K., Yang, S., & Lin, Y. (2026). Learning beyond teacher: Generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125.

[3] Jin, W., Min, T., Yang, Y., Wei, D., Zhou, Y., Kadhe, S.R., Baracaldo, N. & Lee, K.. (2026). Entropy-Aware On-Policy Distillation of Language Models. Proceedings of the 43rd International Conference on Machine Learning.

Dear Reviewer XMSF,

Thank you again for your detailed and constructive review. As we are about halfway through the discussion period, we would like to kindly follow up.

We would like to briefly summarize our main clarifications and additional analyses:

  • Distinction from standard OPD (W1). We clarify that standard OPD optimizes over current student-generated trajectories, whereas LSPD optimizes an objective over historical trajectories stored in a replay buffer. Unlike PPO-based OPD implementations with on-policy components such as clipping, LSPD directly optimizes its proposed objective for off-policy distillation.

  • Connection between theory and algorithm (W1). We clarify how our optimistic KL-regularized RL formulation motivates least-squares reward estimation, confidence-set construction, and the practical squared log-probability matching objective (Section 4). The theoretical analysis in Section 6 further supports historical trajectory reuse and optimistic optimization.

  • OPD with replay (W2). We additionally evaluate OPD with a replay buffer and 256 updates per rollout batch, matching LSPD-RB under the same 2,560-rollout budget. LSPD-RB outperforms OPD-RB on all six benchmarks (31.73 vs. 29.52 average Avg@16), demonstrating that simply introducing replay into the OPD baseline does not recover LSPD-RB's performance.

We would greatly appreciate it if you could let us know whether our responses address your concerns, and we would be happy to clarify any remaining questions.

Hi,

Thank you for the response and the new replay experiment!

The equivalence between the two losses holds on any data, so the real difference is that the OPD baseline uses PPO-style clipping, which naturally limits learning when data is reused many times. The new comparison therefore I think mainly reflects removing clipping, and the requested OPD with an entropy term is still missing.

Therefore, I will keep my score. I find the idea interesting, and encourage the authors to further strength the work in the future submission.

Thanks!

We thank the reviewer for the follow-up question regarding the distinction between our proposed objective and the OPD objective. We would like to further emphasize that the difference goes beyond the clipping operation itself. OPD retains all PPO-style on-policy components as stated in the previous response: in addition to clipping, another key on-policy component is the importance-sampling (IS) ratio used when performing multiple optimization steps on the same rollout batch.

Importantly, we already had an ablation from our earlier experiment history that directly isolates this distinction. Specifically, we evaluated an OPD variant that removes clipping while retaining the IS factor, and additionally enables replay-buffer training and entropy regularization, using Qwen3-4B to distill Qwen3-1.7B Base. We had not included this result in the original submission and the revision, but it is directly relevant to the reviewer’s question, so we report it here:

BenchmarkOPD-RB (IS + entropy, w/o clipping)LSPD-RB
AMC2335.9936.60
AIME2410.4211.67
AIME258.338.33

These results further support that the distinction between LSPD and OPD is not attributable solely to PPO clipping. Even after removing clipping and augmenting OPD with replay-buffer training and entropy regularization, LSPD-RB remains stronger overall. We will incorporate this result, together with a larger-scale ablation across all settings, into the final revision to make this distinction clearer. We hope this additional evidence helps clarify the difference between the two objectives and would be grateful if the reviewer could take it into account for reconsidering the assessment.

Reviewer_gwjhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to improve the reasoning performance and the on-policy distillation efficiency. It trains the student to match the teacher’s probabilities for stored tokens generated by the student, using a Huber-type loss for stability and entropy regularization to encourage diversity. Its replay variant reuses past student responses and cached teacher feedback to reduce the need for new rollouts. The paper also provides an RL version interpretation and analyzes an idealized version of the method. The authors report improved reasoning performance and fewer required rollout batches.

Critical strengths

Reusable teacher feedback. Cached teacher token scores make repeated regression updates quick, and the supplementary material includes an implementation and concrete training settings.

Critical weaknesses

The practical method is insufficiently justified. As the student changes, historical trajectories may become less representative of its current behavior and fail to cover new errors between rollout collections. Entropy regularization may encourage irrelevant tokens and the paper does not demonstrate that entropy regularization provides the optimism required by its theoretical algorithm.

Questions for authors

As the student policy evolves, how do you ensure that historical trajectories remain useful for its current behavior? What evidence supports reusing them rather than collecting fresh on-policy rollouts?

Why does the replay variant use 256 updates per rollout batch, while the settings that use only the latest rollouts perform at most 64? Why use entropy regularization rather than a KL constraint to a recent student policy? Why entropy regularizatio works for this paper?

2 comments

We thank the reviewer for the constructive feedback. We address the reviewer's concerns below:

W1&Q1. How do you ensure that historical trajectories remain useful for its current behavior? What evidence supports reusing them?

Response. We would like to emphasize that our derivation from reverse-KL minimization in Section 4 provides a theoretical basis for using off-policy data in policy distillation. The core insight is that reusing previously collected rollouts is theoretically feasible. Moreover, Figure 2 demonstrates that reusing past rollouts enables stable training and improves sample efficiency. These theoretical and empirical results support the validity and effectiveness of properly reusing historical trajectories as the student evolves, addressing the reviewer’s concern about their continued usefulness between rollout collections.

Regarding the connection between entropy regularization and optimism, we have provided additional justification in Appendix C.7 of the revised manuscript to make this connection clearer.


Q2. Why does the replay variant use 256 updates per rollout batch, while the settings that use only the latest rollouts perform at most 64? Why use entropy regularization rather than a KL constraint to a recent student policy? Why entropy regularizatio works for this paper?

Response To address the difference in update counts, we additionally evaluate LSPD using only the latest rollouts with 256 updates per rollout batch, matching LSPD-RB. Both variants use the same rollout budget of 10 rollout collection steps and 2,560 rollouts in total. The Avg@16 scores on Qwen3-4B distill Qwen3-1.7B-Base are reported below:

BenchmarkLSPD (256 updates, latest rollouts only)LSPD-RB (256 updates)
AMC2334.9436.60
AIME2410.0011.67
AIME256.888.33
MATH-50069.2070.69
Minerva27.0929.57
Olympiad32.5433.50

LSPD-RB achieves higher scores on all six benchmarks with the same number of updates per rollout batch and the same total rollout budget. These results demonstrate that the benefit of replay extends beyond simply increasing the number of updates.

Regarding entropy regularization versus a KL constraint to a recent student policy, we emphasize that our replay batches may contain trajectories generated by multiple historical student policies. The most recent student policy therefore does not represent the behavior distribution of the sampled batch. A standard on-policy trust-region update centered on that policy cannot be directly applied to these data: constraining the updated student to remain close to the recent policy does not resolve the mismatch with the historical policies that generated the samples. Our method is explicitly designed to optimize a fully off-policy distillation objective using these mixed-policy batches. Entropy regularization promotes distributional exploration without anchoring the update to a particular recent student policy, making it consistent with this design.

Beyond its compatibility with our off-policy formulation, we provide theoretical justification for the entropy regularization through its connection to optimism in Section 4. We further clarify this connection in Appendix C.7 of the revised manuscript. Empirically, Table 3 demonstrates the effectiveness of entropy regularization in our method. Our choice is therefore consistent with the intended off-policy update strategy and supported by both the theoretical formulation and the experimental results.

Dear Reviewer gwjh,

Thank you again for your detailed and constructive review. As we are about halfway through the discussion period, we would like to kindly follow up.

We would like to briefly summarize our main clarifications and additional analyses:

  • Off-policy trajectory reuse (W1 & Q1). We clarify the theoretical justification for reusing historical trajectories through our reverse-KL-based formulation in Section 4. Figure 2 further demonstrates that replay enables stable training and improves sample efficiency.

  • Matched-update comparison (Q2). We additionally evaluate LSPD without replay using 256 updates per rollout batch, matching LSPD-RB with the same 2,560-rollout budget. LSPD-RB outperforms LSPD on all six benchmarks (31.73 vs. 30.11 average Avg@16), confirming that the benefits of replay extend beyond additional optimization steps.

  • Entropy regularization and optimism (W1 & Q2). We clarify that KL regularization toward the most recent student policy is not directly suited to replay batches containing trajectories from multiple historical policies. Entropy regularization instead promotes exploration without anchoring updates to a particular student policy. We further provide a detailed theoretical justification connecting entropy regularization to optimism in Appendix C.7.

We would greatly appreciate it if you could let us know whether our responses address your concerns, and we would be happy to clarify any remaining questions.

Comments on the paper

2 comments

Dear Reviewers and Area Chair,

We sincerely thank you for your time and effort in reviewing our manuscript and for your valuable feedback. We have revised the manuscript to incorporate your suggestions and address the concerns raised during the review process. The key revisions are highlighted in red in the updated manuscript and summarized below:

  • Related Work: We have expanded the discussion to clarify the similarities and differences between our work and existing literature.
  • Additional References: We have incorporated additional relevant references in Section 4.1.
  • Limitations: We have added an explicit discussion of the limitations of our work in the Conclusion section.
  • Results on 256 Updates per Batch: We have added experimental results for LSPD with 256 updates per batch in Figure 2.
  • Additional Justification: We have added Appendix C.7, which provides a detailed derivation connecting the optimism principle to the entropy regularization employed in our practical objective.

We appreciate your constructive feedback, which has helped us improve the clarity and completeness of our manuscript.

Best regards,

The Authors

Dear Reviewers and Area Chair,

We sincerely thank you for your constructive feedback and valuable suggestions. We have provided detailed responses, conducted additional experiments, and substantially revised our manuscript to address the concerns raised. We briefly summarize the key updates below:

  1. Methodological novelty and theoretical justification. We clarified the distinctions between LSPD and conventional OPD, particularly its off-policy optimization objective, historical trajectory reuse, and optimistic formulation. We also strengthened the theoretical justification for entropy regularization in Appendix C.7.

  2. Additional controlled experiments. Under matched rollout budgets and 256 updates per batch, LSPD-RB consistently outperforms both LSPD without replay and OPD with replay across all six benchmarks, demonstrating that the improvements cannot be attributed solely to additional optimization steps or replay.

  3. Multi-seed evaluation. Additional evaluations across three random seeds confirm the performance improvements, with LSPD-RB achieving 43.23 ± 0.99 Avg@16 compared with 41.91 ± 0.75 for the strongest baseline.

  4. Manuscript revisions. We expanded the related work discussion, incorporated additional references, clarified the theoretical motivations, and added explicit limitations and training-stability discussions.

We believe these additional results and revisions substantially address the reviewers' principal concerns and strengthen the paper's contributions.

As the discussion period approaches its end, we would greatly appreciate it if the reviewers could revisit our responses and share whether their concerns have been adequately addressed or if any outstanding questions remain.

Thank you again for your time, consideration, and constructive feedback.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.