acceptodds
Under review as a conference paper at ICLR 2027

Off-Policy Policy Optimization

Abstract

Policy gradient methods such as proximal policy optimization (PPO) are typically trained from fresh on-policy rollouts. At the same time, strong teacher policies can provide high-quality demonstrations that expose the student to successful actions and trajectories it may rarely discover on its own. Directly incorporating such demonstrations into policy-gradient training, however, creates substantial policy and state-distribution mismatch, raising concerns about biased or unstable updates and whether learning signal from teacher trajectories remains useful for the student policy. Surprisingly, we find that PPO can learn from teacher demonstrations under extreme policy mismatch when teacher and student samples enter the same standard clipped PPO surrogate with token-level importance ratios. Under explicit assumptions on an unclipped population surrogate, we characterize how teacher demonstrations induce selective imitation, providing action guidance at student-visited prefixes and exploration guidance at prefixes the student has not yet reached, while still permitting improvement beyond the teacher. Empirically, off-policy demonstrations speed up PPO convergence by even when teacher-action importance ratios start as low as , and achieve the highest greedy test accuracy on three language tasks among supervised fine-tuning, on-policy distillation, and on-policy PPO.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Deep Learning, LLM Post Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pEdtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the benefit of including expert data in policy gradient updates without applying per-decision importance sampling correction. The authors argue that off-policy samples can be beneficial, as they can provide "action" and "exploration" guidance. Furthermore, they argue that even actions that are very improbable under the target policy can receive significant gradient updates.

The authors test their hypothesis using PPO on a gridworld and three language tasks of increasing difficulty. The results seem to support the authors' hypothesis.

Critical strengths

Counterintuitive findings The authors claim, essentially, that using an unbiased policy gradient estimator that uses off-policy data without proper importance sampling correction can be beneficial. Their theoretical argument applies to policy gradient without clipping, which makes the statement most interesting, since the clipping mechanism in PPO is exactly designed to prevent off-policy updates (which, even though PPO is considered on-policy, are still present, since the optimization policy diverges from the one used to collect samples as gradient updates are performed). If the authors' reasoning were correct, then this would be an important result. Even more interestingly, the authors define a mechanism to determine whether data is useful, summarized in equation 2.

Small Scale Experiments I like that the authors tested their hypothesis on the Grid-Maze, where they show gradient directions, providing good intuition for the claims, and an important, statistically significant result obtainable with the small problem size (same goes for countdown, word reversal).

Critical weaknesses

Poor concept formalization and sloppy notation: Many mathematical quantities are not defined in the main text (e.g., $\Delta_\rho). The same goes for results or findings, which are often presented at an intuitive (ambiguous) level (e.g., the authors often refer to "improvement" without formalizing what they mean). It is not clear whether the theoretical results admit early episode termination (before H steps). If so, some formulas seem incorrect.

Logical passages missing: In many parts of the paper, I cannot follow the argument's logic. Examples are sections 3.2 and 3.3. I have detailed more precise questions in the appropriate textbox. This is my main concern.

Clear improvement analysis. At the end of the day, beyond all the reasoning, much of which I am not able to follow, the critical question that is important to answer is the following (informally): J(θ+Δθoff−policy)≥J(θ+Δθon−policy)J(\theta + \Delta \theta_{off-policy}) \geq J(\theta + \Delta \theta_{on-policy}) (for small enough steps)? Under what conditions does this hold? The policy gradient theorem already identifies the optimal gradient direction for policy improvement and prescribes using on-policy samples.

Use of clipped objective in the experiments. The authors motivate their approach assuming an unclipped objective (essentially, OffPAC's objective), but test their experiments using PPO. As I have previously remarked, PPO's clipping surrogate objective is actually designed to make off-policy updates less harmful. In a way, this weakens the empirical support for the theory.

GSM8K. Is run only with one seed, which does not tell much about the algorithm's expected performance.

Questions for authors

Clarification

  • Background, line 098. The term dρ(s)=Prρ(st=s)/Hd_\rho(s) = Pr_\rho(s_t=s) / H is unclear to me. 1) On the left side of the equation, there is no time dependence, but such dependence appears on the right-hand side. 2) The right-hand side is "normalized" by the horizon HH. But what happens to episodes that terminate before HH steps?

  • Notice that, in the finite-horizon undiscounted setting, the value and Q-functions and the state visitation are time-dependent. I assume that we can still write Q(s, a) as a shorthand for Q(s, a, t), since in the transformer architecture there is a time embedding.

  • I think the statement 142--151 is vague and unclear. Terms like Δρ\Delta_\rho are not properly defined (one needs to consult the appendix), and it is unclear what kind of improvement line 140 refers to. I.e., an improvement from a policy gradient update?

  • I don't understand how Eq. 4 supports the claim in Section 3.2. It feels completely disconnected to me.

  • Can you concisely explain how you derive eq 5?

  • I don't understand the reasoning in 197--200; in particular, I don't understand where μ(a∣s)πold(a∣s)/μ(a∣s)\mu(a|s) \pi_\text{old}(a|s)/ \mu(a|s) comes from. Why $ \pi_\text{old}(a|s)$ is at the numerator? Is it because we are looking at the first step of the optimization? Why should one assume the teacher contribution is negligible compared to the online contribution? The idea of using importance sampling is to make them "equivalent" in expectation.

  • What is the author's take on my comment in the "Critical Weaknesses"? Clear improvement analysis. At the end of the day, beyond all the reasoning, much of which I am not able to follow, the critical question that is important to answer is the following (informally): J(θ+Δθoff−policy)≥J(θ+Δθon−policy)J(\theta + \Delta \theta_{off-policy}) \geq J(\theta + \Delta \theta_{on-policy}) (for small enough steps)? Under what conditions does this hold? The policy gradient theorem already identifies the optimal gradient direction for policy improvement and prescribes using on-policy samples.

4 comments

Thank you for the detailed review. We address your questions below.

  1. Notation and finite-horizon conventions

The corrected state frequency is

dρ(s)=1H∑t=0H−1Pr⁡ρ(st=s).d_\rho(s)=\frac1H\sum_{t=0}^{H-1}\Pr_\rho(s_t=s).

The state contains the full autoregressive prefix, whose length determines the time step, so QQ and VV need no separate time argument. Episodes terminating early are padded through HH with time-indexed absorbing states and zero subsequent rewards. This preserves episode return and normalizes dρd_\rho. Absorbing-state values and advantages are zero. We have corrected the occupancy definition and added the padding convention in Background and the appendix.

  1. Definition of the advantage margin

For a chosen action a⋆a_\star, let

wb=πold(b∣s)1−πold(a⋆∣s),b≠a⋆.w_b=\frac{\pi_{\mathrm{old}}(b\mid s)}{1-\pi_{\mathrm{old}}(a_\star\mid s)}, \qquad b\ne a_\star.

We define

Δρ(s)=A‾ρ(s,a⋆)−∑b≠a⋆wbA‾ρ(s,b).\Delta_\rho(s)=\overline A_\rho(s,a_\star) -\sum_{b\ne a_\star}w_b\overline A_\rho(s,b).

This compares the chosen action with the student's weighted alternatives, using source ρ\rho's values. A state-only baseline cancels because the weights sum to one. The definition now appears at its first use in Section 3.1.

  1. Meaning of “improvement”

The margin–visitation condition guarantees an increase in the frozen mixed surrogate:

L(πδ)−L(πold)=δ2[dπold(s)Δπold(s)+dμ(s)Δμ(s)]>0,L(\pi_\delta)-L(\pi_{\mathrm{old}}) =\frac\delta2\left[d_{\pi_{\mathrm{old}}}(s)\Delta_{\pi_{\mathrm{old}}}(s) +d_\mu(s)\Delta_\mu(s)\right]>0,

where πδ\pi_\delta increases a⋆a_\star's probability at ss and reduces alternatives proportionally. This does not guarantee increased return. The separate student–teacher equality result concerns return: with unbiased advantage targets, the mixed gradient is H−1∇JH^{-1}\nabla J, so a sufficiently small step increases return if ∇J\nabla J is nonzero. Section 3.1 now distinguishes these claims.

  1. Comparison with on-policy updates

We do not establish

J(θ+Δθoff−policy)≥J(θ+Δθon−policy)J(\theta+\Delta\theta_{\mathrm{off-policy}}) \ge J(\theta+\Delta\theta_{\mathrm{on-policy}})

in general. The exact return gradient gives the steepest first-order increase among parameter-space directions of equal Euclidean norm. Our motivation concerns finite batches: online rollouts may miss useful prefixes or provide weak action-preference signals. Teacher data can supply coverage and favorable value margins at those prefixes. Our analysis characterizes these surrogate signals; whether their benefit outweighs the off-policy bias is evaluated empirically in Sections 4 and 5.

  1. Connection of Equation 4 to Section 3.2

We acknowledge that the performance-difference identity did not provide strong enough support for our claim. We have replaced that discussion in Section 3.2 with a direct explanation of prefix-distribution bias: token-level correction retains the teacher's prefix distribution, supplying learning signals at prefixes that finite student rollouts may miss. This is the source of exploration guidance, provided the teacher's action preferences are useful.

  1. Derivation of Equation 5

With frozen targets and alternative weights, importance weighting gives the conditional surrogate

ℓρ(x;s)=∑aπx(a∣s)A‾ρ(s,a)=σ(x)A‾ρ(s,a⋆)+[1−σ(x)]∑b≠a⋆wbA‾ρ(s,b),gρ(s):=∂xℓρ(x;s)∣x0=ε(1−ε)Δρ(s).\begin{aligned} \ell_\rho(x;s) &=\sum_a\pi_x(a\mid s)\overline A_\rho(s,a)\\\\ &=\sigma(x)\overline A_\rho(s,a_\star) +[1-\sigma(x)]\sum_{b\ne a_\star}w_b\overline A_\rho(s,b),\\\\ g_\rho(s):=\left.\partial_x\ell_\rho(x;s)\right|_{x_0} &=\varepsilon(1-\varepsilon)\Delta_\rho(s). \end{aligned}

Here σ(x)=1/(1+e−x)\sigma(x)=1/(1+e^{-x}) and σ(x0)=ε\sigma(x_0)=\varepsilon. The shared factor is the logistic derivative. After visitation weighting, relative source strength depends on both margins and visitation. Section 3.3 now includes the conditional surrogate and its derivative.

  1. Clarification of lines 197–200

We evaluate the gradient at the start of the update, where πθ=πold\pi_\theta=\pi_{\mathrm{old}}. For the chosen action, teacher sampling probability times importance weight is

μ(a⋆∣s)πold(a⋆∣s)μ(a⋆∣s)=πold(a⋆∣s)=ε.\mu(a_\star\mid s)\frac{\pi_{\mathrm{old}}(a_\star\mid s)}{\mu(a_\star\mid s)} =\pi_{\mathrm{old}}(a_\star\mid s)=\varepsilon.

Online sampling gives the same product, ε×1\varepsilon\times1. This equates the action weighting, not the gradients: importance sampling equates expectations for the same integrand, whereas source targets and visitation distributions differ. We do not assume the teacher gradient is negligible. Both directional gradients share ε(1−ε)\varepsilon(1-\varepsilon), so both can be small absolutely without the teacher being small relative to the online source.

  1. Unclipped analysis and clipped experiments

The exact identities concern the unclipped surrogate. For positive advantage, PPO clipping suppresses the gradient only above ratio 1+ϵ1+\epsilon; a tiny teacher ratio therefore remains active. Action and exploration guidance can persist through active contributions. However, ratio- and sign-dependent clipping changes effective margins, so the unclipped conditions and relative-gradient comparison need not carry over unchanged. We now state this qualification in Sections 3.1 and 3.3.

  1. Multiple training seeds

We have repeated Countdown, Word Reversal, and GSM8K with three independent training seeds. The main language learning curves now report mean development accuracy with ±1 standard deviation bands (Section 4, Figure 6). Held-out test results also report means and standard deviations across three training seeds (Table 1); Appendix B describes the evaluation protocol.

Held-out test accuracy (%) as mean ± standard deviation across three training seeds (Section 4, Table 1). OPD denotes on-policy distillation.

Greedy decoding

MethodCountdownWord ReversalGSM8K
SFT33.7 ± 0.863.74 ± 5.2966.54 ± 2.69
OPD37.0 ± 0.774.54 ± 4.0567.45 ± 0.68
On-Policy PPO21.7 ± 3.195.80 ± 1.2268.49 ± 1.15
Off-Policy PPO43.6 ± 4.996.52 ± 1.2471.17 ± 0.91

Sampled decoding (temperature 0.8)

MethodCountdownWord ReversalGSM8K
SFT21.4 ± 0.761.36 ± 4.5562.52 ± 1.56
OPD26.6 ± 2.971.68 ± 4.8063.58 ± 0.29
On-Policy PPO20.4 ± 2.194.24 ± 1.1950.14 ± 1.55
Off-Policy PPO41.3 ± 3.594.95 ± 1.4561.71 ± 0.62

We would appreciate clarification of any remaining concerns and would be happy to address them.

Reviewer_pEdtOpenReview ↗

Thanks for the Reply

Dear authors, I've read your reply carefully. I will update my review accordingly and consider it during the reviewers-AC discussion.

Thank you for reading our response carefully and considering the revisions. We appreciate your engagement and would be happy to clarify last-minute questions if you have any.

Reviewer_76qxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper shows that PPO with token-level importance weighting can learned from off-policy teacher demonstrations under extreme policy mismatch. The paper empirically characterizes how teacher demonstrations induce "selective imitation" and "exploration guidance." Empirically, off-policy demonstrations improve PPO convergence by even when importance ratios are initially extremely small (because the teacher’s relative gradient contribution depends on advantage margins and visitation, not the magnitude of the importance ratio).

Critical strengths
  1. The grid maze example did a good job of motivating the research question and teasing out what matters when it comes to the demonstrations (i.e. coverage of the optimal route). I always appreciate detailed, pedagogical examples like this prior to the main experiments.

  2. The ideas in the paper are easy to follow, and the core idea itself is, on the surface, is simple: PPO can candle off-policy teacher demonstrations very well. Since we all use PPO and since LLMs are almost always working with off-policy data, this analysis should be broadly useful to the post-training community.

Critical weaknesses

It is difficult to assess the reliability of empirical claims in section 5 because Figures 6-7 and Table 8 report results for only a single training run per method. RL is notoriously high variance. Without multiple independent runs and measures of variability, it is unclear whether these (often small) improvements are statistically meaningful. I strongly encourage the authors to repeat their experiments across multiple seeds and report the mean and variability (e.g. 95% bootstrap CI) of their results.

It's unclear to me if the benefit of the off-policy data is arising because it produces a more accurate critic (which then leads more data efficient learning, see [1]) or if it's improve the policy update somehow. Additional experiments teasing out exactly which part of the agent the demonstrations primarily improve would be extremely useful for understanding the method.

References:

  1. Wang et al. Improving Value Estimation Critically Enhances Vanilla Policy Gradient. ICML 2025. https://openreview.net/forum?id=Nq3oz7vn3j
Questions for authors
  1. Can you provide additional experiments toward understanding exactly which part of PPO is the reason PPO can handle the off-policy data so well? Is it the critic or the actor?
  2. Is it possible to run additional seeds for experiments in section 5 to account for training variation? If the authors believe this to not be necessary, please explain why.
4 comments

Thank you for the review. We have added repeated language experiments and extended the actor/critic ablation to Countdown.

  1. Reliability across training seeds

We have repeated Countdown, Word Reversal, and GSM8K with three independent training seeds. Both the development learning curves and held-out test results now report means and standard deviations (Section 4, Figure 6 and Table 1); the evaluation protocol is described in Appendix B, “Evaluation.” The updated test results are below; values are accuracy (%) as mean ± standard deviation. OPD denotes on-policy distillation.

Greedy decoding

MethodCountdownWord ReversalGSM8K
SFT33.7 ± 0.863.74 ± 5.2966.54 ± 2.69
OPD37.0 ± 0.774.54 ± 4.0567.45 ± 0.68
On-Policy PPO21.7 ± 3.195.80 ± 1.2268.49 ± 1.15
Off-Policy PPO43.6 ± 4.996.52 ± 1.2471.17 ± 0.91

Sampled decoding (temperature 0.8)

MethodCountdownWord ReversalGSM8K
SFT21.4 ± 0.761.36 ± 4.5562.52 ± 1.56
OPD26.6 ± 2.971.68 ± 4.8063.58 ± 0.29
On-Policy PPO20.4 ± 2.194.24 ± 1.1950.14 ± 1.55
Off-Policy PPO41.3 ± 3.594.95 ± 1.4561.71 ± 0.62

Off-policy PPO has the highest mean greedy test accuracy on all three tasks and the highest mean sampled accuracy on Countdown and Word Reversal. SFT has the highest mean sampled accuracy on GSM8K. Word Reversal remains close between on-policy and off-policy PPO.

  1. Actor versus critic contributions

Our original GridMaze ablation showed that actor-only teacher learning nearly matches training both actor and critic, while critic-only learning provides a much smaller benefit (Appendix A.2, Figure 10). We have repeated this ablation on Countdown and observe the same pattern (Appendix B.1, Figure 11). Final Countdown development accuracies are below (mean ± standard deviation across three training seeds).

MethodGreedySampled
Off-policy PPO: actor + critic44.8 ± 4.2%41.9 ± 5.6%
Teacher actor-only45.1 ± 4.0%40.1 ± 4.1%
Teacher critic-only18.2 ± 0.8%15.0 ± 3.6%
Online PPO20.8 ± 3.8%19.5 ± 2.4%

Actor-only learning closely matches the full mixed method, whereas critic-only learning does not improve on online PPO on Countdown. These ablations indicate that actor-side learning accounts for most of the benefit in these two tasks.

We believe these revisions address the concerns raised, and would appreciate clarification of any remaining substantive concerns that would prevent a positive assessment.

As the rebuttal period closes today, we wanted to follow up on your two main concerns. We now report three-seed language results, including held-out accuracy with standard deviations, and provide Countdown actor/critic ablations supporting the original GridMaze finding that actor-side learning accounts for most of the benefit. We would appreciate your assessment of these additions and welcome any remaining questions.

Thank you for the additional experiments! Figure 6 and Table 1 are much easier to interpret now. Figure 6 results look convincing.

I will raise my score.

Thank you for updating your assessment and for the helpful suggestions that strengthened our experiments.

Reviewer_9auKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the PPO method with teacher policies that differ signifcantly from the current state distribution. The analyzed method mixes student and teacher samples equally in the clipped PPO objective and reweights samples by the token-level importance ratio "student(action|state)/teacher(action|state)".

Critical strengths

Off-policy learning is of significant interest, especially given all the recent work on LLM post-training.

The suggested message that standard off-policy PPO works without the need to change the objective is appealing.

Critical weaknesses

In short, the paper fails to conclusively establish the suggested claim.

Section 3 is missing a punchline.

It looks like this is meant to be a theory section. But it's missing concrete results. The secion currently writes out a few expressions, but there aren't any major conclusions.

Section 3.3, in particular, was not very illuminating to me. What's the takeaway here?

Section 4 is the most thorough section, but the setup is a bit too toy to be convincing. In particular, this example can't establish that there is really a "phenomenon" here. The section primarily shows that mixing in a very good policy (diagonal) is effective, while doing on-policy PPO from a bad policy doesn't work well.

Section 5 is a bit superficial. The off-policy PPO method really only wins in the countdown probelm. On Word Reversal on policy PPO is just as good, and on GSM8k, all methods are a toss up in the end.

Throughout, the writing isn't exactly clear on what the paper is trying to show. Is the goal to establish an empirical phenomenon or is the goal to propose an effective method for something. If the former, it needs a broader evaluation. If the latter, then it needs more baselines for comparison.

Questions for authors

Are there non-toy post-training settings where off-policy PPO is effective?

Can you clarify what exactly Section 3 is trying to show? Is there a claim or a proposition that you can formulate?

3 comments

Thank you for the review. We agree that the original presentation did not make the paper's empirical focus or the takeaways of Section 3 sufficiently clear.

  1. Contribution and empirical focus

The paper studies an underexplored use of existing PPO, rather than proposing a new algorithm. Our goals are to demonstrate that PPO can learn effectively under extreme teacher–student policy mismatch, evaluate it as an alternative to the post-training methods we compare, and characterize the learning signals supplied by teacher data. The empirical evidence is the central contribution; Section 3 explains the mechanisms and their limitations. We have restructured the paper to make this empirical focus and the role of the theoretical characterization clearer.

  1. Section 3: purpose and takeaways

Section 3 provides a theoretical characterization under explicit assumptions, rather than a general convergence or return-improvement guarantee. We acknowledge that the original section lacked a clear punchline and have added takeaway boxes to each subsection:

  • Action and exploration guidance: teacher value margins and visitation frequencies determine how demonstrations influence action preferences in the mixed surrogate.
  • Retained prefix coverage: token-level correction preserves teacher training signals at prefixes that finite student batches may miss.
  • Small-ratio behavior: at a fixed prefix and along the analyzed logit direction, teacher and online gradients share the same small-action-probability factor. Tiny ratios alone therefore do not establish a negligible relative teacher contribution; margins and visitation also matter.

These statements characterize why demonstrations can be useful, without claiming that off-policy updates always improve return or outperform on-policy updates. We also clarify how clipping qualifies the unclipped analysis.

  1. What the maze establishes

The maze is a controlled test of whether extreme policy mismatch prevents PPO from benefiting from teacher data. It lets us vary mismatch across orders of magnitude and inspect the resulting learning signals. We agree that this experiment alone cannot establish broad practical effectiveness.

The results go beyond showing that a high-return teacher helps a poor student. A noisy teacher with lower return but better coverage of the optimal route enables faster initial learning than a higher-return detour teacher. Together with the mismatch sweep and compute-matched comparison, this supports the role of teacher coverage rather than teacher quality or additional optimization work alone. The language experiments test whether the approach remains useful beyond this controlled setting.

  1. Language results and evaluation scope

We have repeated Countdown, Word Reversal, and GSM8K with three independent training seeds; the main learning curves now report means and ±1 standard deviation bands. Off-policy PPO outperforms on-policy PPO on Countdown and GSM8K. Mean greedy test accuracy is 43.6% versus 21.7% on Countdown and 71.17% versus 68.49% on GSM8K. Both the learning curves and held-out test results now summarize three training seeds (Section 4, Figure 6 and Table 1; evaluation protocol in Appendix B).

Held-out test accuracy (%) as mean ± standard deviation across three training seeds (Section 4, Table 1). OPD denotes on-policy distillation.

Greedy decoding

MethodCountdownWord ReversalGSM8K
SFT33.7 ± 0.863.74 ± 5.2966.54 ± 2.69
OPD37.0 ± 0.774.54 ± 4.0567.45 ± 0.68
On-Policy PPO21.7 ± 3.195.80 ± 1.2268.49 ± 1.15
Off-Policy PPO43.6 ± 4.996.52 ± 1.2471.17 ± 0.91

Sampled decoding (temperature 0.8)

MethodCountdownWord ReversalGSM8K
SFT21.4 ± 0.761.36 ± 4.5562.52 ± 1.56
OPD26.6 ± 2.971.68 ± 4.8063.58 ± 0.29
On-Policy PPO20.4 ± 2.194.24 ± 1.1950.14 ± 1.55
Off-Policy PPO41.3 ± 3.594.95 ± 1.4561.71 ± 0.62

On Word Reversal, on-policy PPO reaches similar final performance. This is consistent with the task being simple enough for online exploration to suffice, although we have not isolated task simplicity as the cause. We do not claim that off-policy PPO must improve every task.

The language comparisons include SFT, on-policy distillation, and on-policy PPO, with additional AGRO and Score Centering comparisons on Word Reversal. Off-policy PPO is competitive with these evaluated methods; we do not claim superiority over all post-training approaches or decoding metrics.

GSM8K provides a standard mathematical-reasoning benchmark beyond the controlled tasks. The appendix also reports a multi-agent voting experiment: across three seeds, off-policy PPO reaches a mean development social optimum rate of 58.7%, compared with 41.9% for on-policy PPO and 50.3% for on-policy distillation. This experiment uses a scripted teacher and a unit denominator for demonstration actions, and matches fresh-game collection rather than total optimization work. It provides complementary evidence in an interactive setting. We regard this suite as evidence of effectiveness in academic post-training settings, while acknowledging that broader deployment-scale evaluation remains open.

We would appreciate clarification of any remaining concerns and would be happy to address them.

As the rebuttal period closes today, we wanted to briefly follow up. Our response clarifies the paper’s empirical contribution and Section 3’s takeaways, and includes updated language results across three seeds and the additional multi-agent evaluation. We would appreciate your assessment of whether these revisions address your concerns, and welcome any remaining questions.

Comments on the paper

2 comments

Thank you to all three reviewers for the careful reading and constructive suggestions. We have revised the manuscript in three main ways.

  1. More reliable empirical results. We repeated Countdown, Word Reversal, and GSM8K with three independent training seeds. The revised learning curves and held-out test table report means and standard deviations. Off-policy PPO improves over on-policy PPO on Countdown and GSM8K, while achieving comparable performance on Word Reversal.

  2. Additional evidence on the learning mechanism. We extended the actor-versus-critic ablation from GridMaze to Countdown. Actor-only learning closely matches the full actor-and-critic method, whereas critic-only learning provides little benefit on Countdown. These results indicate that direct actor updates on teacher data account for most of the observed gains in these experiments.

  3. Clearer framing and theoretical explanations. We reorganized the paper to foreground the empirical results and clarified that the contribution is an investigation of existing PPO under extreme policy mismatch. We corrected the finite-horizon notation, defined the advantage margin explicitly, added derivations and subsection takeaways, and clarified the scope of the surrogate analysis and its relationship to PPO clipping. The revised discussion directly connects retained teacher prefix coverage to exploration guidance.

Our individual responses address each reviewer’s questions in detail. All section, table, and figure references refer to the revised manuscript.

Dear AC and reviewers,

Thank you for the constructive feedback. We have addressed the concerns raised through additional three-seed experiments, actor/critic ablations on Countdown, clearer theoretical definitions and derivations, and revisions clarifying the paper’s contribution and takeaways.

We appreciate the reviewers’ consideration of these updates, including the positive reassessment of the experimental evidence. We welcome any remaining questions and hope the revised manuscript and responses support a favorable assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.