acceptodds
Under review as a conference paper at ICLR 2027

Off-Policy Learning to Reason Works Because It Is More Pessimistic Than You Think

Abstract

Large scale reinforcement learning has become a central tool for improving reasoning in large language models. At this scale, generation is often lagged or asynchronous, so updates are performed on data collected by older policies. This makes learning inherently off-policy. Most existing approaches nevertheless remain rooted in PPO-style trust-region objectives, treating training as approximately on-policy and using importance weights to correct distribution mismatch. These corrections can introduce high variance, destabilize optimization, and accelerate entropy collapse. Recent work suggests an alternative: rather than correcting the mismatch, one can embrace off-policy data and remove importance weights, often yielding stronger algorithms. In this paper, we provide an intuitive construction of off-policy objectives that include successful off-policy objectives and show that their effectiveness can be understood through implicit pessimism: they optimize toward target policies that are more conservative than their nominal objectives suggest. This perspective explains why some particular implementation choices improve stability: they implicitly control the effective target distribution. We then propose a principled modification that stabilize this induced distribution and improve off-policy learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RLVR, Learning to Reason, Pessimism

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mt4zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discusses issues with large scale RL where generation is lagged due to the sheer scale of the training framework. The main issue is that if the generation lags and is created via older policies, the model updates using these policies will not be good (off-policy). This paper talks about construction of ratio-free off-policy objectives that improves training stability.

Critical strengths

The paper is clearly written with clear problem statement. Paper contains detailed proofs of the propositions.

Critical weaknesses

The models and dataset used are not really large scale, also this is single Qwen3 model family, so it is unclear if this method is generalizable. How does this affect the final performance? There seems to be no accuracy result published, does this actually help the final task we are training for? The main contribution seems to focus on OAPL, which obviously has its downsides. But Is this the correct baseline to compare to?

Questions for authors

See weaknesses

3 comments

Thank you for highlighting the clear problem statement, presentation, and detailed proofs. These are exactly the points we want to convey. Our goal is to explain why existing off-policy objectives work and use this understanding to obtain the desired pessimistic regime directly.

Scale and generalization. OAPL already demonstrates that these objectives work at scale. Our contribution is theoretical understanding, supported by experiments that test its predictions. The analysis concerns the learning objective and does not depend on a particular model family. Our experiments on Qwen3-4B and Qwen3-8B test the predicted behavior across model sizes, regularization strengths, and policy lags. Their purpose is to check the proposed explanation in practical training, rather than repeat OAPL’s validation at scale.

Task accuracy is reported. Section 5 explicitly defines “evaluation reward” as held-out accuracy on Minerva, AIME2025, AMC23, and BRUMO. Figure 2 therefore reports task accuracy alongside entropy. It shows that our change can improve both accuracy and stability, including higher final accuracy with lag L=128L = 128 than near-on-policy GRPO with L=2L = 2.

Choice of baselines. OAPL is the most direct comparison because it uses the same regression loss, allowing us to isolate the effect of advantage normalization. GRPO provides an additional near-on-policy reference.

We have now also tuned OAPL’s advantage temperature β2\beta_2, keeping β1=10−3\beta_1 = 10^{-3}; see the table in our response to Reviewer KXAP. Its final rewards are very close to Lambert, with differences within the observed noise. This agrees with Appendix B.1: appropriate temperature decoupling moves OAPL into the pessimistic regime identified by our theory. Our contribution explains this behavior and guarantees the desired regime through a simple advantage shift, without an extra advantage temperature to tune.


We hope this addresses your concerns and makes the paper’s intended scope and the reported accuracy results clearer. We believe these points mainly needed clarification, while our main findings remain intact. If you have no further concerns, we would appreciate it if you reconsidered your score.

I still have concerns regarding model family types and size of the models tested. Also, the robustness of the baseline still seems to be an issue, so I will keep my scores.

Thank you for the follow-up. We believe there is a big misunderstanding of the paper’s main contribution.

Model size and scope. Fully fine-tuning 4B and 8B models with RL is already a substantial workload on four H100 GPUs. Our experiments test the theoretical predictions across model sizes, regularization strengths, and policy lags. OAPL already demonstrates effectiveness at scale; our goal is to explain its underlying mechanism.

Baseline robustness is the central question of the paper. OAPL can outperform GRPO but relies on temperature decoupling that its nominal theory does not explain. We explain why this helps and how our advantage shift directly obtains the desired pessimistic regime. The added tuned-OAPL comparison gives rewards close to Lambert, within noise, supporting our explanation and showing comparable performance without an extra advantage temperature to tune.

The baseline’s fragility is therefore exactly the phenomenon we study. We hope this clears up the misunderstanding and helps you assess the paper based on its real contribution.

Reviewer_ChvgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies ratio-free off-policy objectives for reinforcement learning of LLM reasoning. For a fixed behavior policy, it rewrites a squared sentence-level log-ratio regression objective and characterizes an interior stationary point through a Lambert-W density ratio. The paper relates the sign of a normalization multiplier to the exponential advantage mass, argues that a nominal finite-group OAPL advantage falls into a negative-multiplier regime, and proposes the shifted-mean advantage r-r_bar+beta to guarantee a positive multiplier at the population level.

Critical strengths
  • The observation that an advantage baseline changes the population target of this off-policy regression objective, rather than merely reducing variance, is conceptually important. This is a useful lens for understanding why seemingly minor advantage-design choices can materially affect asynchronous reasoning training.
  • The positive-multiplier stationary condition and the shifted-mean construction are simple and potentially useful. In particular, the finite-group population analysis in the appendix goes beyond treating a sampled group advantage as an unbiased pointwise quantity.
Critical weaknesses
  • The theoretical characterization is incomplete in the regime used to explain failure. Proposition 3.1 covers an interior stationary point with a positive multiplier, not a unique global optimizer. For a negative multiplier, both Lambert-W branches can occur and global optima may mix branches or be non-unique. The paper nevertheless describes a principal-branch target policy. In addition, its comparison to exp(A/beta) uses an unnormalized ratio and therefore does not establish an ordering between valid policies.
  • The claimed scope and empirical controls are insufficient. The derivation assumes that the sampling distribution and log-ratio denominator are the same behavior policy, which is not the population objective used by A*-PO with current-policy samples and a fixed reference. Practical OAPL, A*-PO, and PPO are absent; GRPO is evaluated at a different lag; policy lag is measured in optimizer steps rather than actual policy divergence; and three-seed curves are not accompanied by numerical confidence intervals or per-benchmark results.
Questions for authors

see above

1 comment

Thank you for highlighting that advantage normalization changes the population target, beyond its usual role in reducing variance, and for recognizing the value of our finite-group analysis. These are exactly the insights we want to convey. OAPL already shows that these objectives work at scale; our goal is to explain why their practical changes help and show how to reach the desired regime directly.

Stationarity and Lambert-W branches. The positive-multiplier result can actually be strengthened beyond stationarity. Writing s=log⁡ρs = \log \rho, the Lagrangian is, up to constants,

L(s,τs)=Eπold[As−β2s2−βτses]+βτs.\mathcal{L}(s,\tau_s) = \mathbb{E}_{\pi_{\mathrm{old}}}\left[As - \frac{\beta}{2}s^2 - \beta\tau_s e^s\right] + \beta\tau_s.

For τs>0\tau_s > 0, this is strictly concave in ss. A normalized stationary solution therefore uniquely maximizes the Lagrangian. Since the constraint term vanishes for every normalized policy, this solution is also the unique global optimizer of the original objective.

The negative-multiplier case does require distinguishing between the two Lambert-W branches, and we will clarify the principal-branch discussion accordingly. This does not affect the finite-group argument establishing Zexp⁡<1Z_{\exp} < 1, or our proposed construction, which guarantees τs≥1\tau_s \ge 1 and avoids this branch ambiguity.

Comparison with the normalized exponential target. We can express the comparison directly through relative odds. For Ai>AjA_i > A_j and τs>0\tau_s > 0, Equation (5) gives

0<log⁡ρiρj=Ai−Ajβ−τs(ρi−ρj)<Ai−Ajβ.0 < \log\frac{\rho_i}{\rho_j} = \frac{A_i-A_j}{\beta} - \tau_s(\rho_i-\rho_j) < \frac{A_i-A_j}{\beta}.

For the normalized exponential target, the corresponding log-ratio difference is exactly (Ai−Aj)/β(A_i-A_j)/\beta, since the normalizer cancels. Thus, the Lambert target reduces the relative reweighting toward higher-advantage responses compared with a valid normalized exponential policy. This is the precise sense in which we mean that the target is more conservative.

Sampling and reference policies. Our characterization assumes that the sampling distribution and the log-ratio denominator are the same behavior policy, as specified in Equation (3). We will make this assumption explicit in the connection to A*-PO. Variants using current-policy samples and a distinct fixed reference are outside this characterization.

Comparison with practical OAPL. The nominal-OAPL comparison deliberately keeps the regression loss and lag fixed and changes only the advantage, isolating the mechanism identified by our theory.

We have now also tuned OAPL’s advantage temperature β2\beta_2, keeping β1=10−3\beta_1 = 10^{-3}; see the table in our response to Reviewer KXAP. Its final rewards are very close to Lambert, with differences within the observed noise. This agrees with Appendix B.1: appropriate temperature decoupling moves OAPL into the pessimistic regime. Our contribution explains this behavior and obtains the desired regime directly, without an extra advantage temperature to tune.

Experimental controls and reporting. GRPO provides a near-on-policy reference, while the matched-lag comparison between nominal OAPL and Lambert tests the effect of advantage normalization. Lag measures operational rollout staleness; we do not assume that equal lag means equal policy divergence.

Figure 2 already reports held-out accuracy on the four-benchmark mixture, with means and standard errors over three independent seeds. Numerical endpoint and per-benchmark tables would make these results easier to inspect. The purpose of these experiments is to test the theoretical predictions and their effect on task performance, building on the evidence of effectiveness at scale already provided by OAPL.


We hope this addresses your concerns and clarifies how these points preserve our main theoretical results and proposed construction. If you have no further concerns, we would appreciate it if you reconsidered your score in light of these clarifications and the additional comparison with tuned OAPL.

Reviewer_KXAPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies ratio-free off-policy reinforcement learning for language-model reasoning. It argues that objectives such as A*-PO and OAPL induce a Lambert-W-tempered target policy rather than the nominal exponentially tilted policy. Based on this analysis, the paper proposes a shifted-mean advantage, A^i=ri−rˉ+β\hat A_i=r_i-\bar r+\beta, which is intended to place the induced target in a conservative, pessimistic regime. Experiments on Qwen3 models suggest improved stability under weak regularization and stale rollouts, with higher reward and better entropy preservation than nominal OAPL and near-on-policy GRPO.

Critical strengths
  1. The paper addresses an important practical problem in large-scale LLM reinforcement learning, where stale rollouts and asynchronous generation create substantial policy mismatch.
  2. The Lambert-W characterization provides an interesting perspective on ratio-free off-policy objectives.
  3. The paper clearly highlights that advantage normalization is part of the effective optimization target rather than merely a variance-reduction choice. 4.The proposed shifted-mean advantage is simple to implement and does not introduce an additional temperature-decoupling hyperparameter. 5.The manuscript is generally well written, and the main derivations are relatively easy to follow.
Critical weaknesses
  1. The theoretical characterization is based on an idealized unrestricted policy space. The derivation treats π(⋅∣x)\pi(\cdot\mid x) as an arbitrary distribution over complete responses with full support. In practice, the method optimizes a finite-parameter autoregressive language model using sampled sequences. 2.Proposition 4.1 contains a formal inconsistency. The proposition states “for any α≥0\alpha\ge 0,” but α\alpha does not appear in the condition or conclusion. 3.The connection between finite-group advantages and the population target needs more justification. The paper defines the population advantage by conditioning the finite-group estimator on one sampled response. This is a reasonable construction, but the resulting objective is not identical to optimizing the empirical group loss on each minibatch.
  2. The baseline comparison is incomplete and potentially unfair. The paper argues that practical OAPL stabilizes training by using a larger temperature for advantage computation than for regularization. However, the main experiments compare the proposed method against nominal OAPL using the same temperature for both quantities. 5.The empirical evidence is relatively limited for the strength of the claims. The main experiments use Qwen3 models on one training dataset and a small held-out benchmark mixture. The results are shown mainly as curves rather than quantitative final-score tables.
Questions for authors

1.How does the proposed method compare with properly tuned OAPL using separate advantage and regularization temperatures? 2.Does the theoretical target remain valid when the policy is an autoregressive neural model rather than an unrestricted response distribution? 3. How sensitive are the results to group size, reward scaling, response length, and the fraction of stale rollouts? 4. Can the authors provide quantitative final results and confidence intervals for every lag and regularization setting?

1 comment

Thank you for highlighting the Lambert-W analysis, the role of advantage normalization, and the simplicity of our change. These are exactly the points we want to convey. OAPL [1] already shows that this approach works at scale. Our goal is to explain why its practical changes help and show how to reach the desired regime directly.

Comparison with tuned OAPL. We compare with nominal OAPL to keep the regression loss fixed and isolate the effect of advantage normalization. Appendix B.1 explains how appropriately separating the temperatures moves OAPL into the positive-multiplier regime.

To check this prediction, we also tuned OAPL’s advantage temperature β2\beta_2, keeping the regularization coefficient fixed at β1=10−3\beta_1 = 10^{-3}. The final rewards were very close to Lambert, with differences within the observed noise. This agrees with our theory: temperature tuning can recover the pessimistic regime. Our shifted advantage guarantees τs≥1\tau_s \ge 1 directly and achieves comparable performance without an extra advantage temperature to tune.

MethodSeparate advantage-temperature tuningQwen3-4B rewardQwen3-8B reward
Nominal OAPLNo: β2=β1\beta_2 = \beta_1$ 16\%$35.935.9\\%
Tuned OAPLYes$ 46.6\%$$ 46.5\%$
LambertNot needed$ 46.5\%$$ 46.3\%$

Reward is held-out accuracy.

Population target and neural models. Our analysis describes the ideal target of the loss. An autoregressive model defines a distribution over responses, so this target is still relevant; model capacity and training determine how closely it can be reached. We do not prove that a finite model reaches it exactly. The point is to understand how advantage normalization changes the target and predict the resulting behavior. Our experiments confirm the predicted patterns with models of finite capacity and practical training.

Finite-group justification. The connection is exact after averaging over sampled groups. Replacing the sampled advantage by its conditional mean changes the expected regression loss only by a term that does not depend on the model parameters. We are therefore analyzing the same expected loss used in training. Appendix A.2 derives

AG(x,y)=G−1G(r(x,y)−Vold(x))+β,A_G(x,y) = \frac{G-1}{G}\bigl(r(x,y) - V_{\mathrm{old}}(x)\bigr) + \beta,

which preserves τs≥1\tau_s \ge 1 for every G≥2G \ge 2.

Experiments and sensitivity. Figure 1 checks the predicted regimes for G=2,4G = 2,4. Figure 2 reports held-out accuracy and standard errors over three independent seeds, across two model sizes, three regularization strengths, and three off-policy lags. These results support our predictions and show that this small change can improve both accuracy and stability.

Scaling rewards and β\beta by the same factor leaves the theoretical target unchanged. We do not divide by the group standard deviation, since this would also change the reward scale and make the effect of our baseline harder to isolate. All experiments use the same response-length limit of T=4096T = 4096 tokens to keep the comparison controlled and the experiments feasible with our available compute resources.

Proposition 4.1. Thank you for catching the typo. The unused α\alpha was left in by mistake. Removing it does not change the statement or proof.

[1] LLMs Can Learn to Reason Via Off-Policy RL. Ritter et al.


We hope this addresses your concerns and shows that these points mainly needed clarification, while our main theoretical results remain unchanged. If you have no further concerns, we would appreciate it if you reconsidered your score.

Comments on the paper

1 comment

We thank the reviewers for their time and constructive feedback. The discussion has helped us clarify our contribution, strengthen the theoretical characterization, and add an important experimental comparison. We summarize these points below for ease of reference.

Our main contribution is to explain how advantage normalization changes the population target of ratio-free off-policy learning, beyond its usual role in reducing variance. We characterize this target through the Lambert-W function, identify a conservative positive-multiplier regime, and propose the simple shift A^i=ri−rˉ+β\hat A_i=r_i-\bar r+\beta to guarantee this regime directly. Our goal is to explain the practical success of existing off-policy objectives and obtain their stabilizing behavior without an additional advantage temperature to tune.

1. Experimental coverage and task performance.

Our experiments evaluate Qwen3-4B and Qwen3-8B across three regularization strengths and three policy lags, with means and standard errors over three independent seeds. As defined in Section 5, “evaluation reward” is held-out accuracy on Minerva, AIME2025, AMC23, and BRUMO. Figure 2 therefore reports task performance alongside entropy, showing that the proposed shift can improve both accuracy and stability. These controlled experiments test the theoretical predictions in practical training, building on OAPL’s existing evidence of effectiveness at scale.

2. Comparison with tuned OAPL.

Following the reviewers’ suggestions, we additionally tuned OAPL’s advantage temperature while fixing the regularization coefficient at β1=10−3\beta_1=10^{-3}. Tuned OAPL reaches 46.6% and 46.5% held-out accuracy on Qwen3-4B and Qwen3-8B, respectively, compared with 46.5% and 46.3% for Lambert; nominal OAPL reaches 16.0% and 35.9%. This supports the explanation in Appendix B.1: temperature decoupling can recover the desired pessimistic regime. Our shift reaches this regime directly and achieves comparable performance without the extra temperature tuning.

3. Global optimality and the meaning of pessimism.

The rebuttal strengthens the positive-multiplier characterization in Proposition 3.1: strict concavity of the Lagrangian implies that a normalized stationary solution is the unique global optimizer. We also express conservatism through relative odds between responses, giving a comparison with a valid normalized exponential target. We acknowledge the branch ambiguity for negative multipliers; the proposed shift guarantees τs≥1\tau_s\geq1 and avoids this ambiguity.

4. Finite-group justification and practical scope.

Appendix A.2 connects the finite-group training loss to the population analysis exactly after averaging over sampled groups: replacing the sampled advantage by its conditional mean changes the expected regression loss only by a parameter-independent constant. The positive-multiplier guarantee is preserved for every group size G≥2G\geq2. We also clarified that the analysis characterizes the objective’s population target, while neural capacity and optimization determine how closely training approaches it, and that the characterization assumes the same behavior policy for sampling and the log-ratio denominator.


Our intended contribution is theoretical understanding supported by empirical evidence: explaining why practical off-policy objectives stabilize learning and providing a simple way to obtain the desired regime. We hope the additional comparison and clarifications provide helpful context for the final assessment. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.