acceptodds
Under review as a conference paper at ICLR 2027

LOCAL: Local-Curvature Advantage Logit Regression for Scalable Policy Optimization

Abstract

Policy optimization should account for how parameter updates change the policy distribution rather than their magnitude in parameter space. Natural policy gradient (NPG) and policy mirror descent (PMD) achieve this through parameter-space and state-wise curvature, respectively, and yield the same Boltzmann policy update for tabular softmax policies. This update determines the target logits only up to a state-dependent offset (SDO), since adding a common constant to all logits leaves the policy unchanged. While therefore negligible in the tabular setting, we show that the SDO becomes consequential under function approximation: fixing it during target fitting alters the curvature beyond the local Fisher geometry. We introduce LOCAL (**Lo**cal-**C**urvature **A**dvantage **L**ogit Regression), which analytically profiles out the SDO and regresses only the policy-relevant relative logit changes. We show that the resulting objective preserves the local Fisher curvature in logit space and that its local linearization recovers NPG. We further derive a first-order implementation and a Top- approximation that makes the method practical for very large action spaces. Across Atari and Procgen controls and LLM post-training with RLHF and RLVR, LOCAL consistently improves policy optimization, showing that properly handling this seemingly redundant degree of freedom is important for policy learning with function approximation.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, Policy Mirror Descent, Natural Policy Gradient, LLM post-training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1x1YOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes LOCAL, a policy optimization algorithm designed to ensure that policy updates are evaluated according to their effect on the policy distribution rather than based on their magnitude in parameter space. It extends insights from NPG and PDM to the function approximation setting. Specifically, it shows that adding a state-dependent offset to all action logits leaves the policy distribution unchanged, but fixing this offset (e.g. to zero) during policy optimization can introduce constraints. LOCAL addresses this by analytically computing and subtracting the optimal state-dependent offset, ensuring that the regression objective focuses on logit changes that actually change the policy distribution and preserves the local Fisher geometry. The paper provides a theoretical result showing that the tightest upper-bound on the regret is achieved by the optimal offset. The paper provides results on various RL pixel environments and environments with LLMs.

Critical strengths
  1. Critical failure mode in actor-based methods. I believe the key insight about the optimal offset is a useful notion, and something I was not explicitly made aware of before. The intuition behind the issue (with Lemma 1) makes this clear. Surprisingly, this offset seems to hurt performance substantially more than I would have thought, especially in the environments in Table 1. I think the knowledge of this failure mode is useful for the community.
Critical weaknesses
  1. Weak LLM results. While I appreciate the effort to extend LOCAL to more topical settings like LLMs, the results are weak, particularly because of lack of statistical rigor. I am less concerned with how much better LOCAL is vs. the baseline, and I am more concerned with how confidently I should be reading these results. I understand that LLMs are expensive to run, but given the high variance observed in pixel based RL (Table 1), it is only natural to think that the LLM-based results could also have high variance and may be unreliable. More seeds with 95% confidence intervals would make me more confident in these results. I refer the authors to: Empirical Design in Reinforcement Learning (Patterson et al. 2024).
Questions for authors
  1. How should I interpret Theorem 1? My understanding is that it is making a general comment on how the tightest upper bound on the regret is related to the optimal offset, but it is not clear to me if whether this is an indication on how LOCAL's regret will be? Nor is it clear it me how it connects to the more typical setting of using an offset of zero i.e. it is not comparing the performance of the policy learned with zero offset vs. LOCAL.
  2. What is the broader implication of the KL loss and entropy results? Naturally, there are "bad" values (too high or too low), but there is no good value. The results are presented as if LOCAL achieves good values but this is not entirely clear me. Additionally, what are the broader implications of these KL loss/entropy results?

If the authors can address my weakness and two questions, I would more inclined towards raising my score. Question 1 will help me also gauge the importance of the theoretical result, especially given this is more of a theoretical paper (I presume). If not a theoretical paper, and it is an empirical paper, then naturally, the LLM experiments definitely need to be more rigorous.

5 comments

We thank the reviewer for the thoughtful and constructive feedback and for the pointer to Patterson et al. (2024), which shaped how we now report the LLM results. Your question on Theorem 1 identified a gap between what the theorem shows and what the text suggested, and it led to a new corollary.

W1. We have rerun every RLVR comparison with 3 training seeds. Table 2 reports the mean and the sample standard deviation, and we report a 95% confidence interval for each difference rather than a hypothesis test (Welch intervals; Appendix C.9.6, Table 23).

Only one of the eight intervals excludes zero, and the pooled difference is +0.47 points (95% CI [-0.14, +1.06]). The largest single-seed gain in the submission (Llama-3.2-3B with PPO advantages, +3.97) becomes +0.20 over 3 seeds (95% CI [-4.75, +5.16]). We therefore no longer claim an improvement in the LLM experiments: the revision states that LOCAL performs comparably to PPO/GRPO on the math benchmarks.

Q1. The question points to a distinction the paper did not state clearly. First, we clarify that Theorem 1 is an upper bound on LOCAL's own regret. However, its comparison with other offset rules is not between the policy sequences that each rule would train. It compares bounds on the same policy sequence, where measuring the fitting error with LOCAL's loss gives the tightest regret bound: εT≤εc,T\varepsilon_T \le \varepsilon_{c,T} for any offsets cc.

To compare the policy sequences trained with different offset rules, we added a new Corollary (Corollary 1 in the revision). Here εˉ(c)\bar\varepsilon(c) is the worst-case fitting error of the update selected by rule cc measured with LOCAL's loss, and εˉc(c)\bar\varepsilon_c(c) is the error of the same update measured with rule cc's own loss. For the sequence trained with any offset rule cc and the one trained with LOCAL, under the same assumption on both sequences, LOCAL has the smaller fitting-error term in the regret bound, εˉ(c⋆)≤εˉc(c)\bar\varepsilon(c^\star)\le\bar\varepsilon_c(c). In addition, since Theorem 1 shows that LOCAL's loss gives the tightest bound on a fixed sequence, we also bound the regret of the sequence trained with rule cc using LOCAL's loss, which gives εˉ(c)\bar\varepsilon(c). LOCAL's sequence still has the tightest bound: εˉ(c⋆)≤εˉ(c)≤εˉc(c)\bar\varepsilon(c^\star)\le\bar\varepsilon(c)\le\bar\varepsilon_c(c).

Beyond the bounds, the same ordering appears empirically: in Example 1 the Fisher residual is 3/803/80 under LOCAL and 3511/100003511/10000 under zero offset, and in the SDO ablations zero offset and uniform centering become unstable or collapse while the optimal SDO trains stably (Figures 2, 7, 8, 10).

Q2. We agree that neither quantity has a "good" value, and the submitted text should not have suggested otherwise. The revised paragraph is purely descriptive: it no longer links lower KL loss to higher performance and states that neither quantity has a preferred value. We report the two diagnostics because recent work on LLM post-training relates entropy collapse to premature convergence and saturating performance (ASPO, Wang et al., 2025, arXiv:2510.06062; Cui et al., 2025, arXiv:2505.22617). The implication we can support is limited to one point: at comparable math accuracy (Table 2), LOCAL ends training closer to the initial policy (lower KL loss) and with higher entropy than PPO/GRPO in most comparisons, so it does not reproduce their update but reaches similar accuracy through a different policy. One possible consequence is suggested by prior work, which reports that entropy collapse in RLVR reduces generation diversity and limits Pass@kk, whereas maintaining entropy sustains it (Liang et al., 2025, arXiv:2508.14029). We did not evaluate Pass@kk, so whether LOCAL's higher entropy brings this benefit is not tested here and is left to future work.

Thanks to the authors for incorporating the feedback. The clarifications on the empirical results, LLM experiments, theory, and writing are helpful. Its unfortunate that the LLM claim does not hold, but I appreciate that it is correct now and that the claims are dialed down. It is OK to be on par with PPO/GRPO. The visual RL experiments still suggest that this algorithm can be practically useful.

I have raised my score in light of the updates. That said, I am curious if there are any other benefits or not in the LLM experiments that are unrelated to performance (your response seems to potentially indicate something related to clipping, compute time etc)?

Thank you for the careful reading and for raising the score.

On clipping, the role that ratio clipping plays in PPO/GRPO is played in LOCAL by the KL term of the mirror-descent update in Definition 1: η\eta sets the strength of this penalty (1/η1/\eta), and the regression target is the exact solution of that KL-regularized problem. So LOCAL needs neither ratio clipping nor an additional KL regularizer, and in fact its loss does not contain a probability ratio to clip.

Compute time is also related to an advantage. LOCAL fits by regression an update whose local linearization is the natural policy gradient, which is a second-order update, yet it only costs as much as ordinary first-order optimization. The wall-clock time is only 4.5-8.3% above GRPO in our LLM experiments. As we noted in our response to Reviewer 4oNu, the current implementation is quite direct, so we expect the achievable overhead to be smaller still.

Beyond these two points, we think one future direction is to incorporate the KL to the reference model, which LLM post-training often cares about more than the KL to the previous policy. Such a term can be added to the update in Definition 1, and the logit target then remains closed-form and is still defined only up to an SDO, so the same construction applies. Another is off-policy training. Since the target and the optimal SDO are computed from the current policy πk\pi_k, samples from a behavior policy μ\mu only need the weight πk(a∣s)/μ(a∣s)\pi_k(a|s)/\mu(a|s). This weight does not involve πθ\pi_\theta and therefore stays fixed during the update. That said, we have not tested either direction, so we leave both to future work.

Reviewer_7bXKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the observation that, when policy mirror descent (PMD) for softmax policies is realized under function approximation, the target logits are determined only up to a state-dependent offset (SDO). Although this degree of freedom does not affect the policy distribution, it does affect the optimization when the target is fitted by neural network regression. The proposed method, LOCAL, analytically profiles out the old-policy-weighted mean of the logit displacement and regresses the relative logit change onto the scaled advantage.

Theoretically, the paper shows that the resulting regression loss corresponds to the state-wise Fisher seminorm, that its local linearization recovers NPG, and that the average regret is upper-bounded in terms of the realized fitting error. To handle large action spaces, the paper further introduces a Top-K approximation. Experimentally, LOCAL is compared with existing methods on Atari, Procgen, and LLM post-training with RLHF and RLVR.

Critical strengths
  • Simple method with a clear theoretical motivation. Focusing on the offset of the target logits, a degree of freedom that is usually ignored, is an interesting perspective. The proposed loss is simple and can be implemented with standard gradient-based optimization, without computing or inverting a Fisher matrix. In particular, Proposition 2 shows that the regression loss with the optimal SDO coincides with the state-wise Fisher seminorm, and together with the connection to NPG, the theoretical picture is clearly organized.
Critical weaknesses
  • The relationship between the Fisher geometry and the actual performance gains is not sufficiently clear. The paper theoretically shows that the LOCAL objective corresponds to the local Fisher geometry. However, it remains unclear to what extent this property actually contributes to the performance improvements observed in practice. In particular, in the LLM experiments, the number of updates per rollout is small, so the LOCAL update may behave similarly to a standard policy gradient update. To clarify this point, it would be helpful to report the evolution of the actual fitting error during training, as well as a comparison with PPO/GRPO without clipping. Such analyses would allow a clearer assessment of the effectiveness of the proposed method.

  • Fairness of the experimental comparison and robustness of the LLM results. Although the LLM experiments cover multiple models and settings, each comparison is based on a single seed, and many of the RLVR improvements in Math Avg are small. Therefore, it remains uncertain how consistently the improvements can be obtained. I also have a concern regarding the interpretation of General Avg in Table 2. For example, for Llama-3.2-3B with GRPO, the GPQA score is 11.08, which is far below the chance level of a four-choice task and also lower than the base model (17.38). Similarly, for Qwen2.5-3B with GRPO, the HotpotQA EM drops to 12.96 from 27.36 for the base model. In these settings, the improvement of LOCAL on these tasks accounts for a large portion of the difference in General Avg. From the current description, it is difficult to determine whether these low scores reflect an actual degradation in capability or are affected by the answer format or answer extraction. In addition, since General Avg aggregates HotpotQA EM and F1 as separate entries, a single task can have a disproportionate influence on the average. Clarification of the evaluation protocol and a more robust aggregation would make it easier to judge the consistency of the performance improvements.

Questions for authors
  • On the actual LOCAL updates. During LLM training, can the authors show the LOCAL regression loss, or how closely the centered logit displacement approaches the target ηA^\eta \hat{A}? In addition, how does LOCAL compare with PPO/GRPO without clipping under the same training conditions (e.g., the same learning rate and number of epochs)? I believe this comparison is important for understanding whether the performance gains of LOCAL stem from the proposed regression objective.

  • On the robustness of the LLM results. Can the authors report results over multiple seeds for representative RLVR settings? Please also describe the answer extraction procedure for GPQA and HotpotQA in Table 2 and how responses that do not match the expected format are handled. In particular, I would like to confirm whether formatting issues or extraction failures affect the results for the baselines whose scores drop substantially. In addition, could the authors report a General Avg in which HotpotQA EM and F1 are not weighted as separate entries?

4 comments

We thank the reviewer for the constructive and specific suggestions. Your observation that some General Avg. scores fall below chance led us to find an answer-extraction problem in our evaluation, and the two analyses you proposed, the regression loss and the comparison without clipping, are now part of the revision.

relationship between the Fisher geometry and the actual performance gains

A small number of updates per rollout does not make LOCAL behave like a standard policy gradient. LOCAL coincides with the policy gradient only at the first optimizer step of each rollout. At every later step, its gradient contains an additional curvature term that depends on the logit displacement from the old policy. It would reduce to the policy gradient only if each rollout were used for a single step, whereas each rollout in our runs gives 4 optimizer steps. In short, LOCAL is not a policy gradient method: it regresses the logits toward the mirror-descent target, and the policy gradient is only the direction of its first step.

  1. On the Regression loss. We now report LOCAL's regression loss during RLVR training for all eight settings (Appendix C.9.3, Figure 14). In LLM training, each minibatch is used for a single optimizer step, so the regression target ηA^\eta \hat{A} changes at every step. The loss therefore measures the residual against a new target each time rather than progress toward a fixed one. It decreases over training in most settings, but mainly because the advantage signal becomes smaller, so on its own it cannot show whether the logits approach the target.

  2. PPO/GRPO without clipping. In the revision, we added a comparison with PPO and GRPO without ratio clipping on Llama-3.2-3B, using the same learning rate, number of epochs, and seeds (Appendix C.9.5). LOCAL has a higher mean Math Avg. than the unclipped objective with both advantage types (58.20 vs. 55.13 with GRPO advantages and 54.51 vs. 51.52 with PPO advantages) and scores higher in 17 of the 18 pairs of runs. The training dynamics differ as well (Figure 15).

Robustness of the LLM results.

All RLVR comparisons now have 3 seeds. We report a Welch 95% CI for each of the eight differences (Appendix C.9.6, Table 23) and a stratified-bootstrap CI for the pooled difference. Pooled, LOCAL's Math Avg. is 0.47 points higher (95% CI [-0.14, +1.06]), and only one of the eight per-setting intervals excludes zero. We now state that LOCAL performs comparably to PPO/GRPO on the math benchmarks and no longer claim a consistent improvement.

The low general-task scores were extraction failures. The original scorer accepts an answer only in the required format (a boxed answer, an <answer> tag, or a "final answer"-type statement) and scores every other response as incorrect. For the two cases you flagged, it found no answer in 48.4% (GPQA, Llama-3.2-3B, GRPO) and 60.1% (HotpotQA, Qwen2.5-3B, GRPO) of the responses. We re-scored all saved generations with a permissive scorer that also extracts answers in other formats, and General Avg. now counts HotpotQA once. The largest General Avg. gap shrinks from 7.34 to 1.16 points, all eight differences lie between -0.11 and +1.16, and none of their Welch 95% intervals excludes zero (Table 2; Tables 11, 12, 20).

Format compliance does differ between the methods, and the revision now reports this explicitly (Table 12). On HotpotQA with the 3B models, PPO/GRPO-trained models answer outside the required format more often than LOCAL-trained ones in three of four comparisons (no answer found in 72.2% vs. 55.3%, 62.2% vs. 45.8%, and 60.1% vs. 25.3% of responses). Elsewhere the two methods are within 3.5 points. This is a difference in output format, not in correctness: once those answers are extracted, the HotpotQA EM gap in these settings shrinks from 7.8-18.4 to 1.8-2.7 points.

We therefore no longer claim that LOCAL preserves general capabilities better than PPO or GRPO. We claim only that it does not degrade them and it better retains the required output format in three of the four 3B comparisons.

Thank you for the response and the additional experiments, which largely address the concerns in my review. I now understand that LOCAL improves on visual control and is on par with PPO/GRPO in LLM post-training, so a practical advantage for LLMs is not yet established. That makes the novelty of the method the deciding point for me, and one thing is unclear. Section 5 and Appendix C.7 distinguish LOCAL from REBEL and PMD-mean, but these methods regress log-probability ratios, which do not depend on the logit offset, so the SDO ablations do not compare LOCAL with them. When do you expect LOCAL's differences from them to matter for training performance?

Thank you for engaging with our rebuttal. We really appreciate your thoughtful review and constructive suggestions.

Novelty

To the best of our knowledge, LOCAL is the first work that shows the seemingly redundant SDO is important for policy learning in ResNet and transformer architectures. Our experiments on visual control and LLM post-training show that the effect of SDO is consistent across different architectures and model sizes. REBEL and PMD-mean each fix one way of handling this offset, LOCAL is the first to optimize it in closed form and to analyze its effect on the curvature.

Distinction from PMD-mean and REBEL

We think there might be a potential misunderstanding here. For a softmax policy, the log-probability ratio is itself a logit displacement with a particular offset, the log-partition difference cLR(s)=log⁡Eb∼πk(⋅∣s)[exp⁡(zθs(b)−zks(b))]c _ {\mathrm{LR}}(s) = \log \mathbb{E} _ {b\sim\pi_k(\cdot|s)}[\exp(z _ \theta^s(b) - z _ k^s(b))] (new Appendix A.7, Eq. 66). So regressing log-ratios does not avoid the SDO, it is one particular way of handling it. We clarify this for the two methods in turn.

  • PMD-mean adds extra terms to the curvature. From this perspective, PMD-mean uses exactly cLRc _ {\mathrm{LR}}, whereas LOCAL uses the mean c⋆(s)c^\star(s) of the displacement, and the two differ by DKL(πks∥πθs)D _ {\mathrm{KL}}(\pi_k^s \| \pi_\theta^s). In the comparison of Proposition 2, this adds to the Fisher curvature FsF_s a rank-one term along the policy change and a Fisher term of the new policy weighted by the KL divergence, whereas the curvature of LOCAL stays at FsF_s throughout the update (new Appendix A.7).

    A Two-state MDP toy example. We added a toy comparison on the two-state MDP of Example 2 between the LOCAL loss and the log-ratio loss (new Appendix A.8). The toy example shows that LOCAL reaches the optimal return in fewer updates than the log-ratio loss. Specifically, with exact updates we test several step sizes η\eta: the log-ratio loss improves the return more slowly than LOCAL, and the gap between the two grows with η\eta. When each update is instead fitted by gradient descent on the loss, we compare different numbers of fitting steps: once each update is fitted with enough steps, LOCAL again reaches the optimal return in fewer updates.

  • REBEL estimates the curvature with higher variance. REBEL takes pairwise differences between two sampled responses, which removes the offset at the sequence level by sampling. Its objective equals LOCAL's in expectation (Section 5), but the offset is in effect estimated from a single other sample, so it has higher variance, which is what Appendix C.7 measures (about 2.1 times the curvature-estimation error at the same number of samples).

To your question of when these differences matter for training performance, we expect two cases. Compared with PMD-mean, they matter when the update is large, i.e., for large η\eta or low success rates, and when each batch is used for more optimizer steps, because the additional curvature above grows with the policy change and is negligible only for small updates (see the toy comparison above, new Appendix A.8). Compared with REBEL, they matter when few samples per state are available and when responses are long: sampled pairing is then noisy, since the variance of the sampled offset accumulates over the tokens, and it is not possible at all in step-level RL with one action per state, such as Atari and Procgen, which is where we observe the gains of LOCAL.

We added the above clarifications and discussions in the revised version. Thank you again. Looking forward to your reply.

Thank you for the reply and Appendices A.7–A.8. The additional analysis clarifies the theoretical distinction between LOCAL and log-ratio regression, and the two-state example is helpful. However, the practical significance of this distinction remains unclear, as the performance comparison between the two is limited to a toy MDP. I therefore maintain my rating of 2.

Reviewer_4oNuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper revisits regression-based policy mirror descent (PMD) with softmax policies. The statewise PMD target fixes the target policy, but its logits are only defined up to a state-dependent offset (SDO). The paper argues that fixing the offset, as zero-offset AMPO-style regression does, changes the fitting problem once logits come from a restricted function class. LOCAL minimizes the squared logit residual over the offset in closed form. The optimal offset is the old-policy mean of the logit displacement. The resulting per-sample actor loss is ½[Δz(a_t) − Σ_b π_old(b|s_t)Δz(b) − ηÂ_t]², which replaces PPO's clipped surrogate inside an otherwise unchanged actor–critic or GRPO pipeline.

Proposition 2 shows that the profiled loss is the statewise categorical Fisher seminorm of the residual. Zero offset and uniform centering add rank-one terms. After local linearization, the exact minimum-norm solution is the NPG direction. Theorem 1 bounds average regret by the realized fitting error. For the same realized updates, the optimal offset minimizes that error term. A Top-K approximation computes the offset from the K most probable old-policy tokens. The experiments cover 57 Atari games and 16 Procgen games with five seeds each, RLHF on hh-rlhf with four 3B–8B models, and RLVR on math with paired PPO and GRPO comparisons.

Critical strengths
  1. Well-controlled results. The comparisons use five seeds with shared architectures, preprocessing, and environment-step budgets. LOCAL uses fewer update epochs than PPO.
    • Atari IQM HNS is 151.39 ± 5.53, versus 105.12 ± 4.84 for the strongest baseline (MDPO) and 104.19 for PPO;
    • The advantage survives removing the two outlier games VideoPinball and Gopher: 137.15 versus 104.19 for PPO;
    • Procgen IQM rises from 26.00 to 42.48;
  2. Clear checkable account of what the offset changes. Proposition 2 makes the distinction precise. Gauge invariance alone is not the point, since uniform full-residual centering is also invariant. Old-policy centering is the choice whose metric is exactly the categorical Fisher. At matched action samples, paired estimation has roughly 2.1× the curvature NMSE of exact LOCAL, and Top-K is close to exact.
Critical weaknesses

The LLM evidence does not yet establish consistent improvement.

  • In Table 2 (p. 8), six of the eight paired math-average gains are 0.09–1.02 points. Only Llama-3.2-3B shows larger gains (+3.97 with PPO advantages, +1.97 with GRPO advantages).
  • The largest general-capability gain is Qwen2.5-3B with GRPO advantages (24.46 → 40.99). It is driven by the baseline's HotpotQA dropping below the base model (EM 12.96 versus 27.36). This looks like a baseline-run pathology rather than evidence that LOCAL preserves capabilities.
  • RLHF reports only the reward of the reward model being optimized (Figure 1, Figure 10). There is no held-out or independent preference evaluation, so reward over-optimization cannot be ruled out.
Questions for authors
  1. LLM robustness. How were checkpoints, η (1.0 versus 5.0 across advantage types; Table 10), and other hyperparameters selected for LOCAL and its baselines? Can you report additional training seeds or paired uncertainty for Table 2, especially the sub-1-point gains? Can you explain the Qwen2.5-3B GRPO HotpotQA drop? For RLHF, is there an evaluation independent of the training reward model, such as held-out win rates against PPO from a separate judge?

  2. practical cost and step-size sensitivity. What are the end-to-end wall-clock time, throughput, and tuning budgets of the matched runs? Figure 3 reports only extra memory. Because LOCAL has no ratio clipping, η acts as the trust-region size. The existing η sweeps each use one model: Qwen2.5-3B for RLVR (Figure 2) and Llama-3.2-3B for RLHF (Figure 9). The Atari/Procgen main results fix η = 1. How sensitive are the control tasks and the other LLMs to η?

3 comments

We thank the reviewer for the careful reading and the precise summary of the paper. Your questions on additional seeds, an independent RLHF evaluation, computational cost, and step-size sensitivity each led to a new experiment in the revision, and we address them in turn below.

LLM robustness. All RLVR comparisons now have 3 seeds. We report a Welch 95% CI for each of the eight differences (Appendix C.9.6, Table 23) and a stratified-bootstrap CI for the pooled difference. Pooled, LOCAL's Math Avg. is 0.47 points higher (95% CI [-0.14, +1.06]), and only one of the eight per-setting intervals excludes zero. We now state that LOCAL performs comparably to PPO/GRPO on the math benchmarks and no longer claim a consistent improvement.

We also report a new RLHF evaluation with two reward models that are not used in training, GRM-Llama3.2-3B and FsfairX-LLaMA3-RM-v0.1 (Appendix C.8.1). The prompts are a fixed subset of the training prompts. LOCAL's advantage under the training reward model carries over to GRM-3B on most models (win rates 51--65%), while under FsfairX-8B the differences are small and mixed in sign (win rates 42%-55%). GRM-3B is closely related to the training reward model (same Llama-3.2-3B backbone, fine-tuned on the preference data that seeds its training set), so FsfairX-8B is the more independent judge. LOCAL does not fall substantially below PPO under any reward model (by at most 0.12 points), so we see no marked over-optimization.

The Qwen2.5-3B GRPO HotpotQA drop. It is an answer-extraction effect, not a loss of capability: the original scorer accepts an answer only in the required format, and the gap largely disappears once answers in other formats are extracted (Table 11, 20). However, it is not a single-run pathology. Over three seeds the GRPO-trained model leaves the required format more often than the base model, while the LOCAL-trained model does not (Table 12). We therefore no longer claim that LOCAL preserves general capabilities better than PPO or GRPO. We claim only that LOCAL does not degrade general capabilities and better retains the output format in three of the four 3B comparisons.

Hyperparameter selection. Because of compute limits, all hyperparameters were selected on Llama-3.2-3B and used unchanged for the other models, separately for GRPO advantages, PPO advantages, and RLHF. The baselines keep the standard clip range 0.2 and choose the learning rate from {10−510^{-5}, 10−610^{-6}}. LOCAL uses the baseline's learning rate and searches only η\eta in {1,5,101, 5, 10}, so the tuning budget is two configurations for the baseline and three for LOCAL. This search selected η=5\eta = 5 with GRPO advantages and η=1\eta = 1 with PPO advantages and in RLHF. For RLVR, configurations are compared by validation accuracy on the fixed 10% subset of MATH-500 and for RLHF by the training reward. For every run we report the final checkpoint, with no checkpoint selection.

Sensitivity to η\eta. We swept η∈{0.1,1,10}\eta \in \{0.1, 1, 10\} on two Procgen and three Atari games (Appendix C.4, Figure 9). As you expect, η\eta acts as the step size: a larger η\eta gives a lower policy entropy on every game, and changing η\eta by a factor of ten in either direction lowers the return on most games. η=0.1\eta = 0.1 is below η=1\eta = 1 on four of the five games and higher on Seaquest. η=10\eta = 10 drives the entropy down early in training, matches η=1\eta = 1 on Coinrun, and has a lower mean return on the other four games. For the other LLMs, we do not have the compute to repeat the η\eta sweeps, so they remain limited to Qwen2.5-3B for RLVR and Llama-3.2-3B for RLHF.

Practical cost: We now report the measured cost of every RLVR run in Appendix B.3. Relative to the corresponding baseline, LOCAL increases the total wall-clock time by 1.4%–8.3% and the average per-step training time by 3.0%-9.6%, and reduces the training throughput by 2.0%-9.9%.

This overhead is already small, at most 8.3% in total wall-clock time, even though we did not optimize the implementation. In each update the current code still computes and copies the full-vocabulary logits before gathering the K+1K+1 logits that the loss uses. Gathering these logits directly, or evaluating the output layer only at these tokens, should reduce the overhead further, so the reported times are likely higher than what LOCAL can achieve.

Thank you for the additional experiments and for narrowing the LLM claims to reflect the new evidence. The multi-seed results and evaluation clarifications help address my concerns. I will maintain my rating as 3.

Thank you for the follow-up and for your careful review. We appreciate your time and the constructive discussion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair

We thank all reviewers. The reviews agree on the core of the paper and ask for stronger statistical evidence in the LLM experiments. We re-ran all RLVR comparisons with three seeds and added the analyses the reviews asked for, summarized below.

Main contribution. We show that the state-dependent offset (SDO), a degree of freedom that leaves the target policy unchanged, matters once logits come from a restricted function class, and we explain why.

  • Theory. Only the optimal SDO measures the fitting error in the state-wise Fisher seminorm; fixed offsets add rank-one terms (Proposition 2). Its local linearization recovers NPG, and it gives the smallest regret upper bound among SDO choices, both for realized updates (Theorem 1) and for training rules (new Corollary 1).

  • Visual Control. On 57 Atari and 16 Procgen games with five seeds, LOCAL raises IQM from 105.12 (strongest baseline) to 151.39 and from 26.00 to 42.48, using one update epoch per rollout where PPO uses four and three.

  • Ablation. When only the offset is changed, zero offset and uniform centering become unstable or collapse in Atari, RLVR and RLHF, while the optimal SDO trains stably (Figures 2, 7, 8, 10).

What the LLM experiments establish. Here LOCAL is a drop-in replacement of the actor loss inside existing PPO/GRPO pipelines, with rollouts, advantages, optimizer and learning rate unchanged.

  • Scale. With the Top-KK approximation, the update runs at vocabulary scale without ratio clipping, at 1.4%-8.3% more wall-clock time.

  • The SDO matters at this scale too. With the offset as the only change, zero offset and uniform centering collapse in RLVR at η∈{1,10}\eta \in \{1, 10\} and become unstable in RLHF at larger step sizes, while the optimal SDO trains stably (Figures 2, 10).

  • Parity with PPO/GRPO. Over three seeds and eight settings, the pooled Math Avg. difference is +0.47 points (95% CI [-0.14, +1.06]), so the interval rules out an average loss larger than 0.14 points. General Avg. differs by -0.11 to +1.16 points, with no interval excluding zero.

  • Comparison without clipping. Against PPO/GRPO without ratio clipping, at the same learning rate, epochs and seeds, LOCAL has a higher mean Math Avg. with both advantage types (58.20 vs. 55.13 and 54.51 vs. 51.52; Appendix C.9.5).

Our claim for LLM post-training is therefore parity as a drop-in replacement without clipping, not an improvement over PPO/GRPO. Designing a training procedure around LOCAL, rather than inserting it into pipelines built for clipped surrogates, is future work.

Revisions.

  • Table 2 reports three seeds, with a 95% confidence interval for each difference (Appendix C.9.6, Table 23). General Avg. counts HotpotQA once and uses the re-scored answers (Appendix B.3; Tables 11, 12, 20).

  • New Corollary 1 compares SDO training rules (Section 3.4, Appendix A.5).

  • New experiments: PPO/GRPO without ratio clipping (Appendix C.9.5), LOCAL's regression loss during RLVR training (Appendix C.9.3), RLHF evaluation with reward models not used in training (Appendix C.8.1), and sensitivity to η\eta on Atari and Procgen (Appendix C.4).

  • Hyperparameter selection and computational cost are reported in Appendix B.3 (Tables 13, 14).

  • Abstract, introduction, experiments and conclusion state the scope above, and the KL/entropy paragraph reports these diagnostics descriptively.

We are happy to provide further analyses during the discussion.

AuthorsOpenReview ↗

Summary of the Discussion and the Revision

We thank the reviewers for the discussion. As the discussion period closes, we summarize the reviewers' concerns, how we addressed them, and what changed in the revision, for the reference of the area chair.

Reviewer 4oNu

  • The LLM evidence did not establish a consistent improvement. We re-ran all RLVR comparisons with three seeds and report 95% confidence intervals (Table 2, Table 23). The pooled Math Avg. difference is +0.47 points (95% CI [-0.14, +1.06]), so we narrowed the claim to parity with PPO/GRPO.

  • The HotpotQA drop of one baseline looked like a run pathology. It was an answer-extraction effect. We re-scored all saved generations with a more permissive scorer, after which the gap largely disappears (Tables 11, 12, 20), and we removed the corresponding claim.

  • RLHF reported only the training reward model. We added an evaluation with two reward models not used in training (Appendix C.8.1, Table 19).

  • Hyperparameter selection, cost, and sensitivity to η\eta. We report the selection protocol and tuning budgets (Appendix B.3), the measured cost (1.4%–8.3% more wall-clock time; Tables 13, 14), and an η\eta sweep on five control games (Appendix C.4, Figure 12).

Reviewer 7bXK

  • Whether LOCAL behaves like a policy gradient when few updates are taken per rollout. We clarified that LOCAL coincides with the policy gradient only at the first optimizer step of each rollout, and added its regression loss during training (Appendix C.9.3, Figure 17) and a comparison with PPO/GRPO without ratio clipping (Appendix C.9.5, Tables 21, 22, Figure 18).

  • Robustness and evaluation of the LLM results. In addition to the three seeds and confidence intervals above, we traced the below-chance scores to answer-extraction failures, re-scored all runs, and now count HotpotQA once in General Avg.

  • Follow-up on novelty: REBEL and PMD-mean regress log-probability ratios, which do not depend on the logit offset. We clarified that, for a softmax policy, the log-ratio is itself a logit displacement with one particular offset, so these methods are two specific ways of handling the SDO. PMD-mean adds extra terms to the Fisher curvature (new Appendix A.7), and REBEL removes the offset by sampled pairs, which has higher estimation variance (Appendix C.7). We added a toy comparison (new Appendix A.8, Figures 4–6) and stated when these differences are expected to matter.

Reviewer 1x1Y

  • Statistical rigor of the LLM results. Addressed by the three seeds, standard deviations, and confidence intervals above, together with the narrowed claim.

  • Interpretation of Theorem 1. We clarified that Theorem 1 compares bounds for the same realized policy sequence, and added Corollary 1, which compares the sequences trained with different SDO rules (Section 3.4, Appendix A.5).

  • Implications of the KL and entropy results. The paragraph is now descriptive and states that neither quantity has a preferred value.

  • Follow-up on benefits other than performance in LLMs. LOCAL needs no ratio clipping, because the KL term of the mirror-descent update plays that role, and it obtains a NPG type update at first-order cost.

Changes in the revision

  • Claims. The abstract, introduction, experiments, and conclusion now state that LOCAL outperforms the baselines on Atari and Procgen and performs comparably to PPO/GRPO in LLM post-training.

  • Theory. New Corollary 1 (Section 3.4, Appendix A.5). New Appendix A.7, which shows that log-ratio regression is an SDO rule and relates LOCAL to PMD-mean and REBEL.

  • New experiments.

    • Three-seed RLVR results with confidence intervals (Table 2; Appendix C.9.6, Table 23).
    • Re-scored general-capability evaluation (Appendix B.3; Tables 11, 12, 20).
    • RLHF evaluation with external reward models (Appendix C.8.1, Table 19).
    • PPO/GRPO without ratio clipping (Appendix C.9.5).
    • Regression loss of LOCAL during training (Appendix C.9.3).
    • Sensitivity to η\eta on Atari and Procgen (Appendix C.4).
    • Toy comparison between the LOCAL loss and the log-ratio loss (Appendix A.8).
  • Reporting. Hyperparameter selection and computational cost (Appendix B.3; Tables 13, 14).

A note on numbering: Appendix A.8 adds three figures, so the appendix figure numbers are shifted by three relative to our earlier responses (for example, Figure 9 there is now Figure 12). Table numbers are unchanged.

We thank the reviewers again for their time and the area chair for handling our submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.