- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes LOCAL, a policy optimization algorithm designed to ensure that policy updates are evaluated according to their effect on the policy distribution rather than based on their magnitude in parameter space. It extends insights from NPG and PDM to the function approximation setting. Specifically, it shows that adding a state-dependent offset to all action logits leaves the policy distribution unchanged, but fixing this offset (e.g. to zero) during policy optimization can introduce constraints. LOCAL addresses this by analytically computing and subtracting the optimal state-dependent offset, ensuring that the regression objective focuses on logit changes that actually change the policy distribution and preserves the local Fisher geometry. The paper provides a theoretical result showing that the tightest upper-bound on the regret is achieved by the optimal offset. The paper provides results on various RL pixel environments and environments with LLMs.
Critical strengths
- Critical failure mode in actor-based methods. I believe the key insight about the optimal offset is a useful notion, and something I was not explicitly made aware of before. The intuition behind the issue (with Lemma 1) makes this clear. Surprisingly, this offset seems to hurt performance substantially more than I would have thought, especially in the environments in Table 1. I think the knowledge of this failure mode is useful for the community.
Critical weaknesses
- Weak LLM results. While I appreciate the effort to extend LOCAL to more topical settings like LLMs, the results are weak, particularly because of lack of statistical rigor. I am less concerned with how much better LOCAL is vs. the baseline, and I am more concerned with how confidently I should be reading these results. I understand that LLMs are expensive to run, but given the high variance observed in pixel based RL (Table 1), it is only natural to think that the LLM-based results could also have high variance and may be unreliable. More seeds with 95% confidence intervals would make me more confident in these results. I refer the authors to: Empirical Design in Reinforcement Learning (Patterson et al. 2024).
Questions for authors
- How should I interpret Theorem 1? My understanding is that it is making a general comment on how the tightest upper bound on the regret is related to the optimal offset, but it is not clear to me if whether this is an indication on how LOCAL's regret will be? Nor is it clear it me how it connects to the more typical setting of using an offset of zero i.e. it is not comparing the performance of the policy learned with zero offset vs. LOCAL.
- What is the broader implication of the KL loss and entropy results? Naturally, there are "bad" values (too high or too low), but there is no good value. The results are presented as if LOCAL achieves good values but this is not entirely clear me. Additionally, what are the broader implications of these KL loss/entropy results?
If the authors can address my weakness and two questions, I would more inclined towards raising my score. Question 1 will help me also gauge the importance of the theoretical result, especially given this is more of a theoretical paper (I presume). If not a theoretical paper, and it is an empirical paper, then naturally, the LLM experiments definitely need to be more rigorous.
5 comments
Thanks to the authors for incorporating the feedback. The clarifications on the empirical results, LLM experiments, theory, and writing are helpful. Its unfortunate that the LLM claim does not hold, but I appreciate that it is correct now and that the claims are dialed down. It is OK to be on par with PPO/GRPO. The visual RL experiments still suggest that this algorithm can be practically useful.
I have raised my score in light of the updates. That said, I am curious if there are any other benefits or not in the LLM experiments that are unrelated to performance (your response seems to potentially indicate something related to clipping, compute time etc)?
Thank you for the careful reading and for raising the score.
On clipping, the role that ratio clipping plays in PPO/GRPO is played in LOCAL by the KL term of the mirror-descent update in Definition 1: sets the strength of this penalty (), and the regression target is the exact solution of that KL-regularized problem. So LOCAL needs neither ratio clipping nor an additional KL regularizer, and in fact its loss does not contain a probability ratio to clip.
Compute time is also related to an advantage. LOCAL fits by regression an update whose local linearization is the natural policy gradient, which is a second-order update, yet it only costs as much as ordinary first-order optimization. The wall-clock time is only 4.5-8.3% above GRPO in our LLM experiments. As we noted in our response to Reviewer 4oNu, the current implementation is quite direct, so we expect the achievable overhead to be smaller still.
Beyond these two points, we think one future direction is to incorporate the KL to the reference model, which LLM post-training often cares about more than the KL to the previous policy. Such a term can be added to the update in Definition 1, and the logit target then remains closed-form and is still defined only up to an SDO, so the same construction applies. Another is off-policy training. Since the target and the optimal SDO are computed from the current policy , samples from a behavior policy only need the weight . This weight does not involve and therefore stays fixed during the update. That said, we have not tested either direction, so we leave both to future work.
We thank the reviewer for the thoughtful and constructive feedback and for the pointer to Patterson et al. (2024), which shaped how we now report the LLM results. Your question on Theorem 1 identified a gap between what the theorem shows and what the text suggested, and it led to a new corollary.
W1. We have rerun every RLVR comparison with 3 training seeds. Table 2 reports the mean and the sample standard deviation, and we report a 95% confidence interval for each difference rather than a hypothesis test (Welch intervals; Appendix C.9.6, Table 23).
Only one of the eight intervals excludes zero, and the pooled difference is +0.47 points (95% CI [-0.14, +1.06]). The largest single-seed gain in the submission (Llama-3.2-3B with PPO advantages, +3.97) becomes +0.20 over 3 seeds (95% CI [-4.75, +5.16]). We therefore no longer claim an improvement in the LLM experiments: the revision states that LOCAL performs comparably to PPO/GRPO on the math benchmarks.
Q1. The question points to a distinction the paper did not state clearly. First, we clarify that Theorem 1 is an upper bound on LOCAL's own regret. However, its comparison with other offset rules is not between the policy sequences that each rule would train. It compares bounds on the same policy sequence, where measuring the fitting error with LOCAL's loss gives the tightest regret bound: εT≤εc,T for any offsets c.
To compare the policy sequences trained with different offset rules, we added a new Corollary (Corollary 1 in the revision). Here εˉ(c) is the worst-case fitting error of the update selected by rule c measured with LOCAL's loss, and εˉc(c) is the error of the same update measured with rule c's own loss. For the sequence trained with any offset rule c and the one trained with LOCAL, under the same assumption on both sequences, LOCAL has the smaller fitting-error term in the regret bound, εˉ(c⋆)≤εˉc(c). In addition, since Theorem 1 shows that LOCAL's loss gives the tightest bound on a fixed sequence, we also bound the regret of the sequence trained with rule c using LOCAL's loss, which gives εˉ(c). LOCAL's sequence still has the tightest bound: εˉ(c⋆)≤εˉ(c)≤εˉc(c).
Beyond the bounds, the same ordering appears empirically: in Example 1 the Fisher residual is 3/80 under LOCAL and 3511/10000 under zero offset, and in the SDO ablations zero offset and uniform centering become unstable or collapse while the optimal SDO trains stably (Figures 2, 7, 8, 10).
Q2. We agree that neither quantity has a "good" value, and the submitted text should not have suggested otherwise. The revised paragraph is purely descriptive: it no longer links lower KL loss to higher performance and states that neither quantity has a preferred value. We report the two diagnostics because recent work on LLM post-training relates entropy collapse to premature convergence and saturating performance (ASPO, Wang et al., 2025, arXiv:2510.06062; Cui et al., 2025, arXiv:2505.22617). The implication we can support is limited to one point: at comparable math accuracy (Table 2), LOCAL ends training closer to the initial policy (lower KL loss) and with higher entropy than PPO/GRPO in most comparisons, so it does not reproduce their update but reaches similar accuracy through a different policy. One possible consequence is suggested by prior work, which reports that entropy collapse in RLVR reduces generation diversity and limits Pass@k, whereas maintaining entropy sustains it (Liang et al., 2025, arXiv:2508.14029). We did not evaluate Pass@k, so whether LOCAL's higher entropy brings this benefit is not tested here and is left to future work.