acceptodds
Under review as a conference paper at ICLR 2027

Gibbs-Gap Policy Optimization for Off-Policy RL with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards often trains language models on responses generated by a different, possibly stale behavior policy. Correcting this mismatch with learner-to-behavior importance weights can introduce high variance for long responses. We introduce Gibbs-Gap Policy Optimization (GGPO), which minimizes a log-mean-exp contrast of KL-adjusted rewards among responses to the same prompt, without learner-to-behavior importance weights. We derive GGPO from KL-regularized policy optimization: on policy, its population loss is exactly the KL-regularized optimality gap, and under suitable coverage the learner's single-response expected reward is at least the behavior policy's best-of- reward, minus terms for the raw finite-group loss, regularization, and finite training data. We characterize reference refresh through exact Gibbs-target composition and a reward bound for averaged window outputs. This refresh is empirically important: in an ablation, fixed-reference GGPO scores on the five-benchmark problem-weighted average, versus with the default refresh schedule. Experiments on mathematical reasoning further show that GGPO outperforms GRPO and AGRO under training–inference mismatch and on fixed offline data, while maintaining its performance as controlled policy staleness grows.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, off-policy reinforcement learning, asynchronous training, policy optimization, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Nu5dOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes using Gibbs-Gap as the objective function, transforming the bias caused by policy mismatch into terms in the performance improvement upper bound, thereby avoiding the use of importance weights. The paper conducts experiments on mathematical reasoning tasks and demonstrates that the proposed method outperforms GRPO and AGRO across multiple benchmarks.

Critical strengths
  1. The paper uses a Gibbs-Gap-based contrastive loss to avoid importance weights, addressing the instability associated with importance weighting.

  2. The paper provides detailed theoretical analyses, explaining why Gibbs-Gap is effective and characterizing its performance guarantees.

  3. The experimental results are promising, with the proposed method outperforming the baselines.

Critical weaknesses
  1. Some theoretical results may have limited practical significance. For example, the constant $ C_x $ in Equation 5 may not exist, rendering the upper bound meaningless. Moreover, Equation 9 and related results only guarantee that $ J(\pi_\theta) $ is not much lower than $ J_K(\pi_{\mathrm{beh}}) $. Since they only provide upper bounds on the performance gap, they cannot guarantee that $ J(\pi_\theta) $ will actually improve.

  2. In Table 6, GGPO with a fixed reference suffers a severe performance collapse. The authors should analyze the reasons behind this behavior.

  3. The experiments are not sufficiently comprehensive. They are limited to small models below 7B parameters, the Qwen3 model family, and mathematical reasoning tasks, making it difficult to draw conclusions about the generalizability of the proposed method.

Questions for authors

None

Reviewer_gthCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary: This paper proposes GGPO, an RLVR algorithm that stabilizes off-policy learning. Building on prior work AGRO that points out the optimal policy has a constant KL-adjusted reward across responses within the same prompt group, GGPO replaces AGRO's variance minimization loss with a log-mean-exp contrastive loss, combined with periodic reference policy resets.

Contributions: The authors show that choosing this specific choice of optimizing towards constant KL-adjusted reward yields several theoretical and algorithmic properties:

  • it implicitly optimizes the learner's single-response reward toward the behavior policy's best-of-K reward as the KL coefficient approaches zero, increasing pass@k;
  • it asymmetrically assigns larger total positive gradient weight to rare successes, benefiting performance on hard problems;
  • its KL-ratio reweighting among correct responses is not affected by a small KL coefficient, preserving diversity of correct responses.

Validation: Across five math reasoning benchmarks on two models, GGPO outperforms GRPO and AGRO, with analysis showing the above properties; controlled staleness and offline RL experiments show its effect on stabilizing off-policy learning.

Critical strengths

Well-justified objective design supported by thorough empirical verification. Although replacing AGRO's variance loss with a log-mean-exp loss is an incremental change in form, the paper clearly justifies it both analytically and empirically, e.g. rare success weighting (Figure 5, Figure 6), diversity preservation (Figure 1d, Table 3), robustness under policy staleness or pure offline training (Figure 2, Figure 3).

Critical weaknesses
  1. Limited experiment scope for evaluating train-infer mismatch or off-policy RL. All experiments are conducted on relatively small dense models (1.7B and single-seed 4B) in single-turn mathematical reasoning with simulated synchronous staleness. More realistic settings such as large scale, truly asynchronous, moe model and multi-turn agentic RL are not evaluated.

  2. Questionable practical benefits in effectiveness, convergence speed, and scalability as a standalone algorithm. While the mechanism analysis is interesting, GGPO's practical advantage is less clear.

  • in Figure 1d, GGPO learns substantially slower than AGRO and even GRPO;
  • in Table 1, when scaling from Qwen3-1.7B to Qwen3-4B, the gain shrinks from +3.94 pp to +1.47 pp over GRPO, or +1.51 pp to +1.17 pp over AGRO, inviting concerns on whether the gains can hold at larger model scales.
Questions for authors
  1. In Figure 1d, GGPO underperforms AGRO and GRPO at early steps, and in Table 1 its margin over the baselines narrows when moving from 1.7B to 4B. How do the authors view GGPO's sample efficiency and scalability to larger models?

  2. Both Table 1 and Figure 2 are results on the final checkpoints. If the checkpoints are selected at its best within the same compute, will GGPO's benefit shrink?

  3. Both GGPO and AGRO are motivated by off-policy learning without explicit importance weights. In Figure 2, what causes AGRO to lose to GGPO, or even GRPO at large policy staleness? Are there other factors other than avoiding importance weights that contributed to the robustness to heavy staleness?

  4. Unlike policy gradient methods that guarantee monotonic local policy improvement, GGPO optimizes towards an optimality condition. Is there any guarantee on why doing gradient descent on the optimality derived objective can improve the policy before reaching the optimum?

  5. Have the authors tested GGPO in more realistic settings (e.g. genuinely asynchronous training or beyond single-turn math)?

I am open to raising my score if the authors can provide insightful explanations or empirical evidence to the questions above.

Reviewer_hmsKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Gibbs-Gap Policy Optimization (GGPO), an off-policy RLVR objective that uses no learner to behavior importance weights. Motivated by the fact that the KL-regularized optimum has constant KL-adjusted reward across responses to a prompt, GGPO minimizes a log-mean-exp contrast of these adjusted rewards within each response group, and periodically refreshes the reference policy to the learner. Under coverage conditions, this finite group loss is shown to bound the gap between the learner's expected reward and the behavior policy's best-of-KK reward. Experiments on five mathematical-reasoning benchmarks show GGPO outperforming GRPO and AGRO in the default setting and holding its accuracy as staleness grows from N=8N = 8 to N=128N = 128, where both baselines degrade. An ablation shows periodic reference refresh to be essential to GGPO's own performance.

Critical strengths
  • The paper offers a principled derivation, with a relation to its closest comparator AGRO. On policy GGPO population loss equals the regularized optimality gap exactly, whereas AGRO's variance objective matches it only to second order, and as stated in Appendix A.9 at the given experimental hyperparameter values AGRO is a distinct objective rather than a special case approximation.

  • The approach is simple to implement and doesn't require hand-tuned thresholds.

Critical weaknesses
  • The paper is hard to follow in its current state, mainly because its theoretical results are not situated. Some results are introduced without motivation or are not followed by an explanation of what insight they add (for example Prop 5 in section 4.2). Several bounds might also be vacuous at the parameters used in the experiments. For example, Prop 2 right hand side needs to be lower than 1 to provide a nontrivial bound for r in [0,1] , but contains an exponential term eDe^D. Also CC can grow exponentially with response length, and no quantitative bounds or empirical estimates for Γ\Gamma are provided for the setting considered where πref≠πbeh\pi_{\mathrm{ref}} \neq \pi_{\mathrm{beh}}.

  • The on-policy N=1 experiment is missing and it might show that GGPO offers a better group update than GRPO. As Figure 2 shows that the performance of GGPO is consistently better than that of GRPO, while the latter barely changes through different staleness N=8,...,64 indicating that the staleness doesn't explain alone the better performance on table 1.

Questions for authors
  • What are the numerical estimates for the constants ? A sampled empirical value for Γ\Gamma and CC in equation (9), and CC and DD in equation (15) at β=0.01\beta = 0.01, would help determine when the bounds are non-vacuous.
  • How do the methods GGPO, AGRO and GRPO compare at N=1N = 1, or the least-stale configuration the setup permits? Since πbeh\pi_{\mathrm{beh}} enters neither the loss nor the gradient, the GGPO update is identical on-policy and off-policy, so a gap that persists at N=1N=1 would indicate the contribution is a better group objective rather than an off-policy correction.
  • Equation (14) shows that TT exact refresh windows at temperature β\beta compose into a final Gibbs target at β/T\beta/T. App D.5 reports that a fixed reference costs GGPO 13.6 points (32.9632.96 versus 46.5846.58 with refresh), and leaving it below fixed reference GRPO. It would be beneficial to know whether the reset of the reference or the annealing of the effective temperature β\beta is the component responsible for making refreshed GGPO work better. An experiment with a fixed temperature at β/T\beta/T, and another annealing β\beta down to β/T\beta/T, both keeping the reference fixed, would help determine the mechanism responsible for the performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.