acceptodds
Under review as a conference paper at ICLR 2027

Diversity-Preserving RLVR via Information Projection

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR), exemplified by Group Relative Policy Optimization (GRPO), is widely used for post-training of modern generative models. Although GRPO improves reasoning accuracy, it can also over-sharpen model output distributions, suppressing alternative correct responses and reducing solution diversity. We mitigate this issue through information projection, seeking the policy that best preserves the response diversity of the base model at a target expected reward. Divergence from the base policy serves as a proxy for diversity loss. To this end, we propose -Bregman Policy Optimization (BPO), a family of representation-based policy updates derived from information-projection dynamics under -Bregman divergences. In a tabular model, we characterize the resulting projection flow and prove that it minimizes the corresponding divergence from the base policy at every attained reward level. Experiments on the Hyper-Grid task illustrate that BPO attains a reward-maximizing policy while preserving diversity among high-reward actions. Extensive experiments on mathematical reasoning further show that BPO maintains competitive Pass@ while improving Pass@ at large sampling budgets, and achieves the KL-divergence-accuracy Pareto frontier, supporting information projection as a principled framework for mitigating over-sharpening during RLVR.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RLVR, Information Projection, Gradient Flow, Bregman Divergence

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_idFKOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce β\betaBPO, a new algorithm based on information projection in order to preserve policy diversity among high reward regions. This algorithm is derived based on the idea of KL information projection, and validated on a toy tabular environment, followed by math reasoning tasks.

Critical strengths
  • This paper is well written.
  • The analysis driven by projection flow and policy velocity is novel and quite interesting.
  • The experiments on Hyper-grid clearly characterized the sharping effect of GRPO, while showing the benefits of β\betaBPO.
Critical weaknesses
  • Much of the mathematical notation should be cleaned up. For example, Algorithm 2 has many repeated operations which can be abstracted.
  • While the authors compare to a number of different methods (RegExp and MaxRL especially) in mathematical reasoning tasks, it would be nice to have some baselines in the Hyper-grid task as well.
  • The LLM results focus solely on math reasoning. It is known that at higher max sequence lengths, math reasoning models behave very differently, thus one more environment would be desirable.
Questions for authors
  • My understanding is that the altered A~\tilde A weights those which are underrepresented in representation space. Is this correct? Similarly, how does this compare with the likelihood and entropy of the generation?
  • How does group size affect the estimation of the representation space here? Do we see that increase in GG likewise decreases the sharpening affect?
  • Can you provide some results on a third environment which uses LLMs?
5 comments

We greatly thank the reviewer for the helpful review, and we provide our response as follows.

Q1: Much of the mathematical notation should be cleaned up. For example, Algorithm 2 has many repeated operations which can be abstracted.

R1: We thank the reviewer for the writing suggestion. We will further polish the mathematical notations in the follow-up revisions. To enhance readability, we revised Algorithm 2 by moving the repeated advantage adjustment steps into a dedicated subroutine (Algorithm 3). We also simplified the indexing and added remarks explaining the purpose of each block of steps. We believe this can ease the reading of our algorithmic design.

Q2: While the authors compare to a number of different methods (RegExp and MaxRL especially) in mathematical reasoning tasks, it would be nice to have some baselines in the Hyper-grid task as well.

R2: We thank the reviewer for the suggestion and we agree with the reviewer that the Hyper-Grid task shows a clear characterization of each method. We further conduct the experiments on Hyper-Grid with other methods with two experimental setups. The empirical results suggest β\betaBPO achieves the minimum divergence at matched accuracy.

  • Single Task HyperGrid (Experiment 1): the same setup as in Section 4.1, we conduct IAC-GRPO and RepExp which adopts a bonus parameter b=0.01b = 0.01, while MaxRL is not applicable to this single task HyperGrid as it is a cross-prompt reweighting method. The models are trained and evaluated across 10 different seeds.
MethodRewardKLPeaks kept ≥0.05 (out of 4)
β\betaBPO2.591 ±\pm 0.0030.59 ±\pm 0.133.8 ±\pm 0.4
RepExp2.595 ±\pm 0.0011.47 ±\pm 0.312.3 ±\pm 0.7
GRPO2.597 ±\pm 0.0001.86 ±\pm 0.191.1 ±\pm 0.3
IAC-GRPO2.598 ±\pm 0.0002.02 ±\pm 0.181.0 ±\pm 0.0

We say a peak mode is maintained if its final probability is at least 0.050.05. The result shows β\betaBPO keeps 3.83.8 out of 44 peak modes and achieves the lowest KL-divergence compared to the other methods.

  • Two Task HyperGrid (Experiment 2): in order to examine the performance of MaxRL, we developed a two-task HyperGrid problem by adding a new grid task. Specifically,
    • Task A: The original task which uses (2,2),(2,5),(5,2),(5,5)(2,2), (2,5), (5,2), (5,5) as 4 peaks.
    • Task B: A new task that uses (1,4),(3,1),(4,6),(6,3)(1,4), (3,1), (4,6), (6,3) as 4 peaks.
    • Other setups: The reward function on the grid is the same. The task index (A or B) is added to the prompt and these two tasks are mixed with ratio 1:11:1 during training.

The result is as follows:

MethodReward (mean of 2 tasks)KL task AKL task BKL meanPeaks kept ≥0.05 (of 8)
β\betaBPO2.591 ±\pm 0.0030.76 ±\pm 0.231.24 ±\pm 0.371.00 ±\pm 0.216.2 ±\pm 1.4
RepExp2.595 ±\pm 0.0011.43 ±\pm 0.441.37 ±\pm 0.211.40 ±\pm 0.254.6 ±\pm 0.5
MaxRL2.592 ±\pm 0.0011.99 ±\pm 0.231.57 ±\pm 0.221.78 ±\pm 0.152.3 ±\pm 0.5
GRPO2.598 ±\pm 0.0002.08 ±\pm 0.281.63 ±\pm 0.231.86 ±\pm 0.202.3 ±\pm 0.5
IAC-GRPO2.598 ±\pm 0.0002.20 ±\pm 0.261.63 ±\pm 0.231.92 ±\pm 0.162.0 ±\pm 0.0

We observe the same result: β\betaBPO achieves the lowest KL-divergence and retains the most peaks.

We hope these results deliver a clearer characterization of each method and thank the reviewer again for the suggestion. We will provide the plots of each trained policy in the follow-up revision.

Q3: My understanding is that the altered A~\tilde{A} weights those which are underrepresented in representation space. Is this correct? Similarly, how does this compare with the likelihood and entropy of the generation?

R3: We thank the reviewer for the question.

The understanding of A~\tilde{A} is correct. It gives greater weight to rollouts that are both rare and correct. This advantage is designed to guide the policy toward the minimum β\beta-Bregman divergence from the base model at a given level of expected reward. The likelihood of the correct rollouts will increase and the entropy will decrease.

To give a more concrete example, Equation (4) gives a property of the KL information projection (β=1\beta=1) at a matched accuracy. As the overall accuracy increases, the mass is transferred from the wrong ones to the correct ones, with the relative probability ratio among the correct ones unchanged. Thus, the likelihood of correct ones increases and the entropy decreases.

Q4: How does group size GG affect the estimation of the representation space here? Do we see that increase in GG likewise decreases the sharpening affect?

R4: According to our policy gradient analysis, increasing the group size will not decrease the sharpening effect. However, it can yield a more accurate estimate of the representation of each method, which can have a positive effect of reducing the sharpening effect. We carry out an ablation study of GG on the Hyper-Grid task to elaborate.

The experiment setup follows Section 4.1 with varying group size G=2k⋅G0,k=−7,−6,…,2G=2^{k}\cdot G_0,k=-7,-6,\ldots,2, where G0=1024G_0=1024 is used in the original submission. We also scale the learning rate as lr=lr0⋅G/G0lr=lr_0\cdot\sqrt{G/G_0} for G<G0G<G_0 so that smaller learning rate is used with smaller group size. We report the KL divergence when the expected reward reaches 2.592.59.

GlrKL at 2.59
82.65e-61.39 ±\pm 0.29
163.75e-60.83 ±\pm 0.29
325.30e-60.77 ±\pm 0.25
647.50e-60.69 ±\pm 0.22
128 (Turning point)1.06e-50.62 ±\pm 0.15
2561.50e-50.61 ±\pm 0.17
5122.12e-50.58 ±\pm 0.13
1,0243e-50.59 ±\pm 0.14
2,0483e-50.62 ±\pm 0.15
4,0963e-50.58 ±\pm 0.10

We firstly observe a phase transition of the behaviours: when G<128G<128, as GG increases, the KL-divergence decreases. When G≥128G\ge 128, the KL-divergence just fluctuates around the 0.600.60 level. Furthermore, the ratio of the KL divergence at G=8G=8 (1.39) and that at G=4096G=4096 (0.58) is 2.402.40, indicating that they are still of the same order. We also incorporate the expected reward v.s. KL-divergence dynamics during training in Figure 5 on page 18 in the updated manuscript.

This suggests that our information projection informed policy update helps the model to keep a low divergence with respect to the base model during training, and increasing GG has a diminishing effect on the improvement of KL-divergence. We believe increasing GG gives more accurate estimates of the leverage score. Once leverage scores are accurate enough, it has a diminishing effect on the divergence.

Q5: Can you provide some results on a third environment which uses LLMs?

R5: We thank the reviewer for the suggestion. We conduct another experiment on HypoSpace [1], which is adopted by IPS-GRPO [2]. The result is consistent with our claim: β\betaBPO is able to keep a good diversity or coverage at matched accuracy.

  • Benchmark introduction: HypoSpace contains 139 training data and 139 testing data spanning 3 tasks, including 50 for causal inference, 50 for 3D reconstruction, and 39 from boolean interaction.
  • Training setup: Qwen3-4B is adopted and trained for 2 epochs, with batch size 128 consisting of 16 rollouts from each of the 8 prompts. The reward is binary and the learning rate is 1e-6 and 3 random seeds are used.
  • Evaluation: For each test question, 64 responses are sampled. We report average accuracy (pass@1) and coverage across all 139 questions, where the coverage of a single prompt is defined as \mboxCoverage=\mboxnumberofdifferentvalidanswersappearedin64responses\mboxnumberofallpossiblevalidanswers\mbox{Coverage} = \frac{\mbox{number of different valid answers appeared in 64 responses}}{\mbox{number of all possible valid answers}} We present the result in the following table:
MethodAccuracy mean ±\pm stdCoverage mean ±\pm std
GRPO0.59 ±\pm 0.150.36 ±\pm 0.08
MaxRL0.79 ±\pm 0.030.45 ±\pm 0.03
RepExp0.67 ±\pm 0.060.40 ±\pm 0.06
IAC-GRPO0.60 ±\pm 0.070.32 ±\pm 0.02
β\betaBPO0.83 ±\pm 0.030.45 ±\pm 0.01

The result shows that β\betaBPO achieves a good coverage and accuracy at the same time.

This additional experiment provides further empirical validation of the proposed β\betaBPO method.

[1] T. Chen, et al. HypoSpace: Evaluating LLM Creativity as Set-Valued Hypothesis Generators under Underdetermination. ICML2026

[2] A. Sinha, et al. Expected return causes outcome-level mode collapse in reinforcement learning and how to fix it with inverse probability scaling. ICML2026.

We hope these responses address the reviewer's concerns and would like to thank the reviewer again for the valuable feedback.

Reviewer_YJebOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies over-concentration among correct responses during reinforcement learning with verifiable rewards. It formulates diversity preservation as minimizing divergence from the base policy at a given expected reward. In a single-context tabular setting, the authors show that ordinary softmax logit gradients amplify probability differences between equally rewarded correct actions, and derive a family of information-projection flows under β-Bregman divergences. The proposed practical algorithm, βBPO, uses representation-based leverage scores to reweight advantages. Experiments on Hyper-grid and three Qwen model sizes report improved large-budget Pass@k while maintaining competitive Pass@1.

Critical strengths
  1. Clear motivation and an interpretable theoretical framework. Proposition 3.1 cleanly separates the probability of success from the conditional distribution within each reward class. Lemma 3.2 then explains why ordinary logit-gradient updates preferentially reinforce already-likely correct actions. The principal derivations for the idealized projection path appear sound under their stated assumptions.

  2. Promising practical results and a useful controlled experiment. The method can be incorporated into existing training pipelines through advantage modification. Hyper-grid allows exact evaluation of terminal distributions, and the LLM experiments include several model sizes and relevant baselines. The reported Qwen3-8B improvement from 74.9% to 81.4% Pass@512 over the runner-up method would be meaningful if reproduced reliably.

Critical weaknesses
  1. The theoretical connection to the practical algorithm is limited. The guarantees rely on tabular representations and asymptotically large sampling groups, whereas the LLM implementation uses learned representations and only eight responses per prompt. The paper does not establish whether these practical updates meaningfully approximate the proposed projection flow. A group-size ablation and direct diagnostics of this approximation would strengthen the central claim.

  2. The evaluation does not directly validate the claimed preservation of solution diversity. Higher Pass@k demonstrates improved problem-solving coverage, but does not distinguish between generating one correct solution more reliably and preserving multiple distinct correct solutions to the same problem. Direct measurements of correct-solution diversity at matched accuracy, together with results across training seeds, are needed to substantiate the claimed mechanism and establish the reliability of the gains.

Questions for authors

Please see weaknesses.

4 comments

We thank the reviewer for the constructive feedback!

Q1: The theoretical connection to the practical algorithm is limited. The guarantees rely on tabular representations and asymptotically large sampling groups, whereas the LLM implementation uses learned representations and only eight responses per prompt. The paper does not establish whether these practical updates meaningfully approximate the proposed projection flow. A group-size ablation and direct diagnostics of this approximation would strengthen the central claim.

R1: We thank the reviewer for raising this question.

We respectfully argue that the tabular representation and the large-group limit simplifications are standard and are commonly adopted in the theoretical analysis[1–5]. In addition, from the error source perspective, this manuscript mainly focuses on the optimization bias whereas the statistical bias is orthogonal and can lead to future research.

Specifically, we agree that our theoretical characterization relies on a tabular representation and the large-group limit, which yield a clear interpretation. The tabular setting, which is widely used in prior work [1-5], allows us to characterize the policy trajectory induced by the limiting update. Existing results for stochastic policy gradient establish convergence and finite-time bounds on the expected reward gap under suitable assumptions (e.g. access to the exact gradient [1,4]).

Furthermore, the error of the final convergent point xτx_{\tau} of a policy gradient algorithm to the minimizer x\*x^\* can be decomposed to the optimization error (OE) and the statistical error (SE) [5]:

\le \underbrace{\lVert x_{\tau} - x_{pg} \rVert}_{\mathrm{SE}} + \underbrace{\lVert x_{pg} - x^{\ast} \rVert}_{\mathrm{OE}}$$ where $x_{pg}$ is the convergent point with large samples. We put emphasis on the **optimization error** in this manuscript, whereas the statistical error is **orthogonal** and influenced by the group size $G$. Extending our trajectory analysis to incorporating the **finite-sample statistical error** can lead to future research. [1] J. Mei, et al. On the global convergence rates of softmax policy gradient methods. ICML2020. [2] N. Lazic, et al. To see the unseen: on the generalization ability of transformers in symbolic reasoning. COLM2026. [3] A. Sinha, et al. Expected return causes outcome-level mode collapse in reinforcement learning and how to fix it with inverse probability scaling. ICML2026. [4] T. Lattimore. A Diffusion Analysis of Policy Gradient for Stochastic Bandits. 2026. [5] M. Lin, et al. Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation: The Case of Multi-Armed Bandits. 2026.

Q2: The evaluation does not directly validate the claimed preservation of solution diversity. Higher Pass@k demonstrates improved problem-solving coverage, but does not distinguish between generating one correct solution more reliably and preserving multiple distinct correct solutions to the same problem. Direct measurements of correct-solution diversity at matched accuracy, together with results across training seeds, are needed to substantiate the claimed mechanism and establish the reliability of the gains.

R2: We acknowledge that Pass@k metric can serve as a useful proxy for diversity, and we provide further diversity measurements in the (1) Hyper-Grid task, and (2) direct measurement of the mode coverage in correct solutions, to augment the diversity preservation claim.

Firstly, β\betaBPO achieves the smallest KL-divergence at matched accuracy in Hyper-Grid task. Figure 1c on page 6 reports the KL divergence and expected reward dynamics during the RL training. It shows that β\betaBPO achieves the minimum divergence at matched accuracy. We further conducted RepExp and IAC-GRPO for this task for comparison and measured how many of the four peak modes retain a final probability of at least 0.050.05.

MethodRewardKLPeaks kept ≥0.05 (out of 4)
β\betaBPO2.591 ±\pm 0.0030.59 ±\pm 0.133.8 ±\pm 0.4
RepExp2.595 ±\pm 0.0011.47 ±\pm 0.312.3 ±\pm 0.7
GRPO2.597 ±\pm 0.0001.86 ±\pm 0.191.1 ±\pm 0.3
IAC-GRPO2.598 ±\pm 0.0002.02 ±\pm 0.181.0 ±\pm 0.0

All four methods approach the maximum expected reward. At these comparable reward levels, β\betaBPO retains 3.8 of the 4 peak modes on average and attains the lowest KL divergence from the base policy.

Furthermore, for LLM, we measure the correct-solution mode coverage as follows:

  • Data: We randomly sampled 100 prompts from the DAPO dataset, generate 32 rollouts for each prompt.
  • LLM-as-judge: We adopt GPT-6.1 Sol as LLM-as-judge to group rollouts by the distinct mathematical approaches they applied to each prompt.
  • Metric: We compute the share of the modes in correct rollouts as: \mboxnumberofdistinctmodesappearingincorrectrollouts\mboxnumberofmodesappearinallrolloutsofallmodels.\frac{\mbox{number of distinct modes appearing in correct rollouts}}{\mbox{number of modes appear in all rollouts of all models}}.
MethodBaseGRPOβ\betaBPORepExpMaxRLIAC-GRPO
mode share54.2%41.7%66.7%58.3%45.8%33.3%

The table shows that β\betaBPO achieves the highest coverage with its correct rollouts.

To summarize, these results provide direct evidence of greater peak-mode retention at comparable expected rewards on Hyper-Grid, and broader correct-solution coverage in the sampled LLM rollouts.

We hope these responses can resolve the concerns and would like to thank the reviewer again for the constructive feedback.

Thank you for your detailed responses. I appreciate the authors' efforts to address my concerns, and I will maintain my original positive score. Wish the authors all the best with the final decision.

We are glad that the reviewer's concerns have been resolved and thank the reviewer for the positive assessment.

Reviewer_AynwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies diversity preservation in RLVR from an information-projection perspective. The authors show that, under a single-context tabular model, GRPO does not follow the KL information-projection path and instead amplifies probability differences among equally rewarded actions. Based on this observation, they introduce β\beta-Bregman Policy Optimization (β\beta BPO), which defines a regularized projection path under β\beta-Bregman divergence and derives the corresponding projection flow, whose policy at each attained reward level minimizes divergence from the base policy. To approximate this flow in practice, β\beta BPO constructs representation-based leverage scores from sampled responses, uses them to reweight the baseline and trajectory advantages, and optimizes a GRPO-style objective with the resulting modified advantages.

Critical strengths
  1. The paper formulates diversity preservation in RLVR as an information-projection problem and shows that, at a fixed reward level, the projected policy preserves the conditional distribution among equally rewarded actions. The analysis further characterizes how GRPO deviates from this path by amplifying existing probability differences among correct actions.
  2. The authors derive the β\beta-Bregman projection flow and use it to construct β\beta BPO. The practical update uses representation-based leverage scores to reweight the baseline and trajectory advantages while retaining a GRPO-style optimization objective.
  3. The experiments cover both controlled Hyper-grid settings and LLM post-training across multiple model scales. β\beta BPO preserves probability mass across high-reward modes in Hyper-grid, achieves a better reward-divergence tradeoff than GRPO, and improves large-budget Pass@k over several RLVR baselines.
Critical weaknesses
  1. The theoretical connection between β\beta BPO and the projection flow is established in the tabular setting, where leverage scores correspond to inverse empirical action probabilities. In the LLM implementation, however, leverage scores are computed from hidden-state representations, but the paper does not verify whether they remain a reliable proxy for response rarity. This weakens the connection between the practical algorithm and the theoretical analysis.
  2. Proposition 3.4 establishes the minimum DβD_\beta divergence from the base policy at a matched expected reward, while the LLM experiments evaluate a sampled, length-normalized token-level KL divergence. This difference is particularly relevant for β=1.5\beta = 1.5 and β=2\beta = 2, where the theoretical guarantee concerns the corresponding β\beta-Bregman divergence rather than KL. Since Figure 4 is used to corroborate Proposition 3.4, the current evaluation provides only limited evidence for the theoretical minimum-divergence property. Evaluating the corresponding DβD_\beta would provide a more direct empirical validation.
  3. The notion of diversity evaluated in the LLM experiments is not fully aligned with the theoretical analysis. Pass@k can reflect correct-response coverage and serve as a useful proxy for diversity, but it is also strongly affected by the total probability mass assigned to correct responses. In contrast, the theory focuses on how probability is distributed among different correct responses. Additional measurements of correct-solution mode coverage or distribution would better support the diversity-preservation claim.
Questions for authors

Please refer to the weaknesses above.

4 comments

We thank the reviewer for the constructive questions and we provide our response as follows.

Q1: The theoretical connection between β\betaBPO and the projection flow is established in the tabular setting, where leverage scores correspond to inverse empirical action probabilities. In the LLM implementation, however, leverage scores are computed from hidden-state representations, but the paper does not verify whether they remain a reliable proxy for response rarity. This weakens the connection between the practical algorithm and the theoretical analysis.

R1: We thank the reviewer for raising this question.

We firstly provide empirical evaluations via LLM-as-judge and trigram analysis to show that the leverage scores computed via the hidden-state representations can serve as a proxy of response rarity. We further clarify the role of the leverage score as a metric of the level of uncertainty due to rareness and its wide usage in the literature.

To start with, to empirically test whether hidden-state leverage scores track response rarity in practice, we randomly sampled 100 prompts from DAPO and generated 32 rollouts per prompt. We ranked the rollouts within each prompt by leverage score and compared the rankings against two measures of rarity.

  • Reasoning-mode frequency: We used GPT-6.1 Sol to classify rollouts by the distinct mathematical approaches they applied to solve the prompt. We labeled a mode as frequent if it appeared in at least 8 of the 32 rollouts and rare if it appeared less than three times.
  • Trigram rarity: For each rollout, we computed the negative base-2 logarithm of its average Jaccard similarity to the rollouts generated by the same model for the same prompt. Jaccard similarity measures overlap between sets of consecutive three-token sequences. Higher scores indicate less textual overlap and thus greater textual rarity.
Leverage rank within groupIn frequent modeIn rare modeIn the lowest quartile of trigram rarity
Lowest 890.1%2.6%40.1%
All rollouts84.2%4.1%25.0%
Highest 880.3%4.4%11.6%

This result shows that, when a rollout has a small leverage score, it is more likely to belong to a frequent mode and less likely to belong to a rare mode. In addition, it also tends to have lower trigram rarity.

Furthermore, we clarify the role of leverage score as a metric of the level of uncertainty due to rareness. The tabular setting, which is widely used in prior work [1–3], makes the link between leverage and rarity explicit through action counts. In the LLM implementation, leverage measures how well a rollout’s representation is supported by the other representations in the group. A higher leverage score indicates lower effective coverage in feature space and hence greater uncertainty. Prior work in RL and LLM uses leverage scores to quantify this uncertainty [4,5], and [5] provides empirical evidence that representation-based leverage scores are useful as exploration bonuses in LLMs.

To summarize, these findings support the use of hidden-state leverage as a proxy for rollout rarity.

[1] J. Mei, et al. On the global convergence rates of softmax policy gradient methods. ICML2020.

[2] N. Lazic, et al. To see the unseen: on the generalization ability of transformers in symbolic reasoning. COLM2026.

[3] A. Sinha, et al. Expected return causes outcome-level mode collapse in reinforcement learning and how to fix it with inverse probability scaling. ICML2026.

[4] M. Soare, et al. Best-arm identification in linear bandits. NeurIPS2014.

[5] J. Tuyls, et al. Representation-based exploration for language models: From test-time to post-training. ICML2026

Q2: Proposition 3.4 establishes the minimum DβD_{\beta} divergence from the base policy at a matched expected reward, while the LLM experiments evaluate a sampled, length-normalized token-level KL divergence. This difference is particularly relevant for β=1.5\beta=1.5 and β=2.0\beta=2.0, where the theoretical guarantee concerns the corresponding β\beta-Bregman divergence rather than KL. Since Figure 4 is used to corroborate Proposition 3.4, the current evaluation provides only limited evidence for the theoretical minimum-divergence property. Evaluating the corresponding DβD_{\beta} would provide a more direct empirical validation.

R2: We thank the reviewer for the suggestion.

We used KL divergence in the original experiments because it is a standard metric in the RL and LLM literature (e.g., [6-8]). In the following, we evaluate the corresponding DβD_\beta, which provides a direct empirical validation of the minimum-divergence property in Proposition 3.4.

We replaced each token-level KL term in Section 4.2 with the corresponding β\beta-Bregman divergence DβD_\beta term, retaining the same sampling procedure and length normalization.

MethodPass@1DβD_\beta (β=1.5\beta=1.5)DβD_\beta (β=2.0\beta=2.0)
GRPO0.2110.1000.053
MaxRL0.1970.1350.072
RepExp0.2030.0640.037
IAC-GRPO0.1980.1540.082
β\betaBPO (β=1.5\beta=1.5)0.2170.039-
β\betaBPO (β=2.0\beta=2.0)0.211-0.022

We also plot these points in Figure 8 on page 19 in the updated manuscript.

For both values of β\beta, β\betaBPO achieves a lower token-level DβD_\beta than every baseline at comparable or higher Pass@1. These results provide token-level empirical evidence consistent with the minimum-divergence property in Proposition 3.4.

[6] J. Schulman, et al. Trust region policy optimization. ICML2015.

[7] L. Ouyang, et al. Training language models to follow instructions with human feedback. NeurIPS2022.

[8] I. Shenfeld, et al. RL’s razor: Why online reinforcement learning forgets less. ICLR2026.

Q3: The notion of diversity evaluated in the LLM experiments is not fully aligned with the theoretical analysis. Pass@k can reflect correct-response coverage and serve as a useful proxy for diversity, but it is also strongly affected by the total probability mass assigned to correct responses. In contrast, the theory focuses on how probability is distributed among different correct responses. Additional measurements of correct-solution mode coverage or distribution would better support the diversity-preservation claim.

R3: We acknowledge that Pass@k metric can serve as a useful proxy for diversity, and we conduct direct evaluation of the retained modes in the correct rollouts to support the diversity-preservation claim.

We measure the correct-solution mode coverage using the rollouts and LLM-as-judge result in R1. Specifically, for each model, we compute the share of the modes in correct rollouts as: \mboxnumberofdistinctmodesappearingincorrectrollouts\mboxnumberofmodesappearinallrolloutsofallmodels.\frac{\mbox{number of distinct modes appearing in correct rollouts}}{\mbox{number of modes appear in all rollouts of all models}}.

MethodBaseGRPOβ\betaBPORepExpMaxRLIAC-GRPO
mode share54.2%41.7%66.7%58.3%45.8%33.3%

The table shows that β\betaBPO achieves the highest coverage with its correct rollouts.

We thank the reviewer again for the constructive questions and suggestions, and hope we have addressed the concerns.

Thank you for the thoughtful response. I have no further questions at this point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.