- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce BPO, a new algorithm based on information projection in order to preserve policy diversity among high reward regions. This algorithm is derived based on the idea of KL information projection, and validated on a toy tabular environment, followed by math reasoning tasks.
Critical strengths
- This paper is well written.
- The analysis driven by projection flow and policy velocity is novel and quite interesting.
- The experiments on Hyper-grid clearly characterized the sharping effect of GRPO, while showing the benefits of BPO.
Critical weaknesses
- Much of the mathematical notation should be cleaned up. For example, Algorithm 2 has many repeated operations which can be abstracted.
- While the authors compare to a number of different methods (RegExp and MaxRL especially) in mathematical reasoning tasks, it would be nice to have some baselines in the Hyper-grid task as well.
- The LLM results focus solely on math reasoning. It is known that at higher max sequence lengths, math reasoning models behave very differently, thus one more environment would be desirable.
Questions for authors
- My understanding is that the altered weights those which are underrepresented in representation space. Is this correct? Similarly, how does this compare with the likelihood and entropy of the generation?
- How does group size affect the estimation of the representation space here? Do we see that increase in likewise decreases the sharpening affect?
- Can you provide some results on a third environment which uses LLMs?
5 comments
Q3: My understanding is that the altered weights those which are underrepresented in representation space. Is this correct? Similarly, how does this compare with the likelihood and entropy of the generation?
R3: We thank the reviewer for the question.
The understanding of is correct. It gives greater weight to rollouts that are both rare and correct. This advantage is designed to guide the policy toward the minimum -Bregman divergence from the base model at a given level of expected reward. The likelihood of the correct rollouts will increase and the entropy will decrease.
To give a more concrete example, Equation (4) gives a property of the KL information projection () at a matched accuracy. As the overall accuracy increases, the mass is transferred from the wrong ones to the correct ones, with the relative probability ratio among the correct ones unchanged. Thus, the likelihood of correct ones increases and the entropy decreases.
Q4: How does group size affect the estimation of the representation space here? Do we see that increase in likewise decreases the sharpening affect?
R4: According to our policy gradient analysis, increasing the group size will not decrease the sharpening effect. However, it can yield a more accurate estimate of the representation of each method, which can have a positive effect of reducing the sharpening effect. We carry out an ablation study of on the Hyper-Grid task to elaborate.
The experiment setup follows Section 4.1 with varying group size , where is used in the original submission. We also scale the learning rate as for so that smaller learning rate is used with smaller group size. We report the KL divergence when the expected reward reaches .
| G | lr | KL at 2.59 |
|---|---|---|
| 8 | 2.65e-6 | 1.39 0.29 |
| 16 | 3.75e-6 | 0.83 0.29 |
| 32 | 5.30e-6 | 0.77 0.25 |
| 64 | 7.50e-6 | 0.69 0.22 |
| 128 (Turning point) | 1.06e-5 | 0.62 0.15 |
| 256 | 1.50e-5 | 0.61 0.17 |
| 512 | 2.12e-5 | 0.58 0.13 |
| 1,024 | 3e-5 | 0.59 0.14 |
| 2,048 | 3e-5 | 0.62 0.15 |
| 4,096 | 3e-5 | 0.58 0.10 |
We firstly observe a phase transition of the behaviours: when , as increases, the KL-divergence decreases. When , the KL-divergence just fluctuates around the level. Furthermore, the ratio of the KL divergence at (1.39) and that at (0.58) is , indicating that they are still of the same order. We also incorporate the expected reward v.s. KL-divergence dynamics during training in Figure 5 on page 18 in the updated manuscript.
This suggests that our information projection informed policy update helps the model to keep a low divergence with respect to the base model during training, and increasing has a diminishing effect on the improvement of KL-divergence. We believe increasing gives more accurate estimates of the leverage score. Once leverage scores are accurate enough, it has a diminishing effect on the divergence.
Q5: Can you provide some results on a third environment which uses LLMs?
R5: We thank the reviewer for the suggestion. We conduct another experiment on HypoSpace [1], which is adopted by IPS-GRPO [2]. The result is consistent with our claim: BPO is able to keep a good diversity or coverage at matched accuracy.
- Benchmark introduction: HypoSpace contains 139 training data and 139 testing data spanning 3 tasks, including 50 for causal inference, 50 for 3D reconstruction, and 39 from boolean interaction.
- Training setup: Qwen3-4B is adopted and trained for 2 epochs, with batch size 128 consisting of 16 rollouts from each of the 8 prompts. The reward is binary and the learning rate is 1e-6 and 3 random seeds are used.
- Evaluation: For each test question, 64 responses are sampled. We report average accuracy (pass@1) and coverage across all 139 questions, where the coverage of a single prompt is defined as We present the result in the following table:
| Method | Accuracy mean std | Coverage mean std |
|---|---|---|
| GRPO | 0.59 0.15 | 0.36 0.08 |
| MaxRL | 0.79 0.03 | 0.45 0.03 |
| RepExp | 0.67 0.06 | 0.40 0.06 |
| IAC-GRPO | 0.60 0.07 | 0.32 0.02 |
| BPO | 0.83 0.03 | 0.45 0.01 |
The result shows that BPO achieves a good coverage and accuracy at the same time.
This additional experiment provides further empirical validation of the proposed BPO method.
[1] T. Chen, et al. HypoSpace: Evaluating LLM Creativity as Set-Valued Hypothesis Generators under Underdetermination. ICML2026
[2] A. Sinha, et al. Expected return causes outcome-level mode collapse in reinforcement learning and how to fix it with inverse probability scaling. ICML2026.
We hope these responses address the reviewer's concerns and would like to thank the reviewer again for the valuable feedback.
We greatly thank the reviewer for the helpful review, and we provide our response as follows.
R1: We thank the reviewer for the writing suggestion. We will further polish the mathematical notations in the follow-up revisions. To enhance readability, we revised Algorithm 2 by moving the repeated advantage adjustment steps into a dedicated subroutine (Algorithm 3). We also simplified the indexing and added remarks explaining the purpose of each block of steps. We believe this can ease the reading of our algorithmic design.
R2: We thank the reviewer for the suggestion and we agree with the reviewer that the Hyper-Grid task shows a clear characterization of each method. We further conduct the experiments on Hyper-Grid with other methods with two experimental setups. The empirical results suggest βBPO achieves the minimum divergence at matched accuracy.
We say a peak mode is maintained if its final probability is at least 0.05. The result shows βBPO keeps 3.8 out of 4 peak modes and achieves the lowest KL-divergence compared to the other methods.
The result is as follows:
We observe the same result: βBPO achieves the lowest KL-divergence and retains the most peaks.
We hope these results deliver a clearer characterization of each method and thank the reviewer again for the suggestion. We will provide the plots of each trained policy in the follow-up revision.