- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BlendPO addresses reinforcement learning with sparse verifiable rewards by controlling the influence of training-only guidance while keeping the guidance text fixed. It interpolates the current policy's next-token logits under original and guided contexts, assigns different interpolation strengths to responses within a GRPO reward group, and trains on those responses under the original context without token-level importance correction. Experiments use abstract mathematical hints and visual-state descriptions, with all learned policies evaluated without guidance. The reported results show improved accuracy over GRPO and several adapted guided-RL baselines, supported by allocation ablations and an estimated training-compute comparison on MATH.
Critical strengths
By interpolating logits under guided and unguided contexts, BlendPO controls guidance strength without modifying the guidance content.
The evaluation covers mathematical reasoning and four visual task families, with improvements over both GRPO and baselines using the same guidance. The main results report means and standard deviations over three training seeds.
The ablations examine the roles of logit interpolation and strength allocation while keeping the guidance content fixed. These controls support the method's effectiveness and indicate that the gains cannot be attributed entirely to access to hints or additional training compute.
Critical weaknesses
1.The within-group sweep does not consistently outperform simpler allocations. The evidence for the full allocation strategy is strongest on Maze8, while its contribution on other tasks remains less clear.
2.The importance-correction ablation uses untruncated weights, whereas OC-GRPO clips its weights to [0.01, 1]. The large degradation may therefore reflect weight variance as well as reduced contributions from useful guided responses.
3.The compute-matched comparison is limited to GRPO on MATH. It does not cover abstract-hint OC-GRPO, 4U+4G, or the visual tasks, where equal training steps need not imply equal cost. Inference costs also differ.
Questions for authors
-
Under what conditions do the gains from the within-group sweep justify the additional cost of interpolation?
-
Does bounded importance correction still cause substantial degradation? With BlendPO's sampling scheme held fixed, how does the [0.01, 1] clipping rule used by OC-GRPO compare with no correction and untruncated correction?
-
How much of the advantage over the strongest baseline using the same guidance remains at comparable cost?
4 comments
Dear Reviewer 3bQr, thank you again for your helpful feedback. We have posted our response with clipped importance-correction results, compute-matched comparisons, and clarification of the sweep’s benefits. When you have a chance, we would appreciate your thoughts on whether these address your concerns.
Thank you for the additional experiments. The ablation and comparisons well address my concerns. I will raise my rating from 2 to 3.
Thank you for your positive reassessment. We are glad that our additional ablation and comparisons addressed your concerns, and we thank you again for your constructive feedback.
Thank you for the thorough review. Following your questions, we added compute-matched comparisons on the visual tasks (Table 18), and updated Tables 14 and 15 with clipped importance correction and Figure 4 with abstract-hint OC-GRPO. We have included these tables and figures in the revised manuscript, and we will update the main text to reflect them and any further discussion.
Table 12 of the original submission compares the within-group sweep with simpler allocations on MATH, AIME, Geometry3K, and PuzzleVQA.
Beyond its gains on Maze8, the sweep outperforms the best simpler allocation on MATH, AIME 26, Geometry3K, and PuzzleVQA. Moreover, the best simpler allocation shows no consistent pattern across these tasks. For example, fixed α = 0.75 is the best simpler allocation on Geometry3K but stays close to GRPO on MATH and PuzzleVQA, while 4U+4G is the best on PuzzleVQA but not on MATH or Geometry3K. Compared with a fixed α, the sweep also adds no interpolation cost. The sweep therefore offers strong performance across these four tasks without requiring a search over allocations for each task.
With BlendPO's sampling scheme held fixed, the [0.01, 1] clipping rule still causes a substantial drop on Maze8, but mitigates the drop on MATH with abstract hints. We added the clipped-correction rows to Tables 14 and 15. Table B1 summarizes the MATH column of Table 14 and the Maze8 results of Table 15.
Clipping improves over untruncated correction on both MATH and Maze8. It thus strikes a balance between limiting weight variance and preserving contributions from useful guided responses. However, with abstract hints and visual states, BlendPO without correction still outperforms the clipped variant on MATH and Maze8.
Table B1. Importance-correction comparison (%)
On MATH, BlendPO remains above abstract-hint OC-GRPO, the strongest baseline using the same hints, at a comparable training budget. An earlier BlendPO checkpoint reaches 78.57 ± 0.65% within GRPO's final mean training-FLOPs budget, compared with 77.12 ± 1.02% for OC-GRPO at a similar budget (Figure 4).
On the visual tasks, BlendPO remains above the baselines and 4U+4G at lower training compute. On Geometry3K, the last checkpoint within the cheapest baseline's budget (step 102) exceeds all listed methods. On PuzzleVQA, the last checkpoint within GRPO's budget (step 28) exceeds GRPO, HINT, and OC-GRPO, and the last checkpoint within 4U+4G's budget (step 36) exceeds 4U+4G. After training, all methods use the same inference procedure, with no additional method-specific computation.
We added the visual-task comparison to the paper as Table 18, and Tables B2 and B3 are identical to its panels (a) and (b). Compute is normalized to the final GRPO checkpoint for each task. We also added abstract-hint OC-GRPO to panels (a) and (b) of Figure 4.
Table B2. Geometry3K
Table B3. PuzzleVQA