acceptodds
Under review as a conference paper at ICLR 2027

BlendPO: Controlling Guidance Strength in Reinforcement Learning without Changing Guidance Content

Abstract

Reinforcement learning with verifiable rewards provides no learning signal when sampled rollouts fail to produce a correct solution. A common approach is to condition rollouts on guidance, such as partial solutions or hints, during training. Stronger guidance makes successful rollouts more likely but can transfer less to the unguided setting used at test time, making guidance strength a key design choice. Solution prefixes allow fine-grained control by adjusting the prefix length. In contrast, for guidance without a natural ordering, such as abstract hints, the guidance strength is typically controlled only through coarse choices, such as selecting a hint level or deciding whether to provide a hint at all. We propose BlendPO (Blended-context Policy Optimization), which enables fine-grained control of guidance strength without changing guidance content by interpolating policy logits obtained with and without guidance. Rather than training with a single guidance strength, BlendPO samples responses at multiple interpolation strengths for each problem and updates the policy using these responses under the original unguided context. We first evaluate BlendPO on mathematical reasoning using abstract hints generated from reference solutions. BlendPO outperforms methods based on solution prefixes as well as those using the same abstract hints. We then extend BlendPO to visual reasoning with visual states, textual descriptions of the visual input that can be obtained from dataset metadata or the task environment without reference solutions. Across four visual reasoning task families, BlendPO consistently improves over GRPO and methods using the same visual states, demonstrating its potential to broaden the practical applicability of guided RL when reference solutions are unavailable.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with Verifiable Rewards, Learning with Privileged Information, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3bQrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BlendPO addresses reinforcement learning with sparse verifiable rewards by controlling the influence of training-only guidance while keeping the guidance text fixed. It interpolates the current policy's next-token logits under original and guided contexts, assigns different interpolation strengths to responses within a GRPO reward group, and trains on those responses under the original context without token-level importance correction. Experiments use abstract mathematical hints and visual-state descriptions, with all learned policies evaluated without guidance. The reported results show improved accuracy over GRPO and several adapted guided-RL baselines, supported by allocation ablations and an estimated training-compute comparison on MATH.

Critical strengths

By interpolating logits under guided and unguided contexts, BlendPO controls guidance strength without modifying the guidance content.

The evaluation covers mathematical reasoning and four visual task families, with improvements over both GRPO and baselines using the same guidance. The main results report means and standard deviations over three training seeds.

The ablations examine the roles of logit interpolation and strength allocation while keeping the guidance content fixed. These controls support the method's effectiveness and indicate that the gains cannot be attributed entirely to access to hints or additional training compute.

Critical weaknesses

1.The within-group sweep does not consistently outperform simpler allocations. The evidence for the full allocation strategy is strongest on Maze8, while its contribution on other tasks remains less clear.

2.The importance-correction ablation uses untruncated weights, whereas OC-GRPO clips its weights to [0.01, 1]. The large degradation may therefore reflect weight variance as well as reduced contributions from useful guided responses.

3.The compute-matched comparison is limited to GRPO on MATH. It does not cover abstract-hint OC-GRPO, 4U+4G, or the visual tasks, where equal training steps need not imply equal cost. Inference costs also differ.

Questions for authors
  1. Under what conditions do the gains from the within-group sweep justify the additional cost of interpolation?

  2. Does bounded importance correction still cause substantial degradation? With BlendPO's sampling scheme held fixed, how does the [0.01, 1] clipping rule used by OC-GRPO compare with no correction and untruncated correction?

  3. How much of the advantage over the strongest baseline using the same guidance remains at comparable cost?

4 comments

Thank you for the thorough review. Following your questions, we added compute-matched comparisons on the visual tasks (Table 18), and updated Tables 14 and 15 with clipped importance correction and Figure 4 with abstract-hint OC-GRPO. We have included these tables and figures in the revised manuscript, and we will update the main text to reflect them and any further discussion.

  • Q1(W1) — Benefits of within-group sweeping over simpler allocations.

Table 12 of the original submission compares the within-group sweep with simpler allocations on MATH, AIME, Geometry3K, and PuzzleVQA.

Beyond its gains on Maze8, the sweep outperforms the best simpler allocation on MATH, AIME 26, Geometry3K, and PuzzleVQA. Moreover, the best simpler allocation shows no consistent pattern across these tasks. For example, fixed α = 0.75 is the best simpler allocation on Geometry3K but stays close to GRPO on MATH and PuzzleVQA, while 4U+4G is the best on PuzzleVQA but not on MATH or Geometry3K. Compared with a fixed α, the sweep also adds no interpolation cost. The sweep therefore offers strong performance across these four tasks without requiring a search over allocations for each task.

  • Q2(W2) — Clipped importance correction analysis.

With BlendPO's sampling scheme held fixed, the [0.01, 1] clipping rule still causes a substantial drop on Maze8, but mitigates the drop on MATH with abstract hints. We added the clipped-correction rows to Tables 14 and 15. Table B1 summarizes the MATH column of Table 14 and the Maze8 results of Table 15.

Clipping improves over untruncated correction on both MATH and Maze8. It thus strikes a balance between limiting weight variance and preserving contributions from useful guided responses. However, with abstract hints and visual states, BlendPO without correction still outperforms the clipped variant on MATH and Maze8.

Table B1. Importance-correction comparison (%)

UpdateMATH, abstract hintMATH, solution prefixMaze8, visual state
w/ correction (untruncated)73.51 ± 0.8372.26 ± 0.7116.34 ± 9.04
w/ clipped correction [0.01, 1]78.35 ± 1.2176.17 ± 1.2536.91 ± 7.15
w/o correction79.40 ± 0.3960.67 ± 3.7069.14 ± 0.90
  • Q3(W3) — Performance comparison under controlled compute budgets.

On MATH, BlendPO remains above abstract-hint OC-GRPO, the strongest baseline using the same hints, at a comparable training budget. An earlier BlendPO checkpoint reaches 78.57 ± 0.65% within GRPO's final mean training-FLOPs budget, compared with 77.12 ± 1.02% for OC-GRPO at a similar budget (Figure 4).

On the visual tasks, BlendPO remains above the baselines and 4U+4G at lower training compute. On Geometry3K, the last checkpoint within the cheapest baseline's budget (step 102) exceeds all listed methods. On PuzzleVQA, the last checkpoint within GRPO's budget (step 28) exceeds GRPO, HINT, and OC-GRPO, and the last checkpoint within 4U+4G's budget (step 36) exceeds 4U+4G. After training, all methods use the same inference procedure, with no additional method-specific computation.

We added the visual-task comparison to the paper as Table 18, and Tables B2 and B3 are identical to its panels (a) and (b). Compute is normalized to the final GRPO checkpoint for each task. We also added abstract-hint OC-GRPO to panels (a) and (b) of Figure 4.

Table B2. Geometry3K

MethodStepComputeAccuracy (%)
GRPO1341.00×56.46 ± 0.58
HINT1340.97×58.02 ± 2.49
OC-GRPO1341.32×59.01 ± 1.50
4U+4G1341.07×61.12 ± 0.82
BlendPO (earlier)1020.86×63.73 ± 2.02
BlendPO (final)1341.14×63.56 ± 0.83

Table B3. PuzzleVQA

MethodStepComputeAccuracy (%)
GRPO401.00×52.08 ± 4.54
HINT401.08×54.42 ± 1.77
OC-GRPO401.39×53.33 ± 1.61
4U+4G401.23×61.25 ± 2.65
BlendPO (earlier)280.94×57.58 ± 1.76
BlendPO (earlier)361.19×62.67 ± 1.15
BlendPO (final)401.31×65.25 ± 1.30

Dear Reviewer 3bQr, thank you again for your helpful feedback. We have posted our response with clipped importance-correction results, compute-matched comparisons, and clarification of the sweep’s benefits. When you have a chance, we would appreciate your thoughts on whether these address your concerns.

Thank you for the additional experiments. The ablation and comparisons well address my concerns. I will raise my rating from 2 to 3.

Thank you for your positive reassessment. We are glad that our additional ablation and comparisons addressed your concerns, and we thank you again for your constructive feedback.

Reviewer_X6RdOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Paper summary: This paper proposes BlendPO, a method that addresses the problem of reinforcement learning with verifiable rewards. Instead of relying on prefix length to change the scale of the guidance, or designing different abstract levels of the hint, BlendPO directly amplifies the level of the hint at the logits level during the rollout phase of RL. More specifically, a weighted linear interpolation of the logits between unguided and guided prediction is implemented to simulate different levels of guidance. Experimental results indicate the strength of the methods compared with different baselines, across different benchmarks.

Paper Contribution:

  1. This paper proposes BlendPO, a logit blending method to address the problem of RLVR.
  2. Experiments show that BlendPO outperforms GRPO and other solution prefix methods.
Critical strengths

Paper strength:

  1. Simple and general design mechanism: This paper proposes blending in logit space to simulate the guidance strength, without requiring extra models or designing different mechanisms for hints.

  2. Experimental results: BlendPO outperforms baselines, including GRPO and prefix methods, on Mathematical reasoning and visual reasoning. The ablation study shows the effect of each design choice in the mechanism.

Critical weaknesses

Below are the weaknesses this paper needs to address:

  1. Ablation study analysis: There are two main components of this paper: the blending weight α\alpha and the within-group sweep. In Figure 3, it seems the within-group sweep is the main improvement between the two components. This can come from different scales within the group generating different success/unsucess response, thus improving the current policy. However, there is one detail worth checking in Figure 3, the setting 6U + 2G underperforms the plain GRPO, which is confusing to me. I suggest conducting another analysis (e.g., counting the number of sucess/unsucess within a group during training) to show where the real improvement of the within-group sweep comes from and why 6U + 2G is bad in this setting.

  2. Generalization of BlendPO across training guidance: Figure 14 suggests that, for solution prefix guidance, BlendPO underperforms other baselines (BREAD, POPE, and OC-GRPO). This means that BlendPO does not generalize well under training guidance. The paper should add analysis to clarify this point. Also, in this setting, the with-correction update is better than the default setting, w/o correction. What can be the explanation for this observation?

  3. Training cost analysis: Compared with the standard GRPO setting, BlendPO requires two generations in the rollout, with and without the hint. Figure 4 only compares FLOPs as the cost with GRPO, and ignores other important metrics such as KV cache size and rollout time.

  4. Comparing with GRPO: In Figure 4, GRPO achieves the peak accuracy before BlendPO (>75% on Math), however, in Table 1, it seems that the comparison with GRPO is conducted at the same checkpoint (<75%). I feel this is not a fair comparison. The results reported in Table 1 with other baselines should come from the highest-accuracy checkpoint of them, not the same number of training steps.

  5. Missing comparison with Privileged distillation for reasoning: The related work includes methods belonging to this direction. Table 1 should also include their performance and further comparison between BlendPO with these methods.

Questions for authors

See Weaknesses section.

6 comments

Thank you for the detailed review. Following your suggestions, we added group-outcome counts (Figures 6 and 7), an analysis of the 6U+2G collapse (Appendix E.1), and a training-cost benchmark (Table 20), and updated Tables 14 and 15 with clipped importance correction and Table 1 with best-checkpoint results and OPSD. We have included these tables and figures in the revised manuscript, and we will update the main text to reflect them and any further discussion.

  • W1 — Analysis of within-group successes and failures during training.

We counted groups in which none, some, or all rollouts succeeded at every training step and added this analysis to the paper (Figures 6 and 7; Appendix E).

Collapse under strong guidance and in 6U+2G. This analysis shows that training collapses under strong guidance (Figure 6). 6U+2G also collapses early in training despite its low mean guidance strength (Figure 7), whereas fixed α=0.25, with the same mean, does not.

Cause of the 6U+2G collapse. Unlike fixed-strength allocation, discrete mixtures generate unguided and guided rollouts from quite different distributions. In mixtures with few guided rollouts, such as 6U+2G, some groups contain no successful rollout. If a guided rollout in such a group earns only the format reward, an incorrect, off-distribution trajectory receives a positive advantage. Reinforcing such incorrect trajectories can lead to collapse, which we also observe in some seeds of 7U+1G and 5U+3G (Figure 9). With more guided rollouts, as in 4U+4G and 2U+6G, more groups contain a success, which reduces the advantage of rollouts that earn only the format reward (Table 21). To examine this, we retrained 6U+2G without the format reward, and it no longer collapses (64.13 ± 1.96%; Table 22).

Within-group sweep components. Neither blending alone (fixed strengths) nor within-group strength variation alone (discrete mixtures) reaches the accuracy of the sweep, so both components appear to contribute. Blending can reduce the distribution gap caused by guidance, and assigning different strengths within a group can help secure a training signal. These properties may contribute to the performance of the sweep.

  • W2 — Interpreting solution-prefix results and the role of importance correction.

Thank you for highlighting this point. The lower solution-prefix accuracy of default BlendPO relative to the baselines is substantially mitigated by importance correction. With the same clipping rule used by OC-GRPO, BlendPO with solution prefixes reaches MATH accuracy comparable to POPE and OC-GRPO (Tables 1 and 14). This suggests that the uncorrected result reflects sensitivity to the update under solution-prefix guidance, rather than a general inability to benefit from this guidance.

With solution-prefix guidance, rollouts are generated with access to a partial solution, while the policy is updated under the original context without that solution. OC-GRPO (Agrawal et al., 2026) reports that this context mismatch can destabilize training and cause collapse when importance correction is omitted. Our low solution-prefix accuracy without correction is consistent with this previously reported issue.

To address this instability, OC-GRPO clips its importance-correction weights to [0.01, 1]. We did the same for BlendPO with all other settings fixed and added the resulting rows to Tables 14 and 15. Table X1 summarizes the MATH column of Table 14 and the Maze8 results of Table 15.

The effect of correction depends on the guidance. Clipped correction helps with solution prefixes. For abstract hints and visual states, where guidance strength cannot be controlled through length, BlendPO without correction is more effective on MATH and Maze8. Importance correction downweights tokens made more likely by guidance, which may also weaken useful learning signals.

Table X1. Importance-correction ablation (%)

UpdateMATH, abstract hintMATH, solution prefixMaze8, visual state
w/ correction (untruncated)73.51 ± 0.8372.26 ± 0.7116.34 ± 9.04
w/ clipped correction [0.01, 1]78.35 ± 1.2176.17 ± 1.2536.91 ± 7.15
w/o correction (default)79.40 ± 0.3960.67 ± 3.7069.14 ± 0.90
  • W3 — Training cost analysis.

Rollout generation takes 1.76× as long as GRPO, and a full step with one update takes 1.68×. The final KV cache is 1.95× as large, but the shared update limits the increase in full-step peak memory to 1.09×. We measured one-step training cost in a controlled setting (Qwen3-1.7B, one H100, 8 prompts × 8 responses, output length fixed at 2,048 tokens, roughly the mean response length) and added the results to the paper as Table 20, which is identical to Table X2 below.

Table X2. Runtime and memory of one training step (Qwen3-1.7B, one H100)

MeasurementGRPOBlendPORatio
Rollout generation (s)92.7 ± 0.1162.8 ± 0.21.76×
Full step: rollout + one update (s)103.2173.31.68×
Final KV cache, padding included (GiB)15.229.61.95×
Peak allocated memory over the full step (GiB)32.135.01.09×
  • W4 — Highest accuracy checkpoint comparison.

Thank you for pointing this out. We revised Table 1 to report every method at its highest-accuracy checkpoint, and BlendPO remains the strongest on MATH and both AIME evaluations. Its MATH accuracy is 79.47 ± 0.30%, compared with 76.26 ± 0.78% for GRPO.

We evaluated the saved checkpoints on MATH, selected the checkpoint with the highest accuracy for each method and seed, and evaluated AIME at that same checkpoint. Baselines with insufficient saved checkpoints were retrained and evaluated.

  • W5 — Comparison with privileged distillation methods.

We added OPSD to Table 1, and Table X3 summarizes its GRPO, OPSD, and BlendPO rows. BlendPO reaches 79.47 ± 0.30% MATH accuracy, versus 71.55 ± 0.43% for OPSD with abstract hints and 71.33 ± 0.65% with reference solutions.

Table X3. Comparison with OPSD, best-checkpoint unguided accuracy (%)

MethodTraining guidanceMATHAIME 24–25AIME 26
GRPONone76.26 ± 0.7818.33 ± 1.4311.25 ± 0.91
OPSDSolution71.33 ± 0.6512.95 ± 1.6910.00 ± 1.10
OPSDAbstract hint71.55 ± 0.4313.51 ± 0.329.31 ± 0.48
BlendPOAbstract hint79.47 ± 0.3022.12 ± 1.4115.90 ± 0.12

Dear Reviewer X6Rd, thank you again for your detailed feedback. We have posted our responses with additional analyses, runtime and memory measurements, best-checkpoint comparisons, and an OPSD baseline. We would appreciate it if you could take a look when convenient, and would be happy to clarify any remaining questions.

Reviewer_PDJTOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

BlendPO samples rollouts by blending policy logits with and without training-only guidance at different strengths within each group, then trains the policy under the unguided context; it is evaluated on mathematical and visual reasoning tasks.

Critical strengths
  • The method offers a simple way to vary guidance strength without rewriting the hint or visual state.

  • The evaluation includes controlled strength comparisons and strong gains on several visual tasks, as well as improvements over relevant mathematics baselines.

Critical weaknesses
  • The claim on "Stronger guidance makes successful rollouts more likely but can transfer less to the unguided setting used at test time, making guidance strength a key design choice" needs some evidence or analysis.

  • The proposed trade-off is not measured directly: the paper compares final unguided accuracy across guidance strengths, but does not systematically report how rollout success and the number of informative reward groups change with strength.

  • Mathematical hints require verified reference solutions and a large model to generate them. Visual states avoid reference solutions, but their availability and acquisition cost in practical settings remain unclear.

  • The within-group sweep is not consistently better than a fixed weak strength on the mathematics benchmarks, where its gains over the strongest same-hint baseline are modest. This limits the evidence that group-level variation is necessary across tasks.

Questions for authors
  • Can you report rollout success rates and the fraction of groups with nonzero reward variation at each guidance strength, alongside final unguided accuracy, to test the proposed success-transfer trade-off directly? Also, this result can support the claim on "Stronger guidance makes successful rollouts more likely but can transfer less to the unguided setting used at test time, making guidance strength a key design choice".

  • What does it cost to construct the mathematical hints and visual states? How does performance change with cheaper or less informative guidance?

  • Under a matched compute budget, when does the within-group sweep outperform a well-chosen fixed weak strength, particularly outside Maze8? This result can support the necessity of the group-level guidance variation.

4 comments

Thank you for the careful review. Following your questions, we measured the success–transfer trade-off directly (Table 19, Figure 6) and tested less informative guidance (Figure 8). We have included these tables and figures in the revised manuscript, and we will update the main text to reflect them and any further discussion.

  • Q1(W1, W2) — Success–transfer analysis.

We analyzed initial rollout success, the proportion of groups containing both successful and unsuccessful rollouts, and final unguided accuracy across guidance strengths. We added these results to the paper as Table 19, which is identical to Table P1 below. Stronger guidance increased initial rollout success, but did not consistently improve final unguided accuracy. This pattern is consistent with HINT’s discussion of shortcut learning (Wang et al., 2026, Figs. 1–2). When guidance provides complete or partial reference solutions, rollout success can improve, but these gains do not necessarily transfer to unguided evaluation.

We also added Figure 6, which plots the proportions of groups with no successful rollouts, mixed successes and failures, or all successful rollouts across training steps. At high guidance strengths, the proportion of groups with no successful rollouts was initially low. However, this proportion increased as training progressed.

Table P1. Initial success, mixed-success groups, and final unguided accuracy across Maze8 guidance strengths (%)

Guidance strength allocationInitial success (%)Proportion of groups with mixed success (%)Final unguided accuracy (%)
α=04.69 ± 4.0811.46 ± 4.7716.15 ± 5.47
α=0.2515.36 ± 2.9343.75 ± 9.3841.08 ± 16.93
α=0.547.92 ± 7.4360.42 ± 9.5518.23 ± 6.76
α=0.7553.13 ± 8.1355.21 ± 1.8015.69 ± 3.22
α=155.99 ± 3.9164.58 ± 6.516.84 ± 1.92
Within-group sweep33.46 ± 4.1075.00 ± 3.1369.14 ± 0.90
  • Q2(W3) — Guidance acquisition cost and effectiveness of less informative guidance.

In our experiments, visual states were obtained only from the annotations provided with the datasets. Abstract hints for the DAPO-Math-17k subset used in training were generated using Qwen3.8-27B, at a generation cost of 0.27 H100 GPU-hours (about 16 min).

BlendPO outperformed the baselines even with incomplete or extraneous information in the visual states. In practical settings, visual states may provide only limited information about a task or contain incorrect or irrelevant details. The visual states in our experiments also have such artifacts. Some PuzzleVQA captions omit relevant visual details, while some Geometry3K annotations and MindCube descriptions include information unavailable from the original input (Figure 5). We further analyzed less informative guidance on Maze8 by independently removing each player, goal, or hole position from the symbolic state with a given probability. We added this analysis to the paper as Figure 8, and Table P2 lists the values plotted in that figure. The final unguided success rate decreases as more information is removed, but remains higher than GRPO on average, even when only about 20% of the position entries are retained.

Table P2. Maze8 guidance-information removal (%, three-seed mean ± sample SD)

RemovalFinal unguided success rate (%)
0% (BlendPO)69.14 ± 0.90
20%58.59 ± 2.20
40%45.25 ± 2.65
60%42.64 ± 3.18
80%21.81 ± 4.80
No additional information (GRPO)16.15 ± 5.47
  • Q3(W4) — Benefits of within-group sweeping over fixed-strength allocation.

Beyond its gains on Maze8, the sweep outperforms the best tested fixed ratio on MATH, AIME 26, Geometry3K, and PuzzleVQA at the same per-token computational cost. Table 12 of the original submission already compares fixed ratios with the within-group sweep on these tasks. Moreover, the best fixed α shows no consistent pattern across these tasks, taking values of 0.25, 0.5, 0.75, and 0.25, respectively. The sweep therefore offers strong performance across these four tasks without requiring task-specific ratio tuning.

Within-group sweeping has the same per-token computational cost as fixed-ratio blending because both use forward passes under the original and guided contexts. The sweep changes only the mixing coefficient assigned to each response.

Dear Reviewer PDJT, thank you again for your thoughtful feedback. We have posted our response and added analyses of the success–transfer trade-off, guidance cost and quality, and within-group sweeping. We would appreciate your thoughts on whether these address your concerns.

Thank you for the additional experiments. My Q1 is addressed by the success–transfer analysis and my Q2 is addressed by the guidance-cost and information-removal results. Also, I find the within-group blending convincing as a robust default that reduces task-specific tuning. More importantly, the technical challenge is tackling the RLVR with sparse reward, which is a significant problem for LLM self-improvement. Overall, I will raise my rating.

For Q3, I would still encourage comparisons with the best fixed weak guidance under matched total training compute, particularly on mathematics tasks where the gains are modest or inconsistent.

Thank you for your positive reassessment and constructive suggestion.

We compared the within-group sweep with fixed α=0.25, the best fixed weak strength on MATH in Table 12, under matched training compute (Figure 10, Appendix E.2). For this comparison, we retrained fixed α=0.25 with three seeds to save the same ten checkpoints as in Figure 4. Its final accuracy is 78.30 ± 0.75%, within one standard deviation of the 78.84 ± 0.81% in Table 12.

The two methods are within seed variation of each other at the first two checkpoints, and the sweep has higher mean accuracy at matched estimated FLOPs from the third checkpoint onward.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.