acceptodds
Under review as a conference paper at ICLR 2027

Let Intuition Teach Deliberation: Rewarding Prefix Success Potential in Reasoning Models

Abstract

Two reasoning trajectories can reach the same answer while differing in whether a correct short answer was already available halfway through. We use this distinction to let verified short-answer probes supervise long-form deliberation. Prefix Success Potential (PSP) measures the probability of producing a correct short answer from a prefix under a fixed protocol. The same policy samples auxiliary answers, a verifier scores them, and their success rates reward the complete original trajectory. Our variant PSP-ME (ME: midpoint and endpoint) favors higher midpoint potential at equal endpoint potential. Probes serve only for scoring; no external teacher or process annotations are required. We train with an 8,192-token budget and evaluate at 32,768/38,912 tokens. Across Qwen3-1.7B, Qwen3-8B, Gemma4-E4B-it, and Gemma4-12B-it, PSP-ME achieves Math-7 scores of 51.06%, 68.70%, 65.56%, and 79.93% respectively. These represent gains of 3.89, 4.45, 5.55, and 6.56 percentage points over matched outcome-reward GRPO. Fixed-prefix evaluation reveals two key findings. First, RL training dissociates answer potential from natural completion: Gemma4-12B-it improves potential by 14.32 pp while completion drops by approximately 29 pp, suggesting that PSP teaches models to maintain extractable reasoning states rather than forcing earlier generation. Second, gains transfer heterogeneously through different mechanisms: Gemma models improve short-answer potential while Qwen3-8B does not, yet both achieve substantial long-budget gains, demonstrating that midpoint supervision enhances reasoning quality through multiple pathways rather than a single uniform mechanism.

Primary area:
reinforcement learning
Keywords:
large language models, mathematical reasoning, reinforcement learning, reward design, prefix success potential, group relative policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_NdodOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Prefix Success Potential (PSP), which rewards the original inference trajectory based on the frequency with which the model generates correct short answers from inference prefixes. The main configuration samples two short answers each at the midpoint and endpoint of the trajectory, and the average validation pass rate is used for GRPO updates, requiring no external teacher or process annotation. The long-budget Math-7 scores of four Qwen/Gemma models are improved by 3.89–6.56 percentage points relative to GRPO. The paper further investigates the relationship between probe location, reward discrimination, computational cost, and the relationship between "extractable answers" and naturally completed solutions.

Critical strengths

Simple reward constructs yielded consistent gains across multiple models. All four models outperformed their corresponding GRPO models in long-budget Math-7. Gemma4-E4B-it achieved a 65.56% improvement, a 9.17 percentage point increase over the pre-training model, demonstrating a clear net capability improvement. Single-endpoint re-answers also improved Qwen3-1.7B, indicating that the gains cannot be solely attributed to finer reward values.

The separation between answer potential and natural output provides valuable diagnostic evidence. In the 8K prefix evaluation, Gemma4-12B-it showed a 14.32 percentage point improvement in short answer potential relative to GRPO, but its natural output accuracy decreased from 70.93% to 41.94%. Similar phenomena were observed in the Clean set using the same validators, thus not simply attributable to validator differences. This finding reveals a noteworthy deviation between training rewards and actual answering behavior.

Critical weaknesses

The main results do not yet separate the effects of baseline degradation and increased inference length. The average test output length of PSP is 1.79–2.74 times that of GRPO, while the natural output accuracy within the 8K prefix is ​​lower on all four models. GRPO for Qwen3-8B and Gemma4-12B-it degrades by 5.33 and 8.46 percentage points from initialization, respectively, and PSP also remains below initialization. Therefore, some gains may reflect mitigation of the long-budget degradation caused by short-budget training, and are insufficient to establish a net gain at comparable inference costs for a relatively well-tuned baseline. The low training cost configuration in Table 19 is a valuable counter-evidence, but test costs are not controlled. (Tables 3, 11, 17, 19)

The incremental value of the core design relative to existing prefix probe reward methods has not been fully established. SPAE has utilized intermediate answer probes to construct process potential signals, and S-GRPO has utilized answer correctness at multiple intermediate positions to provide rewards. This paper uses absolute potential averaging to update the entire trajectory, but lacks direct comparisons with these most relevant methods. The positional assignment comparison with equal probe counts only covers Qwen3-1.7B; in both large models, the gain interval of the midpoint plus endpoint relative to the endpoint only contains zeros, and there is only one training seed. Therefore, support for the general effectiveness of the new design remains limited.

Questions for authors

Baseline Clarification and Necessary Evidence: What are the main sources of the long budget degradation in GRPO for the two large models? After parameter tuning on the validation set and controlling the actual number of generated tokens or inference costs, does the advantage of PSP over GRPO and the initial model still exist?

Supplementary Evidence Influencing Contribution Judgment: Are there comparable results with SPAE or S-GRPO, and midpoint/endpoint ablation with equal probe counts and multiple training seeds, which can confirm the stable gains of the core design beyond typical re-answer supervision?

Mechanism Clarification: How to distinguish between improved prefix inference quality and model better adaptation to specified answer hint formats? After using a fixed answer reading model or multiple hint formats, does the potential improvement of Gemma remain?

4 comments
AuthorsOpenReview ↗

Response to Ndod (1/2): Stronger Baselines and Accuracy–Cost Tradeoffs

We thank Reviewer Ndod for the detailed review and the well-structured questions.

Your three questions provided a clear structure for the additional experiments. The results address each directly: ME's long-budget advantage over truncation-aware baselines holds in both seeds (filtering: +4.20/+3.23 pp, intervals [3.22, 5.22]/[2.10, 4.41]; DAPO: +6.17/+2.90 pp; S-GRPO: +0.45/+1.76 pp); midpoint allocation shows a consistent benefit over endpoint-only on Qwen3-1.7B when probe counts are matched (ME over E4: +2.11/+0.99 pp across both seeds); and ME prefixes improve answer extraction by an unchanged Initial reader under the Boxed protocol (+5.69 pp at 4K, +3.78 pp at 8K).

Q1–Q3 and W1–W2 follow the order of your questions and weaknesses. K = 1,024 tokens.

1. Stronger baselines and actual inference costs [Q1 & W1]

Preserving long-budget capability is an important training objective when rollout budgets are constrained. e3 (Setlur et al., ICLR 2026, Section 6) studies how restrictive training budgets can discourage exploration needed for longer-budget reasoning. Our independent evaluations show a corresponding pattern: after 8K-rollout training, Qwen3-1.7B GRPO improves 8K accuracy from 33.42% to 40.15%, while reducing 32K accuracy from 49.87% to 46.85%. ME reaches 50.19% at 32K in the two-seed average, preserving Initial-level performance.

On Qwen3-1.7B, we added GRPO + overlong filtering, DAPO, and S-GRPO using the same initialization, training data, 8K rollout cap, and 200 updates, each with seeds 42/43. Filtering masks policy/KL/entropy losses for length-stopped responses while retaining their rewards in group normalization. DAPO incorporates dynamic group resampling, asymmetric clipping, and a soft overlong penalty. The independent budget results are:

Method8K accuracy16K accuracy32K accuracyActual mean tokens at 32K
Initial33.4245.0049.8715,102
GRPO40.1545.7146.859,961
GRPO + filtering39.4345.2646.4710,308
DAPO40.5345.2145.658,762
S-GRPO35.4645.1149.2414,375
PSP-E236.6246.0649.4513,152
PSP-ME28.4843.2950.1916,190

Accuracies are percentages; trained rows average seeds 42/43. Math-7 averages responses within each question and weights the seven datasets equally. Actual token costs include all responses, including incorrect and truncated outputs.

ME exceeds filtering at 32K by 4.20/3.23 points, with paired-question 95% intervals [3.22, 5.22]/[2.10, 4.41]. E2 achieves the highest 16K mean, while ME achieves the highest 32K mean. ME’s 32K result uses 7.2% more tokens than Initial, 12.6% more than S-GRPO, and 57.1% more than filtering. At 8K/16K, ME has lower accuracy and higher token use than GRPO and S-GRPO. These measurements establish the budget-dependent accuracy–cost tradeoff; superiority at matched actual inference cost remains unresolved.

Larger-model baselines [Q1 & W1]. In the seed42 comparisons, the Qwen3-8B filtering baseline reaches 64.84%, versus GRPO’s 64.24% and ME’s 68.70%, under the original mixed test caps. ME exceeds filtering by 3.85 points [2.76, 4.99], with actual mean lengths of 15,356 versus 9,023 tokens. Qwen3-8B and Gemma4-12B-it remain below Initial after PSP training: 68.70% versus 69.57%, and 79.93% versus 81.83%. Gemma4-E4B-it improves from 56.39% to 65.56% (+9.17 points). The short-training/long-testing mismatch provides a plausible explanation for degradation; its contribution relative to optimization and stopping behavior remains to be isolated.

Configuration checks [Q1 & W1]. In a seed42 KL ablation under the original mixed test protocol, GRPO scores 47.17% with KL and 45.79% without. A historical 16K-rollout GRPO checkpoint reaches 49.31% under that protocol, using a different training implementation. Our configuration checks cover KL and the added baseline recipes; a validation-selected learning-rate/clipping sweep remains open.

AuthorsOpenReview ↗

Response to Ndod (2/2): S-GRPO, Probe Ablations, and Fixed-Reader Evidence

2. S-GRPO and equal-probe, two-seed ablations [Q2 & W2]

We implemented S-GRPO in the shared Qwen3-1.7B setup, using one original trajectory and eight intermediate exits, with decaying rewards favoring earlier correct answers. Evaluation uses natural generation. PSP averages verified success at fixed prefix positions into one reward for the original trajectory. Under the original protocol—32,768 tokens for GSM8K/MATH500 and 38,912 for competition mathematics—the Math-7 accuracies (%) are:

MethodSeed 42Seed 43
GRPO47.1746.85
GRPO + filtering46.6445.94
DAPO44.7746.27
S-GRPO50.6147.52
Endpoint-149.5249.51
PSP-E249.8049.08
Endpoint-448.9548.29
PSP-ME51.0649.28

Initial scores 50.26% under this protocol. ME exceeds S-GRPO by 0.45/1.76 points, with intervals [−0.27, 1.15]/[0.63, 2.90]. ME uses 11.6%/12.7% more actual output tokens. The seed42 interval includes zero; the seed43 interval is positive.

Midpoint allocation [Q2 & W2]. With four probes in both methods, ME assigns two to the midpoint and two to the endpoint, while E4 assigns all four to the endpoint. ME improves over E4 by 2.11/0.99 points, with intervals [1.20, 3.04]/[0.16, 1.83]. This positional benefit appears in both training seeds. Gains over E2 are 1.26/0.20 points, and Endpoint-1 remains competitive, slightly exceeding ME in seed43. On the two larger models, midpoint-versus-endpoint intervals include zero.

Relation to SPAE [Q2 & W2]. SPAE combines intermediate confidence and correctness into step potential and shapes credit using potential changes and saturation penalties. PSP uses absolute verified success rates to assign a trajectory-level reward. A direct SPAE experiment remains open.

Training costs [Q1 & W1]. Mean training-step hours on four H100 GPUs are 8.52/8.28/17.24/10.09/9.99/10.92 for GRPO/filtering/DAPO/S-GRPO/E2/ME. ME takes 36.7% less time than DAPO. Timing sums 200 updates and includes masked responses, discarded candidates, intermediate exits, and probes; startup, post-training export, and failed attempts are excluded.

3. Fixed reader and answer formats [Q3]

Yes, with the Boxed protocol, the improvement over Initial remains when the reader is held fixed. We evaluated Gemma4-12B-it on 256 Clean questions, with four saved trajectories per generator, prefixes at 4K/8K, and four probes of at most 256 tokens per prefix. For each generator, the same saved prefixes are reused across reader and prompt conditions. The table reports forced-answer success rates (%):

Prefix generator / reader4K Full4K Boxed8K Full8K Boxed
Initial / Initial32.5253.7825.2762.11
GRPO / Initial40.7560.1141.3363.16
ME / Initial31.9359.4721.9765.89
GRPO / GRPO32.5453.4432.8955.08
ME / ME17.6360.578.5966.65

With the common Initial reader and Boxed cue, ME prefixes improve over Initial prefixes by 5.69 points at 4K [3.03, 8.42] and 3.78 points at 8K [1.25, 6.25]. The 8K ME result is 65.89% with the Initial reader and 66.65% with its own reader. This provides direct evidence that the trained prefixes support improved answer extraction by an unchanged reader under the Boxed protocol.

Prompt dependence [Q3]. The two prompt formats give different outcomes. With the Initial reader and Full cue, ME scores 21.97% versus Initial’s 25.27%, a difference of −3.30 points [−5.52, −1.15]. Under Boxed, ME exceeds GRPO by 2.73 points [−0.15, 5.62] at 8K. The fixed-reader gain over Initial is specific to the Boxed protocol.

AuthorsOpenReview ↗

Revised manuscript: strong baselines, positional ablations, and fixed readers

Thank you again. We have uploaded a revised PDF. Selected additions are highlighted in blue and marked NEW. Following your three questions:

  • Strong baselines and inference costs [Q1 & W1]. Section 5.1 and Table 2 report the two-seed Qwen3-1.7B controls, including filtering and DAPO. Section 5.4 and Figure 3 show the independent 8K/16K/32K accuracy and actual-token curves. Table 2 and Appendix G.6 (Table 37) give measured training costs for all six methods.
  • S-GRPO and equal-probe, two-seed ablations [Q2 & W2]. Section 5.2 presents the direct S-GRPO comparison. Table 3 reports both seeds of the positional ablation, including the equal-four-probe midpoint/endpoint comparison; Appendix G.2 gives the original mixed-budget results and paired-question intervals. Section 2 clarifies the methodological relationship to SPAE.
  • Fixed reader and answer format [Q3]. Section 5.3 and Table 5 hold the reader fixed to Initial and compare Full and Boxed extraction from trained Gemma4-12B-it prefixes. Boxed extraction improves on PSP-ME prefixes. Appendix G.4 reports all 20 reader/budget/cue conditions and the corresponding contrasts.

These additions incorporate the evidence from our detailed response into the revised manuscript.

Thank you again for your review. Two additions since our previous response relate to your questions.

Strong baselines on all four models [Q1 & W1]. Table 1 adds DAPO on Qwen3-8B and overlong filtering and DAPO on both Gemma models. PSP-ME exceeds the strongest of GRPO, filtering and DAPO by 3.38–5.91 pp on every model, while generating 1.70–2.45 times as many output tokens. Appendix G.7 reports accuracy, actual mean output tokens and paired-question 95% intervals for every row.

S-GRPO on a second model family [Q2 & W2]. Appendix G.9 adds Gemma4-E4B-it, where S-GRPO reaches 60.11% and PSP-ME 65.56%: a difference of 5.45 pp [3.89, 7.01], with 6,159 versus 12,621 mean output tokens. Its non-mathematical results are in the same appendix. This is a seed-42, step-200 comparison under the original mixed test caps, with intervals conditional on the evaluated checkpoints.

The abstract and Limitations now state that these gains come with longer generations and that matched-cost superiority remains open.

Reviewer_wLhxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Prefix Success Potential (PSP), a trajectory-level reward based on verified short answers sampled from reasoning prefixes. The main variant probes the midpoint and endpoint, averages their success rates, and uses this score in GRPO. Across four models, PSP improves long-budget mathematical accuracy over outcome-reward GRPO. Ablations examine re-answering and probe placement, while fixed-prefix evaluations distinguish forced-answer success from natural-answer correctness.

Critical strengths
  1. PSP provides a simple way to obtain additional reward information without an external teacher, process labels, or a learned critic. Gains over the reported GRPO baseline are consistent across four models. The equal-probe-count ablation on Qwen3-1.7B also supports a benefit from midpoint information beyond endpoint re-answering.
  2. The paper reports important negative results, including longer outputs, worse within-budget accuracy, and performance below initialization. Its fixed-prefix diagnostics reveal a useful distinction between extracting a correct answer and naturally producing one within a budget.
Critical weaknesses
  1. GRPO underperforms initialization on three of four models. PSP remains below initialization on both larger models and improves Qwen3-1.7B by only 0.8 points. Gemma4-E4B-it shows a substantial absolute gain, but stronger outcome-reward baselines are needed to establish how broadly the benefit holds.

  2. PSP can reward unfinished trajectories through short-answer probes, whereas an outcome reward may penalize trajectories that fail to produce a correct answer before the training cap. PSP also generates 79–174% longer responses and has lower natural-answer accuracy at 8K on all four models. These observations are consistent with a change in stopping and truncation incentives, rather than necessarily better reasoning supervision. The additional midpoint benefit is not statistically distinguishable from zero on either larger model. Truncation-aware baselines and accuracy–cost comparisons are therefore central to assessing the contribution.

Questions for authors
  1. How are truncated rollouts handled in the GRPO loss and reward computation? How frequently does truncation occur under each method, and what rewards do truncated PSP trajectories receive compared with completed but incorrect trajectories?
  2. Does PSP retain its advantage against GRPO with overlong-sample masking or another appropriate truncation treatment? This would directly test whether the gains mainly address a short-cap training failure.
  3. Can you compare Initial, GRPO, and PSP across inference budgets and at comparable actual token costs? Together with a stronger GRPO configuration, this would establish whether PSP improves the accuracy–cost tradeoff.
5 comments
AuthorsOpenReview ↗

Response to wLhx (1/2): Truncated Rollouts and Filtering Controls

We thank Reviewer wLhx for the specific questions and careful attention to the method's mechanism.

Your three questions go directly to the method's mechanism. The experiments answer each: truncated ME trajectories receive positive rewards when their prefixes contain extractable answers — 46.55% of length-stopped responses in the Initial policy receive positive ME rewards; PSP's long-budget advantage persists over explicit overlong filtering (+4.41/+3.34 pp, intervals [3.36, 5.48]/[2.34, 4.38]); and the budget sweep places ME's accuracy gain alongside a measured inference cost — 7.2% more tokens than Initial at 32K, 57.1% more than filtering. The advantage over truncation-aware baselines holds in both training seeds.

Q1–Q3 and W1–W2 follow the numbering in your review. K = 1,024 tokens.

1. Scoring and training truncated trajectories [Q1 & W2]

In the original GRPO and PSP experiments, truncated responses retain their original response-token loss masks. GRPO assigns reward 1 when the original answer passes verification and 0 otherwise, including missing or incorrect answers, without an additional truncation penalty. PSP scores verified short-answer probes from the selected prefixes. For ME, the reward averages the midpoint and endpoint success rates, using two probes per position. An unfinished response can therefore receive a positive reward when its prefixes support correct short answers.

For each question, all eight response scores enter the group mean and sample standard deviation. The resulting scalar advantage weights the original response tokens; probe prompts and probe outputs receive no policy-gradient loss. A positive PSP score leads to a positive advantage only when it exceeds the group mean. Completed but incorrect responses are scored by the same probe procedure.

We illustrate this distinction by rescoring the same 512 trajectories from the frozen Initial Qwen3-1.7B policy, grouped using their recorded stopping reasons:

Trajectory categoryCountMean outcome rewardMean ME rewardMean ME advantage
Length stopped3330.0000.321−0.069
Completed, correct1671.0000.811+0.178
Completed, incorrect120.0000.000−0.571

Among length-stopped responses, 46.55% receive positive ME rewards and 34.53% receive positive ME advantages. These measurements demonstrate the intended supervision: trajectories with extractable correct answers can receive learning signal before natural completion.

For the seed42 trained policies, the fraction of responses reaching the 8,192-token cap during the final 20 updates is:

ModelGRPOPSP-E2PSP-ME
Qwen3-1.7B33.67%58.26%79.18%
Qwen3-8B21.09%51.13%73.32%
Gemma4-E4B-it10.59%23.65%70.76%
Gemma4-12B-it3.81%33.16%53.83%

Here, cap hit means that the recorded response length equals 8,192 tokens. The preceding diagnostic table uses explicit length-stop reasons.

2. Overlong filtering and stronger baselines [Q2 & W1 & W2]

Yes. We trained GRPO + overlong filtering with seeds 42/43, matching GRPO’s initialization, data, sampling, optimizer, KL coefficient, 8K rollout cap, and 200 updates. The only change is that responses stopped by the length limit receive zero policy/KL/entropy loss. Their rewards remain in the eight-response group normalization. Normally terminated responses retain their losses; entirely masked groups contribute no response loss. This control uses no dynamic resampling or soft length penalty.

Under the original test protocol—32,768 tokens for GSM8K/MATH500 and 38,912 for competition mathematics—the Math-7 accuracies (%) are:

MethodSeed 42Seed 43Mean ± sample SD
Initial reference——50.26
GRPO47.1746.8547.01 ± 0.23
GRPO + filtering46.6445.9446.29 ± 0.50
DAPO44.7746.2745.52 ± 1.06
S-GRPO50.6147.5249.06 ± 2.18
PSP-E249.8049.0849.44 ± 0.51
PSP-ME51.0649.2850.17 ± 1.26

ME exceeds filtering by 4.41/3.34 percentage points, with paired-question 95% confidence intervals [3.36, 5.48]/[2.34, 4.38]. We also added DAPO, incorporating dynamic group resampling, asymmetric clipping, and a soft overlong penalty; ME exceeds it by 6.29/3.01 points. The long-budget benefit therefore persists with explicit truncation handling in the outcome-reward baselines.

AuthorsOpenReview ↗

Response to wLhx (2/2): Model Comparisons and Accuracy–Cost Tradeoffs

Larger-model baselines and Initial comparisons [Q2 & W1]. In the seed42 comparisons under these mixed caps, the Qwen3-8B filtering control reaches 64.84%, compared with GRPO’s 64.24% and ME’s 68.70%. ME exceeds filtering by 3.85 points [2.76, 4.99]. Relative to initialization, Gemma4-E4B-it improves from 56.39% to 65.56% (+9.17 points). Qwen3-8B and Gemma4-12B-it recover substantial performance over GRPO while remaining below their Initial scores: 68.70% versus 69.57%, and 79.93% versus 81.83%, respectively.

Midpoint allocation [W2]. On Qwen3-1.7B, the equal-four-probe comparison gives ME gains over endpoint-only E4 of 2.11/0.99 points across seeds, with intervals [1.20, 3.04]/[0.16, 1.83]. The midpoint-versus-endpoint intervals on the two larger models include zero. The incremental benefit of midpoint probes is model-dependent.

3. Accuracy and actual inference costs [Q3 & W1 & W2]

We evaluated Initial and all six trained methods under independent 8K, 16K, and 32K generation caps. Each cell reports Math-7 accuracy (%) / actual mean generated tokens. Trained rows average seeds 42/43.

Method8K cap16K cap32K cap
Initial33.42 / 6,78545.00 / 11,43249.87 / 15,102
GRPO40.15 / 5,95045.71 / 8,98646.85 / 9,961
GRPO + filtering39.43 / 6,01845.26 / 9,18046.47 / 10,308
DAPO40.53 / 5,74045.21 / 8,19145.65 / 8,762
S-GRPO35.46 / 6,58545.11 / 10,94849.24 / 14,375
PSP-E236.62 / 6,54246.06 / 10,65649.45 / 13,152
PSP-ME28.48 / 7,14343.29 / 12,26150.19 / 16,190

All methods use the same evaluation questions, repetitions, prompts, sampling settings, and verifiers. Math-7 averages within questions and weights the seven datasets equally. Token costs include every generated response, including incorrect and truncated outputs.

Long-budget preservation [Q3 & W1]. GRPO improves 8K accuracy over Initial while reducing 32K accuracy by 3.02 points. ME retains Initial-level 32K accuracy in the two-seed average. This budget-dependent pattern connects to the challenge studied in e3 (Setlur et al., ICLR 2026, Section 6): restrictive rollout budgets can discourage exploration needed for longer-budget reasoning. Our results show that prefix rewards can preserve this capability while keeping the training rollout cap at 8K.

Accuracy–cost tradeoff [Q3 & W2]. E2 achieves the highest 16K mean, and ME achieves the highest 32K mean. At 32K, ME uses approximately 7.2% more tokens than Initial and 57.1% more than filtering. At 8K/16K, ME has lower accuracy and higher actual token use than GRPO. On Qwen3-8B under the original mixed caps, ME averages 15,356 tokens versus filtering’s 9,023. These results quantify the additional inference cost accompanying the long-budget gains; superiority at matched actual token cost remains unresolved.

Training cost [W1]. Mean training-step hours on four H100 GPUs are 8.52/8.28/17.24/10.09/9.99/10.92 for GRPO/filtering/DAPO/S-GRPO/E2/ME, respectively. ME takes 36.7% less training-step time than DAPO. These measurements sum all 200 updates, including auxiliary generation and discarded or masked responses; startup, post-training export, and failed attempts are excluded.

AuthorsOpenReview ↗

Revised manuscript: truncation handling and accuracy–cost results

Thank you again. We have uploaded a revised PDF. Selected additions are highlighted in blue and marked NEW. Following your three questions:

  • Truncated-rollout scoring and training [Q1 & W2]. Section 4.3 states that original GRPO and PSP retain truncated response tokens in the loss, all eight scores enter group normalization, and an unfinished trajectory with successful probes can receive positive advantage. Appendix G.1 specifies the filtering loss mask. Training cap-hit statistics are in Appendix D.1 (Table 17); the separate frozen-trajectory comparison of length-stopped, completed-correct, and completed-incorrect responses is in Appendix G.8 (Table 42).
  • Truncation-aware controls [Q2 & W1, W2]. Section 5.1 and Table 2 add the two-seed GRPO + overlong-filtering and DAPO controls. Appendix G.2 reports the original mixed-budget scores and paired-question intervals supporting the long-budget gains.
  • Initial, GRPO, and PSP across inference budgets [Q3 & W2]. Section 5.4 and Figure 3 compare accuracy and actual mean output tokens at independent 8K/16K/32K caps, including Initial and all six training methods. Table 2 and Appendix G.6 report the corresponding training costs.

These locations contain the experimental evidence discussed in our detailed response.

Reviewer_VnK8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

So the idea is to get GRPO learning from more than just the final answer. With a right/wrong reward, if all the responses in a group are right, or all wrong, there’s no signal, and that’s a lot of the data. Their fix is to cut the trace partway and again at the end, ask the same model for a quick answer from there, and how often that’s correct becomes the reward for the full response. Nice part is the data doesn’t change, just the scoring, and no teacher or labels.

They train on a math set with a short token limit, then test with a much longer one, and it beats plain GRPO on all the Qwen and Gemma models they tried. There’s also this split where models get better at quick answers from a cut-off trace but finish on their own less often, though not in the same way across the two families. It’s one seed per setup and responses get longer, so that’s the first thing I’d look at with these gains.

Critical strengths
  • The problem they pick is real. When every response in a group gets the same score, GRPO learns nothing from that group. Their reward separates those responses without needing new data, and they show this by rescoring the same responses and counting how many more groups become useful. This is the main reason I think the idea is worth taking seriously.

  • They check two things on the same responses: can the model give a correct short answer when cut off, and does it finish correctly on its own. The two don’t always move together, and that was the most useful finding for me. It means a good forced answer from a cut-off response doesn’t tell you the model would have got there by itself.

Critical weaknesses
  • I can’t tell if the gains come from the reward or just from longer answers. They train with a short token limit but test with a much longer one, and the models trained this way write a lot more than GRPO. When you look at the short limit they trained on, they actually do worse than GRPO on all four models. Without a GRPO baseline matched on length, I don’t know how much of the improvement is better reasoning and how much is the model simply thinking longer. This is the main reason for my score.

  • The baseline is weak and there’s only one seed. GRPO ends up worse than the model they started from on most of the models, and the new method is still below the starting model on two of them. So a good part of the gain looks like recovering what GRPO lost, not going beyond where things started. With one training seed and gains of a few points, I can’t tell how much is real and how much is run-to-run noise.

Questions for authors
  • Can you add a GRPO baseline that produces responses about as long as yours, for example with a length penalty or a matched generation budget? If the gap mostly disappears, that changes how I read the paper.
  • Why does GRPO end up below the starting model? Did you try tuning it (learning rate, KL, clipping), and does better-tuned GRPO close the gap?
  • Can you show results over a few seeds, at least on Qwen3-1.7B, so we can tell whether the gains are bigger than run-to-run noise?
  • Qwen3-8B gets better at long budgets without getting better at short answers from a cut-off response. What do you think the reward is actually changing there?

Suggestion (future work): compare against VinePPO or DAPO at matched compute, and try a task outside math.

5 comments
AuthorsOpenReview ↗

Response to VnK8 (1/2): Long-Budget Capability and Inference Costs

We thank Reviewer VnK8 for the constructive feedback and for identifying the core interpretive challenge.

Your central question — whether the gains reflect better reasoning or simply longer generation — is the right one to ask. We cannot yet provide a length-matched GRPO baseline, so the entanglement with generation length remains partly unresolved. What the new experiments do show: ME's advantage holds over filtering and DAPO, both of which address truncation while generating shorter responses; two training seeds confirm the gains are consistent; and the budget sweep quantifies where ME's accuracy advantage arises and what it costs. The central result is that verified prefix rewards preserve long-budget reasoning performance after short-rollout training, with improvements beyond initialization in some settings.

Q1–Q4 and W1–W2 follow the order of your questions and weaknesses; F1 denotes your future-work suggestion. K = 1,024 tokens.

1. Preserving long-budget capability [Q2 & W2]

You are right that ME's long-budget gains accompany longer generation — Section 2 quantifies this tradeoff across all budgets. The question is whether generation length explains the full gap. e3 (Setlur et al., ICLR 2026, Section 6) shows that restrictive training budgets can discourage exploration; filtering and DAPO both address truncation incentives while keeping generation short, yet both fall below ME at 32K.

Our independent budget evaluations make the tradeoff explicit. After 8K-rollout training, Qwen3-1.7B GRPO improves 8K accuracy from 33.42% to 40.15%, while reducing 32K accuracy from 49.87% to 46.85%. PSP-ME reaches 50.19% at 32K in the two-seed average, preserving Initial-level performance. Improvement beyond initialization also occurs: in the original four-model, seed-42 experiment, Gemma4-E4B-it improves from 56.39% to 65.56% (+9.17 points) under the paper’s long-budget protocol.

2. Accuracy and actual inference costs [Q1 & W1]

We evaluated Initial and all six training methods at independent 8K, 16K, and 32K generation caps. Each evaluation covers the same questions, repetitions, prompts, sampling settings, and verifiers. Each cell below reports Math-7 accuracy (%) / actual mean generated tokens; trained rows average seeds 42/43.

Method8K cap16K cap32K cap
Initial33.42 / 6,78545.00 / 11,43249.87 / 15,102
GRPO40.15 / 5,95045.71 / 8,98646.85 / 9,961
GRPO + filtering39.43 / 6,01845.26 / 9,18046.47 / 10,308
DAPO40.53 / 5,74045.21 / 8,19145.65 / 8,762
S-GRPO35.46 / 6,58545.11 / 10,94849.24 / 14,375
PSP-E236.62 / 6,54246.06 / 10,65649.45 / 13,152
PSP-ME28.48 / 7,14343.29 / 12,26150.19 / 16,190

Math-7 averages responses within each question and weights the seven datasets equally. Token costs average all generated responses, including incorrect and truncated outputs.

The comparison shows a clear budget-dependent tradeoff. E2 achieves the highest 16K mean, and ME achieves the highest 32K mean. At 32K, ME uses approximately 7.2% more tokens than Initial, 57.1% more than filtering, and 62.5% more than GRPO. Its long-budget accuracy accompanies longer generation; its 8K/16K accuracy is lower than GRPO’s. These measurements quantify the accuracy–cost tradeoff; the equal-actual-length GRPO comparison you requested remains open.

AuthorsOpenReview ↗

Response to VnK8 (2/2): Stronger Baselines, Replication, and Mechanism

3. Stronger baselines and two training seeds [Q2 & Q3 & W2]

We added GRPO + overlong filtering, DAPO, and S-GRPO, each with two training seeds. All six methods use the same Qwen3-1.7B initialization, training data, 8K rollout cap, and 200 updates. Filtering masks policy/KL/entropy losses for genuinely length-stopped responses while retaining their rewards in group normalization. DAPO incorporates dynamic group resampling, asymmetric clipping, and a soft overlong penalty; S-GRPO provides a direct comparison using intermediate-answer rewards.

Method32K accuracy, seed 42 / 43Training-step hours, seed 42 / 43
GRPO47.03 / 46.678.29 / 8.76
GRPO + filtering46.76 / 46.188.30 / 8.26
DAPO44.78 / 46.5216.98 / 17.49
S-GRPO50.71 / 47.7710.41 / 9.78
PSP-E249.61 / 49.299.51 / 10.47
PSP-ME50.96 / 49.4210.96 / 10.87

Replication [Q3 & W2]. ME exceeds filtering by 4.20/3.23 points, with paired-question 95% confidence intervals [3.22, 5.22]/[2.10, 4.41], and DAPO by 6.17/2.90 points. Its gains over GRPO are 3.92/2.74 points. These improvements occur in both training seeds.

Training costs [F1]. All timing measurements use four H100 GPUs and sum the 200 training steps, including generation and optimization. Generated-token accounting includes masked filtering responses, discarded DAPO candidates, S-GRPO intermediate exits, and PSP probes. ME averages 10.92 hours, compared with 17.24 for DAPO, a 36.7% reduction. Filtering averages 8.28 hours and GRPO 8.52 hours. Startup, post-training export, and failed attempts are excluded.

4. GRPO configuration and longer-rollout training [Q2 & W2]

In a seed42 KL ablation under the original mixed test protocol (32,768 tokens for GSM8K/MATH500; 38,912 for competition mathematics), GRPO scores 47.17% with KL and 45.79% without, while E2 without KL reaches 48.62%. We also reevaluated a historical 16K-rollout GRPO checkpoint, which reaches 49.31%. This supplemental checkpoint uses a different training implementation. Our configuration checks cover KL and the added baseline recipes; a validation-selected learning-rate/clipping sweep remains open.

5. What changes in Qwen3-8B? [Q4]

Comparing GRPO with ME in the seed42 offline prefix diagnostic, using the first 8K tokens of saved long-budget trajectories, Qwen3-8B’s forced-answer success decreases from 58.17% to 56.95%, and original-answer correctness within that prefix from 54.28% to 32.93%, while long-budget accuracy increases from 64.24% to 68.70%. A candidate explanation is that PSP changes stopping behavior and the distribution of trajectories, allowing useful computation to continue beyond the training cap.

Large-model filtering [Q1 & Q2 & W1 & W2]. The Qwen3-8B seed42 filtering control scores 64.84%, versus ME’s 68.70%, under the original mixed test caps. ME’s advantage is 3.85 points [2.76, 4.99], with actual mean lengths of 15,356 versus 9,023 tokens. Initial scores 69.57%. Thus, on this model, prefix rewards recover much of the long-budget performance lost during training, with longer generation.

Non-mathematical transfer [F1]. We evaluated both Qwen3-1.7B seeds, using a 16K budget for HumanEval. ME averages 38.11% versus GRPO’s 37.05% on GPQA-Diamond, 69.96% versus 69.22% on IFEval, and 81.71% versus 80.49% on HumanEval. Transfer varies across tasks and seeds. Training on a non-mathematical task remains future work.

AuthorsOpenReview ↗

Revised manuscript: long-budget performance and actual costs

Thank you again. We have uploaded a revised PDF. Selected additions are highlighted in blue and marked NEW. The changes map to your review as follows:

  • Inference length and accuracy [Q1 & W1]. Section 5.4 and Figure 3 report accuracy against both maximum generation budget and actual mean output tokens at independent 8K/16K/32K caps. Initial and all six methods are included. This shows where long-budget performance is preserved and the generation cost associated with it.
  • Stronger baselines and training-seed variation [Q2, Q3 & W2]. Section 5.1 and Table 2 report GRPO, filtering, DAPO, S-GRPO, and PSP with two training seeds. The direct S-GRPO comparison is discussed in Section 5.2.
  • Short-rollout degradation and baseline settings [Q2 & W2]. The Introduction connects the short-training/long-testing problem to e3. Appendix G.7 reports the KL ablation and the historical 16K-rollout GRPO checkpoint; Section 5.4 shows the independent budget curves.
  • Qwen3-8B behavior [Q4]. Section 5.3 and Appendix C.7 report the changes in prefix answer potential, natural completion, and long-budget accuracy.
  • Training compute [future-work suggestion]. Table 2 and Appendix G.6 (Table 37) report both seeds' training times and generation totals for all six methods, including filtering and DAPO.

These additions place the evidence from our detailed response alongside the corresponding analyses in the revised paper.

Reviewer_pogLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Prefix Success Potential method, which modifies RLVR for reasoning LLMs by rewarding whether reasoning up to a certain point makes a correct answer more likely (measured by looking at the fraction of successes on several rollouts). The paper's evaluations compare GRPO with PSP_E (prefix = end) and PSP_{ME} (prefix = midpoint, prefix = end, averaged) on 2 model sizes each for 2 model sizes. The paper demonstrates gains on math-7 and other benchmarks.

Critical strengths

The paper is quite clear and explicit about experimental details -- what was done and not done. Its experimentation is fairly fleshed out with multiple models and benchmarks applied.

The paper demonstrates some consistent gains, especially on math and on Gemma-4-E4B-it.

Critical weaknesses

As far as I can tell from what's been run, a lot of this could be truncation effects -- the train reasoning length budget is considerably under initial test mean in many cases. Authors are clear about this, noting GRPO's degradation in some instances. The results would be much more interpretable if you included a baseline of GRPO with overlong filtering at the very least or ideally DAPO.

I'm worried about significance of the improvements, from several angles. The authors note that only one train seed is reported, so CIs are just in test eval (and train seed draw can have pretty substantial variance relative to the gain seen). No CIs were reported for other capabilities, as far as I can tell. As the authors note, both of the larger models perform below their initial checkpoints after RL training, and larger models show no midpoint benefits.

Questions for authors

all raised in concerns.

5 comments
AuthorsOpenReview ↗

Response to pogL (1/2): Long-Budget Preservation and Truncation Baselines

We thank Reviewer pogL for the clear and focused concerns.

Your two concerns — truncation baselines and statistical significance — are addressed directly. We trained GRPO + overlong filtering and DAPO with two seeds each; ME's gains over both are confirmed across seeds (filtering: +4.41/+3.34 pp, intervals [3.36, 5.48]/[2.34, 4.38]; DAPO: +6.29/+3.01 pp). Confidence intervals for non-mathematical benchmarks are now reported. The central finding holds: verified prefix rewards can preserve—and, in some settings, improve—long-budget reasoning performance after short-rollout RL training.

W1–W2 follow your two Critical Weaknesses; your Questions section refers to these concerns. K = 1,024 tokens.

1. Preserving long-budget performance [W1 & W2]

Short-rollout training creates a substantive challenge when a model is deployed with a larger inference budget. As e3 (Setlur et al., ICLR 2026, Section 6) demonstrates, training under a restricted rollout budget can discourage longer exploration and impair performance at larger test-time budgets. Our experiments study how verified prefix rewards address this challenge while keeping the training rollout cap fixed at 8K.

The independent budget evaluations make this effect explicit. On Qwen3-1.7B, GRPO improves 8K accuracy from Initial’s 33.42% to 40.15%, while reducing 32K accuracy from 49.87% to 46.85%. PSP-ME reaches 50.19% at 32K, retaining Initial-level performance in the two-seed average. We also observe a substantial improvement beyond initialization: in the four-model seed-42 experiment, Gemma4-E4B-it increases from 56.39% to 65.56%, a 9.17-point gain, under the paper’s long-budget evaluation protocol.

2. Overlong filtering and DAPO [W1]

We evaluated GRPO + overlong filtering and DAPO on Qwen3-1.7B with seeds 42/43, using the same initialization, training data, 8K rollout cap, and 200 updates. Filtering changes only the loss mask: genuinely length-stopped responses receive zero policy/KL/entropy loss, while their rewards remain in group normalization. Our DAPO configuration uses dynamic group resampling, asymmetric clipping, and a soft overlong penalty.

Under the original test protocol—32,768 tokens for GSM8K/MATH500 and 38,912 for competition mathematics—Math-7 accuracy (%) is shown below. Responses are averaged within each question, then the seven datasets are weighted equally:

MethodSeed 42Seed 43Mean ± sample SD
Initial reference——50.26
GRPO47.1746.8547.01 ± 0.23
GRPO + filtering46.6445.9446.29 ± 0.50
DAPO44.7746.2745.52 ± 1.06
S-GRPO50.6147.5249.06 ± 2.18
PSP-E249.8049.0849.44 ± 0.51
PSP-ME51.0649.2850.17 ± 1.26

ME improves over filtering by 4.41/3.34 points, with paired-question 95% confidence intervals [3.36, 5.48]/[2.34, 4.38], and over DAPO by 6.29/3.01 points. Thus, the long-budget benefit persists when outcome-reward training explicitly handles truncated samples.

Under the same mixed test caps, the Qwen3-8B seed42 filtering control reaches 64.84%, compared with GRPO’s 64.24% and ME’s 68.70%. ME exceeds filtering by 3.85 points [2.76, 4.99]. Together, the small-model replications and larger-model control provide direct evidence for the benefit of prefix rewards beyond overlong-sample masking.

AuthorsOpenReview ↗

Response to pogL (2/2): Replication, Confidence Intervals, and Costs

3. Training variability and midpoint probes [W2]

The table reports individual training runs and their sample SDs. ME’s gains over GRPO are 3.89/2.43 points across seeds. The equal-probe comparison also replicates: with four probes in both methods, midpoint-plus-endpoint ME exceeds endpoint-only E4 by 2.11/0.99 points, with intervals [1.20, 3.04]/[0.16, 1.83].

The incremental benefit over E2 is smaller: 1.26/0.20 points. On the two larger models, the midpoint-versus-endpoint intervals include zero. In seed42, Qwen3-8B and Gemma4-12B-it gain 4.45 and 6.56 points over GRPO, respectively, while their post-training scores remain below Initial. The results therefore distinguish the broad benefit of prefix-based rewards from the model-dependent benefit of adding midpoint probes.

4. Confidence intervals for other capabilities [W2]

We now report both Qwen3-1.7B training seeds on the non-mathematical benchmarks. HumanEval consistently uses a 16,384-token budget. ME minus GRPO differences, in percentage points, are:

BenchmarkSeed 42: difference [95% CI]Seed 43: difference [95% CI]
GPQA-Diamond+2.53 [0.25, 4.80]−0.40 [−2.78, 1.97]
IFEval+0.92 [−2.40, 4.25]+0.55 [−2.77, 4.07]
HumanEval0.00 [−5.49, 5.49]+2.44 [−4.27, 8.54]
BFCL+1.00 [−2.00, 5.00]−2.00 [−5.00, 0.00]

These results show task-dependent transfer.

5. Inference and training costs [W1]

The budget sweep shows where the benefits arise: ME scores 28.48% at 8K, 43.29% at 16K, and 50.19% at 32K; E2 has the highest 16K mean at 46.06%. At 32K, counting all generated answers, ME averages 16,190 output tokens, versus 15,102 for Initial, 10,308 for filtering, and 9,961 for GRPO. Its long-budget accuracy accompanies longer generation.

On four H100 GPUs, mean training-step hours are 8.52 for GRPO, 8.28 for filtering, 17.24 for DAPO, 10.09 for S-GRPO, 9.99 for E2, and 10.92 for ME. ME uses 36.7% less training-step time than DAPO. Timing sums all 200 updates, including sampled candidates and auxiliary generation; startup, post-training export, and failed attempts are excluded.

Intervals are nominal 95% paired-question bootstrap intervals conditional on the evaluated checkpoints; training variation is reported through individual seeds. Differences use unrounded scores.

AuthorsOpenReview ↗

Revised manuscript: truncation controls and two-seed evidence

Thank you again. We have uploaded a revised PDF. Selected additions are highlighted in blue and marked NEW. The additions address your two concerns:

  • Truncation-aware baselines [W1]. Section 5.1 and Table 2 report the Qwen3-1.7B two-seed controls, including GRPO + overlong filtering and DAPO. Appendix G.2 provides the original mixed-budget results and paired-question intervals. Section 5.1 also summarizes the Qwen3-8B filtering control, with full results in Appendix G.7.
  • Training-seed variation and other-capability intervals [W2]. Table 2 reports both Qwen3-1.7B training seeds and their sample SD; Table 3 extends the positional ablations to both seeds. Appendix G.5 reports the non-mathematical results and question-level intervals. The independent 8K/16K/32K sweep in Section 5.4 and Figure 3 shows long-budget performance alongside actual generation costs.

The original four-model seed-42 main table is retained. These additions incorporate the evidence from our detailed response into the manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions

Dear Reviewers and Area Chair,

Thank you for your feedback. We have uploaded a revised PDF. The title, overall structure, and original four-model main table are unchanged. Selected additions are highlighted in blue and marked NEW in the margin. The main text is nine pages; Appendix G provides the full supporting results and protocols.

What changed, by location:

  1. Abstract and Introduction (Section 1). We emphasize preserving long-budget performance after short (8K) RL rollouts. Under independent 32K evaluation, PSP-ME averages 50.19% across two training seeds, compared with Initial's 49.87% and GRPO's 46.85%. We also summarize the fixed-reader extraction finding.

  2. Related work (Section 2). We clarify short-answer supervision and connect the method to the new direct S-GRPO comparison.

  3. Strong baselines and two training seeds (Section 5.1, Table 2). Qwen3-1.7B results now include GRPO + overlong filtering, DAPO, and S-GRPO with seeds 42/43. Under the original mixed test caps—32,768 tokens for GSM8K/MATH500 and 38,912 for competition mathematics—PSP-ME exceeds filtering by 4.41/3.34 percentage points and DAPO by 6.29/3.01 points. Appendix G.2 reports paired-question intervals. Section 5.1 also summarizes the Qwen3-8B filtering control, with its full results in Appendix G.7.

  4. S-GRPO and positional ablations (Section 5.2, Table 3). We add a direct S-GRPO comparison using the same initialization, training data, 8K rollout cap, and 200 updates. The positional ablation now reports both training seeds, including equal-four-probe midpoint/endpoint comparisons (Appendix G.2).

  5. Common-reader experiment (Section 5.3, Table 5). Holding the Gemma4-12B-it reader fixed to Initial, Boxed answer extraction from PSP-ME prefixes improves over Initial prefixes by 3.78 percentage points at 8K and 5.69 points at 4K. Full and Boxed results, including all 20 reader/budget/cue conditions, appear in Appendix G.4.

  6. Inference budgets and measured training cost (Section 5.4). Figure 3 compares accuracy against maximum generation budget and actual mean output tokens for Initial and all six methods at independently generated 8K/16K/32K caps. Table 2 and Appendix G.6 (Table 37) report training-step time and generated tokens for both seeds of all six methods, including filtering. PSP-ME uses 36.7% less training-step time than DAPO in this setup.

  7. Discussion and supporting evidence. Sections 6–7 summarize the findings and their scope. Appendix G also reports two-seed non-mathematical results and intervals (G.5), KL and historical 16K-rollout controls (G.7), and frozen-trajectory truncation diagnostics (G.8).

Our detailed responses and the per-reviewer revision notices provide the corresponding evidence and manuscript locations for each concern.

AuthorsOpenReview ↗

Additional revision: baselines on all four models, Gemma S-GRPO, and a revised abstract

Dear Reviewers and Area Chair,

Thank you for the continued discussion. We have uploaded a further revision. Since our 16 November update we have added DAPO on Qwen3-8B, overlong filtering and DAPO on both Gemma models, and S-GRPO on Gemma4-E4B-it. We have also rewritten the abstract so that it states the generation cost of our gains and the question that remains open.

Truncation-aware baselines now cover all four models. Table 1 includes the new results. Appendix G.7 reports accuracy, actual mean output tokens and paired-question 95% confidence intervals (pp = percentage points):

New baselineAccuracy / mean tokensPSP-ME accuracy / mean tokensPSP-ME minus baseline (pp), 95% CI
Qwen3-8B DAPO62.64 / 7,53768.70 / 15,3566.05 [4.61, 7.59]
Gemma4-E4B-it filtering62.17 / 6,76565.56 / 12,6213.38 [1.74, 5.02]
Gemma4-E4B-it DAPO59.98 / 5,52165.56 / 12,6215.57 [3.84, 7.38]
Gemma4-12B-it filtering72.28 / 6,43479.93 / 13,0027.66 [5.81, 9.56]
Gemma4-12B-it DAPO74.03 / 5,30279.93 / 13,0025.91 [4.18, 7.72]

Accuracy is Math-7 (%). All runs use seed 42, step 200 and an 8K training rollout cap, and are tested under the original mixed caps (32,768 tokens for GSM8K/MATH500, 38,912 for AIME/HMMT). The PSP-ME checkpoints are unchanged. Token counts are averaged over all 8,635 answers. Intervals resample paired questions and condition on the evaluated checkpoints; differences are computed before rounding.

What these results show. On every model, PSP-ME is more accurate than the strongest of GRPO, filtering and DAPO, by 3.38–5.91 pp, and it generates 1.70–2.45 times as many output tokens as that baseline. Relative to the initial model, PSP-ME is higher on Gemma4-E4B-it (+9.17 pp) and Qwen3-1.7B (+0.80 pp), and lower on Qwen3-8B (−0.88 pp) and Gemma4-12B-it (−1.90 pp). The strongest outcome-reward baseline is 3.09, 4.73 and 7.80 pp below the initial model on Qwen3-1.7B, Qwen3-8B and Gemma4-12B-it. We therefore claim that verified prefix rewards largely avoid the long-budget loss that outcome-reward training incurs at a short rollout cap. We do not claim superiority at matched inference cost; the abstract and Limitations state that this remains open.

S-GRPO on a second model family. Appendix G.9 adds Gemma4-E4B-it: S-GRPO reaches 60.11% and PSP-ME 65.56%, a difference of 5.45 pp [3.89, 7.01]. Mean output lengths are 6,159 and 12,621 tokens. The Qwen3-1.7B comparison, where the two methods are close, is unchanged.

Other changes. Tables 11–12 add the non-mathematical results for the new baselines, and Appendix G.9 gives them for Gemma4-E4B-it S-GRPO. Table 1 now states how many sampled replies per question each benchmark averages over, and Tables 1–5 mark metric direction and the best value in each column. Text added in this round is shown in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.