- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Prefix Success Potential (PSP), which rewards the original inference trajectory based on the frequency with which the model generates correct short answers from inference prefixes. The main configuration samples two short answers each at the midpoint and endpoint of the trajectory, and the average validation pass rate is used for GRPO updates, requiring no external teacher or process annotation. The long-budget Math-7 scores of four Qwen/Gemma models are improved by 3.89–6.56 percentage points relative to GRPO. The paper further investigates the relationship between probe location, reward discrimination, computational cost, and the relationship between "extractable answers" and naturally completed solutions.
Critical strengths
Simple reward constructs yielded consistent gains across multiple models. All four models outperformed their corresponding GRPO models in long-budget Math-7. Gemma4-E4B-it achieved a 65.56% improvement, a 9.17 percentage point increase over the pre-training model, demonstrating a clear net capability improvement. Single-endpoint re-answers also improved Qwen3-1.7B, indicating that the gains cannot be solely attributed to finer reward values.
The separation between answer potential and natural output provides valuable diagnostic evidence. In the 8K prefix evaluation, Gemma4-12B-it showed a 14.32 percentage point improvement in short answer potential relative to GRPO, but its natural output accuracy decreased from 70.93% to 41.94%. Similar phenomena were observed in the Clean set using the same validators, thus not simply attributable to validator differences. This finding reveals a noteworthy deviation between training rewards and actual answering behavior.
Critical weaknesses
The main results do not yet separate the effects of baseline degradation and increased inference length. The average test output length of PSP is 1.79–2.74 times that of GRPO, while the natural output accuracy within the 8K prefix is lower on all four models. GRPO for Qwen3-8B and Gemma4-12B-it degrades by 5.33 and 8.46 percentage points from initialization, respectively, and PSP also remains below initialization. Therefore, some gains may reflect mitigation of the long-budget degradation caused by short-budget training, and are insufficient to establish a net gain at comparable inference costs for a relatively well-tuned baseline. The low training cost configuration in Table 19 is a valuable counter-evidence, but test costs are not controlled. (Tables 3, 11, 17, 19)
The incremental value of the core design relative to existing prefix probe reward methods has not been fully established. SPAE has utilized intermediate answer probes to construct process potential signals, and S-GRPO has utilized answer correctness at multiple intermediate positions to provide rewards. This paper uses absolute potential averaging to update the entire trajectory, but lacks direct comparisons with these most relevant methods. The positional assignment comparison with equal probe counts only covers Qwen3-1.7B; in both large models, the gain interval of the midpoint plus endpoint relative to the endpoint only contains zeros, and there is only one training seed. Therefore, support for the general effectiveness of the new design remains limited.
Questions for authors
Baseline Clarification and Necessary Evidence: What are the main sources of the long budget degradation in GRPO for the two large models? After parameter tuning on the validation set and controlling the actual number of generated tokens or inference costs, does the advantage of PSP over GRPO and the initial model still exist?
Supplementary Evidence Influencing Contribution Judgment: Are there comparable results with SPAE or S-GRPO, and midpoint/endpoint ablation with equal probe counts and multiple training seeds, which can confirm the stable gains of the core design beyond typical re-answer supervision?
Mechanism Clarification: How to distinguish between improved prefix inference quality and model better adaptation to specified answer hint formats? After using a fixed answer reading model or multiple hint formats, does the potential improvement of Gemma remain?
4 comments
Response to Ndod (2/2): S-GRPO, Probe Ablations, and Fixed-Reader Evidence
2. S-GRPO and equal-probe, two-seed ablations [Q2 & W2]
We implemented S-GRPO in the shared Qwen3-1.7B setup, using one original trajectory and eight intermediate exits, with decaying rewards favoring earlier correct answers. Evaluation uses natural generation. PSP averages verified success at fixed prefix positions into one reward for the original trajectory. Under the original protocol—32,768 tokens for GSM8K/MATH500 and 38,912 for competition mathematics—the Math-7 accuracies (%) are:
| Method | Seed 42 | Seed 43 |
|---|---|---|
| GRPO | 47.17 | 46.85 |
| GRPO + filtering | 46.64 | 45.94 |
| DAPO | 44.77 | 46.27 |
| S-GRPO | 50.61 | 47.52 |
| Endpoint-1 | 49.52 | 49.51 |
| PSP-E2 | 49.80 | 49.08 |
| Endpoint-4 | 48.95 | 48.29 |
| PSP-ME | 51.06 | 49.28 |
Initial scores 50.26% under this protocol. ME exceeds S-GRPO by 0.45/1.76 points, with intervals [−0.27, 1.15]/[0.63, 2.90]. ME uses 11.6%/12.7% more actual output tokens. The seed42 interval includes zero; the seed43 interval is positive.
Midpoint allocation [Q2 & W2]. With four probes in both methods, ME assigns two to the midpoint and two to the endpoint, while E4 assigns all four to the endpoint. ME improves over E4 by 2.11/0.99 points, with intervals [1.20, 3.04]/[0.16, 1.83]. This positional benefit appears in both training seeds. Gains over E2 are 1.26/0.20 points, and Endpoint-1 remains competitive, slightly exceeding ME in seed43. On the two larger models, midpoint-versus-endpoint intervals include zero.
Relation to SPAE [Q2 & W2]. SPAE combines intermediate confidence and correctness into step potential and shapes credit using potential changes and saturation penalties. PSP uses absolute verified success rates to assign a trajectory-level reward. A direct SPAE experiment remains open.
Training costs [Q1 & W1]. Mean training-step hours on four H100 GPUs are 8.52/8.28/17.24/10.09/9.99/10.92 for GRPO/filtering/DAPO/S-GRPO/E2/ME. ME takes 36.7% less time than DAPO. Timing sums 200 updates and includes masked responses, discarded candidates, intermediate exits, and probes; startup, post-training export, and failed attempts are excluded.
3. Fixed reader and answer formats [Q3]
Yes, with the Boxed protocol, the improvement over Initial remains when the reader is held fixed. We evaluated Gemma4-12B-it on 256 Clean questions, with four saved trajectories per generator, prefixes at 4K/8K, and four probes of at most 256 tokens per prefix. For each generator, the same saved prefixes are reused across reader and prompt conditions. The table reports forced-answer success rates (%):
| Prefix generator / reader | 4K Full | 4K Boxed | 8K Full | 8K Boxed |
|---|---|---|---|---|
| Initial / Initial | 32.52 | 53.78 | 25.27 | 62.11 |
| GRPO / Initial | 40.75 | 60.11 | 41.33 | 63.16 |
| ME / Initial | 31.93 | 59.47 | 21.97 | 65.89 |
| GRPO / GRPO | 32.54 | 53.44 | 32.89 | 55.08 |
| ME / ME | 17.63 | 60.57 | 8.59 | 66.65 |
With the common Initial reader and Boxed cue, ME prefixes improve over Initial prefixes by 5.69 points at 4K [3.03, 8.42] and 3.78 points at 8K [1.25, 6.25]. The 8K ME result is 65.89% with the Initial reader and 66.65% with its own reader. This provides direct evidence that the trained prefixes support improved answer extraction by an unchanged reader under the Boxed protocol.
Prompt dependence [Q3]. The two prompt formats give different outcomes. With the Initial reader and Full cue, ME scores 21.97% versus Initial’s 25.27%, a difference of −3.30 points [−5.52, −1.15]. Under Boxed, ME exceeds GRPO by 2.73 points [−0.15, 5.62] at 8K. The fixed-reader gain over Initial is specific to the Boxed protocol.
Revised manuscript: strong baselines, positional ablations, and fixed readers
Thank you again. We have uploaded a revised PDF. Selected additions are highlighted in blue and marked NEW. Following your three questions:
- Strong baselines and inference costs [Q1 & W1]. Section 5.1 and Table 2 report the two-seed Qwen3-1.7B controls, including filtering and DAPO. Section 5.4 and Figure 3 show the independent 8K/16K/32K accuracy and actual-token curves. Table 2 and Appendix G.6 (Table 37) give measured training costs for all six methods.
- S-GRPO and equal-probe, two-seed ablations [Q2 & W2]. Section 5.2 presents the direct S-GRPO comparison. Table 3 reports both seeds of the positional ablation, including the equal-four-probe midpoint/endpoint comparison; Appendix G.2 gives the original mixed-budget results and paired-question intervals. Section 2 clarifies the methodological relationship to SPAE.
- Fixed reader and answer format [Q3]. Section 5.3 and Table 5 hold the reader fixed to Initial and compare Full and Boxed extraction from trained Gemma4-12B-it prefixes. Boxed extraction improves on PSP-ME prefixes. Appendix G.4 reports all 20 reader/budget/cue conditions and the corresponding contrasts.
These additions incorporate the evidence from our detailed response into the revised manuscript.
Thank you again for your review. Two additions since our previous response relate to your questions.
Strong baselines on all four models [Q1 & W1]. Table 1 adds DAPO on Qwen3-8B and overlong filtering and DAPO on both Gemma models. PSP-ME exceeds the strongest of GRPO, filtering and DAPO by 3.38–5.91 pp on every model, while generating 1.70–2.45 times as many output tokens. Appendix G.7 reports accuracy, actual mean output tokens and paired-question 95% intervals for every row.
S-GRPO on a second model family [Q2 & W2]. Appendix G.9 adds Gemma4-E4B-it, where S-GRPO reaches 60.11% and PSP-ME 65.56%: a difference of 5.45 pp [3.89, 7.01], with 6,159 versus 12,621 mean output tokens. Its non-mathematical results are in the same appendix. This is a seed-42, step-200 comparison under the original mixed test caps, with intervals conditional on the evaluated checkpoints.
The abstract and Limitations now state that these gains come with longer generations and that matched-cost superiority remains open.
Response to Ndod (1/2): Stronger Baselines and Accuracy–Cost Tradeoffs
We thank Reviewer Ndod for the detailed review and the well-structured questions.
Your three questions provided a clear structure for the additional experiments. The results address each directly: ME's long-budget advantage over truncation-aware baselines holds in both seeds (filtering: +4.20/+3.23 pp, intervals [3.22, 5.22]/[2.10, 4.41]; DAPO: +6.17/+2.90 pp; S-GRPO: +0.45/+1.76 pp); midpoint allocation shows a consistent benefit over endpoint-only on Qwen3-1.7B when probe counts are matched (ME over E4: +2.11/+0.99 pp across both seeds); and ME prefixes improve answer extraction by an unchanged Initial reader under the Boxed protocol (+5.69 pp at 4K, +3.78 pp at 8K).
Q1–Q3 and W1–W2 follow the order of your questions and weaknesses. K = 1,024 tokens.
1. Stronger baselines and actual inference costs [Q1 & W1]
Preserving long-budget capability is an important training objective when rollout budgets are constrained. e3 (Setlur et al., ICLR 2026, Section 6) studies how restrictive training budgets can discourage exploration needed for longer-budget reasoning. Our independent evaluations show a corresponding pattern: after 8K-rollout training, Qwen3-1.7B GRPO improves 8K accuracy from 33.42% to 40.15%, while reducing 32K accuracy from 49.87% to 46.85%. ME reaches 50.19% at 32K in the two-seed average, preserving Initial-level performance.
On Qwen3-1.7B, we added GRPO + overlong filtering, DAPO, and S-GRPO using the same initialization, training data, 8K rollout cap, and 200 updates, each with seeds 42/43. Filtering masks policy/KL/entropy losses for length-stopped responses while retaining their rewards in group normalization. DAPO incorporates dynamic group resampling, asymmetric clipping, and a soft overlong penalty. The independent budget results are:
Accuracies are percentages; trained rows average seeds 42/43. Math-7 averages responses within each question and weights the seven datasets equally. Actual token costs include all responses, including incorrect and truncated outputs.
ME exceeds filtering at 32K by 4.20/3.23 points, with paired-question 95% intervals [3.22, 5.22]/[2.10, 4.41]. E2 achieves the highest 16K mean, while ME achieves the highest 32K mean. ME’s 32K result uses 7.2% more tokens than Initial, 12.6% more than S-GRPO, and 57.1% more than filtering. At 8K/16K, ME has lower accuracy and higher token use than GRPO and S-GRPO. These measurements establish the budget-dependent accuracy–cost tradeoff; superiority at matched actual inference cost remains unresolved.
Larger-model baselines [Q1 & W1]. In the seed42 comparisons, the Qwen3-8B filtering baseline reaches 64.84%, versus GRPO’s 64.24% and ME’s 68.70%, under the original mixed test caps. ME exceeds filtering by 3.85 points [2.76, 4.99], with actual mean lengths of 15,356 versus 9,023 tokens. Qwen3-8B and Gemma4-12B-it remain below Initial after PSP training: 68.70% versus 69.57%, and 79.93% versus 81.83%. Gemma4-E4B-it improves from 56.39% to 65.56% (+9.17 points). The short-training/long-testing mismatch provides a plausible explanation for degradation; its contribution relative to optimization and stopping behavior remains to be isolated.
Configuration checks [Q1 & W1]. In a seed42 KL ablation under the original mixed test protocol, GRPO scores 47.17% with KL and 45.79% without. A historical 16K-rollout GRPO checkpoint reaches 49.31% under that protocol, using a different training implementation. Our configuration checks cover KL and the added baseline recipes; a validation-selected learning-rate/clipping sweep remains open.