- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LC-GRPO to address a potential limitation of outcome-supervised RLVR: assigning the same sequence-level advantage to every token can penalize useful prefixes within unsuccessful responses. For verifier-rejected responses, LC-GRPO constructs step-level scores from stored behavior-policy token log-probabilities and uses the largest adjacent drop exceeding a threshold as an estimated failure boundary. Only the boundary and subsequent steps receive non-positive advantages with a bounded confidence-dependent penalty. The paper evaluates Qwen3-4B-Base and Qwen3-8B-Base on six mathematical and code benchmarks and reports accuracy improvements over baselines. Additional evidence includes theoretical properties of the advantage construction, component ablations, first-error localization on ProcessBench, and calibration diagnostics based on a likelihood-derived proxy score.
Critical strengths
-
LC-GRPO uses existing policy-likelihood changes to estimate where negative credit should begin, avoiding an additional PRM or process annotations. The method separates prefix protection from suffix penalization and explicitly constrains the sign and support of the advantages. This provides a simple and potentially practical approach to localized credit assignment under outcome supervision without allowing model confidence to override the verifier.
-
The paper reports improved task accuracy over the evaluated baselines across two model scales and multiple mathematical and code benchmarks. The ablations provide useful evidence about the contributions of individual components.
Critical weaknesses
-
Final-checkpoint localization results do not sufficiently establish the reliability of the signal during training. Table 2 and Appendix F.2 primarily evaluate checkpoints obtained through different training methods. LC-GRPO’s higher localization accuracy shows that the resulting model has a more informative likelihood signal, but does not directly establish that the localization signal guiding policy updates is reliable at initialization or throughout training. The current results also do not adequately distinguish whether improved localization contributes to the task-level gains or emerges as a consequence of training. The missing evidence concerns the training-time mechanism,that if confidence signals can serve as an accurate supervision of failure boundaries during whole training process.
-
The connection between the calibration diagnostics and the central problem is insufficiently established. LC-GRPO primarily aims to improve localized credit assignment under outcome supervision rather than train probabilistic confidence estimates. Its localization mechanism relies on adjacent step log-likelihood changes, which is not served as a judge of answer correctness during training. Improvements in ECE and related metrics can therefore serve as auxiliary observations, but do not directly validate the localization mechanism. The authors should clarify the role of these diagnostics in the central argument.
Questions for authors
Could you provide first-error localization results for the checkpoint before RL and for several intermediate LC-GRPO checkpoints?
If these diagnostics are available, does localization already outperform simple positional priors or random boundaries early in training, and does its performance remain stable throughout optimization?
These results would directly affect my assessment of the reliability of the signal on which the method depends.
We sincerely thank the reviewer for the precise requests, and for recognizing that LC-GRPO uses existing likelihood changes instead of a PRM or process annotations, separates prefix protection from suffix penalization, and constrains the sign and support of the advantages. Below, Q denotes each question and R our response. All numbers come from the re-run described in the general response.
Q1. Is the localization signal reliable at initialization and throughout training, and does it beat positional priors and random boundaries early on?
R: Yes. Before any RL update the signal already localizes three times better than the best control, and it improves at every checkpoint. The new checkpoint study (Table 4a, Page 9; Appendix G, Table 12, Page 21) applies one shared maximum-drop detector to the same 800 ProcessBench failures; step 0 is the SFT checkpoint.
(1) Controls use no model signal: a position prior fitted on 160 disjoint items (.115), always step 2 (.099), always the last step (.103), and a uniform random boundary (.096). The SFT checkpoint (.356) is three times the best of them.
(2) The signal is stable and is sharpened by LC-GRPO's own updates. Exact rises at every checkpoint with no drop, and the fallback rate falls from .520 to .260. After 125 steps LC-GRPO (.484) already exceeds GRPO at step 500 (.398).
(3) At initialization the first error already receives negative credit in 89.4% of failures. At SFT, 35.6% of boundaries are exact and 53.8% are early. An early boundary, including the fallback b=1 (52.0%), still places the first error and every later step in the penalized suffix, and with b=1 the update equals GRPO's. Only a late boundary leaves the first error unpenalized: 10.6% at SFT and 4.9% at step 500.
(4) On-policy check (Table 4a, Page 9; Appendix I, Table 15, Page 22). At each checkpoint we draw 200 failed LC-GRPO training rollouts and label their first error with GPT-5.6 sol; humans re-label 50 per checkpoint and agree with the annotator on 80–88%. Detector–annotator Exact rises from .450 at step 125 to .645 at step 500.
Q2. Does better localization contribute to the task gain, or only follow from training?
R: It contributes. Changing only where the boundary is placed accounts for most of the gain over GRPO, and localization improves before accuracy does.
(1) Mask controls fix the suffix-mask form and remove the confidence penalty (Table 4b, Page 9; Appendix J.2, Table 16, Page 23; 4B, math, two seeds):
The located boundary gains 2.64 points over a random boundary of the same form (2.54 and 2.74 on the two seeds), and the random boundary stays close to GRPO. The confidence penalty adds a further 1.30.
(2) Timing (Tables 10 and 12, Pages 20–21; 8B). By step 125, Exact has risen from .356 to .484 while accuracy on the 1,800 diagnostic responses is still .706 (SFT .721). GRPO at step 500 and LC-GRPO at step 250 reach similar accuracy (.745 and .738), but their Exact is .398 and .541. A more accurate policy therefore does not by itself localize better.
(3) Training rule (Table 3, Page 8; 8B). Every ablated variant trains a policy that localizes worse, from .713 to between .438 (random boundary) and .689 (linear penalty). Table 3 now reports localization only; the accuracy-level evidence for components is (1).
With two seeds in the 4B controls, we report (1) as evidence of a contribution, not as a significance claim.
Q3. What role do the calibration diagnostics play?
R: We agree that they do not validate localization, and the revised paper no longer uses them that way. Localization rests on first-error labels: the fixed ProcessBench set and the human-checked on-policy audit (Q1). The re-run ECE is also worse for LC-GRPO than for GRPO (.180 against .093), so we withdraw the calibration claim. Confidence–correctness separation (AUROC .962 against .644) is reported in §7.4 (Page 8) only as a secondary diagnostic of what the suffix penalty changes: it lowers confidence on rejected responses, while confidence on correct ones stays at .844–.848 (Table 10, Page 20).
We believe these results provide the training-time evidence the reviewer asked for, and we would be grateful if the reviewer would consider raising the score.