acceptodds
Withdrawn from ICLR 2027

Beyond Overconfidence: Logprob-Calibrated Advantage for LLM Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The scaling of Large Language Model (LLM) reasoning via Reinforcement Learning is bottlenecked by prohibitive computational cost and reliance on external Process Reward Models (PRMs). While RM-free self-alignment via Group Relative Policy Optimization (GRPO) offers an efficient alternative, mining valid supervision from sparse outcomes faces three challenges: (1) Confidence-Blind Penalization, where advantage functions ignore internal generative certainty, treating overconfident hallucinations and uncertain guesses with identical severity; (2) Signal Dilution in Long Trajectories, where sparse global rewards fail to resolve credit assignment, masking localized logic fractures within reasoning chains; and (3) Calibration Divergence, where iterative policy updates sharpen output distributions without grounding them in factual truth, destroying the model's intrinsic self-awareness. To address these, we introduce LC-GRPO (Logprob-Calibrated Group Relative Policy Optimization), framing RM-free self-alignment as endogenous confidence calibration. We derive a Step-Level Generative Confidence metric directly from the policy's internal log-probabilities as a zero-cost pseudo-PRM, and introduce a Confidence-Decayed Advantage mechanism to apply non-linear penalties targeting early-onset, high-confidence reasoning errors. Experiments show LC-GRPO delivers a robust, well-calibrated self-evolution path, mitigating compounding errors and achieving superior reasoning efficiency while preserving strict alignment between model confidence and logical accuracy.

Primary area:
reinforcement learning
Keywords:
Reinforcement, Inverse RL

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AeZbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LC-GRPO to address a potential limitation of outcome-supervised RLVR: assigning the same sequence-level advantage to every token can penalize useful prefixes within unsuccessful responses. For verifier-rejected responses, LC-GRPO constructs step-level scores from stored behavior-policy token log-probabilities and uses the largest adjacent drop exceeding a threshold as an estimated failure boundary. Only the boundary and subsequent steps receive non-positive advantages with a bounded confidence-dependent penalty. The paper evaluates Qwen3-4B-Base and Qwen3-8B-Base on six mathematical and code benchmarks and reports accuracy improvements over baselines. Additional evidence includes theoretical properties of the advantage construction, component ablations, first-error localization on ProcessBench, and calibration diagnostics based on a likelihood-derived proxy score.

Critical strengths
  1. LC-GRPO uses existing policy-likelihood changes to estimate where negative credit should begin, avoiding an additional PRM or process annotations. The method separates prefix protection from suffix penalization and explicitly constrains the sign and support of the advantages. This provides a simple and potentially practical approach to localized credit assignment under outcome supervision without allowing model confidence to override the verifier.

  2. The paper reports improved task accuracy over the evaluated baselines across two model scales and multiple mathematical and code benchmarks. The ablations provide useful evidence about the contributions of individual components.

Critical weaknesses
  1. Final-checkpoint localization results do not sufficiently establish the reliability of the signal during training. Table 2 and Appendix F.2 primarily evaluate checkpoints obtained through different training methods. LC-GRPO’s higher localization accuracy shows that the resulting model has a more informative likelihood signal, but does not directly establish that the localization signal guiding policy updates is reliable at initialization or throughout training. The current results also do not adequately distinguish whether improved localization contributes to the task-level gains or emerges as a consequence of training. The missing evidence concerns the training-time mechanism,that if confidence signals can serve as an accurate supervision of failure boundaries during whole training process.

  2. The connection between the calibration diagnostics and the central problem is insufficiently established. LC-GRPO primarily aims to improve localized credit assignment under outcome supervision rather than train probabilistic confidence estimates. Its localization mechanism relies on adjacent step log-likelihood changes, which is not served as a judge of answer correctness during training. Improvements in ECE and related metrics can therefore serve as auxiliary observations, but do not directly validate the localization mechanism. The authors should clarify the role of these diagnostics in the central argument.

Questions for authors

Could you provide first-error localization results for the checkpoint before RL and for several intermediate LC-GRPO checkpoints?

If these diagnostics are available, does localization already outperform simple positional priors or random boundaries early in training, and does its performance remain stable throughout optimization?

These results would directly affect my assessment of the reliability of the signal on which the method depends.

1 comment

We sincerely thank the reviewer for the precise requests, and for recognizing that LC-GRPO uses existing likelihood changes instead of a PRM or process annotations, separates prefix protection from suffix penalization, and constrains the sign and support of the advantages. Below, Q denotes each question and R our response. All numbers come from the re-run described in the general response.


Q1. Is the localization signal reliable at initialization and throughout training, and does it beat positional priors and random boundaries early on?

R: Yes. Before any RL update the signal already localizes three times better than the best control, and it improves at every checkpoint. The new checkpoint study (Table 4a, Page 9; Appendix G, Table 12, Page 21) applies one shared maximum-drop detector to the same 800 ProcessBench failures; step 0 is the SFT checkpoint.

Checkpoint0 (SFT)125250375500
LC-GRPO Exact.356.484.541.619.713 [.681, .744]
GRPO Exact.356.368.381.390.398 [.363, .431]

(1) Controls use no model signal: a position prior fitted on 160 disjoint items (.115), always step 2 (.099), always the last step (.103), and a uniform random boundary (.096). The SFT checkpoint (.356) is three times the best of them.

(2) The signal is stable and is sharpened by LC-GRPO's own updates. Exact rises at every checkpoint with no drop, and the fallback rate falls from .520 to .260. After 125 steps LC-GRPO (.484) already exceeds GRPO at step 500 (.398).

(3) At initialization the first error already receives negative credit in 89.4% of failures. At SFT, 35.6% of boundaries are exact and 53.8% are early. An early boundary, including the fallback b=1b=1 (52.0%), still places the first error and every later step in the penalized suffix, and with b=1b=1 the update equals GRPO's. Only a late boundary leaves the first error unpenalized: 10.6% at SFT and 4.9% at step 500.

(4) On-policy check (Table 4a, Page 9; Appendix I, Table 15, Page 22). At each checkpoint we draw 200 failed LC-GRPO training rollouts and label their first error with GPT-5.6 sol; humans re-label 50 per checkpoint and agree with the annotator on 80–88%. Detector–annotator Exact rises from .450 at step 125 to .645 at step 500.


Q2. Does better localization contribute to the task gain, or only follow from training?

R: It contributes. Changing only where the boundary is placed accounts for most of the gain over GRPO, and localization improves before accuracy does.

(1) Mask controls fix the suffix-mask form and remove the confidence penalty (Table 4b, Page 9; Appendix J.2, Table 16, Page 23; 4B, math, two seeds):

VariantMath Avg.
LC-GRPO (full)58.11
Located suffix mask only, λ=0\lambda=056.81
Random contiguous suffix mask54.17
GRPO53.88

The located boundary gains 2.64 points over a random boundary of the same form (2.54 and 2.74 on the two seeds), and the random boundary stays close to GRPO. The confidence penalty adds a further 1.30.

(2) Timing (Tables 10 and 12, Pages 20–21; 8B). By step 125, Exact has risen from .356 to .484 while accuracy on the 1,800 diagnostic responses is still .706 (SFT .721). GRPO at step 500 and LC-GRPO at step 250 reach similar accuracy (.745 and .738), but their Exact is .398 and .541. A more accurate policy therefore does not by itself localize better.

(3) Training rule (Table 3, Page 8; 8B). Every ablated variant trains a policy that localizes worse, from .713 to between .438 (random boundary) and .689 (linear penalty). Table 3 now reports localization only; the accuracy-level evidence for components is (1).

With two seeds in the 4B controls, we report (1) as evidence of a contribution, not as a significance claim.


Q3. What role do the calibration diagnostics play?

R: We agree that they do not validate localization, and the revised paper no longer uses them that way. Localization rests on first-error labels: the fixed ProcessBench set and the human-checked on-policy audit (Q1). The re-run ECE is also worse for LC-GRPO than for GRPO (.180 against .093), so we withdraw the calibration claim. Confidence–correctness separation (AUROC .962 against .644) is reported in §7.4 (Page 8) only as a secondary diagnostic of what the suffix penalty changes: it lowers confidence on rejected responses, while confidence on correct ones stays at .844–.848 (Table 10, Page 20).


We believe these results provide the training-time evidence the reviewer asked for, and we would be grateful if the reviewer would consider raising the score.

Reviewer_KgBQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Localized-Credit Group Relative Policy Optimization (LC-GRPO) to address the prefix-suppression problem in outcome-supervised reinforcement learning with verifiable rewards (RLVR), where a sequence-level scalar reward indiscriminately penalizes valid early steps in failed trajectories. Gated on the external verifier declaring a response incorrect, LC-GRPO reuses stored behavior-policy log-probabilities from rollouts to compute length-normalized step log-confidence and locates an estimated failure boundary via the largest thresholded drop between adjacent steps. It protects the preceding prefix by assigning zero advantage and applies a bounded, detached non-positive penalty to the failure suffix, while leaving correct-trajectory updates unchanged. Experiments are reported on Qwen3-4B-Base and Qwen3-8B-Base across six mathematical and code reasoning benchmarks.

Critical strengths

Addressing prefix suppression in failed RLVR rollouts without requiring external process reward models (PRMs), process labels, or extra forward passes is a practical and low-overhead direction. The verifier-gated and stop-gradient design is well-motivated, ensuring that endogenous confidence only shapes the support and weight of negative updates on already-failed responses without flipping advantage signs or overriding objective verifier outcomes.

Critical weaknesses

Incomplete Experimental Reporting and Unexecuted Protocols: The paper does not specify the SFT or RL training datasets. Appendices C–H largely describe reporting protocols without providing the promised statistical tests, confidence intervals, matched-compute VPPO results, or absolute systems costs. The AI Use Statement also retains unfilled template placeholders, raising serious concerns about manuscript completeness and reproducibility.

Suspiciously Regular Experimental Results: Across both model scales and all six benchmarks, the seven methods exhibit exactly the same performance ranking without a single reversal. More concerningly, SAS, UCAS, and CAPO show identical arithmetic increments across multiple benchmarks and scales (e.g., AIME24: 62.3 → 62.6 → 62.9). Such unusually uniform patterns across independent RL experiments raise concerns about the reliability of the reported results and warrant verification using raw run-level data.

Questions for authors

Missing Experimental Data and Protocol Templates: Please specify the exact SFT and RL training datasets (names, sources, and sample counts), and explain why Appendices C, D, E, and H contain unexecuted specification text ("must record", "Required report") while the VPPO comparison results, Holm-adjusted pp-values, and hardware systems tables explicitly cited in Section 7.2 are entirely absent from the paper?

Anomalous Monotonicity in Benchmark Scores: Please provide the raw run-level scores across all 3 seeds for every method in Table 5, and explain how distinct baselines (SAS, UCAS, CAPO) produced exact 0.1∼0.30.1 \sim 0.3 arithmetic progressions across all 6 benchmarks and both model scales without a single rank inversion?

AI Use Disclosure: Why does the AI Use Statement on page 10 still contain raw [tasks with required disclosure] template placeholders, and to what extent were generative AI tools used to draft the experimental protocols and tables in this submission?

1 comment

We sincerely thank the reviewer for checking the numbers closely and for recognizing that LC-GRPO addresses prefix suppression without PRMs, process labels, or extra forward passes, with a verifier-gated, stop-gradient design. The concern was justified: several submitted numbers did not match the re-run, and the general response lists the changes.


Q1. Please specify the SFT and RL training datasets.

R: The new Appendix C.2 and Table 6 (Pages 15–16) give the sources, counts, and a complete filter ledger.

(1) Pools. Each domain has an SFT corpus, an RL prompt pool, and a development pool, shared by all methods and scales. Math records come from NuminaMath and from problems a strong model generated with NuminaMath problems as seeds; generated problems pass the same verification and decontamination filters. Final sizes (math / code): SFT 101,471 / 70,530; RL 32,855 / 23,333; development 1,980 / 1,515. Each row of the sequential ledger sums to its final count.

(2) Decontamination. Exact matching, then 5-token MinHash with Jaccard ≥ 0.80, against all six benchmarks. It removed 512 math SFT records (168 exact, 344 near), 481 code SFT, 174 math RL, and 209 code RL; no match remains. Development pools serve only checkpoint selection and hyperparameter search.

(3) ProcessBench (Table 11, Page 20). 1,170 items sampled at random; 160 fit the position prior and are disjoint from evaluation; 210 are ineligible (50 without an annotated error, 90 with a correct answer, 70 parse failures). 800 failures, 200 per subset, are evaluated.


Q2. The appendices contain unexecuted protocol text, and the VPPO results, Holm-adjusted p-values, and systems tables are absent.

R: Every protocol in the revised appendix now has its results, and protocol text without results is removed. No "must record" or "Required report" text remains.

(1) Holm-adjusted p-values: all 24 paired comparisons with 95% bootstrap intervals (Appendix D.4, Table 8, Page 17).

(2) Systems cost: over 12 paired runs on 8×H100, LC-GRPO adds 1.1% wall time (range 0.5–1.9%) and 0.9% peak memory, with 98.9% of GRPO's throughput (Appendix L, Table 18, Page 24).

(3) VPPO: the matched-compute comparison was not run, so the protocol is removed and the paper makes no claim relative to VPPO.


Q3. Rankings are identical everywhere, and SAS, UCAS, and CAPO form arithmetic progressions.

R: We checked the submitted Table 1 column by column (7 columns × 2 scales). The method order was the same in all 14 columns, and the SAS → UCAS → CAPO steps were equal in 5 of them, a fair reason for concern. Because they did not match the re-run, the revision replaces every one of them with re-run scores computed from integer success counts.

The re-run does not show this pattern (Tables 7 and 9, Pages 17 and 19):

(1) SAS < UCAS < CAPO holds in only 4 of 14 columns, and the two steps are equal in 1 (4B All Avg., +0.3 and +0.3).

(2) The strongest baseline changes by task: DAPO, SAS, UCAS, or CAPO at 4B, and SAS, CAPO, or UCAS at 8B. Some baselines fall below simpler ones: DAPO below GRPO on 4B AIME25 (33.5 against 33.8), Dr. GRPO below GRPO on 8B AMC and LCB, and CAPO's 8B All Avg. (60.3) below SAS and UCAS (60.4).

(3) At the seed level, the eight rows appear in 35 different orders across the 42 seed–column cells (7 columns × 2 scales × 3 seeds).

The aggregate trend is the same in both runs: LC-GRPO is highest (51.4 at 4B, 62.4 at 8B), and step- and confidence-aware baselines (48.4–49.0, 60.3–60.4) are at or above outcome-only RL (47.3–48.4, 58.7–59.7). Seed SD of All Avg. is 0.14–0.73.


Q4. Please provide run-level scores for all three seeds.

R: Table 9 (Appendix E, Page 19) lists the seed-level scores of every method, task, and scale (48 rows). Table 7 gives means and seed SDs, and Table 8 the paired tests (10,000 bootstrap replicates and randomization draws, Holm within each scale and comparator). LC-GRPO beats CAPO on all three seeds in every cell except 4B AIME24 (two of three). We will release per-problem success counts and step-level log-confidences with the code.


Q5. The AI Use Statement contains placeholders, and to what extent were AI tools used for the protocols and tables?

R: The placeholder is replaced by the completed statement on Page 11. We apologize for the oversight. Generative AI tools were used for grammar, spelling, and typographical checks, reference verification, and help with drafting and editing text, including the appendix descriptions, and formatting LaTeX tables from the authors' result files. They produced no experimental result. The authors reviewed all suggested edits and take full responsibility for the final content, including all text, claims, proofs, experiments, and artifacts.


We hope the per-seed data, and the completed appendix address the concern, and we would be grateful if the reviewer would reconsider the rating.

Reviewer_nCqMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Localized-Credit GRPO (LC-GRPO), a method for improving credit assignment in outcome-supervised reinforcement learning. The key idea is to identify the potential failure boundary of incorrect reasoning trajectories using adjacent drops in step-level log-probabilities. The approach protects the preceding prefix from negative updates while applying bounded, non-positive advantages to the estimated failure suffix. LC-GRPO requires no additional process supervision or model forward passes. Experiments on Qwen3-4B and Qwen3-8B across six mathematical and coding benchmarks demonstrate improvements over GRPO and several step- or confidence-aware baselines. Additional experiments evaluate error localization, confidence calibration, computational efficiency, and the contributions of individual components.

Critical strengths
  1. The paper addresses a limitation of outcome-supervised RL such as GRPO: uniformly penalizing incorrect trajectories may suppress useful intermediate reasoning steps. LC-GRPO provides a lightweight solution by reusing stored rollout log-probabilities to estimate failure boundaries without additional model evaluations or process annotations. Its integration into GRPO is straightforward and introduces minimal computational overhead.

  2. LC-GRPO improves the six-task average over GRPO by 4.2 and 4.0 points on Qwen3-4B and Qwen3-8B, respectively, and outperforms the strongest reported baseline by 2.2 points at both scales. The paper includes multiple competitive baselines, matched rollout budgets, ablation studies, ProcessBench localization evaluations, and efficiency measurements. These experiments provide encouraging evidence for the effectiveness of the proposed approach.

Critical weaknesses
  1. Better credit assignment does not automatically imply better calibration. The ECE improvement is an empirical observation that requires a more careful mechanistic explanation.

  2. The method assumes that a sharp decrease in step-level log-probability indicates the onset of incorrect reasoning. However, low-probability transitions may arise from valid but uncommon reasoning steps, while common reasoning mistakes may remain highly probable.

  3. The main idea of protecting useful prefixes and penalizing erroneous steps has been explored in prior work. The main distinction is the use of adjacent log-probability drops rather than process supervision or confidence-based step selection. While computationally efficient, the conceptual novelty appears relatively incremental.

  4. Propositions 1–4 primarily establish straightforward properties of the constructed advantages, including sign preservation, bounded correction, and stability under perturbations. They do not provide convergence guarantees or establish that the estimated failure boundaries lead to better policy-gradient estimates. Consequently, the theoretical analysis offers limited justification for the observed performance improvements.

Questions for authors
  1. The proposed method modifies step-level advantages rather than the underlying GRPO optimization procedure. Can the authors clarify whether LC-GRPO is compatible with other policy-gradient algorithms, such as PPO, DAPO? Evaluating LC-based advantage shaping within at least one alternative optimization framework would help establish whether the improvements arise from the proposed localization mechanism or its interaction with GRPO-specific training dynamics.

  2. How strongly is the largest adjacent log-probability drop correlated with the actual first reasoning error? Can the authors analyze localization accuracy across different error types, step positions, and response lengths, particularly cases where incorrect reasoning remains highly confident?

1 comment

We sincerely thank the reviewer for the constructive suggestions, and for recognizing that LC-GRPO reuses stored log-probabilities without extra model evaluations or annotations and adds minimal overhead to GRPO. In the re-run (general response), the quoted gains become 4.1 and 3.7 All Avg. points over GRPO and 2.4 and 2.0 over the strongest baseline (4B, 8B).


Q1. Better credit assignment does not imply better calibration.

R: We agree, and the re-run shows that LC-GRPO does not improve ECE (.180 against .093 for GRPO); we withdraw the calibration claim. The mechanism changes ranking, not calibration (Appendix F, Page 18; Table 10, Page 20):

(1) What changes. λcβ\lambda c^\beta is largest on confident steps of verifier-rejected responses; correct responses are optimized as in GRPO. From step 0 to 500, mean min-step confidence qq on wrong responses falls from .837 to .639 and stays at .844–.848 on correct ones.

(2) Ranking improves. AUROC rises from .532 to .962 (GRPO .644; .949–.973 within length tertiles), Brier falls from .217 to .118 (GRPO .191), and wrong responses with q≥0.8q\ge0.8 fall from 72.7% to 4.9% (GRPO 63.2%).

(3) Why ECE rises. qq is well ranked but compressed toward the middle. The lowest bin holds 303 responses with mean qq .591 and accuracy .086, which alone contributes .085 of the .180; bins with q≥.85q\ge.85 have accuracy .99–1.00 against mean qq .86–.93. The penalty is bounded (λcβ≤0.4\lambda c^\beta\le0.4) and nothing maps qq to a probability (§7.4, Limitations).


Q2. Valid uncommon steps can have low probability; common mistakes can stay probable.

R: We agree that a drop alone is not an error; the design and new checks cover both cases (Appendix H, Tables 13–14, Pages 21–22).

(1) Drops in correct responses are common. 77.5% of 200 correct trajectories without process errors contain a drop of at least δ\delta. LC-GRPO acts on a drop only after the verifier rejects the response, so none receives a negative update.

(2) Uncommon valid steps in failed responses. When the valid prefix contains a low-confidence step (276 of 800 failures), Exact is .649, against .746 otherwise (GRPO .348 and .424). Such a drop can only move the boundary earlier: the first error stays in the penalized suffix and only the protected prefix shrinks.

(3) Confident errors are the real failure mode: when the first error has confidence ≥ 0.8 (203 failures), Exact is .305 and the detector falls back to b=1b=1 (GRPO's update) on 78% of them (Limitations, Page 10).


Q3. The conceptual novelty appears incremental.

R: The contribution is empirical, and the distinction the reviewer names (adjacent drops, not confidence-based step selection) is what matters.

(1) Against confidence-based selection. Training with the minimum-confidence step as boundary gives a policy localizing at .611, against .713 for maximum drop (Table 3, Page 8). The confidence-based baselines SAS, UCAS, and CAPO trail by 2.4–3.0 (4B) and 2.0–2.1 (8B) All Avg. (Table 1, Page 7), significantly in 9 of 12 cells against CAPO (Table 8, Page 17).

(2) Against a random boundary of the same form (Table 4b, Page 9; Table 16, Page 23; 4B, math, two seeds). Without the penalty, a located suffix mask reaches 56.81 Math Avg. and a random contiguous suffix 54.17 (GRPO 53.88; full method 58.11). Most of the gain comes from where the boundary is placed.


Q4. Propositions 1–4 do not give convergence or better gradient estimates.

R: We agree. §6 (Page 6) presents them as properties of the update that do not show convergence, boundary correctness, or accuracy gains; those rest on Q2, Q3, and Q6.


Q5. Is LC-GRPO compatible with PPO or DAPO?

R: Yes for DAPO, and retaining DAPO's discarded groups works best (Table 4b, Page 9; Appendix K, Table 17, Page 23; 4B, math, two seeds, same update and token caps). DAPO reaches 55.57 Math Avg.; DAPO+LC reaches 56.46 when all-wrong groups are discarded (1.04–1.06× DAPO's tokens) and 57.08 when they are retained (0.96–0.98×). Retaining all-wrong groups beats DAPO on both seeds (+0.93, +2.09); discarding them does not on seed 1 (56.07 against 56.22), since dynamic sampling removes LC-GRPO's signal there. Critic-based PPO was not tested (§8).


Q6. How well does the largest drop match the first error by position, length, and confidence?

R: On 800 ProcessBench failures, the trained policy's boundary is the annotated first error in 71.3% of cases (±1 step: 75.8%), against .356 at SFT and .398 under GRPO (Table 12, Page 21). By stratum (Table 13): .766 / .704 / .682 for early / middle / late first errors; .682–.733 across length tertiles; .851 and .305 for non-confident and confident first errors. ProcessBench has no error-type labels, so we cannot split by type.


We would be grateful if the reviewer would consider raising the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Esteemed Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful questions, constructive suggestions, and positive comments on LC-GRPO. We appreciate their recognition of:

  • A practical fix for prefix suppression in outcome-only RL, without PRMs, process labels, or extra forward passes (All reviewers).
  • A clean update design: verifier-gated and stop-gradient, separating prefix protection from suffix penalization with constrained advantage sign and support (Reviewers KgBQ and AeZb).
  • Minimal overhead, reusing stored log-probabilities (Reviewer nCqM).

Novelty. Existing step-credit methods rely on PRMs, process labels, or per-step confidence scores; LC-GRPO locates the first error from the policy's own likelihood changes, at no extra model cost:

  • Adjacent-drop boundary: the largest drop in step log-confidence marks where a failed response goes wrong. It trains a policy that localizes at .713 against .611 for a minimum-confidence boundary (Table 3, Page 8), and LC-GRPO leads the confidence-aware baselines SAS, UCAS, and CAPO by 2.0–3.0 All Avg. (Table 1, Page 7).
  • Verifier-gated credit: only rejected responses are split, so the valid prefix is protected and the error suffix is penalized, with +1.1% wall time and +0.9% memory over GRPO (Table 18, Page 24).

In response to their questions and suggestions, we provided the following key clarifications and additions:

QuestionReviewerOur Response
How strong and consistent are the gains?KgBQLC-GRPO is best at both scales (51.4 at 4B, 62.4 at 8B All Avg.), +4.1 / +3.7 over GRPO and +2.4 / +2.0 over the strongest baseline, and significant over GRPO in all 12 task–scale cells (Holm-corrected; Table 8, Page 17). All 48 seed-level scores, paired bootstrap intervals, and Holm p-values are reported (Tables 8–9, Pages 17–19).
What are the training data, and are they decontaminated?KgBQSources, counts, and a complete filter ledger for SFT, RL, and development pools (Appendix C.2, Table 6, Pages 15–16); exact plus MinHash decontamination against all six benchmarks leaves no remaining match.
Is the localization signal reliable from initialization through training?AeZbOn 800 ProcessBench failures, the SFT checkpoint already localizes at .356 Exact, three times the best positional or random control, and LC-GRPO raises it at every checkpoint to .713 (GRPO .398). At initialization the first error already receives negative credit in 89.4% of failures. A human-checked on-policy audit rises from .450 to .645 (Table 4a, Page 9; Tables 12 and 15, Pages 21–22).
Does better localization drive the task gain?AeZb, nCqMWith the mask form fixed, the located boundary reaches 56.81 Math Avg. against 54.17 for a random boundary (GRPO 53.88), and the confidence penalty brings the full method to 58.11 (Table 4b, Page 9; Table 16, Page 23). Localization improves first: by step 125, Exact rises from .356 to .484 before accuracy moves (Tables 10 and 12, Pages 20–21).
What does the confidence penalty change?nCqM, AeZbIt makes confidence separate correct from wrong responses: AUROC .962 against .644 for GRPO, Brier .118 against .191, and confident wrong responses fall from 63.2% to 4.9%, while confidence on correct responses stays at .844–.848 (Appendix F, Page 18; Table 10, Page 20).
How does LC-GRPO handle uncommon valid steps and different error positions?nCqMDrops act only after the verifier rejects a response, so no correct response is penalized. Localization holds across first-error position (.766 / .704 / .682 early / middle / late) and length tertiles (.682–.733), and stays at .649 even when the valid prefix contains a low-confidence step, against .348 for GRPO (Tables 13–14, Pages 21–22).
Is LC-GRPO compatible with other trainers?nCqMDAPO+LC reaches 57.08 Math Avg. against 55.57 for DAPO, improving on every seed (+0.93, +2.09) with fewer training tokens (Table 17, Page 23).

We thank the reviewers and the AC again for helping strengthen the paper. We hope these clarifications and additional results support favorable consideration of LC-GRPO as a verifier-gated method that assigns step-level credit in outcome-only RL from the policy's own confidence.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.