acceptodds
Under review as a conference paper at ICLR 2027

SkillCritic: Skill-Aware Calibration of Generative Critics for Mathematical Reasoning

Abstract

Reinforcement learning for mathematical reasoning requires finer-grained credit assignment than outcome-level rewards alone can provide. Generative critics are a promising direction because they evaluate partial solutions segment by segment, but we find that they can still assign systematically over-optimistic scores to prefixes that already contain recurring reasoning failure patterns. We propose SkillCritic, a skill-aware calibration mechanism that uses structured failure-pattern priors to convert raw segment scores into risk-calibrated token-level training signals via a bounded soft multiplicative correction, without replacing the critic or introducing an external verifier. Under a unified implementation framework, we compare SkillCritic with matched value-free, conventional critic, and generative critic baselines. SkillCritic shows its clearest gains in lightweight training, where it delivers stronger and more robust improvements, while remaining competitive under full RLPT and showing more concentrated benefits on harder contest-style math subsets. We further diagnose a characteristic instability in prolonged full-RLPT training of generative critics, and find that it is better explained by critic score collapse and downstream baseline miscalibration than by direct explosion of raw value magnitudes. Overall, our results suggest that soft skill-aware calibration is an effective inductive bias for generative-critic reinforcement learning, especially in lightweight regimes and on more challenging mathematical reasoning tasks.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning for Language Models;Mathematical Reasoning;Generative Critics;Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mTKuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SkillCritic, a method for improving credit assignment in generative-critic-based reinforcement learning. It uses a predefined failure-skill bank to identify potential reasoning errors and combines these signals with critic confidence to calibrate the rewards used for actor training.

Experiments show that SkillCritic improves overall accuracy over the GenAC-style baseline with an SFT Critic, while achieving comparable overall performance with an RL-Post-Trained Critic. The paper also analyzes the effectiveness of failure skills and the stability of the critic during training.

Critical strengths

1. Simple and targeted method.

The paper addresses the issue of generative critics assigning overly high scores to flawed reasoning steps. SkillCritic uses a predefined failure-skill bank to selectively calibrate rewards for potentially problematic steps. The method does not require modifying the critic's training objective and can be easily integrated into existing generative-critic-based RL frameworks.

2. Experimental evidence supporting the method.

With an SFT Critic, SkillCritic improves Avg7 accuracy from 35.17% to 37.14% over the GenAC-style baseline. The authors also include randomized-skill controls and skill-hit analysis to examine the role of failure skills.

Critical weaknesses

1. Limited improvements and sensitivity to critic initialization.

While SkillCritic shows clear gains over the GenAC-style baseline with an SFT Critic, the two methods achieve nearly identical Avg7 scores with an RL-Post-Trained Critic. Although SkillCritic improves AIME-family performance, most of the gain comes from AIME24.

2. Insufficient validation of the failure-skill bank.

SkillCritic relies on four manually defined failure skills and text-based rules to identify potential reasoning errors. However, the paper mainly evaluates these rules through their correlation with final-answer correctness, without reporting step-level precision, recall, or false-positive rates. It is therefore unclear how reliably the rules identify reasoning steps that actually need calibration, or how well they generalize to different reasoning patterns.

Questions for authors

1. Limited performance gains.

SkillCritic shows limited overall improvements over GenAC-style, especially with an RL-Post-Trained Critic, where Avg7 remains nearly unchanged. How do the authors explain this result? Does skill-aware calibration still provide meaningful and consistent benefits with a more extensively trained critic?

2. Comparison with related methods.

Prior works such as PathFinder-PRM and Step-wise Rubric Rewards also use structured error information or evaluation criteria to improve reward signals for reasoning. Could the authors clarify how SkillCritic differs from these similar approaches and what advantages its fixed failure-skill rules offer for reward calibration?

Reviewer_9yvhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SkillCritic, a framework for mitigating misleading credit signals caused by over-optimistic generative critics in mathematical reasoning. It identifies recurring reasoning failures through a rule-based failure-skill bank and combines these signals with critic confidence to calibrate actor-facing rewards without altering the critic architecture or training objective. Experiments on seven mathematical reasoning benchmarks show that SkillCritic improves Avg7 from 35.17% to 37.14% with an SFT-initialized critic, while achieving comparable overall performance but higher AIME-family accuracy (13.02% to 14.37%) with an RL-post-trained critic. Additional analyses examine failure-skill relevance and critic stability.

Critical strengths

Practical approach. The paper addresses over-optimistic generative critic signals through a simple, interpretable calibration mechanism. Its ability to integrate into existing actor–critic pipelines without modifying the critic architecture or training objective makes the approach practical and potentially reusable.

Critical weaknesses
  1. Limited methodological novelty and generalizability. The failure-skill bank consists of only four manually defined error patterns detected through heuristic rules. While the calibration mechanism is simple and practical, its methodological novelty is limited, and its generalizability to diverse reasoning errors and tasks remains unclear.

  2. Limited empirical gains. With the RL-post-trained critic, SkillCritic achieves no improvement in overall Avg7 performance (37.71% vs. 37.77%), while its AIME-family gains are largely driven by AIME24. Moreover, experiments are restricted to Qwen3-8B-Base, with limited multi-seed validation, leaving the robustness and generalizability of the reported improvements insufficiently established.

Questions for authors
  1. Failure-skill detection reliability: How were the four manually defined failure patterns selected? Can the authors provide evidence of their detection accuracy, particularly false-positive rates on unseen reasoning trajectories?

  2. Component-wise ablation: Can the authors isolate the contributions of the confidence gate and adaptive correction mechanism? How does SkillCritic compare with simpler fixed reward downweighting using the same failure-skill detectors?

  3. Robustness of empirical gains: Given that the RL-post-trained critic shows no overall Avg7 improvement and the AIME-family gains are largely driven by AIME24, can the authors provide complete multi-seed results or statistical significance analyses?

Reviewer_cturOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper targets over-optimistic value estimates from generative critics (GenAC-style) in RL for math reasoning. It hand-specifies a bank of four recurring “failure skills” (unverified uniformity, unverified aggregation, risky algebraic simplification, missed depletion), detects them with keyword/structural regexes on each reasoning prefix, and when a detection coincides with a high critic score (sigmoid gate around an EMA of the batch 80th percentile), multiplicatively shrinks the actor-facing segment reward by at most 50%, with an adaptive scale and warm-up. The critic, its target, and PPO/GAE are untouched. On Qwen3-8B-Base over seven benchmarks: with an SFT-initialised critic, Avg7 35.17→37.14 and AIME-family 10.72→13.47; with an RL-post-trained critic, Avg7 is flat (37.71 vs 37.77) and AIME-family 13.02→14.37, driven almost entirely by AIME24 (16.00→20.12). Supporting analyses: skill triggers are 3.16× enriched on high-score-but-wrong trajectories vs random triggers; a frequency-matched random-regex control (RandomSkill-B) erases most of the AIME advantage; the gap is 5.5pp on skill-hit trajectories vs 0.3pp on non-hit; and a separate critic-collapse diagnosis.

Critical strengths
  1. Clean, correctly posed problem: critic reliability is failure-type-dependent, and Table 2’s selectivity analysis (3.16× on high-score-wrong vs 1.25× on high-score-right) is the most convincing evidence in the paper.

  2. The intervention is minimal, bounded, actor-side only, and cheap, so it composes with any segment-level critic pipeline.

  3. The authors have the right instincts for mechanism attribution (randomised-trigger control, hit-conditioned split) and are candid about Avg7 parity, the hit-conditioning confound, single-seed status, and critic instability.

Critical weaknesses
  1. Effect sizes are at the noise floor. All results are seed 0 (App. D.6 says seed variants exist only in a “training plan”; the quoted ±0.5 AIME-family spread is not measured).

  2. In the RLPT regime the method is a trade, not a gain: non-AIME benchmarks fall ~1.1pp (56.34→55.22 at best checkpoints; 56.27→55.14 averaged over late checkpoints), AMC23 and MATH-500 regress, and RandomSkill-B beats SkillCritic on AIME25.

  3. The method is under- and inconsistently specified. §3.4 defines w_t, the EMA statistic, and corrects r_k; App. B.3 drops w_t, writes u_k = p_k(1−ρ_k) (critic score, not reward), and introduces a top-k=4 segment restriction absent from the main text. One sentence in §3.4 is grammatically broken mid-definition of τ_t. The regexes themselves are never shown, so the core artefact is unreproducible from the paper.

  4. “Failure skill” overclaims what is four hand-written regexes with a fixed severity 0.7. Detector precision/recall is never measured against step-level ground truth (ProcessBench or Hard2Verify are cited but unused); Table 2 already implies a 22.7% false-positive rate on correct high-score trajectories. No comparison to any PRM baseline, and the GenAC baseline is an unofficial reimplementation whose faithfulness I cannot assess.

Questions for authors

Please see the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.