- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SkillCritic, a method for improving credit assignment in generative-critic-based reinforcement learning. It uses a predefined failure-skill bank to identify potential reasoning errors and combines these signals with critic confidence to calibrate the rewards used for actor training.
Experiments show that SkillCritic improves overall accuracy over the GenAC-style baseline with an SFT Critic, while achieving comparable overall performance with an RL-Post-Trained Critic. The paper also analyzes the effectiveness of failure skills and the stability of the critic during training.
Critical strengths
1. Simple and targeted method.
The paper addresses the issue of generative critics assigning overly high scores to flawed reasoning steps. SkillCritic uses a predefined failure-skill bank to selectively calibrate rewards for potentially problematic steps. The method does not require modifying the critic's training objective and can be easily integrated into existing generative-critic-based RL frameworks.
2. Experimental evidence supporting the method.
With an SFT Critic, SkillCritic improves Avg7 accuracy from 35.17% to 37.14% over the GenAC-style baseline. The authors also include randomized-skill controls and skill-hit analysis to examine the role of failure skills.
Critical weaknesses
1. Limited improvements and sensitivity to critic initialization.
While SkillCritic shows clear gains over the GenAC-style baseline with an SFT Critic, the two methods achieve nearly identical Avg7 scores with an RL-Post-Trained Critic. Although SkillCritic improves AIME-family performance, most of the gain comes from AIME24.
2. Insufficient validation of the failure-skill bank.
SkillCritic relies on four manually defined failure skills and text-based rules to identify potential reasoning errors. However, the paper mainly evaluates these rules through their correlation with final-answer correctness, without reporting step-level precision, recall, or false-positive rates. It is therefore unclear how reliably the rules identify reasoning steps that actually need calibration, or how well they generalize to different reasoning patterns.
Questions for authors
1. Limited performance gains.
SkillCritic shows limited overall improvements over GenAC-style, especially with an RL-Post-Trained Critic, where Avg7 remains nearly unchanged. How do the authors explain this result? Does skill-aware calibration still provide meaningful and consistent benefits with a more extensively trained critic?
2. Comparison with related methods.
Prior works such as PathFinder-PRM and Step-wise Rubric Rewards also use structured error information or evaluation criteria to improve reward signals for reasoning. Could the authors clarify how SkillCritic differs from these similar approaches and what advantages its fixed failure-skill rules offer for reward calibration?