- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper extends standard outcome based RL for LLM reasoning with turn level rewards by proposing Credit Assignment Policy Optimization (CAPO). Specifically, they obtain turn-level correctness scores by prompting a separate (stronger) LLM, which is then used to provide turn-level rewards during RL. In experiments over Llama and Qwen models on math reasonign datasets, they show that it can outperform GRPO and other various baseline methods.
Critical strengths
This paper shows that process reward can be straightforwardly and successfully used for LLM post-training. This is notable as while PRM for test time scaling has been well studied, its success in RL is less so studied. The usage of an off the shelf LLM as the GenPRM also strongly indicates that specially trained reward models are unnecessary. They also provide comprehensive experiments (such as those in section 4.3) that CAPO does in fact reduce process errors.
Critical weaknesses
The method is still missing some ablations that would convince me that the stepwise level credit assignment is truly what mattered. Specifically, what if the reward shaping is still at the trajectory level -- i.e., we add a penalizing term based on the proportion of steps that were labeled as incorrect. I would hypothesize this would still do better than vanilla GRPO but not as good as using granular step level rewards.
Furthermore, the method is limited by requiring explicit stepwise outputs. Is this necessary? What happens if we just arbitrarily partition the model CoT into "steps" i.e., just partition into roughly equal sized chunks without caring whether each semantically represents a full coherent step. See [1] by what I mean (albeit their's was more about simply training a PRM, rather than using it for RL).
[1] Luo et al. Improve Mathematical Reasoning in Language Models by Automated Process Supervision. arXiv 2024.
Questions for authors
See weaknesses.
Also for equation (2), if I understand it correctly, this means that incorrect steps can still receive positive reinforcement -- namely consider the case where most trajectories are incorrect at the outcome level and thus the baseline is skewed low. Here, an incorrect step in a correct-at-the-outcome-level trajectory can still have positive advantage. Is this intended behavior-- since presumably we would like to disincentive incorrect steps regardless of whether the trajectory at a outcome level is correct or not?
Rebuttal by Authors
Thank you for recognizing the practical value of an off-the-shelf GenPRM and the improvements in reasoning quality. Your questions help clarify the particular learning signal provided by CAPO.
The corresponding revisions are highlighted in blue in the revised manuscript.
1. Trajectory-level shaping versus local credit assignment
A trajectory-level error-rate penalty can distinguish responses according to their process quality, but it still assigns the same reward and normalized advantage to every token within a response. CAPO additionally distinguishes the locations receiving that feedback. For example, if a response contains one detected error among several valid segments, a trajectory-level penalty changes the score of every segment uniformly, whereas CAPO directly applies the penalty to the identified segment.
This distinction persists even when the average raw reward is matched. Writing eit for the error mask and eˉi for its mean within response i, the uniform version uses Rittrajectory=Cri−Peˉi, while CAPO uses RitCAPO=Cri−Peit . Their per-response mean raw rewards are equal, but only CAPO retains within-response differentiation.
The reported experiments evaluate the complete CAPO formulation. Section 4.3 shows improvements in independently evaluated process quality, and Table 6 supports balancing the local process signal with the outcome objective.
2. Are explicit stepwise outputs necessary?
The policy does not need to generate indexed steps. As described in Appendix A.3 and Appendix B, paragraph boundaries are identified after generation and the tags are inserted into the judge input. The optimization requires a correspondence between a judged segment and its constituent tokens; equal-sized chunks can therefore be used with the same objective.
Luo et al.’s observation is relevant to this flexibility. Our choice of paragraph boundaries provides a simple and reproducible unit for judging mathematical solutions. Since alternative boundaries may change the semantic content of each unit and the resulting labels, our empirical evidence concerns the stated paragraph-based implementation. We have clarified that the framework’s compatibility with other partitions is separate from their empirical performance.
3. Positive advantages for erroneous steps
Yes, a flagged step in an outcome-correct response can have a positive advantage. This is consistent with the outcome-first hierarchy described in Section 3.2. With a common group standard deviation σ>0, tokens sharing the same outcome reward satisfy
= \frac{W_{\mathrm{process}}}{\sigma} > 0.$$ The flagged segment therefore receives a lower relative advantage even when both advantages are positive. When the group baseline is low, a successful response can retain positive reinforcement, with weaker reinforcement assigned to its flagged segments. This preserves the dominant outcome signal while providing a local distinction within the successful response. When all outcome rewards in the group are identical, the outcome term cancels. If both flagged and unflagged tokens are present, flagged tokens then have negative advantages and unflagged tokens have positive advantages. Thus, the process signal can distinguish reasoning quality even when the outcome reward alone provides no within-group distinction. This behavior also connects to Table 6: the outcome-dominant setting achieves the strongest aggregate performance among the tested reward configurations. The precise claim is relative down-weighting of detected errors; we have made this explicit in the description of Eq. (2).