acceptodds
Under review as a conference paper at ICLR 2027

CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has significantly improved the reasoning capabilities of large language models (LLMs). However, existing RLVR methods provide limited control over intermediate reasoning quality. Value-based approaches such as PPO suffer from inaccurate credit assignment due to limited sampling, while process reward model (PRM)-based methods rely on costly human annotations and often produce unreliable process signals. To address these limitations, we propose Credit Assignment Policy Optimization (CAPO), a simple and efficient method. CAPO avoids training auxiliary models and instead leverages an off-the-shelf, general-purpose LLM as a Generative Process Reward Model (LLM-as-GenPRM) to generate step-wise critiques in a single pass based solely on step correctness. This enables deterministic credit assignment to refine tokens that would otherwise receive identical rule-based rewards. As a result, CAPO significantly simplifies the training pipeline. % while maintaining strong generality across a wide range of powerful, publicly available open-source models. Extensive experiments on LLaMA and Qwen backbones demonstrate that CAPO consistently outperforms six strong baselines across four mathematical benchmarks and three out-of-domain benchmarks. Further analyses show that CAPO’s credit assignment mechanism substantially improves reasoning quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, process reward, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uTSfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends standard outcome based RL for LLM reasoning with turn level rewards by proposing Credit Assignment Policy Optimization (CAPO). Specifically, they obtain turn-level correctness scores by prompting a separate (stronger) LLM, which is then used to provide turn-level rewards during RL. In experiments over Llama and Qwen models on math reasonign datasets, they show that it can outperform GRPO and other various baseline methods.

Critical strengths

This paper shows that process reward can be straightforwardly and successfully used for LLM post-training. This is notable as while PRM for test time scaling has been well studied, its success in RL is less so studied. The usage of an off the shelf LLM as the GenPRM also strongly indicates that specially trained reward models are unnecessary. They also provide comprehensive experiments (such as those in section 4.3) that CAPO does in fact reduce process errors.

Critical weaknesses

The method is still missing some ablations that would convince me that the stepwise level credit assignment is truly what mattered. Specifically, what if the reward shaping is still at the trajectory level -- i.e., we add a penalizing term based on the proportion of steps that were labeled as incorrect. I would hypothesize this would still do better than vanilla GRPO but not as good as using granular step level rewards.

Furthermore, the method is limited by requiring explicit stepwise outputs. Is this necessary? What happens if we just arbitrarily partition the model CoT into "steps" i.e., just partition into roughly equal sized chunks without caring whether each semantically represents a full coherent step. See [1] by what I mean (albeit their's was more about simply training a PRM, rather than using it for RL).

[1] Luo et al. Improve Mathematical Reasoning in Language Models by Automated Process Supervision. arXiv 2024.

Questions for authors

See weaknesses.

Also for equation (2), if I understand it correctly, this means that incorrect steps can still receive positive reinforcement -- namely consider the case where most trajectories are incorrect at the outcome level and thus the baseline is skewed low. Here, an incorrect step in a correct-at-the-outcome-level trajectory can still have positive advantage. Is this intended behavior-- since presumably we would like to disincentive incorrect steps regardless of whether the trajectory at a outcome level is correct or not?

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for recognizing the practical value of an off-the-shelf GenPRM and the improvements in reasoning quality. Your questions help clarify the particular learning signal provided by CAPO.

The corresponding revisions are highlighted in blue in the revised manuscript.

1. Trajectory-level shaping versus local credit assignment

A trajectory-level error-rate penalty can distinguish responses according to their process quality, but it still assigns the same reward and normalized advantage to every token within a response. CAPO additionally distinguishes the locations receiving that feedback. For example, if a response contains one detected error among several valid segments, a trajectory-level penalty changes the score of every segment uniformly, whereas CAPO directly applies the penalty to the identified segment.

This distinction persists even when the average raw reward is matched. Writing eite_{it} for the error mask and eˉiē_i for its mean within response i, the uniform version uses Rittrajectory=Cri−PeˉiR_{it}^{\mathrm{trajectory}} = Cr_i - Pē_i, while CAPO uses RitCAPO=Cri−PeitR_{it}^{\mathrm{CAPO}} = Cr_i - Pe_{it} . Their per-response mean raw rewards are equal, but only CAPO retains within-response differentiation.

The reported experiments evaluate the complete CAPO formulation. Section 4.3 shows improvements in independently evaluated process quality, and Table 6 supports balancing the local process signal with the outcome objective.

2. Are explicit stepwise outputs necessary?

The policy does not need to generate indexed steps. As described in Appendix A.3 and Appendix B, paragraph boundaries are identified after generation and the tags are inserted into the judge input. The optimization requires a correspondence between a judged segment and its constituent tokens; equal-sized chunks can therefore be used with the same objective.

Luo et al.’s observation is relevant to this flexibility. Our choice of paragraph boundaries provides a simple and reproducible unit for judging mathematical solutions. Since alternative boundaries may change the semantic content of each unit and the resulting labels, our empirical evidence concerns the stated paragraph-based implementation. We have clarified that the framework’s compatibility with other partitions is separate from their empirical performance.

3. Positive advantages for erroneous steps

Yes, a flagged step in an outcome-correct response can have a positive advantage. This is consistent with the outcome-first hierarchy described in Section 3.2. With a common group standard deviation σ>0, tokens sharing the same outcome reward satisfy

= \frac{W_{\mathrm{process}}}{\sigma} > 0.$$ The flagged segment therefore receives a lower relative advantage even when both advantages are positive. When the group baseline is low, a successful response can retain positive reinforcement, with weaker reinforcement assigned to its flagged segments. This preserves the dominant outcome signal while providing a local distinction within the successful response. When all outcome rewards in the group are identical, the outcome term cancels. If both flagged and unflagged tokens are present, flagged tokens then have negative advantages and unflagged tokens have positive advantages. Thus, the process signal can distinguish reasoning quality even when the outcome reward alone provides no within-group distinction. This behavior also connects to Table 6: the outcome-dominant setting achieves the strongest aggregate performance among the tested reward configurations. The precise claim is relative down-weighting of detected errors; we have made this explicit in the description of Eq. (2).
Reviewer_ZzUKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Credit Assignment Policy Optimization (CAPO), which uses an off-the-shelf LLM to identify erroneous reasoning steps in policy-generated responses. Each token receives a verifier-based outcome score, with an additional penalty applied to tokens belonging to identified errors. These scores are standardized across the rollout group and directly used as advantages in a clipped policy optimization objective, without accumulating process rewards across steps. CAPO emphasizes outcome supervision through asymmetric reward weighting and optionally aggregates multiple critiques through voting.

Critical strengths

Practical process supervision without specialized reward-model training. CAPO provides a straightforward way to incorporate feedback using an off-the-shelf judge. A single critique can evaluate multiple reasoning steps, and the resulting penalties integrate directly into group-relative policy optimization.

Critical weaknesses
  1. Similar ideas appear to have been explored in several existing papers, suggesting limited novelty.
  2. The claimed “2.5× efficiency improvement” is primarily based on comparing CAPO at step 60 with the baseline at step 176. Could the authors provide performance-versus-GPU-hours curves that include all judge-related overhead, using a common target accuracy or stopping criterion, to verify whether this efficiency advantage holds consistently?
  3. Llama-3 and Qwen2.5 seem rather outdated.
Questions for authors

Can the rewards computed by CAPO be used with PURE’s advantage estimation procedure?

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for highlighting the practical value of obtaining process supervision without specialized reward-model training. We address the contribution, computational cost, policy backbones, and compatibility with PURE below.

The corresponding revisions are highlighted in blue in the revised manuscript.

1. Contribution and relationship to prior work

CAPO’s contribution is a concrete credit-assignment method that turns a general-purpose judge’s error locations into policy-training signals. A generated critique identifies erroneous segments across a rollout; CAPO maps these locations to local token penalties, combines them with verifiable outcome scores, and directly normalizes the resulting scores into advantages without accumulating rewards across steps. The asymmetric weighting prioritizes final correctness while differentiating process quality within the same outcome category.

This design provides process supervision without task-specific PRM training. Its empirical value is supported by comparisons across policy families and scales, the reward-weighting analysis, and process-quality evaluation using a separate GPT-4o evaluator. A lightweight judge is sufficient to obtain competitive results: on the 7B policy, the 4B single-critique configuration reaches an All Mean of 48.22, compared with 47.81 and 47.50 for PURE with 7B and 72B PRMs. We position the contribution in this specific design and its empirical characterization, within the established literature on generative evaluation and process-supervised RL.

2. GPU-hour accounting and judge overhead

Appendix A.1 already reports wall-clock measurements and GPU-hour accounting for Qwen2.5-Math-7B with a locally deployed Qwen2.5-14B-Instruct judge. The measured per-step time increases from approximately 47 seconds for SimpleRL-Zoo to 55 seconds for CAPO, including the additional reward-evaluation latency (12.1 vs. 4.5 seconds). This overhead is incorporated into the reported computational costs.

Using the baseline’s final Math Mean of 52.99 as the performance reference, CAPO reaches 53.28 at step 60 with approximately 88 GPU-hours, compared with approximately 220.5 GPU-hours for the baseline’s reported final checkpoint. The approximately 2.5× figure refers to this explicit performance–compute comparison.

The later checkpoint provides additional evidence: at step 120, CAPO achieves a Math Mean of 55.23 and an All Mean of 50.55, exceeding the baseline’s final scores of 52.99 and 49.03 by 2.24 and 1.52 percentage points, respectively, while using approximately 20% fewer GPU-hours. The efficiency evidence therefore includes both the early mathematical-performance target and a later checkpoint with higher mathematical and overall performance at lower compute cost.

We have foregrounded this existing analysis and specified the target metric and timing accounting.

3. Choice of policy models

The experiments evaluate the training mechanism across two policy families and scales from 1B to 14B. The 14B-policy results provide a further scaling test: Table 5 shows gains of 2.69 points in Math Mean and 2.26 points in All Mean over SimpleRL-Zoo. We believe all the existing evaluation across 1B to 14B are enough to show the effectiveness of our method.

4. Can CAPO’s rewards be used with PURE?

Yes. The generative evaluator and the advantage estimator are separable. The judge’s step labels can be represented as process scores, passed through PURE’s min-form reward transformation and return computation, and combined with a separate outcome component. This would produce a GenPRM+PURE hybrid.

The distinction lies in how the feedback is used: PURE propagates transformed future process rewards to preceding actions, whereas CAPO directly normalizes scores containing local token penalties. Consequently, the same evaluator can support different credit-assignment rules. Compatibility is consistent with the modularity of our feedback mechanism and does not make the two policy updates equivalent.

Reviewer_VvyZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the credit assignment problem faced by RLVR in enhancing the reasoning capabilities of LLMs, and proposes the Credit Assignment Policy Optimization (CAPO) method. The authors point out that existing RLVR methods typically struggle to distinguish the quality of intermediate steps in the reasoning process, and traditional value-based or PRM-based methods may face issues such as inaccurate credit estimation, high additional model training costs, and unreliable reward signals. To this end, CAPO uses a large language model that does not require task-specific fine-tuning as a Generative Process Reward Model (LLM-as-GenPRM), combining the generative error localization capability of a general LLM with reinforcement learning based on final outcomes to build a generative local credit assignment framework that without training a dedicated PRM. This paper conduct experiments on multiple different models and datasets. The results show that CAPO achieves better performance than standard RLVR and some PRM-based methods under multiple settings. In addition, the authors verify the effectiveness of CAPO and its improvement in reasoning quality through further analysis.

Critical strengths
  • The paper focuses on the problem of fine-grained credit assignment in RLVR, which is an interesting and valuable question. At the same time, this paper is simple and easy to understand, and effectively constructs fine-grained rewards by introducing the LLM-as-GenPRM paradigm, and combines them with verifiable reward from the final answer for policy optimization.
  • Well-structured paper and detailed analyses. Results on a variety of experimental settings and datasets further strengthen the conclusions.
  • In addition to the main results, the authors further provide more valuable insights through analytical experiments such as the step error ratio (SER), the token error ratio (TER), the ProcessBench experiment, reward configuration ablation, and training efficiency analysis.
Critical weaknesses
  • The evaluation objectives of ProcessBench are not entirely consistent with the error detection capabilities that CAPO actually requires. The main task of the original ProcessBench is to identify the first erroneous step in the reasoning process, or to determine that there are no errors in the reasoning process. However, the capability actually required by CAPO is more complex. It needs to identify all local erroneous steps throughout the entire reasoning trajectory and convert them into token-level training signals. Therefore, the current experimental results are insufficient to prove that CAPO can reliably identify all erroneous steps.
  • The implementation of CAPO includes multiple designs, but the ablation experiment in Table 6 compare three different reward configuration methods, yet do not sufficiently analyze the contribution of the LLM-as-a-GenPRM paradigm's identification of local erroneous steps and penalty signals to performance improvement.
  • The paper uses "\n \n" as the step separator, and then provides step indices to GenPRM through the <step i> tags, which means that CAPO relies on accurate step segmentation to accomplish local credit assignment. Although this implementation is simple and clear, paragraph divisions in format do not necessarily correspond to logical reasoning steps. This indicates that the optimization behavior of the method depends not only on reasoning correctness, but may also be affected by text structure and segmentation granularity. The paper lacks further empirical results for different segmentation strategies, which may affect the effectiveness and generalization of the method.
  • The paper states in Section 4.2 that CAPO consistently outperforms six baselines across all evaluated model scales and families, but Table 2 shows that this conclusion needs to be qualified according to the specific experimental setup. Although CAPO shows good performance overall, its advantage does not hold simultaneously across all configurations and all tasks.
Questions for authors

Please refer to the weaknesses above.

1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for recognizing the value of fine-grained credit assignment and the breadth of our analyses. We clarify how the existing results support CAPO and distinguish the scope of the different evaluations below.

The corresponding revisions are highlighted in blue in the revised manuscript.

1. Reliability of the process feedback

ProcessBench provides a controlled evaluation against human-annotated first-error locations. In Table 9, the off-the-shelf 4B judge obtains an average localization F1 of 77.2, comparable to the trained 72B PRM at 77.1; majority voting increases this score to 80.4. These results establish first-error localization capability. They complement, rather than replace, our evaluation of the resulting policy’s reasoning quality.

For the latter, Section 4.3 uses GPT-4o, an evaluator distinct from the Qwen judges used during training. Tables 3–4 show lower SER and TER in every reported model–domain aggregate comparison. For example, on Qwen2.5-Math-1.5B, Math SER decreases from 27.24 to 21.91 and Math TER from 33.78 to 24.65. Thus, the evidence extends beyond answer accuracy to the quality of the generated reasoning.

CAPO can apply localized corrections using a partial set of detected errors: unflagged tokens retain the outcome-based raw score in Eq. (1). Exhaustive recovery of every error is therefore not a prerequisite for forming its update. The ProcessBench results support detection capability, while the downstream analyses support the usefulness of this feedback; we have made these roles explicit without presenting first-error evaluation as an all-error annotation benchmark.

2. Contribution of localized penalties and reward weighting

The role of localization follows directly from Eqs. (1)–(2). Uniform trajectory rewards cannot distinguish tokens within the same response. CAPO introduces a within-response advantage difference of W_process/σ between unflagged and flagged tokens, where σ is the group standard deviation. This provides a mechanism for assigning lower relative credit to identified errors while retaining stronger relative credit for the remaining segments.

Table 6 examines a different, complementary question: how to balance this process signal with the outcome objective. With C=1, the asymmetric P=0.1 setting achieves an All Mean of 33.22, compared with 32.48 for P=1 and 31.48 for P=2. Together with the independent process-quality evaluation, these results support the complete outcome-anchored local-credit formulation. We have distinguished this evidence from a component-by-component causal ablation.

3. Step segmentation

CAPO does not require the policy to generate explicit <step i> tags. Appendix A.3 and Appendix B describe how these tags are inserted after generation, using paragraph boundaries to construct the judge input. The optimization only needs a mapping from a judged segment to its tokens, so it does not depend on a particular delimiter or manually annotated step boundaries. Paragraph segmentation offers a simple, reproducible implementation across the evaluated model families.

Alternative partitions, including fixed-length chunks, are compatible with the same optimization rule. Their boundaries may change the judge’s decisions and penalty coverage. We have clarified this distinction in Appendix A.3.

4. Scope of the performance statement

The intended claim concerns aggregate performance under specified configurations. With the same 14B-judge, single-critique configuration, CAPO achieves the highest reported Math Mean and All Mean on all three policy backbones in Table 2. The All Mean gains over the strongest reported baseline are +2.02 points on Qwen2.5-Math-7B (49.83 vs. 47.81), +3.17 on Qwen2.5-Math-1.5B (37.24 vs. 34.07), and +1.72 on Llama-3.2-1B (21.98 vs. 20.26). We have stated this configuration-specific result precisely.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.