acceptodds
Desk rejected at ICLR 2027

Overcoming Skill Credit Blindness: Skill-Conditioned Policy Optimization for LLM Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning reusable reasoning skills through reinforcement learning (RL) is essential for improving both training efficiency and the ability of large language models (LLMs) to solve challenging reasoning tasks. However, most existing methods rely on Group Relative Policy Optimization (GRPO), whose group-relative baseline can absorb systematic skill effects, obscuring whether performance gains arise from policy improvement or injected skills. We term this credit-assignment failure skill credit blindness. To address it, we propose Skill-Conditioned Policy Optimization (SCPO), which explicitly conditions policy optimization on skills and reconstructs the advantage signal for reliable attribution. To isolate policy quality from skill effects, SCPO normalizes rewards only within the same problem and skill condition. It then uses paired skill-conditioned and skill-free rollouts to estimate each skill’s instance-level marginal contribution, which modulates the advantage to reinforce beneficial skills and suppress harmful ones. Finally, SCPO computes cross-problem skill-level gains to support the evolution of the skill library. Experiments with Qwen3-series models across multiple mathematical reasoning benchmarks show that SCPO consistently outperforms GRPO, demonstrating skill-based RL depends not only on learning reusable skills, but also on correctly attributing skill contributions during policy optimization.

Primary area:
reinforcement learning
Keywords:
Skill Learning, Reinforcement Learning, Credit Assignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MmrVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies skill blindness as a credit-assignment failure in skill-based RL. Then it proposes Skill-Conditioned Policy Optimization (SCPO), which address skill credit blindness. Besides, SCPO aggregates skill utility during training to update the skill library. Experimental results show superior performance compared to GRPO.

Critical strengths
  1. This paper identifies skill credit blindness and proposes SCPO to address it.
  2. Experimental results show superior performance compared to GRPO.
Critical weaknesses
  1. The baseline comparison is too limited. The main experiments compare SCPO only against vanilla GRPO, despite this paper being positioned in the broader literature on skill-based RL and self-evolving skill libraries. Comparisons with stronger skill-learning or alternative credit-assignment baselines are necessary to determine whether the gains specifically come from SCPO's proposed credit decomposition.
  2. The statistical evidence on several benchmarks is relatively weak. AIME 2024/2025/2026 and HMMT 2025 contain only 30 test problems, and although the authors report results over 10 seeds, no statistical significance tests or confidence intervals are provided.
  3. The construction of the initial skill library is unclear. SCPO critically relies on an external skill library, but this paper does not explain how this library is initialized before training, including its source, size, and quality-control procedure. The appendix describes how new skills are extracted and evolved during training, but this does not specify the initial library on which the method depends.
  4. The training reward comparison in Figure 2 is potentially confounded by different reward definitions. SCPO introduces additional reward shaping and also defines an optional diversity-aware reward before advantage estimation, so a higher mean training reward than GRPO is not necessarily evidence of a better learned policy unless exactly the same reward is used for both methods. In contrast, the batch pass rate is substantially lower for SCPO early in training and only becomes roughly comparable to GRPO later, which provides limited evidence for the claimed optimization advantage.
  5. It is unclear whether SCPO and GRPO are compared under a matched sampling budget. SCPO requires both skill-conditioned and no-skill rollouts for the same problem to estimate paired skill gain, introducing additional rollout generation beyond standard GRPO. This paper does not report the total number of generated tokens or demonstrate that GRPO receives an equivalent rollout budget, so part of the improvement may come from additional sampling rather than the proposed credit-assignment mechanism.
  6. The choice of mathematical reasoning as the sole evaluation domain limits the empirical validation of SCPO. Skill-based RL is particularly relevant in agentic settings where skills must be repeatedly retrieved, composed, and transferred across heterogeneous tasks, whereas mathematical reasoning provides a relatively narrow and static testbed for these capabilities. A more convincing evaluation would include agentic or interactive environments where skill reuse and transfer are directly exercised.
Questions for authors

Please refer to the Weaknesses.

Reviewer_fAqXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Skill-Conditioned Policy Optimization (SCPO) for skill-augmented LLM reinforcement learning. SCPO separates within-condition behavior credit from skill utility by normalizing rewards within problem–skill groups, estimating same-problem skill/no-skill reward differences, and using these differences to modulate policy advantages. Aggregated utility estimates additionally guide skill-library evolution. Experiments on Qwen3-4B and Qwen3-8B report average Pass@8 improvements of 2.94 and 2.70 percentage points over GRPO across five mathematical reasoning benchmarks.

Critical strengths
  1. Distinguishing the contribution of external guidance from reasoning quality under a fixed condition is a useful research direction.
Critical weaknesses
  1. The argument for a fundamental GRPO credit-assignment failure is not supported by strong evidences.
  2. Missing experimental details and insufficiently specified controls make it difficult to assess reproducibility and attribute the gains to the proposed mechanism.
Questions for authors

Based on the review above and some other issues, the reviewer has the following questions or comments.

  1. Does GRPO use the same skill library, reward shaping, and total rollout/token budget, including library maintenance? A matched skill-augmented GRPO control is needed to isolate SCPO’s contribution.
  2. Are no-skill controls included in Eq. (15), or used only to estimate paired gain? If included, how are their advantages modulated?
  3. How is the skill library operationally maintained? Please specify its initialization, the models/prompts used for extraction and refinement, the transfer-evaluation procedure, and the acceptance/state-transition thresholds.
  4. Are skills available at test time, and do improvements persist without them, indicating internalization rather than dependence on external guidance?
  5. Do the reported ten seeds represent independent training runs or repeated decoding from a fixed checkpoint? Please provide the evaluation sampling settings.
  6. SkillRL already samples groups under fixed retrieved context; Skill-R1 separates rollout and skill-improvement credit; SLIM uses counterfactual utility for library management. SCPO’s specific integration may be distinct, but these individual ideas are not new. The authors may need to elaborate SCPO’s differences and advantages over these previous research works.
Reviewer_jQVDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address the problem of skill credit blind spots in skill-based reinforcement learning of large language models. The authors believe that the existing group-based relative strategy optimization methods cannot accurately distinguish the quality of trajectories from the contribution of injected skills. Therefore, they introduce the decomposition of behavioral credit and skill credit, and update the strategy through causal advantage modulation.

Critical strengths
  1. Paying attention to the difficulties in attributing skills during the inference process of large language models is a research direction worthy of exploration.

  2. The article structure is relatively complete, covering from problem definition to method design and then to experimental verification, with a clear logical chain on the surface.

  3. The concept of skill library evolution was proposed, attempting to combine strategy optimization with the update of external skill libraries, demonstrating a certain level of systematic thinking.

Critical weaknesses
  1. The author claims that the existing group-based strategy optimization has a skill credit blind spot, but fails to provide sufficient ablation experiments or visual evidence to prove that this blind spot actually causes a serious performance bottleneck in actual training.

  2. The introduction of mechanisms such as causal advantage modulation and paired skill gain significantly increases computational cost and implementation difficulty, but the article fails to prove that this complexity brings a matching performance improvement.

  3. The experiments are only conducted on specific mathematical reasoning benchmarks and specific series of models, lacking generalization verification on more extensive tasks and different-sized models. Moreover, the selection of comparison baselines is not comprehensive, and fails to conduct a fair comparison with the current state-of-the-art skill learning methods.

  4. The article assumes that simple paired gain can accurately assess the long-term utility of skills, but in the actual sparse reward environment, this estimation is highly susceptible to noise interference, leading to unstable updates in the skill library.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.