acceptodds
Under review as a conference paper at ICLR 2027

ARISE: Agent Reasoning with Intrinsic Skill Evolution in Hierarchical Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards has become the dominant paradigm for mathematical reasoning in language models, yet most methods treat each problem as a fresh episode and retain discovered strategies only implicitly through policy updates. Abstracting reusable techniques and recognizing when to apply them, the central mechanism of mathematical expertise, therefore stays outside training. Unlike the executable procedures stored in skill libraries for tool-use and embodied agents, a mathematical skill is a reasoning pattern embedded in a solution trace, whose applicability must be inferred from the problem and whose worth is defined only against the same policy solving it unaided. Abstracting and selecting such skills are therefore reasoning tasks that existing designs leave outside policy optimization. We introduce ARISE (Agent Reasoning with Intrinsic Skill Evolution), a hierarchical reinforcement learning framework in which one shared policy serves as the high-level option policy that selects from a persistent skill library and as the low-level intra-option policy that generates solutions conditioned on the selected skill. The policy scores library entries through its own conditional log-likelihood and concludes new skills from ground-truth traces through an inference step. With one option per query, the option-critic gradient reduces to two terms under one group-relative advantage, and generation feedback reaches selection through the tangent kernel between skill documents and rollouts, so a single policy gradient optimizes both levels. The three-level trajectory reward, which separates skill-aided from unaided correct solutions, enters the advantage, while a counterfactual skill quality level governs library maintenance, so reasoning ability and library quality co-evolve. Across two instruction-tuned base models and seven competition and Olympiad-level benchmarks, ARISE outperforms GRPO-family algorithms and memory- and skill-augmented baselines, with the largest gains on out-of-distribution tasks, and the gains persist on a third backbone at 8B scale and in code generation. The implementation is available at [ARISE Codebase](https://anonymous.4open.science/r/ARISE_ICLR-69D2).

Primary area:
reinforcement learning
Keywords:
Hierarchical Reinforcement Learning, Agent Skill, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XQuUOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ARISE, which aims to enable large language models to accumulate and reuse problem-solving skills during reinforcement learning for mathematical reasoning. It uses the same model for skill selection and solution generation, distills new skills from reference solutions, and continuously updates the skill library through verification feedback. Since skill scoring and solution generation share parameters, solution-generation training can indirectly influence skill selection, allowing the model’s capabilities and the skill library to co-evolve. Training consists of a warm-up phase and a skill-augmented phase. Experiments show that ARISE outperforms GRPO and other baselines, improving average accuracy on Omni-MATH from 23.9% to 27.0% with Qwen3-4B, while also yielding gains on larger models and code-generation tasks.

Critical strengths

Explicit accumulation of reusable problem-solving strategies. Reasoning patterns from individual solutions are distilled into structured skills, facilitating reuse across problems and making the accumulated strategies easier to inspect and understand.

A shared model for skill selection and solution generation. No separate scoring network needs to be trained. Solution-generation training can indirectly influence skill scoring through shared parameters, linking skill selection with skill application.

Continuous skill-library updates. Skills are retained, evicted, or reloaded based on verification results for new skills and their subsequent performance, allowing the library to adapt as the model’s capabilities evolve.

Evidence of consistent gains and broader applicability. The method can be combined with GRPO and DAPO. On Qwen3-4B, it improves average accuracy on Omni-MATH from 23.9% to 27.0% with GRPO and from 25.3% to 28.3% with DAPO, with gains also observed on other models and code-generation tasks.

Critical weaknesses

The contributions of reward design and the shared-parameter mechanism are not sufficiently disentangled. Reward scaling in the three-level scheme is largely canceled by within-group normalization, while the binary-reward ablation also changes skill-utility updates. Moreover, the independent scoring head receives zero selection gradient and is therefore not effectively trained. Consequently, the existing experiments do not adequately isolate the contributions of reward design, library maintenance, and parameter sharing.

The reliability of marginal skill-value estimation is limited. In Phase II, the original rollouts may already use an existing skill, so the comparison is not consistently against unaided reasoning. Each new skill is also verified only once on its source problem, making the assessment susceptible to sampling variability and problem-specific hints. This limits the reliability of estimates of its marginal benefit and transferability across problems.

Questions for authors

What drives the gains from the three-level reward scheme? When all rollouts in a group share the same skill, increasing the reward for a correct solution from 1 to 2 leaves the group-normalized advantages essentially unchanged. Are the gains primarily driven by updates to skill utilities? Could the rewards used for policy training and library maintenance be controlled separately to isolate their respective contributions?

Does the shared-parameter advantage persist against a stronger baseline? The selection gradient of the current independent scoring head is identically zero. Would the shared-parameter design still outperform an independent selector trained with an effective gradient signal?

Does the skill quality level reliably reflect marginal value? In Phase II, the original rollouts may already use an existing skill. Does RsR^s therefore measure the new skill's value relative to the existing skill or relative to unaided reasoning? With only one verification rollout on the source problem, how can sampling variability be controlled and source-specific solution hints be distinguished from transferable skills?

2 comments

We thank the reviewer for recognizing the explicit accumulation of reusable strategies, the shared model for selection and generation, the continuous library updates, and the consistent gains across models and code generation. We respond to the weaknesses and the questions below.

W1 / Q1. The three-level reward cancels in the advantage, so the source of its gains is unclear.

We agree that the gains of the three-level reward come from the utility updates, and each reward acts at its own place in the training of one policy that serves as both the selector over skills and the solver. Sec. 3.3 now states the roles of the rewards (Lines 289-291).

  • In the advantage, all GG rollouts of a group share the selected skill, so the trajectory reward enters Eq. (7) only through correctness, ζ0\zeta_0 against ζ1\zeta_1 without a skill or ζ0\zeta_0 against ζ2\zeta_2 with one, and the normalization cancels the value of ζ2\zeta_2.
  • In the library, Update writes the group mean Rˉtw\bar R^w_t into the utility of the selected entry, and Evict and Load keep the entries of high utility in the cache, so ζ2\zeta_2 ranks the entries by their success on later queries.
  • In the policy update, the cache decides which skills later rollouts condition on, so the utility, and with it ζ2\zeta_2, changes the data on which the policy trains its generation and its selection scores, without a direct gradient term.
  • The quality level RsR^s of a new skill only initializes its utility, and later group means overwrite it, with a remaining weight of 0.35 after ten selections (Proposition B.1).

The separate control that the reviewer proposes is therefore already in place: a binary reward in the advantage with the three-level reward in Update coincides with the full system, and ablation (a) differs from it only in Update. Its drop of 1.9 points on AIME 2024, with utilization falling from 73% to 28%, measures the contribution of the reward to library maintenance, separately from parameter sharing.

W1 / Q2. The reviewer asks whether parameter sharing beats an independent selector with an effective gradient.

Yes, the shared-parameter design outperforms an independent selector trained with an effective gradient, and we thank the reviewer for the stronger baseline, which makes the contribution of parameter sharing clearer. We agree that the decoupled head of ablation (f) receives zero gradient under the group-relative objective (Corollary B.1), so it tests only that prediction. Following the suggestion, we train the head by REINFORCE on the group mean reward minus its mean over the batch, with the rest of the protocol unchanged (App. G.2, Lines 1226-1245), and the table reports Pass@1 (%) as the mean over three seeds:

SelectorAMC 2023AIME 2024AIME 2025Omni-MATH
Decoupled head, group-relative, ablation (f)74.255.347.125.7
Decoupled head, batch baseline74.955.847.726.1
ARISE, shared parameters75.656.948.327.0

With a nonzero gradient the head gains 0.4 to 0.7 points over ablation (f), within the seed deviation, and shared-parameter ARISE stays above it by 0.6 to 1.1 points on each benchmark. Parameter sharing therefore contributes beyond a trained independent selector.

W2 / Q3. The single verification rollout may not reliably measure the marginal value of a skill.

We clarify that the single verification rollout only initializes the utility of an entry, and its weight falls to 0.35 after ten selections (Proposition B.1), so later group means decide the rank of the entry. In Phase II the original rollouts follow the current policy over options, so RsR^s measures the value of a new skill over the policy's own choice, the selected entry or the unaided fallback, which is the comparison that cache maintenance needs. Sec. 3.3 of the revised paper now states this reference (Lines 302-304). App. H.1 separates transferable skills from hints specific to the source query, since eight of the ten cached entries keep positive attribution on unseen queries. On a sample of Phase II entries, we rerun eight rollouts on the source query with the entry, with the policy's own choice, and unaided, and report the pass rates (%) by logged level (App. E.2, Lines 1077-1090):

Logged levelEntriesPolicy's choiceUnaidedWith the entry
r1r_13626031
r2r_226766
r3r_333727074
r4r_438131246

The gain of the entry over either reference follows the logged level, with 34 points on r4r_4, 4 on r3r_3, zero on r2r_2, and a loss of 29 on r1r_1, and the two references differ by at most 2 points. The single verification rollout thus orders entries by their marginal value, and the choice of reference changes that value little.

Reviewer_qerAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This Paper integrates a skill library into reinforcement learning.

Critical strengths

This is an interesting approach that lets the model learn from trajectories, summarize the learned skills, and then continue learning with those skills.

Critical weaknesses

See Questions For Authors

Questions for authors

Could the authors evaluate skills collected at early, middle, and late training stages using the same frozen solver and the same held-out problems? Comparing accuracy gains over an unaided baseline would directly test whether skill quality improves or plateaus during training.

1 comment

We thank the reviewer for finding the approach of learning, summarizing, and reusing skills during RL interesting, and for the concrete proposal. We respond to the question of the reviewer below.

Q. The reviewer proposes to evaluate caches from several training stages with a frozen solver.

Table 5 measures quality under the moving policy, which mixes skill quality with solver progress. Following the proposal, we evaluate the caches of steps 500, 1000, 1500, and 2000 on held-out problems with two frozen solvers, the base model and the final checkpoint, against the same solver without skills (App. H.3, Lines 1349-1364). The base model scores the same entries without having learned to select, so the two solvers also separate the cache from the trained selection, and the table reports Pass@1 (%) on AIME 2025 followed by Omni-MATH:

CacheUnaidedStep 500Step 1000Step 1500Step 2000
Base model45.6 / 21.245.8 / 21.445.7 / 21.446.0 / 21.645.9 / 21.6
Final checkpoint47.2 / 25.947.5 / 26.047.7 / 26.548.2 / 26.848.3 / 27.0

Under the final checkpoint, the gain over unaided rises to 1.1 points on both benchmarks and levels off after step 1500. Under the base model the gain stays within 0.4 points for each cache. The caches thus help mainly a solver that has learned to select from them, consistent with the aim of ARISE. The policy selects entries by its own likelihood, and Update and Evict keep the entries that solve later queries, so the efficiency of skill management improves together with the policy.

Reviewer_eP3ROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework that augments standard RLVR post-training of LLMs with an evolving library of skill documents. The framework uses model likelihood scores to select a skill for a given problem and conditions solution generation on the selected skill. It also distills new skills from reference solutions, evaluates them through additional rollouts, and updates the library using estimated skill utility.

Critical strengths
  • The paper advocates treating the skill library as a second axis of evolution alongside model parameters and provides empirical evidence of the effectiveness of this approach.
Critical weaknesses
  • Skill selection relies on a manually specified likelihood-scoring criterion, while the quality scores assigned to newly constructed skills rely heavily on hand-designed heuristics. This approach may therefore be at odds with the design principles of modern agentic LLMs.

  • The paper does not adequately justify assigning a reward of 2 to correct answers generated with a skill. The additional reward reflects the presence of the skill, without measuring its actual contribution to the correct answer.

Questions for authors
  • How is the skill-selection mechanism optimized when all rollouts within a group share the same skill?

  • What evidence shows that the proposed framework produces skills of higher quality or greater transferability?

2 comments

We thank the reviewer for recognizing the skill library as a second axis of evolution alongside the model parameters and the empirical evidence for it. We respond to the weaknesses and the questions below.

W1. Selection uses a hand-specified likelihood score, and the quality levels rely on heuristics.

The selection score of Eq. (4) is the policy's own length-normalized log-likelihood of a skill document given the query, without a separate scorer or hand-set feature. The score changes with each update, since by Eq. (6) documents aligned with correct rollouts gain score, so selection is a trained component of the policy. Random selection, ablation (b), costs 1.5, 1.1, and 1.2 points on AIME 2024, AIME 2025, and Omni-MATH.

Skill quality is likewise measured by the outcomes of the agent itself. The policy concludes each skill by its own inference, and the reward is the binary verifier of RLVR. The levels of Table 2 are the four cells of two binary outcomes on the same query, with and without the new skill, and their values only initialize the utility at the manager level. The levels are thus defined by measured outcomes, and only their numerical values, which order the four cells, are set as hyperparameters. The levels are computed at each step from warm-up on, so the warm-up entries carry measured utilities when Phase II starts to select. Update then refreshes the utility with the outcomes of the later groups that select the entry, so the weight of the initial value falls to 0.35 after ten selections (Proposition B.1). Removing the verification rollout, ablation (c), costs 1.7, 1.4, and 1.6 points, and ranking by insertion order in place of utility, ablation (d), costs 0.6, 0.5, and 0.6 points.

W2. The reward of 2 rewards the presence of a skill without measuring its contribution.

ARISE trains one policy that both selects a skill and solves the query, and the trajectory reward of Eq. (9) assigns ζ0\zeta_0 to incorrect solutions, ζ1\zeta_1 to correct unaided ones, and ζ2\zeta_2 to correct ones under a selected skill. Since all rollouts of a group share the selected skill, the group-normalized advantage of Eq. (7) separates correct from incorrect rollouts and cancels the value of ζ2\zeta_2, so the presence of a skill alone earns no extra gradient. The level ζ2\zeta_2 acts on the library instead, since Update writes the group mean into the utility of the selected entry, so an entry that keeps solving later queries outranks one supported only by its source query. Evict and Load keep the entries of high utility in the cache, and the policy conditions its later rollouts on these entries, so the effect of ζ2\zeta_2 reaches the policy through the skills it trains on. Ablation (a), with a binary reward, changes only this path and costs 1.9 points on AIME 2024, with utilization falling from 73% to 28%, and Sec. 3.3 now states both roles (Lines 289-291).

Q1. The reviewer asks how selection is optimized when a group shares one skill.

Selection is optimized through generation, since the selection score is the policy's own likelihood of a skill document given the query, and the update of generation on a correct skill-conditioned rollout also raises the score of that document by Eq. (6). The explicit selection term has a zero estimate in each group (Proposition 3.2), so the scores are trained through the parameters shared with generation by Eq. (6), while Update, Evict, and Load maintain the candidate set by measured outcomes. For the same reason, a selector with its own parameters receives zero gradient (Corollary B.1), and ablation (f) lands within 0.1 point of random selection. App. G.2 adds three results that compare this training with the baselines that the reviewers suggest (Lines 1226-1279).

  • The decoupled selector trained against a batch baseline receives a nonzero gradient and reaches 74.9, 55.8, 47.7, and 26.1 on AMC 2023, AIME 2024, AIME 2025, and Omni-MATH. Shared-parameter ARISE stays above it by 0.6 to 1.1 points on each benchmark.
  • With the cache fixed at step 2000, the gap between score and random selection grows from 0.3 points at step 500 to 1.0 on AIME 2025 and 1.1 on Omni-MATH at step 2000, while random selection stays within 0.2 points of unaided. Since the cache is fixed, the growth comes from the trained scores.
  • Over Phase II, the Spearman correlation between the scores and the utilities of the cached entries rises from 0.04 to 0.30, and the share of queries that select the highest-utility entry rises from 11% to 30%, against 10% for uniform selection. The scores trained through generation thus come to agree with the outcome-based utility.

Q2. The reviewer asks for evidence that the skills gain quality or transferability.

ARISE jointly optimizes the selection and the application of skills in one agent through hierarchical RL, while the library evolves by measured outcomes, so problem solving and skill use improve together, and the paper measures the quality and the transfer of skills through the agent that uses them. Against GRPO on the same model, Omni-MATH rises from 23.9 to 27.0 with Qwen3-4B, with the largest gains on problems disjoint from training (Table 3). The share of the levels r3r_3 and r4r_4 rises from 40.5% to 75.6% (Table 5), and eight of the ten cached entries keep positive attribution on unseen queries against five of the top ten reservoir entries (App. H.1). The gains also persist on Qwen3-8B-Base and in code generation (App. G.3, App. I).

We hope these replies address the concerns of the reviewer and would be grateful for a reconsideration of the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive comments. Reviewers XQuU and qerA appreciate that ARISE explicitly accumulates and reuses skills during RL, and Reviewer eP3R acknowledges the empirical effectiveness of evolving the skill library together with the model parameters. Reviewer XQuU further notes the shared model for selection and generation and the consistent gains across models and code generation. Revised and added parts of the paper are marked in blue.

In response to Reviewer XQuU, we separated the effects of the reward design, library maintenance, and parameter sharing, added an independent selector trained with a nonzero gradient as a baseline, and rechecked the marginal value of new skills with repeated rollouts (App. E.2 and G.2). In response to Reviewer qerA, we evaluated caches from four training stages with frozen solvers on held-out problems (App. H.3). Both reviewers stated that these responses addressed their concerns.

Reviewer eP3R has not yet responded to our rebuttal or follow-up, so we explain each concern in more detail below. The review rates these concerns as minor or fixable, and several of them overlap with the questions of Reviewer XQuU.

Skill selection (W1, Q1). The review describes the likelihood score as a manually specified criterion and asks how selection is optimized when all rollouts in a group share one skill. The score is the likelihood that the policy itself assigns to a skill document given the query, so ARISE uses neither a separate scorer nor hand-crafted features. When the policy is updated on a correct rollout conditioned on a skill, the same gradient raises the score of that skill document (Eq. (6)), which means that selection is trained together with generation. In App. G.2, ARISE outperforms a decoupled selector trained with a nonzero gradient by 0.6 to 1.1 points, and on a fixed cache the advantage of score-based over random selection grows during training (Lines 1226-1279).

Quality levels (W1). The review also considers the quality levels to be hand-designed heuristics that may not fit the design of agentic LLMs. Each level is determined by two verifier outcomes of the agent on the same query, one with the new skill and one without it, so it records the marginal contribution of the skill, and only the numerical values assigned to the four levels are hyperparameters. The level serves only as the initial utility of an entry, and its weight drops to 0.35 after ten selections (Proposition B.1). Rerunning eight rollouts per entry in App. E.2 shows that the measured gain agrees with the logged level (Lines 1077-1090). In ARISE, the agent selects skills by its own likelihood and the library is maintained by verified outcomes, which we believe is in line with how agentic LLMs are designed.

Role of the skill reward (W2). The review argues that the reward of 2 rewards the presence of a skill rather than its contribution. Because all rollouts in a group share the selected skill, the group-normalized advantage removes the difference between 2 and 1, and using a skill brings no extra gradient. The reward matters for the library, where the group mean is written into the utility of the selected entry (Lines 289-291). Entries stay highly ranked only if they keep solving later queries, so the utility tracks how much each entry actually helps. With a binary reward in ablation (a), which changes only this update, accuracy on AIME 2024 drops by 1.9 points and skill utilization falls from 73% to 28%.

Skill quality and transfer (Q2). On the evidence for skill quality and transfer, the frozen-solver evaluation in App. H.3 shows that caches from later training steps give larger gains on held-out problems, with the gains saturating after step 1500 (Lines 1349-1364). Among the ten cached entries, eight still contribute positively on unseen queries (App. H.1), and the gains carry over to Qwen3-8B-Base and to code generation (App. G.3, App. I).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.