- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ARISE, which aims to enable large language models to accumulate and reuse problem-solving skills during reinforcement learning for mathematical reasoning. It uses the same model for skill selection and solution generation, distills new skills from reference solutions, and continuously updates the skill library through verification feedback. Since skill scoring and solution generation share parameters, solution-generation training can indirectly influence skill selection, allowing the model’s capabilities and the skill library to co-evolve. Training consists of a warm-up phase and a skill-augmented phase. Experiments show that ARISE outperforms GRPO and other baselines, improving average accuracy on Omni-MATH from 23.9% to 27.0% with Qwen3-4B, while also yielding gains on larger models and code-generation tasks.
Critical strengths
Explicit accumulation of reusable problem-solving strategies. Reasoning patterns from individual solutions are distilled into structured skills, facilitating reuse across problems and making the accumulated strategies easier to inspect and understand.
A shared model for skill selection and solution generation. No separate scoring network needs to be trained. Solution-generation training can indirectly influence skill scoring through shared parameters, linking skill selection with skill application.
Continuous skill-library updates. Skills are retained, evicted, or reloaded based on verification results for new skills and their subsequent performance, allowing the library to adapt as the model’s capabilities evolve.
Evidence of consistent gains and broader applicability. The method can be combined with GRPO and DAPO. On Qwen3-4B, it improves average accuracy on Omni-MATH from 23.9% to 27.0% with GRPO and from 25.3% to 28.3% with DAPO, with gains also observed on other models and code-generation tasks.
Critical weaknesses
The contributions of reward design and the shared-parameter mechanism are not sufficiently disentangled. Reward scaling in the three-level scheme is largely canceled by within-group normalization, while the binary-reward ablation also changes skill-utility updates. Moreover, the independent scoring head receives zero selection gradient and is therefore not effectively trained. Consequently, the existing experiments do not adequately isolate the contributions of reward design, library maintenance, and parameter sharing.
The reliability of marginal skill-value estimation is limited. In Phase II, the original rollouts may already use an existing skill, so the comparison is not consistently against unaided reasoning. Each new skill is also verified only once on its source problem, making the assessment susceptible to sampling variability and problem-specific hints. This limits the reliability of estimates of its marginal benefit and transferability across problems.
Questions for authors
What drives the gains from the three-level reward scheme? When all rollouts in a group share the same skill, increasing the reward for a correct solution from 1 to 2 leaves the group-normalized advantages essentially unchanged. Are the gains primarily driven by updates to skill utilities? Could the rewards used for policy training and library maintenance be controlled separately to isolate their respective contributions?
Does the shared-parameter advantage persist against a stronger baseline? The selection gradient of the current independent scoring head is identically zero. Would the shared-parameter design still outperform an independent selector trained with an effective gradient signal?
Does the skill quality level reliably reflect marginal value? In Phase II, the original rollouts may already use an existing skill. Does therefore measure the new skill's value relative to the existing skill or relative to unaided reasoning? With only one verification rollout on the source problem, how can sampling variability be controlled and source-specific solution hints be distinguished from transferable skills?
2 comments
W2 / Q3. The single verification rollout may not reliably measure the marginal value of a skill.
We clarify that the single verification rollout only initializes the utility of an entry, and its weight falls to 0.35 after ten selections (Proposition B.1), so later group means decide the rank of the entry. In Phase II the original rollouts follow the current policy over options, so measures the value of a new skill over the policy's own choice, the selected entry or the unaided fallback, which is the comparison that cache maintenance needs. Sec. 3.3 of the revised paper now states this reference (Lines 302-304). App. H.1 separates transferable skills from hints specific to the source query, since eight of the ten cached entries keep positive attribution on unseen queries. On a sample of Phase II entries, we rerun eight rollouts on the source query with the entry, with the policy's own choice, and unaided, and report the pass rates (%) by logged level (App. E.2, Lines 1077-1090):
| Logged level | Entries | Policy's choice | Unaided | With the entry |
|---|---|---|---|---|
| 3 | 62 | 60 | 31 | |
| 26 | 7 | 6 | 6 | |
| 33 | 72 | 70 | 74 | |
| 38 | 13 | 12 | 46 |
The gain of the entry over either reference follows the logged level, with 34 points on , 4 on , zero on , and a loss of 29 on , and the two references differ by at most 2 points. The single verification rollout thus orders entries by their marginal value, and the choice of reference changes that value little.
We thank the reviewer for recognizing the explicit accumulation of reusable strategies, the shared model for selection and generation, the continuous library updates, and the consistent gains across models and code generation. We respond to the weaknesses and the questions below.
We agree that the gains of the three-level reward come from the utility updates, and each reward acts at its own place in the training of one policy that serves as both the selector over skills and the solver. Sec. 3.3 now states the roles of the rewards (Lines 289-291).
The separate control that the reviewer proposes is therefore already in place: a binary reward in the advantage with the three-level reward in Update coincides with the full system, and ablation (a) differs from it only in Update. Its drop of 1.9 points on AIME 2024, with utilization falling from 73% to 28%, measures the contribution of the reward to library maintenance, separately from parameter sharing.
Yes, the shared-parameter design outperforms an independent selector trained with an effective gradient, and we thank the reviewer for the stronger baseline, which makes the contribution of parameter sharing clearer. We agree that the decoupled head of ablation (f) receives zero gradient under the group-relative objective (Corollary B.1), so it tests only that prediction. Following the suggestion, we train the head by REINFORCE on the group mean reward minus its mean over the batch, with the rest of the protocol unchanged (App. G.2, Lines 1226-1245), and the table reports Pass@1 (%) as the mean over three seeds:
With a nonzero gradient the head gains 0.4 to 0.7 points over ablation (f), within the seed deviation, and shared-parameter ARISE stays above it by 0.6 to 1.1 points on each benchmark. Parameter sharing therefore contributes beyond a trained independent selector.