acceptodds
Under review as a conference paper at ICLR 2027

ARISE: Adapting to Evolving Capability Gaps in Agentic Reinforcement Learning

11 views

Abstract

As a long-horizon agent improves through experience, previously observed weaknesses may recede while new limitations emerge, continually changing what it still needs to learn. Yet the learning process often remains tied to a static view of these needs: fixed behavioral criteria and training priorities can become misaligned with evolving agent capabilities, while sparse task-level feedback makes such misalignment more difficult to detect. Even when capability gaps are identified, rollouts from the current policy may repeatedly reproduce the same failures rather than explore better alternatives. To address this, we introduce Adaptive Rubric–Skill Co-Evolution (ARISE), a reinforcement learning framework that uses rollout evidence to continually adapt evaluation criteria, exploration guidance, and training priorities. Rubrics evolve to reward partial behavioral progress, while their paired skills are refined and selectively activated to guide exploration toward unresolved weaknesses. Alongside this co-evolution, capability-based adaptive sampling prioritizes tasks that target behaviors needing further improvement. Experiments on two challenging long-horizon agent benchmarks, SkillsBench and Terminal-Bench, demonstrate that ARISE successfully enhances both overall task performance and training efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM-Based Agent, Agentic Reinforcement Learning, Large Language Models, Skill Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bRt5OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ARISE. ARISE uses evidence from rollouts to dynamically create and maintain a set of rubrics that provide additional training feedback, rewarding partial progress. Rubrics also have paired skills that help the agent acquire the capabilities needed for success on them. Finally, ARISE also introduces an adaptive task sampling mechanism that favors tasks with applicable rubrics where the agent’ performance is not too low or high. Their experiments on SkillsBench and TerminalBench show benefits from the method.

Critical strengths
  • The presentation is good.

  • The idea of using an LLM to analyze past rollouts in order to generate rubrics, using those rubrics for additional feedback, dynamically adding and removing active rubrics as the agent capabilities evolve, and pairing rubrics with skills to help the acquisition of those capabilities all make sense to me. This part of the method seems reasonable and interesting.

  • The chosen methods the experiments compare to and the ablations seem appropriate. The additional results are also useful.

Critical weaknesses
  • The method relies on a stronger external model as the rubric judge and reflection model, which could limit the applicability of the method to frontier training. Do you think the method would work if the trained model was used for all roles?

  • Results seem to use a single training run for each method and ablation. Moreover, evaluations use only one benchmark for ablations and two for the main comparisons. While these decisions can be reasonable due to compute costs, some of the observed differences (mainly within the ablations) are relatively small, making the evidence for the contribution of each component less convincing.

Questions for authors

Of the components of the method, I found the task sampling component the least convincing.

  • The method relies on predefined task types and capability groups. How much work did it take to develop these? Do you consider this a limitation of the method?
  • Task selection seems to depend purely on performance on capabilities (rubrics) and not on performance on the tasks themselves. Is this desired? Why?
  • A task seems to be more likely to be sampled when it belongs to several types and has applicable rubrics that cover several capabilities. Is this the case? Is this desired? Could this lead to a feedback loop where certain tasks are sampled more frequently, making the reflection model see more trajectories from them and potentially generate more rubrics targeting their behaviors, further increasing their sampling probability?

How was the hyperparameter selection procedure carried out for the method and the baselines? I did not find this information in the paper.

Secondary feedback and questions:

  • When a rubric's performance exceeds the low threshold, the corresponding skill is removed from the model’s context. Is that performance estimated from training rollouts that include skill guidance when the skill is active? If so, could removing the skill cause the performance to drop again into the region where the skill is added? Is there anything preventing this potential oscillation?

  • If you have the results, I think it would be nice to also have available the performance as training progresses for the ablations other than outcome only RL (Figure 3).

  • While the writing is generally good, I think section 3.2.2 could be clearer.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bRt5 (Part 1 / 3)

Thank you for recognizing the rationale behind dynamically evolving rubrics and pairing them with skills, as well as the clear presentation and appropriate baselines and ablations. We appreciate your detailed feedback and address each point below.

W1: The method relies on a stronger external model as the rubric judge and reflection model, which could limit the applicability of the method to frontier training. Do you think the method would work if the trained model was used for all roles?

Thank you for this important question. Our additional experiment shows that ARISE remains effective without a stronger external judge or reflection model. We used Qwen3.5-27B for all three roles, keeping the judge and reflection model fixed at the initial checkpoint while updating only the policy. As shown in the table below, this variant approaches the performance of the original ARISE and outperforms both outcome-only RL and OnlineRubrics on both benchmarks, even though OnlineRubrics elicits criteria by comparing policy responses with reference responses from DeepSeek-V4-Flash. We have added these results to Appendix J.

Method / ModelBase modelPolicy total parametersJudge and reflection modelSkillsBench v1.1 (%)Terminal-Bench v2.1 (%)
Outcome-only RLQwen3.5-27B27BN/A34.543.4
OnlineRubrics (strongest baseline)Qwen3.5-27B27BDeepSeek-V4-Flash40.246.1
ARISEQwen3.5-27B27BQwen3.5-27B43.749.1
ARISEQwen3.5-27B27BDeepSeek-V4-Flash45.650.6

W2: Results seem to use a single training run for each method and ablation. Moreover, evaluations use only one benchmark for ablations and two for the main comparisons. While these decisions can be reasonable due to compute costs, some of the observed differences (mainly within the ablations) are relatively small, making the evidence for the contribution of each component less convincing.

Thank you for this valuable suggestion. Following your feedback, we added experiments with three training seeds and extended the ablation evaluation to Terminal-Bench v2.1.

1. ARISE's advantage over the baselines and strongest ablation persists across training seeds. Following your suggestion, we conducted additional experiments with three training seeds for ARISE, outcome-only RL, OnlineRubrics, and the strongest component ablation, ARISE with outcome-based VADE sampling (w/ VADE Sampling). Each checkpoint is evaluated three times on SkillsBench v1.1, and we report the mean and sample standard deviation across the three seed-level averages. As shown in the table below, ARISE maintains a higher mean score than both baselines and the strongest ablation, supporting its robustness to training variability and the contribution of capability-based sampling. We have added these results to Appendix E.2.

MethodSkillsBench v1.1 pass rate (%)
Outcome-only RL34.87 ± 0.66
OnlineRubrics40.10 ± 1.73
ARISE w/ VADE Sampling42.91 ± 0.77
ARISE46.10 ± 1.23

2. The component ablations show consistent benefits on both benchmarks. To assess whether the component contributions extend beyond SkillsBench, we additionally evaluated all ablation checkpoints on Terminal-Bench v2.1. As shown in the table below, full ARISE outperforms every ablation on both benchmarks, supporting the contributions of dynamic rubric evolution, skill guidance, and capability-based sampling. Moreover, replacing our sampler with VADE sampling reduces performance on both benchmarks, providing further evidence for the value of sampling based on behavioral feedback. We have updated Figure 3(a) in the main paper to include the Terminal-Bench ablations.

SettingSkillsBench v1.1 (%)Terminal-Bench v2.1 (%)
Outcome-only RL34.543.4
w/ Frozen Rubrics39.944.2
w/o Skill Injection41.446.1
w/o Adaptive Sampling42.545.7
w/ VADE Sampling43.747.6
ARISE45.650.6
AuthorsOpenReview ↗

Response to Reviewer bRt5 (Part 2 / 3)

Q1: Of the components of the method, I found the task sampling component the least convincing.

  • The method relies on predefined task types and capability groups. How much work did it take to develop these? Do you consider this a limitation of the method?
  • Task selection seems to depend purely on performance on capabilities (rubrics) and not on performance on the tasks themselves. Is this desired? Why?
  • A task seems to be more likely to be sampled when it belongs to several types and has applicable rubrics that cover several capabilities. Is this the case? Is this desired? Could this lead to a feedback loop where certain tasks are sampled more frequently, making the reflection model see more trajectories from them and potentially generate more rubrics targeting their behaviors, further increasing their sampling probability?

Thank you for these thoughtful questions. We clarify the setup cost, sampling rationale, and potential feedback loop below.

1. Task categorization requires little setup effort, and capability groups are reusable. We consider the need to adapt the task taxonomy a practical limitation when transferring to new environments. However, our automated construction and annotation procedure takes approximately two hours on our corpus, reducing this setup cost. Moreover, the five general capability categories can be reused, and adapting the taxonomy does not require redesigning the sampler. We have added the construction and annotation procedure to Appendix C.2.

2. Rubric-based sampling tracks evolving capabilities beyond binary task outcomes. This choice is intentional: intermediate behaviors may improve while the final task outcome remains unchanged. Moreover, in a large training corpus, individual tasks may be revisited infrequently, so estimates based only on each task's outcomes can lag behind changes in the policy's capabilities. To address this, we aggregate rubric evidence by task type and capability, allowing observations from related tasks to update shared estimates and help the sampler respond to these changes. Consistent with this rationale, replacing our sampler with outcome-based VADE sampling (w/ VADE Sampling in Figure 3(a)) reduces performance on both benchmarks. Nevertheless, we agree that combining rubric feedback and task success rates for sampling is a reasonable extension, and we leave its investigation to future work.

3. Multiple associations increase sampling opportunities, but we constrain concentration and examine the proposed feedback loop. A task associated with more eligible task-type–capability pairs can indeed be selected through more paths. We consider this tendency reasonable because tasks spanning more workflows and capabilities offer diverse training opportunities and may therefore have greater learning value. To limit concentration, however, tasks cannot repeat within a batch, and each pair receives at most eight adaptive selections per batch, as specified in Appendix B.5. In addition, coverage-driven discovery prioritizes insufficiently evaluated tasks, while rubric retirement removes mastered criteria. Beyond these design constraints, we also examined whether frequently sampled tasks dominate rubric generation by measuring the association between each task's rollout count and the number of times its evidence is cited by newly generated rubrics. The resulting weak correlation (Spearman's ρ=0.093\rho=0.093) indicates little association between sampling frequency and rubric source citations.


Q2: How was the hyperparameter selection procedure carried out for the method and the baselines? I did not find this information in the paper.

Thank you for pointing out this omission. ARISE and all baselines use the same validation split and learning-rate search space. We hold out 10% of the 1,728-task corpus across multiple domains and select each method's learning rate from {1e-6, 2e-6, 5e-6} based on its validation performance. Since each training run requires approximately two days on 32 H800 GPUs, we limit the search to learning rate and adopt the original papers' settings for other baseline-specific hyperparameters. Other ARISE-specific hyperparameters were fixed heuristically based on the mechanism design. We apologize for this omission. We have added details of the validation split to Appendix C.1 and the hyperparameter selection procedure to Appendix D.3.

AuthorsOpenReview ↗

Response to Reviewer bRt5 (Part 3 / 3)

Q3: When a rubric's performance exceeds the low threshold, the corresponding skill is removed from the model’s context. Is that performance estimated from training rollouts that include skill guidance when the skill is active? If so, could removing the skill cause the performance to drop again into the region where the skill is added? Is there anything preventing this potential oscillation?

Thank you for raising this concern.

1. Active skills are evaluated using rollouts that include their guidance. Yes, when a skill is active, its rubric pass rate is estimated from training rollouts containing that skill. After the skill is hidden, the rubric continues to evaluate subsequent behavior, and the skill can be reactivated if its pass rate returns to the activation region.

2. We observed no skill reactivation during training. Moreover, the case studies in Appendix F show that rubric pass rates improve after skill guidance is removed. The artifact-specification skill is hidden at step 30, after which its rubric pass rate rises from 34.3% to 66.0% at step 150. Similarly, the tool-response skill is hidden at step 60, and its rubric pass rate rises from 43.2% to 97.2% at step 100.

3. Periodic updates reduce rapid switching. Skill activation and hiding are decided every 10 training steps, as specified in Appendix B.2, using accumulated rubric evaluations to reduce sensitivity to individual rollout fluctuations. Together with the absence of observed reactivation, this indicates stable skill visibility during training.


Q4: If you have the results, I think it would be nice to also have available the performance as training progresses for the ablations other than outcome only RL (Figure 3).

Thank you for this helpful suggestion. We additionally report the training progress of all four component ablations on SkillsBench v1.1. As shown in the table below, ARISE outperforms every ablation at steps 50, 100, and 150, showing a consistent advantage across the evaluated training stages. Pass rates are averaged over three evaluations at each checkpoint. We have added these checkpoint results to Appendix L of the revised paper.

SettingStep 0 (%)Step 50 (%)Step 100 (%)Step 150 (%)
Outcome-only RL23.426.432.234.5
w/ Frozen Rubrics23.428.734.539.9
w/o Skill Injection23.429.935.241.4
w/o Adaptive Sampling23.429.137.242.5
w/ VADE Sampling23.431.837.943.7
ARISE23.434.541.445.6

Q5: While the writing is generally good, I think section 3.2.2 could be clearer.

Thank you for this helpful suggestion. We have refined Section 3.2.2 to make the sampling procedure easier to follow. Specifically, we clarify how task-type tags are constructed and assigned, improve the transitions between the existing equations, and provide clearer pointers to the implementation details in Appendices B.5 and C.2.

Reviewer_PAo5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Previous work often adopts a static view of long-horizon agents, where fixed behavioral criteria and training priorities may become misaligned with actual needs, leading to repeated failures across rollouts. The authors propose ARISE, a reinforcement learning framework that leverages rollout feedback for adaptive training. Experiments on two long-horizon agent benchmarks demonstrate improvements in task performance and training efficiency. The key difference claimed is that the work leverages rollout evidence to dynamically uncover and address emerging bottlenecks compared to other long-horizon agent learning methods.

Critical strengths

The paper is clearly written and provides sufficient methodological details. The method proposed is clearly written, and different components show how the model is organized. The Appendix provides sufficient details of selected parameters and model implementations. The appendix also provides a helpful running example, which makes the setup clearer and easy to follow.

The experiments and analyses are comprehensive and evaluate the proposed method from multiple perspectives. The method is evaluated from different perspectives in two benchmarks. The evaluation includes behavior-level information. The ablation study further validates the effectiveness of skill injection and other components. The proposed method on a single model is compared with different models and setups.

Critical weaknesses

The experiments rely on a single model, Qwen3.5-27B, raising concerns about generalizability across different model families and scales. The model provides evaluation on a single model. Since the claim is general RL, the proposed method needs to be evaluated with different models to demonstrate generalization ability instead of relying on a certain model scale and model structure.

The LLM-based evaluation requires further details and evidence to establish its robustness and reliability, especially given the limited task coverage of the two benchmarks. Further analysis of hyperparameter sensitivity would help assess the stability and effectiveness of the proposed method, considering the number of hyperparameters in the method. So more information about how they are decided needs to be declared instead of only the setup of the used method.

Questions for authors
  1. How well does ARISE generalize to other model families and scales?
  2. How robust are the reported improvements to LLM evaluator choices and key hyperparameters, given the limited task coverage of the benchmarks?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PAo5

Thank you for recognizing the clarity of our presentation, the detailed methodology and running examples, and the breadth of our experiments and behavioral analyses. We appreciate your suggestions and provide additional evidence on generalization and robustness below.

W1: The experiments rely on a single model, Qwen3.5-27B, raising concerns about generalizability across different model families and scales. The model provides evaluation on a single model. Since the claim is general RL, the proposed method needs to be evaluated with different models to demonstrate generalization ability instead of relying on a certain model scale and model structure.

Q1: How well does ARISE generalize to other model families and scales?

Thank you for highlighting the importance of generalization across model families and scales. We additionally trained GLM-4.7-Flash (30B total, 3B active parameters) and Qwen3.5-122B-A10B (122B total, 10B active parameters). As shown in the table below, ARISE achieves substantial improvements over outcome-only RL on SkillsBench v1.1 with all three backbones, supporting its effectiveness across model families and scales. We have added these results and training details to Appendix I.

The table below reports SkillsBench v1.1 pass rates (%).

Training methodQwen3.5-27BGLM-4.7-FlashQwen3.5-122B-A10B
Base model23.410.324.1
Outcome-only RL34.513.831.0
ARISE45.618.440.2

W2: The LLM-based evaluation requires further details and evidence to establish its robustness and reliability, especially given the limited task coverage of the two benchmarks. Further analysis of hyperparameter sensitivity would help assess the stability and effectiveness of the proposed method, considering the number of hyperparameters in the method. So more information about how they are decided needs to be declared instead of only the setup of the used method.

Q2: How robust are the reported improvements to LLM evaluator choices and key hyperparameters, given the limited task coverage of the benchmarks?

Thank you for raising these questions about evaluator robustness and hyperparameter sensitivity. We apologize for the missing details. Following your suggestions, we evaluated alternative LLM evaluators and rubric advantage weights, and added both sensitivity analyses to Appendix E.3. Below, we present the results and clarify the hyperparameter selection procedure.

1. ARISE remains effective across the tested LLM evaluator choices. The LLM evaluator serves as the rubric judge during training, while final benchmark success is determined by task verifiers. We compare DeepSeek-V4-Flash, DeepSeek-V4.1-Flash, and GLM-5.1-FP8 as LLM evaluators, using Qwen3.5-27B as the policy. As shown in the table below, all three settings outperform the strongest baseline, OnlineRubrics, on SkillsBench v1.1, supporting robustness to the choice of LLM evaluator.

LLM evaluator (judge model)SkillsBench v1.1 pass rate (%)
DeepSeek-V4-Flash (paper setting)45.6
DeepSeek-V4.1-Flash47.5
GLM-5.1-FP844.8

2. ARISE's gains persist across the tested rubric advantage weights. Beyond the choice of LLM evaluator, we examine λ\lambda in Eq. (16), which controls the combination of separately normalized task and rubric advantages. Specifically, we test 0.1, 0.3, and 0.5, covering both sides of the default value. As shown in the table below, ARISE consistently outperforms OnlineRubrics on SkillsBench v1.1, indicating that its gains do not depend on a single narrowly chosen weight.

Rubric advantage weight λ\lambdaSkillsBench v1.1 pass rate (%)
0.143.7
0.3 (paper setting)45.6
0.545.2

3. Hyperparameter selection uses a separate validation split and a shared search protocol. For the original experiments, we reserved 10% of the 1,728-task corpus for validation, covering multiple domains. Given the training cost of approximately two days on 32 H800 GPUs per run, we search the same learning-rate grid of {1e-6, 2e-6, 5e-6} for ARISE and all baselines and select each method's best setting on this split. For the remaining baseline-specific hyperparameters, we follow their original papers. Other ARISE-specific hyperparameters are fixed heuristic settings based on the mechanism design and were not tuned through a hyperparameter search due to the high training cost. We have added details of the validation split to Appendix C.1 and the hyperparameter selection procedure to Appendix D.3.

Thanks for the additional clarification. I maintain my positive evaluation.

AuthorsOpenReview ↗

Thank you for your support and positive recommendation

Thank you for your continued positive evaluation and for taking the time to review our additional results and clarifications. We are pleased that these have fully addressed your concerns regarding generalization across policy models, evaluator choices, and hyperparameter sensitivity. Your positive feedback on the clear presentation, detailed methodology, and comprehensive experiments and behavioral analyses is also encouraging. We have incorporated the additional results and a clearer description of hyperparameter selection into the revised paper. We sincerely appreciate your thoughtful feedback throughout the review process.

Reviewer_1RGtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ARISE, an RL framework for long-horizon LLM agents. During training, a reflection model (DeepSeek-V4-Flash) reads recent rollouts and writes pairs of a rubric, which is a binary check on the agent's behavior, and a skill, which is natural-language advice for the same behavior. The same model acts as a judge and scores every applicable rubric on every trajectory. The rubric verdicts are normalized within each rollout group and mixed into the advantage with weight 0.3. The pass rate of each rubric decides what happens to its skill: the skill is shown in the system prompt when the pass rate is at most 0.3, hidden between 0.3 and 0.9, and retired together with its rubric at 0.9 or above. A separate sampler uses the rubric results to favor pairs of task type and capability with intermediate pass rates. Starting from Qwen3.5-27B and training for 150 steps on 1,728 tasks, ARISE reaches 45.6% on SkillsBench v1.1 and 50.6% on Terminal-Bench v2.1, against 40.2% and 46.1% for the best baseline. The authors also report that ARISE reaches the final score of outcome-only RL in a third of the training steps.

Critical strengths

1. The motivation is concrete.

Fixed rubrics stop giving signal once the policy satisfies them, and GRPO gets no signal from groups in which every rollout fails. Pairing each rubric with a skill, and letting the rubric's pass rate decide when the skill is shown, refined or retired, is a simple and sensible answer to both problems. The lifecycle examples in Appendix F make the mechanism easy to follow.

2. Experimental performance is good.

The baselines are recent and relevant (VADE, OnlineRubrics, RuscaRL). All of them start from the same checkpoint with the same data and number of steps. The final numbers come from the official verifiers, not from the LLM judge. The gains also hold on Terminal-Bench, which uses a different agent harness from the one used in training.

3. The appendix is detailed.

It includes all prompts, and the prompts show care about reward leakage. Per-rollout outcome labels are kept out of rubric generation, and rubrics that only restate that the tests pass are not allowed.

Critical weaknesses

1. My main concern is that the gains cannot be separated from the stronger model that runs the loop.

DeepSeek-V4-Flash generates every rubric and skill, rewrites skills that keep failing, and judges every training trajectory (line 922). On SkillsBench this model scores 44.7% (from the official report), about 21 points above the base policy, and ARISE ends at 45.6%. The student thus finishes at about the teacher's level, which is also what plain transfer from the teacher would produce. Table 1 lists 13 reference models, including two versions of DeepSeek-V4-Pro, but not DeepSeek-V4-Flash. Among the baselines, outcome-only RL and VADE never see this model, and OnlineRubrics and RuscaRL use it only for reference answers, rubrics or judging. None of them receives the skills, which carry the teacher's advice on how to act.

The ablations do not resolve this, because every single-component ablation keeps DeepSeek-V4-Flash, and outcome-only RL removes it together with everything else. Using a stronger model is a legitimate design choice. In that case, the baselines should get the same access to it. Or the paper should show that the loop still works when Qwen3.5-27B serves as its own judge and reflection model.

2. The paper does not explain how guidance that is present during training but absent at test time reaches the policy.

Active skills are placed in the system prompt during rollouts (Algorithm 1, line 6) and withheld at evaluation (line 327). The paper says they are meant to be "internalized through policy optimization" (line 1461). I can not find whether the policy loss is computed with or without the skill text in the context, and both answers raise problems.

Without the skill text, the update is off-policy, and the 0.5 < ω < 5 filter in Appendix B.4 drops every token whose probability the skill more than doubled, which are the tokens the skill affected most. With the skill text, the policy learns to follow a skill document in its context. And this is exactly what the curated-skills setting of SkillsBench rewards (Appendix D.1). No experiment separates this skill-following effect from the capabilities that the rubrics target. This train/test asymmetry is the setting of context distillation (Snell et al., 2022) and of RL with off-policy guidance from a stronger model (e.g., LUFFY, Yan et al., 2025). The paper should discuss this line of work and, ideally, compare against it.

3. The comparison is matched in training steps, but not in compute, tuning or seeds.

ARISE adds a judge call on each of the 256 trajectories per step, on traces of up to 131K tokens, plus a reflection call every 10 steps. Appendix H compares only per-step wall-clock time (17.14 vs. 16.46 minutes) and concludes a 65% saving in training time, without counting the tokens or hardware used by DeepSeek-V4-Flash. Outcome-only RL is still improving at step 150 (about 32% at step 100 and 34.5% at step 150, Figure 3b), so a compute-matched run could narrow the gap. I could not find how the hyperparameters were chosen. Each method is also trained only once. The margin over the best baseline is 14 successful attempts out of 261 on SkillsBench and 12 out of 267 on Terminal-Bench, about 4 to 5 tasks. And the ablation gaps are 1.9 to 5.7 points. Appendix E evaluates the same checkpoint three times, which measures evaluation noise, not training variance. A paired test over tasks costs nothing, and three seeds for ARISE, OnlineRubrics and outcome-only RL would make the main claim much more convincing.

4. The presentation makes the method hard to assess and to reproduce.

Figures 1(b) and 2 are icon-level schematics. They do not show which model does what (policy, judge, reflection model, task annotator), how often each update runs, or how the task and rubric advantages are combined. The sampler in Section 3.2.2 depends on details that appear only in Appendices B.5 and C.2. The prompts in Appendix B.6 also insert a "behavior issue taxonomy" and a "behavior pattern catalog" that shape every rubric and skill, but their contents are not given, and neither is the 82-type task registry. Without these lists, and without the code, the method might be hard to be re-implemented from the paper.

5. The claim of "a general paradigm" (line 94) rests on one kind of environment and one policy model.

Both benchmarks and 72% of the training tasks are containerized software, security and data tasks with programmatic verifiers. Several components are built for this setting: a predefined taxonomy of 82 task types assigned by GLM-5.2, five predefined capability categories, and prompts written around pytest-style verifiers. Only Qwen3.5-27B is trained. One structurally different environment, such as web navigation or an embodied text environment, would support the claim much better. At a minimum, the paper should say which components have to be rebuilt for a new environment.

Questions for authors
  1. What would happen when Qwen3.5-27B serves as its own judge and reflection model? Even a 50-step run, compared with the step-50 points in Figure 3(b), would show how much of the gain depends on DeepSeek-V4-Flash.

  2. Is it possible to add DeepSeek-V4-Flash to Table 1, evaluated under the same protocol on both benchmarks? It would also help to know on how many SkillsBench tasks ARISE succeeds where DeepSeek-V4-Flash fails.

  3. Is the skill text kept in the context when the policy loss is computed on guided rollouts? If not, what fraction of tokens does the ratio filter remove? How many rollouts were generated with at least one active skill, and how does ARISE perform on SkillsBench without curated skills?

  4. How many judge and reflection tokens, or GPU-hours, does one ARISE run use, and how do outcome-only RL and VADE perform with the same total compute? Do the baselines use the same length-regularized reward (Eq. 17)?

  5. Figure 3(b) suggests that SkillsBench was evaluated during training, and I could not find a validation split. Is this a common practice for this benchmark? If so, are the hyperparameters selected based on the testing set?

  6. In Eq. (16), a trajectory with no applicable rubric keeps its full task advantage, while the others get only (1−λ) of it. Applicability depends on the agent's own actions, for example whether a tool error occurs. Is this asymmetry intended, is this effect ever checked?

  7. What would have to change to apply ARISE to an environment without a task taxonomy or a programmatic verifier, such as WebArena or ALFWorld?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 1RGt (Part 1 / 5)

Thank you for recognizing ARISE's concrete motivation, the rubric–skill lifecycle design, and the performance gains across agent harnesses. We also appreciate your acknowledgment of the safeguards against reward leakage in our prompts. We address your concerns and present additional evidence below.

W1: My main concern is that the gains cannot be separated from the stronger model that runs the loop.

DeepSeek-V4-Flash generates every rubric and skill, rewrites skills that keep failing, and judges every training trajectory (line 922). On SkillsBench this model scores 44.7% (from the official report), about 21 points above the base policy, and ARISE ends at 45.6%. The student thus finishes at about the teacher's level, which is also what plain transfer from the teacher would produce. Table 1 lists 13 reference models, including two versions of DeepSeek-V4-Pro, but not DeepSeek-V4-Flash. Among the baselines, outcome-only RL and VADE never see this model, and OnlineRubrics and RuscaRL use it only for reference answers, rubrics or judging. None of them receives the skills, which carry the teacher's advice on how to act.

The ablations do not resolve this, because every single-component ablation keeps DeepSeek-V4-Flash, and outcome-only RL removes it together with everything else. Using a stronger model is a legitimate design choice. In that case, the baselines should get the same access to it. Or the paper should show that the loop still works when Qwen3.5-27B serves as its own judge and reflection model.

Q1: What would happen when Qwen3.5-27B serves as its own judge and reflection model? Even a 50-step run, compared with the step-50 points in Figure 3(b), would show how much of the gain depends on DeepSeek-V4-Flash.

Thank you for this important suggestion. We agree that ARISE's contribution should be distinguished from access to a stronger model. Following your suggestion, we test whether its gains persist when the initial policy checkpoint serves as both judge and reflection model.

ARISE's gains persist without a stronger judge and reflection model. We use the fixed initial Qwen3.5-27B checkpoint for both roles. As shown in the table below, ARISE still outperforms outcome-only RL and OnlineRubrics on both benchmarks, even though OnlineRubrics uses DeepSeek-V4-Flash reference responses. This shows that access to a stronger model is not required for the observed improvements. We have added the self-judge results to Appendix J in our revised paper.

Method / ModelBase modelPolicy total parametersJudge and reflection modelSkillsBench v1.1 (%)Terminal-Bench v2.1 (%)
Outcome-only RLQwen3.5-27B27BN/A34.543.4
OnlineRubrics (strongest baseline)Qwen3.5-27B27BDeepSeek-V4-Flash40.246.1
ARISEQwen3.5-27B27BQwen3.5-27B43.749.1
ARISEQwen3.5-27B27BDeepSeek-V4-Flash45.650.6

Q2: Is it possible to add DeepSeek-V4-Flash to Table 1, evaluated under the same protocol on both benchmarks? It would also help to know on how many SkillsBench tasks ARISE succeeds where DeepSeek-V4-Flash fails.

Thank you for this suggestion. We have added DeepSeek-V4-Flash to Table 1. Under the same SkillsBench evaluation protocol, ARISE scores 45.6%, compared with 44.1% for DeepSeek-V4-Flash, and succeeds on 12 tasks where DeepSeek-V4-Flash fails all three attempts.

AuthorsOpenReview ↗

Response to Reviewer 1RGt (Part 2 / 5)

W2.1: The paper does not explain how guidance that is present during training but absent at test time reaches the policy.

Active skills are placed in the system prompt during rollouts (Algorithm 1, line 6) and withheld at evaluation (line 327). The paper says they are meant to be "internalized through policy optimization" (line 1461). I can not find whether the policy loss is computed with or without the skill text in the context, and both answers raise problems.

Q3.1: Is the skill text kept in the context when the policy loss is computed on guided rollouts? If not, what fraction of tokens does the ratio filter remove? How many rollouts were generated with at least one active skill?

Thank you for this important question. We apologize for any confusion and clarify below how ARISE-generated skill guidance is incorporated into policy optimization and later removed from the policy's context.

1. ARISE-generated skill guidance is retained when computing the policy loss on guided rollouts. In Eq. (18) of our paper, both the current and behavior policies condition on the same context containing this guidance, avoiding the context mismatch raised in your question. We have clarified this in Appendix B.4.

2. Removing ARISE-generated skill guidance from the policy's context ensures training–inference consistency in the final training phase. Overall, 65.47% of rollouts include at least one active skill, but training proceeds without this guidance from step 120 through step 150. Moreover, Appendix G shows that the three tracked behaviors maintain pass rates above 90% at later checkpoints without guidance.


W2.2: Without the skill text, the update is off-policy, and the 0.5 < ω < 5 filter in Appendix B.4 drops every token whose probability the skill more than doubled, which are the tokens the skill affected most. With the skill text, the policy learns to follow a skill document in its context. And this is exactly what the curated-skills setting of SkillsBench rewards (Appendix D.1). No experiment separates this skill-following effect from the capabilities that the rubrics target.

Q3.2: How does ARISE perform on SkillsBench without curated skills?

Thank you for raising this distinction. Retaining ARISE-generated skill guidance during loss computation avoids the context mismatch raised in your concern. We further examine whether ARISE's gains extend beyond improved skill following through the evaluations and ablations below.

1. ARISE's gains persist without SkillsBench's curated skills. Following your suggestion, we evaluate all three models without these skills. As shown in the table below, ARISE still outperforms Qwen3.5-27B (base model) and outcome-only RL, showing that its advantage extends beyond using supplied skill documents. We have added these results to Appendix K of the revised paper.

MethodSkillsBench v1.1 without curated skills (%)
Qwen3.5-27B (base model)7.7
Outcome-only RL11.5
ARISE16.1

2. ARISE-generated skill guidance differs from SkillsBench's curated skills in form and delivery. Our guidance consists of brief behavioral instructions injected directly into the system prompt during training, whereas SkillsBench's curated skills comprise documents and resources that the agent accesses through tools during evaluation. Thus, our guidance does not directly train the skill-discovery and loading process used in SkillsBench.

3. The ablation confirms that ARISE-generated skill guidance is not the sole source of improvement. The w/o Skill Injection variant in Figure 3(a) still outperforms outcome-only RL despite receiving no ARISE-generated skill guidance during training, supporting the contribution of ARISE's other mechanisms.


W2.3: This train/test asymmetry is the setting of context distillation (Snell et al., 2022) and of RL with off-policy guidance from a stronger model (e.g., LUFFY, Yan et al., 2025). The paper should discuss this line of work and, ideally, compare against it.

Thank you for pointing out these related approaches. We have added a discussion of context distillation and LUFFY to Appendix B.4. Context distillation optimizes with reduced context using outputs generated with richer context, while LUFFY incorporates external off-policy reasoning trajectories into RL. ARISE instead optimizes its own policy-generated trajectories with rubric feedback, retains ARISE-generated skill guidance in the loss context, and adaptively withdraws this guidance during training.

AuthorsOpenReview ↗

Response to Reviewer 1RGt (Part 3 / 5)

W3: The comparison is matched in training steps, but not in compute, tuning or seeds.

ARISE adds a judge call on each of the 256 trajectories per step, on traces of up to 131K tokens, plus a reflection call every 10 steps. Appendix H compares only per-step wall-clock time (17.14 vs. 16.46 minutes) and concludes a 65% saving in training time, without counting the tokens or hardware used by DeepSeek-V4-Flash. Outcome-only RL is still improving at step 150 (about 32% at step 100 and 34.5% at step 150, Figure 3b), so a compute-matched run could narrow the gap. I could not find how the hyperparameters were chosen. Each method is also trained only once. The margin over the best baseline is 14 successful attempts out of 261 on SkillsBench and 12 out of 267 on Terminal-Bench, about 4 to 5 tasks. And the ablation gaps are 1.9 to 5.7 points. Appendix E evaluates the same checkpoint three times, which measures evaluation noise, not training variance. A paired test over tasks costs nothing, and three seeds for ARISE, OnlineRubrics and outcome-only RL would make the main claim much more convincing.

Q4: How many judge and reflection tokens, or GPU-hours, does one ARISE run use, and how do outcome-only RL and VADE perform with the same total compute? Do the baselines use the same length-regularized reward (Eq. 17)?

Q5: Figure 3(b) suggests that SkillsBench was evaluated during training, and I could not find a validation split. Is this a common practice for this benchmark? If so, are the hyperparameters selected based on the testing set?

Thank you for these valuable suggestions. We apologize for any confusion about the training details. Following your suggestions, we added robustness experiments and clarify compute costs and validation below.

1. Regarding compute cost, both baselines perform worse after extended training. Both outcome-only RL and VADE decline from step 150 to step 200, from 34.5% to 30.7% and from 38.7% to 34.5%, respectively, which means that increasing their training budgets does not narrow the gap to ARISE in these runs. To clarify ARISE's additional cost, one run uses approximately 3.1 billion judge tokens and 8.5 million reflection tokens. Although these calls add inference compute, their latency is already included in the per-step times reported in Appendix H: 17.14 minutes for ARISE and 16.46 minutes for outcome-only RL. The estimated 65% saving therefore refers specifically to elapsed time to reach comparable performance.

2. Regarding hyperparameter selection, all methods use the same validation split and learning-rate search space. Of the full 1,728-task corpus, we reserve 10% across multiple domains for validation and use the remainder for training. Given the training cost of approximately two days on 32 H800 GPUs per run, we restrict tuning to the same learning-rate grid of {1e-6, 2e-6, 5e-6} for ARISE and all baselines, selecting each method's best setting on this split. Other baseline-specific hyperparameters follow their original papers. In addition, all baselines use the same length-regularized reward in Eq. (17). Since hyperparameter selection uses the validation split, SkillsBench evaluations in Figure 3(b) serve only to track performance during training. We have clarified the validation split in Appendix C.1 and the shared tuning and reward settings in Appendix D.3.

3. Regarding robustness, ARISE's gains persist across training seeds and are statistically significant across tasks. Following your suggestion, we train ARISE, OnlineRubrics, outcome-only RL, and ARISE with outcome-based VADE sampling (w/ VADE Sampling) with three seeds and evaluate each checkpoint three times on SkillsBench. The table below reports the mean and sample standard deviation across seed-level averages, showing that ARISE's higher mean performance persists across training runs. We have added these results to Appendix E.2.

MethodSkillsBench v1.1 pass rate (%)
Outcome-only RL34.87 ± 0.66
OnlineRubrics40.10 ± 1.73
ARISE w/ VADE Sampling42.91 ± 0.77
ARISE46.10 ± 1.23

Following your recommendation, we assess task-level significance using the original checkpoints from the main comparison. Specifically, we average each task's three evaluation outcomes and apply two-sided paired permutation tests to mean differences across 87 SkillsBench tasks and 89 Terminal-Bench tasks. We compute 95% confidence intervals using a task-level paired percentile bootstrap. As shown in the table below, all four comparisons remain significant after Holm correction across the four tests.

BenchmarkComparisonGain (pp)95% CI (pp)Holm-adjusted p
SkillsBench v1.1ARISE vs. OnlineRubrics+5.36[1.53, 9.58]0.0306
SkillsBench v1.1ARISE vs. Outcome-only RL+11.11[4.60, 17.62]0.0061
Terminal-Bench v2.1ARISE vs. OnlineRubrics+4.49[0.37, 8.24]0.0424
Terminal-Bench v2.1ARISE vs. Outcome-only RL+7.12[2.25, 12.36]0.0304

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

Dear Area Chair, Senior Area Chair, and Program Chairs,

Thank you for your time and effort in overseeing the review of our paper. We are highly encouraged that all three reviewers have reached a consensus toward acceptance (scores: 4, 3, 3). For ease of assessment, we summarize the key findings and revisions below.

  1. Gains without a stronger external model (1RGt W1/Q1, bRt5 W1): ARISE still outperforms outcome-only RL and OnlineRubrics when the fixed initial policy serves as both judge and reflection model, demonstrating that its gains do not require a stronger model. These results are included in Appendix J.
  2. Gains beyond skill following (1RGt W2/Q3): ARISE retains its advantage without SkillsBench's curated skills (Appendix K). We also clarified the consistent loss context and subsequent removal of ARISE-generated skill guidance, and discussed context distillation and LUFFY in Appendix B.4.
  3. Training robustness and component contributions (1RGt W3, bRt5 W2/Q4): Experiments with additional training seeds (Appendix E.2) and task-level statistical tests reported in our responses support ARISE's improvements. Expanded ablations show consistent component benefits across benchmarks and training stages (Figure 3(a) and Appendix L).
  4. Generalization and sensitivity (1RGt W5/Q7, PAo5 W1/W2): ARISE's gains persist across the tested policy backbones, judge models, and advantage weights (Appendices I and E.3). We also clarified the adaptations needed for environment transfer and which mechanisms remain reusable.
  5. Training costs and fair tuning (1RGt W3/Q4/Q5, bRt5 Q2): We clarified judge and reflection costs and the elapsed-time basis of the reported savings, and found that extended baseline training does not improve performance in these runs. The revised paper also specifies the validation split and shared learning-rate search (Appendices C.1 and D.3).
  6. Sampling and reproducibility (1RGt W4, bRt5 Q1/Q3/Q5): Our responses address taxonomy setup effort, sampling feedback, and skill-switching stability. The revised Figure 2 and Section 3.2.2 clarify the workflow, while Appendices B.6, C.2, and C.3 provide behavioral guidance, taxonomy construction, and the complete task-type registry.

All three reviewers expressed positive recommendations following the discussion:

  • Reviewer bRt5 (Score: 4, Clear Acceptance): Found our responses satisfactory and raised their recommendation to Clear Acceptance.
  • Reviewer 1RGt (Score: 3, Weak Acceptance): Confirmed that their concerns were mostly addressed and raised their score toward acceptance.
  • Reviewer PAo5 (Score: 3, Weak Acceptance): Acknowledged the additional clarification and maintained their positive evaluation.

We sincerely appreciate the reviewers' and chairs' time and guidance, which have significantly strengthened the paper. The revised manuscript incorporates the additional evidence and clarifications summarized above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.