acceptodds
Under review as a conference paper at ICLR 2027

RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillation

Abstract

Training a single LLM agent jointly across diverse interactive environments has attracted increasing attention as a route to generalist agents. Existing curriculum and data-selection strategies often allocate training at the environment level or prioritize local reward-based signals, without explicitly considering relationships between current rollouts across environments for prompt-group selection. Meanwhile, as environments are learned at different rates, all-failure and all-success rollout groups can coexist within a batch, leaving those data without group-relative reward signals. Both challenges highlight limitations of relying solely on scalar rewards in multi-environment RL: they provide limited information about cross-environment relationships and no within-group reward contrast when rewards are identical. This motivates richer textual feedback, such as rubrics describing rollout behaviours, to guide learning. Beyond rubrics' usage in reward construction, we repurpose rubrics to guide both online data selection and policy supervision. An LLM judge tags each rollout using a predefined rubric vocabulary shared across environments. The resulting profiles guide the selection of data that aligns with the overall behavioural composition of the mixed-environment batch while limiting overlap with already-selected data. Available positive rubrics (describing desired behaviours) provide privileged context for an on-policy self-distillation teacher, supplying additional token-level supervision, while negative rubrics (describing undesired behaviours) guide subsequent rollout generation away from recurring failure modes. Together, these components form RISED. Across model backbones, RISED achieves the highest mean pass rate across environments and ranks first or second in every individual environment. Rubric-based analysis of RISED can further characterize the behavioural changes accompanying these gains.

Primary area:
generative models
Keywords:
agentic ai, rubrics, data selection, distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RLszOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses GRPO to train a single LLM agent on ALFWorld, WebShop and DBBench at the same time. For every rollout, an LLM judge assigns rubric labels drawn from one common vocabulary, and these labels serve to choose prompt groups, to give context to a self-distillation teacher, and to guide the training prompts of the following round. Across two Qwen backbones, RISED reaches the best overall pass rate.

Critical strengths
  1. Because all environments use one common rubric vocabulary, rollouts collected in different environments can be compared directly, and a single labeling pass feeds selection, distillation and steering at once (Algorithm 1).
Critical weaknesses
  1. The work offers little novelty, since each of its components closely matches an earlier method. Skill-SD (arXiv 2604.10674) already proposes the distillation objective together with a teacher that receives summaries built from rollouts, RuPI (arXiv 2608.02948) already gives rubrics to the teacher as privileged context, and the selection score takes the MMR form previously applied to embeddings (Sec. 2.4). The only new element is choosing samples by their rubric profiles.

  2. No experiment measures the effect of rubric-based selection by itself, and this is the only new component. RISED improves on the strongest baseline by 1.2 and 0.6 points (Table 2), and I did not find a variant in which distillation and steering stay in place while Eq. 1 is swapped for variance-based or random selection. The component ablation is run with a single seed (Sec. 5).

  3. RISED is compared with the baselines on unequal terms, since it gets knowledge from stronger models. Gemini-2.5-Pro writes the rubric descriptions from a set of rollouts that contains successful ones by Claude-Opus-4.6 (App. A.1), and these descriptions are inserted into both the training prompts and the teacher context (Sec. 3), whereas GRPO and the variance selectors have access to the task prompt alone.

  4. The method is difficult to understand, since the main text has no diagram of it. Every one of the four figures in the main text plots training curves, and the overview and the pseudocode are placed only in the appendix (Figure 5, Algorithm 1).

Questions for authors
  1. What fraction of the self-distillation signal goes to the zero-variance groups that are given as its motivation? The paper motivates distillation with the 62.9% of candidate groups whose rewards are all the same (Sec. 2.4), yet a group with no positive rubric gets no teacher context (App. E.6), and the judge produces positive rubrics only for successful rollouts. Could the authors report, among the selected zero-variance groups, how many are all-success and how many are all-failure, and what fraction of rollouts have a non-empty teacher context?
4 comments

Thank you for your detailed feedback and for recognizing our design of the rubrics twice. We have addressed the weaknesses and questions as follows:

The work offers little novelty, since each of its components closely matches an earlier method. Skill-SD (arXiv 2604.10674) already proposes the distillation objective together with a teacher that receives summaries built from rollouts, RuPI (arXiv 2608.02948) already gives rubrics to the teacher as privileged context, and the selection score takes the MMR form previously applied to embeddings (Sec. 2.4). The only new element is choosing samples by their rubric profiles.

RISED is designed for joint training across heterogeneous, multi-turn environments. Each update draws on trajectories from several environments. This exposes two limitations of scalar rewards, which motivate our two contributions.

  1. Rubric-based selection utility. Our methodological contribution is a behaviour-based selection utility function for mixed-environment RL. Eq. 1 uses an MMR-style score over shared rubric profiles, so selecting a group in one environment can affect which groups are chosen in another while the environment quotas stay fixed. Designing the selection utility is a central part of online data selection, as illustrated by GREATS (Wang et al., 2024).
  2. Complementary supervision from the same annotations. Selected groups with identical rewards have no within-group reward contrast, even when their trajectories contain useful behaviours. We reuse the selection annotations to condition the policy teacher on positive rubrics under Skill-SD’s objective and to guide subsequent rollouts with negative rubrics (Sec. 3, Algorithm 1). We also design the positive distillation and negative steering to route positive and negative rubrics to distinct optimization roles. The role-filter ablation evaluates these two uses (App. C.3). Across 50 sample training steps from one Qwen2.5-3B run, 62.4% of selected groups had identical rewards. 70.6% of rollouts in those groups received positive teacher context. Among all selected prompt groups, 18.3% had identical rewards and provided no positive teacher context to any rollout. Negative tags from these groups can still guide subsequent rollouts (Table 12).

Concurrent RuPI (Bablani et al., 2026) uses task rubrics for open-ended generation. All-Rubric SD adapts its teacher conditioning to interactive, multi-turn, multi-environment RL using RISED’s rollout-derived vocabulary. Its final-checkpoint gains over GRPO-64 on both backbones support the use of rollout-derived rubrics for distillation in this setting (Sec. 4.2 and Table 2).

To our knowledge, RISED is the first to reuse the same rollout annotations for online prompt-group selection and for conditioning an on-policy self-distillation teacher in joint multi-environment agent training.

No experiment measures the effect of rubric-based selection by itself, and this is the only new component. RISED improves on the strongest baseline by 1.2 and 0.6 points (Table 2), and I did not find a variant in which distillation and steering stay in place while Eq. 1 is swapped for variance-based or random selection. The component ablation is run with a single seed (Sec. 5).

To make the effect of rubric-based selection by itself clearer, we added GRPO-64 to Fig. 3 (Fig. 2 in the original submission). The purple curve keeps rubric-based selection and disables distillation and steering. Over the last ten evaluations, it averages 51.87%, compared with 48.15% for GRPO-64 and 50.14% in a separate random-subsampling run with distillation and steering disabled. Both selection runs target 128 candidates and update on 64 (Sec. 5 describes the component ablation). The green curve trains on the standard 64-group batch with rubric ranking disabled and retains distillation and steering. Its last-ten average is 51.27%, compared with 53.75% for full RISED (Fig. 3 and Sec. 5).

To further isolate the contribution of rubric-based ranking, we replaced it with Round-robin Variance while holding distillation and steering fixed. On Qwen2.5-3B, the hybrid averaged 53.34% over the last ten evaluations in one completed run. Round-robin Variance alone averaged 52.30%, and full RISED averaged 53.75%, an observed lead of 0.41 percentage points over the hybrid (App. C.9).


↓ Continued below ↓

RISED is compared with the baselines on unequal terms, since it gets knowledge from stronger models. Gemini-2.5-Pro writes the rubric descriptions from a set of rollouts that contains successful ones by Claude-Opus-4.6 (App. A.1), and these descriptions are inserted into both the training prompts and the teacher context (Sec. 3), whereas GRPO and the variance selectors have access to the task prompt alone.

All methods are evaluated on held-out tasks using the same original task prompts. RISED uses rubric guidance and privileged teacher context during training only. This follows the self-distillation setup of the two works cited in the review: Skill-SD uses an auxiliary model, Seed1.8, to summarize agent rollouts into skills. RuPI uses task-specific rubrics supplied by its datasets, including physician-authored rubrics on HealthBench. In each case the policy-derived teacher produces the token-level targets and the external information supplies privileged context (Wang et al., 2026, and Bablani et al., 2026). RISED likewise uses privileged context during training and evaluates the policy on the original task generations.

To compare with a baseline that also has access to this vocabulary during training, we have included All-Rubric SD, which conditions its teacher on annotations from the same vocabulary. RISED achieves higher overall validation scores than All-Rubric SD on both backbones, supporting the benefit of how RISED uses the shared annotations.

The method is difficult to understand, since the main text has no diagram of it. Every one of the four figures in the main text plots training curves, and the overview and the pseudocode are placed only in the appendix (Figure 5, Algorithm 1).

We agree that readers should be able to see a full RISED pipeline in the main text. In the revision, we moved the full training procedure (Algorithm 1) and the pipeline diagram (Figure 1) into the main text. The performance curves report held-out validation pass rates. RISED and the baselines are evaluated using the same original task prompts, with rubric guidance restricted to training. We make this evaluation protocol explicit in the figure captions (Sec. 3 and Fig. 2).

What fraction of the self-distillation signal goes to the zero-variance groups that are given as its motivation? The paper motivates distillation with the 62.9% of candidate groups whose rewards are all the same (Sec. 2.4), yet a group with no positive rubric gets no teacher context (App. E.6), and the judge produces positive rubrics only for successful rollouts. Could the authors report, among the selected zero-variance groups, how many are all-success and how many are all-failure, and what fraction of rollouts have a non-empty teacher context?

Across 50 sampled training steps from one Qwen2.5-3B RISED run, 62.4% of selected prompt groups had identical rewards. Among those groups, 45.4% were all-success, 31.3% were all-failure, and 23.3% had equal partial rewards. 70.6% of rollouts in these groups had positive teacher context. Coverage was 100% for all-success and equal-partial-reward groups and 6.1% for all-failure groups. Of all selected prompt groups, 18.3% had identical rewards and no positive teacher context for any rollout. Negative tags can still guide subsequent rollouts (App. C.10, Table 12).

Overall, the revision brings the pipeline diagram and Algorithm 1 into the main text, clarifies the component ablations in Sec. 5, adds the variance-plus-feedback control in App. C.9, and reports teacher-context coverage in App. C.10 and Table 12.

Thank you for the detailed response and the new experiments. The coverage statistics answer my question, and moving the method overview into the main text improves readability. However, the new control in App. C.9 shows that rubric-based selection, the only new component, adds 0.41 points over Round-robin Variance with distillation and steering kept fixed, from a single run, which is within the variance of the baselines. The novelty concern also remains

Thank you for acknowledging that the coverage analysis answers your question and that the revised overview improves readability. We would like to clarify the characterization of rubric-based selection as “the only new component.”

RISED adopts Skill-SD’s loss and further designs how rollout-derived rubrics supply supervision in joint multi-environment training. We assign positive and negative rubrics distinct roles: demonstrated behaviours condition the teacher, while violations guide subsequent rollouts. App. C.3 evaluates these choices with rubric selection retained. Over the last ten evaluations, all-rubric teacher conditioning scores 49.68%, positive-only conditioning without steering scores 53.44%, and full RISED averages 53.75%. Although full RISED and positive-only conditioning reach similar late-training performance, the figure shows faster early learning with negative-rubric steering, most clearly on DBBench. These comparisons support using positive rubrics for teacher conditioning and negative rubrics for rollout guidance.

The combination of selection and distillation follows from the training problem. Shared rubric profiles make behaviours comparable across environments for selection, and the same annotations can supply teacher context when selected groups have identical rewards. Our contribution therefore includes the design and evaluation of reusing shared rubric annotations for selection and supervision.

The new comparison is a selector-replacement ablation: it replaces rubric ranking with Round-robin Variance while retaining RISED’s positive-rubric teacher conditioning and negative-rubric steering. It assesses the incremental effect of ranking within a pipeline that already incorporates our supervision design, which App. C.3 evaluates separately.

We hope these clarifications help you reassess the scope and novelty of our contribution.

Reviewer_9UZsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RISED trains a single policy with GRPO jointly on ALFWorld, WebShop and DBBench (AgentBench via AgentRL) and uses LLM-assigned rubric tags in three places. A fixed vocabulary of 59 rubrics in 10 skills (82/11 with AppWorld) is built offline from warm-up rollouts and Claude-Opus-4.6 teacher rollouts using Gemini-2.5-Pro; online, a Qwen3-32B judge tags every rollout with (skill, rubric, role, count). (1) Selection: 128 candidate prompt groups are oversampled and 64 are chosen round-robin per environment by an MMR-style score, cosine similarity to the batch's aggregate rubric profile minus similarity to the already-selected aggregate. (2) Self-distillation: the policy conditioned on descriptions of a rollout's top-3 positive rubrics (plus its group's) acts as teacher for the unconditioned student, with a Skill-SD reverse-KL loss at weight 0.01. (3) Negative steering: when violations outnumber demonstrations in an environment, its top-3 negative rubric descriptions are prepended to that environment's prompts at the next step. A preliminary analysis argues that rubric-profile similarity between training and validation rollouts correlates with reward improvement better than gradient cosine similarity. On Qwen2.5-3B and Qwen3-4B, RISED reports the highest mean pass rate (0.542 vs. 0.530 and 0.585 vs. 0.579 for the strongest baseline) at 1.63x the step time of GRPO-64 plus a dedicated judge server.

Critical strengths
  1. The problem is real and well quantified: 62.9% of candidate groups in this multi-environment setting have zero within-group reward variance and therefore contribute no GRPO signal, and using rubrics as behavioural descriptions rather than as rewards sidesteps the calibration burden of rubric rewards. The design, positive rubrics as privileged teacher context and negative rubrics as rollout guidance, is sensible and the role-filter ablation (App. C.3) gives some support for the split.

  2. The paper is candid about its own limitations where it matters: the reward-conditioned tagging confound (E.4), the zero-gradient case for groups without positive rubrics (E.6), the environment-level rather than group-level nature of the Sec. 2.3 analysis (E.3), the wall-clock overhead (Tab. 9) and the tagger reliability study (C.2). This makes the paper easy to assess, even where the assessment is unfavourable.

Critical weaknesses
  1. The reported gains are within seed noise and the evidence base is too thin to support the claims. Over the strongest baseline, RISED gains 1.2 points on Qwen2.5-3B and 0.6 on Qwen3-4B, comparable to the reported standard deviations, and the number of runs behind "multiple runs" is never stated. Table 2 reports a single final checkpoint, while Fig. 1 shows noisy, interleaved curves; some entries (std 0.000 for GRPO-128, an overall std of 0.001 for RISED on Qwen3-4B against per-environment stds of 0.011 and 0.012) suggest very few seeds. Every ablation (Figs. 2, 4, 6, 8) is single-seed and reported only as curves, so "removing either component reduces overall performance" cannot be read off them at this effect size. Per environment, RISED is not best on ALFWorld on either backbone or on DBBench on Qwen3-4B. The cost side is also not matched: RISED runs at 1.63x the step time of GRPO-64 and adds an eight-GPU server for a 32B judge, roughly ten times the policy size; "reaches GRPO-64's final pass rate in half as many optimizer steps" is 0.82x in wall-clock and about 1.0x in GPU-hours, and no compute-matched control (GRPO-64 run for 1.63x the steps, with the baselines in Fig. 1 still rising at step 1000) is provided. Round-robin Variance captures most of the gain at 1.14x cost and this is not discussed.

  2. The analysis motivating the selection rule is confounded, and the rule does not follow from it. The judge is given the rollout reward and follows outcome-dependent instructions: successes receive 1-3 positive tags with count 1, failures receive 2-4 negative tags with counts. A rubric profile therefore encodes success and failure composition directly, so its correlation with reward improvement is partly a correlation of reward with reward; the paper concedes this in E.4 but offers no reward-statistics-only baseline. The positive-only profile is negatively correlated with held-out improvement (rho = -0.52, Tab. 1), which contradicts the behavioural reading. The headline that rubric similarity "identifies the most-improved environment at all 19 checkpoints" is computed over all four environments including the source WebShop, whose training rollouts trivially resemble its own validation rollouts and which trivially improves most; the held-out-only version of this statistic is not reported. Finally, an environment-level correlation under single-environment training does not imply that selecting groups most similar to the batch's mean profile (the relevance term of Eq. 1) helps learning, as E.3 itself acknowledges, so Sec. 2.4 rests on Sec. 2.3 in name only. A related gap: the "self"-distillation context is built from Claude-Opus-4.6 rollouts and Gemini-written descriptions, and negative steering injects those descriptions into training prompts, so a plain prompt-engineering control (GRPO with a static block of rubric descriptions) is needed to separate textual hints from the selection and distillation machinery.

Questions for authors
  1. How many seeds are behind Table 2? Please report per-seed values, the mean over the last 10 evaluations rather than the final checkpoint, and the component ablations with the same seeds.
  2. Please add two controls: GRPO-64 trained for 1.63x the steps (compute-matched to RISED), and Round-robin Variance selection combined with RISED's distillation and negative steering, which isolates what rubric-based selection adds over the cheap variance selector.
  3. For Sec. 2.3, please report the "most-improved environment" statistic on held-out environments only, and add a reward-only similarity baseline (e.g. similarity of success/failure composition between training and validation rollouts) so that the rubric signal's value beyond outcome composition is visible.

I would move to weak acceptance if (1) shows the overall gain over Round-robin Variance and GRPO-128 is outside seed noise on both backbones and (2) shows the compute-matched GRPO-64 does not close it.

5 comments

Thank you for your constructive feedback and for recognizing our problem, design, and presentations. We have addressed the weaknesses and questions as follows:

The reported gains are within seed noise and the evidence base is too thin to support the claims. Over the strongest baseline, RISED gains 1.2 points on Qwen2.5-3B and 0.6 on Qwen3-4B, comparable to the reported standard deviations, and the number of runs behind "multiple runs" is never stated. Table 2 reports a single final checkpoint, while Fig. 1 shows noisy, interleaved curves; some entries (std 0.000 for GRPO-128, an overall std of 0.001 for RISED on Qwen3-4B against per-environment stds of 0.011 and 0.012) suggest very few seeds. Every ablation (Figs. 2, 4, 6, 8) is single-seed and reported only as curves, so "removing either component reduces overall performance" cannot be read off them at this effect size. Per environment, RISED is not best on ALFWorld on either backbone or on DBBench on Qwen3-4B. The cost side is also not matched: RISED runs at 1.63x the step time of GRPO-64 and adds an eight-GPU server for a 32B judge, roughly ten times the policy size; "reaches GRPO-64's final pass rate in half as many optimizer steps" is 0.82x in wall-clock and about 1.0x in GPU-hours, and no compute-matched control (GRPO-64 run for 1.63x the steps, with the baselines in Fig. 1 still rising at step 1000) is provided. Round-robin Variance captures most of the gain at 1.14x cost and this is not discussed.

How many seeds are behind Table 2? Please report per-seed values, the mean over the last 10 evaluations rather than the final checkpoint, and the component ablations with the same seeds.

Runs and variability. Multi-environment agent training is costly. On 3B, even GRPO-64 takes about 1.8 days per 1,000 steps on 32 H100 GPUs before validation (App. E.7). We ran each Table 2 configuration twice and completed a third run where resources allowed. RISED, GRPO-64, and Global Variance each have three runs on 3B. Runs differ through nondeterminism in the asynchronous pipeline, and we reported the population SD across training runs. We report individual run results in Table 10. Each checkpoint uses four validation rollouts per prompt (Sec. 4.1). For example, the two 3B GRPO-128 runs have the same final ALFWorld pass rate, yielding an SD of 0.000. For 4B RISED, differences across environments partly offset in the overall average, yielding an SD of 0.001.

Checkpoint averages. We average the last ten evaluations within each run before aggregating across runs. We also report the last five. Entries are overall pass rates in percent, mean ± population SD. Table 11 gives the per-environment averages.

Method3B Last 53B Last 104B Last 54B Last 10
GRPO-6448.14 ± 2.0548.15 ± 1.9351.76 ± 4.9051.82 ± 5.10
GRPO-12851.30 ± 1.0350.85 ± 1.1658.09 ± 1.0558.49 ± 1.28
Round-robin Variance52.64 ± 0.4952.30 ± 0.3557.52 ± 2.4957.12 ± 2.50
RISED53.80 ± 0.4553.75 ± 0.5058.19 ± 0.2458.29 ± 0.12

Our primary metric is the unweighted mean pass rate across the three environments because one policy serves all three. RISED has the highest final-checkpoint overall mean on both backbones and remains first at 3B under both averaging windows. At 4B, GRPO-128 leads slightly over the last ten evaluations. However, note that we can treat GRPO-128 as a larger-update stress test. RISED and GRPO-128 each generate 128 groups per step. GRPO-128 updates on all 128, while RISED selects 64. The comparison also captures the effects of a larger update batch and using more rollouts per update (Sec. 4.2).

Component ablations. We used the same configured random seed across the component comparisons in Fig. 3 (Fig. 2 in the original submission). Over the last ten evaluations, selection alone achieves 51.87% and distillation with steering averages 51.27%, compared with 53.75% for full RISED. (Fig. 3 and Sec. 5).

Compute. Our main comparisons use the same number of optimizer steps and report step time separately, as SERL does (Li et al., 2026, Apps. B.1 and B.4). For RISED, most of the extra step time comes from waiting for rubric annotations and scoring with the teacher. Selection itself takes under 0.1 seconds (App. C.6). On Qwen2.5-3B, RISED first reaches the 1,000-step GRPO-64 baseline’s final overall pass rate in roughly half as many optimizer steps. Using Table 9’s first-300-step timings, this corresponds to about 0.82× the elapsed training time excluding validation and roughly equal allocated GPU-hours, including the judge.


↓ Continued below ↓

The analysis motivating the selection rule is confounded, and the rule does not follow from it. The judge is given the rollout reward and follows outcome-dependent instructions: successes receive 1-3 positive tags with count 1, failures receive 2-4 negative tags with counts. A rubric profile therefore encodes success and failure composition directly, so its correlation with reward improvement is partly a correlation of reward with reward; the paper concedes this in E.4 but offers no reward-statistics-only baseline. The positive-only profile is negatively correlated with held-out improvement (rho = -0.52, Tab. 1), which contradicts the behavioural reading. The headline that rubric similarity "identifies the most-improved environment at all 19 checkpoints" is computed over all four environments including the source WebShop, whose training rollouts trivially resemble its own validation rollouts and which trivially improves most; the held-out-only version of this statistic is not reported. Finally, an environment-level correlation under single-environment training does not imply that selecting groups most similar to the batch's mean profile (the relevance term of Eq. 1) helps learning, as E.3 itself acknowledges, so Sec. 2.4 rests on Sec. 2.3 in name only. A related gap: the "self"-distillation context is built from Claude-Opus-4.6 rollouts and Gemini-written descriptions, and negative steering injects those descriptions into training prompts, so a plain prompt-engineering control (GRPO with a static block of rubric descriptions) is needed to separate textual hints from the selection and distillation machinery.

For Sec. 2.3, please report the "most-improved environment" statistic on held-out environments only, and add a reward-only similarity baseline (e.g. similarity of success/failure composition between training and validation rollouts) so that the rubric signal's value beyond outcome composition is visible.

Held-out transfer. With WebShop excluded, combined rubric similarity ranked the most-improved held-out environment first at all 19/19 checkpoints, compared with 8/19 for gradient similarity and 9/19 for binary outcome similarity. Their held-out Spearman correlations are +0.595, +0.183, and −0.127, respectively (App. C.1). Note that a binary outcome counts complete successes, so it misses gains in partial credit. For example, AppWorld’s mean reward can rise while its full-success rate stays zero, helping explain why binary outcome similarity tracks reward improvement weakly.

From transfer to selection. A direct test would start from the same policy checkpoint, update on one candidate prompt group at a time, and compare the reward gains on held-out tasks representing the rest of the batch. The effect of one group can be small, so this would require many matched updates and repeated evaluations (App. E.3). We use the same idea at the environment level, where each source aggregates many rollouts. We train on one source environment at a time, WebShop or DBBench, and evaluate on the others. Under WebShop training, held-out environments with more similar combined rubric profiles tend to improve more (Sec. 2.3 and App. C.1). This motivates the relevance term in Eq. 1, where a candidate group plays the role of the source and the mixed-environment batch plays the role of the target. The selection-only comparison evaluates Eq. 1 as a whole. Additionally, the existing selection-only comparison evaluates Eq. 1 as a whole. On Qwen2.5-3B, selection alone averages 51.87% over the last ten evaluations, above GRPO-64 (48.15%) and a separate random-subsampling run (50.14%).

Rubric supervision. Claude rollouts contributed to the offline rubric vocabulary, which All-Rubric SD also uses to condition its teacher. All-Rubric SD trains without rubric-based rollout steering. Table 9 additionally reports similar teacher-scoring times for All-Rubric SD and RISED (30.5 and 30.6 seconds per step, respectively). RISED’s policy snapshot supplies the distillation targets, conditioned on rubric tags from the policy’s own rollouts. Other self-distillation methods also use privileged training context. SERL uses hindsight environment feedback, Skill-SD uses skills summarized by Seed1.8, and RuPI uses dataset-provided task rubrics (Li et al., 2026, Sec. 3, Wang et al., 2026, App. F, and Bablani et al., 2026). All methods use the original task prompts at validation.


↓ Continued below ↓

Please add two controls: GRPO-64 trained for 1.63x the steps (compute-matched to RISED), and Round-robin Variance selection combined with RISED's distillation and negative steering, which isolates what rubric-based selection adds over the cheap variance selector.

During the rebuttal, we prioritized the Round-robin Variance ablation. Note that many data-selection methods could be paired with different distillation objectives or teacher contexts. RISED studies a specific design motivated by joint training across heterogeneous environments. Shared rubric profiles make rollout behaviours comparable across environments, and the same annotations provide privileged teacher context when selected groups have identical rewards. To further isolate the contribution of rubric-based ranking, we replaced it with Round-robin Variance while holding distillation and steering fixed. On Qwen2.5-3B, the hybrid averaged 53.34% over the last ten evaluations in one completed run. Round-robin Variance alone averaged 52.30%, and full RISED averaged 53.75%, an observed lead of 0.41 percentage points over the hybrid (App. C.9).

Overall, the revision reports held-out transfer and binary outcome comparisons, more ablations, individual-run results, and per-environment averages over the last five and ten evaluations (Apps. C.1, C.7, C.8 and C.9)

Reviewer_2MwZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RISED for multi-environment agentic RL. An LLM judge (Qwen3-32B) tags each rollout with behaviours from a fixed vocabulary of 59–82 positive and negative rubrics, built offline from warm-up rollouts and Claude-Opus-4.6 rollouts. The rubric tags are used in two ways. First, for online data selection: from a 2× oversampled candidate pool, prompt groups are picked with an MMR-style relevance/redundancy score over rubric profiles. Second, for training signal: positive rubrics serve as privileged context for an on-policy self-distillation teacher (using Skill-SD's objective), and negative rubrics are added to the next step's prompts as "anti-patterns". A preliminary analysis shows that rubric-profile similarity tracks cross-environment reward gains better than gradient similarity. On ALFWorld, WebShop and DBBench with Qwen2.5-3B-Instruct and Qwen3-4B, RISED gets the best overall pass rate (0.542 and 0.585), +5.8 and +6.9 points over GRPO-64 and +1.2 and +0.6 points over the strongest baseline.

Critical strengths
  1. Using rubric profiles as a representation is an interesting angle, and the analysis is honest. The comparison of rubric similarity against gradient similarity for predicting cross-environment transfer (Table 1) is a nice diagnostic. The paper also reports its costs openly, including per-step wall-clock time, judge size and tagging stability, which makes the trade-offs easy to judge.
Critical weaknesses
  1. Limited novelty: the method mainly combines existing pieces. Rubric-based training of LLM agents is already an active area. Beyond rubrics as rewards, DR Tulu [1] trains agents with rubrics that evolve during RL, but it is not cited or discussed, even though its evolving rubrics are a direct alternative to this paper's fixed vocabulary. Each component of RISED also follows prior work closely. The distillation loss is taken directly from Skill-SD [2]. Conditioning a self-distillation teacher on rubrics is already done by RuPI [3], which the authors call the closest work and use as a baseline. The selection rule is standard MMR scoring, which the paper notes is already used for online batch selection over embeddings. The new part is applying these to a shared rubric vocabulary, which on its own feels incremental.

  2. The empirical experiment is too weak to support the claims. All experiments use small, older, non-thinking models (Qwen2.5-3B, Qwen3-4B) on three older environments. The one harder environment, AppWorld, appears only in a stress test with near-zero scores. The baselines are simple variants the authors implemented, not the published selection methods they cite (e.g., DAPO, VCRL, PCL/DOTS). The gains over the best baseline are 1.2 and 0.6 points, within or close to the reported standard deviations, and the number of seeds is not stated. The comparison is also not compute-matched: RISED's step time is about the same as GRPO-128's, plus a dedicated 8-GPU judge server, and against GRPO-128 the gain on Qwen3-4B is only 0.6 points. Finally, the rubric vocabulary is partly built from Claude-Opus-4.6's successful rollouts, so the positive rubrics given to the teacher may pass frontier-model knowledge to the policy, which none of the baselines receive. As a result, it is hard to tell how much of the gain comes from the method itself.

[1] Shao et al. DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research. https://arxiv.org/abs/2511.19399

[2] Wang et al. Skill-SD: Skill-Conditioned Self-Distillation for Multi-Turn LLM Agents. https://arxiv.org/abs/2604.10674

[3] Bablani et al. Rubrics as Privileged Information for Open-Ended Generation. https://arxiv.org/abs/2608.02948

Questions for authors
  1. How many seeds are behind Table 2, and are the gains over the best baseline statistically significant? Can you compare at matched compute, counting the judge GPUs, e.g., against GRPO with a larger batch or more rollouts?
  2. How much of the gain comes from the Claude-Opus-4.6-derived rubrics? What happens if the vocabulary is built only from the policy's own rollouts?
  3. Do the gains hold with a more recent or larger backbone (e.g., with thinking enabled), or with harder agentic environments such as tool-use, SWE tasks? How does a fixed vocabulary compare with evolving rubrics as in DR Tulu?
3 comments

Thank you for your time for a detailed review and for recognizing our design of using rubrics as representation. We have addressed the weaknesses and questions as follows:

Limited novelty: the method mainly combines existing pieces. Rubric-based training of LLM agents is already an active area. Beyond rubrics as rewards, DR Tulu [1] trains agents with rubrics that evolve during RL, but it is not cited or discussed, even though its evolving rubrics are a direct alternative to this paper's fixed vocabulary. Each component of RISED also follows prior work closely. The distillation loss is taken directly from Skill-SD [2]. Conditioning a self-distillation teacher on rubrics is already done by RuPI [3], which the authors call the closest work and use as a baseline. The selection rule is standard MMR scoring, which the paper notes is already used for online batch selection over embeddings. The new part is applying these to a shared rubric vocabulary, which on its own feels incremental.

We thank the reviewer for identifying the omitted DR Tulu reference. We now discuss its evolving, instance-specific reward rubrics in Sec. 6. RISED, however, uses a fixed shared vocabulary for selection, keeping behavioural profiles comparable across environments and training steps. Changing the vocabulary would require aligning rubric names before comparing profiles and introduce more challenges for our setting (App. E.1). Joint training across heterogeneous, multi-turn environments motivates two contributions.

  1. Rubric-based selection utility. Our methodological contribution is a behaviour-based selection utility function for mixed-environment RL. Eq. 1 uses an MMR-style score over shared rubric profiles, so selecting a group in one environment can affect which groups are chosen in another while the environment quotas stay fixed. Designing the selection utility is a central part of online data selection, as illustrated by GREATS (Wang et al., 2024).

  2. Complementary supervision from the same annotations. Selected groups with identical rewards have no within-group reward contrast, even when their trajectories contain useful behaviours. We reuse the selection annotations to condition the policy teacher on positive rubrics under Skill-SD’s objective and to guide subsequent rollouts with negative rubrics (Sec. 3, Algorithm 1). We also design the positive distillation and negative steering to route positive and negative rubrics to distinct optimization roles. The role-filter ablation evaluates these two uses (App. C.3). Across 50 sample training steps from one Qwen2.5-3B run, 62.4% of selected groups had identical rewards. 70.6% of rollouts in those groups received positive teacher context. Among all selected prompt groups, 18.3% had identical rewards and provided no positive teacher context to any rollout. Negative tags from these groups can still guide subsequent rollouts (Table 12).

Concurrent RuPI (Bablani et al., 2026) uses task rubrics for open-ended generation. All-Rubric SD adapts its teacher conditioning to interactive, multi-turn, multi-environment RL using RISED’s rollout-derived vocabulary. Its final-checkpoint gains over GRPO-64 on both backbones can be considered as providing new support of the use of rollout-derived rubrics for distillation in this setting (Sec. 4.2 and Table 2). To our knowledge, RISED is the first to reuse the same rollout annotations for online prompt-group selection and for conditioning an on-policy self-distillation teacher in joint multi-environment agent training.


↓ Continued below ↓

The empirical experiment is too weak to support the claims. All experiments use small, older, non-thinking models (Qwen2.5-3B, Qwen3-4B) on three older environments. The one harder environment, AppWorld, appears only in a stress test with near-zero scores. The baselines are simple variants the authors implemented, not the published selection methods they cite (e.g., DAPO, VCRL, PCL/DOTS). The gains over the best baseline are 1.2 and 0.6 points, within or close to the reported standard deviations, and the number of seeds is not stated. The comparison is also not compute-matched: RISED's step time is about the same as GRPO-128's, plus a dedicated 8-GPU judge server, and against GRPO-128 the gain on Qwen3-4B is only 0.6 points. Finally, the rubric vocabulary is partly built from Claude-Opus-4.6's successful rollouts, so the positive rubrics given to the teacher may pass frontier-model knowledge to the policy, which none of the baselines receive. As a result, it is hard to tell how much of the gain comes from the method itself.

Do the gains hold with a more recent or larger backbone (e.g., with thinking enabled), or with harder agentic environments such as tool-use, SWE tasks? How does a fixed vocabulary compare with evolving rubrics as in DR Tulu?

Model choice and scope. We follow AgentRL’s Qwen2.5-3B setup and jointly train on ALFWorld, DBBench, and WebShop (Zhang et al., 2025). The environments differ in action space, rewards, and rollout length. SDAR and SERL also evaluate on ALFWorld and WebShop (Lu et al., 2026, and Li et al., 2026). We extend the evaluation to Qwen3-4B without thinking (Sec. 4.1). Skill-SD likewise uses a non-thinking Qwen3-4B-Instruct-2507 checkpoint for multi-turn agent training (Wang et al., 2026). The four-environment AppWorld experiment is a stress test (Fig. 4). We identify evaluation with larger and thinking-enabled policies as a limitation (Conclusion).

Baseline design. GRPO-64 matches RISED’s 64-group update size. Comparing online selection with uniform GRPO at a fixed update size has precedent in DOTS+RR (Sun et al., 2025). Our variance selectors use the reward-variability signal underlying DAPO and VCRL and match both RISED’s 128-group candidate pool and its 64-group update. GRPO-128 can be treated as a larger-batch stress test*. RISED and GRPO-128 each generate 128 groups per step. GRPO-128 updates on all 128, while RISED updates on 64. The comparison also captures the effects of a larger update batch and using more rollouts per update (Sec. 4.2).

Performance across environments. Our primary metric is the unweighted mean pass rate across the three environments because one policy serves all three (Sec. 4.1). RISED has the highest mean overall pass rate on both backbones at the final checkpoint and over the last five evaluations (Tables 2 and 11). Over the last ten evaluations, RISED remains ahead of GRPO-128 on Qwen2.5-3B and comparable on Qwen3-4B (58.29% vs. 58.49%, Table 11).


↓ Continued below ↓

How many seeds are behind Table 2, and are the gains over the best baseline statistically significant? Can you compare at matched compute, counting the judge GPUs, e.g., against GRPO with a larger batch or more rollouts?

Compute. Our main comparisons use the same number of optimizer steps and report step time separately, as SERL does (Li et al., 2026, Apps. B.1 and B.4). For RISED, most of the extra step time comes from waiting for rubric annotations and scoring with the teacher. Selection itself takes under 0.1 seconds (App. C.6). On Qwen2.5-3B, RISED first reaches the 1,000-step GRPO-64 baseline’s final overall pass rate in roughly half as many optimizer steps. Using Table 9’s first-300-step timings, this corresponds to about 0.82× the elapsed training time excluding validation and roughly equal allocated GPU-hours, including the judge.

Rubric supervision. Claude rollouts contributed to the offline rubric vocabulary, which All-Rubric SD also uses to condition its teacher. All-Rubric SD trains without rubric-based rollout steering. Over the first 300 steps, Table 9 additionally reports wesimilar teacher-scoring times for All-Rubric SD and RISED (30.5 and 30.6 seconds per step, respectively). RISED’s policy snapshot supplies the distillation targets, conditioned on rubric tags from the policy’s own rollouts. Other self-distillation methods also use privileged training context. SERL uses hindsight environment feedback, Skill-SD uses skills summarized by Seed1.8, and RuPI uses dataset-provided task rubrics (Li et al., 2026, Wang et al., 2026, and Bablani et al., 2026). All methods use the original task prompts at validation. A policy-only vocabulary comparison could measure the overall effect of removing Claude rollouts. However, constructing a separate policy-only vocabulary and re-running full multi-environment training plus component ablations exceeded the rebuttal compute window (~at least 2.9 days on 32 H100s per run without validation), so we prioritized controlled online selection and supervision ablations.

Run counts. Multi-environment agent training is costly. On 3B, even GRPO-64 takes about 1.8 days per 1,000 steps on 32 H100 GPUs before validation (App. E.7). We ran each Table 2 configuration two times and completed a third run where resources allowed. RISED, GRPO-64, and Global Variance each have three runs on 3B. Runs differ through nondeterminism in the asynchronous pipeline, and we reported the population SD across training runs. We report individual run results in Table 10.

Overall, the revision reports held-out transfer and binary outcome comparisons, more ablations, individual-run results, and per-environment averages over the last five and ten evaluations (Apps. C.1, C.7, and C.8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.