acceptodds
Under review as a conference paper at ICLR 2027

RISE-RL: Rubric-Informed Selective Exploration for Open-Ended Reinforcement Learning

Abstract

Aligning Large Language Models (LLMs) for open-ended tasks is challenging because responses must satisfy multidimensional criteria without following a single correct generation trajectory. Existing rubric-based reinforcement learning (RL) methods compress fine-grained criterion-level feedback into scalar rewards, making persistent capability gaps difficult to target under limited on-policy exploration. We propose (Rubric-Informed Selective Exploration), which uses frequently missed rubric criteria under natural rollouts to elicit privileged trajectories that are difficult to discover through unguided exploration alone. RISE-RL retains only trajectories whose complete-rubric reward exceeds the mean reward of natural rollouts, and then re-evaluates them under the original prompt to emphasize behaviors that remain weakly supported by the natural policy. The resulting guidance signal is optimized through a separate auxiliary objective and removed once its additional benefit diminishes. Experiments with 4B and 14B models across writing, chat, health, and science show that RISE-RL achieves the highest mean score on every evaluated benchmark under guidance-free evaluation. Compared with standard Rubric-RL, it improves the average score by 1.3 points at the 4B scale and , including a gain on CreativeWriting-V3. It also improves creative-writing diversity and yields gains on objectively scored medical and scientific benchmarks. These results indicate that selective internalization through reward filtering and policy support shaping is effective for open-ended reinforcement learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
post-training, reinforcement learning, rubric

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HWeZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RISE-RL that helps an AI learn from specific mistakes rather than relying only on an overall score. It identifies requirements the AI repeatedly misses and temporarily provides hints to generate better answers. It then uses responses that score above the average of the original responses to train the AI to respond without those hints. The core idea is to give targeted help early in training, then remove it so the AI can perform on its own.

Critical strengths

Instead of relying only on an overall score, the method identifies which requirements the AI repeatedly misses and targets them during training The hints are temporary, and evaluation is conducted without them. This tests whether the AI retains what it learned It shows results across different tasks, the method improves reported performance in writing, chat, health, and science using two model sizes. In the tested writing setting, it exceeds a baseline given similar or greater training resources. The ablations investigate several design choices

Critical weaknesses

Table 9 reports three evaluations of the same trained checkpoint, rather than three independently trained models. This measures variability in generation and judging, but leaves uncertainty about training stability (especially in places where gain is only 0.13 points, or 0.32 points) The paper discusses several papers (e.g., Rubric-Guided Self-Distillation and RGR-GRPO ) but does not evaluate them. These comparisons would help establish the added value of selective guidance, reward filtering, policy support shaping. Some ablations change more than one factor, making the results harder to interpret. For example, in effect of policy support shaping, it changes both which tokens receive more attention and the overall strength of the training update. The comparison does not clearly separate these effects. in the reward filtering, “no-filter” version actively discourages below-average responses rather than simply learning from all guided responses. results favor the proposed settings, but do not establish why they help Both model sizes belong to the Qwen3 family, and the four domains use separately trained models. The results therefore do not establish that one jointly trained model improves across all domains.

Questions for authors

Table 9 reports repeated evaluations of the same checkpoint. Have you repeated training with different random seeds, and do the improvements over Rubric-RL remain consistent? Do you have comparisons with referenced papers (e.g., Rubric-Guided Self-Distillation or RGR-GRPO) that establish the added value of RISE-RL? Why does the “no-filter” variant use signed gains, which actively discourage below-average responses? Compare methods using the same computing budget on more models and tasks

Reviewer_duXQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RISE-RL (Rubric-Informed Selective Exploration), which selects frequently missed rubric criteria under natural rollouts to elicit privileged trajectories that are difficult to discover through unguided exploration alone. It combines GRPO loss and guidance loss at the beginning of model traning, and removes guidance when its reward advantage saturates. The proposed method outperforms baselines in experiments.

Critical strengths
  1. The motivation is clear and the research topic is meaningful. It is difficult to improve failure cases for aggregrating criterion-level feedbacks as scalar reward under existing rubric-RL methods.

  2. The paper conducts extensive experiments across multiple benchmarks and outperforms baselines.

  3. The paper conducts several ablation studies to estimate the influence of criterion selection, reward filtering, policy support shaping, decoupled optimization and guidance removal.

Critical weaknesses
  1. The core contribution is not distinguished from existing works [1-2]. The guidance loss is highly similar to OPSD and RGSD. In experiment, it does not take OPSD and RGSD as baselines. The paper should add GRPO + OPSD-style auxiliary learning and GRPO + RGSD-style auxiliary learning.

  2. The paper lacks several important ablation studies. To understand which module of the proposed method improves performance, it should conduct following extra ablation experiments: (1) GRPO + reward-filtered privileged SFT without support shaping, (2) GRPO + reward-filtered on-policy self-imitation (i.e., an auxiliary NLL loss on high-reward natural-prompt rollouts), and (3) only guidance loss.

  3. The paper does not explain how the proposed method improves exploration. The Proposition 1 shows that guidance loss is related to token-support-adaptive weighted SFT. However, it does not provide theoretic analysis about RL exploration and can not show that token with low probability always correspond to valuable capabilities that the model has yet to acquire.

  4. The evaluation heavily relies on LLM judges, which raises concerns about reward hacking and evaluator bias. It is better to include IFEval and IFBench, which provide programmatically verifiable instruction-following evaluations.

[1] Rubric-Guided Self-Distillation: Post-Training Without Rubric Verifiers [2] Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation

Questions for authors

refer to weakness

Reviewer_mboiOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The article believes that using rubics in open writing tasks will cause reward collapse, so it wants to use guided trajectories mixed with RL to improve model performance. So a set of processes was designed, first filtering the rollout trajectory, then scoring, then doing OPD, and then training RL to improve the score.

Critical strengths
  • The author wrote more formulas to make the optimization results look more credible
  • Conducted a series of ablation experiments to prove the effectiveness and token efficiency of the new method
Critical weaknesses
  • The method has a large number of empirical parameters, although sensitivity analysis is performed on the parameters
  • The paper believes that an excessive teacher gap in OPD will lead to performance degradation, but in fact there are many solutions to an excessive teacher gap in OPD. Most conclusions are drawn on a particularly weak baseline
Questions for authors

Can more detailed experiments be conducted to prove that the method in this article is still helpful in improving model performance while remediating OPD performance loss?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.