acceptodds
Under review as a conference paper at ICLR 2027

CritSel-OPSD: Criterion Selection for Rubric-Conditioned On-Policy Self-Distillation

Abstract

Rubric-conditioned On-Policy Self-Distillation (OPSD) gives the teacher structured criteria as privileged information, but existing formulations typically expose the entire rubric throughout training. We ask whether a rubric, usually treated as an indivisible supervision object, contains redundant teacher-context information. We introduce CritSel-OPSD, a criterion-selection framework that combines leave-one-out teacher-context interventions with Top- subset construction. Selection uses trajectories from the current student and is refreshed periodically, allowing successive training blocks to use student-state-dependent contexts. Sampled Teacher Influence (STI) provides a trajectory-aligned selector. In our evaluated setting with Qwen3-8B and OpenRubrics, compact teacher contexts preserve the observed benefits of full-rubric conditioning across six diverse benchmarks while using only a subset of rubric criteria. Comparisons with random controls indicate that the central result is the viability of compact teacher-context allocation, not the universal superiority of one ranking signal. These results identify criterion selection as a design dimension distinct from rubric design and reward construction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, rubric-conditioned self-distillation, on-policy self-distillation, dynamic criterion selection, rubrics, privileged information

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Yn1NOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Rubric conditioned on policy self distillation uses a rubric as privileged information for the teacher. However, rubrics are typically treated as indivisible. This paper introduces CritSel-OPSD, a framework for constructing subsets of criteria to provide as privileged information using student trajectories. CritSel-OPSD is used to train a Qwen3-8B model and the observed benefits of full rubric conditioning is preserved while only using a subset of rubric criteria.

Critical strengths
  • Related work organizes and represents literature well.
  • Limitations are honest.
Critical weaknesses
  • The motivation of dividing the rubric into criteria subsets is not clear to me, especially given the results are so close to Full-OPSD.
  • It is not clear to me why the selection processes should be related to usefulness. I feel that the similar results of selection approaches show the proposed selection methods may not be correlated to usefulness.
  • The impact of the proposed approach is not clear. Without independent runs it is difficult to determine if reported gains reflect a reproducible improvement.
Questions for authors
  • Empirically how does the student score against the criteria that are NOT selected at refresh boundaries?
  • Does figure 3b show that the position of the criteria in the rubric influences its selection rate?
  • How were hyperparameters chosen? Were the best hyperparameters chosen from a separate validation set separate from the set of reported on in the main results?
Reviewer_drQjOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper works on rubric-based reward modeling. Specifically, it studies the redundancy in generated rubrics, and how pruning some of them can maintain the downstream policy optimization using OPSD. The authors propose to determine each rubric criterion importance based on how the teacher would change its likelihood on the response with and without the criterion being presented in the context. The authors then propose to prune the criteria that have lower importance. The proposed method is tested on Qwen3-8B using an existing rubric dataset and tested on six downstream tasks.

Critical strengths
  1. The problem studied in this paper about adapting existing rubric-based reward modeling for policy optimization is important.

  2. The results look interesting. It is nice to see that the rubrics can be pruned but still provide helpful supervision for policy optimization.

Critical weaknesses
  1. The core motivation about pruning rubric for OPSD is not well justified. After reading through the paper, I am still not convinced by pruning rubrics for OPSD is conceptually beneficial. If a rubric has high-quality, for example, human-written rubrics (e.g., RubricHub) provide comprehensive evaluation criteria for the prompt. By definition they should be important, then why should we prune them for OPSD purpose? If the rubric is redundant in nature, then pruning itself is meaningful for improving the rubric quality, but not for OPSD, and in this case, it is necessary to show that the pruned rubrics can provide broader benefits beyond OPSD. The current results have two possible causes that I cannot distinguish: 1. redundancy in rubrics, and 2. the unique need of OPSD for "pruned" rubrics.

  2. The high-level idea of the proposed method is easy to understand. But the method section is hard to follow. The authors claimed that they used TI-JSD and SI-JSD as different instantiations, but they are not defined in the main body. The authors claimed that the proposed STI is just "a local LOO sensitivity, not an additive casual contribution or Shapley value", then why not use these better alternatives?

  3. The experiment is limited. The authors used Openrubrics for OPSD training of one backbone, and evaluated it on six downstream tasks, which are mostly verifiable tasks. Baselines are OPSD and some variants of the proposed methods. No rubric-based RM, or other OPSD variants were compared. These results were not able to distinguish the possibilities in W1. Moreover, if I understand correctly, the authors used Openrubrics data for OPSD. But Openrubrics does not contain many verifiable data, so it is unclear to me why training on this data would help improve the performance on these downstream tasks.

In overall, I think the problem studied in this paper can be of great importance to the community. But the motivation, the proposed method, the presentation, and the experiments are not sufficient to establish a clear contribution. The paper in its current form is not ready to publish.

Questions for authors

Please see my comments in the weakness section.

Reviewer_LydWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether rubric-conditioned on-policy self-distillation really requires giving the teacher the full rubric. It proposes CritSel-OPSD, which uses leave-one-out interventions on student-generated trajectories to rank rubric criteria and expose only a Top-K subset to the teacher. The default criterion score STI measures how much removing a criterion changes the teacher’s likelihood of the student’s realized tokens. Experiments show that using only four criteria can match or slightly outperform full-rubric conditioning across six downstream benchmarks.

Critical strengths
  • The paper identifies criterion-level teacher-context allocation as a distinct design choice within rubric-conditioned self-distillation. The result that roughly four criteria can retain the downstream gains of a full rubric averaging more than eight criteria is practical and interesting.

  • The experiments include a proper control set including random selection, original-order truncation, bottom-ranked selection etc. These controls help distinguish the feasibility of compact contexts from the effectiveness of a particular ranking rule.

Critical weaknesses
  • The evidence does not support the method over trivial baselines, and effect sizes are within unmeasured noise. The experiments use one model family and one rubric source, and the reported three seeds are decoding seeds for the same trained checkpoint rather than independent training runs. This makes the small differences between selectors difficult to interpret. As a result, the paper makes a good case that the teacher does not need the full rubric, but a much weaker case that the proposed influence-based selection mechanism is necessary. There is also a gap between the score and the intended learning objective. STI measures how much removing one criterion changes the teacher distribution, but a large teacher change does not necessarily imply a more useful learning signal. Similarly, the method scores criteria one at a time while the final Top-K intervention removes many criteria jointly, so interactions or redundancy between criteria are not captured.

  • The contribution of the rubric itself is not fully isolated. Full-OPSD improves Avg-6 from 58.10 to 59.44, but the paper does not include an empty-rubric teacher control, nor report the divergence between full-rubric and rubric-free teachers. Thus, it is unclear how much of the gain comes from the task-specific rubric content versus a more general context-induced change in the teacher.

Questions for authors
  • How were (K=4) and (H=100) chosen? Were these selected using a separate validation procedure, or based on the same downstream benchmarks reported in the paper?

  • What does OPSD with a rubric-free teacher achieve under the identical pipeline?

  • Can you provide independent training-seed results for Full-OPSD, dynamic Random-4, and STI to enable interpreting the relatively small differences between these methods.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

Response to All Reviewers

We thank the reviewers for their constructive feedback and clarify our contributions and limitations.

Motivation and contribution. Our goal is not to improve rubric quality by pruning criteria, but to study which parts of a fixed rubric must be exposed as privileged information to the frozen teacher in OPSD. A criterion may matter for evaluation yet have little influence on teacher predictions along current student trajectories. Thus, evaluation completeness and teacher-context influence are distinct. CritSel-OPSD preserves the original rubric, rubric-free student, and token-level OPSD objective; only the teacher-visible subset changes. Unlike rubric reward modeling or adaptive reward weighting, this is a teacher-context allocation problem.

What the results establish. Table 1 reports Avg-6 scores of 58.10 (Base), 59.44 (Full-OPSD), 59.57 (static Random-4), 59.70 (Original-Order Top-4), 59.77 (dynamic Random-4), and 60.05 (STI). Top-4 exposes ~53% fewer criteria than Full-OPSD (4 versus 8.53). Multiple compact strategies retain the observed performance of Full-OPSD. This, not STI's 0.28-point margin over Random-4, is our central finding. The K=2/4/6 sweep (59.71/60.05/59.59) also shows no monotonic benefit from exposing more criteria. We claim fewer teacher-visible criteria, not lower end-to-end training compute: attribution itself incurs overhead.

Role and limitations of STI. STI scores each criterion by the change in the frozen teacher's log-likelihood of realized student tokens under full-rubric versus leave-one-out contexts, evaluated on the same current-student trajectory. It measures local teacher sensitivity, not causal learning utility or a Shapley contribution. Figure 5 shows STI–TI-JSD Spearman correlations of 0.805–0.836 and Top-4 overlap of 0.838 (Jaccard 0.749), indicating that realized-token scoring largely tracks full-distribution teacher sensitivity. TI-JSD measures the full teacher-distribution change; SI-JSD instead measures changes in teacher–student discrepancy. TI–SI rank correlations of 0.125–0.187 confirm that these signals differ. We agree that high sensitivity need not imply higher student gains: the competitive Random-4 and bottom-ranked controls do not establish STI's superiority. LOO also omits interactions among jointly selected criteria; its advantage is a tractable, linear-in-criterion-count local diagnostic rather than complete attribution.

Dynamics and position effects. Figure 4 uses a fixed probe panel: Top-4 Jaccard similarity is approximately 0.79–0.83 between nearby student checkpoints, versus 0.731 between Steps 100 and 500. This supports student-state-dependent rankings, not a proven benefit of dynamic over static selection. Figure 3b does show position association: selection rates for criterion positions 1 and 11 are 81.33% and 19.52%, versus random-K expectations of 49.08% and 36.36% (Appendix Table 10). STI is not deterministic prefix truncation, but position and original content/order remain confounded.

Baselines, robustness, and hyperparameters. The three reported seeds (42, 1348, 4527) are decoding seeds for the same trained checkpoint, not independent training seeds; therefore, small selector differences have not been shown statistically reliable across training runs. We also agree that Base is not a matched rubric-free-teacher OPSD baseline: Full-OPSD versus Base does not isolate rubric-content effects from other conditioning effects. The evidence concerns one backbone and rubric source, not a causal decomposition or universal generalization. K=4 and H=100 were chosen by empirical hyperparameter tuning: the reported K=2/4/6 sweep yielded 59.71/60.05/59.59, and H=1/25/50/100/250 yielded 58.67/59.70/59.46/60.05/59.91 (Tables 3 and 5). These are observed best settings in the reported sweeps, not universally optimal settings or independently validated gains.

Other clarifications. Our training uses student-generated trajectories and token-level JSD distillation, not verifiable rewards; therefore OpenRubrics need not contain ground-truth verifiable labels for each training example. The downstream benchmark gains are observations, not proof of a particular transfer mechanism. We have not directly measured student satisfaction on unselected criteria, so cannot infer that those criteria are already mastered or never degraded.

In summary, our contribution is to formulate and characterize criterion-level privileged teacher-context allocation while holding the rubric and OPSD objective fixed. We agree that neither STI's superiority nor its correspondence to downstream learning utility has been established; the supported conclusion is that compact contexts can preserve Full-OPSD's observed benefits in this setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.