- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Rubric conditioned on policy self distillation uses a rubric as privileged information for the teacher. However, rubrics are typically treated as indivisible. This paper introduces CritSel-OPSD, a framework for constructing subsets of criteria to provide as privileged information using student trajectories. CritSel-OPSD is used to train a Qwen3-8B model and the observed benefits of full rubric conditioning is preserved while only using a subset of rubric criteria.
Critical strengths
- Related work organizes and represents literature well.
- Limitations are honest.
Critical weaknesses
- The motivation of dividing the rubric into criteria subsets is not clear to me, especially given the results are so close to Full-OPSD.
- It is not clear to me why the selection processes should be related to usefulness. I feel that the similar results of selection approaches show the proposed selection methods may not be correlated to usefulness.
- The impact of the proposed approach is not clear. Without independent runs it is difficult to determine if reported gains reflect a reproducible improvement.
Questions for authors
- Empirically how does the student score against the criteria that are NOT selected at refresh boundaries?
- Does figure 3b show that the position of the criteria in the rubric influences its selection rate?
- How were hyperparameters chosen? Were the best hyperparameters chosen from a separate validation set separate from the set of reported on in the main results?
General Response to All Reviewers
Response to All Reviewers
We thank the reviewers for their constructive feedback and clarify our contributions and limitations.
Motivation and contribution. Our goal is not to improve rubric quality by pruning criteria, but to study which parts of a fixed rubric must be exposed as privileged information to the frozen teacher in OPSD. A criterion may matter for evaluation yet have little influence on teacher predictions along current student trajectories. Thus, evaluation completeness and teacher-context influence are distinct. CritSel-OPSD preserves the original rubric, rubric-free student, and token-level OPSD objective; only the teacher-visible subset changes. Unlike rubric reward modeling or adaptive reward weighting, this is a teacher-context allocation problem.
What the results establish. Table 1 reports Avg-6 scores of 58.10 (Base), 59.44 (Full-OPSD), 59.57 (static Random-4), 59.70 (Original-Order Top-4), 59.77 (dynamic Random-4), and 60.05 (STI). Top-4 exposes ~53% fewer criteria than Full-OPSD (4 versus 8.53). Multiple compact strategies retain the observed performance of Full-OPSD. This, not STI's 0.28-point margin over Random-4, is our central finding. The K=2/4/6 sweep (59.71/60.05/59.59) also shows no monotonic benefit from exposing more criteria. We claim fewer teacher-visible criteria, not lower end-to-end training compute: attribution itself incurs overhead.
Role and limitations of STI. STI scores each criterion by the change in the frozen teacher's log-likelihood of realized student tokens under full-rubric versus leave-one-out contexts, evaluated on the same current-student trajectory. It measures local teacher sensitivity, not causal learning utility or a Shapley contribution. Figure 5 shows STI–TI-JSD Spearman correlations of 0.805–0.836 and Top-4 overlap of 0.838 (Jaccard 0.749), indicating that realized-token scoring largely tracks full-distribution teacher sensitivity. TI-JSD measures the full teacher-distribution change; SI-JSD instead measures changes in teacher–student discrepancy. TI–SI rank correlations of 0.125–0.187 confirm that these signals differ. We agree that high sensitivity need not imply higher student gains: the competitive Random-4 and bottom-ranked controls do not establish STI's superiority. LOO also omits interactions among jointly selected criteria; its advantage is a tractable, linear-in-criterion-count local diagnostic rather than complete attribution.
Dynamics and position effects. Figure 4 uses a fixed probe panel: Top-4 Jaccard similarity is approximately 0.79–0.83 between nearby student checkpoints, versus 0.731 between Steps 100 and 500. This supports student-state-dependent rankings, not a proven benefit of dynamic over static selection. Figure 3b does show position association: selection rates for criterion positions 1 and 11 are 81.33% and 19.52%, versus random-K expectations of 49.08% and 36.36% (Appendix Table 10). STI is not deterministic prefix truncation, but position and original content/order remain confounded.
Baselines, robustness, and hyperparameters. The three reported seeds (42, 1348, 4527) are decoding seeds for the same trained checkpoint, not independent training seeds; therefore, small selector differences have not been shown statistically reliable across training runs. We also agree that Base is not a matched rubric-free-teacher OPSD baseline: Full-OPSD versus Base does not isolate rubric-content effects from other conditioning effects. The evidence concerns one backbone and rubric source, not a causal decomposition or universal generalization. K=4 and H=100 were chosen by empirical hyperparameter tuning: the reported K=2/4/6 sweep yielded 59.71/60.05/59.59, and H=1/25/50/100/250 yielded 58.67/59.70/59.46/60.05/59.91 (Tables 3 and 5). These are observed best settings in the reported sweeps, not universally optimal settings or independently validated gains.
Other clarifications. Our training uses student-generated trajectories and token-level JSD distillation, not verifiable rewards; therefore OpenRubrics need not contain ground-truth verifiable labels for each training example. The downstream benchmark gains are observations, not proof of a particular transfer mechanism. We have not directly measured student satisfaction on unselected criteria, so cannot infer that those criteria are already mastered or never degraded.
In summary, our contribution is to formulate and characterize criterion-level privileged teacher-context allocation while holding the rubric and OPSD objective fixed. We agree that neither STI's superiority nor its correspondence to downstream learning utility has been established; the supported conclusion is that compact contexts can preserve Full-OPSD's observed benefits in this setting.