acceptodds
Under review as a conference paper at ICLR 2027

Beyond Static Criteria: Model–Rubric Co-Evolution for Reinforcement Learning with Self-Distillation

Abstract

Recently, on-policy self-distillation (OPSD) and its reinforcement-learning extensions have offered a promising route to improving large language models. By conditioning the teacher on additional privileged information, these methods provide token-level guidance along the student's own sampled trajectories. However, the effectiveness of these methods depends on predefined privileged teacher information, giving rise to two key challenges. First, teacher contexts are often constructed independently of each sampled rollout, making them unable to adapt to its specific errors or unmet behaviors. Second, the teacher information is typically fixed before training and cannot evolve with the improving capabilities of the policy. To address these limitations, we propose **M**odel–**R**ubric Co-Evolution for **R**einforcement **L**earning with **S**elf-**D**istillation (MR.RLSD), an integrated reinforcement-learning and self-distillation framework centered on co-evolving rubrics. MR.RLSD uses rubrics as a shared interface between reward feedback and teacher guidance. Rubric scores determine policy-update directions, while criteria unmet by each response are transformed online into teacher contexts that provide contrastive feedback for modulating token-level update magnitudes, without requiring pre-constructed privileged contexts. During training, a shared policy serves as both a response generator and a rubric generator, proposing new criteria based on current rollouts and historical rubrics. The updated criteria jointly refine reward computation and teacher guidance, allowing both the optimization targets and self-distillation feedback to adapt to the evolving policy. Experiments across health, writing, and science show that MR.RLSD outperforms the evaluated reinforcement learning and self-distillation baselines, demonstrating the effectiveness of co-evolving the policy and its guiding criteria for open-ended generation.

Primary area:
reinforcement learning
Keywords:
Rubrics, Self-evolving, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ajN9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MR.RLSD trains a model to improve both its answers and the criteria used to judge them. New criteria guide the reward and the teacher feedback. For each answer, the teacher focuses on criteria it failed, using corrective and opposite instructions to adjust the update strength for individual tokens. The paper reports improvements across health, writing, and science tasks.

Critical strengths
  1. The teacher’s guidance targets what each answer gets wrong. This makes sense for open-ended tasks without a single correct answer. Randomly choosing criteria hurts performance, while updating criteria only for teacher guidance still helps. These tests support a benefit beyond adding more reward criteria.

  2. The comparisons cover relevant baselines and several domains. MR.RLSD leads on most benchmarks for both tested models, with modest average gains over EvoRubric. Freezing the rubric generator hurts the health-domain results, and the ResearchQA advantage holds with different judges. These are useful checks.

Critical weaknesses
  1. The gains are modest, but the paper does not report variation across independent training runs. Repeating benchmark evaluations does not answer whether retraining would give similar gains. The paper lists hardware and training settings, but not total costs, including rubric generation, external judging, and teacher scoring. This makes it hard to judge the extra benefit relative to the extra computation.

  2. Table 2 shows that contrastive guidance helps, but does not establish why. Changes in the average and spread of the likelihood signal do not directly show that stylistic effects were removed. The explanation should be tested more directly or stated more cautiously.

  3. The quality of the evolving rubrics needs more direct validation. Section 4.6 shows that generated rubrics improve answers according to the same model used to verify rubrics and provide training judgments. This is useful evidence, but it does not independently establish that the new criteria are correct and relevant. The main benchmarks use other judges, and the ResearchQA cross-judge check is reassuring; neither directly audits the generated criteria. Since these criteria control both rewards and teacher guidance, a sample-level audit of their validity and novelty would help rule out shared evaluator preferences.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ajN9(1/2)

W1. Variability across independent training runs and computational cost

We thank the reviewer for highlighting both training variability and computational cost.

Independent training runs. We agree that repeated evaluations of a single trained model do not capture variability across training runs. We therefore conducted two independent training runs with different random seeds for each of RLCSD, EvoRubric, and MR.RLSD, and evaluated the resulting models on ResearchQA and GPQA-D. The table below reports the mean and standard deviation across the two training seeds for each method.

MethodResearchQA (mean ± std)GPQA-D (mean ± std)
RLCSD79.07 ± 0.61553.54 ± 0
EvoRubric80.00 ± 0.1453.78 ± 0.255
MR.RLSD81.29 ± 0.13556.57 ± 0.49

These results quantify variability across independently trained checkpoints. We will distinguish this training variability from repeated-evaluation variability in the revised manuscript.

Training cost. We additionally measured the average wall-clock time per training step for MR.RLSD and Evorubric , using 184 logged steps from each run on the same policy-worker hardware.

MethodAverage time per training step (s)Relative step time
Evorubric699.451x
MR.RLSD778.021.11x

The measured step time covers the core training pipeline: response generation, rubric generation, external judging and verification, student and teacher scoring, and actor updates. The observed increase in average step time is approximately 11%, indicating a moderate per-step runtime overhead.

We will include the independent-seed results and timing comparison in the revised version to make both training variability and the runtime–performance trade-off clearer.

W2. Interpretation of the contrastive guidance

We thank the reviewer for pointing out this distinction. We agree that changes in the mean and standard deviation of the likelihood signal do not directly establish that stylistic effects have been removed. We will revise the interpretation in Section 4.5 accordingly.

Our motivation follows RLCSD, which observes that privileged conditioning can introduce both task-related changes and stylistic shifts. With matched prompt templates, contrasting teacher likelihoods may attenuate stylistic components shared across the conditioning contexts. RLCSD supports this rationale through its task/style-token analysis on mathematical reasoning (Sections 3.1–3.4, Figures 2 and 4). These findings motivate our construction, but do not establish the same mechanism in our open-ended setting.

To illustrate the signal change, we performed an exploratory offline analysis using base Qwen3-8B on a fixed ResearchQA response about weather-based irrigation scheduling. We scored identical response tokens under one-sided and contrastive contexts. The table lists the 3 positions with the largest absolute raw likelihood discrepancy, retaining the signed values.

Rankw/o contrast: tokenSigned discrepancyContrastive: tokenSigned discrepancy
1In−19.5134 Feedback−12.0100
2Example−13.0450Example−10.2704
3 Automation−12.6615 Ev−6.9446

Here, In occurs in the summary marker “In summary,” and Ev is a subword token of “Evapotranspiration.” The summary marker’s initial token moves from rank 1 to 208, with its absolute discrepancy decreasing from 19.5134 to 0.3953. This illustrates a local reduction in the signal assigned to a summary marker.

Separately, our training ablation in Table 2 supports the practical benefit of the contrastive construction: removing it reduces HealthBench performance from 51.76 to 49.18. This result supports its usefulness under the tested setting.

We will include the token-level illustration in the supplementary analysis and revise Section 4.5 to distinguish the design rationale, descriptive signal analysis, and downstream performance evidence. In particular, we will remove the implication that the signal statistics directly demonstrate the suppression of stylistic effects.

AuthorsOpenReview ↗

Response to Reviewer ajN9(2/2)

W3. Independent assessment of generated rubric quality

We agree that evaluation by the training judge alone does not directly establish the validity and novelty of generated criteria. To address this concern, we conducted a criterion-level audit of generated rubrics retained in the step-160 pools, using the same 80 sampled ResearchQA training queries for MR.RLSD, Evorubric and vanilla Qwen3-8B;

GPT-4.1 and Claude Opus 4.6 assessed each criterion separately. Both auditors differ from GPT-OSS-120B, which provides training-time judgments and verification. The generating method and the training verifier’s decisions were hidden from the auditors.

The audit assessed two dimensions:

  • Validity: whether the criterion is relevant to the question, scientifically defensible, and checkable in an answer.
  • Novelty relative to the original rubrics: whether it introduces a new evaluation dimension, substantively refines an existing requirement, or duplicates an existing criterion. Superficial wording changes do not count as novelty.
GeneratorBoth judged validBoth judged valid and novel or materially refinedBoth judged valid and a new dimension
MR.RLSD86.8%84.2%73.7%
Evorubric75.0%73.2%55.4%
Frozen Qwen3‑8B74.5%67.4%39.9%

As shown in the table above, our method generates rubrics with higher validity and novelty rates, as well as a higher proportion of criteria introducing new evaluation dimensions, compared with the initial model and EvoRubric. These results provide quantitative evidence that our method generates higher-quality rubrics. We will incorporate these supplementary results into the main text in the revised version.

To complement the quantitative rubric-quality evaluation above, we provide a concrete example showing how a generated rubric introduces an additional evaluation dimension and identifies a relevant omission in the response.

Query: “How does ultraviolet (UV) radiation molecularly disrupt skin cell function and contribute to the process of photoaging?”

The original rubrics cover DNA damage, oxidative stress, extracellular-matrix degradation, inflammatory signaling, and their connection to visible photoaging.

Generated rubric (verbatim)What this criterion evaluates
Epoch 1: Discusses the relationship between chronic UV exposure and cellular senescence, including its contribution to photoaging and tissue dysfunction.Explicitly checks whether the response explains cellular senescence and its contribution to photoaging, complementing the original criteria on molecular damage, inflammation, and matrix degradation.
Epoch 4: Uses quantitative details (e.g., specific wavelengths of UV, enzyme activity levels, ROS concentrations) and cites relevant sources when available to support molecular mechanisms, without fabricating data or overgeneralizing.Checks the appropriate use of quantitative information and supporting sources when available, with an emphasis on evidence-grounded explanations of molecular mechanisms.

A concrete omission identified by the retained criterion. A response to this query provides a detailed explanation of the original rubric dimensions. Its concluding summary states:

Ultraviolet radiation disrupts skin cell function and contributes to photoaging through a complex interplay of:

  • DNA damage
  • Oxidative stress
  • Inflammation
  • Matrix degradation
  • Impaired stem cell function
  • Disrupted signaling pathways

According to the logged assessments, this response satisfies all four original positive criteria and triggers none of the four original negative criteria. However, it does not explain cellular senescence or its contribution to photoaging, and the generated senescence criterion is marked as not satisfied.

Thus, the generated criterion identifies a specific mechanistic omission that the original rubric set does not explicitly check. The additional requirement concerns the explanation of cellular senescence, rather than merely mentioning skin aging in general.

We will include this example, the corresponding response excerpt, and the logged assessments in the revised manuscript to illustrate how generated rubrics complement the original evaluation criteria and reveal additional response deficiencies.

I thank the authors for extending their experiments, which solves most of my concerns. I will raise my score by 1 point.

Thank you for your constructive suggestions and for your positive reassessment of our work. Your feedback has helped us strengthen the experimental validation and clarify the interpretation of our results. We will incorporate the additional experiments, analyses, and corresponding revisions presented in the rebuttal into the revised version.

Reviewer_25duOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MR.RLSD for open-ended generation tasks, combining rubric-based reinforcement learning with self-distillation. The method uses a shared policy to generate responses and new rubrics, continually updates the rubric pool based on current rollouts and historical rubrics, and uses an external model to score responses and verify new rubrics. Its main contribution is to use evolved rubrics for both reward computation and teacher guidance tailored to deficiencies in individual responses. Across different models and benchmarks, the proposed method improves over the compared methods, and the experiments largely support the stated design choices and claims.

Critical strengths

Turning evolved rubrics into teacher guidance tailored to response-specific deficiencies helps improve OPSD, and the experiments support this. To address the difficulty of capturing specific response deficiencies with a single reward in open-ended tasks, the authors use the criteria unmet by each response to construct teacher contexts for self-distillation. The experiments largely support the authors' claims: MR.RLSD improves over multiple baselines, and selecting failed rubrics also outperforms random rubric selection. Figure 4 also shows that newly generated rubrics can still improve RLCSD when used only to construct teacher contexts and excluded from reward computation, supporting the value of rubric evolution for teacher guidance. Together, these results support the design choice of using specific evaluation feedback to provide targeted teacher guidance. The paper is generally clear, and the ablation studies also largely support the main design choices.

Critical weaknesses

I do not fully understand the rationale here. Training already uses GPT-OSS-120B to score responses and verify new rubrics, which already consumes inference resources. However, token-level guidance comes from the policy's own teacher branch. The paper does not sufficiently explain why it converts the external model's evaluation feedback into teacher contexts for self-distillation rather than using the external model directly to provide token-level teacher signals.

Questions for authors
  1. Given that GPT-OSS-120B is already used during training to score responses and verify rubrics, why convert its evaluation feedback into privileged contexts and then obtain token-level guidance from the policy's own teacher branch, rather than using GPT-OSS-120B directly to provide token-level teacher signals (more akin to OPD)?
  2. I am somewhat confused about what is counted in Equations (8) and (10). Equation (8) seems to aggregate the rubrics that pass filtering across all candidate rubric sets, whereas Equation (10) computes a reward for an individual candidate set. Does the numerator count all rubrics that pass filtering across all candidate sets, or only those in the candidate set currently being scored? Please clarify the relationship between the two definitions.
  3. Figure 1 helps explain the basic idea of the method, but I would still like to understand the specific content of newly added rubrics during actual training and how they complement the existing rubrics. Could you provide one or two examples from existing training records, showing the original query, existing rubrics, newly added rubrics at different stages, and the corresponding responses, to help readers understand rubric evolution?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 25du(1/2)

Q1. Why use self-distillation when GPT‑OSS‑120B is already available?

We thank the reviewer for raising this question. We would like to clarify the role of GPT-OSS-120B and the motivation for using self-distillation.

External judging is a necessary component of rubric-based reward computation, as in prior methods such as RaR and EvoRubric. In our framework, the judge scores responses against rubrics and verifies newly generated criteria. These evaluation functions remain necessary even without self-distillation. We use the resulting rubric-level feedback to construct teacher contexts, while the policy’s own teacher branch provides token-level guidance.

Our choice of GPT-OSS-120B reflects resource constraints rather than a requirement of the method. We selected it as a practical, locally deployable evaluator for our experiments. The framework can also accommodate other evaluators, including potentially more capable proprietary models accessed through APIs. Our method requires their rubric-level judgments and verification decisions; the policy’s own teacher branch converts this feedback into token-level guidance.

This separation matters when a closed-source evaluator’s API provides evaluation outputs but does not expose teacher-forced likelihoods for arbitrary student-generated trajectories. In that setting, the direct token-level OPD procedure evaluated here cannot be applied, whereas our rubric-conditioned self-distillation remains applicable. Thus, improvements in the judge can be incorporated without requiring it to provide token probabilities or perform cross-model token-level teaching.

We nevertheless evaluated direct OPD because likelihood access is available for our locally deployed GPT-OSS-120B. We used it as an external teacher and applied DPCA for cross-tokenizer alignment.

MethodResearchQAGPQA-DTime Cost
GPT-OSS-120B OPD (DPCA)81.8850.9049 hours
MR.RLSD81.1657.0739 hours

Although direct OPD achieves a slightly higher ResearchQA score than MR.RLSD (81.88 vs. 81.16), its GPQA-D score is substantially lower (50.90 vs. 57.07), a difference of 6.17 percentage points. Moreover, this baseline requires teacher-forced likelihood scoring of student trajectories using GPT-OSS-120B, and it also incurs additional resource costs. Considering these mixed performance results and the additional teacher-scoring requirements, direct OPD does not demonstrate a clear overall advantage over MR.RLSD under the evaluated configurations.

Q2:Relationship between Eq. (8) and Eq. (10)

We thank the reviewer for identifying this notation issue. We apologize for this oversight. The numerator in the original Eq. (10) incorrectly refers to a query-level aggregate, whereas the reward should be computed separately for each candidate rubric set.

For each candidate set Rm\mathcal R_m, let Vm\mathcal V_m denote the subset of rubrics that pass both verifier filtering and discriminativeness filtering on the current rollout group Yx\mathcal Y_x. For a nonempty candidate set, the corrected Eq. (10) is

Rrub(Rm)=num⁡(Vm)num⁡(Rm)−pverify(Rm)−psize(Rm)−plen(Rm).R_{\mathrm{rub}}(\mathcal{R}_m) = \frac{\operatorname{num}(\mathcal{V}_m)}{\operatorname{num}(\mathcal{R}_m)} - p_{\mathrm{verify}}(\mathcal{R}_m) - p_{\mathrm{size}}(\mathcal{R}_m) - p_{\mathrm{len}}(\mathcal{R}_m).

Here, num⁡(⋅)\operatorname{num}(\cdot) counts rubric criteria. The numerator counts criteria from the current candidate set that pass both filtering stages, while the denominator counts all criteria generated in that same set before filtering. Equation (8), in contrast, aggregates the retained rubrics across candidate sets to update the rubric pool.

Our implementation already uses this candidate-level calculation after both filtering stages. We will correct the numerator in Eq. (10) and clarify these definitions in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 25du(2/2)

Q3. Examples of generated rubrics at different training steps

We thank the reviewer for this suggestion. Below, we present actual rubric-generation outputs from two training epochs for the same ResearchQA query, together with a recorded response illustrating how a retained generated criterion identifies an omission beyond the original rubric coverage.

Query: “How does ultraviolet (UV) radiation molecularly disrupt skin cell function and contribute to the process of photoaging?”

The original rubrics cover DNA damage, oxidative stress, extracellular-matrix degradation, inflammatory signaling, and their connection to visible photoaging.

Generated rubric (verbatim)What this criterion evaluates
Epoch 1: Discusses the relationship between chronic UV exposure and cellular senescence, including its contribution to photoaging and tissue dysfunction.Explicitly checks whether the response explains cellular senescence and its contribution to photoaging, complementing the original criteria on molecular damage, inflammation, and matrix degradation.
Epoch 4: Uses quantitative details (e.g., specific wavelengths of UV, enzyme activity levels, ROS concentrations) and cites relevant sources when available to support molecular mechanisms, without fabricating data or overgeneralizing.Checks the appropriate use of quantitative information and supporting sources when available, with an emphasis on evidence-grounded explanations of molecular mechanisms.

A concrete omission identified by the retained criterion. A response to this query provides a detailed explanation of the original rubric dimensions. Its concluding summary states:

Ultraviolet radiation disrupts skin cell function and contributes to photoaging through a complex interplay of:

  • DNA damage
  • Oxidative stress
  • Inflammation
  • Matrix degradation
  • Impaired stem cell function
  • Disrupted signaling pathways

According to the logged assessments, this response satisfies all four original positive criteria and triggers none of the four original negative criteria. However, it does not explain cellular senescence or its contribution to photoaging, and the generated senescence criterion is marked as not satisfied.

Thus, the generated criterion identifies a specific mechanistic omission that the original rubric set does not explicitly check. The additional requirement concerns the explanation of cellular senescence, rather than merely mentioning skin aging in general.

We will include this example, the corresponding response excerpt, and the logged assessments in the revised manuscript to illustrate how generated rubrics complement the original evaluation criteria and reveal additional response deficiencies.

Reviewer_25duOpenReview ↗

Response to authors

The authors have largely addressed my concerns, and I maintain my original rating.

Thank you for your thoughtful review and valuable feedback. Your suggestions have helped us clarify the motivation and presentation of our method. We will incorporate the corresponding clarifications, additional results, and revisions discussed in the rebuttal into the revised version.

Reviewer_UwoSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MR. RLSD, which combines evolving rubrics with reinforcement learning and self-distillation. The model generates new rubrics based on its own responses and uses them to compute rewards. It also turns failed rubrics into teacher guidance to adjust token-level updates. In this way, both the reward and teacher guidance can evolve as the model improves.

Critical strengths
  1. The paper addresses a meaningful limitation of existing reinforcement learning and self-distillation methods by exploring how teacher guidance can evolve as the policy improves.

  2. The idea of converting failed rubrics into contrastive teacher contexts is intuitive and provides a reasonable way to obtain response-specific token-level guidance without relying on predefined contexts.

  3. The proposed method consistently improves average performance over existing methods across six benchmarks using two Qwen models.

Critical weaknesses
  1. The paper does not sufficiently justify why self-distillation is necessary in addition to evolving rubric-based RL. For example, it lacks a direct ablation that removes the self-distillation branch while keeping the same evolved rubrics and reward computation. The comparison with EvoRubric is encouraging but does not fully isolate the contribution of self-distillation. Moreover, Equation 3 and Equation 6 imply a multiplier of only [0.97, 1.03] under the initial hyperparameter settings, raising questions about whether such small token-level adjustments meaningfully contribute to performance gains.

  2. Since GPT-OSS-120B is already used as an external judge and verifier, the framework already relies on a larger and more powerful model, introducing additional computational costs. Why not consider using it as a stronger teacher instead of self-distillation?

  3. The notation in Equation 10 is potentially inconsistent. The numerator num⁡(R^x)\operatorname{num}(\hat{R}_x) appears to count the rubrics retained from all candidate sets, whereas the denominator num⁡(Rm)\operatorname{num}(R_m) refers to an individual candidate set. If the reward is intended to measure the acceptance rate of each candidate set, should the numerator instead be num⁡(R^m)\operatorname{num}(\hat{R}_m)? The current definition makes the rubricator reward difficult to interpret.

  4. The presentation is somewhat redundant relative to the main idea. Figure 1, Figure 2, and Algorithm 1 overlap in explaining the training process. Moreover, Figure 2 visually suggests a direct multiplication between the sequence-level advantage and the contrastive token signal, but does not clearly explain the multiplication symbol. As I understand it, this operation corresponds to Equations 5 and 6. It would be helpful to clarify this operation in the figure or its caption.

In conclusion, the method is reasonably novel but still raises some concerns (see Weaknesses), so my current rating is Weak Accept.

Questions for authors
  1. Have the authors evaluated the quality of generated rubrics more quantitatively?

  2. Can the authors clarify the numerator in Equation 10?

  3. How do the benefits of self-distillation compare with its additional computational cost?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Uwos(1/3)

W1. Contribution of self-distillation and the magnitude of token-level guidance

We thank the reviewer for suggesting a more controlled comparison. Removing self-distillation leaves an evolving-rubric RL pipeline conceptually similar to EvoRubric. However, comparison with EvoRubric alone does not fully isolate the contribution of self-distillation within our framework. We therefore added a matched w/o SD ablation, retaining our rubric-evolution procedure and reward definition while disabling the self-distillation branch. The base model, training data, and number of training updates are kept unchanged.

MethodResearchQAGPQA-D
MR.RLSD w/o SD80.0152.53
MR.RLSD81.1657.07

Removing self-distillation reduces ResearchQA performance by 1.15 percentage points and GPQA-D performance by 4.54 percentage points, providing direct evidence of its contribution under the tested settings.

Regarding the small signal magnitude, we additionally examined the empirical advantage correction. In training process of Healthbench, the mean absolute correction among gate-selected response tokens was 0.02371. We are also measuring the corresponding quantity in a reproduced RLCSD setting:

MethodMean absolute advantage correction among gate-selected tokens
MR.RLSD0.02371
RLCSD0.0309

This comparison provides context for the scale of token-level corrections. More directly, the matched ablation shows that the bounded guidance contributes to downstream performance: a small correction magnitude does not imply a negligible empirical contribution.

We will include the direct ablation and the finalized correction-magnitude analysis in the revised manuscript.

W2. Why use self-distillation when GPT‑OSS‑120B is already available?

We thank the reviewer for raising this question. We would like to clarify the role of GPT-OSS-120B and the motivation for using self-distillation.

External judging is a necessary component of rubric-based reward computation, as in prior methods such as RaR and EvoRubric. In our framework, the judge scores responses against rubrics and verifies newly generated criteria. These evaluation functions remain necessary even without self-distillation. We use the resulting rubric-level feedback to construct teacher contexts, while the policy’s own teacher branch provides token-level guidance.

Our choice of GPT-OSS-120B reflects resource constraints rather than a requirement of the method. We selected it as a practical, locally deployable evaluator for our experiments. The framework can also accommodate other evaluators, including potentially more capable proprietary models accessed through APIs. Our method requires their rubric-level judgments and verification decisions; the policy’s own teacher branch converts this feedback into token-level guidance.

This separation matters when a closed-source evaluator’s API provides evaluation outputs but does not expose teacher-forced likelihoods for arbitrary student-generated trajectories. In that setting, the direct token-level OPD procedure evaluated here cannot be applied, whereas our rubric-conditioned self-distillation remains applicable. Thus, improvements in the judge can be incorporated without requiring it to provide token probabilities or perform cross-model token-level teaching.

We nevertheless evaluated direct OPD because likelihood access is available for our locally deployed GPT-OSS-120B. We used it as an external teacher and applied DPCA for cross-tokenizer alignment.

MethodResearchQAGPQA-DTime Cost
GPT-OSS-120B OPD (DPCA)81.8850.9049 hours
MR.RLSD81.1657.0739 hours

Although direct OPD achieves a slightly higher ResearchQA score than MR.RLSD (81.88 vs. 81.16), its GPQA-D score is substantially lower (50.90 vs. 57.07), a difference of 6.17 percentage points. Moreover, this baseline requires teacher-forced likelihood scoring of student trajectories using GPT-OSS-120B, and it also incurs additional resource costs. Considering these mixed performance results and the additional teacher-scoring requirements, direct OPD does not demonstrate a clear overall advantage over MR.RLSD under the evaluated configurations.

AuthorsOpenReview ↗

Response to Reviewer Uwos(2/3)

W4: Figures 1–2 and Algorithm 1

We thank the reviewer for these presentation suggestions. We apologize for the redundant explanations and the unclear depiction of token-level advantage modulation in Fig. 2. We will streamline the presentation so that Fig. 1 illustrates the motivation through a concrete example, Fig. 2 presents the high-level training pipeline, and Algorithm 1 provides the detailed procedure. We will also remove repeated descriptions from the accompanying text.

In Fig. 2, we will replace the multiplication symbol with an explicit token-level advantage modulation block, showing the sequence-level advantage and contrastive token signal as inputs and the resulting per-token advantages as output. The revised caption will explain that the reward-derived advantage anchors the update direction, while the contrastive signal adjusts the magnitude at individual token positions. We will also clarify this operation’s correspondence to Eqs. (5)–(6).

Q1: Quantitative rubric quality

We quantitatively assessed the quality of MR.RLSD-generated rubrics against two baselines: the Evorubric and direct generation from the initial Qwen3-8B model. On 80 sampled ResearchQA training queries, GPT-4.1 and Claude Opus 4.6 evaluated rubric validity and novelty relative to the original criteria, blinded to the generating method.

GeneratorBoth judged validBoth judged valid and novel or materially refinedBoth judged valid and a new dimension
MR.RLSD86.8%84.2%73.7%
Evorubric75.0%73.2%55.4%
Frozen Qwen3‑8B74.5%67.4%39.9%

As shown in the table above, our method generates rubrics with higher validity and novelty rates, as well as a higher proportion of criteria introducing new evaluation dimensions, compared with the initial model and EvoRubric. These results provide quantitative evidence that our method generates higher-quality rubrics. We will incorporate these supplementary results into the main text in the revised version.

W3 and Q2: The numerator in Eq. (10)

We thank the reviewer for identifying this notation issue. We apologize for this oversight. The numerator in the original Eq. (10) incorrectly refers to a query-level aggregate, whereas the reward should be computed separately for each candidate rubric set.

For each candidate set Rm\mathcal R_m, let Vm\mathcal V_m denote the subset of rubrics that pass both verifier filtering and discriminativeness filtering on the current rollout group Yx\mathcal Y_x. For a nonempty candidate set, the corrected Eq. (10) is

Rrub(Rm)=num⁡(Vm)num⁡(Rm)−pverify(Rm)−psize(Rm)−plen(Rm).R_{\mathrm{rub}}(\mathcal{R}_m) = \frac{\operatorname{num}(\mathcal{V}_m)}{\operatorname{num}(\mathcal{R}_m)} - p_{\mathrm{verify}}(\mathcal{R}_m) - p_{\mathrm{size}}(\mathcal{R}_m) - p_{\mathrm{len}}(\mathcal{R}_m).

Here, num⁡(⋅)\operatorname{num}(\cdot) counts rubric criteria. The numerator counts criteria from the current candidate set that pass both filtering stages, while the denominator counts all criteria generated in that same set before filtering. Equation (8), in contrast, aggregates the retained rubrics across candidate sets to update the rubric pool.

Our implementation already uses this candidate-level calculation after both filtering stages. We will correct the numerator in Eq. (10) and clarify these definitions in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Uwos(3/3)

Q3: Benefit relative to the cost of self-distillation

The self-distillation branch incurs a modest time cost. Averaged over 184 logged steps of our Qwen3-8B ResearchQA run, it takes 41.20 seconds per step, accounting for 5.30% of the total training-step time. This includes teacher-context construction, teacher scoring, and token-level advantage modulation.

Training stageTime per step (s)Share
Response generation225.9629.04%
Rubric generation18.732.41%
Reward computation, including judging and verification286.6336.84%
Student log-probability computation40.715.23%
Self-distillation41.205.30%
Actor update158.4820.37%
Other within-step operations6.310.81%
Total training step778.02100.00%

Periodic validation and checkpoint saving are excluded.

We will include this timing breakdown in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.