- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MR.RLSD trains a model to improve both its answers and the criteria used to judge them. New criteria guide the reward and the teacher feedback. For each answer, the teacher focuses on criteria it failed, using corrective and opposite instructions to adjust the update strength for individual tokens. The paper reports improvements across health, writing, and science tasks.
Critical strengths
-
The teacher’s guidance targets what each answer gets wrong. This makes sense for open-ended tasks without a single correct answer. Randomly choosing criteria hurts performance, while updating criteria only for teacher guidance still helps. These tests support a benefit beyond adding more reward criteria.
-
The comparisons cover relevant baselines and several domains. MR.RLSD leads on most benchmarks for both tested models, with modest average gains over EvoRubric. Freezing the rubric generator hurts the health-domain results, and the ResearchQA advantage holds with different judges. These are useful checks.
Critical weaknesses
-
The gains are modest, but the paper does not report variation across independent training runs. Repeating benchmark evaluations does not answer whether retraining would give similar gains. The paper lists hardware and training settings, but not total costs, including rubric generation, external judging, and teacher scoring. This makes it hard to judge the extra benefit relative to the extra computation.
-
Table 2 shows that contrastive guidance helps, but does not establish why. Changes in the average and spread of the likelihood signal do not directly show that stylistic effects were removed. The explanation should be tested more directly or stated more cautiously.
-
The quality of the evolving rubrics needs more direct validation. Section 4.6 shows that generated rubrics improve answers according to the same model used to verify rubrics and provide training judgments. This is useful evidence, but it does not independently establish that the new criteria are correct and relevant. The main benchmarks use other judges, and the ResearchQA cross-judge check is reassuring; neither directly audits the generated criteria. Since these criteria control both rewards and teacher guidance, a sample-level audit of their validity and novelty would help rule out shared evaluator preferences.
Questions for authors
See weaknesses.
4 comments
Response to Reviewer ajN9(2/2)
W3. Independent assessment of generated rubric quality
We agree that evaluation by the training judge alone does not directly establish the validity and novelty of generated criteria. To address this concern, we conducted a criterion-level audit of generated rubrics retained in the step-160 pools, using the same 80 sampled ResearchQA training queries for MR.RLSD, Evorubric and vanilla Qwen3-8B;
GPT-4.1 and Claude Opus 4.6 assessed each criterion separately. Both auditors differ from GPT-OSS-120B, which provides training-time judgments and verification. The generating method and the training verifier’s decisions were hidden from the auditors.
The audit assessed two dimensions:
- Validity: whether the criterion is relevant to the question, scientifically defensible, and checkable in an answer.
- Novelty relative to the original rubrics: whether it introduces a new evaluation dimension, substantively refines an existing requirement, or duplicates an existing criterion. Superficial wording changes do not count as novelty.
| Generator | Both judged valid | Both judged valid and novel or materially refined | Both judged valid and a new dimension |
|---|---|---|---|
| MR.RLSD | 86.8% | 84.2% | 73.7% |
| Evorubric | 75.0% | 73.2% | 55.4% |
| Frozen Qwen3‑8B | 74.5% | 67.4% | 39.9% |
As shown in the table above, our method generates rubrics with higher validity and novelty rates, as well as a higher proportion of criteria introducing new evaluation dimensions, compared with the initial model and EvoRubric. These results provide quantitative evidence that our method generates higher-quality rubrics. We will incorporate these supplementary results into the main text in the revised version.
To complement the quantitative rubric-quality evaluation above, we provide a concrete example showing how a generated rubric introduces an additional evaluation dimension and identifies a relevant omission in the response.
Query: “How does ultraviolet (UV) radiation molecularly disrupt skin cell function and contribute to the process of photoaging?”
The original rubrics cover DNA damage, oxidative stress, extracellular-matrix degradation, inflammatory signaling, and their connection to visible photoaging.
| Generated rubric (verbatim) | What this criterion evaluates |
|---|---|
| Epoch 1: Discusses the relationship between chronic UV exposure and cellular senescence, including its contribution to photoaging and tissue dysfunction. | Explicitly checks whether the response explains cellular senescence and its contribution to photoaging, complementing the original criteria on molecular damage, inflammation, and matrix degradation. |
| Epoch 4: Uses quantitative details (e.g., specific wavelengths of UV, enzyme activity levels, ROS concentrations) and cites relevant sources when available to support molecular mechanisms, without fabricating data or overgeneralizing. | Checks the appropriate use of quantitative information and supporting sources when available, with an emphasis on evidence-grounded explanations of molecular mechanisms. |
A concrete omission identified by the retained criterion. A response to this query provides a detailed explanation of the original rubric dimensions. Its concluding summary states:
Ultraviolet radiation disrupts skin cell function and contributes to photoaging through a complex interplay of:
- DNA damage
- Oxidative stress
- Inflammation
- Matrix degradation
- Impaired stem cell function
- Disrupted signaling pathways
According to the logged assessments, this response satisfies all four original positive criteria and triggers none of the four original negative criteria. However, it does not explain cellular senescence or its contribution to photoaging, and the generated senescence criterion is marked as not satisfied.
Thus, the generated criterion identifies a specific mechanistic omission that the original rubric set does not explicitly check. The additional requirement concerns the explanation of cellular senescence, rather than merely mentioning skin aging in general.
We will include this example, the corresponding response excerpt, and the logged assessments in the revised manuscript to illustrate how generated rubrics complement the original evaluation criteria and reveal additional response deficiencies.
I thank the authors for extending their experiments, which solves most of my concerns. I will raise my score by 1 point.
Thank you for your constructive suggestions and for your positive reassessment of our work. Your feedback has helped us strengthen the experimental validation and clarify the interpretation of our results. We will incorporate the additional experiments, analyses, and corresponding revisions presented in the rebuttal into the revised version.
Response to Reviewer ajN9(1/2)
We thank the reviewer for highlighting both training variability and computational cost.
Independent training runs. We agree that repeated evaluations of a single trained model do not capture variability across training runs. We therefore conducted two independent training runs with different random seeds for each of RLCSD, EvoRubric, and MR.RLSD, and evaluated the resulting models on ResearchQA and GPQA-D. The table below reports the mean and standard deviation across the two training seeds for each method.
These results quantify variability across independently trained checkpoints. We will distinguish this training variability from repeated-evaluation variability in the revised manuscript.
Training cost. We additionally measured the average wall-clock time per training step for MR.RLSD and Evorubric , using 184 logged steps from each run on the same policy-worker hardware.
The measured step time covers the core training pipeline: response generation, rubric generation, external judging and verification, student and teacher scoring, and actor updates. The observed increase in average step time is approximately 11%, indicating a moderate per-step runtime overhead.
We will include the independent-seed results and timing comparison in the revised version to make both training variability and the runtime–performance trade-off clearer.
We thank the reviewer for pointing out this distinction. We agree that changes in the mean and standard deviation of the likelihood signal do not directly establish that stylistic effects have been removed. We will revise the interpretation in Section 4.5 accordingly.
Our motivation follows RLCSD, which observes that privileged conditioning can introduce both task-related changes and stylistic shifts. With matched prompt templates, contrasting teacher likelihoods may attenuate stylistic components shared across the conditioning contexts. RLCSD supports this rationale through its task/style-token analysis on mathematical reasoning (Sections 3.1–3.4, Figures 2 and 4). These findings motivate our construction, but do not establish the same mechanism in our open-ended setting.
To illustrate the signal change, we performed an exploratory offline analysis using base Qwen3-8B on a fixed ResearchQA response about weather-based irrigation scheduling. We scored identical response tokens under one-sided and contrastive contexts. The table lists the 3 positions with the largest absolute raw likelihood discrepancy, retaining the signed values.
InFeedbackExampleExampleAutomationEvHere,
Inoccurs in the summary marker “In summary,” andEvis a subword token of “Evapotranspiration.” The summary marker’s initial token moves from rank 1 to 208, with its absolute discrepancy decreasing from 19.5134 to 0.3953. This illustrates a local reduction in the signal assigned to a summary marker.Separately, our training ablation in Table 2 supports the practical benefit of the contrastive construction: removing it reduces HealthBench performance from 51.76 to 49.18. This result supports its usefulness under the tested setting.
We will include the token-level illustration in the supplementary analysis and revise Section 4.5 to distinguish the design rationale, descriptive signal analysis, and downstream performance evidence. In particular, we will remove the implication that the signal statistics directly demonstrate the suppression of stylistic effects.