- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper argues that pairwise preference labels leave the reason for a preference implicit, and proposes RubricOpt, which has three components: (1) a diagnosis that uses a fixed eight-factor ontology and an LLM judge to compare what benchmarks demand with what preference pairs contrast; (2) Semantic Rubric Steering, which appends a short natural-language rubric to the prompt during DPO training while keeping the pairs, labels and objective unchanged, withdraws the rubric gradually, and evaluates without it; and (3) a gated refinement loop that edits rubric clauses based on recurring benchmark failures and accepts an edit only if deterministic checks pass. Experiments use three models from two families (Qwen3-4B, Qwen3-8B, Mistral-7B) and two preference corpora (Tulu3, HelpSteer3), each compared with a matched ordinary-DPO control. The paper reports gains on BBH and on Arena-Hard, and acknowledges that the one trained refinement edit yields no detectable gain beyond an additional epoch of training.
Critical strengths
Strength 1: The diagnosis is simple and informative. Measuring task demand and preference-data supply on the same set of factors is easy to interpret, and it identifies behaviors that the benchmarks require but the preference pairs rarely contrast (Figure 3, Sec. 5.2).
Strength 2: Matched controls and candid reporting of negative results. Each rubric arm is compared with an ordinary-DPO control that shares the model, pairs, labels, objective and seed, and the refinement study uses two continuation branches that differ only in the rubric, which separates the effect of an edit from that of continued training. The authors report the resulting null result in the abstract rather than omitting it, which makes the claims checkable.
Critical weaknesses
Weakness 1: The effectiveness of the contributions is not demonstrated. (a) Rubric content. The central claim is that the rubric tells the model which behavioral distinction to learn, but the formal experiments lack the basic control of conditioning on a length-matched, content-irrelevant text. Without this control, the gains cannot be attributed to the semantics of the rubric rather than to the presence of extra training context. (b) Diagnosis. All formal experiments use the full set of eight factors, so the diagnosis does not determine the rubric. The comparison fixed in advance to test whether the diagnosis matters (Table 17, comparison E: diagnosis-selected factors vs. low-demand factors at matched length) finds the two indistinguishable on the seven benchmarks, and the authors conclude that the diagnosis “describes the model–task–data state rather than predicting where the gains land” (Appendix H.2). (c) Recursive refinement. The method named in the title has no positive result. The only trained edit yields no gain over the matched unedited continuation, and the authors state: “These results attribute the BBH gain to continued training under the rubric rather than to this wording edit” (Appendix E.1). Taken together, none of the three contributions is shown to be responsible for the reported gains. This is the main reason for my recommendation.
Weakness 2: The presentation is unclear and internally inconsistent. Several statements and tables contradict each other, so it is unclear which results are authoritative. The caption of Table 18 states that the full table is Table 6, but differences computed from Table 6 do not match Table 18 in the BBH, MATH and Avg. columns; the BBH difference stated in Sec. 5.2 agrees with Table 6, not with Table 18. Sec. 5.3 and Table 4 describe the refined rubric R1 as changing only f4 (the f3 proposal is rejected by the gate), whereas Appendix H.4, describing the Tulu3 continuation trained under R1 (labeled RUBRICOPT-R2 (REFINE) in Table 1), states that its rubric differs from R0 in f3, f4 and f8. Likewise, Mistral-7B is reported in the Tulu3 table (Table 1), but Appendix H.14 states that it was trained on the HelpSteer3 training split. In addition, several cross-references are unresolved and render as “??”, e.g., in Appendix A.3 (the figure that supports the composition claim) and Appendix C.2 (the summary of the mechanistic probes). These problems make it difficult to verify the evidence.
Questions for authors
Q1: Are the claims in the introduction and Sec. 5.1 supported by the results? What supports the statement in the introduction that “The pattern holds across different preference corpora, model families, and model scales,” when on Arena-Hard, the protocol behind the Qwen3-4B open-ended result, both Qwen3-8B and Mistral-7B score below their ordinary-DPO controls (Table 2)? Similarly, what is the basis for the statement in Sec. 5.1 that the third family shows “the same correction of ordinary-DPO drift on the open-ended protocols,” when Mistral-7B’s Arena-Hard score is again below its ordinary-DPO control (Table 2)? Will these statements be revised to match the results?
Q2: Which of the inconsistent tables and statements are correct? Why do the differences in Table 18 not match those computed from Table 6, and which is correct? Which clauses differ between R0 and the refined rubric used for the Tulu3 continuation: only f4 (Sec. 5.3, Table 4), or f3, f4 and f8 (Appendix H.4)? Which corpus was Mistral-7B trained on for the results in Table 1: Tulu3, as in Table 1, or HelpSteer3, as in Appendix H.14?
3 comments
Responses 2/2
W2 / Q2: Why do the differences in Table 18 not match those computed from Table 6, and which is correct?
TLDR: The revised manuscript reports one protocol, and every reported difference matches its component scores.
The submitted tables came from different harness versions. The revised manuscript reports full-subtask scores in Tab. 10 and contrasts in Tab. 9 and Appendix D.2. All 15 comprehensive-capability averages in Tabs. 1, 2, and 10 match their component scores to displayed precision.
Q2: Which clauses differ between R0 and the refined rubric used for the Tulu3 continuation: only f4 ... or f3, f4 and f8 ...?
TLDR: The HelpSteer3 matched revision changes only f4, and the Tulu3 continuation changes f3, f4, and f8.
The f4 revision is the HelpSteer3 KEEP/REFINE experiment in Tab. 3. The Tulu3 continuation in Tab. 1 uses a separate revised rubric (Appendix D.2). RubricOpt-R1/R2 name policy rounds, and R0/R1 name the rubric used in each round.
Q2: Which corpus was Mistral-7B trained on for the results in Table 1: Tulu3, as in Table 1, or HelpSteer3, as in Appendix H.14?
TLDR: Tab. 1 combines Tulu3 capability checkpoints and HelpSteer3 open-ended checkpoints.
The capability block uses Tulu3-trained Mistral checkpoints, and the open-ended block uses separate HelpSteer3-trained checkpoints, one per dataset (Tab. 2 and Appendix D.2). Tab. 2 labels both blocks in its headings and caption.
W2: several cross-references are unresolved and render as “??”, e.g., in Appendix A.3 ... and Appendix C.2 ... These problems make it difficult to verify the evidence.
TLDR: The revised manuscript has no unresolved references.
The compiled PDF contain no unresolved “??” references or undefined citations. The factor composition is defined in Sec. 3.1 and evaluated in Tab. 10, and the mechanistic probes are in Sec. 4.1.
Thanks for the responses. Three things remain open: f2 vs f7 compares two real rubrics rather than a content-irrelevant control at matched length, so it still cannot separate the rubric's semantics from added instruction; your comparison E differs by +0.01, which concedes that the diagnosis does not determine the rubric; and BBH is null, so one f4 edit on one seed does not establish the mechanism named in the title. The presentation fixes are noted but were not what drove my decision.
Responses 1/2
Thank you for the attribution and consistency review. We address each point below.
TLDR: At matched length, the rubric's named behavior changes outcomes, which added context alone does not explain.
The revision adds a Length-Matched Controls paragraph in Sec. 4.2. The f2 and f7 rubrics differ by one token and condition the same rows: MuSR pass@4 is 70.8% vs. 68.6%, with 17 prompts passing only under f2 and 6 only under f7. The mismatched f7 rubric also falls below ordinary DPO on MuSR (3 vs. 14 prompts). The f5 rubric's margin slope is 0.0226 nats per Δz5 vs. 0.0116 against Δz2, and increasing the rubric from 70 to 234 tokens changes the comprehensive-capability average from 52.64% to 52.74%.
TLDR: The revision frames diagnosis as a measurement that localizes failures to factors.
The paper states that the shared factor ontology links diagnosis, data contrasts, and clause-level feedback by localizing failures to the factors that revision targets (Sec. 3.1). All eight factors enter the initial rubric from the frozen clause bank (Sec. 3.1). Comparison E's aggregate difference is +0.01 points (Tab. 10 reports the per-benchmark contrasts).
TLDR: The revision is a small, targeted update that improves the open-ended aggregate.
Revision selects the factor with the most consistent cross-benchmark evidence and applies the smallest clause revision that addresses it (Sec. 3.2). Both branches share the starting policy, reference, and 566-step budget, differing only in the f4 wording. The revision raises MT-Bench 6.419→6.825 and WildBench 3.781→3.836, and the open-ended aggregate AvgΔ rises from +1.66 to +4.06 (Tab. 3).
TLDR: Setting-dependent effects follow from the semantic alignment account.
The rubric aligns the training specification with task demand and preference contrasts, so differences across models and corpora predict setting-dependent effects. The introduction states this directly: “with gains and regressions varying by setting.” Qwen3-8B BBH and Arena-Hard are above their controls, and Mistral's open-ended Arena-Hard is 25.8% against 24.0% for DPO (Tab. 2).