acceptodds
Under review as a conference paper at ICLR 2027

RubricOpt: Recursive Refinement of Semantic Rubrics for Preference Optimization

Abstract

Pairwise preference labels specify which response is preferred but leave the behavioral basis of that preference implicit. We introduce RubricOpt, a framework that makes a semantic rubric an explicit, editable specification for preference optimization. RubricOpt measures semantic demand—how behaviors separate task success from failure—and semantic supply—how often preference pairs contrast those behaviors—in a shared eight-factor space. Semantic Rubric Steering uses this diagnosis to condition training on a short natural-language rubric while keeping the response pairs, labels, and preference objective fixed. The rubric is gradually withdrawn before training ends, and all evaluation is rubric-free. A gated refinement loop revises clauses when recurring failures reveal requirements missing from the current wording. Our analysis gives sufficient conditions for criterion-free behavioral improvement and for retaining success-probability gains after withdrawal. Experiments use two preference corpora and three models from two families, each compared with a matched ordinary-DPO control. On Tulu3, steering improves BBH by – percentage points and the seven-benchmark average by – points across Qwen3-4B, Qwen3-8B, and Mistral-7B. On HelpSteer3, four rubric compositions improve Qwen3-4B Arena-Hard win rates by – points, without a material increase in response length. Matched HelpSteer3 continuation on Qwen3-4B adds BBH points under the unchanged rubric, while the tested wording edit yields no detectable additional BBH gain, separating the benefit of continued training from the effect of refinement.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Preference optimization, Semantic rubrics, Recursive refinement, LLM alignments

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_oSgXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that pairwise preference labels leave the reason for a preference implicit, and proposes RubricOpt, which has three components: (1) a diagnosis that uses a fixed eight-factor ontology and an LLM judge to compare what benchmarks demand with what preference pairs contrast; (2) Semantic Rubric Steering, which appends a short natural-language rubric to the prompt during DPO training while keeping the pairs, labels and objective unchanged, withdraws the rubric gradually, and evaluates without it; and (3) a gated refinement loop that edits rubric clauses based on recurring benchmark failures and accepts an edit only if deterministic checks pass. Experiments use three models from two families (Qwen3-4B, Qwen3-8B, Mistral-7B) and two preference corpora (Tulu3, HelpSteer3), each compared with a matched ordinary-DPO control. The paper reports gains on BBH and on Arena-Hard, and acknowledges that the one trained refinement edit yields no detectable gain beyond an additional epoch of training.

Critical strengths

Strength 1: The diagnosis is simple and informative. Measuring task demand and preference-data supply on the same set of factors is easy to interpret, and it identifies behaviors that the benchmarks require but the preference pairs rarely contrast (Figure 3, Sec. 5.2).

Strength 2: Matched controls and candid reporting of negative results. Each rubric arm is compared with an ordinary-DPO control that shares the model, pairs, labels, objective and seed, and the refinement study uses two continuation branches that differ only in the rubric, which separates the effect of an edit from that of continued training. The authors report the resulting null result in the abstract rather than omitting it, which makes the claims checkable.

Critical weaknesses

Weakness 1: The effectiveness of the contributions is not demonstrated. (a) Rubric content. The central claim is that the rubric tells the model which behavioral distinction to learn, but the formal experiments lack the basic control of conditioning on a length-matched, content-irrelevant text. Without this control, the gains cannot be attributed to the semantics of the rubric rather than to the presence of extra training context. (b) Diagnosis. All formal experiments use the full set of eight factors, so the diagnosis does not determine the rubric. The comparison fixed in advance to test whether the diagnosis matters (Table 17, comparison E: diagnosis-selected factors vs. low-demand factors at matched length) finds the two indistinguishable on the seven benchmarks, and the authors conclude that the diagnosis “describes the model–task–data state rather than predicting where the gains land” (Appendix H.2). (c) Recursive refinement. The method named in the title has no positive result. The only trained edit yields no gain over the matched unedited continuation, and the authors state: “These results attribute the BBH gain to continued training under the rubric rather than to this wording edit” (Appendix E.1). Taken together, none of the three contributions is shown to be responsible for the reported gains. This is the main reason for my recommendation.

Weakness 2: The presentation is unclear and internally inconsistent. Several statements and tables contradict each other, so it is unclear which results are authoritative. The caption of Table 18 states that the full table is Table 6, but differences computed from Table 6 do not match Table 18 in the BBH, MATH and Avg. columns; the BBH difference stated in Sec. 5.2 agrees with Table 6, not with Table 18. Sec. 5.3 and Table 4 describe the refined rubric R1 as changing only f4 (the f3 proposal is rejected by the gate), whereas Appendix H.4, describing the Tulu3 continuation trained under R1 (labeled RUBRICOPT-R2 (REFINE) in Table 1), states that its rubric differs from R0 in f3, f4 and f8. Likewise, Mistral-7B is reported in the Tulu3 table (Table 1), but Appendix H.14 states that it was trained on the HelpSteer3 training split. In addition, several cross-references are unresolved and render as “??”, e.g., in Appendix A.3 (the figure that supports the composition claim) and Appendix C.2 (the summary of the mechanistic probes). These problems make it difficult to verify the evidence.

Questions for authors

Q1: Are the claims in the introduction and Sec. 5.1 supported by the results? What supports the statement in the introduction that “The pattern holds across different preference corpora, model families, and model scales,” when on Arena-Hard, the protocol behind the Qwen3-4B open-ended result, both Qwen3-8B and Mistral-7B score below their ordinary-DPO controls (Table 2)? Similarly, what is the basis for the statement in Sec. 5.1 that the third family shows “the same correction of ordinary-DPO drift on the open-ended protocols,” when Mistral-7B’s Arena-Hard score is again below its ordinary-DPO control (Table 2)? Will these statements be revised to match the results?

Q2: Which of the inconsistent tables and statements are correct? Why do the differences in Table 18 not match those computed from Table 6, and which is correct? Which clauses differ between R0 and the refined rubric used for the Tulu3 continuation: only f4 (Sec. 5.3, Table 4), or f3, f4 and f8 (Appendix H.4)? Which corpus was Mistral-7B trained on for the results in Table 1: Tulu3, as in Table 1, or HelpSteer3, as in Appendix H.14?

3 comments
AuthorsOpenReview ↗

Responses 1/2

Thank you for the attribution and consistency review. We address each point below.

W1(a): The central claim is that the rubric tells the model which behavioral distinction to learn, but the formal experiments lack the basic control of conditioning on a length-matched, content-irrelevant text. ... the gains cannot be attributed to the semantics of the rubric rather than to the presence of extra training context.

TLDR: At matched length, the rubric's named behavior changes outcomes, which added context alone does not explain.

The revision adds a Length-Matched Controls paragraph in Sec. 4.2. The f2 and f7 rubrics differ by one token and condition the same rows: MuSR pass@4 is 70.8% vs. 68.6%, with 17 prompts passing only under f2 and 6 only under f7. The mismatched f7 rubric also falls below ordinary DPO on MuSR (3 vs. 14 prompts). The f5 rubric's margin slope is 0.0226 nats per Δz5\Delta z_5 vs. 0.0116 against Δz2\Delta z_2, and increasing the rubric from 70 to 234 tokens changes the comprehensive-capability average from 52.64% to 52.74%.

W1(b): All formal experiments use the full set of eight factors, so the diagnosis does not determine the rubric. ... (Tab. 17, comparison E: diagnosis-selected factors vs. low-demand factors at matched length) finds the two indistinguishable on the seven benchmarks, and the authors conclude that the diagnosis “describes the model–task–data state rather than predicting where the gains land” (Appendix H.2).

TLDR: The revision frames diagnosis as a measurement that localizes failures to factors.

The paper states that the shared factor ontology links diagnosis, data contrasts, and clause-level feedback by localizing failures to the factors that revision targets (Sec. 3.1). All eight factors enter the initial rubric from the frozen clause bank (Sec. 3.1). Comparison E's aggregate difference is +0.01 points (Tab. 10 reports the per-benchmark contrasts).

W1(c): The method named in the title has no positive result. The only trained edit yields no gain over the matched unedited continuation ... “These results attribute the BBH gain to continued training under the rubric rather than to this wording edit” ...

TLDR: The revision is a small, targeted update that improves the open-ended aggregate.

Revision selects the factor with the most consistent cross-benchmark evidence and applies the smallest clause revision that addresses it (Sec. 3.2). Both branches share the starting policy, reference, and 566-step budget, differing only in the f4 wording. The revision raises MT-Bench 6.419→6.825 and WildBench 3.781→3.836, and the open-ended aggregate AvgΔ rises from +1.66 to +4.06 (Tab. 3).

Q1: What supports the statement in the introduction that “The pattern holds across different preference corpora, model families, and model scales,” ... both Qwen3-8B and Mistral-7B score below their ordinary-DPO controls ... what is the basis for ... “the same correction of ordinary-DPO drift on the open-ended protocols,” ... Will these statements be revised to match the results?

TLDR: Setting-dependent effects follow from the semantic alignment account.

The rubric aligns the training specification with task demand and preference contrasts, so differences across models and corpora predict setting-dependent effects. The introduction states this directly: “with gains and regressions varying by setting.” Qwen3-8B BBH and Arena-Hard are above their controls, and Mistral's open-ended Arena-Hard is 25.8% against 24.0% for DPO (Tab. 2).

AuthorsOpenReview ↗

Responses 2/2

W2 / Q2: Why do the differences in Table 18 not match those computed from Table 6, and which is correct?

TLDR: The revised manuscript reports one protocol, and every reported difference matches its component scores.

The submitted tables came from different harness versions. The revised manuscript reports full-subtask scores in Tab. 10 and contrasts in Tab. 9 and Appendix D.2. All 15 comprehensive-capability averages in Tabs. 1, 2, and 10 match their component scores to displayed precision.

Q2: Which clauses differ between R0 and the refined rubric used for the Tulu3 continuation: only f4 ... or f3, f4 and f8 ...?

TLDR: The HelpSteer3 matched revision changes only f4, and the Tulu3 continuation changes f3, f4, and f8.

The f4 revision is the HelpSteer3 KEEP/REFINE experiment in Tab. 3. The Tulu3 continuation in Tab. 1 uses a separate revised rubric (Appendix D.2). RubricOpt-R1/R2 name policy rounds, and R0/R1 name the rubric used in each round.

Q2: Which corpus was Mistral-7B trained on for the results in Table 1: Tulu3, as in Table 1, or HelpSteer3, as in Appendix H.14?

TLDR: Tab. 1 combines Tulu3 capability checkpoints and HelpSteer3 open-ended checkpoints.

The capability block uses Tulu3-trained Mistral checkpoints, and the open-ended block uses separate HelpSteer3-trained checkpoints, one per dataset (Tab. 2 and Appendix D.2). Tab. 2 labels both blocks in its headings and caption.

W2: several cross-references are unresolved and render as “??”, e.g., in Appendix A.3 ... and Appendix C.2 ... These problems make it difficult to verify the evidence.

TLDR: The revised manuscript has no unresolved references.

The compiled PDF contain no unresolved “??” references or undefined citations. The factor composition is defined in Sec. 3.1 and evaluated in Tab. 10, and the mechanistic probes are in Sec. 4.1.

Thanks for the responses. Three things remain open: f2 vs f7 compares two real rubrics rather than a content-irrelevant control at matched length, so it still cannot separate the rubric's semantics from added instruction; your comparison E differs by +0.01, which concedes that the diagnosis does not determine the rubric; and BBH is null, so one f4 edit on one seed does not establish the mechanism named in the title. The presentation fixes are noted but were not what drove my decision.

Reviewer_EXmAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

The paper introduces RubricOpt, a method that incorporates natural-language rubrics into DPO training to make the behavioral preferences more explicit. The rubric covers eight factors: task comprehension, reasoning validity, verification and calibration, factual grounding, completeness, instruction adherence, clarity, and conciseness. Each factor has a short Prefer clause and an Avoid clause. The authors use a Qwen3-8B judge to score responses on these factors from 0 to 4, and use these scores to measure two things: semantic demand, which captures the differences between successful and failed benchmark responses, and semantic supply, which measures how frequently chosen and rejected responses differ along each factor. During training, the rubric is included in the prompt with probability q. It is always included during the first 10% of training, gradually phased out over the next 80% using a cosine schedule, and completely removed for the final 10%. Importantly, the preference pairs, labels, and DPO objective remain unchanged, and the rubric tokens are masked from the loss. The paper also introduces a refinement process in which the policy examines benchmark failures and proposes edits to the rubric, subject to a deterministic acceptance gate. On the theoretical side, Proposition 1 and Appendix K provide local conditions under which rubric conditioning can improve behavior or task success even after the rubric is removed, together with a bound on how much of the improvement may be lost.

The experiments are conducted on Qwen3-4B, Qwen3-8B, and Mistral-7B-Instruct-v0.3, using LoRA-based DPO with HelpSteer3 and Tulu3. Each experiment uses one training epoch and one random seed, with standard DPO as the matched baseline. On Tulu3, RubricOpt improves BBH performance by 3.80 to 8.35 points and the average performance across seven benchmarks by 1.06 to 2.96 points (Table 1). On HelpSteer3, four different rubric compositions improve the Arena-Hard win rate of Qwen3-4B by 4.1 to 7.1 points (Tables 2 and 3). However, the evidence for the proposed rubric refinement mechanism is much more limited. The only trained refinement consists of a seven-word change to the factual-grounding clause. In the matched continuation experiment, simply training for another epoch with the original rubric already improves BBH by 8.15 points, while the refined rubric provides no detectable additional improvement (Appendix E.1).

Overall, I find the idea promising, as natural-language rubrics could make DPO more interpretable and controllable without modifying its core objective. The reported gains are encouraging, but the single-seed experiments and limited evidence for rubric refinement leave some important questions open. More comprehensive validation is needed to establish how robust and generalizable the benefits are.

Critical strengths

One strength of this paper is the promising research direction it explores.

Conditioning DPO training on natural-language rubrics, while keeping the preference pairs, labels, and optimization objective unchanged, provides an intuitive way to make the behavioral basis of preferences more explicit. This could improve both the explainability and human controllability of preference optimization, allowing humans to better understand and guide the behavioral criteria used to align models.

Critical weaknesses

Two major weaknesses remain that, in my view, prevent the paper from reaching the acceptance bar for ICLR.

W1. The presentation and overall completeness need substantial improvement. The paper is difficult to follow, partly because it introduces many new terms (semantic demand and supply, Semantic Rubric Steering, R0/R1/R2, KEEP and REFINE) without a sufficiently clear explanation of how they fit together. The 9-page main text also relies heavily on a 39-page appendix, which contains unresolved "??" references and inconsistent numbers (e.g., different seven-benchmark averages for the same runs in Tables 6 and 26). In addition, the mandatory AI use statement is minimal and refers to details that are not provided. The recommended ethics and reproducibility statements are missing, and there is no mention of code release. These issues make the paper feel insufficiently polished and difficult to assess.

W2. The reported improvements are not sufficiently robust, and it remains unclear how much of the gains can actually be attributed to the rubric. For example, the highlighted +8.35 BBH improvement for Qwen3-4B on Tulu3 (Table 1) comes from a generative, tag-aware BBH evaluation. However, an earlier evaluation of the same runs reports nearly identical scores for ordinary DPO (55.54) and RubricOpt-R1 (55.67) in Table 31. Appendix H.10 also acknowledges that "what the criterion changes on BBH is the format of the answer". Similarly, for Qwen3-8B, the matched comparison in Appendix H.13 "shows no measurable difference on this corpus at this budget", despite the +3.80 BBH improvement highlighted in the abstract. Furthermore, all experiments use a single seed, there is no content-matched placebo control, and the only trained refinement produces a BBH change of -0.50 (95% CI [-2.1, +1.1]). Together, these results raise important questions about the robustness of the improvements and whether they genuinely come from rubric conditioning.

Overall, while I appreciate the promising research direction, I believe the paper still falls short of the ICLR acceptance bar in terms of completeness, presentation, and empirical soundness. I encourage the authors to address these issues, strengthen the experimental evidence, and present the work more clearly in a future revision.

Questions for authors

Clarification requests (If the authors can address properly, that may increase any score or overall score assessment)

  1. Which evaluation protocol should be considered the primary one? The main results use a generative, tag-aware BBH evaluation (Appendix H.12), while an earlier evaluation of the same Qwen3-4B Tulu3 runs reports almost identical scores for ordinary DPO (55.54) and RubricOpt-R1 (55.67) in Table 31. For Qwen3-8B, two format-robust evaluations of the same checkpoints also give different results (Appendix H.13: 0.7400 vs. 0.7000 for ordinary DPO, and 0.7535 vs. 0.7380 for RubricOpt-R1). Could the authors clarify whether the primary evaluation protocol was selected before examining the results? It would also be helpful to report both evaluation results for all runs in Table 1, along with their answer-parsing success rates.

  2. Do the improvements come from the actual content of the rubric, or simply from adding extra context during training? A useful control would be to replace the rubric with an unrelated, length-matched instruction (approximately 234 tokens, matching the full rubric in Table 26), while keeping the same exposure mask and training schedule. Could the authors include this comparison against RubricOpt-R1 on BBH and Arena-Hard? This would help establish whether the behavioral guidance in the rubric is responsible for the observed improvements.

  3. How much of the improvement can be attributed to additional training? On HelpSteer3, another epoch of training with the unchanged rubric already improves BBH by 8.15 points (Appendix E.1). However, there is no corresponding ordinary-DPO control with the same additional training budget. Could the authors include this control, ideally starting from the same checkpoint, to better separate the benefits of rubric conditioning from those of simply training longer?

4 comments
AuthorsOpenReview ↗

Responses 1/3

Thank you for the suggestions on evaluation and completeness. We address each point below.

W1.1: The paper is difficult to follow, partly because it introduces many new terms ... without a sufficiently clear explanation of how they fit together. ... relies heavily on a 39-page appendix, which contains unresolved "??" references and inconsistent numbers ...

TLDR: The revision states the core claim directly and collects the concept definitions in the appendix.

RubricOpt writes the behavioral criteria behind a preference label as a semantic rubric: clauses keyed to a shared eight-factor ontology that links task demand, preference data, and failures. The policy checks its own failures against the rubric, revises the clauses that leave them unaddressed, and trains under the revised rubric.

The revision states this claim in the introduction, condenses the appendix from 39 to 14 pages, removes the unresolved references, and collects the symbols in Appendix A.1 with the factor ontology and worked demand/supply examples in Appendix A.2.

W1.2: the mandatory AI use statement is minimal and refers to details that are not provided. The recommended ethics and reproducibility statements are missing, and there is no mention of code release.

TLDR: The revision adds the required statements and includes the core implementation.

The manuscript now includes Reproducibility and Broader Impact statements (p.10) and an expanded LLM Usage Disclosure (p.15). Appendices A.2–A.3 and C.2 specify the ontology, blind scoring, and five-stage revision, and the full codebase will be released publicly. The core implementation is below.

AuthorsOpenReview ↗

Responses 2/3

import math, re

# (1) Frozen canonical clause bank (Appendix A.2).
CLAUSE_BANK = {
 "f1": ("task comprehension",
        "show a correct understanding of what was asked",
        "answering a different question or ignoring stated conditions"),
 "f2": ("reasoning validity",
        "reasoning steps that are logically grounded and internally consistent",
        "unsupported intermediate inferences or invalid deductions"),
 "f3": ("verification and calibration",
        "verify key intermediate results and express uncertainty in a calibrated way",
        "unchecked steps or unsupported overconfidence"),
 "f4": ("factual and evidential grounding",
        "factual claims are grounded in the given information or common knowledge",
        "invented facts or unsupported assertions"),
 "f5": ("completeness",
        "responses that include the reasoning and information needed to fully address the task",
        "missing necessary reasoning, information, or important parts of the task"),
 "f6": ("instruction adherence",
        "follow the stated instructions and stay on task",
        "drifting off-topic or violating requested formats"),
 "f7": ("clarity and organization",
        "clearly expressed and logically organized",
        "jumbled steps or answers that are hard to locate"),
 "f8": ("efficiency and conciseness",
        "keep only the necessary content",
        "repeating reasoning or padding with filler"),
}

# (2) Renderer (Appendix C.1). The same string is appended to every training row.
def render_rubric(factors, bank=CLAUSE_BANK):
    names  = ", ".join(bank[f][0] for f in factors)
    prefer = ";\n".join(f"- {bank[f][1]}" for f in factors)
    avoid  = ";\n".join(f"- {bank[f][2]}" for f in factors)
    return (f"Capability priorities: {names}\n"
            f"Prefer:\n{prefer}.\n"
            f"Avoid:\n{avoid}.")

# (3) Deterministic gate (Appendix C.2), round-1 checks.
TOKEN = re.compile(r"[A-Za-z]+")
INTENSIFIERS = {"very", "extremely", "highly", "significantly", "substantially"}
BENCH_NAMES = {"bbh", "gsm8k", "math", "musr", "arc", "truthfulqa", "mmlu",
               "tulu3", "helpsteer3", "arena", "mt-bench", "wildbench"}

def _content_tokens(s):
    return [t.lower() for t in TOKEN.findall(s)]

def gate(proposal, bank=CLAUSE_BANK, attribution=None, max_edits=3):
    """proposal[f] = {"input": (prefer, avoid), "output": (prefer, avoid),
                      "decision": "keep" | "refine"}."""
    checks = {}
    checks["all_factors_once"] = (
        set(proposal) == set(bank) and
        all(proposal[f]["decision"] in ("keep", "refine") for f in proposal))

    refined = [f for f in proposal if proposal[f]["decision"] == "refine"]
    checks["input_matches_canonical"] = all(
        proposal[f]["input"] == (bank[f][1], bank[f][2]) for f in refined)

    ok_len, ok_lex = True, True
    for f in refined:
        for new, old in zip(proposal[f]["output"], (bank[f][1], bank[f][2])):
            new_t, old_t = _content_tokens(new), set(_content_tokens(old))
            recall = len(set(new_t) & old_t) / max(1, len(old_t))
            ok_len &= recall >= 0.60
            ok_len &= -5 <= len(new.split()) - len(old.split()) <= 10
            ok_lex &= not any(t in BENCH_NAMES for t in new_t)
            ok_lex &= not re.search(r"\d", new)
            ok_lex &= not any(t in INTENSIFIERS for t in new_t)
    checks["content_retention_and_length"] = ok_len
    checks["no_leakage_or_digits"] = ok_lex
    checks["matched_pair"] = all(p and a for p, a in (proposal[f]["output"] for f in proposal))
    checks["attribution_trace"] = (not refined) or (
        attribution is not None and set(refined) <= set(attribution))
    checks["edit_budget"] = len(refined) <= max_edits
    return all(checks.values()), checks
AuthorsOpenReview ↗

Responses 3/3

W2 / Q1: Which evaluation protocol should be considered the primary one? ... an earlier evaluation of the same Qwen3-4B Tulu3 runs reports almost identical scores ... (55.54) ... (55.67) ... For Qwen3-8B, two format-robust evaluations of the same checkpoints also give different results ... Could the authors clarify whether the primary evaluation protocol was selected before examining the results? ... report both evaluation results for all runs in Table 1, along with their answer-parsing success rates.

TLDR: The revision evaluates every run on all 27 BBH subtasks with shared prompt/sample keys.

The formal study uses one harness protocol for every baseline, and BBH covers all 27 subtasks (Appendix D.1). The current Tulu3 results and parsing rates are:

ModelMethodBBH accuracy (%)Harness parsing rate (%)
Qwen3-4BOrdinary DPO15.4396.21
Qwen3-4BRubricOpt-R119.64 (+4.20)95.70
Qwen3-8BOrdinary DPO38.6693.07
Qwen3-8BRubricOpt-R148.26 (+9.60)94.19

Both methods use the same prompt/sample keys within each model block, so the comparison is item-paired. The reported comparison uses the full 27-subtask evaluation. Appendix H.10 is an inference-time rubric reinsertion experiment, separate from training-time conditioning.

Q2: replace the rubric with an unrelated, length-matched instruction (approximately 234 tokens, matching the full rubric in Table 26), while keeping the same exposure mask and training schedule. Could the authors include this comparison against RubricOpt-R1 on BBH and Arena-Hard?

TLDR: Existing comparisons hold length fixed while changing the named behavior.

The revision adds a Length-Matched Controls paragraph in Sec. 4.2. The f2 and f7 rubrics differ by one token and condition the same rows, with MuSR pass@4 at 70.8% versus 68.6% and 17 versus 6 unique passes. The factor-set comparison uses mid-size compositions of 151 and 153 tokens, with per-benchmark contrasts in Tab. 10.

Q3: How much of the improvement can be attributed to additional training? ... there is no corresponding ordinary-DPO control with the same additional training budget. Could the authors include this control, ideally starting from the same checkpoint, to better separate the benefits of rubric conditioning from those of simply training longer?

TLDR: Both reported gains are measured against controls with identical training budgets.

For first-round conditioning, each rubric condition and its ordinary-DPO control train for the same number of epochs within a dataset (Appendix C.3). For the revision, KEEP and REFINE start from the same first-round checkpoint and share the same additional training budget. The revision raises MT-Bench 6.419→6.825 and WildBench 3.781→3.836, and the open-ended aggregate AvgΔ rises from +1.66 to +4.06 (Tab. 3).

W2: all experiments use a single seed, there is no content-matched placebo control, and the only trained refinement produces a BBH change of -0.50 (95% CI [-2.1, +1.1]). ... questions about the robustness of the improvements and whether they genuinely come from rubric conditioning.

TLDR: Every comparison is paired on model, pairs, labels, objective, and seed.

Each rubric condition shares these with its ordinary-DPO control, and the open-ended aggregate improves (AvgΔ +1.66→+4.06, Tab. 3). The −0.50 interval belongs to the earlier sampled evaluation, and the study varies model, corpus, and benchmark within one training seed per setting.

Reviewer_EXmAOpenReview ↗

Response to all above 3 sections

Thanks to the authors for the response and the revised PDF, which I have read.

The paper is much easier to follow now. The appendix is down to 14 pages, the broken references are gone, the notation and factor definitions are in one place, and the missing statements have been added. I also like that the abstract and conclusion are more careful about what the results show. I have raised my Presentation score from 1 to 2. On code, the snippet in the response covers the clause bank, the renderer and the gate, but not the judge, the attribution step or training, so an anonymous repository would still help.

My main concern, though, is the BBH evaluation, and the revision makes me more worried about it, not less. I asked whether the protocol was chosen before the results were known, and for both earlier measurements with parse rates. Neither is in the response or the paper. Instead, Tab. 1 now uses a third protocol. Under it, Qwen3-4B on Tulu3 gets 15.43 (DPO) and 19.64 (RubricOpt-R1), where the original Tab. 31 gave about 55.5 for both. For Qwen3-8B the gain is now +9.61, while the original App. H.13 said there was no measurable difference. The absolute numbers also look wrong to me. The untrained Qwen3-4B-Instruct gets 12.82 on BBH (Tab. 10) and 5.2% in the pilot (Tab. 5), which is far below what this model normally scores and below chance on most BBH subtasks. Tab. 6 then reports 43.1% for a Qwen3-4B R2 checkpoint without a rubric, while Tabs. 1 and 3 give 17.80–22.73 for the R2 checkpoints. With a 96% parse rate, scores this low suggest that the harness setup is driving the BBH numbers more than the models are. Mistral is evaluated with a different extraction method, and there BBH drops by 7.95 points. Since BBH carries most of the capability claims, I do not know which of these numbers to trust.

On Q2, I was asking for an unrelated instruction of the same length, trained with the same mask and schedule. The new paragraph compares rubrics with each other (f2 vs f7, and the 70-token vs 234-token rubric), so rubric text is present on both sides and the question stays open. The pilot also does not support the claim: f2 and plain DPO both get 70.8% MuSR pass@4, and App. C.1 says no benchmark shows a significant difference between them. What the pilot shows is that f7 does worse than f2, not that a rubric does better than no rubric.

On Q3, I agree that the first-round comparisons are matched in epochs, and Tab. 3 now says which baseline Avg. ∆ uses. My question was about the second round. KEEP and REFINE both train with a rubric, so neither tells us what one more epoch of plain DPO would have done. That run is still missing. The revision results are also mixed. Compared with KEEP, REFINE is slightly lower on BBH, Arena-Hard and MATH-Hard and higher on MT-Bench and WildBench, and both branches end below R1 on MT-Bench (6.419 and 6.825 vs 6.93). On Tulu3, the second round costs about 6 points on both GSM8K and MATH-Hard, which brings the average back to the DPO level (48.24 vs 48.20).

Finally, a shared seed between an arm and its control does not tell us how much the results move across seeds, and everything here is still one run. Many of the remaining gains are small. On Qwen3-4B with HelpSteer3, BBH goes up by 0.23 points and the average by 0.18. Without more seeds or confidence intervals, I cannot tell which of these differences are real.

The idea is still interesting, and the paper reads much better now. But the main question is still open: do the gains come from the rubric, and do they hold up across evaluation setups? None of the three experiments I asked for were run, and the BBH numbers are not consistent. I am keeping my overall rating.

Reviewer_6Z7EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Preference optimization collapses multiple behaviors of two responses into a single binary label to perform optimization, which leads to degenerate performance. RubricOpt is motivated by the supply-demand discrepancy between the task-performance and the dataset, where the discrepancy is measured across eight factors, scored by an LLM-as-a-judge. The authors propose semantic rubric steering, which appends a rubric during training-time only, so that the model can adhere to the semantically defined rubric, to allow behavior-aligned preference optimization. The authors additionally propose a refinement stage, where the pipeline discovers recurring failures to provide feedback to the eight-factors. Experiments show that RubricOpt improves various benchmark performances.

Critical strengths

The need of semantically aligned preference optimization

I strongly agree with the authors that semantically aligned preference optimization is important. There are multiple behaviors hidden in the two responses for a DPO (x,y1,y2)(x,y_1,y_2) triplet, where a single preference label collapses the fine-grained behavioral preferences into one.

Critical weaknesses

The demand-supply analysis is not sufficient to support the claim

  1. The demand-supply analysis is not sufficient to support the claim. The authors show the effectiveness of RubricOpt in the main table (Table 1). However, this is insufficient to show that RubricOpt operates according to the method's key idea: semantically aligning the model across the eight factors. To show RubricOpt's efficacy, the authors can conduct a demand diagnosis on multiple models: (1) the base model, (2) the Ordinary DPO model, and (3) the RubricOpt model. The following results would support the corresponding interpretations.

  2. Ordinary DPO does not match the demand of the benchmark. If Ordinary DPO does not match the benchmark's demand and shows a trend similar to the supply, we can interpret this result as indicating that Ordinary DPO adheres strictly to the dataset. Its eight-factor profile therefore resembles the dataset's supply profile, leading to degraded benchmark performance due to the demand-supply gap.

  3. RubricOpt's REFINE version shows a similar eight-factor profile to that of the demand. If the authors show this result, we can interpret it as follows: the refinement version alters the rubric to match the benchmark. This applies to the refinement version rather than vanilla RubricOpt, which does not include benchmark diagnostics. The refinement version therefore successfully steers training toward the demanded behaviors and ultimately leads to superior performance in Table 1.

The refinement stage does not show substantial benefit.

  1. The refinement stage is the most important part of this work. In my view, the refinement stage is the most important part of this work. The vanilla version of RubricOpt can be seen as a simple instruction change during training with a scheduler. I expect the refinement stage to analyze the supply-demand gap and make the most of the supply to meet the demand (e.g., re-weighting the data examples to match the semantic demand distribution). However, the current refinement loop does not seem to minimize this gap. Instead, it discovers a failure and attributes it to one of the eight factors. I suspect that the loosely coupled method resulted in lower performance for Qwen3-4B in Table 1.
Questions for authors
  1. How consistent is the score zk(x,y)z_k(x,y) across different judges?
  2. With what standards have the authors chosen the eight factors?
2 comments

Thank you for focusing on the behavioral mechanism. We address each point below.

W1.1–W1.3: conduct a demand diagnosis on multiple models: (1) the base model, (2) the Ordinary DPO model, and (3) the RubricOpt model. ...

If Ordinary DPO does not match the benchmark's demand and shows a trend similar to the supply ... Its eight-factor profile therefore resembles the dataset's supply profile ...

RubricOpt's REFINE version shows a similar eight-factor profile to that of the demand. ... the refinement version alters the rubric to match the benchmark. ... successfully steers training toward the demanded behaviors ...

TLDR: The three models share the same demand ranking, and the R1 revision raises demand on high-demand factors on MATH and ARC.

The demand ranking is led by completeness (F5), reasoning validity (F2), and conciseness (F8) in Base, ordinary DPO, and RubricOpt-R1. Completeness demand is 0.269, 0.249, and 0.252 on the common benchmarks. Ordinary DPO's demand profile stays close to Base's. The R1 revision raises completeness demand on MATH-Hard from 0.115 to 0.211 and conciseness demand on ARC from 0.148 to 0.233.

W2: The refinement stage is the most important part of this work. ... I expect the refinement stage to analyze the supply-demand gap and make the most of the supply to meet the demand (e.g., re-weighting the data examples to match the semantic demand distribution). However, the current refinement loop does not seem to minimize this gap. ... I suspect that the loosely coupled method resulted in lower performance for Qwen3-4B in Table 1.

TLDR: Revision makes the same training pairs emphasize the diagnosed factor by revising the rubric.

Reweighting data by semantic supply is a natural use of the diagnosis and a promising direction for training-data selection, which we have added as future work. The current operator acts on the rubric: the policy identifies a factor where its rubric-free benchmark behavior is deficient, revises the corresponding clause, and the revised rubric conditions both responses in each pair. The same pairs then emphasize that factor during DPO. With starting policy fixed, the f4 revision raises MT-Bench +0.406+0.406 and WildBench +0.055+0.055, and the open-ended aggregate improves (AvgΔ +1.66→+4.06, Tab. 3).

Q1: How consistent is the score zk(x,y)z_k(x,y) across different judges?

TLDR: Every condition uses the same self-judge, so consistency is across conditions.

The self-judge prompt is the same for all conditions and blind to task outcome, gold answer, preference role, and generating model. Appendix A.3 reports its reliability check.

Q2: With what standards have the authors chosen the eight factors?

TLDR: The eight factors synthesize pragmatic maxims with fine-grained evaluation taxonomies.

RubricOpt synthesizes them from Grice's cooperative maxims [1] and fine-grained LLM evaluation taxonomies [2] (Sec. 1 and Sec. 3.1), and Appendix A.2 defines each factor with a matched Prefer/Avoid pair. The ontology is held constant across models and corpora.

[1] Grice, H. P. Logic and Conversation. In Syntax and Semantics, Vol. 3: Speech Acts, 1975.

[2] Ye, S., et al. FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets. ICLR 2024.

I thank the authors for responding to my concerns.

  • W1: the scores the authors have introduces are the success-failure factor gaps, which doesn’t show the following: (1) the ordinary DPO followed the training data’s supply, due to lack of factor-wise guidance during training; (2) the RubricOpt has improved on the factors that the base model’s lacked on, via the factor-wise guidance. Therefore, my concern remains.
  • W2: I acknowledge that the refinement allows some performance increase. However, I am not convinced on that the current methodology of refinement will allow smaller gap between the supply and the demand, which is the motivation of this research.
  • W3: For the judge consistency, the authors should show that the scores are consistent across different judges.
  • W4: I appreciate the authors for clarifying the ontology's reliability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.