acceptodds
Under review as a conference paper at ICLR 2027

Rubrics Always Miss: Robust Preference Optimization under Incomplete Criteria

Abstract

Rubrics provide fine-grained and interpretable supervision for aligning large language models by evaluating responses against explicit criteria, such as factual correctness, relevance, and instruction following. Recent methods generate rubrics that specify what makes a good response to a given instruction, then use rubric scores as rewards or to construct preferences for training. However, it is difficult to ensure that these rubrics cover all relevant criteria, so their scores may not fully reflect response quality. To address this issue, we propose a robust preference optimization framework that accounts for errors in preferences caused by incomplete rubrics. Since the missing criteria are unobserved, we assess rubric completeness indirectly by examining whether the observed rubrics capture differences between responses. For each instruction, we perturb the policy to generate response variations and measure how consistently larger response changes are accompanied by larger changes in rubric scores. This agreement serves as an instance-specific proxy for rubric completeness. We then use the estimated completeness to construct a feasible set around the preference probabilities inferred from the observed rubric scores, allowing larger deviations for samples with lower estimated completeness, and therefore a larger feasible set. The policy minimizes the worst-case training loss over this set, thereby allocating greater protection against preference errors to samples with less complete rubrics. Our framework can be integrated with any margin-aware preference optimization methods. Experiments across multiple datasets demonstrate consistent improvements over existing methods. Code is on anonymous GitHub.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, Rubrics, Robust learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gyytOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: To address preference labeling errors caused by incomplete rubrics, this paper generates response variants through gradient direction perturbation, estimates instance-level rule completeness using the "ranking consistency between response changes and rule score changes," and then constructs an uncertain set of preference probabilities weighted by completeness for min-max robust training. It can be integrated with various margin-aware preference optimization methods.

Critical strengths

Strengths

The problem has real-world significance. Rubric incompleteness is a real pain point in rubric-based alignment, and the quantification in Table 1 is convincing.

The idea of ​​"adjusting the robust radius with instance-level completeness" is novel and plug-and-play. The Uniform/Shuffled ablation design is reasonable, and the occlusion experiment directly verifies the surrogate metric.

Critical weaknesses

Weaknesses

The effectiveness of the completeness surrogate is insufficiently demonstrated.

The Wasserstein distance of the response representation measures superficial or semantic changes, not qualitative changes.

A fragmented rule with only a "length" parameter may be highly correlated with the embedding distance; while a complete rule may not have a score that remains unchanged in the face of rewrite changes. Theorem 1 depends on assumption (12), which states that "the score variation of the complete rule is bidirectionally Lipschitz related to the embedding distance," a premise that needs to be proven.

When a_i ≪ b_i, the margin g_i is likely to be 0, rendering the lower bound meaningless.

The estimation is noisy.

Each sample has at most 8 perturbation directions, and the standard error of Kendall τ is approximately 0.29 when k=8. Adding the scoring noise from the LLM judges and the randomness of generation, s_i is likely primarily noise.

The perturbation-generated answers are independent of the training pair (y_w, y_l). The rule's ability to distinguish perturbation answers does not mean it can distinguish the training pair.

The masking experiment suffers from mechanical confounding.

Masking rules shorten the dimension of D^R and increase tie, thus mechanically decreasing Kendall τ with tie correction. A comparison is needed that "keeps the number of rules constant and replaces them with irrelevant ones."

Key relevant work and baselines are missing.

Performing worst-case optimization around q essentially falls under the category of "robust DPO with noisy preference labels." cDPO (label smoothing), rDPO (Provably Robust DPO, Chowdhury et al., 2024), Dr. DPO (divided robust DPO), etc., are not cited or compared; only the paper's own uniform robust comparison is insufficient.

For MMPO, the closed-form solution q* = σ(logit p − Δ/(λ·s_i)) indicates that its effect is close to an adaptive label smoothing.

The evaluation exhibits cyclicality.

In the setting using generated rules, the win rate and average score are also calculated using generated rules, while the paper's premise is precisely that these rules are incomplete.

Independent evaluation methods such as human-written rules, human preferences, or AlpacaEval/Arena-Hard should be used.

Additionally, the average rating on UltraFeedback only increased from 8.75 to 8.80, but the win rate reached 68%. How draws were handled needs to be explained.

Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Response to W1, W2 & W3

We sincerely thank the reviewer for the detailed assessment and valuable suggestions. We address each concern below.

[W1] Wasserstein distance between response representations may capture superficial or semantic changes rather than changes in response quality.

Response: We thank the reviewer for this valuable comment. We agree that response Wasserstein distance is not a measure of response quality. We appreciate the reviewer’s suggestion and conduct an additional emperical analysis to examine whether larger response shifts tend to correspond to larger quality changes.

  • Evaluate verifiable and open-ended tasks.

    • We use APPS prompts dataset with executable tests and SummEval dataset with open-ended task.
  • Measure the quality change.

    • For nominal response yi0y_{i0} and perturbed response yihy_{ih}, define DihQD^Q_{ih} as quality change measurement DihQ=∣Qi(yih)−Qi(yi0)∣,D^Q_{ih}=\left|Q_i(y_{ih})-Q_i(y_{i0})\right|,

    • On APPS, the quality score QQ is the ratio of passed test Qi(y)=number of tests passednumber of total tests.Q_i(y)=\frac{\text{number of tests passed}}{\text{number of total tests}}.

    • On SummEval, Qi(y)Q_i(y) is the rating judged by GPT-4o using provided evaluation creteria in dataset.

  • Assess the association between response shifts and quality changes.

    • For each prompt ii, we compare the rankings induced by DihYD^Y_{ih} (response shift) and DihQD^Q_{ih} (quality change) across perturbation directions, in order to exam whether larger response shifts more consistently correspond to larger changes in response quality.
    • Let nicn_i^c and nidn_i^d denote the numbers of concordant pairs (matching orders, i.e., larger response shift corresponds to larger quality change) and discordant pairs (opposite orders, i.e., larger response shift corresponds to smaller quality change). We report Kendall’s $\tau_b$ and pair concordance nic/(nic+nid)n_i^c/(n_i^c+n_i^d).
DatasetMean Kendall's τb\tau_bPair concordance (%)
APPS0.58883.32
SummEval0.40174.09

Response shifts are positively associated with quality changes on both tasks, and the number of concordant pairs is substantially larger than the number of discordant pairs. These results support larger response shifts tend to correspond to larger quality changes.

[W2] A length-only rubric may correlate strongly with response embedding distance

Response: We thank the reviewer for this valuable comment. Individual distances or rubric scores may change without changing ss, provided both rankings are preserved. We examine reviewer's concerns through additional experiments.

  • Length-only rubric.

    • We reuse the same RubricBench prompts and their original probe responses, replacing the full rubric with rlen(y)=L(y)r_{\mathrm{len}}(y)=L(y), where L(y)L(y) is the response token count. The corresponding rubric-score shift is

    DihR,len=∣L(yih)−L(yi0)∣.D_{ih}^{R,\mathrm{len}}=\left\lvert L(y_{ih})-L(y_{i0})\right\rvert.

  • Recompute completeness.

    • Keeping response distances fixed, we recompute rank agreement and ss using the original procedure.
RubricMean ssProportion with s≥0.8s\geq 0.8
RubricBench0.809064.6%
Length-only0.18732.2%

Length-only scoring yields substantially lower completeness scores and rarely produces high-ss instances. This suggests that response length alone does not induce the high completeness scores.

[W3] Justification of the two-sided Lipschitz condition in Eq. (12), and a zero lower bound when ai≪bia_i\ll b_i.

Response: We thank the reviewer for the insightful question.

  • The condition in Eq. (12) means that every nonzero response shift changes at least one complete-rubric score. This is the required nondegeneracy: Dih⋆>0D^\star_{ih}>0 whenever DihY>0D^Y_{ih}>0. For example, a change in factual correctness should appear in the corresponding complete-rubric score. Based on this example, this is a mild condition for task-relevant response changes.

  • We provide an example in which Eq. (12) holds. Specifically, let DihY>0D_{ih}^{Y}>0 and Dih⋆>0D_{ih}^{\star}>0, we define

    \qquad b_i=\max_h\frac{D^\star_{ih}}{D^Y_{ih}}.$$ We have $0<a_i\le b_i<\infty$ and hence Eq. (12). Therefore, the constants exist in this case without requiring a global bi-Lipschitz mapping.
  • We refine the lower bound, so that even ai≪bia_i\ll b_i, we can obtain a strictly positive lower bound. Specifically, it suffices that DihR<aiDihYD^R_{ih}<a_iD^Y_{ih} for at least one probe. This condition is also mild. Then, for weights ωi≠0\omega_i\ne0, using the lower inequality in Eq. (12), the same proof gives a stronger bound with a strictly positive right-hand side:

    E[(qi⋆−pi)2]≥∑h=1ki[ai2(DihY)2−(DihR)2]+⋅αi2κi∥ωi∥22ki.\mathbb E[(q_i^{\star}-p_i)^2] \ge \sum_{h=1}^{k_i} \left[a_i^2(D_{ih}^Y)^2-(D_{ih}^R)^2\right]_+ \cdot \frac{\alpha_i^2\kappa_i\|\omega_i\|_2^2}{k_i}.

AuthorsOpenReview ↗

Response to W4, W5, W6 & W7

[W4] Even a complete rubric may change its scores under paraphrasing.

Response: We thank the reviewer for this valuable comment. We add the following experiment to show the sensitivity to paraphrasing.

  • Paraphrase perturbed responses.

    • On RubricBench, we use GPT-4o to paraphrase each existing perturbed response, instructing it to change wording and syntax while preserving meaning.
  • Recompute shifts.

    • Using the same frozen encoder and rubric evaluator, we recompute response distances D~ihY\tilde D^Y_{ih} and rubric-score shifts D~ihR\tilde D^R_{ih} relative to the nominal response.
  • Re-rank and recompute completeness.

    • We independently re-rank the probe directions by the two measures and recompute the ss:
ResponsesMean ssMean ∣s~−s∣\lvert\tilde s-s\rvert
Original0.80900
Paraphrased0.80130.0263

Completeness scores remain relatively stable under the tested paraphrases. Together, the experiments support the usefulness of our score design.

[W5] Is the completeness score dominated by noise

Response: We thank the reviewer for raising this concern.

  • The value of approximately 0.29 assumes that response shifts and rubric-score shifts are unrelated and no tie among the values of either shift.

  • In our setting, these shifts show positive rank association and contain ties. Therefore, the value of approximately 0.29 cannot be taken as the standard error of our method.

  • To further assess estimation noise, we independently repeat the full calibration pipeline five times with different Kmax⁡K_{\max}, and report the mean standard error of τb\tau_b.

Maximum directions Kmax⁡K_{\max}Mean Standard error of τb\tau_b ↓
40.042
80.037
160.034

The mean standard error remains below 0.05, suggesting limited noise in our method.

[W6] A rubric’s ability to distinguish perturbation-generated responses does not necessarily imply that it can correctly distinguish the original training pair.

Response: We thank the reviewer for raising this point. We clarify that the perturbation-generated answers are used to assess rubric incompleteness and determine robustness weights, not to infer the preference between the training responses. In other words, the rubric is decided by prompt, instead of ywy_w and yly_l, so we assess its completeness using response variations for the same prompt.

Specifically:

  • The actual training pair determines the nominal preference and the optimization loss.

  • The perturbation-generated answers aim to calculate score to determine how complete the rubric is.

[W7] Removing criteria reduces the dimension of rubric-score vectors and may increase ties. A control that replaces criteria with irrelevant ones while keeping the total criterion count fixed is needed.

Response: We thank the reviewer for this useful suggestion. We agree that removing criteria may increase tie frequency. However, an increase in ties does not mechanically decrease Kendall's correlation. With tie correction,

τb=C−D(C+D+TY)(C+D+TR).\tau_b=\frac{C-D}{\sqrt{(C+D+T_Y)(C+D+T_R)}}.

Here, CC and DD count concordant and discordant pairs, while TYT_Y and TRT_R count pairs tied only in DYD^Y or DRD^R, respectively.

  • Counterexample: With the other relations unchanged, the correlation can increase.

    \qquad \tau_b:0.333\rightarrow0.816.$$

To empirically address reviewer's concern, we add additional experiment below.

  • Compare deletion and random-rubric replacement. On the same RubricBench examples, we apply deletion or random-rubric replacement to the first m=0,…,5m=0,\ldots,5 criteria in their original order:

    • Deletion: We remove the selected criteria from the rubric-score vectors and compute rubric-score distances using the remaining criteria.

    • Random-rubric replacement: We retain all criterion dimensions but replace the selected criteria’s scores with independently sampled random rubrics scores.

  • Experimental results. We recompute rubric-score distances and Kendall’s τb\tau_b and report tie rates and mean τb\tau_b for both settings.

mmDeletionRandom
Tie rateMean τb\tau_bTie rateMean τb\tau_b
019.67%0.79919.67%0.799
124.81%0.68220.33%0.708
228.90%0.45914.35%0.433
333.93%0.23314.08%0.267
438.84%0.10916.59%0.133
544.96%0.08319.43%0.105

These results support that our completeness measure Kendall’s τb\tau_b is more related to the loss of informative rubric-score signals, rather than tie frequency.

AuthorsOpenReview ↗

Response to W8, W9, W10 & W11

[W8] The method is related to robust preference optimization under noisy labels, and comparisons with cDPO, rDPO, and Dr.DPO are needed.

Response: We thank the reviewer for this valuable suggestion. These methods share the goal of mitigating unreliable supervision, but our method specifically targets inaccurate preference caused by incomplete rubrics.

  • cDPO: Fixed label smoothing.

  • rDPO: Loss correction under random preference flips.

  • Dr.DPO: Preference-pair reweighting through robust loss aggregation.

  • Ours: Instance-specific rubric completeness constrains adversarial deviations from rubric-derived preference probabilities. Omitted criteria can distort both preference strength and direction even when all observed criteria are scored accurately.

  • Added experiments. We evaluate all methods on UltraFeedback (UF), WildChecklists (WC), and HelpSteer2 (HS2), reporting preference accuracy (%) and average rubric score (Avg).

MethodUF Pref. Acc.UF AvgWC Pref. Acc.WC AvgHS2 Pref. Acc.HS2 Avg
cDPO85.738.0277.197.0361.987.55
rDPO86.778.2678.247.7363.087.71
Dr.DPO86.128.1177.967.7063.027.68
MMPO + Ours-KL88.268.8079.778.4764.867.83

MMPO + Ours-KL achieves the highest preference accuracy and average rubric score among the compared methods across all three datasets.

[W9] Closed form in MMPO is related to adaptive label smoothing.

Response: We thank the reviewer for this valuable suggestion.

  • We agree that the MMPO is a special case of our method, which is close to an adaptive soft-target interpretation.

  • However, our approach is a general robust preference optimization framework that also applies to ODPO and Scaled DPO. Using the same completeness-aware uncertainty set, adversarial preferences modify offsets in ODPO and pair weights in Scaled DPO, together with preference directions in both cases, instead of the adaptive label smoothing.

[W10] Independent evaluations, such as human-written rubrics, human preferences, or AlpacaEval and Arena-Hard, are needed.

Response: We thank the reviewer for this useful suggestion. We add additional evaluations on AlpacaEval and Arena-Hard.

  • Evaluation protocols and score interpretation.

    • AlpacaEval: length-controlled win rate. We evaluate each model and compare its responses against the benchmark’s reference responses. We report the length-controlled (LC) win rate (%), which estimates preference over the reference model.

    • Arena-Hard: win-rate score based on dataset-specific rubrics. We evaluate each model and compares its responses against the benchmark’s reference responses. We report a win-rate score (%) against the reference model.

  • Experimental results.

MethodAlpacaEval (LC)Arena-Hard
DPO53.4483.27
MMPO53.6485.19
MMPO + Ours-KL54.3187.36
ODPO54.0285.33
ODPO + Ours-KL55.8986.94
Scaled DPO54.7385.18
Scaled DPO + Ours-KL55.2786.74

Adding Ours method improves both benchmark scores. These consistent gains support the effectiveness of our method.

[W11] The small increase in average score on UltraFeedback appears inconsistent with a pairwise win rate of approximately 68%, and the treatment of ties needs clarification.

Response: We thank the reviewer for this valuable comment. The key distinction is preference frequency versus score magnitude, and the original pairwise protocol did not include a tie option.

  • Win rate.

    • We forced LLM judgments to choose one answer in our method and its corresponding baseline. Note that there is no tie option. Win rate measures how frequently our responses are preferred.
  • Average score.

    • Each response is rated against the rubric from 0-10 intergers. Note that model can rate both 8.3 and 8.1 to 8 to reduce the performance gap. So a small gain is compatible with a relatively high pairwise win rate.
  • Added experiments.

    • To validate the above, we add experiment under a more fine-grained rating, from 0-10 with 0.2 as the step size, on all three datasets. The results are below.
MethodUF CoarseUF FineWC CoarseWC FineHS2 CoarseHS2 Fine
MMPO8.758.798.058.267.747.88
MMPO + Ours-KL8.808.948.478.707.838.06
ODPO8.618.687.157.447.697.83
ODPO + Ours-KL8.638.857.467.767.968.21
Scaled DPO8.638.726.816.937.717.97
Scaled DPO + Ours-KL8.969.377.027.267.888.30

The results show that our method has more advantages in fine-grained setting, indicating that it is reasonable to obtain 68% win rate under small rating gain in coarse setting (since we naturally obtain small difference).


Please let us know if we have properly addressed your questions and we are happy to discuss more!

Dear Reviewer gyyt,

As the discussion deadline approaches, we are wondering whether our responses have properly addressed your concerns? Your feedback would be extremely helpful to us. If you have further comments or questions, we hope for the opportunity to respond to them.

Many thanks,

The authors of #40463

Reviewer_epgrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies preference optimization when evaluation rubrics omit relevant criteria, potentially distorting response preferences. It proposes an instance-specific proxy for rubric completeness by perturbing the policy along gradient-informed directions and measuring rank agreement between response changes and rubric-score changes. This proxy controls a distributionally robust objective that allows greater deviations from nominal preference probabilities for instances with lower estimated completeness. The framework is integrated with MMPO, ODPO, and Scaled DPO, and evaluated on UltraFeedback, WildChecklists, and HelpSteer2 using both dataset-provided and automatically generated rubrics. The paper reports consistent alignment improvements, complemented by criterion-masking experiments, ablations, hyperparameter analyses, and evaluations on six general capability benchmarks. A theoretical analysis establishes a conditional lower bound connecting ranking disagreement to preference error.

Critical strengths
  1. The paper addresses a meaningful limitation of rubric-based alignment. Even accurately scored rubrics can produce misleading preferences when relevant criteria are omitted. Accounting for this residual uncertainty during training complements efforts to improve rubric construction and could benefit a broad range of rubric-based alignment pipelines.

  2. The empirical improvements extend across several training objectives and rubric sources. The reported gains span three datasets, three preference optimization objectives, and multiple rubric generation methods. This breadth supports the framework’s practical potential, while the criterion-masking and shuffled-score experiments provide initial evidence that the instance-specific completeness estimates contribute to its effectiveness.

Critical weaknesses
  1. The interpretation of the completeness score needs stronger support. Response changes can reflect wording or style without changing quality, so disagreement between response shifts and rubric-score shifts does not necessarily indicate missing criteria. Could the authors directly test whether their score predicts preference errors under controlled criterion masking? It would also be helpful to briefly discuss related analyses of preference-data quality, such as AIR(arXiv:2504.03612), to contextualize rubric incompleteness among other sources of unreliable preference supervision.

  2. The advantage over uniform robust optimization remains uncertain. Table 4 shows a 0.48 percentage-point improvement over Uniform Robust, but only one random seed is reported, and using the same robustness budget may not match the effective uncertainty across methods. Does this advantage persist after independently tuning the uniform baseline’s budget and repeating the comparison across seeds? This would clarify whether instance-specific completeness estimation justifies its additional complexity and preprocessing cost.

Questions for authors
  1. Does a lower completeness score predict larger preference errors between masked and full rubrics, beyond the average trend in Figure 3a?
  2. Was Uniform Robust’s budget independently tuned, and is the 0.48-point gain in Table 4 consistent across random seeds?
3 comments
AuthorsOpenReview ↗

Response to W1 & Q1

We sincerely thank the reviewer for the thoughtful evaluation and constructive suggestions. We address each concern below.

[W1 & Q1] The interpretation of the completeness score needs stronger empirical support.

Response: We appreciate the reviewer’s suggestion and agree that additional empirical evidence is needed to examine whether larger response shifts tend to correspond to larger quality changes.

  • Evaluate verifiable and open-ended tasks.

    • We use APPS prompts dataset with executable tests and SummEval dataset with open-ended task.
  • Measure the quality change.

    • For nominal response yi0y_{i0} and perturbed response yihy_{ih}, define DihQD^Q_{ih} as quality change measurement DihQ=∣Qi(yih)−Qi(yi0)∣,D^Q_{ih}=\left|Q_i(y_{ih})-Q_i(y_{i0})\right|,

    • On APPS, the quality score QQ is the ratio of passed test Qi(y)=number of tests passednumber of total tests.Q_i(y)=\frac{\text{number of tests passed}}{\text{number of total tests}}.

    • On SummEval, Qi(y)Q_i(y) is the rating judged by GPT-4o using provided evaluation creteria in dataset.

  • Assess the association between response shifts and quality changes.

    • For each prompt ii, we compare the rankings induced by DihYD^Y_{ih} (response shift) and DihQD^Q_{ih} (quality change) across perturbation directions, in order to exam whether larger response shifts more consistently correspond to larger changes in response quality.
    • Let nicn_i^c and nidn_i^d denote the numbers of concordant pairs (matching orders, i.e., larger response shift corresponds to larger quality change) and discordant pairs (opposite orders, i.e., larger response shift corresponds to smaller quality change). We report Kendall’s $\tau_b$ and pair concordance nic/(nic+nid)n_i^c/(n_i^c+n_i^d).
DatasetMean Kendall's τb\tau_bPair concordance (%)
APPS0.58883.32
SummEval0.40174.09

Response shifts are positively associated with quality changes on both tasks, and the number of concordant pairs is substantially larger than the number of discordant pairs. These results support larger response shifts tend to correspond to larger quality changes.

[W1 & Q1] Does the completeness score predict preference errors under controlled criterion masking, and how does this relate to AIR's analysis of preference-data quality?

Response: We thank the reviewer for this valuable suggestion. We use the same experiment setting on RubricBench in RQ3. We keep the response pairs fixed and remove the first m=0,…,5m=0,\ldots,5 criteria.

  • Assess whether completeness predicts preference errors.
    • To examine whether lower completeness scores are associated with larger preference errors, we measure preference changes under criterion masking. For a response pair (yiA,yiB)(y_i^A,y_i^B), we compute rubric-derived preference probabilities using the Bradley–Terry mapping:
    p_i^{\mathrm{full}} =\sigma\left(\gamma\left[r_i^{\mathrm{full}}(y_i^A)-r_i^{\mathrm{full}}(y_i^B)\right]\right),$$ where $r_{i,m}$ and $r_i^{\mathrm{full}}$ denote response scores under the corresponding rubric. - We then define the **preference probability error** relative to the full-rubric reference: $$e_{i,m}=\left|p_{i,m}^{\mathrm{obs}}-p_i^{\mathrm{full}}\right|,$$ At each masking level, we report: - **Mean completeness:** the average of $s$. - **Mean preference error:** larger $e_{i,m}$ indicate greater deviation from the full-rubric reference. - **Spearman correlation between $1-s_{i,m}$ and $e_{i,m}$:** larger positive values indicate a stronger association between lower completeness and larger preference errors.
Masked criteria mmMean ssMean preference error eeSpearman (1−s,e)(1-s,e)
00.80900N/A
10.69820.0830.287
20.48590.1270.346
30.27160.1790.353
40.15340.2110.338
50.12920.2470.301

We found that both incompleteness 1−s1-s and preference error ee increases as more criteria are removed and lower completeness scores are associated with larger preference errors, supporting the usefulness of ss for predicting errors induced by criterion omission.

  • Relation to AIR.
    • AIR (He et al., 2025) analyzes preference-data quality through annotations, instructions, and response pairs. It improves dataset construction through point-wise generative scoring, variance-based instruction selection, and response-pair filtering, while keeping the training objective fixed to standard DPO.
    • Our work complements this approach through completeness-aware robust preference optimization, addressing inaccurate preference margin caused by omitted rubric criteria.
AuthorsOpenReview ↗

Response to W2 & Q2

[W2&Q2] Was Uniform Robust's budget independently tuned, and is its performance gap to Ours-KL consistent across random seeds?

Response: We thank the reviewer for this useful suggestion. Following the reviewer’s suggestion, we additionally evaluate both methods over the different budget with repeated experiments.

  • Specifically, we evaluate across ρ∈[0,0.0005,0.001,0.005,0.01]\rho \in [0,0.0005,0.001,0.005,0.01] and report mean ± standard deviation across three random seeds (13, 42, and 100).
Robustness budget ρ\rhoUniform Robust: Validation Pref. Acc. (%)Ours-KL: Validation Pref. Acc. (%)
087.46 ± 0.0287.46 ± 0.02
0.000587.49 ± 0.0387.64 ± 0.04∗^{*}
0.00187.78 ± 0.0388.26 ± 0.02∗^{*}
0.00587.52 ± 0.0287.75 ± 0.03∗^{*}
0.0187.44 ± 0.0187.51 ± 0.03∗^{*}

{*} means statistical significant under paired t-test with p<0.01p<0.01. Independent tuning selects the same budget for both methods and recovers the budget used in the original ablation. Ours-KL retains its validation advantage with small variation across seeds, supporting the robustness of our method.


We are eager to hear your feedback. We’d deeply appreciate it if you could let us know whether your concerns have been addressed.

Dear Reviewer epgr,

As the discussion deadline approaches, we are wondering whether our responses have properly addressed your concerns? Your feedback would be extremely helpful to us. If you have further comments or questions, we hope for the opportunity to respond to them.

Many thanks,

The authors of #40463

Reviewer_SYC4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a robust preference optimization framework to address preference errors caused by incomplete rubrics. The method estimates instance-specific rubric completeness by measuring the rank agreement between response shifts and rubric-score changes induced by parameter perturbations. These completeness scores control a feasible set around the nominal preference probabilities, and the policy minimizes the worst-case training loss over this set. The framework can be integrated with different margin-aware preference optimization methods, and experiments report consistent improvements across three datasets.

Critical strengths
  1. The research question has practical significance. It highlights that even when every observed criterion is scored accurately, omitted criteria can still introduce preference errors, addressing a practical problem in rubric-based preference alignment.
  2. The framework is applicable to different margin-aware preference optimization methods, including MMPO, ODPO, and Scaled DPO. Experiments report consistent improvements across multiple datasets and different rubric construction methods.
Critical weaknesses
  1. The reliability of the rubric completeness score as a proxy for rubric completeness is not sufficiently established. A limited number of parameter perturbations does not guarantee coverage of all relevant quality dimensions, and larger response shifts do not necessarily correspond to larger changes in response quality. Therefore, rank agreement between response shifts and rubric-score changes may not reliably indicate rubric completeness. The justification for the completeness-controlled feasible set also remains limited. The inner maximization selects the worst-case preference probabilities for the current policy, rather than estimating the preferences under a complete rubric set. The theoretical analysis does not establish that these complete-rubric preferences lie within the feasible set. Moreover, the worst-case preference probabilities may reverse an originally correct preference ordering, and the impact of such reversals is not directly examined.

  2. The experiments do not directly establish whether the framework mitigates the adverse effects of incomplete rubrics. The main rubric-based evaluation still uses dataset-provided or generated rubrics, so improvements under these rubrics do not necessarily imply improvements on omitted criteria.

Questions for authors
  1. How do the selected gradient-informed perturbation directions capture the quality dimensions relevant to each instance? Since response shifts do not necessarily correspond to changes in response quality, how do the authors verify that the rubric completeness score distinguishes omitted criteria from variations in expression that preserve response quality?
  2. The worst-case preference probabilities may deviate from the preferences under a complete rubric set and may even reverse an originally correct preference ordering. How do the authors justify the completeness-controlled feasible set and assess whether such reversals impair correct preference supervision?
  3. The main rubric-based evaluation uses dataset-provided or generated rubrics. How do the authors establish that the improvements result from mitigating preference errors caused by incomplete rubrics? Could the authors withhold selected reference criteria during training, evaluate the trained models using the full reference rubric set, and separately report performance on the withheld criteria?
4 comments
AuthorsOpenReview ↗

Response to W1 & Q1

We sincerely thank the reviewer for the valuable comments. We address each concern below.

[W1.1] How do the method capture the quality dimensions relevant to each instance? Larger response shifts do not necessarily correspond to larger changes in response quality.

Response: We thank the reviewer for this valuable comment. First, we use Fisher-inspired gradient directions to capture the directions corresponding to the larger response shift, as the proxy of quality changes. Then, we agree that response distance is not a measure of response quality, therefore we validate larger response shifts tend to correspond to larger quality changes below.

  • Evaluate verifiable and open-ended tasks.

    • We use APPS prompts dataset with executable tests and SummEval dataset with open-ended task.
  • Measure the quality change.

    • For nominal response yi0y_{i0} and perturbed response yihy_{ih}, define DihQD^Q_{ih} as quality change measurement DihQ=∣Qi(yih)−Qi(yi0)∣,D^Q_{ih}=\left|Q_i(y_{ih})-Q_i(y_{i0})\right|,

    • On APPS, the quality score QQ is the ratio of passed test Qi(y)=number of tests passednumber of total tests.Q_i(y)=\frac{\text{number of tests passed}}{\text{number of total tests}}.

    • On SummEval, Qi(y)Q_i(y) is the rating judged by GPT-4o using provided evaluation creteria in dataset.

  • Assess the association between response shifts and quality changes.

    • For each prompt ii, we compare the rankings induced by DihYD^Y_{ih} (response shift) and DihQD^Q_{ih} (quality change) across perturbation directions, in order to exam whether larger response shifts more consistently correspond to larger changes in response quality.
    • Let nicn_i^c and nidn_i^d denote the numbers of concordant pairs (matching orders, i.e., larger response shift corresponds to larger quality change) and discordant pairs (opposite orders, i.e., larger response shift corresponds to smaller quality change). We report Kendall’s $\tau_b$ and pair concordance nic/(nic+nid)n_i^c/(n_i^c+n_i^d).
DatasetMean Kendall's τb\tau_bPair concordance (%)
APPS0.58883.32
SummEval0.40174.09

Response shifts are positively associated with quality changes on both tasks, and the number of concordant pairs is substantially larger than the number of discordant pairs. These results support larger response shifts tend to correspond to larger quality changes.

[W1.2] The inner maximization selects worst-case preferences rather than the complete-rubric preferences, whose inclusion in the feasible set is not guaranteed.

Response: We thank the reviewer for the insightful question.

  • Even when the complete preference is outside our feasible set, a positive robustness budget ensures that the set contains a preference closer to complete preference than the nominal preference is. It is straightforward since a sufficiently small step from the nominal preference toward the complete preference remains feasible.

  • We provide a bound to show this. Specifically, let p,q⋆∈[ϵ,1−ϵ]np,q^\star\in[\epsilon,1-\epsilon]^n denote the nominal and complete preferences, respectively, and define

    \qquad d(u,v)=\frac1n\sum_i|u_i-v_i|.$$ When $q^\star\notin\mathcal U$ and $\rho>0$, we have $J>\rho>0$. For either KL or Wasserstein distance, convexity implies that $\widetilde q=(1-\rho/J)p+(\rho/J)q^\star$ belongs to $\mathcal U$. Therefore, $$\inf_{q\in\mathcal U}d(q^\star,q) \le \left(1-\frac{\rho}{J}\right)d(q^\star,p) <d(q^\star,p).$$ This gives a strictly smaller approximation error even without full coverage.
  • We can also bound the complete-preference training loss without assuming full coverage. Specifically, for MMPO, let Rθ(q)R_\theta(q) denote the average training loss and Bθ=max⁡i∣Δθ,i∣B_\theta=\max_i|\Delta_{\theta,i}|, where Δθ,i\Delta_{\theta,i} is the reward difference in MMPO. We have

    Rθ(q⋆)≤max⁡q∈URθ(q)+Bθinf⁡q∈Ud(q⋆,q).R_\theta(q^\star) \le \max_{q\in\mathcal U}R_\theta(q) + B_\theta\inf_{q\in\mathcal U}d(q^\star,q).

    Therefore, the complete-preference loss is bounded by the optimized worst-case loss plus a term controlled by the remaining approximation error.

AuthorsOpenReview ↗

Response to W2, Q2 & Q3

[Q2] Worst-case preferences may reverse the original ordering, and the impact of these reversals has not been directly examined.

Response: We thank the reviewer for this useful comment. Missing criteria may affect both preference strength and direction, our framework handles preference reversals, as already detailed in Appendix C.

To examine the empirical impact of these reversals, we add additional experiments. We compare MMPO + Ours-KL with a no-reversal variant on UltraFeedback and WildChecklists.

  • No-reversal variant.

    • Fix the original winner/loser ordering and let m(r)=γ−1logit⁡(r)m(r)=\gamma^{-1}\operatorname{logit}(r). For a negative adversarial margin, restore the original direction and its magnitude:

    m~=m(q)ifm(q)≥0,elsem=min⁡{∣m(p)∣,∣m(q)∣},\widetilde m = m(q) \quad \text{if} \quad m(q) \geq 0, \text{else} \quad m = \min\{|m(p)|,|m(q)|\}, q~=σ(γm~).\widetilde q=\sigma(\gamma\widetilde m).

DatasetMethodReversal rate (%)Pref. Acc. (%)Avg. score
UltraFeedbackMMPO + Ours-KL, reversals allowed16.3788.268.80
UltraFeedbackMMPO + Ours-KL, reversals disallowed087.928.77
WildChecklistsMMPO + Ours-KL, reversals allowed20.0879.778.47
WildChecklistsMMPO + Ours-KL, reversals disallowed079.338.29

Allowing reversals (as already shown in Appendix C) improves preference accuracy and average scores on both datasets.

[W2&Q3] How do the authors establish that the improvements result from mitigating preference errors caused by incomplete rubrics

Response: We thank the reviewer for this valuable suggestion.

  • As the reviewer suggested, we withhold selected reference criteria during training, evaluate the trained models using the full reference rubric set, and separately report performance on the withheld criteria.
  • Specifically, we use UltraFeedback's four criteria: instruction following, truthfulness, honesty, and helpfulness. We consider the following settings:
SettingCriteria retained during trainingWithheld criteria
NoneInstruction following, Honesty, Helpfulness, TruthfulnessN/A
H1Instruction following, Honesty, HelpfulnessTruthfulness
H2Instruction following, HelpfulnessTruthfulness, Honesty
  • Train with the retained criteria. We construct training using only the retained criteria in each setting.

  • Evaluate with the full rubric: we compare MMPO and MMPO + Ours-KL under each setting using the full reference rubric.

  • Evaluate with the withheld criteria: we compare MMPO and MMPO + Ours-KL under each setting using the full rubric and the withheld rubric.

Training settingMethodFull: Pref. Acc. (%)Full: Avg. scoreWithheld criteria: Avg. score
NoneMMPO87.468.75N/A
NoneMMPO + Ours-KL88.268.80N/A
H1MMPO80.277.838.07
H1MMPO + Ours-KL83.508.038.39
H2MMPO73.267.187.72
H2MMPO + Ours-KL77.607.588.10

Both methods degrade as more training criteria are withheld, but our method's advantage in preference accuracy and full-rubric scores increases with more criterion omission, supporting improved robustness to incomplete rubric supervision.


Please let us know if we have properly addressed your questions and if you have further questions -- thank you so much!

Thank you for the additional analyses and experiments. The response-shift analysis and withheld-criteria evaluation provide useful evidence and address important parts of my concerns. I still have the following reservations.

  1. The new loss bound provides theoretical support, but its practical significance remains unclear because the remaining approximation-error term is not shown to be sufficiently small.

  2. The no-reversal variant restores the original direction and caps the margin magnitude using the nominal margin. This is a post-processing rule rather than re-solving the inner maximization under a no-reversal constraint. The performance difference therefore does not isolate the effect of allowing reversals.

  3. The withheld-criteria experiments support improved robustness over MMPO. However, without an independently tuned Uniform Robust baseline under the same omission settings, they do not isolate the additional benefit of instance-specific completeness estimation.

AuthorsOpenReview ↗

Response to Additional Concerns

We sincerely thank the reviewer for the detailed assessment and valuable suggestions! We are glad that our rebuttal has already addressed important parts of your concerns. We address the remaining concern below.

[1] Practical significance remains unclear because the remaining approximation-error term is not shown to be sufficiently small.

Response: We appreciate the reviewer’s valuable suggestion. We empirically quantify approximation-error in our controlled criterion-omission experiments, following the setting in the response to [W2&Q3].

  • For each omission setting, we use the MMPO + Ours-KL trained with the retained criteria setting H1/H2.
  • We use preferences derived from complete reference criteria as q⋆q^\star to compute the full-reference loss Rθ(q⋆)R_\theta(q^\star), and use preferences derived from retained criteria for the worst-case loss max⁡q∈URθ(q)\max_{q\in\mathcal U}R_\theta(q) and the approximation-error term Bθinf⁡q∈Ud(q⋆,q)B_\theta\inf_{q\in\mathcal U}d(q^\star,q).
  • Since our method achieved the best performance at ρ=0.001\rho=0.001 under H1 and ρ=0.005\rho=0.005 under H2, we report the results under theses two budget.
ρ\rhoSettingRθ(q⋆)R_\theta(q^\star)max⁡q∈URθ(q)\max_{q\in\mathcal U}R_\theta(q)BθB_\thetainf⁡q∈Ud(q⋆,q)\inf_{q\in\mathcal U}d(q^\star,q)Bθinf⁡q∈Ud(q⋆,q)B_\theta\inf_{q\in\mathcal U}d(q^\star,q)
0.001H10.6930.6970.6130.00130.0008
0.001H20.6940.7061.3520.01020.0138
0.005H10.7010.7030.67700
0.005H20.6920.7071.26800
  • At ρ=0.001\rho=0.001, the approximation-error term amounts to 0.122% of the full-reference training loss under H1 and 1.997% under H2. At ρ=0.005\rho=0.005, the term is zero in both settings, since U\mathcal{U} is large enough to include the ground truth. This approximation-error term is sufficiently small, and these quantiles are consistent with the theoretical bound.

  • In addition, we can see the bound is tight in practice, the difference between the left-hand side and right-hand side is small, further showing the practical meaning.

[2] The no-reversal variant is a post-processing rule rather than re-solving the inner maximization under a no-reversal constraint.

Response: We thank the reviewer for this useful comment. To isolate the effect of allowing reversals, we add a new ablation that imposes the no-reversal constraint in the inner maximization.

  • For the new no-reversal variant, we train MMPO + Ours-KL with the restricted feasible set UNR\mathcal U_{\mathrm{NR}}, where UNR=UKL∩[q:qi≥1/2, ∀i].\mathcal U_{\mathrm{NR}} = \mathcal U_{\mathrm{KL}}\cap [q:q_i\ge1/2,\ \forall i].
DatasetMethodReversal rate (%)Pref. Acc. (%)Avg. score
UltraFeedbackMMPO + Ours-KL, reversals16.3788.268.80
UltraFeedbackMMPO + Ours-KL, no reversals (old)087.928.77
UltraFeedbackMMPO + Ours-KL, no reversals (new)087.518.76
WildChecklistsMMPO + Ours-KL, reversals20.0879.778.47
WildChecklistsMMPO + Ours-KL, no reversals (old)079.338.29
WildChecklistsMMPO + Ours-KL, no reversals (new)079.018.25

Allowing reversals yielded higher preference accuracy and average scores on both datasets. This support the the design of our method.

[3] Need independently tune Uniform Robust baseline under the same omission settings to verify the benefit of instance-specific completeness estimation.

Response: We thank the reviewer for this valuable comment. We add experiments by independently tuning the robustness budget for Uniform Robust under each omission setting.

SettingBudget ρ\rhoMMPOUniform RobustOurs-KL
None087.46 ± 0.0287.46 ± 0.0287.46 ± 0.02
None0.000587.46 ± 0.0287.49 ± 0.0387.64 ± 0.04∗^{*}
None0.00187.46 ± 0.0287.78 ± 0.0388.26 ± 0.02∗^{*}
None0.00587.46 ± 0.0287.52 ± 0.0287.75 ± 0.03∗^{*}
None0.0187.46 ± 0.0287.44 ± 0.0187.51 ± 0.03
H1080.27 ± 0.0380.27 ± 0.0380.27 ± 0.03
H10.000580.27 ± 0.0381.15 ± 0.0481.86 ± 0.02∗^{*}
H10.00180.27 ± 0.0382.64 ± 0.0383.50 ± 0.01∗^{*}
H10.00580.27 ± 0.0382.04 ± 0.0383.28 ± 0.03∗^{*}
H10.0180.27 ± 0.0381.47 ± 0.0282.70 ± 0.04∗^{*}
H2073.26 ± 0.0573.26 ± 0.0573.26 ± 0.05
H20.000573.26 ± 0.0575.44 ± 0.0176.82 ± 0.02∗^{*}
H20.00173.26 ± 0.0576.72 ± 0.0377.60 ± 0.02∗^{*}
H20.00573.26 ± 0.0576.83 ± 0.0377.96 ± 0.03∗^{*}
H20.0173.26 ± 0.0576.09 ± 0.0677.13 ± 0.05∗^{*}

Ours-KL outperformed the baseline and Uniform Robust at every budget under each omission setting. These results verify the benefit of instance-specific completeness estimation.


We are very willing to know whether there are remaining concerns? If all concerns are addressed, would you please kindly raise your score? We appreciate again for the time and efforts!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank the reviewers for their valuable comments. Additions and substantive updates are highlighted in blue in the revised PDF file.

  1. Response shifts and quality changes (§6.3, Table 5; Appendix A.3). We add evaluations on APPS with executable tests and SummEval with open-ended tasks. Mean Kendall’s τb\tau_b is 0.588 / 0.401, and pair concordance is 83.32% / 74.09%. Response shifts are positively associated with quality changes on both tasks, supporting that larger response shifts tend to correspond to larger quality changes.
  2. Strengthened theoretical analysis (§5; Appendices E.1–E.3).
    • Updated Theorem 1 and proof (E.1–E.2). We refine the preference-error lower bound. Under the stated assumptions, it is strictly positive even when ai≪bia_i\ll b_i under mild condition. We also clarify that the sensitivity condition concerns the selected probes, without requiring a global bi-Lipschitz mapping.
    • Guarantees without full coverage (E.3). Even when the complete preference is outside our feasible set, a positive robustness budget ensures that the set contains a preference closer to the complete preference than the nominal preference is. We also bound the complete-preference training loss by the optimized worst-case loss plus a term controlled by the remaining approximation error.
  3. Evidence under criterion omission.
    • Learning with withheld criteria (§6.2, Table 3). We withhold selected reference criteria during training, evaluate using the full reference and withheld rubric set. Our method’s advantage increases with more criterion omission, supporting improved robustness to incomplete rubric supervision.
    • Completeness and preference errors (Appendix A.4, Table 7). We progressively remove criteria. Both incompleteness 1−s1-s and preference error increase as more criteria are removed, and lower completeness scores are associated with larger preference errors.
  4. Additional controls for the completeness score (Appendices A.5–A.7).
    • Length-only rubrics (Table 8). Length-only scoring yields substantially lower completeness scores and rarely produces high-ss instances, suggesting that response length alone does not induce high completeness scores.
    • Paraphrased responses (Table 9). We paraphrase perturbed responses while preserving meaning and recompute both shifts. Completeness scores remain stable under the paraphrases.
    • Estimation noise (Table 10). We repeat the full calibration pipeline five times. The mean standard error remains below 0.05, suggesting limited estimation noise.
    • Fixed-dimension control (Table 11). We compare criterion deletion with random-score replacement while retaining all criterion dimensions. The results support that Kendall’s τb\tau_b reflects the loss of informative rubric-score signals rather than tie rate alone.
  5. Comparisons with robust preference learning and discussion on AIR (§2, Table 2; Appendix F.1). We add comparisons with cDPO, rDPO, and Dr.DPO.
    • MMPO + Ours-KL achieves higher preference accuracy and average rubric scores than these baselines across all three datasets. Our work also complements AIR through completeness-aware robust preference optimization, while AIR improves preference-data construction and keeps the training objective fixed to standard DPO.
  6. Preference reversals and independent budget tuning (Appendices G.3–G.4).
    • Preference reversals (Table 14). We compare MMPO + Ours-KL with a no-reversal variant. Allowing reversals improves preference accuracy and average scores on both datasets.
    • Budget tuning and random seeds (Table 15). We independently tune Uniform Robust and our method and report mean ± standard deviation across three random seeds. Our method retains its validation advantage across seeds.
  7. Additional benchmark evaluations (§6.4; Appendix G.5, Table 16). We add evaluations on AlpacaEval and Arena-Hard, reporting length-controlled win rate and dataset-specific rubric win-rate score. Adding our method improves both benchmark scores.
  8. Method and evaluation clarifications.
    • Applicability beyond MMPO (§4.3). We clarify the adaptive soft-target interpretation for MMPO. The completeness-aware uncertainty set also applies to methods such as ODPO and Scaled DPO, modifying offsets or pair weights together with preference directions.
    • Win rate and rating aligment (§6.1; Appendix G.6, Table 17). We clarify that the original pairwise comparison has no tie option, while average rubric scores use integers from 0 to 10. We add finer ratings with a 0.2 step size on all three datasets. Our method shows larger gains in this setting.

We hope these revisions can fully address reviewers' concerns, and we are very eager to hear the feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.