- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary: To address preference labeling errors caused by incomplete rubrics, this paper generates response variants through gradient direction perturbation, estimates instance-level rule completeness using the "ranking consistency between response changes and rule score changes," and then constructs an uncertain set of preference probabilities weighted by completeness for min-max robust training. It can be integrated with various margin-aware preference optimization methods.
Critical strengths
Strengths
The problem has real-world significance. Rubric incompleteness is a real pain point in rubric-based alignment, and the quantification in Table 1 is convincing.
The idea of "adjusting the robust radius with instance-level completeness" is novel and plug-and-play. The Uniform/Shuffled ablation design is reasonable, and the occlusion experiment directly verifies the surrogate metric.
Critical weaknesses
Weaknesses
The effectiveness of the completeness surrogate is insufficiently demonstrated.
The Wasserstein distance of the response representation measures superficial or semantic changes, not qualitative changes.
A fragmented rule with only a "length" parameter may be highly correlated with the embedding distance; while a complete rule may not have a score that remains unchanged in the face of rewrite changes. Theorem 1 depends on assumption (12), which states that "the score variation of the complete rule is bidirectionally Lipschitz related to the embedding distance," a premise that needs to be proven.
When a_i ≪ b_i, the margin g_i is likely to be 0, rendering the lower bound meaningless.
The estimation is noisy.
Each sample has at most 8 perturbation directions, and the standard error of Kendall τ is approximately 0.29 when k=8. Adding the scoring noise from the LLM judges and the randomness of generation, s_i is likely primarily noise.
The perturbation-generated answers are independent of the training pair (y_w, y_l). The rule's ability to distinguish perturbation answers does not mean it can distinguish the training pair.
The masking experiment suffers from mechanical confounding.
Masking rules shorten the dimension of D^R and increase tie, thus mechanically decreasing Kendall τ with tie correction. A comparison is needed that "keeps the number of rules constant and replaces them with irrelevant ones."
Key relevant work and baselines are missing.
Performing worst-case optimization around q essentially falls under the category of "robust DPO with noisy preference labels." cDPO (label smoothing), rDPO (Provably Robust DPO, Chowdhury et al., 2024), Dr. DPO (divided robust DPO), etc., are not cited or compared; only the paper's own uniform robust comparison is insufficient.
For MMPO, the closed-form solution q* = σ(logit p − Δ/(λ·s_i)) indicates that its effect is close to an adaptive label smoothing.
The evaluation exhibits cyclicality.
In the setting using generated rules, the win rate and average score are also calculated using generated rules, while the paper's premise is precisely that these rules are incomplete.
Independent evaluation methods such as human-written rules, human preferences, or AlpacaEval/Arena-Hard should be used.
Additionally, the average rating on UltraFeedback only increased from 8.75 to 8.80, but the win rate reached 68%. How draws were handled needs to be explained.
Questions for authors
See Weaknesses
4 comments
Response to W4, W5, W6 & W7
[W4] Even a complete rubric may change its scores under paraphrasing.
Response: We thank the reviewer for this valuable comment. We add the following experiment to show the sensitivity to paraphrasing.
-
Paraphrase perturbed responses.
- On RubricBench, we use GPT-4o to paraphrase each existing perturbed response, instructing it to change wording and syntax while preserving meaning.
-
Recompute shifts.
- Using the same frozen encoder and rubric evaluator, we recompute response distances and rubric-score shifts relative to the nominal response.
-
Re-rank and recompute completeness.
- We independently re-rank the probe directions by the two measures and recompute the :
| Responses | Mean | Mean |
|---|---|---|
| Original | 0.8090 | 0 |
| Paraphrased | 0.8013 | 0.0263 |
Completeness scores remain relatively stable under the tested paraphrases. Together, the experiments support the usefulness of our score design.
[W5] Is the completeness score dominated by noise
Response: We thank the reviewer for raising this concern.
-
The value of approximately 0.29 assumes that response shifts and rubric-score shifts are unrelated and no tie among the values of either shift.
-
In our setting, these shifts show positive rank association and contain ties. Therefore, the value of approximately 0.29 cannot be taken as the standard error of our method.
-
To further assess estimation noise, we independently repeat the full calibration pipeline five times with different , and report the mean standard error of .
| Maximum directions | Mean Standard error of ↓ |
|---|---|
| 4 | 0.042 |
| 8 | 0.037 |
| 16 | 0.034 |
The mean standard error remains below 0.05, suggesting limited noise in our method.
[W6] A rubric’s ability to distinguish perturbation-generated responses does not necessarily imply that it can correctly distinguish the original training pair.
Response: We thank the reviewer for raising this point. We clarify that the perturbation-generated answers are used to assess rubric incompleteness and determine robustness weights, not to infer the preference between the training responses. In other words, the rubric is decided by prompt, instead of and , so we assess its completeness using response variations for the same prompt.
Specifically:
-
The actual training pair determines the nominal preference and the optimization loss.
-
The perturbation-generated answers aim to calculate score to determine how complete the rubric is.
[W7] Removing criteria reduces the dimension of rubric-score vectors and may increase ties. A control that replaces criteria with irrelevant ones while keeping the total criterion count fixed is needed.
Response: We thank the reviewer for this useful suggestion. We agree that removing criteria may increase tie frequency. However, an increase in ties does not mechanically decrease Kendall's correlation. With tie correction,
Here, and count concordant and discordant pairs, while and count pairs tied only in or , respectively.
-
Counterexample: With the other relations unchanged, the correlation can increase.
\qquad \tau_b:0.333\rightarrow0.816.$$
To empirically address reviewer's concern, we add additional experiment below.
-
Compare deletion and random-rubric replacement. On the same RubricBench examples, we apply deletion or random-rubric replacement to the first criteria in their original order:
-
Deletion: We remove the selected criteria from the rubric-score vectors and compute rubric-score distances using the remaining criteria.
-
Random-rubric replacement: We retain all criterion dimensions but replace the selected criteria’s scores with independently sampled random rubrics scores.
-
-
Experimental results. We recompute rubric-score distances and Kendall’s and report tie rates and mean for both settings.
| Deletion | Random | |||
|---|---|---|---|---|
| Tie rate | Mean | Tie rate | Mean | |
| 0 | 19.67% | 0.799 | 19.67% | 0.799 |
| 1 | 24.81% | 0.682 | 20.33% | 0.708 |
| 2 | 28.90% | 0.459 | 14.35% | 0.433 |
| 3 | 33.93% | 0.233 | 14.08% | 0.267 |
| 4 | 38.84% | 0.109 | 16.59% | 0.133 |
| 5 | 44.96% | 0.083 | 19.43% | 0.105 |
These results support that our completeness measure Kendall’s is more related to the loss of informative rubric-score signals, rather than tie frequency.
Response to W8, W9, W10 & W11
[W8] The method is related to robust preference optimization under noisy labels, and comparisons with cDPO, rDPO, and Dr.DPO are needed.
Response: We thank the reviewer for this valuable suggestion. These methods share the goal of mitigating unreliable supervision, but our method specifically targets inaccurate preference caused by incomplete rubrics.
-
cDPO: Fixed label smoothing.
-
rDPO: Loss correction under random preference flips.
-
Dr.DPO: Preference-pair reweighting through robust loss aggregation.
-
Ours: Instance-specific rubric completeness constrains adversarial deviations from rubric-derived preference probabilities. Omitted criteria can distort both preference strength and direction even when all observed criteria are scored accurately.
-
Added experiments. We evaluate all methods on UltraFeedback (UF), WildChecklists (WC), and HelpSteer2 (HS2), reporting preference accuracy (%) and average rubric score (Avg).
| Method | UF Pref. Acc. | UF Avg | WC Pref. Acc. | WC Avg | HS2 Pref. Acc. | HS2 Avg |
|---|---|---|---|---|---|---|
| cDPO | 85.73 | 8.02 | 77.19 | 7.03 | 61.98 | 7.55 |
| rDPO | 86.77 | 8.26 | 78.24 | 7.73 | 63.08 | 7.71 |
| Dr.DPO | 86.12 | 8.11 | 77.96 | 7.70 | 63.02 | 7.68 |
| MMPO + Ours-KL | 88.26 | 8.80 | 79.77 | 8.47 | 64.86 | 7.83 |
MMPO + Ours-KL achieves the highest preference accuracy and average rubric score among the compared methods across all three datasets.
[W9] Closed form in MMPO is related to adaptive label smoothing.
Response: We thank the reviewer for this valuable suggestion.
-
We agree that the MMPO is a special case of our method, which is close to an adaptive soft-target interpretation.
-
However, our approach is a general robust preference optimization framework that also applies to ODPO and Scaled DPO. Using the same completeness-aware uncertainty set, adversarial preferences modify offsets in ODPO and pair weights in Scaled DPO, together with preference directions in both cases, instead of the adaptive label smoothing.
[W10] Independent evaluations, such as human-written rubrics, human preferences, or AlpacaEval and Arena-Hard, are needed.
Response: We thank the reviewer for this useful suggestion. We add additional evaluations on AlpacaEval and Arena-Hard.
-
Evaluation protocols and score interpretation.
-
AlpacaEval: length-controlled win rate. We evaluate each model and compare its responses against the benchmark’s reference responses. We report the length-controlled (LC) win rate (%), which estimates preference over the reference model.
-
Arena-Hard: win-rate score based on dataset-specific rubrics. We evaluate each model and compares its responses against the benchmark’s reference responses. We report a win-rate score (%) against the reference model.
-
-
Experimental results.
| Method | AlpacaEval (LC) | Arena-Hard |
|---|---|---|
| DPO | 53.44 | 83.27 |
| MMPO | 53.64 | 85.19 |
| MMPO + Ours-KL | 54.31 | 87.36 |
| ODPO | 54.02 | 85.33 |
| ODPO + Ours-KL | 55.89 | 86.94 |
| Scaled DPO | 54.73 | 85.18 |
| Scaled DPO + Ours-KL | 55.27 | 86.74 |
Adding Ours method improves both benchmark scores. These consistent gains support the effectiveness of our method.
[W11] The small increase in average score on UltraFeedback appears inconsistent with a pairwise win rate of approximately 68%, and the treatment of ties needs clarification.
Response: We thank the reviewer for this valuable comment. The key distinction is preference frequency versus score magnitude, and the original pairwise protocol did not include a tie option.
-
Win rate.
- We forced LLM judgments to choose one answer in our method and its corresponding baseline. Note that there is no tie option. Win rate measures how frequently our responses are preferred.
-
Average score.
- Each response is rated against the rubric from 0-10 intergers. Note that model can rate both 8.3 and 8.1 to 8 to reduce the performance gap. So a small gain is compatible with a relatively high pairwise win rate.
-
Added experiments.
- To validate the above, we add experiment under a more fine-grained rating, from 0-10 with 0.2 as the step size, on all three datasets. The results are below.
| Method | UF Coarse | UF Fine | WC Coarse | WC Fine | HS2 Coarse | HS2 Fine |
|---|---|---|---|---|---|---|
| MMPO | 8.75 | 8.79 | 8.05 | 8.26 | 7.74 | 7.88 |
| MMPO + Ours-KL | 8.80 | 8.94 | 8.47 | 8.70 | 7.83 | 8.06 |
| ODPO | 8.61 | 8.68 | 7.15 | 7.44 | 7.69 | 7.83 |
| ODPO + Ours-KL | 8.63 | 8.85 | 7.46 | 7.76 | 7.96 | 8.21 |
| Scaled DPO | 8.63 | 8.72 | 6.81 | 6.93 | 7.71 | 7.97 |
| Scaled DPO + Ours-KL | 8.96 | 9.37 | 7.02 | 7.26 | 7.88 | 8.30 |
The results show that our method has more advantages in fine-grained setting, indicating that it is reasonable to obtain 68% win rate under small rating gain in coarse setting (since we naturally obtain small difference).
Please let us know if we have properly addressed your questions and we are happy to discuss more!
Dear Reviewer gyyt,
As the discussion deadline approaches, we are wondering whether our responses have properly addressed your concerns? Your feedback would be extremely helpful to us. If you have further comments or questions, we hope for the opportunity to respond to them.
Many thanks,
The authors of #40463
Response to W1, W2 & W3
We sincerely thank the reviewer for the detailed assessment and valuable suggestions. We address each concern below.
Response: We thank the reviewer for this valuable comment. We agree that response Wasserstein distance is not a measure of response quality. We appreciate the reviewer’s suggestion and conduct an additional emperical analysis to examine whether larger response shifts tend to correspond to larger quality changes.
Evaluate verifiable and open-ended tasks.
Measure the quality change.
For nominal response yi0 and perturbed response yih, define DihQ as quality change measurement DihQ=∣Qi(yih)−Qi(yi0)∣,
On APPS, the quality score Q is the ratio of passed test Qi(y)=number of total testsnumber of tests passed.
On SummEval, Qi(y) is the rating judged by GPT-4o using provided evaluation creteria in dataset.
Assess the association between response shifts and quality changes.
Response shifts are positively associated with quality changes on both tasks, and the number of concordant pairs is substantially larger than the number of discordant pairs. These results support larger response shifts tend to correspond to larger quality changes.
Response: We thank the reviewer for this valuable comment. Individual distances or rubric scores may change without changing s, provided both rankings are preserved. We examine reviewer's concerns through additional experiments.
Length-only rubric.
DihR,len=∣L(yih)−L(yi0)∣.
Recompute completeness.
Length-only scoring yields substantially lower completeness scores and rarely produces high-s instances. This suggests that response length alone does not induce the high completeness scores.
Response: We thank the reviewer for the insightful question.
The condition in Eq. (12) means that every nonzero response shift changes at least one complete-rubric score. This is the required nondegeneracy: Dih⋆>0 whenever DihY>0. For example, a change in factual correctness should appear in the corresponding complete-rubric score. Based on this example, this is a mild condition for task-relevant response changes.
We provide an example in which Eq. (12) holds. Specifically, let DihY>0 and Dih⋆>0, we define
\qquad b_i=\max_h\frac{D^\star_{ih}}{D^Y_{ih}}.$$ We have $0<a_i\le b_i<\infty$ and hence Eq. (12). Therefore, the constants exist in this case without requiring a global bi-Lipschitz mapping.We refine the lower bound, so that even ai≪bi, we can obtain a strictly positive lower bound. Specifically, it suffices that DihR<aiDihY for at least one probe. This condition is also mild. Then, for weights ωi=0, using the lower inequality in Eq. (12), the same proof gives a stronger bound with a strictly positive right-hand side:
E[(qi⋆−pi)2]≥∑h=1ki[ai2(DihY)2−(DihR)2]+⋅kiαi2κi∥ωi∥22.