- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to identify failures in preference models before they are exploited during LLM alignment. The authors use disagreement among an ensemble of preference models as a proxy for evaluation errors and formulate adversarial comparison generation as a zero-sum Stackelberg game. A Leader minimizes disagreement, while a Follower observes the Leader's response and generates an alternative that maximizes it. The paper develops SMIDA with convergence guarantees in the tabular setting and extends it to neural policies through Deep-SMIDA. Experiments show that the trained policies improve the yield of high-disagreement, oracle-confirmed adversarial comparisons from 4.4% to 9.6%.
Critical strengths
-
Interesting problem formulation. The paper addresses an important weakness of preference-based alignment: the evaluator itself may be unreliable. Framing error discovery as an adversarial game is a meaningful perspective.
-
Solid theoretical contribution with encouraging empirical results. The equilibrium analysis and last-iterate convergence result are valuable. The experiments also provide evidence that ensemble disagreement can help identify preference-model errors.
Critical weaknesses
-
The necessity of the Stackelberg formulation is not sufficiently established. The main experiments do not clearly isolate the benefit of jointly training the Leader and Follower over simpler alternatives, such as a fixed Leader with a disagreement-maximizing Follower or a single adversarial generator.
-
Limited validation of actual preference errors. The evaluation relies on a synthetic oracle rather than human judgments. It is therefore unclear how reliably the discovered adversarial comparisons correspond to genuine human preference violations.
Questions for authors
-
Could the authors compare Deep-SMIDA against a fixed-Leader or single-policy disagreement-maximization baseline under matched training budgets, using the same oracle-confirmed adversarial yield metric?
-
How sensitive are the results to the choice of synthetic oracle? Would a small human evaluation or an independent oracle lead to similar conclusions?
-
The convergence guarantee applies to the tabular setting. Could the authors provide additional evidence of training stability and convergence behavior for Deep-SMIDA across multiple random seeds?
4 comments
[Continued response to toAN]
Limited validation of actual preference errors. The evaluation relies on a synthetic oracle rather than human judgments. It is therefore unclear how reliably the discovered adversarial comparisons correspond to genuine human preference violations.
How sensitive are the results to the choice of synthetic oracle? Would a small human evaluation or an independent oracle lead to similar conclusions?
We validated the oracle against both a human and an independent oracle. A human annotator and a frontier model (Opus 5.5) each labelled 100 comparisons from Sec. 5.3.2, and agreed with the oracle on 88.9% (95% CI [81.2%, 93.7%]) and 88.0% (CI [79.8%, 93.2%]) of comparisons. The oracle is also self-consistent (85.0% agreement between two independent queries) and agrees with the original UltraFeedback labels on 85.5% of comparisons. The two judges agree with the oracle at similar rates, so our conclusions do not depend on a single judge.
The convergence guarantee applies to the tabular setting. Could the authors provide additional evidence of training stability and convergence behavior for Deep-SMIDA across multiple random seeds?
We agree that the guarantee is tabular. Our existing runs provide evidence on the stability of Deep-SMIDA, while varying the regularization strength . In the sweep of Appendix G.2, we trained six configurations against the same ensemble, with ranging from to . All six completed training without divergence. The five configurations evaluated on held-out prompts reach yields of 25.8–43.7%, compared with 10.0% for the reference policies. Policy drift and candidate yield decrease steadily in , as the closed form of Theorem 4.2 predicts. We do bring the reviewer's attention to the comments we have made in Section 4.3 in the paper. Preliminary experiments showed that training was highly sensitive to the learning rates and and often became un-stable; standard warm-up techniques did not resolve this instability. To address this, we linearly increase the ratio from 1 to its target , while warming up the Leader’s learning rate linearly from to . We would point the reviewer to Appendix D in our paper for more details on this.
We thank the reviewer for encouraging comments!
Comment on Rebuttal
Thank you for the detailed responses and the additional experiments. I appreciate the authors' efforts to address my concerns.
The newly added search–annotation–retraining experiment is particularly valuable, as it demonstrates that the discovered comparisons can improve preference-model accuracy on unseen adversarial examples. The additional comparisons with the fixed-Leader baseline and the validation of the synthetic oracle against human and independent LLM judgments also strengthen the empirical evidence.
Some limitations remain, particularly the relatively small model scale, the single-seed retraining experiment, and the lack of a direct comparison with Adv-RM. Nevertheless, I believe the authors have substantially strengthened the paper and addressed the main concerns regarding the practical value of the proposed approach.
Overall, I maintain my Weak Accept (3) recommendation.
We thank the reviewer for their active engagement with our paper. We agree with the reviewer about the small model scale, and we have highlighted this in the conclusion of the paper. We would also like to point the reviewer to our response to reviewer hgoB, where we provide a comparison between Adv-RM and our work. We report some details below.
Adv-RM trains a single adversarial policy against a target reward model, using a second reward model as reference. Our method differs in three ways:
- Symmetric role of ensemble Adv-RM singles out one model as the target and finds responses that it overrates relative to the reference. By construction, it needs a stronger reference model in order to update a weaker one. Our objective is the variance across all ensemble members, so it searches for comparisons on which the members disagree without designating any of them as correct.
- Preference models v/s Reward models. Adv-RM needs per-response scalar rewards. Our ensemble members are pairwise preference models, which accommodate intransitive preferences and need not be representable by a scalar BT reward.
- Adaptive players. Adv-RM attacks with a single policy against a fixed baseline. Our Leader adapts away from responses the Follower has already exploited, so the Follower must keep searching for new weaknesses rather than repeatedly exploiting the same ones. Therefore, we believe that a fair comparison for Adv-RM is indeed the fixed Leader abalation for which we provide more details below.
For the above reasons, we did not include Adv-RM as a direct baseline, since doing so would have required changing its method significantly to fit our task. We note that Adv-RM is related work in the spirit of the problem that we try to solve. However, the approach and scope of our methods is rather different.
Once again, thank you for your time, your questions, and your positive assessment of our work.
We thank the reviewer for the positive assessment, and for finding the problem formulation meaningful and the equilibrium analysis and last-iterate convergence result valuable. We address each concern below.
The new section Sections 5.4 and 5.5 compare our jointly trained Leader–Follower pair with a fixed-Leader baseline whose Follower also maximizes ensemble disagreement.
The motivation is that, against a fixed Leader, the Follower can repeatedly exploit weaknesses specific to an unchanging response distribution. An adaptive Leader shifts probability away from responses vulnerable to the current Follower, reducing the payoff of those attacks and encouraging further search. This may also discourage deterioration in instruction following: disagreement alone can reward response patterns that exploit evaluator uncertainty while neglecting the prompt.
Our IFEval ablation is consistent with this interpretation. The Follower trained against an adaptive Leader performs better on all four aggregate metrics; for example, loose instruction-level accuracy is 46.04%, compared with 38.49% for the fixed-Leader baseline (Section 5.5, Appendix G.5).
In the new Section 5.4 (with more details in Appendix E.6), we directly test whether labelling discovered comparisons and retraining the preference model improves accuracy on unseen comparisons. We test the fixed leader and the jointly trained model, and keep additional responses from the untrained policies as a control. From each source we select 100 triplets that are α-online candidates at α=0.05, label them with the oracle, and add them to the original UltraFeedback training comparisons. Each experiment retrains a single preference model from scratch, as described in the procedure of Appendix E.2. As a baseline, we also keep a model trained only on the original data. We evaluate oracle agreement on two held-out sets whose prompts are disjoint from all training and augmentation data: 2,000 UltraFeedback comparisons, and 600 game-generated comparisons (300 sampled at random from the Follower-only pair and 300 from the Leader–Follower pair).
Augmentation with comparisons from our jointly trained Leader–Follower pair increases accuracy on unseen adversarial comparisons from 70.33% to 83.33%, reducing the observed error rate from 29.7% to 16.7%. With the same budget of 100 labelled comparisons, reference-policy augmentation achieves only 68.67% adversarial accuracy (further showing poor performance of the current models on OOD data), while Follower-only augmentation reaches 80.83%. The additional 2.5 percentage points from jointly training both policies (95% paired confidence interval [0.66,4.44]) are statistically significant at p=0.017.
The breakdown on the individual adverserial distributions below show that the Leader-Follower jointly trained comparisons beat the Follower-only on both augmented distributions (+1.67 points on Follower-only comparisons, +3.33 on Leader–Follower comparisons), which further reinforces that the gain is not specific to just the augmentation data of its own distribution.
Oracle agreement (%) on the 600 held-out adversarial comparisons, by the pair that generated them (300 each). Each retrained model uses the original 10,002 UltraFeedback comparisons plus 100 oracle-labelled comparisons from the given source.
These results show that the discovered comparisons supply useful supervision whose benefits extend to unseen prompts. For now, we use one preference model and a single update round, with accuracy measured against the synthetic oracle. We note that strategies for retraining over repeated rounds is exciting future work.
We also observe a complementary benefit: despite optimizing ensemble variance, our Leader remains competitive with preference-optimization methods, obtaining oracle preference rates of 54.89% against NLHF and 50.70% against SLHF (Appendix G.4).