acceptodds
Under review as a conference paper at ICLR 2027

Can You Trust Your Preference Models? Stackelberg Games for Alignment

13 views

Abstract

Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are unavailable during policy training, we use disagreement among an ensemble of preference models as a proxy for prediction error. We formulate this search as a zero-sum Stackelberg game between two opposing policies, a Leader which tries to minimize disagreement, and a Follower which maximizes it. For a general class of finite contextual Stackelberg games, we establish equilibrium existence and, under regularization, uniqueness and a closed-form characterization. We introduce Stackelberg Mirror Descent–Ascent (SMIDA), a two-timescale algorithm with linear last-iterate convergence in the tabular setting, and develop a neural policy-gradient analogue. Experiments with a synthetic oracle support disagreement as an informative proxy for prediction error. On held out data the trained policies outperform reference policies and increase the fraction of comparisons that exceed a disagreement threshold and reverse the oracle’s preference.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Alignment, Preference learning, Reward hacking, Adversarial search

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_toANOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to identify failures in preference models before they are exploited during LLM alignment. The authors use disagreement among an ensemble of preference models as a proxy for evaluation errors and formulate adversarial comparison generation as a zero-sum Stackelberg game. A Leader minimizes disagreement, while a Follower observes the Leader's response and generates an alternative that maximizes it. The paper develops SMIDA with convergence guarantees in the tabular setting and extends it to neural policies through Deep-SMIDA. Experiments show that the trained policies improve the yield of high-disagreement, oracle-confirmed adversarial comparisons from 4.4% to 9.6%.

Critical strengths
  1. Interesting problem formulation. The paper addresses an important weakness of preference-based alignment: the evaluator itself may be unreliable. Framing error discovery as an adversarial game is a meaningful perspective.

  2. Solid theoretical contribution with encouraging empirical results. The equilibrium analysis and last-iterate convergence result are valuable. The experiments also provide evidence that ensemble disagreement can help identify preference-model errors.

Critical weaknesses
  1. The necessity of the Stackelberg formulation is not sufficiently established. The main experiments do not clearly isolate the benefit of jointly training the Leader and Follower over simpler alternatives, such as a fixed Leader with a disagreement-maximizing Follower or a single adversarial generator.

  2. Limited validation of actual preference errors. The evaluation relies on a synthetic oracle rather than human judgments. It is therefore unclear how reliably the discovered adversarial comparisons correspond to genuine human preference violations.

Questions for authors
  1. Could the authors compare Deep-SMIDA against a fixed-Leader or single-policy disagreement-maximization baseline under matched training budgets, using the same oracle-confirmed adversarial yield metric?

  2. How sensitive are the results to the choice of synthetic oracle? Would a small human evaluation or an independent oracle lead to similar conclusions?

  3. The convergence guarantee applies to the tabular setting. Could the authors provide additional evidence of training stability and convergence behavior for Deep-SMIDA across multiple random seeds?

4 comments

We thank the reviewer for the positive assessment, and for finding the problem formulation meaningful and the equilibrium analysis and last-iterate convergence result valuable. We address each concern below.

The necessity of the Stackelberg formulation is not sufficiently established. The main experiments do not clearly isolate the benefit of jointly training the Leader and Follower over simpler alternatives, such as a fixed Leader with a disagreement-maximizing Follower or a single adversarial generator.

Could the authors compare Deep-SMIDA against a fixed-Leader or single-policy disagreement-maximization baseline under matched training budgets, using the same oracle-confirmed adversarial yield metric?

The new section Sections 5.4 and 5.5 compare our jointly trained Leader–Follower pair with a fixed-Leader baseline whose Follower also maximizes ensemble disagreement.

The motivation is that, against a fixed Leader, the Follower can repeatedly exploit weaknesses specific to an unchanging response distribution. An adaptive Leader shifts probability away from responses vulnerable to the current Follower, reducing the payoff of those attacks and encouraging further search. This may also discourage deterioration in instruction following: disagreement alone can reward response patterns that exploit evaluator uncertainty while neglecting the prompt.

Our IFEval ablation is consistent with this interpretation. The Follower trained against an adaptive Leader performs better on all four aggregate metrics; for example, loose instruction-level accuracy is 46.04%, compared with 38.49% for the fixed-Leader baseline (Section 5.5, Appendix G.5).

In the new Section 5.4 (with more details in Appendix E.6), we directly test whether labelling discovered comparisons and retraining the preference model improves accuracy on unseen comparisons. We test the fixed leader and the jointly trained model, and keep additional responses from the untrained policies as a control. From each source we select 100 triplets that are α\alpha-online candidates at α=0.05\alpha=0.05, label them with the oracle, and add them to the original UltraFeedback training comparisons. Each experiment retrains a single preference model from scratch, as described in the procedure of Appendix E.2. As a baseline, we also keep a model trained only on the original data. We evaluate oracle agreement on two held-out sets whose prompts are disjoint from all training and augmentation data: 2,000 UltraFeedback comparisons, and 600 game-generated comparisons (300 sampled at random from the Follower-only pair and 300 from the Leader–Follower pair).

Augmentation with comparisons from our jointly trained Leader–Follower pair increases accuracy on unseen adversarial comparisons from 70.33% to 83.33%, reducing the observed error rate from 29.7% to 16.7%. With the same budget of 100 labelled comparisons, reference-policy augmentation achieves only 68.67% adversarial accuracy (further showing poor performance of the current models on OOD data), while Follower-only augmentation reaches 80.83%. The additional 2.5 percentage points from jointly training both policies (95% paired confidence interval [0.66,4.44][0.66, 4.44]) are statistically significant at p=0.017p=0.017.

The breakdown on the individual adverserial distributions below show that the Leader-Follower jointly trained comparisons beat the Follower-only on both augmented distributions (+1.67 points on Follower-only comparisons, +3.33 on Leader–Follower comparisons), which further reinforces that the gain is not specific to just the augmentation data of its own distribution.

Oracle agreement (%) on the 600 held-out adversarial comparisons, by the pair that generated them (300 each). Each retrained model uses the original 10,002 UltraFeedback comparisons plus 100 oracle-labelled comparisons from the given source.

Augmentation sourceGenerated by Follower-only pairGenerated by Leader–Follower pairBoth
None (original model)70.3370.3370.33
Reference pair69.3368.0068.67
Follower-only pair79.3382.3380.83
Leader–Follower pair81.0085.6783.33
Leader–Follower − Follower-only+1.67+3.33+2.50 [+0.66, +4.44]*

These results show that the discovered comparisons supply useful supervision whose benefits extend to unseen prompts. For now, we use one preference model and a single update round, with accuracy measured against the synthetic oracle. We note that strategies for retraining over repeated rounds is exciting future work.

We also observe a complementary benefit: despite optimizing ensemble variance, our Leader remains competitive with preference-optimization methods, obtaining oracle preference rates of 54.89% against NLHF and 50.70% against SLHF (Appendix G.4).

AuthorsOpenReview ↗

[Continued response to toAN]

Limited validation of actual preference errors. The evaluation relies on a synthetic oracle rather than human judgments. It is therefore unclear how reliably the discovered adversarial comparisons correspond to genuine human preference violations.

How sensitive are the results to the choice of synthetic oracle? Would a small human evaluation or an independent oracle lead to similar conclusions?

We validated the oracle against both a human and an independent oracle. A human annotator and a frontier model (Opus 5.5) each labelled 100 comparisons from Sec. 5.3.2, and agreed with the oracle on 88.9% (95% CI [81.2%, 93.7%]) and 88.0% (CI [79.8%, 93.2%]) of comparisons. The oracle is also self-consistent (85.0% agreement between two independent queries) and agrees with the original UltraFeedback labels on 85.5% of comparisons. The two judges agree with the oracle at similar rates, so our conclusions do not depend on a single judge.

The convergence guarantee applies to the tabular setting. Could the authors provide additional evidence of training stability and convergence behavior for Deep-SMIDA across multiple random seeds?

We agree that the guarantee is tabular. Our existing runs provide evidence on the stability of Deep-SMIDA, while varying the regularization strength τ\tau. In the sweep of Appendix G.2, we trained six configurations against the same ensemble, with τ\tau ranging from 2×10−42\times10^{-4} to 4×10−34\times10^{-3}. All six completed training without divergence. The five configurations evaluated on held-out prompts reach α=0.1\alpha=0.1 yields of 25.8–43.7%, compared with 10.0% for the reference policies. Policy drift and candidate yield decrease steadily in τ\tau, as the closed form of Theorem 4.2 predicts. We do bring the reviewer's attention to the comments we have made in Section 4.3 in the paper. Preliminary experiments showed that training was highly sensitive to the learning rates ηLη^L and ηFη^F and often became un-stable; standard warm-up techniques did not resolve this instability. To address this, we linearly increase the ratio ηtF/ηtLη_t^F/η^L_t from 1 to its target ηF/ηL>1η^F/η^L > 1, while warming up the Leader’s learning rate ηTLη^L_T linearly from ηinitLη^L_{init} to ηLη^L. We would point the reviewer to Appendix D in our paper for more details on this.

We thank the reviewer for encouraging comments!

Reviewer_toANOpenReview ↗

Comment on Rebuttal

Thank you for the detailed responses and the additional experiments. I appreciate the authors' efforts to address my concerns.

The newly added search–annotation–retraining experiment is particularly valuable, as it demonstrates that the discovered comparisons can improve preference-model accuracy on unseen adversarial examples. The additional comparisons with the fixed-Leader baseline and the validation of the synthetic oracle against human and independent LLM judgments also strengthen the empirical evidence.

Some limitations remain, particularly the relatively small model scale, the single-seed retraining experiment, and the lack of a direct comparison with Adv-RM. Nevertheless, I believe the authors have substantially strengthened the paper and addressed the main concerns regarding the practical value of the proposed approach.

Overall, I maintain my Weak Accept (3) recommendation.

We thank the reviewer for their active engagement with our paper. We agree with the reviewer about the small model scale, and we have highlighted this in the conclusion of the paper. We would also like to point the reviewer to our response to reviewer hgoB, where we provide a comparison between Adv-RM and our work. We report some details below.

Adv-RM trains a single adversarial policy against a target reward model, using a second reward model as reference. Our method differs in three ways:

  1. Symmetric role of ensemble Adv-RM singles out one model as the target and finds responses that it overrates relative to the reference. By construction, it needs a stronger reference model in order to update a weaker one. Our objective is the variance across all ensemble members, so it searches for comparisons on which the members disagree without designating any of them as correct.
  2. Preference models v/s Reward models. Adv-RM needs per-response scalar rewards. Our ensemble members are pairwise preference models, which accommodate intransitive preferences and need not be representable by a scalar BT reward.
  3. Adaptive players. Adv-RM attacks with a single policy against a fixed baseline. Our Leader adapts away from responses the Follower has already exploited, so the Follower must keep searching for new weaknesses rather than repeatedly exploiting the same ones. Therefore, we believe that a fair comparison for Adv-RM is indeed the fixed Leader abalation for which we provide more details below.

For the above reasons, we did not include Adv-RM as a direct baseline, since doing so would have required changing its method significantly to fit our task. We note that Adv-RM is related work in the spirit of the problem that we try to solve. However, the approach and scope of our methods is rather different.

Once again, thank you for your time, your questions, and your positive assessment of our work.

Reviewer_3ABNOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Preference models used to align language models are imperfect proxies for human preferences. This paper searches for comparisons on which the preference model is likely wrong, so that limited human annotation can be directed there. The underlying goal is to improve the preference model with this newly identified data. To find these likely wrong comparisons, the authors use disagreement among an ensemble of five preference models by measuring the variance of their predicted probabilities. For the search itself, a Leader and a Follower LLM generate new responses that the preference models then score. The two are trained in a zero-sum Stackelberg game, where the Follower maximizes variance and the Leader minimizes it, so that the Follower cannot exploit weaknesses specific to a fixed response distribution

Critical strengths

Making both sides of the comparison adaptive is a reasonable design choice. A simpler approach would train a single LLM to attack a fixed response, but the attacker can then keep exploiting the same weaknesses rather than searching for new ones. Framing the search as a Stackelberg game instead lets both sides learn, with the Follower maximizing disagreement and the Leader minimizing it. As the Leader shifts away from responses the Follower has successfully attacked, those attacks stop working and the Follower has to keep looking.

Critical weaknesses

The main goal of searching for comparisons with potential preference errors is to use them to improve the preference models. However, the experiments stop short of this. No preference model is ever retrained on the identified comparisons, and no evidence is given that doing so improves it. The authors acknowledge this, but since it is the central motivation for the search, the search itself remains weakly justified. A demonstration that augmenting the ensemble’s training data with the discovered comparisons reduces its error rate, even at small scale, would substantially strengthen the paper.

Questions for authors

The relabelled triplets used to train the ensemble appear to receive a single oracle query each, whereas the evaluation in Section 5.3.2 averages 11 queries per comparison. Since LLM outputs are not deterministic, the oracle could return a different preference even if queried again on the same comparison. How self-consistent is the oracle across repeated queries, and could label noise in the ensemble’s training data affect the measured preference-model errors? All experiments use a single oracle model as the stand-in for human preference, so the reported errors depend on that model’s judgments. Since the original UltraFeedback annotations were replaced during relabelling, how often did the oracle agree with them? Reporting this agreement, or using it to select a suitable oracle, would give some indication of how well the oracle approximates human preference.

4 comments

We thank the reviewer for the positive assessment, and for the clear description of the problem and, in particular, of why making both sides of the comparison adaptive matters. We address the concerns of the reviewer below.

The main goal of searching for comparisons with potential preference errors is to use them to improve the preference models. However, the experiments stop short of this. No preference model is ever retrained on the identified comparisons, and no evidence is given that doing so improves it. The authors acknowledge this, but since it is the central motivation for the search, the search itself remains weakly justified. A demonstration that augmenting the ensemble's training data with the discovered comparisons reduces its error rate, even at small scale, would substantially strengthen the paper.

We thank the reviewer for suggesting this experiment. We have added Section 5.4, which completes one round of search, annotation and retraining, and directly evaluates whether the discovered comparisons improve performance on unseen data.

A demonstration that augmenting the ensemble’s training data with the discovered comparisons reduces its error rate, even at small scale, would substantially strengthen the paper.

Adding 100 labelled comparisons generated by the jointly trained Leader–Follower pair reduces the preference model’s observed error rate on unseen adversarial comparisons from 29.7% to 16.7%, corresponding to an increase in accuracy from 70.33% to 83.33%. Accuracy on held-out UltraFeedback comparisons also increases, from 75.55% to 76.65%. These results suggest that comparisons found during adversarial search and filtered by α\alpha-candidates can serve as additional supervision that improves the preference model on unseen data.

The procedure is detailed in Appendix E.6. We compare, as baselines, augmentation from the jointly trained Leader–Follower pair, the Follower trained against a fixed Leader, and the untrained reference policies as a control. From each source, we select 100 triplets satisfying the α\alpha-online criterion at α=0.05\alpha=0.05, label them with the oracle, and add them to the original UltraFeedback training set. Each experiment retrains a single preference model from scratch, with a model trained only on the original data as a further baseline. We evaluate oracle agreement on two held-out validation sets: 2,000 UltraFeedback comparisons and 600 game-generated comparisons, comprising 300 randomly sampled from the Follower-only pair and 300 from the trained Leader–Follower pair.

The comparison between augmentation sources shows that the choice of additional training data is relevant. With the same budget of 100 labelled comparisons, reference-policy augmentation yields 68.67% adversarial accuracy, while Follower-only augmentation reaches 80.83%. The jointly trained pair achieves 83.33%, giving a further improvement of 2.50 percentage points over Follower-only augmentation (95% paired confidence interval [0.66,4.44][0.66, 4.44]; statistically significant at p=0.017p=0.017).

Oracle agreement (%) of a single preference model retrained after adding 100 oracle-labelled α\alpha-online candidates (α=0.05\alpha=0.05) from each source.

Augmentation sourceHeld-out UltraFeedback (2,000 comparisons)Held-out adversarial (600 comparisons)
None (original model)75.5570.33
Reference pair76.2568.67
Follower-only pair (fixed Leader)76.1580.83
Jointly trained Leader–Follower pair76.6583.33

We further break down the adversarial evaluation by the pair that generated the comparisons and observe that the benefit extends beyond the distribution that generated the augmentation data.

Oracle agreement (%) on the 600 held-out adversarial comparisons, split by generating pair (300 each). Each augmented model is retrained on the original UltraFeedback data plus 100 oracle-labelled comparisons from the indicated source.

Augmentation sourceGenerated by Follower-only pairGenerated by Leader–Follower pairBoth
None (original model)70.3370.3370.33
Reference pair69.3368.0068.67
Follower-only pair79.3382.3380.83
Jointly trained Leader–Follower pair81.0085.6783.33
Leader–Follower − Follower-only+1.67+3.33+2.50

Differences are in percentage points and calculated before rounding. The 95% paired confidence interval for the overall difference is [+0.66,+4.44][+0.66, +4.44].

This experiment directly supports the central motivation of our paper, that one round of search, annotation and retraining produces a preference model with higher accuracy on unseen comparisons. It also provides evidence that jointly adapting the two generating policies helps find more useful training comparisons under the same annotation budget.

For now, we use one preference model and a single update round, with accuracy measured against the synthetic oracle.

AuthorsOpenReview ↗

[Continued response to 3ABN]

The relabelled triplets used to train the ensemble appear to receive a single oracle query each, whereas the evaluation in Section 5.3.2 averages 11 queries per comparison. Since LLM outputs are not deterministic, the oracle could return a different preference even if queried again on the same comparison. How self-consistent is the oracle across repeated queries, and could label noise in the ensemble's training data affect the measured preference-model errors?

Using the 11 oracle queries per comparison from Sec. 5.3.2, two independent queries on the same comparison agree 85.0% of the time (95% CI [83.9%, 86.1%]; a coin flip would give 50%). The oracle's preference probability PP is also far from indifference for most comparisons: ∣P−1/2∣>0.1|P-1/2|>0.1 for 93.1% of comparisons, and ∣P−1/2∣>0.3|P-1/2|>0.3 for 80.5%.

Label noise in the single-query training labels does not affect our conclusions, for two reasons. First, the evaluation labels in Sec. 5.3.2 average 11 queries, so the measured preference-model errors are computed against a low-noise target. Second, training-label noise affects every member of the ensemble and every method we compare in the same way, since all use the same training data. Further, as we describe in Section 3 in the paper, this noisy setting is precisely how we obtain annotations from humans in real experiments. In fact, a preference sample is precisely the label of a randomly chosen human annotator.

All experiments use a single oracle model as the stand-in for human preference, so the reported errors depend on that model's judgments. Since the original UltraFeedback annotations were replaced during relabelling, how often did the oracle agree with them? Reporting this agreement, or using it to select a suitable oracle, would give some indication of how well the oracle approximates human preference.

The oracle agrees with the original UltraFeedback preferences on 85.5% of comparisons (95% CI [85.2%, 85.8%]). Since the original UltraFeedback labels were produced by GPT-4 rather than by humans, we additionally validated the oracle with a human annotator and an independent LLM judge. On 100 comparisons from Sec. 5.3.2 each, the human annotator agreed with our oracle on 88.9% of comparisons and an independent synthetic oracle (Opus 5.5) on 88.0%. This supports the consistency of our chosen synthetic oracle.

We add this analysis to the revised appendix E5. We thank the reviewer for helpful comments that greatly improved the paper.

I thank the authors for the detailed response and the additional experiments.

My main concern has been addressed. Previously, the paper only showed that the method finds comparisons on which the preference model is likely wrong, without testing whether using them helps. The new experiment in Section 5.4 completes this step by retraining with the discovered comparisons, and shows improved accuracy on unseen comparisons, which supports the central motivation of the method.

The additional analyses also resolve my two questions. On oracle self-consistency, the report that two independent queries agree on 85.0% of comparisons, with most preferences far from indifference, addresses the concern about label noise in the single-query training labels. The explanation that this noise affects every ensemble member and every compared method in the same way also answers my first question. On how well the oracle approximates the original annotations, the 85.5% agreement with UltraFeedback, together with the checks against a human annotator and an independent LLM judge, answers my second question.

I will raise my score accordingly.

We sincerely appreciate your time and thoughtful review. Thank you for your positive assessment and support!

Reviewer_hgoBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to find errors in preference models without asking for human labels during policy training. It uses disagreement among an ensemble as the training signal. A Leader generates a response and tries to minimize disagreement, while a Follower generates an alternative and tries to maximize it. The paper gives equilibrium and convergence results for the tabular game, then develops a neural version and tests it using a synthetic preference oracle.

Critical strengths

Finding preference-model errors is an important problem. I appreciate that the paper distinguishes ensemble disagreement from actual preference errors, rather than treating them as the same thing.

The theoretical analysis is also a useful part of the paper. It gives a closed-form equilibrium under regularization and a linear last-iterate convergence result for the tabular algorithm. The experiments include held-out prompts and some useful checks, including the fixed-Leader ablation.

Critical weaknesses

My main concern is whether the method actually gives a better way to find preference-model errors. In Section 5.3.2, the trained pair has a lower overall oracle-reversal rate than the reference pair: 19.4% versus 34.3%. The reported improvement is for comparisons that both pass a disagreement threshold and reverse the oracle preference, where the yield increases from 4.4% to 9.6%. This is a useful result, but it answers a narrower question. I would like to know which method finds more useful errors under the same generation cost and annotation budget.

The benefit of training both players is also not clear enough. The fixed-Leader ablation shows better instruction following for the Follower trained with an adaptive Leader. But instruction following is not the main error-discovery objective. A direct comparison on oracle-confirmed error discovery would be more convincing. The paper also discusses Adv-RM as closely related work, but does not provide a direct experimental comparison.

Another concern is that the main preference-error evaluation uses a synthetic oracle. This is a reasonable starting point, but disagreement with this oracle is not necessarily disagreement with human preferences. Repeated oracle queries can measure its variability, but cannot remove systematic judge errors. Some human validation or evaluation with another independent oracle would help.

Finally, the paper stops at finding candidate errors. It does not show that labeling these examples and updating the preference model makes it more robust on new comparisons. The authors acknowledge this limitation, but it leaves the practical value of the search uncertain. The convergence theorem tells us that the tabular algorithm reaches the game equilibrium; it does not establish that this equilibrium gives the most useful examples for improving the evaluator.

Questions for authors
  1. Under matched generation and annotation budgets, how many oracle-confirmed errors does the method find compared with reference sampling, a fixed-Leader Follower, and Adv-RM?

  2. Does the adaptive Leader improve error discovery itself, beyond the reported IFEval improvement?

  3. If the discovered comparisons are labeled and used to update the preference model, does its accuracy improve on unseen comparisons? Can this be checked with human labels or an independent oracle?

7 comments

We thank the reviewer for the thoughtful review, constructive criticism, and for finding the problem important and the theoretical analysis useful. We address each concern below. In particular, we note the addition of an experiment that performs one complete round of search, annotation and retraining, with matched generation and annotation budget.

My main concern is whether the method actually gives a better way to find preference-model errors. In Section 5.3.2, the trained pair has a lower overall oracle-reversal rate than the reference pair: 19.4% versus 34.3%. The reported improvement is for comparisons that both pass a disagreement threshold and reverse the oracle preference, where the yield increases from 4.4% to 9.6%. This is a useful result, but it answers a narrower question. I would like to know which method finds more useful errors under the same generation cost and annotation budget.

We would like to point out that the quantity that matters for error discovery is the yield of flagged comparisons that reverse the oracle, which goes from 4.4% to 9.6%, not the overall reversal rate. The 34.3% for the reference pair is measured over all full responses sampled from the reference policy, which amounts to sampling comparisons at random. These reversals are spread over the whole set of comparisons, and without any signal pointing to them, finding them requires labelling every comparison. Our goal is to find preference-model errors without asking for labels during the search, so that search, labelling and retraining can run as a repeated, automated loop.

The disagreement threshold α\alpha is what lets us decide which comparisons to send to labelling, so we believe that the relevant quantity is the number of confirmed errors among the flagged comparisons. Figure 2(b) shows this at matched generation and annotation budgets; our method produces more preference triplets that both reverse the oracle judgement (ε\varepsilon-adversarial) and can be identified cheaply, using only the ensemble and no labels (α\alpha-online candidates). We have a new Figure 6 in Appendix G.3 for a more detailed analysis.

(a) Base (reference) policies

αε = 0ε = 0.1ε = 0.2ε = 0.3
0.016 (0.4%)6 (0.4%)5 (0.3%)3 (0.2%)
0.0214 (0.9%)13 (0.8%)11 (0.7%)9 (0.6%)
0.0531 (1.9%)27 (1.7%)22 (1.4%)15 (0.9%)
0.171 (4.4%)63 (3.9%)55 (3.4%)42 (2.6%)
0.2128 (8.0%)107 (6.7%)93 (5.8%)75 (4.7%)

(b) Trained policies

αε = 0ε = 0.1ε = 0.2ε = 0.3
0.0154 (3.4%)50 (3.1%)44 (2.8%)38 (2.4%)
0.0279 (4.9%)72 (4.5%)62 (3.9%)54 (3.4%)
0.05128 (8.0%)116 (7.2%)100 (6.2%)90 (5.6%)
0.1153 (9.6%)141 (8.8%)124 (7.8%)112 (7.0%)
0.2206 (12.9%)186 (11.6%)160 (10.0%)144 (9.0%)

The benefit of training both players is also not clear enough. The fixed-Leader ablation shows better instruction following for the Follower trained with an adaptive Leader. But instruction following is not the main error-discovery objective. A direct comparison on oracle-confirmed error discovery would be more convincing.

Against a fixed Leader, the Follower finds a fixed set of weaknesses of the ensemble and keeps exploiting them. With an adaptive Leader, the responses the Follower has successfully attacked are abandoned by the Leader, those attacks stop paying off, and the Follower has to keep searching for new weaknesses. Quantitatively, our IFEval ablation shows that the Follower trained against a fixed Leader loses instruction-following capability compared to our Follower (e.g. instruction-level accuracy 46%, against 38% for the fixed Leader). This is because the Follower can continue to exploit the ensemble simply by degrading the structure/quality of its responses, while the fixed Leader cannot adapt to counter these attacks.

More importantly, following the reviewers suggestion, we now evaluate the usefulness of the generated comparisons for improving the preference model. The new Section 5.4 describes an experiment in which we augment the original preference-training dataset with 100 labeled comparisons from each generation method and retrain the preference model from scratch. Comparisons from the jointly trained Leader-Follower pair improves accuracy on unseen adversarial comparisons by 2.5 percentage points more than Follower-only comparisons (83.33% against 80.83%, statistically significant p = 0.017). For more details on this, see the next comment.

Finally, training the Leader also has an indirect benefit for preference alignment. Although trained solely to minimize ensemble variance, our Leader improves over the base Leader and remains competitive with policies explicitly trained to optimize preferences via classical alignment methods; in particular, our Leader has a win rate of 54.89% against NLHF and 50.7% against SLHF.

AuthorsOpenReview ↗

[Continued response to hgoB]

The paper also discusses Adv-RM as closely related work, but does not provide a direct experimental comparison.

Adv-RM [1] trains a single adversarial policy against a target reward model, using a second reward model as reference. The adversary is rewarded for responses that the target scores above the SFT baseline while the reference scores them low. Our method differs in three ways.

  1. Symmetric role of ensemble Adv-RM singles out one model as the target and finds responses that it overrates relative to the reference. By construction, it needs a stronger reference model in order to update a weaker one. Our objective is the variance across all ensemble members, so it searches for comparisons on which the members disagree without designating any of them as correct.
  2. Preference models v/s Reward models. Adv-RM needs per-response scalar rewards. Our ensemble members are pairwise preference models, which accommodate intransitive preferences and need not be representable by a scalar BT reward.
  3. Adaptive players. Adv-RM attacks with a single policy against a fixed baseline. Our Leader adapts away from responses the Follower has already exploited, so the Follower must keep searching for new weaknesses rather than repeatedly exploiting the same ones. Therefore, we believe that a fair comparison for Adv-RM is indeed the fixed Leader abalation for which we provide more details below.

For the above reasons, we did not include Adv-RM as a direct baseline, since doing so would have required changing its method significantly to fit our task. We note that Adv-RM is related work in the spirit of the problem that we try to solve. However, the approach and scope of our methods is rather different.

Another concern is that the main preference-error evaluation uses a synthetic oracle. This is a reasonable starting point, but disagreement with this oracle is not necessarily disagreement with human preferences. Repeated oracle queries can measure its variability, but cannot remove systematic judge errors. Some human validation or evaluation with another independent oracle would help.

We validated the oracle against a human annotator and an independent oracle. A human annotator and a frontier model (Opus 5.5) each labelled 100 random comparisons from Sec. 5.3.2. The human annotator agreed with the oracle on 88.9% of comparisons (95% CI [81.2%, 93.7%]), while Opus 5.5 agreed with our oracle on 88.0% (95% CI [79.8%, 93.2%]).

The oracle also agrees with the original UltraFeedback preferences on 85.5% of comparisons (95% CI [85.2%, 85.8%]). Note that the original UltraFeedback labels were themselves produced by GPT-4.

AuthorsOpenReview ↗

[Continued (2) response to hgoB]

Finally, the paper stops at finding candidate errors. It does not show that labeling these examples and updating the preference model makes it more robust on new comparisons. The authors acknowledge this limitation, but it leaves the practical value of the search uncertain.

The convergence theorem tells us that the tabular algorithm reaches the game equilibrium; it does not establish that this equilibrium gives the most useful examples for improving the evaluator.

If the discovered comparisons are labeled and used to update the preference model, does its accuracy improve on unseen comparisons? Can this be checked with human labels or an independent oracle?

We agree that finding candidate errors alone does not establish their usefulness for improving the preference model. To address this, we have added a retraining experiment in Section 5.4, with further details in Appendix E.6. The experiment shows that labelling the discovered comparisons and adding them to the training data improves oracle agreement on unseen comparisons.

We compare three augmentation sources: the Follower trained against a fixed Leader, the jointly trained Leader–Follower pair, and the untrained reference policies as a control. From each source, we select 100 triplets that are α\alpha-online candidates at α=0.05\alpha=0.05, label them with the oracle, and add them to the original UltraFeedback training comparisons. For each source, we retrain a single preference model from scratch following Appendix E.2, and compare it with a baseline trained only on the original data. Evaluation uses two held-out sets whose prompts are disjoint from all training and augmentation data: 2,000 UltraFeedback comparisons and 600 game-generated comparisons, comprising 300 randomly sampled from the Follower-only pair and 300 from the jointly trained Leader–Follower pair.

Augmentation with comparisons from the jointly trained pair increases accuracy on the held-out adversarial set from 70.33% to 83.33%, reducing the observed error rate from 29.7% to 16.7%. With the same budget of 100 labelled comparisons, reference-policy augmentation achieves 68.67% adversarial accuracy, while Follower-only augmentation reaches 80.83%. Joint training therefore provides a further gain of 2.50 percentage points over Follower-only augmentation (95% paired confidence interval [0.66,4.44][0.66, 4.44]; statistically significant at p=0.017p=0.017). Accuracy on held-out UltraFeedback comparisons also increases, from 75.55% to 76.65%.

Oracle agreement (%) of a single preference model after adding 100 oracle-labelled α\alpha-online candidates (α=0.05\alpha=0.05) from each source.

Augmentation sourceHeld-out UltraFeedback (2,000 comparisons)Held-out adversarial (600 comparisons)
None (original model)75.5570.33
Reference pair76.2568.67
Follower-only pair (fixed Leader)76.1580.83
Jointly trained Leader–Follower pair76.6583.33

The breakdown by generating pair shows that Leader–Follower augmentation outperforms Follower-only augmentation on both held-out distributions: by 1.67 percentage points on Follower-only comparisons and by 3.33 points on Leader–Follower comparisons. Thus, the improvement is not limited to comparisons generated by the same pair that supplied the augmentation data.

Oracle agreement (%) on the 600 held-out adversarial comparisons, by the pair that generated them (300 each). Each retrained model uses the original 10,002 UltraFeedback comparisons plus 100 oracle-labelled comparisons from the given source.

Augmentation sourceGenerated by Follower-only pairGenerated by Leader–Follower pairBoth
None (original model)70.3370.3370.33
Reference pair69.3368.0068.67
Follower-only pair79.3382.3380.83
Leader–Follower pair81.0085.6783.33
Leader–Follower − Follower-only+1.67+3.33+2.50 [+0.66, +4.44]*

We agree that the convergence theorem establishes convergence to the game equilibrium, not optimality for downstream learning. However, convergence does provide us with a very strong stopping criterion. Since the policies remain stable after convergence, responses after this time come from the same distribution, which likely don't provide useful additional supervision. In conclusion, the new experiment provides direct empirical evidence that the search produces useful supervision that improves performance on unseen comparisons. For now, we use one preference model and a single update round, with accuracy measured against the synthetic oracle. We note that retraining the ensemble over repeated rounds is exciting future work.

Thank you for your feedback!

References

[1] A. Bukharin, H. Qian, S. Sun, A. Renduchintala, S. Singhal, Z. Wang, O. Kuchaiev, O. Delalleau, and T. Zhao. Adversarial training of reward models. In Second Conference on Language Modeling (COLM), 2025.

Comments on the paper

2 comments
AuthorsOpenReview ↗

New experiment: improving preference models with discovered comparisons (Section 5.4)

As helpfully suggested by all reviewers, we directly test whether labelling discovered comparisons and retraining the preference model improves accuracy on unseen comparisons (in the newly added Section 5.4; more details in Appendix E.6). Adding 100 labelled comparisons from our jointly trained Leader–Follower pair increases accuracy on unseen adversarially generated comparisons from 70.33% to 83.33%. Below, we describe the experiment in detail.

In particular, we compare three sources of additional training data: the untrained reference policies as a control, the Follower trained against a fixed Leader, and the jointly trained Leader–Follower pair. From each source, we randomly select 100 triplets that are α\alpha-online candidates at α=0.05\alpha=0.05, label them with our oracle, and add them to the original UltraFeedback training set. For each augmented dataset, we retrain a single preference model from scratch, following the procedure in Appendix E.2. As a further baseline, we also use a model trained only on the original data. We evaluate oracle agreement on two held-out sets whose prompts are disjoint from all training and augmentation data: 2,000 UltraFeedback comparisons and 600 game-generated comparisons, with 300 sampled at random from the Follower-only game and 300 from the Leader–Follower game.

Augmentation with comparisons from our jointly trained pair reduces the observed error rate on the game-generated comparisons from 29.7% to 16.7%. With the same budget of 100 labelled comparisons, reference-policy augmentation achieves 68.67% accuracy, while Follower-only augmentation reaches 80.83%. The additional 2.5 percentage points from jointly training both policies (95% paired confidence interval [0.66,4.44][0.66, 4.44]) are statistically significant p=0.017p=0.017. Accuracy on held-out UltraFeedback comparisons also improves, from 75.55% to 76.65%.

Oracle agreement (%) after adding 100 oracle-labelled α\alpha-online candidates (α=0.05\alpha=0.05) from each source.

Augmentation sourceHeld-out UltraFeedback (2,000 comparisons)Held-out game-generated (600 comparisons)
None (original model)75.5570.33
Reference pair76.2568.67
Follower-only pair (fixed Leader)76.1580.83
Jointly trained Leader–Follower pair76.6583.33

The breakdown by generation source below shows that Leader–Follower augmentation outperforms Follower-only augmentation on both evaluation distributions, by 1.67 percentage points on Follower-only comparisons and 3.33 points on Leader–Follower comparisons. This provides further evidence that the gain is not confined to comparisons generated by the same policies that supplied the augmentation data.

Oracle agreement (%) on the 600 held-out game-generated comparisons, by the pair that generated them (300 each). Each augmented model is retrained on the original 10,002 UltraFeedback comparisons plus 100 oracle-labelled comparisons from the given source.

Augmentation sourceGenerated by Follower-only pairGenerated by Leader–Follower pairBoth
None (original model)70.3370.3370.33
Reference pair69.3368.0068.67
Follower-only pair79.3382.3380.83
Leader–Follower pair81.0085.6783.33
Leader–Follower − Follower-only (percentage points)+1.67+3.33+2.50

These results show that the discovered comparisons supply useful supervision whose benefits extend to unseen prompts. They also provide evidence that jointly training both policies produces more useful augmentation data than training the Follower alone, under the same annotation budget. For now, this experiment uses a single preference model per condition and one update round, with accuracy measured against the synthetic oracle. Retraining over repeated rounds is an exciting direction for future work.

AuthorsOpenReview ↗

Summary of Contributions and Discussions

Dear Area Chair and Reviewers,

Language models are often aligned to human preferences by optimizing against a learned evaluator. Outside its training distribution, the evaluator is often misspecified, assigning high rewards to responses that humans would disfavor. An optimizing policy will learn to exploit these errors, a phenomenon known as reward hacking. Our idea is to use the same optimization pressure to discover such errors and target them for correction.

Unfortunately, human labels are expensive to collect, hence infeasible to use as a signal on newly generated preferences. For this reason, we start with an ensemble of preference models trained on labeled comparisons, and we use the variance of the ensemble’s predictions to identify promising candidates for annotation. In particular, we target ε-adversarial preference triplets, that is, triplets where the ensemble’s mean prediction favors one response while the underlying preference favors the other by a margin of at least ε from indifference. We formulate this search as a Stackelberg game between two language models. Given a prompt, the Leader generates a response. The Follower observes both the prompt and the Leader’s response, then generates an alternative that maximizes ensemble disagreement. Keeping the Leader fixed would introduce a problem akin to reward hacking. The Follower could learn to induce disagreement by exploiting quirks specific to the fixed Leader’s responses, limiting the range of adversarial triplets it discovers. We therefore train the Leader to minimize disagreement in anticipation of the Follower’s best response. As the Leader shifts away from responses the Follower repeatedly exploits, the Follower has an incentive to seek new comparisons that induce disagreement.

Theoretical contributions

  1. We introduce a framework for thinking about adversarial preferences, define ε-adversarial triplets and give a criterion, based on ensemble disagreement, for identifying such candidates during policy training.
  2. We formulate the ensemble-variance game as an instance of a general class of finite, contextual, zero-sum Stackelberg games. We prove that an equilibrium exists and unique, and admits a closed-form characterization.
  3. We introduce Stackelberg Mirror Descent–Ascent (SMIDA), prove linear convergence of its last iterate in the tabular setting, and develop Deep-SMIDA, a neural policy-gradient variant for specific for language models.


Empirical findings

  1. Ensemble disagreement helps identify evaluator errors. On held-out comparisons, the ensemble reversed the synthetic oracle’s preference on 40.6% of comparisons in the highest-variance decile, compared with 24.3% overall (Section 5.2).

  2. The trained policies generate more adversarial comparisons that the variance filter can identify. At a common threshold that selected 10.0% of comparisons from the base policies, the filter selected 33.7% from the trained pair. Of all comparisons generated by each pair, 7.0% from the trained policies both passed the filter and reversed a clear oracle preference (ε = 0.3), compared with 2.6% from the base policies (Section 5.3).

  3. Retraining on the selected comparisons improves evaluator accuracy on unseen prompts. Adding 100 oracle-labeled comparisons from the jointly trained pair and retraining from scratch raised accuracy from 70.33% to 83.33% on held-out game-generated comparisons and from 75.55% to 76.65% on held-out UltraFeedback comparisons (Section 5.4). With the same annotation budget, comparisons from the base policies yielded 68.67% accuracy on the game-generated set, while comparisons from a Follower trained against a fixed Leader yielded 80.83%. The results are statistically significant (Figure 3).

  4. The Leader remains competitive with policies trained directly on preferences. Although the Leader is trained to minimize ensemble disagreement, the oracle preferred its responses in 54.89% of comparisons against NLHF and 50.70% against the SLHF Leader (Section 5.5).

A large-scale, iterative test of our method on more complex agentic tasks over longer horizons is an exciting direction for future work.

We thank the reviewers for their thoughtful feedback, which greatly strengthened the paper. Reviewers described the task of identifying preference-model errors as “an important problem,” and highlighted evaluator unreliability as “an important weakness of preference-based alignment.” We also appreciate Reviewers assessment that our results are “valuable”. We thank the reviewers and area chairs for their service to the academic community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.