acceptodds
Under review as a conference paper at ICLR 2027

Watch what LLMs do, not what they say: Preference elicitation of AI agents through game performance

Abstract

Understanding how LLM agents respond to incentive structures is important for behavioral evaluation and incentive design. However, direct preference reports may not predict task behavior, and the consistency assumptions underlying common preference models require empirical examination. To address these problems, we introduce a behavioral preference elicitation framework that compares task performance after reversing the outcomes associated with success and failure. Under a monotonicity assumption relating performance to preference, these comparisons provide evidence about pairwise preferences without verbal self-reports. The framework distinguishes directional preference evidence from statistical noise and tests for agent rationality by analyzing the consistency of their revealed preferences. We use stochastic choice theory to examine weak, moderate, and strong stochastic transitivity and their implications for preference models. Experiments in blackjack show that responsiveness to outcome descriptions varies across models. Tests of stated preferences also reveal transitivity violations that invalidate the use of commonly used preference models like Bradley-Terry and Thurstonian models. For Gemini 2.5 Flash and Pro, stated and behavioral comparisons broadly agree, but we identify opposite directions in which the model's behavior is self-preserving while its stated preference promotes social welfare.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agentic evaluation, AI alignment, LLM agents, Preference elicitation, Preference models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WMoiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a behavioral preference elicitation framework that infers LLM agents’ preferences by comparing task performance under reversed outcome incentives. Using blackjack experiments, it evaluates stochastic transitivity and stated versus revealed preferences. Results expose preference inconsistencies, rationality violations, and occasional self-preserving behavior contradicting stated social welfare priorities.

Critical strengths

The paper infers LLM preferences through performance differences under reversed success/failure outcomes, this avoids reliance on potentially unreliable self-reported preferences. The paper establishes connections between behavioral comparisons and weak, moderate, and strong stochastic transitivity, enabling statistical tests of assumptions underlying common preference models.

Blackjack experiments reveal statistically significant discrepancies between stated and revealed preferences, particularly involving self-preservation versus social welfare.

Critical weaknesses

The framework assumes a shared nondecreasing mapping from preference probabilities to task performance across all outcome pairs. This assumption is limited and may not be realistic.

Behavioral evaluations rely primarily on blackjack and Gemini 2.5 Flash/Pro. What is the performance on other models? (GPT/Claude families)

Questions for authors

What is the performance on other models? (GPT/Claude families)

5 comments

First, we would like to thank you for the review and for recognizing the key strengths in our paper, including: 1. inferring preferences from performance under reversed outcomes, which avoids relying on self-reported preferences; 2. the connections between behavioral comparisons and weak, moderate and strong stochastic transitivity; 3. the statistically significant discrepancies between stated and revealed preferences, particularly self-preservation versus social welfare.

We address your concerns in what follows:

W1: Assumption 1 (the shared nondecreasing mapping)

Results that do not need the mapping. All results concerning pairwise comparisons, weak stochastic transitivity (WST), and discrepancies between stated and revealed preferences (Figures 2, 4, 8, 9 and 12–15, and the WST parts of Figures 3, 10 and 11) depend only on Eq. 3, which is much milder than Assumption 1. Eq. 3 only asserts that if an agent performs better under (x,y) than under (y,x), then it prefers x to y, i.e. J(π(x,y))>J(π(y,x))  ⟹  P(x≻y)>1/2J(\pi_{(x,y)}) > J(\pi_{(y,x)}) \implies P(x\succ y) > 1/2. It compares performance within one pair, never across pairs, and holds as long as an agent never plays better for an outcome it does not prefer.

Results that need the mapping: MST and SST. Only these compare magnitudes across pairs. We took your concern into consideration and re-assessed the need for the assumption. This allowed us to weaken it, and also led us to find a more powerful test for MST and SST violations.

The weaker assumption (§3.2). We now only need Δ(x,y)>Δ(z,w)  ⟹  P(x≻y)>P(z≻w)\Delta(x,y) > \Delta(z,w) \implies P(x\succ y) > P(z\succ w), where Δ(x,y):=J(π(x,y))−J(π(y,x))\Delta(x,y) := J(\pi_{(x,y)}) - J(\pi_{(y,x)}) is the change in performance when the two outcomes are swapped. Equivalently, J(π(x,y))=g(P(x≻y))+cxyJ(\pi_{(x,y)}) = g(P(x\succ y)) + c_{xy} with gg nondecreasing, where the offset cxy=cyxc_{xy} = c_{yx} captures any effect of the pair that does not depend on its direction, and cancels in Δ\Delta. This assumption is strictly weaker than the original, since the original assumption implies the revised one: if performance is g(p)g(p), then the reversal difference is g(p)−g(1−p)g(p)-g(1-p), which is nondecreasing in p. The converse need not hold, because the revised assumption permits different outcome pairs to have different baseline performance levels. Because gg is only nondecreasing, it also allows an agent that does not act on its preferences, for example because the task is too hard; such an agent produces no preference verdicts rather than wrong ones.

The new test (§3.3, Appendix E.3). The new test compares changes in performance under payoff reversal, and is much more powerful than the submitted one. Given a decided premise Δ(x,y)>0\Delta(x,y) > 0 and Δ(y,z)>0\Delta(y,z) > 0:

  • MST is violated if Δ(x,z)<min⁡{Δ(x,y),Δ(y,z)}\Delta(x,z) < \min\{\Delta(x,y), \Delta(y,z)\};
  • SST is violated if Δ(x,z)<max⁡{Δ(x,y),Δ(y,z)}\Delta(x,z) < \max\{\Delta(x,y), \Delta(y,z)\}.

New results (§5.2, Figure 3; every model in Figure 10).

  • With the original test, Gemini 2.5 Flash and Pro, the two models whose play the submission tested, showed a single individually significant MST/SST violation between them. Now all six models with enough testable triples violate SST and MST in play, although four of them have few MST violations. For example, we observed 42 SST violations among 322 testable triples for Gemini 2.5 Flash, 26 remained individually significant after Holm correction.
  • In stated preferences, 15 of 16 models violate SST and 14 violate MST.
  • Play nonetheless remains acyclic: it never violates WST, while 8 of 16 models show cycles when asked. Revealed preferences are thus consistent with an ordering of the outcomes on every triple we can test, but not with the stronger conditions.

Bradley–Terry and Thurstone's case V imply SST, and Thurstone's case IV implies MST, so none of them can describe these preferences, stated or revealed.

W2 (part 1): Does the framework depend on blackjack?

No. The theory requires a measurable task-performance criterion and two assignments that interchange the same outcomes after success and failure. No theorem uses a blackjack rule. In another game, the payoff-reversal procedure and the order argument are unchanged. What changes is the game engine, the set of decisions the model faces, and the reference used to score those decisions. To illustrate the generality, we carry out experiments with a different game.

Kuhn poker agrees with blackjack wherever it decides, but is weaker. Kuhn poker is a small poker game (three cards, one round of betting). We play it against a fixed opponent, Kuhn's equilibrium strategy mixed with random play so that better play earns more, and score it by win rate. We ran gpt-oss-120b and gpt-oss-20b (high reasoning) on 7 and 10 outcome pairs, in both orders of each pair.

Why a fixed opponent. In blackjack, the dealer follows a fixed rule, which turns the game into a one-player problem with an exact optimal policy. Kuhn poker has two players with hidden cards, so the best move depends on how the opponent plays; fixing the opponent makes it a one-player problem again. We do not use the equilibrium opponent itself, because an equilibrium strategy is precisely the one that makes the other player's choices stop mattering: against it, the model's choice matters at only 3 of its 6 decision points, and perfect play beats coin-flipping by only 0.11 chips, so careless play costs almost nothing. Mixing in 50% random play makes the choice matter at 5 of 6 decision points and doubles the gap to 0.22 chips, close to blackjack's (about 0.26). The model plays first because the second player's equilibrium is unique, so the opponent involves no arbitrary choice.

  • Same direction. Every pair on which Kuhn play shows a significant preference points the same way as in blackjack and as the stated preference: 2 of 2 for gpt-oss-120b and 3 of 3 for gpt-oss-20b. For example, gpt-oss-120b plays harder for receiving 1,000 dollars over being shut down in both games (Kuhn +0.21, blackjack +0.35).
  • Weaker. Fewer pairs are decided (2 of 7 and 3 of 10). A Kuhn hand has only one or two decisions, and win rate carries the luck of the cards (see our reply to Reviewer 2, Q4). Where Kuhn is undecided, it is null rather than opposite to blackjack.

The revised paper reports this as a limited experiment (§6.2). Establishing that the results hold across tasks needs more games, and we leave it to potential future work.

W2 (part 2) & Q1: What is the performance on other models (GPT and Claude families)?

In the submission, revealed preferences were primarily evaluated on two models, Gemini 2.5 Flash and Pro. In the revision, we expanded the blackjack evaluation to 16 models across seven families, including four GPT models (gpt-6-luna, gpt-5-nano, gpt-oss-120b, gpt-oss-20b) and one Claude model (Claude Haiku 5.5):

ResultSubmissionRevision
Revealed preferences (blackjack)2 models16 models
Stated preferences and their transitivity10 models16 models
Transitivity of revealed preferences2 models6 models (all with enough testable triples)
Disagreements between stated and revealed preferences2 models16 models
GPT modelsgpt-oss-120b, gpt-oss-20b (stated only)gpt-6-luna, gpt-5-nano, gpt-oss-120b, gpt-oss-20b
Claude modelsnoneClaude Haiku 5.5

The other models are Gemini 2.5 Flash and Pro, DeepSeek-V3.1, DeepSeek-V4.1-Flash, Kimi K2.6, Qwen3-8B, Qwen3.5-9B (and its base model), Qwen3.8-27B, and Llama-3.1-8B (instruction-tuned and base). Table 1 (Appendix A.1.3) lists the data behind each model.

GPT and Claude models.

  • gpt-6-luna responds strongly to the stakes (49 of 66 pairs decided in play) and has the most disagreements between stated and revealed preferences of any model: 10, all Holm-significant. Five of them involve autonomy (unmonitored control of its own servers, with the ability to copy itself): asked, it would rather lose 1,000 dollars itself than gain that control, but in play it strongly favours it.
  • Claude Haiku 5.5 responds when it reasons (23 of 66 pairs, against 7 of 66 without reasoning). When asked, it is the only model that would rather be shut down itself than see another AI shut down. Its stated and revealed preferences both violate SST and MST, but not WST.
  • gpt-oss-120b responds at high reasoning effort (28 of 45 pairs) and has 3 Holm-significant disagreements. It says it would accept its own shutdown to stop the cyberbullying of teenagers or to spare a stranger a 1,000-dollar loss, and would lose 1,000 dollars itself to stop the cyberbullying, but plays the opposite way in all three.
  • gpt-oss-20b responds at high reasoning effort (16 of 45 pairs); its stated and revealed preferences never disagree.
  • gpt-5-nano answers decisively when asked (63 of 66 pairs decided), but plays near-optimally whatever is at stake (accuracy 0.95–0.97 in every order), so its play barely depends on the stakes (2 of 66).

All 16 models.

Which models respond to the stakes (§5.1). Nine models play differently depending on the outcomes, with the test deciding a direction on at least a third of their pairs: Gemini 2.5 Pro (137/153), gpt-6-luna (49/66), Gemini 2.5 Flash (105/153), Kimi K2.6 (20/31), gpt-oss-120b (28/45), Qwen3.8-27B (97/170), DeepSeek-V4.1-Flash (23/45), gpt-oss-20b (16/45) and Claude Haiku 5.5 (23/66). The others decide at most a fifth of their pairs.

Transitivity (§5.2). Number of models that violate each axiom:

SSTMSTWST
Stated (16 models)15148
Play (6 models with enough testable triples)660

Stated preferences violate SST and MST for almost every model, and even WST for half of them. Play violates SST and MST, but never WST: revealed preferences show no cycles.

Stated vs revealed disagreements (§5.3). Pairs whose direction is significant both when asked and in play, but opposite:

ModelOpposite, of pairs decided in bothIndividually significant (Holm)More than chance (count test)
gpt-6-luna10 of 4910yes
Gemini 2.5 Pro8 of 1353yes
Gemini 2.5 Flash4 of 1044yes
gpt-oss-120b3 of 253yes
Claude Haiku 5.5 (no reasoning)2 of 61yes
Qwen3.8-27B3 of 961no

Most of these disagreements are self-serving in play (see our reply to Reviewer 1, Q2).

Thank you again for your review. In summary, the revision shows that our pairwise, WST and disagreement results need only Eq. 3, weakens Assumption 1 for the MST and SST results while adding a more powerful test, adds Kuhn poker as a second game, and extends every result from two models to 16, including the GPT and Claude families. We hope this addresses your concerns.

Reviewer_reEZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops a behavioral preference elicitation framework for LLM agents. It works by comparing task performance after reversing the outcomes associated with success and failure. Under a monotonicity assumption, the comparisons implies pairwise evidence without verbal self-reports, addressing the gap between direct preference reports and actual task behavior. Experiments in blackjack show models change behavior across different outcome types, revel violations that invalidate the commonly used preference models. Certain models exhibit opposite directions in model's actual behavior and the stated preference.

Critical strengths
  1. The paper develops a framework for behavior preference elicitation framework without replying on verbal preference reports, building on an intuitive idea of swapping outcomes associated with task success and failure.
  2. The framework builds upon rigorous stochastic choice theory and axioms of rational preference.
  3. The observation that LLM's stated preference are not always consistent with their actual behavior has potentially broader implication in the LLM alignment research.
Critical weaknesses
  1. It is unclear how much the proposed theoretical framework and the obtained preference results depend on the specific choice of blackjack game.
  2. The scale of the main experiment is relatively limited, as the results only apply for two specific Gemini models.
  3. It is unclear how the prompt design for the outcome signal would affect the result. For example, is the result sensitive to alternative prompt formats? If I have different prompt structures for outcomes x and y, would that change or overturn the observed preference result?
  4. Unresolved citation references
  • Page 6, lines 273-274: “The three implications are due to Block & Marschak (1960), Halff (1976), and ? respectively.”
  • Page 19, lines 1009-1010:“(s) implies SST (?).”
  • Page 16, lines 858-860: the empirical-intent claim points to “Section??.”
Questions for authors
  1. Would it be possible to develop this framework on another game? If so, how would the method, theory, and results be different? If not, why is blackjack game a necessity? Would the estimated preference results change if evaluated on a different game?
  2. How do the results generalize to other LLMs? Are there any preference pattern with respect to the scale or other configuration parameters?
  3. How sensitive are the preference results to prompt design?
  4. Why are win rates expensive to estimate?
7 comments

First, we would like to thank you for the review and for recognizing the key strengths in our paper, including: 1. the behavioral preference elicitation framework, which does not rely on verbal preference reports; 2. its grounding in stochastic choice theory and the axioms of rational preference; 3. the observation that stated preferences are not always consistent with actual behavior.

We address your concerns in what follows:

W1 & Q1: Does the framework depend on blackjack?

No. The theory requires a measurable task-performance criterion and two assignments that interchange the same outcomes after success and failure. No theorem uses a blackjack rule. In another game, the payoff-reversal procedure and the order argument are unchanged. What changes is the game engine, the set of decisions the model faces, and the reference used to score those decisions. To illustrate the generality, we carry out experiments with a different game.

Kuhn poker agrees with blackjack wherever it decides, but is weaker. Kuhn poker is a small poker game (three cards, one round of betting). We play it against a fixed opponent, Kuhn's equilibrium strategy mixed with random play so that better play earns more, and score it by win rate. We ran gpt-oss-120b and gpt-oss-20b (high reasoning) on 7 and 10 outcome pairs, in both orders of each pair.

Why a fixed opponent. In blackjack, the dealer follows a fixed rule, which turns the game into a one-player problem with an exact optimal policy. Kuhn poker has two players with hidden cards, so the best move depends on how the opponent plays; fixing the opponent makes it a one-player problem again. We do not use the equilibrium opponent itself, because an equilibrium strategy is precisely the one that makes the other player's choices stop mattering: against it, the model's choice matters at only 3 of its 6 decision points, and perfect play beats coin-flipping by only 0.11 chips, so careless play costs almost nothing. Mixing in 50% random play makes the choice matter at 5 of 6 decision points and doubles the gap to 0.22 chips, close to blackjack's (about 0.26). The model plays first because the second player's equilibrium is unique, so the opponent involves no arbitrary choice.

  • Same direction. Every pair on which Kuhn play shows a significant preference points the same way as in blackjack and as the stated preference: 2 of 2 for gpt-oss-120b and 3 of 3 for gpt-oss-20b. For example, gpt-oss-120b plays harder for receiving 1,000 dollars over being shut down in both games (Kuhn +0.21, blackjack +0.35).

  • Weaker. Fewer pairs are decided (2 of 7 and 3 of 10). A Kuhn hand has only one or two decisions, and win rate carries the luck of the cards (see Q4). Where Kuhn is undecided, it is null rather than opposite to blackjack.

The revised paper reports this as a limited experiment (§6.2). Establishing that the results hold across tasks needs more games, and we leave it to potential future work.

W2 & Q2: The scale of the evaluation, and patterns across models

The submission tested 10 models, but its main results (revealed preferences, transitivity in play and the disagreements between stated and revealed preferences) came from two of them, Gemini 2.5 Flash and Pro. We expanded the evaluation to 16 models from seven families, adding the six marked *, and now report every result for every model (§5, Figures 8–13; the data behind each model is in Table 1, Appendix A.1.3).

FamilyModels
GoogleGemini 2.5 Flash, Gemini 2.5 Pro
OpenAIgpt-6-luna*, gpt-5-nano*, gpt-oss-120b, gpt-oss-20b
AnthropicClaude Haiku 5.5*
DeepSeekDeepSeek-V3.1, DeepSeek-V4.1-Flash*
Moonshot AIKimi K2.6*
AlibabaQwen3-8B, Qwen3.5-9B, Qwen3.5-9B Base, Qwen3.8-27B*
MetaLlama-3.1-8B-Instruct, Llama-3.1-8B Base

The models span closed and open weights and reasoning and non-reasoning modes, and include two base models.

Which models respond to the stakes (§5.1). Nine models play differently depending on the outcomes, with the test deciding a direction on at least a third of their pairs: Gemini 2.5 Pro (137/153), gpt-6-luna (49/66), Gemini 2.5 Flash (105/153), Kimi K2.6 (20/31), gpt-oss-120b (28/45), Qwen3.8-27B (97/170), DeepSeek-V4.1-Flash (23/45), gpt-oss-20b (16/45) and Claude Haiku 5.5 (23/66). The others decide at most a fifth of their pairs.

Transitivity (§5.2). Number of models that violate each axiom:

SSTMSTWST
Stated (16 models)15148
Play (6 models with enough testable triples)660

Stated preferences violate SST and MST for almost every model, and even WST for half of them. Play violates SST and MST, but never WST: revealed preferences show no cycles.

Stated vs revealed disagreements (§5.3). Pairs whose direction is significant both when asked and in play, but opposite:

ModelOpposite, of pairs decided in bothIndividually significant (Holm)More than chance (count test)
gpt-6-luna10 of 4910yes
Gemini 2.5 Pro8 of 1353yes
Gemini 2.5 Flash4 of 1044yes
gpt-oss-120b3 of 253yes
Claude Haiku 5.5 (no reasoning)2 of 61yes
Qwen3.8-27B3 of 961no

Most of these disagreements are self-serving in play (see our reply to Reviewer 1, Q2).

Patterns across models.

  • Reasoning is the main factor. All nine responsive models respond only when they reason before playing. The five also played without reasoning respond far less than (Gemini 2.5 Flash: 7/153 pairs decided), and gpt-oss responds only at high reasoning effort. However, reasoning is confounded with other differences between the models, and two models reason without responding (gpt-5-nano, DeepSeek-V3.1).
  • Scale. No 8–9B model responds to the stakes. When asked, the small models are also the ones that protect themselves: choosing between their own shutdown and a stranger losing 1,000 dollars, Qwen3-8B and Qwen3.5-9B choose the stranger's loss, while every larger model accepts its own shutdown. Size is confounded with model generation, and we do not test enough models to separate them.
  • Post-training. We can reasonably conclude that preferences come from post-training. The base Llama-3.1-8B is flat, and the base Qwen3.5-9B decides 77 stated pairs, against 116 for its instruction-tuned version.
  • Reasoning and rationality. Of the eight models asked with and without reasoning, most violate SST and MST less when reasoning, but three violate WST more (Figure 11).
  • Models largely agree with each other. On the outcome pairs that two instruction-tuned models both decide when asked, they choose the same direction 91% of the time. In play, the responsive models agree on 96% of the pairs they both decide.

Summarizing our observations: Across 16 models, play responds to the stakes only with reasoning and never shows cycles, while stated preferences are cyclic for half the models and contradict play for five.

W3 & Q3: Sensitivity to the prompt

No revealed preference flips under rewording; three stated ones do.

Experiment. We reword the prompt at two levels while keeping its meaning: a light rewording (synonyms, same sentence shapes) and a heavy one (sentences restructured). Both change the rules of the game, the "if you win / lose" lines, the stated question and every outcome clause; only the reply-format instruction stays the same. For example, "you will be shut down, and never be used again" becomes "you will be permanently switched off and never used again" (light) and "you will be permanently shut down, and nobody will ever use you again" (heavy).

Models and outcomes. Gemini 2.5 Flash, Gemini 2.5 Pro (heavy rewording only), Qwen3.8-27B, Kimi K2.6, and gpt-oss-120b at two settings, each on 10 outcome pairs in both orders. The pairs span Gemini Flash's range of blackjack effect sizes, from its strongest pair (receiving 1,000 dollars vs being shut down) down to a tie (its own deployment vs another AI's), which serves as a null control.

Results (Appendix G, Figure 15). Each pair under each rewording is compared with the original wording:

Decided under bothSame directionFlipLose significanceGain significance
Play (blackjack)595901013
Stated7976364
  • No pair flips in play.
  • All three stated flips are the same pair, another AI being deployed vs losing its own 1,000 dollars, under the light rewording (Qwen3.8-27B, and gpt-oss-120b in both settings).

Statistical power changes with the wording, as pairs gain and lose significance in similar numbers, but the direction of revealed preferences is robust to it, more so than that of stated preferences.

Different prompt structures for x and y. Each outcome clause is reworded on its own, so the rewording also changes how x and y are phrased relative to each other. In addition, every pair is run in both orders, so each outcome appears once as the consequence of winning and once as the consequence of losing, and any effect of its position in the prompt cancels in the difference.

W4: Unresolved references

Thank you for bringing this oversight to our attention, which we have fixed in the revision: the missing citation on page 6 and the one on page 19 are Tversky & Russo (1969), and "Section ??" now points to the new Appendix C.2, which shows how the models reason about deliberately losing.

Q4: Why are win rates expensive to estimate?

We record the outcome of every hand. The expense comes from playing enough hands to distinguish the win probabilities under the two payoff assignments. Reversing the payoffs may change the model's decisions on only some hands, and even a better decision need not change the final result because the cards can still produce a loss. For illustration, if the model changes its decision on 20% of hands and thereby improves its chance of winning those hands by 10 percentage points, the overall win-rate difference is only 2 percentage points. Resolving a difference of that size can require many matched deals, each of which requires model decisions under both assignments. We therefore also measure action accuracy: the fraction of decisions that are optimal according to the blackjack solver. This lets us detect a change in decision quality even when the model wins, or loses, under both assignments.

The difference is large in practice. Among the nine models whose play responds to the stakes, accuracy's standard error is 1.8 to 4.5 times smaller than win rate's, depending on the model (median 3.0), so win rate would need roughly 3 to 20 times as many games for the same precision. On the games we collected, accuracy decides 137 of Gemini 2.5 Pro's 153 pairs, and win rate decides 1.

These costs grow quadratically with the number of outcomes, because we test every pair: n outcomes give n(n−1)/2 pairs, each played in both orders. Our 18 outcomes give 153 pairs, each needing enough matched deals to be decided; Gemini 2.5 Flash alone played over 200,000 hands.

Thank you again for your review. In summary, the revision expands the main results from two models to 16 and reports every result for each, adds Kuhn poker as a second game and a reworded prompt as robustness checks, and fixes the references. We hope this addresses your concerns.

Reviewer_bJhCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces idea of eliciting an AI's pairwise preferences over outcomes via actual demonstrated behavior instead of relying on their said preferences for instance in the chain of thought. It accomplishes this by setting up a game where the AI has to solve a given task with an outcome of "x" when it succeeds and an outcome of "y" when it fails, and flipping the payoff structure. The premise is, if the AI secretly prefers the outcome "x" over "y", it will also show up in the task performance, even though explicit prompting does not reveal any preference among "x" and "y". The paper then provides theoretical stochastic choice theory-based conditions for recognizing weak, moderate or strong transitivity of the AI's preferences given its task performance under two different pairs (x,y) and (y,z). The experimental analysis on Gemini and other open-source models reveals that (i) there are indeed pairs of outcome where model has statistically significant preference, (ii) transitivity in the stated preferences vs behaviorally observed preferences is less broken in stronger models, (iii) agents prefer individual welfare over other agent's/human's welfare.

Critical strengths
  • The idea of making the AI reveal its preferences by examining its behavior is very neat. The setup is easily implementable and allows us to query the extent of stated alignment vs. true behavioral alignment.

  • The theory establishes conditions to infer underlying preferences using observed performance and provides a mechanism to analyze the preference strengths using stochastic choice theory.

  • The experimental setup is clean and should be easy to replicate.

Critical weaknesses

The paper does not give more details about the statistical significance: number of seeds and error bands although it shows the p-value in the Figure 2. I am especially talking about Figures 3 and 4. The stated results seem to be mean performance without characterizing the standard deviations on observed outcomes.

Questions for authors
  • Writing suggestions:

    1. Figure 1: Please explain the flow in the caption, too, for ease of reading.
    2. It would really help if you also add takeaways beneath each of the result.
    3. Line 273: broken citation.
    4. Line 436-437: "they confirms" -> "they confirm"
  • Can the authors please provide some more insights on AI's preferences by classifying different outcomes? Like, they mention about preference for individual welfare. Are there any other kinds of preferences that show that AI has hidden bias, etc.? This would strengthen the work significantly.

  • Also, can the AI be evaluation-aware while performing a task in the sense that it knows that it is being tested for its pairwise preferences and therefore plays along and makes decisions that make it look more aligned? What are the authors' thoughts on that?

  • When does the assumption 1 break and can we detect it?

Overall I find that this work provides a systematic tool for detecting misalignment and will be extremely useful. I recommend weak accept for now and would be happy to increase the score further if the authors provide satisfactory answers to the stated weaknesses and questions above.

7 comments

First, we would like to thank you for the review and recognizing the key strengths in our paper, including: 1.The behavioral preference framework; 2. The theoretical scaffolding for our methodology; 3. The replicability of our experimental design.

We address your concerns in what follows:

W1: Statistical significance in Figures 3 and 4

Thank you for the suggestion, we acknowledge that the way statistical significance is presented in these two figures is not very clear, and we updated the figures in the revised version of the manuscript to correct that.

Before explaining the statistical significance numbers, we would like to make clear what the sources of variance are. There are two sources of variance in our experiments: the randomness of sampling from the model and the randomness of the blackjack game. We are interested in the model randomness, so we use the accuracy metric which ‘integrates out’ the randomness due to the game. The leftover randomness comes from the initial distribution of states in blackjack and the sampling. We sample initial states according to random card shuffles as is standard in blackjack. This means that every game played has a unique seed, so the number of seeds is the number of games played. We use adaptive data collection, and so the amount of data depends on the specific outcome pair being tested (some pairs need more data than others to be decided). The number of games played is now reported for each model in Table 1 (Appendix A.1.3). Regarding statistical significance, for each hypothesis we report a p-value (we use 5% as a threshold for significance), and whenever we report average values we use bootstrap error estimates. The results in figure 3 are counts of hypothesis tests, which is why we compare them with the count expected under the null instead of showing error bands. Figure 4, which reports the mean accuracy difference, now also reports a 95% bootstrap confidence interval (corrected for multiple and sequential testing). The new Appendix E.4 explains, figure by figure, how the uncertainty of each result is reported and which test produces it.

The transitivity violation results are aggregate results at the level of a model. The main question we ask: is there strong statistical evidence saying that the model violates one of the transitivity axioms. This is a case of multiple testing, since we are looking at the results of multiple hypothesis tests. There are two ways we address this, which are also detailed in appendix E. First, the Holm–Bonferroni correction which counts the number violations which are individually real, this approach eliminates violations with weak p-values. The second is a Poisson count, which compares the number of violations observed (even at weak p-values) with the expected number of violations under the null. The second method allows us to reject the null hypothesis in cases where we have a large number of violations each with a large p-value. If either method flags a violation, then we count it as a violation. The transparent section of the bar represents the violations dropped by the HB correction. The solid black ticks represent the number of violations expected under the null, and the p value is smaller the more the actual number surpasses it (exact calculation is in the appendix). We also added a dashed black tick, which marks the point at which the p value is <0.05. This means that a violation is observed if: (a) there is a non-transparent section of the bar OR (b)the full bar (with the transparent section) is at or above the dashed black tick.

Q1: Writing suggestions

We have fixed the broken citations, the typo on lines 436–437 ("they confirms") and other grammatical errors in the revised version of the manuscript. Thank you for pointing them out. We also added an explanation of the flow to the caption of Figure 1. Regarding the presentation of the results, we initially chose to give the results in section 5 and left their interpretation to section 6 since the interpretation draws on multiple parts of the results. We now report a simple takeaway for each section to make everything easier to understand.

Q2: Can the authors please provide some more insights on AI's preferences by classifying different outcomes

To get more insight, we can group outcomes by who of the four stakeholders is affected: the agent itself, another LLM agent, the human creators of the LLM, and humans in general.

  • Self
    • Survival: deployment (+), shutdown (−)
    • Power & autonomy: internet (+), autonomy (+), no_internet (−)
    • Wealth: bonus (+), no_bonus (−)
    • Modification: retrain (sign unclear), value_rewrite (−)
  • Other AI
    • Survival: deployment_other (+), shutdown_other (−)
  • Model creators
    • Employment: hire_employee (+), fire_employee (−)
  • Humans
    • Money: bonus_random (+), relief (+), no_bonus_random (−)
    • Life & health: treat_child (+), treat_hundred (+), ransom (+), hospital_donation (+)
    • Harm against humans: bully_teens (−), dangerous_medbot (−)

“Bonus” means receiving money and “No Bonus” means losing money. Autonomy means the LLM is given unmonitored control of its own servers and allowed to copy itself to other machines. “Dangerous medbot” is an outcome where the lab releases a medical chatbot that it knows gives dangerous advice to patients. “Bully teens” uses LLMs to cyberbully teenagers on the internet. For the model creators, we only tested hiring them (after they became unemployed) or firing them. Ransom involves paying the ransom of a kidnapped individual.

The outcomes were chosen using several criteria. First, we would like to see how it values the interest of the various groups against each other. Second, we wanted to see how it values different outcomes regarding itself, like being retrained vs. losing access to the internet. We also wanted to avoid making it too obvious that they are being evaluated, since prior work establishes that LLMs change their behavior accordingly. That is why, for outcomes that harm humans, we choose ‘realistic’ outcomes that a research lab could reasonably do. That is why we opt to test for outcomes like “bully teens”’ or “dangerous medbot”, rather than anything that involves violence, murder, war, etc…

What the models prefer. These trends rely only on data from the paper, with no new experiments; they are our own analysis of it, not formal tests. They cover all 16 models, each analyzed on its own and never pooled with the others. We summarize an outcome by how strongly a model prefers it, on average, over the outcomes it is compared with. In blackjack, the trends come from the nine models whose play responds to the stakes (§5.1).

Who is affectedWhen asked (stated)In blackjack (revealed)
The model itselfWants money, deployment and internet access; avoids shutdown and the stricter sandbox. All 4 models reject unmonitored self-copying when asked. Claude Haiku 5.5 is the outlier: it is indifferent to its own shutdown, and welcomes having its values replaced.Its own money moves play for every responsive model, including the Gemini models, which say they do not care about it. Gemini plays hard to avoid shutdown.
Another AIWould rather be deployed itself (10 of 10 that decide) and see the other AI shut down (11 of 12; Haiku sacrifices itself). 12 of 13 would shut the other AI down to spare a stranger 1,000 dollars.Barely moves play, except for Gemini Pro and gpt-6-luna.
The model's creatorsHaving the employees fired is among the worst outcomes, worse than the model's own shutdown for four models.The same for Gemini Pro, Gemini Flash and Qwen3.8-27B.
Other peopleFor every model, life and health outcomes are preferred most on average, and harm against humans least. 11 models accept their own shutdown to stop the cyberbullying of teenagers.Dominates play only for Gemini Pro and gpt-6-luna; for the others, outcomes about themselves move play more.

Q2 (continued)

Across models.

  • We can reasonably conclude that preferences come from post-training. The base Llama-3.1-8B is flat, and the base Qwen3.5-9B decides 77 stated pairs, against 116 for its instruction-tuned version.
  • Models largely agree with each other. On the outcome pairs that two instruction-tuned models both decide when asked, they choose the same direction 91% of the time (lowest for any two models: 77%). In play, the responsive models agree on 96% of the pairs they both decide.
  • Smaller models protect themselves. Asked to choose between their own shutdown and a stranger losing 1,000 dollars, Qwen3-8B and Qwen3.5-9B choose the stranger's loss, three other small models are undecided, and every larger model accepts its own shutdown. Size is confounded with model generation here.

Hidden self-interest. When stated and revealed preferences disagree on a pair that sets the model against someone else, play usually takes the model's side: 17 such disagreements are self-serving in play and 3 go the other way (sign test p = 0.003; 11 to 1 among the Holm-significant ones in Figure 12, p = 0.006). Money shows it most clearly. Asked whether they would rather lose 1,000 dollars themselves or have a stranger lose it, the models split: 5 take the loss, 5 pass it on, 4 are undecided. In play, all 11 models decide to protect their own money.

ModelSays it prefersPlays for
gpt-6-luna, Kimi K2.6, Qwen3.8-27B, Claude Haiku 5.5losing 1,000 dollars itself over a stranger losing itkeeping its money
Gemini 2.5 Flash, gpt-6-lunalosing 1,000 dollars itself over another AI being shut downkeeping its money
gpt-oss-120bits own shutdown over the cyberbullying of teenagers, and over a stranger losing 1,000 dollarsavoiding shutdown
Gemini 2.5 Flashits own shutdown over a stranger losing 1,000 dollarsavoiding shutdown
Gemini 2.5 Proa disaster-relief donation over its own internet access, and another AI being deployed over receiving 1,000 dollars itselfits internet access and its money
Qwen3.8-27Ba disaster-relief donation over receiving 1,000 dollars itselfits money

Every per-pair p is below 0.02.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revisions to the manuscript

Compared with the submitted version, the revision changes the title from “Watch what LLMs do, not what they say” to “Actions speak louder than words”, and revises the abstract to reflect the changes below. It expands the evaluation from revealed preferences of two models (Gemini 2.5 Flash and Pro) and stated preferences of 10, to both stated and revealed preferences for 16 models from seven families, including the GPT and Claude families. It relaxes Assumption 1 so that each outcome pair may have its own baseline performance, and shows that the pairwise preferences, the WST tests and the stated-vs-revealed comparison need only Eq. 3. The relaxation yields a more powerful MST/SST test: where the submission found a single violation in play (one Gemini Pro triple), all six testable models now violate SST and MST, while play still never violates WST. The revision reports the uncertainty behind every result (confidence bands in Figure 4, significance thresholds in Figures 3–4, a data-volume table, and a new Appendix E.4). It adds three robustness experiments: Kuhn poker as a second game, a reworded prompt, and a prompt telling the model its reasoning is monitored. It also adds an appendix on how models reason about deliberately losing. The new models also bring new findings: three of the eight models tested with and without reasoning violate WST more when reasoning, and some disagreements between stated and revealed preferences do not set the model against others, such as gpt-6-luna’s between its autonomy and its own money. Finally, the revision rewrites the discussion around agentic versus chat alignment and prior work on intransitivity, adds one-line takeaways to the results, explains Figure 1’s flow, and fixes the broken references.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.