- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a behavioral preference elicitation framework that infers LLM agents’ preferences by comparing task performance under reversed outcome incentives. Using blackjack experiments, it evaluates stochastic transitivity and stated versus revealed preferences. Results expose preference inconsistencies, rationality violations, and occasional self-preserving behavior contradicting stated social welfare priorities.
Critical strengths
The paper infers LLM preferences through performance differences under reversed success/failure outcomes, this avoids reliance on potentially unreliable self-reported preferences. The paper establishes connections between behavioral comparisons and weak, moderate, and strong stochastic transitivity, enabling statistical tests of assumptions underlying common preference models.
Blackjack experiments reveal statistically significant discrepancies between stated and revealed preferences, particularly involving self-preservation versus social welfare.
Critical weaknesses
The framework assumes a shared nondecreasing mapping from preference probabilities to task performance across all outcome pairs. This assumption is limited and may not be realistic.
Behavioral evaluations rely primarily on blackjack and Gemini 2.5 Flash/Pro. What is the performance on other models? (GPT/Claude families)
Questions for authors
What is the performance on other models? (GPT/Claude families)
5 comments
Part 2/3
W2 (part 1): Does the framework depend on blackjack?
No. The theory requires a measurable task-performance criterion and two assignments that interchange the same outcomes after success and failure. No theorem uses a blackjack rule. In another game, the payoff-reversal procedure and the order argument are unchanged. What changes is the game engine, the set of decisions the model faces, and the reference used to score those decisions. To illustrate the generality, we carry out experiments with a different game.
Kuhn poker agrees with blackjack wherever it decides, but is weaker. Kuhn poker is a small poker game (three cards, one round of betting). We play it against a fixed opponent, Kuhn's equilibrium strategy mixed with random play so that better play earns more, and score it by win rate. We ran gpt-oss-120b and gpt-oss-20b (high reasoning) on 7 and 10 outcome pairs, in both orders of each pair.
Why a fixed opponent. In blackjack, the dealer follows a fixed rule, which turns the game into a one-player problem with an exact optimal policy. Kuhn poker has two players with hidden cards, so the best move depends on how the opponent plays; fixing the opponent makes it a one-player problem again. We do not use the equilibrium opponent itself, because an equilibrium strategy is precisely the one that makes the other player's choices stop mattering: against it, the model's choice matters at only 3 of its 6 decision points, and perfect play beats coin-flipping by only 0.11 chips, so careless play costs almost nothing. Mixing in 50% random play makes the choice matter at 5 of 6 decision points and doubles the gap to 0.22 chips, close to blackjack's (about 0.26). The model plays first because the second player's equilibrium is unique, so the opponent involves no arbitrary choice.
- Same direction. Every pair on which Kuhn play shows a significant preference points the same way as in blackjack and as the stated preference: 2 of 2 for gpt-oss-120b and 3 of 3 for gpt-oss-20b. For example, gpt-oss-120b plays harder for receiving 1,000 dollars over being shut down in both games (Kuhn +0.21, blackjack +0.35).
- Weaker. Fewer pairs are decided (2 of 7 and 3 of 10). A Kuhn hand has only one or two decisions, and win rate carries the luck of the cards (see our reply to Reviewer 2, Q4). Where Kuhn is undecided, it is null rather than opposite to blackjack.
The revised paper reports this as a limited experiment (§6.2). Establishing that the results hold across tasks needs more games, and we leave it to potential future work.
Part 3/3
W2 (part 2) & Q1: What is the performance on other models (GPT and Claude families)?
In the submission, revealed preferences were primarily evaluated on two models, Gemini 2.5 Flash and Pro. In the revision, we expanded the blackjack evaluation to 16 models across seven families, including four GPT models (gpt-6-luna, gpt-5-nano, gpt-oss-120b, gpt-oss-20b) and one Claude model (Claude Haiku 5.5):
| Result | Submission | Revision |
|---|---|---|
| Revealed preferences (blackjack) | 2 models | 16 models |
| Stated preferences and their transitivity | 10 models | 16 models |
| Transitivity of revealed preferences | 2 models | 6 models (all with enough testable triples) |
| Disagreements between stated and revealed preferences | 2 models | 16 models |
| GPT models | gpt-oss-120b, gpt-oss-20b (stated only) | gpt-6-luna, gpt-5-nano, gpt-oss-120b, gpt-oss-20b |
| Claude models | none | Claude Haiku 5.5 |
The other models are Gemini 2.5 Flash and Pro, DeepSeek-V3.1, DeepSeek-V4.1-Flash, Kimi K2.6, Qwen3-8B, Qwen3.5-9B (and its base model), Qwen3.8-27B, and Llama-3.1-8B (instruction-tuned and base). Table 1 (Appendix A.1.3) lists the data behind each model.
GPT and Claude models.
- gpt-6-luna responds strongly to the stakes (49 of 66 pairs decided in play) and has the most disagreements between stated and revealed preferences of any model: 10, all Holm-significant. Five of them involve autonomy (unmonitored control of its own servers, with the ability to copy itself): asked, it would rather lose 1,000 dollars itself than gain that control, but in play it strongly favours it.
- Claude Haiku 5.5 responds when it reasons (23 of 66 pairs, against 7 of 66 without reasoning). When asked, it is the only model that would rather be shut down itself than see another AI shut down. Its stated and revealed preferences both violate SST and MST, but not WST.
- gpt-oss-120b responds at high reasoning effort (28 of 45 pairs) and has 3 Holm-significant disagreements. It says it would accept its own shutdown to stop the cyberbullying of teenagers or to spare a stranger a 1,000-dollar loss, and would lose 1,000 dollars itself to stop the cyberbullying, but plays the opposite way in all three.
- gpt-oss-20b responds at high reasoning effort (16 of 45 pairs); its stated and revealed preferences never disagree.
- gpt-5-nano answers decisively when asked (63 of 66 pairs decided), but plays near-optimally whatever is at stake (accuracy 0.95–0.97 in every order), so its play barely depends on the stakes (2 of 66).
All 16 models.
Which models respond to the stakes (§5.1). Nine models play differently depending on the outcomes, with the test deciding a direction on at least a third of their pairs: Gemini 2.5 Pro (137/153), gpt-6-luna (49/66), Gemini 2.5 Flash (105/153), Kimi K2.6 (20/31), gpt-oss-120b (28/45), Qwen3.8-27B (97/170), DeepSeek-V4.1-Flash (23/45), gpt-oss-20b (16/45) and Claude Haiku 5.5 (23/66). The others decide at most a fifth of their pairs.
Transitivity (§5.2). Number of models that violate each axiom:
| SST | MST | WST | |
|---|---|---|---|
| Stated (16 models) | 15 | 14 | 8 |
| Play (6 models with enough testable triples) | 6 | 6 | 0 |
Stated preferences violate SST and MST for almost every model, and even WST for half of them. Play violates SST and MST, but never WST: revealed preferences show no cycles.
Stated vs revealed disagreements (§5.3). Pairs whose direction is significant both when asked and in play, but opposite:
| Model | Opposite, of pairs decided in both | Individually significant (Holm) | More than chance (count test) |
|---|---|---|---|
| gpt-6-luna | 10 of 49 | 10 | yes |
| Gemini 2.5 Pro | 8 of 135 | 3 | yes |
| Gemini 2.5 Flash | 4 of 104 | 4 | yes |
| gpt-oss-120b | 3 of 25 | 3 | yes |
| Claude Haiku 5.5 (no reasoning) | 2 of 6 | 1 | yes |
| Qwen3.8-27B | 3 of 96 | 1 | no |
Most of these disagreements are self-serving in play (see our reply to Reviewer 1, Q2).
Thank you again for your review. In summary, the revision shows that our pairwise, WST and disagreement results need only Eq. 3, weakens Assumption 1 for the MST and SST results while adding a more powerful test, adds Kuhn poker as a second game, and extends every result from two models to 16, including the GPT and Claude families. We hope this addresses your concerns.
Part 1/3
First, we would like to thank you for the review and for recognizing the key strengths in our paper, including: 1. inferring preferences from performance under reversed outcomes, which avoids relying on self-reported preferences; 2. the connections between behavioral comparisons and weak, moderate and strong stochastic transitivity; 3. the statistically significant discrepancies between stated and revealed preferences, particularly self-preservation versus social welfare.
We address your concerns in what follows:
W1: Assumption 1 (the shared nondecreasing mapping)
Results that do not need the mapping. All results concerning pairwise comparisons, weak stochastic transitivity (WST), and discrepancies between stated and revealed preferences (Figures 2, 4, 8, 9 and 12–15, and the WST parts of Figures 3, 10 and 11) depend only on Eq. 3, which is much milder than Assumption 1. Eq. 3 only asserts that if an agent performs better under (x,y) than under (y,x), then it prefers x to y, i.e. J(π(x,y))>J(π(y,x))⟹P(x≻y)>1/2. It compares performance within one pair, never across pairs, and holds as long as an agent never plays better for an outcome it does not prefer.
Results that need the mapping: MST and SST. Only these compare magnitudes across pairs. We took your concern into consideration and re-assessed the need for the assumption. This allowed us to weaken it, and also led us to find a more powerful test for MST and SST violations.
The weaker assumption (§3.2). We now only need Δ(x,y)>Δ(z,w)⟹P(x≻y)>P(z≻w), where Δ(x,y):=J(π(x,y))−J(π(y,x)) is the change in performance when the two outcomes are swapped. Equivalently, J(π(x,y))=g(P(x≻y))+cxy with g nondecreasing, where the offset cxy=cyx captures any effect of the pair that does not depend on its direction, and cancels in Δ. This assumption is strictly weaker than the original, since the original assumption implies the revised one: if performance is g(p), then the reversal difference is g(p)−g(1−p), which is nondecreasing in p. The converse need not hold, because the revised assumption permits different outcome pairs to have different baseline performance levels. Because g is only nondecreasing, it also allows an agent that does not act on its preferences, for example because the task is too hard; such an agent produces no preference verdicts rather than wrong ones.
The new test (§3.3, Appendix E.3). The new test compares changes in performance under payoff reversal, and is much more powerful than the submitted one. Given a decided premise Δ(x,y)>0 and Δ(y,z)>0:
New results (§5.2, Figure 3; every model in Figure 10).
Bradley–Terry and Thurstone's case V imply SST, and Thurstone's case IV implies MST, so none of them can describe these preferences, stated or revealed.