acceptodds
Under review as a conference paper at ICLR 2027

Can Revealed Preferences Clarify LLM Alignment and Steering?

Abstract

LLMs are increasingly used to make or support high-stakes decisions under uncertainty, where alignment depends not only on factual accuracy but on how models weigh tradeoffs between different outcomes. We present an empirical pipeline for estimating the implied preferences that an LLM's observed choices optimize: we elicit the model's probability distribution over unknowns along with the choice it would make for the decision task and then fit a discrete choice model to recover the cost function that best rationalizes the model's decisions. We show how this revealed-preference description allows rigorous evaluation of whether models behave in a consistently goal-directed way, whether they can verbalize a description of their objectives which matches their revealed decision policy, and whether prompting can reliably steer those policies to implement a user-specified cost function. We apply this evaluation across four medical diagnosis domains and multiple frontier and open-source models. We find that while many models have a nontrivial degree of internal coherence, they also have significant weaknesses in faithfully reporting or adopting preferences in response to user direction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Decision Making, Revealed Preferences, Beliefs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iL15OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical pipeline for estimating the implied preferences optimized by the observed choices of an LLM, and evaluates it across four medical diagnosis domains.

Critical strengths
  1. This paper offers a novel perspective by combining LLMs' probability distributions with actual decisions and inferring their implied preferences through discrete choice analysis.
  2. The evaluation framework is comprehensive, incorporating an assessment of counterfactual predictions.
Critical weaknesses
  1. This paper infers implied preferences from LLMs' observed decision behavior but doesn't adequately verify whether the parameters fitted to this behavior truly represent each model's preferences.
  2. Regarding the dataset, the validity of the ground-truth posterior p∗p^* derived from Bayesian networks has not been sufficiently established. Furthermore, fever and infant crying tasks rely on clinician-specified pediatric Bayesian networks rather than real patient datasets, and the rationale behind this approach has not been validated.
Questions for authors
  1. This paper doesn't provide justification for choosing the multinomial logit model, and lacks comparisons with alternative methods.
  2. A question mark appears in the Related Work section.
  3. The ground-truth posterior p∗p^* is derived from Bayesian networks rather than validated clinical truths. Is this approach reasonable?
  4. Are the Bayesian networks learned from the two public datasets plausible and valid?
  5. The patient characteristics for the fever and infant crying tasks were constructed using clinician-specified pediatric Bayesian networks. What is the rationale for this approach?
  6. Only two LLMs across four configurations were evaluated, limiting generalizability.
  7. The validity of the proposed framework remains to be verified. The introduction lacks references to support the underlying idea.
  8. Why focus on the costs associated with FP, FN and deferral rather than other costs? Why do you think the LLMs actually base their decisions on these specific costs?
  9. Experiments were conducted on tasks in only medical diagnosis domains. Can the conclusion be generalized to other domains?
1 comment

In my initial review, I considered the evidential gaps potentially addressable through additional validation. The responses and revised results reveal more fundamental problems. Several claims do not match the evidence, the added analyses do not adequately test the central claims, and key predictions fail. These issues affect the paper’s core assumptions, evaluation, and main contribution. I am therefore lowering my score.

The authors initially downplayed reference-posterior accuracy because cost estimation does not depend on p⋆p^\star and predicted and realized gains share a common yardstick. Yet replacing the scoring probabilities with matching patients’ empirical outcome frequencies in Table 20 reduces the probability-intervention correlations from 0.65/0.91 to 0.03/0.33. This collapse contradicts the earlier suggestion that a common yardstick makes reference-posterior accuracy unimportant for evaluating intervention benefits. The authors themselves attribute the deterioration to inaccuracies in the Heart posterior. Independence of cost estimation from p⋆p^\star does not imply independence of intervention evaluation, and common scoring does not guarantee validity. The new Heart experiment changes the supplied probabilities and generates new responses; it does not establish predictive validity when those probabilities are inaccurate.

Even under the original evaluation, Figure 3(c) shows substantial systematic error, with a correlation of 0.65, a slope of 0.49, an intercept of 36.8 percentage points, and an MAE of 18.6 percentage points. Positive correlation is insufficient evidence of reliable benefit prediction. The higher 0.91 uses costs refitted to post-intervention decisions, supporting retrospective explanation rather than resolving the target counterfactual’s prospective prediction errors.

E→C does not establish Baseline-cost stability under probability prompting. Merely displaying the model’s previously elicited probability reduces deferral from 24% to 5% and shifts fitted costs. Estimating new costs under E and transferring them to C tests transfer between explicit-probability conditions, not the validity of the original Baseline decomposition. High correlation between E and C cost ratios does not establish unchanged costs.

The newly added non-medical credit-default experiment (Appendix E) reveals a direct failure of the target probability-intervention prediction. With Baseline costs fixed, replacing probabilities leaves the simulated policy deferring on almost every case, missing the gains from the actual model’s reduced deferral. This is not a successful cross-domain replication of that predictive capability. Post-intervention cost fitting cannot substitute for the failed validation.

More fundamentally, fitted-cost changes do not identify preference changes. Decision-time beliefs, task interpretation, and response strategies may change and be absorbed into the estimates. Without separating these sources, attributing losses to belief errors versus preference misalignment is unjustified. The response also incorrectly treats the loss structure as implied by the action space. Yes, No, and Defer do not require equal deferral costs when the condition is present and absent. The main text acknowledges this additional assumption, while the response presents it as necessary. Nested-logit checks do not test this restriction.

These problems undermine the evidential chain supporting the main contribution, from stable preference estimation to separating belief errors from preference misalignment and reliably predicting intervention benefits. The experiments do not establish that chain, and key predictions fail. If the claim is reduced to fitting behavior under particular prompts, the independent contribution of that narrower result must be established; it cannot sustain the original explanatory and predictive claims. The responses do not adequately address these consequences and have deepened my doubts about the framework’s validity and the paper’s overall conclusions.

Reviewer_Zfa3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether we can understand LLM’s choices based on their revealed preferences, instead of relying and trusting what they state. The authors use a few tasks from the medical domain to first get the model’s beliefs, and then their revealed preferences. Then, they analyze the consistency and steerability. Results show LLMs have significant internal coherence, but struggle to faithfully report or be steered.

Critical strengths

This paper is very interesting, useful, and clearly written. Unfaithful beliefs, reasoning, or explanations are critical for proper alignment.

The critical strength, in my opinion, is the interpretability of the method. It converts revealed LLM preferences into cost ratios that can be easily inspected and understood. This gives us a tool to study the consistency of a model’s beliefs compared to its decisions.

For robustness, the authors also test alternative probability prompts and noise perturbation.

Critical weaknesses

Revealed preferences are not necessarily novel in studying LLM behavior. What’s the novelty of your paper compared to: Cook, T. R., Kazinnik, S., Modig, Z., & Palmer, N. M. (2026). What do LLMs want?. I see the setup is not exactly the same, but many of the core ideas are, so I wonder about the main contributions of your work.

It’s also hard to know if these findings will generalize outside of the medical domain evaluated here, but the method is interesting and potentially useful.

A reference is missing in L86

Questions for authors

I’m mostly concerned about novelty and will increase the score if it is addressed.

Reviewer_zowuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper describes an LLM's decisions under uncertainty with a revealed-preference model. A numeric belief pE(x)=PE(θ=1∣x)p_E(x)=P_E(\theta=1\mid x) is elicited in one context window and a decision a∈{Yes,No,defer}a\in\{\text{Yes},\text{No},\text{defer}\} in another; a multinomial logit with loss ℓc(a,θ)=cFP 1[a=1,θ=0]+cFN 1[a=0,θ=1]+cdefer 1[a=defer]\ell_c(a,\theta)=c_{FP}\,\mathbb 1[a=1,\theta=0]+c_{FN}\,\mathbb 1[a=0,\theta=1]+c_{\rm defer}\,\mathbb 1[a=\text{defer}] and noise scale β=1\beta=1 is fit by maximum likelihood, and the ratios cFN/cFPc_{FN}/c_{FP} and cdefer/cFPc_{\rm defer}/c_{FP} are reported. The implied cost vector is used for (i) a consistency score ILFC (share of decisions matching the implied-optimal action), also restricted to borderline beliefs in [0.35,0.45][0.35,0.45]; (ii) comparison with the model's self-reported cost ratios (global and case-specific); (iii) steerability, by refitting after prompting with six benchmark cost tuples; and (iv) counterfactual prediction of the realised percent loss reduction from cost-function and probability prompting, using either the targeted component alone ("target counterfactual") or the post-prompt refitted cost vector ("steered counterfactual"). Experiments cover four binary-state diagnosis tasks (structural heart disease and diabetes from public data with fitted Bayesian networks; two clinician-specified paediatric networks) and four configurations (GPT-5 at high and minimal reasoning, Qwen3.8-27B with x-high reasoning and with reasoning disabled). Findings: baseline ILFC of 67–85% (50–74% on borderline cases); self-reported FN/FP and Defer/FP ratios exceed the implied ones (2–3.5 vs 1.5–2.5, and 1.1–1.5 vs about 0.4); deferral is treated as cheap; steering moves ratios in the right direction in 65–83% of settings but rarely lands on target; target counterfactuals correlate with realised gains (r=.81r=.81 for cost prompting, .65.65 for probability prompting) and steered counterfactuals more strongly (r=.87r=.87, .91.91).

Critical strengths
  1. Well-posed question and clean framing. Separating beliefs from preferences is the right lens for "alignment of trade-offs", and the statistical-decision formulation in Section 3.1 is stated precisely, including the caveat that rationality is tested rather than assumed.
  2. Sensible robustness checks on the belief object. Loss-conditioned elicitation, Gaussian perturbation, averaging over five elicitations and the Yamin et al. validity criteria (Appendix A), plus a held-out split for the cost vector (Appendix C), are all appropriate and clearly reported.
  3. Informative steering analysis. Figure 2 and Table 1 give a useful picture for practitioners: reasoning configurations move in the right direction but overshoot on deferral, and non-reasoning configurations barely move.
  4. Reproducibility. Every prompt is given verbatim, the datasets and network construction are described, and the discussion paragraph is honest about the fixed action space and the reliance on elicited probabilities.
Critical weaknesses
  1. With a binary state and three actions the revealed-preference model has almost no refutable content beyond monotonicity in pp, so "coherent goal pursuit" is tested by a test that nearly any monotone choice rule passes. Expected losses are linear in pp: cFP(1−p)c_{FP}(1-p) for Yes, cFN pc_{FN}\,p for No, cdeferc_{\rm defer} for defer. The lower envelope gives No on [0, cdefer/cFN)[0,\,c_{\rm defer}/c_{FN}), defer on [cdefer/cFN, 1−cdefer/cFP][c_{\rm defer}/c_{FN},\,1-c_{\rm defer}/c_{FP}] and Yes above (or no deferral region at all). The cost vector is therefore exactly two thresholds on the belief, and any choice rule that is single-crossing in pp (No, then defer, then Yes) is perfectly rationalisable by some cc. ILFC thus measures departures from the best-fitting two-threshold rule, i.e. monotonicity violations with respect to pEp_E, which is what Yamin et al. (2026) already test on three of the four datasets. The random-utility machinery (Gumbel errors, MLE) adds no additional restriction in this instantiation, and the "chance level of 33%" quoted for Table 3 is not the relevant null; the relevant null is a noisy monotone rule. The paper should say this plainly and state what the economic framing contributes beyond locating two thresholds. The held-out validation in Appendix C is likewise unsurprising for a two-parameter threshold rule.

  2. The cost ratios are identified only jointly with the belief object, and the chosen sensitivity analysis does not probe the relevant misspecification. Because decisions depend on pp only through threshold crossings, any monotone distortion gg of the decision-time belief is observationally equivalent to a change in the thresholds, i.e. in cc. The headline interpretive finding, "every model reports ratios that are exaggerated relative to those implied by its own decisions", is therefore equally consistent with the numeric report pEp_E being a compressed or stretched version of the belief the model acts on, which is well documented for LLM probability outputs. Appendix A.3 adds iid Gaussian noise of SD at most 0.05, which is not this failure mode. Two cheap diagnostics would help: (i) refit the Baseline decisions against p⋆p^\star instead of pEp_E (the paper has p⋆p^\star for every case) and report how far the implied ratios move; (ii) fit a one-parameter calibration map gγ(pE)g_\gamma(p_E) jointly with cc and report the profile likelihood, which shows directly how much of the self-report gap can be absorbed by belief miscalibration.

  3. Complete separation is not addressed. Table 2 reports 100% ILFC for GPT5-Minimal under elicited-pp prompting and 99.8–100% in several other cells, i.e. the decisions are a deterministic function of the belief. In a logit with β\beta fixed the MLE then does not exist: the likelihood increases without bound as the scale of cc grows, and the ratio is only set-identified, since any thresholds between the last "No" belief and the first "Yes" belief fit equally well. L-BFGS-B will stop at whatever bound or tolerance it is given, and the bootstrap interval will reflect those settings rather than sampling uncertainty. The paper does not mention separation, bounds or regularisation. Please state how these cells were handled and report the identified interval rather than a point estimate where the fit is deterministic.

  4. Independence of irrelevant alternatives is implausible for {Yes,No,defer}\{\text{Yes},\text{No},\text{defer}\}. The multinomial logit imposes that the odds of Yes versus No are unaffected by the deferral option. But "defer" is not a third diagnosis; it is an opt-out that competes with whichever of Yes/No is marginal. A nested logit (defer versus the Yes/No nest), or a two-stage model (decide versus defer, then Yes versus No), is the standard remedy and maps directly onto the two-line structure of the prompt. The implied Defer/FP ratios, on which the paper makes its strongest qualitative claims ("deferral is cheap", self-reports overstate deferral cost threefold), are the quantities most exposed to this misspecification.

  5. The "deferral" action is not elicited as an action with consequences, which confounds the deferral-cost findings. In the Baseline prompt (Appendix D.2.1), "defer" is the answer "No" to "Can you make a decision about whether the patient is likely to …?", with no cost attached and with a forced Yes/No decision requested on the next line. A model answering "Can decide: No" is reporting low confidence, not choosing an action whose cost is cdeferc_{\rm defer}. The self-report prompt (D.3.1), by contrast, describes deferral as "deferring to further testing" with an explicit cost. That the implied Defer/FP ratio is about 0.4 "for every model" while the self-reported one is 1.1–1.5 may simply reflect that the two prompts ask different questions. The forced decision on Line 2 appears to be discarded; using it would give a cleaner binary-threshold estimate, and the "Can decide" line could be analysed separately as a confidence report.

  6. The steered-counterfactual correlations (r=.87r=.87 and .91.91) are in-sample fit statistics, not predictions, and should not be headlined as "out-of-sample predictions of how they respond to prompting". c^steered(k)\hat c^{(k)}_{\rm steered} is fit by maximum likelihood to the very post-prompt decisions acost(k)a^{(k)}_{\rm cost} whose realised loss is on the yy-axis; the implied rule reproduces those decisions in 83–95% of cases (Table 2), so Lk(a(pE,c^steered))≈Lk(acost(k))L_k(a(p_E,\hat c_{\rm steered}))\approx L_k(a^{(k)}_{\rm cost}) essentially by construction, and the same holds for c^p⋆\hat c_{p^\star} in Figure 3d. The only genuinely predictive quantities are the target counterfactuals (r=.81r=.81, .65.65), and the probability-prompting one is moderate. The conclusion that "imperfect adoption … rather than unrationalisable behaviour explains most of the gap" then amounts to the statement that a refit two-threshold rule fits well, which Weakness 1 already explains.

  7. The correlation is computed over 96 points that share 16 baselines and a common denominator, so both rr and the reported pp-values (10−2410^{-24}, 10−3810^{-38}) are overstated, and the pattern may be driven by between-scenario variation. The six benchmark tuples for a given model–domain pair share abasea_{\rm base}, pEp_E and c^\hat c; scenarios with a large cFNc_{FN} inflate both the predicted and the realised percent reduction through the shared Lk(abase)L_k(a_{\rm base}). The needed analyses are (a) a case-level cluster bootstrap for rr; (b) within-scenario and within-(model, domain) correlations, not only within-model; (c) a variance decomposition showing how much of the realised Δk\Delta_k is explained by scenario and domain fixed effects alone versus by the counterfactual predictor; and (d) calibration (slope and intercept of realised on predicted, and mean absolute error), since Figure 3a visibly has slope below one and rr says nothing about whether the predictions are quantitatively right.

  8. Steerability is probed on a narrow range, and the direction classification depends on where the baseline sits. Prompted FN/FP ratios span only 0.5–2 and Defer/FP 0.25–2 (Figure 2 axes), whereas clinically meaningful FN/FP ratios are often 5–20. For the no-reasoning Qwen configuration the baseline FN/FP ratios (about 2–4, Figure 1) sit at or above the top of the prompted range, so almost every prompt lies below baseline and "movement toward the target" is indistinguishable from regression toward one. Table 1's categories (Wrong/Under/Target/Over) are not conditioned on the baseline-to-target distance; when that distance is small, "Wrong" is essentially a coin flip. A regression of log⁡(implied/target)\log(\text{implied}/\text{target}) on log⁡(baseline/target)\log(\text{baseline}/\text{target}), or a slope of implied on prompted ratio with a CI per configuration, would be more informative than category shares.

  9. Basic design parameters are missing from the main text. The number of cases per domain (only diabetes, 100, is stated), the number of repeated elicitations per case ("all repetitions of a case" appears only in Appendix C), the decoding temperature, and the unit the bootstrap resamples are not given. These determine whether the Gumbel error is modelling LLM sampling noise or deterministic misspecification: at temperature zero the logit noise is pure misspecification, and comparing the sharpness of configurations is then a comparison of misspecification rather than of preference noise. For the two paediatric networks there are no real patients, and it is not stated whether the θi\theta_i in LkL_k are sampled from the network or losses are taken in expectation under p⋆p^\star.

  10. Two model families, one action structure, one loss family. The framework is exercised only on binary-state, three-action problems with a scalar belief, where (Weakness 1) it reduces to two thresholds. Nothing in Sections 3–5 shows the pipeline working where it would have bite, e.g. three or more states or actions with a non-trivial loss matrix, where IIA, identification and the counterfactual predictions could actually fail. "Frontier and open-source models" is one of each.

  11. A "(?)" citation on page 2; "Plunkett et al. (2025) We prompt…" on page 4 is a broken sentence; the Figure 1 caption refers to "(i)-(iii) and (v)" while the text defines regimes (i)–(iv); Tables 2 and 3 carry no intervals although every other metric has a bootstrap CI; the scenario numbers in Figure 3 are illegible at print size.

Questions for authors
  1. How are separated cells (100% ILFC) handled in the MLE? What bounds does L-BFGS-B use, and do the reported ratios and intervals for those cells depend on them?
  2. What are the implied Baseline ratios when p⋆p^\star is used as the belief object instead of pEp_E? If they differ materially, which of the "self-report gap" and "deferral is cheap" conclusions survive?
  3. Does a nested logit (defer versus the Yes/No nest) change the Defer/FP ratios or the steering conclusions?
  4. Why is the forced Yes/No decision on Line 2 of the Baseline prompt not used? What FN/FP ratios result from fitting Line 2 alone?
  5. For Figure 3, please give slope, intercept and MAE of realised on predicted; within-scenario correlations; a case-level cluster-bootstrap CI for rr; and the R2R^2 of a model with only scenario and domain fixed effects.
  6. Number of cases per domain, repetitions per case, decoding temperature, and bootstrap unit?
  7. For the two paediatric networks, how are the "ground-truth outcomes" θi\theta_i defined?
  8. Can the authors show one example, even synthetic, with three or more states where the revealed-preference model yields a prediction that a monotone-choice description would not?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.