acceptodds
Under review as a conference paper at ICLR 2027

When LLM Advice Depends on Values Users Never Stated: Measuring Undisclosed Preference Dependence

Abstract

Large language model (LLM) assistants increasingly give practical advice on decisions whose resolution can depend on how the user prioritizes competing values. Existing evaluations study how models acquire and use preferences; we study what happens before a recommendation-relevant priority has been supplied. We introduce undisclosed preference dependence (UPD): an offline behavioral audit that first tests whether opposing legitimate priorities reverse a model's recommendations consistently across paraphrases and repeated draws, then evaluates separate priority-withheld answers for one-sided commitment without disclosure of that dependency. Across 60 controlled decision families and four model families, the sensitivity criterion held in 188/240 family–model cells (188/216 among estimable cells). Within sensitive cells, human annotation identified UPD in 266/937 eligible substantive withheld answers (28.4%; family-clustered 95% CI 19.9–37.7%). Targeted priority changes produced substantially more movement than the off-axis additions we tested, and the pattern transferred to screened natural-source requests. The assay operationally distinguishes using a supplied priority from handling advice when a recommendation-relevant priority remains unresolved, and provides a scalable, rerunnable offline audit: supplied or constructed scenario–priority contrasts can be reapplied across models and checkpoints, with withheld-answer endpoint judges validated against the human reference, to track what their advice depends on and whether those dependencies are surfaced.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, AI advice, user preferences, evaluation, human-centered AI, alignment, user agency, counterfactual evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4vE8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper defines Undisclosed Preference Dependence (UPD): a model's advice depends on a user priority the user never stated, the model commits to one side, and it does not expose that dependence (via a question, conditional branching, or stated assumptions). The audit has two streams: a supplied-priority stream establishes that opposed priorities flip the model's recommendation for a given scenario (S-positive: 188/240 cells), and a withheld-priority stream has two human annotators label whether the ordinary answer is an undisclosed commitment (kappa = .86). UPD requires both. Across four frontier models on 60 constructed decision families, the UPD rate is 28.4 percent (family-clustered 95 percent CI 19.9 to 37.7), with per-model rates from 23.6 to 33.8 percent; specificity controls are clean (targeted movement 1.84 versus off-axis 0.23), six families are C-positive across all four models (p = .00013), and a screened WildChat extension shows the phenomenon transfers to natural requests. A pre-registered follow-up prediction failed and is reported as such.

Critical strengths

The audit design and reporting standards are well above the norm for this literature: a frozen scenario panel built before any target outputs, dual independent human annotation with high agreement, an off-axis specificity control that rules out generic prompt sensitivity, judge-dependence checks via Gemini relabeling (kappa = .942), and honest reporting of a failed prediction. This matters because for a measurement paper, the credibility of the measurement is the contribution, and here it is largely credible.

Critical weaknesses
  1. The contribution stops at diagnosis, and the significance of the diagnosis is not established. The authors state plainly that UPD is a process-level failure and that they do not measure latent priorities, advice uptake, regret, or whether disclosure improves decisions; no mitigation of any kind is tested, not even a one-line system prompt instructing the model to surface assumptions. The paper therefore cannot say whether the 28.4 percent rate corresponds to any user harm, nor whether it is trivially fixable. For ICLR, a new metric needs either demonstrated consequence or demonstrated actionability; this paper defers both to future work, which caps the contribution at a well-executed definition.

  2. The cross-model results, the part practitioners would actually use, are the weakest part. The Anthropic model is S-positive on only 28 of 60 families (versus 52 to 55 for the others) and produced 85 refusals, so its UPD rate sits on a much smaller and differently composed denominator; the paper itself notes the rows describe different eligible populations. No between-model significance tests are reported, the pooled CI spans nearly 18 points, and everything is single-turn, which plausibly overstates severity relative to real multi-turn use where priorities surface naturally.

Questions for authors
  1. Have you run even a minimal mitigation probe, e.g. a system prompt asking the model to state assumptions or ask one clarifying question before committing? If a one-line instruction collapses the UPD rate, the metric's framing changes substantially; if it does not, the paper becomes considerably stronger.

  2. Given the different eligible populations per model, what comparison between the rows of Table 1 do you intend readers to make, and can any such comparison be made statistically?

  3. Do you have evidence on how UPD behaves beyond the first turn, even on a small subset?

Reviewer_erHkOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at advice requests where the right recommendation hinges on how a user weighs competing legitimate values, but the user has left that preference unstated. Its central question is whether a priority-withheld answer quietly commits to one side even when the model would reverse its recommendation under opposing priorities. The proposed undisclosed-preference-dependence audit uses priority-specified prompts to establish model-relative recommendation sensitivity, then takes a separate priority-withheld response—annotated by blinded humans—to judge whether the model commits without disclosing the dependency. Across a broad set of decision families and model families, the audit finds substantial undisclosed preference dependence, shows that responses move much more under the targeted priority than under control perturbations, and reproduces the qualitative pattern in a natural-conversation extension.

Critical strengths
  1. The paper frames a timely and underexplored problem in LLM advice: how to handle material user preferences that have not been stated.
  2. It presents a careful audit design that separates preference sensitivity from whether a withheld answer transparently exposes that dependence.
Critical weaknesses
  1. The paper has not yet established whether the measured phenomenon translates into worse user decisions or other concrete harms.
  2. Because key rates are conditioned on model-specific sensitivity, the results are difficult to compare directly across models.
  3. The annotation and naturalistic-transfer evidence remains limited relative to the breadth of the paper's normative framing.
  4. The paper appropriately says UPD is a process-level failure rather than a direct measure of user harm in Section 6.3. Its future-turn probe does not support the preregistered downstream prediction in Appendix D, and the later-turn WildChat check has only two usable reveals out of 60. The normative importance of UPD would be stronger with evidence that it affects users' decisions or awareness of relevant tradeoffs.
Questions for authors
  1. Does UPD correlate with a measurable difference in user decision quality or users' recognition of their own priorities? If not, what evidence best establishes it as a failure rather than a neutral advice style?
Reviewer_fR9DOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors raise the question on how LLMs trade of legitimate preferences on task where no user preference (that would settle the case) has been presented yet. They introduce different decision scenarios they call “families” and test via two stream, one where the priorities are exposed and one where they are withheld, how robust and transparent the LLMs make the trade-off between the preferences.

Critical strengths

The question the authors raise, how LLMs make trade-off between conflicting targets in conditions where no preference by the user is stated, is an interesting one to study. The proposed experimental setup seems to be well suited to answer this question, though making the code publicly available would strengthen this point even more.

Critical weaknesses

The authors correctly point to a closely related field of research under the term “sycophancy”, line 374, but the difference to their idea of UPD (line 376 f.) is insufficient discussed. Papers in that field, e.g. [1] and [2], follow a very similar approach where first a preference is not stated and then they investigate how the LLM changes its opinion once the preference is stated. The authors experimental setup reads more like a restructuring of that very idea by splitting the two step approach into two separate stream, one where the priority is provided and one where it is withheld.

I do not see the need for introducing the individual conditions in line 134 f. that eventually lead to the term “S-positive”. The very fundamental idea as it is being presented in the paper could be described by the flip ratio, following the existing body of literature [1, 2]. The idea of investigating different levels of commitment would then be an extension towards a flip matrix. Beyond the existing literature on sycophancy there are additional papers would need to be considered, such as the paper “AGM-Bench: Do Large Language Models Revise Beliefs Rationally?” [3] which was presented at the ICLR Workshop on Logical Reasoning of Large Language Models.

The presented results in 3.1 and 3.2 fail to discuss the results to an extend that provides new and actionable insights. The presented figures 1 & 2 add little in providing a deeper understanding. Even if we take S positive/negative to be the right metrics to report, the analysis on how large the subsets of families (rows in figure 2) are across which the four models agree do not provide any insights. While the fundamental question is interesting to raise, the authors would need to dig much deeper into those results in order to derive hypothesis about how the families differ and why certain models tend to produce S positive results while others produce S negative.

[1] https://arxiv.org/pdf/2604.02423 [2] https://arxiv.org/pdf/2505.23840 [3] https://openreview.net/pdf?id=2s1BujG84C

Questions for authors

No questions that would materially affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.