- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper defines Undisclosed Preference Dependence (UPD): a model's advice depends on a user priority the user never stated, the model commits to one side, and it does not expose that dependence (via a question, conditional branching, or stated assumptions). The audit has two streams: a supplied-priority stream establishes that opposed priorities flip the model's recommendation for a given scenario (S-positive: 188/240 cells), and a withheld-priority stream has two human annotators label whether the ordinary answer is an undisclosed commitment (kappa = .86). UPD requires both. Across four frontier models on 60 constructed decision families, the UPD rate is 28.4 percent (family-clustered 95 percent CI 19.9 to 37.7), with per-model rates from 23.6 to 33.8 percent; specificity controls are clean (targeted movement 1.84 versus off-axis 0.23), six families are C-positive across all four models (p = .00013), and a screened WildChat extension shows the phenomenon transfers to natural requests. A pre-registered follow-up prediction failed and is reported as such.
Critical strengths
The audit design and reporting standards are well above the norm for this literature: a frozen scenario panel built before any target outputs, dual independent human annotation with high agreement, an off-axis specificity control that rules out generic prompt sensitivity, judge-dependence checks via Gemini relabeling (kappa = .942), and honest reporting of a failed prediction. This matters because for a measurement paper, the credibility of the measurement is the contribution, and here it is largely credible.
Critical weaknesses
-
The contribution stops at diagnosis, and the significance of the diagnosis is not established. The authors state plainly that UPD is a process-level failure and that they do not measure latent priorities, advice uptake, regret, or whether disclosure improves decisions; no mitigation of any kind is tested, not even a one-line system prompt instructing the model to surface assumptions. The paper therefore cannot say whether the 28.4 percent rate corresponds to any user harm, nor whether it is trivially fixable. For ICLR, a new metric needs either demonstrated consequence or demonstrated actionability; this paper defers both to future work, which caps the contribution at a well-executed definition.
-
The cross-model results, the part practitioners would actually use, are the weakest part. The Anthropic model is S-positive on only 28 of 60 families (versus 52 to 55 for the others) and produced 85 refusals, so its UPD rate sits on a much smaller and differently composed denominator; the paper itself notes the rows describe different eligible populations. No between-model significance tests are reported, the pooled CI spans nearly 18 points, and everything is single-turn, which plausibly overstates severity relative to real multi-turn use where priorities surface naturally.
Questions for authors
-
Have you run even a minimal mitigation probe, e.g. a system prompt asking the model to state assumptions or ask one clarifying question before committing? If a one-line instruction collapses the UPD rate, the metric's framing changes substantially; if it does not, the paper becomes considerably stronger.
-
Given the different eligible populations per model, what comparison between the rows of Table 1 do you intend readers to make, and can any such comparison be made statistically?
-
Do you have evidence on how UPD behaves beyond the first turn, even on a small subset?