acceptodds
Under review as a conference paper at ICLR 2027

Evaluating LLM-Generated Preference Distributions

Abstract

Large Language Models (LLMs) are increasingly used as probabilistic generators for simulation, synthetic data generation, and decision support in settings where real-world data are unavailable. Yet, the structure and reliability of the distributions they produce remain understudied. Here, we systematically analyze LLM-generated distributions of preferences for air travel, restaurants, and consumer products. Encouragingly, repeated calls to the same model yield comparatively self-coherent distributions, with the most probable outcomes often stabilizing within the recorded calls. At the same time, we observe substantial discordance across differently sized models within each of three families, with little consensus even among their most probable outcomes. We examine these patterns across nine open-weight models and three choice domains, with additional checks under temperature changes, greedy decoding, and perturbations of prompt and ordering. Our findings indicate that changing the model can shift outcomes as much as rewording the prompt, challenging the common assumption that sufficiently capable LLMs produce similar preference distributions when used as stand-ins for survey respondents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM-generated preference distributions, verbalized probability elicitation, repeated sampling, distributional evaluation, prompt sensitivity, persona conditioning, synthetic data, LLM-based simulation, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_MHyEOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the consistency of preference probability distributions generated by LLMs. The authors use nine open-weight models from three families (Llama, Qwen, and Mistral) and examine choices related to air travel, restaurants, and consumer products. Instead of asking LLMs to make one choice, they ask the models to generate explicit probability distributions over a list of alternatives. The paper then studies how these distributions change across repeated calls, different models, and prompt variations.

The main finding is that the same model tends to produce relatively consistent distributions across repeated calls, especially for its top choices. However, different models produce quite different distributions, even when belonging to the same model family. For example, the top-five choices of Llama 70B and 405B have little overlap, and transferring a top-five shortlist between these two models reduces its probability coverage by about 17.7 percentage points, evaluated against the target model's own outputs. The paper also shows that prompt wording and the processing of generated responses can substantially affect the resulting distributions.

Critical strengths

I like that the paper goes beyond simply showing LLM outputs are sensitive to prompts. It asks whether changing models changes an entire probability distribution, and how much this matters for reusing model-generated choices. The 17.7 percentage point difference in shortlist coverage is a useful way to quantify model disagreement, although the coverage is evaluated against the LLM's own distribution rather than actual human choices. This is relevant for applications where simulated distributions are used repeatedly.

In addition, I appreciate the effort in auditing the outputs and distinguishing between matched and unmatched probabilities. For example, the paper shows that only around 69.8% of Yelp and 65.1% of Amazon probability mass is retained under the respective matching rules. The authors also recognize that some stabilization patterns arise mechanically from averaging repeated calls, and they use a stationary random baseline to check this. These are useful methodological details that are often overlooked in LLM simulation studies.

Critical weaknesses

My biggest concerns is that I am not convinced that the paper is measuring preference distributions in an economically meaningful sense.

The paper asks LLMs to provide probability distributions over 300 alternatives, but what exactly is this probability supposed to represent? For example, the travel prompt asks about travel plans over the next 12 months, and the restaurant and product prompts ask about visiting or purchasing items in the next month. These are not necessarily mutually exclusive events. A consumer may visit multiple restaurants, buy multiple products, or travel to multiple destinations. However, the prompts force the probabilities across all alternatives to sum to one. This is not the same as either marginal purchase probabilities or a standard discrete-choice probability distribution.

There is also very little information about the actual choice environment. For travel, there are no ticket prices, budgets, trip purposes, or other constraints. For restaurants, distance, price and quality information are largely absent from the prompt. The generated distributions may therefore reflect what destinations or product names are more salient in the model's training data, rather than actual consumer preferences.

More importantly, repeated calls with the same demographic persona do not generate a sample of different consumers. They are repeated numerical responses from the same model under the same instruction. Averaging these responses may reduce generation noise, but it does not necessarily recover preference heterogeneity in a population.

I understand the authors explicitly state that they do not measure human-choice accuracy. But I think the issue here is more fundamental. Even before comparing with real human data, it is unclear what behavioral object the generated probability vector represents.

Without a clearer definition or validation, I find it difficult to interpret the results as evidence about the reliability of LLM-generated preferences, rather than the reproducibility of numbers written by LLMs.

Questions for authors
  1. Clarification regarding the meaning of preference probability.

What is the precise interpretation of the probability assigned to each alternative? Is it the probability of selecting one alternative from the entire choice set, or the probability of visiting/purchasing that alternative within a period? If it is the latter, why should probabilities sum to one when multiple events can occur? This clarification is important for interpreting the main results.

  1. Evidence based on a more meaningful economic choice task.

Would the major cross-model findings remain if the prompt defined one mutually exclusive decision, such as the destination of the next single trip, or the restaurant for one particular dining occasion, with an outside option? I think this would help establish whether the results reflect differences in preference predictions or simply differences in how models generate numerical lists. Even a relatively small controlled experiment would be informative.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MHyE (1/2)

Thank you for the thoughtful review and for valuing the shortlist-coverage measure and the audit of matched versus unmatched probability. This part answers Question 1 and the concerns about the choice environment and repeated calls; Part 2 answers the validation concern and Question 2.

Question 1: What is the precise interpretation of the probability assigned to each alternative? Is it the probability of selecting one alternative from the entire choice set, or the probability of visiting/purchasing that alternative within a period?

It is the former, as the sum-to-one instruction implies: the probability of selecting each alternative on one choice occasion (a trip, a dining visit, a purchase) within the stated window; over several occasions, it is the expected share of occasions. This is the standard discrete-choice probability conditional on a choice occasion [1, 2], and it mirrors travel demand modeling, where trip frequency (trip generation) is modeled separately from where each trip goes (destination choice) [3]. The window sets the context without assuming exclusivity, and for the restaurant and product prompts the sum-to-one instruction resolves the wording toward choice shares. Section 3.2 will state this definition.

Weaknesses, paragraph 3: There is also very little information about the actual choice environment.

The context is minimal and, within each compared pair, identical (the same persona text, origin, and choice set; the adapted Mistral-24B prompt adds one sentence, Appendix I.5), so missing attributes cannot by themselves explain why the two models of a pair disagree. If outputs partly reflect salience from training data, the finding that differently sized models of one family emphasize different alternatives is what a practitioner needs to know before reusing such a distribution. Section 6 will state that the elicited distributions are conditional on this minimal context, which includes no prices, budgets, or trip purposes.

Weaknesses, paragraph 4: More importantly, repeated calls with the same demographic persona do not generate a sample of different consumers.

Repeated calls are not used to represent different consumers. They measure how stable one model's stated distribution is for one fixed profile (Section 3.3), a precondition for using that distribution in a simulator. Heterogeneity across profiles is examined separately with a persona grid of five age groups by four education groups (Appendix C.1). In the specified contrasts (Appendix K), changing the age within an education group, or the education within an age group, moves the 70B distribution by mean JS 0.330 and 0.258 (Table 18), and no pair of Llama models gives an identical top five for any of these 20 personas (Table 3).

References

[1] McFadden, D. (1974). Conditional logit analysis of qualitative choice behavior. In P. Zarembka (Ed.), Frontiers in econometrics (pp. 105–142). Academic Press.

[2] Train, K. E. (2009). Discrete choice methods with simulation (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511805271

[3] Ortúzar, J. de D., & Willumsen, L. G. (2011). Modelling transport (4th ed.). Wiley. https://doi.org/10.1002/9781119993308

AuthorsOpenReview ↗

Response to Reviewer MHyE (2/2)

Weaknesses, paragraph 6: Without a clearer definition or validation, I find it difficult to interpret the results as evidence about the reliability of LLM-generated preferences, rather than the reproducibility of numbers written by LLMs.

In the simulation uses we target, the numbers a model writes are what the simulator consumes, so their reproducibility within a model and their agreement across models are the reliability properties at stake. They are also necessary conditions for any behavioral reading: two models that disagree on their most probable outcomes under an identical prompt cannot both match the same behavioral target. For the same reason, shortlist coverage is scored on the target model's own held-out calls: it measures what is lost when one model's shortlist stands in for another's, not human-choice accuracy (Section 5.3). Validation against human choices is the next step (Section 7).

Question 2: Would the major cross-model findings remain if the prompt defined one mutually exclusive decision, such as the destination of the next single trip, or the restaurant for one particular dining occasion, with an outside option?

Question 2: I think this would help establish whether the results reflect differences in preference predictions or simply differences in how models generate numerical lists.

Thank you for this concrete suggestion. We will run this controlled experiment for the Llama 70B/405B and Qwen 72B/235B pairs (20 calls per model): the prompt asks for the destination of the traveler's next single trip, with "no trip" as an outside option, keeping the persona, choice list, and decoding settings of the matched pairs, and the analysis follows Tables 15 and 16. The Llama-3.1-405B endpoint used in the paper has since been retired, and no serverless inference provider we checked offers this checkpoint, so we will deploy it on our own GPU server, which takes longer than the discussion period; we will include those experiment results in the camera-ready version. For Llama-3.3-70B, the most probable destination in the 20 pooled verbalized IND calls, ORD (Figure 9), is the airport returned in all 300 single-choice draws at temperature 0.2 (Appendix A). We will also compare models on token-level single-choice probabilities, which involve no numerical list, and we will include those experimental results in the camera-ready version.

Reviewer_TVwVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper systematically analyzes LLM-generated preference distributions across three choice domains (air travel, restaurants, and products) and nine open-weight models. The main findings are that repeated calls to the same model yield comparatively self-coherent distributions, while different models disagree even on their most probable outcomes, and changing the model can shift outcomes as much as rewording the prompt.

Critical strengths

As LLMs are increasingly used for simulation and survey data, it is crucial for practitioners to understand the reliability of the elicited data and the corresponding distributions. This paper studies a practically relevant question.

Critical weaknesses
  1. The paper relies on verbalized probabilities, which may not reflect the model’s token-level probabilities. This limits what the results say about the underlying distributions. It would strengthen the paper to compare verbalized distributions with alternatives such as token-level probabilities or empirical distributions from repeated single-choice sampling.
  2. The task format is narrow. The experiments ask the model to output a JSON list of probabilities over a fixed set of options, which differs from the preference data typically used in LLM-based simulation, such as pairwise comparisons, or open-ended and longer-form responses. Also, many key comparisons rely on very limited model calls (15–20 per setting). It would strengthen the paper to show whether the observed patterns hold for these more common forms of preference elicitation and under larger-scale sampling.
Questions for authors

Have the authors compared verbalized distributions with token-level probabilities or repeated single-choice sampling at larger scale?

2 comments
AuthorsOpenReview ↗

Response to Reviewer TVwV

Thank you for highlighting that practitioners need to know how reliable LLM-elicited distributions are.

Question: Have the authors compared verbalized distributions with token-level probabilities or repeated single-choice sampling at larger scale?

We study verbalized distributions by design, because the stated vector is what a simulator receives when an LLM serves as its probabilistic generator (Section 1); Section 3.2 accordingly defines the measured object as the distribution a model states under a given prompt and decoding procedure. Stated and token-level distributions can differ: prompted judgments and direct probability measurements can diverge [1], and LLMs describe a target opinion distribution more accurately than they simulate it by sampling [2].

The paper does compare against repeated single-choice sampling, with more draws than verbalized calls (Appendix A, Table 7). For Llama-3.3-70B at IND, all 300 single-choice draws at temperature 0.2 returned ORD, and at temperature 1.0 ORD and MDW took 97% and 3% of 300 draws, whereas 20 and 18 verbalized calls named 35 and 34 destinations. In this pilot, single-choice draws concentrate on one or two answers and do not recover the distribution that each verbalized call states. Because the checkpoints are open-weight, we will obtain token-level probabilities by scoring every listed option under a single-choice prompt with locally served Llama-3.1-8B, Qwen2.5-7B, Mistral-7B, and Llama-3.3-70B checkpoints and compare the result with the verbalized vectors and across models; we will include those experiment results in the camera-ready version.

Weakness 2: The task format is narrow.

Weakness 2: It would strengthen the paper to show whether the observed patterns hold for these more common forms of preference elicitation and under larger-scale sampling.

Probability vectors over a fixed choice set are the input of the choice simulators that motivate the study (Section 1). Pairwise comparisons and open-ended responses are valuable formats, but turning them into such a distribution requires an additional aggregation model with its own assumptions. Section 6 will state this scope and name these formats as extensions. Single-choice elicitation is already compared with the verbalized vectors (Appendix A), and the token-level comparison above extends this comparison across models, including Llama-3.1-8B and Llama-3.3-70B; larger-scale sampling is addressed below.

Weakness 2: Also, many key comparisons rely on very limited model calls (15–20 per setting).

Each call is converted into one probability vector over the 300-option list (Section 3.1), so a 20-call batch pools 20 distributions rather than 20 single choices. Models are compared on groups of five calls with top-five shortlists scored on held-out calls, and the differences are large relative to the variation within the recorded calls: for Llama 70B/405B, between-minus-within JS is 0.150 [0.103, 0.197] and the shortlist loss is 17.7 percentage points [11.5, 24.0] (Tables 15 and 16). We will extend the Llama 70B/405B and Qwen 72B/235B pairs to larger batches; because the 405B endpoint used in the paper has been retired, this checkpoint must be deployed on our own GPU server, and we will include those experiment results in the camera-ready version.

References

[1] Hu, J., & Levy, R. (2023). Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 5040–5060). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.306

[2] Meister, N., Guestrin, C., & Hashimoto, T. (2025). Benchmarking distributional alignment of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 24–49). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.naacl-long.2

Thanks for the response. However, I still do not think the current paper is qualified for this venue. I will maintain my score.

Reviewer_YNZmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates LLM-generated preference distributions. Models are prompted to assign probabilities to lists of 300 options and return JSON outputs. The study examines repeated calls, model choice, prompt wording, and output processing. Experiments cover nine open-weight models from three families, mainly on travel choices, with additional restaurant and product tests. The authors compare distribution distances and top-choice overlap, and evaluate shortlists on held-out calls. In most main travel comparisons, agreement is higher within models than between models. Changing the model can shift the distribution as much as changing the prompt, though this comparison depends on the baseline batch. Parsing and normalization also affect the distributions.

Critical strengths
  1. The main comparisons are well designed and provide clear evidence that model choice matters. They use equal numbers of calls and evaluate shortlists on held-out calls. Between-model differences exceed within-model variation in eight of nine model pairs. Transferring a shortlist between Llama 70B and 405B reduces coverage by 17.7 percentage points on average.

  2. The analysis of output processing offers practical guidance for LLM-based simulation. Parsing, matching, and normalization can substantially change the final distributions. About 30% of the probability mass in Yelp outputs does not match the choice list. Processing changes alone reduce the number of active Yelp names from 103 to 77.

Critical weaknesses
  1. Ambiguous probability elicitation. The prompts ask about potentially overlapping events, such as visiting multiple restaurants within a month, but require probabilities to sum to one. Models may interpret this task differently, so the observed disagreement may reflect task ambiguity rather than differences in preferences.

  2. Limited control and generalizability. Several model comparisons differ in generation budgets, prompts, or response filtering, while the main repeated-call study uses only one airport and persona. These limitations make it difficult to attribute disagreement to model choice or generalize the central finding.

Questions for authors

Q1: Which model comparisons used identical prompts, generation budgets, and response-selection rules? Does the main finding hold when restricted to these comparisons?

Q2: What random variable do the elicited probabilities represent? How should the sum-to-one constraint be interpreted when multiple visits or purchases can occur within the specified period?

Q3: Do you have independent-batch or additional-context results showing that between-model disagreement consistently exceeds within-model variation? Please distinguish these from resampling the same recorded calls.

2 comments
AuthorsOpenReview ↗

Response to Reviewer YNZm (1/2)

Thank you for the careful review and for recognizing that the matched, held-out comparisons give clear evidence that model choice matters. This part answers Weakness 1 with Question 2 and the control part of Weakness 2 with Question 1; Part 2 answers its generalization part with Question 3, and the baseline-batch point in your summary.

Weakness 1: Models may interpret this task differently, so the observed disagreement may reflect task ambiguity rather than differences in preferences.

Question 2: What random variable do the elicited probabilities represent?

We define each vector as a choice-share distribution: the probability that one choice occasion within the stated window (a trip, a dining visit, a purchase) selects each listed alternative. This is the standard discrete-choice probability conditional on a choice occasion [1, 2]; travel demand modeling likewise separates the destination of each trip (destination choice) from how often a person travels (trip generation) [3], and our travel prompt elicits the former. The window sets the context rather than imposing exclusivity; with several occasions, the shares describe how occasions are allocated across alternatives. At equal numbers of calls, within-model JS is 0.035–0.144 against 0.185–0.394 between Llama models (Section 5.3), so each model resolves the instruction consistently across calls. Whichever reading a model adopts, a simulator receives the stated vector. Section 3.2 will define the estimand explicitly. The single-occasion experiment with an outside option proposed by Reviewer MHyE fixes one mutually exclusive decision and tests whether the cross-model disagreement persists; we will include those experiment results in the camera-ready version.

Weakness 2: Several model comparisons differ in generation budgets, prompts, or response filtering, while the main repeated-call study uses only one airport and persona.

Question 1: Which model comparisons used identical prompts, generation budgets, and response-selection rules? Does the main finding hold when restricted to these comparisons?

Five of the nine pairs share prompt text, temperature 0.2, and token budget within their family: Llama 70B/405B (1,024 tokens; Appendix K.2), and the three Qwen pairs and Mistral-7B/Mixtral-8x7B (4,096 tokens; Appendix I.1). All calls pass through one parsing and matching policy. Apart from Mistral-24B, response selection in the Qwen and Mistral logs is not content-based, although complete attempt histories are not retained for every batch (Appendix I.6), and the Llama 70B/405B collection code makes a fixed number of calls without retries. Restricted to these five pairs, between-minus-within JS ranges from 0.081 to 0.200 and the shortlist-transfer loss from 6.9 to 17.7 percentage points, with every 95% resampled range above zero (Tables 15 and 16). The other four pairs differ in settings or in what was recorded: two involve the adapted Mistral-24B prompt with retry-selected responses (Appendix I.5), and the two Llama-8B pairs lack a per-call record of the 8B token budget (Appendix E); of these, only Mistral-7B/24B has ranges that cross zero. The nine-model comparison is at IND, and Section 4 will state this scope.

References

[1] McFadden, D. (1974). Conditional logit analysis of qualitative choice behavior. In P. Zarembka (Ed.), Frontiers in econometrics (pp. 105–142). Academic Press.

[2] Train, K. E. (2009). Discrete choice methods with simulation (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511805271

[3] Ortúzar, J. de D., & Willumsen, L. G. (2011). Modelling transport (4th ed.). Wiley. https://doi.org/10.1002/9781119993308

AuthorsOpenReview ↗

Response to Reviewer YNZm (2/2)

Question 3: Do you have independent-batch or additional-context results showing that between-model disagreement consistently exceeds within-model variation?

Yes, from separately collected calls rather than resampling. Within Llama-3.3-70B, the repeated-call design covers four departure airports with 20 calls each (Table 1). A second 70B batch of 15 fresh calls with the original wording stays closer to the primary batch (JS 0.065) than the 405B model does (JS 0.184); the difference of 0.119 has a 95% range of [0.048, 0.188] (Table 4, last row). Across contexts, each of 20 personas receives a fresh call from each Llama model: between-model JS averages 0.359–0.486 (Table 3), against 0.055–0.187 for consecutive calls to 70B at a fixed airport and persona (Table 2). No Llama pair gives an identical top five for any of the 20 personas (Table 3), so the between-model gap is not specific to one persona, and the within-model reference already spans all four airports (Table 2). We also computed the comparison on the separately collected temperature-zero batches (Appendix M) for the seven pairs that have them (the two Llama-8B pairs lack a separately generated batch): between-minus-within JS is above zero for every pair, from 0.117 [0.083, 0.148] to 0.290 [0.247, 0.332] (Llama 70B/405B), and the shortlist loss is positive for every pair, from 2.0 points (Qwen 7B/72B, range [0.0, 5.7]) to 33.7. These batches also differ in serving (a dedicated 405B endpoint with a 4,096-token limit, and local vLLM for Qwen-72B and Mistral-24B, which keeps its adapted prompt), so they show the pattern across collection conditions rather than isolating temperature; the revision will state these details in Appendix M. We will also add new 100-call batches of the Llama 70B/405B and Qwen 72B/235B pairs, and we will include those experiment results in the camera-ready version.

Summary: Changing the model can shift the distribution as much as changing the prompt, though this comparison depends on the baseline batch.

The paper reports both baseline batches for this reason: under the primary batch, one of the four rewordings matches the 70B/405B model change, and under the second, the model change exceeds every rewording on average (Section 5.4; Table 20). The supported claim is that the model is a setting to fix and report alongside the prompt wording, not a fixed ordering of model and wording effects.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the AC and All Reviewers

We thank the reviewers for their constructive feedback. The reviews note that the matched comparisons with held-out shortlist scoring give clear evidence that model choice matters (YNZm), that the 17.7-point shortlist-coverage loss is a useful measure of model disagreement (MHyE), that the processing audit offers practical guidance (YNZm) and methodological detail often overlooked in LLM simulation studies (MHyE), and that the question is practically relevant (TVwV).

We define each elicited vector as a choice-share distribution, the standard discrete-choice probability for one choice occasion; low within-model variation shows that each model resolves the instruction consistently across calls (YNZm, MHyE). Restricted to the five model pairs with identical prompts, temperature, and token budgets, every between-minus-within JS and shortlist-transfer range stays above zero, and a separately collected 70B batch, 20 personas, and the temperature-zero batches of seven pairs also show between-model JS above within-model JS (YNZm). The paper already compares verbalized vectors with 300 single-choice draws per temperature, which concentrate on one or two destinations (TVwV).

We will add the single-occasion experiment with an outside option (MHyE), a token-level comparison with locally served checkpoints (TVwV, MHyE), and larger batches for the Llama 70B/405B and Qwen 72B/235B pairs (TVwV, YNZm). The Llama-3.1-405B endpoint used in the paper has since been retired and no serverless inference provider we checked offers it, so the 405B runs require deployment on our own GPU server, which takes longer than the discussion period. We will include those experiment results in the camera-ready version, together with an explicit estimand definition and scope statement in Sections 3.2, 4, and 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.