acceptodds
Under review as a conference paper at ICLR 2027

Same Choice, Different Response: Evaluating LLM Simulators Under Attribute Changes

Abstract

Large language models (LLMs) are increasingly used to simulate human choices, but reproducing observed choices may not predict how people respond when decision attributes change. We study interventional response alignment: agreement between human and model changes in the full choice-probability vector. We show that low observational choice risk does not, without additional assumptions, guarantee low response risk. On SwissMetro, we evaluate responses to travel-time and fare changes against fitted population-level human references. Prompted LLMs show response discrepancies even when their baseline choices match recorded human choices. More strikingly, on a fixed respondent-disjoint test set, choice-only SFT and DPO raise a 7B model's accuracy from 44.3% to 69.3–70.8%, yet improve time alignment while worsening fare alignment and the prespecified combined endpoint. Supervising paired response differences reduces the combined gap by 41% relative to SFT and outperforms choice augmentation on the same inputs. This gain does not extend to a held-out attribute, comes with worse negative log-likelihood, and has an unresolved accuracy difference. Because responses to the constructed changes are unobserved, these findings concern agreement with fitted population references rather than individual counterfactual fidelity. They show why choice accuracy alone is insufficient to evaluate simulators intended for use under change.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Human Behavior Simulation, Interventional Response Alignment, Counterfactual Evaluation, Preference Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tZ3POpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether reproducing observed human choices can generalize to simulating human choices at large. To do this, they rely on counterfactuals for things like price and time in the SwissMetro domain. Their results show discrepancies even when baseline choices actually match humans (as fitted populations).

Critical strengths

This is a very important problem. So much research goes into getting LLMs to simulate human behavior (and so much capital in industry) without any sort of verification. I think the counterfactual approach is the right one, and the paper is well formulated.

Critical weaknesses

I think this work has potential to be great, but as of now, it’s quite unambitious. Collecting (or even finding) a human dataset with counterfactuals seems rather inexpensive. Instead, the paper is compromising the potential analysis by using fitted human responses. This, combined with using just one domain and one model family, makes the paper underwhelming.

I say this because I believe that LLM simulators of human behavior are overhyped, and I have been waiting for work to prove it. If I had to play devil’s advocate, I could say that perhaps frontier models can in fact simulate behavior, that one domain is not enough to justify your claims, or that it’s not enough data to simulate effectively. I encourage the authors to think bigger and help us show the limitations of LLM simulators.

Although not critical, you might want to review more existing literature. Other researchers are using counterfactuals to study AI systems:

  • Cherep, M., Ma, C., Xu, A., Shaked, M., Maes, P., & Singh, N. (2026, April). A framework for studying ai agent behavior: Evidence from consumer choice experiments. In International Conference on Learning Representations (Vol. 2026, pp. 63296-63326).
  • Karvonen, A., Ong, E., Kantamneni, S., & Marks, S. (2026). Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments. arXiv preprint arXiv:2608.16747.
  • Yang, T., Mi, Z., Huang, Y., Zhang, G., Chen, J., Yang, Y., & Yi, J. (2026). AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents. arXiv preprint arXiv:2609.33658.
Questions for authors

I don’t have any questions that could change my assessment. I think this work should be accepted once it tackles this problem more rigorously, and that should not be possible to do in a rebuttal period.

2 comments

We thank the reviewer for this assessment and for recognizing the potential value of the work. We agree that the central methodological concern requires a more rigorous and comprehensive treatment than can realistically be completed within a rebuttal period, and we do not regard the additional analyses reported here as a definitive resolution of that concern. Nevertheless, we have taken concrete first steps to clarify and strengthen the evidence: benchmarking the associated human models on a common reconstructed test set, adding a nested-logit robustness check, constructing semi-synthetic settings with known post-change choices and independent response ground truth, and directly quantifying the circularity of closed-reference evaluation. These analyses help delineate what the current evidence does and does not establish. In particular, they show that reference-based evaluation can materially understate residual response error, while external validity beyond SwissMetro and generalization across LLM scales and families remain open. Accordingly, we will revise the manuscript to narrow its claims, distinguish alignment with a fitted response operator from true human counterfactual fidelity, and present the new analyses as preliminary stress tests rather than comprehensive validation. We agree that a full resolution would require broader real intervention data, stronger independent evidence about human responses, and controlled cross-model experiments beyond the rebuttal period. We appreciate the reviewer’s clear articulation of the standard that this line of work should ultimately meet.

I'm considering this more an acknowledgement than a rebuttal, but if there's something you'd like to discuss, I'd appreciate a human-written response.

Reviewer_m5XBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether LLM simulators that reproduce human choices also respond like humans when a decision attribute changes, such as travel time or fare. Since human responses to these changes are not observed, the human side is a fitted multinomial logit on the SwissMetro dataset. Prompted LLMs show response gaps even when their baseline choice matches the human one. On Qwen2.5-7B, SFT and DPO improve choice accuracy but make the response gap worse.

Critical strengths

The question is well motivated. LLM simulators are mostly evaluated on matching observed choices, but people want to use them for "what if" questions. Showing that higher choice accuracy can come with worse response alignment (Table 3) is a useful lesson.

Critical weaknesses
  1. My biggest concern is the human reference. The "human response" is the response of a fitted MNL. The response-aware loss is also trained on targets from the same MNL and evaluated against it, which is somewhat circular.
  2. The dataset coverage is too narrow. All intervention results come from one dataset, i.e., SwissMetro, with only time and fare changed. OPeRA and the sports study do not seem to test intervention responses, so it's not clear the findings hold beyond SwissMetro.
  3. The training study is small. Does the finding generalize beyond Qwen2.5-7B to a larger model or model from different family?
Questions for authors

See weaknesses.

4 comments

For the first weakness, We agree that training and evaluating exclusively against the same fitted MNL can overstate response fidelity. The synthetic ground-truth experiment quantifies this issue directly. Under the misspecified data-generating process, response-aware training achieves a mean core response gap of 0.0054 when evaluated against the same reference that generated its targets, but its independently measured ground-truth gap is 0.0170—approximately 3.18 times larger. Closed-reference evaluation therefore materially overstates the final level of alignment. At the same time, the independent evaluation does not eliminate the relative training benefit. Response-aware training reduces the ground-truth core gap by 57.1% relative to choice-only training under misspecification, and by 76.7% under correctly specified MNL ground truth. It also outperforms same-input choice augmentation under the misspecified process: the paired augmentation-minus-response-aware difference in ground-truth core gap is +0.0195 [0.0165, +0.0227]. Thus, in the tested surrogate settings, the absolute residual error is underestimated by the closed reference, but the relative improvement remains visible against independent ground truth. We will therefore report reference-based and independently evaluated response endpoints separately. The original reduction against the fitted reference should be described only as improved alignment with that specified response operator, not as an equivalent improvement in true human counterfactual fidelity. We also avoid generalizing the preserved strategy ordering beyond the three surrogate strategies tested here.

For the second weakness, We agree that the present intervention evidence is narrow. The current results directly support response alignment for SwissMetro time and fare interventions; OPeRA and the sports study provide evidence about observed-choice simulation, not validated intervention responses. The semi-synthetic experiment strengthens the SwissMetro analysis by supplying known post-change choices and independent response ground truth, but it does not constitute evidence from a second real dataset or from a new intervention domain. We will therefore scope the claims accordingly and avoid presenting cross-dataset observed-choice performance as evidence of cross-domain intervention validity. Establishing external validity requires at least one additional choice dataset with a well-defined intervention and either observed post-change outcomes or a separately justified response model. Until that experiment is completed, generalization beyond SwissMetro time and fare remains an open limitation rather than a demonstrated finding.

For the last weakness, The current training study does not establish that generalization. Its direct LLM evidence is limited to Qwen2.5-7B, and the newly completed synthetic training experiments use tabular neural surrogates rather than additional LLM architectures. They clarify the evaluation mechanism and the circularity issue, but they cannot substitute for training a larger model or a model from another family. We will therefore narrow the conclusion to the tested architecture and training setup and identify model scale and model family as explicit threats to generalization. A definitive answer requires a controlled comparison in which choice-only, choice augmentation, and response-aware training are applied to at least one larger model and one different model family, with matched data, seeds, and training budgets. Both reference-based and independently evaluated response metrics should be reported. Until those runs are completed, we make no claim that the relative benefit of response-aware training has been demonstrated across LLM scales or families.

Thank you so much for your attention for the paper, please let us know if there's any problem or question.:)

Reviewer_kbJjOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper highlights a research gap in current discrete choice modelling: a well-fitted discrete choice model on a snapshot attribute matrix XX doesn't ensure reliable prediction when the attribute matrix changes. The authors propose a new metric for the model in addition to the current choice fitness, which includes the discrepancy between the machine learning model and human observations in the change in choice probability under attribute changes. Two empirical datasets are used to test the efficiency of the proposed loss function. The authors find that the proposed loss function yields better hold-out performance compared to the choice-only loss function.

Critical strengths
  1. The research gap is well clarified. The discrete choice models based on the observed attribute matrix are at risk of being misspecified when the attribute matrix changes.
  2. The methodology of LLM training considers not only the choice accuracy but also its alignment with the classic discrete choice models. Therefore, some explainable assumptions on the classic discrete choice models, such as "random utility maximization" and coefficient direction, are also implicitly learned by LLMs.
Critical weaknesses

The proposed method seems limited in addressing the stated research gap.

  • The loss function for training an LLM is based on the classic discrete choice model's prediction under attribute change as a penalty term. This is based on the assumption that the classic discrete choice model has an interpretable and reasonable prediction on unobserved attribute change. If so, why not directly use the classic discrete choice model (such as pooled MNL)?
  • The stated LLM's predictive performance on changed attributes is never formally tested. A Monte Carlo experiment synthesizing some observed interventions to validate the proposed LLM's performance is necessary but missing.
Questions for authors
  1. How is the performance of the LLM choice simulator compared to its associated human models?
  2. What if the underlying discrete choice model is misspecified?
  3. How about the prediction performance on synthetic dataset that the changed attribute chosen choice is known?
8 comments

For the first weakness, we agree that when the research object is a structured choice setting with explicitly specified attributes and interventions, and the analyst trusts the fitted discrete-choice specification, directly using that model is the simpler and more transparent solution. Pooled MNL or a richer reference model should therefore be a direct benchmark, not merely a source of training targets. Our results do not demonstrate that an LLM is superior to a classical discrete-choice model for SwissMetro time or fare prediction, and we will revise the motivation to avoid implying such a general advantage. The intended role of response-aware training is different: the discrete-choice model supplies an explicit, interpretable response operator for regularizing an LLM, while the LLM provides a flexible language-based interface for rich respondent descriptions, unstructured context, and choice settings that are not completely represented by a prespecified utility function. The research question is therefore whether such a general-purpose simulator can be steered to respect specified intervention responses while retaining that flexibility—not whether it should replace MNL in a fully structured problem where MNL already suffices. We acknowledge that the present evidence establishes this broader motivation only partially, and that reliance on a fitted teacher also inherits the teacher's specification errors.

For the second weakness, we agree that a formal intervention-based Monte Carlo evaluation is necessary. As a first step, we added a semi-synthetic experiment using real SwissMetro contexts, known pre- and post-intervention choices, and complete true response vectors. The design includes both correctly specified MNL ground truth and a deliberately misspecified nonlinear, respondent-specific, nested-logit process. Using tabular neural surrogates, response-aware training reduces the mean ground-truth core gap from 0.0190 to 0.0044 under correct specification and from 0.0397 to 0.0170 under misspecification, with improvements in post-intervention NLL in both cases. However, these surrogate results validate the evaluation protocol and the behavior of the training mechanism in a controlled setting; they do not constitute the requested formal validation of the proposed LLM. We therefore do not present them as closing this gap. A complete revision must apply the same Monte Carlo protocol to the actual Base, SFT, DPO, and response-aware LLMs, compare them directly with pooled and richer discrete-choice baselines, and report post-change choice accuracy, NLL, and independently defined response errors. Until that LLM-level experiment is completed, the claimed intervention benefit remains preliminary and will be explicitly scoped as such.

For the first question, we added an independent same-test-set reconstruction using the official SwissMetro data and the published split seed. On the reconstructed 1,611-choice test set, the pooled MNL, primary MNL with interactions, nonlinear reference, and nested-logit reference achieve accuracies of 0.6567, 0.6754, 0.6878, and 0.6747, respectively. Their corresponding NLLs are 0.8267, 0.7173, 0.7063, and 0.7204. Five-fold respondent-disjoint validation gives the same qualitative ordering in log loss: the primary and nonlinear references outperform pooled MNL, with mean log losses of 0.7344 and 0.7310 versus 0.8094. For context, the paper reports Base, SFT, and DPO accuracies of 0.443, 0.701, and 0.701, and NLLs of 1.330, 0.669, and 3.270. Thus, SFT performs favorably on both observed-choice accuracy and probabilistic fit, whereas DPO achieves similar accuracy but substantially worse NLL. These results reinforce our distinction between predicting observed choices and aligning intervention responses: a model can perform well on the former without being reliable on the latter. Because the original split identifiers and LLM row-level predictions were not available for this independent reconstruction, we do not claim paired statistical significance between these reference-model results and the published LLM numbers. The comparison should therefore be interpreted as a common-cohort benchmark rather than a fully paired head-to-head test.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.