acceptodds
Under review as a conference paper at ICLR 2027

Right on Average, Wrong per Respondent: A Mechanistic Diagnosis of LLM-Simulated Choice

Abstract

Language models are increasingly used as synthetic survey respondents, and what those applications need is how a choice moves when an attribute changes, not how plausible a single answer looks. Existing evaluations score the level of an answer and not its response to a change. We introduce MagBench, which perturbs one attribute of a choice situation at two sizes, the smaller half the larger, on five datasets of real human choices, and asks whether the response scales in proportion. In aggregate it does. For an individual respondent it does not, and most of that respondent's movement has nothing to do with the size of the change. The aggregate is right because those respondent-specific parts cancel, and what cancels is not noise but a displacement that reappears at both sizes of change and under a different prompt. Inside the models the size of the change is decodable and reaches the subspace that carries the response, and neither fact predicts behavior. Writing the displacement one respondent's perturbation produces into another respondent's run moves the response to the donor and not the recipient, at 0.89 against over 67 runs and five model families. It holds even in a model whose choices follow the human direction 51 percent of the time, so whether this structure is present and whether the behavior it should drive is valid are separate questions. Encoding a quantity is not using it.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, activation patching, linear probing, causal mediation, LLM simulation of human respondents, survey simulation, representation-behavior gap

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ykxSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests whether simulated choices respond proportionally to two sizes of change in an attribute. Aggregate responses often look closer to proportional than individual responses. Probing and activation interventions then show that a model's response to a perturbation can be carried by a low-dimensional displacement and transferred between respondents.

Critical strengths
  • Testing responses to changes is a useful direction for evaluating synthetic respondents. The probability-based readout and relabeling controls help make the test cleaner. I also appreciate the reference-validity checks and the authors' decision to exclude electricity from the corrected conclusions. The intervention work includes several useful controls.
Critical weaknesses
  • My main concern is the interpretation of “wrong per respondent”. The individual human counterfactuals are not observed; the proportionality target follows from a fitted choice model. The curvature checks help, but they do not exclude other forms of nonlinear human response. Matching a small-to-large ratio also does not mean that the aggregate response has the correct absolute magnitude. I would describe this as a test against an operational reference, rather than direct evidence that the simulated individuals are wrong and the average is right.

  • I also have some concerns on the “systematic share”. As defined in Appendix B, it is the absolute mean response divided by the mean absolute response. This measures cancellation or signed coherence, rather than the fraction of movement caused by change size. Perfectly proportional individual responses with opposing signs could have a low share. The separate two-size analysis is evidence of scaling inconsistency, but it does not justify saying that the remaining movement has nothing to do with size.

  • The held-out respondents are used to select the peak probe layer before interventions, so they are not fully held out from analysis selection. More importantly, the transplanted displacement contains both proportional and non-proportional effects. Showing that it transfers, or scales approximately linearly along the tested path, does not isolate the error component or rule out downstream contributions in general.

Questions for authors
  1. Do the behavioral conclusions survive more flexible human references, and can proportionality be reported separately from absolute human accuracy?

  2. Can layer selection use a separate validation split, and can an intervention target the non-proportional component while preserving the common response?

Reviewer_5kVEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors investigate how appropriate LLMs are in simulating human choices. They put the LLM in the position of e.g. picking a mode of transportation based on different price information and observe how the response changes when input attributes, such as the price of one mode of transportation, changes. They compare the change of LLM outputs to estimated log-odds of human responses. They find, that on average the effect quantified in the LLMs is similar to the effect in human responses, but the LLMs fail to mimic individual responses.

Critical strengths

The authors have identified an interesting problem to investigate. I agree with the authors that LLMs seem to replace real human studies and that it would be interesting to better understand if LLMs tend to make similar trade-off like humans. The idea of changing one input value and observing the effect it has on the LLM output follows well established best practices in research.

Critical weaknesses

I acknowledge the “harder problem” the authors have identified, namely deriving the quantity to compare the LLMs change in output against. But I disagree with the authors assumptions about the human preference. They model the change in target by assuming the learned beta-coefficients to have linear contribution on the log-odds of choosing one option over the other. Which I do not think is the case. Take the “one rendered prompt” the authors report in line 1009 f. For example. Imagine a human that has a strict monetary incentive and no other preference for any mode of transportation. With the authors assumption we assume the log-odds to linearly in-/decrease with in-/decreasing price of the train ticket. But this is not the case. The response function of the human in this case is more like a step function, where at the moment one mode of transportation is cheaper the log-odds flip completely. Therefore I believe the very fundamental assumption about the human preference function to be violated. In consequence all investigation down the line are invalid.

Questions for authors

No questions.

Reviewer_dmfiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether LLMs can reliably simulate how people change their choices when prices or other attributes change in given scenarios. It introduces MagBench, which compares responses to small and large changes against a statistical reference fitted to human choices. They find that the group average responds roughly as expected, but individual responses often do not. Individual errors cancel out, making the average look more reliable than the underlying predictions. Experiments on internal activations suggest that models encode the size of a change but do not consistently use it correctly.

Critical strengths

The paper provides a controlled test of whether simulated choices respond consistently to changes in prices or other attributes. Across multiple models and datasets, it finds individual inconsistencies that group averages conceal. This offers a useful evaluation beyond average agreement.

Critical weaknesses

The reference does not establish whether the simulated responses match real people. Agreement or disagreement with a fitted statistical model does not directly show human accuracy. Appendix A provides useful checks, but some tested values fall outside the range covered by the human data: 22% for Swissmetro headway and about 50% for electricity prices. The authors exclude electricity from the corrected evaluation, but still report it in Table 6 and list it in Figure 2.

The presentation makes the main argument difficult to assess. Key terms such as “systematic share,” “displacement,” and “response-carrying subspace” are introduced without enough intuitive explanation. The paper moves between group averages, individual responses, and internal model experiments without clearly explaining how each result supports the next claim. This makes it difficult to distinguish what the experiments demonstrate from what the authors infer.

Questions for authors
  1. What evidence supports using the fitted model to judge individual human responses? Should “right” and “wrong” be understood only as agreement or disagreement with this reference?

  2. Which findings explain the failure to scale, beyond identifying where it has already formed?

  3. Is electricity included in the results in Section 4.1? If so, how do those results change when it is excluded, given that its reference is invalid?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.