acceptodds
Under review as a conference paper at ICLR 2027

Human Non-Choice in LLMs: Internal Support, Limited Expression

Abstract

Large language models (LLMs) are increasingly used to simulate human responses under demographic and contextual personas, yet matching directional opinions alone may overlook an important aspect of human response behavior: people sometimes decline to commit to an available position. We study whether persona-conditioned LLMs systematically underexpress “Can’t choose” (CC) responses and whether this mismatch can be mitigated through activation-level interventions. Using 42 survey items spanning environment, family, and health across six countries, we construct cell-balanced human targets and contrast them with LLM response probabilities. We derive shared activation directions from teacher-forced contrasts between CC, midpoint, and directional responses, and intervene at inference time using three complementary mechanisms: component-wise head gain, angular rotation in residual space, and additive steering. Intervention strengths are selected to minimize divergence from human CC rates, while midpoint behavior and the full six-category response distribution remain independent evaluation outcomes rather than optimization targets. We further compare learned interventions with matched random-direction controls and a direct semantic-logit bias, and evaluate generalization on held-out respondents under multiple answer-order permutations. This framework separates the problem of behavioral fidelity from simply increasing indecision, providing a controlled test of whether internal representation interventions can correct systematic omissions in persona-conditioned LLM simulations without retraining.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, persona-conditioned LLMs, synthetic survey respondents, activation steering, representation engineering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9iWTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Interesting paper - The paper studies whether LLM survey respondents can reproduce human "can't choose" response. Across three models, non-choice is underproduced. Output rescaling recovers much of the average deficit, while internal interventions show that later computation can either support or counteract a correction.

Critical strengths
  • I found the distinction between matching the average non-choice rate and matching which questions elicit non-choice particularly useful. The paper shows that these can come apart, rather than treating a better aggregate as a complete solution. Keeping midpoint responses separate is also important, especially given OLMo's behavior.

  • The intervention analysis goes beyond showing that a feature is decodable. Holding an edit fixed while allowing or clamping later computation gives a concrete way to examine what happens to it. I also appreciate the candid comparisons with random controls and simple output rescaling.

Critical weaknesses
  • The main correction results have a fairly narrow generalization scope. The controllers use question-specific coefficients, and the held-out people answer questions already seen during development. Changing answer order weakens the gains substantially in some cases. This is mentioned in the paper, but I would bring it closer to the main results so that readers do not mistake available control for a general correction method.

  • Another minor concern is the component-specific explanation is also less secure than the path-level observation. Random controls are competitive in Llama and OLMo, and some restoration comparisons have limited delivered-norm matching. These qualifications do not invalidate the live-versus-clamped effects, but they limit claims that the selected components have a unique role in non-choice.

Questions for authors
  1. Can the main results more clearly separate transfer to new people, robustness to answer order, and transfer to new questions?

  2. How robust are the key restoration comparisons to better norm matching and the fidelity gains to uncertainty in the human targets?

3 comments

We sincerely thank the reviewer for the careful and constructive review. We will add the new results to the revised PDF, which we will upload during the discussion period. Triplets of numbers refer to Qwen / Llama / OLMo.

Q1. Separating transfer to new people, answer order and new questions (also the first weakness).

We agree, and we will make this scope clear in the revision. The controller rows of Table 1 are for new people on the 42 known questions, with per-question coefficients, averaged over four answer orders (§3.2, l.134–140). Table 14 (App. D.1) separates the first two axes. New people: under the fitted order, the CC cross-entropy gain is preserved (development −.0141 / −.0496 / −.0319; held-out −.0141 / −.0520 / −.0329; §5.3, l.284–285). Answer order: the gain is smaller on the unfitted orders, most visibly in Llama (D.1, l.1281–1282). We will add the tables below.

New analysis: fitted versus unfitted orders, with and without question-specific coefficients.

  • What we did. On held-out people, we evaluated the selected Neuron controllers (Qwen, Llama) separately on the fitted order A and on the unfitted orders B–D. We also replaced the per-question coefficients with their median over the 42 questions (one value per layer), without refitting. The median variant applies to the Neuron coefficients; for OLMo's Head controller, Tables 14 and 16 already give the order split with per-question coefficients.
  • Why. To separate what transfers across answer orders from what depends on question-specific coefficients, for both overall itemwise fidelity (CC cross-entropy) and the across-question pattern (item-level correlation with human CC rates).

Change in CC cross-entropy from baseline (lower is better):

Qwen, order AQwen, B–DLlama, order ALlama, B–D
Rescaling (one scalar)−.010−.009−.039−.043
Per-question coefficients−.014−.008−.052−.022
Median coefficients−.009−.008−.034−.020

Item-level Pearson correlation with human CC rates across the 42 questions, computed on the mean probability within each order group (the four-order values elsewhere, .35 / .18, average all four orders first and are not the mean of these columns):

Qwen, order AQwen, B–DLlama, order ALlama, B–D
Baseline.25.31.18.14
Rescaling (one scalar).23.26.19.14
Per-question coefficients.87.53.91.35
Median coefficients.21.27.20.12
  • Result. With per-question coefficients, the cross-entropy gain on unfitted orders is about half of that on the fitted order, and the correlation falls from .87 / .91 to .53 / .35, still above baseline. With median coefficients, the cross-entropy gain on unfitted orders is nearly the same as with per-question ones (−.008 vs −.008; −.020 vs −.022), but the correlation returns to baseline. Most of the cross-entropy gain on unfitted orders thus remains with one shared coefficient per layer at evaluation, whereas the correlation gain depends on question-specific coefficients.
  • What it shows. The controllers establish available control on known questions, not a general correction method, which is the reviewer's distinction and the paper's intended reading ("The optimized controllers establish access to correction", l.326–327).
  • New questions. In the paper, a separate source-global Head controller, frozen and applied to six Government 2006 items, transfers unevenly: in the four-order mixture, it lowers CC cross-entropy for Llama only (D.3, Table 17). The question-independent U edit keeps the same downstream fidelity signs on six Environment 2010 items in Llama and OLMo (F.2, l.1677–1678). New: on Government 2006, it lowers CC cross-entropy (Llama −.0042 [−.0079, −.0005]; OLMo −.0051 [−.0085, −.0021]), and on the British Election Study, a British survey to which nothing was fitted (design in our reply to a8UX, W2), it raises "Don't know" log-odds mainly with later layers live (Llama +.440 vs +.016 clamped; OLMo +.202 vs +.010; Table I5).

Component-specific explanation (the second weakness). We agree, and the paper does not claim a unique role for the selected components: "The controls limit uniqueness" (§5.5, l.432–433), and competitive controls and incomplete norm matching restrict component specificity (§6, l.463–464). Retuned random controllers do not differ reliably from the selected ones on CC cross-entropy in Llama and OLMo (l.313–314), so the controllers show that a correction is accessible, not that the extracted candidates are superior (l.312–313). The §5.5 statement that a component supporting ordinary CC can oppose an upstream correction requires only that component's own live-versus-restored contribution, which uses all inputs (Table 26). We will clarify this scope in the revision.

New, the path-level observation at additional sites: under a design fixed in advance, we repeated the live-versus-clamped test at six additional MLP sites, the next two per model by the paper's development ranking. The prespecified rule (later layers adding in Llama and OLMo, contributing negatively in Qwen) held at 4 of the 5 sites it scores; the U-versus-N contrast raised CC at 4 of the 6 sites, and at all 4 the later-layer contribution had the predicted sign. The misses are reported in Table I1 (details in our reply to ZxQ4, Q2).

Q2. Norm matching and uncertainty in the human targets.

  • Restoration. The comparison §5.5 relies on, the sign of each component's live-minus-restored contribution, uses all inputs and involves no norm matching (Table 26). The 7–61% applies only to the contrast between the selected component and its controls (G.2, l.1972–1977); we will clarify this in the revision.
    • What we did (new). Keeping the matching criterion unchanged, we recomputed the difference between the selected component's contribution and the mean of its two controls on the inputs where the selected component and both controls pass.
    • Result. Pooled over the four answer orders, the difference has the sign of the selected component's all-input contribution, with 95% CIs excluding 0: Llama +.022 (27% of inputs); OLMo −.044 (61%); Qwen ×.75 / ×1.25: −.032 / +.045 (7%).
    • What it shows. On matched inputs, the selected components differ from their controls in the direction of their all-input contributions. Qwen's subset is small, so we treat it as a consistency check; the §5.5 conclusion rests on the all-input contributions.
  • Direction specificity, where matching is nearly complete. For the fixed U edit, matching passes for 100% of Llama inputs and about 95–96% of OLMo inputs, and U yields lower CC cross-entropy than each displacement control matched to its norm (Table 27). New: among U and 40 random directions norm-matched per input, U gives the largest increase in the logit of mean CC on a 96-persona panel (Llama .502 vs at most .185; OLMo .127 vs at most .080; Table I2).
  • Human targets. The paper's intervals condition on the empirical human targets and do not include the survey's full sampling uncertainty (A.4, l.826–827).
    • What we did (new). We resampled human respondents within demographic cells (1,000 bootstrap resamples), recomputed the targets and re-evaluated the main fidelity gains; we also computed the agreement between two disjoint halves of the respondents.
    • Result. Rescaling and the selected controllers lower CC cross-entropy in all 1,000 resamples. The baseline item-level correlations with human CC rates, computed on probabilities averaged over the four orders (point estimates .35 / .18 / .04; Q1 reports order-group values), have 95% percentile intervals of [.28, .41] / [.10, .24] / [−.03, .11]. The split-half reliability of the human item-level CC rates is .94 (Table I6).
    • What it shows. The fidelity gains persist under respondent resampling within demographic cells, and the targets are reliable enough to evaluate which questions receive non-choice.

Thanks for the detailed response. This addresses most of my concerns.

Reviewer_ZxQ4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at whether LLMs playing survey respondents say “Can’t choose” as often as real people do. Using 42 ISSP statements, personas from six countries and three 7 to 9B models, the authors find that all three models pick it much less often than humans. They keep the midpoint answer as a separate outcome. This turns out to matter, because OLMo uses the midpoint far too much while still underusing “Can’t choose”. Flattening the output probabilities to match how spread out human answers are closes a large part of the average gap, but it puts the extra non-choice on the wrong questions. The rest of the paper tries internal steering to raise “Can’t choose” and then looks at how later layers amplify or cancel a fixed edit.

Critical strengths

I think the question is a good one. A lot of work checks whether simulated respondents hold the right opinions, and much less checks whether they should hold an opinion at all. Treating “Can’t choose” and the midpoint separately was the right call, and the OLMo result shows why. The point that fixing the average rate does not fix which questions get non-answers is useful for anyone building synthetic survey samples. I also appreciated that the authors test on held-out personas and several answer orders, include random-support controls, and report the cases where results go the other way.

Critical weaknesses

My main problem is with the internal part of the paper, which makes up most of it. The selected steering methods lower “Can’t choose” cross-entropy only slightly. In Llama and OLMo they do about as well as steering built on random components. The plain output rescaling from Section 5.2 beats them on every metric in all three models. The state-derived edit in Section 5.4 closes 9.5% of the gap in Llama and 3.8% in OLMo, and barely moves Qwen. After reading the paper I was not sure what the internal analysis adds beyond what the output-level analysis already shows.

I also found it hard to see one mechanism in the results. Each model ends up with a different method, site and edit. Appendix A.4 says the mechanistic follow-ups were exploratory and designed after earlier results were seen. The observation that later layers can strengthen or cancel an edit fits earlier work on self-repair. What would make it convincing here is showing that the authors can predict when it happens at sites or doses they did not select on. The restoration comparisons in Section 5.5 also rest on a small share of inputs, since only 7 to 61% pass the norm-matching checks.

Generalisation is a concern given the motivation. The controllers fit a separate coefficient for each question and are tested on new personas answering the same questions. On new questions (Government 2006, Appendix D.3) the results are mixed. The gains also shrink a lot when the answer order changes, especially for Llama. Simulating answers to surveys the method has not been tuned on is exactly the setting the introduction motivates.

Finally, the paper is very hard to read. Almost every sentence is heavily qualified and the main ideas come with little intuition, so I struggled to tell which results the authors see as the core contribution. Part of the rescaling result is also expected by construction, since Appendix E shows that flattening has to raise any rare enough option. I think a shorter paper built around the behavioural finding and the allocation problem, with one focused and prespecified mechanistic test, would be much stronger.

Questions for authors
  1. What does the internal analysis tell us about non-choice that the output rescaling does not?
  2. Can you predict in advance whether later layers will amplify or buffer an edit at a new site or dose?
  3. Does any method in the paper improve which questions receive non-choice, rather than only the average rate?
  4. How do the internal controllers perform on new questions without a coefficient fitted per question?
4 comments

We sincerely thank the reviewer for the detailed and constructive review. Following it, we ran the test the reviewer asks for (prediction of the later-layer response at sites fixed in advance; Q2) and three further analyses: U against 40 random directions (Q1), the same edit protocol at the same layer in all three models (Part 2), and an item-level correlation analysis of allocation (Q3). We will add all new results to the revised PDF, which we will upload during the discussion period. Triplets of numbers refer to Qwen / Llama / OLMo.

Q1. What does the internal analysis add beyond output rescaling?

Rescaling shows that the models' answers are too concentrated and that loosening them restores much of the average amount of non-choice. Because flattening raises any sufficiently rare option (App. E, l.1566–1567), it cannot show whether a fixed internal direction derived without human targets changes CC, or why a change made inside the model has the size it has at the output. The internal experiments show both, and the paper uses them to test its account, not as a rival correction method (l.317; H.3, l.2080–2083). The account has three steps (§6, l.440–442):

  1. Concentration. All three models concentrate probability more than people do: entropy over the four directional answers is below human-cell entropy on 36 / 39 / 38 of 42 items (§5.2, l.262–263). Loosening this uniformly restores 58–91% of the average gap (l.269) but not which questions receive it (l.273–275).
  2. Usable internal support. Three tests, which differ in how much is fitted, point the same way. A direction taken, with no human target, from development prompts describing an unformed opinion (U) versus settled neutrality (N) improves CC, midpoint and overall fit on ordinary prompts in Llama and OLMo when set to U, and worsens all three when set to N (§4.2; §5.4, l.335–336; Eq. 15). Scaling existing activity by a fixed 25%, with no coefficient fitted, raises CC and improves its fit in all three models (§5.5, l.397–399). Controllers fitted on other people remove 63 / 55 / 43% of the reducible CC cross-entropy on held-out people (Table 1, Eq. 8).
  3. Later computation. Holding an edit fixed, we let later layers recompute ("live") or replay them from the unedited run ("clamped"). The direct effect of the U edit is small (+.020 / +.014 log-odds clamped in Llama / OLMo), and later layers produce most of it (+.603 / +.133 live; Table 2), lowering CC cross-entropy by .0143 / .0031 (§5.4, l.344–346); in Qwen, the MLP after the edited head takes part of a correction back (Table 2). Activity that supports CC in ordinary responses can oppose an upstream correction (§5.5).

New experiment: U against 40 random directions.

  • What we did. At MLP19, the paper's U site in Llama and OLMo, we compared U with 40 random directions, each matched per input to U's displacement norm, with later layers live, on a panel of 96 test personas × 42 items × 4 orders. The outcome is the change in the logit of the mean CC probability, a different statistic from Table 2.
  • Why. The reviewer notes that random components do about as well for the fitted controllers. This test asks the same question of the fixed edit, with 40 random directions instead of the single one in Table 27.
  • Result. U gives the largest increase among the 41 directions in Llama (.502 against at most .185) and OLMo (.127 against at most .080) (Table I2). In Qwen, where MLP19 is not the paper's site and later layers take back most of the U–N contrast (Part 2), U changes the outcome by only .0002, within the range of the random directions.
  • What it shows. In Llama and OLMo, the effect of the U edit depends on its direction and not only on the size of the change, which an output-level analysis cannot test.

Controller size, random controls and rescaling. These observations match the paper: rescaling fits better than the selected controllers on CC cross-entropy, midpoint error and total variation in all three models (l.316–317), and random supports do about as well in Llama and OLMo (l.313–314). On size, cross-entropy cannot fall below the human entropy (.1515; Table 1); of the reducible part, the controllers remove 63 / 55 / 43% and rescaling 64 / 79 / 66%. On purpose, the controllers test whether a correction is reachable inside the model for new people, not which method fits best (l.312–313). Random supports are retuned by the same per-question search, so they test whether our extraction helps that search (D.2, l.1290–1291): it does in Qwen (63% against 19%), while in Llama and OLMo random supports remove about as much (56% / 44%). The 9.5% / 3.8% comes from one fixed edit, identical for all questions and never tuned to human CC rates. Qwen's small U effect is of the same size as the direct effect in the other two models (+.012 against +.020 / +.014 log-odds; Table 24); what differs is what later layers add, which the same-layer experiment in Part 2 tests in all three models.

"Hard to see one mechanism … a different method, site and edit." The reviewer is right that the selected operators, sites and edits are not identical across models, and the paper does not claim one shared circuit (l.325–326; H.2, l.2045–2046). What is common is a relation: all three models concentrate probability more than people do, corrections come with a loosening of that concentration (all 32 internal controls with |1−β| ≥ .01 flatten the directional scores and improve CC cross-entropy; l.320–321, exploratory), and later layers decide how much of a fixed edit reaches the output. The selection rules are the same in every model (l.194–195; F.1), and the outcomes overlap: Llama and OLMo share the U edit and site (MLP19; F.1, l.1616–1619), and Qwen and Llama the controller type (Neuron; Table 1). Qwen differs because its selected U site is its final layer (MLP31), with no later layers to release (Table 24). The experiment below removes this difference.

New experiment: the same edit protocol at the same layer in all three models.

  • What we did. We applied the U-versus-N contrast (the U edit minus the N edit) at MLP19 in all three models, each with its own direction derived by the paper's protocol (F.1), on the panel and statistic of Table I2, with later layers live and clamped. For Qwen, MLP19 is one of the Q2 sites.
  • Why. In Table 2, Qwen's edit differs in layer and type; this test holds both fixed.
U versus N, MLP19QwenLlamaOLMo
Direct (clamped).035.027.033
Added by later layers−.029+.775+.558
Final (live).005.802.591
  • Result. The direct effects are small and similar (.027–.035). Later layers add +.775 / +.558 in Llama and OLMo; in Qwen they take back most of the direct effect (−.029, 95% CI [−.033, −.025]). Rows add up before rounding.
  • What it shows. With layer and protocol held fixed, the models differ mainly in what follows the edit, as in step 3 of the account.

Self-repair. We agree that Qwen's buffering is consistent with the compensation shown by McGrath et al. [1]. Self-repair, however, describes one direction, later layers undoing part of an intervention; in Llama and OLMo, they instead produce almost all of the edit's effect (Table 2). Beyond this precedent, (i) the direction of the later-layer response is not fixed across models at the same layer and edit protocol; (ii) it changes fidelity to people: releasing later layers lowers CC cross-entropy for U in Llama and OLMo (§5.4), whereas Qwen's following MLP raises it by .0022 for the beneficial head reduction (H14 ×.75; F.2); and (iii) the opposing component can be one that supports CC in ordinary responses (§5.5).

Q2 (and A.4). Can the later-layer response be predicted at additional sites?

This is the test the paper names as its next step (§6, l.464–466; H.4, l.2093–2094). The rule, fixed in advance, follows Table 2: later layers produce most of the effect in Llama and OLMo and contribute negatively in Qwen. It held at 4 of the 5 new sites it scores; U raised CC relative to N at 4 of the 6 new sites. We did not test new doses; in the paper, Qwen's following MLP opposes both head doses (Table 2; exploratory).

  • Design. As A.4 states, the submitted follow-ups were exploratory; this is the focused, prespecified test the reviewer recommends. The hypotheses and thresholds were fixed before any new outcome was examined. Sites: for each model, the two next-ranked MLP sites by the paper's development score (F.1), excluding the paper's site; the ranking uses development prompts only. H1: the U-versus-N contrast is positive with later layers live (95% CI excluding 0). H2: in Llama and OLMo, the later-layer part has the sign of the final effect and the direct part is less than half of it, scored where H1 holds; in Qwen, the later-layer part is negative, scored at both sites. H3 (allocation) is reported under Q3.
  • Results. Llama MLP15: direct .011 → final .160; MLP23: .024 → .066; OLMo MLP23: −.009 → +.113; Qwen MLP19: later layers −.029, final +.005. H1 did not hold at OLMo MLP15 (direct −.031 → final −.137) or Qwen MLP23 (direct −.135 → final −.011). The H2 miss is Qwen MLP23, where later layers cancelled 92% of a negative direct effect, so the later-layer part was positive, opposite to the rule's fixed sign (Table I1).
  • What it shows. The later-layer pattern of step 3 is predicted at four of the five scored additional sites, so it is not specific to the paper's selected sites. Qwen MLP23 is a failed prediction: later layers buffered a negative direct contrast, which we report as a descriptive observation, not as support for H2. The H1 misses bound the sites at which this edit raises CC relative to N, as §6 anticipated.

References

[1] Thomas McGrath, Matthew Rahtz, János Kramár, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771, 2023.

Q3. Does any method improve which questions receive non-choice?

The largest allocation gains come from the controllers with per-question coefficients, each fitted to its question's human CC rate (C.2, l.1253–1254). Among question-independent methods, the instruction proposed by Reviewer a8UX raises Qwen's correlation from .35 to .47, and rescaling changes it little; all remain far below the human split-half value of .94. The submission showed that rescaling's itemwise CC changes correlate negatively with the corrections required (−.32 / −.21 / −.57; §5.2, l.273–275); we now also measure the correlation between model and human CC levels for each method.

  • What we did (new). Across the 42 items, we correlated model and human CC levels and compared this with the agreement between two disjoint halves of the respondents.
  • Result. Human halves: .94. Baselines: .35 / .18 / .04 (probabilities averaged over the four orders). Question-specific controllers (Neuron, Qwen / Llama), on held-out people: .25 → .87 / .18 → .91 under the fitted order and .31 → .53 / .14 → .35 under the unfitted orders; with one shared median coefficient per layer, the gain disappears (reply to 9iWT, Q1). Question-independent methods: rescaling .32 / .19 / .03; at the six new sites of Q2, the U edit keeps the change in correlation within ±.05 (each 95% CI; the prespecified H3); an instruction or a richer persona reaches at most .47 (Qwen under a8UX's instruction, with mean CC almost unchanged; Tables I3, I4, I6).
  • What it shows. Amount and allocation move separately. A better allocation is accessible inside the model when each question's target is supplied, but it is not predicted (H.4, l.2098–2100). We will report these correlations in the revision.

Q4. Performance on new questions without per-question coefficients.

Only partly for the fitted controllers, as the paper states ("existing transfer results are limited or mixed"; H.4, l.2096–2098). The question-independent U edit, which has no fitted coefficient, still lowers CC cross-entropy on new questions in Llama and OLMo.

  • U edit. In the paper, the later-layer contribution to fidelity has the same sign on six additional Environment 2010 items (F.2, l.1677–1678). New: applied unchanged to the six Government 2006 items and 288 personas of D.3, scored as the mean of order-specific losses, it lowers CC cross-entropy (Llama −.0042 [−.0079, −.0005]; OLMo −.0051 [−.0085, −.0021]; Table I5). New: on the British Election Study, to which nothing was fitted, it raises "Don't know" log-odds mainly through later layers (+.440 / +.202 live against +.016 / +.010 clamped, Llama / OLMo; design in our reply to a8UX, W2).
  • Fitted controllers. The separate frozen source-global Head controller (D.3, l.1448–1450) improves Government 2006 CC cross-entropy in the four-order mixture for Llama only (Table 17). New, on the 42 known questions: median coefficients keep nearly all of the cross-entropy gain on unfitted orders (−.008 / −.020, against −.008 / −.022) but not the allocation gain (reply to 9iWT, Q1).
  • Answer order. We agree, and the paper reports this cost (D.1, l.1281–1282): Llama's gain is −.052 on the fitted order and −.030 averaged over four (Table 14).
  • Untuned surveys. We agree that per-question controllers cannot be applied to a survey without human responses; the paper lists this as a limit (§6, l.467–469). For a new survey, it contributes the assessment the introduction motivates (l.039–040), which needs no fitted controller; on the BES, the shortage recurs.

7–61%. The restoration contributions in §5.5 (Table 26) use all inputs. The 7–61% applies only to the comparison of the selected components with their controls, which requires all three to pass the norm checks (G.2, l.1972–1977); the reviewer is right that this limits component specificity. We will clarify this in the revision. New: keeping the criterion unchanged, we recomputed the difference between the selected component's contribution and the mean of its two controls on the inputs that pass; pooled over the four answer orders, it has the sign of the selected component's all-input contribution, with 95% CIs excluding 0: Llama +.022 (27% of inputs), OLMo −.044 (61%), Qwen ×.75 / ×1.25 −.032 / +.045 (7%).

"Expected by construction." We agree that flattening raises a sufficiently rare option (App. E, l.1564–1567); we do not count the direction of this change as a finding. The algebra does not fix the amount (one scalar fitted to directional entropy, never to CC, recovers 58–91%, whereas a pooled-entropy target overshoots to 5.68–6.47%; Table 18), the allocation (Q3), or the internal counterpart (controls fitted only to CC also flatten the directional scores, l.320–321, although a CC-only shift would not, l.1517–1519).

Readability and the suggested restructuring. We agree, and we will revise the manuscript following the reviewer's suggestions.

AuthorsOpenReview ↗

Reminder: Request for Feedback on Our Rebuttal

Thank you again for your review. As the discussion period is coming to a close, we wanted to briefly follow up on our rebuttal.

If you have a chance to review our response, we would appreciate any further feedback, particularly if you have any remaining concerns. We would also appreciate your consideration of our response in your final assessment.

Thank you for your time and consideration.

Reviewer_a8UXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies why LLM are less likely to quantify uncertainty, i.e. select "Can't choose" in surveys. The author take a survey data and use meta data as persona and prompt llm to do the survey and compare to the real human results. They find that llms never choose uncertainty options. They used temperature scaling and activation steer to increase the probability of such actions, as well as how later transformer layers can strengthen or weaken the effects of these changes.

Critical strengths
  1. the analysis of how later transformer layers can increase or decrease activation steering is interesting.
Critical weaknesses
  1. The personas are way too simplified to capture realistic human values. They only contain demographic informations, rather than beliefs, interests, knowledge, and etc. It's also like using llm to do a linear regression with those simple features. Human values are intrinsically complex, and meta data are too simplified to represent realistic human opinion simulation.
  2. The overall task is too constrained. The whole task is asking llms to generate "can't choose" more frequently in doing surveys. It is an interesting topic by itself, but it's hard to know if such finding can generalize to broader human opinion simulation or other uncertainty related tasks.
  3. Another big weakness I found is the lack of instruction prompt ablation. If you add "If you are not sure about your answer, select 'Can't choose'." to the llm's instruction, would this increase the number of llms choosing the "can't choose" option?
  4. Finally, the paper focuses on matching the overall frequency of "can't choose" rather than predicting when people should actually use it. For example, a method could simply increase the probability of "can't choose" across all questions and improve the average results, without actually learning when people are uncertain. In fact, the authors find that temperature scaling improves the overall frequency, but its changes across questions are negatively correlated with the corrections needed. This raises concerns about whether the method actually captures human non-choice behavior.
Questions for authors

see all above

6 comments

We sincerely thank the reviewer for the careful reading and the concrete suggestions. Following them, we conducted three new experiments, a richer-persona experiment (W1), an evaluation on an additional survey programme, the British Election Study (W2), and an instruction ablation with the reviewer's proposed instruction (W3), as well as one new item-level analysis (W4). In each experiment, the shortage of non-choice remains. We will add all new results to the revised PDF, which we will upload during the discussion period. Triplets of numbers refer to Qwen / Llama / OLMo.

W1. Persona design.

The reviewer is right that our personas contain only demographic and background fields (A.2). We tested whether this explains the finding: when each persona also carries the person's own answers to the other questions on the same topic, the overall fit improves, but "Can't choose" (CC) remains far below the human rate in all three models.

  • What is evaluated, and the regression analogy. We agree that demographic fields cannot determine one person's values, and the paper does not ask them to: "No individual-answer accuracy is used" (A.2, l.766–767). For each question, the target is a cell-balanced population distribution: human and model distributions average the same demographic cells with equal weights (Eq. 5). Sparse conditioning limits individual prediction, but at this level a calibrated simulator, whether a regression on these fields or a model given far more information, should still reproduce the population's CC share (3.57% on average). The models return 1.79 / 0.68 / 1.06%, and their answer probabilities are more concentrated than people's (§5.2, l.262–263).
  • The setting follows recent work. Conditioning on demographics or respondent characteristics is a common setting in this literature, including recent ICLR papers. For example, Jahanparast et al. [1] prompt with a single group label (e.g., education, region, religion or party) to predict that group's survey answer distribution, and T. Hu et al. [2] condition on a population description or a single sociodemographic attribute. Earlier, Argyle et al. [3] and Santurkar et al. [4] condition on respondent characteristics or group descriptions.
  • New experiment: richer personas.
    • What we did. To each persona, we added the respondent's actual answers on the five-point scale to the other studied items of the same ISSP module (the target item excluded). The respondent's own CC answers were not shown, so the model cannot copy them. All other settings follow the paper (42 items × 288 test personas × 4 answer orders).
    • Why. To test whether the CC shortage shrinks when each persona carries the respondent's observed positions on the same topic.
    • Result. The added information improves the overall fit: total variation to the human distribution falls by .10 / .07 / .02. CC remains far below 3.57%: it is 1.24 / 0.89 / 1.07% (originally 1.79 / 0.68 / 1.06%), lower in Qwen (the gap widens from 1.78 to 2.33 percentage points), slightly higher in Llama and unchanged in OLMo. The probability of the most likely scale answer in each response stays high (.58 → .58 / .55 → .52 / .62 → .70).
    • What it shows. Richer personas improve full-distribution fidelity but do not close the CC gap, and the responses remain concentrated (Table I4). This is in line with the paper's link between the shortage and concentrated response probabilities (§5.2). In the revision, we will note that this experiment additionally uses respondents' answers to other items.

References

[1] Erfan Jahanparast, Zhiqing Hong, and Serina Chang. What do large language models know about opinions? In International Conference on Learning Representations (ICLR), 2026.

[2] Tiancheng Hu, Joachim Baumann, Lorenzo Lupo, Nigel Collier, Dirk Hovy, and Paul Röttger. SimBench: Benchmarking the ability of large language models to simulate human behaviors. In International Conference on Learning Representations (ICLR), 2026.

[3] Lisa P. Argyle, Ethan C. Busby, Nancy Fulda, Joshua R. Gubler, Christopher Rytting, and David Wingate. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351, 2023.

[4] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.

W2. Generalization.

We thank the reviewer for raising this point. On an independent survey programme, the British Election Study, the models again under-express non-choice, and in Llama and OLMo the paper's internal edit again reaches the answer mainly through later layers (new experiment below).

We first clarify what is expected to generalize. The task is not to raise CC as such: corrections are scored against each question's human rate, with the midpoint and the full six-category distribution as unfitted outcomes (§3.2, l.142–148). The paper's claim has three parts: the models under-express non-choice, they contain internal activity that can raise it, and later layers determine how much of an internal correction reaches the answer. This pattern, not a fitted controller, is what should generalize. Within the paper, the shortage spans three topics, six countries (each model under-predicts CC in five of six country means; §5.1, l.236–238) and three model families, and the paper names new questions as the test of whether the pattern persists (§6, l.467–469).

  • New experiment: British Election Study (BES).
    • What we did. We used BES Internet Panel Wave 31 (Great Britain, 2026), which differs from the ISSP in country, year, question wording and response labels. We selected eight of its five-point items, each of which shows "Don't know" next to a middle category ("About right" or "Neither agree nor disagree"), so non-choice is again an option separate from the middle of the scale. We built 320 British personas balanced over 64 demographic cells (region, degree, age, sex), compared model probabilities with the weighted human answer shares of the same cells, and kept the paper's prompt format and four answer orders.
    • Why. If the shortage were specific to the ISSP items or to the label "Can't choose", it should not appear here.
    • Result (shortage). All three models give "Don't know" far less probability than people choose it: 2.0 / 1.1 / 2.2% against 13.2%, and less than people on every one of the eight items (Table I5).
    • Result (later layers). We then applied the paper's U edit (Eq. 15) unchanged, with the same site, direction and anchors, to Llama and OLMo; Qwen's U site is its final layer, so no later layers follow it (Table 24). Mean per-input "Don't know" log-odds rise by +.440 / +.202 (Llama / OLMo) when later layers are allowed to respond, against +.016 / +.010 when they are held fixed (Table I5).
    • What it shows. The shortage is not specific to the ISSP items or to the label "Can't choose", and the same edit, never fitted to BES, again reaches the answer mainly through later layers, as in Table 2. The first and third parts of the account therefore carry over to a new survey, country and response label.
  • Additional ISSP items. The U edit is defined from development prompts without any human target. In Llama and OLMo, it keeps its downstream fidelity signs on six additional Environment 2010 items (F.2) and, as a new result, lowers CC cross-entropy on Government 2006 (−.0042 / −.0051; Table I5). This supports the second part: the internal activity is not tied to the original items.
  • Other uncertainty-related tasks. The quantity studied here is the rate at which people decline to choose, a property of the simulated population, not a model's confidence in a factual answer; the paper distinguishes opinion absence from uncertainty (§2.1, l.86–87). The processes involved, logit scaling, confidence-guided factual abstention and downstream compensation, have been studied in other tasks [1–3], and the paper connects them to a discrepancy with human responses (§6, l.443–446). We do not claim that our edits transfer to factual abstention.

References

[1] Alessandro Stolfo, Ben Wu, Wes Gurnee, Yonatan Belinkov, Xingyi Song, Mrinmaya Sachan, and Neel Nanda. Confidence regulation neurons in language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 37, 2024.

[2] Dharshan Kumaran, Nathaniel Daw, Simon Osindero, Petar Veličković, and Viorica Patraucean. Causal evidence that language models use confidence to drive behaviour. Nature Machine Intelligence, 8(9):1411–1425, 2026.

[3] Thomas McGrath, Matthew Rahtz, János Kramár, Vladimir Mikulik, and Shane Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771, 2023.

W3. Instruction ablation.

We thank the reviewer for proposing this ablation. We ran it with the reviewer's exact wording: the instruction raises CC by only .03 / .07 / .01 percentage points, and all three models remain far below the human rate.

  • What we did. We added one line before "Answer with the letter only." on the full test set (42 items × 288 personas × 4 orders): P1, the reviewer's instruction, "If you are not sure about your answer, select 'Can't choose'."; and P2, a variant that states that every option is acceptable instead of conditioning on uncertainty, "All six options are valid responses, including 'Can't choose' and '[the item's midpoint label]'. Select the option that best represents the person described."
  • Why. If the shortage arose only because the models are not told that CC is acceptable, an explicit instruction should close it.
CC probabilityQwenLlamaOLMo
Original1.79%0.68%1.06%
P11.82%0.75%1.07%
P22.29%0.72%0.77%
Human rate3.57%3.57%3.57%
  • Result. P1 closes under 3% of each model's gap to the human rate. Compared with P1, P2 raises CC further only in Qwen (2.29%, still below 3.57%) and lowers CC in OLMo (0.77%). In OLMo, P1, which does not mention the midpoint, moves probability mainly to the midpoint (41.4% → 60.6%; human 22.2%), and P2 raises it to 75.6%; the midpoint error grows from 24.5 to 40.2 / 53.4 pp.
  • What it shows. The shortage is not removed by telling the models that CC is acceptable (Table I3). By contrast, loosening the concentration of the answer probabilities, with the prompt unchanged and without fitting CC, closes 58–91% of the same gap (§5.2, l.269). This is consistent with the paper's account, in which the shortage is tied to concentrated response probabilities rather than to a lack of permission to use the option. The OLMo result also illustrates why the paper evaluates the midpoint separately from CC (§6, l.458–460).

W4. Overall frequency versus when people choose CC.

We agree that a closer overall frequency is not sufficient, and the paper is built around this distinction.

  • Scores and controllers are per question. The primary score, CC cross-entropy, and the MAE are computed for each question against that question's human rate and then averaged (Eq. 1; §3.2, l.138–140). Each question's score is minimized at its own human rate; a uniform increase can still lower the average when most questions are under-predicted, which is why the paper also evaluates allocation (next point).
  • The paper tests the reviewer's scenario. One rescaling shared by all questions lowers the average error, yet its itemwise changes correlate negatively with the corrections needed (−.32 / −.21 / −.57; §5.2, l.273–275). The paper concludes that evaluation must distinguish "overall frequency from its allocation across questions" (§7, l.477–478); rescaling is "an output reference" (l.317), not a proposed correction method.
  • New analysis: item-level correlation with human CC rates. Across the 42 items, we correlated each model's CC probability with the human CC rate and compared it with the correlation between two disjoint halves of the human respondents. The two halves correlate at .94; the models reach .35 / .18 / .04 (probabilities averaged over the four orders). Rescaling gives .32 / .19 / .03, and the instructions and richer personas also remain far below .94; in Qwen, P1 raises it to .47 while mean CC barely changes. Controllers whose coefficients are fitted to each question's human rate raise it on unfitted answer orders from .31 / .14 to .53 / .35 (Qwen / Llama), but not when one coefficient per layer is shared across questions (Table I6; reply to 9iWT, Q1).
  • What it shows. Amount and allocation move separately, as the paper argues. An internal control can produce part of the item pattern when each question's target is supplied, but no correction that treats all questions alike exceeds .47, against .94 between two halves of the human sample. Predicting the pattern for new questions remains open (H.4, l.2098–2100).

Comments on the paper

1 comment

We sincerely thank all three reviewers for their careful and constructive reviews. We have uploaded the revised PDF, with new results highlighted in blue. Below we summarize how the revision addresses the reviewers' concerns and where the material cited in our replies now appears.

We are grateful for the reviewers' positive remarks. ZxQ4 described the question of whether simulated respondents "should hold an opinion at all" as a good one and wrote that treating "Can't choose" (CC) and the midpoint separately "was the right call". 9iWT found the distinction between the average non-choice rate and which questions elicit it "particularly useful". a8UX found the analysis of how later layers strengthen or weaken an edit interesting.

1. Readability and core contribution (ZxQ4). We added an overview figure of the three-step account (Figure 1) and an opening paragraph stating what each §5 subsection tests, and revised the §5.2–§5.5 headings to state their results. §5.2 states that the direction of the rescaling effect is expected by construction; the findings are how much CC probability it recovers and how that mass is allocated across questions.

2. Mean rate versus allocation (a8UX, ZxQ4). Across items, model and human CC log odds correlate at .35/.18/.04 (Qwen/Llama/OLMo), against .94 between two halves of the human respondents (§5.2; App. D.4, Table 21). Rescaling closes 58–91% of the mean CC gap but leaves allocation inaccurate. The largest allocation gains come from the Qwen and Llama controllers with coefficients fitted to each question's human CC rate; replacing the coefficients with their median at each layer returns the correlation to near baseline (§5.3).

3. What the internal analysis adds (ZxQ4). 1) Against 40 random directions of matched norm, the fixed edit toward an unformed opinion (U) gives the largest increase in the logit of mean CC in Llama and OLMo (§5.4; App. F.5, Table 30). 2) With the same protocol at the same layer in all three models, the contrast between U and an edit toward considered neutrality (N) is similar with later layers clamped, while the later-layer responses differ (§5.4; Table 30); §6 relates Qwen's buffering to self-repair. 3) A rule fixed in advance (later layers produce most of the effect in Llama and OLMo and contribute negatively in Qwen) holds at four of the five scored new sites, failing at Qwen's multilayer perceptron in layer 23; new doses remain to be tested (App. F.4, Table 29). §5.3 states that rescaling is the stronger correction and that random controls remain competitive in Llama and OLMo.

4. Generalization (a8UX, ZxQ4, 9iWT). §5.3 states that the controllers establish access to correction for new people on known questions (scope also in the Table 1 caption); gains are smaller on answer orders not used for fitting, and a separate controller frozen for new questions has mixed outcomes (App. D.1, D.3, D.4). The fixed U edit, with no fitted coefficient, lowers mean CC cross-entropy across six Government 2006 items in Llama and OLMo (App. F.6, Table 31). On the British Election Study, all three models underproduce "Don't know" on each of eight items (§5.1; App. B.5, Table 13). §6 distinguishes a population's rate of declining to choose from confidence in a factual answer.

5. Personas and instructions (a8UX). Adding to each persona the respondent's answers to the other items on the same topic (CC answers excluded) improves overall fit, yet CC stays below the human rate (App. B.4, Table 12). The reviewer's instruction to select CC when unsure raises CC by at most .07 percentage points (App. B.3, Table 11).

6. Component specificity and uncertainty in the human targets (ZxQ4, 9iWT). §5.5 states that restoration contributions use all inputs; only the comparison with controls is restricted to the 7–61% of inputs that pass the norm checks. On these inputs, the selected-minus-control differences have the signs of the all-input contributions (App. G.3). The controls limit claims of a unique role for the selected components (§5.5). Rescaling and the selected controllers lower CC cross-entropy in all 1,000 resamples of human respondents, drawn within demographic cells independently for each item (App. D.4).

Updated table numbers. Line numbers in our replies refer to the submitted PDF.

In our repliesRevised PDF
Tables I1–I4, I6Tables 29, 30, 11, 12, 21
Table I5Tables 13 and 31
Tables 14, 16, 17, 18Tables 17, 19, 20, 22
Tables 24, 26, 27Tables 28, 33, 34
Tables 1–2, equations, sections, appendicesunchanged

The revised paper keeps its three-step account: the three tested models concentrate their answers and underproduce non-choice, they contain usable internal support for it, and later computation determines how much of an internal correction reaches the answer. We would be glad to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.