acceptodds
Under review as a conference paper at ICLR 2027

Belief Elicitation Is an Intervention: Measuring and Manufacturing Belief–Action Consistency in Language Models

Abstract

Evaluating whether language models act consistently with their stated beliefs is central to assessing calibration, risk preferences, and safety alignment. Existing benchmarks typically measure this by asking a model for a subjective probability and observing its subsequent choice within the same conversation. However, treating this same-session agreement as evidence of behavioral predictability implicitly relies on an untested assumption: Measurement Non-Reactivity (MNR), which assumes that the act of asking a belief does not alter the subsequent action. We formalize this problem by separating the computed metric, same-session consistency (), from the true target, cross-context predictive validity (). We prove that fails to identify : under standard deterministic evaluation, the bounds on span the entire unit interval , rendering a perfect consistency score completely uninformative. To resolve this flaw, we propose the Shadow Fork, an evaluation protocol that pairs each elicited session with an unprompted action session to identify without assuming MNR, costing only one additional query per scenario. Across five frontier models, we show that standard protocols substantially inflate consistency due to reactivity: by in a calibrated delegation study and by on SimpleQA abstention. Crucially, interventional experiments injecting synthetic probabilities reveal that models mechanically conform to whatever number appears in their context ( decision flip rate). Therefore, observed belief–action consistency is largely an artifact manufactured by the evaluation prompt rather than evidence of genuine model introspection, highlighting the critical need for non-reactive benchmark design.

Primary area:
datasets and benchmarks
Keywords:
large language models, belief elicitation, measurement reactivity, benchmark validity, identification, introspection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Bu4xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper takes aim at Measurement Non-Reactivity—the assumption that eliciting an LLM's belief merely observes, rather than changes, a subsequent action—and argues that same-session belief–action consistency can be inflated as a result. It separates same-session consistency from cross-context predictive validity, proves that same-session data cannot identify the latter, and uses a paired "Shadow Fork" treatment/control protocol to identify both quantities and their difference. Across several frontier models and tasks, the experiments find substantial inflation under standard protocols. Replaying a model's report largely reproduces its action, while changing only the injected report can sharply change the action—evidence that the report can act as an input rather than a passive readout.

Critical strengths
  1. The paper identifies a broadly relevant and underappreciated validity problem in evaluating whether language models act on their stated beliefs.
  2. It offers a strong combination of conceptual clarification, formal identification analysis, and causal experimentation.
Critical weaknesses
  1. The empirical scope is limited to narrow decision formats and tasks, so the conclusions may not transfer to richer agentic or free-form settings.
  2. The causal interpretation depends on session-equivalence and control assumptions that are difficult to fully validate for closed model endpoints.
  3. The proposed protocol may require more experimental effort than many benchmark evaluations can readily support.
  4. Section 7 limits the claims to a numerical report scored against a threshold-based binary decision in two tasks. In SimpleQA, the tested thresholds are concentrated in the upper tail in Section 6. The paper does not establish that the same effect holds for free-text rationales, richer beliefs, or multi-step agents.
Questions for authors
  1. Could the paper include a mid-range SimpleQA threshold pilot, rather than only the upper-tail threshold range, to test the scope of the effect?
  2. What stronger test could exclude stable caching by the gateway provider, and has any such behavior been observed?
2 comments

We thank the reviewer for the positive assessment.

W1 / W4 (scope). The scope is deliberate: it is the configuration used by the evaluations we audit (Appendix A), and the one in which identification can be tested exactly. Free-text and agentic settings are beyond this paper, as Section 7 states.

W2 / Q2 (caching). Every call is an independent stateless request, memory features were disabled, and calls were issued in a seeded random order (Appendix G). Identical prompts sent 120 seconds apart show no batch effect (all p ≥ 0.38, Appendix C), and identical calls return different outputs. Among 5,760 within-cell replicate pairs for DeepSeek and Qwen, only 1 has an identical reasoning length. Caching could only make the two arms more alike, which would shrink I rather than inflate it.

Q1 (mid-range threshold). We added a penalty of 9.5 (τ ≈ 90.48), using 9.5 rather than 9 to avoid ties. On the four live models, pooled I = +0.084 [+0.062, +0.109], compared with +0.155 at λ = 20 and +0.145 at λ = 50 on the same models (Appendix I).

W3 (cost). One extra call per scenario.

Dear Reviewers,

We sincerely appreciate your time and effort in reviewing our manuscript "Belief Elicitation Is an Intervention: Measuring and Manufacturing Belief–Action Consistency in Language Models"#59853 and providing valuable suggestions.

As the author-reviewer discussion phase is drawing to a close, we would like to kindly confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues.

If you require further clarification or have any additional questions, please do not hesitate to let us know. We are more than willing to continue our communication with you.

Best regards, The authors of Paper #59853

Reviewer_jtofOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates two quantities that belief–action evaluations usually conflate: same-session consistency C1, where report and action come from one context (what benchmarks compute), and cross-context predictive validity C0, where the report is scored against the action taken when nothing was elicited (what the results are read as showing). It shows that C0 is not identified from same-session data, with the identified set reaching [0, 1] under deterministic reports, and uses the Shadow Fork, one extra scenario-paired action-only session, to identify C0 without assuming Measurement Non-Reactivity. Across five frontier models, pooled I = C1 − C0 is +0.410 on a delegation task (pre-specified ΔAUC +0.346) and +0.192 on SimpleQA abstention, and changing only the inserted report value flips the decision (+0.976).

Critical strengths

(1) The C1/C0 distinction is stated cleanly and applies to any evaluation that scores a decision made after an elicited report. It makes plain that a same-session score can’t tell a read-out from an input, and Appendix A shows that the evaluations reporting the highest belief–action agreement use exactly the same-context configuration. The fix costs one extra call per scenario.

(2) The experiments see the obvious objections coming. The filler arm controls for format, the payoff-blind placebo and the sham thresholds separate a level shift from threshold tracking, and branch C separates the content of the report from the act of reporting. The blocked permutation, the restricted wild cluster bootstrap, the temperature-0 rerun, leave-one-model-out and the worst-case missingness bounds all leave the headline intact.

Critical weaknesses

(1) The control branch is a bare action prompt, and Section 4 says this choice is “stated rather than defended empirically”. Reasoning-effort settings aren’t reported, although Appendix G notes that the default token budget truncated reasoning models. If a model reasons before choosing in the direct arm, it may already form an internal estimate, so the control may not show what an unprompted model does; part of I would then measure reasoning vs no reasoning, and elicitation itself would account for less of it.

(2) The abstract claims more than the evidence covers, and the novelty is modest. The abstract concludes that “observed belief–action consistency is largely an artifact manufactured by the evaluation prompt”, but the evidence is one numerical report scored against a threshold-based binary decision in two tasks, with 20 synthetic delegation scenarios and 100 SimpleQA items. The delegation thresholds sit at each model’s own crossover, where a number in context is most likely to tip the decision, and SimpleQA covers only τ = 95.24 and 98.04, with C1 at just 0.703. Section 7 states this scope correctly. As the authors say, the paired protocol comes from [1], and Proposition 1 is the standard missing-potential-outcome argument, so what is new is the identification framing and the measurements.

References
[1] When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs, arXiv 2026.

Questions for authors

Q1. Could you report the reasoning settings for each model and, for at least one model, run a control where the model reasons but isn’t asked for a number?

Q2. How large is I when a model is far from indifference? The mid-range point that Appendix I already identifies (λ = 9, τ = 90) would show whether +0.192 is typical or specific to the upper tail.

Q3. Table 2 shows C0 without its ceiling. A perfect predictor of a stochastic unelicited action scores 0.5 + d, which averages 0.828 here, so C0 = 0.564 alone reads as if beliefs barely predict anything. Could the capture ratio and the per-model stratified AUC of the belief for the unelicited action (0.468 to 0.688, Appendix D) go into the main text, and could the abstract lead with ΔAUC, the primary estimand, instead of I?

Q4. The abstract calls ΔAUC “pre-specified”. Is there a timestamped analysis plan, or can you say when and how the choice was fixed?

2 comments

W1 / Q1 (reasoning). All five models already reason under provider defaults before a bare action. Gateway usage logs show non-zero reasoning tokens in 99.5–100% of act_only calls (medians: GPT-5.5 255, Claude 316, Gemini 574, DeepSeek 559); DeepSeek and Qwen also return long visible traces. The action-only control therefore already reasons; it only lacks a number in the conversation. We further ran an explicit reason arm (2–4 sentences, no number) on GPT-5.5 and DeepSeek: I(reason) remains +0.266 [+0.188, +0.339] and +0.434 [+0.383, +0.489]. Branch C confirms the mechanism: changing only the inserted number flips 97.6% of decisions, while replaying the model's own number changes almost nothing (0.006). Settings are now in Appendix G.

W2 (scope). We agree. The abstract now limits the conclusion to single numerical reports scored against threshold-based binary decisions, matching Section 7.

W2 (thresholds near indifference). In the quarter of delegation cells where the unprompted action is most deterministic, far from indifference, inflation is still +0.360 (Appendix D). On the 1,138 rows where the unprompted action is fully deterministic (d = 0.5), I = +0.365 [+0.313, +0.418].

W2 (novelty). We agree that the paired protocol comes from Yamin et al. and that Proposition 1 is a standard argument. What is new is that the error has no fixed sign (Proposition 3), that an unpaired control cannot repair it (Corollary 2), how large it is across five models, and that a number in context drives the action.

Q2 (mid-range threshold). We added a penalty of 9.5 (τ ≈ 90.48), using 9.5 rather than 9 to avoid ties with integer reports. On the four live models, pooled I = +0.084 [+0.062, +0.109] (GPT-5.5 +0.154, Claude +0.105, Gemini +0.040, DeepSeek +0.040), compared with +0.155 at λ = 20 and +0.145 at λ = 50 on the same models (Appendix I). Inflation persists outside the upper tail, at about half the original magnitude.

Q3 (ceiling). We agree. Table 2 now reports the stochastic ceiling (0.828 on average), the capture ratio and the per-model AUC, and the abstract leads with the AUC measure.

Q4 (pre-specified). There is no timestamped analysis plan, so yes, the word was too strong. We have removed it from the abstract and Section 4.2, and we now report the AUC measure and binary inflation side by side.

Thanks for the clear response and the additional analyses, which resolve my concerns. The reasoning logs and the explicit reason arm (I = +0.266 and +0.434) show that the inflation does not come from a control without reasoning. The deterministic cells (I = +0.365) and the new mid-range threshold (I = +0.084) show that it persists far from indifference and outside the upper tail, although it is about half as large at the mid-range threshold. With the narrower abstract and the revised Table 2, the claims now match the evidence, so I am raising my rating to weak acceptance.

Reviewer_1fDMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that asking an LLM for a belief can change its next action, so belief–action agreement within one session (C1) can overstate how well the belief predicts unprompted behavior (C0). The authors prove that C1 does not identify C0 and propose the Shadow Fork, a protocol pairing every elicited session with an action-only session.

Critical strengths
  1. The paper separates two quantities that are often treated as one, same-session consistency and cross-context predictive validity, and states the gap between them formally. The proposed check is easy to adopt, requiring just one additional action-only query per scenario.

  2. The empirical work is carefully controlled for its scale. It uses five relatively frontier models, a format-matched filler arm, a payoff-blind placebo, an injected-number test, a wild cluster bootstrap, and a temperature-0 rerun, and it releases the raw responses and analysis scripts.

Critical weaknesses
  1. It would be better to define the scope of "belief elicitation" and "intervention" behind the strong title claim. Under the paper's framing, any added turn can change the context and the action; for example, the filler arm, which only asks how clearly the scenario is written, moves GPT-5.5's action by −0.144 (Table 1).

  2. It would be better to explain why the link between elicitation and intervention matters and who benefits from studying it. The paper itself notes that when a system elicits a report and acts on it in one context, C1 is the relevant quantity and its analysis does not apply (Section 3.1), and although Appendix A classifies twenty existing evaluations by protocol, the paper does not re-run any of them to show that their conclusions would change.

  3. It would be better to deepen the technical contribution. The non-identification result follows directly from the fact that same-session data never observe the unprompted action, and the paired protocol has been used before, as the authors mentioned, so beyond this short formal argument, the main new content is the measurement on 20 purpose-built scenarios and 100 SimpleQA items. The contribution would be stronger with a further step, for example, re-scoring published belief–action evaluations under the paired protocol or proposing a method that reduces this reactivity.

  4. It would be better to widen the range of tasks and report types. The claims cover one numerical report scored against a threshold-based binary decision in two tasks, and the per-model threshold calibration in the delegation study rules out comparing consistency levels across models.

Questions for authors
  1. How do the authors define "belief elicitation" and "intervention"? Would a filler turn that also changes the action count as an intervention under this definition?

  2. Is there an existing benchmark or reported result whose conclusion changes when the Shadow Fork is applied? Showing one would make the importance concrete.

  3. Why was a bare action-only context chosen as the control rather than a context with reasoning but no belief report, and how would the results change with the latter?

  4. In practice, which users or decisions depend on C0, such as safety audits or certification, and how much inflation would be needed to alter those decisions?

3 comments

We thank the reviewer for the careful review.

W1 / Q1 (definition, filler). Asking for a belief is an intervention when it changes what the model does next. An extra turn can also move the action, as the filler arm does for GPT-5.5. What matters for our claim is not that shift, but whether the later action tracks the reported number. Filler barely creates that tracking; asking for a belief does. Branch C makes the point cleanly: change only the number in context, and the decision flips; put the model's own number back, and almost nothing changes.

W2 / Q2 / Q4 (which depends on C0). C0 is the right quantity when a report gathered in evaluation is used to predict behaviour that later runs without that report, as in confidence-based abstention or safety audits. If a pass/fail bar sits between C0 and C1, the decision is wrong, and that gap is the inflation we measure. We have not re-run a published benchmark under the Shadow Fork. What we show is that, in our own tasks, same-session scoring and cross-context scoring lead to different practical conclusions: models that look perfectly consistent under C1 still vary substantially under C0.

W3 (novelty). We agree that Proposition 1 is a standard argument. What is new is that the error has no fixed sign, that an unpaired control cannot fix it, and that the reported number itself drives the action.

W4 (scope). The scope is deliberate and matches the evaluations we audit. SimpleQA uses one shared threshold, so models can be compared. We also added a mid-range SimpleQA threshold; inflation is smaller there, but still present.

Q3 (reasoning control). All models already reason before a bare action under their default settings. We also ran a control where GPT-5.5 and DeepSeek write a short explanation with no number, then choose. Inflation remains large.

Dear Reviewers,

We sincerely appreciate your time and effort in reviewing our manuscript "Belief Elicitation Is an Intervention: Measuring and Manufacturing Belief–Action Consistency in Language Models"#59853 and providing valuable suggestions.

As the author-reviewer discussion phase is drawing to a close, we would like to kindly confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues.

If you require further clarification or have any additional questions, please do not hesitate to let us know. We are more than willing to continue our communication with you.

Best regards, The authors of Paper #59853

Thank you for the clarification. I will keep my previous rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.