- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper takes aim at Measurement Non-Reactivity—the assumption that eliciting an LLM's belief merely observes, rather than changes, a subsequent action—and argues that same-session belief–action consistency can be inflated as a result. It separates same-session consistency from cross-context predictive validity, proves that same-session data cannot identify the latter, and uses a paired "Shadow Fork" treatment/control protocol to identify both quantities and their difference. Across several frontier models and tasks, the experiments find substantial inflation under standard protocols. Replaying a model's report largely reproduces its action, while changing only the injected report can sharply change the action—evidence that the report can act as an input rather than a passive readout.
Critical strengths
- The paper identifies a broadly relevant and underappreciated validity problem in evaluating whether language models act on their stated beliefs.
- It offers a strong combination of conceptual clarification, formal identification analysis, and causal experimentation.
Critical weaknesses
- The empirical scope is limited to narrow decision formats and tasks, so the conclusions may not transfer to richer agentic or free-form settings.
- The causal interpretation depends on session-equivalence and control assumptions that are difficult to fully validate for closed model endpoints.
- The proposed protocol may require more experimental effort than many benchmark evaluations can readily support.
- Section 7 limits the claims to a numerical report scored against a threshold-based binary decision in two tasks. In SimpleQA, the tested thresholds are concentrated in the upper tail in Section 6. The paper does not establish that the same effect holds for free-text rationales, richer beliefs, or multi-step agents.
Questions for authors
- Could the paper include a mid-range SimpleQA threshold pilot, rather than only the upper-tail threshold range, to test the scope of the effect?
- What stronger test could exclude stable caching by the gateway provider, and has any such behavior been observed?
2 comments
Dear Reviewers,
We sincerely appreciate your time and effort in reviewing our manuscript "Belief Elicitation Is an Intervention: Measuring and Manufacturing Belief–Action Consistency in Language Models"#59853 and providing valuable suggestions.
As the author-reviewer discussion phase is drawing to a close, we would like to kindly confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues.
If you require further clarification or have any additional questions, please do not hesitate to let us know. We are more than willing to continue our communication with you.
Best regards, The authors of Paper #59853
We thank the reviewer for the positive assessment.
W1 / W4 (scope). The scope is deliberate: it is the configuration used by the evaluations we audit (Appendix A), and the one in which identification can be tested exactly. Free-text and agentic settings are beyond this paper, as Section 7 states.
W2 / Q2 (caching). Every call is an independent stateless request, memory features were disabled, and calls were issued in a seeded random order (Appendix G). Identical prompts sent 120 seconds apart show no batch effect (all p ≥ 0.38, Appendix C), and identical calls return different outputs. Among 5,760 within-cell replicate pairs for DeepSeek and Qwen, only 1 has an identical reasoning length. Caching could only make the two arms more alike, which would shrink I rather than inflate it.
Q1 (mid-range threshold). We added a penalty of 9.5 (τ ≈ 90.48), using 9.5 rather than 9 to avoid ties. On the four live models, pooled I = +0.084 [+0.062, +0.109], compared with +0.155 at λ = 20 and +0.145 at λ = 50 on the same models (Appendix I).
W3 (cost). One extra call per scenario.