acceptodds
Under review as a conference paper at ICLR 2027

Reading Is Not Simulating: Calibrating Chain-of-Thought Interventions on an Executable Cognitive Model

Abstract

Chain-of-thought interventions probe answer dependence on written reasoning. Interpreting that dependence as consequence prediction requires a reference for what an edit should change. We build one from an executable cognitive process: an ACT-R-derived driver runs in a closed-loop simulator, and 1.5B–7B language models are fine-tuned to generate its templated state traces before predicting outcomes. Shared-randomness re-executions supply mechanism-specific counterfactuals; fresh seeds at the same specification supply empirical consequence references. At 1.5B, whole-trace replacement reaches 74% answer-direction agreement on outcome-moving pairs, yet donor replacements without the corresponding mechanism intervention yield comparable or higher agreement against their own outcome changes. Edited states persist in 99–100% of evaluated continuation clauses where reference traces disagree. In the matched-boundary assay, all three students fall below the specification-matched majority for lane-consequence accuracy on the hands mechanism; the 7B student also falls below arm-wise modal prediction and re-execution agreement. Separately, hands answer-direction agreement is 79–85% with consequence-supplied or full-trace inputs, versus 7–52% after state-only continuation. Reference decision rules change which shortfalls are supported, and a layout audit exposes termination leakage. In this setting, we empirically distinguish following supplied traces from predicting consequences of edited states, and provide an executable protocol that makes this distinction auditable through explicit information boundaries, decision rules, and evaluation populations.

Primary area:
interpretability and explainable AI
Keywords:
chain-of-thought faithfulness, counterfactual intervention, intervention calibration, executable reference, information-boundary assay, donor control, re-execution baselines, decision rules, interpretability, verbalized reasoning, language models, cognitive architecture, ACT-R, closed-loop simulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GxzeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks what a chain-of-thought intervention should be compared against. Instead of asking whether an edited trace changes the answer, the authors build a reference that says what the edit ought to change: an ACT-R-style driver model runs in a simulator, its internal state is written out as a templated trace, and re-running the same seed with one mechanism altered gives a ground-truth counterfactual. Small Qwen and Llama models (1.5B to 7B) are fine-tuned to produce these traces before predicting the crash-severity outcome.

The main finding is that the students track whatever trace they are given but do not work out what an edited state implies. Replacing the whole trace moves the answer in the right direction most of the time, but so does pasting in a trace from an unrelated run, so the replacement effect is not evidence about the mechanism. On the hands mechanism all three students fall short of a simple majority rule; on the peripheral-vision mechanism the 3B student matches it. Along the way the authors catch their own trace format leaking the run duration and rebuild it. They conclude that reading a trace and simulating from it are different things, and offer the protocol as a way to tell them apart.

Critical strengths
  • The executable reference process is a useful methodological contribution. Shared randomness supports paired mechanism interventions, while fresh-seed re-executions provide empirical predictive references. The paper carefully distinguishes these finite-sample references from performance bounds.

  • The donor controls demonstrate why a large replacement effect alone is insufficient evidence of mechanism-specific consequence prediction. OTHER-CELL DONOR reaches 0.821 / 0.830 compared with REPLACE at 0.735 / 0.784 in the two donor-tested models. Each condition is evaluated against its own outcome change, making the comparison informative about the specificity of effect magnitude.

  • The leakage and selection audits are valuable. The authors identify outcome information in termination metadata, retrain with a revised layout, and show that an apparent conditioning gain disappears under selection matching. The analysis chronology and limitations are reported candidly.

Critical weaknesses
  • The training support for the hands intervention is unclear, limiting the interpretation of its predictive shortfall. From Appendices C and E, hand position appears tied to glance type: a body lean removes steering authority, while the training campaign varies glance type and duration without listing hand position as an independent factor. If the counterfactual combination of body lean with hands on the wheel is absent from training, the hands assay also tests generalization to an unseen state combination. This does not invalidate the observed shortfall, but it leaves unclear how much it reflects that distribution shift rather than a more general limitation in predicting consequences from supplied states. The peripheral intervention may also introduce an unseen combination, so this possibility alone does not explain the asymmetry. Training-set support counts and a controlled intervention-exposure experiment would make the interpretation stronger.

  • The assay’s ability to distinguish consequence prediction from simpler strategies needs further validation. The paper appropriately states that observable prediction accuracy does not diagnose internal computation, and a positive control is not necessary to establish that these students fall below a specified benchmark. However, the title and broader framing suggest a distinction between reading and simulating that the current comparisons establish only indirectly. A predictor that explicitly propagates the supplied state under the reference dynamics, operating under the same information boundary and scoring procedure, would help validate the assay. Alternatively, the framing could more consistently emphasize predictive performance under intervention rather than simulation as a computational capability.

  • The abstract and conclusion could better reflect heterogeneity across mechanisms and reference rules. The introduction and main text do discuss peripheral, but the headline empirical contrast emphasizes hands. On peripheral, the 3B student is above B1* and is not significantly different from B2*, while the 7B student is below B2*. This is neither uniform failure nor demonstrated equivalence to the reference. Similarly, the 3B peripheral continuation has higher answer DA than full-trace input (0.875 versus 0.819), but substantially lower invariance (0.677 versus 0.938), so it should not be described as an overall improvement. A concise statement of this heterogeneity in the abstract and conclusion would better delimit the evidence.

  • The presentation makes the central comparisons difficult to track. The paper introduces several reference rules, two outcome endpoints, and both conditional and nominal denominators, with important qualifications distributed across the main text and appendices. A compact guide connecting each headline claim to its endpoint, evaluation population, reference rule, and robustness result would substantially improve readability.

Questions for authors

Q1. What is the training support for the intervention states?

Could you report the joint distribution of glance type and hand position in the training examples, including whether body lean with hands on the wheel ever occurs? Please provide the analogous support information for the peripheral intervention. If these combinations are absent, how do you distinguish intervention generalization failure from other sources of consequence-prediction error? An experiment varying hand position independently of glance type would be particularly informative.

Q2. Can you validate the assay using a known consequence predictor?

Could a predictor that explicitly propagates the supplied state under the reference dynamics be evaluated using the same information boundary and scoring procedure? Since the boundary omits some seed-dependent information, this need not imply perfect agreement. The useful check is whether a known consequence-prediction procedure behaves as expected relative to the empirical references. This would strengthen the interpretation of the assay beyond documenting the students’ benchmark shortfalls.

Q3. Can the abstract and conclusion state the mechanism-dependent results more explicitly?

The introduction already discusses the peripheral findings. Could the abstract and conclusion also acknowledge that peripheral 3B is above B1* without a statistically supported difference from B2*, whereas peripheral 7B is below B2*? For hands, could the headline statement also preserve the distinction between the all-model shortfall against B2* and the more reference-dependent comparisons against B1* and B∆?

Reviewer_yYYwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I admit I did not understand what this paper was trying to show. They claim they are showing that "Donor replacements without the corresponding mechanism intervention yield direction agreement comparable to whole-trace replacement". I did not think the paper demonstrated this. The methodological choices were questionable and the presentation of the results was haphazard.

Critical strengths

Multiple models are trained. The ACT-R action domain could be valuable.

Critical weaknesses
  • Does not clearly explain how ACT-R works, and does not explain why ACT-R was chosen.
  • It's not clear why driving is the domain that was chosen here
  • The paper would strongly benefit from diagrams explaining the ACT-R setup and the driving setup
  • The paper is very difficult to follow, in my opinion. It relies on equations without first giving motivating examples or a clear explanation
  • It is using an action domain (driving). Would a predictive domain be sufficient? That would be clearer.

The writing quality is poor, and Pangram registers it as 100% AI. This is not a problem in itself, but I believe it's contributing to poor prose and inscrutable presentation.

Questions for authors

Why did you choose ACT-R?

Why did you choose driving as a domain?

Is there a simpler way that you could show what you're trying to show?

Reviewer_Q4ikOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates LLMs’ ability to predict the consequences of state changes using a driving simulator as an executable reference. Models are fine-tuned to generate state traces and predict outcomes, then tested through trace interventions and controlled information boundaries. The results show that preserving edited states or responding to supplied traces does not necessarily translate into accurate consequence prediction. The paper emphasizes appropriate controls and reference baselines when interpreting chain-of-thought interventions.

Critical strengths
  1. The paper explores an interesting topic: the simulation capabilities of LLMs, it's actually testing whether we need the world model or that we can use the exsiting well-trained LLM to serve as the simulation here.
  2. Driving simulation is a mature testbed and a suitable setting for controlled evaluation.
Critical weaknesses
  1. The findings are limited to observable behavior and do not establish whether the models perform internal simulation.
  2. The main consequence-prediction evaluation covers only two interventions: whether the driver’s hand remains on the wheel and whether peripheral vision is available. A broader range of interventions would strengthen the evaluation.
  3. The analysis of generated reasoning chains is limited. More detailed comparisons with simulator trajectories would help clarify where the models successfully predict the process and where they fail.
Questions for authors
  1. How do the authors account for missing information, such as the exact initial vehicle position and the driver’s reaction time, when evaluating trajectory predictions?
  2. Which edited texts undergo overall consistency checks, and how do the authors limit their conclusions for samples that fail these checks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.