acceptodds
Under review as a conference paper at ICLR 2027

Identifying Changes in Causal Attribution under Reasoning Step Replacement

Abstract

Language models can replace written reasoning steps with continuous hidden states. How does this change the contribution of the reasoning that remains visible? We measure how a change in the input affects the answer by intervening on the states stored for the prompt, written steps, and hidden steps. These parts interact, so assigning credit to one depends on how their joint effects are divided. We compare two reasoning modes using the same rule for assigning credit. For rules based on different orderings of the parts, we derive the exact range of possible changes in attribution. Unchanged interactions cancel; only changes in interactions make this comparison ambiguous. We apply the method to GPT-2 models trained on a synthetic policy task, holding each model's weights fixed across modes. Across three adaptation runs, replacing more steps increases the contribution of states at text positions, while accuracy and correct policy use decline. None of the runs supports an increase in the contribution of hidden-step states. In matched comparisons within two models, interventions on text inputs produce smaller attribution changes than interventions on the stored states. The results separate the influence carried by text-associated states from the model's ability to apply the policy correctly.

Primary area:
interpretability and explainable AI
Keywords:
causal attribution, latent reasoning, chain-of-thought, mechanistic interpretability, activation patching, interaction effects

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9fBDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper divides the internal states preceding answer generation into three carriers, i.e., prompt, visible text steps, and continuous hidden steps, performing paired activation-patching interventions with respect to changes in an input factor. Given the interactions among these carriers, this work adopts random-order attribution rules from cooperative game theory, demonstrating that when a common attribution rule is held fixed across two reasoning modes, unchanged interaction terms cancel out. Empirical evaluations on a synthetic policy reasoning task reveal that as more reasoning steps are replaced by hidden states, the contribution of text-position states increases, whereas the model's overall accuracy and its ability to apply the policy correctly decline.

Critical strengths
  1. The paper introduces a cooperative game attribution approach based on random orders and demonstrates the cancellation property of non-drifting interaction terms in paired comparisons via Möbius inversion analysis.
  2. The experiments show that an increase in internal cache attribution does not correspond to correct policy execution by the model; instead, it is accompanied by an elevated spurious response to inapplicable policies and reduced accuracy, providing negative diagnostic evidence for understanding continuous latent reasoning.
  3. By incorporating input recomputation comparisons and subset experiments on shared padding tokens, this work rules out spurious attribution artifacts caused merely by output format degradation.
Critical weaknesses
  1. In the analysis of latent reasoning, recent studies have explored the faithfulness and causal flow of continuous hidden-state reasoning, which shares similarities with this paper. For example, a. CausalFlow: Causal Attribution and Counterfactual Repair for LLM Agent Failures. b. Thought Anchors: Which LLM Reasoning Steps Matter; c. Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs; d. LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning. Furthermore, does this fundamentally act as a form of process reward that assigns values to different rollout components? Please further clarify the core innovations of this paper and its distinctions from prior work.
  2. All experiments are conducted exclusively on the original GPT-2 model with only 124M parameters. The contemporary motivation for internalizing chains of thought is to compress lengthy context compute overheads in large language models with tens of billions of parameters. Small-scale models lack multi-hop symbolic reasoning capabilities, and their accuracy drops significantly when m=3m=3. Are the phenomena observed on such severely degraded, extremely small models a limitation of the continuous reasoning paradigm itself, or simply a consequence of GPT-2's insufficient capacity to sustain stable reasoning?
  3. The synthetic rule world used in the experiments contains only two entities, two three-hop membership chains, and three policies. This setup is highly artificial with an extremely constrained state space, failing to represent real-world scenarios. Moreover, combining unpaired key-value pairs from different runs in the self-attention mechanism inevitably introduces out-of-distribution feature perturbations. Although the authors provide an initial check by inspecting the probability mass of legal action labels, an analysis of intermediate and hidden representations is lacking, making it impossible to exclude artifacts induced by the hybrid caches.
  4. The experiments reveal that identical padding tokens account for a substantial positive attribution change at the cache level. The text attributes this to padding tokens absorbing preceding contextual information; however, in causal interventions, this could just as well arise as a positional encoding artifact introduced by the hard-coded fixed slot width. Comparative ablations with varying slot lengths or dynamic masking without padding should be supplemented.
  5. When comparing different modes, the number of continuous latent steps and the physical length of the text steps change simultaneously. This means that the total computational step budget is confounded with the text slot allocation, making it difficult to decouple the independent effects of these distinct factors.
Questions for authors

Please see the above comments.

Reviewer_UwyYOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores how a change in input can affect the answers of a model, in this case, GPT-2, which was fine-tuned to produce step-by-step reasoning with some steps replaced by continuous hidden states (i.e., the coconut (chain-of-continuous-thoughts) curriculum). The authors do so by intervening on states related to prompt, written steps, and hidden steps. When the same model replaces more of its steps with continuous hidden states, they find that the contributions of the text-position states increase while the accuracy and correct policy use decline.

Critical strengths

S1. The problem setup is interesting because continuous hidden steps are not really interpretable at the moment. Hence, having a setup that measures how much the prompt, written steps, and hidden steps each contribute to the model’s response is useful.

Critical weaknesses

W1. The paper is relatively hard to understand, particularly section 3 and the results section. See Q1.

W2. The experiment setup doesn’t seem to test what it’s meant to evaluate. The only reasoning step that depends on the policy flip is written out in both modes, while the hidden steps (which only cover membership hops) are not really affected by the flip. Based on this understanding, I believe that the lack of increase in hidden step contribution is because of the experimental setup rather than the property of the model.

W3. The experiments are only tested on GPT2, which limits the evaluation result claims.

Questions for authors

Q1. In section 3, what does proposition 1 really answer and how does it connect to the experiments? Can you provide a better explanation of all the different variables in this section and what each is really for? This section was really dense and hard to read.

Also consider W2 and W3 as questions of clarification and additional requests for additional evaluation.

Reviewer_D2ErOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper investigates the problem of causal attribution under reasoning step replacement. Specifically, the paper presents a method for measuring the causal effect of interventions on the prompt, written steps as well as the hidden states of the model. The empirical evaluation is carried out using a GPT2 model that is fine-tuned on a synthetic dataset. The results show that certain interventions lead to a decrease in performance while other interventions do not seem to have a major impact.

Critical strengths
  • The paper looks at causal attribution of interventions in the reasoning chain of a model which has recently received increased attention. Providing new insights into the reasoning processes of language models is important to understand how they reach a certain decision or answer.
Critical weaknesses
  • Quality of the presentation. In my opinion the paper looks heavily AI-assisted, and the content is not easy to follow. There are quite a few notions introduced throughout the paper that don't really mean anything. For example there's no need to define a "coalition". A is just a subset.

  • I think when discussing causal effects one needs to refer to a (structural) causal model. The paper does not define one. Therefore, it is hard to interpret the quantities defined by the paper. For example, eq (3) looks like a total effect, but it is not clear what it means.

  • The empirical evaluation is fairly limited. Studying just one model (GPT2) is simply not enough. Furthermore, I'm not even sure that GPT2 is a reasoning model.

Questions for authors

No questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.