acceptodds
Withdrawn from ICLR 2027

Therefore I am. I Think.

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

We consider the question: when a large language reasoning model makes a choice, did it think first and then decide to, or decide first and then think? In this paper, we present evidence that detectable, early-encoded decisions shape chain-of-thought in reasoning models. Specifically, we show that a simple linear probe successfully decodes tool-calling decisions from pre-generation activations with very high confidence, and in some cases, even before a single reasoning token is produced. Activation steering supports this causally: perturbing the decision direction leads to inflated deliberation, and flips behavior in many examples (between 7 - 79% depending on model and benchmark). We also show through behavioral analysis that, when steering changes the decision, the chain-of-thought process often rationalizes the flip rather than resisting it. Together, these results suggest that reasoning models can encode action choices before they begin to deliberate in text.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Probing, Tool-calling, LLMs, Interpretability, Efficient Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LJcPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the question whether choices are made in the computations that produce the reasoning tokens or before that.

Critical strengths

Paper aims to study an interesting and timely question. The implication of an answer to this question can impact both the development of new methods (e.g early exits) and impact trust calibration (if models dont rely on the reasoning tokens this is an additional evidence that we should not rely on the reasoning text as an explanation).

The paper is written clearly, easy to follow and present its results clearly.

Critical weaknesses

It is not clear if how the results in the paper would extend to other models, what puts in doubt the way the paper draw conclusions. Results on other models could be impacted by a number of factors, and mostly by their training regime. The amount of RL training and the constraints that were put during that training might have a crucial impact on such results. If models were RL trained for small portion out of their entire training they might exhibit less reliance on the process of producing reasoning tokens. Also, if models were trained to support several reasoning "efforts" they might tend to reach a conclusion earlier. An investigation into those factors would be very beneficial to this paper.

A comparison between models that would verify that this phenomena is consistent across more models of different sizes, companies etc. would benefit this paper and support the claims the authors drew.

Without such further experiments the claims in the paper could be interpreted in a way that will make them overclaiming in some cases. In order to mitigate that Id suggest the follwoing change:

  • All 3 conclusions in the last paragraph in the intro should be hedged to communicate that those are things that can happend in LLMs, rather then present those as typical behavior of LLMs.
  • The abstract should communicate the scope of the experiments and either specify the models that were tested or at least that 3 small models were tested.
Questions for authors

The examples in the main body of the paper are not all needed and mostly can go into the appendix.

Reviewer_8xqDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, they see that does the LLMs make their decisions before the CoT, or that the decision is made based on the CoT. They do this with tool-calling decisions with when2call and bfcl benchmarks on two models. First, they do this with training a linear probe on hidden states at different steps to find that tool-calling is in a way that makes the decision before the CoT or that the CoT changes it to make that decision. Also, they use steering to change the decision and also see that how the CoT changes based on these decisions. Then they use LLM judges to classify reasoning behaviours after steering. The results show that tool-calling decisions can be predicted before generation, and that steering can change the decision, and then also the CoT changes to support the change in decision, and it is kind of acting as a post-hoc justification.

Critical strengths

The way that they do not only see that can the final action be predicted from hidden states, but they also see that in a way that do these representations have a causal effect on the model's behavior, and they do that with combining linear probing and steering, is solid. In 4 and figure 2, they show that the tool-calling decision can be predicted before that the model starts to generate, and it is a solid results (not convincing). Also, in 4, besides the flip rates, they also see how the content of CoT changes after steering. Using reversed presentation order to make this analysis more reliable is also solid.

Critical weaknesses

It is not clear that they make a causal relation between that the decision-making process happens early and the reasoning trajectory. In 4, they show that linear probes can predict the tool-calling decision with high AUROC before any reasoning, but this can not mean that the model has made the decision at that point and instead ​​it can be in a way that the information in the query and the tool definitions is enough to make the tool-calling decision. So, it means that pre-generation activations are just encoding these task-related features instead of a decision.

As mentioned in 3.4, the steering vector is from pre-generation activations, but it is used for pre-fill and decoding. It is not clear that the final action changes because that the decision that has been made early was changed, or because the intervention affects the model during the CoT.

The control experiment with a random vector (unrelated) can not separate the effect of the pre-generation intervention, and because the question is that do models make decisions before CoT, but this can not make any difference between a prediction that is early and a decision that has been made early.

In 4, they show that steering can change the behavior (with llm judges for classifying), but it is should not mean that these are post-hoc because steering can change both the trajectory and the final action, and these two can change the model to make a new decision instead of that, it generates a CoT to be aligned with the changed decision that has been made. So llm judges can help with finding differences in behavior, but they cannot make the causal order between decision and trajectory. On a side note, the behavioural categories of judges can overlap because they are informed about which response is steered.

As mentioned in 2, previous works have explored that hidden states can predict model behavior, how steering can change the model outputs, and also does CoT faithfully shows the process of the model, and they just try to kind of combine these approaches and apply them to tool-calling decisions. This is a thin novelty, while tool-calling for analyzing early decisions is solid, but it is not clear what does this new understanding of reasoning adds beyond previous works.

On a side note, you should not include the acknowledgement in the double-blind review process.

Questions for authors

In 3.2 and 4, can you compare the linear probe with a baseline that predicts the final action with the query and tool definition to have a sort of control experiment that can help to make a difference between predicting the final action and the case that the model made a decision before CoT?

In 3.4, can you explain that why the steering vector is used for pre-fill and decoding, but your hypothesis is that decisions are made before?

Can you explain how the behavioral analysis in 4 makes a difference between post-hoc and reasoning that is changed by steering? How can you know that steering is changing the trajectory or the decision?

Reviewer_VdxpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether reasoning models encode tool/no-tool decisions before generating visible chain-of-thought, and whether subsequent reasoning rationalizes those decisions. It trains linear probes on hidden states and constructs mean-difference steering vectors to influence tool use. Experiments primarily use Qwen3-4B and GLM-Z1-9B on When2Call and a dataset constructed from BFCL, with supplementary probing results on GPT-OSS-20B. The authors report strong pre-generation predictability, action changes under steering, and frequently increased reasoning length. Two LLM judges categorize changes in the generated reasoning to support the interpretation of rationalization.

Critical strengths

The paper provides evidence that eventual tool/no-tool actions are predictable from pre-generation activations in the evaluated models. Cross-validated probing across two benchmarks, combined with interventions that change tool-use behaviour, gives this observation empirical support. This could be useful for understanding tool-use behaviour, although the broader interpretation remains insufficiently established.

Critical weaknesses
  1. The scope and resulting contribution are limited.

The experiments concern one binary behavior - whether to call a tool - with causal evaluation restricted to two small dense models. The paper applies established probing and steering methods but provides limited insight beyond showing that another behavior is predictable and steerable. My concern is the combination of narrow scope, insufficiently motivated experimental choices, and limited new insight. A focused study using simple methods can make a substantial contribution, but this requires deeper understanding of the behavior or its mechanism than the paper currently provides. The broader framing about deciding before thinking exceeds what the experiments demonstrate.

  1. Predictability before reasoning does not establish that a decision has already been made.

The probes show that pre-generation activations contain information predictive of the eventual tool-use action. This could reflect an initial tendency or information about the request and available tools. It does not establish that the decision is settled or that subsequent reasoning merely justifies it. The paper repeatedly moves from evidence of predictability to claims about early decisions without establishing this distinction.

  1. The steering results admit other explanations, some of which may better explain the observations.

The paper interprets the results as: Steering changes an already-made tool-use decision => reasoning changes to justify that decision. However, other explanations are possible. For example: Steering makes the model more or less critical of the request => its reasoning changes => its tool-use decision changes.

This example may explain the longer reasoning and greater steerability in thinking mode more naturally. The broader issue is that the paper does not establish what the steering direction represents or why the authors' interpretation should be preferred over other plausible, potentially more likely explanations.

  1. Steering during decoding prevents attribution of the effect to the early state.

Section 3.4 states that steering is applied during both prefill and decoding. Consequently, the observed changes cannot be attributed specifically to a decision encoded before reasoning begins.

  1. The behavioral analysis provides limited insight into the research question.

The motivation for the six categories is unclear, and the categories overlap. Agreement between judges supports consistency of categorization, but does not establish that the reasoning rationalizes an already-made decision.

The analysis accompanying Tables 3-4 in Section 4 mainly describes numbers without providing insight into the research question. As a result, it adds little evidence for the paper's central interpretation.

Questions for authors

These questions concern the validity and interpretation of the findings. Addressing them alone would not resolve my main concern about scope and significance. Broader evidence across behaviors or a substantially wider range of recent models, and/or deeper insight into the latent mechanism behind tool-use decisions, would be needed to address that concern.

  1. How do you rule out the alternative causal explanation described in Weakness 3 ?

  2. Given that steering is applied during both prefill and decoding, what evidence attributes the effect to the pre-generation state ? Please distinguish the effects of steering only during prefill, only during decoding, and during both.

  3. Why does activation-probe predictability indicate an early decision ? Information such as available tools or query-tool compatibility may already be sufficient to predict whether the model will call a tool. How do you distinguish a representation of this information from a decision the model has made ?

  4. How were the six behavioral categories developed, and how does this categorization help answer the research question ?

  5. What is the motivation for comparing predictions with the think_end probe in Figure 3 ?

Suggestions

  • Explain how the layers for extracting and applying the steering vector were selected.
  • Explain what specifically about GPT-OSS's MoE architecture requires a different steering technique and motivates its exclusion from causal evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.