acceptodds
Under review as a conference paper at ICLR 2027

Thinking Leakage: A Causal Audit of NoThink Post-Training in Hybrid Reasoning Models

Abstract

Post-training hybrid reasoning models in NoThink mode has attracted growing interest as a way to improve performance while keeping inference fast. However, these gains may draw on thinking behavior already accessible through the base model's Think mode. We formulate this thinking leakage in a causal mediation framework and audit its contribution using bidirectional interventions along a simple base-derived activation direction. Across three models and three post-training methods on competition math benchmarks, we find that leakage is real, causal, and substantial: behavioral and representational analyses reveal shifts toward Think, steering the base model along this direction reproduces most of the post-training accuracy gain, and counter-steering a checkpoint removes a substantial share of what it gains. Across nine aligned checkpoints with positive NoThink gains, the resulting leakage ratio ranges from 42% to 79%. These interventions support a substantial causal contribution of thinking leakage. Our findings show that a post-training method's apparent advantage can therefore reflect greater drift toward Think, obscuring whether it improves capability within NoThink or more effectively re-invokes existing Think behavior.

Primary area:
interpretability and explainable AI
Keywords:
thinking leakage, hybrid reasoning models, causal mediation, activation steering, post-training evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FYFBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether improvements from NoThink post-training partly arise from reactivating thinking behavior already available in the base model. It constructs a Think–NoThink activation direction and uses bidirectional interventions to measure its contribution to performance gains across several models and training methods.

Critical strengths
  • Relevant question. The paper examines an important ambiguity in evaluating hybrid reasoning models: improved NoThink performance may reflect greater use of existing thinking behavior. Distinguishing these explanations can help interpret post-training gains.

  • Bidirectional evidence. The combination of representational analysis, steering in the base model, and counter-steering in trained checkpoints provides stronger evidence than behavioral observations alone. Evaluating multiple models and training methods also broadens the empirical support.

Critical weaknesses
  • Unclear specificity. The Think–NoThink direction is constructed from averages over generated tokens and may capture differences in length, token position, and response style alongside deliberation. The clean-answer reference and random-direction controls do not fully separate these factors. Consequently, performance changes under intervention do not uniquely establish how much improvement is caused by thinking leakage.
  • Limited methodological novelty. CoT Vectors [1] already extracts reasoning vectors by contrasting activations with and without CoT. The paper should clarify what its Think–NoThink direction and subsequent interventions reveal beyond this existing approach.
  • Practical implications need clarification. Reactivating existing thinking ability can still be a useful outcome of NoThink post-training, particularly if it improves accuracy while reducing inference cost. Although Appendix B.1 acknowledges this possibility, the paper could explain more concretely how its leakage estimates should inform the evaluation of such gains.

[1] CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs

Questions for authors

Do the main findings hold when the direction is estimated from length-matched, position-matched responses, or from the thinking region alone?

What additional insight does the proposed audit provide over existing CoT-versus-direct-answer steering methods? Would those directions produce similar estimates of the contribution to performance gains?

Given that reusing existing thinking ability can still improve reasoning efficiency, how should the leakage estimate affect our assessment of a post-training method’s value?

5 comments
AuthorsOpenReview ↗

Response to Reviewer FYFB: Part 1

We sincerely thank the reviewer for the constructive and insightful comments. Below we address the comments point by point.

Response to Weakness 1 and Question 1: We thank the reviewer for this question. We agree that vleakv_{\text{leak}} is a composite: as a difference of means over complete rollouts, it bundles deliberation with what accompanies it. To test whether the effect reduces to any one of these factors, we added control directions, each built to isolate one factor, and applied both interventions along each on Qwen3-8B (new Appendix H, changes are highlighted in the revised PDF).

directioncos with vleakv_{\text{leak}}steering base: acc. (base 0.178)counter-steering GRPO@500: acc. (untreated 0.472)
vleakv_{\text{leak}}10.3350.349
think-head (thinking region, first 256 tokens)0.760.4600.362
think-end (thinking region, last 256 tokens)0.870.2960.379
length (long − short NOTHINK responses)0.330.1790.500
marker (windows with − without marker words)−0.0030.1670.469

Thinking region alone, position- and length-matched. Think-head and think-end are estimated from the thinking region only, contrasting the first (or last) 256 tokens of the thinking block with the first (or last) 256 tokens of the NOTHINK response. This matches position and the number of tokens averaged. Both directions reproduce the gain when added to the base model and remove it when subtracted from the checkpoint, so the main findings hold. Interestingly, think-head gives the strongest steering gain (0.460 against 0.335 for vleakv_{\text{leak}}) but no stronger counter-steering loss (0.362 against 0.349). The signal that best induces thinking in the base model thus sits near the start of deliberation, while what post-training installs in the checkpoint is not concentrated there; the drift aligns less with think-head than with vleakv_{\text{leak}} (cosine 0.75 against 0.95). How leakage is distributed across the stages of deliberation is an interesting question we leave for future work.

Not length or style. A length direction leaves accuracy unchanged in both interventions. Its counter-steering value of 0.500 is noise rather than a gain: the bootstrap interval includes zero, marker density and response length are unchanged. After checking the rollouts, we found it is concentrated in a few HMMT-Feb problems while AIME25 is unchanged, a pattern more consistent with sampling variation than with a change in reasoning. A direction targeting marker words, nearly orthogonal to vleakv_{\text{leak}}, has no effect either. Besides, the clean-answer reference axis, anchored on the answer a THINK rollout writes after deliberation is complete, shows almost no drift (figure 8). The drift that post-training induces also lies mostly along the thinking-derived directions (cosine 0.61–0.95 with the drift), against 0.34 for length and −0.06 for marker.

We do not claim vleakv_{\text{leak}} isolates pure deliberation. The effect follows the THINK–NOTHINK contrast and does not reduce to length, position, or style on its own. These controls cover Qwen3-8B only, and we will note this in Appendix H.

AuthorsOpenReview ↗

Response to Reviewer FYFB: Part 2

Response to Weakness 2 and Question 2: We thank the reviewer for the pointer; we have added CoT Vectors to Sec. 7 Related Work.

CoT Vectors measures a different contrast. Its vector is the difference in answer-segment activations with and without a CoT prefix, applied to strengthen reasoning at low cost. This corresponds to our reference vector (Appendices D.1, E.1, F.2), which contrasts NOTHINK with the answer THINK writes after deliberation. vleakv_{\text{leak}} instead contrasts the full THINK rollout, deliberation included, with NOTHINK, i.e., the two modes themselves. The two behave differently in our experiments: post-training barely moves NOTHINK along the reference direction, and steering along it does not raise accuracy. It is also worth noting that CoT Vectors reports gains of +4.6 and +1.9 points on two non-hybrid models even with learnable vectors, whereas a plain difference-of-means vector raises NOTHINK accuracy by up to 27 points on our hybrid models (Sec. 4). This is consistent with our central point: in hybrid models, existing reasoning is unusually easy to re-invoke in NOTHINK, which is what makes leakage a concern.

Our contribution is an audit, not a steering method. Steering is a widely used technique, from controlling model preferences to adjusting reasoning speed or backtracking (Lin et al., 2025; Venhoff et al., 2025; Sinii et al., 2025; the first paragraph of Sec. 7). In this line of work, including CoT Vectors, the direction is the product: a vector injected to enhance or control behavior. We use it as an instrument for a different question: for a checkpoint that reports a NOTHINK gain, how much of that gain depends on the checkpoint having moved along this direction. As the reviewer noted among the strengths, our contribution is this question and the causal framework that answers it with bidirectional interventions. To our knowledge, this is the first representation-level audit of leakage in hybrid models.

  • What is audited. Steering alone, including Sinii et al.'s result that a vector can match RL, shows that a gain is reachable without weight updates. It does not show how much of a specific checkpoint's gain actually runs through that direction. We measure how far post-training moved each checkpoint along it (Sec. 3), remove that movement (Sec. 5), and decompose the reported gain (Sec. 6), checking the reading against steering the base model (Sec. 4).
  • Where the direction comes from. Prior directions rely on engineered extraction: CoT-prompted inputs, length-matched stimuli, annotated token spans, or optimized vectors. Hybrid models supply both modes from the same weights under a template switch, so a plain difference of means suffices, and it is stable at mid-depth, where CoT Vectors reports extracted vectors to be unstable.

Would CoT-versus-direct directions give similar estimates? Our new controls (Appendix H) suggest yes, provided they align with the THINK–NOTHINK contrast. Directions estimated from different parts of THINK rollouts reproduce and remove the gain much as vleakv_{\text{leak}} does, while length, marker, and random directions do neither (Appendix H; response to Weakness 1 and Question 1). The estimate depends on alignment with the mode contrast, not on the exact extraction recipe.

AuthorsOpenReview ↗

Response to Reviewer FYFB: Part 3

Response to Weakness 3 and Question 3: We thank the reviewer and agree that re-invoking existing reasoning can be a genuine benefit, particularly when it lowers inference cost (Appendix B.1). The leakage estimate does not diminish a gain; it tells us which claim the gain supports and what it should be compared against. The decomposition in Sec. 6 splits a gain into two parts:

  • The leakage part is reasoning the base model already has, now produced in NOTHINK. Its value is efficiency, so the right baseline is base THINK, not base NOTHINK (the λ\lambda in Sec. 6). For example, Qwen3-8B GRPO@500 in NOTHINK keeps about 85% of base THINK's accuracy at about half the tokens. That said, part of this saving may come from the training setup rather than the method: under RL with a fixed generation budget, rollouts that exceed it receive no reward, which implicitly favors shorter outputs.
  • The residual part (R0R_0) is what remains once the drift is removed. Under our audit, it is the part that supports a claim of improved direct-answering capability.

This suggests three practical uses:

  1. Reporting NOTHINK gains. Report response length and the leakage ratio alongside accuracy. Since, under our audit, leakage is linear in drift with one coefficient per model (Sec. 6), the drift alone gives a cheap estimate once that coefficient is known. A NOTHINK gain that doubles response length also changes the cost users expect when they choose NOTHINK. To support such audits, we will release all checkpoints and rollouts.
  2. Comparing methods beyond accuracy. Two methods with the same NOTHINK accuracy are not equivalent if their leakage differs. At matched accuracy, SFT drifts about twice as far as GRPO (Appendices D.4, G.3), so more of GRPO's gain lies outside leakage.
  3. Claiming a higher ceiling. A method that only re-invokes existing reasoning will not exceed base THINK across inference budgets; one that raises the attainable level will. Comparing against base THINK across budgets separates the two.

We have expanded Appendix B.1 with these points.

We appreciate your careful reading and thank you for your patience with this long response. If any point remains unclear, please let us know and we will gladly elaborate.

Reviewer_wqcmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether NOTHINK post-training gains depend on thinking behavior already accessible through the base model’s THINK mode. It defines a base-derived leakage axis from THINK–NOTHINK activation means, measures checkpoint drift and reasoning-marker density, and applies steering and counter-steering during generation. Across three hybrid reasoning models and GRPO, SFT, and OPSD on competition math problems, it reports leakage ratios of 42%–79% for nine positive-gain aligned checkpoints. The proposed decomposition separates the observed gain into a leakage-dependent component and a residual component.

Critical strengths

1. A concrete framework for auditing NOTHINK post-training gains. In Sections 2 and 3, the paper operationalizes thinking leakage through a base-derived leakage axis constructed from the native THINK–NOTHINK contrast. It connects projected engagement and checkpoint drift with reasoning-marker density, providing complementary representational and behavioral measurements. This framework makes the source of reported gains an explicit part of the assessment of NOTHINK post-training methods.

2. Substantive evidence from bidirectional interventions. In Sections 4 and 5, steering the base model reproduces much of the post-training accuracy gain, while counter-steering post-trained checkpoints removes a substantial share of their improvement. The dose-response analyses, reverse-drift cases, and reference-axis controls further characterize these effects. Together, these experiments provide substantive evidence that displacement along the audited leakage direction affects NOTHINK performance and contributes to the observed post-training gains.

Critical weaknesses

1. Restoration of mean projected engagement needs empirical verification. In Section 5, counter-steering at γ=1\gamma = 1 is described as returning mean projected engagement to the base NOTHINK level “by construction.” Equation 10 applies a constant projection shift, but counter-steering also changes the generated tokens and thus the activations at subsequent steps. Appendix F.8 shows that generated tokens themselves carry engagement changes. The paper should therefore report the achieved mean projected engagement before treating the counter-steered outcome as Y1,0Y_{1,0}. Without this check, the causal interpretation of the leakage ratio in Section 6 rests on an unverified restoration assumption.

2. Thinking leakage is not fully separated from intervention-induced disruption. The controls in Section 5 support the relevance of the leakage axis to accuracy. However, restricting the analysis to normally terminating rollouts addresses degenerate generation without ruling out other forms of intervention-induced disruption. Some reference and random interventions, particularly on the Qwen models, also receive negligible doses, so they provide little evidence about the effects of comparably sized perturbations. These controls leave uncertainty about how much of the accuracy loss is specific to thinking leakage. This matters because Section 6 uses that loss to quantify the leakage-dependent component and calculate the leakage ratio.

Questions for authors
  1. At γ=1\gamma = 1, does mean projected engagement measured over the counter-steered rollouts actually return to the base NOTHINK level? Please report the achieved values across checkpoints and clarify how any remaining deviation affects the identification of the counter-steered outcome with Y1,0Y_{1,0} and the leakage ratio in Section 6.

  2. What evidence distinguishes the removal of thinking leakage from other forms of intervention-induced disruption? In particular, do the reference and random interventions include doses comparable to counter-steering on the Qwen models? If so, how do their accuracy effects compare when the analysis is restricted to normally terminating rollouts?

6 comments
AuthorsOpenReview ↗

Response to Reviewer wqcm:

We sincerely thank the reviewer for the constructive and insightful comments. Below we address the comments point by point.

Response to Weakness 1 and Question 1: We thank the reviewer; "by construction" overstated this. We revised Sec.5, 6, and added two subsections in the appendix (F.9, G.4) to address this (changes are highlighted in the revised PDF).

Does engagement return to the base NOTHINK level? Not exactly. At the intervened layer the achieved projection decomposes as πtot(20)=πbeh(20)+πimp\pi^{(20)}_ {\text{tot}} = \pi^{(20)}_ {\text{beh}} + \pi_ {\text{imp}}, where πimp\pi_ {\text{imp}} is the applied shift and πbeh(20)\pi^{(20)}_ {\text{beh}} is the shift carried by the generated text, measured by re-running it without intervention. Since πbeh(20)\pi^{(20)}_ {\text{beh}} points the same way, the achieved engagement overshoots base NOTHINK, by 6–21% of ∥vleak∥\|v_ {\text{leak}}\| on the nine aligned checkpoints. At deeper layers, self-repair undoes part of πimp\pi_ {\text{imp}} (Appendix F.8). Values for all fifteen checkpoints:

modelcheckpointπ1−π0\pi_1-\pi_0Δπ\Delta\piπbeh−πbare∥vleak∥\frac{\pi_{\text{beh}}-\pi_{\text{bare}}}{\|v_{\text{leak}}\|}
Qwen3-8BGRPO@500+9.96+64.6%-15.8%
Qwen3-8BGRPO@200+8.78+56.9%-16.6%
Qwen3-8BSFT@50+13.88+90.0%-9.8%
Qwen3-8BOPSD@25+11.78+76.4%-21.4%
Qwen3-8BOPSD@100+7.31+47.4%-14.3%
Qwen3-4BGRPO@450+5.19+60.2%-15.3%
Qwen3-4BGRPO@200+4.69+54.3%-16.6%
Qwen3-4BSFT@75+7.91+91.6%-11.4%
Qwen3-4BOPSD@50+3.13+36.3%-9.4%
Qwen3-4BOPSD@25+2.41+27.9%-15.3%
MiniCPM4.1-8BGRPO@500+4.95+52.4%-9.1%
MiniCPM4.1-8BGRPO@200+3.68+38.9%-5.9%
MiniCPM4.1-8BSFT@100-1.21-12.8%+11.0%
MiniCPM4.1-8BOPSD-LoRA@75+0.01+0.1%+4.7%
MiniCPM4.1-8BOPSD@50-1.32-14.0%+3.4%

How this affects Y1,0Y_{1,0} and the leakage ratio. The counter-steered outcome is therefore an approximate rather than exact realization of Y(θ1,π0)Y(\theta_1, \pi_0). We have revised Sec. 5 to describe the intervention as an applied displacement equal to the drift rather than an exact restoration, and added the table above to Appendix F.9. Appendix G.4 (new) reports, for reference, λ/(1+o)\lambda/(1+o), where oo is the overshoot as a fraction of the drift; this gives 0.34–0.62 against the reported 0.42–0.79. Since self-repair undoes much of the extra shift before it reaches the output, we keep the reported λ\lambda in the main text, where it is defined by the applied displacement on both decomposition paths of Sec. 6; the conclusions are the same on either reading. The end of Sec. 6 now states the rescaled values with a pointer to Appendix G.4, which gives the full justification.

Response to Weakness 2 and Question 2: We thank the reviewer and agree that the original controls received small doses on the Qwen models.

Dose-matched controls. We have now added them (new Appendix H). On Qwen3-8B GRPO@500, a random direction pushed exactly as far as the vleakv_{\text{leak}} counter-steering shift (∥Δh∥=9.96\|\Delta h\| = 9.96) leaves accuracy at 0.475 against 0.472 untreated, and marker density and length unchanged; counter-steering along vleakv_{\text{leak}} at the same dose lowers accuracy to 0.349. Directions targeting marker words or response length also leave the checkpoint unchanged, whereas every direction derived from THINK rollouts lowers accuracy. A generic perturbation of comparable size therefore does not reproduce the loss. Restricted to normally terminating rollouts, the gap remains: 0.490 under the random direction against 0.369 under vleakv_{\text{leak}}.

Further evidence beyond the controls.

  • Disruption can only lower accuracy, but the same operator raises it on the two reverse-drift checkpoints, where training moved NOTHINK away from THINK and counter-steering therefore adds vleakv_{\text{leak}} back (e.g., SFT-MiniCPM4.1-8B@100: 0.162 → 0.179 → 0.201, table 10).
  • The second decomposition path in Sec. 6 uses LbaseL_{\text{base}}, the accuracy gained by steering the base model, which disruption cannot inflate. It agrees with LtrainedL_{\text{trained}} to within 2.6 points on average.

We appreciate your careful reading and thank you for your patience with this response. If any point remains unclear, please let us know and we will gladly elaborate.

Dear Reviewer wqcm,

Thank you again for your time and thoughtful feedback on our submission. As the author-reviewer discussion period ends in less than 48 hours, we wanted to kindly check whether our response has addressed your concerns.

If anything remains unclear, we would be glad to respond while the discussion is still open.

Best regards, The Authors

Thanks for the additional experiments. The random control at the same dose helps address my concern about disruption. I'm still unsure how much the engagement overshoot affects the leakage estimate. Could you adjust the dose on one or two representative checkpoints so that measured engagement matches the base level, then report the accuracy and leakage estimate there? This would help check the linear correction. For now, I’ll keep my rating.

Reviewer_XHT4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper defines thinking leakage, measured as the movement of NOTHINK activations/representations along the THINK-NOTHINK direction of the base model. The paper establishes that (1) the gap between the THINK and NOTHINK activations decreases after post-training, (2) that the NOTHINK representation moves closer to the THINK representation. They also run experiments that show that steering the base model in the THINK direction, without weight updates, can recover most of the gain from post-training. Similarly, counter-steering the post-trained model removes a substantial share of what it gains.

Critical strengths
  • The paper studies an interesting hypothesis -- NOTHINK post-training shifts the NOTHINK representations and behaviors towards THINK.
Critical weaknesses
  • The main leakage metric measures the projection of the post-trained NOTHINK representation moves along the THINK-NOTHINK direction of the base model. Essentially, it measures if post-training moved the NOTHINK representation towards the original THINK representation. However, the THINK representation also changes during post-training. It would be more convincing if the metric instead measured if the THINK-NOTHINK gap is smaller after post-training compared to before post-training.

  • The paper observes that the think and no-think gap reduces after post-training, and interprets this as, specifically, the original nothink representation moving towards the think representation. However, it is also possible that this is a consequence of general post-training and distribution sharpening, similar to entropy collapse, and not unique to thinking behaviors.

  • Sections 4 and 5 establish that the corresponding steering vectors can induce the gain (when steering nothink towards think of the base model) and loss (when steering think towards original nothink for the post-trained models) observed w/ and without post-training. However, an overall improvement in accuracy does not establish that the steering induced the same representation change.

  • Confusing notation and descriptions: At several places, I was confused by the description. Section 2.1 mentions "engagement" and uses it to describe their causal structure, but the term is never defined. Section 3 mentions "reasoning markers" but leaves it to the reader to sus out what those mean. I finally found the markers in Appendix C, page 20.

Often times, the metrics are under-explained. For e.g. for figure 2, drift-alignment measures whether NOTHINK moves in the direction of the original THINK. It only measures directionality. The second metric, projection drift, measures quantifies the shift. This is very unclear from the text.

It will significantly improve the paper if the writing is improved.

Questions for authors
  1. What is the reason for running all analysis on layer=20? How sensitive are the conclusions to this choice?
  2. The evaluation set is only 120 examples. What is the variance / confidence intervals for different training runs?
  3. The increase in reasoning markers are VERY minimal and likely not significant (was significance testing performced?) Figure 2c shows that the # of reasoning markers per 1000 tokens increased by ~10 tokens on average (rough estimate). This increase (1%) is too minimal to make strong claims that behavior is steered towards THINK. moreover, this measures the reasoning markers in the answers and not the reasoning. But, it is never established if the THINK behaviors exhibit these markers in the asnwers as well.
9 comments
AuthorsOpenReview ↗

Response to Reviewer XHT4: Part 1

We sincerely thank the reviewer for the constructive and insightful comments. Below we address the comments point by point.

Response to Weakness 1. We thank the reviewer for raising this point. We measure along the base model's direction for three reasons.

  1. The research question fixes the reference. We ask whether post-training re-invokes the thinking ability the base model already has, which is the definition of thinking leakage in this work (line 48-50), so the reference must be anchored in the base model.
  2. The checkpoint's own gap measures a distance, not a displacement. It describes how far apart the two modes are now, but leakage is how far NOTHINK has moved toward THINK, which requires a starting point. On the checkpoint's own representations, ideally, that starting point would be a leakage-free NOTHINK vector : where NOTHINK would sit after all of post-training except the leakage. Leakage would then be the fraction of the way from hˉNfree\bar h^{\text{free}}_ {\mathcal N} to the checkpoint's THINK vector that its NOTHINK vector has covered, with hˉT−hˉNfree\bar h_ {\mathcal T} - \bar h^{\text{free}}_ {\mathcal N} as the checkpoint's own leakage vector. But hˉNfree\bar h^{\text{free}}_ {\mathcal N} is a counterfactual and cannot be observed. Base NOTHINK is the closest observable substitute, which is why we anchor on the base model.
  3. The gap measure is in the paper, and agrees. In Sec. 3, we recompute both modes on rollouts generated by the Qwen3-8B checkpoint at 200 and 500 GRPO steps (eq. (8), line 190-199, figure 3, Appendix D.3), and the angle between them shrinks (11.31° → 6.66° → 5.48°).

The checkpoint's own angle alone is also symmetric: it cannot tell NOTHINK moving toward THINK from THINK moving toward NOTHINK, which would mean the opposite of leakage.

Beyond Qwen3-8B GRPO in the paper, we additionally compared each checkpoint's own THINK vector hˉT\bar h_ {\mathcal T} with the corresponding base THINK vector on MiniCPM4.1-8B GRPO and Qwen3-8B SFT:

checkpointcos (post THINK vs. base THINK)anglenorm (base THINK → post THINK)
Qwen3-8B GRPO@5000.99981.2°72.32 → 71.93
MiniCPM4.1-8B GRPO@5000.99990.8°72.26 → 71.89
Qwen3-8B SFT@50 (mask-think)0.99585.3°72.32 → 74.47

So the gap closes mainly because NOTHINK moves toward THINK. This also supports point 2: since THINK changes little, base NOTHINK is a reasonable substitute for the missing leakage-free starting point. We have also added a note to clarify in Sec. 3 (iv):"Since ∥vleak∥\|v_ {\mathrm{leak}}\| is set jointly by the two mode norms and the angle between them, we report ϕ\phi alongside it: a gap can shrink because the representations themselves shrink, or because the two modes come to occupy the same direction. Only the latter is the geometric signature of thinking leakage."

Response to Weakness 2: We thank the reviewer for this insightful point. We see the two views as complementary. Entropy collapse describes post-training in terms of how certain the model's outputs become; our measurements describe it in terms of which direction the representations move. A distribution can sharpen toward many different targets, and our results identify the target in hybrid models: the model's own THINK mode. To our knowledge, this is the first representation-level audit which direction post-training moves the NOTHINK mode. Previous work has touched on this only at the text level, for example by tracking reasoning-marker density, without examining internal representations or testing whether the shift causes the gain, as mentioned in lines 51–52.

This movement is also not general sharpening, which would pull the two modes toward a common place. As lines 216–225 (revision) describe, it is asymmetric: the THINK centroid moves only 10.2% of the base gap, while the NOTHINK centroid moves 66.2% toward it (Sec. 3, Fig. 3; Appendix D.3). THINK also changes little on the other models and methods we checked (see our response to Weakness 1).

More importantly, Sec. 4 and Sec. 5 show that this movement causes the performance gain, beyond correlation. Steering the untouched base model along this direction, with no training and hence no sharpening, reproduces most of the gain, while the same dose along a reference direction does not (Sec. 4). Counter-steering a checkpoint along it removes most of the gain (Sec. 5).

Additional control experiment (new Appendix H) further verifies that the direction of this shift in behavioral patterns is not general sharpening; it is moving to THINK mode.

AuthorsOpenReview ↗

Response to Reviewer XHT4: Part 2

Response to Weakness 3: We thank the reviewer for this point. We do not claim that steering reproduces the full representation of a post-trained checkpoint; doing so is essentially impossible, even with fine-grained token-wise, multi-layer steering. Our goal is narrower: to test whether the axis we identify is causally related to the performance gain. This does not require steering to reproduce the representation exactly.

The logic of Sec. 4 and Sec. 5 is that of testing whether an ingredient is what makes a treatment work from a causal perspective. Suppose sailors recover from scurvy after eating oranges, and we want to know whether vitamin C is the reason. Oranges change many things in the body, just as post-training changes many things in the model. To test vitamin C, we give pure vitamin C to sick sailors and see whether they recover (sufficiency, Sec. 4), and we block the absorption of vitamin C in sailors who recovered on oranges and see whether the benefit is lost (necessity, Sec. 5). The pill does not reproduce everything that eating oranges does, and it does not need to: if recovery follows the vitamin C dose in one direction and is lost when vitamin C is blocked in the other, vitamin C is causally responsible. In our setting, post-training plays the role of the oranges, and vleakv_{\text{leak}} plays the role of vitamin C. Steering supplies it to the base model (Sec. 4), and counter-steering removes it from a checkpoint (Sec. 5).

Causality is established by the trends these interventions produce, not by matching a single representation. And the trends are not limited to accuracy. Reasoning-marker density rises with steering dose and falls under counter-steering, and response length follows the same pattern (Len. column of Tables 4 and 8). Accuracy, marker density, and length together point to the same conclusion: moving along this one direction drives the model toward or away from THINK behavior, in proportion to the dose.

Response to Weakness 4 and unclear metrics: We thank the reviewer for the careful reading; these are fair points. We have rewritten the first half of Sec. 3 and revised Sec. 2.1 accordingly (changes are highlighted in the revised PDF).

  • "Engagement" (Sec. 2.1). It is our name for the mediator in the causal graph. We now introduce it as such where it first appears: "how much THINK-style computation a nominally direct NOTHINK answer draws on, measured in Sec. 2.2 as a projection onto a single direction."
  • Reasoning markers (Sec. 3). We now define them where they are first used: "the density of reasoning markers, short connecting words such as wait, alternatively and hence that signal a turn, a second thought or a conclusion in a line of reasoning and are characteristic of THINK traces (full list in Appendix C)." The full list remains in Appendix C, now referenced from the main text.
  • The roles of the metrics (Sec. 3). The reviewer's reading is exactly right, and we now state it in the text. Drift alignment is a cosine, so it measures only direction: whether NOTHINK moves toward THINK at all. Paired Cohen's dd measures magnitude: how far NOTHINK moves along that direction, per question, relative to question-to-question variation. Marker density measures in the generated text. We have added one sentence introducing each metric by the question it answers, before its formula.

Response to Question 1: We thank the reviewer for this question.

Why layer 20. We chose layer 20 based on prior work and our own experiments. Layer 20 sits at mid-depth, where prior work finds residual-stream representations most abstract and most causally manipulable (the second paragraph of Sec. 4). We then verified this choice empirically (see the following response to the sensitivity of the conclusions).

Sensitivity of the conclusions. We repeated the main measurements at other depths:

  • Leakage detection (Appendix D.2). At shallow, middle, and deep layers (8, 20, and 32/28), drift along vleakv_{\text{leak}} tracks NOTHINK accuracy for every model at every depth (ρ=0.83\rho = 0.83–0.930.93).

  • Steering (Appendix E.2). Steering along each layer's own vleakv_{\text{leak}} at six depths (12–32) raises both accuracy and marker density above base at every depth. Layer 20 gives the largest effect on both, roughly twice the accuracy gain of the next best layer, and the lowest repetition rate.

So the existence and causal role of leakage do not depend on the layer; layer 20 is where the effect is strongest and the intervention is cleanest, which is why we use layer 20 for auditing.

AuthorsOpenReview ↗

Response to Reviewer XHT4: Part 3

Response to Question 2: We thank the reviewer for this question.

Choice of evaluation set. Evaluating reasoning on competition-level math with multiple rollouts per problem and reporting mean accuracy is standard practice. We use 120 problems from AIME and HMMT, each sampled 16 times. Easier benchmarks leave too little room for our question: on MATH-500, Qwen3-8B already reaches 87.4 in NOTHINK and 97.4 in THINK, whereas on AIME24 it reaches 29.1 and 76.0, and on AIME25 20.9 and 67.3 [1]. Leakage is easier to measure where the gap between the two modes is large, so we need problems hard enough to leave that gap open.

Evaluation variance. We bootstrap over problems (1,000 resamples) to obtain 95% confidence intervals. Single-condition accuracy intervals reach about ±0.07\pm 0.07, but our claims compare two conditions on the same problems, and those paired differences are much tighter (about ±0.02\pm 0.02–0.060.06), since problem difficulty cancels.

The intervals also match the three groups of checkpoints in Sec. 5 and Appendix F.1:

  • Aligned (9 checkpoints). Both the training gain and the accuracy removed by counter-steering (LtrainedL_{\text{trained}}) have intervals that exclude zero on every checkpoint.
  • No usable gain (4 checkpoints). None has a training gain significantly above zero, and three of the four have LtrainedL_ {\text{trained}} intervals that include zero.
  • Reverse drift (2 checkpoints). Both the training gain and LtrainedL_{\text{trained}} have intervals that lie entirely below zero: these checkpoints lost accuracy, and moving them back along the axis recovers part of it.

The intervals also agree with our grouping of the remaining checkpoints: those that barely drifted have LtrainedL_ {\text{trained}} intervals that include zero, and the two MiniCPM checkpoints that drifted backward have LtrainedL_ {\text{trained}} intervals that are entirely negative.

checkpointcheckpoint ACCcounter-steer γ=1 ACCtrain gain (checkpoint − base NoThink)L_trained (checkpoint − counter-steer)
GRPO-Qwen3-8B@5000.472 [0.400, 0.544]0.349 [0.286, 0.418]+0.294 [+0.233, +0.349]+0.123 [+0.085, +0.164]
GRPO-Qwen3-8B@2000.447 [0.380, 0.519]0.315 [0.250, 0.378]+0.269 [+0.214, +0.324]+0.132 [+0.095, +0.172]
SFT-Qwen3-8B@500.417 [0.347, 0.489]0.267 [0.205, 0.335]+0.239 [+0.188, +0.291]+0.150 [+0.114, +0.188]
OPSD-Qwen3-8B@250.331 [0.265, 0.400]0.210 [0.154, 0.268]+0.153 [+0.103, +0.204]+0.121 [+0.077, +0.169]
OPSD-Qwen3-8B@1000.150 [0.106, 0.197]0.131 [0.090, 0.173]-0.028 [-0.077, +0.017]+0.019 [-0.016, +0.055]
GRPO-Qwen3-4B@4500.445 [0.375, 0.514]0.312 [0.249, 0.384]+0.285 [+0.233, +0.339]+0.133 [+0.101, +0.167]
GRPO-Qwen3-4B@2000.414 [0.347, 0.486]0.284 [0.221, 0.348]+0.254 [+0.207, +0.304]+0.130 [+0.097, +0.166]
SFT-Qwen3-4B@750.408 [0.342, 0.474]0.261 [0.201, 0.325]+0.248 [+0.201, +0.294]+0.147 [+0.117, +0.179]
OPSD-Qwen3-4B@500.191 [0.140, 0.248]0.152 [0.104, 0.206]+0.031 [-0.001, +0.063]+0.039 [+0.020, +0.059]
OPSD-Qwen3-4B@250.119 [0.076, 0.167]0.102 [0.061, 0.151]-0.041 [-0.076, -0.008]+0.017 [-0.009, +0.043]
GRPO-MiniCPM4.1-8B@5000.436 [0.365, 0.513]0.304 [0.241, 0.366]+0.257 [+0.206, +0.318]+0.133 [+0.100, +0.166]
GRPO-MiniCPM4.1-8B@2000.304 [0.243, 0.371]0.243 [0.183, 0.306]+0.124 [+0.089, +0.162]+0.061 [+0.034, +0.090]
SFT-MiniCPM4.1-8B@1000.162 [0.117, 0.214]0.179 [0.134, 0.232]-0.018 [-0.034, -0.001]-0.017 [-0.034, -0.001]
OPSD-LoRA-MiniCPM4.1-8B@750.196 [0.151, 0.248]0.189 [0.141, 0.246]+0.016 [-0.002, +0.035]+0.007 [-0.009, +0.023]
OPSD-MiniCPM4.1-8B@500.068 [0.041, 0.100]0.085 [0.055, 0.123]-0.112 [-0.149, -0.077]-0.018 [-0.035, -0.003]

Base NoThink: Qwen3-8B 0.178 [0.122, 0.236]; Qwen3-4B 0.160 [0.108, 0.213]; MiniCPM4.1-8B 0.180 [0.135, 0.234]

Variance across training runs. We train each method–model configuration once. Multiple seeds matter when the research question is whether one training method outperforms another on average. Ours is different: for a given checkpoint, how much of its gain depends on thinking leakage. Each checkpoint is the object of the audit, and every comparison we make is between conditions on the same models, so the relevant variance is the evaluation variance above, which is small relative to the effects. Our conclusions also do not rest on any single run: they rest on trends that hold across 65 checkpoints spanning three methods, three models, and multiple training steps (Sec. 3), and across 15 checkpoints in the audit (Sec. 4,5,6).

[1] Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., ... & Qiu, Z. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their time and constructive feedback. We are encouraged that the reviewers found the question relevant and important for interpreting NOTHINK post-training gains (FYFB, wqcm, XHT4), and that the combination of representational analysis with bidirectional interventions provides substantive evidence beyond behavioral observation alone (wqcm, FYFB).

We have revised the paper and added new experiments in response; all changes are highlighted in the revised PDF. The main updates are summarized below, with details in the individual responses.

1. Clarity of the framework and measures (XHT4). We now define thinking engagement as the mediator in the causal graph where it first appears (Sec. 2.1), and have rewritten the first half of Sec. 3 so that each measure states what it captures. Reasoning markers are now defined in the main text.

2. Achieved engagement under counter-steering (wqcm). Counter-steering does not return engagement exactly to the base level, and we no longer describe it as doing so "by construction". We now report the achieved engagement for every checkpoint (Appendix F.9): at the intervened layer it overshoots base by 6–21% of ∥vleak∥\|v_{\text{leak}}\|. Sec. 5 now defines the intervention as an applied displacement equal to the drift, and Appendix G.4 reports the leakage ratio rescaled by the overshoot (0.34–0.62, against 0.42–0.79 as reported); the conclusions are unchanged.

3. Specificity and disruption controls (FYFB, wqcm). A new appendix (Appendix H) repeats both interventions along control directions, each isolating one factor. Directions estimated from the thinking region alone and matched in position and token count (first or last 256 tokens of the THINK's reasoning block) reproduce and remove the gain much as vleakv_{\text{leak}} does. Directions targeting response length or marker words, and a random direction pushed exactly as far as vleakv_{\text{leak}}, have no comparable effect. The effect therefore follows the THINK–NOTHINK contrast and does not reduce to length, position, style, or generic perturbation.

4. Direction of the shift (XHT4). In addition to the mode-gap analysis in Sec. 3, we compared each checkpoint's own THINK centroid with the base model's on MiniCPM4.1-8B GRPO and Qwen3-8B SFT. THINK changes little in both, so the gap closes mainly because NOTHINK moves toward THINK.

5. Statistical reliability (XHT4). We added paired bootstrap confidence intervals over problems. On all nine checkpoints with a genuine gain, the training gain, the accuracy removed by counter-steering, and the change in marker density all have intervals excluding zero.

6. Positioning and practical implications (FYFB). We have added CoT Vectors to Related Work and clarified that our contribution is an audit of post-training gains rather than a new steering method. Appendix B.1 now explains how the leakage estimate should inform the evaluation of a method: the leakage share supports a claim of efficiency, to be compared against base THINK, while the residual supports a claim of improved direct answering.

7. Generalization to another benchmark (XHT4). A new appendix (Appendix I) applies the same vleakv_{\text{leak}} and the same doses, without re-estimating either, to Minerva Math (272 undergraduate-level STEM problems) on Qwen3-8B. Both interventions show a monotone dose–response: steering the base model recovers 34% and 73% of the GRPO gain at the two doses, and counter-steering the checkpoint removes 69% and 111% of it. The direction therefore generalizes beyond the competition-math data it was estimated on.

We will release all code, checkpoints, and rollouts upon publication. We thank the reviewers again and welcome any further questions.

Area_Chair_pwLNOpenReview ↗

rebuttal ddl

Dear reviewers, the ICLR deadline is only 4 days away. The authors have already rebuttaled the issues raised. If you still have questions, please participate in the rebuttal.

AuthorsOpenReview ↗

Update following the reviewer discussion

We thank the reviewers for their follow-up questions. We have three updates to our earlier response.

  1. Direction of drift (XHT4). Our earlier statement that “THINK changes little” was too broad for Qwen3-8B SFT@50. Its THINK centroid moves by 45.5% of the original mode gap in the full activation space. Along the base NOTHINK-to-THINK axis (v_{leak}), however, THINK moves −5.3% of that gap and NOTHINK moves +88.0%. NOTHINK thus accounts for about 94% of the reduction in their projected separation. The full Euclidean distance between the two centroids decreases by 82.2%.

  2. Direct test of the overshoot correction (wqcm). We tested the linear correction on three checkpoints by adjusting the counter-steering dose to bring layer-20 engagement back to the base level. The remaining gap was at most 0.6% of the base mode gap, and the measured leakage ratios differed from the linear predictions by at most 3.4% in relative terms (Appendix G.4, Table 18).

  3. Scope of the causal claim (XHT4). We revised the abstract and introduction to state precisely what the interventions quantify: how much of the observed NOTHINK accuracy gain depends on the audited direction. In Sec. 3, we clarified the comparison with the final-answer reference direction and removed the broader concluding statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.