- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper extracts fixed activation directions from differences between pre- and post-training checkpoints on shared reasoning trajectories. Adding these directions to the starting model recovers substantial accuracy gains. Subtracting them from the trained model reduces performance. Experiments cover six checkpoint pairs involving RL and on-policy distillation, with additional transfer from mathematics to code. The authors also examine token-level effects and propose AdaTIRD, which uses supplied difficulty labels to vary suppression strength and reduce generation length.
Critical strengths
The recovery experiment is the strongest part of the paper. Adding the OPD-derived direction raises Qwen3-4B Base’s AIME24 accuracy from 18.3% to 69.2%, with no weight updates. Random-direction and subtraction controls help establish that the extracted orientation matters. I also find the mathematics-to-code transfer useful. It shows that the intervention is not simply recovering behavior specific to the extraction task.
Critical weaknesses
I am less convinced by the explanation of what TIRD contributes during reasoning. Response length changes substantially alongside accuracy. The experiments therefore leave open how much of the gain comes from additional computation. The token analysis has a separate limitation: positions are selected for large probability drops. Table 6 shows comparable drops for ordinary-language tokens, but provides no category frequencies over the full responses. This makes the interpretation of the selected examples difficult to assess.
For AdaTIRD, the missing comparison is fixed suppression at the same average token cost. The decoding baselines do not answer whether assigning different coefficients by difficulty actually helps. This matters more to my assessment than further comparisons with temperature schedules.
Questions for authors
-
Does Base+TIRD retain an accuracy advantage when actual generated-token expenditure is matched?
-
What fraction of each token category passes the probability-drop threshold in Section 3.3? Do category differences remain after matching the original target-token probabilities?
-
On MATH500, how does AdaTIRD compare with a fixed coefficient selected on held-out data, at similar mean output length? This would help establish the contribution of difficulty-dependent allocation
4 comments
To further investigate this issue, we compared the changes in predictive entropy and KL divergence between token groups with mean probability drops greater than 0.3 and those below this threshold.
Token groups with mean probability drop > 0.3
| Category | Groups | Mean probability decrease | Mean entropy change | Mean KL change |
|---|---|---|---|---|
| Mathematical content | 372 | 0.4210 | 1.0466 | 0.7330 |
| Reasoning control | 110 | 0.4080 | 1.0514 | 0.6839 |
| Discourse structure | 63 | 0.3921 | 1.0111 | 0.6407 |
| Formatting | 33 | 0.4103 | 0.5983 | 0.7080 |
| Ordinary language | 220 | 0.4133 | 1.1142 | 0.6929 |
Token groups with mean probability drop ≤ 0.3
| Category | Groups | Mean probability decrease | Mean entropy change | Mean KL change |
|---|---|---|---|---|
| Mathematical content | 3,283 | 0.0887 | 0.4823 | 0.2330 |
| Reasoning control | 759 | 0.0886 | 0.7242 | 0.4292 |
| Discourse structure | 377 | 0.0779 | 0.5719 | 0.3646 |
| Formatting | 423 | 0.0507 | 0.2134 | 0.1849 |
| Ordinary language | 1,843 | 0.0874 | 0.6689 | 0.3327 |
These results explain why we focus our analysis on tokens with probability drops greater than 0.3. As shown in the tables, these positions exhibit substantially larger changes in mean predictive entropy and KL divergence than other positions. Such larger distributional changes suggest that these positions are more likely to play a causal role in altering the model's reasoning trajectory, allowing us to better understand how TIRD influences the reasoning process.
To further investigate the role of these tokens, we conducted an additional experiment using Qwen3-4B RL on AIME24. Specifically, we let the Target model generate responses through autoregressive sampling. Whenever the probability of a sampled token under Target exceeded its probability under Target−TIRD by more than 0.3, we replaced that token with the highest-probability token predicted by Target−TIRD at the same position. The results are shown below.
| Condition | Accuracy | Mean Length |
|---|---|---|
| Target | 62.50% | 11,830.24 |
| Target with Token Replacement | 36.67% | 9,281.90 |
Across all generated sequences, only 5.74% of tokens were replaced, yet this intervention caused a substantial 25.83 percentage-point drop in accuracy, from 62.50% to 36.67%. These results demonstrate that tokens whose probabilities are substantially suppressed by removing TIRD play an important role in the reasoning process. Replacing only a small fraction of these tokens with the most probable alternatives under Target−TIRD is sufficient to substantially degrade reasoning performance, providing further evidence of their functional importance in shaping the model's reasoning trajectories.
3. AdaTIRD vs. Fixed-Coefficient Suppression
We thank the reviewer for suggesting this important comparison. To demonstrate the effectiveness of difficulty-dependent coefficient allocation relative to fixed-coefficient designs, we compared AdaTIRD against fixed-coefficient suppression baselines.
We also introduced two additional experimental settings using pretrained difficulty classifiers, namely Jev API and the open-source JevK5, to evaluate AdaTIRD when ground-truth difficulty labels are unavailable.
| Method | Accuracy | Mean tokens | Token savings |
|---|---|---|---|
| Target | 95.50% | 3,382.18 | 0.00% |
| Fixed α = −0.2 | 94.20% | 2,960.05 | 18.39% |
| Fixed α = −0.4 | 93.20% | 2,498.77 | 30.85% |
| Fixed α = −0.6 | 91.25% | 2,001.31 | 40.83% |
| Fixed α = −0.8 | 88.95% | 1,756.04 | 48.08% |
| Fixed α = −1 | 83.65% | 1,522.70 | 54.98% |
| Temperature | 95.75% | 3,416.46 | −1.01% |
| Length | 94.50% | 3,264.39 | 3.48% |
| Temperature + Length | 94.10% | 3,311.76 | 2.08% |
| AdaTIRD Standard | 95.30% | 2,921.37 | 13.62% |
| AdaTIRD + Jev API | 94.15% | 2,529.00 | 25.23% |
| AdaTIRD + JevK5 | 93.40% | 2,243.73 | 33.66% |
The results demonstrate that even without access to ground-truth difficulty labels, and relying only on existing difficulty classifiers without additional training, our method achieves a clear advantage in the trade-off between accuracy and reasoning length, demonstrating its practical applicability.
Furthermore, compared with the fixed α = −0.2 baseline, AdaTIRD Standard achieves higher accuracy at a comparable generation length. Specifically, AdaTIRD Standard achieves 95.30% accuracy with 2,921 tokens, whereas fixed α = −0.2 achieves 94.20% accuracy with 2,960 tokens.
Similarly, the Jev API and JevK5 settings also achieve higher accuracy than fixed-coefficient baselines with comparable generation lengths. These results demonstrate the value of our difficulty-dependent coefficient allocation strategy, showing that assigning intervention strengths according to problem difficulty provides advantages over applying a uniform coefficient to all problems.
We sincerely appreciate the reviewer's thoughtful feedback and valuable suggestions. We hope that these additional experiments and analyses further demonstrate the effectiveness of TIRD and the benefits of difficulty-adaptive reasoning allocation.
References
[1] Andy Zou et al. Representation Engineering: A Top-Down Approach to AI Transparency. 2023.
[2] DeepSeek-AI et al. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. 2025.
reply
Thanks for this feedback; I maintain my original scores, but I will update my confidence
We sincerely thank the reviewer for the thoughtful evaluation and constructive feedback. We particularly appreciate the insightful questions regarding the relationship between reasoning capability and generation cost, the token-level effects of TIRD, and the effectiveness of difficulty-adaptive suppression. To further investigate these questions, we conducted additional experiments and extended our analyses as detailed below.
1. Reasoning Accuracy Under Comparable Generation Costs
We thank the reviewer for this valuable question. To investigate whether the accuracy improvements achieved by TIRD arise simply from increased token expenditure, we constructed three alternative steering directions using RepE [1]: increasing response length, suppressing EOS, and encouraging reasoning behavior.
We applied these directions to the model and obtained the following results.
As shown in the results, both the response-length direction and the EOS-suppression direction substantially increase generation length, reaching token expenditures close to that of TIRD. Although these directions improve accuracy to some extent, their improvements remain far smaller than those achieved by TIRD.
This demonstrates that TIRD enhances reasoning capability, rather than merely increasing reasoning effort.
As analyzed in Section 3 of our paper, what changes during model reasoning is the way the model thinks. This behavioral change resembles observations made during reinforcement learning, such as those reported for DeepSeek-R1-Zero [2]. As training progresses, the model generates longer responses and becomes more inclined to carefully check its reasoning rather than directly produce an answer.
Reasoning capability and reasoning effort are inherently related. Additional reasoning effort can indeed increase the probability of arriving at the correct answer, but genuinely improving the model's capability requires effective reasoning rather than merely generating more tokens.
This is precisely what TIRD enables. TIRD shifts the model's internal activations toward representations that favor reflective reasoning, thereby activating genuinely effective emergent reasoning capabilities, rather than simply using more tokens to guess the answer.
2. Token Category Frequencies and Probability-Matched Analysis
We appreciate the reviewer's insightful suggestion. To provide a more comprehensive understanding of the token-level effects of TIRD, we extended our analysis to token categories across the full responses and examined probability changes under different original Target-token probability ranges.
Using Qwen3-4B RL as the Target model, we first collected statistics for the different token categories.
To analyze token probability changes in greater detail across different initial probability ranges, we grouped tokens according to their original sampling probabilities under Target and calculated the following statistics.
One important consideration is that averaging probability changes inevitably obscures finer-grained information. Consequently, it is difficult to determine from these averaged statistics alone how suppressing TIRD in activation space specifically affects the model and its reasoning decisions. This is precisely why we focus on tokens with probability changes greater than 0.3 in Section 3.3 of our paper.