acceptodds
Under review as a conference paper at ICLR 2027

Dr.CoT: Diagnosing and Regulating Chain-of-Thought via Sparse Feature Coverage

Abstract

Large reasoning models improve complex problem solving through extended chain-of-thought (CoT) reasoning, but excessive reasoning increases inference cost and can even degrade accuracy, a phenomenon known as overthinking. Existing approaches mitigate overthinking by estimating reasoning sufficiency from trajectory-derived signals, yet lack explicit prompt-conditioned modeling of the reasoning state, limiting their ability to diagnose sufficiency and realign the trajectory when reasoning deviates from the prompt. To address these limitations, we propose **Dr.CoT**, a framework for diagnosing and regulating chain-of-thought reasoning via prompt-conditioned sparse feature coverage. Specifically, Dr.CoT first represents the prompt and reasoning trajectory in a shared sparse feature space, and uses prompt-conditioned feature coverage as an indicator of the current reasoning state. It then tracks feature coverage and its gain to diagnose reasoning sufficiency, adaptively stopping reasoning when coverage is sufficient and gains remain persistently low, which improves the accuracy–efficiency trade-off. During continued reasoning, it further uses uncovered prompt features to steer the model toward addressing the prompt, correcting deviations of the reasoning trajectory from the prompt. Extensive experiments across three model scales show that Dr.CoT reduces average token usage by **17.01%** while improving average accuracy by **2.16%** over standard CoT, and also outperforms all evaluated baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Reasoning Models, Chain of Thought, Overthinking, Sparse Autoencoders, Reasoning Efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YmrTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Dr.CoT, an inference-time framework that uses prompt-conditioned sparse feature coverage to improve the accuracy-efficiency trade-off of large reasoning models. It uses a sparse autoencoder (SAE) to map the input prompt and generated chain-of-thought (CoT) into a shared sparse feature space. The framework consists of two components:

  1. Coverage-Guided Sufficiency Diagnosis. Measures prompt feature coverage and its recent gains to determine when reasoning is sufficient. Reasoning stops when coverage is sufficiently high and gains remain low across consecutive windows.
  2. Residual Feature Steering. Uses uncovered prompt features to construct dynamic steering directions, guiding the model toward information that has not yet been addressed.

Experiments on three Qwen3.5 models (2B, 9B, and 27B) across four math and science reasoning benchmarks show that Dr.CoT reduces token usage by 17.01% while improving accuracy by 2.16 percentage points over standard CoT.

Critical strengths
  • The paper proposes a novel unified framework that uses sparse feature representations for both reasoning termination and trajectory steering. Combining reasoning state modeling, sufficiency diagnosis, and dynamic steering is an interesting and meaningful contribution.
  • Dr.CoT achieves a competitive accuracy-efficiency trade-off across different model scales and reasoning benchmarks. The improvements are particularly notable on smaller models.
  • The paper provides useful ablation studies and computational cost analysis. Table 2 shows that coverage-guided stopping mainly reduces token usage, while residual feature steering improves accuracy. Table 4 compares different steering directions and provides evidence for the effectiveness of uncovered feature steering. Table 6 reports FLOPs, throughput, and GPU memory usage, which helps assess the practical cost of the method.
Critical weaknesses
  1. Although Dr.CoT improves overall accuracy and reduces token usage across all three model scales, Table 1 shows accuracy drops on some individual benchmarks compared with standard CoT. It would be helpful if the authors could further analyze the reasons for these drops.
  2. In Table 3, the authors replace the current problem's prompt features with those from another problem as a baseline. However, using unrelated prompt features to assess reasoning sufficiency is likely to hurt performance. Therefore, this comparison is not particularly convincing. More meaningful baselines would help better validate the effectiveness of coverage-guided sufficiency diagnosis.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer YmrT

We appreciate your careful assessment and recognition of our prompt-conditioned feature modeling for sufficiency diagnosis and trajectory steering. Our responses to your comments W1–W2 are as follows:


W1: The paper's Table 1 shows accuracy drops on some individual benchmarks. What causes these drops?

We inspected cases where Vanilla answered correctly but Dr.CoT did not and identified overthinking as one remaining failure mode.

  1. With Qwen3.5-9B on HMMT problem 9, Dr.CoT reaches the correct result, 50−502101−150-\frac{50}{2^{101}-1}, at about 22k tokens. It continues for another 10k tokens, while the coverage ratio rises only from 83.9% to 84.4%. Subsequent reconsideration introduces a boundary error, and the final answer becomes 5050, omitting the correction term retained in Vanilla's final answer.

  2. With Qwen3.5-2B on GPQA problem 120, Dr.CoT initially derives ∣dNdp∣∝p−4\left|\frac{dN}{dp}\right|\propto p^{-4} and selects A. It then reinterprets the question, replaces three-dimensional spherical-shell counting with two-dimensional ring counting, and switches to B (p−3p^{-3}). Vanilla also reconsiders the question but ultimately outputs A, illustrating that continued reasoning does not necessarily overturn a correct answer.

Although Dr.CoT reduces overthinking overall, these cases show room for more precise coverage-guided sufficiency diagnosis. Finer-grained stopping signals could better distinguish useful verification from redundant reconsideration and preserve correct intermediate results.


W2: Beyond the shuffled-prompt comparison, can more meaningful baselines further validate coverage-guided sufficiency diagnosis?

We compared fixed-budget Vanilla with Dr.CoT without residual feature steering on AIME 2026 using Qwen3.5-9B. For Vanilla, we varied only the total token budget, keeping the paper’s decoding settings and final-answer protocol. All results are averaged over four samples per question.

Table 1. Fixed-budget and coverage-guided stopping on Qwen3.5-9B and AIME 2026.

MethodStopping ruleAccuracy (%) ↑Mean generated tokens ↓
VanillaFixed budget (20,480 tokens)76.6720,084
VanillaFixed budget (24,576 tokens)76.6723,770
VanillaFixed budget (28,672 tokens)73.3327,106
Dr.CoT (w/o steering)Coverage-guided stopping77.5026,982

Increasing the fixed budget produced longer outputs, yet accuracy plateaued and then declined, a pattern consistent with overthinking. At the closest mean generation length (only 124 tokens apart), Dr.CoT (w/o steering) achieved 4.17 percentage points higher accuracy than the 28,672-token baseline. This result supports using the coverage ratio and coverage gain to determine when to stop reasoning for each question.


We hope our responses and the additional evidence have addressed your concerns. We sincerely thank you again for your constructive feedback, which has helped us strengthen the evaluation and presentation of our work.

Thanks for the authors' response, and I will maintain my positive scores.

Thank you very much for taking the time to review our rebuttal. We sincerely appreciate your thoughtful feedback and continued positive assessment of our work.

Reviewer_HTQKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dr.CoT, an inference-time method for reducing overthinking in reasoning models. The main idea is to represent the prompt and the generated reasoning in a shared sparse feature space, measure how much of the prompt features have been covered, and use this signal both for adaptive stopping and for steering the model toward uncovered prompt features. The method is evaluated on three Qwen3.5 model sizes and four math/science benchmarks. The authors report reduced token usage together with improved average accuracy over vanilla CoT and several early-stopping baselines.

Critical strengths

The main idea is simple and interesting. Instead of relying only on output-side signals such as confidence or redundancy, Dr.CoT explicitly measures reasoning progress relative to the prompt and uses the same representation for both stopping and steering.

The empirical results are promising. The method reduces token usage while improving average accuracy, and the ablations on shuffled prompts and alternative steering directions provide useful evidence that the prompt-conditioned features matter.

Critical weaknesses
  • Limited generalization: All experiments use the Qwen3.5 model family with Qwen-Scope SAEs, so it is unclear whether the method transfers to other model architectures or SAE representations.

  • Incomplete baseline comparison: The evaluation focuses mainly on early-stopping baselines, while closely related activation-steering methods discussed in the paper are not experimentally compared.

  • Potential generation-budget confound: Vanilla generations, especially for the 2B model, often approach the 32,768-token limit, while responses without a boxed answer are marked incorrect. Since Dr.CoT explicitly terminates reasoning and generates a final answer, part of the accuracy gain may come from avoiding truncation rather than better reasoning.

  • Questionable interpretation of feature coverage: The method treats accumulated prompt-feature activation as reasoning progress, and coverage is monotonically non-decreasing. However, activating a feature does not necessarily mean that the corresponding reasoning step has been correctly resolved.

  • Limited statistical analysis: The paper reports point estimates averaged over four samples per question but provides no confidence intervals or significance analysis, which is important for relatively small gains on larger models.

Questions for authors

My questions are related to the weaknesses mentioned above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer HTQK (Part 1 / 3)

We sincerely appreciate your detailed and constructive feedback. We believe our response has addressed most of your concerns, and we respectfully ask if you're comfortable reconsidering the rating in light of our efforts and dedication. Below, we provide detailed responses to all the comments:


W1: All experiments use Qwen3.5 with Qwen-Scope SAEs. Does Dr.CoT transfer to other architectures and SAE representations?

We evaluated Dr.CoT on DeepSeek-R1-Distill-Llama-8B [1] using a publicly available pretrained ReLU SAE [2] (layer 19, 65,536 features). Both methods follow the paper’s evaluation protocol with four responses per question.

Table 1. Results on DeepSeek-R1-Distill-Llama-8B. Acc (%) and Tokens are averaged over four samples per question.

\hline \begin{array}{l} \vphantom{\text{GPQA-Diamond}} \\\\ \begin{array}{l} \text{Method} \end{array} \end{array} & \begin{array}{c} \text{AIME 2026} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{HMMT 2026} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{GPQA-Diamond} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{HLE-STEM} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{Overall} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} \\\\ \hline \begin{array}{l} \text{Vanilla} \\\\ \text{Dr.CoT} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{39.17} & \hphantom{\text{Tokens}}\llap{12{,}767} \\\\ \hphantom{00.00}\llap{\mathbf{41.67}} & \hphantom{\text{Tokens}}\llap{\mathbf{12{,}340}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{18.18} & \hphantom{\text{Tokens}}\llap{15{,}459} \\\\ \hphantom{00.00}\llap{\mathbf{21.21}} & \hphantom{\text{Tokens}}\llap{\mathbf{14{,}479}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{50.51} & \hphantom{\text{Tokens}}\llap{8{,}175} \\\\ \hphantom{00.00}\llap{\mathbf{52.53}} & \hphantom{\text{Tokens}}\llap{\mathbf{7{,}941}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{9.20} & \hphantom{\text{Tokens}}\llap{5{,}351} \\\\ \hphantom{00.00}\llap{\mathbf{10.34}} & \hphantom{\text{Tokens}}\llap{\mathbf{4{,}744}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{29.26} & \hphantom{\text{Tokens}}\llap{10{,}438} \\\\ \hphantom{00.00}\llap{\mathbf{31.44}} & \hphantom{\text{Tokens}}\llap{\mathbf{9{,}876}} \end{array} \\\\ \hline \end{array}$$ As shown in Table 1, Dr.CoT achieves higher accuracy with fewer generated tokens than Vanilla on all four benchmarks. These benefits extend to the evaluated Llama-based backbone and ReLU SAE, providing evidence of Dr.CoT’s generalization beyond the original Qwen3.5/Qwen-Scope combination. We have added these results and analysis to Appendix F.2 of the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer HTQK (Part 2 / 3)

W2: The evaluation focuses on early-stopping baselines. How does Dr.CoT compare with related activation-steering methods?

We compared Dr.CoT with two activation-steering baselines, SEAL [3] and ReBalance [4], on Qwen3.5-9B. Both baselines follow the method-specific settings in their original papers and use the same benchmarks, decoding settings, token budget, and final-answer protocol as our main experiments. All results are averaged over four samples per question.

Table 2. Activation-steering comparison on Qwen3.5-9B.

\hline \begin{array}{l} \vphantom{\text{GPQA-Diamond}} \\\\ \begin{array}{l} \text{Method} \end{array} \end{array} & \begin{array}{c} \text{AIME 2026} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{HMMT 2026} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{GPQA-Diamond} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{HLE-STEM} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{Overall} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} \\\\ \hline \begin{array}{l} \text{SEAL} \\\\ \text{ReBalance} \\\\ \text{Dr.CoT (ours)} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{77.50} & \hphantom{\text{Tokens}}\llap{29{,}324} \\\\ \hphantom{00.00}\llap{76.67} & \hphantom{\text{Tokens}}\llap{30{,}453} \\\\ \hphantom{00.00}\llap{\mathbf{81.67}} & \hphantom{\text{Tokens}}\llap{\mathbf{27{,}247}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{59.09} & \hphantom{\text{Tokens}}\llap{30{,}650} \\\\ \hphantom{00.00}\llap{\mathbf{61.36}} & \hphantom{\text{Tokens}}\llap{31{,}039} \\\\ \hphantom{00.00}\llap{56.82} & \hphantom{\text{Tokens}}\llap{\mathbf{27{,}664}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{80.81} & \hphantom{\text{Tokens}}\llap{23{,}455} \\\\ \hphantom{00.00}\llap{83.33} & \hphantom{\text{Tokens}}\llap{24{,}004} \\\\ \hphantom{00.00}\llap{\mathbf{85.35}} & \hphantom{\text{Tokens}}\llap{\mathbf{23{,}083}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{\mathbf{27.59}} & \hphantom{\text{Tokens}}\llap{23{,}707} \\\\ \hphantom{00.00}\llap{26.72} & \hphantom{\text{Tokens}}\llap{24{,}887} \\\\ \hphantom{00.00}\llap{24.71} & \hphantom{\text{Tokens}}\llap{\mathbf{20{,}855}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{61.25} & \hphantom{\text{Tokens}}\llap{26{,}784} \\\\ \hphantom{00.00}\llap{62.02} & \hphantom{\text{Tokens}}\llap{27{,}596} \\\\ \hphantom{00.00}\llap{\mathbf{62.14}} & \hphantom{\text{Tokens}}\llap{\mathbf{24{,}712}} \end{array} \\\\ \hline \end{array}$$ As shown in Table 2, Dr.CoT achieves 0.89 percentage points higher overall accuracy than SEAL with 7.74% fewer generated tokens. Compared with ReBalance, Dr.CoT achieves comparable overall accuracy with 10.45% fewer tokens. Accordingly, we have added these comparisons to Table 1 of the revised manuscript. --- ***W3: Could part of the accuracy gain come from avoiding truncation and missing boxed answers rather than better reasoning?*** Vanilla and Dr.CoT share the same final-answer protocol and reserved token budget. When Vanilla reaches its reasoning budget, the decoder inserts the fixed answer prefix `The boxed answer is \boxed{`, and the model uses the reserved tokens for final-answer generation. Dr.CoT can enter this same phase earlier when its stopping criterion is met. Thus, even when the 2B model reaches its reasoning limit, Vanilla still enters the final-answer phase with the same reserved budget as Dr.CoT.
AuthorsOpenReview ↗

Response to Reviewer HTQK (Part 3 / 3)

W4: Activating a prompt feature does not necessarily mean that the corresponding reasoning step has been correctly resolved.

We agree that a model can activate a prompt feature even when the corresponding reasoning step is incorrect. Let FxF_x denote the prompt features and GtG_t the accumulated reasoning features over the first tt reasoning tokens. For ∥Fx∥1>0\lVert F_x\rVert_1 > 0, Eq. (4) defines the coverage ratio as ct=∑j=1Mmin⁡(Fx,j,Gt,j)∥Fx∥1c_t=\frac{\sum_{j=1}^{M}\min(F_{x,j},G_{t,j})}{\lVert F_x\rVert_1}. An incorrect step can therefore increase ctc_t if it adds activation to a prompt feature whose accumulated activation is still below its value in the prompt.

However, coverage can still provide useful guidance for regulating the reasoning trajectory. Correctly applying a condition or carrying out a calculation depends on the reasoning ability of the backbone, but the same model can still overthink by repeatedly checking the same information or continuing to reason after reaching a correct answer. Dr.CoT keeps the backbone frozen and regulates the direction and length of the reasoning trajectory without explicitly verifying or correcting individual steps. Residual feature steering uses uncovered prompt features to guide subsequent reasoning. Coverage-guided sufficiency diagnosis switches the model to final-answer generation when the coverage ratio is high and the coverage gain remains low across consecutive windows.


W5: The four-sample point estimates lack confidence intervals or significance analysis.

We provide sample standard deviations (SDs) for accuracy and 95% confidence intervals (CIs) for the accuracy difference between Dr.CoT and Vanilla. Each method generates four responses per question on every benchmark, following the paper’s evaluation protocol. CIs are computed using Welch’s t procedure across the four sampling rounds on the fixed benchmarks.

Table 3. Accuracy and uncertainty across Qwen3.5 backbones. Accuracy is reported as mean ± SD (%), with gains and 95% CIs in percentage points. For each backbone, accuracy is averaged equally across the four benchmarks.

BackboneVanilla Acc. ↑Dr.CoT Acc. ↑Accuracy gain95% CI of gainToken reduction
Qwen3.5-2B24.10 ± 0.9027.69 ± 1.35+3.59[1.54, 5.64]31.49%
Qwen3.5-9B59.68 ± 0.4962.14 ± 0.68+2.46[1.42, 3.51]11.24%
Qwen3.5-27B71.28 ± 0.9471.73 ± 0.89+0.46[−1.13, 2.04]8.30%
Overall51.68 ± 0.2753.85 ± 0.39+2.17[1.58, 2.77]17.01%

As shown in Table 3, Dr.CoT improves mean accuracy by 2.17 percentage points (95% CI [1.58, 2.77]) with an average token reduction of 17.01% across the three backbones. For Qwen3.5-27B, the main observed benefit is an 8.30% reduction in generated tokens, with an estimated accuracy gain of 0.46 percentage points (95% CI [−1.13, 2.04]).

Detailed benchmark-wise results with standard deviations are provided in Appendix F.1 (Table 12) of the revised manuscript.


References

[1] DeepSeek. DeepSeek-R1-Distill-Llama-8B. https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B, 2025.

[2] qresearch. DeepSeek-R1-Distill-Llama-8B-SAE-l19. https://huggingface.co/qresearch/DeepSeek-R1-Distill-Llama-8B-SAE-l19, 2025.

[3] Runjin Chen et al. SEAL: Steerable Reasoning Calibration of Large Language Models for Free. In Conference on Language Modeling (COLM), 2025.

[4] Yulin Li et al. Efficient Reasoning with Balanced Thinking. In International Conference on Learning Representations (ICLR), 2026.


We are encouraged by your recognition of the strengths of our work. In light of the additional experiments and clarifications provided above, we respectfully ask whether you might reconsider your rating. Thank you again for your time and thoughtful engagement; we welcome any further questions or discussion.

Dear Reviewer HTQK,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards,

The authors

Reviewer_zWirOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Dr.CoT, a framework for diagnosing and regulating CoT reasoning via prompt-conditioned sparse feature coverage. Using a Sparse Autoencoder, it projects both the prompt and reasoning trajectory into a shared sparse feature space, decomposing prompt features into covered and uncovered components. It adaptively stops reasoning when coverage is sufficient and gains remain persistently low, and uses uncovered prompt features to steer the model toward unresolved information. Experiments on Qwen3.5 with different parameters across 4 benchmarks show Dr.CoT reduces average token usage while improving average accuracy over standard CoT.

Critical strengths
  1. Using prompt-conditioned sparse feature coverage as an explicit measure of reasoning progress is genuinely novel. Unlike trajectory-derived signals, coverage directly measures how much prompt-relevant information has been addressed, providing a more principled stopping criterion.

  2. The three components form a well-integrated system: feature modeling provides the representation, coverage-guided diagnosis provides adaptive stopping, and residual feature steering provides trajectory correction. The mathematical formulation is clean with formal proofs of coverage properties (monotonicity, boundedness) in the appendix.

  3. The evaluation spans three model scales and four benchmarks, with particularly well-designed ablation studies covering component contributions, prompt reference specificity, steering direction, SAE training budget , and inference-time cost. The four case studies provide intuitive understanding of how the method operates in practice.

  4. The paper addresses real-world deployability by showing that as few as 0.1M SAE training tokens yield reasonable results, and that Dr.CoT achieves a 7.2% FLOPs reduction with only a little throughput decrease and additional memory overhead.

Critical weaknesses
  1. AIME 2026 and HMMT 2026are very small. The accuracy difference between Dr.CoT and Vanilla on AIME-2B represents roughly 0.5 problems of 30. No confidence intervals, standard deviations, or significance tests are provided, making it difficult to assess whether reported improvements are statistically meaningful.

  2. On Qwen3.5-27B, Dr.CoT improves limited overall accuracy. While the token reduction is still meaningful, the near-zero accuracy improvement on the largest model raises questions about whether the approach provides meaningful benefits at scale. This trend is not discussed in depth.

  3. The paper does not evaluate on other reasoning domains such as code generation, commonsense reasoning, or multi-hop QA. Since the coverage metric is in principle domain-agnostic, demonstrating generalization beyond STEM would significantly strengthen the contribution.

  4. The paper claims that uncovered features represent "unresolved prompt-relevant information," but this claim is only indirectly supported by the shuffled-prompt ablation. A more direct validation could involve identifying the top uncovered features at various reasoning stages and examining their semantic meaning through SAE feature interpretation.

Questions for authors
  1. Could you provide confidence intervals or significance tests for the main results, especially on the smaller benchmarks ( such as AIME with 30 problems)?

  2. On Qwen3.5-27B, the accuracy improvement is quite limited. Do you expect the method to provide meaningful benefits at even larger scales? What explains the diminishing returns?

  3. How was the SAE layer (20 for 2B, 28 for 9B, 60 for 27B) selected? Were other layers tested? What is the sensitivity to layer choice?

  4. Can you provide any analysis of what the "uncovered features" semantically represent at different stages of reasoning? This would help validate the claim that they capture "unresolved prompt-relevant information."

4 comments
AuthorsOpenReview ↗

Response to Reviewer zWir (Part 1 / 3)

Thank you for recognizing the novelty of prompt-conditioned sparse feature coverage and for your thoughtful suggestions. We have added experiments and analyses to provide further evidence for our findings. Below, we respond to your comments and questions:


W1 / Q1: Could you provide confidence intervals or significance tests for the main results, especially on the smaller benchmarks?

We report standard deviations (SDs) and 95% confidence intervals (CIs) for the comparison between Vanilla and Dr.CoT. Both methods generate four responses per question on every benchmark, following the paper’s evaluation protocol. The 95% CIs are computed using Welch’s t procedure across four sampling rounds on the fixed benchmarks.

Table 1. Accuracy and uncertainty across Qwen3.5 backbones. Accuracy is reported as mean ± SD (%), with gains and 95% CIs in percentage points. For each backbone, accuracy is averaged equally across the four benchmarks.

BackboneVanilla Acc. ↑Dr.CoT Acc. ↑Accuracy gain95% CI of gainToken reduction
Qwen3.5-2B24.10 ± 0.9027.69 ± 1.35+3.59[1.54, 5.64]31.49%
Qwen3.5-9B59.68 ± 0.4962.14 ± 0.68+2.46[1.42, 3.51]11.24%
Qwen3.5-27B71.28 ± 0.9471.73 ± 0.89+0.46[−1.13, 2.04]8.30%
Overall51.68 ± 0.2753.85 ± 0.39+2.17[1.58, 2.77]17.01%

As shown in Table 1, Dr.CoT improves mean accuracy by 2.17 percentage points (95% CI [1.58, 2.77]) with an average token reduction of 17.01% across the three backbones.

Detailed results are provided in Appendix F.1 (Table 12) of the revised manuscript. For example, on AIME 2026 with Qwen3.5-9B, Dr.CoT improves mean accuracy from 73.33 ± 2.72% to 81.67 ± 1.92%.


W2 / Q2: On Qwen3.5-27B, the accuracy improvement is limited. What explains the diminishing returns, and is the method useful at larger scales?

On the same benchmarks, larger models may be less likely to deviate from the prompt during reasoning. Dr.CoT may therefore have fewer opportunities to correct such deviations through residual feature steering and improve accuracy. This is one possible explanation for the smaller accuracy gain observed on Qwen3.5-27B. At this scale, the main observed benefit is improved reasoning efficiency, with an 8.3% reduction in token consumption. Coverage-guided sufficiency diagnosis helps reduce unnecessary reasoning associated with overthinking.

As task complexity increases, even larger models may gradually deviate from the prompt over a long reasoning trajectory. In long-horizon agent tasks, for example, a model must adhere to the constraints in the original prompt across multiple steps, yet it may overlook one of them at a later step. In such tasks, steering subsequent reasoning with uncovered prompt features may become more important and potentially yield larger accuracy gains. Moreover, even after gathering enough information to complete a task, an agent may continue reasoning or repeatedly call tools to verify the same result. Extending coverage-guided sufficiency diagnosis to reduce this overhead is therefore a direction for future work.

AuthorsOpenReview ↗

Response to Reviewer zWir (Part 2 / 3)

W3: The paper does not evaluate other reasoning domains such as code generation, commonsense reasoning, or multi-hop QA.

We evaluated Dr.CoT on HumanEval [1] for code generation, StrategyQA [2] for commonsense multi-hop reasoning, and three BIG-bench [3] tasks covering argument evaluation, social reasoning, and counterfactual reasoning.

All experiments used Qwen3.5-9B with the paper’s Dr.CoT settings. Within each task, Vanilla and Dr.CoT shared identical prompts, decoding parameters, and answer budgets. Both methods generated four responses per question.

Table 2. Cross-domain evaluation on Qwen3.5-9B. Acc (%) ↑ and Tokens ↓ denote mean accuracy and mean generated tokens, respectively.

\hline \begin{array}{l} \vphantom{\text{BIG-bench}} \\\\ \begin{array}{l} \text{Method} \end{array} \end{array} & \begin{array}{c} \text{HumanEval} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{StrategyQA} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} & \begin{array}{c} \text{BIG-bench} \\\\ \hline \begin{array}{rr} \hphantom{00.00}\llap{\text{Acc}} & \text{Tokens} \end{array} \end{array} \\\\ \hline \begin{array}{l} \text{Vanilla} \\\\ \text{Dr.CoT} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{97.56} & \hphantom{\text{Tokens}}\llap{16{,}408} \\\\ \hphantom{00.00}\llap{\mathbf{98.17}} & \hphantom{\text{Tokens}}\llap{\mathbf{13{,}369}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{80.35} & \hphantom{\text{Tokens}}\llap{19{,}252} \\\\ \hphantom{00.00}\llap{\mathbf{81.22}} & \hphantom{\text{Tokens}}\llap{\mathbf{18{,}965}} \end{array} & \begin{array}{rr} \hphantom{00.00}\llap{92.25} & \hphantom{\text{Tokens}}\llap{8{,}937} \\\\ \hphantom{00.00}\llap{\mathbf{95.35}} & \hphantom{\text{Tokens}}\llap{\mathbf{6{,}942}} \end{array} \\\\ \hline \end{array}$$ As shown in Table 2, Dr.CoT achieves higher accuracy with fewer generated tokens across all three evaluations. Token consumption is reduced by 18.5% on HumanEval and 22.3% on BIG-bench. These results support Dr.CoT’s applicability beyond mathematical and scientific reasoning without task-specific hyperparameter tuning. --- ***Q3: How was the SAE layer selected, were other layers tested, and how sensitive is Dr.CoT to the layer choice?*** We selected the first linear-attention layer in the final group to capture prompt-relevant information in late-layer representations while leaving subsequent layers to integrate residual feature steering. To assess sensitivity, we evaluated six additional layers on Qwen3.5-9B using AIME 2026. All results are averaged over four samples per question. **Table 3. Sensitivity to SAE layer selection on Qwen3.5-9B and AIME 2026. Acc (%) and Tokens are averaged over four samples per question.** | Method | SAE layer | Acc (%) ↑ | Tokens ↓ | |---|---:|---:|---:| | Vanilla | — | 73.33 | 30,574 | | Dr.CoT | 16 | 75.00 | 30,165 | | Dr.CoT | 20 | 78.33 | 28,907 | | Dr.CoT | 24 | 76.67 | 28,875 | | Dr.CoT (main setting) | 28 | 81.67 | 27,247 | | Dr.CoT | 29 | 79.17 | 27,969 | | Dr.CoT | 30 | 76.67 | 28,303 | | Dr.CoT | 31 | 78.33 | 27,234 | As shown in Table 3, Dr.CoT achieves higher accuracy with fewer generated tokens than Vanilla at every tested layer, indicating robust performance across layers. We have added the layer sensitivity results and analysis to the “SAE Layer Selection” paragraph in Section 4.3 of the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer zWir (Part 3 / 3)

W4 / Q4: What do uncovered features semantically represent at different stages of reasoning?

We analyzed a Dr.CoT reasoning trajectory from Qwen3.5-9B on HMMT problem 22. The geometry problem specifies angle constraints and three segment lengths: HO=20HO=20, SE=26SE=26, and OS=10OS=10. Table 4 shows the coverage of two SAE features, F54689 and F57301, at different reasoning stages.

Table 4. Reasoning progress and cumulative feature coverage in HMMT problem 22.

Reasoning prefix (tokens)Reasoning progressF54689 coverageF57301 coverage
1–128Lists angle constraints; the given segment lengths have not been addressed.0.0%3.2%
1–512Derives HO∥ESHO\parallel ES from perpendicularity; the given segment lengths have only been restated.30.0%43.7%
1–768Uses HO=20HO=20 to set O=(20,0)O=(20,0) and introduces a distance expression for SESE.75.7%100.0%
1–1024Uses SE=26SE=26 to consider S=(26,yE)S=(26,y_E).78.8%100.0%
1–2048Continues examining the resulting coordinate and angle relationships.94.7%100.0%

As shown in Table 4, the trajectory first addresses angle constraints and then incorporates the given segment lengths into the derivation. Independent high-activation contexts associate F54689 with specified length values and F57301 with magnitude and distance. In this case, F54689’s prompt activation is entirely localized to the three given segment lengths, while F57301 activates when the trajectory expresses SESE as ∣xS∣|x_S|.

At 512 reasoning tokens, the given segment lengths have been restated but not yet used in the derivation, and 70.0% of F54689’s prompt activation remains uncovered. Its coverage subsequently increases as the trajectory uses HO=20HO=20 and SE=26SE=26 in the coordinate derivation.

This case study provides qualitative evidence that uncovered prompt features can reflect unresolved prompt-relevant information at different reasoning stages.


References

[1] Mark Chen et al. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374, 2021.

[2] Mor Geva et al. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics, 2021.

[3] Aarohi Srivastava et al. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. Transactions on Machine Learning Research, 2023.


We appreciate your thoughtful assessment and hope that the additional analyses and experiments have addressed your concerns. We welcome any further questions or discussion.

Dear Reviewer zWir,

I hope this message finds you well. As the discussion period is nearing its end, I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the Area Chair and all three reviewers—zWir, HTQK, and YmrT—for their thoughtful and constructive feedback. We are encouraged by their recognition of Dr.CoT's prompt-conditioned feature modeling for sufficiency diagnosis and trajectory steering.

We have carefully responded to all reviewer comments. Below, we summarize the core contributions of our study and the principal updates made during the rebuttal.


Core Contributions of Our Work

  1. Prompt-Conditioned Feature Modeling: Dr.CoT represents the prompt and reasoning trajectory in a shared sparse feature space using a sparse autoencoder (SAE), explicitly quantifying reasoning progress by measuring how the generated CoT covers prompt-relevant information.
  2. Coverage-Guided Sufficiency Diagnosis: Dr.CoT tracks feature coverage and its gain to diagnose reasoning sufficiency, adaptively stopping reasoning when coverage is sufficient and gains remain persistently low.
  3. Residual Feature Steering: Dr.CoT decodes uncovered prompt features into a residual-stream direction to guide reasoning toward unresolved prompt-relevant information and correct reasoning drift, without task-specific fine-tuning.

Updates of Experimental Results During Rebuttal

  • Generalization Across Backbones and Domains: Extended evaluation to DeepSeek-R1-Distill-Llama-8B with a pretrained ReLU SAE. Dr.CoT achieves an overall accuracy gain of 2.18% with 5.38% fewer tokens than Vanilla. Added cross-domain evaluations on HumanEval, StrategyQA, and three BIG-bench tasks on Qwen3.5-9B. Dr.CoT achieves an overall accuracy gain of 1.53% with 11.93% fewer tokens than Vanilla.
  • Steering and Stopping Comparisons: Added comparisons with SEAL and ReBalance on Qwen3.5-9B. Dr.CoT achieves higher overall accuracy with 7.74%/10.45% fewer tokens, respectively. Added a fixed-budget comparison on AIME 2026. Dr.CoT without steering achieves an accuracy gain of 4.17% over Vanilla with comparable token usage.
  • SAE Layer Sensitivity: Added evaluations at six additional SAE layers on Qwen3.5-9B and AIME 2026. Dr.CoT achieves higher accuracy with fewer tokens than Vanilla across all seven tested layers.
  • Statistical Analysis: Added accuracy standard deviations and 95% confidence intervals across four sampling rounds on the fixed benchmarks. Across three Qwen3.5 backbones, Dr.CoT's overall accuracy gain over Vanilla has a 95% CI of [1.58, 2.77] percentage points, with an average token reduction of 17.01%.

Updates of In-Depth Discussions During Rebuttal

  • Interpretation of Feature Coverage: Clarified how uncovered features can reflect unresolved prompt-relevant information through a geometry case study, and explained how changes in feature coverage relate to the use of prompt constraints during reasoning.
  • Evaluation Fairness: Clarified that Vanilla and Dr.CoT share the same final-answer protocol and reserved answer budget. Both methods enter the final-answer phase under these shared conditions for a fair comparison.
  • Analysis of Reasoning Dynamics and Model Scale: Analyzed how continued reasoning can introduce errors after a correct intermediate result. Further discussed the role of coverage-guided sufficiency diagnosis in reducing unnecessary reasoning across model scales and its potential extension to long-horizon agent tasks.

In response to the reviewers' feedback, we have revised the manuscript to strengthen the experimental evaluation and clarify the method. We hope these revisions help address their concerns, and sincerely thank the Area Chair and reviewers for their time and effort.

We respectfully look forward to their favorable consideration and remain grateful for their valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.