acceptodds
Under review as a conference paper at ICLR 2027

ASTra: Mitigating overthinking via tracking attention sink

19 views

Abstract

The overthinking problem has recently attracted significant attention from researchers. While the introduction of chain of thought substantially enhances the model's ability to solve complex tasks, the overthinking problem emerges when solving relatively simple questions. In these cases, overthinking can be observed as repeated verifications and divergent thoughts, which do not translate into improved problem-solving, yet generate considerable computational overhead. In this paper, we propose ASTra (Attention Sink Tracker), an inference-time method to mitigate overthinking problems in LLM. By monitoring attention weight distribution, we discovered an optimal truncation point to prompt models to terminate the chain of thought. Extensive experiments show that our method reduces 15% to 70% of chain of thought tokens, while maintaining final output quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM overthinking, Attention mechanism, optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MLgcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ASTra, a training-free, inference-time method to mitigate overthinking in LLMs. ASTra dynamically identifies attention sinks, tracks their cumulative attention weights, and triggers early termination when a threshold is exceeded, aiming to reduce redundant chain-of-thought (CoT) tokens without sacrificing answer quality. The experimental validation shows the effectiveness of the proposed method.

Critical strengths
  1. ASTra uses attention sink saturation as an internal signal to identify when continued reasoning may yield diminishing returns. This offers a conceptually distinct alternative to existing stopping strategies based on fixed token budgets or output-level signals, providing a potentially useful perspective on inference-time reasoning control.

  2. Experiments across multiple models and diverse reasoning and knowledge benchmarks report significant token reductions while largely preserving task performance. The ablations and statistical analyses further support the effectiveness of the approach. Although monitoring introduces computational overhead, the substantial reduction in generated tokens suggests practical potential for improving inference efficiency without additional training.

Critical weaknesses
  1. ASTra assumes that attention sink saturation indicates diminishing returns from continued reasoning, but this relationship does not necessarily establish that the remaining reasoning tokens are redundant or can be safely removed. Without stronger evidence linking the proposed signal to reasoning progress across different tasks and models, the stopping criterion may terminate useful reasoning and limit the reliability of the approach.
Questions for authors
  1. Could the authors provide evidence that attention sink saturation reliably indicates diminishing returns from continued reasoning, rather than merely correlating with generation length? In particular, how does ASTra perform when the stopping threshold is varied across tasks and models, and how often does it truncate reasoning before a correct answer is reached?

  2. Can ASTra be adapted to non-softmax attention mechanisms, where attention weights may not sum to one and conventional attention sinks may not emerge?

Reviewer_wdJKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

ASTra aims to reduce overthinking in LLMs by tracking attention sinks, which are tokens that receive a large share of attention. When the tracked signal reaches a threshold, ASTra inserts a prompt that asks the model to stop reasoning and give its answer. The paper connects this signal to reduced reasoning progress and reports token savings with small changes in answer quality across six models and four main benchmarks.

Critical strengths
  • The paper proposes an interesting way to decide when to stop reasoning. It uses information from the model’s own attention rather than a separate model that judges the answer. This could be a useful approach to reducing unnecessary computation.

  • The evaluation looks at more than token savings. It includes several models, actual latency measurements, comparisons of stopping signals using the same intervention, failure analysis, and a transfer test on Omni-MATH. These checks help explain the method’s practical value.

Critical weaknesses
  • The presentation and the main line of reasoning need to be clearer. The paper moves from observations about attention sinks to claims about reasoning saturation and optimal stopping, but the steps between these claims are hard to follow. It is often unclear whether a statement is a measured result, an assumption, or a proved claim. Important details are also spread across the main text and several appendices. The paper should state its main argument in a clear order and connect each claim to its evidence. This is more than a wording issue: the current structure makes it difficult to judge how much the theory and experiments actually establish.

  • The theory does not fully explain the actual stopping rule. A bound on attention outputs does not by itself show that the full model has stopped making useful progress. The model also uses residual connections and MLPs, while ASTra combines information from several layers and heads and uses historical attention averages. It is unclear how far the theory covers these parts of the method. Small changes at one step also do not rule out useful reasoning later. In addition, stable sink weights do not establish that redundant reasoning builds sink mass faster than useful reasoning. The authors should narrow the theoretical claims to what is shown, or add evidence for the missing links. The empirical method may still be useful even if the theory serves mainly as motivation.

  • The tested tasks cover a limited range of reasoning problems. The main benchmarks focus on math and factual or scientific question answering. Testing several models is useful, but it does not show whether the stopping signal works across different task types. The Omni-MATH transfer test adds useful evidence within math, but does not substantially broaden task coverage. Results on tasks such as code generation with execution tests, long-document or multi-document reasoning, and planning would help establish the method’s scope. Tasks that require long intermediate reasoning or later error correction are especially relevant, since stopping too early could remove necessary work. If broader tests are not feasible, the claims should be limited to the tested task types.

  • The tuning setup makes the baseline comparison harder to interpret. ASTra receives task-dependent tuning, while competing methods use settings from their original papers. The results therefore do not clearly separate the value of the attention-sink signal from the benefit of choosing better settings for each task. Giving the baselines a similar tuning budget and comparing methods at similar quality or cost would make the claimed advantage more convincing. Alternatively, the claims should be limited to the tested settings.

Questions for authors
  1. What does the theory guarantee when residual connections, MLPs, multiple layers and heads, and historical attention averages are taken into account? Which parts of the claim that further reasoning has little value are proved, and which remain assumptions?
  2. Does the attention-sink signal remain useful for code generation, long-document or multi-document reasoning, and planning? Are there results on tasks that require long intermediate reasoning before reaching an answer?
  3. Appendix L pools GPQA results from Qwen3-8B, 14B, and 32B and reports 6.9% wrong-to-correct changes and 7.1% correct-to-wrong changes. Can this analysis be broken down by model and dataset, with results by question difficulty and type? In particular, are correct-to-wrong changes more common on questions that require longer reasoning?
  4. Can the authors show a concrete CoT example where ASTra stops too early, while the baseline continues and reaches the correct answer? A side-by-side comparison should mark ASTra’s stopping point and the later reasoning steps that the baseline needed to solve the problem. This would help explain when the stopping signal mistakes unfinished reasoning for overthinking.
Reviewer_4LTmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Attention Sink Tracker (ASTra), an inference-time method for shortening chain-of-thought reasoning without additional model training. It identifies attention sinks using attention concentration and stability over a sliding window, aggregates their attention mass separately for prompt and generated tokens, and inserts a stopping prompt when the decode-side sink ratio exceeds a threshold. The motivation connects sink concentration with reduced reasoning progress through attention interventions, confidence diagnostics, and a theoretical bound on attention outputs.

Experiments evaluate six models on mathematical reasoning, factual question answering, and graduate-level scientific questions. The paper reports reductions in generated tokens with mostly small changes in answer quality, alongside latency measurements, alternative stopping-signal comparisons, threshold sensitivity, cross-task transfer, prompt ablations, and failure analysis.

Critical strengths
  1. Monitoring attention concentration and persistence provides a plausible internal signal for adaptive early stopping without additional training. The intervention in Algorithm 3 reuses the ongoing generation stream rather than starting a separate answer-generation pass, making the approach practically appealing for models whose attention statistics are accessible.
  2. The efficiency evaluation measures end-to-end latency rather than equating token savings with speedup. Table 4 explicitly reports reduced token-generation throughput from monitoring overhead, while still showing lower per-query latency, which provides useful evidence that the shorter outputs can outweigh the additional computation in the measured settings.
  3. The cross-task experiment in Table 7 applies a GPQA-calibrated threshold to Omni-MATH without recalibration and reports substantial token savings with a small aggregate accuracy change. This offers a useful transfer check beyond independently tuning each benchmark, while the difficulty breakdown makes the nonuniform effects visible.
Critical weaknesses
  1. The single-layer geometric bound can hold under its assumptions without establishing the stronger claim that further reasoning has no value. In Appendix B.6, the final attention output is treated as the representation directly mapped to logits, without accounting for the residual and feed-forward pathways that also determine the model's output. Moreover, stable attention to individual sink tokens does not establish a fixed weighted sink centroid across decoding steps, which the subsequent marginal-gain argument requires. The concern is therefore the extension of a conditional geometric result to reasoning stagnation and optimal stopping in the full model, rather than the geometric bound itself.
  2. The implemented sink statistic does not exactly match the probability-mass partition used in the theoretical justification. Algorithm 2 replaces the current first-token attention with its cumulative historical average while retaining current-step weights for other sinks. Consequently, the combined prefill and decode quantities need not form a subset of one normalized attention distribution, and the reported decode ratio is not necessarily bounded by one. Without a bound on this smoothing discrepancy, the claimed exact correspondence between the implemented monitor and the decode-conditional theoretical bound remains uncertain.
  3. The reported accuracies are difficult to reconcile with the stated evaluation protocol, beyond a single anomalous entry. In Table 2, none of the twelve Baseline and ASTra accuracies for either MATH500 or GPQA can arise, at the reported precision, from a single binary-scored evaluation on the stated 500 or 448 examples, respectively. Baseline accuracies also differ substantially across the main, threshold-sensitivity, and stopping-signal experiments. Different subsets, configurations, or aggregation rules could explain these differences, but their correspondence is insufficiently documented to audit the reported gains and paired statistical analysis.
  4. Appendix H.1 interprets nonsignificant paired tests as support for near-lossless output quality, but not rejecting equality does not establish equivalence or non-inferiority within the stated tolerance. The reported p-values do not bound the possible accuracy or F1 degradation, particularly on smaller evaluation sets. Consequently, the statistical evidence supports a more limited conclusion than the claim that quality preservation has been established.
  5. The stopping-signal comparison in Figure 7 does not separate stopping-signal quality from truncation intensity. The entropy and fixed-length controls use fewer tokens and intervene more frequently than ASTra, so their lower accuracy is also compatible with more aggressive truncation. Sharing the inserted stopping prompt controls the intervention wording, but the unmatched operating points do not establish a better accuracy-cost trade-off for the attention-based signal.

Paper Formatting and Clarity Concerns

  1. Several labels and equations in Figure 5 are small at normal page scale, making the monitoring and threshold calculations difficult to inspect.
  2. The caption of Table 6 states that accuracy remains within 1.5 percentage points of the baseline throughout the sweep, but the first threshold setting decreases accuracy from 0.825 to 0.785, a 4-point drop. The subsequent paragraph excludes this setting, so the caption overstates the demonstrated robustness.
  3. Table 12 and its accompanying discussion state that token reductions are significant in 20 of 24 settings, but the table contains five nonsignificant settings, leaving 19. This counting discrepancy makes the statistical summary inconsistent with the reported results.
  4. Table 14 aggregates results over four Qwen3 models, but its group counts sum to 534, and the counting unit and its relationship to the 448 GPQA examples in Table 9 are unclear. In the hardest group, the displayed accuracies decrease from 0.538 to 0.512, implying approximately -2.6 percentage points rather than the reported -2.06.
Questions for authors
  1. What sample counts, scoring and aggregation rules, and evaluation configurations underlie the MATH500 and GPQA results? In particular, why is the Qwen3-8B MATH500 baseline 93.21% in Table 2 but 82.5% in Table 6, and its GPQA baseline 55.84% in Table 2 but 47.1% in Figure 7? How do these evaluations map to the paired observations in Appendix H and the pooled counts in Table 14?
  2. How does the representation bound extend to the complete residual-stream computation, and what evidence supports the stationary sink-centroid assumption across decoding steps?
  3. How is the discrepancy between the historical first-token average and current attention mass handled? Can the implemented decode ratio exceed one, and is any clipping or renormalization applied?
  4. Can paired confidence intervals or a non-inferiority analysis establish that quality degradation remains within the stated tolerance, rather than merely failing to reject equality?
  5. How do the stopping signals compare along accuracy-token trade-off curves at multiple matched average token budgets? Which held-out data and selection criteria determine their thresholds, including the GPQA calibration used in the main evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.