acceptodds
Withdrawn from ICLR 2027

Transition Thinking: Training-Free Prediction Arbitration for Reasoning Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training-free methods such as Soft Thinking seek latent reasoning in a frozen model by feeding probability-weighted mixtures of token embeddings back into the Transformer. However, their advantage over chain-of-thought is inconsistent across models and benchmarks. We find that the soft mixture usually stays near its dominant token embedding and rarely changes the next generated tokens, so Soft Thinking behaves much like argmax decoding. This helps explain why its gains vary. Argmax trajectories can be as accurate as sampled ones when they finish but are vulnerable to repetition loops. Soft Thinking terminates looping generations with Cold Stop, yet these generations are often incorrect. Layerwise next-token predictions reveal a transition at a specific intermediate layer, where entropy falls sharply while predictions still differ from the final layer. At this layer, hesitation and repetition loops produce opposite confidence patterns in the two layers. When the final layer hesitates, the transition layer can hold a confident alternative, whereas inside a loop the final layer is nearly deterministic and the transition layer is diffuse. We propose *Transition Thinking*, which keeps argmax decoding for thinking, arbitrates hesitations with the transition-layer prediction, and controls repetition with a thinking-time penalty and an in-place escape from verified loops. The transition layer is selected once per model from an unlabeled profile, and answer generation is unchanged. Across three Qwen3 models, Transition Thinking improves average accuracy over chain-of-thought on mathematics and science by 0.5 to 2.3 points, stays close to it on coding tasks, and shortens correct generations. It requires no training, additional forward passes, or task-specific layer search.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent Reasoning, Inference-Time Intervention, Training-Free Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w97eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a new decoding scheme "Transition Thinking". Their method is based on insights they find through experiments which span:

  • Evaluation of latent reasoning methods at inference time (no finetuning)
  • Intermediate layer decoding
  • Loop detection in CoT

The paper starts by evaluating soft thinking methods and find uncertainty is very low in the thinking states corroborating previous results in the literature.

The paper then shows that there is a critical layer ℓ⋆\ell^\star in models at which confidence shoots up (i.e. entropy goes down sharply). They find that this phenomena has correlation with looping: when the final layer hesitates, ℓ⋆\ell^\star sometimes holds a confident alternative token. Inside repetition loops, they find the opposite (final layer certain, ℓ⋆\ell^\star diffuse).

Finally, the paper presents In-place Collapse Escape (ICE) as a novel method to detect and mitigate loops in the CoT. Their method is a simple algorithm which monitors if entropy has been low (under a threshold) for some amount of time. If it is the case, the method checks for repetitions of strings up to a period of about ~1024. If a loop is indeed detected, the method forces random (almost uniform) sampling of top tokens to break the loop.

Together the paper's results shed light on different aspects which play a part in test-time decoding in the context of reasoning tasks.

Critical strengths

The main strength of the paper for me is the novelty of the finding/approach. The authors cleverly combine insights from three different subfields of LLM reasoning (latent thinking, intermediate layer decoding, repetition detection).

Critical weaknesses
  1. I found the paper hard to read. The paper strings together findings from different subfields of LLM reasoning (latent reasoning, intermediate layer decoding, repetition detection). This culminates in the novel method they present. However, upon first read (especially of the intro) I found that it is hard to identify the research question/gap in the literature highlighted by the paper. It feels like multiple disjoint findings are reported. I think a sharper narrative could help this out. I feel a sharper intro is not a far cry away; the findings culminating in the method proposed already hint at this.

  2. The method, although interesting, does not improve performance meaningfully compared to the proposed baselines. Results reported in Table 2 show that performance increase is only about 1-2%. Moreover, when performing component analysis in Table 5, just using repetition penalty (RP + CoT) seems to have a huge effect on performance, even beating TT on AIME26 for Qwen 8B.

Questions for authors
  • what is rtr_t? This is defined nowhere in the main text.
  • The analysis is only done on Qwen models, which are known to exhibit specific phenomena not found in other model families. Would it be possible for the authors to report/confirm that their findings replicate on other LLM families?
  • Could the authors report how many loops are avoided compared to e.g. cold stop or even just how many loops avoided compared to simple argmax baseline? It would be interesting to have a principled testing of ICE in isolation, not just results which report increase in accuracy, but also results which report meaningful decrease in loop count.
  • Many parameters (1-nat threshold, ½ factor, ρ = 1.05, 0.01 nat, 256 steps, 64 tokens, top-20, 3 escapes, 90% match) are not ablated over. Were these hard to tune? It is hard to get a sense of how finicky the method is to tune without this information.
  • The ablations in Table 5 do not discuss length. Could the authors provide the length of the generations for the component-wise analysis? This would be interesting as it would shed light on which components are most important to driving the token count down.
Reviewer_hDkSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces "Transition Thinking," a method to help language models reason better without needing extra training. The authors noticed that existing "soft thinking" methods often act like basic decoding and get stuck in repetitive loops. Their solution uses confident predictions from a middle layer of the AI to guide the final output, while also including built-in tricks to break out of those loops.

Critical strengths
  • Author makes great observations about why current training-free methods get stuck in loops or fail to improve reasoning.
  • This method doesn't require retraining the model, making it very easy to plug into existing systems, I hope they also designed the codebase to be model agnostic.
  • The ablations are useful to see how individual part of their method contributes to the overall result.
Critical weaknesses
  • The idea is not very original, as older methods (like DoLa : https://sls.csail.mit.edu/publications/2024/Yung-SungChuang-1_ICLR-2024.pdf and Confident Layer Decoding: https://arxiv.org/pdf/2606.21906) already use middle layers to fix final predictions.
  • The actual performance improvements are very small for example, it only beat a simple baseline by 0.1% in one of the tests.
  • There isn't enough statistical proof to show these small gains are reliable and not just a fluke.
  • Authors only tested it on one specific model family (Qwen3), so it's unclear if this works for other architectures.
Questions for authors
  1. How exactly does your approach make different choices compared to Confident Layer Decoding?
  2. Can you prove these small improvements are statistically significant across your test problems?
  3. Does this really offer much benefit over a standard baseline that is properly tuned to control repetition?
  4. Will your rule for picking the "middle layer" actually work on entirely different AI models?
Reviewer_kswMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Transition Thinking is a training-free decoding method motivated by two observations: Soft Thinking often behaves like greedy decoding and inherits its repetition loops, while intermediate Transformer layers sometimes provide confident alternatives when the final layer hesitates. It selects a fixed “transition layer” at the sharpest drop in intermediate-prediction entropy, then selectively mixes that layer’s logits with the final logits to resolve uncertain decisions. Because this arbitration contributes little inside highly confident repetition loops, the method separately applies a mild repetition penalty and brief randomized decoding to escape verified loops and continue reasoning. Across three Qwen3 sizes, it improves average accuracy over sampled chain-of-thought by 0.7–1.2 percentage points and shortens correct generations by 5.2–6.5%. However, its advantage over sampling with a repetition penalty is small on the tested subset, and actual runtime savings remain unproven.

Critical strengths
  1. The method addresses a concrete failure mode of LLM reasoning in that it became repeating and overconfident and proposes a rule that is simple and label-free
  2. The experimental controls are reasonably careful, the paper contains substantive ablations
Critical weaknesses

The strongest simple baseline nearly eliminates the advantage on the tested subset. This is in table 5 where the numbers show that cot with repetition penalty actually crawls back a lot of the improvement from TT. For 14B, the reported average advantage over that baseline is only 0.1 point. Without uncertainty estimates, that is insufficient evidence of a meaningful advantage. For 8B, the 0.9-point difference is more promising, but still requires validation.

The authors should evaluate CoT + repetition penalty across the complete seven-benchmark suite, including 4B, and report its generation lengths and runtime. A CoT + repetition penalty + ICE baseline would further help isolate the contribution of intermediate-layer information.

Questions for authors

The paper should report how often the gate activates, how often it actually changes the selected token, and what happens at those changed decisions. A gate can be active without changing the argmax; disagreement between the two layers also does not guarantee a changed token after mixing and penalization.

The strongest experiment would collect identical prefixes at actual arbitration events and compare continuations after:

  1. The final-layer choice.
  2. The TT choice.
  3. A matched final-layer-only alternative or randomized intervention.

The outcome should be eventual answer correctness, since several next tokens can begin valid reasoning paths. Holding the remaining decoding procedure fixed would test whether the intermediate prediction supplies useful decision information at the claimed moments.

That would support the “correcting hesitant decisions” explanation much more directly than confidence and disagreement statistics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.