acceptodds
Under review as a conference paper at ICLR 2027

NormCaST: Effective Latent-Space Reasoning via Soft Embedding Norm Calibration

Abstract

Recent latent reasoning approaches enable large language models (LLMs) to reason in a continuous space and improve reasoning performance. As a representative method, Soft Thinking utilizes the probability-weighted average of candidate token embeddings as next input embedding. However, we identify that such averaging operation results in norm shrinkage at multi-candidate positions, where Soft Thinking departs from standard Chain-of-Thought (CoT) reasoning. Further analysis reveals that the resulting embeddings fall below the typical norm range of discrete embeddings, creating an input-scale mismatch that limits performance. Motivated by these observations, we propose **NormCaST** (Norm-Calibrated Soft Thinking), the first approach to analyze and optimize latent-space reasoning from the perspective of representation magnitude. Specifically, NormCaST calibrates each soft embedding to the probability-weighted average norm of its candidate token embeddings, restoring the typical input representation scale. This training-free calibration introduces no substantial computational overhead and serves as a plug-and-play enhancement for existing Soft Thinking variants. Extensive experiments across four models ranging from 1.7B to 32B parameters and multiple reasoning benchmarks demonstrate that NormCaST consistently outperforms its non-calibrated counterparts, improving average Pass@1 by up to 2.97 points while maintaining comparable generation length.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM Reasoning, Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_GsB4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

NormCaST addresses norm shrinkage in soft token embeddings by rescaling them to the probability-weighted average norm of their candidates. It preserves direction and requires no training. The paper reports average accuracy gains across four models, with additional coding and SwiReasoning evaluations.

Critical strengths
  1. The paper connects a valid norm bound to the shrinkage observed in the examined Qwen3 trajectories and tests a simple correction (Eq. 8 and Figs. 1-2). Calibration preserves direction and adds no model forward passes (Eq. 9 and Appendix B2). All three tested norm targets improve reported average accuracy (Table 2).

  2. I lean toward weak acceptance for this useful correction, but uncertainty about the reliability of the gains make me keep the recommendation borderline. All eight model/variant pairs improve in average accuracy over their uncalibrated counterparts, with broadly comparable generation lengths (Table 1). Calibrated Gumbel exceeds CoT’s average on 3 of 4 models, although calibrated Vanilla does so only on Qwen3-8B. SwiReasoning integration also improves average accuracy, though its calibrated variants remain longer than CoT (Appendix D2 and Table 9).

Critical weaknesses
  1. The paper does not quantify uncertainty in the gains, which limits my confidence in the practical benefit and target ranking. Each problem has three samples, and each AIME set contains only 30 problems. For Qwen3-8B Gumbel, AIME accounts for about 91% of the 2.95-point average accuracy gain (Table 1). The small differences between targets and small coding gains are also uncertain with both calibrated variants decline on LiveCodeBench (Tables 2-3).
  2. The claim that calibration recovers semantic expressiveness goes beyond the evidence and should be narrowed. Norm analysis and rescaling experiments support the relevance of input scale, but they do not measure semantic recovery.
Questions for authors

Could you use the existing outcomes to report uncertainty for the paired calibration gains and target comparisons, accounting for repeated samples of each problem? If the main gains are unstable, the case for acceptance would be weaker. Uncertainty about the differences between targets would mainly weaken the claim that the weighted target is better.

Reviewer_TQJiOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies Soft Thinking, which feeds the probability-weighted average of candidate token embeddings back into the model instead of a sampled token. It shows that this averaging shrinks the embedding norm at steps with more than one candidate, so many soft embeddings fall below the norm range of tokens the model usually reads. The proposed NormCaST keeps each soft embedding's direction and rescales its norm to the probability-weighted average norm of the candidates, without training or extra forward passes. On four models and five math and science benchmarks, it raises the average Pass@1 of Soft Thinking and its Gumbel variant by 0.36 to 2.97 points, with small gains on three code benchmarks and when combined with SwiReasoning.

Critical strengths
  1. The diagnosis is simple and clearly argued. Jensen's inequality explains why averaging shrinks the norm, and separating single-candidate from multi-candidate steps (Figure 1, Table 5) shows exactly where Soft Thinking departs from CoT and where the shrinkage happens.

  2. The fix is cheap and easy to adopt. This method is one rescaling step with no training, no extra forward pass and no new hyperparameter, and I like that it plugs into Soft Thinking, its Gumbel variant and SwiReasoning, improving the average in all eight paired comparisons of Table 1.

Critical weaknesses

The experiments test whether the fix works, not whether the diagnosis is right. Half of the contribution is the claim that norm shrinkage is why Soft Thinking underperforms, but the only evidence that it causes the drop is that rescaling raises accuracy (Table 1, Table 2). A direct test would shrink the input norm of ordinary CoT tokens to the levels seen in Soft Thinking and check whether CoT gets worse, which also takes the mixed direction out of the picture.

Questions for authors

What happens if the soft embedding is scaled directly to about 0.5 to 1.5 times the target norm? Table 2 compares three ways of choosing the target, all inside the candidate norm range (L456-459) and at most 0.4 points apart. A peak near 1 would support the input-scale explanation, while a steady rise would suggest that a larger input norm simply helps.

Reviewer_ifnxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines training-free latent reasoning with Soft Thinking, which feeds the probability-weighted convex combination of candidate token embeddings back into the LLM. Most Soft Thinking steps collapse to one candidate and reduce to discrete CoT; only minority multi-candidate steps cause L2 norm shrinkage by Jensen’s inequality: ∥∑pkE[k]∥2<∑pk∥E[k]∥2\|\sum p_k E[k]\|_2 < \sum p_k \|E[k]\|_2. Analysis on Qwen3-1.7B and Qwen3-8B shows these compressed embeddings often enter a low-norm region dominated by symbols and non-Latin tokens, and the norm decay propagates within the Pre-LN Transformer residual stream. The authors introduce NormCaST, a training-free, hyperparameter-free, direction-preserving rescaling method that adjusts the soft embedding norm to the probability-weighted average norm of candidate embeddings (it makes no change for single-candidate steps). Evaluated on four models and eight benchmarks covering math, science and coding, NormCaST outperforms vanilla Soft Thinking, Gumbel-Softmax and SwiReasoning consistently, bringing +0.36 to +2.97 macro-average Pass@1 gains with similar generation length; combining NormCaST with SwiReasoning yields further improvements.

Critical strengths
  1. It provides a thorough diagnosis of a real geometric effect. The work uses the Jensen inequality bound. Statistics show most steps collapse to one candidate. Scatter plots illustrate norm deficits. Vocabulary analysis identifies a low-norm region dominated by non-Latin and byte tokens. Only separating multi-candidate steps reveals the shrinkage. This deep diagnostic analysis is the paper’s main strength.
  2. NormCaST is a free, plug-and-play fix. It requires no training and no learnable parameters, introduces no extra forward passes, and meanwhile preserves embedding direction.
  3. The paper includes solid ablations. It tests alternative norm targets and runs experiments with multiple seeds, reports generation lengths and adopts deterministic evaluation, and also carries out combined tests with SwiReasoning.
Critical weaknesses
  1. The improvements are not fully quantified for reliability. Results lack seed-level variance, confidence intervals and significance tests. Much of the macro gain concentrates on the small AIME subsets, while gains are tiny on math and coding benchmarks, and some coding settings degrade. The averaged positive results cannot prove consistent, robust advantages.
  2. Similar performance across different norm targets suggests general norm inflation may explain the improvement, rather than the specific Jensen-based design. Recovering semantic expressiveness is only a hypothesis.
  3. NormCaST-enhanced Soft Thinking cannot reliably outperform standard discrete CoT.
  4. The generality is limited and generation length acts as a confounder. Diagnostics and experiments are restricted to Qwen3 and Soft Thinking. Calibrated outputs are sometimes shorter, and the accuracy-length tradeoff is not isolated. Wall-clock speed and precomputation details are unreported, so the low-cost claim is based only on theoretical operation counts.
Questions for authors
  1. Can you provide paired confidence intervals for Table 1–3 results with more rollouts on AIME? Are conclusions stable when the two AIME subsets are evaluated separately?
  2. Can you run a scale sweep and sham rescaling control, and check whether most gains come from low-norm steps?
  3. Can you show layer-wise residual norms and logit differences to verify how the rescaling effect propagates past RMSNorm?
  4. Does this norm calibration work on models outside Qwen3 or other latent reasoning methods?
  5. Do gains hold on length-matched samples after controlling for ColdStop? Can you report measured latency and clarify if candidate norms are precomputed
Reviewer_5zsEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at Soft Thinking, which feeds the probability-weighted average of candidate token embeddings back into the model instead of a sampled token. The authors note that at multi-candidate steps, roughly 18–26% of steps and exactly where Soft Thinking differs from CoT, this average has a much smaller norm than real token embeddings. They argue this pushes the input into a low-norm, out-of-distribution region. Their fix, NormCaST, simply rescales the soft embedding to the probability-weighted average norm of its candidates. It's training-free, has no hyperparameters, and can be added to other Soft Thinking variants. On four models and five math/science benchmarks, it raises average Pass@1 by 0.36–2.97 points over the uncalibrated versions, with small gains also on code and with SwiReasoning.

Critical strengths
  1. It's basically free and plug-and-play. It's one line of code with no parameters and no extra compute, and average Pass@1 goes up in every pairing the authors tried. Anyone already using Soft Thinking can adopt it with almost no effort.

  2. The diagnosis is clean. Linking the norm shrinkage to the few multi-candidate positions, where Soft Thinking actually differs from CoT, is a nice framing, and Figs. 1 and 2 show it clearly.

Critical weaknesses
  1. The gains are small and may just be noise. Results come from 3 seeds with one sample each and no error bars, and on AIME one problem in one seed moves the score by 1.11 points. Both ~3-point headline gains come almost entirely from AIME. Per benchmark, calibration helps in 23 of 40 cells and hurts in 13, and vanilla NormCaST still trails plain CoT on 3 of 4 models.

  2. The mechanism isn't really shown, and the novelty is overstated. Beyond Jensen's inequality the evidence is only correlational: there's no intervention and no layer-wise analysis, even though RMSNorm makes each block's input scale-invariant. In Table 2, the random, top-1 and weighted targets land within 0.4 points of each other, which looks more like "any upscaling helps". Also, HRPO already normalizes its probability-weighted embedding to preserve scale, so the "first approach" claim doesn't hold.

Questions for authors
  1. Can you report std or confidence intervals across seeds, and use more samples on AIME? Also, is the soft phase deterministic across seeds?

  2. Is there direct causal evidence? For example, does shrinking the embeddings in standard CoT by the same factors hurt accuracy?

  3. How does NormCaST compare with simple controls such as a fixed global scale, HRPO-style 1/∥p∥1/\|p\| normalization, or slerp?

  4. It would help to run the norm analysis on 32B and gpt-oss-20b as well, not only 1.7B/8B.

1 comment

Thanks, you completely addressed my concerns. I have updated my rating

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.