NormCaST: Effective Latent-Space Reasoning via Soft Embedding Norm Calibration

Recent latent reasoning approaches enable large language models (LLMs) to reason in a continuous space and improve reasoning performance. As a representative method, Soft Thinking utilizes the probability-weighted average of candidate token embeddings as next input embedding. However, we identify that such averaging operation results in norm shrinkage at multi-candidate positions, where Soft Thinking departs from standard Chain-of-Thought (CoT) reasoning. Further analysis reveals that the resulting embeddings fall below the typical norm range of discrete embeddings, creating an input-scale mismatch that limits performance. Motivated by these observations, we propose **NormCaST** (Norm-Calibrated Soft Thinking), the first approach to analyze and optimize latent-space reasoning from the perspective of representation magnitude. Specifically, NormCaST calibrates each soft embedding to the probability-weighted average norm of its candidate token embeddings, restoring the typical input representation scale. This training-free calibration introduces no substantial computational overhead and serves as a plug-and-play enhancement for existing Soft Thinking variants. Extensive experiments across four models ranging from 1.7B to 32B parameters and multiple reasoning benchmarks demonstrate that NormCaST consistently outperforms its non-calibrated counterparts, improving average Pass@1 by up to 2.97 points while maintaining comparable generation length.