- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies representation collapse in soft-token latent reasoning, distinguishing temporal covariance saturation, attenuation of input perturbations, and loss of diversity across rollouts. It proposes SoftSphere, which projects each soft token onto a sphere whose radius matches the mean vocabulary-embedding norm. Conditional theoretical results establish preservation of temporal effective rank and slower state alignment. Experiments across two model families and five tasks show improved temporal rank, reduced repetition.
Critical strengths
-
The paper clearly distinguishes variation within a trajectory, local input sensitivity, and diversity across sampled trajectories. It also explicitly recognizes that covariance shrinkage does not necessarily reduce effective rank. These distinctions make the analysis of latent collapse more precise and provide a useful diagnostic framework for continuous feedback.
-
SoftSphere is a simple intervention requiring no additional parameters or auxiliary objective.
Critical weaknesses
-
The contribution beyond standard normalization remains insufficiently established. Fixed-norm normalization has clear precedents [1,2]. Although the paper studies partial and centered variants, it does not isolate the benefit of per-step norm correction from a constant rescale. Its comparative value relative to collapse-mitigation methods such as WIDER [3] and SeLaR [4] also remains unclear.
-
The cross-rollout diversity claim lacks direct evidence. Cross-rollout collapse (Fig. 2c) is an important motivation, but SoftSphere’s effect on this collapse is not shown. The reported temporal-rank and pass@k improvements do not by themselves establish greater diversity across rollouts.
[1] Zhang & Sennrich. Root Mean Square Layer Normalization. NeurIPS 2019.
[2] Loshchilov et al. nGPT: Normalized Transformer with Representation Learning on the Hypersphere. ICLR 2025.
[3] Hao & Yang. Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning. 2026.
[4] Fu & Luo. SeLaR: Selective Latent Reasoning in Large Language Models. ACL 2026.
Questions for authors
-
What evidence distinguishes the benefit of per-step norm correction from ordinary scale adjustment, and what does SoftSphere contribute beyond established normalization and collapse-mitigation methods [1–4]?
-
Which reported result directly supports mitigation of cross-rollout collapse, rather than improved within-trajectory temporal rank or answer coverage?
-
Which observed behavior in the actual unprojected trajectories corresponds to the sustained radial decay assumed by Theorem A, given the reported mean norm of approximately 0.99R?