- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper propose NF-CoT, a latent CoT method that models continuous reasoning with an autoregressive normalizing flow inside the LLM. This gives latent thoughts the same interface as text tokens, allow the latent thoughts embeddings to be sampled left-to-right, scored with an exact likelihood, and can be optimized with RL. The architecture first compress the textual CoTs into fixed-length continuous latents by a frozen VAE, and shallow invertible flow blocks map these latents into a space that is easier to model. The LLM backbone then acts as a deep autoregressive flow, generating latent thoughts through a Gaussian NF head and answer text through the standard LM head within a single causal stream.
Critical strengths
Using jointly trained invertible flow blocks to learn a target space that suits autoregressive Gaussian modeling, which provides a principled answer to how a continuous latent CoT can be fit by exact maximum likelihood inside a single LLM causal stream.
Critical weaknesses
-
The paper claims that it develops "a likelihood-based formulation of continuous CoT that preserves the sampling, scoring, and decoding interface of explicit CoT.", so that it allows to perform sft and rl on the latent thinking tokens. However, some existing work already did this, could you justify the difference and compare with them? (E.g. Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains NeurIPS 2025)
-
The necessity of NF module is not empirically justified. It seems that the purpose for the flow module is to produce an embedding space that is easier for the LLMs to learn. But what if the LLM directly models the VAE latents autoregressively (i.e., removing the flow module, but using an NLL objective instead of the L2 objective used in LaVAE)?
Questions for authors
- How does your method compares to the deterministic hidden states like Coconut and its variations?
- Why LaVAE performs so badly (Table 1)?
- The gain of NF-CoT over Standard SFT (69.1 vs. 57.8 on average) is surprisingly large, given that the latent thoughts are distilled from the same explicit CoT data. Could the authors provide the training and evaluation details of the Standard SFT baseline, and justify where the gain comes from?
6 comments
Q1: Comparison with Deterministic Latent Reasoning (e.g., Coconut)
We thank the reviewer for raising this question. Unlike methods that propagate deterministic latent states without an explicit latent likelihood, NF-CoT models a probability distribution over latent trajectories and samples from it autoregressively. This has two advantages:
- Avoiding mode averaging. A problem may admit multiple valid reasoning trajectories. L2 regression tends to predict their conditional mean, which may fall between distinct reasoning modes and fail to represent a coherent solution path. Probabilistic modeling instead allows NF-CoT to capture and sample different reasoning trajectories rather than collapsing them into a single averaged prediction.
- Enabling latent-space RL. Exact likelihoods of sampled latent trajectories allow direct policy-gradient optimization of latent generation using downstream rewards, rather than optimizing only answer tokens.
Controlled deterministic ablation. We implement a deterministic counterpart with the same backbone, training data, latent layout, and training steps. It directly predicts each continuous latent token using L2 loss instead of a Gaussian distribution; at inference, each prediction is fed back autoregressively as the next input embedding, without sampling.
| Continuous thoughts | HumanEval / HE+ | MBPP / MBPP+ |
|---|---|---|
| Deterministic (L2 regression) | 75.3 / 69.1 | 78.3 / 65.4 |
| Probabilistic (NF-CoT) | 85.8 / 78.3 | 83.9 / 72.1 |
NF-CoT consistently outperforms deterministic regression, supporting the benefit of probabilistic latent modeling. We will include this ablation in the revised paper.
Q2: Why Does LaVAE Perform Poorly?
We thank the reviewer for this question. LaVAE's performance highlights two limitations of directly modeling complex latent CoT distributions.
First, LaVAE relies on a simple Gaussian assumption for the original latent CoT distribution, using an L2 regression objective to predict VAE latents. However, latent CoT distributions can be highly complex and multimodal, as a single problem may admit multiple distinct reasoning strategies. A simple Gaussian may fail to capture such distributions, leading to mode averaging and less meaningful latent representations. In contrast, NF-CoT uses normalizing flows to learn an invertible transformation, allowing more complex distributions to be modeled in a transformed space with a simpler Gaussian density.
Second, LaVAE generates the entire latent block in a single step, without explicitly modeling dependencies between latent tokens. In contrast, NF-CoT autoregressively factorizes the joint distribution into 64 conditional distributions, each conditioned on previously generated latents. This enables more expressive joint distribution modeling, capturing complex dependencies between reasoning steps and allowing the reasoning trajectory to evolve progressively.
Q3: Standard SFT Baseline and the Source of NF-CoT's Gains
We thank the reviewer for this insightful question. Although NF-CoT and Standard SFT learn from the same explicit CoT data, we attribute NF-CoT's performance gains to two key advantages of continuous latent reasoning: smoother semantic transitions and the ability to represent diverse reasoning possibilities in continuous latent space.
Standard SFT supervises both explicit CoT traces and final answers, not answers alone, using next-token cross-entropy. Its configuration is:
| Setting | Standard SFT |
|---|---|
| Backbone / Data | Qwen3-8B-Base / 733K Ling-Coder Python samples |
| Training | Full fine-tuning, 2 epochs (22,902 steps), global batch 64 |
| Optimizer | AdamW, β=(0.9, 0.95), weight decay 0.01 |
| Implementation | 8×B200, bf16, FSDP, no packing |
| Evaluation | vLLM, temperature 0.6, top-p 1.0, 16 samples/problem, average pass@1 |
These settings match NF-CoT's training setup and evaluation protocol.
Why can latent reasoning outperform explicit CoT? Explicit CoT serializes reasoning into discrete language tokens, progressively committing to particular textual expressions. Continuous latents can encode richer semantic information and preserve multiple plausible reasoning paths, allowing more flexible and potentially parallel reasoning. Appendix A provides qualitative evidence: different latent samples for the same problem decode into distinct strategies (e.g., dynamic programming, tabulation, and memoization), each producing correct solutions.
Continuous latents also allow smoother semantic transitions. Rather than repeatedly expressing intermediate steps in discrete natural-language tokens, the model operates directly on continuous representations, enabling gradual and flexible updates of reasoning states, including intermediate computations difficult to verbalize. By reducing the constraints of discrete tokenization, NF-CoT can support more natural and effective latent-space reasoning.
Follow-up on Our Response
Dear Reviewer,
Thank you again for your positive assessment of our work and your constructive feedback.
We hope our responses have satisfactorily addressed your questions and suggestions. Please feel free to let us know if any further clarification would be helpful.
Best regards,
The Authors
Clarifications on Methodological Novelty, Baseline Comparisons, and Additional Ablations
We sincerely thank the reviewer for the thoughtful questions. We clarify the distinctions from prior methods and provide additional controlled ablations below.
W1: Differences from Prior Likelihood-Based Latent Reasoning Methods
We thank the reviewer for highlighting this relevant work. Our key distinctions lie in more expressive latent distribution modeling through normalizing flows and semantic reasoning representations learned through VAE encoding.
Distribution modeling. CoLaR directly models each latent step using a Gaussian distribution whose mean and variance are predicted autoregressively by the LLM. However, latent CoT distributions can be complex and potentially multimodal, since one problem may admit multiple reasoning strategies. NF-CoT instead learns an invertible nonlinear transformation using normalizing flows, allowing complex distributions in the original latent space to be modeled through simpler Gaussians in a transformed space, while preserving exact likelihood computation and autoregressive sampling.
Latent representations. CoLaR compresses local groups of token embeddings through scaled summation, whereas NF-CoT uses a VAE to learn compact semantic representations of entire reasoning traces. Thus, NF-CoT models learned CoT representations rather than directly compressed token embeddings.
W2: Necessity of the Normalizing Flow Module
We thank the reviewer for this suggestion. We conducted a controlled ablation removing the shallow flow blocks while retaining autoregressive Gaussian modeling of the same VAE latents with the same NLL objective. All other training and evaluation settings remain unchanged.
Latent density modeling. We compute exact NLL on 128 real VAE latent samples and compare it with Gaussian noise matched to their per-dimension mean and variance.
Removing the shallow blocks increases NLL by approximately 50%, with worse likelihood on all 128 samples. NF-CoT also better distinguishes real latents from moment-matched Gaussian noise (2.93 vs. 1.62 bits/dim), indicating that the flow captures complex, non-Gaussian structure in the latent CoT distribution.
Downstream reasoning performance. Under the same evaluation protocol, we compare against the original NF-CoT results:
Performance drops substantially on all four benchmarks without the shallow blocks. Autoregressive Gaussian NLL modeling alone is therefore insufficient to match NF-CoT; the flow improves both density modeling and downstream reasoning performance. We will include this ablation in the revised paper.