acceptodds
Under review as a conference paper at ICLR 2027

Latent Thought Flow: Efficient Latent Reasoning in Large Language Models

Abstract

Large Language Models (LLMs) increasingly rely on intermediate reasoning, yet explicit Chain-of-Thought (CoT) suffers from a linguistic space bottleneck: each thought must be decoded into tokens, causing high inference overhead. Latent reasoning moves deliberation into continuous space, but existing methods mostly learn deterministic or reward-maximizing paths, lacking a principled way to allocate probability across trajectories with different correctness and costs. We propose **Latent Thought Flow (LTF)**, which models reasoning as variable-length continuous trajectories and trains a sampler to match a reward-induced posterior over answer quality and computation cost. We instantiate this with a continuous GFlowNet using stochastic latent transitions. To handle sparse answer supervision, we introduce an Entropy-Weighted Subtrajectory Balance objective for intermediate rewards and a reference-prior regularizer to anchor exploration. Experiments under finetuning and transfer learning settings show that LTF outperforms explicit and latent reasoning baselines, improving accuracy by 9.5% while reducing reasoning length by 27.2% on average compared with strong latent reasoning methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Latent Reasoning, Efficient Reasoning, Generative Flow Networks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AFZ5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Latent Thought Flow (LTF), a method for latent reasoning in LLMs. LTF learns variable-length continuous latent trajectories using a GFlowNet objective, plus rewards combining answer quality and reasoning cost. It adds entropy-weighted subtrajectory balance and an auxiliary reference-prior regularizer, to encourge concise but useful latent reasoning paths. Experiments ({four models} X {arithmetic, symbolic reasoning, planning, code generation benchmarks}) report consistent accuracy and latency improvements over explicit CoT and several latent reasoning baselines.

Critical strengths

The empirical gains are consistent across several tasks and model sizes. For example, Table 1 reports the 1B model's average accuracy increases from 55.56% to 59.68% over ReGuLaR while latency decreases from 124.56 to 118.12 ms. Measuring acutal latency plus reasoning length makes the evidence more convincing.

Critical weaknesses
  1. The experimental comparison isn't fully convincing. The explicit CoT baseline is implausibly weak. For example, LLaMA-3.1-8B gets 19% on GSM8K-Aug, far below commonly reported number for this model, which can get ~40% even with No-CoT and bump to ~70% with CoT-SFT (similar for other model sizes). Also, there is no CoT-SFT baseline, as the paper repeatedly claims superior accuracy-efficiency tradeoffs over explicit CoT, but avoids comparing against explicit CoT-SFT. Comparing a fine-tuned latent model against zero-shot prompting creates an artificially weak baseline.

  2. The length penalty is hard to interpret. Different length trajectories have densities on spaces of different dimension, so without a clear base measure and consistent treatment of Jacobians, the weight of cost relative to the log-density terms in Eq. 14 can depend on the arbitrary scale of zz. Without a λc=0\lambda_c = 0 ablation, it is unclear whether the penalty drives short trajectories. The ablations also point the same way. DB, TB and SubTB share the same balance fixed point for a fixed reward, but DB and TB produce trajectories about 4x longer than SubTB (7.28 and 7.51 vs. 1.90). If all three reach the same fixed target, their length distributions will match. Without the reference prior, LTF remains more accurate than ReGuLaR (59.25 vs. 55.56) but uses longer trajectories (2.75 vs. 2.04). Thus, the efficiency gain over ReGuLaR appears coming from the prior and the choice of balance loss, not from reward-proportional sampling.

Questions for authors

See weaknesses. I will raise my score if any of above gets properly addressed/explained.

3 comments

Thanks for the added experiments. They help address several of my concerns. However, Appendix C.3 says no 1B responses hit the 256-token limit, so the budget explanation remains unclear. Did the experiments also use the bounded sampler assumed in the theory? I would appreciate clarification on these points. For now, I will keep my score.

Thanks for clarifying. The stronger 8B CoT result needs a latency comparison to support the efficiency claim. Also, clipping differs from the truncated Gaussian in the paper, do the density calculations account for this? I’ll keep my score for now.

Thanks for the followup. The latency results make the trade-off clear, and the explanation addresses my concern. I’ll raise my score to 3.

Reviewer_bpDbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Latent Thought Flow (LTF) learns to reason through a sequence of continuous vectors before generating an answer. A Gaussian transition model chooses each vector, and a learned stopping probability controls the number of reasoning steps. The intended training target assigns probability to trajectories in proportion to a reward combining answer quality and a length penalty. The method adapts a GFlowNet objective that constrains probability flow along parts of a trajectory, weights these constraints by transition entropy, and adds a rationale-based reference branch and an answer loss. Experiments cover LLaMA backbones and a Qwen-derived DeepSeek checkpoint, with arithmetic, symbolic reasoning, planning, code generation, and transfer evaluations. The reported accuracy and latency gains are promising, but the continuous target distribution and the comparison with reward maximization require clarification.

Critical strengths

1.The method jointly learns the content, length, and sampling probabilities of latent reasoning paths. Sections 3.1-3.3 aim to retain multiple correct, efficient paths rather than train only toward one reference chain or one highest-reward path. The stopping mechanism also allows the amount of internal computation to vary by input. The balance constraints specify a concrete training objective for this idea. This is an interesting direction for learning efficient latent reasoning, provided the target distribution is made precise.

2.The reported gains cover several tasks and include variability estimates. Tables 1-3 evaluate multiple backbones and both short-answer and structured-output tasks. For LLaMA-1B, Table 1 reports average accuracy of 59.68% versus 55.56% for ReGuLaR, while latency falls from 124.56 to 118.12 ms. Tables 4-5 examine training components and alternative objectives. Appendix B reports five independent runs, and Table 7 gives confidence intervals for selected results. These findings support a practical benefit within the tested settings.

Critical weaknesses

1.The paper does not establish that its continuous target can be normalized into a probability distribution. Equation (2) allows latent vectors in an unbounded Euclidean space, and Equation (9) normalizes trajectory rewards by integrating them over that space. This requires the reward integral to be finite. Equations (6)-(8) penalize trajectory length, but do not establish this condition for the vectors at a fixed length. The Gaussian sampling policy does not itself multiply the target reward by a decaying density, and the separate loss in Equation (19) does not specify such a factor in Equation (9). The authors should state the integration measure and explain what makes the normalizing constant finite. This is an unresolved condition for the probability interpretation, not evidence that the training procedure necessarily diverges. It nevertheless affects the central justification for reward-proportional sampling.

2.The reward-maximization comparison may also change the auxiliary supervision. Appendix B says that Latent-GRPO replaces the complete objective in Equation (21), whereas DB and TB replace only the flow-loss term. If interpreted literally, GRPO would lose the answer cross-entropy and reference-prior losses that the other variants retain. Table 5 would then compare both the trajectory objective and the supervision available to each method. The large performance gap could not be attributed to the trajectory objective alone. Please clarify the implementation and compare objectives while retaining the same auxiliary losses, rewards, and rollout budgets. The positive main results do not resolve this ambiguity or the target-definition issue, which currently leave me leaning toward weak rejection.

Questions for authors

1.What integration measure is used in Equation (9), and what ensures that its normalizing constant is finite? If bounded support or a reward-weighted reference density is used, please show how it enters Equations (11)-(17).

2.Does Latent-GRPO retain the same answer cross-entropy and reference-prior losses as LTF, DB, and TB? If not, what happens when only the trajectory-learning objective changes and the auxiliary losses, rewards, rollout budgets, and tuning conditions remain fixed?

3.With 20 rollouts per training example and rewards evaluated at latent prefixes, what are the training GPU-hours and reward-evaluation costs relative to ReGuLaR and the optimization baselines? These costs would clarify the practical value of the inference savings.

1 comment

I still have two remaining concerns:

  1. Consistency between the theoretical normalization assumptions and the actual implementation. I would appreciate further clarification on whether the probability distribution used in practice is consistent with the bounded-support assumption introduced in the theoretical analysis. 2. The trade-off between training cost and inference efficiency. I remain concerned that LTF requires approximately 30% more training computation than ReGuLaR while achieving only about a 5% reduction in inference latency. Is this trade-off practically worthwhile?

I do not think additional experiments are necessary at this stage. I would simply appreciate further clarification from the authors on these two points.

Reviewer_YLF1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Latent Thought Flow (LTF), which models reasoning as a variable-length sequence of continuous latent states followed by a learned stopping decision and answer generation. Each possible stopping prefix receives a reward combining answer correctness, ground-truth answer likelihood, and a penalty for latent steps. The sampler is trained with an entropy-weighted continuous Subtrajectory Balance objective, alongside an answer loss and an optional reference-rationale prior.

Critical strengths
  • The formulation of which latent states to sample and when to stop is interesting.
  • Results are consistently better than the reported latent-reasoning baselines across several backbones and tasks.
  • The paper includes comparisons among balance objectives and ablations of entropy weighting and the reference prior.
Critical weaknesses
  • I don't understand why is reward-proportional sampling the right objective for accuracy and efficiency? The balance condition does not help improve expected correctness or minimizes computation. Even with perfectly known rewards, (Q(\tau)\propto R(\tau)) retains probability on paths below the maximum reward. The paper motivates this choice by preserving multiple useful paths, but needs evidence that this trade-off improves the stated task objectives.

  • Positive reward for incorrect trajectories may be problematic. The ground-truth likelihood term gives a positive reward even when a decoded answer fails the verifier. Under a reward-proportional target, many individually low-reward trajectories can collectively receive large probability mass if they occupy a large region of latent space.

  • This work is a combination of efficient thinking and latent reasoning, so more baselines should be used. The latent reasoning baselines are nice, but I think there are more efficient thinking/adaptive reasoning baseline work that the authors should be considered, such as https://arxiv.org/abs/2511.21581 and https://arxiv.org/abs/2505.13379.

Questions for authors
  • How are candidate answers decoded and scored at every intermediate prefix? Is decoding deterministic, and are any computations shared across prefixes?
  • Can Latent-GRPO be evaluated with the same answer and reference-prior losses as LTF?
  • Can the authors report training compute and a within-input measure of successful trajectory diversity?
1 comment
Area_Chair_fURQOpenReview ↗

Reminder: Rebuttal Discussion

Dear Reviewer,

The discussion deadline is only a few hours away. Could you please respond to the authors' rebuttal before the deadline?

Best regards, Area Chair

Comments on the paper

1 comment

Dear Reviewers,

I hope you're doing well.

Please respond to the authors' rebuttal if you have not already done so.

Thank you for your valuable time and engagement.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.