acceptodds
Under review as a conference paper at ICLR 2027

From Chain-of-Thought to Loops: Non-Autoregressive Latent Reasoning via Looped Transformers

Abstract

Chain-of-thought (CoT) reasoning often improves language-model performance by giving models additional computation before answering. However, explicit CoT expresses this computation as a sequence of autoregressively generated tokens. Latent reasoning replaces these tokens with compact continuous states, but most autoregressive latent-reasoning methods retain a left-to-right dependency among latent vectors. We introduce LLoCoT: a looped latent-reasoning framework that replaces left-to-right latent generation with iterative refinement of a compact latent workspace. A shared transformer is reapplied for a small number of refinement iterations, jointly updating the latent slots based on the prompt and the evolving workspace state. Using the refined state, a probabilistic head predicts a distribution from which latent tokens are sampled in parallel and used to condition an autoregressive decoder for answer generation. Training uses continuous representations derived from explicit CoT together with a final-answer prediction loss and likelihood-based supervision of the latent states. Across HumanEval, HumanEval+, MBPP, and MBPP+, the method achieves the highest four-benchmark mean among the evaluated methods, performing on par with Reasoning SFT while outperforming the base model, answer-only SFT, and NF-CoT. Furthermore, compact latent states and parallel slot sampling reduce the serial thought-generation bottleneck of explicit CoT and other autoregressive methods while preserving probabilistic latent modeling.

Primary area:
generative models
Keywords:
latent reasoning, looped transformers

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_uysuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LLoCoT. LLoCoT abandons standard left-to-right latent thought generation and instead refines a fixed set of K=64 learned slot vectors. A shared transformer, the full Qwen3 1.7B Base backbone, runs R=2 times over the slots and prompt; a diagonal Gaussian head outputs K 2560-dimensional latent thoughts, and predicted means from non-final loops feed back into hidden states. Final latents are sampled in parallel to drive an autoregressive answer decoder. Training targets follow NF-CoT: a standalone transformer VAE trained for 180k steps encodes reasoning traces into 64 continuous vectors, then a shallow conditional autoregressive flow maps these vectors into the model’s latent space. The loss combines likelihood over final latents (including the flow’s log determinant), an intermediate prediction auxiliary likelihood weighted at 0.25, and answer cross-entropy for the decoder conditioned on target latents. Training uses two phases: roughly 1.5k steps with frozen backbone, then around 45k full fine-tuning steps at batch size 64. Evaluated on HumanEval, HumanEval+, MBPP and MBPP+ with 16 samples per task, its average score across the four benchmarks hits 63.6, versus 63.2 for Reasoning SFT, 62.7 for answer-only SFT, 62.0 for NF-CoT and 57.3 for the base model. Tested at batch size 1 on a single H100, time to first answer token falls from 3.17 s (Reasoning SFT) to 0.09 s and end-to-end latency from 15.16 s to 11.47 s, though peak memory rises 1.9× and analytical FLOPs go up 1.6×.

Critical strengths

The efficiency measurement is carefully done and honestly described for what it shows. The serial reasoning phase falls from about 2.5 s (Reasoning SFT) and about 2.0 s (NF-CoT) to 0.06 s, with a clear protocol (128 prompts, three runs after ten warm ups, batch size one, phase by phase timings) and forward call counts that explain the gain. The authors say plainly that memory and FLOPs are worse and that the speedup comes from parallelism rather than less compute. That is the right framing, and the idea of replacing serial latent generation with a small number of parallel refinement steps is a sensible direction.

Second, the evaluation protocol is unusually clean for this area. Appendix B states that failed, truncated and malformed completions count as zero rather than being dropped, that every task by sample record must exist before results are promoted, and that pass@1 is averaged over 16 samples per task. The paper is also candid in its limitations section (one 1.7B model, code only, no adaptive depth).

Critical weaknesses

the accuracy claim is not supported at the level the abstract implies. LLoCoT's four benchmark mean (63.6) is 0.4 points above Reasoning SFT (63.2) and 0.9 above answer only SFT (62.7). It is behind Reasoning SFT on HumanEval (69.3 vs 70.3) and HumanEval+ (63.9 vs 64.4) and ahead only on MBPP and MBPP+. The intervals the paper reports are fixed task sampling intervals (half widths of 0.6 to 1.3 per benchmark), which by the paper's own words (Appendix B.3) do not capture variation over training seeds, and each system appears to be a single training run. The ablation conclusions are even smaller than that: final configuration vs no feedback is +0.5, auxiliary loss alone is minus 0.2, sampled feedback is minus 0.7, and the loop depth sweep moves the mean by 0.1 to 0.6 for R=1 to 4. These differences are well inside plausible seed to seed variation for a 1.7B model, so the statements about which feedback or loop setting is best are not established. The honest claim the data support is "on par with Reasoning SFT in accuracy".

Questions for authors

1/ What data were used to train the VAE and the LLoCoT stages (problems, reasoning trace source, size), and what exactly is "Reasoning SFT"?

2/How many training seeds underlie each row in Table 1 and what is the seed to seed spread? Do Base SFT, Reasoning SFT and NF-CoT receive the same number of optimizer steps and examples as LLoCoT?

Reviewer_d7j2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LLoCoT, which replaces autoregressive latent-thought generation with non-autoregressive, iterative refinement of a fixed-size latent workspace. Training uses VAE-encoded CoT targets transformed by a conditional normalizing flow, together with an answer-generation objective. On the evaluated code-generation benchmarks, LLoCoT achieves performance comparable to Reasoning SFT and improves over NF-CoT, while substantially reducing reasoning latency.

Critical strengths
  1. This paper addresses the sequential latent-generation bottleneck through shared-workspace refinement and parallel sampling while retaining probabilistic modeling and likelihood-based training.
  2. LLoCoT substantially reduces reasoning latency while maintaining accuracy comparable to Reasoning SFT and outperforming NF-CoT on average across the evaluated coding benchmarks.
Critical weaknesses
  1. Evaluation is limited to a single 1.7B backbone and code-generation tasks, leaving generalization to larger models and other reasoning domains, such as math, unverified.
  2. It lacks loop-by-loop analysis, such as decoded-latent probes and failure analysis, to explain what successive iterations improve, particularly given the modest gains from additional refinement.
Questions for authors

See weakness

Reviewer_877AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new latent reasoning method, LLoCoT, that iteratively refines a fixed-size set of latent tokens before outputting a final answer. First a VAE is trained to compress full reasoning traces into 64 vectors, which are then transformed into latent thoughts via a conditional flow transformation. The LLM then learns to predict these latent thoughts in parallel, over multiple refining steps, and to predict the final answer. Experiments on code-generation tasks with Qwen3-1.7B-Base show comparable performance to other latent reasoning and SFT baselines, with reduced latency.

Critical strengths

The proposed method provides a novel way of adapting LLMs to iteratively refine a latent workspace, and good performance relative to comparable latent reasoning work. The significant drops in latency are promising relative to autoregressive reasoning, despite the higher flop count.

Critical weaknesses

LLoCoT introduces substantial architectural and training complexity that may be difficult to adapt to new models and tasks. For example, the latent workspace size, refinement depth, and VAE and conditional flow training may all require tuning in a new setting. Although the ablations in Section 4.2 are helpful, they only examine a limited set of configurations. Without more comprehensive ablations it is unclear how these design choices interact or how robust the method is across different settings.

Questions for authors

Why was batch size 1 selected for inference efficiency methods? Large (and continuous) batch sizes may be a more realistic setting.

What training data was used to train the VAE, and can you compare the training compute costs of the different methods?

Reviewer_jnT7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LLoCoT, a latent reasoning framework that replaces autoregressive latent generation with iterative refinement of a fixed-size workspace using a shared transformer. The method combines VAE- and flow-derived supervision, intermediate Gaussian mean feedback, and auxiliary likelihood objectives, before sampling latent vectors in parallel for autoregressive answer generation. Experiments on code generation tasks using Qwen3-1.7B demonstrate accuracy comparable to Reasoning SFT, while reducing reasoning latency relative to both Reasoning SFT and NF-CoT.

Critical strengths
  • The motivation is intuitive and clear. Based on the observation that compressing reasoning into continuous states does not eliminate autoregressive dependencies, the paper explores whether parallel latent prediction with iterative refinement can reduce the sequential reasoning bottleneck.
Critical weaknesses
  • The methodological contribution is incremental, and the added value of looped refinement is insufficiently established. LLoCoT builds on NF-CoT’s VAE- and flow-based target construction and replaces autoregressive latent generation with shared-transformer refinement. However, Table 2 shows that the single-loop variant already achieves 62.9, compared with 63.1 for two loops, while increasing the depth to six or eight loops reduces performance to 61.2 and 61.7. The final model reaches 63.6 by combining mean feedback with auxiliary supervision, leaving the added value of looped feedback insufficiently established.

  • The experimental coverage is insufficient to establish the method’s broader effectiveness. All experiments use Qwen3-1.7B on code generation. The lack of evaluation on larger models or mathematical reasoning and conversational benchmarks leaves its scalability and cross-domain generalization unclear.

Questions for authors
  • Could the authors clarify the training datasets, reasoning-trace sources, and training budgets for all methods, and provide controlled comparisons to disentangle architectural gains from differences in training settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.