acceptodds
Under review as a conference paper at ICLR 2027

Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers

Abstract

Language models typically reason via explicit chain-of-thought (CoT), generating intermediate steps token-by-token. Latent CoT offers an alternative: it performs multi-step reasoning in the model's hidden states, replacing decoded tokens with continuous representations for greater efficiency. However, existing latent CoT methods underperform explicit CoT beyond 1B parameters, and the gap widens with scale. Looped, or recurrent-depth, Transformers, which reuse their weights to increase computation depth without adding parameters, are a natural fit for latent reasoning. We therefore ask whether looped Transformers can bridge this gap. We answer affirmatively with a simple recipe: a looped padded Transformer that processes K latent blocks in parallel for R iterations, with a cross-entropy loss on each latent position's gold CoT-step token, similar to explicit CoT supervision. We instantiate it as LOTUS (Looped Transformers with parallel supervision on latents). LOTUS is, to our knowledge, the first latent-CoT method to bridge the gap to explicit CoT at the 3B scale, while cutting thought-phase latency by 2.5x-6.9x from compact math expressions to natural language. Projecting LOTUS's post-loop latents through the base LM head recovers the gold reasoning steps and even surfaces alternative valid intermediate steps, evidence that its latent space is interpretable and CoT-aligned. Ablations confirm that both the looped backbone and the parallel supervision on gold CoT tokens are essential.

Primary area:
foundation or frontier models, including LLMs
Keywords:
transformers, latent reasoning, CoT

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CNcaOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LOTUS, which combines a looped padded Transformer with direct, position-aligned supervision on gold CoT tokens. A fixed set of latent blocks is refined in parallel, after which the model generates the answer conditioned on the resulting continuous representations. Experiments across GPT-2 and Llama-3.2 1B/3B show that LOTUS substantially narrows the accuracy gap to explicit CoT while reducing inference latency. The paper also studies supervision choices, latent width, loop depth, and the reasoning content recoverable from the learned latents.

Critical strengths
  1. Clear motivation and sound methodology. The paper addresses a well-motivated problem: the accuracy gap between latent and explicit reasoning. Combining looped latent refinement with direct CoT supervision provides a simple and technically sound approach.
  2. Strong empirical results. LOTUS substantially narrows the gap between latent and explicit CoT at the 3B scale while reducing thought-phase latency. The ablation studies further support the importance of the proposed components.
  3. Clear presentation. The paper is well written and easy to follow. The figures and tables effectively illustrate the architecture, training objectives, and experimental results.
Critical weaknesses
  1. The evaluation is limited to models up to 3B and GSM8K-related benchmarks. Results on an 8B backbone and harder math tasks, such as MATH-500, would strengthen the evidence that LOTUS’s accuracy–latency trade-off generalizes to larger models and more complex reasoning.
  2. The training overhead is not quantified. LOTUS undergoes additional fine-tuning from an explicit-CoT checkpoint, but its training costs and the comparability of baseline training budgets remain unclear. This limits assessment of the overall computational trade-off.
Questions for authors
  1. Could the authors evaluate LOTUS on an 8B backbone and MATH-500, comparing accuracy and latency with explicit CoT under matched settings?
  2. On harder math problems, does increasing latent tokens or loop iterations beyond the training budgets improve accuracy? Please vary these independently using the same checkpoint and report end-to-end latency.
  3. What is LOTUS’s training overhead over explicit CoT at 3B, in GPU-hours and peak memory? Please separate initial CoT training from additional LOTUS training and clarify whether baseline training budgets are comparable.
1 comment
AuthorsOpenReview ↗

Response to Reviewer CNca

We thank the reviewer for the positive assessment and the concrete suggestions!

> "The evaluation is limited to models up to 3B"

> "harder math tasks, such as MATH-500"

Larger models. The trend up to 3B is encouraging: as the backbone grows from GPT-2 to 3B, prior latent methods fall further behind explicit CoT, while LOTUS stays within about 1.5 points at each scale (Section 4.2). Training at 8B scale exceeds our compute budget, but SIM-CoT's results suggest that for Llama models fine-tuned on GSM8k-Aug, explicit CoT has already plateaued: it gains only 0.2 points from Llama-3.2-3B to Llama-3.1-8B (71.5 to 71.7). An 8B Llama comparison would therefore likely add little beyond our 3B results.

Harder math tasks. Our evaluation includes a harder variant of GSM8K: GSM-Hard, whose questions use much larger numbers, is one of our out-of-domain benchmarks (Table 1). At 3B, LOTUS comes within 1.0 point of explicit CoT on it (16.0 vs 17.0) and is ahead of prior latent methods. Our step beyond compact arithmetic expressions is the natural-language experiment, whose full-sentence CoT traces are longer and less regular; there, LOTUS matches explicit CoT in accuracy and its thought phase is 6.9× faster than explicit CoT's (Table 3). Unfortunately, training on harder math datasets currently exceeds our compute budget.

> "On harder math problems, does increasing latent tokens or loop iterations beyond the training budgets improve accuracy? Please vary these independently using the same checkpoint and report end-to-end latency."

Tables 6 and 7 vary RR and cc independently on the same 3B checkpoint (trained at R=6R=6, c=25c=25). One more iteration does not help (R=7R=7: 69.3 vs 70.0), and wider blocks add 0.5 points (c=30c=30 and c=50c=50: 70.5 vs 70.0). R=7R=7 adds one more pass, about 19 ms (Table 2: 133 ms over 7 passes); c=30c=30 and c=50c=50 add 3 and 8 ms over c=25c=25. Prefill and answer time do not change, so these add directly to LOTUS's 181.2 ms end-to-end latency (Table 2).

On harder problems with deeper serial dependencies, a larger budget could help, given available CoT supervision for the extra steps: depth pays off when it is trained (Table 5: 52.6% at R=4R=4 to 70.0% at R=6R=6).

> "What is LOTUS’s training overhead over explicit CoT at 3B, in GPU-hours and peak memory? Please separate initial CoT training from additional LOTUS training and clarify whether baseline training budgets are comparable."

Overall, LOTUS's training is slower per step than explicit CoT's, mainly due to its six sequential passes, but comparable to that of latent baselines with six sequential steps; its wider latent blocks mainly add memory. Training is a one-time cost; in exchange, LOTUS's thought phase is 2.5× to 6.9× faster than explicit CoT's (Tables 2 and 3).

GPU-hours. For Llama-3.2-3B-Instruct on B200 GPUs with a global batch of 128, initial explicit-CoT fine-tuning took 14 GPU-hours (3.4 h on 4 GPUs, 10 epochs), and additional LOTUS fine-tuning took 246 GPU-hours (30.7 h on 8 GPUs, 30 epochs including the curriculum). The first stage also produces the explicit-CoT baseline and is the same starting point Coconut uses. The two stages ran on different numbers of GPUs, so their training time and peak memory are not directly comparable; we compare both on a single GPU below.

Controlled comparison of per-step time and peak memory. We measured single-GPU bf16 forward+backward cost with batch size 4 on Llama-3.2-3B-Instruct as an example on H100. CoT SFT takes 55.9 ms / 13.5 GB. Coconut with six sequential latent steps takes 379.8 ms / 15.0 GB. LOTUS with R=6R=6, c=25c=25 takes 535.1 ms / 29.5 GB. Thus, LOTUS is more expensive than CoT SFT, but the extra time over six-step Coconut is moderate (1.41×). A sanity check with LOTUS at R=6R=6, c=1c=1 takes 483.4 ms / 14.4 GB, showing that most of the time cost comes from the six sequential refinement passes, while the larger parallel block width mainly increases activation memory.

Baseline training budgets. Per training step, the baselines are comparable to LOTUS with different cc and RR. As stated in Section 4.1, CODI and CODI + SIM-CoT "share the sequential latent reasoning budget (R=6R=6) with LOTUS and differ only in the number of latent tokens per step", i.e., they correspond to c=1c=1. We report their published numbers and did not time their training, but each of their steps also runs a teacher pass over the explicit CoT, and SIM-CoT adds an auxiliary decoder of the backbone's size. Since widening from c=1c=1 to c=25c=25 costs mostly memory and only about 10% more time (above), LOTUS's per-step time is likely comparable to theirs. PCCoT and KaVa use "3 sequential steps and 24 latent tokens in total" (Section 4.1), i.e., LOTUS with R=3R=3 and a much smaller cc (24 latent tokens instead of our 150); PCCoT reports accuracy peaking at about 3 iterations and degrading beyond that, which is why KaVa keeps 3 (Appendix F).

Reviewer_8F4WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LOTUS, a latent reasoning framework that combines Looped Transformers with parallel supervision of latent reasoning states. By iteratively refining latent blocks and aligning them with gold Chain-of-Thought (CoT) tokens, LOTUS aims to reduce sequential reasoning overhead while maintaining the accuracy of explicit CoT. Experiments on mathematical reasoning benchmarks show that LOTUS approaches explicit CoT performance at the 3B scale while achieving lower inference latency. The paper also provides ablation studies investigating latent supervision, block width, and loop depth.

Critical strengths
  1. Promising empirical results. LOTUS achieves competitive accuracy compared with explicit CoT and several existing latent reasoning approaches. The reported improvements in reasoning efficiency are encouraging, particularly at the 3B scale.

  2. Useful experimental analysis. The paper includes ablations on supervision strategies, latent block width, and recurrent depth, which help explain the effectiveness of the proposed design.

Critical weaknesses
  1. Limited methodological novelty. The proposed approach primarily combines existing Looped Transformer architectures with direct CoT supervision using a standard cross-entropy objective. Although this combination is effective, the methodological contribution appears relatively incremental compared with existing work on recurrent computation and supervised latent reasoning.

  2. Limited experimental generalization. The experiments focus almost exclusively on mathematical reasoning benchmarks, with the largest evaluated backbone containing only 3B parameters. Consequently, it remains unclear whether the reported advantages extend to larger language models, more complex reasoning tasks, or domains beyond mathematical reasoning. Moreover, the fixed latent block configuration may limit its applicability to problems requiring substantially longer reasoning chains.

Questions for authors
  1. Beyond combining recurrent computation and parallel CoT supervision, what is the key methodological contribution that distinguishes LOTUS from existing latent reasoning approaches? Could the authors further clarify this distinction?

  2. How well does LOTUS generalize to larger language models and non-mathematical reasoning tasks? Additionally, how sensitive are its accuracy and efficiency advantages to substantially longer reasoning chains that exceed the predefined latent block budget?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8F4W

We thank the reviewer for the thoughtful feedback and for finding our results promising and our ablations useful.

> "The proposed approach primarily combines existing Looped Transformer architectures with direct CoT supervision using a standard cross-entropy objective."

> "the methodological contribution appears relatively incremental compared with existing work on recurrent computation and supervised latent reasoning."

> "Beyond combining recurrent computation and parallel CoT supervision, what is the key methodological contribution that distinguishes LOTUS from existing latent reasoning approaches?"

First, the key methodological contribution is aligning each CoT step with blocks of the padded latent workspace across all loops, so LOTUS is not a mere combination of existing Looped Transformer architectures with CoT supervision: Existing looped language models such as LoopLM loop per token for next-token prediction and have no parallel latent workspace for the CoT (Appendix D). This alignment is what lets the standard cross-entropy of explicit CoT supervise all steps in parallel. Among the methods we compare with, LOTUS is the only one that refines its latents in parallel and trains them directly on the gold CoT tokens (Appendix F), and the only one within 1.5 points of explicit CoT at 3B (Section 4.2; Tables 1 and 12).

Second, neither part works alone: the looped model without latent supervision reaches 63.3% (Table 4), and dense step supervision on a sequential model (CODI + SIM-CoT) reaches 62.3%; together they reach 70.0%.

Third, we see the simplicity of LOTUS as a strength rather than a limitation: Prior methods added increasingly elaborate supervision while the gap to explicit CoT widened with scale (Section 4.2); LOTUS shows that once each CoT token has a fixed latent position, the plain cross-entropy of explicit CoT is enough.

> "it remains unclear whether the reported advantages extend to larger language models"

The trend up to 3B is encouraging: as the backbone grows from GPT-2 to 3B, prior latent methods fall further behind explicit CoT, while LOTUS stays within about 1.5 points at each scale (Section 4.2). Training LOTUS at 8B exceeds our compute budget, but SIM-CoT's results suggest that, for Llama models fine-tuned on GSM8k-Aug, explicit CoT has already plateaued: it gains only 0.2 points from Llama-3.2-3B to Llama-3.1-8B (71.5 to 71.7). An 8B Llama comparison would therefore likely add little beyond our 3B results. Looped Transformers have been pretrained stably at billions of parameters (Geiping et al., 2025a; Zhu et al., 2025c), so the architecture is not an obstacle to scaling.

> "more complex reasoning tasks, or domains beyond mathematical reasoning"

GSM8K is the common benchmark of latent reasoning methods: Coconut, CODI, SIM-CoT, PCCoT and KaVa all train on its augmented version, GSM8k-Aug, and evaluate on its test set. We follow the same setup so that results are directly comparable. For harder math, our out-of-domain benchmarks include GSM-Hard, whose questions use much larger numbers; at 3B, LOTUS comes within 1.0 point of explicit CoT on it (16.0 vs 17.0) and is ahead of prior latent methods (Table 1). Our step beyond compact arithmetic expressions, toward more general reasoning, is the natural-language experiment, whose full-sentence CoT traces are longer and less regular; there, LOTUS still matches explicit CoT (68.13 vs 68.41) and beats every latent baseline (Table 13).

> "the fixed latent block configuration may limit its applicability to problems requiring substantially longer reasoning chains."

First, even with a fixed KK, longer chains are still possible: As stated in the submission, "Longer problems fall back to autoregressive completion" (Appendix G; Section 4.1, footnote 2); in training, only the first KK steps become latent blocks and "the rest remain visible tokens" (Appendix G), and at inference the model writes them out autoregressively as text rather than as latent thoughts. Since training and inference use the same format, this fallback introduces no train-test mismatch that could hurt accuracy on longer chains or make it sensitive to chain length.

Second, we also show LOTUS still matches explicit CoT on longer chains with a larger cc per block: On natural-language CoT, whose full-sentence steps make the chains much longer (Section 4.2), we tune the block configuration by doubling the block width to c=50c=50 tokens (Appendix G), and LOTUS stays on par with explicit CoT (68.13 vs 68.41) while its thought phase is 6.9× faster (Table 3).

Thank you for your rebuttal and clarification.

Regarding novelty, I am only concerned that there have been multiple works on looped transformers in the field of latent reasoning (e.g., [1], [2], [3], [4], [5]), and the novelty of applying looped transformers to latent reasoning is definitely gone. The novelty of this paper is mainly reflected in the implementation method and details. 

Using only mathematical reasoning tasks for the experiment section is definitely not enough. Because a common situation is that the effect is good on a certain type of task, but not very good on another task. For the top conference, using only one type of dataset is not convincing enough. Other task datasets, such as General Knowledge and Commonsense Reasoning, Programming, Language Modeling and Reading Comprehension, Complex Multi-hop and Multidisciplinary QA, Multi-modal Reasoning. You can refer to the summary in a survey paper [6].

Coconut and CODI are relatively old papers in the field of latest reasoning. In 2026, a large number of new papers will emerge, and using only mathematical datasets is not enough. Currently, top conference papers usually need to conduct experiments on tasks in multiple fields.

If the method only works well in the field of mathematical, it means that it is only applicable to the mathematical domain, and the generalization of the method is very poor. 

I will reconsider the rating.

Ref.

[1] looped transformer (Reasoning with latent thoughts: On the power of looped transformers, ICLR 2025), 

[2] Huginn (Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach, arXiv, 2025), 

[3] Ouro (Scaling Latent Reasoning via Looped Language Models), 

[4] RD-VLA (Recurrent-Depth VLA: Implicit Test-Time Compute Scaling of Vision-Language-Action Models via Latent Iterative Reasoning, arXiv 2026)  

[5] RELAY (Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning, EACL, 2026) 

[6] Implicit Reasoning in Large Language Models: A Comprehensive Survey, arXiv, 2025

Thank you for the reply!

> "the novelty of applying looped transformers to latent reasoning is definitely gone"

We appreciate the references, and would like to clarify that the loops in [1]–[5] are not the same as LOTUS's loop and [1]–[3] are already discussed in Appendix D. LOTUS loops over a workspace of padded latent positions in which each thinking step has its own block, refined in parallel and supervised directly by that step's explicit CoT reference; none of [1]–[5] loops over such a workspace:

  • [1]–[3] loop the layers applied to each token, so their latent reasoning is extra depth per token (Appendix D). Their reported gains are in accuracy for a given number of parameters ([1], [3]) or from spending more test-time compute ([2]); as opposed to LOTUS, they report no gain in reasoning latency, and they do not supervise their loops with CoT steps.
  • [4] loops the action head of a vision-language-action model for robot control, a different problem from replacing a language model's CoT.
  • [5] is the closest: it aligns CoT steps with loop iterations, but uses the looped model to generate CoT data for fine-tuning an autoregressive model, which still writes its CoT token by token, so it does not improve reasoning latency. LOTUS instead aligns each CoT step with a fixed block of latent positions, refines all blocks in parallel, and decodes only the answer, so its thought phase is 2.5× to 6.9× faster than explicit CoT's at nearly the same accuracy (Tables 1–3).

We will add discussion of [4] and [5] to Appendix D.

> "Using only mathematical reasoning tasks for the experiment section is definitely not enough."

We choose math since this is where prior latent reasoning methods find it hard to match the performance of explicit CoT: beyond 1B parameters, latent methods fall behind explicit CoT on math, and the gap widens with scale (Section 1). Common non-math benchmarks have not shown this gap. On CommonsenseQA, CODI reports that training on explicit CoT "actually hurts performance" for LLaMA-1b, and that CODI, a latent method, beats explicit CoT for both GPT-2 and LLaMA-1b, so there is no gap to bridge. PCCoT's authors also tried CLUTRR, CommonsenseQA and StrategyQA, but report that they "cannot distinguish the performance of different approaches on these datasets" as well.

Nothing in LOTUS is specific to math: it only needs CoT traces split into steps, and it already works with full-sentence traces as well as arithmetic expressions. Beyond math, we also ran LOTUS with GPT-2 on ProsQA, the multi-hop logical reasoning benchmark: on its validation split, LOTUS reaches 96.7%, 20 points above explicit CoT (76.7%) and on par with Coconut (97.0%). As on CommonsenseQA, explicit CoT is weak on this task, so there is no gap to close, but the result shows that LOTUS is not limited to GSM8K.

> "Currently, top conference papers usually need to conduct experiments on tasks in multiple fields."

The most related latent-CoT methods we compare with also evaluate only on math: SIM-CoT (ICLR 2026) and KaVa (ICLR 2026). We follow their benchmarks so that our results are directly comparable to theirs.

After considering (1) the author's rebuttal, (2) the comments of other reviewers, (3) the contribution of the paper to the field, and (4) especially the overall quality of the current manuscript, I tend to maintain my initial rating.

Reviewer_WjbhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LOTUS uses a looped Transformer with direct latent supervision on position-aligned CoT tokens. It iteratively refines latent states and generates answers from continuous representations; LOTUS-aux adds an auxiliary decoder for training supervision. Evaluated on arithmetic benchmarks with GPT-2 and Llama-3.2 1B/3B, LOTUS matches explicit CoT accuracy while drastically cutting latency.

Critical strengths
  1. On Llama-3.2-3B, LOTUS yields a good accuracy-latency tradeoff: comparable accuracy to explicit CoT, better OOD performance, and substantial latency reduction. The 6.84× speedup is only for the thought phase.
  2. Controlled ablations confirm that post-loop direct supervision is the key. Negative findings (weak LOTUS-aux on small models, no gain from more inference iterations) further support the design.
Critical weaknesses
  1. The core efficient inference procedure is not fully specified. There are inconsistencies between Eq. (2), Figure 2 and Section 3.2 regarding the number of forward passes and token delimiters. Eq. (4) does not clarify how latent states form the KV context, and the autoregressive fallback lacks clear trigger and stopping rules. These ambiguities affect whether all computation is properly accounted for.
  2. PCCoT / KaVa baselines adopt different latent budgets and iteration counts from LOTUS. There is no shared hardware accuracy-latency frontier for fair comparison, and the consistency of training checkpoints needs clarification.
  3. The paper mixes reference numbers from different sources. Recalculation shows the performance gap is larger than claimed, so this statement needs correction.
  4. The OOD improvement is partial and limited. The overall OOD gain is not universal, and LOTUS is still worse than explicit CoT on GSM-Hard.
Questions for authors
  1. Can you provide inference pseudocode to clarify forward passes, delimiter handling, latent KV cache construction, and the trigger, termination, frequency and overhead of the autoregressive fallback after K steps?
  2. Can you fix the inconsistent KaVa references, clarify which baselines are rerun vs imported, and report checkpoint/seed details for the key comparisons?
  3. Could you add a common-hardware comparison with parallel baselines and uncertainty for accuracy gaps?
  4. Which data split was used for Table 9 (test set or filtered training set)? Does the intermediate-number recovery result prove coherent causal reasoning on unseen chains?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Wjbh (1/3)

We thank the reviewer for the positive assessment, careful reading, and recognizing the good accuracy-latency tradeoff and our controlled ablations.

> "Can you provide inference pseudocode to clarify forward passes, delimiter handling, latent KV cache construction, and the trigger, termination, frequency and overhead of the autoregressive fallback after K steps?"

Section 3.2 and Section 3.3.1 (Inference) already specify inference, formalized in Eqs. (2) and (4). For extra clarity, here it is as pseudocode for one question:

C = f([Q, e_BoT]).kv                    # question and BoT, encoded once
h = f(E | C).h_lat                      # initial latent pass: h(0)
for t in 1..R:                          # R looped iterations
    out = f(E + h | C)                  # input injection
    h = out.h_lat                       # h(t)
KV = C + out.kv                         # cache of the final iteration
y = argmax f(e_EoT | KV)                # answer pass starts at EoT
while y != EOS and len(ans) < max_len:  # greedy decoding; each pass appends to KV
    ans.append(y); y = argmax f(e_y | KV)

> "There are inconsistencies between Eq. (2), Figure 2 and Section 3.2 regarding the number of forward passes and token delimiters."

We thank the reviewer for pointing out a small mistake in Figure 2's caption, which we will fix: it lists ⟨BoT⟩\langle BoT\rangle and ⟨EoT⟩\langle EoT\rangle in the loop and omits the initial latent pass. Section 3.2 and Eq. (2) are correct and consistent with each other. Section 3.2's text says that LOTUS "first runs a single forward pass over the prefix [Q,⟨BoT⟩][Q, \langle BoT\rangle]" and "then iterates fθf_\theta for RR iterations, each refining the latent embeddings while attending to CpreC_{\text{pre}} (the trailing ⟨EoT⟩\langle EoT\rangle and the answer suffix AA do not enter the loop)." Eq. (2) writes out the initial latent pass, h(0)=fθ(E∣Cpre)h^{(0)} = f_\theta(E \mid C_{\text{pre}}), followed by these RR iterations (t=1,…,Rt = 1, \dots, R), so the latents take R+1R+1 passes, all included in Table 2's thought time; ⟨EoT⟩\langle EoT\rangle then starts the answer pass (Eq. (4), Figure 2(b)).

> "Eq. (4) does not clarify how latent states form the KV context"

Sections 3.2 and 3.3.1 specify this. Section 3.2: "LOTUS first runs a single forward pass over the prefix [Q,⟨BoT⟩][Q, \langle BoT\rangle] to populate a KV cache CpreC_{\text{pre}}, which is reused throughout training and inference without recomputation." Section 3.3.1, introducing Eq. (4): "Lans\mathcal L_{\text{ans}} is computed in a separate final forward of fθf_\theta that reuses the prefix cache CpreC_{\text{pre}} and inserts the post-loop latent hidden states h(R)h^{(R)} at the latent rows" (drawn in Figure 2(b)). Concretely, the keys and values at the latent rows are those computed in the final looped iteration, which produces h(R)h^{(R)}; no extra pass over the latents is run.

> "the autoregressive fallback lacks clear trigger and stopping rules"

There is no separate fallback to trigger. As described in Section 3.3.1 (Inference), LOTUS caches the question once, performs RR looped refinements, and then decodes autoregressively after ⟨EoT⟩\langle EoT\rangle. The fallback in footnote 2 (Section 4.1: "falls back to autoregressive completion otherwise") is part of this.

In training, steps after the KK-th stay as text after ⟨EoT⟩\langle EoT\rangle ("the rest remain visible tokens", Appendix G), so the model learns to write them before the answer. At inference, decoding after ⟨EoT⟩\langle EoT\rangle thus writes any such steps and then the answer, and stops at end-of-sequence or after max_len tokens. KK covers most chains (only 0.81% of training examples and 30 of the 1,319 test questions have more than KK steps), and LOTUS outputs only 3.2 tokens after ⟨EoT⟩\langle EoT\rangle on average, including any CoT steps beyond KK, so the fallback adds little overhead.

AuthorsOpenReview ↗

Response to Reviewer Wjbh (2/3)

> "PCCoT / KaVa baselines adopt different latent budgets and iteration counts from LOTUS."

> "There is no shared hardware accuracy-latency frontier for fair comparison"

> "Could you add a common-hardware comparison with parallel baselines"

Table 2 times LOTUS, explicit CoT and our main latent baselines, SIM-CoT and CODI (which share LOTUS's sequential latent budget, R=6R=6), on the exact same H100. With Table 1's accuracies, this already gives the shared-hardware accuracy-latency frontier for these methods.

We treat PCCoT and KaVa as extra baselines (Table 12) since their published settings differ: 24 latent tokens over 3 Jacobi iterations (Appendix F). More iterations reportedly hurt them: PCCoT reports lower accuracy and unstable training, and KaVa reports the same accuracy drop. At their best they both stay well below our performance at 3B. LOTUS instead keeps improving with more iterations up to R=6R=6 (Table 5).

We did not find official KaVa code or checkpoint, which makes the common-hardware comparison on it difficult. However, PCCoT and KaVa correspond to LOTUS with R=3R=3 and a much smaller cc: 24 latent tokens in total instead of our 150 (Section 4.1). In our measurements, thought time depends mainly on the number of sequential passes, not on width: going from 6 to 300 latent positions adds only 30 ms (Table 7). With 3 iterations instead of 6, PCCoT and KaVa can therefore be faster than LOTUS, but at 1B and 3B they are less accurate. At 3B, LOTUS is the only latent method within 1.5 points of explicit CoT (Section 4.2), and its thought phase is still 2.5× faster than explicit CoT's (Table 2).

> "The paper mixes reference numbers from different sources. Recalculation shows the performance gap is larger than claimed, so this statement needs correction."

SIM-CoT reports 58.4 (1B) and 71.5 (3B) for explicit CoT, and our own explicit-CoT checkpoints match them. The latest version of KaVa's paper reports 63.4 and 73.2 respectively. Although KaVa reportedly uses the same data and backbones, we cannot reproduce these numbers, and we did not find its code or checkpoint to check them against. We therefore use only explicit-CoT numbers we can verify with our own runs:

  • LOTUS trails SIM-CoT's explicit CoT by 1.1 (1B) and 1.5 (3B).
  • KaVa trails the same reference by 1.9 and 5.8.
  • Natural-language CoT is where our explicit CoT number matches KaVa's, and LOTUS reaches 68.13 versus KaVa's 60.0.

So "within 1.5 points" holds against the explicit CoT we can verify; LOTUS's gap grows only against KaVa's 63.4 and 73.2, which we cannot verify and, as a precaution, did not use in the paper. Even against the explicit CoT numbers from KaVa, LOTUS's gap still stays smaller than KaVa's own.

> "Can you fix the inconsistent KaVa references"

Yes. We will cite the latest version of KaVa's paper throughout, since all our KaVa numbers come from it. As explained above, we will also state in Section 4.2 and Table 12 that every gap uses explicit CoT we can verify.

> "clarify which baselines are rerun vs imported"

Explicit CoT performance is imported from SIM-CoT and verified with our own runs as discussed above; all other baselines are imported from published results. Run by us: all LOTUS variants, both natural-language models (explicit CoT and LOTUS), and all latency measurements.

AuthorsOpenReview ↗

Response to Reviewer Wjbh (3/3)

> "report checkpoint/seed details for the key comparisons"

> "the consistency of training checkpoints needs clarification"

> "uncertainty for accuracy gaps"

We reported both checkpoint and seed details: the Table 1 caption states "Results are reported as mean ± std across 3 seeds", and Appendix G that we "select the best checkpoint by GSM8k validation accuracy". Tables 3 and 13 also report mean ± std for the natural-language comparison, and Table 3 states that LOTUS is "statistically on par with explicit CoT accuracy". For the other gaps where imported baselines have no std, the 95% interval for a difference between two accuracies near 70% on 1,319 test questions is about ±3.5 points: the 1.5-point gap to explicit CoT is inside it; the gaps to CODI + SIM-CoT (7.7) and KaVa (4.3) are not.

> "The overall OOD gain is not universal"

> "LOTUS is still worse than explicit CoT on GSM-Hard."

Our claim is about the average (63.9 vs 62.1 at 3B). LOTUS beats explicit CoT on MultiArith (99.9 vs 98.3) and SVAMP (75.7 vs 71.0), and trails it only on GSM-Hard, by 1.0 point (16.0 vs 17.0), which is within test-set noise (about ±2.8); even there, LOTUS is ahead of every prior latent method.

> "Which data split was used for Table 9 (test set or filtered training set)?"

As stated in Appendix I.2, Table 9 uses training questions: the multi-path bank "is built from the original GSM8K problems, which are also in the training set." "For each question we identify its trained path (its gold chain in the training data) and an unseen-but-valid alternative drawn from a bank of up to ten correct paths obtained by rejection-sampling Llama-3.1-8B (Grattafiori et al., 2024) on the original GSM8K training questions (7,444 questions with 58,155 verified paths in total), with each alternative verified absent from training." Filtering then leaves "the final 341 questions". "Unseen" refers to the chain, so we need questions whose trained chain is known. Table 8 uses the 1,319 test questions.

> "Does the intermediate-number recovery result prove coherent causal reasoning on unseen chains?"

On training questions, no: Table 9 uses training questions, so the greedy readout follows the trained path; the unseen chain's intermediates are high-probability alternatives, most ranking in the top 5 but few top 1. On the 1,319 test questions, where no path was trained, the greedy readout can indeed follow a coherent chain, even "a valid alternative decomposition" of the reference (Appendix I.1, Example 2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.