- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces LOTUS, which combines a looped padded Transformer with direct, position-aligned supervision on gold CoT tokens. A fixed set of latent blocks is refined in parallel, after which the model generates the answer conditioned on the resulting continuous representations. Experiments across GPT-2 and Llama-3.2 1B/3B show that LOTUS substantially narrows the accuracy gap to explicit CoT while reducing inference latency. The paper also studies supervision choices, latent width, loop depth, and the reasoning content recoverable from the learned latents.
Critical strengths
- Clear motivation and sound methodology. The paper addresses a well-motivated problem: the accuracy gap between latent and explicit reasoning. Combining looped latent refinement with direct CoT supervision provides a simple and technically sound approach.
- Strong empirical results. LOTUS substantially narrows the gap between latent and explicit CoT at the 3B scale while reducing thought-phase latency. The ablation studies further support the importance of the proposed components.
- Clear presentation. The paper is well written and easy to follow. The figures and tables effectively illustrate the architecture, training objectives, and experimental results.
Critical weaknesses
- The evaluation is limited to models up to 3B and GSM8K-related benchmarks. Results on an 8B backbone and harder math tasks, such as MATH-500, would strengthen the evidence that LOTUS’s accuracy–latency trade-off generalizes to larger models and more complex reasoning.
- The training overhead is not quantified. LOTUS undergoes additional fine-tuning from an explicit-CoT checkpoint, but its training costs and the comparability of baseline training budgets remain unclear. This limits assessment of the overall computational trade-off.
Questions for authors
- Could the authors evaluate LOTUS on an 8B backbone and MATH-500, comparing accuracy and latency with explicit CoT under matched settings?
- On harder math problems, does increasing latent tokens or loop iterations beyond the training budgets improve accuracy? Please vary these independently using the same checkpoint and report end-to-end latency.
- What is LOTUS’s training overhead over explicit CoT at 3B, in GPU-hours and peak memory? Please separate initial CoT training from additional LOTUS training and clarify whether baseline training budgets are comparable.
Response to Reviewer CNca
We thank the reviewer for the positive assessment and the concrete suggestions!
> "The evaluation is limited to models up to 3B"
> "harder math tasks, such as MATH-500"
Larger models. The trend up to 3B is encouraging: as the backbone grows from GPT-2 to 3B, prior latent methods fall further behind explicit CoT, while LOTUS stays within about 1.5 points at each scale (Section 4.2). Training at 8B scale exceeds our compute budget, but SIM-CoT's results suggest that for Llama models fine-tuned on GSM8k-Aug, explicit CoT has already plateaued: it gains only 0.2 points from Llama-3.2-3B to Llama-3.1-8B (71.5 to 71.7). An 8B Llama comparison would therefore likely add little beyond our 3B results.
Harder math tasks. Our evaluation includes a harder variant of GSM8K: GSM-Hard, whose questions use much larger numbers, is one of our out-of-domain benchmarks (Table 1). At 3B, LOTUS comes within 1.0 point of explicit CoT on it (16.0 vs 17.0) and is ahead of prior latent methods. Our step beyond compact arithmetic expressions is the natural-language experiment, whose full-sentence CoT traces are longer and less regular; there, LOTUS matches explicit CoT in accuracy and its thought phase is 6.9× faster than explicit CoT's (Table 3). Unfortunately, training on harder math datasets currently exceeds our compute budget.
> "On harder math problems, does increasing latent tokens or loop iterations beyond the training budgets improve accuracy? Please vary these independently using the same checkpoint and report end-to-end latency."
Tables 6 and 7 vary R and c independently on the same 3B checkpoint (trained at R=6, c=25). One more iteration does not help (R=7: 69.3 vs 70.0), and wider blocks add 0.5 points (c=30 and c=50: 70.5 vs 70.0). R=7 adds one more pass, about 19 ms (Table 2: 133 ms over 7 passes); c=30 and c=50 add 3 and 8 ms over c=25. Prefill and answer time do not change, so these add directly to LOTUS's 181.2 ms end-to-end latency (Table 2).
On harder problems with deeper serial dependencies, a larger budget could help, given available CoT supervision for the extra steps: depth pays off when it is trained (Table 5: 52.6% at R=4 to 70.0% at R=6).
> "What is LOTUS’s training overhead over explicit CoT at 3B, in GPU-hours and peak memory? Please separate initial CoT training from additional LOTUS training and clarify whether baseline training budgets are comparable."
Overall, LOTUS's training is slower per step than explicit CoT's, mainly due to its six sequential passes, but comparable to that of latent baselines with six sequential steps; its wider latent blocks mainly add memory. Training is a one-time cost; in exchange, LOTUS's thought phase is 2.5× to 6.9× faster than explicit CoT's (Tables 2 and 3).
GPU-hours. For Llama-3.2-3B-Instruct on B200 GPUs with a global batch of 128, initial explicit-CoT fine-tuning took 14 GPU-hours (3.4 h on 4 GPUs, 10 epochs), and additional LOTUS fine-tuning took 246 GPU-hours (30.7 h on 8 GPUs, 30 epochs including the curriculum). The first stage also produces the explicit-CoT baseline and is the same starting point Coconut uses. The two stages ran on different numbers of GPUs, so their training time and peak memory are not directly comparable; we compare both on a single GPU below.
Controlled comparison of per-step time and peak memory. We measured single-GPU bf16 forward+backward cost with batch size 4 on Llama-3.2-3B-Instruct as an example on H100. CoT SFT takes 55.9 ms / 13.5 GB. Coconut with six sequential latent steps takes 379.8 ms / 15.0 GB. LOTUS with R=6, c=25 takes 535.1 ms / 29.5 GB. Thus, LOTUS is more expensive than CoT SFT, but the extra time over six-step Coconut is moderate (1.41×). A sanity check with LOTUS at R=6, c=1 takes 483.4 ms / 14.4 GB, showing that most of the time cost comes from the six sequential refinement passes, while the larger parallel block width mainly increases activation memory.
Baseline training budgets. Per training step, the baselines are comparable to LOTUS with different c and R. As stated in Section 4.1, CODI and CODI + SIM-CoT "share the sequential latent reasoning budget (R=6) with LOTUS and differ only in the number of latent tokens per step", i.e., they correspond to c=1. We report their published numbers and did not time their training, but each of their steps also runs a teacher pass over the explicit CoT, and SIM-CoT adds an auxiliary decoder of the backbone's size. Since widening from c=1 to c=25 costs mostly memory and only about 10% more time (above), LOTUS's per-step time is likely comparable to theirs. PCCoT and KaVa use "3 sequential steps and 24 latent tokens in total" (Section 4.1), i.e., LOTUS with R=3 and a much smaller c (24 latent tokens instead of our 150); PCCoT reports accuracy peaking at about 3 iterations and degrading beyond that, which is why KaVa keeps 3 (Appendix F).