- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LoopSpec, a method to decode specific steps within a looped transformer and compare with the actual decoded results, to decide whether a branch is accepted or pruned. This way, the recurrent steps within a loop transformer are less as some branches are initiated exactly as the original final decoded token, thus the decoding latency is reduced. The author also designed a gating mechanism to control the size of the branching tree.
The experiments demonstrates the efficiency gain of the proposed method. I would have scored higher. However, the variance is not reported, and it is not clear in the paper what is the cost of expanding a tree in a looped transformer.
Critical strengths
- The idea and the proposed method is reasonable and very interesting to me.
- The efficiency gain is clear and substantial.
- Experiment details are provided, along with code for reproduction.
Critical weaknesses
- The scope of the experiments is somewhat limited. Only two transformers are compared in this paper.
- The cost of decoding branches is not declared.
- The rationale for choosing the proposal depths in Table 1 is not given.
- The variance is not reported in any experiment。
Questions for authors
- Can you explain why no variation is provided?
- I didn't quite get the gating mechanism, could you further explain this? I think adding a bit illustration in Fig. 1 would help, instead of just stating "gate closed, no second proposal"
- Could you further elaborate the concept a power-law upper envelope in Fig. 3(b)?
4 comments
Initial Response (2/2)
W2. The cost of decoding branches is not declared
To quantify the computational overhead of decoding branches, we profile the latency of a single recurrent-block of Raven-Llama under LoopSpec (, ) with a 1024-token context. Each branch size is measured 10 times and we report the mean and std.
| Branches | 1 | 2 | 4 | 6 | 18 | 35 | 68 | 130 | 249 |
|---|---|---|---|---|---|---|---|---|---|
| Latency per recurrent (μs) | 696.2 ± 0.5 | 703.4 ± 0.8 | 723.5 ± 0.8 | 755.3 ± 0.5 | 773.5 ± 0.4 | 840.8 ± 0.5 | 972.3 ± 0.3 | 1459.3 ± 1.1 | 1909.8 ± 1.4 |
| Relative to 1 branch | 1.00x | 1.01x | 1.04x | 1.09x | 1.11x | 1.21x | 1.40x | 2.10x | 2.74x |
Branch size 1 to 249 covers the full range of the number of branches in LoopSpec for Raven-Llama model. As shown in the table, the latency increases substantially slower than the number of branches.
Furthermore, the branch size distribution (Figure 4) shows that the average branch size is 34.06 and mostly under 60. Therefore, the additional computation from branching introduces moderate latency overhead due to a memory-bound behavior, which is outweighed by the pipelined speculation of LoopSpec, resulting in the end-to-end speedups reported in our experiments.
W3. Rationale for choosing the proposal depths in Table 1
We would like to clarify that Section 5 of the paper provides a theoretically motivated method for selecting the proposal depths (Thm 2). For the Raven model family with 32 loops, the theorem gives . We empirically evaluate several nearby proposal depths configurations in Table 1. The results confirm that is the best configuration, which closely matches the theoretical result.
Reminder: Discussion Period Ending in a Few Days
Dear Reviewer SF7N,
Just a brief reminder that the author–reviewer discussion period concludes in a few days.
In our response, we report speedups over 10 random seeds with standard deviations of only 0.02–0.07x, e.g., 2.40 ± 0.03x on GSM8K and 2.34 ± 0.06x on AIME25 for Ouro-1.4B-Thinking. We also profiled the branch cost on Raven-Llama: per-recurrent latency grows from 696 μs at 1 branch to 1,910 μs at 249 branches, i.e., 2.74x for a 249x increase in branches, with a mean branch size of 34.06. We clarified questions including the gating mechanism, the power-law envelope in Fig. 3(b), and the choice of proposal depths via Theorem 2. To address the model scope, we added a third architecture, Dory1.0-4B, with 2.29x speedup on GSM8K.
If you feel these responses address your concerns, we would really appreciate it if you could acknowledge this and consider reassessing your rating. If anything remains unclear or you have any follow-up questions, we would be happy to address them within the remaining discussion period.
Thank you again for the time and effort you put into the review.
Best regards,
Authors
I appreciate the authors for their further explanations and added experiments. My concerns have been addressed by the additional variance and cost experiments, and I am happy to raise my score accordingly.
Initial Response (1/2)
We appreciate reviewer's insightful feedback. We would like to clarify and explain the questions and provide additional experiment results.
Q1/W4. Variance of the Speedup.
We address the reviewer’s concern on missing variance of speedup by the additional experiment in the following table. We ran the Ouro Thinking models under temperature=1.0 and top-p=0.7 and 10 random seeds. The results show consistent speedup over the vanilla models.
As shown in the table, LoopSpec consistently achieves substantial speedups, with low standard deviations (0.02–0.07x) across random seeds. These results demonstrate that LoopSpec's acceleration is stable and is not sensitive to the choice of random seed.
Q2. Explanation of Gating Mechanism
Overall, we introduce the gating mechanism to tackle the exponentially growing speculative branches in LoopSpec. Intuitively, the gate asks whether the deeper recurrent state still trusts the first draft x~(1). When a branch reaches depth d2, we get a second prediction pd2 about the same token x~(1) that was drafted at depth d1. We compare how much probability each depth assigns to the first draft x~(1):
Without gating, every branch forks at both depths and the number of branches grows exponentially (up to 249 for R=32, d1=2, d2=8). With gating, the gate opens for only 19-40% of branches (Table 8), which reduces the mean effective batch size from 43.75 to 34.06 (Figure 4).
Q3. Explanation of Power-Law Upper Envelope
Figure 3(b) measures how far the intermediate prediction at depth r is from the final prediction at depth R, using total variation (TV) distance. This distance empirically decreases as r grows. To reflect this in our analysis (Theorem 2), we find a simple formula that upper bounds every measured point. In particular, we choose the power law function ϕ(r)=β⋅r−α, which appears as a straight line on the log-log plot. The envelope is the tightest line that stays above all measured points: we pick α and β to minimize the maximum ratio between ϕ(r) and the measured TV.
W1. Limited models are compared
We would like to clarify that in the paper, we conducted experiments across 2 model families, 4 Transformer architectures, and 7 model checkpoints (Table 6), rather than only two Transformers.
To further strengthen the experiment, we additionally provide NVIDIA Nemotron-Labs Dory1.0-4B [1], a recently released Looped Transformer based on normalized-GPT (nGPT) with hybrid sliding-window attention and full attention, as well as GQA, differing from Ouro and Raven's Transformer architectures. We use R=4, and (d1,d2)=(1,2) for Dory.
As the table shows, LoopSpec performs substantial speedup on the Dory model, demonstrating its capability for more Transformer architectures.
[1] M. Fishman, S. Sun, S. Acharya, F. Ladhak, B. Chmiel, S. A. Serano, R. Banner, B. Ginsburg, and K. C. Puvvada. Dory1.0-4B: A Looped Normalized-GPT. arXiv:2611.03998, 2026.