- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents MixerLoop, a linear attention architecture that loops the Gated Delta Net block N times within each individual layer of the language model. Additionally, the authors optimize their method specifically to the KV260 accelerator. The paper conducts experiments on 10M-450M param models that are trained on 10B tokens and downstream evaluated on common benchmarks such as Arc, PIQA, CSQA and others. The authors show their optimization's throughput gains compared to baselines, as well as ablate the number of loop steps at test time in their 4-step trained model, finding that performance increases until 4 loop steps are reached.
Critical strengths
- The idea of looping language models, also linear attention architectures, is relevant, even though it is not novel
Critical weaknesses
Weakness 1) The idea is not novel: looped linear attention variants have been explored before [1, 2], as well as looping just the sequence mixer [3, 4]. I do not see a contribution beyond these papers.
Weakness 2) The method is severely underexplained
- You only briefly talk about how your method actually works in section 3.1. From this I understand that the main idea is to loop the Gated Delta Net layer, but the specifics remain quite unclear. For example, do you repeat the block within the residual stream, as in
h_{t+1} = h_t + GDN(h_t), or do you loop the GDN directly on itself as inGDN(GDN(...GDN(h_t))? I would guess it's the former, because you briefly mention each mixer iteration adding a scaled copy of its input to the output. But generally, expanding this section with formulae interleaved with explanations that clearly show the pipeline would have been much better. I also do not think that the methods section is the best location for defining your baselines, that should be done in the experiments section. - The rest of the methods section talks about how optimize the method for a GPU, and you gather activations for later analyses. This is not part of the method, rather the experiments/analysis section
Weakness 4) The GPU specific optimization is unscientific
- After the one-paragraph explanation of your actual method, the largest part of the methods section talks about hardware optimizations specific to one accelerator (KV260 FPGA). If this is meant to be the contribution of your paper, since it takes so much space in the methods section, then I would expect a much more scientific treatment of the matter: (1) theoretical motivations leading towards the optimizations you propose, (2) substantially more explanation of the different architectural components of the GPU components in the theoretical background, (3) a treatment of prior work on gpu optimizations for GDN and looped models, and (4) most importantly optimizations across different types of GPUs as well as ablations of different parts of your optimization method. This way, your manuscript seems more like an engineering report in which you present how you made a certain model run really fast on your specific hardware, rather than a scientific report. In my opinion, the paper would be substantially strengthened if you focused on your architectural contribution (MixerLoop) in the main text, extended your treatment of hardware optimizations to other GPU types and presented this in the appendix. The speed comparison of your model compared to equally optimized baselines could nonetheless be presented in the main paper results, just the method section shouldn't overwhelmed by engineering-heavy unscientific hardware considerations.
Weakness 4) The experiments as well as results are highly questionable
- You never mention what dataset you pre-train your models on
- The experimental results are questionable at the least:
-
- evaluating 13-100M parameter models on benchmarks like Arc, PIQA, CSQA will not give any measurable performance because of the lack of capacity and trained tokens. The technical reports of Pythia, Olmo, SmallLM, and Fineweb ablations show that e.g. ARC around 1B parameters minimum as well as ~100B training tokens, and PIQA and CSQA need at least 50B training tokens for above-chance performance
-
- Performance on benchmarks like ARC and PIQA is below random (50% for PIQA, 20% for CSQA, 25% for ARC)
-
- I would guess you report accuracy here, although you never mention it, and what does "baseline adjusted" mean? Does it mean you report the difference to the baseline, but if so, then what is the baseline? That remains unclear. And if what you are reporting is raw accuracy, then how can some numbers be negative?
-
- In no setting does MixerLoop (meaningfully) outperform FullLoop or even NoLoop, except "CS Algorithms" (although again I think these numbers are essentially meaningless because of scale)
-
- Regarding the hardware optimization experiments, again these seem more like an engineering report because of the lack of motivation and derivation, but even then the comparisons are unfair at best, because the other methods you compare to are not hardware-optimized in the same way your method is.
- Section 4.4 analyzes how performance improves with more loops, and the figures suggest that performance increases monotonically with more test-time compute. However, you show this on your 4-step looped model which is only ever trained to loop exactly four times, making any other number of loops off-distribution. For a scientific comparison, you should have trained your model on varying number of loops (e.g. poisson-distributed as in [5]) or trained separate models that loop 1, 2, 3 or 4 times. You also did not show what happens if you loop more than four times, e.g. 5, 6 or more times, and it is likely that performance would radically fall again, for the same reason that the states become off-distribution.
- Figure 6 is never mentioned in the main text, and its relevance remains unclear.
To convincingly show the effect of your method, you would need:
- appropriately sized models that can be downstream tested, at least 1-3B parameters trained on ~100-500B tokens
- appropriate ablations of each component of your method, for example different numbers of loops trained separately, models that train on varying numbers of loops (as in [5]), different token mixers (GDN2, Linear Attention, Gated Linear Attention, Softmax Attention), etc.
- appropriate comparisons not only to baselines but the current state of the art in looped language models, e.g. Huggin and others
[1] LT2: Linear-Time Looped Transformers https://arxiv.org/abs/2605.20670 [2] CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models https://arxiv.org/abs/2607.10110 [3] Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping https://arxiv.org/abs/2603.23998 [4] DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products https://arxiv.org/abs/2502.10297 [5] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach https://arxiv.org/abs/2502.05171
Questions for authors
- Can you make the definition of your method more specific. What exactly do you loop?
- Please provide a general optimization treatment for other accelerators as well as motivation and background for hardware optimization
- Can you explain what you mean by "baseline adjusted" in Table 1? Is this the reason why performances are sub-random or even negative?
- Can you provide appropriate ablations as well as comparisons to state of the art baselines?
Response to Reviewer oBfe (1/4)
We sincerely thank the reviewer for the detailed feedback and constructive questions.
W1 & Q1: Novelty and Architectural Motivation
We cite SGT and LT2 in Section 2 precisely because they establish two important starting points for our work. SGT demonstrates that recurrence can be localized within the attention mechanism rather than applied to the entire Transformer block, with its benefits attributed to selectively looping high-entropy attention heads. LT2, on the other hand, investigates full-block looping with linear-time token mixers and demonstrates that GDN is particularly promising for recurrent computation, exhibiting strong performance and training stability. These two observations motivate our central question: if selective attention looping is beneficial in softmax-based Transformers, can looping only the GDN mixer also provide benefits, even though GDN does not have softmax attention entropy?
Unlike SGT, which selectively loops high-entropy attention heads, MixerLoop repeats the entire GDN mixer within each layer. Unlike LT2, which repeats the full block stack, including the FFNs, MixerLoop performs multiple GDN iterations while applying the FFN only once. Our investigation therefore focuses on whether recurrence in the context-dependent mixer is sufficient to preserve most of the benefits of full-block looping, without repeatedly executing the position-wise FFN.
We selected GDN because LT2 had already demonstrated its strong suitability for recurrent computation among linear-time mixers. Our experiments further show that this choice of recurrence boundary preserves performance close to FullLoop while reducing backbone projection computation by approximately 43.3%.
The hardware contribution follows directly from this architectural choice. By keeping the GDN mixer weights on-chip and overlapping additional mixer iterations with FFN weight transfers, we demonstrate that four-step MixerLoop can achieve nearly the same decoding throughput as NoLoop under the evaluated KV260 configuration. Beyond this implementation, we also derive a general analytical condition relating the number of iterations, model dimensions, and memory bandwidth, which characterizes when the additional mixer computation can be hidden behind weight transfers.
We do not claim that our conclusions about recurrence allocation necessarily generalize to softmax attention or every other token mixer. Rather, our contribution is to investigate and demonstrate this trade-off specifically for GDN, building on the complementary findings of SGT and LT2.