acceptodds
Under review as a conference paper at ICLR 2027

Allocating Recurrent Compute in Looped Language Models

Abstract

Looped language models improve reasoning and knowledge manipulation by applying shared computation repeatedly. Existing systems usually repeat an entire layer stack, although a mixer and a dense feed-forward network (FFN) perform different operations and have different costs. We ask a narrower question: what should loop? We view recurrence as iterative refinement of a hidden representation and separate two effects that full-block recurrence couples: repeated context-dependent state updates in the mixer and repeated position-wise transformation in the FFN. This view motivates MixerLoop, which repeats each Gated DeltaNet mixer while applying its dense FFN once. Across matched pretraining experiments, MixerLoop preserves the performance benefits of full-block recurrence while reducing backbone projection compute by 43%, and consistently outperforms NoLoop. We observe that successive mixer applications continue to reduce next-token cross-entropy. MixerLoop also creates an opportunity during inference because mixer weights are reused across recurrent steps. We implement a custom FPGA accelerator on a KV260 that keeps these weights on chip and overlaps additional mixer computation with the transfer of the larger FFN weights. Using the same hardware and compute datapath, NoLoop and four-step MixerLoop both achieve approximately 152 tokens/s. In memory-bound decoding, MixerLoop therefore turns weight-transfer time into useful recurrent computation with nearly unchanged latency. Together, these results identify the token mixer as an effective recurrence boundary, retaining the benefits of recurrent depth while placing additional computation where it can be efficiently reused by hardware during inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Language Models, Linear Attention, Inference Acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jZxROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents MixerLoop, a linear attention architecture that loops the Gated Delta Net block N times within each individual layer of the language model. Additionally, the authors optimize their method specifically to the KV260 accelerator. The paper conducts experiments on 10M-450M param models that are trained on 10B tokens and downstream evaluated on common benchmarks such as Arc, PIQA, CSQA and others. The authors show their optimization's throughput gains compared to baselines, as well as ablate the number of loop steps at test time in their 4-step trained model, finding that performance increases until 4 loop steps are reached.

Critical strengths
  1. The idea of looping language models, also linear attention architectures, is relevant, even though it is not novel
Critical weaknesses

Weakness 1) The idea is not novel: looped linear attention variants have been explored before [1, 2], as well as looping just the sequence mixer [3, 4]. I do not see a contribution beyond these papers.

Weakness 2) The method is severely underexplained

  • You only briefly talk about how your method actually works in section 3.1. From this I understand that the main idea is to loop the Gated Delta Net layer, but the specifics remain quite unclear. For example, do you repeat the block within the residual stream, as in h_{t+1} = h_t + GDN(h_t), or do you loop the GDN directly on itself as in GDN(GDN(...GDN(h_t))? I would guess it's the former, because you briefly mention each mixer iteration adding a scaled copy of its input to the output. But generally, expanding this section with formulae interleaved with explanations that clearly show the pipeline would have been much better. I also do not think that the methods section is the best location for defining your baselines, that should be done in the experiments section.
  • The rest of the methods section talks about how optimize the method for a GPU, and you gather activations for later analyses. This is not part of the method, rather the experiments/analysis section

Weakness 4) The GPU specific optimization is unscientific

  • After the one-paragraph explanation of your actual method, the largest part of the methods section talks about hardware optimizations specific to one accelerator (KV260 FPGA). If this is meant to be the contribution of your paper, since it takes so much space in the methods section, then I would expect a much more scientific treatment of the matter: (1) theoretical motivations leading towards the optimizations you propose, (2) substantially more explanation of the different architectural components of the GPU components in the theoretical background, (3) a treatment of prior work on gpu optimizations for GDN and looped models, and (4) most importantly optimizations across different types of GPUs as well as ablations of different parts of your optimization method. This way, your manuscript seems more like an engineering report in which you present how you made a certain model run really fast on your specific hardware, rather than a scientific report. In my opinion, the paper would be substantially strengthened if you focused on your architectural contribution (MixerLoop) in the main text, extended your treatment of hardware optimizations to other GPU types and presented this in the appendix. The speed comparison of your model compared to equally optimized baselines could nonetheless be presented in the main paper results, just the method section shouldn't overwhelmed by engineering-heavy unscientific hardware considerations.

Weakness 4) The experiments as well as results are highly questionable

  • You never mention what dataset you pre-train your models on
  • The experimental results are questionable at the least:
      1. evaluating 13-100M parameter models on benchmarks like Arc, PIQA, CSQA will not give any measurable performance because of the lack of capacity and trained tokens. The technical reports of Pythia, Olmo, SmallLM, and Fineweb ablations show that e.g. ARC around 1B parameters minimum as well as ~100B training tokens, and PIQA and CSQA need at least 50B training tokens for above-chance performance
      1. Performance on benchmarks like ARC and PIQA is below random (50% for PIQA, 20% for CSQA, 25% for ARC)
      1. I would guess you report accuracy here, although you never mention it, and what does "baseline adjusted" mean? Does it mean you report the difference to the baseline, but if so, then what is the baseline? That remains unclear. And if what you are reporting is raw accuracy, then how can some numbers be negative?
      1. In no setting does MixerLoop (meaningfully) outperform FullLoop or even NoLoop, except "CS Algorithms" (although again I think these numbers are essentially meaningless because of scale)
  • Regarding the hardware optimization experiments, again these seem more like an engineering report because of the lack of motivation and derivation, but even then the comparisons are unfair at best, because the other methods you compare to are not hardware-optimized in the same way your method is.
  • Section 4.4 analyzes how performance improves with more loops, and the figures suggest that performance increases monotonically with more test-time compute. However, you show this on your 4-step looped model which is only ever trained to loop exactly four times, making any other number of loops off-distribution. For a scientific comparison, you should have trained your model on varying number of loops (e.g. poisson-distributed as in [5]) or trained separate models that loop 1, 2, 3 or 4 times. You also did not show what happens if you loop more than four times, e.g. 5, 6 or more times, and it is likely that performance would radically fall again, for the same reason that the states become off-distribution.
  • Figure 6 is never mentioned in the main text, and its relevance remains unclear.

To convincingly show the effect of your method, you would need:

  1. appropriately sized models that can be downstream tested, at least 1-3B parameters trained on ~100-500B tokens
  2. appropriate ablations of each component of your method, for example different numbers of loops trained separately, models that train on varying numbers of loops (as in [5]), different token mixers (GDN2, Linear Attention, Gated Linear Attention, Softmax Attention), etc.
  3. appropriate comparisons not only to baselines but the current state of the art in looped language models, e.g. Huggin and others

[1] LT2: Linear-Time Looped Transformers https://arxiv.org/abs/2605.20670 [2] CHASE: Cache-Hole-Adapted Skip Exit for Looped State-Space Language Models https://arxiv.org/abs/2607.10110 [3] Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping https://arxiv.org/abs/2603.23998 [4] DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products https://arxiv.org/abs/2502.10297 [5] Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach https://arxiv.org/abs/2502.05171

Questions for authors
  1. Can you make the definition of your method more specific. What exactly do you loop?
  2. Please provide a general optimization treatment for other accelerators as well as motivation and background for hardware optimization
  3. Can you explain what you mean by "baseline adjusted" in Table 1? Is this the reason why performances are sub-random or even negative?
  4. Can you provide appropriate ablations as well as comparisons to state of the art baselines?
Reviewer_oBfeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies which components should be repeated in looped language models built with Gated DeltaNet (GDN) mixers. It proposes MixerLoop, which repeats each layer's GDN mixer four times with shared weights and separate recurrent states, followed by one SwiGLU FFN application. The authors compare it with NoLoop and FullLoop at 13M and 100M parameters, and report a 450M comparison with published models. MixerLoop obtains similar aggregate downstream performance to FullLoop while reducing backbone projection compute by approximately 43%. Fixed-checkpoint ablations show that additional mixer iterations reduce validation cross-entropy. The paper also implements a complete quantized batch-1 decoder on a KV260 FPGA. By caching mixer weights and overlapping additional mixer iterations with FFN weight transfers, the 13M MixerLoop achieves approximately the same throughput as NoLoop at the evaluated single-port operating point, around 152 tokens/s.

Critical strengths

None.

Critical weaknesses

1. The novelty and significance of the contribution are insufficiently established.

The architectural change is modest, and the paper does not sufficiently distinguish it from prior work on selective attention looping and linear-time looped transformers. Section 2 acknowledges SGT as prior work that localizes recurrence within a block and cites LT2, but the discussion does not make sufficiently clear what new architectural insight MixerLoop establishes beyond these approaches. The choice to focus on GDN is also insufficiently justified, and no softmax-attention comparison establishes whether the conclusion about recurrence allocation extends beyond this mixer. The connection to FPGA execution is explicitly explained, but I find the broader scientific contribution of the implementation limited: the main demonstrated benefit is a conditional opportunity to overlap computation with weight transfers on a small model and one device. Together, these points leave the significance of the contribution below the level needed for acceptance.

2. The experiments do not adequately establish the breadth or robustness of the conclusions.

The controlled comparisons are limited to 13M and 100M models. The 450M results use published baselines rather than a complete matched comparison; in particular, the 400M NoLoop baseline uses 15B training tokens, whereas MixerLoop uses 100B. The small aggregate CORE advantage over FullLoop at 100M is sensitive to the CommonsenseQA result and does not show a consistent advantage across tasks.

Although the paper reports training loss and validation cross-entropy for truncated MixerLoop checkpoints, it does not report a direct held-out NLL/perplexity comparison across the trained architectures at the main comparison scales. Finally, the hardware result is demonstrated only for the quantized 13M model; task accuracy after quantization and throughput at the larger evaluated model scales are not reported. These limitations make the general claims about recurrence allocation and practical efficiency insufficiently convincing.

Questions for authors
  1. Architectural comparison: How does MixerLoop differ consequentially from the selective attention recurrence in SGT and the GDN recurrence in LT2? Why was GDN selected, and which conclusions do you intend to claim specifically for GDN?
  2. Controlled language-model comparison: Do you have held-out NLL or perplexity results comparing the independently trained NoLoop, MixerLoop, and FullLoop models under matched data and parameter budgets? How do you account for the differences in recurrence placement and initialization when attributing the result to repeating the mixer rather than the FFN?
  3. CORE clarification: Appendix A specifies baseline normalization and CORE V2 corrections. Could you give the numerical task baselines and the raw CommonsenseQA accuracies at 100M, and explain the large MixerLoop–FullLoop difference on that task? This materially affects the interpretation of the small aggregate CORE lead.
  4. Deployment scope: What task-quality measurements support the W8A8 deployment, and what evidence supports extending the observed throughput result to larger models or GPUs? In particular, how would the required mixer residency, persistent states, and FFN prefetch capacity be realized at those scales?
4 comments
AuthorsOpenReview ↗

Response to Reviewer oBfe (1/4)

We sincerely thank the reviewer for the detailed feedback and constructive questions.

W1 & Q1: Novelty and Architectural Motivation

We cite SGT and LT2 in Section 2 precisely because they establish two important starting points for our work. SGT demonstrates that recurrence can be localized within the attention mechanism rather than applied to the entire Transformer block, with its benefits attributed to selectively looping high-entropy attention heads. LT2, on the other hand, investigates full-block looping with linear-time token mixers and demonstrates that GDN is particularly promising for recurrent computation, exhibiting strong performance and training stability. These two observations motivate our central question: if selective attention looping is beneficial in softmax-based Transformers, can looping only the GDN mixer also provide benefits, even though GDN does not have softmax attention entropy?

Unlike SGT, which selectively loops high-entropy attention heads, MixerLoop repeats the entire GDN mixer within each layer. Unlike LT2, which repeats the full block stack, including the FFNs, MixerLoop performs multiple GDN iterations while applying the FFN only once. Our investigation therefore focuses on whether recurrence in the context-dependent mixer is sufficient to preserve most of the benefits of full-block looping, without repeatedly executing the position-wise FFN.

We selected GDN because LT2 had already demonstrated its strong suitability for recurrent computation among linear-time mixers. Our experiments further show that this choice of recurrence boundary preserves performance close to FullLoop while reducing backbone projection computation by approximately 43.3%.

The hardware contribution follows directly from this architectural choice. By keeping the GDN mixer weights on-chip and overlapping additional mixer iterations with FFN weight transfers, we demonstrate that four-step MixerLoop can achieve nearly the same decoding throughput as NoLoop under the evaluated KV260 configuration. Beyond this implementation, we also derive a general analytical condition relating the number of iterations, model dimensions, and memory bandwidth, which characterizes when the additional mixer computation can be hidden behind weight transfers.

We do not claim that our conclusions about recurrence allocation necessarily generalize to softmax attention or every other token mixer. Rather, our contribution is to investigate and demonstrate this trade-off specifically for GDN, building on the complementary findings of SGT and LT2.

AuthorsOpenReview ↗

Response to Reviewer oBfe (2/4)

(continued)

W2 & Q3: CORE Scores and CommonsenseQA

Thank you for raising this important concern.

Regarding the numerical task baselines, we follow the CORE V2 evaluation protocol provided by DataComp-LM (DCLM). The baseline-adjusted score is calculated by subtracting the task-specific baseline from the raw accuracy and normalizing by the remaining margin to 100%. The task-specific baseline values can be found in the DCLM evaluation metadata.

For CommonsenseQA, the raw accuracies of NoLoop, MixerLoop, and FullLoop at 100M are 25.72%, 39.39%, and 22.03%, respectively.

However, your question prompted us to investigate this unusually large difference further. We found that among the 1,221 questions in the CORE version of CommonsenseQA, 492 have C as the correct answer, corresponding to 40.3% of the dataset. This answer imbalance is also documented in the CORE V2 baseline corrections. Further analysis revealed that MixerLoop selects option C for approximately 95% of the questions, while FullLoop selects option B for approximately 84%.

To investigate the effect of answer-position bias, we conducted additional evaluations by redistributing the correct answers evenly across options A, B, C, and D. Under this setting, the accuracies of MixerLoop and FullLoop become 25.18% and 25.39%, respectively, nearly eliminating the original gap. In another experiment, we moved the original option C to position B for each question. On the same set of questions, the accuracies changed to 19.57% for MixerLoop and 38.08% for FullLoop.

These results strongly suggest that the large original CommonsenseQA difference is primarily driven by answer-position bias rather than a meaningful difference in commonsense reasoning ability. We therefore agree with your concern that this result should not be interpreted as an evidence of MixerLoop outperforming FullLoop.

Excluding CommonsenseQA, the average scores across the remaining 21 CORE tasks are approximately 11.85 for MixerLoop and 12.86 for FullLoop. Nevertheless, the results in the other two task groups remain unchanged. At 100M, FullLoop leads in the Language and Reading group (10.19 vs. 8.18), while MixerLoop leads in the Symbolic and Analytical Reasoning group (13.73 vs. 11.70). This is also why we report results separately by task group in the main paper. We have never intended to claim that MixerLoop consistently outperforms FullLoop. Our central claim is that MixerLoop retains much of FullLoop's performance while requiring substantially less backbone projection computation.

We sincerely appreciate the reviewer for raising this question, which led us to identify and better understand this important evaluation artifact.

AuthorsOpenReview ↗

Response to Reviewer oBfe (3/4)

(continued)

W3 & Q2: Controlled Comparisons and Held-Out NLL

Thank you for raising these questions regarding the controlled comparisons and language modeling performance.

We do have held-out NLL results comparing the independently trained NoLoop, MixerLoop, and FullLoop models. As shown in Fig. 3, all three 13M models were trained on ClimbMix for 10B tokens, with final recorded training losses of 3.197, 3.157, and 3.159, respectively. To directly address your question about held-out NLL, we additionally evaluated the final checkpoints of all three architectures on the same validation texts. Following the same model order, the held-out NLL values are 3.1488, 3.1167, and 3.1170 for 13M, and 2.5139, 2.4958, and 2.4422 for 100M. At both scales, MixerLoop achieves lower NLL than NoLoop, with a particularly close match to FullLoop at 13M.

We agree that differences in initialization may affect the final scores. However, this is a variable that cannot be fully controlled, as the three architectures have fundamentally different recurrence configurations, just as FullLoop cannot be perfectly matched with an equivalent NoLoop model in terms of parameter count and initialization. Nevertheless, the held-out NLL results provide additional evidence that repeating the mixer alone can improve language modeling performance over NoLoop, without repeatedly executing the FFN. At 13M, MixerLoop achieves nearly identical NLL to FullLoop, while at 100M, FullLoop achieves a lower NLL. These observations are consistent with our central claim that MixerLoop preserves much of the benefit of full-block looping at a lower computational cost.

AuthorsOpenReview ↗

Response to Reviewer oBfe (4/4)

(continued)

Q4: W8A8 Deployment and Hardware Generalization

For the 13M MixerLoop, we evaluated the floating-point model and its W8A8 CPU reference implementation on the same 5,312 next-token prediction positions from held-out ClimbMix texts. The NLL values are 3.5504 and 3.5513, respectively, while the numbers of correct top-1 predictions are 1,965 and 1,962. These results indicate that W8A8 quantization surprisingly introduces only a negligible difference in short-context next-token prediction under the evaluated conditions.

Regarding hardware generalization, we deliberately chose an FPGA rather than a GPU because of the explicit control it provides over on-chip memory allocation and data movement. Our KV260 implementation keeps the mixer weights on-chip and overlaps the additional mixer iterations with FFN weight transfers. The underlying principle is that additional recurrent computation can be hidden behind memory transfers when its execution time does not exceed the available transfer window. We derive an analytical condition relating the iteration count, model dimensions, and memory bandwidth to characterize when this overlap is possible. While our experimental demonstration is limited to the KV260, this condition provides a basis for evaluating the applicability of the design to other hardware platforms.

On GPUs, achieving the same degree of explicit memory control is more challenging. To our knowledge, the lifetime of shared memory is generally limited to the execution of a thread block, while L2 cache persistence mechanisms provide preferential retention rather than deterministic control over data residency. Consequently, CUDA or Triton implementations cannot guarantee the exact timing of data retrieval in the same way as an explicitly scheduled FPGA design. One possible implementation approach would be to fuse the mixer iterations within each layer into a single kernel, allocating on-chip storage for mixer weights and recurrent states while using the remaining available resources to prefetch FFN weights.

We do not claim that the measured KV260 throughput directly generalizes to GPUs. Our contribution is to demonstrate this computation–memory overlap on a complete FPGA decoder and provide an analytical framework for determining when similar benefits may be achievable on other platforms through software-hardware co-design. Extending the implementation to larger accelerators is a valuable direction for future work.

We sincerely appreciate your detailed questions and constructive feedback.

Reviewer_E9ANOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MixerLoop, which repeats the Gated DeltaNet mixer four times per layer, with separate recurrent states per iteration, and applies the Feed-Forward Network only once. They demonstrate that it cuts projection compute by about 43% relative to looping the full block. They compare their method against NoLoop, FullLoop at 13M and 100M parameters using the CORE Suite. They also train a larger MixerLoop and show it stays close to an existing full-block looped model on zero-shot benchmarks. Moreover, they build an FPGA accelerator that keeps the mixer weights on chip and runs the extra mixer iterations while the FFN weights are being transferred, so four-step MixerLoop decodes at about the same speed as the model without looping.

Critical strengths
  1. The question is well framed and the design is simple. Separating context-dependent state updates in the mixer from position-wise transformation in the FFN is a clear way to ask where recurrence pays off.

  2. The paper was relatively easy to follow, and the findings seemed to have a smooth flow. Overall the lit review was sufficient and the method design was clear.

  3. The authors build a complete decoder. NoLoop and MixerLoop run on the same bitstream, and the overlap condition gives a usable design linking loop count, model geometry and memory bandwidth.

Critical weaknesses
  1. The main comparisons are not fully controlled. The three models use different initialisation schemes, so part of the difference between MixerLoop, FullLoop and NoLoop could come from initialisation rather than from where the loop is placed. Moreover, the gaps between the models are small and reported without any variance across seeds. The larger-scale comparison is also not matched, since the FullLoop model is taken from prior work with a different training setup and the NoLoop baseline is trained on far fewer tokens. Essentially, the claim that MixerLoop keeps the benefits of full-block looping is not yet well supported, especially since it still falls slightly behind FullLoop at that scale.

  2. Removing mixer iterations from a model that was trained with four only shows that the model depends on all four, not that the extra iterations add value over training with fewer. On the hardware side, the matched decoding speed holds only in the most bandwidth-limited setting, and MixerLoop falls behind once more bandwidth is available. In my opinion, a FullLoop model on the same accelerator would also be needed, since that is the comparison that would show the benefit of this particular loop boundary.

Questions for authors

A few questions considering which i can update my score:

  1. How do models trained with fewer mixer iterations compare with the four-iteration model truncated to the same count? Does the last iteration still help when the model is trained for it from the start?

  2. How much of the decoding time on the accelerator is spent on the vocabulary projection, and what throughput would FullLoop reach on the same hardware?

4 comments
AuthorsOpenReview ↗

Response to Reviewer E9AN (1/4)

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the motivation, clarity, and hardware–software co-design of our work.

W1: Controlled Comparisons and Initialization

We agree that differences in initialization may affect the final scores in our current comparisons. However, this is a variable that cannot be fully controlled, as the three models have fundamentally different recurrence configurations, just as FullLoop cannot be perfectly matched with an equivalent NoLoop model in terms of parameter count and initialization.

We did calculate the sample standard deviations across seeds from our original evaluation records. The CORE scores and standard deviations are as follows: NoLoop: 1.39 ± 0.67; MixerLoop: 1.72 ± 0.81; FullLoop: 1.49 ± 1.41. These results support our claim that MixerLoop retains most of the benefits of full-block looping, even though FullLoop achieves a higher peak score, while also supporting your caution regarding the interpretation of small score differences.

We acknowledge that the 100M results are based on a single seed, and cross-seed stability can be further evaluated in future experiments. Similarly, for the larger-scale comparison, we could supplement our experiments with a NoLoop baseline trained under the same budget. We would like to emphasize that, in the reported experiments, MixerLoop achieves an aggregate score close to FullLoop while reducing backbone projection computation.

AuthorsOpenReview ↗

Response to Reviewer E9AN (2/4)

(continued)

W2: Mixer Iterations and Hardware Efficiency

We view Gated DeltaNet (NoLoop) as a single-step solver. Removing mixer iterations from a model trained with four iterations and observing a dramatic performance drop demonstrates the transition from a single-step solver to a multi-step solver. While this experiment does not necessarily guarantee that multiple steps outperform a single step in all settings, our language modeling results indicate that this is the right direction.

On the hardware side, we transparently report that the matched decoding speed holds only in the most bandwidth-limited setting. This is because we have already exhausted the computational resources of the KV260 edge platform, particularly its URAM capacity (64/64), while sufficient memory bandwidth remains available.

In larger-scale deployments, memory bandwidth constraints will only become more severe. The compute-to-memory-bandwidth ratio of the KV260 is approximately 4 MAC/byte, compared with 216 MAC/byte for the U250. As a more extreme example, the H100 has a ratio of approximately 295 MAC/byte for batch-1 decoding (although this is not a typical deployment scenario). Therefore, MixerLoop is applicable to a broader range of scenarios than those illustrated in our figure.

For FullLoop, excluding computations such as the LM head, throughput decreases almost linearly with the number of iterations. For example, at T = 4, throughput is reduced by a factor of four.

AuthorsOpenReview ↗

Response to Reviewer E9AN (3/4)

(continued)

Q1: Contribution of Individual Mixer Iterations

Thank you for this excellent question.

In our experiments, the independently trained NoLoop model substantially outperforms the truncated MixerLoop. For example, for the 13M model, NoLoop achieves an NLL that is 2.380 nats/token lower than that of the truncated MixerLoop. Regarding the fourth iteration, restoring the fourth mixer iteration in three original 13M checkpoints reduces NLL by 0.245, 0.309, and 0.325 nats/token, respectively. For the 100M model, the reduction is 0.179 nats/token. These results demonstrate that the trained four-step models indeed make use of the fourth iteration. However, we believe it is important to clarify what "help" means in this context. As discussed above, recurrence turns MixerLoop into a multi-step solver, which achieves higher average scores on the symbolic and analytical reasoning benchmark group. Therefore, the additional iterations (including the final one) do not merely reduce NLL, but also provide tangible benefits on downstream tasks.

More advanced training approaches do exist, such as assigning a separate loss to each iteration. Models trained in this manner would almost certainly perform better at T = 4 than at T = 1. E.g., Ouro.

AuthorsOpenReview ↗

Response to Reviewer E9AN (4/4)

(continued)

Q2: Vocabulary Projection and FullLoop Throughput

To maintain tokenizer consistency, our 13M model uses the same Llama 2 vocabulary of approximately 32K tokens as the 100M model. The vocabulary projection accounts for approximately 8.192M parameters and 60% of the decoding time on our accelerator.

For FullLoop, the implementation repeats the entire five-layer network four times, followed by a single vocabulary projection. Under our current layer-by-layer weight-loading strategy, which caches only the weights of the current layer, each traversal requires transferring at least 5.253 MB of mixer and FFN matrix weights.

Therefore, the total matrix-weight transfer volume per token is:

4 × 5.25312 + 9.216 = 30.23 MB/token.

Under the single-port bandwidth limit of 2.4 GB/s, these transfers alone require at least 12.60 ms/token, corresponding to a throughput upper bound of 79.4 tokens/s. The actual throughput would be lower.

Thank you again for your thoughtful review and constructive questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.