acceptodds
Under review as a conference paper at ICLR 2027

Loop the Loopies!

Abstract

We present the Loopie series, consisting of two Mixture-of-Experts (MoE) models: a 20B-parameter model with 2B active parameters and a 6B-parameter model with 0.6B active parameters. Looped Transformers have long faced a challenge: given an N times increase in pre-training compute, increasing the parameter count by a factor of N usually outperforms looping a model N times. Loopie addresses this challenge. Extensive ablation studies, including comparisons with a vanilla 30B-A3B model, show that Loopie substantially outperforms vanilla Transformer baselines trained with the same compute budget. With a novel post-training method, Loopie develops strong reasoning abilities and achieves frontier-level reasoning performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped transformer, architecture, pretraining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eejoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether looped (weight-tied recurrent) Transformers can beat standard non-recurrent Transformers under the same pre-training wall-clock budget rather than the same parameter count. The authors introduce Loopie-20B-A2B and Loopie-6B-A0.6B (built on Qwen3-MoE) with three main components:

  1. Layer-loop recurrence: Instead of repeating the entire layer stack (model-loop), Loopie repeats each individual layer twice in place.
  2. The Loopie Recipe: Starting from a 48-layer Qwen3-30B-A3B reference, the authors halve the stored layers, loop each layer twice inside a single activation-checkpointing unit, and use the freed memory to double the microbatch size (MBS from 1 to 2). They reinvest this throughput gain into wider layers and slightly more blocks, arriving at Loopie-20B-A2B with approximately matched optimizer-step time.
  3. Large-scale training: The models are pre-trained on 3.54T tokens, followed by 2T tokens of massive-batch Supervised Pre-training and GSPO/DAPO reinforcement learning.
Critical strengths
  1. Systems-friendly recurrence via layer-loop. Unlike whole-model looping (Ouro, Huginn), which routes activations from the last layer back to the first and causes pipeline bubbles, layer-loop keeps recurrence local to each pipeline stage and shares weights across adjacent depths. Figure 2 shows that layer-loop overtakes model-loop after ~1.2T tokens at the 6B MoE scale, offering a clean and practical design for distributed MoE training.

  2. Multi-epoch Supervised Pre-training (SPT) at massive batch scale. Running target-masked SFT loss at pre-training batch scale (128M nominal token positions per step) for approximately 10 epochs (2T tokens) yields smooth loss curves without epoch-boundary cliffs and improves reasoning benchmarks without degrading MMLU or ARC-Challenge. This is a useful empirical takeaway for large-scale post-training.

Critical weaknesses
  1. Missing systems details and checkpointing controls limit verification and attribution of the central wall-clock-matched pre-training results (Sections 2.3–2.5). Loopie-20B-A2B has a leading-order Transformer-block compute proxy (LRD2LRD^2) 42.4% higher than that of the reproduced 48-layer Qwen3-30B-A3B baseline (Section 2.3). The authors attribute the efficiency gain that offsets this extra nominal work to doubling the per-device microbatch size; Figure 3 reports MBS=2 for Loopie and MBS=1 for the baseline. However:

    • Missing systems details: The paper does not report the GPU model, number of GPUs/nodes, selected TP/PP degrees, actual optimizer-step times (tstept_{\text{step}}), or peak memory breakdowns for Figure 3 or the Scaling Ladder in Figure 4. Figure 3 reports EP=8, the microbatch sizes, and throughput in TFLOPS/s, but these do not suffice to independently verify the claimed wall-clock matching from the reported information.
    • Checkpointing sensitivity and memory attribution: In Section 2.3 (lines 225–237), Loopie wraps both passes of a layer into a single activation-checkpointing unit (checkpointing every 2 executed layers) and claims the efficiency gain comes "solely from the reduced stored depth, which lowers activation memory." However, the paper does not report the checkpoint-unit granularity used for the baseline or whether grouping adjacent baseline layers was evaluated. Moreover, as Appendix F (Page 34) notes, MBS=2 feasibility depends on total memory including parameters and optimizer states, where Loopie also has substantially fewer stored parameters. Without reporting memory breakdowns and a baseline with matched 2-layer checkpointing granularity, the results do not establish whether the wall-clock advantage is robust to this natural checkpointing alternative. Resolving this concern may require rerunning the central pre-training comparison, and the outcome could change the main conclusion.
  2. Contradictory ablation claims and missing statistical and post-training controls weaken the evidence for the proposed mechanism and token-efficiency claims (Sections 2.2–2.6, 4.1).

    • Contradictory interpretation of Figure 5: Section 2.6 claims the ablation on Loopie-6B-A0.6B tests whether recurrence ordering is important while preserving "the overall looped computation budget" so the gain "cannot simply be attributed to additional computation." Yet the Figure 5 caption states that it compares R=2R=2 vs. R=1R=1 at fixed stored size and "is not a FLOP-matched comparison" (and labels a "2.14x speedup" along a token-count axis despite the two runs having different per-token compute). While Figure 2 validly compares layer-loop against model-loop, Figure 5 halves effective depth and the number of Transformer-block applications rather than isolating loop ordering.
    • Missing statistical details and controlled post-training baselines: Figures 2–5 report aggregate performance plots (with gains of 0.6–2.2 percentage points in Figure 4 and a small plotted gap at 800B tokens in Figure 3) without specifying the number of independent runs or providing uncertainty estimates and per-benchmark numerical tables. In post-training, Table 2 supports SPT over standard SFT and PT using only qualitative arrows (↑,↓,≈\uparrow, \downarrow, \approx) without a controlled quantitative ablation on the same dataset. Furthermore, the paper does not report a non-recurrent baseline trained through the full pipeline (2.28T initial pre-training + 1.26T annealing + 2T SPT + RL), so using final Table 1 scores to claim superior pre-training token efficiency (3.54T vs. 25T) does not control for post-training stages; Stage 2 also includes "MMLU-style general QA" data (Page 30), while decontamination is only described for the RL stage.
Questions for authors

Clarification requests:

  1. What GPU model, GPU count, TP/PP/EP settings, checkpoint-unit granularity, step times, and peak memory breakdowns were used in Figures 3 and 4? If the 48-layer Qwen3-30B-A3B baseline groups two adjacent layers per checkpoint unit, matching Loopie's two executed layer applications per unit, does this configuration permit MBS=2? If evaluated, please report its peak memory and measured optimizer-step time under the same hardware allocation and tokens per optimizer step.

  2. Could you clarify the discrepancy between Section 2.6, which claims a fixed computation budget to test recurrence ordering, and the Figure 5 caption, which describes an R=2R=2 versus R=1R=1 comparison at fixed stored model size that is not FLOP-matched? Which ablation setup was actually run? How is the "2.14x speedup" in Figure 5 calculated, and does it refer to tokens required to reach a given score, FLOPs, or elapsed training time?

  3. What quantitative evidence underlies the SPT-versus-SFT comparison in Table 2, and were the starting checkpoint, dataset, and evaluation protocol controlled? For Figures 2–5, please state the number of independent training runs and share any existing uncertainty estimates or per-benchmark results. Was overlap with evaluation sets checked for Stage 2 annealing, including the "MMLU-style general QA" data on page 30, and for SPT? If so, how?

11 comments
AuthorsOpenReview ↗

Rebuttal (1/5)

Thanks for your review! The review identifies several real reporting omissions. However, it also repeatedly turns an omitted table, or one imprecise sentence, into a claim that the underlying experiment was not performed or that the central result is unsound. That inference is not warranted.

The paper makes a specific operational claim:

Under a fixed hardware allocation and matched end-to-end optimizer-step time, can a layer-loop MoE outperform a vanilla MoE?

The paper does not claim that Loopie and the vanilla baseline have identical analytical FLOPs, identical stored parameter counts, or identical token counts. Figure 3 and Figure 4 are the central compute-matched pre-training experiments. Figure 2 compares layer-loop and model-loop recurrence. Figure 5 is a fixed-stored-capacity recurrence ablation. Table 1 is an end-to-end final-model comparison and is not intended to isolate the architecture from the complete post-training pipeline.

We address the review’s points below.


1. Systems accounting and checkpointing

1.1 The 1.424× nominal compute ratio is explicitly acknowledged

The review correctly observes that Loopie-20B-A2B has a higher leading-order Transformer-block compute proxy than the reproduced Qwen3-30B-A3B baseline. This is not a hidden issue in the paper. Section 2.3 explicitly reports

C(Loopie)C(baseline)=27×2×2304248×20482≈1.424\frac{C(\mathrm{Loopie})}{C(\mathrm{baseline})} = \frac{27 \times 2 \times 2304^2}{48 \times 2048^2} \approx 1.424

The paper therefore does not claim FLOP equality. Calling this ratio a contradiction ignores the explicit distinction in Section 2.3 between nominal block work and the final matching criterion.

The final criterion is the measured end-to-end optimizer-step time:

tstep(D1,L1,R1;2b0,g0/2)≈tstep(D0,L0,R0;b0,g0)t_{\mathrm{step}}(D_1,L_1,R_1;2b_0,g_0/2) \approx t_{\mathrm{step}}(D_0,L_0,R_0;b_0,g_0)

This criterion is used because realized training cost includes effects that are absent from the simple LD2RLD^2R proxy, including MoE routing, expert communication, tensor and pipeline parallelism, kernel efficiency, optimizer overhead, activation rematerialization, memory pressure, gradient accumulation, and GPU utilization.

The comparison protocol fixes the hardware allocation, sequence length, tokens per optimizer step, optimizer, training data, and activation-checkpointing policy. For both models, we perform a parallel-configuration search over TP, PP, EP, and MBS. Figure 3 reports the selected EP and MBS values: Loopie uses EP=8 and MBS=2, while the baseline uses EP=8 and MBS=1. The corresponding measured system throughput is 261.53 TFLOP/s for Loopie and 189.65 TFLOP/s for the baseline.

These TFLOP/s numbers are system-benchmark measurements; they are not being presented as token-throughput equality or as proof of FLOP equality. The final comparison is based on the measured optimizer-step time under a fixed number of tokens per optimizer update.

1.2 The missing systems table is a reporting defect, not evidence that the experiment was not run

The reviewer is correct that the manuscript does not report enough information to independently reconstruct the Figure 3 and Figure 4 measurements. In particular, the current version omits the exact GPU model, GPU and node counts, selected TP/PP values, measured optimizer-step times, and a peak-memory breakdown.

This is a reproducibility omission, and we will fix it. The revision will report, for every central comparison:

  • GPU model, GPU count, and node count;
  • TP, PP, EP, and MBS;
  • gradient-accumulation steps;
  • tokens per optimizer step;
  • measured optimizer-step time;
  • tokens per second and total wall-clock time;
  • MFU or the corresponding utilization metric;
  • parameter, optimizer-state, activation, temporary-buffer, and communication memory;
  • checkpoint-unit granularity.

The reviewer’s conclusion goes further than the evidence supports. An omitted systems table does not show that the reported comparison was fabricated, nor does it invalidate the measured trajectory in Figure 3. It does mean that the current version does not expose the comparison with sufficient implementation detail. We will correct that.

AuthorsOpenReview ↗

Rebuttal (2/5)

1.3 The checkpointing attribution in the current text is too strong

Loopie encloses the two recurrent applications of each stored layer in a single activation-checkpointing unit. This is the implementation used for the reported result. The reduced stored depth lowers the activation footprint and allows a larger per-device microbatch, which reduces gradient-accumulation steps.

The current wording that the efficiency gain comes “solely” from reduced stored depth is too strong. Appendix F already states that actual feasibility is determined by the full memory footprint, including parameters, optimizer states, temporary buffers, and communication workspaces. The correct attribution is:

Under the reported implementation, the measured wall-clock gain is enabled by the combination of reduced stored depth, reduced stored parameter and optimizer-state footprint, activation rematerialization behavior, and the resulting larger microbatch configuration.

We will replace the stronger “solely” wording with this more precise statement.

1.4 A two-layer checkpoint unit in the baseline is a valid sensitivity control, but it is not equivalent to Loopie

The reviewer asks whether the vanilla baseline was evaluated with two adjacent untied layers grouped into one checkpoint unit. The current manuscript does not report this control. This is a legitimate request, and we will add the result or explicitly report its availability and outcome.

However, grouping two vanilla layers into one checkpoint unit does not make the baseline equivalent to Loopie. The baseline still stores two distinct parameter sets and two distinct optimizer-state sets and applies two different functions. Loopie stores one layer and applies the same function twice at adjacent effective depths. Matching the number of executed layers inside a checkpoint unit therefore matches only one rematerialization detail; it does not match parameter-state memory, function sharing, or the recurrence graph.

The control is still important because it tests how much of the measured MBS advantage is sensitive to checkpoint granularity. We will report whether the grouped baseline supports MBS=2, its peak-memory profile, and its measured optimizer-step time. Until that sensitivity is shown, we will narrow the systems claim to the measured Loopie Recipe under the stated checkpointing implementation.

This is a limitation on the scope of the systems claim. It is not evidence that the central experiment was absent.


2. Figure 5 and the recurrence ablation

The reviewer is correct about one specific problem: the prose in Section 2.6 is imprecise.

2.1 What Figure 5 actually compares

The actual Figure 5 experiment is:

  • the same stored backbone;
  • the same stored parameter count;
  • the same optimizer;
  • the same data mixture;
  • the same training-token budget;
  • Loopie with R=2R=2;
  • a non-recurrent variant with R=1R=1.

The caption already states that this is not a FLOP-matched comparison.

Therefore, Figure 5 is a fixed-stored-capacity recurrence ablation. It asks whether adding layer recurrence improves performance when the stored backbone is held fixed. It does not isolate recurrence ordering, and it does not show that the improvement cannot be attributed partly to the additional executed computation.

The sentence in Section 2.6 claiming that the experiment preserves the “overall looped computation budget” and that the gain “cannot simply be attributed to additional computation” is not accurate. We will remove it.

The corrected interpretation is:

Figure 5 shows that, at fixed stored capacity, the R=2R=2 layer-loop model performs better than the corresponding R=1R=1 non-recurrent model. Because this comparison is not FLOP-matched, it does not separate the benefit of recurrence from the additional executed computation.

The review is right to demand this correction. We will make it.

2.2 Figure 2 is the ordering comparison

Figure 2 is the experiment that compares the ordering of recurrent computation:

  • layer-loop: L1→L1→L2→L2→⋯L_1 \rightarrow L_1 \rightarrow L_2 \rightarrow L_2 \rightarrow \cdots;
  • model-loop: L1→L2→⋯→LM→L1→L2→⋯L_1 \rightarrow L_2 \rightarrow \cdots \rightarrow L_M \rightarrow L_1 \rightarrow L_2 \rightarrow \cdots.

Both variants use the same number of recurrent applications and the same nominal number of block executions. The result shows that model-loop is initially ahead, but layer-loop overtakes it after approximately 1.2T tokens and continues improving faster.

Thus, the evidence for the ordering of recurrence comes from Figure 2, whereas Figure 5 provides a different fixed-capacity ablation. The review is correct that these two experiments should not be described as if they measured the same quantity. We will separate their roles explicitly.

AuthorsOpenReview ↗

Rebuttal (3/5)

2.3 The “2.14× speedup” annotation is under-defined

The “2.14× speedup” annotation in Figure 5 is not sufficiently defined in the current caption. It should not be interpreted as a FLOP speedup, throughput speedup, or elapsed wall-clock speedup, because the comparison is not FLOP-matched.

We will not defend an ambiguous label. We will either remove it or replace it with:

  • the exact score threshold;
  • the token count required by each model to reach that threshold;
  • the explicit ratio used to compute the token-axis efficiency number.

If retained, it will be described as a token-to-score ratio, not as a generic “speedup.”


3. Statistical reporting and numerical results

The review is correct that Figures 2–5 do not report independent-run counts, random seeds, uncertainty intervals, or per-benchmark numerical tables.

The current experiments are large-scale training runs whose primary purpose is to compare training trajectories and scaling behavior under the stated compute protocol. However, the manuscript should not leave the statistical status of these plots implicit. The revision will:

  • state the number of independent runs for every configuration;
  • state the random seeds where applicable;
  • identify configurations for which only one run was performed;
  • provide per-benchmark numerical results instead of only aggregate eight-benchmark means;
  • provide checkpoint-level values underlying Figures 2–5;
  • include uncertainty estimates where repeated runs exist;
  • avoid language suggesting statistical significance where no repeated-run estimate is available.

In particular, the 0.6–2.2 percentage-point differences in Figure 4 should be described as observed empirical gaps across the evaluated scaling ladder, not as statistically significant estimates with confidence intervals that were never computed.

At the same time, the absence of error bars does not make the observed trajectories disappear. It limits the precision with which we can characterize run-to-run variance. The appropriate correction is to expose the run count and narrow the claim, rather than to treat an omitted uncertainty table as evidence that the underlying comparisons were not conducted.


4. SPT versus SFT and PT

4.1 What Table 2 does and does not establish

Table 2 is a qualitative regime-definition table. The upward, downward, and approximately-equal arrows summarize the observed behavior of representative configurations. They are not a controlled quantitative ablation on a common checkpoint and dataset.

The paper does establish an objective-level distinction:

  • SPT and SFT use the same target-only loss mask;
  • SPT uses pre-training-scale optimization;
  • SPT uses a global batch size of at least 1,024;
  • SPT uses sequence lengths of at least 128K;
  • SPT is run for a 2T-token budget;
  • conventional SFT uses much smaller batches, shorter sequences, and much smaller token budgets.

Figures 6–8 show the behavior of Loopie during its SPT run: smooth loss over approximately ten epochs, continued reasoning gains, and no sustained degradation on MMLU or ARC-Challenge.

However, these figures do not constitute a controlled causal comparison against SFT and PT from the same starting checkpoint, on the same dataset, with the same evaluation schedule. The reviewer is correct on this point. The arrows in Table 2 should not be treated as if they were a quantitative matched ablation.

We will add a quantitative SPT-versus-SFT-versus-PT comparison with:

  • the same starting checkpoint;
  • the same supervised dataset where applicable;
  • explicit loss masking;
  • matched evaluation checkpoints;
  • per-benchmark numerical results;
  • training-token and optimizer-update accounting.

If the full matched comparison cannot be included, we will narrow the claim to an empirical observation about the evaluated Loopie run rather than presenting the Table 2 arrows as a general causal result.

Reviewer_C8vKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Loopie, a looped Mixture-of-Experts (MoE) Transformer architecture that introduces layer-level looping within the model, rather than repeatedly executing the full model stack. The idea is relatively straightforward and easy to follow. The paper also introduces a supervised pre-training strategy and presents experiments across different configurations, reporting that Loopie can reduce pre-training compute while outperforming vanilla Transformers under comparable compute budgets.

Critical strengths
  1. Simple and intuitive design. The core idea of introducing layer-level looping into an MoE Transformer is relatively straightforward and easy to understand.

  2. Promising compute-performance trade-off. The experiments suggest that Loopie can achieve better performance than vanilla Transformers under the authors' reported compute-matching settings, potentially offering a more efficient approach to pre-training. The work trains multiple looped MoE model checkpoints across several scales, includes comprehensive hyper‑parameter documentation, appendix details for architecture, data composition and training recipes. SPT training regime is an interesting practical contribution for large‑scale post‑training. This body of empirical work represents significant compute investment, which is a plus for the submission.

Critical weaknesses
  1. Insufficient motivation and mechanistic analysis. The explanations of why layer-level looping works are relatively superficial. The three analyses intended to explain the underlying mechanisms do not provide sufficiently convincing mechanistic insights or experimental evidence to establish why the proposed architecture leads to the reported improvements.

  2. Insufficient attribution of architectural gains versus system-level efficiency. The experiments do not adequately disentangle the benefits of the proposed architecture from engineering gains at the hardware and systems levels. Although the paper claims that the layer-looping architecture achieves better performance under the same pre-training wall-clock time, the comparisons rely on measured optimizer-step runtime rather than theoretical floating-point operations (FLOPs) as the basis for compute alignment. Consequently, the reported gains may partly arise from differences in batching efficiency, hardware utilization, or throughput, rather than from intrinsic architectural advantages. More controlled comparisons are needed to establish that the proposed looping mechanism itself accounts for the claimed improvements.

  3. Limited competitiveness and insufficiently rigorous compute comparisons. The results in Table 1 are not particularly compelling, and the use of pre-training token counts as a proxy for computational cost may not provide a sufficiently rigorous basis for comparing different architectures. A more informative evaluation would compare Loopie and relevant strong baselines under clearly matched token and computational budgets. For example, it would be useful to evaluate Loopie with the same 36T-token budget and compare its performance against Qwen3-30B-A3B-Thinking, provided that the comparison is appropriate. In addition, the scalability of Loopie remains unclear: it is not established whether the reported advantages would persist at substantially larger model sizes, such as 100B or 1T parameters.

Questions for authors
  1. Would the authors provide stronger mechanistic analyses or targeted ablation studies to explain why layer-level looping improves performance? What evidence distinguishes the benefits of the looping architecture itself from those arising from other training or implementation choices?

  2. Would the authors provide more detailed and controlled comparisons that separately report FLOPs, training tokens, optimizer steps, wall-clock time, and throughput? How much of the reported improvement can be attributed to the architecture itself, rather than to batching efficiency or hardware utilization?

  3. How does Loopie perform under an identical 36T-token budget, and how does it compare with relevant strong baselines, including Qwen3-30B-A3B-Thinking where appropriate? Do the authors have evidence that the claimed advantages persist as model scale increases substantially, for example to 100B or 1T parameters?

11 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thanks for your review. We must first clarify that several assertions in this review are inconsistent with experiments already reported in the paper. We address these issues point by point.

1. Insufficient Motivation and Mechanistic Analysis of Layer-Level Looping

The reviewer argues that the paper does not provide sufficiently convincing mechanistic evidence to explain why layer-level looping works. However, this assessment overlooks the most direct and important evidence presented in the paper: the large-scale pre-training experiment in Figure 2.

We trained two 6B-A0.6B models and conducted full, long-horizon pre-training over approximately 2 trillion tokens, comparing the training dynamics of layer-loop and model-loop. This is neither a small-scale toy experiment nor a one-off downstream evaluation conducted only at the final checkpoint. The results show that model-loop initially has a slight advantage, but layer-loop overtakes it after approximately 1.2T tokens and improves at a faster rate as training continues. This indicates that the advantage of layer-loop is not merely the result of an accidental initialization or evaluation fluctuation, but rather stems from differences in the long-horizon optimization and representation-learning dynamics induced by the two recurrence schedules.

Furthermore, we will supplement the revised manuscript with residual-stream and gradient-norm diagnostics. Our existing diagnostics show that layer-loop produces significantly larger residual-stream updates, indicating that the second loop is not simply repeating an almost ineffective computation, but instead performs meaningful additional representation refinement on top of the preceding update. Meanwhile, the gradient norms across the two loop passes are more stable, suggesting a more stable optimization process under this local recurrence pattern. Together, these observations support the following explanation: layer-loop allows each Transformer layer to iteratively refine its representation locally before passing it to the next layer, rather than requiring the same parameters to serve widely separated and functionally different positions in effective depth.

Section 2.6 and Figure 5 also provide a direct architectural ablation. This experiment keeps the stored backbone, parameter count, optimizer, data mixture, and training-token budget fixed, while removing the layer-loop recurrence schedule, comparing the R=2 layer-loop model against its corresponding non-recurrent variant. Layer-loop substantially improves average downstream performance. Therefore, the improvement cannot simply be explained by "having more parameters" or "training on more tokens."

Importantly, Figure 5 is specifically designed to isolate the contribution of the recurrence schedule itself, and the paper explicitly states that this is not a FLOP-matched comparison. It addresses a different question from the long-horizon training curves in Figure 2 and the end-to-end compute-matched experiments in Sections 2.3–2.5. Characterizing these experiments as a lack of "mechanistic analysis or targeted ablations" directly contradicts what is already presented in the paper.

Response to Question 1: The benefits of layer-level looping are supported by three complementary lines of evidence:

  1. Figure 2: Two 6B-A0.6B models trained over approximately 2T tokens, demonstrating that layer-loop overtakes model-loop in the later stages of training and continues to improve more rapidly.
  2. Figure 5: A controlled experiment with the stored backbone, parameter count, optimizer, training data, and token budget held fixed, directly demonstrating the importance of the recurrence schedule itself.
  3. Additional residual-stream and gradient-norm visualizations: We will provide these diagnostics to further demonstrate that the second loop produces stronger and more stable internal representation updates.

Therefore, Loopie's improvements cannot be attributed to SPT or other post-training choices. The architectural comparisons discussed above are conducted during pre-training, with both models using the same training data and basic optimization settings.

AuthorsOpenReview ↗

Rebuttal (2/3)

2. Insufficient Separation of Architectural Gains from System-Level Efficiency

The review implicitly assumes a premise that does not hold: matching theoretical FLOPs is equivalent to matching actual pre-training cost. For large-scale MoE training, this assumption is clearly incorrect. Theoretical FLOP counts do not capture the effects of communication, routing, optimizer overhead, kernel launches, activation checkpointing, or microbatch scheduling on hardware utilization. Two models with identical nominal FLOPs may incur substantially different actual training costs due to differences in MFU. Indeed, some architectures appear significantly cheaper in terms of theoretical FLOPs, yet substantially reduce MFU, ultimately resulting in much higher realized training costs.

For this reason, our comparisons align the actual end-to-end training cost incurred during pre-training, rather than analytical FLOPs that are disconnected from execution efficiency. We have never claimed that Loopie and the baseline have identical theoretical FLOP counts. On the contrary, Section 2.3 explicitly states that Loopie performs more nominal leading-order computation, with a compute-proxy ratio of approximately 1.42 relative to the Qwen3-like reference. This additional nominal work is offset by Loopie's higher measured execution efficiency, which is precisely the practical pre-training efficiency investigated in our paper.

To ensure that this comparison does not favor Loopie, we performed extensive parallelism-configuration searches for both Loopie and the vanilla baseline on actual hardware. Specifically, we jointly searched over TP, PP, EP, and microbatch size, selecting configurations that maximize the achievable measured throughput and realized efficiency for each architecture independently. We fixed the hardware allocation, sequence length, number of tokens per optimizer step, activation-checkpointing policy, optimizer, and training data. When the two architectures required different per-device microbatch sizes, we adjusted gradient accumulation accordingly to ensure that the number of tokens processed per optimizer step remained identical.

Figure 3 reports precisely this end-to-end comparison. Loopie and the Qwen3-like vanilla baseline are compared under matched per-step wall-clock time. Loopie achieves a highest measured throughput of 261.53 TFLOPS/s, compared with 189.65 TFLOPS/s for the baseline. This difference was not obtained by assigning an inferior parallelism configuration to the baseline; both architectures underwent their own extensive parallelism-configuration searches.

More importantly, the "more rigorously controlled experiments" requested by the reviewer have already been conducted in the paper. Section 2.6 and Figure 5 specifically examine whether the layer-loop recurrence pattern itself contributes to the improvement. Meanwhile, Section 2.3, Figure 3, and Section 2.4 evaluate whether Loopie outperforms a strong vanilla baseline under matched real end-to-end training costs. These two categories of experiments are complementary:

  • Figure 5 answers: Does the recurrence schedule itself provide a benefit when stored capacity and the training recipe are held fixed?
  • Figure 3 answers: Does Loopie achieve an overall pre-training advantage on real hardware under matched optimizer-step wall-clock time?

Therefore, simply attributing Loopie's improvements to differences in batching efficiency or hardware utilization is not a valid conclusion. More precisely, the layer-loop architecture itself changes both the representation-update mechanism and the stored depth, enabling more effective local iteration as well as higher achievable hardware efficiency. The latter is not architecture-independent noise or an unrelated implementation artifact; it is a direct consequence of how this architecture operates in practical large-scale training.

We will additionally report the FLOPs proxy, training tokens, optimizer steps, wall-clock time, throughput, and MFU side by side in the revised manuscript to make the compute-alignment procedure easier to verify. However, it is not meaningful to forcibly decompose two physically coupled effects into arbitrary percentages of "architectural contribution" and "systems contribution." Figure 5 already identifies the contribution of the recurrence schedule itself, while Figure 3 identifies its overall advantage under a real training system.

AuthorsOpenReview ↗

Rebuttal (3/3)

3. Limited Competitiveness, Insufficiently Rigorous Compute Comparisons, and Scalability

The reviewer has misunderstood the purpose of Table 1. Table 1 is not intended to replace rigorous compute-matched architectural comparisons, nor is it intended to establish FLOP-level causal relationships between different architectures. Table 1 serves two specific purposes, to demonstrate that the final trained Loopie models achieve:

  1. frontier-level training token efficiency.
  2. frontier-level post-training performance.

For example, Loopie-20B-A2B uses only 3.54T pre-training tokens, whereas Nemotron 3 Nano and Nemotron Cascade 2 each use 25T tokens, and Qwen3-30B-A3B-Thinking uses 36T tokens (using exactly the same training data sources). Table 1 demonstrates that, despite using substantially fewer pre-training tokens than these models, Loopie still achieves competitive performance across multiple knowledge, general and reasoning benchmarks. It is a summary of training-token efficiency and final model capabilities, not an architectural attribution experiment.

The "rigorous comparisons between different architectures" requested by the reviewer have already been extensively conducted in Sections 2.1–2.6. Specifically:

  • Section 2.3 explains the Loopie Recipe and the measured end-to-end cost-matching methodology.
  • Figure 3 and Section 2.4 compare Loopie against a Qwen3-like vanilla MoE baseline under matched per-step wall-clock time.
  • Section 2.5 constructs a four-level scaling ladder, conducting compute-matched comparisons against vanilla baselines with 0.15B, 0.25B, 0.5B, and 1B active parameters.
  • Figure 4 demonstrates that Loopie outperforms its corresponding vanilla baseline at all four scales.
  • Section 2.6 and Figure 5 further isolate the contribution of layer-loop recurrence scheduling itself.

Therefore, concluding that the paper lacks fair architectural comparisons merely because Table 1 is not designed as a rigorous architectural ablation conflates the purpose of a results table with the experimental evidence supporting the architectural claims.

Regarding the reviewer's suggestion to train Loopie on 36T tokens and compare it with Qwen3-30B-A3B-Thinking, we must distinguish two fundamentally different research questions. Such a comparison is unnecessary, as we have already pretrained Qwen3-30B-A3B on exactly the same data and demonstrated Loopie's significant advantage. The central question of this paper is: Does the layer-loop architecture outperform the standard Transformer architecture under the same actual pretraining cost?

Moreover, Table 1 compares final post-training checkpoints. Different models use different supervised-training, reasoning-training, and post-training recipes. Therefore, treating 36T raw pre-training tokens as a direct measure of the complete computational cost cannot constitute a rigorous architectural comparison. The actual architectural comparisons have already been conducted in Figures 3, 4, and 5.

Regarding scalability to 100B or 1T parameters, we have never claimed to have trained or experimentally validated 100B- or 1T-parameter models. Our conclusions about scalability are based on the experimental scales that we have actually evaluated. The four-level scaling ladder covers vanilla baselines with 0.15B–1B active parameters, and Loopie consistently outperforms its corresponding compute-matched vanilla model at every scale. In addition, our main experiments include Loopie-20B-A2B, a model with 20B total parameters and 2B active parameters.

"Experiments at 100B or 1T parameters have not been conducted" does not imply "the existing results provide no evidence of scalability," much less "Loopie's advantages will disappear at larger scales." It simply limits our empirical conclusions to the model scales actually tested. Treating an untested, substantially larger scale as grounds for dismissing the currently validated results goes beyond the experimental claims of this work.

Response to Question 3: Table 1 is intended to demonstrate the final models' frontier-level training-token efficiency and final performance, whereas Sections 2.1–2.6 provide the rigorous architectural comparisons and compute-matched experiments. Qwen3's 36T-token results are already included as a final-model reference, but training Loopie for another 36T tokens is not a necessary condition for answering the compute-matched architectural question investigated in this paper. Regarding scalability, we have already demonstrated consistent improvements across a four-level scaling ladder covering vanilla baselines with 0.15B–1B active parameters, and we have additionally demonstrated large-scale Loopie results with a 20B-total-parameter model. Direct validation at 100B or 1T parameters is a direction for future scaling, not a prerequisite for the validity of the experiments already presented.

Reviewer_nu9AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper study a problem: can looped Transformers match or exceed vanilla Transformers under the same pre-training compute budget? The ablation studies show that 20B-A2B loop Transformers (2 layer-loop) outperforms a vanilla 30B-A3B model. With a post-training method, the model Loopie-20B-A3B-Thinking achieves comparable performance with two Nemotron models but performs worse than Qwen3-30B-A3B-Thinking.

Critical strengths
  1. The questions "can looped Transformers match or exceed vanilla Transformers under the same pre-training compute budget?" is important and interesting. The paper match models by realized pre-training cost rather than by exact theoretical FLOPS. And the answer is yes.
  2. Scaling ladder in figure 4 show the improvement under different model size (0.15B, 0.25B, 0.5B, 1B).
Critical weaknesses
  1. The paper claim its novel post-training method but ablation analysis about it. And the novelty needs to be clarified.
  2. Loopie-20B-A2B match Qwen3-30B-A3B by realized pre-training cost on author's GPU setup, but the match settings may vary across different GPU. It need more GPU setup to claim the realized pre-training cost.
  3. Loopie 20B-A2B Thinking is still worse than Qwen3 30B-A3B Thinking.
Questions for authors

See the weaknesses; Other questions:

  1. Have you tried 15B-A1.5B?
  2. Why choose layer-loop 2? How about layer-loop 3, 4 ...?
6 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

Thank you for the careful review and for recognizing the importance of the central question and the evidence from the scaling ladder. We agree that several aspects of the manuscript, especially the scope of SPT and the reporting of hardware-dependent compute matching, should be explained more clearly. We address these points below.

1. Scope and status of the post-training method

The primary focus of this paper is the pre-training behavior of layer-loop Transformers under a matched realized compute budget. In particular, the central question is whether a looped Transformer can match or outperform a vanilla Transformer when the two models are trained with comparable end-to-end pre-training cost.

The post-training experiments have a narrower purpose. They demonstrate that a Loopie base model remains compatible with large-scale post-training and can acquire strong reasoning capabilities. They are not intended to make the post-training method carry the main architectural claim, nor are they intended to provide a complete standalone study of SPT.

The methodological component introduced in the post-training section is Supervised Pre-Training (SPT), which combines:

  • the target-only supervision pattern of SFT;
  • pre-training-scale global batch sizes;
  • long sequence lengths;
  • a multi-trillion-token training budget; and
  • optimization over multiple epochs.

Thus, SPT uses the same type of loss masking as SFT, but operates at a substantially different optimization scale. In the representative Loopie configuration, SPT uses a global batch size of at least 1,024, sequence lengths of at least 128K, approximately 128M nominal token positions per update, and a 2T-token budget over approximately ten epochs.

The current manuscript already provides initial evidence for this regime in Appendix C. Table 2 defines the differences among SPT, conventional SFT, and PT. Figure 6 shows a smooth loss trajectory throughout the multi-epoch SPT run without clear epoch-boundary cliffs. Figures 7 and 8 show continued improvement on reasoning metrics without sustained degradation on MMLU or ARC-Challenge.

That said, we agree that the current presentation is too concise to establish SPT as a fully characterized standalone method. In the camera-ready version, we will add more systematic and controlled SPT ablations and make the distinction between the main pre-training contribution and the supporting post-training study explicit. A more comprehensive treatment of SPT, including broader ablations and analysis, will be developed separately. In this paper, the main role of SPT is to show that layer-loop recurrence does not prevent effective large-scale post-training or strong reasoning performance.

2. Hardware dependence of realized-cost matching

We agree that realized optimizer-step time can depend on the GPU platform and parallel configuration. Our claim is therefore conditional on the stated hardware allocation and training setup; we do not intend to claim GPU-independent equality of training cost.

For the main comparison in Figure 3, we fix the hardware allocation, sequence length, tokens per optimizer step, optimizer, training data, and activation-checkpointing policy. We then search over TP, PP, EP, and microbatch-size configurations and select configurations based on measured end-to-end optimizer-step time. The selected configurations reported in the current manuscript use EP=8 and MBS=2 for Loopie, and EP=8 and MBS=1 for the Qwen3-like vanilla baseline. The corresponding peak benchmark throughputs are 261.53 TFLOP/s and 189.65 TFLOP/s, respectively.

The nominal leading-order Transformer-block compute is not identical: for the main comparison, the proxy ratio is approximately 1.424 in favor of Loopie’s executed block work. This is precisely why the paper uses measured optimizer-step time as the final matching criterion rather than claiming analytical FLOP equality. The realized comparison includes memory pressure, activation rematerialization, gradient accumulation, kernel efficiency, communication, routing, and GPU utilization.

AuthorsOpenReview ↗

Rebuttal (2/3)

The reviewer is correct that the current manuscript does not expose enough systems information to independently reconstruct the measurements. In the revision, we will add a consolidated systems table containing:

  • GPU model, GPU count, and node count;
  • TP, PP, EP, and MBS;
  • gradient-accumulation steps;
  • tokens per optimizer step;
  • measured optimizer-step time;
  • tokens per second and total wall-clock time;
  • nominal FLOPs and realized throughput; and
  • peak-memory breakdowns for parameters, optimizer states, activations, temporary buffers, and communication workspaces.

We will also make the scope of the claim more explicit. The intended statement is that, under the evaluated hardware allocation and the reported implementation, the Loopie Recipe produces a matched realized pre-training cost and a better downstream trajectory than the corresponding vanilla baseline. The optimal parallel configuration may vary across GPU platforms, as already noted in the caption of Figure 3. We will report the platform details and clarify this conditional scope in the camera-ready version.

3. Loopie-20B-A2B-Thinking and Qwen3-30B-A3B-Thinking

We agree that Loopie-20B-A2B-Thinking is not uniformly better than Qwen3-30B-A3B-Thinking. The two models use different pre-training data sources and post-training pipelines, so their final reasoning scores should not be interpreted as an architecture-only comparison.

The results in Table 1 are mixed, as expected. Loopie scores higher than Qwen3 on AIME 2024 (92.10 vs. 90.10), ties Qwen3 on AIME 2025 (83.75 vs. 83.75), and is lower on AMC (94.21 vs. 96.73). We therefore do not claim that Loopie comprehensively outperforms Qwen3 on final reasoning benchmarks.

For the token-efficiency comparison, the closest reference models in Table 1 are Nemotron 3 Nano and Nemotron Cascade 2. Loopie and these two models use the same underlying pre-training and post-training data sources, while differing in data-selection strategy, training recipe, and architecture. This does not make the comparison a perfectly controlled architectural ablation, but it makes it more informative for evaluating performance under a common data-source family than the comparison with Qwen3.

Loopie uses 3.54T pre-training tokens, whereas both Nemotron models use 25T tokens. Despite this difference, Loopie matches or exceeds at least one of the two Nemotron models on every reported knowledge and general-capability benchmark, and outperforms each of them on five of the seven such benchmarks. For example:

  • MMLU: Loopie 81.28, Nemotron 3 Nano 80.52, Nemotron Cascade 2 81.22;
  • ARC-Challenge: Loopie 93.52, Nemotron 3 Nano 91.96, Nemotron Cascade 2 93.86;
  • IFEval: Loopie 84.72, Nemotron 3 Nano 77.05, Nemotron Cascade 2 79.21; and
  • BBH: Loopie 82.28, Nemotron 3 Nano 68.76, Nemotron Cascade 2 75.86.

These results support the more limited claim that Loopie can achieve strong general and reasoning capabilities with substantially fewer pre-training tokens under the evaluated data-source setting. They should not be read as evidence that Loopie must outperform every model trained with a different corpus and post-training recipe.

We will revise the wording around Table 1 to make this distinction clearer. Figures 3 and 4 provide the main controlled evidence for the pre-training architecture and the Loopie Recipe under matched realized cost. Table 1 provides an end-to-end capability comparison after post-training.

Question 1: Have you tried 15B-A1.5B?

We have not trained a separate 15B-A1.5B model. The current study prioritizes a scaling ladder with four compute-matched baseline scales, from 0.15B to 1B, together with the larger Loopie-20B-A2B experiment. These points were selected to test whether the observed advantage persists across scale rather than to evaluate every intermediate parameter count.

A 15B-A1.5B model would be an additional interpolation point, but it would not test a distinct mechanism beyond the scaling trend already evaluated in Figure 4 and the larger flagship comparison. We will state explicitly in the revision that this configuration was not evaluated, rather than leaving its status implicit.

AuthorsOpenReview ↗

Rebuttal (3/3)

Question 2: Why use two loop steps? What about three or four?

The relevant analysis is already included in Appendix J.1 and Figure 10, although we agree that it is too easy to miss in the current organization. The appendix compares N-layer-loop models with models using N times as many stored layers for N=2, 3, and 4, using the same general architecture and a 250B-token training budget.

The analysis reflects the compute constraint that motivates this paper. Increasing the number of loop steps increases per-token training work approximately linearly, while the marginal performance benefit of additional recurrence decreases in the evaluated large-scale pre-training setting. Under this fixed-budget trade-off, two loop steps provided the best observed balance among recurrent refinement, stored capacity, throughput, and total pre-training cost.

Two loop steps are also the smallest nontrivial recurrent setting: each stored layer is applied once and then performs one additional local refinement before passing its representation to the next layer. This gives recurrence a meaningful computational role while keeping the compute multiplier small enough for a stringent comparison with vanilla MoE models.

We emphasize that this choice is not intended to imply that three or four loop steps are ineffective in every setting. Larger loop counts may be useful when inference-time computation is abundant, when adaptive computation is available, or when the goal is latent iterative reasoning rather than compute-efficient pre-training. The scope of our claim is narrower: for the frontier-scale pre-training regime studied here, two loop steps offered the most favorable trade-off in our experiments.

We will move the discussion currently in Appendix J.1 into the main text, where it will become Section 2.7 in the camera-ready version. We will also make the limitations of the exploratory comparison clearer, including the fact that the N=2 stored-layer reference is not a perfectly FLOP-matched control.

We appreciate the reviewer’s comments and the opportunity to clarify these points. The revisions will make the scope of SPT, the hardware dependence of realized-cost matching, the distinction between Qwen3 and the data-source-aligned Nemotron comparison, and the rationale for two loop steps more explicit. The central conclusion remains supported by Figures 3 and 4: under the evaluated realized pre-training budget, the Loopie Recipe can outperform the corresponding vanilla Transformer baselines across the reported scales.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.