- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies whether looped (weight-tied recurrent) Transformers can beat standard non-recurrent Transformers under the same pre-training wall-clock budget rather than the same parameter count. The authors introduce Loopie-20B-A2B and Loopie-6B-A0.6B (built on Qwen3-MoE) with three main components:
- Layer-loop recurrence: Instead of repeating the entire layer stack (model-loop), Loopie repeats each individual layer twice in place.
- The Loopie Recipe: Starting from a 48-layer Qwen3-30B-A3B reference, the authors halve the stored layers, loop each layer twice inside a single activation-checkpointing unit, and use the freed memory to double the microbatch size (MBS from 1 to 2). They reinvest this throughput gain into wider layers and slightly more blocks, arriving at Loopie-20B-A2B with approximately matched optimizer-step time.
- Large-scale training: The models are pre-trained on 3.54T tokens, followed by 2T tokens of massive-batch Supervised Pre-training and GSPO/DAPO reinforcement learning.
Critical strengths
-
Systems-friendly recurrence via layer-loop. Unlike whole-model looping (Ouro, Huginn), which routes activations from the last layer back to the first and causes pipeline bubbles, layer-loop keeps recurrence local to each pipeline stage and shares weights across adjacent depths. Figure 2 shows that layer-loop overtakes model-loop after ~1.2T tokens at the 6B MoE scale, offering a clean and practical design for distributed MoE training.
-
Multi-epoch Supervised Pre-training (SPT) at massive batch scale. Running target-masked SFT loss at pre-training batch scale (128M nominal token positions per step) for approximately 10 epochs (2T tokens) yields smooth loss curves without epoch-boundary cliffs and improves reasoning benchmarks without degrading MMLU or ARC-Challenge. This is a useful empirical takeaway for large-scale post-training.
Critical weaknesses
-
Missing systems details and checkpointing controls limit verification and attribution of the central wall-clock-matched pre-training results (Sections 2.3–2.5). Loopie-20B-A2B has a leading-order Transformer-block compute proxy () 42.4% higher than that of the reproduced 48-layer Qwen3-30B-A3B baseline (Section 2.3). The authors attribute the efficiency gain that offsets this extra nominal work to doubling the per-device microbatch size; Figure 3 reports MBS=2 for Loopie and MBS=1 for the baseline. However:
- Missing systems details: The paper does not report the GPU model, number of GPUs/nodes, selected TP/PP degrees, actual optimizer-step times (), or peak memory breakdowns for Figure 3 or the Scaling Ladder in Figure 4. Figure 3 reports EP=8, the microbatch sizes, and throughput in TFLOPS/s, but these do not suffice to independently verify the claimed wall-clock matching from the reported information.
- Checkpointing sensitivity and memory attribution: In Section 2.3 (lines 225–237), Loopie wraps both passes of a layer into a single activation-checkpointing unit (checkpointing every 2 executed layers) and claims the efficiency gain comes "solely from the reduced stored depth, which lowers activation memory." However, the paper does not report the checkpoint-unit granularity used for the baseline or whether grouping adjacent baseline layers was evaluated. Moreover, as Appendix F (Page 34) notes, MBS=2 feasibility depends on total memory including parameters and optimizer states, where Loopie also has substantially fewer stored parameters. Without reporting memory breakdowns and a baseline with matched 2-layer checkpointing granularity, the results do not establish whether the wall-clock advantage is robust to this natural checkpointing alternative. Resolving this concern may require rerunning the central pre-training comparison, and the outcome could change the main conclusion.
-
Contradictory ablation claims and missing statistical and post-training controls weaken the evidence for the proposed mechanism and token-efficiency claims (Sections 2.2–2.6, 4.1).
- Contradictory interpretation of Figure 5: Section 2.6 claims the ablation on Loopie-6B-A0.6B tests whether recurrence ordering is important while preserving "the overall looped computation budget" so the gain "cannot simply be attributed to additional computation." Yet the Figure 5 caption states that it compares vs. at fixed stored size and "is not a FLOP-matched comparison" (and labels a "2.14x speedup" along a token-count axis despite the two runs having different per-token compute). While Figure 2 validly compares layer-loop against model-loop, Figure 5 halves effective depth and the number of Transformer-block applications rather than isolating loop ordering.
- Missing statistical details and controlled post-training baselines: Figures 2–5 report aggregate performance plots (with gains of 0.6–2.2 percentage points in Figure 4 and a small plotted gap at 800B tokens in Figure 3) without specifying the number of independent runs or providing uncertainty estimates and per-benchmark numerical tables. In post-training, Table 2 supports SPT over standard SFT and PT using only qualitative arrows () without a controlled quantitative ablation on the same dataset. Furthermore, the paper does not report a non-recurrent baseline trained through the full pipeline (2.28T initial pre-training + 1.26T annealing + 2T SPT + RL), so using final Table 1 scores to claim superior pre-training token efficiency (3.54T vs. 25T) does not control for post-training stages; Stage 2 also includes "MMLU-style general QA" data (Page 30), while decontamination is only described for the RL stage.
Questions for authors
Clarification requests:
-
What GPU model, GPU count, TP/PP/EP settings, checkpoint-unit granularity, step times, and peak memory breakdowns were used in Figures 3 and 4? If the 48-layer Qwen3-30B-A3B baseline groups two adjacent layers per checkpoint unit, matching Loopie's two executed layer applications per unit, does this configuration permit MBS=2? If evaluated, please report its peak memory and measured optimizer-step time under the same hardware allocation and tokens per optimizer step.
-
Could you clarify the discrepancy between Section 2.6, which claims a fixed computation budget to test recurrence ordering, and the Figure 5 caption, which describes an versus comparison at fixed stored model size that is not FLOP-matched? Which ablation setup was actually run? How is the "2.14x speedup" in Figure 5 calculated, and does it refer to tokens required to reach a given score, FLOPs, or elapsed training time?
-
What quantitative evidence underlies the SPT-versus-SFT comparison in Table 2, and were the starting checkpoint, dataset, and evaluation protocol controlled? For Figures 2–5, please state the number of independent training runs and share any existing uncertainty estimates or per-benchmark results. Was overlap with evaluation sets checked for Stage 2 annealing, including the "MMLU-style general QA" data on page 30, and for SPT? If so, how?
11 comments
Rebuttal (2/5)
1.3 The checkpointing attribution in the current text is too strong
Loopie encloses the two recurrent applications of each stored layer in a single activation-checkpointing unit. This is the implementation used for the reported result. The reduced stored depth lowers the activation footprint and allows a larger per-device microbatch, which reduces gradient-accumulation steps.
The current wording that the efficiency gain comes “solely” from reduced stored depth is too strong. Appendix F already states that actual feasibility is determined by the full memory footprint, including parameters, optimizer states, temporary buffers, and communication workspaces. The correct attribution is:
Under the reported implementation, the measured wall-clock gain is enabled by the combination of reduced stored depth, reduced stored parameter and optimizer-state footprint, activation rematerialization behavior, and the resulting larger microbatch configuration.
We will replace the stronger “solely” wording with this more precise statement.
1.4 A two-layer checkpoint unit in the baseline is a valid sensitivity control, but it is not equivalent to Loopie
The reviewer asks whether the vanilla baseline was evaluated with two adjacent untied layers grouped into one checkpoint unit. The current manuscript does not report this control. This is a legitimate request, and we will add the result or explicitly report its availability and outcome.
However, grouping two vanilla layers into one checkpoint unit does not make the baseline equivalent to Loopie. The baseline still stores two distinct parameter sets and two distinct optimizer-state sets and applies two different functions. Loopie stores one layer and applies the same function twice at adjacent effective depths. Matching the number of executed layers inside a checkpoint unit therefore matches only one rematerialization detail; it does not match parameter-state memory, function sharing, or the recurrence graph.
The control is still important because it tests how much of the measured MBS advantage is sensitive to checkpoint granularity. We will report whether the grouped baseline supports MBS=2, its peak-memory profile, and its measured optimizer-step time. Until that sensitivity is shown, we will narrow the systems claim to the measured Loopie Recipe under the stated checkpointing implementation.
This is a limitation on the scope of the systems claim. It is not evidence that the central experiment was absent.
2. Figure 5 and the recurrence ablation
The reviewer is correct about one specific problem: the prose in Section 2.6 is imprecise.
2.1 What Figure 5 actually compares
The actual Figure 5 experiment is:
- the same stored backbone;
- the same stored parameter count;
- the same optimizer;
- the same data mixture;
- the same training-token budget;
- Loopie with ;
- a non-recurrent variant with .
The caption already states that this is not a FLOP-matched comparison.
Therefore, Figure 5 is a fixed-stored-capacity recurrence ablation. It asks whether adding layer recurrence improves performance when the stored backbone is held fixed. It does not isolate recurrence ordering, and it does not show that the improvement cannot be attributed partly to the additional executed computation.
The sentence in Section 2.6 claiming that the experiment preserves the “overall looped computation budget” and that the gain “cannot simply be attributed to additional computation” is not accurate. We will remove it.
The corrected interpretation is:
Figure 5 shows that, at fixed stored capacity, the layer-loop model performs better than the corresponding non-recurrent model. Because this comparison is not FLOP-matched, it does not separate the benefit of recurrence from the additional executed computation.
The review is right to demand this correction. We will make it.
2.2 Figure 2 is the ordering comparison
Figure 2 is the experiment that compares the ordering of recurrent computation:
- layer-loop: ;
- model-loop: .
Both variants use the same number of recurrent applications and the same nominal number of block executions. The result shows that model-loop is initially ahead, but layer-loop overtakes it after approximately 1.2T tokens and continues improving faster.
Thus, the evidence for the ordering of recurrence comes from Figure 2, whereas Figure 5 provides a different fixed-capacity ablation. The review is correct that these two experiments should not be described as if they measured the same quantity. We will separate their roles explicitly.
Rebuttal (3/5)
2.3 The “2.14× speedup” annotation is under-defined
The “2.14× speedup” annotation in Figure 5 is not sufficiently defined in the current caption. It should not be interpreted as a FLOP speedup, throughput speedup, or elapsed wall-clock speedup, because the comparison is not FLOP-matched.
We will not defend an ambiguous label. We will either remove it or replace it with:
- the exact score threshold;
- the token count required by each model to reach that threshold;
- the explicit ratio used to compute the token-axis efficiency number.
If retained, it will be described as a token-to-score ratio, not as a generic “speedup.”
3. Statistical reporting and numerical results
The review is correct that Figures 2–5 do not report independent-run counts, random seeds, uncertainty intervals, or per-benchmark numerical tables.
The current experiments are large-scale training runs whose primary purpose is to compare training trajectories and scaling behavior under the stated compute protocol. However, the manuscript should not leave the statistical status of these plots implicit. The revision will:
- state the number of independent runs for every configuration;
- state the random seeds where applicable;
- identify configurations for which only one run was performed;
- provide per-benchmark numerical results instead of only aggregate eight-benchmark means;
- provide checkpoint-level values underlying Figures 2–5;
- include uncertainty estimates where repeated runs exist;
- avoid language suggesting statistical significance where no repeated-run estimate is available.
In particular, the 0.6–2.2 percentage-point differences in Figure 4 should be described as observed empirical gaps across the evaluated scaling ladder, not as statistically significant estimates with confidence intervals that were never computed.
At the same time, the absence of error bars does not make the observed trajectories disappear. It limits the precision with which we can characterize run-to-run variance. The appropriate correction is to expose the run count and narrow the claim, rather than to treat an omitted uncertainty table as evidence that the underlying comparisons were not conducted.
4. SPT versus SFT and PT
4.1 What Table 2 does and does not establish
Table 2 is a qualitative regime-definition table. The upward, downward, and approximately-equal arrows summarize the observed behavior of representative configurations. They are not a controlled quantitative ablation on a common checkpoint and dataset.
The paper does establish an objective-level distinction:
- SPT and SFT use the same target-only loss mask;
- SPT uses pre-training-scale optimization;
- SPT uses a global batch size of at least 1,024;
- SPT uses sequence lengths of at least 128K;
- SPT is run for a 2T-token budget;
- conventional SFT uses much smaller batches, shorter sequences, and much smaller token budgets.
Figures 6–8 show the behavior of Loopie during its SPT run: smooth loss over approximately ten epochs, continued reasoning gains, and no sustained degradation on MMLU or ARC-Challenge.
However, these figures do not constitute a controlled causal comparison against SFT and PT from the same starting checkpoint, on the same dataset, with the same evaluation schedule. The reviewer is correct on this point. The arrows in Table 2 should not be treated as if they were a quantitative matched ablation.
We will add a quantitative SPT-versus-SFT-versus-PT comparison with:
- the same starting checkpoint;
- the same supervised dataset where applicable;
- explicit loss masking;
- matched evaluation checkpoints;
- per-benchmark numerical results;
- training-token and optimizer-update accounting.
If the full matched comparison cannot be included, we will narrow the claim to an empirical observation about the evaluated Loopie run rather than presenting the Table 2 arrows as a general causal result.
Rebuttal (1/5)
Thanks for your review! The review identifies several real reporting omissions. However, it also repeatedly turns an omitted table, or one imprecise sentence, into a claim that the underlying experiment was not performed or that the central result is unsound. That inference is not warranted.
The paper makes a specific operational claim:
The paper does not claim that Loopie and the vanilla baseline have identical analytical FLOPs, identical stored parameter counts, or identical token counts. Figure 3 and Figure 4 are the central compute-matched pre-training experiments. Figure 2 compares layer-loop and model-loop recurrence. Figure 5 is a fixed-stored-capacity recurrence ablation. Table 1 is an end-to-end final-model comparison and is not intended to isolate the architecture from the complete post-training pipeline.
We address the review’s points below.
1. Systems accounting and checkpointing
1.1 The 1.424× nominal compute ratio is explicitly acknowledged
The review correctly observes that Loopie-20B-A2B has a higher leading-order Transformer-block compute proxy than the reproduced Qwen3-30B-A3B baseline. This is not a hidden issue in the paper. Section 2.3 explicitly reports
C(baseline)C(Loopie)=48×2048227×2×23042≈1.424The paper therefore does not claim FLOP equality. Calling this ratio a contradiction ignores the explicit distinction in Section 2.3 between nominal block work and the final matching criterion.
The final criterion is the measured end-to-end optimizer-step time:
tstep(D1,L1,R1;2b0,g0/2)≈tstep(D0,L0,R0;b0,g0)This criterion is used because realized training cost includes effects that are absent from the simple LD2R proxy, including MoE routing, expert communication, tensor and pipeline parallelism, kernel efficiency, optimizer overhead, activation rematerialization, memory pressure, gradient accumulation, and GPU utilization.
The comparison protocol fixes the hardware allocation, sequence length, tokens per optimizer step, optimizer, training data, and activation-checkpointing policy. For both models, we perform a parallel-configuration search over TP, PP, EP, and MBS. Figure 3 reports the selected EP and MBS values: Loopie uses EP=8 and MBS=2, while the baseline uses EP=8 and MBS=1. The corresponding measured system throughput is 261.53 TFLOP/s for Loopie and 189.65 TFLOP/s for the baseline.
These TFLOP/s numbers are system-benchmark measurements; they are not being presented as token-throughput equality or as proof of FLOP equality. The final comparison is based on the measured optimizer-step time under a fixed number of tokens per optimizer update.
1.2 The missing systems table is a reporting defect, not evidence that the experiment was not run
The reviewer is correct that the manuscript does not report enough information to independently reconstruct the Figure 3 and Figure 4 measurements. In particular, the current version omits the exact GPU model, GPU and node counts, selected TP/PP values, measured optimizer-step times, and a peak-memory breakdown.
This is a reproducibility omission, and we will fix it. The revision will report, for every central comparison:
The reviewer’s conclusion goes further than the evidence supports. An omitted systems table does not show that the reported comparison was fabricated, nor does it invalidate the measured trajectory in Figure 3. It does mean that the current version does not expose the comparison with sufficient implementation detail. We will correct that.