- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes Loopback Transformer, a parameter-efficient architecture that improves residual information reuse in looped Transformers. The paper introduces a unified write–transport–read formulation to analyze how different residual pathways store and aggregate intermediate computations. Based on this, Loopback stores individual block updates, preserves a separate input path, and combines geometric decay with attention-based allocation over historical updates. Experiments on ClimbMix show that Loopback achieves competitive or improved validation loss compared with matched-depth dense Transformers while using substantially fewer Transformer parameters at several model scales.
Critical strengths
- The proposed write–transport–read framework provides a clear and useful perspective for understanding residual information reuse across looped Transformer architectures.
- The method is relatively simple and well motivated. Separating the original input from accumulated updates and explicitly controlling the total update weight are sensible architectural choices.
- The evaluation covers multiple model scales and compares against a broad range of residual architectures, including Hyper-Connections, Hyperloop, and Attention Residuals. The paper also reports useful training-throughput and memory measurements.
Critical weaknesses
-
A central contribution of Loopback is the combination of geometric decay and learned attention over historical updates. However, Table 3 shows that geometric decay alone accounts for the overwhelming majority of the improvement over the plain loop. At 191M, decay-only achieves a validation loss of 2.6220, compared with 2.6187 for full Loopback. At 374M, the corresponding losses are 2.3999 and 2.3971. Thus, learned allocation provides only 0.0033 and 0.0028 nats of additional improvement. These results suggest that the primary effectiveness of Loopback may come from controlling residual accumulation rather than learning content-dependent update allocation. The paper would benefit from comparisons with simpler alternatives, including learned residual scaling, per-visit gating, and static learned depth-wise coefficients. It is currently unclear whether the additional complexity of historical attention is justified by its incremental benefit.
-
The analysis emphasizes the distinction between delta storage and full-output storage, and the experiments in Table 10 suggest that update-based readers outperform state-based readers. However, as acknowledged in Appendix C, these alternatives also differ in head count, source count, and output blending. Consequently, the observed improvements cannot be attributed solely to the representation being stored. A controlled comparison that changes only the stored representation while keeping the reader architecture, normalization, and allocation mechanism identical would strengthen the central methodological claim.
-
Several improvements are relatively small. For example, at 191M, Loopback improves validation loss over Dense by only 0.0047 nats and over mHC by 0.0063 nats. The paper does not report variability across independent training seeds. The initialization experiments in Table 3(b) measure sensitivity to different initialization schemes rather than training-run variability under the same configuration. Similarly, the bootstrap analysis of downstream benchmarks does not capture training randomness. Moreover, the results at 1.4B are mixed: Loopback achieves worse validation loss and lower seven-task average accuracy than Dense, despite a small improvement in thirteen-task average accuracy. This raises questions about whether the claimed parameter-efficiency advantages generalize consistently across scales and training configurations.
-
Although Loopback substantially reduces stored Transformer parameters and introduces little additional arithmetic FLOPs, its historical update reads incur significant runtime and memory overhead. For example, at 374M, training throughput decreases from 128k to 98k tokens/s/GPU, while prefill throughput decreases from 492k to 352k tokens/s. Peak training memory also increases from 26 GiB to 32 GiB. The paper acknowledges these overheads, but a more explicit quality–parameter–latency trade-off would help establish the practical usefulness of the proposed architecture. In particular, it would be valuable to identify the deployment settings in which reduced parameter storage outweighs the additional activation memory and latency.
Questions for authors
- Can the authors compare Loopback against simpler learned residual-scaling or per-visit gating baselines, and explain whether content-dependent allocation provides benefits beyond static learned coefficients?
- Can the authors provide a controlled delta-vs-state ablation that keeps the reader structure and other architectural choices identical?
- How stable are the small validation-loss improvements across independent training seeds? Can the authors report mean and standard deviation for at least the smaller-scale configurations?
- What explains the weaker validation-loss results at 1.4B? How sensitive are the results to loop topology, decay strength, and training duration?
- Under what practical memory or latency constraints would Loopback be preferable to Dense or simpler looped architectures?
6 comments
Response to Reviewer AFGN(Part2)
W4 / Q5: Practical quality–memory–latency trade-offs.
Thank you for the question. We optimized the kernel that reads past block updates, raising Loopback’s training throughput at 374M from 98.2k to 114.9k tokens/s/GPU (1.17×) on eight H100s. Peak reserved training memory was 29.08 GiB/GPU for Loopback and 27.82 for Standard. At 3B, Loopback reached 98% of Standard’s training throughput (24.2k vs. 24.7k tokens/s/GPU) with 0.7% more peak reserved memory (44.6 vs. 44.31 GiB/GPU).
The benefit of sharing weights is clearest at inference. At matched training tokens and executed depth, the trained 3B Loopback reached 1.9849 validation loss versus 2.0005 for Standard with 75% of its Transformer parameters. Against Plain Loop at 374M, Loopback also achieved lower validation loss in all three training seeds at matched depth and training tokens (W3/Q3). We measured peak inference memory on one H100 using 4,096-token chunks for prefill.
| Peak inference memory (GiB) ↓ | 1.4B Standard / Loopback | 3B Standard / Loopback |
|---|---|---|
| Decoding, batch 1 | 2.94 / 2.29 | 6.20 / 5.07 |
| Prefill, 8K tokens | 4.71 / 4.22 | 8.29 / 7.35 |
| Prefill, 16K tokens | 6.37 / 5.87 | 10.56 / 9.61 |
| Prefill, 32K tokens | 9.54 / 9.02 | 15.11 / 14.15 |
To measure when lower weight storage allows more requests to run together, we benchmarked a 100B reference architecture with random weights in vLLM 0.31 on four H100s. At the same 85% GPU memory budget, Loopback could cache 303,888 tokens versus 210,256 for Standard (+44.5%), with the same KV storage per token. Each request generated 8K tokens after prefill. The table reports aggregate decoding throughput at the maximum concurrency that fits in memory for each model. Paired entries are Standard / Loopback.
| Prompt | Max concurrent requests ↑ | Aggregate decode throughput (tokens/s) ↑ | Throughput ratio |
|---|---|---|---|
| 32K | 5 / 7 | 181.5 / 226.1 | 1.25× |
| 64K | 2 / 4 | 78.1 / 127.2 | 1.63× |
| 128K | 1 / 2 | 38.1 / 64.1 | 1.68× |
Loopback is most useful for long-context serving when GPU memory limits how many requests can run at once. Our trained 3B model achieved lower validation loss than Standard while using less peak memory during both decoding and prefill. In the 100B reference benchmark, the same four H100s could serve four concurrent requests with 64K prompts, up from two with Standard, increasing aggregate decoding throughput by 63%.
Response to Reviewer AFGN(Part3)
Q4: Larger-scale performance and sensitivity to topology, decay, and training duration.
At 1.4B, we selected [3→8×2→6] through empirical topology comparisons. We evaluated this configuration at the full training budget. We reran Standard on the same eight H100s. At 3B, Loopback uses [6→6×2→6] against a 24-block, width-3,072 Standard; Loopback achieves lower validation loss at both scales while retaining only 68% and 75% of Standard’s Transformer parameters. Each pair matches training tokens and executed depth;
| Zero-shot task accuracy (%) / metric | 1.4B Standard | 1.4B Loopback | 3B Standard | 3B Loopback |
|---|---|---|---|---|
| PIQA | 73.88 | 74.48 | 77.04 | 75.90 |
| HellaSwag | 55.02 | 56.43 | 62.84 | 63.72 |
| WinoGrande | 56.51 | 56.99 | 57.22 | 58.48 |
| OpenBookQA | 37.80 | 36.60 | 40.40 | 41.80 |
| CommonsenseQA | 23.34 | 18.67 | 19.90 | 21.87 |
| BoolQ | 62.51 | 63.24 | 62.23 | 64.95 |
| RACE | 34.64 | 33.49 | 35.12 | 36.17 |
| ARC-Easy | 66.16 | 67.00 | 69.19 | 71.13 |
| ARC-Challenge | 36.35 | 37.03 | 40.53 | 40.96 |
| MMLU | 24.70 | 26.81 | 26.26 | 28.86 |
| ANLI-R1 | 31.30 | 34.80 | 31.00 | 32.90 |
| RTE | 57.40 | 55.23 | 61.01 | 56.68 |
| LAMBADA | 42.64 | 43.72 | 48.50 | 49.45 |
| 13-task mean ↑ | 46.33 | 46.50 | 48.56 | 49.45 |
| Validation loss ↓ | 2.1283 | 2.1073 | 2.0005 | 1.9849 |
| Transformer parameters retained | 100% | 68% | 100% | 75% |
Longer training at 374M also preserves Loopback’s advantage:
| Training budget | Seed | Standard loss ↓ | mHC loss ↓ | Loopback loss ↓ |
|---|---|---|---|---|
| 50 TPP | 42 | 2.3171 | 2.3135 | 2.3051 |
| 50 TPP | 1337 | 2.3227 | 2.3155 | 2.3077 |
| 100 TPP | 42 | 2.2620 | 2.2633 | 2.2561 |
Loopback outperforms both Standard and mHC in these comparisons while retaining 58.3% of Standard’s Transformer parameters. The advantage is reproduced across two seeds at 50 TPP and persists through 100 TPP at this scale.
We also tested whether allocating additional training resources to the baselines closes the gap. The smaller Standard baselines nearly match Loopback’s parameter count and receive slightly more total training FLOPs. The time comparison uses the same eight H100 GPUs and batch configuration, with Loopback’s final training kernel.
| Comparison | Reference scale | Baseline | Transformer params (baseline / Loopback) | Baseline TPP | Baseline loss ↓ | Loopback loss ↓ |
|---|---|---|---|---|---|---|
| Baseline uses 1.01× Loopback’s training FLOPs | 191M | Standard, 8 blocks | 56.64M / 56.65M | 45 | 2.6428 | 2.6187 |
| Baseline uses 1.06× Loopback’s training FLOPs | 374M | Standard, 14 blocks | 179.86M / 179.88M | 35 | 2.4193 | 2.3971 |
| Matched training time | 374M | Standard, 24 blocks | 308.33M / 179.88M | 22.23 | 2.4071 | 2.3971 |
| Matched training time | 374M | Plain Loop | 179.86M / 179.88M | 22.51 | 2.4275 | 2.3971 |
Loopback uses 20.00 TPP, with a common reference parameter count within each comparison. At equal training time, the baseline losses come from the training curves between the 20- and 25-TPP checkpoints. Loopback achieves lower validation loss than both Standard and Plain Loop.
Response to Reviewer AFGN(Part1)
Thank you for recognizing the simplicity of our method and the effectiveness of our unified residual connection view. We address each of your points below.
Thank you for your question. In a plain loop, every pass adds new block updates while carrying earlier ones forward at full weight. Each loop boundary multiplies an update’s prior weight by λ (0 < λ < 1), so its weight falls to λ after one boundary, λ² after two, and λ³ after three. This geometric decay favors recent passes and keeps the total update weight bounded as the same stack repeats. Attention can still give an older update more weight when its content is useful for the current token. The original input keeps weight one throughout. We tested the two controls you suggested, learned block scaling and per-visit gating, at 20 TPP.
Loopback achieved the lowest ClimbMix validation loss at both scales, 2.6187 at 191M and 2.3971 at 374M. Decay alone outperformed both controls. With the same decay schedule and total update weight, Loopback’s content-dependent allocation yielded lower validation loss than decay-only in all six paired runs (three seeds per scale). The mean reduction was 0.0050 nats at 191M and 0.0049 nats at 374M.
Thank you for your question. In Table 10, the looped Transformer and the memory update rule for each family stay fixed. GDN, Mamba-2, and linear attention each use their standard recurrence at loop boundaries. What changes within each pair is the residual reader. The state reader combines the original loop input with past loop outputs to form a candidate hidden state. The delta reader combines the changes made by past passes and adds the result to the original input, so the input has its own path through the loop.
We ran two ablations at 191M (20 TPP, seed 42). First, to test sensitivity to head count, we kept the state reader’s five sources and total width of 768 fixed and compared eight 96-dimensional heads with one 768-dimensional head. Second, to test what each slot should store, we fixed the one-head reader, its four history slots, and the separate input path. Both runs used the same normalization, query construction, gates, allocation rule, and output mixing. Each slot held either the cumulative state relative to the original loop input (State) or the change made by that pass (Delta).
ClimbMix validation loss
Across all three readers, switching from cumulative states to per-pass deltas reduced validation loss more than switching from eight heads to one. Changing the head count moved loss by at most 0.0014 nats for Mamba-2 and linear attention. With the reader fixed, storing deltas lowered loss by 0.0108 nats for GDN, 0.0043 for Mamba-2, and 0.0079 for linear attention.
We trained each model with three seeds, keeping its configuration fixed within each scale. The tables report ClimbMix validation loss after 20 TPP for each seed and the mean ± sample standard deviation across seeds. Bold marks the lowest loss in each column.
191M, 20 layers
374M, 24 layers
At 191M, Loopback achieves comparable validation loss to Standard with 40% of its Transformer parameters. Its improvement over mHC is supported by a paired difference of −0.0103 [−0.0196, −0.0010] nats. At 374M, Loopback improves over Standard in every seed with 58.3% of its Transformer parameters, with a paired difference of −0.0211 [−0.0360, −0.0061].