acceptodds
Under review as a conference paper at ICLR 2027

Loopback Transformer: Towards a Unified View of Residual Looped Transformers

11 views

Abstract

Looped Transformers reduce parameter count by repeatedly applying shared blocks, but parameter sharing alone does not determine how the results of repeated computation are reused. Building on the depth-memory view of residual connections, we analyze how the written representation and weighting rule constrain the contribution of each update. Storing a full block output also writes back its input, changing how earlier information is carried forward. Reading individual updates with a separate input path exposes independent controls over input strength, total update weight, and allocation. Guided by this analysis, we develop the Loopback Transformer, which stores individual block updates and keeps the shared stack’s input on a separate path. Geometric decay controls total update weight and supplies a recency prior, while attention learns how to allocate that weight. On ClimbMix, Loopback uses only about 40% and 58% of the dense controls’ Transformer parameters at the 191M and 374M scales, respectively, while achieving lower validation loss and better downstream performance at the studied training budgets. Comparisons use the same training-token budget and computational depth within each scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped transformers, recurrent depth, residual connections

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AFGNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes Loopback Transformer, a parameter-efficient architecture that improves residual information reuse in looped Transformers. The paper introduces a unified write–transport–read formulation to analyze how different residual pathways store and aggregate intermediate computations. Based on this, Loopback stores individual block updates, preserves a separate input path, and combines geometric decay with attention-based allocation over historical updates. Experiments on ClimbMix show that Loopback achieves competitive or improved validation loss compared with matched-depth dense Transformers while using substantially fewer Transformer parameters at several model scales.

Critical strengths
  1. The proposed write–transport–read framework provides a clear and useful perspective for understanding residual information reuse across looped Transformer architectures.
  2. The method is relatively simple and well motivated. Separating the original input from accumulated updates and explicitly controlling the total update weight are sensible architectural choices.
  3. The evaluation covers multiple model scales and compares against a broad range of residual architectures, including Hyper-Connections, Hyperloop, and Attention Residuals. The paper also reports useful training-throughput and memory measurements.
Critical weaknesses
  1. A central contribution of Loopback is the combination of geometric decay and learned attention over historical updates. However, Table 3 shows that geometric decay alone accounts for the overwhelming majority of the improvement over the plain loop. At 191M, decay-only achieves a validation loss of 2.6220, compared with 2.6187 for full Loopback. At 374M, the corresponding losses are 2.3999 and 2.3971. Thus, learned allocation provides only 0.0033 and 0.0028 nats of additional improvement. These results suggest that the primary effectiveness of Loopback may come from controlling residual accumulation rather than learning content-dependent update allocation. The paper would benefit from comparisons with simpler alternatives, including learned residual scaling, per-visit gating, and static learned depth-wise coefficients. It is currently unclear whether the additional complexity of historical attention is justified by its incremental benefit.

  2. The analysis emphasizes the distinction between delta storage and full-output storage, and the experiments in Table 10 suggest that update-based readers outperform state-based readers. However, as acknowledged in Appendix C, these alternatives also differ in head count, source count, and output blending. Consequently, the observed improvements cannot be attributed solely to the representation being stored. A controlled comparison that changes only the stored representation while keeping the reader architecture, normalization, and allocation mechanism identical would strengthen the central methodological claim.

  3. Several improvements are relatively small. For example, at 191M, Loopback improves validation loss over Dense by only 0.0047 nats and over mHC by 0.0063 nats. The paper does not report variability across independent training seeds. The initialization experiments in Table 3(b) measure sensitivity to different initialization schemes rather than training-run variability under the same configuration. Similarly, the bootstrap analysis of downstream benchmarks does not capture training randomness. Moreover, the results at 1.4B are mixed: Loopback achieves worse validation loss and lower seven-task average accuracy than Dense, despite a small improvement in thirteen-task average accuracy. This raises questions about whether the claimed parameter-efficiency advantages generalize consistently across scales and training configurations.

  4. Although Loopback substantially reduces stored Transformer parameters and introduces little additional arithmetic FLOPs, its historical update reads incur significant runtime and memory overhead. For example, at 374M, training throughput decreases from 128k to 98k tokens/s/GPU, while prefill throughput decreases from 492k to 352k tokens/s. Peak training memory also increases from 26 GiB to 32 GiB. The paper acknowledges these overheads, but a more explicit quality–parameter–latency trade-off would help establish the practical usefulness of the proposed architecture. In particular, it would be valuable to identify the deployment settings in which reduced parameter storage outweighs the additional activation memory and latency.

Questions for authors
  1. Can the authors compare Loopback against simpler learned residual-scaling or per-visit gating baselines, and explain whether content-dependent allocation provides benefits beyond static learned coefficients?
  2. Can the authors provide a controlled delta-vs-state ablation that keeps the reader structure and other architectural choices identical?
  3. How stable are the small validation-loss improvements across independent training seeds? Can the authors report mean and standard deviation for at least the smaller-scale configurations?
  4. What explains the weaker validation-loss results at 1.4B? How sensitive are the results to loop topology, decay strength, and training duration?
  5. Under what practical memory or latency constraints would Loopback be preferable to Dense or simpler looped architectures?
6 comments
AuthorsOpenReview ↗

Response to Reviewer AFGN(Part1)

Thank you for recognizing the simplicity of our method and the effectiveness of our unified residual connection view. We address each of your points below.

W1 / Q1: Learned allocation and simpler alternatives.

Thank you for your question. In a plain loop, every pass adds new block updates while carrying earlier ones forward at full weight. Each loop boundary multiplies an update’s prior weight by λ (0 < λ < 1), so its weight falls to λ after one boundary, λ² after two, and λ³ after three. This geometric decay favors recent passes and keeps the total update weight bounded as the same stack repeats. Attention can still give an older update more weight when its content is useful for the current token. The original input keeps weight one throughout. We tested the two controls you suggested, learned block scaling and per-visit gating, at 20 TPP.

Model191M loss ↓374M loss ↓
Plain Loop2.68022.4415
+ learned block-update scaling2.67852.4393
+ learned per-visit gating2.65852.4314
Decay-only2.62202.3999
Loopback2.61872.3971

Loopback achieved the lowest ClimbMix validation loss at both scales, 2.6187 at 191M and 2.3971 at 374M. Decay alone outperformed both controls. With the same decay schedule and total update weight, Loopback’s content-dependent allocation yielded lower validation loss than decay-only in all six paired runs (three seeds per scale). The mean reduction was 0.0050 nats at 191M and 0.0049 nats at 374M.

W2 / Q2: Stored representations and the reader interface.

Thank you for your question. In Table 10, the looped Transformer and the memory update rule for each family stay fixed. GDN, Mamba-2, and linear attention each use their standard recurrence at loop boundaries. What changes within each pair is the residual reader. The state reader combines the original loop input with past loop outputs to form a candidate hidden state. The delta reader combines the changes made by past passes and adds the result to the original input, so the input has its own path through the loop.

We ran two ablations at 191M (20 TPP, seed 42). First, to test sensitivity to head count, we kept the state reader’s five sources and total width of 768 fixed and compared eight 96-dimensional heads with one 768-dimensional head. Second, to test what each slot should store, we fixed the one-head reader, its four history slots, and the separate input path. Both runs used the same normalization, query construction, gates, allocation rule, and output mixing. Each slot held either the cumulative state relative to the original loop input (State) or the change made by that pass (Delta).

ClimbMix validation loss

Reader8 heads → 1 headState → Delta
GDN2.6685 → 2.65952.6602 → 2.6494
Mamba-22.6759 → 2.67452.6587 → 2.6544
Linear attention2.6846 → 2.68512.6756 → 2.6677

Across all three readers, switching from cumulative states to per-pass deltas reduced validation loss more than switching from eight heads to one. Changing the head count moved loss by at most 0.0014 nats for Mamba-2 and linear attention. With the reader fixed, storing deltas lowered loss by 0.0108 nats for GDN, 0.0043 for Mamba-2, and 0.0079 for linear attention.

W3 / Q3: Reliability across independent training seeds.

We trained each model with three seeds, keeping its configuration fixed within each scale. The tables report ClimbMix validation loss after 20 TPP for each seed and the mean ± sample standard deviation across seeds. Bold marks the lowest loss in each column.

191M, 20 layers

ModelSeed 42Seed 1337Seed 2027Mean ± SD
Standard2.62342.61442.62332.6204 ± 0.0052
Plain Loop2.68022.68152.68852.6834 ± 0.0045
mHC2.62502.62922.62792.6274 ± 0.0022
Loopback2.61872.61552.61692.6170 ± 0.0016

374M, 24 layers

ModelSeed 42Seed 1337Seed 2027Mean ± SD
Standard2.41982.42572.41752.4210 ± 0.0042
Plain Loop2.44152.44092.43722.4399 ± 0.0023
mHC2.40622.40792.40432.4061 ± 0.0018
Loopback2.39712.39962.40312.3999 ± 0.0030

At 191M, Loopback achieves comparable validation loss to Standard with 40% of its Transformer parameters. Its improvement over mHC is supported by a paired difference of −0.0103 [−0.0196, −0.0010] nats. At 374M, Loopback improves over Standard in every seed with 58.3% of its Transformer parameters, with a paired difference of −0.0211 [−0.0360, −0.0061].

AuthorsOpenReview ↗

Response to Reviewer AFGN(Part2)

W4 / Q5: Practical quality–memory–latency trade-offs.

Thank you for the question. We optimized the kernel that reads past block updates, raising Loopback’s training throughput at 374M from 98.2k to 114.9k tokens/s/GPU (1.17×) on eight H100s. Peak reserved training memory was 29.08 GiB/GPU for Loopback and 27.82 for Standard. At 3B, Loopback reached 98% of Standard’s training throughput (24.2k vs. 24.7k tokens/s/GPU) with 0.7% more peak reserved memory (44.6 vs. 44.31 GiB/GPU).

The benefit of sharing weights is clearest at inference. At matched training tokens and executed depth, the trained 3B Loopback reached 1.9849 validation loss versus 2.0005 for Standard with 75% of its Transformer parameters. Against Plain Loop at 374M, Loopback also achieved lower validation loss in all three training seeds at matched depth and training tokens (W3/Q3). We measured peak inference memory on one H100 using 4,096-token chunks for prefill.

Peak inference memory (GiB) ↓1.4B Standard / Loopback3B Standard / Loopback
Decoding, batch 12.94 / 2.296.20 / 5.07
Prefill, 8K tokens4.71 / 4.228.29 / 7.35
Prefill, 16K tokens6.37 / 5.8710.56 / 9.61
Prefill, 32K tokens9.54 / 9.0215.11 / 14.15

To measure when lower weight storage allows more requests to run together, we benchmarked a 100B reference architecture with random weights in vLLM 0.31 on four H100s. At the same 85% GPU memory budget, Loopback could cache 303,888 tokens versus 210,256 for Standard (+44.5%), with the same KV storage per token. Each request generated 8K tokens after prefill. The table reports aggregate decoding throughput at the maximum concurrency that fits in memory for each model. Paired entries are Standard / Loopback.

PromptMax concurrent requests ↑Aggregate decode throughput (tokens/s) ↑Throughput ratio
32K5 / 7181.5 / 226.11.25×
64K2 / 478.1 / 127.21.63×
128K1 / 238.1 / 64.11.68×

Loopback is most useful for long-context serving when GPU memory limits how many requests can run at once. Our trained 3B model achieved lower validation loss than Standard while using less peak memory during both decoding and prefill. In the 100B reference benchmark, the same four H100s could serve four concurrent requests with 64K prompts, up from two with Standard, increasing aggregate decoding throughput by 63%.

AuthorsOpenReview ↗

Response to Reviewer AFGN(Part3)

Q4: Larger-scale performance and sensitivity to topology, decay, and training duration.

At 1.4B, we selected [3→8×2→6] through empirical topology comparisons. We evaluated this configuration at the full training budget. We reran Standard on the same eight H100s. At 3B, Loopback uses [6→6×2→6] against a 24-block, width-3,072 Standard; Loopback achieves lower validation loss at both scales while retaining only 68% and 75% of Standard’s Transformer parameters. Each pair matches training tokens and executed depth;

Zero-shot task accuracy (%) / metric1.4B Standard1.4B Loopback3B Standard3B Loopback
PIQA73.8874.4877.0475.90
HellaSwag55.0256.4362.8463.72
WinoGrande56.5156.9957.2258.48
OpenBookQA37.8036.6040.4041.80
CommonsenseQA23.3418.6719.9021.87
BoolQ62.5163.2462.2364.95
RACE34.6433.4935.1236.17
ARC-Easy66.1667.0069.1971.13
ARC-Challenge36.3537.0340.5340.96
MMLU24.7026.8126.2628.86
ANLI-R131.3034.8031.0032.90
RTE57.4055.2361.0156.68
LAMBADA42.6443.7248.5049.45
13-task mean ↑46.3346.5048.5649.45
Validation loss ↓2.12832.10732.00051.9849
Transformer parameters retained100%68%100%75%

Longer training at 374M also preserves Loopback’s advantage:

Training budgetSeedStandard loss ↓mHC loss ↓Loopback loss ↓
50 TPP422.31712.31352.3051
50 TPP13372.32272.31552.3077
100 TPP422.26202.26332.2561

Loopback outperforms both Standard and mHC in these comparisons while retaining 58.3% of Standard’s Transformer parameters. The advantage is reproduced across two seeds at 50 TPP and persists through 100 TPP at this scale.

We also tested whether allocating additional training resources to the baselines closes the gap. The smaller Standard baselines nearly match Loopback’s parameter count and receive slightly more total training FLOPs. The time comparison uses the same eight H100 GPUs and batch configuration, with Loopback’s final training kernel.

ComparisonReference scaleBaselineTransformer params (baseline / Loopback)Baseline TPPBaseline loss ↓Loopback loss ↓
Baseline uses 1.01× Loopback’s training FLOPs191MStandard, 8 blocks56.64M / 56.65M452.64282.6187
Baseline uses 1.06× Loopback’s training FLOPs374MStandard, 14 blocks179.86M / 179.88M352.41932.3971
Matched training time374MStandard, 24 blocks308.33M / 179.88M22.232.40712.3971
Matched training time374MPlain Loop179.86M / 179.88M22.512.42752.3971

Loopback uses 20.00 TPP, with a common reference parameter count within each comparison. At equal training time, the baseline losses come from the training curves between the 20- and 25-TPP checkpoints. Loopback achieves lower validation loss than both Standard and Plain Loop.

Reviewer_iyEqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how a looped Transformer should store and reuse the outputs of repeated block applications. To do so, the authors propose LoopBack transformer which uses a Huginn layout. It works by appending each block's net update to a per-token history bank and multiplies all stored coefficients by a fixed lambda at each visit, and then redistributes the total with a softmax from a learned token-shared query and matches against RMS-normalized updates. Empirical experiments show LoopBack has lower validation loss compare to dense (with the same effective depth) at 191M, 374M and 778M, while worst for larger 1.4B model.

Critical strengths

The paper treats how the shared middle's accumulated outputs are stored and read as a separate design axis. This is a useful framing because it lets individual components be isolated and tested. The empirical study covers a wide set of baselines and includes informative ablations, especially Tables 3 and 9, which show that most of the gain comes from decay-controlled total update weight rather than from learned allocation or recency alone. The parameter-matched shallow Dense controls (Appendices E.1 and E.3) also show that repeated computation matters beyond parameter count. The authors are also transparent about the regression at the 1.4B scale.

Critical weaknesses

There are several weaknesses I hope that the authors can address.

First, I am not fully convinced on the usefulness of Eq.13. Table 3a shows adding a learned queries provide very small gain, around 0.0033/0.0028 lower validation losses for 191M and 374M respectively. In Appendix F.2 reruns of 191M, I see that for the same configuration, zero-query decay models scores 2.6245 instead of 2.6220, and Dense scores 2.6188 instead of 2.6234. Therefore it is reasonable to assume run-to-run variation can reach 0.0046 nats, which is as large as the allocation gain. As it is, I believe the contribution looks like anchor damped iteration rather than learned iteration.

Second, Table 9 shows that bounding total update weights accounts for most of the gain. A simple explanation is that residual stream of a plain loops grows with KR additions and drowns out hθh_\theta. Perhaps there are simpler way to fix this stream growth such as:

  1. a plain loop with input injection
  2. a plain loop with learned scalar gate per visit
  3. a plain loop with RMSnorm at visit boundary

Without these simple baselines, I do not know whether the improvement comes from the proposed memory design or just generic stabilization.

Third, I find the proposed solution claim of "near-Dense arithmetic cost" a bit misleading. Table 11 shows step time jump 33% / 31% / 19%, similarly peak training memory up 11% / 23% / 16%, and prefill throughput 0.68/ 0.72/ 0.81 for 191M / 374M / 778M model respectively when compare to dense training. At equal GPU-hours dense could see roughly 20-30% more tokens. Therefore the practical saving is mostly in weight storage, and I would like to see a comparison at matched wall-clock time.

Finally, I do not think this technique scales cleanly. Results for 1.4B models need a deeper exploration. The validation loss difference goes from −0.0187 at 778M to +0.0110 at 1.4B, where the learning rates are equal.At 1.4B the training difference is −0.0032, so training and validation disagree by about 0.014 nats, which needs explanation. I would like to see more than one run at this or larger scales.

Questions for authors

Please see weakness

7 comments
AuthorsOpenReview ↗

Response to Reviewer iyEq(Part1)

Thank you for recognizing the effectiveness of our unified framework and our broad results. We address each of your points below.

W1: Reliability of the learned-allocation gain.

Geometric decay favors recent loop iterations and gives equal prior weight to updates from the same iteration. Two such updates can carry different information about a token, and an older update can still be useful. In Eq. 13, the learned query scores each update by its content, so the next block can give it more weight while preserving both the total update weight and the original input’s coefficient of one. To isolate this effect, we initialize the queries to zero in both models. Loopback learns them during training, while decay-only keeps them at zero and follows the geometric weights throughout.

ScaleModelSeed 42Seed 1337Seed 2027
191MDecay-only2.62202.62222.6220
191MLoopback2.61872.61552.6169
374MDecay-only2.39992.40572.4089
374MLoopback2.39712.39962.4031

Learning how to weight past block updates reduced ClimbMix validation loss by an average of 0.0050 nats at 191M (paired 95% t-interval [0.0008, 0.0093]) and 0.0049 nats at 374M ([0.0004, 0.0094]). Loopback outperformed decay-only in all three seeds at both scales, showing a consistent gain from learned allocation.

W2: Comparison with simpler stabilization baselines.

We tested all three suggested controls at 191M and 374M: additive input injection at each loop boundary, a learned scalar gate on each loop's new update, and RMSNorm at loop boundaries. The table reports ClimbMix validation loss after 20 TPP.

Model191M loss ↓374M loss ↓
Plain Loop2.68022.4415
Plain Loop + input injection2.64972.4315
Plain Loop + learned per-loop gate2.65852.4314
Plain Loop + loop-boundary RMSNorm7.28277.3175
Decay-only2.62202.3999
Loopback2.61872.3971

With loop-boundary RMSNorm, token representations at 191M collapsed toward a single direction. At 374M, training loss plateaued near 7.3, close to the unigram baseline of 7.44.

The reviewer’s input-injection suggestion has a direct connection to our forgetting rule. Let yy be the output of one pass through the shared blocks. At the loop boundary, the decay-only rule gives h0+λ(y−h0)=λy+(1−λ)h0h_0+\lambda(y-h_0)=\lambda y+(1-\lambda)h_0. Applying this rule at successive boundaries gives an update that crosses bb boundaries a weight of λb\lambda^b, while the original input retains weight one. The additive baseline reinjects the input while carrying earlier updates forward at full weight. Full Loopback also learns how to distribute the weight set by decay across individual block updates.

Input injection and learned gating improve Plain Loop at both scales, but neither matches decay-only; Loopback achieves the lowest loss. These tested stabilizers therefore do not reproduce the gains of our residual design. The additional benefit of learned allocation is tested across seeds in W1.

W3: Practical cost and performance at matched training time.

We agree that “near-Dense arithmetic cost” could suggest comparable training time. At matched width, executed depth, and training tokens, Loopback and Standard perform the same forward and backward computations inside their Transformer blocks. Loopback also stores each block update and reads past updates to compute an attention-weighted sum. This adds 0.04–0.08% to forward FLOPs. Moving these updates and their gradients through GPU memory adds time to each training step. We will clarify in the revision that this comparison refers specifically to forward FLOPs. Thank you for pointing this out.

To reduce this overhead, we optimized the kernel that reads and combines past updates. At 374M on eight H100s, Loopback’s training throughput rose from 98.2k to 114.9k tokens/s/GPU (+17.0%). In a separate prefill test on one H100 with eight 4,096-token sequences, an earlier version of the optimized kernel raised throughput from 353k to 409k tokens/s (+15.9%).

To compare validation loss at equal GPU-hours, we gave each baseline the time Loopback takes to train for 20 TPP. We estimated each baseline’s loss at that budget from its seed-42 training curve.

ModelTraining TPP at equal GPU-hoursValidation loss ↓
Standard22.232.4071
Plain Loop22.512.4275
Loopback20.002.3971

In the same time, Standard can process 11.2% more training tokens. Loopback still reaches a lower validation loss with 58.3% of Standard’s Transformer parameters.

AuthorsOpenReview ↗

Response to Reviewer iyEq(Part2)

W4: Performance at larger scales.

To test whether the validation loss reversal persisted at larger scales, we revisited the 1.4B experiment and extended the comparison to 3B. In the original 1.4B run, Loopback fit the training data slightly better than Standard but did slightly worse on held-out text. For the 1.4B follow-up, we compared loop topologies in short training runs at the same 25-layer depth. We paired our Loopback residual with the better-performing topology for full training and reran Standard on the same hardware.

With the revised 1.4B configuration, Loopback had lower training and validation loss than Standard by 0.0212 and 0.0210 nats, respectively. At 3B, its validation loss was 1.9849, compared with 2.0005 for Standard. At matched executed depth and training tokens (20 TPP, seed 42), Loopback scored higher on 9 of 13 zero-shot tasks at 1.4B and 11 of 13 at 3B, using 68% and 75% of Standard’s Transformer parameters, respectively.

Task / metric1.4B Standard1.4B Loopback3B Standard3B Loopback
PIQA73.8874.4877.0475.90
HellaSwag55.0256.4362.8463.72
WinoGrande56.5156.9957.2258.48
OpenBookQA37.8036.6040.4041.80
CommonsenseQA23.3418.6719.9021.87
BoolQ62.5163.2462.2364.95
RACE34.6433.4935.1236.17
ARC-Easy66.1667.0069.1971.13
ARC-Challenge36.3537.0340.5340.96
MMLU24.7026.8126.2628.86
ANLI-R131.3034.8031.0032.90
RTE57.4055.2361.0156.68
LAMBADA42.6443.7248.5049.45
Seven-task mean ↑52.6253.1856.5357.35
13-task mean ↑46.3346.5048.5649.45
WikiText perplexity ↓21.0920.2917.2816.74
ClimbMix validation loss ↓2.12832.10732.00051.9849
Transformer parameters (% of Standard)100%68%100%75%

To assess variation across training seeds, we repeated the comparison at 778M (32 layers) and 374M (24 layers), matching training tokens and executed depth within each pair.

ScaleSeedStandard loss ↓Loopback loss ↓
374M422.41982.3971
374M13372.42572.3996
374M20272.41752.4031
778M422.25232.2410
778M13372.24602.2384

Loopback achieved lower validation loss than Standard with both seeds at 778M and all three seeds at 374M, using 56.3% and 58.3% of Standard’s Transformer parameters, respectively.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Before Discussion Closes

Dear Reviewer,

We submitted our rebuttal on November 12, four days ago, and have not yet received follow-up feedback. With the discussion closing on November 18, we would appreciate the opportunity to address any remaining concerns.

Our rebuttal and revised manuscript include additional experiments and analyses responding to your comments. Could you please review our rebuttal content and let us know whether they address your concerns and warrant reconsideration of your assessment?

Thank you for your time and consideration.

Best regards, The authors

Reviewer_88RdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how looped Transformers should store and recombine the updates produced by repeated applications of shared blocks. Building on the depth-memory view of residual connections, the authors cast residual addition, Attention Residuals, Hyper-Connections, and linear-attention and state-space recurrences into a common write-transport-read interface. From this they derive two observations. First, storing a block's full output also re-writes its input, so constraints placed on the carry do not hold for the effective transport. Second, any such model's output decomposes into an input strength, a total update weight, and an allocation across updates, and existing rules couple these: joint softmax ties input and update weight together, while plain addition lets the total update weight grow with depth. The proposed Loopback Transformer stores raw per-block updates, keeps the prefix output h0 on a separate path with a fixed coefficient of one, uses a geometric visit-end decay to set a recency prior and bound the total update weight, and uses a lightweight learned softmax attention to reallocate that fixed total across stored updates.

Critical strengths
  1. Clear and useful unifying analysis. The write-transport-read formulation cleanly places a wide range of residual and recurrent rules in one framework. The decomposition into input strength, total update weight, and allocation is a nice conceptual tool that explains why plain looping underperforms and what each existing method can and cannot control.
  2. A nice finding on decay. The ablations show that bounding the total update weight via anchored geometric decay recovers most of the gap between plain looping and dense models.
  3. Simple, well-initialized design. Loopback leaves the shared blocks almost unchanged and starts exactly from the decay-only trajectory via zero-initialized queries.
  4. Broad baseline coverage. The comparison includes mHC, HC, iHC, Hyperloop, MHAR, and Attention Residuals, plus eleven recurrent and regression readers adapted along depth, under a shared protocol
Critical weaknesses
  1. Limited novelty of the main gain. As the paper itself describes, many of the building blocks of the proposed method including delta storage, learned depth-wise softmax reads, and the prefix-loop-coda layout appear in prior works and the authors state that they do not claim delta storage as new. Most of the improvement comes from anchored decay.

  2. No statistical evidence for small margins. All results come from single training runs, and the models are small. Many of the reported differences, including Loopback's margin over Dense at the smallest scale and the gain attributed to learned allocation, are small enough that they could plausibly be explained by run-to-run variation, and the paper's own reruns of comparable settings suggest such variation is not negligible. Without repeated seeds and confidence intervals, it is unclear which of the reported improvements are reliable. Downstream differences are mostly fractions of a point, and the taskwise diagnostics resample evaluation items rather than training runs, so they are not significance tests.

  3. The advantage does not persist with scale. Loopback's loss advantage over Dense holds at the three smaller scales but reverses at the largest scale, where the seven-task mean also drops. The small thirteen-task gain at that scale is concentrated in a few tasks that are either small or erratic across models, and on many tasks Loopback is behind Dense. Topology, parameter ratio, learning rate, and hardware all change across scales, so no scaling trend can be inferred. The abstract highlights only the two smallest scales.

  4. Missing compute-matched, parameter-matched baseline. The compute-matched Dense model has more parameters, while the parameter-matched Dense models (8 or 14 blocks) see the same tokens with far less compute, so neither comparison holds both fixed. A parameter-matched dense model trained to the same total FLOPs (that is, on proportionally more tokens) would show whether looping is a better use of extra compute than more data for a model of the same size.

  5. No generation or long-context evaluation. All downstream tasks are likelihood-scored, plus perplexity. For a new generative architecture, there is no free-form generation evaluation (for example, summarization, translation, or open-ended continuation with MAUVE or a judge, and repetition rates), no measured token-by-token decoding speed, and no check that incremental decoding matches full-forward logits. Training and evaluation are limited to 4,096 tokens.

Questions for authors
  1. Can you report multiple seeds, with confidence intervals, for at least Dense, Plain, decay-only, and Loopback at 191M and 374M?
  2. How does Loopback compare with a dense model of the same stored parameters trained on proportionally more tokens to match total training FLOPs?
  3. Why does the advantage reverse at 1.4B? Was lambda re-tuned at larger scales, and could a different topology or lambda recover the gain?
  4. How do you explain the lower training loss but higher validation loss at 1.4B?
  5. Can you provide any free-form generation evaluation (open-ended continuation quality, repetition rates, or few-shot summarization), with measured token-by-token decoding latency and a check that incremental decoding matches full-forward logits?
  6. What is the measured peak inference memory with chunked prefill at longer contexts (8K-32K)?
  7. How does Loopback compare with Plain and Dense in a wall-clock-matched setting?
  8. Does the benefit persist under longer training (for example, 100+ tokens per parameter)?
7 comments
AuthorsOpenReview ↗

Response to Reviewer 88Rd (Part1)

Thank you for recognizing how our framework unifies residual connections and recurrent memory, and for highlighting the gains from anchored geometric decay. We address each of your points below.

W1: Novelty of the residual design and the role of decay.

Our novelty has two folds. First, we provide a unified view of residual mechanisms in looped Transformers. To the best of our knowledge, this is the first work to bring residual variants into the write–transport–read framework of recurrent memory to study how shared layers reuse information across passes. Second, guided by this view, we designed Loopback, a parameter-efficient looped Transformer inspired by how recurrent memory forgets and retrieves information. Loopback stores individual block updates and geometrically discounts their prior weights at each loop boundary. Attention can still give an older update more weight when its content is useful for the current token. Loopback achieved the lowest validation loss and the highest mean accuracy across 13 zero-shot tasks among the seven looped residual baselines and the Standard Transformer in our 191M, 374M, and 778M comparisons. At 1.4B and 3B, Loopback also outperformed Standard on both metrics at matched training tokens and executed depth, using only 68% and 75% of Standard’s Transformer parameters, respectively.

All thirteen zero-shot task accuracies (%) are listed below.

Task / metric1.4B Standard1.4B Loopback3B Standard3B Loopback
PIQA73.8874.4877.0475.90
HellaSwag55.0256.4362.8463.72
WinoGrande56.5156.9957.2258.48
OpenBookQA37.8036.6040.4041.80
CommonsenseQA23.3418.6719.9021.87
BoolQ62.5163.2462.2364.95
RACE34.6433.4935.1236.17
ARC-Easy66.1667.0069.1971.13
ARC-Challenge36.3537.0340.5340.96
MMLU24.7026.8126.2628.86
ANLI-R131.3034.8031.0032.90
RTE57.4055.2361.0156.68
LAMBADA42.6443.7248.5049.45
13-task mean ↑46.3346.5048.5649.45
Validation loss ↓2.12832.10732.00051.9849
Transformer parameters retained100%68%100%75%

Loopback also achieves the highest thirteen-task mean among Standard and all seven evaluated looped residual baselines in the following comparisons.

Residual design191M, 20 layers ↑374M, 24 layers↑778M, 48 layers ↑
Standard38.2840.0443.04
Plain Loop36.6440.3442.78
mHC38.3040.6943.07
HC38.1540.4443.17
iHC38.4039.5043.14
Hyperloop38.3140.9643.23
MHAR38.3040.0943.09
AttnRes38.1939.3943.50
Loopback38.6440.9743.58

Each comparison matches training tokens and executed depth. Loopback achieves the lowest validation loss and highest thirteen-task mean in all three settings while using fewer Transformer parameters than Standard. Guided by a unified view of residual connections and recurrent memory, Loopback achieves the lowest validation loss and highest thirteen-task mean in these comparisons while retaining only 40–75% of Standard’s Transformer parameters.

AuthorsOpenReview ↗

Response to Reviewer 88Rd (Part2)

W2 / Q1: Statistical reliability across training seeds.

We addressed this concern by completing three training seeds (42, 1337, 2027) for five models at both scales. Decay-only fixes the read queries to zero, so each update’s weight falls geometrically with the number of loop boundaries it has crossed.

The tables report ClimbMix validation loss after 20 TPP. The last column gives the mean difference, Loopback minus the row model, with a paired 95% t-interval across training seeds; negative values favor Loopback. Bold marks the lowest loss in each seed.

191M, 20 layers

ModelSeed 42Seed 1337Seed 2027Mean difference [95% CI]
Standard2.62342.61442.6233−0.0033 [−0.0131, +0.0064]
Plain Loop2.68022.68152.6885−0.0664 [−0.0789, −0.0538]
mHC2.62502.62922.6279−0.0103 [−0.0196, −0.0010]
Decay-only2.62202.62222.6220−0.0050 [−0.0093, −0.0008]
Loopback2.61872.61552.6169—

374M, 24 layers

ModelSeed 42Seed 1337Seed 2027Mean difference [95% CI]
Standard2.41982.42572.4175−0.0211 [−0.0360, −0.0061]
Plain Loop2.44152.44092.4372−0.0399 [−0.0531, −0.0268]
mHC2.40622.40792.4043−0.0062 [−0.0170, +0.0046]
Decay-only2.39992.40572.4089−0.0049 [−0.0094, −0.0004]
Loopback2.39712.39962.4031—

Loopback improves over Plain Loop, decay-only, and mHC in every seed at both scales. Learned allocation improves loss in all six paired comparisons, with paired 95% intervals below zero at both scales. At 374M, Loopback also improves over Standard in every seed while retaining 58.3% of its Transformer parameters; its thirteen-task mean increases by 0.83 [+0.37, +1.30] percentage points across training seeds. At 191M, Loopback achieves comparable validation loss to Standard with only 40% of its Transformer parameters. These repeated runs support the statistical evidence for our method and additional benefit of learned allocation beyond the fixed decay prior.

W3 / Q3 / Q4: Larger-scale results and the 1.4B reversal.

Our follow-up 1.4B and 3B comparisons match training tokens and executed depth, with λ = 1/4 unchanged. The table reports all thirteen zero-shot task accuracies (%), their means, and ClimbMix validation loss after 20 TPP (seed 42). Parameter ratios exclude token embeddings and the output head.

Task / metric1.4B Standard1.4B Loopback3B Standard3B Loopback
PIQA73.8874.4877.0475.90
HellaSwag55.0256.4362.8463.72
WinoGrande56.5156.9957.2258.48
OpenBookQA37.8036.6040.4041.80
CommonsenseQA23.3418.6719.9021.87
BoolQ62.5163.2462.2364.95
RACE34.6433.4935.1236.17
ARC-Easy66.1667.0069.1971.13
ARC-Challenge36.3537.0340.5340.96
MMLU24.7026.8126.2628.86
ANLI-R131.3034.8031.0032.90
RTE57.4055.2361.0156.68
LAMBADA42.6443.7248.5049.45
Seven-task mean ↑52.6253.1856.5357.35
13-task mean ↑46.3346.5048.5649.45
Validation loss ↓2.12832.10732.00051.9849
Transformer parameters (% of Standard)100%68%100%75%

The original 1.4B result is consistent with a generalization gap. It motivated us to revisit the balance between shared and unshared blocks at fixed executed depth. Through short topology comparisons (5 TPP), we selected [3→8×2→6] for full training. We reran Standard on the same eight H100s with matched training tokens. At 20 TPP, the follow-up Loopback configuration achieves lower training and validation losses by 0.0212 and 0.0210 nats, respectively.

At 3B, Loopback uses [6→6×2→6] against a 24-block Standard; both use width 3,072, peak learning rate 0.001, and eight H100s. Loopback improves 11 of 13 tasks. We will include these results in the abstract as per-scale parameter-efficiency comparisons.

AuthorsOpenReview ↗

Response to Reviewer 88Rd (Part3)

W5 / Q5 / Q6: Generation, incremental decoding, and long-context evaluation.

We added few-shot dialogue summarization on SAMSum using the LongBench prompts and greedy decoding, scored by ROUGE-L. Inputs are capped at 3,500 tokens. Loopback achieves higher ROUGE-L at both scales:

ScaleStandard ROUGE-L ↑Loopback ROUGE-L ↑
1.4B18.5322.47
3B24.2228.66

We also evaluated translation on WMT14 French→English (3,003 sentences) and WMT16 German→English (2,999 sentences), using lm-eval's 5-shot prompts, greedy decoding, and sacreBLEU.

ScaleTranslation taskBLEU ↑ (Standard / Loopback)chrF ↑ (Standard / Loopback)
1.4BWMT14 Fr→En3.12 / 7.4622.23 / 31.28
1.4BWMT16 De→En3.31 / 5.8520.98 / 27.85
3BWMT14 Fr→En11.40 / 16.2536.81 / 43.58
3BWMT16 De→En9.24 / 14.3532.90 / 39.48

Loopback scores higher on both metrics in all four comparisons. At 3B, paired 95% bootstrap intervals for the BLEU gains are [+4.34, +5.37] and [+4.54, +5.71], respectively.

We evaluated open-ended continuation at 1.4B and 3B on 2,000 held-out ClimbMix prompts, generating 256 tokens from each 128-token prompt. Sampled generation uses nucleus sampling with p = 0.9.

Evaluation1.4B Standard / Loopback3B Standard / Loopback
MAUVE, sampled ↑0.955 / 0.9620.962 / 0.949
Repeated 4-grams, sampled ↓0.040 / 0.0380.038 / 0.038

Loopback achieves higher continuation MAUVE and lower sampled repetition at 1.4B. At 3B, Standard scores higher on MAUVE (0.962 vs. 0.949), and both models have a sampled four-gram repetition rate of 0.038 at the reported precision.

We also implemented KV-cached decoding and checked trained checkpoints against full-forward evaluation. In FP32, relative logit errors are 1.8–3.4 × 10⁻⁶, with 100% argmax agreement and identical tested 32-token greedy continuations. BF16 argmax agreement is 99.2–99.7% across the two architectures.

After extensive kernel optimization, we measured the trained 3B checkpoints using vLLM 0.31 on eight H100 GPUs, with tensor parallelism 8, one active request, and 8K generated tokens. The table reports single-request decoding throughput, excluding prefill:

Prompt lengthStandard (tokens/s) ↑Loopback (tokens/s) ↑
32K511.8489.5
64K413.6400.0
128K304.4297.1

With a 32K prompt, single-request decoding takes 1.954 ms/token for Standard and 2.043 ms/token for Loopback. Across the three prompt lengths, Loopback adds only 0.08–0.09 ms per generated token, from additional history-read kernels and cross-GPU all-reduce operations. In the memory-constrained 100B benchmark below, reduced weight storage enables more concurrent requests and higher aggregate throughput within the same GPU memory budget.

Loopback’s parameter savings also reduce inference memory. In a single-H100 benchmark with 4,096-token prefill chunks, Loopback uses less peak memory at both trained scales:

Peak prefill memory (GiB) ↓1.4B Standard / Loopback3B Standard / Loopback
8K tokens4.71 / 4.228.29 / 7.35
16K tokens6.37 / 5.8710.56 / 9.61
32K tokens9.54 / 9.0215.11 / 14.15

To test how these memory savings affect serving capacity, we benchmark a 100B reference architecture with random weights on four H100 GPUs using vLLM 0.31 and the latest kernels, which reduce history-read workspace. With vLLM’s memory budget set to 85% on each GPU and the same KV storage per token, Standard can cache 210,256 tokens and Loopback 303,888 (+44.5%). Each request generates 8K tokens. We report aggregate decoding throughput after prefill at the maximum number of requests each model can serve concurrently.

Prompt tokensMax concurrent requests (Standard / Loopback)Standard (tokens/s)Loopback (tokens/s)Throughput ratio
32K5 / 7181.5226.11.25×
64K2 / 478.1127.21.63×
128K1 / 238.164.11.68×

To evaluate long-context prediction, we additionally continued training both 374M models on the same approximately 2.6B Books3 tokens with 16K sequences. We evaluated next-token loss on 64 held-out documents, each extending to 32K tokens:

Token-position intervalStandard loss ↓Loopback loss ↓
0–1K2.77742.7423
1–2K2.69982.6783
2–4K2.66072.6342
4–8K2.58752.5632
8–16K2.56512.5394
16–32K2.57992.5502

When both 374M models were trained only on 4K-token sequences, Loopback’s loss was 0.186 nats higher than Standard’s at positions 16K–32K. After matched continuation on 16K-token sequences, Loopback had lower loss in every interval, including 16K–32K (2.5502 vs. 2.5799).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

General Response to All Reviewers

We thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of the unified analysis of residual and recurrent designs, the simplicity of Loopback, and the breadth of the baseline comparisons. At matched training tokens and executed depth, Loopback achieves lower validation loss and higher thirteen-task mean accuracy than Standard at both 1.4B and 3B, retaining only 68% and 75% of its Transformer parameters, respectively. Their suggestions guided additional experiments on statistical reliability, residual design, training budgets, generation, and practical efficiency.

We summarize the additional results below and address each reviewer’s points in the individual responses.

  • Unified view and larger-scale performance. We provide a unified view of residual mechanisms through the write–transport–read framework of linear attention and recurrent memory. This view led to Loopback: geometric forgetting discounts earlier block updates, while attention learns which updates remain useful for each token. Loopback achieves the lowest validation loss and highest thirteen-task mean among Standard and seven looped residual baselines in our 191M, 374M, and 778M comparisons. At 1.4B and 3B, it also outperforms Standard on both metrics at matched training tokens and executed depth, retaining only 68% and 75% of Standard’s Transformer parameters, respectively.

  • Reliability across training seeds. We completed three seeds for five models at both 191M and 374M. Learned allocation improves over decay-only in all six paired comparisons, reducing validation loss by 0.0050 and 0.0049 nats on average, with both paired 95% intervals excluding zero. Loopback also improves over Standard in all three 374M seeds and both 778M seeds.

  • Performance across depths. We added a new depth study to the revision to test whether Loopback’s gains persist across different model depths. We trained all nine designs from 16 to 64 layers at 374M and 778M. Loopback achieves lower validation loss than Standard throughout this depth sweep, while retaining only 56.3–58.3% of Standard’s Transformer parameters.

  • Controlled residual ablations. We tested learned block scaling, per-loop gating, and input injection at 191M and 374M. Loopback achieves lower validation loss than all three controls at both scales. These comparisons complement the repeated-seed ablation of learned allocation.

  • Training resources and longer training. At 191M and 374M, Loopback achieves lower loss than nearly parameter-matched Standard models trained with slightly more total FLOPs. At 374M, Loopback also achieves lower validation loss than Standard under the same training-time budget. Its advantage persists through 100 TPP: validation loss is 2.2561, versus 2.2620 for Standard and 2.2633 for mHC. The advantage at 50 TPP is reproduced across two seeds.

  • Generation and long-context evaluation. On SAMSum, ROUGE-L improves from 18.53 to 22.47 at 1.4B and from 24.22 to 28.66 at 3B. Loopback also achieves higher BLEU and chrF on both tested translation tasks at both scales. We added open-ended continuation results and verified incremental decoding against full-forward evaluation. After matched continuation with 16K sequences at 374M, Loopback achieves lower next-token loss in every evaluated position interval through 32K.

  • Practical efficiency. Optimizing the history-read kernel increases 374M training throughput by 17.0% over the previous implementation. At 3B, Loopback reaches 98% of Standard’s training throughput with 0.7% more peak reserved training memory. Peak prefill memory is lower at 8K–32K at both trained scales, including 14.15 versus 15.11 GiB at 3B and 32K. In a separate 100B reference benchmark with random weights on four H100s, Loopback supports twice as many concurrent 64K requests within the same memory budget, delivering 63% higher aggregate decoding throughput.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.