acceptodds
Under review as a conference paper at ICLR 2027

Asymmetric Looped Transformers: Fixed Global Representations for an Iterative Decoder

Abstract

Looped Transformers increase effective depth by repeatedly applying shared lay- ers, but full-stack recurrence does not explicitly distinguish context construction from iterative state refinement. We introduce the Asymmetric Looped Trans- former (AsymLoop), which separates a single-pass causal encoder from a weight- tied recurrent decoder. The encoder produces layer-specific Global key–value caches that remain fixed across iterations; the decoder jointly attends to these caches and Local keys and values derived from its evolving state under a shared softmax. We further adapt existing heavy-tail-guided layerwise learning-rate allo- cation into a Dynamic Learning Rate Schedule (DLRS) for differentiated matrix- wise updates. Experiments on FineWeb-Edu across four nominal model tiers, from 60M to 1B, show lower validation perplexity and higher average zero-shot accuracy across seven downstream tasks than Vanilla Full Loop at matched train- ing budgets and effective depth. AsymLoop reduces perplexity by 0.98 and 0.30 at the 135M and 350M tiers, respectively, and improves the seven-task average from 49.58% to 50.85% at 1B. Ablations support complementary roles for fixed and evolving KV, while showing that advantages over tuned learning rates and strengthened full-loop baselines are not uniform. These single-seed results moti- vate asymmetric information exchange as a design direction for recurrent language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7Y6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AsymLoop, dividing the model into a causal encoder executed once and a weight-tied decoder repeated K times. The encoder output is mapped into layer-specific Global KV caches that remain constant across iterations, while each decoder layer attends to both these Global KV and the Local KV from its current state using a single softmax. For training, the authors employ a previously established heavy-tail-guided layerwise learning-rate method, referred to as ALRS. The authors pre-train LLaMA-style models ranging from 60M to 1B parameters and report lower validation perplexity and higher average zero-shot accuracy compared to Dense and Full Loop at matched effective depth. Additional improvements are reported with extended training budgets, an OLMo-style backbone, sparse MoE layers, GSM8K fine-tuning, and a FLOPs-matched wider Full Loop.

Critical strengths
  1. The design is straightforward and well-articulated. Integrating a single-pass encoder with a looped decoder via a fixed Global KV is straightforward to implement, and the additional parameters and FLOPs are explicitly analyzed.
  2. The experimental scope is comprehensive. Models are trained from scratch at four different scales up to 1B parameters, and the evaluation encompasses downstream tasks, a 5x Chinchilla budget, recurrent depths from K = 1 to 5, an alternative backbone, MoE layers, and a compute-matched baseline.
Critical weaknesses
  1. The contribution of the architecture is not clearly distinguished from the impact of the learning-rate method. The paper does not clarify whether Dense and Full Loop models in the main results are also trained with ALRS, and only AsymLoop is labeled with ALRS in Table 2 and Appendices E and G. The 60M results in Table 3 display ratios similar to those of the uniform-LR baselines compared to ALRS AsymLoop in Tables 4 and 8. The paper's ablation studies indicate that ALRS alone reduces Full Loop perplexity by approximately 0.8 (Table 4c), and a Dense model with layerwise learning rates already surpasses the uniform-LR Full Loop at both K = 2 and K = 3 (Table 8). Therefore, a significant portion of the reported improvements may be attributable to the optimizer rather than the asymmetric design.
  2. The evidence supporting the claim that performance gains arise from recurrence or role separation is limited. AsymLoop with K = 1, which lacks recurrence, already outperforms Full Loop with K = 2 and K = 3 at 60M and 135M parameters (Table 3), and additional iterations yield only marginal improvements. In Table 7, both the encoder–decoder split without Global KV and the additive injection variant perform worse than Full Loop, suggesting that the primary benefit may result from the additional Global KV attention path derived from CED. Furthermore, the advantage over Full Loop decreases from approximately 7% lower perplexity at 60M to about 0.5% at 1B in a single run.
  3. Results for identical configurations are inconsistent across tables. For example, at 60M, AsymLoop with K = 3 reports a perplexity of 21.45 in Table 3, 23.75 in Table 4, and approximately 19.5 based on the cross-entropy in Table 10. Uniform-LR Full Loop with K = 2 is reported as 25.57 in Tables 4c and 8, but 25.05 in Table 7. The evaluation protocols underlying these results are not specified. Additionally, Figure 6 indicates 59.6K training steps for the 1B models, which equates to approximately 31B tokens, rather than the stated 20B.
Questions for authors

Most of my concerns are included in the weakness section. Specifically, I would like to ask the authors:

  1. Are the Dense and Full Loop baselines in the main results and Appendices E, G, and I trained with ALRS or with uniform learning rates? If the latter, how do the comparisons change when all models employ the same learning-rate method?
  2. Given that AsymLoop with K = 1 already outperforms Full Loop with K = 2 and K = 3 at 60M and 135M, could the authors clarify the extent to which the observed gains are attributable to recurrence as opposed to the Global KV path?
  3. Why do identical 60M configurations yield different perplexities in Tables 3, 4, 7, and 10, and which evaluation protocol is employed in each table?
  4. What is the actual training budget for the 1B models, considering that 59.6K steps in Figure 6 correspond to approximately 31B tokens with a global batch size of 512 and a sequence length of 1024?
  5. Are the 1B results based on a single seed? Given that the perplexity gap at 1B is only 0.05, what is the seed variance at this scale?
7 comments
AuthorsOpenReview ↗

[W1 & Q1]: Do the main and appendix baselines use ALRS, and how do comparisons change when all models use the same learning-rate method?

We sincerely thank you for your time and insightful comments. We address your concerns point-by-point below.

Thank you for raising this concern.

(1) Learning-rate method used by each baseline.

In the original main experiments, Dense and Full Loop use uniform learning rates, while AsymLoop uses ALRS; all models within each scale share the same base learning rate. To assess whether AsymLoop’s advantage persists when the baselines also use ALRS, we conduct six additional Dense and Full Loop runs at 135M and 350M, and train the 1B Dense baseline with ALRS using three seeds. These runs retain the corresponding original architectures and training-token budgets. We report the comparisons below.

(2) Comparisons with ALRS applied to the baselines at 135M and 350M.

We completed six additional Dense and Full Loop runs with ALRS at 135M and 350M, retaining the original architectures, training-token budgets, and effective-depth pairings. As shown in Rebuttal Table 1, AsymLoop achieves lower perplexity in all six comparisons. These results show that the gap persists when ALRS is applied to the baselines.

Rebuttal Table 1. Main baselines with and without ALRS

ScaleModelKEff. depthFFNPPL ↓
135MDense112204817.46
135M{\color{#b45309}\text{135M}}Dense + ALRS{\color{#b45309}\text{Dense + ALRS}}1{\color{#b45309}\text{1}}12{\color{#b45309}\text{12}}2048{\color{#b45309}\text{2048}}17.22{\color{#b45309}\text{17.22}}
135MAsymLoop112204816.54
135MFull Loop224204817.12
135M{\color{#b45309}\text{135M}}Full Loop + ALRS{\color{#b45309}\text{Full Loop + ALRS}}2{\color{#b45309}\text{2}}24{\color{#b45309}\text{24}}2048{\color{#b45309}\text{2048}}16.56{\color{#b45309}\text{16.56}}
135MAsymLoop324204816.14
135MFull Loop336204816.81
135M{\color{#b45309}\text{135M}}Full Loop + ALRS{\color{#b45309}\text{Full Loop + ALRS}}3{\color{#b45309}\text{3}}36{\color{#b45309}\text{36}}2048{\color{#b45309}\text{2048}}16.19{\color{#b45309}\text{16.19}}
135MAsymLoop536204816.09
350MDense124273612.55
350M{\color{#b45309}\text{350M}}Dense + ALRS{\color{#b45309}\text{Dense + ALRS}}1{\color{#b45309}\text{1}}24{\color{#b45309}\text{24}}2736{\color{#b45309}\text{2736}}12.62{\color{#b45309}\text{12.62}}
350MAsymLoop124273612.20
350MFull Loop248273612.30
350M{\color{#b45309}\text{350M}}Full Loop + ALRS{\color{#b45309}\text{Full Loop + ALRS}}2{\color{#b45309}\text{2}}48{\color{#b45309}\text{48}}2736{\color{#b45309}\text{2736}}12.35{\color{#b45309}\text{12.35}}
350MAsymLoop348273612.00
350MFull Loop372273612.15
350M{\color{#b45309}\text{350M}}Full Loop + ALRS{\color{#b45309}\text{Full Loop + ALRS}}3{\color{#b45309}\text{3}}72{\color{#b45309}\text{72}}2736{\color{#b45309}\text{2736}}12.10{\color{#b45309}\text{12.10}}
350MAsymLoop572273611.92
AuthorsOpenReview ↗

[W2 & Q2]: Given AsymLoop’s advantage at , how much of the gain comes from recurrence rather than the Global KV path?

Thank you for raising this question about the source of AsymLoop’s gains. To distinguish the contributions of recurrence and the Global KV path, we conduct a 2×2 ablation at 135M, varying K∈{1,3}K\in\{1,3\} and the presence of Global KV. All variants share the same 6+6 encoder–decoder split, QKNorm, ALRS, seed, global batch size, data budget, and rank count.

Rebuttal Table 2. Recurrence and Global KV ablation at 135M

Model variantParametersCE ↓, K=1K=1CE ↓, K=3K=3
Without Global KV134.30M2.838852.81738
With Global KV141.38M2.806642.78120

Increasing KK from 1 to 3 reduces validation CE by 0.0215 without Global KV and by 0.0254 with Global KV, without adding parameters in either case. Adding the Global KV path reduces validation CE by 0.0322 at K=1K=1 and by 0.0362 at K=3K=3. These results support complementary contributions: recurrence provides additional improvement beyond the non-recurrent Global KV configuration, while Global KV increases the benefit of recurrent computation. Their combination achieves the lowest CE under the same training data budget.

AuthorsOpenReview ↗

[W3 & Q3]: Why do identical 60M configurations yield different perplexities across Tables 3, 4, 7, and 10 of the submitted paper, and which evaluation protocol does each use?

We thank the reviewer for pointing out that our experimental settings were insufficiently described.

(1) Unified evaluation settings and the protocol for each table.

In the original manuscript, some experiments in the main text and appendix used different training-token accounting methods and validation subsets, resulting in different reported values for the same configuration. To resolve this ambiguity, we have re-evaluated all experiments under unified evaluation settings and matched training budgets, and updated the relevant tables in both the main text and appendix so that corresponding results use consistent settings. All changes are marked in blue. We also document the experimental settings for each table in Appendix J, including training budgets and token accounting, validation subsets, and evaluation protocols.

(2) Why identical 60M configurations differ across tables.

The numerical discrepancies in the original manuscript mainly arose from batch counting during evaluation and training-token budgets. Table 3 of the submitted paper computed validation loss by averaging over batches, using different batch-indexing and counting conventions from Table 4(c). Table 10 used a training budget of 1.6B valid tokens, excluding padding, which differed from the actual training budget in Tables 3 and 4(c). All comparisons in the paper were conducted within individual tables under shared training and evaluation settings; no conclusions relied on comparing absolute perplexities across tables. These cross-table differences therefore do not affect the conclusions supported by the comparisons within each experiment.

Reviewer_9gS6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AsymLoop, a recurrent Transformer architecture that separates single-pass causal context encoding from iterative state refinement. The encoder produces layer-specific Global KV representations that remain fixed across decoder iterations. A weight-tied decoder jointly attends to these representations and Local KV computed from its evolving hidden states, using a shared softmax. Training adopts an existing spectral layerwise learning-rate allocation method, termed ALRS in this configuration. Experiments compare AsymLoop with Dense and Full Loop models across four configurations up to approximately 1.4B parameters, with additional studies on longer training, an OLMo-style backbone, sparse MoE models, and GSM8K fine-tuning. Ablations investigate the attention components and optimization choices, while Appendix E includes approximately FLOPs-matched comparisons with widened Full Loop baselines.

Critical strengths
  1. The architecture provides a clear mechanism for repeatedly accessing contextual representations without rebuilding them inside every recurrent iteration. Joint attention to fixed Global KV and evolving Local KV is a concrete architectural contribution. The ablations under uniform learning rates show that removing either source worsens validation perplexity, and the contribution of Global KV increases with recurrent depth in Appendix C.1. The parameter-matched ablation and additive-injection comparison provide additional evidence that the particular context-access mechanism matters.

  2. The paper makes a substantive effort to control computational cost. Appendix E compares AsymLoop with widened Full Loop models and verifies that measured training matrix FLOPs differ by only approximately 1–2%. At 135M, three-seed results show lower validation cross-entropy for AsymLoop than for the widened baseline, with separation larger than the reported run-to-run standard deviations. Together with the results across model scales and training budgets, this supports the empirical promise of the proposed architecture beyond a single endpoint comparison.

Critical weaknesses
  1. The architectural contribution is not sufficiently isolated from optimization and normalization choices in the comparisons supporting the main scaling claims. Table 4(c) shows that ALRS alone improves Full Loop perplexity from 25.571 to 24.787, so the optimization configuration has a material effect. The paper provides useful architecture ablations under uniform learning rates and an ALRS control for Full Loop at 60M, but it does not clearly establish equivalent optimization and normalization controls throughout the larger-scale and widened-baseline experiments. Moreover, the main scaling comparison emphasizes Full Loop. The no-Global-KV and additive-injection ablations are relevant partial-loop controls, but they remain small-scale tests rather than comparisons with well-tuned partial-loop alternatives at the scales supporting the main claims. This limits the evidence that fixed Global KV provides a robust advantage over established ways of organizing recurrent computation.

  2. The mapping between reported results and experimental protocols is insufficiently documented. For example, the 60M Full Loop model has perplexity 23.03 in Table 3, 25.571 in Table 4(c), and cross-entropy 3.0434 in Table 10, corresponding to perplexity approximately 20.98. Different training or evaluation configurations could explain these values, but their correspondence is not adequately specified. There is also an apparent budget/indexing discrepancy: Appendix A states a 20B-token budget, global batch size 512, and sequence length 1024 for the 1B configuration, implying approximately 38.1k optimizer steps, whereas Figure 6 extends to 59.6k on an axis labeled Training Step. These ambiguities make it difficult to verify which experiments share the claimed budgets and evaluation conditions. The main downstream results also lack uncertainty estimates, which matters particularly for the smaller accuracy differences.

Questions for authors
  1. Please specify, for every main and compute-matched baseline, whether QKNorm and ALRS are enabled and how the base learning rate was selected. Does AsymLoop retain its advantage over widened Full Loop when both use the same normalization and spectral learning-rate allocation procedure?

  2. Please reconcile the 60M Full Loop results in Tables 3, 4(c), and 10 by listing the checkpoint, training budget, learning-rate configuration, validation split, and evaluation procedure for each. Please similarly clarify the relationship between the stated 20B-token budget and the 59.6k training-step endpoint in Figure 6.

  3. How does AsymLoop compare with a well-tuned partial-loop model using persistent input injection at 135M or larger scale under matched training FLOPs and optimization settings? The small-scale no-Global-KV and additive-injection comparisons are useful, but a larger-scale control would materially strengthen the architectural conclusion.

  4. What uncertainty accompanies the downstream accuracy differences? Please provide paired evaluation uncertainty where possible, and clarify whether the main 1B result and extended-budget results have been checked across training seeds. This would help distinguish consistent gains from differences sensitive to the particular run or evaluation sample.

10 comments
AuthorsOpenReview ↗

[W1 & Q1]: How are base learning rates selected, and does AsymLoop retain its advantage when normalization and ALRS are matched across baselines?

Thank you for raising this point. We specify the settings first. In the submitted main experiments, Dense and Full Loop use uniform learning rates and QKNorm without ALRS; within each scale all three models share the same base learning rate, taken from the grid-searched scaling ladder of LLR (ICML 2026) [1] (10−310^{-3} at 60M, 135M and 350M; 5×10−45\times10^{-4} at 1B), on top of which ALRS applies matrix-wise multipliers. We agree that this leaves the architecture entangled with the optimization and normalization settings, and we have therefore run the matched controls described below; the revised manuscript states the QKNorm and ALRS status of every main and compute-matched baseline.

(1) Settings of all baselines: QKNorm, ALRS, and base learning rates.

The original results reproduced in Rebuttal Table 1 follow the settings above: Dense and Full Loop with uniform learning rates and QKNorm enabled, and AsymLoop with both QKNorm and ALRS, at the same base learning rate within each scale.

(2) Additional controls with ALRS on the baselines.

To assess whether AsymLoop’s advantage persists once the learning-rate allocation is aligned, we conduct six additional Dense and Full Loop runs at 135M and 350M with both ALRS and QKNorm enabled. These runs retain the original architectures and training-token budgets and are compared with AsymLoop at matched effective depths. AsymLoop achieves lower perplexity in all six comparisons, which indicates that its advantage is not attributable to ALRS alone.

Rebuttal Table 1. Main baselines with and without ALRS

ScaleModelKEff. depthFFNPPL ↓
135MDense112204817.46
135M{\color{#b45309}\text{135M}}Dense + ALRS{\color{#b45309}\text{Dense + ALRS}}1{\color{#b45309}\text{1}}12{\color{#b45309}\text{12}}2048{\color{#b45309}\text{2048}}17.22{\color{#b45309}\text{17.22}}
135MAsymLoop112204816.54
135MFull Loop224204817.12
135M{\color{#b45309}\text{135M}}Full Loop + ALRS{\color{#b45309}\text{Full Loop + ALRS}}2{\color{#b45309}\text{2}}24{\color{#b45309}\text{24}}2048{\color{#b45309}\text{2048}}16.56{\color{#b45309}\text{16.56}}
135MAsymLoop324204816.14
135MFull Loop336204816.81
135M{\color{#b45309}\text{135M}}Full Loop + ALRS{\color{#b45309}\text{Full Loop + ALRS}}3{\color{#b45309}\text{3}}36{\color{#b45309}\text{36}}2048{\color{#b45309}\text{2048}}16.19{\color{#b45309}\text{16.19}}
135MAsymLoop536204816.09
350MDense124273612.55
350M{\color{#b45309}\text{350M}}Dense + ALRS{\color{#b45309}\text{Dense + ALRS}}1{\color{#b45309}\text{1}}24{\color{#b45309}\text{24}}2736{\color{#b45309}\text{2736}}12.62{\color{#b45309}\text{12.62}}
350MAsymLoop124273612.20
350MFull Loop248273612.30
350M{\color{#b45309}\text{350M}}Full Loop + ALRS{\color{#b45309}\text{Full Loop + ALRS}}2{\color{#b45309}\text{2}}48{\color{#b45309}\text{48}}2736{\color{#b45309}\text{2736}}12.35{\color{#b45309}\text{12.35}}
350MAsymLoop348273612.00
350MFull Loop372273612.15
350M{\color{#b45309}\text{350M}}Full Loop + ALRS{\color{#b45309}\text{Full Loop + ALRS}}3{\color{#b45309}\text{3}}72{\color{#b45309}\text{72}}2736{\color{#b45309}\text{2736}}12.10{\color{#b45309}\text{12.10}}
350MAsymLoop572273611.92

(3) Widened Full Loop with matched optimization and training FLOPs.

We further compare AsymLoop with widened Full Loop baselines at 135M and 350M, controlling learning-rate scheduling and training FLOPs. Both models use ALRS and QKNorm, with matched effective depth and training-token budgets at each scale. We widen the Full Loop FFN to approximately match AsymLoop’s training FLOPs. As shown in Rebuttal Table 2, AsymLoop achieves lower perplexity at both scales: 16.14 versus 16.32 at 135M, and 12.00 versus 12.24 at 350M. These results show that AsymLoop retains its advantage in the tested configurations when the learning-rate allocation is aligned and training FLOPs are approximately matched.

Rebuttal Table 2. Widened Full Loop with matched optimization and training FLOPs

ScaleModelEff. depthFFNPPL ↓
135MA: AsymLoop24204816.14
135M{\color{#b45309}\text{135M}}D: Full Loop, widened{\color{#b45309}\text{D: Full Loop, widened}}24{\color{#b45309}\text{24}}2286{\color{#b45309}\text{2286}}16.32{\color{#b45309}\text{16.32}}
350MA: AsymLoop48273612.00
350M{\color{#b45309}\text{350M}}D: Full Loop, widened{\color{#b45309}\text{D: Full Loop, widened}}48{\color{#b45309}\text{48}}2974{\color{#b45309}\text{2974}}12.24{\color{#b45309}\text{12.24}}
References
[1] He et al. One LR Doesn’t Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs. ICML 2026. Paper
AuthorsOpenReview ↗

[W2 & Q2]: What explains the inconsistent 60M results across tables and the discrepancy between the stated 1B token budget and training steps?

We thank the reviewer for pointing out that our experimental settings were insufficiently described.

(1) Unified evaluation settings and per-table protocols.

In the original manuscript, some experiments in the main text and appendix used different training-token accounting methods and validation subsets, resulting in different reported values for the same configuration. To resolve this ambiguity, we have re-evaluated all experiments under unified evaluation settings and matched training budgets, and updated the relevant tables in both the main text and appendix so that corresponding results use consistent settings. All changes are marked in blue. We also document the experimental settings for each table in Appendix J, including training budgets and token accounting, validation subsets, and evaluation protocols.

(2) Why identical 60M configurations differ across tables.

The numerical discrepancies in the original manuscript mainly arose from batch counting during evaluation and training-token budgets. Table 3 of the submitted paper computed validation loss by averaging over batches, using different batch-indexing and counting conventions from Table 4(c). Table 10 used a training budget of 1.6B valid tokens, excluding padding, which differed from the actual training budget in Tables 3 and 4(c). All comparisons in the paper were conducted within individual tables under shared training and evaluation settings; no conclusions relied on comparing absolute perplexities across tables. These cross-table differences therefore do not affect the conclusions supported by the comparisons within each experiment.

(3) The 1B token budget and the 59.6K training steps.

For the 1B experiments, the 20B-token budget excludes padding, whereas multiplying optimizer updates by batch size and sequence length includes padding positions. The recorded 59,633 updates correspond to approximately 31.265B input positions but 20.0002B valid prediction tokens. These counts are identical across Dense, Full Loop, and AsymLoop, reconciling the stated budget with the endpoint in Figure 6.

AuthorsOpenReview ↗

[W1 & Q3]: How does AsymLoop compare with a well-tuned partial-loop model using persistent input injection at larger scales under matched FLOPs and optimization settings?

Thank you for this suggestion. At 135M and 350M, AsymLoop achieves lower perplexity than both the no-Global-KV variant and the persistent additive-injection baseline in the configurations reported below. We extend the Global KV ablations to 135M and 350M, comparing AsymLoop with two variants: removing Global KV, and replacing it with additive injection of the fixed encoder output at every decoder iteration. Both variants retain the same encoder–decoder split, effective depth, and training-token budget. We widen the injection baseline’s FFNs to approximately match AsymLoop’s training FLOPs. These controls assess the contribution of Global KV and compare its attention-based context access with persistent additive injection at larger scales.

Rebuttal Table 3. Larger-scale context-access controls

ScaleModelEnc. + decoderDecoder KEff. depthFFNPPL ↓
135MA: AsymLoop6 + 6324204816.14
135M{\color{#b45309}\text{135M}}B: w/o Global KV{\color{#b45309}\text{B: w/o Global KV}}6 + 6{\color{#b45309}\text{6 + 6}}3{\color{#b45309}\text{3}}24{\color{#b45309}\text{24}}2048{\color{#b45309}\text{2048}}16.86{\color{#b45309}\text{16.86}}
135M{\color{#b45309}\text{135M}}C: additive injection, widened{\color{#b45309}\text{C: additive injection, widened}}6 + 6{\color{#b45309}\text{6 + 6}}3{\color{#b45309}\text{3}}24{\color{#b45309}\text{24}}2286{\color{#b45309}\text{2286}}16.61{\color{#b45309}\text{16.61}}
350MA: AsymLoop12 + 12348273612.00
350M{\color{#b45309}\text{350M}}B: w/o Global KV{\color{#b45309}\text{B: w/o Global KV}}12 + 12{\color{#b45309}\text{12 + 12}}3{\color{#b45309}\text{3}}48{\color{#b45309}\text{48}}2736{\color{#b45309}\text{2736}}12.54{\color{#b45309}\text{12.54}}
350M{\color{#b45309}\text{350M}}C: additive injection, widened{\color{#b45309}\text{C: additive injection, widened}}12 + 12{\color{#b45309}\text{12 + 12}}3{\color{#b45309}\text{3}}48{\color{#b45309}\text{48}}2974{\color{#b45309}\text{2974}}12.37{\color{#b45309}\text{12.37}}

AsymLoop achieves the lowest perplexity at both scales. Removing Global KV is the most costly change, raising perplexity by 0.72 at 135M and 0.54 at 350M. Replacing attention-based access with persistent additive injection recovers part of this gap but still trails AsymLoop by 0.47 and 0.37, even with a widened FFN that approximately matches AsymLoop’s training FLOPs. The ranking is the same at both scales, indicating that attending to a fixed Global KV cache is more effective than additively injecting the same encoder output.

Reviewer_8ZTxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Asymmetric Looped Transformer (AsymLoop), which decomposes the Full Looped Transformer into a single pass encoder and a multiple pass decoder. The 1-pass encoder provides Global KV and the recurrent decoder provides Local KV for the decoding process. Joint attention is applied to acquire info from both global KV and local KV. Empirical result shows competitive performance over dense and full loop baselines for different model scales.

Critical strengths
  1. Compared with the full loop baseline, the 1-pass encoder of AsymLoop provides a stable representation for both training and potential downstream tasks.
  2. The empirical result shows AsymLoop achieves a better performance over the full loop baseline at the matched depths and parameters. Proving parts of the efficiency of AsymLoop.
Critical weaknesses
  1. The paper lacks a full landscape of AsymLoop's performance on (parameters, depths, number of recurrence K). In table 3, the comparison is at the matched effective depths and approximate parameters. However, the comparison is missing at the matched K. For instance, "4 Loop / 4 Total, K = 3, FullLoop" is a comparison of "3 Loop / 6 Total, K = 3, AsymLoop" at the matched depths and K. "6 Loop / 6 Total, K = 3, FullLoop" is a comparison at the matched K and approximate parameters.

  2. There lacks direct evidence to show the advantage of 1-pass encoder design. The ablation shows the necessity of global KV. But the necessity of global KV doesn't mean the necessity of 1-pass encoder design. There exists other way to establish a global KV. For example, considering a full loop variant that derives a fixed global KV from an intermediate first-pass representation while still repeating all layers.

Questions for authors

See Weakness.

4 comments
AuthorsOpenReview ↗

[W1]: How does AsymLoop compare with Full Loop at matched recurrence count, with either effective depth or parameter count matched?

Thank you for this suggestion. We compare Full Loop and AsymLoop under two settings, and AsymLoop achieves lower perplexity in both at the same iteration count (K=3K=3): (1) matched KK and effective depth, using a shallower Full Loop stack; and (2) matched KK and approximately matched parameter counts, retaining the original Full Loop stack. Rebuttal Table 1 reports setting (1) and Rebuttal Table 2 reports setting (2).

Rebuttal Table 1. Matched recurrence count and effective depth

ScaleModelEnc. + loopKKEff. depthPPL
60MFull Loop (shallow)0+40+431220.6616
60MFull Loop (original)0+60+621220.9768
60MAsymLoop3+33+331219.4600
135MFull Loop (shallow)0+80+832417.0475
135MFull Loop (original)0+120+1222416.2667
135MAsymLoop6+66+632416.1384

(1) Matched KK and effective depth.

With both iteration count (K=3K=3) and effective depth matched, AsymLoop achieves lower perplexity than the shallow Full Loop baseline at both scales: 19.46 versus 20.66 at 60M, and 16.14 versus 17.05 at 135M. These results show that AsymLoop’s advantage persists when both factors are controlled.

Rebuttal Table 2. Matched recurrence count with approximately matched parameters

ScaleModelKKEff. depthGFLOPs/tokPPL
60MFull Loop3180.360822.77
60MAsymLoop3120.292421.45
135MFull Loop3360.672216.81
135MAsymLoop3240.535416.14
350MFull Loop3722.182112.15
350MAsymLoop3481.678012.00

(2) Matched KK and approximately matched parameter counts.

With K=3K=3 and approximately matched parameter counts, AsymLoop achieves lower perplexity than Full Loop at all three scales, despite Full Loop having 50% greater effective depth. Nevertheless, AsymLoop achieves lower perplexity by 1.32, 0.67, and 0.15 at 60M, 135M, and 350M, respectively, while requiring 19.0%, 20.4%, and 23.1% fewer forward FLOPs per token. These results further support AsymLoop’s effectiveness at matched KK across model scales.

AuthorsOpenReview ↗

[W2]: Does the single-pass encoder offer advantages over alternative ways of constructing fixed Global KV?

Thank you for this suggestion. We compare AsymLoop’s Global KV construction with two alternatives, KV-Reuse and Mid-Proj. In the configurations reported below, AsymLoop achieves lower perplexity than both alternatives at 135M and 350M. In AsymLoop, the single-pass encoder output is projected into layer-specific Global KV caches. KV-Reuse instead stores the keys and values computed by each Full Loop layer during the first pass and reuses them as that same layer’s fixed Global KV in the second pass, without additional projections. Mid-Proj takes the hidden state at layer L/2L/2 during the first pass and applies L/2L/2 independent projections to construct Global KV caches. These caches remain fixed and are reused throughout all subsequent recurrent computation, matching AsymLoop in both the number of caches and the number of times each cache is reused. Both variants jointly attend to fixed Global KV and evolving Local KV using a shared softmax. We report the results at 135M and 350M below.

Rebuttal Table 3. Alternative constructions of fixed Global KV

ScaleModelEff. depthFFNParams (M)PPL ↓
135MAsymLoop K3242048141.3816.14
135MFull Loop K2242048134.3017.12
135M{\color{#b45309}\text{135M}}Full Loop + KV-Reuse{\color{#b45309}\text{Full Loop + KV-Reuse}}24{\color{#b45309}\text{24}}2048{\color{#b45309}\text{2048}}134.30{\color{#b45309}\text{134.30}}17.01{\color{#b45309}\text{17.01}}
135M{\color{#b45309}\text{135M}}Mid-Proj{\color{#b45309}\text{Mid-Proj}}24{\color{#b45309}\text{24}}2048{\color{#b45309}\text{2048}}141.38{\color{#b45309}\text{141.38}}16.63{\color{#b45309}\text{16.63}}
350MAsymLoop K3482736393.4012.00
350MFull Loop K2482736368.2312.30
350M{\color{#b45309}\text{350M}}Full Loop + KV-Reuse{\color{#b45309}\text{Full Loop + KV-Reuse}}48{\color{#b45309}\text{48}}2736{\color{#b45309}\text{2736}}368.23{\color{#b45309}\text{368.23}}12.22{\color{#b45309}\text{12.22}}
350M{\color{#b45309}\text{350M}}Mid-Proj{\color{#b45309}\text{Mid-Proj}}48{\color{#b45309}\text{48}}2736{\color{#b45309}\text{2736}}393.40{\color{#b45309}\text{393.40}}12.19{\color{#b45309}\text{12.19}}

Both alternatives improve over the plain Full Loop baseline at matched effective depth, reducing perplexity from 17.12 to 17.01 (KV-Reuse) and 16.63 (Mid-Proj) at 135M, and from 12.30 to 12.22 and 12.19 at 350M. Access to a fixed context memory is therefore beneficial in itself, and the ordering is consistent across both scales. The two constructions nevertheless remain behind AsymLoop. Mid-Proj has exactly the same parameter count (141.38M and 393.40M) and the same effective depth as AsymLoop, yet is 0.49 and 0.19 perplexity higher at 135M and 350M; KV-Reuse, which introduces no additional parameters, is 0.87 and 0.22 higher. These comparisons indicate that neither construction of the fixed Global KV matches the dedicated single-pass encoder used in AsymLoop.

Thank you again for your feedback, which is valuable in helping us improve the quality of our manuscript. Both of the weaknesses you raised are now addressed with new experiments: matched-recurrence comparisons against Full Loop under both effective-depth and parameter matching (Rebuttal Tables 1 and 2), and an ablation of two alternative constructions of fixed Global KV at 135M and 350M (Rebuttal Table 3). AsymLoop retains its advantage in every one of these new comparisons. As these were the two concerns behind your assessment, we would be grateful if you could consider whether the added evidence warrants revisiting your rating. We would also be happy to run further controls during the discussion period should any point remain open.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Experiments

Dear Reviewer,

Thank you again for your time and thoughtful review. As the discussion period draws to a close, we would appreciate your feedback on whether our rebuttal addresses your concerns. In particular, we have added comparisons at matched recurrence counts under both effective-depth and parameter matching, as well as experiments with alternative constructions of fixed Global KV. If these additions address your concerns, we would be grateful if you would consider revising your rating upward. We remain happy to discuss any remaining questions or provide further clarification.

Best regards,

The authors

AuthorsOpenReview ↗

A Kind Reminder Before the Discussion Closes

Dear Reviewer 8ZTx,

As the discussion period will close soon, we would like to kindly follow up on our rebuttal, particularly the added comparisons at matched recurrence counts and the experiments with alternative constructions of fixed Global KV. We would greatly appreciate hearing whether these additions address your concerns, or whether any questions remain. Even a brief response would be very helpful, and we remain available to provide further clarification. If your concerns have been addressed, we would be grateful if you would consider updating your rating accordingly. Thank you again for your time and thoughtful feedback.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.