- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces AsymLoop, dividing the model into a causal encoder executed once and a weight-tied decoder repeated K times. The encoder output is mapped into layer-specific Global KV caches that remain constant across iterations, while each decoder layer attends to both these Global KV and the Local KV from its current state using a single softmax. For training, the authors employ a previously established heavy-tail-guided layerwise learning-rate method, referred to as ALRS. The authors pre-train LLaMA-style models ranging from 60M to 1B parameters and report lower validation perplexity and higher average zero-shot accuracy compared to Dense and Full Loop at matched effective depth. Additional improvements are reported with extended training budgets, an OLMo-style backbone, sparse MoE layers, GSM8K fine-tuning, and a FLOPs-matched wider Full Loop.
Critical strengths
- The design is straightforward and well-articulated. Integrating a single-pass encoder with a looped decoder via a fixed Global KV is straightforward to implement, and the additional parameters and FLOPs are explicitly analyzed.
- The experimental scope is comprehensive. Models are trained from scratch at four different scales up to 1B parameters, and the evaluation encompasses downstream tasks, a 5x Chinchilla budget, recurrent depths from K = 1 to 5, an alternative backbone, MoE layers, and a compute-matched baseline.
Critical weaknesses
- The contribution of the architecture is not clearly distinguished from the impact of the learning-rate method. The paper does not clarify whether Dense and Full Loop models in the main results are also trained with ALRS, and only AsymLoop is labeled with ALRS in Table 2 and Appendices E and G. The 60M results in Table 3 display ratios similar to those of the uniform-LR baselines compared to ALRS AsymLoop in Tables 4 and 8. The paper's ablation studies indicate that ALRS alone reduces Full Loop perplexity by approximately 0.8 (Table 4c), and a Dense model with layerwise learning rates already surpasses the uniform-LR Full Loop at both K = 2 and K = 3 (Table 8). Therefore, a significant portion of the reported improvements may be attributable to the optimizer rather than the asymmetric design.
- The evidence supporting the claim that performance gains arise from recurrence or role separation is limited. AsymLoop with K = 1, which lacks recurrence, already outperforms Full Loop with K = 2 and K = 3 at 60M and 135M parameters (Table 3), and additional iterations yield only marginal improvements. In Table 7, both the encoder–decoder split without Global KV and the additive injection variant perform worse than Full Loop, suggesting that the primary benefit may result from the additional Global KV attention path derived from CED. Furthermore, the advantage over Full Loop decreases from approximately 7% lower perplexity at 60M to about 0.5% at 1B in a single run.
- Results for identical configurations are inconsistent across tables. For example, at 60M, AsymLoop with K = 3 reports a perplexity of 21.45 in Table 3, 23.75 in Table 4, and approximately 19.5 based on the cross-entropy in Table 10. Uniform-LR Full Loop with K = 2 is reported as 25.57 in Tables 4c and 8, but 25.05 in Table 7. The evaluation protocols underlying these results are not specified. Additionally, Figure 6 indicates 59.6K training steps for the 1B models, which equates to approximately 31B tokens, rather than the stated 20B.
Questions for authors
Most of my concerns are included in the weakness section. Specifically, I would like to ask the authors:
- Are the Dense and Full Loop baselines in the main results and Appendices E, G, and I trained with ALRS or with uniform learning rates? If the latter, how do the comparisons change when all models employ the same learning-rate method?
- Given that AsymLoop with K = 1 already outperforms Full Loop with K = 2 and K = 3 at 60M and 135M, could the authors clarify the extent to which the observed gains are attributable to recurrence as opposed to the Global KV path?
- Why do identical 60M configurations yield different perplexities in Tables 3, 4, 7, and 10, and which evaluation protocol is employed in each table?
- What is the actual training budget for the 1B models, considering that 59.6K steps in Figure 6 correspond to approximately 31B tokens with a global batch size of 512 and a sequence length of 1024?
- Are the 1B results based on a single seed? Given that the perplexity gap at 1B is only 0.05, what is the seed variance at this scale?
7 comments
[W2 & Q2]: Given AsymLoop’s advantage at , how much of the gain comes from recurrence rather than the Global KV path?
Thank you for raising this question about the source of AsymLoop’s gains. To distinguish the contributions of recurrence and the Global KV path, we conduct a 2×2 ablation at 135M, varying and the presence of Global KV. All variants share the same 6+6 encoder–decoder split, QKNorm, ALRS, seed, global batch size, data budget, and rank count.
Rebuttal Table 2. Recurrence and Global KV ablation at 135M
| Model variant | Parameters | CE ↓, | CE ↓, |
|---|---|---|---|
| Without Global KV | 134.30M | 2.83885 | 2.81738 |
| With Global KV | 141.38M | 2.80664 | 2.78120 |
Increasing from 1 to 3 reduces validation CE by 0.0215 without Global KV and by 0.0254 with Global KV, without adding parameters in either case. Adding the Global KV path reduces validation CE by 0.0322 at and by 0.0362 at . These results support complementary contributions: recurrence provides additional improvement beyond the non-recurrent Global KV configuration, while Global KV increases the benefit of recurrent computation. Their combination achieves the lowest CE under the same training data budget.
[W3 & Q3]: Why do identical 60M configurations yield different perplexities across Tables 3, 4, 7, and 10 of the submitted paper, and which evaluation protocol does each use?
We thank the reviewer for pointing out that our experimental settings were insufficiently described.
(1) Unified evaluation settings and the protocol for each table.
In the original manuscript, some experiments in the main text and appendix used different training-token accounting methods and validation subsets, resulting in different reported values for the same configuration. To resolve this ambiguity, we have re-evaluated all experiments under unified evaluation settings and matched training budgets, and updated the relevant tables in both the main text and appendix so that corresponding results use consistent settings. All changes are marked in blue. We also document the experimental settings for each table in Appendix J, including training budgets and token accounting, validation subsets, and evaluation protocols.
(2) Why identical 60M configurations differ across tables.
The numerical discrepancies in the original manuscript mainly arose from batch counting during evaluation and training-token budgets. Table 3 of the submitted paper computed validation loss by averaging over batches, using different batch-indexing and counting conventions from Table 4(c). Table 10 used a training budget of 1.6B valid tokens, excluding padding, which differed from the actual training budget in Tables 3 and 4(c). All comparisons in the paper were conducted within individual tables under shared training and evaluation settings; no conclusions relied on comparing absolute perplexities across tables. These cross-table differences therefore do not affect the conclusions supported by the comparisons within each experiment.
[W1 & Q1]: Do the main and appendix baselines use ALRS, and how do comparisons change when all models use the same learning-rate method?
We sincerely thank you for your time and insightful comments. We address your concerns point-by-point below.
Thank you for raising this concern.
(1) Learning-rate method used by each baseline.
In the original main experiments, Dense and Full Loop use uniform learning rates, while AsymLoop uses ALRS; all models within each scale share the same base learning rate. To assess whether AsymLoop’s advantage persists when the baselines also use ALRS, we conduct six additional Dense and Full Loop runs at 135M and 350M, and train the 1B Dense baseline with ALRS using three seeds. These runs retain the corresponding original architectures and training-token budgets. We report the comparisons below.
(2) Comparisons with ALRS applied to the baselines at 135M and 350M.
We completed six additional Dense and Full Loop runs with ALRS at 135M and 350M, retaining the original architectures, training-token budgets, and effective-depth pairings. As shown in Rebuttal Table 1, AsymLoop achieves lower perplexity in all six comparisons. These results show that the gap persists when ALRS is applied to the baselines.
Rebuttal Table 1. Main baselines with and without ALRS