acceptodds
Under review as a conference paper at ICLR 2027

LatticeLoop: Learning Loop-Specific Weights from Shared Low-Bit Codes

Abstract

Looped Transformers apply one stack of core layers several times. Full weight tying keeps the model small, but every loop must apply the same weights. Independent weights for each loop remove this restriction, but the core storage grows with the number of loops. Partial untying and per-loop LoRA add less storage, but they limit the change between loops to a few rows or to a low-rank update. Block-scaled low-bit formats such as NVFP4 already split every weight tensor into two parts: 4-bit codes set the pattern of each block of 16 weights, and 8-bit block scales set its magnitude. LatticeLoop shares the codes across loops and learns one table of block scales for each loop. Each additional loop costs 0.5 bits per core weight, one ninth of an independent copy, and every loop runs as one standard block-scaled matrix multiplication. In language model pretraining at matched storage, LatticeLoop closes 92.5% of the loss gap between full tying and independent weights, while partial untying and per-loop LoRA close less than a third of the gap. The advantage holds up to 12 loops and 80B training tokens, and decoding speed stays close to full tying.

Primary area:
foundation or frontier models, including LLMs
Keywords:
loop, nvfp4

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_CxUGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Within this manuscript, authors proposed a method, called LatticeLoop, that shares the codes across all loops within the formulation of looped transformers and learns a separate block-scale table and tensor scale for each loop.

Critical strengths
  • The central claim of this paper is that loop-specific block scales use extra storage much better than the existing relaxations. The evidence supports it.
  • LatticeLoop introduces no new format, kernel or extra computation. Each loop is an ordinary block-scaled matmul that reads the same code tensor with a different scale table. The authors verify this with real FP4 kernels.
Critical weaknesses

From my opinion, the major weakness per this work is the innovation or how much inspiring towards to other researchers. As what I could recall, within related domains not far from looped transformer, there are already large mount of prior arts making use of the thoughts/ideas/tricks which makes use of the scales to dramatically reduce the trainable parameter size, especially within meta-learning and test-time-adaptation communities.

Per not finishing a literature review here, I could manage to list certain works here:

  • Liu et al., NeurIPS 2024, "BitDelta: Your Fine-Tune May Only Be Worth One Bit", which compresses the model weight delta to 1-bit signs plus a learned scale.
  • Lian et al., NeurIPS 2022, "Scaling & shifting your features: a new baseline for efficient model tuning", which fine-tunes by learning only scale and shift parameters.
  • Mudrakarta et al., ICLR 2019, "K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning", which uses one shared model serves multiple tasks, and only the norm scale and bias of a small model patch finetuned per task.
Questions for authors

Except the parameter efficiency as well as the corresponding training design and potential advantage per less parameters coming along, I would like to see what this work fundamentally improve compared with classic looped transformer baseline, or what this work really brings or add on that pervious work is not able to handle within the community.

2 comments
AuthorsOpenReview ↗

Response to Reviewer CxUG [1 / 1]

W1. "From my opinion, the major weakness per this work is the innovation or how much inspiring towards to other researchers. As what I could recall, within related domains not far from looped transformer, there are already large mount of prior arts making use of the thoughts/ideas/tricks which makes use of the scales to dramatically reduce the trainable parameter size, especially within meta-learning and test-time-adaptation communities."

A1. Thank you for these references. We will cite and discuss them in the related work. These methods and LatticeLoop both use scales, but they solve different problems under different structural constraints. BitDelta compresses the weight change of an existing fine-tuned model, SSF tunes a frozen pretrained model for a new task, and K for the Price of 1 gives each task its own small patch. LatticeLoop instead pretrains shared codes and loop-specific scales together, so that the loops of a single model specialize. Their parameterizations are also much coarser than the change that the loops of our model learn:

  • SSF and K for the Price of 1 modulate features with one scale and one shift per channel. On the output of a linear layer, such a scale acts as one multiplier per output row of the weight matrix. In our trained model, the best per-row multipliers explain only 29.6% of the weight change across loops (Table 3), which motivates one scale for every block of 16 weights.
  • BitDelta stores a 1-bit sign matrix and one scale per matrix, α Sign(Δ)\alpha\,\mathrm{Sign}(\Delta). For each extra loop, the sign matrix costs 1.0 bit per weight, twice our 0.5 bits. At inference, it either adds a separate matmul per projection or, if merged, gives up code sharing.
  • LatticeLoop adds no new parameter type or operator to the exported model. Its loop-specific scales are the 16-weight E4M3 block scales that NVFP4 already stores and reads in every matmul.

Q1. "Except the parameter efficiency as well as the corresponding training design and potential advantage per less parameters coming along, I would like to see what this work fundamentally improve compared with classic looped transformer baseline, or what this work really brings or add on that pervious work is not able to handle within the community."

A2. Our starting point is not parameter compression. It is a question that only block-scaled 4-bit formats raise: must the loops of a looped Transformer differ in their codes, or is it enough that they differ in their scales? The answer breaks the trade-off between quality, storage and speed that limits classic looped Transformers.

Full tying is compact and fast, but forcing every loop to apply the same linear operator caps quality. Prior relaxations cannot lift this ceiling without losing speed or sharing. Unmerged LoRA decodes 18% to 27% slower on a B200 (Table 10) and closes only 7.5% of the gap to independent weights at matched storage, merged LoRA stores a full copy per loop, and partial untying closes 30.8% (Table 1). LatticeLoop keeps the decoding speed of full tying (160.3 vs. 161.4 tok/s) and its shared storage (+4.3% payload at K=3K=3), while closing 92.5% of the gap. At K=12K=12 and 80B tokens, it gains 5.6 accuracy points over full tying at the same speed, and 6-loop LatticeLoop already outperforms 12-loop full tying (47.9 vs. 43.1 mean accuracy, 13.52 vs. 13.98 perplexity; Table 2) with half the recurrent compute.

Loops do not need different signs, zeros or within-block ratios. Rescaling shared 4-bit codes at a granularity of 16 weights is enough to close 92.5% of the gap (Table 1), and the change between loops is indeed block-level rather than row- or matrix-level (Section 4.5). As block-scaled 4-bit formats (NVFP4, MX) become standard on modern GPUs, LatticeLoop shows that the scales of the quantization format can serve as a modulation primitive with no compute overhead, a design option that dense formats do not offer.

Dear Reviewer CxUG,

Thank you again for your thoughtful feedback and for pointing us to prior work on scale tuning. As the discussion draws to a close, we wanted to follow up on your question about what LatticeLoop adds to a standard looped Transformer.

SSF and K for the Price of 1 use channel-wise modulation for task adaptation. We asked whether the loops within a single model could specialize while sharing the same 4-bit codes. In our analysis, the best per-row scaling explained only 29.6% of the weight variation across loops. In a separate from-scratch experiment, per-row scales closed 37.6% of the loss gap between full tying and independent weights, while scales learned per 16-weight block closed 91.7% on average across three seeds.

These block scales are already part of the NVFP4 format, so each loop still uses a single standard NVFP4 matmul. In our 80B-token runs, a 6-loop LatticeLoop model achieved higher mean accuracy than a 12-loop fully tied model (47.9% vs. 43.1%) with half as many recurrent passes.

We’d welcome your thoughts on whether this clarifies the contribution and would be happy to discuss any remaining questions.

Best regards,

The Authors

Reviewer_p58BOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LatticeLoop targets the storage/quality trade-off in looped Transformers. In between the two extreme cases, i.e. fully tied weights and independent weights, this paper proposes to add loop-dependent scale parameters based on the NVFP4 microscaling format. During pretraining, LatticeLoop shares a single FP4 weight tensor across all looped layers but allows the FP8 block scales (and a global tensor scale) to be optimized individually for each loop. As the scale is shared for every 16 elements, the total increase in storage is one-ninth the cost of a fully independent copy, or 0.5 extra bits per weight element per added loop. This keeps inference cost the same as one ordinary NVFP4 matmul, unlike LoRA-based methods, which incur an additional low-rank matmul.

The proposed method is tested on a 360M-parameter (SmolLM-360M) looped Transformer pretrained from scratch on a 150B-token sample of Dolma 3 Mix. LatticeLoop shows comparable accuracy to independent-weight variants (Table 1) while matching the decoding throughput of the typical (full-tying) baseline (Figure 4); LatticeLoop is correspondingly faster than the unmerged LoRA variant, by a margin that grows with batch size and depth, from roughly 10% to 60%.

Critical strengths
  1. A simple approach that is compatible with modern GPU-native inference formats. By giving each loop only its own block/tensor scales on top of a shared E2M1 tensor, LatticeLoop fits directly into the NVFP4 block-scaled matmul already used for both inference and training.
  2. Real inference performance verified on hardware, not just estimated. Decode throughput is measured directly on an NVIDIA B200 GPU across batch sizes and depths (Figure 4, Table 10): LatticeLoop tracks full tying's throughput closely while outperforming unmerged LoRA by 10-60% depending on configuration — attributed concretely to LoRA's extra small-matrix kernel launches per decode step versus LatticeLoop's single block-scaled matmul.
Critical weaknesses
  1. Insufficient data. Every headline number (Tables 1, 2, 8, 9, and all speed measurements) comes from one 360M-parameter model (SmolLM-360M) on one corpus (Dolma 3 Mix), for at most 80B tokens, with a single seed and no confidence intervals anywhere — no evidence the 92.5%-gap-closed result holds at the 1B–8B+ scales where this would matter in deployment. Coverage is further thinned by uneven baseline sweeps: full tying and LoRA are only trained/reported at K=12 (Table 2), the one point where LatticeLoop has added the most loop-dependent parameters — its extra storage over full tying is largest there (5.5 bits), versus just 1.0 bit at K=3 — so the "advantage holds at every depth" claim is unverified at exactly the budgets (K=3/6/9) where LatticeLoop is granted the least extra storage relative to the cheapest baseline. Separately, the mechanistic analysis in Section 4.5 ("What the Scales Learn") uses the same K=3 architecture but a separately-trained checkpoint — trained on a different corpus (FineWeb-Edu, not Dolma) for only ~9.6B tokens, versus a minimum of 20B Dolma tokens for every model in Tables 1 and 2 — so the resulting weights never appear in the accuracy results, and the "what the scales learn" conclusions are not shown to generalize to the actual models being evaluated.
  2. Pretraining from scratch is expensive, with no discussion or test of a cheaper post-training alternative. The shared FP4 (E2M1) weights and per-loop scales are learned jointly from random initialization during pretraining (Section 3.2), rather than from an already-trained checkpoint. The paper never tests whether starting from a real pretrained tied model and only fine-tuning the per-loop scale tables on top could work instead. This would be a much cheaper post-hoc adaptation. LoRA and partial untying, by contrast, derive their practical appeal from being applicable after pretraining — leaving LatticeLoop's path to adoption on existing models untested.
  3. Evaluation skips the task category looping is claimed to help most. As noted in Related Work, a foundational motivation for looped architectures (Saunshi et al., 2025) is their advantage on reasoning tasks. That citation shows "closed book QA" has the smallest benefit from looping (34-48% of the iso-flop gap closed), versus much larger benefits on tasks requiring active reasoning over context. LatticeLoop's seven zero-shot tasks (ARC-E/C, HellaSwag, OpenBookQA, PIQA, Winogrande, SIQA, Appendix G) are all closed-form commonsense/knowledge benchmarks with no retrieved context passage — exactly the category its own motivating citation says benefits least from looping, leaving the method's reasoning-task advantage untested.
  4. (Minor) The paper's use of "code" for the shared E2M1 values is imprecise terminology. F4={0,±0.5,±1,±1.5,±2,±3,±4,±6}\mathcal{F}_4 = \{0, \pm0.5, \pm1, \pm1.5, \pm2, \pm3, \pm4, \pm6\} (Eq. 1) is the value set of the 4-bit E2M1 floating-point format — a fixed radix/exponent-mantissa representation including subnormals, not an arbitrary code (e.g., the bit pattern decoding to +6+6 cannot be reassigned to +7+7 or any value outside F4\mathcal{F}_4). LatticeLoop's only degree of freedom per loop is a multiplicative rescaling of this fixed set, never an arbitrary remapping — the use of "code" risks overstating how much flexibility is actually shared across loops.
Questions for authors

please see Weaknesses above

6 comments
AuthorsOpenReview ↗

Response to Reviewer p58B [3 / 3]

W3. "Evaluation skips the task category looping is claimed to help most. As noted in Related Work, a foundational motivation for looped architectures (Saunshi et al., 2025) is their advantage on reasoning tasks. That citation shows "closed book QA" has the smallest benefit from looping (34-48% of the iso-flop gap closed), versus much larger benefits on tasks requiring active reasoning over context. LatticeLoop's seven zero-shot tasks (ARC-E/C, HellaSwag, OpenBookQA, PIQA, Winogrande, SIQA, Appendix G) are all closed-form commonsense/knowledge benchmarks with no retrieved context passage — exactly the category its own motivating citation says benefits least from looping, leaving the method's reasoning-task advantage untested."

A5. We chose the seven benchmarks to match the zero-shot suites of the closest sub-billion and recursive models, MobileLLM and Relaxed Recursive Transformers. Following the suggestion, we evaluated the 80B-token checkpoints with lm-evaluation-harness on four benchmarks that require reading a context passage: BoolQ, RACE-Middle, RACE-High and LAMBADA.

Method (80B tokens)KKCore bitsBoolQRACE-MRACE-HLAMBADAMean (%)
Full tying124.558.435.229.841.641.3
LoRA (r=18r=18)1210.060.137.431.144.243.2
LatticeLoop35.559.636.830.943.542.7
LatticeLoop67.062.339.932.847.445.6
LatticeLoop1210.063.541.233.948.946.9

At K=12K=12, LatticeLoop is 5.6 points above full tying and 3.7 points above equal-storage LoRA on these tasks, and LatticeLoop at K=6K=6 is 2.4 points above LoRA at K=12K=12. From K=3K=3 to K=12K=12, LatticeLoop gains 4.2 points. We will add these benchmarks to Appendix G.


W4. "(Minor) The paper's use of "code" for the shared E2M1 values is imprecise terminology. F4 = {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6} (Eq. 1) is the value set of the 4-bit E2M1 floating-point format — a fixed radix/exponent-mantissa representation including subnormals, not an arbitrary code (e.g., the bit pattern decoding to +6 cannot be reassigned to +7 or any value outside F4). LatticeLoop's only degree of freedom per loop is a multiplicative rescaling of this fixed set, never an arbitrary remapping — the use of "code" risks overstating how much flexibility is actually shared across loops."

A6. Thank you for pointing this out. We use "code" in its usual scalar-quantization sense, the stored 4-bit symbol of each weight before block scaling, not a learned codebook. Section 3.1 states the restriction that the reviewer describes. Eq. 1 defines the fixed E2M1 value set F4\mathcal{F}_4, and the paragraph "What varies across loops" states that every block keeps its signs, its zero entries and the ratios between its weights in all loops, and that only one positive multiplier per block changes. We will add one sentence at the first use of "code" stating that F4\mathcal{F}_4 is the fixed E2M1 value set of the hardware format and not a learned codebook. This restriction is also the finding of the paper: a positive rescaling of shared E2M1 values, one multiplier per block of 16 weights, is enough to close most of the gap to independent weights.

AuthorsOpenReview ↗

Response to Reviewer p58B [2 / 3]

W1 (part 3). "Separately, the mechanistic analysis in Section 4.5 ("What the Scales Learn") uses the same K=3 architecture but a separately-trained checkpoint — trained on a different corpus (FineWeb-Edu, not Dolma) for only ~9.6B tokens, versus a minimum of 20B Dolma tokens for every model in Tables 1 and 2 — so the resulting weights never appear in the accuracy results, and the "what the scales learn" conclusions are not shown to generalize to the actual models being evaluated."

A3. We re-ran the weight-change fits and the scale interventions of Section 4.5 on the K=3K=3, 20B-token Dolma checkpoint of Table 1. Per-matrix, per-input-column and per-output-row multipliers explain 21.4%, 25.3% and 31.2% of the squared cross-loop weight change on Dolma, against 20.1%, 24.0% and 29.6% in Table 3. After removing the scale of the whole matrix, 67.1% of the blocks still change by more than a factor of 1.25 between loops, against 65.3% in Appendix F. Averaging the effective scales over the three loops raises the Dolma held-out loss by 0.184, 4.7 times the 0.039 loss gap between full tying and independent weights, and swapping the scale tables of loops 1 and 3 raises it by 1.62 (0.149 and 1.499 for the analysis model in Figure 6).

We also trained the coarse granularities directly on the Table 1 protocol (K=3K=3, 20B tokens, one seed each), with loop-specific scales per matrix, per input column and per output row learned jointly with the shared codes. They give the same ranking:

Loop-specific scaleGap closed (%)
Per matrix (tensor)22.4
Per input column33.1
Per output row37.6
Per 16-weight block (LatticeLoop, 3 seeds)91.7 ± 0.4

We will report the analysis of the Dolma checkpoint in Section 4.5, so that the analysis and the accuracy results use the same models.


W2. "Pretraining from scratch is expensive, with no discussion or test of a cheaper post-training alternative. The shared FP4 (E2M1) weights and per-loop scales are learned jointly from random initialization during pretraining (Section 3.2), rather than from an already-trained checkpoint. The paper never tests whether starting from a real pretrained tied model and only fine-tuning the per-loop scale tables on top could work instead. This would be a much cheaper post-hoc adaptation. LoRA and partial untying, by contrast, derive their practical appeal from being applicable after pretraining — leaving LatticeLoop's path to adoption on existing models untested."

A4. LatticeLoop can start from a pretrained tied checkpoint without changing its outputs. With all log gains A=0A=0, αk,b=βb\alpha_{k,b}=\beta_b in every loop, which is the fully tied model (Section 3.2). LoRA and partial untying also need further training to make the loops differ when they start from a tied model.

We tested this directly. We loaded the 20B-token full-tying checkpoint of Table 1, froze its shared E2M1 codes, and trained only the loop-specific scale tables for 2B more Dolma tokens, 10% of the pretraining budget. For comparison, we trained equal-storage LoRA (r=18r=18) on the same frozen checkpoint for 2B tokens.

Setting (K=3K=3, 5.5 core bits)Extra tokensHeld-out PPLGap closed (%)
Frozen tied model + LoRA (r=18r=18)2B15.6918.2
Frozen tied codes + LatticeLoop scales2B15.3867.3
LatticeLoop, joint pretraining (Table 1)0 (20B total)15.2392.5

Training only the scales for 2B tokens closes 67.3% of the gap, against 18.2% for LoRA with the same tokens and storage. Joint pretraining closes 92.5% (91.7% ± 0.4% over three seeds), since the shared codes can then adapt to the per-loop scales. On pretraining cost: in Tables 1 and 2 all sharing rules are pretrained from scratch with the same token budget, and LatticeLoop adds no matrix multiplication over full tying. We will add this experiment to the revised paper.

AuthorsOpenReview ↗

Response to Reviewer p58B [1 / 3]

W1 (part 1). "Insufficient data. Every headline number (Tables 1, 2, 8, 9, and all speed measurements) comes from one 360M-parameter model (SmolLM-360M) on one corpus (Dolma 3 Mix), for at most 80B tokens, with a single seed and no confidence intervals anywhere — no evidence the 92.5%-gap-closed result holds at the 1B–8B+ scales where this would matter in deployment."

A1. On seeds: every configuration in Table 1 was trained with three seeds. The submitted Table 1 reports the best run of each method. Over all three seeds (mean ± sample standard deviation; gap closed is computed per seed and then averaged):

MethodHeld-out lossGap closed (%)
Full tying2.761 ± 0.0010.0
Partial untying2.750 ± 0.00227.8 ± 5.9
LoRA (r=18r=18)2.758 ± 0.0016.9 ± 2.8
LatticeLoop2.725 ± 0.00291.7 ± 0.4
Independent weights2.722 ± 0.002100.0

The standard deviation of the held-out loss is at most 0.002, and LatticeLoop is below both baselines on every seed. At K=12K=12 and 20B tokens, the three-seed perplexity is 14.22 ± 0.02 for LatticeLoop, 15.13 ± 0.02 for full tying and 14.79 ± 0.02 for LoRA. The 80B runs use one seed because of their compute cost; their margins over full tying and LoRA are 0.90 and 0.48 perplexity.

On scale: our width and head configuration matches SmolLM-360M and MobileLLM-350M. MobileLLM focuses on layer sharing at 125M and 350M parameters, a standard scale for from-scratch weight-sharing studies where memory footprint is a key constraint. Larger width does not change the comparison with LoRA in LoRA's favor. At the same budget of 0.5 extra bits per weight per loop, the rank of a BF16 LoRA update is r=doutdin32(dout+din)r = \frac{d_{\text{out}}d_{\text{in}}}{32(d_{\text{out}}+d_{\text{in}})}, so r/dr/d stays fixed as the width grows (1/641/64 for square matrices). The number of block scales grows as doutdin/16d_{\text{out}}d_{\text{in}}/16, and the difference between two loops can have full rank at every width. Along the two axes we do scale, depth and tokens, the margin over full tying does not shrink: at K=12K=12, the perplexity margin is 0.91 at 20B tokens and 0.90 at 80B, and the accuracy margin goes from 4.5 to 5.6 points (Table 2). Pretraining all five sharing rules from scratch at 1B–8B parameters is beyond our compute budget for the rebuttal.


W1 (part 2). "Coverage is further thinned by uneven baseline sweeps: full tying and LoRA are only trained/reported at K=12 (Table 2), the one point where LatticeLoop has added the most loop-dependent parameters — its extra storage over full tying is largest there (5.5 bits), versus just 1.0 bit at K=3 — so the "advantage holds at every depth" claim is unverified at exactly the budgets (K=3/6/9) where LatticeLoop is granted the least extra storage relative to the cheapest baseline."

A2. The point with the least extra storage, K=3K=3 with 1.0 extra bit per core weight over full tying, is evaluated for all five sharing rules in Table 1. At this budget, LatticeLoop reaches 15.23 perplexity and 40.5% accuracy, against 15.75 and 37.1% for equal-storage LoRA, and over three seeds it closes 91.7% ± 0.4% of the gap, against 6.9% ± 2.8% for LoRA.

To complete the depth sweep, we trained full tying and equal-storage LoRA (r=18r=18) at K=6K=6 and K=9K=9 for 20B tokens on Dolma:

Loops KKCore bits (Full / LoRA and LatticeLoop)Full tying PPL / Avg. (%)LoRA (r=18r=18) PPL / Avg. (%)LatticeLoop PPL / Avg. (%)
34.5 / 5.515.80 / 36.815.75 / 37.115.23 / 40.5
64.5 / 7.015.41 / 38.415.19 / 39.814.62 / 43.0
94.5 / 8.515.23 / 39.214.94 / 41.014.35 / 43.8
124.5 / 10.015.12 / 39.714.78 / 41.814.21 / 44.2

At every loop count, LatticeLoop has lower perplexity and higher accuracy than both baselines. We will include all four loop counts for all three methods in the revised Table 2.

Reviewer_H6ALOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces LatticeLoop: a looped transformer that tries to find the sweet spot between fully tied and independent weights. The core idea is using NVFP4's block scales to specialize weights across loops while sharing the 4-bit codes. Adding just 0.5 bits per weight per loop to get full-matrix variation solves the initial trade-off.

The authors also share an empirical study showing that LatticeLoop consistently outperforms equal-budget LoRA and partial untying without adding any inference latency.

Critical strengths

The paper is well written, the figure are very informative. The hardware-native design is a relevant systems insight: each loop runs as a standard block-scaled FP4 matmul that simply reads a different scale table, so decoding matches full-tying speed without the extra matmuls of unmerged LoRA. The storage accounting is careful, and at matched storage LatticeLoop clearly outperforms LoRA and partial untying, with gains that persist up to 12 loops and 80B tokens.

Critical weaknesses

The paper is clean, but I would point out a few small weaknesses. The claim that block granularity matters rests on post-hoc fits of the trained model (Table 3), not on trained ablations with per-loop tensor, row or column scales. Also, the results looks single-seed on a small loss gap and the LoRA baseline may be under-tuned.

Questions for authors

How many seeds were used? Please report the variance of held-out loss for Table 1. How were the LoRA and partial-untying baselines tuned?

1 comment
AuthorsOpenReview ↗

Response to Reviewer H6AL [1 / 1]

W1. "The claim that block granularity matters rests on post-hoc fits of the trained model (Table 3), not on trained ablations with per-loop tensor, row or column scales."

A1. We trained these ablations. The variants learn one loop-specific scale per matrix, per output row or per input column, jointly with the shared codes from scratch, with K=3K=3, 20B tokens and the protocol of Table 1. Each variant was trained with one seed due to the limited rebuttal period.

Loop-specific scaleGap closed (%)
Per matrix (tensor)22.4
Per input column33.1
Per output row37.6
Per 16-weight block (LatticeLoop, 3 seeds)91.7 ± 0.4

Table 3 measures a different quantity, the fraction of the weight change of the analysis model that a coarse fit explains, so its numbers are not directly comparable with gap closed. The ranking is the same in both: per row is the best coarse granularity, followed by per column and per matrix. The best trained coarse variant, per row, closes 37.6% of the gap, against 91.7% for block scales. We will add these ablations to the paper.


W2 & Q1. "Also, the results looks single-seed on a small loss gap" / "How many seeds were used? Please report the variance of held-out loss for Table 1."

A2. Each configuration of Table 1 was trained with three seeds. The submitted Table 1 reports the best run of each method, and its gap closed (92.5% for LatticeLoop) is computed from these best runs. Over all three seeds, the results are:

MethodHeld-out lossGap closed (%)
Full tying2.761 ± 0.0010.0
Partial untying2.750 ± 0.00227.8 ± 5.9
LoRA (r=18r=18)2.758 ± 0.0016.9 ± 2.8
LatticeLoop2.725 ± 0.00291.7 ± 0.4
Independent weights2.722 ± 0.002100.0

Values are mean ± sample standard deviation over three seeds. Gap closed is computed for each seed against the full-tying and independent-weight models of that seed, then averaged.

The standard deviation of the held-out loss is at most 0.002. On average, LatticeLoop is 0.025 below the best equal-storage baseline, and on every seed it is below both baselines. Per seed, LatticeLoop closes 91.2% to 92.0% of the gap, partial untying 22.6% to 34.2%, and LoRA 3.8% to 9.1%. At K=12K=12, LatticeLoop is also below equal-storage LoRA in perplexity under both token budgets (14.21 vs. 14.78 at 20B, 13.08 vs. 13.56 at 80B; Table 2). We will report the three-seed mean ± std in the revised Table 1.


W2 & Q2. "the LoRA baseline may be under-tuned" / "How were the LoRA and partial-untying baselines tuned?"

A3. We swept the peak learning rate of LoRA and of partial untying over {5×10−4, 1×10−3, 3×10−3, 1×10−2}\{5\times10^{-4},\,1\times10^{-3},\,3\times10^{-3},\,1\times10^{-2}\} at K=3K=3 and 20B tokens. For both, 3×10−33\times10^{-3} was best, which is the peak learning rate used for all methods in Table 1. All other settings are shared across the five sharing rules: optimizer, schedule, weight decay, clipping and token budget (Appendix A). LoRA rank 18 and the untied row fraction 1/9 are set by giving every relaxation the same extra storage as LatticeLoop (Section 3.3). Since all rows are symmetric at random initialization, the choice of untied rows is not a hyperparameter. All three relaxations start from the same tied function: the LoRA update starts at zero, and untied rows start as copies of the shared rows (Appendix B).

With the larger storage at K=12K=12, the same LoRA setup improves full tying from 15.12 to 14.78 perplexity at 20B tokens and from 13.98 to 13.56 at 80B (Table 2). For reference, in Table G.2 of Relaxed Recursive Transformers, rank-64 LoRA at width 2048 improves the best tied model by at most 0.7 accuracy points on Gemma, TinyLlama and Pythia. Our rank 18 at width 960 is a smaller fraction of the width.

Reviewer_mWSLOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a principled exploration of using low-bit weight adaptation of a shared weights in a looped Transformer. They contrast this to other approaches of adapting weights (e.g. LoRA) for loop Transformers and motivate why they try it: minimal overhead in both storage and compute. They adapt best practices around low-bit block quantization and adaptation to build a recipe for training the model. In their evaluation, they show rigorous results and ablations justifying the use of the approach over others and the accuracy relative to the fully shared Looped Transformer baseline and the full untied baseline.

Critical strengths

The paper tackles a single modification on the Looped Transformer using existing techniques to make a well quantified improvement.

Critical weaknesses

No significant weaknesses. The paper maps a clear, if marginal, improvement.

Questions for authors

No questions

1 comment
AuthorsOpenReview ↗

Response to Reviewer mWSL [1 / 1]

Thank you very much for your positive review and for recognizing the clarity and rigor of our study and ablations. We are glad the design and trade-offs came across clearly. Just to share one practical takeaway we found encouraging: as depth scales (K=12K=12, 80B tokens), the shared-code design translates into a +5.6 accuracy-point gain over full tying (+3.4 over equal-storage LoRA) at full-tying decode speed, and allows a 6-loop model to match and exceed a 12-loop fully tied model with half the recurrent passes. Thank you again for your time and supportive evaluation!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.