- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Within this manuscript, authors proposed a method, called LatticeLoop, that shares the codes across all loops within the formulation of looped transformers and learns a separate block-scale table and tensor scale for each loop.
Critical strengths
- The central claim of this paper is that loop-specific block scales use extra storage much better than the existing relaxations. The evidence supports it.
- LatticeLoop introduces no new format, kernel or extra computation. Each loop is an ordinary block-scaled matmul that reads the same code tensor with a different scale table. The authors verify this with real FP4 kernels.
Critical weaknesses
From my opinion, the major weakness per this work is the innovation or how much inspiring towards to other researchers. As what I could recall, within related domains not far from looped transformer, there are already large mount of prior arts making use of the thoughts/ideas/tricks which makes use of the scales to dramatically reduce the trainable parameter size, especially within meta-learning and test-time-adaptation communities.
Per not finishing a literature review here, I could manage to list certain works here:
- Liu et al., NeurIPS 2024, "BitDelta: Your Fine-Tune May Only Be Worth One Bit", which compresses the model weight delta to 1-bit signs plus a learned scale.
- Lian et al., NeurIPS 2022, "Scaling & shifting your features: a new baseline for efficient model tuning", which fine-tunes by learning only scale and shift parameters.
- Mudrakarta et al., ICLR 2019, "K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning", which uses one shared model serves multiple tasks, and only the norm scale and bias of a small model patch finetuned per task.
Questions for authors
Except the parameter efficiency as well as the corresponding training design and potential advantage per less parameters coming along, I would like to see what this work fundamentally improve compared with classic looped transformer baseline, or what this work really brings or add on that pervious work is not able to handle within the community.
2 comments
Dear Reviewer CxUG,
Thank you again for your thoughtful feedback and for pointing us to prior work on scale tuning. As the discussion draws to a close, we wanted to follow up on your question about what LatticeLoop adds to a standard looped Transformer.
SSF and K for the Price of 1 use channel-wise modulation for task adaptation. We asked whether the loops within a single model could specialize while sharing the same 4-bit codes. In our analysis, the best per-row scaling explained only 29.6% of the weight variation across loops. In a separate from-scratch experiment, per-row scales closed 37.6% of the loss gap between full tying and independent weights, while scales learned per 16-weight block closed 91.7% on average across three seeds.
These block scales are already part of the NVFP4 format, so each loop still uses a single standard NVFP4 matmul. In our 80B-token runs, a 6-loop LatticeLoop model achieved higher mean accuracy than a 12-loop fully tied model (47.9% vs. 43.1%) with half as many recurrent passes.
We’d welcome your thoughts on whether this clarifies the contribution and would be happy to discuss any remaining questions.
Best regards,
The Authors
Response to Reviewer CxUG [1 / 1]
W1. "From my opinion, the major weakness per this work is the innovation or how much inspiring towards to other researchers. As what I could recall, within related domains not far from looped transformer, there are already large mount of prior arts making use of the thoughts/ideas/tricks which makes use of the scales to dramatically reduce the trainable parameter size, especially within meta-learning and test-time-adaptation communities."
A1. Thank you for these references. We will cite and discuss them in the related work. These methods and LatticeLoop both use scales, but they solve different problems under different structural constraints. BitDelta compresses the weight change of an existing fine-tuned model, SSF tunes a frozen pretrained model for a new task, and K for the Price of 1 gives each task its own small patch. LatticeLoop instead pretrains shared codes and loop-specific scales together, so that the loops of a single model specialize. Their parameterizations are also much coarser than the change that the loops of our model learn:
Q1. "Except the parameter efficiency as well as the corresponding training design and potential advantage per less parameters coming along, I would like to see what this work fundamentally improve compared with classic looped transformer baseline, or what this work really brings or add on that pervious work is not able to handle within the community."
A2. Our starting point is not parameter compression. It is a question that only block-scaled 4-bit formats raise: must the loops of a looped Transformer differ in their codes, or is it enough that they differ in their scales? The answer breaks the trade-off between quality, storage and speed that limits classic looped Transformers.
Full tying is compact and fast, but forcing every loop to apply the same linear operator caps quality. Prior relaxations cannot lift this ceiling without losing speed or sharing. Unmerged LoRA decodes 18% to 27% slower on a B200 (Table 10) and closes only 7.5% of the gap to independent weights at matched storage, merged LoRA stores a full copy per loop, and partial untying closes 30.8% (Table 1). LatticeLoop keeps the decoding speed of full tying (160.3 vs. 161.4 tok/s) and its shared storage (+4.3% payload at K=3), while closing 92.5% of the gap. At K=12 and 80B tokens, it gains 5.6 accuracy points over full tying at the same speed, and 6-loop LatticeLoop already outperforms 12-loop full tying (47.9 vs. 43.1 mean accuracy, 13.52 vs. 13.98 perplexity; Table 2) with half the recurrent compute.
Loops do not need different signs, zeros or within-block ratios. Rescaling shared 4-bit codes at a granularity of 16 weights is enough to close 92.5% of the gap (Table 1), and the change between loops is indeed block-level rather than row- or matrix-level (Section 4.5). As block-scaled 4-bit formats (NVFP4, MX) become standard on modern GPUs, LatticeLoop shows that the scales of the quantization format can serve as a modulation primitive with no compute overhead, a design option that dense formats do not offer.