- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates why standard post-training quantization (PTQ) can fail on looped transformers and identifies two key failure modes: feedback exposure and calibration blindness. First, the authors show that quantization errors can be particularly damaging when introduced into recurrent state transitions without an identity bypass, as demonstrated by the severe degradation from quantizing Huginn's non-residual adapter compared to its residual core. Second, they find that standard GPTQ calibration using only first-step activations fails to capture the changing hidden-state distributions across recurrence steps, sometimes performing worse than simple round-to-nearest quantization. To address this, the authors propose trajectory calibration, which accumulates GPTQ Hessians across recurrence steps. Experiments on nine checkpoints spanning seven looped transformer architectures show consistent improvements, including recovering full-precision-level accuracy on Huginn.
Critical strengths
- Timely and relevant research topic with extensive empirical analysis. Quantization remains relatively underexplored for looped transformers, making this a timely research direction. The paper presents extensive experiments and detailed analyses to investigate the underlying failure modes, which could provide valuable insights for the community.
Critical weaknesses
I am not an expert in quantization but I tried my best to familiarize myself with recent related work while reviewing this paper. I have a few concerns while reading:
-
Limited generality of the "Feedback Exposure" analysis. It seems like "Feedback Exposure" problem you studied only applies models with specific structures (Huginn and R-Llama), because it focuses on the "Adaptor" structure only. However, other loop transformers like Ouro doesn't have any adaptor structure, which means that your analysis is not focusing on general loop transformers. This seems to be an overclaiming issue, considering that your title is "Quantizing Looped Transformers" but half of the paper only focus on a specific loop model.
-
I think the proposed method is still incremental. Regarding the "Feedback Exposure" part, I think the finding that adapters are vulnerable in loop transformer is interesting. However, the fact that errors can propagate through recurrent loops is not new, so the main contribution here is primarily empirical evidence on the importance of adaptors. As for the "Calibration Blindness" part, the paper itself acknowledges that the prior work "Hyperloop Transformers" already explicitly employed GPTQ with Hessian estimation based on activations accumulated across all recurrent loops, using the same INT4 quantization and group size of 128. Therefore, the Calibration Blindness part does not offer methodological novelty.
Questions for authors
See weaknesses.
2 comments
We thank the reviewer for the follow-up.
The paper is titled "Quantizing Looped Transformers," yet a substantial portion of its analysis focuses on a failure mode that appears specific to certain architectural designs, rather than broadly applicable to looped transformers. In particular, the authors have not established how this mechanism applies to other looped transformer architectures such as Ouro.
Feedback exposure is one of two distinct failure modes we identify, and its condition also applies across designs. A layer is feedback-exposed when it perturbs the recurrent state without an identity path around it, and the resulting error is fed back at later steps (Section 3). In Ouro's loop, every quantized projection sits inside an attention or MLP block with an identity path. We therefore expect a smaller feedback effect from the rounding error of any one Ouro layer than from a comparable error on the recurrent state.
We tested this prediction with two new experiments on Ouro-1.4B and Ouro-2.6B. In the first experiment, we inserted a linear map either on the state entering Ouro's first layer or at the start of that layer's attention block. The map applies a random rotation rounded to per-channel INT4 and then the inverse of the unrounded rotation, so it equals the identity plus INT4 rounding error. At the layer input, the map changes the whole recurrent state, as Huginn's adapter does. When the map is inside the attention block, the identity path carries the original state unchanged. We also quantized each Ouro layer alone with per-channel INT4 on all seven of its projections. The map's 14% relative weight error is below that of each of these projections.
The first recurrence step receives token embeddings and no fed-back state. On 50 GSM8K training prompts, we therefore compared two rollouts with the same weights and the same bf16 state entering the second recurrence step. In one rollout, each later step received the previous step's output. The other followed the λ = 0 control of Section 3.1 and was reset to the bf16 state at every later step. The table shows how much final-step top-1 agreement with bf16 these resets restore.
| Final-step agreement lost to feedback (points) | Ouro-1.4B | Ouro-2.6B |
|---|---|---|
| Inserted map on the recurrent state | 14.1 | 30.8 |
| Inserted map inside the attention block | 2.4 | 1.6 |
| Worst Ouro layer, quantized alone | 4.8 | 2.4 |
Even the worst Ouro layer quantized alone is much closer to the attention-block result than to the whole-state result.
We then tested Ouro's own RMSNorm at the end of each recurrence step because it is the only component that changes the recurrent state without an identity path. The norm stays in bf16 in the paper's INT4 runs, so we added Gaussian noise to its weights and compared it with a norm inside the last layer's MLP block. On training prompts we scaled the noise so that perturbing either norm gave a mean relative L2 state error of 0.5%, 1% or 2% after one step from a bf16 state.
Using the same bf16 resets, we found that perturbing the end-of-step norm adds more error to the final state through feedback than perturbing the MLP norm. The mean per-prompt ratio of the two added errors ranged from 1.4 to 2.9 across models and noise levels. In each setting, the gap held on all 50 GSM8K test prompts in each of five random noise directions, and the end-of-step norm also lost more final-step agreement with bf16 on average. Perturbing the end-of-step norm thus reveals the same kind of feedback exposure that Section 3 found in Huginn's adapter.
The additional comparisons and analyses help explain the effectiveness of this approach, but do not establish methodological novelty.
We view the diagnosis of the two failure modes as the main contribution. Feedback exposure occurs even with RTN, which uses no calibration data (Section 3). Calibration blindness explains why one-step GPTQ can be worse than RTN on looped models (Section 4). To our knowledge, neither Hyperloop nor any other earlier work on looped transformers tests the two failure modes separately or measures which directions a step-0 Hessian misses.
We will add both Ouro experiments to Appendix E and summarize them in Section 3 in the final version.
Thank you for the detailed response and clarifications!
Regarding W1, I acknowledge that feedback exposure is not limited to Huginn-style adapters, and the additional experiments help demonstrate its applicability to broader recurrent systems. However, my main concern about overclaiming remains. The paper is titled "Quantizing Looped Transformers," yet a substantial portion of its analysis focuses on a failure mode that appears specific to certain architectural designs, rather than broadly applicable to looped transformers. In particular, the authors have not established how this mechanism applies to other looped transformer architectures such as Ouro. I appreciate the planned clarification of scope, but I still think the paper's framing is broader than what its evidence supports.
Regarding W2, I acknowledge that the controlled experiments and Hessian subspace analyses provide useful empirical insights beyond simply observing error propagation. However, my main concern about novelty remains. In particular, the authors confirm that Hyperloop Transformers already uses GPTQ with Hessians accumulated across recurrence steps. The additional comparisons and analyses help explain the effectiveness of this approach, but do not establish methodological novelty.
Overall, I appreciate the extensive empirical analysis, but I remain concerned about the paper's generality and novelty. I will therefore maintain my original rating of 2.