acceptodds
Under review as a conference paper at ICLR 2027

Quantizing Looped Transformers: Feedback Exposure and Calibration Blindness

Abstract

Looped transformers reuse weights across recurrence steps, making low-bit quantization especially attractive. We identify two distinct failure modes of standard post-training quantization. On Huginn-3.5B, per-channel INT4 fails primarily at the non-residual loop-entry adapter, while quantizing the residual core is much less damaging. We call this *feedback exposure*: a quantized layer perturbs the recurrent state without an identity path, and the resulting error is fed back at later steps. Controlled experiments on linear filters and Mamba state-space models show that feedback exposure also occurs outside transformers. Grouped INT4 reveals a separate failure, *calibration blindness*: our one-step GPTQ baseline builds its Hessian from step-0 activations, leaving input directions used later in the recurrence nearly unweighted. Across nine checkpoints from seven looped architectures, one-step GPTQ is worse than round-to-nearest (RTN) on the primary task metric for five checkpoints. Accumulating the GPTQ Hessian across recurrence steps outperforms both one-step GPTQ and RTN on all nine checkpoints and recovers bf16-level accuracy on Huginn. These results separate two questions for PTQ on looped models: where quantization error enters the recurrence, and which states calibration sees.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped transformers, post-training quantization, weight-only quantization, GPTQ, recurrent inference, low-bit inference, model compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GCLaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why standard post-training quantization (PTQ) can fail on looped transformers and identifies two key failure modes: feedback exposure and calibration blindness. First, the authors show that quantization errors can be particularly damaging when introduced into recurrent state transitions without an identity bypass, as demonstrated by the severe degradation from quantizing Huginn's non-residual adapter compared to its residual core. Second, they find that standard GPTQ calibration using only first-step activations fails to capture the changing hidden-state distributions across recurrence steps, sometimes performing worse than simple round-to-nearest quantization. To address this, the authors propose trajectory calibration, which accumulates GPTQ Hessians across recurrence steps. Experiments on nine checkpoints spanning seven looped transformer architectures show consistent improvements, including recovering full-precision-level accuracy on Huginn.

Critical strengths
  1. Timely and relevant research topic with extensive empirical analysis. Quantization remains relatively underexplored for looped transformers, making this a timely research direction. The paper presents extensive experiments and detailed analyses to investigate the underlying failure modes, which could provide valuable insights for the community.
Critical weaknesses

I am not an expert in quantization but I tried my best to familiarize myself with recent related work while reviewing this paper. I have a few concerns while reading:

  1. Limited generality of the "Feedback Exposure" analysis. It seems like "Feedback Exposure" problem you studied only applies models with specific structures (Huginn and R-Llama), because it focuses on the "Adaptor" structure only. However, other loop transformers like Ouro doesn't have any adaptor structure, which means that your analysis is not focusing on general loop transformers. This seems to be an overclaiming issue, considering that your title is "Quantizing Looped Transformers" but half of the paper only focus on a specific loop model.

  2. I think the proposed method is still incremental. Regarding the "Feedback Exposure" part, I think the finding that adapters are vulnerable in loop transformer is interesting. However, the fact that errors can propagate through recurrent loops is not new, so the main contribution here is primarily empirical evidence on the importance of adaptors. As for the "Calibration Blindness" part, the paper itself acknowledges that the prior work "Hyperloop Transformers" already explicitly employed GPTQ with Hessian estimation based on activations accumulated across all recurrent loops, using the same INT4 quantization and group size of 128. Therefore, the Calibration Blindness part does not offer methodological novelty.

Questions for authors

See weaknesses.

2 comments

Thank you for the detailed response and clarifications!

Regarding W1, I acknowledge that feedback exposure is not limited to Huginn-style adapters, and the additional experiments help demonstrate its applicability to broader recurrent systems. However, my main concern about overclaiming remains. The paper is titled "Quantizing Looped Transformers," yet a substantial portion of its analysis focuses on a failure mode that appears specific to certain architectural designs, rather than broadly applicable to looped transformers. In particular, the authors have not established how this mechanism applies to other looped transformer architectures such as Ouro. I appreciate the planned clarification of scope, but I still think the paper's framing is broader than what its evidence supports.

Regarding W2, I acknowledge that the controlled experiments and Hessian subspace analyses provide useful empirical insights beyond simply observing error propagation. However, my main concern about novelty remains. In particular, the authors confirm that Hyperloop Transformers already uses GPTQ with Hessians accumulated across recurrence steps. The additional comparisons and analyses help explain the effectiveness of this approach, but do not establish methodological novelty.

Overall, I appreciate the extensive empirical analysis, but I remain concerned about the paper's generality and novelty. I will therefore maintain my original rating of 2.

We thank the reviewer for the follow-up.

The paper is titled "Quantizing Looped Transformers," yet a substantial portion of its analysis focuses on a failure mode that appears specific to certain architectural designs, rather than broadly applicable to looped transformers. In particular, the authors have not established how this mechanism applies to other looped transformer architectures such as Ouro.

Feedback exposure is one of two distinct failure modes we identify, and its condition also applies across designs. A layer is feedback-exposed when it perturbs the recurrent state without an identity path around it, and the resulting error is fed back at later steps (Section 3). In Ouro's loop, every quantized projection sits inside an attention or MLP block with an identity path. We therefore expect a smaller feedback effect from the rounding error of any one Ouro layer than from a comparable error on the recurrent state.

We tested this prediction with two new experiments on Ouro-1.4B and Ouro-2.6B. In the first experiment, we inserted a linear map either on the state entering Ouro's first layer or at the start of that layer's attention block. The map applies a random rotation rounded to per-channel INT4 and then the inverse of the unrounded rotation, so it equals the identity plus INT4 rounding error. At the layer input, the map changes the whole recurrent state, as Huginn's adapter does. When the map is inside the attention block, the identity path carries the original state unchanged. We also quantized each Ouro layer alone with per-channel INT4 on all seven of its projections. The map's 14% relative weight error is below that of each of these projections.

The first recurrence step receives token embeddings and no fed-back state. On 50 GSM8K training prompts, we therefore compared two rollouts with the same weights and the same bf16 state entering the second recurrence step. In one rollout, each later step received the previous step's output. The other followed the λ = 0 control of Section 3.1 and was reset to the bf16 state at every later step. The table shows how much final-step top-1 agreement with bf16 these resets restore.

Final-step agreement lost to feedback (points)Ouro-1.4BOuro-2.6B
Inserted map on the recurrent state14.130.8
Inserted map inside the attention block2.41.6
Worst Ouro layer, quantized alone4.82.4

Even the worst Ouro layer quantized alone is much closer to the attention-block result than to the whole-state result.

We then tested Ouro's own RMSNorm at the end of each recurrence step because it is the only component that changes the recurrent state without an identity path. The norm stays in bf16 in the paper's INT4 runs, so we added Gaussian noise to its weights and compared it with a norm inside the last layer's MLP block. On training prompts we scaled the noise so that perturbing either norm gave a mean relative L2 state error of 0.5%, 1% or 2% after one step from a bf16 state.

Using the same bf16 resets, we found that perturbing the end-of-step norm adds more error to the final state through feedback than perturbing the MLP norm. The mean per-prompt ratio of the two added errors ranged from 1.4 to 2.9 across models and noise levels. In each setting, the gap held on all 50 GSM8K test prompts in each of five random noise directions, and the end-of-step norm also lost more final-step agreement with bf16 on average. Perturbing the end-of-step norm thus reveals the same kind of feedback exposure that Section 3 found in Huginn's adapter.

The additional comparisons and analyses help explain the effectiveness of this approach, but do not establish methodological novelty.

We view the diagnosis of the two failure modes as the main contribution. Feedback exposure occurs even with RTN, which uses no calibration data (Section 3). Calibration blindness explains why one-step GPTQ can be worse than RTN on looped models (Section 4). To our knowledge, neither Hyperloop nor any other earlier work on looped transformers tests the two failure modes separately or measures which directions a step-0 Hessian misses.

We will add both Ouro experiments to Appendix E and summarize them in Section 3 in the final version.

Reviewer_nT9BOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies PTQ in looped Transformers, which reuse weights over multiple recurrence steps. It separates feedback exposure, in which quantization errors return through the recurrent state, from calibration blindness, in which first-step calibration misses later states. Controlled perturbations, Jacobian analysis, and Hessian measurements support the study. Trajectory-based GPTQ calibration beats first-step GPTQ and round-to-nearest (RTN) on nine checkpoints from seven architectures.

Critical strengths
  • The feedback intervention is convincing. Quantizing only Huginn's adapter reduces final top-1 agreement to 10.5%, while quantizing 16 residual-core projections retains 87.9% (Figure 1 and Table 3). Removing error from the prior state raises agreement from 10.4% to 69.2% in Section 3.1. This directly tests the role of recirculated error.

  • The equal-row INT3 control addresses a simple alternative explanation. In Table 10, trajectory calibration improves strict GSM8K accuracy from 5.84% to 17.44% with the same activation-row budget. Its advantage is therefore not explained only by collecting more rows.

Table 2 also covers nine checkpoints from seven architectures, with gains over one-step GPTQ and RTN in every case. The real-kernel test adds practical evidence. The paper is well organized, the notation is mostly consistent, and the distinction between feedback and calibration helps explain the results. Its main value is as a diagnostic and empirical study.

Critical weaknesses

1. The calibration code and the stated data split do not match. This is the most important issue for my recommendation. Section 6 and Appendix Q, Table 25 say Huginn and R-Llama use the first 50 GSM8K training questions for calibration and the separate 1,319-question test set for evaluation. In the supplied trajectory_gptq_huginn.py, however, lines 2057–2060 construct calibration, validation, and evaluation prompts from data/gsm_test.json. The supplied prepare_gsm8k.py creates that file from the official test split.

The code shows a mismatch with the stated protocol. It does not establish which implementation produced Table 2. The exact run settings and existing records are needed to determine whether the main results used training-only calibration.

2. The paper needs to separate its new diagnosis from established methods and limit the causal claims. Hyperloop Transformers, Section 4.2 already accumulates GPTQ Hessians across loop iterations. LoopQ, Sections 3–4 discusses recurrent error accumulation, state reuse, and trajectory-aware calibration. Both are cited, but accumulating H_t is not new by itself. The paper should identify which controlled finding or distinction goes beyond them, and make that comparison earlier than Section 7.

The Hessian-rank evidence also needs care. Section 4 reports an increase from 43 to 335 using a relative eigenvalue threshold. Appendix P notes that a better layer-reconstruction proxy does not always mean better model perplexity. These observations support a connection between trajectory coverage and recovery, but do not by themselves show that the rank change causes the recovery.

Finally, the claim of reaching bf16 accuracy applies to strict GSM8K exact match: 34.9% versus 35.0%. For flexible accuracy, the reported quantized result is about 35.3%, below bf16 at 42.23%. The recovery claim should name the metric.

Overall assessment

I currently lean toward rejection, mainly because the origin of the main calibration results is unresolved. The feedback experiments and equal-budget controls are strong. Evidence that the results used the stated training-only split, together with a clearer distinction from prior work and narrower claims, could support a higher rating.

Basis for confidence

My background includes PTQ, Hessian-based calibration, and recurrent error propagation. I examined the paper and supplied code. The main uncertainty is which code and data produced the final results; the provided files alone cannot settle that.

Questions for authors

Calibration records. Please provide the script, command-line arguments, calibration-question IDs, and existing evaluation records for the Huginn and R-Llama results in Table 2. Were calibration questions drawn only from the training split? How does the supplied test-split entry point relate to those runs?

Scientific contribution and interpretation. What is the main finding beyond Hyperloop and LoopQ: the controlled localization of feedback-sensitive components, the calibration-direction mismatch, or their relation? Please also explain whether Hessian rank is a causal explanation or a diagnostic indicator, and limit the bf16-recovery statement to the metric supported by the results. These questions can be addressed through the existing evidence and records.

Reviewer_2CncOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of quantizing Looped Transformers using PTQ. The authors study and identify two failure modes, 1) feedback exposure and 2) calibration blindness. For the first they show that rounding error can become dangerous when it enters the recurrent state transition without an identity path. For the latter they show that calibrating only on step 0 can be detrimental. They propose trajectory based calibration to overcome the ‘step-0’ GPTQ shortcomings and experimentally validate its effectiveness on 7 different looped transformers.

Critical strengths
  • The paper is very timely and it is an important problem to understand what are challenges with looped transformer quantization and how do existing PTQ algorithms translate to them.
  • I appreciate that this paper focuses strongly on the analysis and problem part and is not just ‘yet another quantization algorithm’ that works for a specific model family. Especially the controlled injections (sec 3.1 and fig 1) lead to interesting insights as well as the cross-architecture rank survey (sec 4.1 and fig 2).
  • The empirical study of the trajectory calibration is very broad and covers 9 models across 7 different model families.
Critical weaknesses
  • While some of the analysis parts are interesting and novel, a big part of the paper are build on an arguably unnatural and overly simplistic created baseline (step-0 Hessian / one-step GPTQ) which significantly limits the contribution and novelty of this paper. As the authors correctly note, “Conventional calibration-based PTQ uses activations from a single forward pass.” (L473), this already implies that with shared parameters calibration data is collected and aggregated over all loop steps as they happen in one forward pass of the model. The authors further note that also in diffusion models it is common practice to calibrate over all denoising time-steps (cf sec 2 and 7). Given it is ‘common practice’ to select calibration data from all occurrences of a weight/quantizer in the model, this makes big parts of section 5 and 6 (and some of 4) trivial/obvious.
  • The feedback exposure is a nice analysis and insights, but it is a shortcoming that the authors do not further investigate or propose a technique to mitigate the effect (except keeping it in BF16 and Hadamard rotation as the suggestion in L215).
  • The main empirical study is largely restricted to GPTQ style weight-optimization. Given it is more of an analysis paper, it would have been good to see how the insights hold across more algorithms in the same category (weight rounding/optimization) and how it translates to different PTQ techniques (e.g. transformation learning etc).
  • While the overall writing is clear and easy to follow, there are lot’s of in-line results discussion which can make parts harder to follow (some of these results are even not in tables or figures nor the appendix). Especially around section 3.2 the story gets a bit blurry and the many in-line results that make it hard to follow what exactly the authors want to convey. Also a clear motivation or research question seems to be missing here.
Questions for authors

Where is the data for the controlled injection experiments? (L140-147, it references appendix E but I could not find a clear table or figure with the results)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.