acceptodds
Under review as a conference paper at ICLR 2027

LoopQ: Quantization for Looped Language Models

14 views

Abstract

Looped language models (LoopLMs) increase effective computational depth by repeatedly applying shared Transformer blocks without increasing the parameter count. However, this reuse makes them particularly sensitive to post-training quantization (PTQ). We systematically study PTQ in LoopLMs and identify three sources of degradation: distribution shift across roles, state reuse across loop transitions, and recursive error accumulation. We propose LoopQ, a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines loop-aware activation scaling and selective transformations to accommodate changing hidden-state distributions, cross-loop state alignment to stabilize transitions, and trajectory-aware calibration to limit accumulated error. Under W4A4 quantization, LoopQ achieves a 67.8% average relative improvement in five-task mean accuracy across four backbones and an average relative perplexity reduction of at least 75.2% across eight backbone–dataset pairs, compared with the strongest static PTQ baseline for each comparison.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
quantization, post-training quantization, looped language models, low-bit inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9W35OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Conventional post-training quantization (PTQ) can severely degrade looped language models (LoopLMs) because recursive block reuse introduces activation distribution shifts, state mismatch across loop transitions, and recursive error accumulation. Assigning separate transformations to every loop requires additional stored weight representations when those transformations are folded into the weights, weakening the parameter-sharing advantage. The authors propose LoopQ, which combines loop-aware activation scaling (LAS), selective loop-aware transformation (SLT), a cross-loop transition adapter (CTA), and trajectory-aware calibration (TAC) while retaining a shared quantized backbone. TAC jointly optimizes these adaptations using predictive KL and boundary-state matching against the full-precision model. Experiments on four backbones from three LoopLM architectures demonstrate substantial accuracy and perplexity recovery, particularly under W4A4 quantization.

Critical strengths
  • Well-motivated problem and insightful error analysis. The paper identifies quantization difficulties arising from recursive parameter sharing. Propositions 1 and 2 characterize the cost of shared quantization configurations and bound loop-wise error propagation, while Figures 1 and 2 empirically show pronounced error peaks near loop transitions. This distinction provides a useful basis for adapting conventional PTQ to LoopLMs.
  • Broad empirical validation and deployment profiling. W4A4/W4A8 evaluations cover Ouro 1.4B/2.6B, LoopFormer, and Parcae, with additional KV4, dynamic INT4/NVFP4, and early-exit evaluations on Ouro 1.4B. Packed W4A4KV4 achieves 2.13× BF16 decode throughput on an RTX 5090 at 8,192 prompt tokens and 1,024 generated tokens, with concurrency selected separately for each runtime (Table 9, Figure 3c). Estimated deployment memory is 1.13 GiB, adding 0.10 GiB over shared FlatQuant (Table 3).
  • Clear presentation and intuitive design. Figure 1 connects the identified quantization challenges to LAS, SLT, CTA, and TAC, making the overall framework easy to follow.
Critical weaknesses

W1. Missing heuristic baselines for SLT. Error peaks occur near loop transitions on Ouro (Figures 1 and 2), and selected linear-path groups concentrate near the end of the shared stack (Figure 4). These observations motivate testing a simple first/last-layer selection heuristic against GGN-based selection (Eqs. 5, 6, and 10). The authors note that restricting selection to late layers would exclude earlier key-cache sites, but do not evaluate a heuristic that controls cache selections separately. Random SLT lowers mean accuracy from 58.96% to 56.82% and raises OP PPL from 8.98 to 10.48 (Table 2), demonstrating a benefit over random selection without establishing an advantage over this natural heuristic. Such a comparison should match group counts and types and report deployment memory, since linear-path groups require loop-dependent weight representations whereas key-cache groups act on the cache path.

W2. Incomplete separation of transformation and calibration effects. In Table 2, adding both SLT and TAC raises mean accuracy from 38.54% for LAS Only to 57.46% for w/o CTA, a gain of 18.92 percentage points. TAC combines predictive KL with boundary-state matching (Eq. 8). Appendix B.2 describes MSE-based scale initialization followed by joint optimization, but does not specify the objective used for the LAS Only ablation. Unrolled FlatQuant achieves 61.43% through independent loop calibration, supporting the value of separate transformations at a higher storage cost, but does not isolate their contribution under LoopQ's selective budget. An objective-matched comparison between LAS Only + TAC and w/o CTA is needed to distinguish the contribution of SLT from that of trajectory-aware calibration.

W3. Unreported calibration cost. LoopQ optimizes through the unrolled recurrence while keeping backbone weights frozen. For Ouro 2.6B, this corresponds to 48 shared layers executed over four loops, or 192 effective layers. Calibration also includes FlatQuant MSE initialization and GGN curvature estimation. However, the paper reports no calibration wall-clock time, FLOPs, or peak GPU memory. Tables 3, 8, and 9 address deployment memory and inference performance, leaving the calibration overhead relative to conventional PTQ unquantified and limiting assessment of the method's practical preparation cost.

Questions for authors

Q1. Comparison with Boundary-Layer Selection Heuristics. How does Full LoopQ compare in accuracy and deployment memory with a heuristic selecting linear-path transformations from the first and/or last layers without GGN-based search? Please match the number and types of selected groups and hold key-cache selections fixed.

Q2. Separating Transformation and Calibration Effects. Which objective is used for LAS Only in Table 2? Could you compare LAS Only + TAC, without SLT or CTA, against w/o CTA using the same Eq. 8 objective and schedule? Could you also evaluate LoopQ w/o KL Loss using reconstruction MSE and fixed, KL-independent boundary weights, holding selected groups fixed, since Eq. 13 also depends on KL?

Q3. Calibration Cost. Could you report total calibration wall-clock time, FLOPs, and peak GPU memory for LoopQ, FlatQuant, and QuaRot on Ouro 1.4B/2.6B under the same quantization setting and hardware?

3 comments

Thank you for recognizing our analysis of quantization error in looped models and our deployment evaluation. We add a boundary-layer heuristic comparison under W4A4KV4 and an ablation that removes KL dependence from the training objective.

W1 / Q1: Boundary-layer heuristic

How does Full LoopQ compare in accuracy and deployment memory with a heuristic selecting linear-path transformations from the first and/or last layers without GGN-based search? Please match the number and types of selected groups and hold key-cache selections fixed.

On Ouro-1.4B-Thinking under W4A4KV4, we compare full LoopQ with a heuristic that selects four fixed boundary-layer gate_up groups.

Table R3-1. Comparison with fixed boundary-layer selection.

MethodWikiText-2 PPL ↓LAMBADA PPL ↓
Fixed four Gate-up groups13.769.57
LoopQ13.418.98

LoopQ achieves approximately 2.5% lower WikiText-2 PPL and 6.2% lower LAMBADA PPL than the heuristic. These results support our selection method for identifying modules that benefit from loop-specific transformations.

W2 / Q2: LAS calibration and the KL ablation

Which objective is used for LAS Only in Table 2? Could you compare LAS Only + TAC, without SLT or CTA, against w/o CTA using the same Eq. 8 objective and schedule? Could you also evaluate LoopQ w/o KL Loss using reconstruction MSE and fixed, KL-independent boundary weights, holding selected groups fixed, since Eq. 13 also depends on KL?

LAS Only uses MSE calibration. To isolate SLT's contribution, we compare shared transformations and SLT with and without CTA. All four configurations use LAS, the same TAC objective, and the same calibration budget.

Table R3-2. SLT and CTA under the same TAC objective and calibration budget.

SLTCTALAMBADA PPL ↓WikiText-2 PPL ↓
OffOff9.8815.11
OnOff9.5914.03
OffOn9.1914.28
OnOn8.9813.41

Without CTA, SLT reduces LAMBADA PPL from 9.88 to 9.59 and WikiText-2 PPL from 15.11 to 14.03, demonstrating its contribution under the same calibration objective and budget. With CTA, SLT further reduces the two metrics by approximately 2.3% and 6.1%, respectively.

This control separates the two changes combined in the original LAS Only versus no-CTA comparison: both configurations now use TAC, so the remaining improvement measures the benefit of selective loop-specific transformations under that objective.

For the KL ablation, we replace predictive KL with logit reconstruction MSE and use fixed boundary weights.

Table R3-3. Predictive KL versus logit MSE with fixed boundary weights.

Calibration objectiveLAMBADA PPL ↓WikiText-2 PPL ↓
Logit MSE + fixed boundary weights12.2514.37
Full LoopQ8.9813.41

This replacement raises LAMBADA PPL from 8.98 to 12.25 and WikiText-2 PPL from 13.41 to 14.37. The gains from full TAC on both datasets support predictive distribution matching with adaptive boundary weights.

Predictive KL aligns the terminal predictive distributions, while logit MSE matches raw scores. The lower PPL with full TAC supports the combined objective in Eq. (8). Fixed boundary weights remove the additional KL dependence in Eq. (13), so this ablation removes KL from both terms of the training objective.

W3 / Q3: Calibration cost

Could you report total calibration wall-clock time, FLOPs, and peak GPU memory for LoopQ, FlatQuant, and QuaRot on Ouro 1.4B/2.6B under the same quantization setting and hardware?

Table R3-4. Calibration time for Ouro 1.4B on H200.

Method or stageOuro 1.4B calibration time (H200, Approximately)
QuaRot / MixQuant / SpinQuant30 minutes each
FlatQuant4 hours
LoopQ joint calibration4 hours

The additional cost comes from joint calibration: we backpropagate through the unrolled recurrence to optimize the quantization parameters and CTA together, with peak GPU memory around 70 GiB. Initial FlatQuant calibration and LoopQ joint calibration each account for roughly half of the 8-hour total. With an existing shared FlatQuant checkpoint, only the additional 4 hours are needed.

Thank you for the detailed response and for conducting the additional experiments within the rebuttal period. While your response clarifies several aspects of the paper, a few key concerns remain only partially resolved:

  • On W1 / Q1: Thank you for providing the additional comparison in Table R3-1. However, there is insufficient explanation as to why only four gate_up (MLP gate/up projection) groups were selected rather than matching the group types selected by SLT at the first/last boundary layers (e.g., the four linear groups of the final layer in Figure 4a). Moreover, the comparison omits the 5-task downstream benchmark accuracies reported in Table 2, so this concern is not fully resolved.

  • On W2 / Q2: The results in Table R3-2 actually reinforce our concern that the performance recovery is predominantly driven by TAC rather than SLT. Specifically, starting from LAS Only (MSE calibration, with 143.87 LAMBADA PPL and 37.89 WikiText-2 PPL in Table 2), applying TAC alone without SLT or CTA (SLT: Off, CTA: Off) already reduces LAMBADA PPL from 143.87 to 9.88 (and WikiText-2 PPL from 37.89 to 15.11), whereas turning on SLT only further reduces LAMBADA PPL from 9.88 to 9.59.

  • On W3 / Q3 (Calibration cost): We understand that conducting the full set of profiling experiments (such as Ouro 2.6B and exact FLOPs) may have been difficult given the time constraints of the rebuttal period, and we appreciate that you provided the calibration wall-clock time (8 hours total) and peak GPU memory (~70 GiB) for Ouro 1.4B.

In summary, while I appreciate the authors' efforts during the rebuttal, my primary concerns regarding the ablation and heuristic comparisons remain only partially addressed.
Therefore, I will maintain my original score.

Thank you for the follow-up and for highlighting these two points.

Q1. Boundary-layer selection. Table R3-1 tests a simple size-based rule. We chose gate_up because it combines the gate and up projections and covers more weights than any other candidate linear group in an Ouro 1.4B layer. We selected four such groups to match the number of linear groups selected by SLT. With the rebuttal period now drawing to a close, we do not have time to run the additional experiments before the deadline. We will extend the comparison to more heuristics and models in the revised manuscript.

Q2. Contributions of TAC and SLT. We agree that TAC accounts for most of the recovery in this setting. Calibrating shared transformations over the full trajectory already brings PPL down substantially. From this lower baseline, adding either SLT or CTA further reduces PPL on both datasets under the same calibration objective and budget. Using both gives the lowest PPL on LAMBADA and WikiText-2. We see value in these additional gains and will discuss them alongside TAC’s larger contribution in the revised manuscript.

We appreciate your careful reading and the suggestions for improving the paper.

Reviewer_J459OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new post-training quantization framework for LoopLMs, named LoopQ. Firstly, LoopQ identifies two key challenges in LoopLM quantization: (1). shared quantization scales and transformations may be poorly suited to the geometry of individual loops; (2). quantization will carry state mismatch across boundaries and its error will propagate through subsequent computation and can be amplified. Based on these observations, LoopQ introduces four coordinated adaptations: (1) LAS uses loop-dependent activation scales; (2) SLT dynamically selects transformation groups according to a sharing-sensitivity score; (3) CTA introduces an adapter to adjust the quantized hidden state before the next loop; (4) TAC jointly optimizes these components with the full-precision model. Experiments demonstrate that LoopQ achieves better average downstream task accuracy and lower ppl across different backbone models with strong baselines. Evaluation on reasoning and code generation tasks further demonstrates the strengths of LoopQ under complex tasks. Ablation studies also demonstrate the effectiveness of the proposed components.

Critical strengths
  1. This paper identifies the key challenges in LoopLM quantization and introduces four coordinated adaptations to guide post-training quantization, which help reduce the performance degradation.
  2. Experiments demonstrate higher average downstream task accuracy and lower ppl across different backbone models compared with strong baselines, with a further evaluation on reasoning and code generation tasks. Efficiency analysis demonstrates the deployment efficiency of LoopQ.
Critical weaknesses
  1. This paper does not report the end-to-end calibration cost. LoopQ introduces four components during the PTQ stage to accommodate the loop-specific characteristics of LoopLMs, but these components also introduce additional calibration overhead. It remains unclear whether this cost is practical.
  2. The ablation studies do not demonstrate whether the gate, low-rank residual bottleneck and the orthogonal projection are necessary. It is strongly recommended to isolate their contributions.
Questions for authors
  1. Could you report your end-to-end calibration cost? For example, report the end-to-end time of the PTQ stage and peak GPU memory usage. It is recommended to compare them with those of the baselines.
  2. Could you ablate the individual contributions of the gate, low-rank residual network, and the orthogonal projection in CTA? For example, compare the gtg_t-only, RtR_t-only, and CTA without orthogonal projection (eq.7). This would help us further assess the contribution of each component.
2 comments

Thank you for recognizing LoopQ's design and deployment evaluation. We report calibration costs and CTA branch ablations below.

W1 / Q1: Calibration cost

Could you report your end-to-end calibration cost? For example, report the end-to-end time of the PTQ stage and peak GPU memory usage. It is recommended to compare them with those of the baselines.

Table R2-1. Calibration time for Ouro 1.4B on H200.

Method or stageOuro 1.4B calibration time (H200, Approximately)
QuaRot / MixQuant / SpinQuant30 minutes each
FlatQuant4 hours
LoopQ joint calibration4 hours

The additional cost comes from joint calibration: we backpropagate through the unrolled recurrence to optimize the quantization parameters and CTA together, with peak GPU memory around 70 GiB. Initial FlatQuant calibration and LoopQ joint calibration each account for roughly half of the 8-hour total. With an existing shared FlatQuant checkpoint, only the additional 4 hours are needed.

W2 / Q2: CTA branch ablations

Could you ablate the individual contributions of the gate, low-rank residual network, and the orthogonal projection in CTA? For example, compare the gt{g}_{t}-only, Rt{R}_{t}-only, and CTA without orthogonal projection (eq.7). This would help us further assess the contribution of each component.

We train gate-only and residual-only variants from the same MSE checkpoint, matching full CTA's calibration budget and group-selection procedure. Gate-only modulates the current loop update; residual-only retains the low-rank correction and its orthogonal projection.

Table R2-2. CTA branch ablations.

CTA configurationLAMBADA PPL ↓WikiText-2 PPL ↓
No CTA9.5914.03
Gate-only9.2313.91
Residual-only9.3913.72
w/o orthogonal9.1713.45
Full LoopQ8.9813.41

Each branch reduces PPL on both datasets. Gate-only gives a larger reduction on LAMBADA, while residual-only gives a larger reduction on WikiText-2. Combining them reduces LAMBADA PPL from 9.59 to 8.98, below either branch alone.

As described in Section 4.3, the gate modulates the existing update, while the projected low-rank branch supplies a complementary directional correction before the state is reused. The combined result supports this design.

Thank you for your response. Most of my concerns have been addressed, and I will keep my positive rating.

Reviewer_YmS7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LoopQ addresses post-training quantization for language models that repeatedly apply shared Transformer blocks. It combines loop-specific activation scales, selective loop-specific transformations, transition adapters, and joint trajectory calibration using terminal prediction KL and boundary-state matching. Static W4A4/W4A8 experiments cover two Ouro sizes, LoopFormer, and Parcae; additional Ouro experiments test dynamic INT4/NVFP4 and quantized KV caches. The paper also reports component ablations, early exit, depth extrapolation, and packed inference measurements. The main result is substantial recovery from static W4A4 failures while retaining a relatively compact model representation; gains are smaller in some other formats.

Critical strengths
  1. The recovery is large in an important setting and is demonstrated across several architectures. The static baselines already collect activations over the full recurrence, so the improvement cannot simply be attributed to correcting a baseline calibrated only on the first loop. For Ouro 1.4B at W4A4, the five-task mean rises from 25.62% for the strongest static baseline to 59.09%, while LAMBADA perplexity falls from over 1,000 to 8.20. The fully unrolled quantized model provides a useful reference for both accuracy and the cost of giving up sharing. These results make a convincing case that recurrent reuse deserves specific attention in PTQ.
  2. The paper examines both component effects and deployment costs. In the W4A4KV4 ablation, LAS alone reaches 38.54%, removing CTA gives 57.46%, and the full method reaches 58.96%; random group selection also performs worse than GGN selection. The additional formats, early-exit results, and packed RTX 5090 measurements broaden the evaluation. I also appreciate that the storage estimates include loop-specific derived weight copies. The reported 1.24–2.13× throughput improvements use separately chosen concurrency for the two runtimes, so they should be read as achieved throughput under that protocol.
Critical weaknesses
  1. The ablations leave the contribution of full-trajectory calibration partly mixed with that of loop-specific parameters. Table 2 does establish benefits from CTA, boundary matching, and GGN rather than random selection. However, its LAS-only to no-CTA comparison adds both SLT and trajectory-aware calibration, and there is no fully shared configuration optimized with the same terminal prediction KL and optimization budget. Such a control would clarify how much recovery comes from stronger calibration of shared parameters and how much requires relaxing sharing. This is a limitation of component attribution, not an absence of useful ablations or evidence against the overall recovery.
  2. Some numerical settings and calibration resources needed for reproduction remain unspecified in the paper. Appendix B describes the selection procedure, final group budget, adapter architecture, and adaptive loss, but does not give the total optimization steps, optimizer/learning rate, numerical CTA rank, probe count, refresh interval, or numerical warmup and budget-reduction schedule. Calibration time and peak memory are also not reported. These omissions matter because the method backpropagates through the recurrence and estimates curvature. The deployment appendix does give storage accounting, timing protocols, and component overheads, so the remaining resource concern is specifically calibration. The NVFP4 advantage over MixQuant is about 0.28 percentage points; without uncertainty estimates, I would interpret this as a modest extension result rather than evidence of a large advantage in that format.
Questions for authors

As shown in the weakness.

1 comment

Thank you for recognizing LoopQ's effectiveness across looped architectures. We address the shared-transformation control and calibration costs below.

W1: Shared transformations and trajectory-aware calibration

The ablations leave the contribution of full-trajectory calibration partly mixed with that of loop-specific parameters. Table 2 does establish benefits from CTA, boundary matching, and GGN rather than random selection. However, its LAS-only to no-CTA comparison adds both SLT and trajectory-aware calibration, and there is no fully shared configuration optimized with the same terminal prediction KL and optimization budget. Such a control would clarify how much recovery comes from stronger calibration of shared parameters and how much requires relaxing sharing. This is a limitation of component attribution, not an absence of useful ablations or evidence against the overall recovery.

We compare shared transformations and SLT on Ouro 1.4B-Thinking under static W4A4KV4, with and without CTA. All four configurations use LAS, the same TAC objective, and the same calibration budget.

Table R1-1. SLT and CTA under the same TAC objective and calibration budget.

SLTCTALAMBADA PPL ↓WikiText-2 PPL ↓
OffOff9.8815.11
OnOff9.5914.03
OffOn9.1914.28
OnOn8.9813.41

Without CTA, SLT reduces LAMBADA PPL from 9.88 to 9.59 and WikiText-2 PPL from 15.11 to 14.03. With CTA, SLT reduces them by approximately 2.3% and 6.1%, respectively. CTA also improves both metrics with or without SLT. Thus, each component contributes under the same TAC objective, and full LoopQ achieves the lowest PPL on both datasets.

The no-CTA comparison directly addresses component attribution: the shared transformations already receive trajectory-aware calibration, yet allowing selected transformations to vary across loops improves both metrics. SLT therefore provides a benefit beyond applying TAC to shared parameters.

Sharing both activation scales and transformations across loops, while retaining TAC and disabling CTA, yields LAMBADA and WikiText-2 PPL of approximately 2.26 × 10⁶ and 2,152.86, respectively. Performance collapses under this fully shared configuration, highlighting the importance of loop-specific activation scales.

W2: Reproducibility and calibration cost

Some numerical settings and calibration resources needed for reproduction remain unspecified in the paper. Appendix B describes the selection procedure, final group budget, adapter architecture, and adaptive loss, but does not give the total optimization steps, optimizer/learning rate, numerical CTA rank, probe count, refresh interval, or numerical warmup and budget-reduction schedule. Calibration time and peak memory are also not reported. These omissions matter because the method backpropagates through the recurrence and estimates curvature. The deployment appendix does give storage accounting, timing protocols, and component overheads, so the remaining resource concern is specifically calibration. The NVFP4 advantage over MixQuant is about 0.28 percentage points; without uncertainty estimates, I would interpret this as a modest extension result rather than evidence of a large advantage in that format.

Joint calibration uses Adam for 160 updates, a peak learning rate of 10⁻³, and CTA rank 8. On an H200, this stage takes approximately 4 hours for Ouro 1.4B from a shared FlatQuant checkpoint, with peak GPU memory around 70 GiB. Including the initial FlatQuant calibration, the total is approximately 8 hours.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.