- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Conventional post-training quantization (PTQ) can severely degrade looped language models (LoopLMs) because recursive block reuse introduces activation distribution shifts, state mismatch across loop transitions, and recursive error accumulation. Assigning separate transformations to every loop requires additional stored weight representations when those transformations are folded into the weights, weakening the parameter-sharing advantage. The authors propose LoopQ, which combines loop-aware activation scaling (LAS), selective loop-aware transformation (SLT), a cross-loop transition adapter (CTA), and trajectory-aware calibration (TAC) while retaining a shared quantized backbone. TAC jointly optimizes these adaptations using predictive KL and boundary-state matching against the full-precision model. Experiments on four backbones from three LoopLM architectures demonstrate substantial accuracy and perplexity recovery, particularly under W4A4 quantization.
Critical strengths
- Well-motivated problem and insightful error analysis. The paper identifies quantization difficulties arising from recursive parameter sharing.
Propositions 1 and 2characterize the cost of shared quantization configurations and bound loop-wise error propagation, whileFigures 1 and 2empirically show pronounced error peaks near loop transitions. This distinction provides a useful basis for adapting conventional PTQ to LoopLMs. - Broad empirical validation and deployment profiling. W4A4/W4A8 evaluations cover
Ouro 1.4B/2.6B,LoopFormer, andParcae, with additional KV4, dynamic INT4/NVFP4, and early-exit evaluations on Ouro 1.4B. Packed W4A4KV4 achieves2.13×BF16 decode throughput on an RTX 5090 at 8,192 prompt tokens and 1,024 generated tokens, with concurrency selected separately for each runtime (Table 9,Figure 3c). Estimated deployment memory is 1.13 GiB, adding 0.10 GiB over shared FlatQuant (Table 3). - Clear presentation and intuitive design.
Figure 1connects the identified quantization challenges toLAS,SLT,CTA, andTAC, making the overall framework easy to follow.
Critical weaknesses
W1. Missing heuristic baselines for SLT. Error peaks occur near loop transitions on Ouro (Figures 1 and 2), and selected linear-path groups concentrate near the end of the shared stack (Figure 4). These observations motivate testing a simple first/last-layer selection heuristic against GGN-based selection (Eqs. 5, 6, and 10). The authors note that restricting selection to late layers would exclude earlier key-cache sites, but do not evaluate a heuristic that controls cache selections separately. Random SLT lowers mean accuracy from 58.96% to 56.82% and raises OP PPL from 8.98 to 10.48 (Table 2), demonstrating a benefit over random selection without establishing an advantage over this natural heuristic. Such a comparison should match group counts and types and report deployment memory, since linear-path groups require loop-dependent weight representations whereas key-cache groups act on the cache path.
W2. Incomplete separation of transformation and calibration effects. In Table 2, adding both SLT and TAC raises mean accuracy from 38.54% for LAS Only to 57.46% for w/o CTA, a gain of 18.92 percentage points. TAC combines predictive KL with boundary-state matching (Eq. 8). Appendix B.2 describes MSE-based scale initialization followed by joint optimization, but does not specify the objective used for the LAS Only ablation. Unrolled FlatQuant achieves 61.43% through independent loop calibration, supporting the value of separate transformations at a higher storage cost, but does not isolate their contribution under LoopQ's selective budget. An objective-matched comparison between LAS Only + TAC and w/o CTA is needed to distinguish the contribution of SLT from that of trajectory-aware calibration.
W3. Unreported calibration cost. LoopQ optimizes through the unrolled recurrence while keeping backbone weights frozen. For Ouro 2.6B, this corresponds to 48 shared layers executed over four loops, or 192 effective layers. Calibration also includes FlatQuant MSE initialization and GGN curvature estimation. However, the paper reports no calibration wall-clock time, FLOPs, or peak GPU memory. Tables 3, 8, and 9 address deployment memory and inference performance, leaving the calibration overhead relative to conventional PTQ unquantified and limiting assessment of the method's practical preparation cost.
Questions for authors
Q1. Comparison with Boundary-Layer Selection Heuristics. How does Full LoopQ compare in accuracy and deployment memory with a heuristic selecting linear-path transformations from the first and/or last layers without GGN-based search? Please match the number and types of selected groups and hold key-cache selections fixed.
Q2. Separating Transformation and Calibration Effects. Which objective is used for LAS Only in Table 2? Could you compare LAS Only + TAC, without SLT or CTA, against w/o CTA using the same Eq. 8 objective and schedule? Could you also evaluate LoopQ w/o KL Loss using reconstruction MSE and fixed, KL-independent boundary weights, holding selected groups fixed, since Eq. 13 also depends on KL?
Q3. Calibration Cost. Could you report total calibration wall-clock time, FLOPs, and peak GPU memory for LoopQ, FlatQuant, and QuaRot on Ouro 1.4B/2.6B under the same quantization setting and hardware?
3 comments
Thank you for the detailed response and for conducting the additional experiments within the rebuttal period. While your response clarifies several aspects of the paper, a few key concerns remain only partially resolved:
-
On W1 / Q1: Thank you for providing the additional comparison in
Table R3-1. However, there is insufficient explanation as to why only fourgate_up(MLP gate/up projection) groups were selected rather than matching the group types selected bySLTat the first/last boundary layers (e.g., the four linear groups of the final layer inFigure 4a). Moreover, the comparison omits the 5-task downstream benchmark accuracies reported inTable 2, so this concern is not fully resolved. -
On W2 / Q2: The results in
Table R3-2actually reinforce our concern that the performance recovery is predominantly driven byTACrather thanSLT. Specifically, starting fromLAS Only(MSE calibration, with143.87LAMBADA PPL and37.89WikiText-2 PPL inTable 2), applyingTACalone withoutSLTorCTA(SLT: Off, CTA: Off) already reduces LAMBADA PPL from143.87to9.88(and WikiText-2 PPL from37.89to15.11), whereas turning onSLTonly further reduces LAMBADA PPL from9.88to9.59. -
On W3 / Q3 (Calibration cost): We understand that conducting the full set of profiling experiments (such as
Ouro 2.6Band exact FLOPs) may have been difficult given the time constraints of the rebuttal period, and we appreciate that you provided the calibration wall-clock time (8 hourstotal) and peak GPU memory (~70 GiB) forOuro 1.4B.
In summary, while I appreciate the authors' efforts during the rebuttal, my primary concerns regarding the ablation and heuristic comparisons remain only partially addressed.
Therefore, I will maintain my original score.
Thank you for the follow-up and for highlighting these two points.
Q1. Boundary-layer selection. Table R3-1 tests a simple size-based rule. We chose gate_up because it combines the gate and up projections and covers more weights than any other candidate linear group in an Ouro 1.4B layer. We selected four such groups to match the number of linear groups selected by SLT. With the rebuttal period now drawing to a close, we do not have time to run the additional experiments before the deadline. We will extend the comparison to more heuristics and models in the revised manuscript.
Q2. Contributions of TAC and SLT. We agree that TAC accounts for most of the recovery in this setting. Calibrating shared transformations over the full trajectory already brings PPL down substantially. From this lower baseline, adding either SLT or CTA further reduces PPL on both datasets under the same calibration objective and budget. Using both gives the lowest PPL on LAMBADA and WikiText-2. We see value in these additional gains and will discuss them alongside TAC’s larger contribution in the revised manuscript.
We appreciate your careful reading and the suggestions for improving the paper.
Thank you for recognizing our analysis of quantization error in looped models and our deployment evaluation. We add a boundary-layer heuristic comparison under W4A4KV4 and an ablation that removes KL dependence from the training objective.
W1 / Q1: Boundary-layer heuristic
On Ouro-1.4B-Thinking under W4A4KV4, we compare full LoopQ with a heuristic that selects four fixed boundary-layer
gate_upgroups.Table R3-1. Comparison with fixed boundary-layer selection.
LoopQ achieves approximately 2.5% lower WikiText-2 PPL and 6.2% lower LAMBADA PPL than the heuristic. These results support our selection method for identifying modules that benefit from loop-specific transformations.
W2 / Q2: LAS calibration and the KL ablation
LAS Only uses MSE calibration. To isolate SLT's contribution, we compare shared transformations and SLT with and without CTA. All four configurations use LAS, the same TAC objective, and the same calibration budget.
Table R3-2. SLT and CTA under the same TAC objective and calibration budget.
Without CTA, SLT reduces LAMBADA PPL from 9.88 to 9.59 and WikiText-2 PPL from 15.11 to 14.03, demonstrating its contribution under the same calibration objective and budget. With CTA, SLT further reduces the two metrics by approximately 2.3% and 6.1%, respectively.
This control separates the two changes combined in the original LAS Only versus no-CTA comparison: both configurations now use TAC, so the remaining improvement measures the benefit of selective loop-specific transformations under that objective.
For the KL ablation, we replace predictive KL with logit reconstruction MSE and use fixed boundary weights.
Table R3-3. Predictive KL versus logit MSE with fixed boundary weights.
This replacement raises LAMBADA PPL from 8.98 to 12.25 and WikiText-2 PPL from 13.41 to 14.37. The gains from full TAC on both datasets support predictive distribution matching with adaptive boundary weights.
Predictive KL aligns the terminal predictive distributions, while logit MSE matches raw scores. The lower PPL with full TAC supports the combined objective in Eq. (8). Fixed boundary weights remove the additional KL dependence in Eq. (13), so this ablation removes KL from both terms of the training objective.
W3 / Q3: Calibration cost
Table R3-4. Calibration time for Ouro 1.4B on H200.
The additional cost comes from joint calibration: we backpropagate through the unrolled recurrence to optimize the quantization parameters and CTA together, with peak GPU memory around 70 GiB. Initial FlatQuant calibration and LoopQ joint calibration each account for roughly half of the 8-hour total. With an existing shared FlatQuant checkpoint, only the additional 4 hours are needed.