acceptodds
Under review as a conference paper at ICLR 2027

TReQ: Refitting Output Transforms Without Requantizing Language Models

Abstract

Low-bit weight–activation quantization introduces errors that can accumulate across language-model layers. We present TReQ, a post-training quantization method with block-diagonal transformations before and after low-precision matrix multiplication. The output transformations mix channels within small blocks, providing additional reconstruction freedom while keeping quantized weights and scales fixed during each refit. We combine two-sided transformation construction with reconstruction of attention scores, attention context, and downstream projection outputs. The output transformations can be fused into the GEMM epilogue, avoiding a separate kernel and intermediate-memory round trip. Across Ministral-3-8B, Llama-3.1-8B, and Qwen3-8B, TReQ reduces block reconstruction error relative to matched WUSH baselines in INT4, MXFP4, and NVFP4. Under INT4, it lowers task Kullback–Leibler (KL) divergence and improves accuracy on all five benchmarks for every model, while reducing WikiText-2 perplexity. End-to-end improvements in the FP4 formats are less consistent. A fixed-checkpoint Ministral INT4 comparison shows that output refitting alone reduces held-out KL by 8.63%. On NVIDIA B200, fused output transformations add only 0.1–1.5% to MXFP4 full-model prefill latency at the tested workloads with at least 2,048 total tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-training quantization, Large language models, Weight–activation quantization, Output transformation refitting, Quantization error correction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z2dQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TReQ, a post-training quantization method that improves weight–activation quantization by introducing block-diagonal output transformations. Unlike conventional approaches that primarily optimize input transformations or quantized weights, TReQ refits output transformations while keeping quantized weight codes and scales fixed. The method further incorporates attention-aware reconstruction to reduce errors in attention scores, contexts, and downstream projections.

Critical strengths
  1. The paper addresses an important problem in low-bit LLM quantization: reducing accumulated quantization errors without modifying existing quantized weights and scales.
Critical weaknesses
  1. To the best of my knowledge,the method largely builds on existing transformation-based quantization and reconstruction techniques, particularly WUSH and TurboBoA. While block-diagonal output refitting provides additional flexibility, the conceptual advance beyond these methods appears incremental.
  2. The main experiments primarily compare against WUSH, without direct comparisons to other competitive quantization approaches under comparable conditions.
Questions for authors
  1. Why do substantial reductions in block reconstruction error fail to consistently improve downstream performance under MXFP4 and NVFP4? Does this indicate a limitation of the reconstruction objective?
  2. Could the authors provide a component-wise ablation separating two-sided transformation construction, output refitting, and attention-aware reconstruction? Which component contributes most to the final improvements?
  3. On the same fixed quantized checkpoint, how much improvement comes from per-channel diagonal scaling versus 32×3232 \times 32 block-diagonal mixing? Please keep weight codes, scales, input transformations, data splits, and fitting procedures identical, and report held-out KL divergence and representative task accuracy.
  4. Could the authors provide pseudocode for the complete algorithm, clearly indicating when input transformations, output transformations, quantized weight codes, and scales are updated or frozen at each stage? In particular, please clarify which steps involve requantization to facilitate reproducibility.
2 comments

Thank you for your feedback.

Contribution beyond WUSH and TurboBoA. TReQ introduces an explicit block-diagonal output map that remains adjustable after quantization. WUSH provides the input-transform construction, while TurboBoA’s attention-wise refinement updates per-output-channel scales with integer codes fixed, corresponding to diagonal output scaling. Our output transformations additionally mix channels within each block while keeping the original weight codes and quantization scales intact. We fit these maps against projection outputs, attention scores, and attention context. The fixed-checkpoint comparison below directly measures the additional value of channel mixing.

External baseline and applicability beyond WUSH. We added a FlatQuant W4A4 adaptation for Ministral, obtaining 0.04908 held-out KL and 74.23% MMLU on the identical trajectories and 14,042 questions used in our fixed-checkpoint comparison. FlatQuant substantially outperforms these fixed-checkpoint arms and exceeds full TReQ’s submitted MMLU score of 61.79%. We will report this result and narrow our claim to the contribution of output refitting.

To examine applicability beyond WUSH, we froze all original FlatQuant quantization parameters and fitted additional block-diagonal output maps using gradient-based block-MSE fitting. On 32 independent held-out conversations, these maps reduce mean per-layer MSE by 7.28%. However, model-level KL changes from 0.04908 to 0.04972 and MMLU from 74.23% to 74.29%, with no statistically detected improvement on either endpoint. These results show that output transformations can improve held-out local reconstruction with another quantization method, while also supporting your concern about the reconstruction objective: lower local MSE does not guarantee lower end-to-end KL or higher accuracy. Why these gains fail to improve model-level quality, including in the submitted MXFP4/NVFP4 results, remains an open question.

Component-wise ablation. We ran a cumulative component ablation on layer 12 of Ministral-3-8B INT4, keeping grouping, quantization settings, input statistics, and data splits fixed. On 32 independent held-out conversations, complete-block relative MSE, including residuals, is:

Cumulative configurationHeld-out relative MSE ×10⁻³ ↓
Two-sided WUSH initialization + RTN13.709
+ Projection output fitting11.507
+ Four alternating construction rounds11.579
+ GPTQ10.485
+ Attention-aware Q/K/V refitting8.404
+ O/down teacher refitting7.826

Overall, the pipeline reduces held-out block MSE on layer 12 by 42.9%, despite a small 0.63% intermediate increase when adding alternating construction in this specific setup. Attention-aware Q/K/V refitting gives the largest individual reduction in this sequence (19.84%). Appendix B.6 provides fixed-checkpoint evidence for the output refit as a whole.

Diagonal scaling versus block-diagonal mixing. We ran the requested three-way comparison on one fixed Ministral-3-8B INT4 WUSH checkpoint, keeping weight codes, scales, input transformations, and data splits identical. We fitted diagonal and 32 × 32 transformations under their respective constraints using the same refitting protocol.

Output transformationHeld-out Tulu KL ↓MMLU ↑
Identity (OFF)0.4136953.38%
Diagonal0.3853355.60%
Block-320.3728955.93%

Diagonal scaling already helps. Block-32 gives a further 3.23% relative KL reduction over diagonal scaling, and the MMLU point estimate is 0.33 percentage points higher. This isolates the additional value of channel mixing on this checkpoint.

Algorithm and parameter updates. The update schedule is:

Initialize T_in and T_out with two-sided WUSH; quantize weights with RTN.

Alternate:
  Update T_in and requantize weights with RTN; keep T_out fixed.
  Fit T_out; keep T_in, weight codes, and scales fixed.

Select the best fitting-set candidate.
Apply GPTQ to update weight codes/scales; keep T_in and T_out fixed.

Freeze T_in, weight codes/scales, and activation-quantizer configuration.
Refit only T_out: projection outputs → Q/K scores and V context → O/down teacher outputs.

Requantization occurs only during construction and the final GPTQ pass; subsequent output refits preserve the quantized checkpoint.

Thank you for your detailed response. However, I still consider the novelty of this manuscript to be insufficient, and therefore I maintain my original score.

Reviewer_k7oROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces TReQ, a post‑training quantization method for 4‑bit weight‑activation quantization (A4W4) of large language models. Built upon the prior WUSH two‑sided block‑diagonal transformation framework, TReQ’s core novelty lies in introducing adjustable block‑diagonal output transformations ToutT_{out} after low‑precision matrix multiplication to compensate for quantization‑induced errors. Experiments on three 8B models show consistent improvements under INT4 in block‑level MSE, perplexity, KL‑divergence, and downstream accuracy. However, despite reduced block‑MSE, MXFP4/NVFP4 yield inconsistent end‑to‑end performance. A fixed‑checkpoint ablation validates that refitting only ToutT_{out} already reduces held‑out KL by 8.63%. B200 kernel measurements demonstrate small MXFP4 prefill latency overhead.

Critical strengths
  1. New technical perspective for LLM PTQ. Most existing transformation‑based quantization applies transforms before quantization. TReQ explores compensating quantization errors after low‑precision GEMM via block‑diagonal output transforms. The ablation proves that pure‑ToutT_{out} refitting on frozen quantized checkpoints yields tangible quality gains, which opens a practical possibility for post‑hoc quality repair for existing quantized models.
  2. Attention‑aware multi‑stage refitting. Instead of independently fitting each projection’s output, the method targets coupled intermediate quantities inside self‑attention: attention scores from Q‑K dot‑product and attention context from value projection. This better accounts for error propagation across Q/K/V modules inside Transformer blocks, which is one key reason for strong performance under INT4.
  3. Hardware‑friendly design. Block‑diagonal output transformation can be fused into the GEMM epilogue. Under large token‑count prefill workloads, the added latency for MXFP4 is limited (0.1‑1.5%).
Critical weaknesses
  1. Inconsistent performance for FP4 formats (MXFP4 / NVFP4). Even though block‑level reconstruction MSE is mostly reduced across blocks, such block‑level improvement cannot consistently translate to end‑to‑end KL and downstream accuracy. For Llama‑3.1‑8B and Qwen3‑8B, some tasks show degraded results.

  2. High complexity of offline calibration pipeline. The full workflow involves WUSH initialization, alternating iterative construction, RTN candidate selection, bidirectional GPTQ pass, plus multi‑stage attention‑aware refitting with Gauss‑Newton solves, line‑search, and validation roll‑back. Many hyper‑parameters (regularization λ for different stages, ridge damping, condition‑number threshold, block size) are introduced. Calibration wall‑clock overhead is non‑negligible.

  3. Baseline comparison scope is limited. Main comparisons are against the authors‑adapted WUSH baseline under matched calibration settings. There is no full‑fledged end‑to‑end comparison against other representative A4W4 quantization methods.

  4. Potential misunderstanding risk for readers. The paper’s abstract emphasizes “without requantizing language models”, but weight requantization and updates do happen inside the construction‑phase iterative loop. Weight‑freezing only applies to the later attention‑aware refitting phase. This distinction is not sufficiently emphasized in the high‑level abstract/introduction.

Questions for authors
  1. The timing baseline is an internal “no‑ToutT_{out}” control derived from the same checkpoint, rather than a native WUSH CUDA kernel implementation. Would it be possible to provide a direct latency comparison between TReQ and other quantization methods?

  2. The pipeline contains many hyper‑parameters across different refitting stages. Could you discuss how you selected these hyper‑parameters, and how sensitive final performance is to λ and other numerical thresholds?

  3. Why choose fixed block‑size=32 for block‑diagonal transforms? Have you ablated different block sizes (e.g., 16, 64) to trade‑off reconstruction capacity versus compute / storage overhead?

  4. In construction‑phase alternating loop, you alternate updating ToutT_{out} and requantizing weights. Have you tried an alternative pipeline that skips construction‑phase weight re‑quantization: start directly from an off‑the‑shelf quantized checkpoint and only run your attention‑aware ToutT_{out} refitting? How much performance gap would this simplified pipeline bring compared to full TReQ?

  5. Even with regularization and internal‑validation roll‑back, refitting ToutT_{out} on calibration data may still overfit the calibration distribution. Would it be possible to provide analysis on the generalization gap between fitting‑set KL and held‑out KL across different model/format combinations?

2 comments

Thank you for your feedback.

Latency comparison. We focus on quantized linear layers and integrate our output transformation into the GEMM epilogue. For other quantization methods using the same activation format, weight format, and Tensor Core GEMM, the core matrix multiplication follows the same execution framework. The relevant question is therefore how much additional latency our input/output transformations and their grouping introduce over a standard quantize-and-GEMM implementation.

We measured this directly on NVIDIA B200 using three CUTLASS-based paths: quantize-only, input transformation + quantization, and input/output transformations + quantization. All start from BF16 activations and four-bit weights and use MXFP4 Tensor Core GEMM with the same tile configuration. Input transformations are fused with activation quantization, and output transformations are fused into the GEMM epilogue.

For full-model Ministral-3-8B prefill with at least 2,048 total tokens, the two-sided path adds 3.1–8.4% latency over quantize-only. Input-only adds 2.2–7.1% over quantize-only (overhead of WUSH), while adding the output transformation on top of WUSH only costs 0.7–1.2%. Input transformations are fused with activation quantization, and output transformations are fused into the GEMM epilogue.

Hyperparameters and regularization sensitivity. Appendix B specifies the stage-specific regularization coefficients, fixed before full-model evaluation. The objectives normalize reconstruction and effective-weight terms by their Gram-matrix curvature, making the coefficients less dependent on their raw numerical scales. Numerical damping is separate: 10−410^{-4} for single-layer solves and 10−610^{-6} for attention and downstream teacher solves.

We added a sensitivity experiment on Ministral layer 12, independently halving and doubling each of the five regularization coefficients while keeping all other settings fixed. Across ten perturbations, complete-block MSE on 32 independent held-out conversations changes by −2.34% to +1.53% relative to the default. This shows modest local sensitivity around the chosen values.

Block size. We choose block size 32 as the largest mixing width that fits efficiently within our current fused epilogue design. A width of 16 restricts channel mixing, whereas 64 requires a larger epilogue tile and greater on-chip resources, compromising the intended execution efficiency. Thus, 32 is a hardware-driven design choice.

The fixed-checkpoint comparison also confirms the value of mixing beyond diagonal scaling: block-32 reduces held-out KL from 0.38533 to 0.37289, a further 3.23% relative reduction, with MMLU increasing from 55.60% to 55.93%.

Output-only refitting without construction-phase requantization. Yes. Appendix B.6 already evaluates this simplified pipeline: starting from a fixed WUSH checkpoint, we fit only output transformations while preserving input transformations, weight codes, scales, and activation-quantizer settings. It reduces fitting-set KL by 10.85% and held-out KL by 8.63%.

Our additional paired Ministral INT4 experiment gives:

Fixed-checkpoint configurationHeld-out KL ↓MMLU ↑
Output refitting OFF0.4136953.38%
Block-32 output refitting ON0.3728955.93%

Thus, output-only refitting improves both endpoints without requantization. Full TReQ’s submitted MMLU score is 61.79%, compared with 55.93% for this simplified pipeline. Full TReQ additionally optimizes construction and changes quantized weights, whereas the fixed-checkpoint experiment isolates output refitting.

Fitting versus held-out generalization. Appendix B reports the following full-pipeline KL reductions relative to WUSH on separate fitting and held-out conversations; internal-validation conversations are excluded.

ModelFormatFitting KL reductionHeld-out KL reduction
MinistralINT428.97%27.47%
LlamaINT457.31%59.11%
Qwen3INT432.58%36.53%
MinistralMXFP45.24%5.01%
LlamaMXFP4−5.83%−2.49%
Qwen3MXFP4−11.99%−12.97%

Positive values indicate improvement. INT4 gains transfer to held-out conversations across all three models. MXFP4 behavior is also broadly consistent across splits: Ministral improves on both, while Qwen3 regresses on both. These observations distinguish fitting-set overfitting from the reconstruction–model-quality mismatch.

Our frozen-FlatQuant extension provides further evidence: output fitting reduces independently held-out per-layer MSE by 7.28%, yet produces no statistically detected KL or MMLU improvement. Local reconstruction gains can therefore generalize to unseen inputs without yielding better model-level quality; understanding this disconnect remains an open question.

I thank the authors for their response. However, some of my concerns remain, such as the complexity of the offline calibration pipeline and the inconsistent performance across the FP4 formats (MXFP4/NVFP4). I therefore maintain my original score.

Reviewer_PYqhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TReQ, a W4A4 post-training quantization method that refits block-diagonal output transformations while keeping quantized weights fixed. It combines WUSH-based transformation initialization, two-sided GPTQ, and attention-aware reconstruction. Experiments show substantial improvements over WUSH under INT4, while gains under MXFP4 and NVFP4 are less consistent.

Critical strengths
  1. Block-diagonal output transformations enable more flexible error correction than channel-wise scaling.
  2. TReQ achieves consistent INT4 improvements across three models and five downstream benchmarks.
Critical weaknesses
  1. The paper does not isolate the contributions of block-diagonal channel mixing, alternating optimization, two-sided GPTQ, and attention-aware refitting. In particular, a diagonal versus block-diagonal comparison is missing.

  2. Experiments primarily compare against WUSH. Additional comparisons with existing W4A4 methods, such as FlatQuant [1] and MR-GPTQ [2], would better establish TReQ's competitiveness.

  3. Despite lower block reconstruction MSE, MXFP4 and NVFP4 sometimes show worse perplexity or downstream accuracy.

  4. Output transformations increase effective bits per weight, while the incremental calibration cost relative to WUSH is not quantified.

  5. Related methods such as GuidedQuant [3], YAQA [4], and KronQ [5] also incorporate output-side sensitivity into quantization, although their objectives and optimization procedures differ. The paper should clarify this distinction and discuss the implications for quantization error compensation.

[1] Sun, Yuxuan, et al. "Flatquant: Flatness matters for llm quantization." arXiv preprint arXiv:2410.09426 (2024).

[2] Egiazarian, Vage, et al. "Bridging the gap between promise and performance for microscaling fp4 quantization." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Kim, Jinuk, et al. "GuidedQuant: Large language model quantization via exploiting end loss guidance." arXiv preprint arXiv:2505.07004 (2025).

[4] Tseng, Albert, Zhaofeng Sun, and Christopher De Sa. "Model-preserving adaptive rounding." arXiv preprint arXiv:2505.22988 (2025).

[5] Lee, Donghyun, et al. "KronQ: LLM Quantization via Kronecker-Factored Hessian." arXiv preprint arXiv:2607.07964 (2026).

Questions for authors
  1. Could the authors provide end-to-end decoding throughput measurements, including input transformations and activation quantization?

  2. How does TReQ perform at lower bit-widths, such as W2 or W3 quantization?

  3. How sensitive is TReQ to the output transformation block size (ex, 8, 16, 32, or 64) in terms of accuracy, memory overhead, and latency?

2 comments

Thank you for your feedback.

Diagonal scaling versus block-diagonal mixing. We ran the requested three-way comparison on one fixed Ministral-3-8B INT4 WUSH checkpoint. We fitted the diagonal and 32×3232\times 32 transformations under their respective constraints using the same refitting protocol.

Output transformationHeld-out Tulu KL ↓MMLU ↑
Identity (OFF)0.4136953.38%
Diagonal0.3853355.60%
Block-320.3728955.93%

Diagonal scaling already helps. Block-32 gives a further 3.23% relative KL reduction over diagonal scaling, and the MMLU point estimate is 0.33 percentage points higher. This isolates the additional value of channel mixing on this checkpoint. We additionally ran a cumulative component ablation on layer 12 of Ministral to separate out the contribution of different components.

Cumulative configurationHeld-out relative MSE ×10⁻³ ↓
Two-sided WUSH initialization + RTN13.709
+ Projection output fitting11.507
+ Four alternating construction rounds11.579
+ GPTQ10.485
+ Attention-aware Q/K/V refitting8.404
+ O/down teacher refitting7.826
Overall, the pipeline reduces held-out block MSE on layer 12 by 42.9%, despite a small 0.63% intermediate increase when adding alternating construction under this setup. Appendix B.6 provides fixed-checkpoint evidence for the output refit as a whole.

External baseline and applicability beyond WUSH We did a FlatQuant W4A4 adaptation for Ministral, obtaining 0.04908 held-out KL and 74.23% MMLU on identical trajectories and 14042 questions used in our fixed-checkpoint comparison. FlatQuant substantially outperforms and exceeds full TReQ's submitted MMLU score of 61.79%. To examine whether output transformations are useful beyond WUSH, we froze all original FlatQuant quantization parameters and fitted additional block-diagonal output maps against block reconstruction MSE. On 32 independent held-out conversations, these maps reduce mean per-layer MSE by 7.28%. However, model-level KL changes from 0.04908 to 0.04972 and MMLU from 74.23% to 74.29%, with no statistically detected improvement on either endpoint. These results show that the output-transformation parameterization can be adapted to another quantization method and improve held-out local reconstruction, however, why such gain does not yield lower end-to-end KL or higher accuracy remains an open question.

Calibration and storage costs. On NVIDIA GH200 GPUs, constructing the Ministral-3-8B INT4 with WUSH takes 0.22 GPU-hours across all 34 blocks. Fitting block-32 output transformations on this fixed checkpoint requires 2.12 additional GPU-hours, without updating its quantized weights or scales. These timings exclude initial model loading and shared calibration-data collection. Including scales and transformations, storage increases from 4.110 bits per weight for WUSH to 4.230 with the added output transformations; the full TReQ configuration uses 4.356 bits per weight (Appendix B.7).

Difference with related works. GuidedQuant uses end-loss gradients to guide quantization; YAQA uses model-output curvature to guide adaptive rounding; KronQ uses activation and gradient covariance for quantization and precision allocation. TReQ's fixed-checkpoint refit acts at a different stage: it fits an explicit post-GEMM map while leaving the already selected codes and scales intact. We will clarify this distinction.

Lower bit widths. We focus on four-bit formats because our kernels load packed four-bit operands and consume them directly with native Tensor Core instructions. Compact W2/W3 storage would require additional unpacking or dequantization into a supported compute format before GEMM, changing the execution pipeline and its performance trade-offs; we have not evaluated those configurations.

Decoding performance. For decoding, Appendix B.8, Table 15 reports B200 W4A8 projection-operator latency for 1–16 input rows, including input transformation, activation quantization, GEMM, and output transformation. Adding the output transformation costs 0.23–0.42 μs in the GEMM kernel, or 2.8–8.6% of total operator latency. These are complete projection-operator measurements; full-model autoregressive decoding throughput remains unmeasured.

Block size. We choose block size 32 as the largest mixing width that fits efficiently within our current fused epilogue design. A width of 16 restricts channel mixing, whereas 64 would require a larger epilogue tile and greater on-chip resources, compromising the intended execution efficiency. Thus, 32 is a hardware-driven design choice rather than an empirically established accuracy optimum.

Thank you for the responses and additional experiments. However, the new FlatQuant comparison shows that FlatQuant substantially outperforms TReQ (74.23% vs. 61.79% MMLU). Moreover, adding TReQ’s output transformations to FlatQuant does not improve end-to-end KL or accuracy, and channel mixing gives only marginal gains over diagonal scaling. I therefore maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.