- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TReQ, a post-training quantization method that improves weight–activation quantization by introducing block-diagonal output transformations. Unlike conventional approaches that primarily optimize input transformations or quantized weights, TReQ refits output transformations while keeping quantized weight codes and scales fixed. The method further incorporates attention-aware reconstruction to reduce errors in attention scores, contexts, and downstream projections.
Critical strengths
- The paper addresses an important problem in low-bit LLM quantization: reducing accumulated quantization errors without modifying existing quantized weights and scales.
Critical weaknesses
- To the best of my knowledge,the method largely builds on existing transformation-based quantization and reconstruction techniques, particularly WUSH and TurboBoA. While block-diagonal output refitting provides additional flexibility, the conceptual advance beyond these methods appears incremental.
- The main experiments primarily compare against WUSH, without direct comparisons to other competitive quantization approaches under comparable conditions.
Questions for authors
- Why do substantial reductions in block reconstruction error fail to consistently improve downstream performance under MXFP4 and NVFP4? Does this indicate a limitation of the reconstruction objective?
- Could the authors provide a component-wise ablation separating two-sided transformation construction, output refitting, and attention-aware reconstruction? Which component contributes most to the final improvements?
- On the same fixed quantized checkpoint, how much improvement comes from per-channel diagonal scaling versus block-diagonal mixing? Please keep weight codes, scales, input transformations, data splits, and fitting procedures identical, and report held-out KL divergence and representative task accuracy.
- Could the authors provide pseudocode for the complete algorithm, clearly indicating when input transformations, output transformations, quantized weight codes, and scales are updated or frozen at each stage? In particular, please clarify which steps involve requantization to facilitate reproducibility.
2 comments
Thank you for your detailed response. However, I still consider the novelty of this manuscript to be insufficient, and therefore I maintain my original score.
Thank you for your feedback.
Contribution beyond WUSH and TurboBoA. TReQ introduces an explicit block-diagonal output map that remains adjustable after quantization. WUSH provides the input-transform construction, while TurboBoA’s attention-wise refinement updates per-output-channel scales with integer codes fixed, corresponding to diagonal output scaling. Our output transformations additionally mix channels within each block while keeping the original weight codes and quantization scales intact. We fit these maps against projection outputs, attention scores, and attention context. The fixed-checkpoint comparison below directly measures the additional value of channel mixing.
External baseline and applicability beyond WUSH. We added a FlatQuant W4A4 adaptation for Ministral, obtaining 0.04908 held-out KL and 74.23% MMLU on the identical trajectories and 14,042 questions used in our fixed-checkpoint comparison. FlatQuant substantially outperforms these fixed-checkpoint arms and exceeds full TReQ’s submitted MMLU score of 61.79%. We will report this result and narrow our claim to the contribution of output refitting.
To examine applicability beyond WUSH, we froze all original FlatQuant quantization parameters and fitted additional block-diagonal output maps using gradient-based block-MSE fitting. On 32 independent held-out conversations, these maps reduce mean per-layer MSE by 7.28%. However, model-level KL changes from 0.04908 to 0.04972 and MMLU from 74.23% to 74.29%, with no statistically detected improvement on either endpoint. These results show that output transformations can improve held-out local reconstruction with another quantization method, while also supporting your concern about the reconstruction objective: lower local MSE does not guarantee lower end-to-end KL or higher accuracy. Why these gains fail to improve model-level quality, including in the submitted MXFP4/NVFP4 results, remains an open question.
Component-wise ablation. We ran a cumulative component ablation on layer 12 of Ministral-3-8B INT4, keeping grouping, quantization settings, input statistics, and data splits fixed. On 32 independent held-out conversations, complete-block relative MSE, including residuals, is:
Overall, the pipeline reduces held-out block MSE on layer 12 by 42.9%, despite a small 0.63% intermediate increase when adding alternating construction in this specific setup. Attention-aware Q/K/V refitting gives the largest individual reduction in this sequence (19.84%). Appendix B.6 provides fixed-checkpoint evidence for the output refit as a whole.
Diagonal scaling versus block-diagonal mixing. We ran the requested three-way comparison on one fixed Ministral-3-8B INT4 WUSH checkpoint, keeping weight codes, scales, input transformations, and data splits identical. We fitted diagonal and 32 × 32 transformations under their respective constraints using the same refitting protocol.
Diagonal scaling already helps. Block-32 gives a further 3.23% relative KL reduction over diagonal scaling, and the MMLU point estimate is 0.33 percentage points higher. This isolates the additional value of channel mixing on this checkpoint.
Algorithm and parameter updates. The update schedule is:
Requantization occurs only during construction and the final GPTQ pass; subsequent output refits preserve the quantized checkpoint.