acceptodds
Withdrawn from ICLR 2027

GLF-Q: Global-Local Feature-based Quantization for Vision Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

23 views

Abstract

Post-training quantization (PTQ) efficiently compresses Vision Transformers (ViTs) without retraining, yet suffers severe accuracy degradation at low bit-widths. Existing optimization-based PTQ methods guide block reconstruction via either soft logits or second-order Hessian proxies. Logit supervision is prone to overfitting on limited calibration data, while Hessian approximations incur structural truncation errors. To address these limitations, we propose GLF-Q, a novel PTQ framework guided by Global-Local Feature alignment. GLF-Q propagates quantized block outputs through downstream full-precision layers to align penultimate-layer representations under local output regularization, providing downstream feature supervision without explicitly approximating the Hessian or using a Taylor expansion. Furthermore, offline Hadamard transformations are introduced with zero runtime overhead to disperse activation outliers across channels, effectively contracting dynamic ranges and reducing quantization errors. Meanwhile, optimizing this loss via a Straight-Through Estimator (STE) achieves rapid convergence, bypassing continuous relaxation rounding formulations such as AdaRound. Extensive experiments across representative ViT architectures demonstrate that GLF-Q with standard uniform quantizers substantially outperforms state-of-the-art methods under 3-bit quantization on image classification. In addition, GLF-Q exhibits strong out-of-domain calibration robustness and achieves speedups under 8-bit GPU deployment.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Model Compression, Post-training quantization, Vision Transformer

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PkiZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GLF-Q, a block-wise post-training quantization framework for Vision Transformers. The authors argue that logit-guided block reconstruction tends to overfit the limited calibration data, and that Hessian-guided block reconstruction suffers from approximation errors at low bit-widths.

The framework consists of three components. First, the Global-Local Feature (GLF) loss propagates the output of the quantized block through the frozen full-precision downstream layers and aligns the representation immediately before the task head, together with a local MSE on the block output as regularization. Second, offline Hadamard rotations are applied after converting LayerNorm into RMSNorm, which spreads activation outliers across channels without runtime overhead. Third, GELU is replaced with ReLU followed by MLP reconstruction. The weights and quantization scales are then optimized with STE for 3k iterations per block.

The method is evaluated on ImageNet with ViT, DeiT, and Swin at W3A3 and W4A4, and on COCO object detection and instance segmentation at W4A4. The authors report state-of-the-art results with standard uniform quantizers, particularly at W3A3. Additional experiments include ablations on the loss and each component, out-of-domain calibration with SUN397, 8-bit inference latency on TensorRT, and reconstruction time.

Critical strengths
  1. Strong W3A3 accuracy with uniform quantizers. GLF-Q achieves the best W3A3 accuracy on all seven ImageNet models while using only standard uniform quantizers, and the results are stable across seeds (Table 10). This is the main practical value of the paper for hardware-friendly deployment.
  2. Reconstruction efficiency. Most existing reconstruction methods require 20k iterations, whereas GLF-Q converges within 3k iterations. As a result, it is 2.0–2.6× faster than FIMA-Q in total training (reconstruction) time (Table 7). This matters for PTQ, where low calibration cost is the main advantage over QAT.
Critical weaknesses
  1. Limited novelty. The GLF-Q framework is a combination of existing techniques: downstream supervision with local regularization from PD-Quant [1], offline rotation from QuaRot [2] and SliceGPT [3], and MLP reconstruction from APHQ-ViT [4]. GLF differs from PD-Quant mainly in moving the supervision point from the logits to the penultimate-layer representation. The paper does not present any new analysis, observation, or motivation of its own. The stated motivation for each component largely restates that of the original work. As a result, the state-of-the-art performance appears to come from stacking existing methods rather than from a new idea.
  2. Insufficient contribution of the GLF loss. As the name GLF-Q suggests, GLF is presented as the main contribution, so it should account for a substantial part of the performance gain. However, the ablation in Table 4 shows the opposite. Without Hadamard rotation and MLP reconstruction, GLF alone performs worse than FIMA-Q [5] on six of the seven models, even though FIMA-Q uses neither of these components. On ViT-S, for example, GLF alone reaches only 60.37%, while FIMA-Q achieves 64.09%. Even with MLP reconstruction, which APHQ-ViT also uses, GLF performs only comparably to APHQ-ViT and is clearly worse on ViT-B. These results suggest that the state-of-the-art performance comes mainly from the borrowed components rather than from GLF itself.

[1] Liu, Jiawei, et al. "Pd-quant: Post-training quantization based on prediction difference metric." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.

[2] Ashkboos, Saleh, et al. "Quarot: Outlier-free 4-bit inference in rotated llms." Advances in Neural Information Processing Systems 37 (2024): 100213-100240.

[3] Ashkboos, Saleh, et al. "Slicegpt: Compress large language models by deleting rows and columns." International Conference on Learning Representations. Vol. 2024. 2024.

[4] Wu, Zhuguanyu, et al. "Aphq-vit: Post-training quantization with average perturbation hessian based reconstruction for vision transformers." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[5] Wu, Zhuguanyu, et al. "FIMA-Q: Post-training quantization for vision transformers by fisher information matrix approximation." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors
  1. What limitation of ViT PTQ does this work address? The paper identifies two limitations. The first is that logit-guided reconstruction tends to overfit the limited calibration data, and Hessian-guided reconstruction suffers from approximation errors. The second is that activation outliers and the negative tail of GELU waste the quantization range. However, the overfitting of global supervision was already identified and addressed by PD-Quant, activation outliers by QuaRot, and the negative tail of GELU by APHQ-ViT. Which of these limitations is not already addressed by existing methods? If the existing methods have not fully resolved them, the authors should point out where these methods fall short. Could the authors clarify the specific problem in ViT PTQ that this work aims to solve?
  2. What is the novelty of this work? The GLF-Q framework combines downstream supervision with local regularization from PD-Quant, offline rotation from QuaRot and SliceGPT, and MLP reconstruction from APHQ-ViT. Could the authors clarify how their work differs from these methods in terms of observation, motivation, analysis, and proposed method? In particular, PD-Quant already identified the overfitting of global supervision and addressed it with local regularization. What new insight does GLF provide beyond moving the supervision point from the logits to the penultimate-layer representation?
  3. Can the authors provide an analysis of the approximation errors in Hessian-guided block reconstruction? In reconstruction-based methods such as BRECQ [6], QDrop [7], APHQ-ViT, and FIMA-Q, it is widely accepted that Hessian approximations introduce structural and truncation errors. Although the paper presents this as one of its main problems, it does not provide any analysis to support it. For example, how large is the gap between the loss change predicted by the second-order approximation and the actual loss change at low bit-widths?
  4. How sensitive is GLF to the size of the calibration set? Since the main motivation is overfitting on limited calibration data, an evaluation with different calibration set sizes is needed. It would be helpful to show how the gap between GLF and logit-guided block reconstruction changes as the calibration set becomes smaller.
  5. Can GLF be extended to dense prediction tasks? Since GLF supervises the representation immediately before the task head, extending it to other tasks with ViT backbones would be a natural direction. For dense prediction tasks such as depth estimation and super-resolution, however, there is no single global representation like the class token. It would be interesting to see how the global supervision target should be defined for such tasks and whether GLF remains effective on them.

[6] Li, Yuhang, et al. "Brecq: Pushing the limit of post-training quantization by block reconstruction." arXiv preprint arXiv:2102.05426 (2021).

[7] Wei, Xiuying, et al. "Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization." arXiv preprint arXiv:2203.05740 (2022).

Reviewer_czvNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GLF-Q, a post-training quantization framework for Vision Transformers. Its main idea is a Global-Local Feature (GLF) loss that combines penultimate-layer feature alignment through a frozen full-precision downstream network with local block reconstruction regularization. The framework additionally incorporates offline Hadamard transformations, MLP reconstruction, and STE-based optimization. Experiments across ViT, DeiT, and Swin architectures demonstrate improvements in low-bit quantization accuracy, calibration robustness, and reconstruction efficiency.

Critical strengths

1. Consistent accuracy improvements. GLF-Q achieves consistent improvements across multiple Vision Transformer architectures under 3-bit and 4-bit uniform quantization.

2. Extensive experimental validation. The paper evaluates multiple architectures and tasks, including ablation studies, out-of-domain calibration, and inference efficiency.

Critical weaknesses

1. Limited methodological novelty. The proposed framework largely builds on existing techniques, including global-local reconstruction similar to PD-Quant [1] and MLP reconstruction from APHQ-ViT [2]. The offline orthogonal reparameterization builds upon established techniques such as SliceGPT [3] and QuaRot [4], while its adaptation to Vision Transformers does not appear to introduce a fundamentally new transformation mechanism. The paper also overlooks MGRQ [5], which combines Extra-Block Global Supervision (EBGS) with local feature reconstruction for ViT quantization. Although the reported accuracy improvements are consistent, the paper offers limited methodological insights beyond modifying and combining existing approaches.

2. Insufficient analysis of the main contribution. Given the limited novelty discussed above, the primary contribution appears to be replacing the prediction-level supervision used in PD-Quant [1] and MGRQ [5] with penultimate feature matching. Rather than primarily emphasizing performance gains from combining existing techniques, the paper should provide a deeper analysis of why this modification is effective and what new insights it offers. Although Table 3 and Appendix D demonstrate accuracy and generalization improvements, they do not sufficiently explain why feature-level supervision better addresses calibration overfitting. A more comprehensive analysis, particularly in comparison with PD-Quant and MGRQ, should be central to the paper to establish its methodological significance.

Questions for authors

Q1. How does the proposed Global Feature Loss compare directly with the Extra-Block Global Supervision (EBGS) in MGRQ [5]? What advantages does penultimate feature matching provide over EBGS under identical quantization and reconstruction settings?

Q2. Given that SliceGPT [3] already introduces LayerNorm-to-RMSNorm conversion and offline orthogonal reparameterization, and QuaRot [4] employs Hadamard rotations for quantization, what specific contributions does GLF-Q introduce beyond adapting these existing techniques to Vision Transformers?

Q3. Why does penultimate feature matching improve accuracy and generalization over prediction-level supervision? What theoretical or mechanistic insights explain its effectiveness, particularly in mitigating calibration overfitting?


References

[1] Liu et al., PD-Quant: Post-Training Quantization Based on Prediction Difference Metric, CVPR 2023.

[2] Wu et al., APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers, CVPR 2025.

[3] Ashkboos et al., SliceGPT: Compress Large Language Models by Deleting Rows and Columns, ICLR 2024.

[4] Ashkboos et al., QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs, NeurIPS 2024.

[5] Yang et al., MGRQ: Post-Training Quantization for Vision Transformer with Mixed Granularity Reconstruction, ICIP 2024.

Reviewer_YWNROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GLF-Q, a post-training quantization framework for Vision Transformers. Its central component is a global–local feature reconstruction objective: during block-wise reconstruction, the current quantized block’s output is propagated through the frozen full-precision downstream network to match pre-classifier features, while local block-output matching serves as a regularizer. The framework further incorporates offline Hadamard rotations to mitigate activation outliers, MLP reconstruction that replaces GELU with ReLU, and STE-based optimization of weights and quantization scales. Evaluations on ImageNet classification and COCO object detection and instance segmentation report improved low-bit accuracy with standard uniform quantization, particularly for 3-bit classification, alongside results on out-of-domain calibration, reconstruction time, and INT8 deployment efficiency.

Critical strengths
  1. Strong quantization accuracy and deployment compatibility. The method achieves strong low-bit accuracy using standard uniform quantizers, without requiring specialized non-uniform quantization formats. Among the compared methods, GLF-Q achieves the best ImageNet classification results across multiple Vision Transformer architectures, with consistent gains particularly under 3-bit quantization.

  2. Broad empirical evaluation and ablation analysis. The evaluation covers ImageNet classification and COCO object detection and instance segmentation, across multiple architectures and quantization bit-widths. Extensive ablation studies further demonstrate the effectiveness of the proposed method.

Critical weaknesses
  1. Limited novelty of the core objective. The central design of GLF is a direct weighted combination of MSE on pre-classifier features and MSE on local block outputs. Relative to PD-Quant, the main change is to replace global prediction-distribution matching with pre-classifier feature matching. Although the experiments demonstrate its effectiveness, the objective largely amounts to a simple combination of two conventional feature-matching losses, with limited methodological novelty beyond changing the supervision location and distance metric.

  2. Insufficient justification of the loss design and interaction between its terms. APHQ-ViT and FIMA-Q demonstrate advantages of Hessian-guided reconstruction over plain MSE in their evaluated settings. However, the paper does not sufficiently explain why plain MSE remains the preferred local regularizer when global feature supervision is introduced. Table 3 compares DPLR-FIM, Logits + LF, and GLF, but does not evaluate global feature alignment combined with Hessian-weighted local reconstruction. This comparison would help establish whether global feature supervision and curvature weighting are complementary.

  3. Unclear organization of the method section. “Global-Local Feature Loss” and “GLF-Q Framework” are presented as parallel subsections, while the latter contains rotation, MLP reconstruction, and STE-based optimization. This organization makes the relationship between the central objective and the overall pipeline less immediately clear.

  4. The independent contribution of STE is not fully established. Table 3 isolates the effect of the reconstruction loss under STE and includes a FIMA-Q + MR + Hadamard baseline using AdaRound. However, it does not directly compare STE and AdaRound under the same GLF objective. Consequently, it remains unclear whether STE offers an accuracy or convergence advantage over AdaRound when optimizing GLF. A comparison between GLF + STE and GLF + AdaRound, with other components held fixed, would help clarify this point, particularly through accuracy curves against both iteration count and wall-clock time.

  5. The sources of inference speedup are not sufficiently disentangled. Table 6 demonstrates INT8 inference speedups over FIMA-Q, but it remains unclear how much comes from replacing GELU with ReLU, replacing LayerNorm with RMSNorm, or other deployment differences. Since GELU-to-ReLU MLP reconstruction is already used in APHQ-ViT, Table 6 should include APHQ-ViT under the same hardware and TensorRT settings, alongside separate evaluations of activation and normalization replacements. This would clarify the deployment benefits of the complete framework and the contribution of its new design choices.

Overall, the paper reports strong empirical results, but the method primarily combines existing techniques, with insufficient explanation of the new design choices and limited isolation of their individual contributions. Consequently, the current manuscript reads more like a technical report integrating and benchmarking existing techniques than a research paper demonstrating the methodological novelty or insight that I would expect for acceptance at ICLR.

Questions for authors

Please refer to the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.