acceptodds
Under review as a conference paper at ICLR 2027

Flow-Q: Flow Consistency for Post-Training Quantization of Vision Transformers

Abstract

We present Flow-Q, a general post-training quantization framework for Vision Transformers. Our analysis shows that the existing sensitivity-weighted reconstruction paradigm induces feature distortion during quantization, degrading the robustness of quantized feature representations. To overcome this challenge, we first propose an evolution-aware learning (EAL) strategy that captures feature evolution information to construct a manifold representation with global topological structure. Moreover, a geometric constraint rectification (GCR) module is designed to utilize the manifold representation to rectify quantized features from both direction and magnitude perspectives, thereby alleviating feature distortion. EAL provides the geometric structural foundation for GCR, enabling the quantized model to inherit the robustness of visual representations from the full-precision network. Extensive experiments on nine tasks across twelve benchmarks demonstrate the strong generalizability of Flow-Q, which achieves new state-of-the-art quantization performance across DINOv3, conventional ViTs, and mobile-efficient models.

Primary area:
optimization
Keywords:
Post-Training Quantization, Vision Transformers

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YG9oOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Flow-Q, an auxiliary objective for Hessian-based block-wise reconstruction in post-training quantization of Vision Transformers. It aims to preserve full-precision representation quality, which the authors argue sensitivity-weighted reconstruction alone fails to do. Flow-Q builds a sensitivity-weighted Gram matrix from FP feature evolution and takes its top-µ SVD subspace. It then aligns the quantized and FP feature evolutions in this subspace with a direction loss and a magnitude loss. Integrated into APHQ-ViT and LS-ViT, it shows consistent gains on quantized DINOv3 across dense downstream tasks, on ImageNet classification with ViT/DeiT/Swin, and on mobile-efficient models at low bit-widths.

Critical strengths
  1. A novel perspective on the quality of quantized feature representations. This work raises a question about the quality of the quantized feature representation itself, offering a new perspective to existing quantization research.
  2. Extensive experiments demonstrating generalizable performance. This work achieves strong and generalizable performance across 9 tasks and 12 benchmarks that use vision transformers as the backbone.
Critical weaknesses

Incomplete description of the method. The paper does not describe how Ldirect\mathcal{L}_{direct} and Lmagn\mathcal{L}_{magn} are specifically used in the reconstruction. By examining the supplementary material, I confirmed that Eq. (5) and Eq. (6) are added to the reconstruction loss of the baseline method into which Flow-Q is integrated (e.g., LS-ViT in the provided code). I suggest explicitly stating this in the Geometric Rectification paragraph of the paper.


Minor Weaknesses (These points do not affect my evaluation of the paper, but are suggested to improve its overall quality.)

  1. In the Implementation Details paragraph of Section 4, the paper states: "The learning rates for weight and activation quantization are set to 1e-3 and 4e-5, respectively." - This expression is inaccurate. These are not learning rates for quantization itself, but learning rates used in the reconstruction for optimizing the quantization parameters.

  2. In Tables 1 and 2, the evaluation metrics for each task are described in the Evaluation Metrics paragraph rather than in the tables themselves, which reduces readability. If the paper is accepted and additional space becomes available in the camera-ready version, I suggest indicating the metric for each task directly in the tables.

Questions for authors

Q1. Why was Flow-Q not applied to FIMA-Q? Can the authors apply Flow-Q to FIMA-Q as well, to demonstrate its generalizability?

Q2. Section 3.1 states that sensitivity does not reflect feature evolution well, whereas Section 3.2 states that sensitivity indicates the importance of the corresponding evolution information. These two statements seem contradictory to me, though I may have misunderstood. Please clarify how they are consistent.

Q3. From the provided code, it appears that attention quantization is not applied to DINOv3, whereas prior ViT PTQ works typically quantize the attention modules. Is there a reason the authors did not apply attention quantization to DINOv3? Were the compared methods also evaluated under the same quantization setting?

Q4. From the provided code, flow_loss appears to be added to the optimization objective of LS-ViT. Could the authors provide an ablation on the coefficient multiplied by flow_loss to show the sensitivity to this value?

Q5. Could the authors provide an ablation study on the size of the optimization set?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank reviewer YG9o for the valuable time and constructive feedback. Since the issues in "Critical Weaknesses" and "Questions for Authors" are different, we renumber the questions sequentially from Q1 to Q8, followed by our point-by-point responses below.

Q1: Incomplete description of the method in the Geometric Rectification paragraph.

A1: Thank you for your insightful question. We have added an explanation of how Ldirect and Lmagn\mathcal{L}_{\mathrm{direct}}\text{ and }\mathcal{L}_{\mathrm{magn}} are used in reconstruction to the Geometric Rectification paragraph (§3.3\S\color{red}3.3). For ease of review, we present it as follows:

  • Flow-Q can be seamlessly integrated into the existing work through L=Lhw+α(Ldirect+Lmagn)\mathcal{L} = \mathcal{L}_{hw} + \alpha (\mathcal{L}_{direct} + \mathcal{L}_{magn}), where Lhw\mathcal{L}_{hw} denotes the Hessian-weighted reconstruction loss in the current methods (Wu et al., 2025b;a; Hwang et al., 2026). An ablation study on parameter α\alpha is provided in §4.2\S\color{red}4.2.

Q2: An expression error in the implementation details paragraph.

A2: Thank you for pointing this out. We have corrected this in the implementation details paragraph (§4\S\color{red}4). For ease of review, we present it as follows:

  • The learning rates used to optimize the quantization parameters of weight and activation in the reconstruction are set to 1e-3 and 4e-5, respectively.

Q3: Adding the metrics to Tables 1 and 2 improves readability.

A3: We fully agree with your point. The metrics for each task have been added to Tables 1\color{red}1 and 2\color{red}2 to improve readability.

Q4: Applied to FIMA-Q to demonstrate generalization.

A4: Thank you for your question. We integrated our method into the FIMA-Q framework and validated its generalization on DINOv3 and conventional ViTs under specific quantization settings. The experimental results are shown in the table below:

MethodW/ASemantic Matching ([email protected])Image Retrieval (mAP)Saliency Detection (FmaxF_{max})Multi-view Corresp. (mRec)Video Object Segmentation ((J\\&F)_m / J_m / F_m)
FIMA-Q4/483.27 / 65.81 / 50.3142.7656.10 / 41.2647.8741.43 / 40.02 / 42.84
+ Ours4/484.46 / 66.39 / 51.3444.2556.71 / 41.6949.2342.77 / 42.24 / 43.30
MethodW/AViT-SViT-BDeiT-SDeiT-BDeiT-LSwin-SSwin-BAvg.
FIMA-Q2/338.8261.1552.5766.2476.2559.5763.1559.68
+ Ours2/340.3962.2653.8867.3577.2160.9564.9961.00
FIMA-Q2/456.7674.4165.3873.3280.6670.6472.3670.50
+ Ours2/458.2775.2966.3174.1181.3871.7973.9771.59

The above experimental results have been added to §4.1\S\color{red}4.1 of the revision and highlighted for clarity. Due to the limited time available for the current experiments, we plan to conduct a more comprehensive evaluation in future work by integrating our method into the FIMA-Q framework under various quantization settings on DINOv3, conventional ViTs, and mobile-efficient networks.

Q5: Inaccurate description.

A5: We apologize for the confusion. This was an inaccurate statement made during the writing process. We have corrected it in §3.2\S\color{red}3.2 of the paper, with the changes highlighted in blue. For ease of review, we present it as follows:

  • The Hessian matrix H inherently characterizes the sensitivity of each position to quantization errors, indicating the importance of the corresponding local positions.

Q6: Attention quantization and experimental settings.

A6: We thank the reviewer for this reminder. First, we ensure consistent quantization settings across all methods on quantized DINOv3. Second, our public code applies partial attention quantization to the QKV and output projections (weights and input activations), while leaving the Q–K multiplication, softmax, and attention–V multiplication unquantized. The reasons are as follows:

  • Trade-off. Unlike prior ViT PTQ works, which typically use CLS features for global tasks such as image classification, DINOv3 focuses on patch features for downstream dense prediction tasks (e.g., segmentation), where attention plays a critical role. In our preliminary experiments, fully quantizing the attention modules resulted in substantial performance degradation. Therefore, we adopt partial attention quantization to achieve a trade-off between quantization efficiency and downstream task performance.

All methods share the same DINOv3 quantization settings for fairness.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q7: Sensitivity analysis of the loss coefficient.

A7: Thank you for spotting this. We have added an ablation study on the loss coefficient α\alpha in §4.2\S\color{red}4.2 of the paper. The experimental results show that the model achieves its best performance when α\alpha is set to 0.01.

α\alpha0.0010.0050.010.020.05
Accuracy54.4154.7754.8954.8154.60

Q8: Ablation study on the size of the optimization set.

A8: Apologies for this missing ablation study. We define the size of the optimization set as S\mathcal{S} and add an ablation study on different optimization set sizes under the W2A3 quantization setting on DeiT-S, together with the corresponding experimental analysis, in §4.2\S\color{red}4.2. For a fair comparison, we follow the settings of FIMA-Q, APHQ-ViT, and LS-ViT, and set the size of the optimization set to 1024.

S\mathcal{S}768896102411521280
Baseline52.6953.0053.8654.0554.63
+ Ours53.9254.2754.8955.2455.60

Thank you for your valuable time and constructive feedback. We have revised the manuscript with more discussions and experimental results according to your comments. It is welcome if there are any further questions or suggestions.

I have carefully read the authors’ response, and all of my concerns have been addressed. The revised manuscript is more complete than the original submission. I believe this paper deserves acceptance, as it offers a new perspective on post-training quantization. I will maintain my score and wish the authors the best of luck.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you very much for your acknowledgment of our work. Your careful review has greatly helped improve the quality of our work. We sincerely appreciate your support and encouragement.

Reviewer_NbzCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FLOW-Q aims to preserve how features change between neighboring blocks after quantization. It extracts the main directions of change from a full-precision model, with sensitivity weighting, and trains the quantized model to follow similar directions and magnitudes. Adding it to APHQ-ViT and LS-ViT improves results across several models and tasks.

Critical strengths
  1. The experiments cover several models and tasks, with gains when the method is added to either baseline. The five downstream tasks requiring no additional training provide useful evidence that features are preserved (Tables 1–4 and S1).
  2. The full-precision model provides a fixed reference for optimizing the quantized model, making the method easy to add to existing approaches. Comparisons of direction selection, loss functions, and reference features support these choices (Tables 7–9).
Critical weaknesses
  1. The problem solved by the added constraint needs to be clearer. Existing methods already match full-precision features at each position and use sensitivity to weight the errors. The observation that similar sensitivities can accompany different feature changes does not, by itself, show that these differences are missed or explain why EAL is needed (§3.1).
  2. Large feature changes may not identify the most harmful quantization errors. Equations 2–4 retain a few directions based on the weighted size of full-precision feature changes. Discarded directions may still be sensitive to quantization, and GCR cannot constrain errors in those directions. The paper should explain this trade-off and whether the original reconstruction loss covers the gap.
  3. The claim of preserving geometric structure needs more explanation. Zero loss in Eqs. 5–6 does recover the projected features, but a small loss does not mean that every local region is well preserved. Table 8 shows that the method works, yet it does not explain whether the gains come from better structural preservation or from rescaling and reweighting reconstruction errors.
Questions for authors
  1. What exactly does weighted reconstruction miss, and how does EAL recover that information?
  2. Why prioritize directions with large feature changes? What constrains quantization errors in the directions that are left out?
  3. How does GCR preserve local structure beyond what can be achieved by changing the scale and weights of reconstruction errors?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We thank reviewer NbzC for the valuable time and constructive feedback. Since the issues raised in "Critical Weaknesses" correspond one-to-one with those in "Questions for Authors", we have numbered the questions sequentially from Q1 to Q3 and provide our point-by-point responses below.

Q1: Clarification of the problem addressed by the added constraint.

A1: We thank the reviewer for the valuable feedback. We fully agree that the problem addressed by the added constraint needs to be clarified. To make our response clearer, we divide the explanation into two aspects: the information missed by weighted reconstruction (Q1.1) and how EAL recovers the missing information (Q1.2).

Q1.1: What information is missing from weighted reconstruction?

A1.1: Thank you for pointing this out. Existing weighted reconstruction methods primarily focus on point-wise local numerical alignment, limiting their ability to capture rich contextual structural information for quantization. Consequently, quantized features cannot inherit the representational robustness of full-precision features, as demonstrated in Figure 2\color{red}2 (Left).

Q1.2: How EAL recovers the missing information?

A1.2: Thank you for your insightful comment. Figure 2\color{red}2 (Right) provides a quantitative analysis demonstrating that feature evolution offers complementary guidance for block reconstruction. However, feature evolution information remains discrete and point-wise. Therefore, transforming such discrete information into rich structural correlations is crucial, which motivates the design of EAL. Specifically, EAL achieves this through simple element-wise multiplication and Gram anchoring. As shown in Table 5\color{red}5 of the original paper, EAL delivers substantial performance gains with only 0.76% additional training time and 0.17 GB of extra GPU memory.

We have revised §3.1\S\color{red}3.1 of the paper to provide a clearer explanation.

Q2: Large feature changes may not identify the most harmful quantization errors.

A2: Thank you for this thoughtful question. We agree that large feature changes cannot always identify the most harmful quantization errors. Nevertheless, our method works collaboratively with the original reconstruction loss to jointly identify and mitigate these errors. To make our response clearer, we divide the discussion into two aspects: the prioritization of large feature changes (Q2.1) and the quantization constraints on discarded directions in our method (Q2.2).

Q2.1: Why prioritize directions with large feature changes?

A2.1:. Thank you for your insightful comment. In our work, prioritizing directions with larger feature changes enables more effective geometric correction while reducing redundant constraints. It also achieves a favorable trade-off among information preservation, computational efficiency, and model performance:

  • Better Geometric Rectification and Less Redundancy. Directions with larger feature changes typically capture more significant feature evolution information. By concentrating the limited correction capacity on these dominant directions, GCR prioritizes the recovery of important geometric changes during quantization, avoids redundant constraints on less informative directions, and better preserves geometric consistency between quantized and full-precision features. The table below validates our claim. Let aa and bb denote the pairwise feature distances in the full-precision and quantized feature maps, respectively. The relative pairwise distance error is defined as ∣a−b∣a\frac{|a-b|}{a}, measuring the relative distortion introduced by quantization. Structure preservation is the arithmetic mean of KNN@5, KNN@10, and KNN@20. Each KNN@k measures the overlap between the full-precision and quantized models’ k nearest patch-feature neighbors, averaged over patches and images.
SettingmIoU ↑Relative pairwise distance ↓Structure preservation ↑
Baseline42.0713.1973.02
+ GCR42.927.4477.83
  • Trade-off across Three Aspects. As shown in Table 10\color{red}10 (see below) of the original paper, CEVR denotes the cumulative explained variance ratio, measuring the proportion of original information preserved. The results demonstrate that, for a channel dimension of 384, emphasizing a small number of directions with larger feature changes achieves a trade-off among information preservation, efficiency, and performance.
μ\mu128144192256300360
CEVR94.3595.8098.4899.6299.8599.99
Time (ms)19.0122.2228.8336.1042.9447.46
W2A3 (Cls.)54.6154.7054.8954.9254.9354.93
W2A4 (Cls.)66.3066.4366.5066.5166.5266.53

We have revised §4.2\S\color{red}4.2 to provide a clearer explanation.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q2.2: What constrains quantization errors in the directions that are left out?

A2.2: Thank you for your constructive feedback. The directions discarded by our method remain constrained by the original reconstruction loss with varying sensitivity weights. Specifically, our method models feature directions with significant changes to provide rich contextual structural information and directional rectification for quantization. In contrast, the original reconstruction loss performs point-wise modeling across all directions with different sensitivity weights.

As demonstrated in Table 5\color{red}5 of the original paper, FLOW-Q and the original reconstruction loss work complementarily, with our method improving performance by 2.23% over the original reconstruction loss alone.

BaselineEALGCRAccuracyTime (min)GPU (GB)
√42.05105.33.87
✓✓42.75106.14.04
✓✓42.92115.94.78
✓✓✓44.28118.64.93

We have revised the corresponding content in §3.3\S\color{red}3.3 of the paper.

Q3: The claim of preserving geometric structure needs more explanation.

A3: Thank you for your insightful question. The small loss values do not necessarily guarantee that every local position is well preserved. The performance improvements brought by the GCR module genuinely stem from better structural preservation. We discuss these two aspects separately in Q3.1 and Q3.2 below.

Q3.1: A small loss does not mean that every local region is well preserved.

A3.1: We fully agree with your comment. The small losses resulting from Eqs. (5\color{red}5)–(6\color{red}6) indeed cannot guarantee that every local detail is well preserved. However, it is important to emphasize that our method is integrated as a plug-in into existing sensitivity-weighted reconstruction frameworks (i.e., LS-ViT and APHQ-ViT). Therefore, the resulting framework not only provides contextual modeling information for geometric correction during quantization but also retains the point-wise local correction mechanisms of the original methods. Geometric rectification and point-wise correction work collaboratively to mitigate quantization errors.

Q3.2: Does the performance improvement of GCR stem from better structural preservation?

A3.2: Thank you for raising this insightful question. The performance improvement brought by GCR stems from better structural preservation rather than merely changing the scale and weights of reconstruction errors.

  • Mechanism. The proposed GCR module performs geometric calibration within the subspace, using projected full-precision features to complementarily correct the directions and magnitudes of projected quantized features. This enables the quantized model to learn the geometric structures embedded in the full-precision features.

  • Validation. We conduct ablation experiments on the ADE20K semantic segmentation task under the DINOv3 W3A4 quantization setting to verify that GCR improves performance through better structural preservation. The metrics and settings in the table have already been explained in A2.1.

SettingmIoU ↑Relative pairwise distance ↓Structure preservation ↑
Baseline42.0713.1973.02
+ GCR42.927.4477.83

We have added the experiments and corresponding analysis to §4.2\S\color{red}4.2 of the revision.  


Thank you for your valuable time and constructive feedback. We have revised the manuscript with more discussions and experimental results according to your comments. It is welcome if there are any further questions or suggestions.

Dear Reviewer NbzC,

As the discussion period is approaching its end, we kindly ask whether our responses have addressed your concerns. We would be happy to provide further clarification if needed.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_dmszOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FLOW-Q, a post-training quantization framework that supplements sensitivity-based reconstruction with block-wise feature evolution matching. Evolution-aware learning constructs a sensitivity-weighted Gram matrix from full-precision block updates. Geometric constraint rectification then projects full-precision and quantized updates onto its dominant subspace and aligns their directions and magnitudes. The paper reports consistent improvements when integrated into APHQ-ViT and LS-ViT, covering DINOv3 downstream tasks, conventional vision transformers, and lightweight architectures. The empirical scope is a strength. However, the formulation leaves important implementation details unspecified, and the evidence does not fully establish the necessity of the proposed geometric interpretation over simpler feature-matching alternatives.

Critical strengths
  1. Experiments are extensive. The experiments cover classification, dense correspondence, retrieval, segmentation, detection, and depth estimation. Training-free downstream evaluations are particularly useful for assessing preservation of pretrained features.

  2. Improvements over strong baselines are consistent. Integration with both APHQ-ViT and LS-ViT supports the practical usefulness of the approach as an additional reconstruction objective.

Critical weaknesses
  1. About method: The motivation analysis does not isolate sensitivity weighting as the cause of distortion. The visualizations demonstrate differences between full-precision and quantized representations, but do not establish that sensitivity weighting causes those differences rather than quantization itself. Furthermore, selecting pairs with very small sensitivity differences can inflate the ratio of evolution discrepancy to sensitivity difference. The ratio is also sensitive to the scales of the two quantities. Standardized correlation analysis and comparisons across multiple layers would provide stronger evidence that feature evolution contributes information beyond sensitivity.

  2. About ablation study: The EAL-only and GCR-only variants in Table 5 still require precise definitions. EAL constructs the reference subspace used by GCR, so their independent operation cannot be inferred from the full implementation.

  3. About code: In the supplied LS-ViT implementation, subspace construction combines a diagonal sensitivity estimate with the elementwise square of a low-rank factor before taking a stabilized square root. The manuscript instead describes weighting by ∣H∣\sqrt{|H|}, without explaining how the structured sensitivity estimate is reduced to the feature-shaped weights used in practice. The code also clarifies that, for the supplied DINOv3 entry point, KL divergence is computed from softmax-normalized backbone outputs. Since the backbone uses an identity head, these outputs are normalized CLS features rather than class logits. The authors should explicitly describe and justify this sensitivity objective and explain the corresponding construction for APHQ-ViT.

  4. Minor issues:

4.1 There should be a numerical safeguard to the magnitude ratio and initial FLOW-Q loss normalization; the inspected expressions do not explicitly protect their denominators.

4.2 How truncation rank is selected across architectures with different channel dimensions?

4.3 What's overhead of the small EAL among the complete method’s overhead? Table 5 reports increases from 105.3 to 118.6 minutes and from 3.87 to 4.93 GB.

Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/4)

We thank reviewer dmsz for the valuable time and constructive feedback. Since the issues in "Critical Weaknesses" and "Questions for Authors" are different, we number the questions sequentially from Q1 to Q6 and provide our point-by-point responses below.

Q1: Concerns regarding the motivation analysis & standardized correlation analysis.

A1: We appreciate this constructive suggestion. To provide a clearer response, we divide our clarification into two parts: the motivation analysis (Q1.1) and the standardized correlation analysis (Q1.2).

Q1.1: Concerns arising from the imprecise presentation of the motivation analysis.

A1.1: Thank you for spotting this. The statement in the abstract of our initial paper that "the existing sensitivity-weighted reconstruction paradigm induces feature distortion during quantization" was not sufficiently accurate. More precisely, the existing sensitivity-weighted reconstruction paradigm is insufficient to prevent quantization-induced feature distortion, which is precisely the core problem addressed in our work. Other parts of the original manuscript are also consistent with this interpretation.

Specifically, feature distortion primarily originates from the quantization process itself. Hessian-based sensitivity weighting methods and our work mitigate feature distortion from different perspectives. However, as illustrated in Figure 2\color{red}2 (Left), existing Hessian-based weighting mechanisms still exhibit considerable limitations in preserving the feature representations of the full-precision model after quantization. The following table compares the quantization performance of the native MSE criterion, Hessian sensitivity weighting, and our proposed approach across three architectures: ViT-B, DeiT-B, and Swin-S.

MethodW/AViT-BDeiT-BSwin-S
MSE2/353.3554.5557.02
Hessian2/362.8967.0665.77
Ours2/364.9868.4967.49

We have revised the abstract and related work of the paper and highlighted the changes.

Q1.2: Standardized correlation analysis and comparisons across multiple layers would provide stronger evidence for feature evolution beyond sensitivity.

A1.2: Thank you for your question. We agree that selecting pairs with very small sensitivity differences may lead to insufficiently robust statistical evidence. Therefore, we made revisions in the following two aspects to provide stronger evidence that feature evolution captures complementary information beyond sensitivity.

  • Expand the Sensitivity Difference Range. We expanded the threshold ranges for sensitivity differences. Specifically, instead of considering only the top 1% and 2%, we now examine four ranges: the top 1%, 2%, 10%, and 20%, to investigate how the ratio between feature evolution differences and sensitivity differences varies across different thresholds.
  • Addition of Standardized Correlation Analysis. We added a standardized correlation analysis across different layers. Concretely, before calculating the ratio, we normalized both feature evolution differences and sensitivity differences to ensure that they are on the same scale. Taking sensitivity differences as an example, the absolute difference between paired values is divided by the maximum absolute value of the two, thereby scaling the difference to the range of [0, 1]. The same normalization procedure is applied to feature evolution differences. The following table presents five key statistics from the box plots, summarizing the mean ratios of normalized feature evolution differences to normalized sensitivity differences across different threshold ranges.
LayerLower_whiskerQ1MedianQ3Upper_whisker
20.000033152652106
50.000062203569143
70.000058213876157
100.000017173162128

We have revised the motivation analysis and Figure 2\color{red}2 (Right) in §3.1\S\color{red}3.1, while adding a standardized correlation analysis.

AuthorsOpenReview ↗

Official Comment by Authors (2/4)

Q2: Clarification of the EAL-only and GCR-only ablation settings.

A2: We apologize for the ambiguity. To address your concerns, we provide a detailed explanation of the EAL-only and GCR-only settings, as follows:

  • EAL-only refers to first generating the manifold representation of the full-precision model using Eq. 2\color{red}2 in Evolution Aware Learning, and then replacing ΔXFP\Delta X^{FP} in Eq. 2\color{red}2 with ΔXQ\Delta X^{Q} to obtain the manifold representation of the quantized model. The quantized manifold representation is then aligned with its full-precision counterpart by minimizing the mean squared error (MSE), thereby enabling the quantized model to capture structural information.

  • GCR-only represents the construction of the subspace basis by replacing the manifold representation G\mathcal{G} in Eq. 3\color{red}3 with the Hessian matrix H. Subsequently, ΔXQ\Delta X^{Q} and ΔXFP\Delta X^{FP} in Eq. 4\color{red}4 are replaced with the outputs of the current quantized block XQX^{Q} and the corresponding full-precision block XFPX^{FP}, respectively. These outputs are then projected onto the subspace for geometric correction of both direction and magnitude.

We have added the definitions of EAL-only and GCR-only to the component efficiency analysis in §4.2\S\color{red}4.2 of the paper.

Q3: Clarification of sensitivity weight construction, the KL divergence objective in DINOv3, and the corresponding operations in APHQ-ViT.

A3: We thank the reviewer for this insightful comment. To make our response clearer, we divide the issues into three aspects: sensitivity weight construction (Q3.1), the use of normalized CLS features in DINOv3 (Q3.2), and the corresponding construction in APHQ-ViT (Q3.3).

Q3.1: Sensitivity weight construction notation about ∣H∣\sqrt{|H|} and unexplained feature-wise weight derivation.

A3.1: Thank you for your careful review. We apologize for the confusion caused by the lack of sufficient explanations in the original paper. To address your concerns, we provide the following detailed clarifications:

  • Explanation of the Hessian Formula and Implementation. Our provided code is built upon the LS-ViT framework, which constructs the Hessian matrix HH by combining diagonal and low-rank components. Then, we adopt its official implementation for Hessian construction and derive ∣H∣\sqrt{|H|} using the following operation in our code: sensitivity = torch.sqrt(abs(hessian_weight) + 1e-12). It is important to emphasize that HH represents the loss-sensitivity weights in the current framework, whereas ∣H∣\sqrt{|H|} denotes the weights used in our method to modulate the feature evolution matrix.

  • How to Reduce to Feature-Shaped Weights. The Hessian matrix measures the sensitivity of each position in the feature map. Therefore, both the diagonal and low-rank Hessian components have the same dimensions as [1, (T$\times$C)], where T denotes the number of tokens and C denotes the number of channels. In contrast, the dense features have a shape of [B, T, C], where B is the batch size. We repeat the Hessian matrix along the batch dimension B times to obtain feature-shaped weights aligned with the dense features.

To improve readability, we have added a brief description of the Hessian construction process in LS-ViT to §3.2\S\color{red}3.2.

AuthorsOpenReview ↗

Official Comment by Authors (3/4)

Q3.2: Why are normalized CLS features used to construct the Hessian sensitivity objective in DINOv3 quantization?

A3.2: We appreciate this constructive suggestion. Using normalized CLS features for DINOv3 quantization achieves the best quantization performance while enabling task-agnostic Hessian estimation across different downstream tasks. We address your concern through a combination of mechanistic explanations and supporting experiments.

  • Mechanism Explanation. The strength of DINOv3 lies in its ability to produce robust feature representations that generalize across various downstream tasks. In contrast, existing PTQ studies typically employ different ViT architectures and use class logits obtained through classification heads, which are tailored to specific image classification tasks. To design a task-agnostic Hessian estimation scheme that does not rely on task-specific classification heads, we compute the KL divergence between the softmax-normalized CLS features of the full-precision and quantized models. As CLS features aggregate global visual information, this objective allows the model to characterize quantization-induced changes in the global feature representations.
  • Supporting Experiment. To verify the effectiveness of this design, we further compare the performance of DINOv3 quantization under the APHQ-ViT framework when using class logits and normalized CLS features to compute the KL divergence. We freeze the DINOv3 parameters and train a classification head on the ImageNet dataset. The trained classification head is then integrated into the DINOv3 quantization process to provide class logits. In addition, we conduct image classification experiments on ViT-S under both KL divergence computation settings and compare their respective performance. The experimental results are presented below.
SettingW/ADINOv3 (Semantic Matching)ViT-S (Image Classification)
Class logits2/481.83 / 61.65 / 42.5957.29
CLS features2/482.25 / 62.17 / 42.7356.85

We have added the corresponding experiments and analysis to §4.2\S\color{red}4.2 of the revised paper.

Q3.3: The corresponding sensitivity objective construction in APHQ-ViT?

A3.3: Sorry for this confusion. To address your concerns, we provide a detailed explanation of the sensitivity objective construction in APHQ-ViT in the following order: first, the KL divergence computation target, followed by the Hessian generation.

  • KL Divergence Computation Target. During DINOv3 quantization, both APHQ-ViT and LS-ViT use normalized CLS features to compute the KL divergence. This choice is not determined by the quantization framework itself, but by the inherent robustness of the feature representations learned by DINOv3.

  • Hessian Generation. After backpropagating the KL divergence loss to obtain the gradients, we follow the perturbation-based Hessian generation strategy of APHQ-ViT to construct the sensitivity objective.

We have updated the corresponding descriptions in §3.2\S\color{red}3.2 and §4.2\S\color{red}4.2 of the revised manuscript.

Q4: Numerical stability of magnitude ratios and FLOW-Q Loss normalization.

A4: We fully agree with your insightful comment. Numerical safeguards have been implemented at the code level for both the magnitude ratio and the initial FLOW-Q loss normalization. Our experiments further confirm that incorporating these safeguards results in virtually identical performance to the original implementation.

Q5: Truncation rank selection across different architectures.

A5: Thank you for your careful review. For architectures with channel dimensions of 384, 768, and 1024, including ViT-S, ViT-B, DeiT-S, DeiT-B, DeiT-L, and the ViT-B-based DINOv3 model, the truncation rank is uniformly set to 192. For the other architectures considered in this paper, the truncation rank is set to half of the corresponding channel dimension.

We have added an explanation of the truncated rank selection across different architectures to §3.3\S\color{red}3.3 of the revision.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of changes for quick review

We express our sincere thanks to the ACs and reviewers for their time and effort. Below, we summarize how we address the key concerns for a quick review.

1. Motivation and Standardized Correlation Analysis (R-dmsz,R-NbzC). Regarding motivation, we corrected the description of the limitation of weighted reconstruction in the abstract and §\S3.1. Before computing the ratio between the feature evolution difference and the sensitivity weight difference, we regularized each difference separately to ensure they remain on the same scale. We then showed how the ratio of their regularized distances varies across multiple layers to form a standard correlation analysis.

2. Justification and Experimental Validation of Components (R-dmsz,R-NbzC). We explained the importance of the EAL strategy from the perspective that EAL can capture the information missing from weighted reconstruction (§\S3.1). For the GCR module, we first demonstrated the advantage of GCR in prioritizing directions with large feature changes, and then further highlighted that the directions discarded by GCR remain constrained by the weighted reconstruction loss (§\S3.3). Second, ablation experiments were conducted on the GCR module in terms of relative pairwise distance and structure preservation, to ensure that the GCR module can leverage contextual information for geometric calibration and that the performance improvement brought by GCR stems from better structure preservation (§\S4.2). The settings for each component when it exists independently are newly defined in §\S4.2.

3. Generalization and Parameter Ablation (R-YG9o). To further emphasize the generalization of our work, we integrated our method into the FIMA-Q framework and conducted quantization experiments on DINOv3 and conventional ViT architectures. For the parameter ablation, we added ablation experiments on the loss coefficient α\alpha and the optimization set size to demonstrate the effectiveness of the parameters. All added experiments and their corresponding analyses have been integrated into §\S4.1 and §\S4.2.

4. Confusion in the Code (R-dmsz,R-YG9o). We illustrated in detail the Hessian matrix generation strategies of APHQ-ViT, LS-ViT, and FIMA-Q in §\S3.2 to match the code, as well as why they can maintain the same size as the feature maps. When quantizing DINOv3, the use of CLS features for the KL divergence is explained by the need to achieve task-agnostic Hessian estimation and to obtain better experimental performance (§\S4.2). In addition, we elaborated on why the attention module in DINOv3 adopts partial quantization in downstream dense prediction tasks.

5. Detailed Clarification (R-dmsz,R-YG9o). To make our work easier to follow, we clarified the selection of truncation rank across different network architectures, described how the proposed loss is integrated into reconstruction (§\S3.3), and added evaluation metrics to Tables 1 and 2 to enhance readability (§\S4.1). In addition, imprecise descriptions in the Implementation Details were corrected.

The major revised contents in the revised version are highlighted in blue\color{blue}blue.

We have made every effort to address the reviewers' concerns and revise the manuscript accordingly. The key concerns and revisions are listed below.

Motivation and Standardized Correlation Analysis (R-dmsz,R-NbzC)

  1. Clarified the limitation of weighted reconstruction (Abstract and §3.1\S3.1).

  2. Regularized feature evolution and sensitivity differences separately to ensure they are measured on the same scale, and added standardized correlation analysis across multiple layers to highlight their correlation (§3.1\S3.1).

Justification and Experimental Validation of Components (R-dmsz,R-NbzC)

  1. Specified the configurations of individual components (EAL-only and GCR-only) when evaluated independently (§4.2\S4.2).

  2. Emphasized the importance of the EAL strategy in capturing information missing from weighted reconstruction (§3.1\S3.1).

  3. Explained the advantages of GCR in emphasizing directions with larger feature variations and the continued constraints imposed by the weighted reconstruction loss on directions discarded by GCR (§3.3\S3.3).

  4. Conducted ablation experiments on relative pairwise distance and structure preservation, demonstrating that GCR performance gains stem from improved feature structure preservation (§4.2\S4.2, Table 13).

Generalization and Parameter Ablation (R-YG9o)

  1. Evaluated generalization by integrating the method into FIMA-Q and conducting quantization experiments on DINOv3 and conventional ViT architectures (§4.1\S 4.1, Tables 1 and 3).

  2. Added ablation experiments on the loss coefficient α\alpha and optimization set size to evaluate the effectiveness of these parameter settings (§4.2\S4.2, Tables 11 and 12).

Confusion in the Code (R-dmsz,R-YG9o)

  1. Detailed the Hessian matrix generation strategies of APHQ-ViT, LS-ViT, and FIMA-Q to align with the code implementations, and explained why the resulting matrices have the same dimensions as the feature maps (§3.2\S3.2).

  2. Compared CLS features and class logits for KL divergence in DINOv3 quantization, explaining the choice of CLS features for task-agnostic Hessian estimation and improved experimental performance (§4.2\S4.2, Table 14).

Important Clarification (R-dmsz,R-NbzC,R-YG9o)

  1. Described the selection criteria for truncation ranks across different network architectures (§3.3\S3.3).

  2. Explained the integration of the proposed loss into the reconstruction process (§3.3\S3.3).

  3. Added evaluation metrics to Tables 1 and 2 to improve readability (§4.1\S4.1).

  4. Corrected imprecise descriptions in the Implementation Details (§4\S4).

Point-by-point replies are provided in the reviewer-specific responses. We welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.