- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PVP is a post-training quantization method for ultra-low-bit vision encoders. It weights feature-reconstruction errors per coordinate by task-gradient magnitudes under a pseudo-Huber potential, and adds a patch-direction constraint and a next-block attention-value aggregation constraint. A three-stage pipeline adjusts activation coordinates by learned block-wise rotations and a block Hadamard, calibrates quantization granularity, then optimizes AdaRound-style weight codes and activation scales. Experiments cover seven ImageNet classifiers at W2A2/W2A3/W2A4 (+5.31 mean over LS-ViT at W2A3), self-supervised and vision-language encoders, COCO detection, and a packed W2A4 runtime.
Critical strengths
S1. The accuracy improvement is clear and practically useful. PVP is best on all seven models at W2A3, with a mean gain of 5.31 points over LS-ViT (Table 2), and Table 11 reports real speedup and memory savings for a packed W2A4 runtime shipped as source, which few ViT PTQ papers provide.
S2. The central diagnostic is well designed. Using the exact Hessian, Table 13 shows that the second-order KL prediction is accurate at α = 10⁻³ (0.078–0.409% block-median error) but fails at the realized W2A3 residual (54.6–93.3%), and Fig. 4a isolates the coordinate weight within a single pipeline.
Critical weaknesses
W1. The method described in the paper does not match the released configuration. In the README's canonical entry point (run_pvp.sh with default.yaml), both visual pins have weight 0, the pseudo-Huber field is used only in the first third of the blocks, the last block is trained with pure KL, which Table 1 presents as collapsing, and the pseudo-Huber scale is 0.5·RMS rather than the 0.25·RMS of Eq. 3. It is therefore unclear whether Table 2 was produced by the method described in §3, and the promised per-cell manifests are a comment-only stub.
W2. The headline gain cannot be attributed to the proposed objective. The W2 baselines were run by the authors without a stated protocol, while PVP additionally uses the rotation stage, four times as many calibration images (4,096 vs. 1,024), finer activation granularity and QDrop. The only matched objective swap yields +0.64 / +0.28, whereas the rotation stage alone gives +5.61. Results for the same configuration also differ across tables (59.54, 59.21, 63.154, where even AdaRound reaches 62.729 in Table 10), and no variance is reported (Tables 2, 5, 10; Fig. 5b).
Questions for authors
- Which exact configuration produced the main tables? Please explain the relation between the recommended entry point (pins off, depth-mixed objective, pure KL on the last block, c = 0.5) and the method of §3, and release the as-run per-cell configurations.
- In the main ImageNet comparison, are the calibration count, granularity, high-precision exceptions and rotation stage matched across methods, and how were the W2 baseline rows produced? Beyond absolute values and pseudo-Huber, how does d_j differ from LS-ViT's diagonal?
- Given that direct KL also has the worse fit metric (2.282 vs. 1.467 on DeiT-T), what supports the overfitting explanation rather than optimization difficulty — especially since PD-Quant, which also optimizes a prediction-difference KL, reaches 25.56 / 31.45 at W2A3?
- Since Table 5 states one 50-epoch linear-probe protocol for both arms, how do you explain CIFAR-100 Top-1 falling from 62.84 to 53.02 while retrieval improves by 3.38 after adding the rotation stage? Please also reconcile the conflicting entries above and state which are canonical.
6 comments
Response to Reviewer fhFT (Part 2)
2. Regarding W2/Q2: ImageNet comparison protocol and sources of gains
The baselines use the same 4,096 reconstruction images and 256 calibration images. We align the quantization granularity at the L stage and in the final model, as well as the input interfaces and higher-precision exceptions.
| Comparison item | Relationship between configurations |
|---|---|
| Rotation transforms | A PVP component that the other methods do not use. |
| Number of calibration samples | All methods use the same 256 images to initialize quantization parameters and the same 4,096 reconstruction images during training. |
| Quantization granularity | All methods use per-token activation quantization and per-channel weight quantization in linear layers, plus per-head-channel matmul quantization for QK and PV in attention, as in LS-ViT. All methods are compared at the same quantization granularity. |
| High-precision exceptions | All methods use W8A8 patch embeddings with uint8 image inputs. First, folding preprocessing into the weights allows direct uint8 input and avoids CPU overhead. Second, INT8 Tensor Cores let the first layer run without unpacking weights at a point where there is no earlier computation to overlap with unpacking. The other modules use the bit widths of each experiment. |
For the quantization granularity settings, see the official LS-ViT W2A3 configuration, which enables matmul_head_channel_wise, token_channel_wise, and qdrop.
To isolate the effect of rotation, we compared the ImageNet Top-1 accuracy of PVP and LS-ViT at W2A3 without the G stage.
| Model | PVP (no G) | LS-ViT (no G) |
|---|---|---|
| ViT-S | 53.947 ± 0.121% | 49.984 ± 0.239% |
| DeiT-T | 46.723 ± 0.217% | 43.127 ± 0.334% |
Each setting used three random seeds. The table reports the mean across three runs, and the value after ± reflects variation across seeds due to calibration-set sampling, QDrop, and other sources of randomness. For the model, bit width, and dataset settings raised in your question, which gave the +5.61 pp gain from rotation, PVP retains an accuracy advantage of approximately 3–4 pp even without the G stage.
Rotation complements the reconstruction objective. On some models and at some bit widths, adding rotation to existing methods can improve accuracy more than changing the objective. We study rotation as a set of optimizable parameters to address activation outliers that earlier work handled with specialized quantizers. PVP gains more from rotation than the other methods. Applying rotation before reconstruction also makes activation scales easier to optimize, so PVP calibrates substantially faster than the baseline implementations.
Response to Reviewer fhFT (Part 3)
3. Regarding Q2: How the LS-ViT and PVP optimization objectives differ
PVP uses the absolute task gradient to weight activation errors and a Pseudo-Huber loss to control large errors. LS-ViT combines a diagonal term with a low-rank term . Let denote the student-minus-teacher activation for calibration sample , and let be the gradient of the final task loss with respect to that activation.
Weighting rule. LS-ViT uses gradient samples to fit a second-order approximation of KL divergence with a diagonal term and a low-rank term, . The corresponding overall objective is
PVP derives its weights from the task change caused by reducing an activation error. Reducing coordinate gives an absolute task-change term , while the corresponding change in weighted squared error is . Matching their sums across calibration samples gives
\quad\Longrightarrow\quad d_j=\frac{\sum_n|g_{nj}e_{nj}|}{\sum_n e_{nj}^2} =\underset{d\ge0}{\operatorname{argmin}} \sum_n\bigl(|g_{nj}|-d|e_{nj}|\bigr)^2.$$ **Final reconstruction losses.** LS-ViT constructs its final reconstruction loss from the combined matrix $\widehat H=\widehat D+\widehat u\widehat u^\top$. For the activation error $e$ of one sample, the loss is $$\mathcal L_{\mathrm{LS\text{-}ViT}}(e) =\frac12 e^\top \bigl(\widehat D+\widehat u\widehat u^\top\bigr)e =\frac12\sum_j\widehat D_{jj}e_j^2 +\frac12(\widehat u^\top e)^2.$$ Here $\widehat u$ is the fitted low-rank vector, and the second term captures joint changes across activation coordinates. These definitions correspond to [LS-ViT Section 3.2, Equations (12)–(21)](https://openaccess.thecvf.com/content/CVPR2026/papers/Hwang_LS-ViT_Least-Squares_Hessian_Based_Block_Reconstruction_for_Low-Bit_Post-Training_Quantization_CVPR_2026_paper.pdf). PVP uses the following final reconstruction loss: $$\mathcal L_{\mathrm{rw}}(e) =\sum_j \frac{c^2\left(\sqrt{1+e_j^2/c^2}-1\right) \sum_n|g_{nj}e_{nj}|}{\sum_n e_{nj}^2}, \qquad c>0.$$
Response to Reviewer fhFT (Part 1)
Response to the Third Reviewer
Thank you for raising questions about the implementation settings, the role of the loss functions, and how the reported results relate to one another.
1. Regarding W1/Q1: Configuration differences between the paper and the released code
The supplementary code was extracted from part of our original repository and includes implementation variants for the main experiments, motivation experiments, and ablations. To meet the anonymity requirements, we extracted and cleaned the code, which left some settings missing or misapplied. We apologize for the resulting confusion. In the original experiments, we set hyperparameters and other options through command-line arguments.
run_pvp.shanddefault.yamlprovide a runnable demo; their default DeiT-T model and settings are intended for smoke testing and do not fully reproduce the best results in the paper.The manuscript incorrectly described the Pseudo-Huber scale as fixed. It is a runtime hyperparameter, and we use the setting in Table 14. We have corrected this in the manuscript. In the main comparison in Table 2, we use Pseudo-Huber in all quantized Transformer blocks. These blocks share the same configuration, except that the final block falls back to a setting without the next-block loss because there is no subsequent block. The captions will list the separate configurations for ablations and mechanism experiments. We used Python eager execution for experiments that collected diagnostic measurements or changed components.
Layer-dependent Huber settings and pure KL at the final block were exploratory configurations in the code. Neither was used in Table 2. Mixing losses by layer was an experimental candidate in the code ablations, motivated by the depth-dependent effect of direct KL shown in Table 1 and Figure 2.
We have updated Table 14. The table below summarizes the main experimental settings.