acceptodds
Under review as a conference paper at ICLR 2027

Pinning Visual Paths:Preserving Visual Structure for Ultra-Low-Bit Vision Encoders with Task-Informed Reconstruction

Abstract

Vision encoders increasingly rely on low-bit inference to cut computation and memory. Post-training quantization (PTQ) therefore calibrates a low-bit student to sequentially reconstruct the block-wise activations of a full-precision teacher using a subset of images. Since uniform objectives like MSE spend limited low-bit capacity on reducing all feature errors equally, regardless of their task relevance, recent methods instead optimize the loss derived from gradient-curvature-objective pipeline: they propagate quantization perturbations through a full-precision suffix (the full-precision layers following the quantized block) to obtain exact task-loss gradients, derive second-order curvature information from these gradients, and use the resulting Taylor approximation of the task loss as the optimization objective. However, these methods mainly attribute the improvement to more accurate prediction of the task loss, leaving the reliability of the estimation-and-optimization pipeline itself largely unexamined. The curvature inferred from sampled gradients may not match the Hessian required by the local Taylor expansion, and ultra-low-bit quantization produces perturbations far beyond the range where such a local approximation holds. More fundamentally, even removing this approximation and directly optimizing the exact KL divergence can overfit both the small calibration set and the full-precision suffix, while also introducing strong cross-batch variation in the optimization signal. This motivates a different objective: incorporating task information into reconstruction while keeping updates stable and reliable. We propose Pinning Visual Paths (PVP). It uses gradient magnitudes to weight block-output errors without fitting a full endpoint task-loss model, thereby avoiding unstable endpoint-KL update directions and overfitting the task-blind part of the KL update space. It adds local constraints on patch features and the next attention block's token aggregation to address the mismatch introduced by a quantized suffix. It first rotates and rescales activation coordinates to reduce outliers, then refines weight codes and activation scales to stabilize discrete optimization within a restricted update space that limits overfitting. Across seven ImageNet encoders, PVP improves mean W2A3 Top-1 accuracy by 5.31 percentage points over SOTA methods, and remains effective for self-supervised, vision-language, and downstream models. Packed kernels provide a geometric-mean 2.39× speedup and 2.99× Inmemory reduction over BF16.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
post-training quantization, ultra-low-bit quantization, vision encoders, model compression, block reconstruction, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fhFTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PVP is a post-training quantization method for ultra-low-bit vision encoders. It weights feature-reconstruction errors per coordinate by task-gradient magnitudes under a pseudo-Huber potential, and adds a patch-direction constraint and a next-block attention-value aggregation constraint. A three-stage pipeline adjusts activation coordinates by learned block-wise rotations and a block Hadamard, calibrates quantization granularity, then optimizes AdaRound-style weight codes and activation scales. Experiments cover seven ImageNet classifiers at W2A2/W2A3/W2A4 (+5.31 mean over LS-ViT at W2A3), self-supervised and vision-language encoders, COCO detection, and a packed W2A4 runtime.

Critical strengths

S1. The accuracy improvement is clear and practically useful. PVP is best on all seven models at W2A3, with a mean gain of 5.31 points over LS-ViT (Table 2), and Table 11 reports real speedup and memory savings for a packed W2A4 runtime shipped as source, which few ViT PTQ papers provide.

S2. The central diagnostic is well designed. Using the exact Hessian, Table 13 shows that the second-order KL prediction is accurate at α = 10⁻³ (0.078–0.409% block-median error) but fails at the realized W2A3 residual (54.6–93.3%), and Fig. 4a isolates the coordinate weight within a single pipeline.

Critical weaknesses

W1. The method described in the paper does not match the released configuration. In the README's canonical entry point (run_pvp.sh with default.yaml), both visual pins have weight 0, the pseudo-Huber field is used only in the first third of the blocks, the last block is trained with pure KL, which Table 1 presents as collapsing, and the pseudo-Huber scale is 0.5·RMS rather than the 0.25·RMS of Eq. 3. It is therefore unclear whether Table 2 was produced by the method described in §3, and the promised per-cell manifests are a comment-only stub.

W2. The headline gain cannot be attributed to the proposed objective. The W2 baselines were run by the authors without a stated protocol, while PVP additionally uses the rotation stage, four times as many calibration images (4,096 vs. 1,024), finer activation granularity and QDrop. The only matched objective swap yields +0.64 / +0.28, whereas the rotation stage alone gives +5.61. Results for the same configuration also differ across tables (59.54, 59.21, 63.154, where even AdaRound reaches 62.729 in Table 10), and no variance is reported (Tables 2, 5, 10; Fig. 5b).

Questions for authors
  1. Which exact configuration produced the main tables? Please explain the relation between the recommended entry point (pins off, depth-mixed objective, pure KL on the last block, c = 0.5) and the method of §3, and release the as-run per-cell configurations.
  2. In the main ImageNet comparison, are the calibration count, granularity, high-precision exceptions and rotation stage matched across methods, and how were the W2 baseline rows produced? Beyond absolute values and pseudo-Huber, how does d_j differ from LS-ViT's diagonal?
  3. Given that direct KL also has the worse fit metric (2.282 vs. 1.467 on DeiT-T), what supports the overfitting explanation rather than optimization difficulty — especially since PD-Quant, which also optimizes a prediction-difference KL, reaches 25.56 / 31.45 at W2A3?
  4. Since Table 5 states one 50-epoch linear-probe protocol for both arms, how do you explain CIFAR-100 Top-1 falling from 62.84 to 53.02 while retrieval improves by 3.38 after adding the rotation stage? Please also reconcile the conflicting entries above and state which are canonical.
6 comments
AuthorsOpenReview ↗

Response to Reviewer fhFT (Part 1)

Response to the Third Reviewer

Thank you for raising questions about the implementation settings, the role of the loss functions, and how the reported results relate to one another.

1. Regarding W1/Q1: Configuration differences between the paper and the released code

The supplementary code was extracted from part of our original repository and includes implementation variants for the main experiments, motivation experiments, and ablations. To meet the anonymity requirements, we extracted and cleaned the code, which left some settings missing or misapplied. We apologize for the resulting confusion. In the original experiments, we set hyperparameters and other options through command-line arguments. run_pvp.sh and default.yaml provide a runnable demo; their default DeiT-T model and settings are intended for smoke testing and do not fully reproduce the best results in the paper.

The manuscript incorrectly described the Pseudo-Huber scale as fixed. It is a runtime hyperparameter, and we use the setting in Table 14. We have corrected this in the manuscript. In the main comparison in Table 2, we use Pseudo-Huber in all quantized Transformer blocks. These blocks share the same configuration, except that the final block falls back to a setting without the next-block loss because there is no subsequent block. The captions will list the separate configurations for ablations and mechanism experiments. We used Python eager execution for experiments that collected diagnostic measurements or changed components.

Layer-dependent Huber settings and pure KL at the final block were exploratory configurations in the code. Neither was used in Table 2. Mixing losses by layer was an experimental candidate in the code ablations, motivated by the depth-dependent effect of direct KL shown in Table 1 and Figure 2.

We have updated Table 14. The table below summarizes the main experimental settings.

Configuration itemMain experimental setting
G: Quantization granularityScalar-wise activation quantization and per-channel weight quantization
G: Optimization budget3,000 steps; 4,096 calibration rows; 64 images per fold
C0: Quantization granularity calibration256 calibration images; batch size = 32; eq_n = 128; quantization parameters are reinitialized after materializing the G-stage transformations, and the granularity is converted to that of LS-ViT
Final quantization configurationLS-ViT activation quantization granularity; Patch Embedding weights remain at 8-bit; QK/PV bit widths are adjusted according to the target activation precision A
L: Reconstruction budget5,000 updates per reconstruction unit; logical batch size = 96; reconstruction and Hessian estimation each use 4,096 images
Hessian refresh strategyquarter_full
L: Weight optimizeradam_soft_code; learning rate = 0.003; β = (0.729, 0.997002999); maximum code step = 0.01
L: Activation optimizeradam_log_scale; initial learning rate = 0.00168; β = (0.85, 0.995); cosine learning-rate schedule; maximum log step = 0.02
Soft-code RebaseWarmup = 2,048; interval = 2,048; minimum outward steps = 2,048; fraction = 0.01
Task distillation objectiveTraditional vision models: KL divergence with temperature T = 1; mechanism analyses and ablation experiments: T = 20; modern vision models: Global Feature / Prototype Score / Patch Token, with respective weights of 0.45 / 0.35 / 0.20
QDropLS-ViT's element-wise QDrop: quantized and floating-point values are each selected with probability 0.5
Response-weighted reconstructionPseudo-Huber loss with scale coefficient c=0.5 RMSc=0.5\,\mathrm{RMS}
Local visual constraintslocal_patch_next_attention = true; Patch Direction Pin weight = 0.25; Next-block Attention Aggregation Pin weight = 0.25
Numerical precisionFP32 teacher activation cache; BF16 student weights; FP32 quantization parameters and gradients; TF32 enabled
Execution backendCompiled CUDA Graph; Triton kernels fuse STE linear/attention paths, loss reduction/fitting, and log-scale + soft-lattice Adam updates; disabled for mechanism analyses and ablation experiments
AuthorsOpenReview ↗

Response to Reviewer fhFT (Part 2)

2. Regarding W2/Q2: ImageNet comparison protocol and sources of gains

The baselines use the same 4,096 reconstruction images and 256 calibration images. We align the quantization granularity at the L stage and in the final model, as well as the input interfaces and higher-precision exceptions.

Comparison itemRelationship between configurations
Rotation transformsA PVP component that the other methods do not use.
Number of calibration samplesAll methods use the same 256 images to initialize quantization parameters and the same 4,096 reconstruction images during training.
Quantization granularityAll methods use per-token activation quantization and per-channel weight quantization in linear layers, plus per-head-channel matmul quantization for QK and PV in attention, as in LS-ViT. All methods are compared at the same quantization granularity.
High-precision exceptionsAll methods use W8A8 patch embeddings with uint8 image inputs. First, folding preprocessing into the weights allows direct uint8 input and avoids CPU overhead. Second, INT8 Tensor Cores let the first layer run without unpacking weights at a point where there is no earlier computation to overlap with unpacking. The other modules use the bit widths of each experiment.

For the quantization granularity settings, see the official LS-ViT W2A3 configuration, which enables matmul_head_channel_wise, token_channel_wise, and qdrop.

To isolate the effect of rotation, we compared the ImageNet Top-1 accuracy of PVP and LS-ViT at W2A3 without the G stage.

ModelPVP (no G)LS-ViT (no G)
ViT-S53.947 ± 0.121%49.984 ± 0.239%
DeiT-T46.723 ± 0.217%43.127 ± 0.334%

Each setting used three random seeds. The table reports the mean across three runs, and the value after ± reflects variation across seeds due to calibration-set sampling, QDrop, and other sources of randomness. For the model, bit width, and dataset settings raised in your question, which gave the +5.61 pp gain from rotation, PVP retains an accuracy advantage of approximately 3–4 pp even without the G stage.

Rotation complements the reconstruction objective. On some models and at some bit widths, adding rotation to existing methods can improve accuracy more than changing the objective. We study rotation as a set of optimizable parameters to address activation outliers that earlier work handled with specialized quantizers. PVP gains more from rotation than the other methods. Applying rotation before reconstruction also makes activation scales easier to optimize, so PVP calibrates substantially faster than the baseline implementations.

AuthorsOpenReview ↗

Response to Reviewer fhFT (Part 3)

3. Regarding Q2: How the LS-ViT and PVP optimization objectives differ

PVP uses the absolute task gradient to weight activation errors and a Pseudo-Huber loss to control large errors. LS-ViT combines a diagonal term DD with a low-rank term uu⊤uu^\top. Let ene_n denote the student-minus-teacher activation for calibration sample nn, and let gng_n be the gradient of the final task loss with respect to that activation.

Weighting rule. LS-ViT uses gradient samples to fit a second-order approximation of KL divergence with a diagonal term and a low-rank term, H=D+uu⊤H=D+uu^\top. The corresponding overall objective is

min⁡D,u∑n∥gn−(D+uu⊤)en∥22.\min_{D,u}\sum_n\|g_n-(D+uu^\top)e_n\|_2^2.

PVP derives its weights from the task change caused by reducing an activation error. Reducing coordinate jj gives an absolute task-change term ∣gnjenj∣|g_{nj}e_{nj}|, while the corresponding change in weighted squared error is djenj2d_je_{nj}^2. Matching their sums across calibration samples gives

\quad\Longrightarrow\quad d_j=\frac{\sum_n|g_{nj}e_{nj}|}{\sum_n e_{nj}^2} =\underset{d\ge0}{\operatorname{argmin}} \sum_n\bigl(|g_{nj}|-d|e_{nj}|\bigr)^2.$$ **Final reconstruction losses.** LS-ViT constructs its final reconstruction loss from the combined matrix $\widehat H=\widehat D+\widehat u\widehat u^\top$. For the activation error $e$ of one sample, the loss is $$\mathcal L_{\mathrm{LS\text{-}ViT}}(e) =\frac12 e^\top \bigl(\widehat D+\widehat u\widehat u^\top\bigr)e =\frac12\sum_j\widehat D_{jj}e_j^2 +\frac12(\widehat u^\top e)^2.$$ Here $\widehat u$ is the fitted low-rank vector, and the second term captures joint changes across activation coordinates. These definitions correspond to [LS-ViT Section 3.2, Equations (12)–(21)](https://openaccess.thecvf.com/content/CVPR2026/papers/Hwang_LS-ViT_Least-Squares_Hessian_Based_Block_Reconstruction_for_Low-Bit_Post-Training_Quantization_CVPR_2026_paper.pdf). PVP uses the following final reconstruction loss: $$\mathcal L_{\mathrm{rw}}(e) =\sum_j \frac{c^2\left(\sqrt{1+e_j^2/c^2}-1\right) \sum_n|g_{nj}e_{nj}|}{\sum_n e_{nj}^2}, \qquad c>0.$$
Reviewer_83ybOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Pinning Visual Paths (PVP), a post-training quantization framework that combines task-informed reconstruction with constraints on patch features and attention aggregation. Gauge Alignment (G) transforms activation coordinates, while Lattice Pinning (L) optimizes discrete weight codes and activation scales. The theoretical analysis and diagnosis of existing reconstruction methods are thorough. However, evaluation on complete large VLMs and VQA tasks is still missing, and the deployment results do not cover the three-bit activation settings central to the accuracy experiments.

Critical strengths
  1. The theoretical analysis is solid. The paper examines curvature mismatch, approximation errors under finite-radius perturbations, calibration instability, and quantized-suffix mismatch, providing a clear motivation for the reconstruction objective.

  2. The analysis of existing reconstruction methods is detailed. The controlled metric-ranking analyses and block-loss diagnostics help distinguish accurate approximation of the task loss from effective guidance for low-bit optimization.

Critical weaknesses
  1. Evaluation on complete large VLMs and additional tasks is missing. The paper evaluates recent encoders such as DINOv3 and SigLIP2, but encoder-level results cannot substitute for experiments on complete generative VLMs. Experiments with high-capacity VLMs, such as LLaVA-1.5-13B [1] and Qwen3-VL-32B [2], are needed to establish the applicability of the method to more advanced VLM systems. Performance on VQA and other multimodal tasks should also be reported to determine whether the preserved visual features remain useful to the downstream language model.

  2. Deployment experiments do not cover key precision settings. Measured deployment results are needed for W3A3, W2A3, W4A4, and larger models, with accuracy, GPU memory usage, and latency reported for the same quantized checkpoints. Please describe the CUDA implementations for these bit widths, including bit packing, arithmetic data types, scale handling, quantization/dequantization, and floating-point fallbacks. If three-bit values are stored or processed using wider data types, the associated overhead should be included in the measurements.

  3. Key ablations remain insufficient to explain the transfer cost and isolate the sources of the overall gains. In Appendix A.1/Table 5, PVP G+L achieves 53.02% CIFAR-100 Top-1 accuracy, compared with 62.84% for Identity+PVP—a decrease of 9.82 percentage points—despite achieving better retrieval performance. Aggregating results across seven encoders and four bit-width settings obscures which configurations primarily drive this trade-off. Per-encoder and per-precision ablations are needed to explain the effect of G and clarify the scope of the feature-preservation claims.

Questions for authors
  1. Why focus on optimizing vision Transformers? In embodied systems such as π0.5 [3] and world models such as V-JEPA 2 [4], vision Transformers such as ViT and Swin are no longer the dominant model architectures; VLMs are. Additional experiments on VLMs are needed.

  2. The experimental backbones and quantization comparisons are mainly drawn from older model families. Can the authors provide experiments on larger, more recent VLMs under matched experimental protocols, reporting VQA performance together with measured GPU memory usage and inference latency? Without such evidence, I remain unconvinced that the conclusions extend to current large-model deployments. The existing encoder-level comparisons and W2A4 deployment results do not establish these end-to-end benefits, particularly for W3A3 and W2A3.

  3. Why does CIFAR-100 Top-1 accuracy decrease by 9.82 percentage points for G+L relative to Identity+PVP? Which encoders and precision settings primarily account for this decrease? Please isolate the effect of G and its interaction with L.

[1] H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[2] S. Bai et al., “Qwen3-VL Technical Report,” arXiv preprint arXiv:2511.21631, 2025.

[3] Physical Intelligence et al., “π0.5: A Vision-Language-Action Model with Open-World Generalization,” arXiv preprint arXiv:2504.16054, 2025.

[4] M. Assran et al., “V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning,” arXiv preprint arXiv:2506.09985, 2025.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 83yb (Part 1)

Thank you for your specific suggestions on complete VLM evaluation, practical deployment, and transfer ablations. We have added VLM experiments and details of the kernel implementation.

1. Regarding W1/Q1/Q2: The value of visual-encoder optimization and complete VLM evaluation

The importance of vision Transformers. Your concern about ViT applications is reasonable, but ViTs remain an important component of VLMs. For example, π₀.₅ (Physical Intelligence, 2025, Appendix A-E) uses a SigLIP ViT encoder (PaliGemma, Section 3.1). V-JEPA (Bardes et al., 2024, Section 3.3) likewise uses a ViT backbone. ViT optimization has received limited attention in current applications. For example, LLM Compressor's Qwen3-VL example optimizes only the language model.

To show that ViTs remain a substantial part of prefill latency even after the language side has been optimized, we loaded the QuantTrio/Qwen3.5-xB-AWQ W4A16 models with vLLM. We then ran three prefills on a single NVIDIA A100 40GB GPU with 1922×2016 ArxivQA images and measured time with CUDA events.

ModelViT latencyFull prefill latencyViT share
2B-AWQ220.5 ms308.5 ms71.46%
4B-AWQ222.7 ms432.0 ms51.55%
9B-AWQ328.9 ms693.4 ms47.43%
27B-AWQ329.3 ms1633.5 ms20.16%

The ViT accounts for a substantial share of VLM prefill latency, especially for model sizes suitable for edge deployment. These results support improving ViT inference speed.

Accuracy evaluation on complete multimodal VQA tasks. We focus on compressing vision encoders. To isolate the effect of our method and compare fairly with prior ViT methods, the paper reports only ViT task results. To address your question about full-VLM performance, we added experiments on LLaVA and Qwen.Only the vision encoder was quantized and calibrated; all other modules remained in BF16.

LLaVA-1.5-13B

Method / precisionGQATextVQAVQAv2OCR-VQAVGSEED-BenchRefCOCO val
BF1663.1052.7578.4066.6038.4066.5063.70
RTN W4A436.306.3040.4626.0012.4035.901.20
QDrop W4A441.808.5349.3128.5021.2041.2023.30
PVP W4A462.3041.1477.9659.4036.7066.5062.40
RTN W3A36.001.3316.043.800.806.200.00
QDrop W3A336.606.9641.6927.1015.3031.605.50
PVP W3A360.5025.1075.5936.5035.1064.2057.70

LLaVA-Next-13B

Method / precisionGQATextVQAVQAv2OCR-VQAVGSEED-BenchRefCOCO val
BF1666.1064.7580.6272.4041.3071.9071.40
RTN W4A435.606.3040.4626.0014.3038.501.80
QDrop W4A444.1011.5351.6127.9021.2044.9024.80
PVP W4A464.9056.3180.1469.0040.7070.1069.20
RTN W3A32.102.2521.425.101.208.800.00
QDrop W3A337.206.4642.7726.2016.7035.1011.40
PVP W3A362.5031.6976.2951.4037.3068.3063.90

Qwen3.5-4B

Method / precisionGQATextVQAVQAv2OCR-VQAVGSEED-BenchRefCOCO val
BF1661.8067.8981.8969.2042.5080.0090.70
RTN W4A432.105.2636.6512.1014.0038.406.30
QDrop W4A433.8010.3243.3315.0020.2046.8012.60
PVP W4A443.7012.1850.1219.9024.0052.2045.00
RTN W3A333.008.1837.5613.1011.9040.404.40
QDrop W3A334.008.9541.6815.1018.5040.505.60
PVP W3A337.109.0744.0720.2019.5041.9025.30

Qwen3.5-9B

Method / precisionGQATextVQAVQAv2OCR-VQAVGSEED-BenchRefCOCO val
BF1661.4071.4282.0671.6044.5080.4092.40
RTN W4A431.105.2539.2019.3013.4038.2010.80
QDrop W4A435.709.7248.6720.9016.3043.8026.70
PVP W4A439.5011.2251.3321.5019.9048.1049.50
RTN W3A329.005.8135.9018.6012.1039.308.60
QDrop W3A331.408.0844.2119.2012.5039.9011.40
PVP W3A333.509.5847.8421.9017.2041.4022.90

These results show that our method compresses the VLM ViT while preserving some of the model’s capabilities across multiple vision tasks, and performs better than QDrop and RTN across multiple models.

AuthorsOpenReview ↗

Response to Reviewer 83yb (Part 2)

2. Regarding W2/Q2: W2A4 CUDA deployment and other precisions

The current CUDA deployment implementation targets W2A4. The W2A3, W3A3, and W4A4 results in the paper evaluate the quantization algorithm.

Measured GPU acceleration in complete VLMs. To address your request, we measured the latency and GPU memory usage of our CUDA kernel implementation in VLMs. We used the same setup as in the ViT-share experiment, with 384×384 input images. Parentheses in the GPU memory column indicate the reduction relative to the BF16 vision encoder.

ModelBF16 ViT eager latencyw2a4 library latencySpeedupW2A4 ViT GPU memory
Qwen3.5-2B-AWQ53.89 ms40.31 ms1.337×129.05 MiB (−503.10 MiB)
Qwen3.5-4B-AWQ69.52 ms58.92 ms1.180×133.05 MiB (−503.11 MiB)
Qwen3.5-9B-AWQ70.66 ms51.93 ms1.361×184.85 MiB (−684.95 MiB)
Qwen3.5-27B-AWQ130.426 ms111.823 ms1.166×194.33 MiB (−685.01 MiB)

Although the ViT changes little in size as the full model scales up, our kernels achieve 1.16× to 1.36× speedup in full-VLM inference and reduce GPU memory usage.

W2A4 kernel implementation details are as follows:

ComponentImplementation
Weight packingThe W2 source format packs 4 weight codes per uint8. A tile-major layout supports Tensor Core memory access, with physical dimensions aligned to the selected tiles.
Activation packingA4 packs 2 activation codes per uint8, or 8 codes into a 32-bit fragment word. Quantization includes scaling, round-to-nearest-even, clipping, and packing.
GEMM arithmeticW2 is expanded within the CTA into INT4 operands and combined with A4 using INT4 Tensor Core MMA and INT32 accumulation. Outputs are rescaled to BF16.
Pre-expanded pathSome fixed-shape paths use a pre-expanded centered-W4 cache at 4 bits per weight. Source W2, caches, and padding are included in storage and GPU-memory accounting according to actual residency.
Scale handlingWeights use path-specific per-channel or groupwise metadata; activations use static or per-token scales. Scale calculations and output rescaling use floating-point arithmetic.
Asymmetric zero pointsWeights and activations retain their respective zero points. Integer dot products are corrected using row sums, column sums, and constant terms.
Quantization / dequantizationActivation quantization is fused with writes to the packed layout. Dequantization, rescaling, and some bias/activation operations run in the epilogue or dedicated fused kernels.
Attention and floating-point componentsQK and PV use the corresponding packed A4 paths. Softmax exponentiation, normalization, and scale handling use floating-point arithmetic. LayerNorm, GELU, residuals, and module interfaces use BF16/FP32 according to the fused path. Patch embedding uses W2A8 for the seven traditional encoders and BF16 for the six modern encoders; readouts retain their configured precision.

Current deployment development focuses on W2A4. NVIDIA hardware provides 4-bit Tensor Core operations for this computation. The main challenges for W2A3 and W3A3 concern 3-bit layouts and the attention pipeline. Asymmetric quantization also requires fused weight and activation zero-point corrections, row/column sums, and rescaling. For A3 attention, Q/K/V packing, QK computation, softmax, and requantization of probabilities for PV must be designed together to turn low-bit representations into execution gains.

W2A3 and W3A3 have received more attention in dedicated hardware research. Roth et al. (JMLR 2024, Section 5.2.2 and Figure 6) report VGG/CIFAR-10 classification accuracy and throughput for both precisions on a Xilinx Ultra96 FPGA. Zhao et al. (ISCAS 2025) design an ASIC architecture supporting 2–8 bit weights and activations, with 2-bit and 3-bit weight-loading modes. BitSystolic (Yang and Li, TCAS-I 2021) demonstrates a fabricated edge-inference NPU with 2–8 bit weights and activations.

Since 3-bit deployment currently requires substantial hardware development, we focus implementation effort on W2A4. Dedicated W2A3 and W3A3 implementations are left for future work as application needs and hardware support evolve.

AuthorsOpenReview ↗

Response to Reviewer 83yb (Part 3)

3. Regarding W3/Q3: Table 5 corrections, configuration-level results, and the role of G

We apologize for the data-pasting error in Table 5's Identity classification column. During copying and aggregation, the original table mixed ImageNet evaluation results with CIFAR-100 linear-probe results. This created the appearance that removing components improved accuracy. We have corrected the table using consistent CIFAR-100 sources.

The corrected evaluation uses a 50-epoch CIFAR-100 linear probe on frozen quantized backbones and cosine retrieval with 10K queries.

The corrected CIFAR-100 classification results in Table 5 compare Identity with the PVP G stage as follows.

G stageG stepsL methodL stepsTop-1 (%)Top-5 (%)
PVP3KPVP5K53.0281.83
PVP3KLS-ViT5K52.3881.05
PVP3KAPHQ-ViT5K48.6077.79
PVP3KFIMA-Q5K51.3880.26
IdentityN/APVP5K49.5778.82
IdentityN/ALS-ViT5K49.3478.57
IdentityN/AAPHQ-ViT5K44.2773.87
IdentityN/AFIMA-Q5K48.4177.72

Classification and retrieval results before and after adding G are compared below. Identity + PVP achieves 49.57% Top-1 and PVP G+L achieves 53.02%, a gain of 3.45 percentage points. R@1 increases from 45.94% to 49.32%, a gain of 3.38 percentage points. These gains demonstrate the contribution of the G stage.

L methodIdentity Top-1With G Top-1Δ Top-1 (pp)Identity R@1With G R@1Δ R@1 (pp)
PVP49.5753.02+3.4545.9449.32+3.38
LS-ViT49.3452.38+3.0445.6448.24+2.61
APHQ-ViT44.2748.60+4.3339.8343.19+3.37
FIMA-Q48.4151.38+2.9844.7047.35+2.64

Dear Reviewer 83yb,

Since today is the final day of the author-reviewer discussion period, we would like to gently follow up on our rebuttal. If our clarifications have resolved your concerns, we kindly ask you to consider updating your rating accordingly. Thank you very much for your time and support!

Best regards,

The Authors

Reviewer_a3JHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Pinning Visual Paths (PVP), which weights block-output errors using gradient magnitudes without fitting a full endpoint task-loss model. This avoids unstable endpoint-KL update directions and overfitting in task-insensitive parts of the KL update space. It shows accuracy improvement under very low-bit settings, surpassing SOTA methods.

Critical strengths
  1. it shows accuracy improvement under very low-bit settings for vision encoders, compared to other SOTA baselines.
  2. it highlights a new source of degradation for reconstruction-based PTQ: mismatch between the approximated and actual loss curvature induced by limited precision of low-bit, and biased gradient updates by data overfitting.
Critical weaknesses
  1. The experimental evidence and discussion are insufficient to support the claimed cause of quantization degradation.
    • The provided analysis (Figure 2, Table 1) lacks an overall explanation of how the experiment was conducted to obtain the results and findings. This leads to great confusion for readers about what the analysis is for. Checking the questions below would be helpful to address this weakness.
    • The manuscript is recommended to revise the description of each analysis: the findings, configurations, and the experiment result itself should be more thoroughly provided within the text. Please provide the experimental setup, key findings, and results in the main text, and explain how they support the authors’ claims. In the current version, the figures and results are presented largely as references, with insufficient explanation of their significance or connection to the claims.
  2. The entire presentation is too hard to follow, and this impacts the overall quality of the manuscript. This may stem from several reasons as below.
    • There are broken sentences that make the main flow of the paper hard to understand. For example, what does “ranking audit” refer to in the first sentence of the final paragraph of Section 2.2.1? What does “this issue” refer to in the opening sentence of Section 2.2.2?
    • The manuscript uses some very vague terms and phrases overall, without any prior context, making it hard for readers without background to follow. The manuscript is encouraged to be more self-contained. e.g., "teacher-student residual" (even if this is described in section 3, it should be defined earlier if it is going to be used very frequently in the prior sections), "code-cell boundaries", "nonlinear activation geometry", "Direct KL", "local pins", "cuts", "cross-batch code/scale-gradient cosine", "suffix-state mismatch", "ranking audits".
    • No section for background or related work: to understand the new findings, we need to know what optimization pipeline and loss function the prior works follow for quantization. However, there is no section providing the details.
    • Figure 3 for the overall method is hard to understand, as equations and figure components are provided without any context. e.g., is the FP line and the Quant line supposed to be a model? And what does the blue and green curves depict? Is the gap between them important? If so, what is the gap? There are too many components that are not explained.
Questions for authors
  1. The manuscript should explain more clearly how the experimental results support its claims.
    • What finding does Table 1 depict? the metrics in the table are hard to understand, as there is no explanation (Fit, Held, H-F).
    • How do the experiments in Figure 2 support the claimed cause of quantization degradation? The figure caption and main manuscript lack the clarity needed to understand the connection.
  2. Some clarification and revision should be made for the manuscript to enhance readability.
    • What does “ranking audit” refer to in the first sentence of the final paragraph of Section 2.2.1?
    • What does “this issue” refer to in the opening sentence of Section 2.2.2?
    • Figure 3 is encouraged to be revised.
4 comments
AuthorsOpenReview ↗

Response to Reviewer a3JH (Part 1)

Thank you for your specific comments on the experimental evidence and presentation. Owing to space constraints, the background and explanations in the original manuscript were not sufficiently detailed or clear, which made the paper harder to understand. We apologize for this lack of clarity.

In the revised manuscript, Related Work has been moved to Section 2, immediately after the introduction. The captions of Figure 2 and Table 1 now explain their experimental settings and metrics. The original Figure 3 has been removed, and its analysis is now explained with equations and text in Section 3.2.1. Original Sections 2.2.1 and 2.2.2 are now Sections 3.2.1 and 3.2.2, respectively.

1. Regarding W1/Q1: Experimental settings and main findings of Table 1

Table 1 compares activation-weighted reconstruction, block-output MSE, and full-model output KL as three block reconstruction losses. Direct KL leads to severe overfitting and accuracy collapse, while block-output MSE constrains the outputs and limits overfitting. Here, reconstruction means adjusting quantization parameters so that the quantized model behaves as closely as possible to the original full-precision model. We call the full-precision model the teacher and the quantized model the student.

The experiment uses DeiT-T and ViT-S at W2A3, meaning 2-bit weights and 3-bit activations. All branches use 1,024 ImageNet calibration images, a shared starting point, and 20K reconstruction steps per block. We compare PVP's activation-weighted reconstruction term Lrw\mathcal L_{\mathrm{rw}}, Block MSE, and direct KL computed through a full-precision suffix. The Lrw\mathcal L_{\mathrm{rw}} row reports the reconstruction performance of PVP's activation-weighted component.

The three objectives differ in the signal used to adjust the current block. Block MSE compares the student's and teacher's block outputs element by element, treating each squared error equally. Activation-weighted reconstruction assigns weights to block-output errors according to their effects on the downstream task output. Direct KL passes the current block output through the remaining network, compares the final class probability distributions, and uses that difference to adjust the current block. The suffix is simply the part of the network after the current block; a full-precision or floating-point suffix means that this part remains at full precision while the current block is calibrated.

The table below defines the metrics and their purposes.

MetricDefinition and purpose
Fit ↓Teacher–student KL evaluated on the calibration views used for reconstruction. It measures how much the student still differs from the teacher on the fitting inputs after reconstruction.
Held ↓Teacher–student KL evaluated on separately augmented views of the same calibration images. It measures the output difference after changing the image view. Fit and Held use two sets of views from the same source images.
H−FHeld − Fit measures the increase in KL when moving from fitting views to newly augmented views. It measures generalization beyond the calibration-data distribution.
acc ↑Top-1 accuracy on the full ImageNet test set, reported as a percentage to measure final classification performance.

Fit, Held, and H−F are all reported in units of 10−310^{-3}.

Table 1 compares three losses with a shared starting point, bit width, and reconstruction budget. Direct KL optimization gives Held KL values of 2.641 and 7.298 and H−F values of 0.359 and 0.785 on DeiT-T and ViT-S, respectively. These are the highest among the three branches for each model. Its final output therefore differs more from the teacher, and it incurs greater information loss on the validation views. Section 3.2.2 explains this finding in two ways.

First, direct KL constrains the current block through the final classification output but does not align intermediate module outputs. Two different block outputs can produce similar class probabilities on the current images and different activations on another batch.

Second, block calibration and final deployment use different suffix states. Later blocks remain at full precision while the current block is optimized and use quantized representations at deployment. The same current-block output can therefore produce different final results through these two suffixes. The effect of a block adjustment therefore depends on the downstream network used during calibration and on its behavior in the complete quantized model. This is the meaning of suffix-state mismatch.

Block MSE directly constrains every coordinate of the block output so that the student representation stays close to the teacher's. It reaches 25.49% and 33.82% Top-1 on DeiT-T and ViT-S, respectively. Both exceed the direct KL branch and demonstrate the role of block-output constraints.

AuthorsOpenReview ↗

Response to Reviewer a3JH (Part 2)

2. Regarding W1/Q1: How the experiments relate to the motivation

Our experiments and theory test whether the gains of existing methods come from accurate second-order KL predictions and examine what quantization-loss design should aim for. We find that these methods do not accurately predict KL and that their strong results do not come from accurate KL prediction. KL is also not an effective objective for direct optimization in this setting.

ExperimentContentResultConclusion
Figure 2(a)Compare the rankings from three existing loss functions with the rankings of measured Block MSE and KL changes.In early layers, their optimization behavior resembles Block MSE more closely than KL.These methods behave more like local block losses that incorporate task information. Their gains cannot be directly attributed to accurate KL prediction.
Figure 2(b)Use curvature at the full-precision block output to predict KL as perturbations grow from one thousandth of the actual quantization error to the full error.KL prediction errors increase substantially from small perturbations near the full-precision output to the actual quantized output.The second-order predictions near the full-precision output described by existing methods do not accurately predict KL at the actual quantization error.
Figure 2(c)Compare quantization-parameter gradient directions across calibration batches for different loss functions.KL update directions across batches are orthogonal, while weighted block-output MSE shows clear aligned or opposing trends.Direct KL update directions are nearly uncorrelated across batches. This explains its optimization disadvantage in the small-sample calibration results of Table 1 and a possible cause of final accuracy degradation.

Figure 2(a): What explains how the losses of strong prior methods behave during calibration?

We fix an already quantized DeiT-T W2A3 model and construct 128 feasible quantization-code choices for each block. Each choice replaces one weight's current integer code with an adjacent code, such as changing 1 to 2, within the representable W2 range.

We first estimate the reconstruction losses of APHQ-ViT, FIMA-Q, and LS-ViT on 64 images. Another set of 64 images is used to measure the actual block-output MSE and final classification-output KL changes for each proposal. Thus, loss estimation and evaluation use different images.

The curves show the mean correlation across the three metrics, which come from strong baseline methods. The results show that they behave more like local block losses than direct task losses. Their gains therefore do not necessarily come from the more accurate KL predictions proposed in those studies.

Figure 2(b): How KL prediction accuracy changes from small perturbations near the full-precision output to actual quantization errors

We use ViT-S W2A3 and examine the output error ee produced by the block's final rounded quantized values. Here, ee is the quantized block output minus its full-precision output.

We compute H(0)H(0) at the full-precision block output and add a perturbation αe\alpha e along the direction of the output error e=oquant−ofpe = o_{quant} - o_{fp} caused by quantizing the block. At α=10−3\alpha=10^{-3}, its magnitude is one thousandth of the actual quantization error, so the output remains near the full-precision output. At α=1\alpha=1, adding the full error reaches the quantized block output. Let K(αe)K(\alpha e) denote the teacher–student KL measured by passing the perturbed block output through the full-precision downstream network. We then use the exact Hessian at the full-precision block output to predict KL as follows.

Q(αe)=α22e⊤H(0)e.Q(\alpha e)=\frac{\alpha^2}{2}e^\top H(0)e.

This use of curvature near the full-precision output to estimate loss changes is the second-order prediction examined here.

This experiment examines how much quantization error the second-order formula can accommodate while accurately predicting KL. Prediction errors increase substantially from small perturbations to the actual quantized output. The success of these methods is therefore difficult to attribute to accurate KL prediction.

Figure 2(c): How agreement between quantization updates across calibration batches explains Table 1

We fix the ViT-S W2A3 model, compute gradients separately on different calibration minibatches, and compare pairs of gradient directions. The parameters include the current block's weight-code optimization parameters and activation scales.

The results show that the activation-weighted loss gives stronger agreement across batches, while using KL directly produces parameter updates in different, nearly perpendicular directions. This explains the accuracy results in Table 1. It also agrees with the theoretical explanation based on the gradient null space in the main text and shows that KL updates are unstable with few calibration samples.

AuthorsOpenReview ↗

Response to Reviewer a3JH (Part 3)

3. Regarding W2/Q2: Clarifying “ranking audit” and “this issue”

The manuscript now replaces “ranking audit” with a direct description of the experiment: Figure 2(a) shows the correlation between loss-function rankings of weight-code changes and the measured MSE and KL changes. The opening of Section 3.2.2 also replaces “this issue” with “previous approaches do not appear to stem from accurate KL prediction”.

4. Regarding W2: Additional background and terminology

We moved Related Work from Appendix C of the original manuscript to Section 2 of the main text, after the introduction and before the mechanism analysis, while retaining the existing discussion and citations.

5. Regarding W2/Q2: Removal of Figure 3 and its explanation

We removed the original Figure 3 and replaced it with path-integral formulas and text in Section 3.2.1. Equation (2) expresses the gradient and KL as

\qquad K(e)=\int_0^1(1-t)e^\top H(te)e\,\mathrm dt=e^\top\left(\int_0^1(1-t)H(te)\,\mathrm dt\right)e.$$ The formulas show that the gradient weights curvature uniformly along the path, while KL weights it by $1-t$. Thus, $\nabla K(e)=He$ and $K(e)=e^\top \frac12He$ do not use the same $H$. The curvature $H$ fitted from gradients does not necessarily predict KL accurately. As quantization errors grow, the gradient-fitted curvature can deviate further from the curvature needed for the second-order KL estimate.

Dear Reviewer a3JH,

Since today is the final day of the author-reviewer discussion period, we would like to gently follow up on our rebuttal. If our clarifications have resolved your concerns, we kindly ask you to consider updating your rating accordingly. Thank you very much for your time and support!

Best regards,

The Authors

Comments on the paper

1 comment

General Response

We sincerely thank all reviewers for their time and effort in evaluating our paper. We appreciate their recognition of the practical value of PVP, its task-aware reconstruction design, and its effectiveness across image classification and complete vision-language models. Their constructive suggestions motivated additional experiments and clearer explanations of the method and evaluation. Below, we summarize the contributions recognized by the reviewers and our main updates during the rebuttal.

Contributions recognized by reviewers

Understanding low-bit reconstruction. Controlled experiments distinguish accurate task-loss prediction from effective reconstruction. They show why a loss that predicts the final task metric well need not provide the best local reconstruction signal, and motivate PVP's task-gradient weighting, patch-feature constraint, and next-block attention constraint. [a3JH, 83yb, fhFT]

Method design and accuracy. PVP uses task sensitivity to assign larger reconstruction weights to changes that matter more for the downstream output, while its two visual constraints preserve patch information and the attention input used by the next block. PVP outperforms LS-ViT on all seven ImageNet models at W2A3, with a mean Top-1 gain of 5.31 points (Table 2). [a3JH, 83yb, fhFT]

Generalization and deployment. The added evaluations extend the evidence beyond ImageNet classification. The VLM experiments cover multiple models, tasks, and ViT bit widths, and the packed W2A4 implementation provides measured latency and GPU-memory savings (Table 11). [83yb, fhFT]

Additional experiments and clarifications

Why PVP uses task gradients to weight block errors. We clarified the Fit and Held views of each calibration image and their joint interpretation. Early-block reconstruction losses rank weight-code changes more like block MSE than final-output KL; even an exact full-precision-output Hessian predicts KL poorly at actual W2A3 errors, while direct KL gives less consistent updates across calibration batches. [a3JH, fhFT]

Accuracy gains without rotation and reproducibility. With Stage G disabled, PVP improves ImageNet W2A3 Top-1 over LS-ViT by 3.96 points on ViT-S and 3.60 on DeiT-T across three seeds. We documented the shared calibration data, quantization settings, demo differences, and settings for the main results and ablations. [fhFT]

Full VLM accuracy and deployment. On four LLaVA and Qwen3.5 models and seven tasks, PVP outperformed QDrop and RTN at both W4A4 and W3A3 with the remaining modules in BF16. With W4A16 AWQ language models, the W2A4 ViT implementation sped up image prefill plus the first-token LM head by 1.166× to 1.361× across Qwen3.5 2B to 27B, while ViT memory use fell by about 503 to 685 MiB. We separated these measurements from the accuracy experiments. [83yb]

CIFAR-100 transfer correction. The original Identity column mixed ImageNet and CIFAR-100 linear-probe results. Under one CIFAR-100 protocol, adding G to PVP improves Top-1 from 49.57% to 53.02% and R@1 from 45.94% to 49.32%; corrected rows and all four method comparisons are now provided. [83yb, fhFT]

Changes to the manuscript

  • Sections 1 to 3, Figure 2, and Table 1: We moved Related Work into the main text, defined the teacher-student output difference earlier, and clarified the diagnostic experiments and captions. We replaced the original Figure 3 with Equation (2) and an explanation of the different weights used for gradient fitting and KL prediction.
  • Section 4.1 and Appendix D.2 (Table 15): We replaced the Pseudo-Huber scale with a configurable hyperparameter and clarified calibration settings, task losses, and execution modes.
  • Appendix A.1 (Table 5): We corrected Identity Top-1 and Top-5 using the CIFAR-100 linear-probe protocol used for the G rows.
  • Appendix A (Table 9): We clarified recalibrated and fixed modules in the precision ablation.
  • Appendix A.7 (Table 12): We added ViT coverage for modern architectures and measured its share of Qwen3.5 prefill latency after language-model quantization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.