- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a function-space sensitivity measure for mixed-precision neural network quantization. Instead of measuring quantization damage only through parameter changes or loss curvature, the method uses JVPs to estimate how specific quantization perturbations affect the final model outputs. It also considers cross-layer tangent interference and uses the resulting sensitivity scores for budget-constrained bit allocation. Experiments on ResNet-18, MobileNetV3-Small, and MobileNetV2 compare different sensitivity measures and evaluate their prediction accuracy and allocation performance.
Critical strengths
- The paper provides a useful comparison of parameter-space, loss-space, and function-space sensitivity. The proposed measure achieves strong correlations with actual quantization damage, especially on ResNet-18, and the experiments include several meaningful baselines.
- The analysis of cross-layer interference is interesting. The authors also evaluate whether better sensitivity prediction actually leads to better bit allocation, rather than relying only on correlation metrics. These results help clarify both the usefulness and limitations of the proposed approach.
Critical weaknesses
- My biggest concern is whether the proposed JVP-based sensitivity provides sufficient practical advantages over directly measuring isolated output distortion. In Table 2, the direct KL baseline achieves a correlation of 0.805 on MobileNetV3-Small, compared with 0.705 for additive functional sensitivity and 0.776 with tangent interference. More importantly, Appendix Table 16 shows that JVP profiling is actually slower than direct isolated profiling (93.81s vs. 16.35s on MobileNetV3-Small). This makes the practical motivation less convincing.
- The novelty compared with existing output-space methods is somewhat limited. The paper acknowledges that Jacobian-projected quantization perturbations have already been studied in BLOB-Q. The main distinction seems to be the choice of Euclidean logit geometry and the retention of tangent responses. However, the results do not clearly demonstrate that this choice is consistently better than other output-space metrics, especially CE-GGN.
- The experimental scope is relatively narrow. All experiments use Imagenette and three CNN architectures, with full allocation experiments on only two of them. The improvement from tangent interference is also not consistent across architectures. It would be useful to see whether the proposed sensitivity remains effective on larger models or more challenging quantization settings.
Questions for authors
- Given the higher profiling cost and mixed prediction results, what is the main practical advantage of JVP sensitivity over direct isolated-output measurements?
- Can the authors provide stronger evidence that the proposed output geometry offers advantages over existing Jacobian-based sensitivity methods?
- Does the improvement in final bit allocation generalize beyond the currently tested CNN architectures and Imagenette?
2 comments
Additional Details on Transformers
Following the reviewer's concern about evaluation only on CNN architectures, we added an ImageNet-pretrained DeiT-Tiny transformer as a cross-family architecture check, using the same weight-only FP32/INT8/INT4 protocol.
Across 125 heterogeneous configurations, additive functional sensitivity remains strong:
\rho_{\mathrm{logit}}=0.965.$$ This also provides stronger evidence concerning output geometry. CE-GGN obtains $0.920/0.886$, and the functional-minus-GGN improvements are statistically significant for both KL and logit MSE: $$+0.0256\;[0.0015,0.0542], \qquad +0.0798\;[0.0490,0.1230].$$ The transformer produces a different interaction pattern from MobileNetV3-Small: adding tangent-interference terms decreases correlation to $0.921/0.937$, with both paired decreases statistically significant. This strengthens the conclusion that the value of cross-layer tangent composition is architecture-dependent. We additionally performed exact-budget allocation at $B=8,9,10,11$ and re-evaluated selected configurations on all 3,925 Imagenette validation images. All objectives coincide at $B=8$; at higher budgets, functional, direct-output, and CE-GGN allocations remain closely competitive with small target-dependent differences. We therefore no longer restrict the empirical evidence to CNNs. We still avoid claiming broad transformer generalization from a single DeiT-Tiny model and explicitly retain larger datasets and larger/diverse transformer families as limitations.
Response to the reviewe
We thank the reviewer for the constructive assessment. We agree that the key issues are the practical role of JVPs, the distinction from prior Jacobian/output-space methods, and the experimental scope. We revised the paper and added direct-vector, low-bit, and exact-budget controls to sharpen these claims.
1. What is the practical advantage over direct isolated-output measurements?
We agree that JVP profiling is not cheaper in our current implementation. Direct isolated profiling takes 56.89 s vs. 194.07 s on ResNet-18 and 16.35 s vs. 93.81 s on MobileNetV3-Small. We have removed any implication of a profiling-speed advantage.
We also added the stronger control suggested by the reviews: cache the exact isolated output vectors
dl,a(x)=fθ+Δθl,a(x)−fθ(x)
and compose them directly. On MobileNetV3-Small, this gives
\rho=0.846\;(\mathrm{logit\ MSE}),$$ compared with $0.776/0.802$ for JVP tangent interaction. Thus, exact direct vectors can indeed outperform the first-order representation. This changes our claim. The practical value of JVPs in this study is **analytical and reusable rather than computational**: the same cached compression-direction responses support additive scoring, alternative output geometries such as CE-GGN, and explicit cross-layer interference analysis without further quantized-model evaluations. We no longer claim that vector preservation itself is unique to JVPs. ### 2. Novelty relative to BLOB-Q and other Jacobian-based methods We agree that neither Jacobians nor final-output sensitivity are new, and we now state this explicitly. BLOB-Q already uses Jacobian-projected quantization perturbations for output-distortion-based allocation, and Chen et al. derive related Jacobian-mediated loss geometry. Our contribution is therefore narrower: we use the common form $$S_M(\Delta w)=\Delta w^\top J^\top M J\Delta w$$ to isolate what changes when the **same quantization perturbation** is measured under different output geometries and when its vector response is retained rather than immediately reduced to a scalar. This produces several distinctions not available from a single scalar sensitivity score: - Euclidean logit geometry ($M=I$) vs. CE-GGN loss geometry; - first-order JVP responses vs. exact nonlinear isolated responses; - scalar direct costs vs. vector-preserving direct composition; - additive scoring vs. explicit first-order cross-layer interference. The results do **not** show that Euclidean geometry universally dominates. On ResNet-18 it is significantly stronger than additive CE-GGN for KL and logit MSE, while CE-GGN is stronger for Top-1; on MobileNetV3-Small, all paired functional-vs-GGN confidence intervals cross zero. We therefore revised the paper to present the contribution as a controlled geometric/diagnostic framework rather than a universally superior metric. ### 3. Generalization beyond the current CNN/Imagenette setting We agree that the experimental scope remains limited and do not claim generalization to transformers or full ImageNet-scale evaluation. We did, however, add two checks that broaden the original evidence. First, MobileNetV2 provides an external architecture check: additive functional sensitivity remains strong, with $$\rho=0.935\;(\mathrm{KL}),\qquad \rho=0.904\;(\mathrm{logit\ MSE}),$$ while tangent interaction does not improve ranking. This reinforces that interaction benefits are architecture-dependent rather than universal. Second, we added controlled INT4/INT3/INT2 experiments on ResNet-18. Local JVP-vs-exact correlation decreases from $0.990$ at INT4 to $0.887$ at INT3 and $0.687$ at INT2. Nevertheless, viable INT3 configurations remain strongly ranked: $$\rho_{\mathrm{KL}}=0.829,\qquad \rho_{\mathrm{logit}}=0.936.$$ INT2 then exposes the expected breakdown of the local first-order approximation. Finally, all allocation comparisons now use the same exact parameter-bit constraint and full 3,925-image validation set. These experiments show useful allocations in some regimes, but also confirm that stronger global ranking does not guarantee better allocation at every budget. ### Summary We have therefore narrowed the paper's claim. JVP sensitivity is not presented as universally faster or superior to direct output measurements, nor as a new use of Jacobians in quantization. Its contribution is a reusable first-order representation that makes output geometry, local linearization error, vector retention, and cross-layer composition separately testable. We also now state clearly that validation on larger datasets and transformer architectures remains future work.