- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents GraphPTQ, a unified evaluation framework for post-training quantization of graph models across eight architectures and eleven quantization modes. It jointly evaluates hardware-measured GPU memory, in-distribution performance, and OOD/zero-shot generalization under different weight and activation precisions. The paper further decomposes quantization and distribution-shift perturbations and proposes a directional-curvature metric, , as an ID-only diagnostic of quantization sensitivity. Empirically, it identifies W8A16 as the most consistently safe configuration and reports increasingly architecture-dependent behavior under more aggressive low-bit quantization.
Critical strengths
-
The paper projects the ID Hessian onto the actual PTQ perturbation direction, $ \Delta W$, and uses $ |\Delta W^\top H_{\mathrm{ID}}\Delta W|$ to characterize quantization sensitivity. This directly evaluates the direction introduced by PTQ rather than relying on direction-agnostic sharpness measures such as SAM, Hessian trace, or the largest Hessian eigenvalue.
-
The paper separates weight quantization, activation quantization, distribution shift, and their interaction terms in the forward perturbation analysis. This provides a concrete way to analyze why the same low-bit quantization can produce different OOD behaviors across different architectures.
Critical weaknesses
-
The directional-curvature experiments do not provide strong enough evidence for the claimed OOD-risk prediction. Most of the reported OOD correlations are weak or statistically non-significant. W4A16, for example, has only with , although W4A16 is one of the main settings for which the paper recommends using this diagnostic before deployment. W4A4 also has only with . Across the five reported modes, only W4A8 reaches the stated threshold for OOD. These results show some correlation in selected settings, but do not yet show that the metric can reliably rank OOD risk across quantization configurations. The claimed strengthening with decreasing weight bit-width is also difficult to establish from the current comparison because activation precision changes across several of the modes, while W4A16, W4A8, and W4A4 share the same weight perturbation used by the diagnostic.
-
Some experimental results are internally inconsistent. The paper states that zero-shot performance changes are reported as relative percentages, but GIMLET HIV changes from 0.662 under FP32 to 0.375 under W4A4. This corresponds to a decrease of 28.7 percentage points, but a relative decrease of approximately 43.4%. The paper reports 28.7%, which corresponds to the absolute difference rather than the stated relative-percentage definition. This is not only a labeling issue. Under W4A16, the same task changes from 0.662 to 0.566, corresponding to a 9.6 percentage-point decrease or approximately a 14.5% relative decrease, while the text reports 13.2%. Neither calculation gives 13.2%. This makes it unclear which calculation and aggregation procedure are used for some of the reported conclusions.
-
The comparison between the explicit INT4 path and the fused path changes several factors simultaneously. The explicit GraphGPT W4A16 result discussed as unstable uses per-channel quantization, while the fused path uses per-group quantization with and BF16 fused computation. The corresponding explicit per-group INT4 experiment was not completed. The observed difference therefore cannot be attributed specifically to kernel fusion because quantization granularity and computation precision also change at the same time.
Questions for authors
-
Could the authors provide stronger validation for the use of directional curvature as an OOD-risk predictor? In particular, how should the W4A16 result of , and the W4A4 result of , be reconciled with the claim that the metric can rank OOD deployment risk from ID data alone? It would also be useful to report whether the increase in correlation under lower weight precision remains when activation precision is held fixed.
-
Could the authors clarify the exact definition and aggregation procedure used for zero-shot ? For GIMLET HIV, W4A4 changes from 0.662 to 0.375, which is a 28.7 percentage-point decrease but approximately a 43.4% relative decrease, while the paper reports 28.7%. Under W4A16, the score changes from 0.662 to 0.566, which is a 9.6 percentage-point decrease or approximately a 14.5% relative decrease, while the text reports 13.2%. Please clarify which values are used in the main conclusions and reconcile the corresponding tables and figures.
-
Could the authors provide a controlled comparison between the explicit and fused INT4 paths in which quantization granularity and computation precision are held fixed? In the current experiments, the unstable explicit GraphGPT W4A16 result uses per-channel quantization, while the fused path uses per-group quantization with and BF16 fused computation. Without the corresponding explicit per-group experiment, it is difficult to determine how much of the improvement comes from kernel fusion rather than quantization granularity or numerical precision.
-
Could the authors clarify the perturbation decomposition in Eq. (2)? The paper states that both and exist only when quantization and distribution shift are jointly present, but can remain nonzero when as long as both weight and activation quantization are present. Please clarify this statement and provide evidence for the claim that the sign of the interaction determines the direction of the observed OOD performance change.
Thank you for finding the zero-shot Δ error, the confounded INT4 comparison and the missing control for the bit-width trend.
How the revision addresses your review. We refined the analysis into two hypotheses, when ID validation reflects the OOD change and what better rounding can and cannot fix, and tested them on the 132 GOOD and DrugOOD models (4 seeds per configuration, 16 random roundings per bit-width). Zero-shot Δ is redefined and recomputed (GIMLET at W4A4: +1.0% under the new GraphPTQ recipe). The INT4 comparison was redone one factor at a time (head, layer coverage, granularity), which changed our explanation.
Weakness 1 & Question 1 — H̃ correlates weakly with the OOD change (W4A16 ρ=0.15, W4A4 ρ=0.34), and the bit-width trend mixes weight and activation bits. You are right. With 4 seeds per configuration (60 GOOD models instead of 15), the correlation is stronger than the earlier estimates at W4A16 and W4A4 and positive in every mode (Spearman 0.33–0.53; W4A16 0.47, W4A4 0.53; App. F.2), but it remains moderate, and part of its signal is shared with the quantized model's own ID drop. The bit-width trend does not hold at fixed activation precision (W8A16→W4A16: +0.14 [−0.13, +0.38]; W8A8→W4A8: −0.01), and W4A4 is dominated by activation error, so we withdraw it. The revision explains the gap: H̃ captures only the systematic, curvature part of the loss change, while under shift a first-order quantization × shift term carries most of the variation between roundings (median R² ≈0.83 at 4 bits; App. E). H̃ remains useful as that systematic part: it is what curvature-aware rounding reduces, and GPTQ's reduction carries over to OOD (H2, §6.2). Revision: the trend claim is withdrawn; H̃ is reported as a moderate ranker of the size of the OOD change and as the systematic term in H2.
Weakness 2 & Question 2 — Zero-shot Δ is inconsistent (GIMLET HIV 28.7 vs. 13.2). You are right: both were absolute ROC-AUC points mislabelled as relative %. Table 2 now uses one definition, the per-dataset relative change vs. FP32 averaged per column, with the absolute scores in App. C.1 (Table 12); every table number is now generated by script from the stored per-run results. Under the new recipe GIMLET changes by +0.7% at W4A16 and +1.0% at W4A4 (torchao W4A4: −9.5%).
Weakness 3 & Question 3 — The explicit vs. fused INT4 comparison changes several factors at once. You are right, and the controlled comparison changed our explanation: the accuracy gap came from the quantized output head and layer coverage, not from fusion or granularity. At matched head policy and coverage, per-channel and per-group agree within 0.3 points (GraphGPT +4.9% vs. +4.6%, TGT +5.2% vs. +5.5% MAE; App. C.3), while quantizing GraphGPT's head alone moves the MAE change from +5.1% to +161.9% (App. C.4). Fusion affects memory, not accuracy; GraphPTQ now runs on our own kernels (GQ-Kernels), and accuracy and memory are measured on the same kernels (App. A.3). Revision: the attribution to fusion is withdrawn; head policy and layer coverage are reported per model.
Question 4 — ΔWΔa is nonzero without shift; show that the sign of ΔWε sets the direction of the OOD change. You are right on both. Eq. 2 now states that only ΔWε needs shift, while ΔWΔa appears whenever weights and activations are both quantized (§3). The sign claim holds only in part: on the split where it is measured, the sign of the first-order OOD term matches the direction of the loss change in 100/120 cells, but on held-out OOD test data only in 69/120, i.e. chance (App. F.4). The term explains the direction, but the direction does not carry over to new data. What holds is distributional: across random roundings, the spread of the OOD loss change and the share of roundings that lower it follow the noise model (slope 1.03 / 1.07; calibration error 0.041 / 0.039), but not the sign of one rounding. Whether ID validation reflects it depends on the shift: the ID-best of 16 roundings is OOD-better in 88% of DrugOOD but 66% of GOOD cases. This needs OOD data, so it is an analysis, not a deployment diagnostic. Revision: Eq. 2 corrected; the sign claim narrowed to this statement (§4.3, §6.2).