- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how changes in graph topology after PTQ calibration affect the numerical difference between quantized and not quantized GNNs. The authors propose CP-DPR, a rewiring method that preserves some graph properties, and EQDZ, a metric to measure the numerical divergence after topology changes. The experiments show that topology changes often increase divergence especially with 4-bit quantization. Roman-empire with GCN shows the opposite result.
Critical strengths
- The paper asks a precise and concise but at the same time interesting research question.
- The crossed bootstrap is appropriate for the experimental setup. The analyses of rounding, clipping, and recalibration are also useful, especially the observation that activation distributions can change without affecting min–max calibration parameters.
Critical weaknesses
- I found the paper difficult to follow, especially at the beginning. The authors start introducing formulas already in the abstract, without defining all the terms. In my opinion, the abstract and introduction should first explain the story of the paper. Introducing formulas with undefined symbols so early makes it harder to understand what the authors want to show. Moreover, for a reader who is not very familiar with quantization, some technical terms are used without enough explanation.
- My biggest concern is on EQDZ metric. QDZ normalizes the FP32–PTQ logit difference by the norm of FP32 logits. However, this norm can change when the graph topology changes. For example EQDZ could increase because the FP32 logit norm becomes smaller, even if the absolute quantization error stays the same.I suggest reporting: i) the absolute FP32–PTQ logit difference before and after perturbation; ii) how the FP32 logit norm changes with perturbation amount; iii) results with normalization fixed to the original graph.
- The paper shows changes in numerical divergence, but it is less clear if these changes affect actual predictions. I think there was enough space in the paper to investigate additional related research questions and make the work more complete and interesting.
- The paper reports larger EQDZ values for W4A4 than W8A8. However W4A4 already has a higher quantization error before perturbation. On Cora the relative increase compared to baseline is about 19% for W8A8 and 16% for W4A4; W4A4 has larger absolute divergence, but it is not clear if it is more sensitive in relative terms. Reporting baseline normalized EQDZ for all datasets and separating absolute and relative sensitivity can be useful.
- The opposite behavior on roman empire is interesting but the evidence comes from only one heterophilous dataset. It is difficult to know if this is caused by the architecture, heterophily, or something specific to roman empire. Testing more heterophilous datasets can be useful.
- In the introduction, the authors motivate the problem by explaining that real graphs change over time. I understand the choice of a controlled rewiring method, since it helps to isolate specific numerical effects. However, given this motivation, I think it would also make sense to test the method on temporal graph datasets, where the topology actually changes over time.
- CP-DPR preserves degrees and connected components, but the are others structural properties that can vary (for example homophily, and this is connect to W5). Analyzing which graph properties are most related to divergence could help explain the results.
Questions for authors
See weaknesses.