- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Quantization-Aware Preconditioning (QAP) for physics-informed neural networks, motivated by the observation that preconditioning affects not only optimization conditioning but also the amplification of quantization-induced residual errors. The authors develop a theoretical framework that separates optimization contraction from quantization noise injection, and propose isotropic and covariance-aware regularization strategies based on the physical tangent geometry and estimated quantization residual statistics. Several covariance estimation and low-rank approximation methods are introduced, with experiments on heterogeneous diffusion problems showing improvements over unpreconditioned and full-precision-derived baselines.
Critical strengths
The paper makes an interesting contribution by explicitly considering how residual preconditioning influences quantization-induced errors, an interaction that is often overlooked in existing preconditioning methods. The authors further develop a theoretically motivated optimization framework that balances conditioning improvement and quantization noise amplification, and propose practical preconditioner constructions consistent with this analysis. I find this connection between quantization geometry and preconditioner design novel and potentially valuable.
Critical weaknesses
-
The experimental results do not always clearly demonstrate the advantages of the proposed method, despite some seemingly substantial numerical improvements. In particular, the FP-derived baseline is described as "deliberately strong", yet in the 2D experiment it performs about 3.7× worse than no preconditioning (0.2623 vs. 0.0716). Thus, the reported 5.32× improvement over this baseline is less convincing, while the improvement over identity is 1.45×. Moreover, as the authors acknowledge in Appendix I, covariance-aware QAP is not statistically significantly better than QAP-I in the three main 1D end-to-end settings, although the gated 2D variant does show a significant improvement. I would like to see a clearer discussion of what these numerical results actually establish.
-
The baseline comparisons seem relatively limited. The principal end-to-end experiments compare against no preconditioning and a full-precision-derived preconditioner, besides the proposed QAP variants. However, PINN preconditioning methods such as De Ryck et al. (2024) and Liu et al. (2024) are discussed in the related work but not evaluated experimentally. Could the authors provide comparisons with stronger existing preconditioning methods, such as operator-based or ILU-based approaches where applicable? This would better establish the practical significance of QAP beyond the current baselines.
-
The accuracy and reliability of the estimated quantization covariance deserve further investigation. While the theoretical motivation for covariance-aware regularization is reasonable, the experiments suggest that different covariance estimation and approximation strategies can lead to substantially different results. In particular, some covariance-aware variants perform worse than isotropic QAP, and the ungated QAP-Cov+ exhibits a large failure in the 2D experiment. Appendix K.2 provides encouraging agreement between the analytic and source-based geometry estimates, but this does not directly establish their accuracy relative to the joint residual perturbation geometry of the fully quantized network. Moreover, there is limited analysis of how the number of calibration probes and covariance estimation errors affect final performance. Additional experiments investigating these relationships, particularly in cases where covariance-aware regularization degrades performance, would make the proposed mechanism more convincing.
-
QAP-I does not explicitly incorporate directional quantization residual covariance, although it uses the quantized tangent geometry. Nevertheless, it matches or outperforms covariance-aware variants in several settings, including Helmholtz, ungated 2D QAP-Cov+, and W1A8. In addition, the covariance-aware gains in the main 1D end-to-end experiments are not statistically significant. While Appendix E and the gated 2D experiments provide evidence that directional covariance information can be beneficial, it remains unclear under what conditions the additional covariance estimation is necessary and whether its computational cost is justified. A more systematic explanation of when covariance-aware regularization helps or hurts would strengthen the paper's central motivation.
Questions for authors
-
Can the authors provide comparisons with stronger existing PINN preconditioning methods and clarify the significance of the reported improvements, particularly when the FP-derived baseline performs substantially worse than no preconditioning?
-
How accurately do the estimated covariance matrices capture the actual quantization residual geometry? Can the authors further investigate how covariance estimation errors and the number of calibration probes affect final performance, especially in cases where covariance-aware methods perform worse?
-
Appendix E and the 2D experiment provide evidence for the benefit of directional covariance information. However, the improvement over QAP-I is not significant in the main 1D end-to-end settings, and covariance-aware variants perform worse in some other cases. What explains these differences? Can the authors further validate whether the proposed reliability criteria consistently predict when directional covariance information will be beneficial across different PDEs?
-
Appendix J.3 describes a general low-rank quantization-aware correction on top of an existing preconditioner. Could the authors evaluate this approach with an ILU-based or another structured preconditioner? This is not a major concern, but such an extension could further demonstrate the general applicability of QAP.