- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This manuscript studies the robustness of image classification with respect to optical aberrations in camera lenses. The authors propose a mathematical framework with eight Zernike coefficients, whose randomized smoothing yields a robustness certificate. Extensive experiments, including ImageNet and several Transformers, were performed.
Critical strengths
I like the use of the optical approach with the mathematical framework, which is also appropriately connected to obtain the robustness certificate.
Critical weaknesses
- I would like to acknowledge extensive experiments including ImageNet, but the target models are still Transformers in tiny size. More large-scale models would make this study more convincing.
- The ALC attack is 24-dimensional, which is projected into 8-dimension in CROCO. The low flip rate in Table 18 could indicate the effect of projection itself. A more controlled ablation study is required.
- For comparison of modes such as defocus and spherical, the same Zernike coefficients were used, but the RMS wavefront error itself could be affected by the mode choice. In other words, the comparison could also be affected by the aberration severity. This also requires more controlled experiments.
- Eq. 3 uses Gaussian smoothing with \sigma^2 I_d, which is replaced with \sigma^2 \Pi_S in Appendix B to yield Eq. 6. The two are different things, right?
- After Eq. 10, the authors used R=0.79 as average certified radius. However, the RMS threshold with the Marechal criterion requires a worse-case, lower bound, not the average.
- Is the dimension-free correct? For example, squared-norm is dependent on the dimension d; a 3x dimension (from 8 to 24) would yield \sqrt{3}x increase.
- Writing should be improved:
- “That ellipsoid” → “That ball”
- “Thus, amplitude based imaging would therefore be resolvable.”
- “the whole sphere” → “the whole ball”
- “across noise level for ResNet50” → “across noise levels for ResNet50”
- “non adversarially” → “non-adversarially”
- “safety critical systems” → “safety-critical systems”
- “low dimensional set” → “low-dimensional set”
- “pixel space certification” → “pixel-space certification” → I doubt that the authors intentionally removed all hyphens, possibly to humanize AI writing.
Questions for authors
See the weakness above.
5 comments
Thank you for the detailed responses and revisions. Most of my concerns have been adequately addressed. I have only three remaining points:
- Section 4.3 and Table 22 still seem to suggest that the 8D certificate covers the original 24D attack, although these are different settings.
- The equal-RMS results are close to the sampling limit, making it difficult to conclude that all modes are equally robust.
- The SwinV2-base experiment is still in progress, so this concern remains open.
Thank you for the follow-up. We uploaded a revision that addresses the three remaining points. A new Table 6 shifts the later tables by one, so the former Table 22 is now Table 23.
8D certificate and the 24D ALC attack
Section 4.3 now separates two settings. The 8D reduction of the chromatic ALC kernels is a size comparison only, since a chromatic kernel lies outside the space the 8D certificate covers, and the 69 to 84 percent is no coverage claim. The test of the original attack is the full 24D ALC against a ResNet50 certified in all 24 dimensions. Inside the radius, the smoothed classifier never flips on the 501 images, including the 34 on which ALC fools the base classifier. Appendix L and the caption of Table 23 now state this and refer to Table 25.
Equal-RMS results near the sampling limit
We agree and removed the claim of equal robustness. Raising σ does not help, because the ceiling of 3.199σ scales with it. On ImageNet1k (ResNet50, 0.125 waves, Table 6) the tolerances range from 0.275 to 0.294 waves, and every other mode lies 2 to 9 standard errors below the strongest. The flip rate 1 − n_A/n, which has no ceiling, is 0.39 percent for astigmatism and 1.49 percent for spherical aberration. Section 4.2 and Appendix B now state only this.
Larger transformer model
The SwinV2 base on ImageNet100 with σ=1 reaches acc@0 75.2 percent and ACR 2.01, against 74.4 percent and 2.00 for SwinV2 tiny, at 43.6 s per image. A newly trained ViT-B/16 reaches 77.5 percent and an ACR of 2.10. Both are in Table 2, with costs and recipes in Appendix G. The SwinV2 base runs at σ=2, σ=4 and without noise are still training and will be added to Table 2 in the final version.
Response to Reviewer fjUx
Thank you for your thorough review and the valuable comments on the theoretical details.
We agree that the projection has to be separated from the smoothing. Appendix L and the caption of Table 23 now state that this table uses the channel-averaged kernel and restrict its conclusions to this variant. Our ablation with original and averaged kernels confirms that averaging weakens the kernel. We therefore tested the attack without projection, running the original 24-dimensional ALC against a ResNet50 certified over all 24 dimensions (Imagenette, σ=1). Inside the certified radius the smoothed classifier never flips, on all 501 images, including 34 on which ALC breaks the base classifier. Outside the radius, where the certificate makes no claim, it flips the smoothed classifier on 31.8 percent of the 85 images on which ALC succeeds. Appendix L reports both.
Thank you for this point. A unit coefficient contributes 1/N_i waves RMS, which ranges from 1/3 for defocus to 1/8 for coma and trefoil, so equal coefficient norm does not mean equal wavefront error. Following your suggestion, we ran the controlled experiment. We certified the ResNet18 on Imagenette once per mode with noise σ_i = 0.125·N_i, which gives every mode the same wavefront error of 0.125 waves RMS. All eight modes are then certified to a tolerance between 0.343 and 0.350 waves RMS, so at equal wavefront error the model is equally robust to all modes. The finding that defocus and spherical aberration are weakest holds at equal coefficient norm. The abstract, introduction, Section 4.2 and Appendix B now state this, and Table 12 reports the new result.
Yes, they are different. N(0, σ^2 Π_S) is a degenerate Gaussian on R^d, which yields a different smoothed classifier and certifies only perturbations in the range of Π_S, which is what a per-mode certificate requires. Appendix B now makes the reduction explicit.
You are right that the example after Equation 10 should use the radius of each image rather than the average. The guarantee itself holds per image and is not affected, only the illustration changes. The Maréchal ball is contained in an image's certificate if that image's radius is at least 8/14, and the revised Appendix F reports this per image. In the configuration with the smallest average radius, it holds for 87.4 percent of the correctly classified images and for 55.5 percent of all evaluated images.
The radius formula in Equation 5 has no explicit dependence on d, and this is what we meant by "dimension-free". However, the noise norm grows as σ√d, so p_A can change with d. In addition, equal radii are not directly comparable across dimensions. A monochromatic aberration repeats its eight coefficients in all three color channels, thus its norm is √3 times larger at d=24. A d=24 certificate of radius r therefore covers monochromatic aberrations only up to r/√3. We clarified this in Section 3, Section 4.2, Appendix C and the caption of Table 6.
Certification treats the classifier as a black box, so the method places no constraint on model size, and the cost scales with the forward pass (Appendix G for the models we use). We agree that a larger model would make the study more convincing. We are therefore training and certifying a SwinV2 base on ImageNet100 with the same protocol as Table 2. After 37 of 90 epochs it reaches 73.0 percent test accuracy, and we will add its certification to the paper as soon as it finishes.
Thank you for the list. All listed points are corrected, and we fixed the missing hyphens throughout.