- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes to apply the framework of Besov spaces to problems of robustness in neural networks. Besov spaces provide a means of simultaneously characterizing the smoothness of functions at multiple scales. The authors propose to apply this to problems of robustness in neural networks, proposing to evaluate and design for perturbations at different scales. They observe that this provides a more general framework than, say, criteria based on Lipschitz constants or Hessian matrices, and try to illustrate the utility of this framework via two applications. The first is an attempt to robustify vision classifiers by adding a roughness penalty to the loss function. The authors demonstrate increased robustness to noise. The second is post-training quantization of LLMs, where the authors attempt to identify blocks of weights which are more sensitive to quantization error via probes whose scale is matched to that of the quantization error, and claim that their predictions are more accurate than Hessian-based techniques. Unfortunately, the authors have not chosen to compare their schemes against the SOTA for each application, and thus have not demonstrated the utility of their proposed approach.
Critical strengths
In principle, Besov spaces might provide a useful mental model for designing robustness into neural networks, motivating a more systematic study of the impact of perturbations at multiple scales
Critical weaknesses
It appears that the idea of Besov spaces is mainly used here to motivate the concept of perturbations at multiple scales, not to provide a fundamentally new framework for robustness. As such, the authors seem to be dressing up a natural idea (match the evaluation criterion for robustness to the scale of perturbations you expect) in a mathematical framework. I do not see any fundamental consequences, unlike, say, applications to neural network approximation and generalization bounds, where Besov smoothness assumptions play a critical role.
The authors have not demonstrated the utility of their framework in either of the applications they select. For the vision application, robustness to noise can be easily obtained by data augmentation techniques with no degradation in clean accuracy. For post-training quantization of LLMs, the authors have chosen to compare against an HAWQ-V2 baseline (modifying a scheme originally proposed for CNNs in 2020) rather than SOTA for mixed precision quantization of LLMs developed over the last two years.
The paper is poorly written and organized, with no clear claims of innovation and no clear description of related work which is SOTA for the applications considered.
Note on revised version: The revised paper has an improved and more self-contained presentation.
I do not appreciate being bombarded with a 70 page appendix. If the authors are counting on me reading it to provide an evaluation of their contribution, they are out of luck!
Questions for authors
In the vision application, what exactly are your claims of innovation? You have not compared against SOTA augmentation strategies to evaluate your approach to noise robustness.
For post-training mixed precision quantization of LLMs, why have you not considered SOTA over the past couple of years (e.g., SqueezeLLM, SpQR etc)?
Note on revised version: I appreciate the authors' effort to combine their ideas with SOTA approaches in each of the two applications, and am upgrading my rating of the paper.
5 comments
-
Measuring loss after quantization can identify important blocks that local derivatives miss. Theorem 6 constructs blocks with the same loss, slope, and curvature at the original weights but different losses after quantization. For example, , with , describes the loss increase along a block's quantization error. At the original weights (), the cubic term is invisible to the first two derivatives; after the full error (), it adds to the damage. Measuring this endpoint reveals which blocks benefit most from higher precision.
On this additive class, any deterministic allocator using only the starting value, slope, and curvature has worst-case extra loss , while exact endpoint allocation is optimal; of blocks receive higher precision. Theorem 5 bounds the gap from the best allocation in terms of block interactions and calibration error: accurate scores and small interactions give a near-optimal allocation. The controlled Taylor experiment tests the practical importance of the missing information: changing one protected block on Qwen2.5-1.5B GPTQ lowers PPL from 12.5915 to 11.0104, a 12.56% reduction at the same budget.
-
Precision should protect the probability of the correct token. KL rewards matching the original output distribution; task loss rewards assigning high probability to the observed correct token. For example, moving probability between two wrong tokens while keeping the correct token at 60% leaves its prediction loss unchanged. Lowering the correct token from 60% to 50% increases that loss, even if the whole distribution changes less by KL. Matching the original output more closely can therefore select an allocation that predicts the correct token less well. Theorem 4 gives the exact difference through the target-alignment term and shows that the block rankings can reverse. In the preregistered Pythia-12B check, task-damage measurements correlate with block damage at , compared with for drift. Measuring the loss saved by extra precision directs the budget toward prediction quality. The EvoPress-style comparison below tests this choice within a recent allocation search.
The Besov construction turns the deployment-scale principle into a measurement framework with concrete decisions attached. RSC uses finite-band second differences to control margin slopes at chosen scales and directions. For allocation, the signed task-loss change adds the target information needed to distinguish harmful variation from harmless variation or improvement. The separation results establish why these measurements contain information that local summaries can miss; the bounds state when using that information controls prediction changes and allocation loss. We have placed these consequences, their assumptions, and the corresponding empirical tests in revised Secs. 2–4.
W2 / Q1: RSC adds robustness to strong augmentation recipes
RSC improves robustness even when the training recipe already includes AugMix and JSD consistency. We keep that recipe fixed and test the additional benefit of controlling how the classification margin changes under noise. RSC evaluates the original image and two equally spaced perturbed versions, then penalizes the squared difference between the two margin slopes. A constant slope contributes zero. The radius sets the perturbation size, and the direction family sets its structure. Matched comparisons use the same perturbed images for additional augmentation, consistency, or RSC, allowing us to examine the contribution of the objective. A separate control tests the benefit when training already includes Gaussian-noise augmentation.
We compare with AugMix (https://arxiv.org/abs/1912.02781) and PRIME (https://arxiv.org/abs/2112.13547) and measure what RSC adds to AugMix. We use AugMix with its JSD consistency term and PRIME with the authors' augmentation implementation. The ResNet-18 comparison below uses five training seeds per row. AugMix and AugMix + RSC share the same 200-epoch training protocol; RSC uses the submitted band and . Auxiliary forwards retain the submission's training-mode BatchNorm updates; we also test whether the added benefit persists when these updates are disabled, below. PRIME on CIFAR-100 uses its own 100-epoch one-cycle recipe with gradient clipping; its CIFAR-10 reference uses the common 200-epoch recipe. Values are mean ± sample SD; C-all averages all 15 CIFAR-C corruptions and all five severities.
| Dataset | Method | Clean | Gaussian | Gaussian | C-all |
|---|---|---|---|---|---|
| CIFAR-10 | AugMix + JSD | 95.71 ± 0.14 | 89.49 ± 0.26 | 67.92 ± 1.59 | 89.35 ± 0.16 |
| CIFAR-10 | AugMix + JSD + RSC | 95.58 ± 0.25 | 90.88 ± 0.39 | 73.54 ± 0.96 | 89.80 ± 0.22 |
| CIFAR-10 | PRIME + JSD | 92.59 ± 0.16 | 90.84 ± 0.30 | 87.09 ± 0.34 | 89.38 ± 0.13 |
| CIFAR-100 | AugMix + JSD | 76.89 ± 0.31 | 61.02 ± 0.65 | 32.73 ± 0.75 | 65.00 ± 0.27 |
| CIFAR-100 | AugMix + JSD + RSC | 76.70 ± 0.30 | 65.98 ± 1.01 | 40.22 ± 1.41 | 66.17 ± 0.18 |
| CIFAR-100 | PRIME + JSD, own training recipe | 77.22 ± 0.26 | 71.59 ± 0.23 | 63.19 ± 0.39 | 68.25 ± 0.11 |
RSC improves AugMix across both datasets, both Gaussian noise levels, and mean CIFAR-C accuracy, while keeping clean accuracy within 0.2 points of AugMix. The gains are 1.40 / 4.96 points at , 5.62 / 7.48 points at , and 0.45 / 1.16 points on mean CIFAR-C accuracy, for CIFAR-10 / CIFAR-100. The clean-accuracy changes are −0.13 / −0.19 points.
RSC retains more clean accuracy at comparable Gaussian robustness in a matched AugMix ablation on CIFAR-100. We compared the additional objectives in a separate three-seed experiment, using and the same isotropic/high-frequency perturbations. RSC uses , with the same setting on both datasets. The CIFAR-100 results are:
| AugMix-based recipe | Clean | Gaussian | Gaussian | C-all |
|---|---|---|---|---|
| AugMix + JSD | 76.95 ± 0.29 | 63.20 ± 1.03 | 35.09 ± 1.51 | 65.41 ± 0.07 |
| + matched noise augmentation | 75.38 ± 0.56 | 68.93 ± 0.57 | 44.40 ± 2.13 | 64.89 ± 0.58 |
| + matched KL consistency | 75.89 ± 0.11 | 69.44 ± 0.11 | 46.46 ± 1.11 | 66.46 ± 0.15 |
| + RSC | 76.67 ± 0.29 | 69.48 ± 0.44 | 46.15 ± 0.42 | 66.98 ± 0.18 |
At similar Gaussian accuracy, RSC retains 1.30 more clean-accuracy points than matched augmentation and 0.78 more than matched KL consistency, and has the highest mean C-all accuracy in this comparison. On CIFAR-10, the same RSC setting improves AugMix from 90.34% to 91.95% Gaussian accuracy and from 89.48% to 90.24% C-all accuracy, with clean accuracy 95.38% → 95.54%. These comparisons establish the benefit of the RSC objective on the same perturbed images.
The benefit also extends to an augmented vision transformer, where both clean and noise accuracy improve. The submitted ViT-S/4 CIFAR-10 recipe uses RandAugment(2,9), random crop, and horizontal flip in both arms (Appendix E). Adding the default RSC objective raises clean accuracy from 90.31% to 91.44%, CIFAR-C noise accuracy from 73.01% to 79.50%, and Gaussian accuracy at from 77.28% to 85.72% (Table 8). These are means over two training seeds per arm.
The added benefit persists when auxiliary probes cannot change BatchNorm running statistics. This control tests whether the RSC objective improves AugMix when those statistical updates are excluded. In a separate seven-seed FP32 experiment, RSC adds 1.44 points of CIFAR-100 Gaussian accuracy, paired 95% CI [0.57, 2.31], with all seven pairs improving. The full protocol and clean-accuracy results are reported under ZcQB W3.
RSC also improves corruption robustness after explicit Gaussian-noise augmentation. In six additional, independent CIFAR-100 seed pairs, adding RSC improves C-all from 59.04 ± 0.45% to 59.74 ± 0.35%, a paired gain of 0.70 points [0.21, 1.19], with all six pairs improving. Clean accuracy changes by −0.10 points [−0.49, 0.29]. The added benefit therefore persists when training already includes the Gaussian noise used in the primary noise test.
These experiments establish the application contribution: margin-slope control adds robustness to augmented classifiers, with a favorable clean/robustness trade-off in the matched AugMix comparison. The gain persists with explicit Gaussian-noise augmentation and when auxiliary BatchNorm updates are excluded. The radius and direction ablations in our response to ZcQB show how to target that control. We present the mechanism, comparisons, and uncertainty together in revised Sec. 4.1 and Appendix M.1–M.2.
W2 / Q2: Comparisons with SpQR, SqueezeLLM, and EvoPress (ICML 2025)
The new comparisons show that the allocation criterion adds value within modern quantizers and recent allocation search. We use the authors' SpQR (https://arxiv.org/abs/2306.03078) and SqueezeLLM (https://proceedings.mlr.press/v235/kim24f.html) reconstruction routines and compare with a blockwise reproduction of EvoPress (ICML 2025) (https://proceedings.mlr.press/v267/sieberling25a.html). Task-loss allocation improves over Fisher allocation in all 12 SpQR/SqueezeLLM settings under common storage caps. The EvoPress-style comparison establishes competitive direct allocation and shows that using task loss also improves the search itself. These results test the criterion both when constructing a fixed allocation and when searching over combinations.
Thank you for asking what the Besov formulation enables and whether its benefits extend to strong existing methods. The framework supports two concrete uses: training image classifiers to resist noise and choosing where quantized language models need more precision. Both use responses measured at perturbation sizes and directions relevant to the task. Besov theory supplies the finite-difference tools for measuring these responses and their changes in slope. The results below show what this information enables mathematically and how using it adds robustness to AugMix, improves allocation within the authors' SpQR and SqueezeLLM reconstruction routines, and improves the search criterion in a blockwise reproduction of EvoPress (ICML 2025). The vision comparison also includes PRIME.
W1: Why Besov is a natural tool, and what it enables
Besov theory is a natural and powerful tool for deployment-scale sensitivity because its basic measurements are finite differences across scales. These measurements compare responses a finite distance apart, remain meaningful when that step crosses an activation boundary, and can separate changes in slope from affine variation. Scale normalization makes responses at different perturbation sizes comparable. This is precisely the structure needed to ask which perturbations change a model's behavior and at what scale.
Theorem 1 makes the connection precise: under its sampling assumptions, the canonical full-grid population aggregate, including its Lp anchor, is equivalent to a Besov quasi-norm. The finite-band, direction-specific measurements then specialize this construction to deployment perturbations. The resulting tools identify information that scalar summaries miss, connect margin variation to prediction stability, and guide the measurements used for precision allocation. Four concrete consequences follow.
The same Lipschitz constant and Hessian can hide different patterns of sensitivity. Consider triangular waves with slopes +1 and −1. Some bend frequently; others have long straight segments. All have Lipschitz constant 1 and second derivative zero away from the bends. Measuring finite differences at several step sizes reveals how closely the bends are spaced. Theorem 2 makes this distinction quantitative: for fN(x)=dist(Nx,Z)/N, these two summaries stay fixed while the homogeneous Besov seminorm grows as Ns−1 for 1<s<1+1/p. The spectrum therefore measures information that these summaries leave undetermined. In piecewise-affine networks, this matters when a perturbation crosses an activation or margin boundary.
The geometry also explains the scale correction. Under the stated continuous-density assumptions and a positive boundary-crossing coefficient, second-difference amplitude scales as r3/2 for continuous piecewise-affine responses. The s=1.5 diagnostic removes this expected step-size effect; the s=1 training objective measures changes in slope directly.
Controlling margin variation has a direct link to stable predictions. The margin is the correct class's score minus its strongest competitor's score. A perturbation can turn a correct prediction into an error by driving this margin below zero. Theorem 3 bounds the margin change using two measurable parts: the overall change at a larger radius, called the coarse anchor, and changes in slope across smaller radii, measured by second differences. For label-preserving probes with the stated common-direction setup across scales, these terms and the clean-margin distribution bound the probability of a prediction change. RSC directly controls the slope-change part of this bound. The radius and direction specify which perturbations receive this control.