acceptodds
Under review as a conference paper at ICLR 2027

Scalar Smoothness Is One Point on a Besov Spectrum

Abstract

Studies of neural network generalization and robustness often summarize smoothness with a single number, such as a Lipschitz constant or a Hessian curvature statistic. However, Lipschitz constants summarize global worst-case sensitivity and Hessian statistics summarize local curvature; neither captures how model responses vary with the size, direction, and location of perturbations encountered in real-world deployment. To measure smoothness under these perturbations, we introduce a finite-difference roughness spectrum that profiles model responses across these conditions. Our roughness spectrum generalizes scalar smoothness measures, bringing Lipschitz sensitivity and directional Hessian curvature into a common framework across perturbation scales. We connect this spectrum to classical Besov theory, which describes how a function's variation changes with scale. Our theoretical analysis also proves that a larger output shift can cause less task damage than a smaller one, and that local curvature alone can miss the blocks most vulnerable to finite perturbations. In image classification, Roughness Source Control (RSC) suppresses margin roughness along noise directions and improves noise robustness in a comparison of 80 checkpoints across four model–dataset combinations. In post-training quantization, we identify blocks that need higher precision by measuring task-loss increases from perturbations matched to the size and direction of each block's quantization error. Our mixed-precision allocation achieves lower perplexity than the Hessian-based HAWQ-V2 baseline under the same GPTQ weights and bit budgets in all 14 settings across seven held-out large language models.

Primary area:
learning theory
Keywords:
neural network smoothness, model robustness, Besov spaces, finite differences, perturbation sensitivity, post-training quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Mu5tOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper proposes to apply the framework of Besov spaces to problems of robustness in neural networks. Besov spaces provide a means of simultaneously characterizing the smoothness of functions at multiple scales. The authors propose to apply this to problems of robustness in neural networks, proposing to evaluate and design for perturbations at different scales. They observe that this provides a more general framework than, say, criteria based on Lipschitz constants or Hessian matrices, and try to illustrate the utility of this framework via two applications. The first is an attempt to robustify vision classifiers by adding a roughness penalty to the loss function. The authors demonstrate increased robustness to noise. The second is post-training quantization of LLMs, where the authors attempt to identify blocks of weights which are more sensitive to quantization error via probes whose scale is matched to that of the quantization error, and claim that their predictions are more accurate than Hessian-based techniques. Unfortunately, the authors have not chosen to compare their schemes against the SOTA for each application, and thus have not demonstrated the utility of their proposed approach.

Critical strengths

In principle, Besov spaces might provide a useful mental model for designing robustness into neural networks, motivating a more systematic study of the impact of perturbations at multiple scales

Critical weaknesses

It appears that the idea of Besov spaces is mainly used here to motivate the concept of perturbations at multiple scales, not to provide a fundamentally new framework for robustness. As such, the authors seem to be dressing up a natural idea (match the evaluation criterion for robustness to the scale of perturbations you expect) in a mathematical framework. I do not see any fundamental consequences, unlike, say, applications to neural network approximation and generalization bounds, where Besov smoothness assumptions play a critical role.

The authors have not demonstrated the utility of their framework in either of the applications they select. For the vision application, robustness to noise can be easily obtained by data augmentation techniques with no degradation in clean accuracy. For post-training quantization of LLMs, the authors have chosen to compare against an HAWQ-V2 baseline (modifying a scheme originally proposed for CNNs in 2020) rather than SOTA for mixed precision quantization of LLMs developed over the last two years.

The paper is poorly written and organized, with no clear claims of innovation and no clear description of related work which is SOTA for the applications considered.

Note on revised version: The revised paper has an improved and more self-contained presentation.

I do not appreciate being bombarded with a 70 page appendix. If the authors are counting on me reading it to provide an evaluation of their contribution, they are out of luck!

Questions for authors

In the vision application, what exactly are your claims of innovation? You have not compared against SOTA augmentation strategies to evaluate your approach to noise robustness.

For post-training mixed precision quantization of LLMs, why have you not considered SOTA over the past couple of years (e.g., SqueezeLLM, SpQR etc)?

Note on revised version: I appreciate the authors' effort to combine their ideas with SOTA approaches in each of the two applications, and am upgrading my rating of the paper.

5 comments

Thank you for asking what the Besov formulation enables and whether its benefits extend to strong existing methods. The framework supports two concrete uses: training image classifiers to resist noise and choosing where quantized language models need more precision. Both use responses measured at perturbation sizes and directions relevant to the task. Besov theory supplies the finite-difference tools for measuring these responses and their changes in slope. The results below show what this information enables mathematically and how using it adds robustness to AugMix, improves allocation within the authors' SpQR and SqueezeLLM reconstruction routines, and improves the search criterion in a blockwise reproduction of EvoPress (ICML 2025). The vision comparison also includes PRIME.

W1: Why Besov is a natural tool, and what it enables

Besov theory is a natural and powerful tool for deployment-scale sensitivity because its basic measurements are finite differences across scales. These measurements compare responses a finite distance apart, remain meaningful when that step crosses an activation boundary, and can separate changes in slope from affine variation. Scale normalization makes responses at different perturbation sizes comparable. This is precisely the structure needed to ask which perturbations change a model's behavior and at what scale.

Theorem 1 makes the connection precise: under its sampling assumptions, the canonical full-grid population aggregate, including its LpL^p anchor, is equivalent to a Besov quasi-norm. The finite-band, direction-specific measurements then specialize this construction to deployment perturbations. The resulting tools identify information that scalar summaries miss, connect margin variation to prediction stability, and guide the measurements used for precision allocation. Four concrete consequences follow.

  1. The same Lipschitz constant and Hessian can hide different patterns of sensitivity. Consider triangular waves with slopes +1+1 and −1-1. Some bend frequently; others have long straight segments. All have Lipschitz constant 1 and second derivative zero away from the bends. Measuring finite differences at several step sizes reveals how closely the bends are spaced. Theorem 2 makes this distinction quantitative: for fN(x)=dist⁡(Nx,Z)/Nf_N(x)=\operatorname{dist}(Nx,\mathbb{Z})/N, these two summaries stay fixed while the homogeneous Besov seminorm grows as Ns−1N^{s-1} for 1<s<1+1/p1 < s < 1 + 1/p. The spectrum therefore measures information that these summaries leave undetermined. In piecewise-affine networks, this matters when a perturbation crosses an activation or margin boundary.

    The geometry also explains the scale correction. Under the stated continuous-density assumptions and a positive boundary-crossing coefficient, second-difference amplitude scales as r3/2r^{3/2} for continuous piecewise-affine responses. The s=1.5s=1.5 diagnostic removes this expected step-size effect; the s=1s=1 training objective measures changes in slope directly.

  2. Controlling margin variation has a direct link to stable predictions. The margin is the correct class's score minus its strongest competitor's score. A perturbation can turn a correct prediction into an error by driving this margin below zero. Theorem 3 bounds the margin change using two measurable parts: the overall change at a larger radius, called the coarse anchor, and changes in slope across smaller radii, measured by second differences. For label-preserving probes with the stated common-direction setup across scales, these terms and the clean-margin distribution bound the probability of a prediction change. RSC directly controls the slope-change part of this bound. The radius and direction specify which perturbations receive this control.

  1. Measuring loss after quantization can identify important blocks that local derivatives miss. Theorem 6 constructs blocks with the same loss, slope, and curvature at the original weights but different losses after quantization. For example, ϕi(t)=ct2/2+βit3\phi_i(t)=ct^2/2+\beta_i t^3, with βi∈{0,M}\beta_i\in\lbrace 0,M\rbrace, describes the loss increase along a block's quantization error. At the original weights (t=0t=0), the cubic term is invisible to the first two derivatives; after the full error (t=1t=1), it adds βi\beta_i to the damage. Measuring this endpoint reveals which blocks benefit most from higher precision.

    On this additive class, any deterministic allocator using only the starting value, slope, and curvature has worst-case extra loss min⁡(K,L−K)M\min(K,L-K)M, while exact endpoint allocation is optimal; KK of LL blocks receive higher precision. Theorem 5 bounds the gap from the best allocation in terms of block interactions and calibration error: accurate scores and small interactions give a near-optimal allocation. The controlled Taylor experiment tests the practical importance of the missing information: changing one protected block on Qwen2.5-1.5B GPTQ lowers PPL from 12.5915 to 11.0104, a 12.56% reduction at the same budget.

  2. Precision should protect the probability of the correct token. KL rewards matching the original output distribution; task loss rewards assigning high probability to the observed correct token. For example, moving probability between two wrong tokens while keeping the correct token at 60% leaves its prediction loss unchanged. Lowering the correct token from 60% to 50% increases that loss, even if the whole distribution changes less by KL. Matching the original output more closely can therefore select an allocation that predicts the correct token less well. Theorem 4 gives the exact difference through the target-alignment term and shows that the block rankings can reverse. In the preregistered Pythia-12B check, task-damage measurements correlate with block damage at +0.973+0.973, compared with −0.221-0.221 for drift. Measuring the loss saved by extra precision directs the budget toward prediction quality. The EvoPress-style comparison below tests this choice within a recent allocation search.

The Besov construction turns the deployment-scale principle into a measurement framework with concrete decisions attached. RSC uses finite-band second differences to control margin slopes at chosen scales and directions. For allocation, the signed task-loss change adds the target information needed to distinguish harmful variation from harmless variation or improvement. The separation results establish why these measurements contain information that local summaries can miss; the bounds state when using that information controls prediction changes and allocation loss. We have placed these consequences, their assumptions, and the corresponding empirical tests in revised Secs. 2–4.

W2 / Q1: RSC adds robustness to strong augmentation recipes

RSC improves robustness even when the training recipe already includes AugMix and JSD consistency. We keep that recipe fixed and test the additional benefit of controlling how the classification margin changes under noise. RSC evaluates the original image and two equally spaced perturbed versions, then penalizes the squared difference between the two margin slopes. A constant slope contributes zero. The radius sets the perturbation size, and the direction family sets its structure. Matched comparisons use the same perturbed images for additional augmentation, consistency, or RSC, allowing us to examine the contribution of the objective. A separate control tests the benefit when training already includes Gaussian-noise augmentation.

We compare with AugMix (https://arxiv.org/abs/1912.02781) and PRIME (https://arxiv.org/abs/2112.13547) and measure what RSC adds to AugMix. We use AugMix with its JSD consistency term and PRIME with the authors' augmentation implementation. The ResNet-18 comparison below uses five training seeds per row. AugMix and AugMix + RSC share the same 200-epoch training protocol; RSC uses the submitted band {0.5,1,2,4}\lbrace 0.5,1,2,4\rbrace and λ=0.05\lambda=0.05. Auxiliary forwards retain the submission's training-mode BatchNorm updates; we also test whether the added benefit persists when these updates are disabled, below. PRIME on CIFAR-100 uses its own 100-epoch one-cycle recipe with gradient clipping; its CIFAR-10 reference uses the common 200-epoch recipe. Values are mean ± sample SD; C-all averages all 15 CIFAR-C corruptions and all five severities.

DatasetMethodCleanGaussian σ=0.05\sigma=0.05Gaussian σ=0.1\sigma=0.1C-all
CIFAR-10AugMix + JSD95.71 ± 0.1489.49 ± 0.2667.92 ± 1.5989.35 ± 0.16
CIFAR-10AugMix + JSD + RSC95.58 ± 0.2590.88 ± 0.3973.54 ± 0.9689.80 ± 0.22
CIFAR-10PRIME + JSD92.59 ± 0.1690.84 ± 0.3087.09 ± 0.3489.38 ± 0.13
CIFAR-100AugMix + JSD76.89 ± 0.3161.02 ± 0.6532.73 ± 0.7565.00 ± 0.27
CIFAR-100AugMix + JSD + RSC76.70 ± 0.3065.98 ± 1.0140.22 ± 1.4166.17 ± 0.18
CIFAR-100PRIME + JSD, own training recipe77.22 ± 0.2671.59 ± 0.2363.19 ± 0.3968.25 ± 0.11

RSC improves AugMix across both datasets, both Gaussian noise levels, and mean CIFAR-C accuracy, while keeping clean accuracy within 0.2 points of AugMix. The gains are 1.40 / 4.96 points at σ=0.05\sigma=0.05, 5.62 / 7.48 points at σ=0.1\sigma=0.1, and 0.45 / 1.16 points on mean CIFAR-C accuracy, for CIFAR-10 / CIFAR-100. The clean-accuracy changes are −0.13 / −0.19 points.

RSC retains more clean accuracy at comparable Gaussian robustness in a matched AugMix ablation on CIFAR-100. We compared the additional objectives in a separate three-seed experiment, using r=4r=4 and the same isotropic/high-frequency perturbations. RSC uses λ=0.2\lambda=0.2, with the same setting on both datasets. The CIFAR-100 results are:

AugMix-based recipeCleanGaussian σ=0.05\sigma=0.05Gaussian σ=0.1\sigma=0.1C-all
AugMix + JSD76.95 ± 0.2963.20 ± 1.0335.09 ± 1.5165.41 ± 0.07
+ matched noise augmentation75.38 ± 0.5668.93 ± 0.5744.40 ± 2.1364.89 ± 0.58
+ matched KL consistency75.89 ± 0.1169.44 ± 0.1146.46 ± 1.1166.46 ± 0.15
+ RSC76.67 ± 0.2969.48 ± 0.4446.15 ± 0.4266.98 ± 0.18

At similar Gaussian accuracy, RSC retains 1.30 more clean-accuracy points than matched augmentation and 0.78 more than matched KL consistency, and has the highest mean C-all accuracy in this comparison. On CIFAR-10, the same RSC setting improves AugMix from 90.34% to 91.95% Gaussian accuracy and from 89.48% to 90.24% C-all accuracy, with clean accuracy 95.38% → 95.54%. These comparisons establish the benefit of the RSC objective on the same perturbed images.

The benefit also extends to an augmented vision transformer, where both clean and noise accuracy improve. The submitted ViT-S/4 CIFAR-10 recipe uses RandAugment(2,9), random crop, and horizontal flip in both arms (Appendix E). Adding the default RSC objective raises clean accuracy from 90.31% to 91.44%, CIFAR-C noise accuracy from 73.01% to 79.50%, and Gaussian accuracy at σ=0.05\sigma=0.05 from 77.28% to 85.72% (Table 8). These are means over two training seeds per arm.

The added benefit persists when auxiliary probes cannot change BatchNorm running statistics. This control tests whether the RSC objective improves AugMix when those statistical updates are excluded. In a separate seven-seed FP32 experiment, RSC adds 1.44 points of CIFAR-100 Gaussian accuracy, paired 95% CI [0.57, 2.31], with all seven pairs improving. The full protocol and clean-accuracy results are reported under ZcQB W3.

RSC also improves corruption robustness after explicit Gaussian-noise augmentation. In six additional, independent CIFAR-100 seed pairs, adding RSC improves C-all from 59.04 ± 0.45% to 59.74 ± 0.35%, a paired gain of 0.70 points [0.21, 1.19], with all six pairs improving. Clean accuracy changes by −0.10 points [−0.49, 0.29]. The added benefit therefore persists when training already includes the Gaussian noise used in the primary noise test.

These experiments establish the application contribution: margin-slope control adds robustness to augmented classifiers, with a favorable clean/robustness trade-off in the matched AugMix comparison. The gain persists with explicit Gaussian-noise augmentation and when auxiliary BatchNorm updates are excluded. The radius and direction ablations in our response to ZcQB show how to target that control. We present the mechanism, comparisons, and uncertainty together in revised Sec. 4.1 and Appendix M.1–M.2.

W2 / Q2: Comparisons with SpQR, SqueezeLLM, and EvoPress (ICML 2025)

The new comparisons show that the allocation criterion adds value within modern quantizers and recent allocation search. We use the authors' SpQR (https://arxiv.org/abs/2306.03078) and SqueezeLLM (https://proceedings.mlr.press/v235/kim24f.html) reconstruction routines and compare with a blockwise reproduction of EvoPress (ICML 2025) (https://proceedings.mlr.press/v267/sieberling25a.html). Task-loss allocation improves over Fisher allocation in all 12 SpQR/SqueezeLLM settings under common storage caps. The EvoPress-style comparison establishes competitive direct allocation and shows that using task loss also improves the search itself. These results test the criterion both when constructing a fixed allocation and when searching over combinations.

Reviewer_TAeYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  1. The authors measure network responses across pertubation variants (size, direction, location) by developing a roughness spectrum (finite-difference based). They connect their formulation to Besov regularity.
  2. They develop a regularizer called as Roughness source control (RSC) for mixed-precision quantization of language modes.
  3. They make an interesting obervation on the output distribution shift and performance damage - a large shift need not cause a massive dip in performance.
  4. PTQ experiments use per-block calibration-loss increases to select blocks for higher precision, results are shown across vision architectures as well as language models over GPTQ.
Critical strengths
  1. Drift being negatively correlated with block-level damage is an interesting finding. It offers practical insight into why output-error based proxies can select the wrong blocks, even independently of adopting the full Besov framing.

  2. The empirical validation is extensive and contains important baselines. The allocation rule improves over HAWQ-v2. RSC improves noise/corruption performance across multiple vision architectures. The appendix includes an uncertainty analysis, a Hessian calibration-budget sweep, and a wall-clock-matched SAM control.

Critical weaknesses
  1. The empirical comparison does not cleanly isolate perturbation scale from direction and representation. A trace-based score can mis-rank errors even for a purely quadratic loss if those errors occupy different eigen directions. A local Taylor baseline would therefore be more direct evidence that finite scale. This concern affects the mechanistic interpretation rather than the empirical results.

  2. The practical role of the full spectrum needs clearer separation from the endpoint interventions The framework may have substantial unifying value, but the evidence that its measured spectrum explains or is necessary for the intervention gains is not clearly stated.

Questions for authors
  1. How does the allocation advantage compare against a local Taylor approximation?
  2. Which useful allocation decisions require the spectrum rather than Equation 8's direct task-loss endpoint?
  3. Is the intended PTQ contribution primarily a diagnostic explanation of existing sensitivity scores, or does the spectrum provide an additional operational advantage?
3 comments

Thank you for proposing the same-direction Taylor comparison. Even with the exact error direction and full directional curvature, Taylor can miss a block choice that changes final PPL by 12.56%. The new control traces this gain from a missed change in block priority to the resulting precision allocation, evaluated on held-out data.

The framework gives two interventions: RSC trains vision models (Eq. 9), and task-loss allocation assigns LLM precision (Eq. 8). We have given each its own short algorithm box in revised Sec. 3.

W1 / Q1: Same quantization error, local Taylor prediction versus direct measurement

Taylor knows both the direction and size of the actual error. It predicts the result using slope and curvature at the original weights; endpoint scoring applies the error and measures the result. Both methods use the same weight perturbation, block, task loss, and calibration data. The comparison therefore isolates the information gained by evaluating the loss at the actual error size.

For block ii, let δi=Q4(θi)−θi\delta_i=Q_4(\theta_i)-\theta_i be its actual quantization error, with zero entries outside that block. Let Fi(t)F_i(t) be the calibration loss after applying a fraction tt of this error: t=0t=0 is the original model and t=1t=1 applies the full error.

F_i(t) &= \mathbb{E}_{\mathrm{cal}}\thinspace\ell(\theta + t\delta_i), \\\\ T_i &= F_i'(0) + \tfrac{1}{2}F_i''(0), \\\\ D_i &= F_i(1) - F_i(0). \end{aligned}$$ Here $T_i$ predicts the loss increase from the starting slope and curvature, while $D_i$ measures the increase after the full error is applied. The Taylor score includes the linear term and the full Hessian contribution along the actual error, including all cross-coordinate terms. For a quadratic loss along this path, Taylor is exact, even if some directions have much greater curvature than others. This directly addresses the concern about trace-based scores: the baseline already accounts for the actual error direction. A remaining gap means that the starting slope and curvature do not describe the loss over the full step. Endpoint scoring measures this missing response directly. Below, we verify the derivatives independently and check the gap at higher numerical precision. We compute the derivatives by automatic differentiation in FP32 with eager attention. Both scores use the same 32 WikiText-103 training windows, each with 1,024 prediction targets. Evaluation uses 128 disjoint validation windows. Within each model–quantizer case, the 4-bit and 8-bit candidate weights and high-precision budget are identical across allocation rules. The complete comparison covers three models, two quantizers, and three budgets. Each PPL entry is Taylor / endpoint; lower is better. The correlation column reports calibration-score Spearman correlation with independently measured single-block validation damage. $K$ of the model's $L$ blocks receive higher precision. | Model | Quantizer | $\rho$: Taylor / endpoint | $K=\lfloor L/8\rfloor$ PPL | $K=\lfloor L/4\rfloor$ PPL | $K=\lfloor L/2\rfloor$ PPL | |---|---|---:|---:|---:|---:| | Pythia-1.4B | RTN | 0.955 / 0.957 | 17.1575 / 17.5162 | 15.8982 / 15.8982 | 14.3884 / 14.3884 | | Qwen2.5-1.5B | RTN | 0.951 / 0.943 | 13.4191 / 13.4191 | 12.2637 / 12.2637 | 11.1495 / 11.1495 | | Phi-3.5-mini-instruct | RTN | 0.575 / 0.843 | 10.4578 / 9.9610 | 9.3956 / 9.2503 | 8.3572 / 7.9533 | | Pythia-1.4B | GPTQ | 0.871 / 0.896 | 14.4571 / 14.4729 | 14.1611 / 14.0880 | 13.5970 / 13.6155 | | Qwen2.5-1.5B | GPTQ | 0.808 / 0.951 | 13.2264 / 11.4714 | 12.5915 / 11.0104 | 11.9067 / 10.5133 | | Phi-3.5-mini-instruct | GPTQ | 0.394 / 0.830 | 8.0001 / 7.8548 | 7.8427 / 7.6445 | 7.3708 / 7.2477 | Direct measurement improves GPTQ PPL at every tested budget on Qwen2.5-1.5B and Phi-3.5, with average reductions of 12.51% and 2.00%. The simultaneous 95% CIs are [11.07%, 14.12%] and [1.41%, 2.95%], respectively. Across all three models, endpoint scoring has lower PPL in seven of the nine GPTQ settings; Pythia's average reduction is 0.09% [−0.03%, 0.22%]. These intervals use paired eight-window validation blocks and adjust across the six model–quantizer averages. Phi's endpoint allocation also improves PPL at every RTN budget. The extra information matters when it changes which blocks receive protection. With Qwen RTN, both scores select the same blocks at all three budgets, producing the same models and PPLs. With Qwen GPTQ, measuring the actual damage improves PPL at every budget. The following case traces the gain to one changed block choice.

One corrected block choice produces a 12.56% PPL reduction. At K=7K=7, the two Qwen GPTQ allocations share six high-precision blocks. Endpoint scoring upgrades block 1 in place of block 4, using zero-based indices. PPL falls from 12.5915 to 11.0104, with simultaneous 95% CI [10.93%, 14.35%] for the relative reduction across the 18 allocation comparisons. The budget and candidate weights are unchanged; the gain comes from where the same precision budget is spent.

The scale profile shows why the choice changes. Here Di(r)=Fi(r)−Fi(0)D_i(r)=F_i(r)-F_i(0) is the damage after applying fraction rr of block ii's actual error. The difference D1(r)−D4(r)D_1(r)-D_4(r) changes from −0.000232 nats at r=1/16r=1/16 to +0.007434 at r=1/4r=1/4: block 4 is more vulnerable at the smaller error, but block 1 becomes more vulnerable as the errors grow. Each error keeps its direction throughout this sweep. Taylor continues to favor protecting block 4 across the probed radii. The spectrum exposes the reversal that Taylor misses; the endpoint identifies the priority at the actual error; protecting that block lowers final PPL.

High-precision checks confirm this effect. For Qwen's decisive block 1 under GPTQ, the same-window FP64 audit gives a Taylor prediction of 0.002443 nats and measured damage of 0.098285 nats. The FP32/FP64 Taylor discrepancy is orders of magnitude smaller than this gap, and independent small-step derivative checks corroborate the automatic derivatives. Substituting the audited FP64 scores leaves all 18 primary allocation decisions unchanged.

This weight-space test complements the submitted Table 1 scale sweep: with each residual-error direction and injection site fixed, mean rank correlation across its seven upper-group models rises from 0.53 at r=1/16r=1/16 to 0.92 at r=1r=1. The new GPTQ test uses the submitted one-shot quantizer construction with the FP32, 32-window protocol above. The seven-model Table 2 re-evaluation in our response to ZcQB further tests allocation gains across held-out models and budgets.

W2 / Q2: How the spectrum guides the precision decision

The spectrum answers how block priorities change as the error changes; Eq. 8 measures the damage from one specified error. For a fixed quantization candidate, allocation uses that candidate's endpoint. A scale profile tests whether a ranking from smaller errors carries over to the actual error, as the Qwen reversal demonstrates. When choosing between bit widths, the operational question is how much damage each upgrade removes. We answer it by measuring the actual candidates being compared, each with its own error size and direction.

Allocation questionMeasurements usedDecision supported
Which blocks should receive higher precision for a specified low-bit error?Task damage at each block's actual errorProtect the blocks with the greatest measured damage
Where is an extra bit most valuable?Damage at both actual candidates, Di(3)D_i^{(3)} and Di(4)D_i^{(4)}Protect the blocks where the upgrade prevents the most damage
Can a ranking be reused at a different error magnitude or source?A scale/direction profileIdentify changes in block priority

Spend the extra bit where it saves the most prediction loss. For a 3-bit versus 4-bit choice, the per-block benefit is

Di(3)−Di(4).D_i^{(3)}-D_i^{(4)}.

Each Di(b)D_i^{(b)} is the loss increase when block ii is quantized to bb bits. Subtracting the 4-bit damage from the 3-bit damage tells us how much loss the extra bit avoids.

For illustration, a block whose damage falls from 0.11 to 0.10 gains 0.01, while a block whose damage falls from 0.20 to 0.02 gains 0.18. The extra bit is more valuable for the second block, even though its 4-bit damage is smaller. A single 4-bit endpoint cannot determine this upgrade benefit; measuring both candidates can.

We tested this rule on 13 LLMs with groupwise RTN, group size 128, assigning either one quarter or one half of the blocks to int3 and the remainder to int4. Each comparison uses the same candidate weights and number of int3 blocks. Below, wins and losses mean lower and higher final PPL for the two-endpoint rule; percentages are the median excess PPL of the comparator relative to that rule.

ComparatorTwo-endpoint wins / ties / losses, 26 settingsMedian comparator excess PPL
Reuse the int4 endpoint alone23 / 1 / 23.18%
Extrapolate along the int4 error direction to twice its magnitude21 / 1 / 41.33%
HAWQ-V2 scoring23 / 0 / 33.28%

Measuring the actual precision change improves allocation in 23 of 26 settings over reusing the int4 endpoint, and in 21 of 26 over extrapolating its error. A change in bit width can change both the size and direction of the error. The two-candidate measurement captures both and assigns the bit budget according to the task loss it saves. The SpQR and SqueezeLLM comparisons in our response to Mu5t apply this same criterion with budgets that also account for metadata and sparse storage.

The practical gain is to measure the errors relevant to the decision instead of reusing a ranking from another condition. The scale profile explains when priorities reverse; the candidate measurements quantify which upgrades save task loss. This gives PTQ a direct implementation: evaluate the candidate endpoints required by the bit-width choice. In vision, the corresponding choices are the scales and directions controlled by RSC, whose benefits are established by the radius and direction-transfer experiments in our response to ZcQB. We present each measurement alongside the decision and outcome it supports in revised Secs. 3–4 and Appendix M.

Q3: Better allocation with lower scoring time and memory

The PTQ contribution is operational: better precision choices, lower final PPL, and an efficient score computed with forward passes. The Taylor control identifies a specific block choice improved by measuring the actual error. All 14 Table 2 re-evaluations improve significantly over HAWQ-V2 after simultaneous correction. The new comparisons establish gains within the authors' SpQR/SqueezeLLM reconstruction routines and improve a blockwise reproduction of EvoPress (ICML 2025) by using task loss as its search criterion.

Endpoint scoring obtains the loss by running the perturbed model; Taylor scoring also computes derivatives. Using the same official-engine SpQR errors on Qwen2.5-1.5B, an A100-40GB benchmark of three blocks, four windows per block, and three timing repetitions gives:

Scoring procedureRelative scoring timeAdditional peak allocated memory
Realized task-loss endpoint1.00×1.34 GiB
Full directional second-order Taylor3.25×16.77 GiB

The benchmark uses FP32/eager execution, cached clean losses, and the same resident model and perturbations. It measures scoring time and memory, excluding candidate construction, loading, and allocation solving. Direct measurement therefore supplies the finite-scale information at substantially lower scoring cost in this controlled comparison.

KL asks whether the output still resembles the original model; task loss asks whether the correct next token has become less likely. For illustration, suppose the correct token is A and the original probabilities of A, B, and C are 60%, 20%, and 20%. Changing them to 60%, 39%, and 1% changes the wrong-token probabilities but leaves the correct token's probability—and its prediction loss—unchanged. Changing them to 50%, 25%, and 25% is closer to the original distribution by forward KL, yet makes the correct token less likely. An allocation should protect the precision that prevents this loss of correct-token probability.

For an observed correct token yxy_x, our damage score is simply

Di=Exlog⁡Px(yx)Qi,x(yx).D_i=\mathbb{E}_x\log\frac{P_x(y_x)}{Q_{i,x}(y_x)}.

Here PxP_x and Qi,xQ_{i,x} are the original and single-block-quantized probabilities. A decrease in the correct token's probability gives positive damage; an increase gives negative damage. Theorem 4 expresses the difference between this score and output KL exactly through the target-alignment term.

This distinction changes real allocation decisions. In the EvoPress-style comparison reported in our response to Mu5t, every KL-search solution is closer to the original outputs on calibration data than endpoint allocation, yet five have worse calibration NLL and worse held-out PPL. Switching the search criterion from KL to task NLL improves held-out PPL in 10 of 12 settings. The benefit therefore extends to a recent search method: evaluating allocations by their effect on the correct tokens improves prediction quality on held-out data. We have brought this explanation into revised Sec. 4.4.

Reviewer_ZcQBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies the limitations of using scalar smoothness measures (Lipschitz constants and Hessian statistics) to characterize neural network sensitivity. The authors introduce a finite-difference roughness spectrum that measures model responses across perturbation scales, directions, and representations, with a theoretical connection to Besov regularity. Two applications are presented with experiments across vision and language models show improvements in noise robustness in image classification and quantization performance over baselines.

Critical strengths
  • Well-motivated problem with a strong mathematical base. The paper points out an important limitation of conventional scalar smoothness measures and provides a more flexible spectrum framework for characterizing model sensitivity to finite, structured perturbations. The arguments are structured with explicit assumptions.
  • Promising empirical results. The paper has a balance between definitions with analysis and empirical evidence. For quantization, the proposed damage-based allocation outperforms HAWQ-V2 in all settings across 7 held-out LLMs under controlled configurations in section 4.
  • Broad applicability. The proposed framework is evaluated across different perturbation cases and tasks. It includes input robustness against noise in image classification and model weight quantization in LLMs. It is nice to see the broad applicable settings.
Critical weaknesses
  • Lack of systematic configuration evaluation. RSC uses fixed perturbation scales, direction families, and regularization weights. Although the choices are well-motivated, no overall understanding on their individual contributions or the trade-offs of different combinations. Those can be important design choices in practice.
  • Limited isolation of the proposed framework’s contributions. The paper introduces a unified roughness-spectrum perspective, but its empirical application uses different measurements. For example, the quantization method relies directly on the increase of task loss. It is unclear how much of the improvement comes from the proposed multiscale Besov formulation compared to finite-scale task sensitivity.
  • Limited statistical power for the empirical results. Many of the results reported in the main paper (table 1 and 2 for example) do not show the variance and confidence explicitly. Some improvements are small, especially without explicit statistical reliability reported.
Questions for authors

Thanks for submitting to ICLR 2027! Here are a few questions (please also refer to the previous section) that can help strengthen the claims.

  • More ablation results on the key factors of RSC are interesting to see. For example, what if RSC is compared with perturbation-matched noise augmentation or consistency regularization with the same direction families and scales? Ablations of single-scale versus multiscale training and regularization strength would help establish which components contribute to the robustness improvement.
  • The authors acknowledge previous direct loss-based sensitivity probing in the appendix, could the authors clarify which empirical improvements depend specifically on the proposed spectrum instead of evaluating task damage at the realized perturbation scale?
5 comments

Thank you for recognizing the motivation, mathematical foundation, and controlled allocation results. The additional experiments answer your questions directly: matched training comparisons identify the effect of the objective, scale, and direction; allocation experiments connect finite-scale measurements to better block choices; and all 14 Table 2 comparisons show statistically significant improvements after simultaneous correction.

W1 / Q1: Controlled ablations of the objective, scale, and direction

We added controlled experiments to distinguish the effects of the objective, perturbation size, direction, and regularization strength. We first compare objectives with the images, perturbations, and training protocol fixed, then vary the RSC settings one at a time in separate ablations. Each objective uses the original image xx and two increasingly perturbed versions, x+rux+ru and x+2rux+2ru. The classification margin is the correct class's score minus the strongest competing class's score. The three objectives use these predictions as follows:

  • Noise augmentation: predict the correct label on the perturbed images, using cross-entropy.
  • JSD consistency: keep the three predicted probability distributions close to one another.
  • RSC: keep the margin's slope from changing sharply between the two steps, by penalizing the squared difference between the slopes. A constant slope contributes zero to this penalty.

The comparison fixes ResNet-18, 200 training epochs, three seeds, initialization, data order, and the sampled perturbations. We sample rr from {0.5,1,2,4}\lbrace 0.5,1,2,4\rbrace and use isotropic/high-frequency directions. Auxiliary forwards use batch statistics while preserving the running BatchNorm statistics; the auxiliary methods share the same optimization safeguards and five-epoch loss warmup. The regularization coefficients for JSD and RSC are listed below.

This comparison isolates how the same perturbed inputs contribute through each objective. Accuracy is reported as mean ± sample standard deviation across training seeds; Gaussian noise has pixel-space σ=0.05\sigma=0.05.

ObjectiveCIFAR-10 cleanCIFAR-10 GaussianCIFAR-100 cleanCIFAR-100 Gaussian
Standard training95.43 ± 0.2168.00 ± 2.8978.23 ± 0.4134.58 ± 0.42
Matched noise augmentation95.51 ± 0.1283.31 ± 1.3278.04 ± 0.0753.76 ± 0.92
Matched JSD consistency, coefficient 1294.97 ± 0.1489.53 ± 0.3376.49 ± 0.2464.47 ± 0.27
RSC, coefficient 0.0595.46 ± 0.1279.77 ± 0.3978.44 ± 0.0846.93 ± 1.18

RSC delivers substantial robustness gains while retaining baseline clean accuracy. Gaussian accuracy increases by 11.77 points on CIFAR-10 and 12.35 points on CIFAR-100, with paired-seed 95% confidence intervals of [4.27, 19.27] and [8.88, 15.81] points. Its second difference penalizes changes in slope and leaves affine margin variation unpenalized. The augmentation and consistency controls also let us test how this objective adds to an already robust training recipe.

On top of AugMix, RSC preserves more clean accuracy at comparable Gaussian robustness than adding matched augmentation or consistency. A separate three-seed CIFAR-100 experiment adds each objective to the same AugMix + JSD training recipe. The additional objectives use the same probes at r=4r=4 with isotropic/high-frequency directions; RSC uses λ=0.2\lambda=0.2. C-all averages all 15 CIFAR-C corruptions and all five severities.

AugMix-based recipeClean accuracyGaussian σ=0.05\sigma=0.05 accuracyC-all accuracy
AugMix + JSD76.95 ± 0.2963.20 ± 1.0365.41 ± 0.07
+ matched noise augmentation75.38 ± 0.5668.93 ± 0.5764.89 ± 0.58
+ matched KL consistency75.89 ± 0.1169.44 ± 0.1166.46 ± 0.15
+ RSC76.67 ± 0.2969.48 ± 0.4466.98 ± 0.18

At similar Gaussian accuracy, RSC retains 1.30 more clean-accuracy points than matched augmentation and 0.78 more than matched KL consistency, and achieves the highest mean C-all accuracy in this comparison. RSC therefore improves robustness on top of AugMix while retaining more clean accuracy than the matched alternatives.

The added benefit persists with auxiliary BatchNorm updates disabled: AugMix + RSC improves CIFAR-100 Gaussian accuracy by 1.44 points, paired 95% CI [0.57, 2.31], with all seven seed pairs improving. We give the independent replication under W3 below, addressing your statistical-reliability concern. The comparison with explicit Gaussian-noise augmentation is reported in our response to Mu5t.

Radius controls the perturbation size, direction controls its structure, and λ\lambda controls the penalty strength. We vary these choices one at a time using the same controlled three-seed CIFAR-10 protocol as the first table. Unless changed in the first column, the setting is the submitted scale band, both direction families, and λ=0.05\lambda=0.05.

RSC configurationClean accuracyGaussian accuracy
Band {0.5,1,2,4}\lbrace 0.5,1,2,4\rbrace95.46 ± 0.1279.77 ± 0.39
Single radius r=0.5r=0.595.37 ± 0.1272.66 ± 1.26
Single radius r=1r=195.30 ± 0.1276.11 ± 1.85
Single radius r=4r=495.22 ± 0.1483.66 ± 0.49
λ=0.01\lambda=0.0195.38 ± 0.1173.81 ± 0.74
λ=0.10\lambda=0.1095.43 ± 0.0682.34 ± 0.24
Isotropic directions only95.32 ± 0.0681.51 ± 0.27
High-frequency directions only95.38 ± 0.1877.39 ± 1.06

Selecting the training scale improves Gaussian accuracy by 11.00 points under the same controlled protocol. Moving from r=0.5r=0.5 to r=4r=4 raises accuracy from 72.66% to 83.66%. Concentrating the penalty at r=4r=4 also improves on the uniform band's 79.77% at this test noise level. These results make scale selection a concrete source of improvement: RSC can direct training toward the perturbation sizes relevant to the target condition. Increasing the coefficient strengthens robustness within the tested range, and the direction family determines which perturbations receive the most protection.

The radius–coefficient grid gives a practical way to choose the robustness target. This separate three-seed sweep uses the submission's original probe-forward/BatchNorm update policy. Each entry below is clean / Gaussian σ=0.05\sigma=0.05 / Gaussian σ=0.1\sigma=0.1 accuracy, in percent.

Training radiiλ=0.05\lambda=0.05λ=0.2\lambda=0.2λ=0.8\lambda=0.8
Band {0.5,1,2,4}\lbrace 0.5,1,2,4\rbrace95.17 / 83.74 / 36.3395.07 / 87.00 / 40.6594.84 / 89.26 / 46.12
r=4r=494.92 / 87.35 / 43.2594.99 / 89.90 / 51.6694.62 / 91.54 / 55.22
r=8r=894.27 / 90.61 / 64.0794.15 / 91.66 / 71.6094.24 / 92.24 / 72.53

The same pattern holds across this grid: larger radii particularly improve robustness to stronger noise, and the coefficient adjusts the clean/robustness trade-off. For example, at λ=0.2\lambda=0.2, moving from r=4r=4 to r=8r=8 increases accuracy under σ=0.1\sigma=0.1 noise from 51.66% to 71.60%, with clean accuracy changing from 94.99% to 94.15%. This gives a concrete basis for selecting a radius and coefficient according to the expected perturbation and acceptable clean-accuracy change.

Direction matters independently of perturbation magnitude. Returning to the controlled protocol, we tested isotropic and high-frequency noise at equal pixel RMS 0.05. The three-seed results are:

RSC training directionsIsotropic test noiseHigh-frequency test noise
Isotropic only81.53 ± 0.0984.72 ± 0.62
High-frequency only77.65 ± 0.5885.84 ± 0.72

The training direction determines where the protection goes. Isotropic training gives higher isotropic-noise accuracy; high-frequency training gives higher high-frequency-noise accuracy. The training-direction × test-direction interaction is 5.01 points, with simultaneous 95% CI [3.57, 6.44] after adjustment across the three tested noise magnitudes. Because the noise magnitudes are matched, this identifies the effect of noise structure. The robustness gain also holds with GroupNorm: RSC improves Gaussian accuracy from 79.27 ± 0.94% to 90.21 ± 0.27%, with a paired gain of 10.94 points [8.98, 12.89].

W2 / Q2: How the spectrum guides measurement, training, and allocation

The same measurement framework supports a training penalty for image classifiers and a precision score for language models. RSC penalizes changes in margin slope along selected noise directions and scales during training. Allocation measures the loss caused by actual quantization candidates and assigns precision where it prevents the most damage. A scale profile shows how block priorities change with the error: a block that appears safe under a small error can become the more vulnerable block under the actual error, even along the same direction. Eq. 8 measures the task damage at each actual quantization candidate. A fixed candidate calls for one endpoint measurement, and a choice between bit widths calls for measuring the candidates being compared.

Three results connect this information to concrete outcomes.

  1. Finite-scale information changes which block is protected and improves PPL. The Taylor controls in our response to TAeY give both scores exactly the same weight error. Taylor includes its full directional curvature, so it already accounts for the error's direction. It predicts the result from slope and curvature at the original weights; endpoint scoring measures the loss after applying the error. The loss, representation, and calibration data are fixed. Endpoint allocation reduces Qwen2.5-1.5B GPTQ perplexity by 11.70–13.27% across three budgets relative to the full second-order Taylor rule. At K=7K=7, the allocations share six high-precision blocks; protecting block 1 in place of block 4, using zero-based indices, reduces PPL from 12.5915 to 11.0104, a 12.56% improvement. The scale profile explains why: the blocks exchange priority as the error grows. Measuring at the actual error captures the priority that improves the final model.
  2. Scale and direction give RSC effective training controls. The vision ablations show an 11.00-point Gaussian-accuracy improvement from selecting the radius, and the direction-transfer test shows that matching the training direction changes protection at equal noise magnitude. The diagnostic also transfers across training recipes: the radius selected using a 91-checkpoint pool was fixed before evaluation on 25 additional checkpoints. At σ=0.05\sigma=0.05, the relative margin-roughness reading correlates with accuracy at ρ=−0.86\rho=-0.86, compared with +0.29+0.29 for the measured Lipschitz-slope proxy. The training comparisons establish the intervention effects; the held-out diagnostic identifies a strong relationship between roughness and robustness across recipes.
  3. Measuring each precision candidate directs the bit budget to the upgrades that save the most task loss. For a 3-bit versus 4-bit choice, Di(3)−Di(4)D_i^{(3)}-D_i^{(4)} is the loss increase at 3 bits minus the loss increase at 4 bits. It measures how much prediction loss the extra bit saves for block ii. A large 4-bit damage score alone does not tell us this benefit; both candidates are needed. Measuring them improves over reusing the int4 endpoint in 23 of 26 groupwise-RTN settings. The complete comparison in our response to TAeY shows the practical benefit of measuring the precision change the allocator is choosing.

The theory establishes the information advantage that the experiments put to use. Theorems 2 and 6 construct responses with matching scalar or local second-order information and different finite-scale sensitivity or allocation damage. A method given those same local summaries cannot distinguish the responses; measurements at finite scales can. The experiments connect that distinction to training outcomes and block selection.

Besov theory gives this measurement problem a natural mathematical structure. Its finite differences remain meaningful across changes in slope, and its scale weights specify how variation at different distances is compared. Theorem 1 establishes equivalence between the canonical full-grid population aggregate, including its LpL^p anchor, and a Besov quasi-norm under the stated sampling assumptions. RSC specializes the finite-band second-difference construction to margin-slope control, using (m,s,p,q)=(2,1,2,2)(m,s,p,q)=(2,1,2,2); the diagnostic uses the s=1.5s=1.5 readout. The Besov aggregate measures the magnitude of response variation across scales. Eq. 8 adds the signed expected task-loss difference needed for allocation: average loss after quantization minus average loss before quantization. A positive value measures prediction damage; retaining the sign distinguishes damage from improvement. We have placed these definitions next to the two algorithms in revised Secs. 2–3.

Comments on the paper

1 comment

Thank you for the detailed reviews. We use one measurement framework for two tasks: training image classifiers to resist noise and allocating precision in quantized language models. Both applications compare a model's responses before and after perturbations. Measuring these changes across perturbation sizes and directions gives the roughness spectrum. Besov theory provides the mathematical tools for studying these finite differences across scales.

For image classifiers, RSC uses finite differences as a training penalty to limit abrupt changes in the slope of the classification margin—the correct class's score minus its strongest competitor's. For language models, task-loss allocation measures the loss caused by actual quantization candidates and gives extra bits to the blocks where they save the most prediction loss. The additional experiments test both applications with strong existing methods and use controlled comparisons to examine what produces the gains:

  • The loss at the actual error can change a precision decision that local slope and curvature miss. Taylor receives the exact error, including its direction and size, but predicts its effect from derivatives at the original weights. Endpoint scoring applies that error and measures what happens. With the weights, calibration data, candidates, and budget fixed, direct measurement improves Qwen2.5-1.5B GPTQ PPL by 11.70–13.27% across three budgets. At one budget, changing a single protected block reduces PPL by 12.56%. The derivatives are independently verified, and the comparison reports every tested model–quantizer–budget case.
  • The criterion improves allocation within SpQR and SqueezeLLM under the same storage caps, and improves recent allocation search. Using the authors' reconstruction routines, task-loss allocation improves over Fisher allocation in all 12 tested settings; all four model–quantizer averages remain significant after simultaneous correction. In a blockwise reproduction of EvoPress (ICML 2025), choosing allocations by correct-token loss (NLL) instead of similarity to the original output (KL) improves held-out PPL in 10 of 12 settings. Direct endpoint scoring is also competitive with this search, which uses 9.76–39.27× as many calibration-forward tokens.
  • RSC adds robustness even when training already uses strong augmentation and consistency regularization. We start from AugMix + JSD, then compare adding RSC, noise augmentation, or KL consistency using the same perturbed images. At comparable Gaussian accuracy on CIFAR-100, RSC retains 1.30 more clean-accuracy points than added noise augmentation and 0.78 more than added KL consistency. The benefit persists when auxiliary probes cannot change BatchNorm running statistics: a separate seven-seed experiment gives a 1.44-point Gaussian-accuracy gain over AugMix, with paired 95% CI [0.57, 2.31]. In a separate control where training already includes Gaussian-noise augmentation, RSC adds 0.70 points of mean CIFAR-C accuracy, with paired 95% CI [0.21, 1.19].
  • Controlled experiments isolate the roles of the objective, scale, direction, and regularization strength. We compare augmentation, consistency, and RSC objectives using the same model, initialization, data order, training schedule, and original and perturbed images. Separate ablations change one factor at a time. Increasing the RSC radius from 0.5 to 4 raises Gaussian accuracy by 11.00 points in the controlled CIFAR-10 comparison. Direction comparisons match the noise magnitude, so they test which perturbation patterns receive protection independently of noise size.
  • The allocation gains over HAWQ-V2 are consistent and statistically significant across all seven Table 2 models and both budgets. All 14 simultaneous 95% confidence intervals for improvement remain above zero after multiple-comparison correction.

The responses below address each concern with the corresponding results and completed manuscript revisions. Table numbers below refer to the submitted manuscript: Tables 1, 2, and 8 correspond to revised Tables 31, 3, and 10; the earlier SpQR-style Tables 25–26 correspond to revised Tables 27–28.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.