- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies what finite representation observations can establish about local stability, identifying the limitations of static activations and incomplete perturbation coverage. It derives sensitivity bounds and prediction-invariance conditions on covered subspaces under independently established smoothness bounds, and proposes Secant-Frame Regularization (SFR) to penalize the largest reconstructed logit response without explicitly computing input Jacobians. Experiments demonstrate conditional certification on controlled functions and trained smooth models, while CIFAR-10 experiments show improved robustness to selected transformations compared with matched-query pairwise consistency.
Critical strengths
- The analysis clearly identifies what finite observations can and cannot certify, providing a rigorous framework connecting perturbation coverage, smoothness, and prediction margins to local stability.
- SFR improves evaluated transformation robustness over matched-query pairwise consistency across all five seeds, achieving a 2.46-point average gain with a 0.30-point clean-accuracy decrease and demonstrating practical value.
Critical weaknesses
- The certificates require independently established curvature bounds, and their application to trained networks is demonstrated only on shallow smooth models in PCA coordinates, limiting the demonstrated practical scope of the theoretical guarantees.
- Using the same regularization weight for SFR and pairwise consistency does not account for their different penalty scales, leaving it unclear whether the reported advantage persists after independently tuning both methods.
Questions for authors
- How costly and conservative are the required curvature bounds, and how does this requirement limit the applicability of the certificates beyond the shallow smooth models evaluated?
- Were the regularization weights tuned independently for SFR and pairwise consistency, and does SFR retain its robustness advantage when both methods achieve comparable clean accuracy?
4 comments
Rebuttal - Q1Q2
3. Question 1
How the reported bound is obtained. For the trained models in experiment 7, Appendix D.8, Lemma D.6 proves a global sufficient bound for the unthresholded Softplus classifier. The bound follows analytically from the activation’s derivative bounds and the trained weights. ''Independently supplied'' means justified separately from the finite secant observations, and it can depend on the learned parameters. Appendix E.8 evaluates this bound in float64 with an upward safety factor. Sampled Jacobian differences serve as diagnostics, rather than as the basis for its neighborhood-wide validity.
Computational cost. Obtaining this particular bound requires two weight-matrix spectral norms and a maximum row norm. Because it is global, it can be computed once for a fixed checkpoint and reused at all test anchors. Secant queries and margins are still evaluated per example. The manuscript does not report separate wall-clock timings for bound acquisition, so we do not claim a measured speed advantage for this step. Its cost is separate from SFR training, which does not use .
Observed conservatism. Appendix E.8 reports a maximum sampled Jacobian-difference quotient divided by the analytic bound of 0.3240. This measures slack relative to the observed variation, and it does not identify the unknown exact neighborhood-wide curvature constant. The certificate calculation nevertheless yields positive radii (Table 15). These radii are in standardized PCA coefficient-space units. SFR and exact-Jacobian regularization attain the same reported median radius in this experiment. The dependence on conservatism is in Theorem 4.3: a larger valid increases both the secant-reconstruction error bound and the quadratic displacement correction, which can reduce the certified radius.
Beyond the evaluated architecture. Independent bounds also exist for deeper smooth networks. For instance, [1] derived compositional curvature bounds for smooth sequential and residual networks. Their global bounds offer a route to supplying the required assumption when the map, domain, and norms match. Layer-based estimates can become conservative as depth and layer gains increase, so obtaining a useful radius for a particular deep model remains dependent on the quality of that independent bound. This requirement also explains the separation of our experiments: a ReLU network or transformation pipeline crossing nonsmooth boundaries does not automatically satisfy the smoothness assumption. Accordingly, the CIFAR-10 results support empirical robustness to covered transformations. The formal certificate results are reported for the settings in which the assumptions are established.
4. Question 2
The primary CIFAR-10 protocol in Table 18 specifies and . The results of this experiment concern these fixed coefficients and do not establish superiority over independently optimized clean-robust trade-off curves. Checkpoint selection is performed on validation data: each regularized method is selected by covered robustness subject to the seed-matched ERM clean accuracy floor, and the final test metrics are reported afterward. This rule controls model selection and incorporates clean accuracy (Table 18).
At the reported operating points, pairwise consistency and SFR achieve 95.5% and 95.2% clean accuracy, respectively, with 82.6% and 85.1% evaluated joint robust correctness (Table 19). The paired robustness gain is 2.46 percentage points, and the paired clean accuracy change is −0.304 points (Table 21). The accuracy and prediction-invariance differences quoted above are in Table 22. All differences are SFR minus pairwise consistency. The covered robustness endpoint uses the evaluation protocol described in Table 18. These show that SFR produces a consistent robustness improvement with a modest clean accuracy trade-off under the documented protocol. The reported experiments do not resolve the outcome after independently tuning both methods to equal clean accuracy. We respectfully argue that this qualification preserves the demonstrated empirical contribution while keeping its scope precise, alongside the paper’s information-theoretic and conditional-certification results.
We appreciate the opportunity to make clarifications and hope our responses address your concerns. Thank you again!
Reference:
[1] Entesari, T., Sharifi, S., & Fazlyab, M. (2024). Compositional curvature bounds for deep neural networks. ICML’24: Proceedings of the 41st International Conference on Machine Learning.
Thank you for the detailed responses and for clarifying the scope and limitations of the results.
Thank you for the detailed responses and for clarifying the scope and limitations of the results. These clarifications have addressed my concerns, and I will raise my rating accordingly.
Dear Reviewer 26Nf,
Thank you again for taking the time to review our paper and for raising the score! If you have any follow-up questions or comments, we would be happy to clarify further.
Rebuttal - W1W2
Thank you for recognizing the rigor of the finite-observation analysis and the consistent improvement in robustness across all five seeds. We also appreciate the concerns about practical certification and the strength of regularization. To clarify the relationship between our contributions: finite perturbation queries reveal a restricted response operator; an independently justified smoothness bound controls the uncertainty between these queries; and a prediction margin turns the resulting displacement bound into a conditional certificate. The SFR approach directly regularizes the empirical response operator and can be trained without a curvature bound.
1. Weakness 1
The trained-model experiment has a specific purpose: to instantiate the complete certification argument on learned classifiers while making its assumptions verifiable. Theorem 4.3 and Corollary 4.5 apply whenever the stated coverage, smoothness, and readout conditions hold. Their statements do not impose a one-hidden-layer architecture or PCA representation. The independently supplied bound is the information that allows finite secants to support a neighborhood-wide guarantee. Making this requirement explicit is central to the paper’s account of what finite observations can establish.
Experiment 7 demonstrates that this conditional result can be instantiated on trained models and real data. Appendix E.8 evaluates certificates on all official MNIST test examples, with PCA and standardization fitted on the optimization-training split. The experiments include coordinate dimensions 8, 16, and 32. These coordinates define the declared perturbation domain and provide complete directional coverage, and the guarantees hold within that domain. The analytic weight-based curvature bound and the resulting positive certified radii provide a concrete test of the full argument.
The CIFAR-10/ResNet-18 experiment addresses the complementary question of whether the finite-query objective helps train a standard image classifier. It evaluates SFR empirically without requiring a curvature bound. Thus, the practical scope demonstrated for formal certification and the scope demonstrated for empirical regularization are different and identified. The scope of the trained certificate experiment does not restrict where SFR can be evaluated as a regularizer. We give the bound’s cost and measured conservatism.
2. Weakness 2
The existing comparison establishes a consistent benefit under a controlled, specified training protocol. Pairwise consistency and SFR share the architecture, data splits, optimizer schedule, perturbation setup, forward-query budget, and validation-based checkpoint-selection rule. Under that protocol, SFR improves covered joint robust correctness in every seed, with a mean gain of 2.46 percentage points and a mean clean accuracy decrease of 0.304 points (Tables 18-22). This supports the practical value of the proposed objective at the reported operating points.
You are correct that a shared numerical coefficient does not equalize effective regularization strength. Proposition 4.6 makes the distinction precise. In its orthonormal-frame setting, let A=Jf(x;V) denote the reconstructed response operator and let r be the number of orthonormal directions. With ζ=0, the per-example penalties are Rpair=r1∥A∥F2 and RSF=∥A∥op2. Pairwise consistency penalizes the average squared singular response, and SFR penalizes the largest squared response. Their ratio depends on the singular-value distribution. When multiple singular modes are possible, no single constant rescaling makes the two objectives equivalent across all response operators. This establishes a substantive difference in how they penalize directional sensitivity.
At the same time, that mathematical distinction does not establish that the measured improvement is caused exclusively by spectral emphasis, or that SFR dominates an independently tuned pairwise baseline. The finite-query information limits, conditional guarantees, and distinction between average and worst-direction control remain valid independently of this hyperparameter-comparison limitation.