acceptodds
Under review as a conference paper at ICLR 2027

On the Formal Verification of 1-Lipschitz Neural Networks

Abstract

Certifying neural networks against -norm adversarial perturbations is essential for safety-critical deployment. While 1-Lipschitz architectures offer scalable, computationally inexpensive certified radii, a persistent gap remains between their empirical robustness and guaranteed bounds. To bridge this gap, we investigate formal verification (e.g., LiRPA) on 1-Lipschitz models. Focusing on Gradient Norm Preserving (GNP) networks—a state-of-the-art approach for certified robustness—we establish a necessary condition for LiRPA to outperform \certrad, proving that this condition fails when parsing standard GNP architectures off-the-shelf. To resolve this limitation, we introduce a LiRPA-compatible graph factorization of GNP activations (e.g. GroupSort2). Leveraging this formulation, we conduct the first evaluation comparing LiRPA against standard Lipschitz certified radii across diverse hyperparameter configurations. Notably, our analysis reveals the first family of networks where analytical Lipschitz bounds outperform consistently LiRPA. We identify two key hyperparameter dimensions governing this behavior: optimization trade-offs and Lipschitz constraint strictness. Specifically, relaxing optimization to enable higher clean accuracy, combined with looser Lipschitz constraints, creates a regime where LiRPA yields provably superior certificates. Based on this insight, we introduce a hybrid certification framework that unifies the scalable geometry of 1-Lipschitz networks with the localized precision of linear bound propagation. This hybrid strategy scales LiRPA certification to deeper architectures. Empirically, on Imagenette—where standalone solvers fail—our framework scales to a VGG16 architecture, achieving 84.5% clean accuracy and advancing certified robust accuracy from 54.0% (standard Lipschitz bounds) to 72.5%.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Lipschitz Neural Networks, LiRPA, Formal Verification, Machine Learning, CNN

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_geKMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper theoretically analyzes the limitations of analytical Lipschitz certificates and proposes two complementary approaches to improve verification: an exact reformulation of GroupSort2 that preserves dependencies in the computational graph, and a hybrid certification framework that decomposes the network into a 1-Lipschitz backbone and a shallow head verified using linear bound propagation. These approaches improve certification tightness and scalability in selected architectural and training regimes. The paper’s central finding is that more sophisticated verifiers do not necessarily yield tighter robustness certificates: for tightly constrained 1-Lipschitz networks, inexpensive analytical bounds can match or outperform LiRPA, whereas LiRPA becomes more beneficial when architectural constraints are relaxed.

Critical strengths

Strong points:

  1. The geometric analysis of Lipschitz-certified radii motivates the proposed framework and provides a clear, useful comparison between analytical Lipschitz certification and formal verification.
  2. The computational graph reformulation substantially improves certified robust accuracy in the evaluated settings.
  3. Ablation studies, repeated training runs, and evaluations on an additional test subset strengthen the empirical evidence.
  4. The experiments identify regimes in which computationally expensive verification offers little or no improvement over inexpensive analytical certificates.
Critical weaknesses

Weaknesses:

  1. The interpretation of Proposition 3.1 is overstated. The proposition establishes conditional gradient attenuation, rather than proving that every expanding architecture must violate the Eikonal condition. A more precise conclusion is that orthogonal expansion does not guarantee global gradient-norm preservation.

  2. The temperature explanation requires clarification. The appendix defines the loss as CE(τf(x),y)\mathrm{CE}(\tau f(x),y) (p. 19), meaning that τ\tau acts as an inverse temperature. Increasing τ\tau reduces the unscaled logit margin required to achieve a given confidence. Therefore, the claim that larger τ\tau requires larger logit margins (p. 5) does not follow from the stated formulation. Although the empirical trend may hold, the proposed causal explanation needs further justification.

  3. The soundness of some analytical certificates is unclear. The paper uses power iteration to estimate certain Lipschitz constants (p. 7 and Appendix E.4). Finite power iteration alone does not guarantee an upper bound on the spectral norm. The authors should clarify how these estimates are converted into sound bounds, since underestimating the Lipschitz constant can invalidate the corresponding CRA values. This concern does not, by itself, invalidate the LiRPA certificates.

  4. Claims of broad superiority require qualification. Table 10 reports an Imagenette VGG13 baseline with 85.0% clean accuracy and 84.0% CRA, both higher than the highlighted hybrid VGG16 results of 84.5% clean accuracy and 72.5% VRA. Although this comparison involves different architectures, it shows that hybridization does not uniformly dominate the reported constrained baselines. The paper should distinguish improvements in certification for a fixed model from improvements in the overall accuracy–robustness trade-off.

  5. The empirical evaluation could be strengthened. Five-run experiments support the smaller-model results, but comparable variability estimates are not reported for the main hybrid results. With only 200 test examples, a 0.5-percentage-point difference corresponds to a single example and should be interpreted cautiously. More complete runtime and peak-memory measurements, together with ablations on the backbone–head split, would better substantiate the scalability claims.

  6. Practical applicability remains insufficiently demonstrated. The experiments address image classification under bounded ℓ2\ell_2 perturbations; in particular, the reported radius of 8/2558/255 should not be interpreted as the commonly used ℓ∞\ell_\infty threat model. This evaluation supports a specific robustness guarantee, but provides limited evidence for the broader safety-critical applications invoked in the motivation.

Questions for authors

No further questions

5 comments

We thank Reviewer geKM for their careful evaluation of our paper and for the thoughtful comments that have helped us clarify our contributions.

(W1) The Proposition 3.1 conclusion can indeed be clarified. We will apply this modification in the final version, clearly stating that “orthogonal expansion does not guarantee global gradient-norm preservation”. As Reviewer 7SYd mentioned, we will also acknowledge the prior result in [6].

(W2) We thank the reviewer for pointing out this incorrect statement. The sentence should have read "higher temperatures allow smaller logit margins" and we will revise accordingly. More precisely, in case the surrounding paragraph was not sufficiently clear, at low temperatures, achieving high confidence requires larger unscaled logit margins, encouraging the network to push decision boundaries away from the training data. Conversely, higher temperatures allow smaller unscaled logit margins, reducing this margin-maximizing pressure and potentially leading to more complex decision boundaries that weave and fold around local data clusters (Figure 1b), particularly when the gradient norm is capped at one (Figure 5). We will also clarify that this interpretation is intended as an intuitive explanation of the empirical observations rather than a formal theoretical consequence.

(W3) We agree that, formally, Power Iteration (PI) does not provide a guaranteed upper bound on the spectral norm. However, since the final Lipschitz constant estimation is performed only once, we can afford a large number of iterations to obtain an accurate approximation, as is common practice in the literature [2]. We have also cross-checked the estimates using SVD or Gram iteration (which provide an upper bound) [1] to assess their numerical accuracy. We will add a clarification in the revised manuscript to explain this validation procedure and support the reliability of the reported results.

(W4) We agree that it is not stated clearly in the paper that we consider the Pareto front associated with a given architecture. This aspect will be clarified in the final version.

(W5) (Note: As this concern was raised by all reviewers, we provide a common response below.) We initially restricted our evaluation to the 200 test points and perturbation radii established by SDP-CROWN. This was done to ensure a fair comparison with the baseline. However, we fully agree with all the reviewers that 200 samples can be limited, and that drawing conclusions from a 1-to-3 point difference (representing just a few images) requires nuance. To address this, we ran evaluations on 1000 test points for the CIFAR-10 architectures (CNN-A, ConvLarge, VGG13 and hybrid VGG13). These extended benchmarks confirm the trends with higher statistical confidence and will be reported in the revised appendix. In light of this, we will qualify the claims in the main text. Where the margins are thin, we will modify statements like 'Lipschitz-certified radii consistently surpass formal verification' to state that they 'consistently match or outperform' formal verification on fully constrained models. In such cases, it is preferable to use the CRA as it is immediate to compute. We provide the results for Convlarge in the following tab, the complete results will be added to the final version.

Table A

Model (Setting)ArchitectureClean Acc (%)ERA (%)CRA (%)SDP-CROWN VRA (%)
ConvLarge (Accurate)RKO (Björck)82.979.118.626.7
ConvLarge (Accurate)AOC73.769.756.158.0
ConvLarge (Robust)RKO (Björck)68.166.864.865.3
ConvLarge (Robust)AOC67.065.264.264.1

To substantiate our scalability claims, we profiled the peak GPU memory and runtime on our deepest model (VGG16 on Imagenette). Standard bound propagation physically cannot scale to this depth, instantly triggering an Out-Of-Memory (OOM) error by attempting to allocate over 2,048 GB of VRAM on an 80GB GPU. This confirms the fundamental memory bottleneck described in our theoretical analysis. In contrast, our hybrid framework successfully bypasses this barrier, completing the verification with a peak memory of only 0.62 GB and an average time of 121.59 seconds per sample. We will include these results in the appendix of the final version.

We agree that ablating the backbone-head split provides important context for the scalability of our hybrid framework. We evaluated varying depths for the unconstrained classification head and present the results for VGG13 architecture (CIFAR-10). If we restrict the head to only 1 linear layer, the lack of non-linearities means SDP-CROWN provides no verification advantage over standard last-layer normalization (e.g., as seen in LiResNet). Conversely, if we detach the Lipschitz constraint earlier (3 or 4 free layers), the robustness property seems to vanish. As shown in the table below, even with high weight decay regularization (wd0.5), the bounds become too loose for the verifier to recover robustness, causing both analytical CRA and formal VRA to collapse to near zero. Therefore, the 2-layer unconstrained head configuration in our main experiments (Section 6.4) represents a good structural balance: it contains sufficient non-linearity to leverage SDP-CROWN's localized precision, while maintaining the global Lipschitz constraint deep enough to preserve verifiable robustness. We observed the same phenomenon on Imagenette models, and we will include the ablation tables for both datasets in the final version.

Table D

Head Depth (Unconstrained Layers)Weight Decay (wd)Clean Accuracy (%)Analytical CRA (%)Formal VRA (%)
3 Layers1.084.50.011.5
3 Layers0.585.00.04.5
4 Layers1.085.50.00.0
4 Layers0.586.50.00.0

To further demonstrate the stability and reliability of our hybrid certification approach, we present the statistical results for 3 VGG13 models on Imagenette below. The variance results for the remaining hybrid evaluations, specifically VGG13 on CIFAR-10 and VGG16 on Imagenette, will be added in the appendix of the final version.

Table E

Model (Setting)Clean Acc (%)CRA (%)Hybrid VRA (%)
VGG13 (Hybrid, wd=0.5)87.5 ± 1.069.8 ± 4.973.2 ± 4.0
VGG13 (Hybrid, wd=0.1)88.5 ± 0.537.5 ± 1.363.0 ± 3.0

(W6) We agree that an ℓ2\ell_2 radius of 8/255 represents a different radius than the conventional 36/255 in the Lipschitz community. We selected this specific radius to ensure a direct comparison with the experimental setup used in the SDP-CROWN baseline (Chiu et al., 2025). We also evaluated the models at a larger ℓ2\ell_2 radius of 36/255, please refer to Table C in the answer to Reviewer 7SYd. However, the shallow architectures standard to formal verification benchmarks lack the capacity to maintain a meaningful accuracy trade-off at this larger perturbation level. We will revise the introduction to clarify that our 8/255 evaluation establishes a standardized verification baseline. We will reframe our current methods as a necessary algorithmic foundation for scaling toward future safety-critical deployments.

[1] Delattre et al., Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration

[2] Leino et al., Globally-Robust Neural Networks

[3] Béthune et al., Pay Attention to your Loss

[4] Tsuzuku et al., Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

[5] Boissin et al., An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures

[6] Li et al., Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks

Reviewer_nRL4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the interaction between formal verification and 1-Lipschitz neural networks for l_2-robustness certification. The authors analyze why native Lipschitz-certified radii can be loose, derive a necessary condition under which LiRPA can outperform Lipschitz certification, and show that standard GroupSort2 parsing may violate this condition. They then propose an exact ReLU-based factorization of GroupSort2 that is LiRPA-compatible and reduces memory cost. Finally, they introduce a hybrid certification framework that propagates perturbations through a Lipschitz backbone and applies LiRPA only to a shallow head, enabling certification of deeper models such as VGG16.

Critical strengths
  1. Clear technical motivation and contribution. The paper identifies a concrete incompatibility between standard LiRPA relaxations and GroupSort2-based GNP networks, and derives a necessary condition for LiRPA to match or outperform the native Lipschitz certificate.

  2. The proposed GroupSort2 factorization is simple, exact, and effective. Rewriting GroupSort2 through a shared ReLU preserves the operation while maintaining the relevant dependency structure and reducing nonlinear nodes. Empirically, this substantially improves verified robust accuracy and avoids OOM failures on deeper models.

  3. The paper provides useful insight into when analytical versus formal certification is preferable. In strongly constrained, near-isometric regimes, Lipschitz certificates can match or outperform LiRPA, whereas LiRPA is more useful in higher-temperature or more contractive regimes.

  4. The hybrid verification framework improves scalability. Applying Lipschitz propagation through the backbone and LiRPA only on the head allows certification of VGG-scale models; on Imagenette, the reported VGG16 result improves certified robustness from 58.0% to 72.5% while maintaining 84.5% clean accuracy.

Critical weaknesses
  1. Some theoretical statements should be scoped more carefully. The key condition in Theorem 4.1 is necessary but not sufficient. The proposed factorization guarantees this necessary norm condition, but this alone does not guarantee that LiRPA will produce a tighter certificate. The paper should more clearly separate the theoretical guarantee from the empirical observation of improved verification.

  2. The transition from binary/scalar theory to multiclass certification is somewhat implicit. Much of the formal development is written for scalar classifiers, while the experiments are multiclass. A cleaner theorem or derivation for pairwise logit margins in the multiclass setting would strengthen the theoretical presentation.

  3. The main empirical verification tables use relatively small test subsets. The primary VRA evaluation uses 200 test points. The additional independent subset and five-run statistics in the appendix are helpful, but broader-scale evaluation would make the empirical conclusions more convincing.

  4. The hybrid setting is conceptually different from the fully 1-Lipschitz setting. In the VGG experiments, the backbone remains Lipschitz-constrained while the final classification head is unconstrained and verified separately. This is a reasonable design, but the paper should distinguish more clearly between claims about formally verifying fully 1-Lipschitz networks and claims about the hybrid partially constrained architecture.

Questions for authors
  1. Could you clarify more explicitly which guarantees follow directly from satisfying (|\omega|_2 \le L), given that the condition is necessary but not sufficient for LiRPA to outperform the Lipschitz certificate?

  2. Can the main theoretical results be stated directly for multiclass pairwise logit margins, rather than relying on the scalar/binary presentation?

  3. How sensitive are the conclusions to the relatively small 200-sample verification subsets? Could you provide larger-scale verification results for at least the main CIFAR-10 settings?

  4. In the hybrid VGG setting, how should the reader interpret the method relative to the paper’s original focus on fully 1-Lipschitz networks? Could you more clearly separate these two regimes in the claims and conclusions?

4 comments

We thank the Reviewer nRL4 for their positive assessment and for recognizing the value of our GroupSort2 reformulation. We particularly appreciate the highlighting of our hybrid certification strategy. We strongly agree that dividing the network into a 1-Lipschitz backbone and a symbolic LiRPA head is essential for bypassing memory limits. This is the core mechanism that enables us to successfully scale verification to deeper architectures.

(W1 - Q1) The assessment about Theorem 4.1 and the GS2 graph factorization is developed in Appendix B.1, where we prove the condition is necessary but not sufficient. However, we will make this distinction clearer in the main text. We will revise Section 4 to explicitly state that our condition does not automatically guarantee tighter bounds. We will reframe the motivation for the factorization. It fulfills a necessary mathematical prerequisite. Beyond that, it halves the memory cost and enables SDP-CROWN compatibility. We will emphasize that the tighter bounds are an empirical result.

(W2 - Q2) We chose to present the theoretical results in binary setup because it is simpler to understand and generalizable to the multiclass setting. Even if prior works [3,4] treat this reduction as a standard property, we recognize the need for an explanation in the final version: we will add a proposition and derivation to the appendix.

We can provide a sketch of the explanation: multiclass certification requires the true class logit to remain greater than all other logits. We decompose this multiclass problem into independent pairwise margin functions defined as gj(x)=fc(x)−fj(x)g_j(x) = f_c(x) - f_j(x), cc being the true class. Each margin gj(x)g_j(x) operates as a standard scalar function. We apply binary certification independently to each margin to prove gj(x)≥0g_j(x) \geq 0. The final multiclass certified radius is simply the minimum of these independent binary radii. Consequently, any theoretical proof established for a scalar binary classifier applies directly and identically to the multiclass architecture without loss of generality.

(W3 - Q3) (Note: As this concern was raised by all reviewers, we provide a common response below.) We initially restricted our evaluation to the 200 test points and perturbation radii established by SDP-CROWN. This was done to ensure a fair comparison with the baseline. However, we fully agree with all the reviewers that 200 samples can be limited, and that drawing conclusions from a 1-to-3 point difference (representing just a few images) requires nuance. To address this, we ran evaluations on 1000 test points for the CIFAR-10 architectures (CNN-A, ConvLarge, VGG13 and hybrid VGG13). These extended benchmarks confirm the trends with higher statistical confidence and will be reported in the revised appendix. In light of this, we will qualify the claims in the main text. Where the margins are thin, we will modify statements like 'Lipschitz-certified radii consistently surpass formal verification' to state that they 'consistently match or outperform' formal verification on fully constrained models. In such cases, it is preferable to use the CRA as it is immediate to compute. We provide the results for Convlarge in the following tab, the complete results will be added to the final version.

Table A

Model (Setting)ArchitectureClean Acc (%)ERA (%)CRA (%)SDP-CROWN VRA (%)
ConvLarge (Accurate)RKO (Björck)82.979.118.626.7
ConvLarge (Accurate)AOC73.769.756.158.0
ConvLarge (Robust)RKO (Björck)68.166.864.865.3
ConvLarge (Robust)AOC67.065.264.264.1

(W4 - Q4) We agree that the hybrid architecture operates under fundamentally different constraints than the fully 1-Lipschitz networks. To clarify this distinction, the revised manuscript will separate our claims into two distinct regimes: the 1-Lipschitz setting—where the superiority of formal verification over Lipschitz certificates is dictated by architectural constraints and training temperature—and the hybrid setting, where an unconstrained classification head helps achieve better clean accuracy and makes formal verification essential to recover unproven robustness. As shown in Appendix E.4, applying formal verification to the final layers of a fully constrained network provides no certifiable advantage over native Lipschitz bounds. Substantial gains only emerge when formal methods are applied to an unconstrained classification head. This hybrid design is directly built upon the conclusions of our benchmark. It leverages the demonstrated phenomenon that Lipschitz bounds excel for highly constrained feature extraction, while formal verification is capable of recovering unproven robustness in relaxed layers.

[1] Delattre et al, Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration

[2] Leino et al., Globally-Robust Neural Networks

[3] Béthune et al., Pay Attention to your Loss

[4] Tsuzuku et al., Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

[5] Boissin et al., An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures

[6] Li et al., Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks

Thank you again for your review. As the discussion period is coming to an end, we believe we have addressed all your concerns and would appreciate your feedback.

Reviewer nRL4, thanks for your review. Could you take a look at the authors’ response and comment on which of your concerns have or have not been resolved?

Reviewer_7SYdOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies formal verification (LiRPA: CROWN, α-CROWN, SDP-CROWN) applied to 1-Lipschitz Gradient-Norm-Preserving (GNP) networks with GroupSort2 (GS2) activations. It makes four claims:

  • Native Lipschitz certificates are loose for two reasons: channel expansion attenuates the backward gradient norm (Prop. 3.1), and high training temperature τ forces the decision boundary to fold (Sec. 3, Fig. 1, Fig. 6).
  • A necessary condition for LiRPA to beat the Lipschitz radius is ‖ω‖₂ ≤ L for the equivalent affine lower bound (Thm. 4.1). The standard min/max relaxation of GS2 can violate this condition (Thm. 4.2). A factorization r = ReLU(y − x), z_max = x + r, z_min = y − r gives a local bounding operator with spectral norm exactly 1 (Thm. 4.4), halves the non-linear node count, and makes SDP-CROWN applicable.
  • A hybrid certification: propagate the ℓ₂ ball through a 1-Lipschitz backbone, then run LiRPA only on a shallow head.
  • Experiments on MNIST, CIFAR-10 (ε = 8/255) and Imagenette (ε = 8/255), 200 test points. The factorization raises CROWN VRA from 0–3.5% to 35–65.5% (Table 1). In the “Robust” regime, CRA matches or exceeds SDP-CROWN VRA; in the “Accurate” regime with RKO, SDP-CROWN exceeds CRA (Table 2). The hybrid VGG16 on Imagenette reaches 84.5% clean / 72.5% VRA (Table 10).
Critical strengths
  • The GS2 factorization is a simple, correct, and effective engineering contribution. The sum-preservation identity min + max = x + y is exact. Theorem 4.4 is a short, verifiable 2×2 eigenvalue computation. The empirical effect is large and consistent (Table 1, Table 5): CROWN VRA goes from 0.0% to 50–65% and α-CROWN no longer runs out of memory on ConvLarge. This part of the paper is useful to practitioners of auto_LiRPA with GroupSort networks, and it is the main reason I lean toward acceptance.
  • The empirical regime map (Sec. 6.3, App. A.5) is informative. The paper identifies when post-hoc LiRPA adds value (RKO, high τ) and when it does not (AOC, low τ). The 5-run variance study (Table 7) and the independent test subset (Table 8) support the direction of the trend. This is a practical result for the Lipschitz community, which has not systematically run SDP-CROWN on its models.
Critical weaknesses
  • The headline hybrid result is a comparison between two certifiers on one model, not an improvement over the best baseline. The abstract states “advancing certified robust accuracy from 58.0% to 72.5%”. The 58.0% is a CRA computed by power iteration on an unconstrained ReLU head (Sec. E.4). This global product-of-spectral-norms bound is known to be loose, so LiRPA is expected to beat it. The fair comparison is against the fully constrained baselines at equal clean accuracy in Table 10. On Imagenette, VGG16 (Accurate) AOC gives 84.0% clean / 71.0% CRA with zero verification cost; the hybrid gives 84.5% / 72.5%. The gain is 1.5 points on 200 samples (3 images). On CIFAR-10, VGG13 AOC (Accurate) gives 83.5% / 60.0%; the hybrid wd = 0.5 gives 85.0% / 55.5%, which is lower robustness. The claim in Sec. 6.4 that the hybrid “escapes the Pareto frontier” is not supported for CIFAR-10. The abstract and conclusion must be rewritten to reflect this.
  • Inconsistency between Table 2 and Table 7 that undermines the main benchmark. Table 7 reports mean ± std over 5 runs. Several Table 2 entries lie far outside those ranges. Examples: ConvLarge (Accurate) AOC CRA is 61.5% in Table 2 but 37.2 ± 1.2% in Table 7; ConvLarge (Robust) AOC CRA is 69.0% vs 62.4 ± 1.7%; ConvLarge (Accurate) RKO CRA is 21.5% vs 13.4 ± 2.2%. If the Table 2 models are selected checkpoints, the selection procedure must be stated. If the settings differ, that must be stated. As written, I cannot tell which numbers support the central claim “Lipschitz-certified radii consistently surpass formal verification”. Also, the key margins (e.g., 69.0% CRA vs 68.5% VRA) equal one image out of 200.
  • Theoretical contributions are thin or overstated.
    1. Thm. 4.1 is a two-line consequence of the definition of distance to a hyperplane. It should be a remark, not a theorem.
    2. Thm. 4.2 proves that one local 2×2 block can have norm > 1. The theorem statement claims the global vector ω violates the condition. A single block with norm > 1 does not imply the product has norm > 1. The statement and proof do not match.
    3. Prop. 3.1 is a standard fact (orthogonal decomposition ℝ^d_out = im(W) ⊕ ker(Wᵀ)), and gradient attenuation under expansion is already the subject of Li et al. (2019). The paper does not state this.
    4. The counterexample in A.4 claims “no single valid weight matrix can satisfy the Eikonal condition for all inputs”. The argument fixes W = (1, 0)ᵀ. Take instead W = (1/√2, 1/√2)ᵀ and W_head = (1/√2, 1/√2). Then WᵀW = 1, ‖W_head‖₂ = 1, GS2 is the identity on (x/√2, x/√2), and the output is ŷ = x with gradient norm 1 for all x. The stated general claim is false. The proof only shows that one specific expansion fails.
  • Benchmark radius is not comparable to the Lipschitz literature. The paper uses ε = 8/255 (ℓ₂) from Chiu et al. (2025). The 1-Lipschitz literature (SOC, AOC, LiResNet, BRONet, LipNeXt) reports ε ∈ {36/255, 72/255, 108/255}. At 8/255, the certified numbers cannot be placed relative to any published Lipschitz baseline, and the paper gives no such baseline. The conclusion that the framework “paves the way” to certify LiResNet/LipNeXt is not tested.

Minor (do not drive the recommendation): “Table 7” in Sec. 6.3 should be “Table 2”; “Theorem 4.3” in Sec. 6.1 should be “Definition 4.3”; “Section 6.4” in Sec. 6.3 should be “Section 5”; “Section C.5” for reproducibility details points to Hardware; the Reproducibility Statement says proofs for Sec. 4 are in Appendix A (they are in Appendix B). The backbone orthogonalization method (AOC or RKO) of the hybrid models in Table 10 is not stated.

Questions for authors
  • Explain the discrepancy between Table 2 and Table 7 for ConvLarge AOC and RKO CRA values. Are the Table 2 models the best of 5 runs, a different training configuration, or a different test subset? What is the maximum CRA across the 5 runs in Table 7?
  • For the hybrid VGG16 / VGG13 models: which orthogonalization (AOC or RKO) is used in the backbone? What is the Lipschitz constant of the backbone used in Phase 1?
  • Report the hybrid framework at a standard Lipschitz radius (e.g., ε = 36/255 on CIFAR-10) and compare to at least one published 1-Lipschitz baseline at the same radius.
  • Does Theorem 4.2 hold at the global level, or only for one local block? If only local, revise the statement.
  • In A.4, address the counterexample W = (1/√2, 1/√2)ᵀ with head (1/√2, 1/√2). Does the “structurally unavoidable” claim survive?
  • In Thm. 4.4, the backward pass selects the upper or lower ReLU relaxation by the sign of the combined coefficient on r. Confirm that the single-γ 2×2 analysis covers both cases in auto_LiRPA, including the lower-bound slope α for the “unstable” ReLU.
5 comments

We thank Reviewer 7SYd for their deep and thorough evaluation. Their detailed feedback will directly improve the final manuscript. We are grateful the reviewer recognizes the mathematical correctness, effectiveness, and practical utility of our GS2 graph factorization. We also appreciate the positive feedback on our benchmarks. We agree that it could represent a practical guide for determining exactly when formal verification adds value.

(W1) We agree the abstract was comparing two certification methods on the hybrid model. While this demonstrates the benefit of our certification method on hybrid models, we acknowledge that the comparison with the fully constrained baseline should also be reported. We will update the abstract to reflect this correct comparison. Our claim of "escaping the Pareto frontier" refers to expanding the reachable trade-off space. We will explicitly define this in the revised text. On CIFAR-10,structural constraints cap the baseline's clean accuracy at 83.5%. The hybrid model breaks this ceiling to reach clean accuracies up to 87.5%. This creates a new high-accuracy operating point that baseline models cannot reach. On Imagenette, the hybrid model improves the certified robustness over the baseline (72.5% vs 71%) at comparable clean accuracy. We will outline both of these distinct trade-off dynamics in the final manuscript.

The reviewer notes that LiRPA is expected to beat the loose global bound on the unconstrained head. We completely agree. This is the exact design motivation of the hybrid framework. Our benchmark demonstrated that formal verification can recover certified robustness in relaxed optimization regimes. We designed the hybrid architecture and hybrid verification to exploit this phenomenon achieving either better clean or certified accuracy. We will rewrite the abstract and conclusion to clarify the baseline comparison and the architecture-specific Pareto definition.

(Note: As this concern was raised by all reviewers, we provide a common response below.) We initially restricted our evaluation to the 200 test points and perturbation radii established by SDP-CROWN. This was done to ensure a fair comparison with the baseline. However, we fully agree with all the reviewers that 200 samples can be limited, and that drawing conclusions from a 1-to-3 point difference (representing just a few images) requires nuance. To address this, we ran evaluations on 1000 test points for the CIFAR-10 architectures (CNN-A, ConvLarge, VGG13 and hybrid VGG13). These extended benchmarks confirm the trends with higher statistical confidence and will be reported in the revised appendix. In light of this, we will qualify the claims in the main text. Where the margins are thin, we will modify statements like 'Lipschitz-certified radii consistently surpass formal verification' to state that they 'consistently match or outperform' formal verification on fully constrained models. In such cases, it is preferable to use the CRA as it is immediate to compute. We provide the results for Convlarge in the following tab, the complete results will be added to the final version.

Table A

Model (Setting)ArchitectureClean Acc (%)ERA (%)CRA (%)SDP-CROWN VRA (%)
ConvLarge (Accurate)RKO (Björck)82.979.118.626.7
ConvLarge (Accurate)AOC73.769.756.158.0
ConvLarge (Robust)RKO (Björck)68.166.864.865.3
ConvLarge (Robust)AOC67.065.264.264.1

(W3.3) For Proposition 3.1, we acknowledge this relies on standard orthogonal decomposition. We will explicitly cite [6] and clarify that gradient attenuation under expansion is a known phenomenon that we are applying to our specific context.

(W3.1) The proof of Theorem 4.1 is indeed a direct geometric consequence. However, we propose to set it as a Proposition instead of a Remark due to its impact on the paper. Its value lies not in mathematical complexity, but in establishing the necessary condition that dictates whether any linear relaxation can outperform Lipschitz bounds. This central proposition leads to our ReLU graph factorization.

(W3.2 - Q4) We agree that a local block with a norm ≥1\ge 1 does not guarantee ∥ω∥2≥1\lVert \omega \rVert_2 \ge 1. In LiRPA, bounds propagate backward through a matrix product: ω⊤=∏l=M1DlWlDσlAl\omega^\top = \prod_{l=M}^1 D_l W_l D_{\sigma_l} A_l. The diagonal selection matrices DD contain ±1\pm 1 entries (∥D∥2=1\lVert D \rVert_2 = 1). GNP architectures enforce weight norms of 11 (∥Wl∥2=1\lVert W_l \rVert_2 = 1). With the property of submultiplicativity (∥AB∥2≤∥A∥2∥B∥2\lVert AB \rVert_2 \le \lVert A \rVert_2 \lVert B \rVert_2), the upper bound of the global norm simplifies to ∏∥Al∥2\prod \lVert A_l \rVert_2. Because ∏∥Al∥2>1\prod \lVert A_l \rVert_2 > 1, we cannot guarantee the necessary condition ∥ω∥2≤1\lVert \omega \rVert_2 \le 1. We will update the proof and statement of Theorem 4.2 to make this explicit, adding a counterexample demonstrating that standard min/max cannot guarantee the necessary condition ∥ω∥2≤1\lVert \omega \rVert_2 \le 1.

(Q6) The single-γ\gamma 2x22x2 analysis successfully covers both cases. During backward propagation, auto_LiRPA dynamically selects the relaxation slope based on the sign of the accumulated coefficient. Depending on this sign and the bounding objective, the framework routes to either the upper bound secant slope (γ=uu−l\gamma=\frac{u}{u-l}) or the parameterized lower bound slope (γ=α\gamma = \alpha). Crucially, verification frameworks restrict both the upper and lower ReLU relaxation slopes such that they remain in [0,1][0,1] across all regimes (active, inactive, and unstable). This guarantees the selected slope γ\gamma always satisfies 0≤γ≤10 \leq \gamma \leq 1. We will add this clarification in the final version.

(Q5) Our example in Appendix A.4 was indeed misleading. We only intended to show a particular case where the gradient norm is forced to decrease, but the reviewer correctly points out that this does not generalize. Because the main theorem is self-sufficient, we will remove this counterexample from the final version.

(W2 - Q1) We thank the reviewer for catching this discrepancy. The mismatch stems from two reporting errors: the Table 7 statistics inadvertently included AOC models trained with a different learning rate, and the Table 2 RKO baseline omitted Power Iteration. The models in Table 2 were the first ones trained, not selectively chosen checkpoints. We have corrected these values below and in the manuscript.

Table B

Model (Setting)ArchitectureClean Acc (%)ERA (%)CRA (%)α-CROWN VRA (%)SDP-CROWN VRA (%)
ConvLarge (Accurate) Max CRA 27.5RKO (Björck)82.0 ± 3.278.8 ± 3.522.2 ± 3.30.029.0 ± 3.8
ConvLarge (Accurate) Max CRA 61.5AOC79.7 ± 3.375.1 ± 2.455.1 ± 5.30.059.0 ± 4.3
ConvLarge (Robust) Max CRA 70RKO (Björck)71.7 ± 1.770.4 ± 1.668.2 ± 1.761.1 ± 1.869.0 ± 1.5
ConvLarge (Robust) Max CRA 70AOC71.1 ± 1.270.0 ± 1.468.8 ± 1.241.4 ± 10.868.7 ± 1.2

(W4 - Q3) The choice of 8/2558/255 was to provide comparison with the SDP-CROWN paper, but as requested, we have evaluated our hybrid models at the standard ϵ=36/255\epsilon = 36/255 radius. The results are provided below and will be added to the appendix.

Table C

ModelSetting / ArchitectureClean (%)ERA (%)CRA (%)Hybrid VRA (%)
Fully Constrained Baselines
VGG13 (Accurate)GS2 RKO (Björck)84.574.00.00.0
VGG13 (Accurate)GS2 AOC83.564.51.51.5
VGG13 (Robust)GS2 RKO (Björck)82.073.527.520.0
VGG13 (Robust)GS2 AOC82.073.062.558.0
Hybrid Framework Models(Weight Decay Tuning)
VGG13wd=0.585.069.50.000.00
VGG13wd=0.187.562.50.000.00

Regarding the comparison to published 1-Lipschitz baselines (LiResNet, SOC, AOC, etc.), we want to clarify the primary objective of our work. Our goal is not to beat the SOTA certified accuracy at ϵ=36/255\epsilon=36/255, which relies on significantly larger architectures, and additional training data beyond the scope of this paper. Rather, our core contribution is methodological: we are the first to bridge two previously isolated communities—Lipschitz architectures and linear bound propagation (LiRPA)—by demonstrating when and why one outperforms the other, and combining them to scale certification via hybrid verification.

Because our objective is to isolate the impact of formal verification versus Lipschitz certificates, all our experiments require comparable architecture and training regimes. To ensure our baseline is well-grounded, we built our framework upon Adaptive Orthogonal Convolutions, which has already been shown in the literature to be SOTA [5].

Finally, we have revised our conclusion to clarify that extending our framework to architectures like LiResNet is an avenue for future work. LiResNet’s specific regularization (EMMA loss) dictates a fundamentally different approach to hybrid models; adapting our method to such architectures would require integrating LiRPA bounds directly into the training objective.

(Q2)

In the hybrid architectures (VGG13 and VGG16), the feature extraction backbone is constrained to be 1-Lipschitz with AOC constraints. We apply AOC orthogonalization to linear/convolutional backbone layers, as it empirically provides the highest baseline robustness. We will add this clarification to the final version.

We finally thank the reviewer for noticing the minor presentation errors. We will modify them in the final version.

[1] Delattre et al., Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iteration

[2] Leino et al., Globally-Robust Neural Networks

[3] Béthune et al., Pay Attention to your Loss

[4] Tsuzuku et al., Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks

[5] Boissin et al, An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures

[6] Li et al, Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.