- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the robustness verification of quantized neural networks (QNNs) when bit flips occur simultaneously across multiple parameters. The authors propose mBFV, which constructs per-parameter contribution bounds within a CROWN-style bound propagation framework. By employing a top-m aggregation strategy to account for perturbations in up to parameters, the method avoids the need to explicitly enumerate all possible combinations of perturbed parameters. To enhance the precision of the relaxation, the authors further introduce a Branch-and-Bound (BaB) approach tailored to the locations of the perturbations. Experimental evaluations—covering seven fully connected architectures across the ACAS Xu, MNIST, and CIFAR-10 datasets—demonstrate the effectiveness of the proposed verification method through comparisons with baselines and an ablation study of the BaB component.
Critical strengths
-
The research problem is meaningful, and the methodology is driven by a clear computational motivation. Verifying multi-parameter bit flips is more challenging than verifying single-parameter perturbations. This paper incorporates a cardinality constraint on parameter perturbations into bound propagation and combines it with a perturbation-location-based BaB approach, offering a viable alternative to the direct enumeration of parameter combinations.
-
The BaB approach yields clear empirical gains. In the reported experiments, the number of verified instances increased from 184 to 293 when BaB was applied. This ablation study demonstrates that partitioning based on perturbation locations helps mitigate the excessive looseness of a uniform relaxation, serving as a key piece of evidence for the effectiveness of the proposed method.
Critical weaknesses
Section 3 lacks a clear narrative, which severely compromises the readability of the core methodology. The authors present a dense sequence of formulas, often accompanied by only one or two sentences of explanation. It is difficult for the reader to discern: what specific problem is being addressed at this stage? Why is this particular formula introduced? What is its function, and how does it relate to the subsequent step? Such explanations should be provided by the authors; instead, the reader is left to piece them together from the formulas themselves.
My primary concern is not the omission of derivations for certain formulas—it is acceptable to skip routine computational steps, provided that the underlying logic of the method is clearly articulated. This is especially critical regarding the paper’s core contribution: how the margin lower bound is derived when multiple parameters are perturbed simultaneously, and why this bound is reliable. Rather than developing a coherent explanation around these points, Section 3 relies on a barrage of formulas to carry the entire explanatory burden. This makes the core contribution difficult to grasp and hinders the reader’s ability to assess its soundness.
In my view, Section 3 requires a structural reorganization, rather than merely the addition of a few missing derivation steps. The authors should first outline the overall approach, then clarify the purpose of each step and how these steps interconnect. Formulas should serve to support the narrative, not replace it. Incorporating a small, running example—such as a simple network—throughout the section would also be beneficial; however, the essential task remains to clearly explain the methodology. This current style—a mechanical enumeration of formulas—is my principal reservation regarding the paper.
Questions for authors
-
Narrative and Soundness in Section 3. The current presentation in Section 3 makes it difficult to assess the soundness of the proposed method. While the authors provide numerous formulas, the purpose of each step—and how these steps collectively yield a reliable lower bound on the margin—remains unclear. Please reorganize this section to clearly articulate the core logic and the basis for its soundness. I will not adjust my current rating until this issue is addressed.
-
Practical Application of Certificates. How are the certificates produced in this paper intended to be used? Are they meant for verifying specific input regions prior to deployment, or for verifying new inputs individually at runtime? If the goal is to ensure that all inputs classified as "A" are robust against parameter faults, is it necessary to verify each input individually, or can the verified regions be designed to cover these inputs? How would such coverage be implemented in a practical system, and what is the associated computational overhead?
-
Model Scale and Experimental Contribution. Why do the experiments focus primarily on fully connected networks of the current scale, rather than evaluating larger or more complex model architectures? Is this a limitation inherent to the method itself, or is it due to excessive computational cost? If the choice of models is based solely on the fact that "prior work used them," I find this justification insufficient.
6 comments
Response to Reviewer FvAJ - Experiment
Q3. Model scale and experimental contribution
The fully connected models in the submission were chosen to provide direct comparisons with existing state-of-the-art tools (BFAVerifier and MILP) on standard verification benchmarks. The scale is not an inherent limitation of our method: while existing baselines time out on networks with more than 13k parameters even for single bit flips (), mBFV scales to 1.15M parameters at and 575k parameters at .
To demonstrate applicability to more complex architectures, the revised paper evaluates a convolutional model: CIFAR-10 CNN-A (215k parameters, two convolutional and two fully connected layers) from SDP-CROWN [a]. Verified instances out of 15 selected inputs (60 s timeout):
| radius 0 | radius 1/255 | |
|---|---|---|
, mBFV w/o BaB | 15 | 14 |
, mBFV | 15 | 15 |
, mBFV w/o BaB | 0 | 0 |
, mBFV | 15 | 10 |
, mBFV | 0 | 0 |
At radius 1/255, each method is evaluated on instances it verified at radius 0. While BFAVerifier and MILP verify zero instances (timing out on all 15 inputs), mBFV verifies all 15 inputs at (radius 0) and 10 inputs at radius 1/255. Crucially, branch and bound is essential: without BaB, zero instances verify at (Sec. 5.1, Tab. 1).
References
[a] Chiu et al. SDP-CROWN: Efficient Bound Propagation for Neural Network Verification with Tightness of Semidefinite Programming. ICML 2025.
Thanks for your answer and clarification. I have one more question.
Have you considered comparing the bounds obtained under the original model and the bit-reversed model in same setting? Have you considered comparing the bounds obtained under the original model and the bit-reversed model?
I'm sure you'll come to some interesting conclusions.
Response to Reviewer FvAJ - Revised Narrative in Sec. 3
We thank the reviewer for the thorough critique and constructive suggestions. We have substantially revised Sec. 3 to address your concerns regarding narrative and readability, and we address each question below.
Q1 & W1. Narrative and soundness in Sec. 3
We restructured Sec. 3 to prioritize conceptual clarity over formula density. Instead of having derivations in isolation, the section is now organized around the two core challenges of multi-parameter bit flip verification:
The first challenge is propagating bounds through non-linear layers when parameters are perturbed. Sec. 3.1 shows that backward linear bound propagation separates unperturbed propagation from additive parameter perturbations. This isolates the parameter effect into sensitivity coefficients gp≤0, which are evaluated in constant time from bounding hyperplanes and perturbation intervals.
The second challenge is selecting the worst-case combination of m perturbed parameters out of K without enumeration. Sec. 3.2 resolves this by relaxing binary parameter indicators to the continuous domain Cm=s∈[0,1]K:∑psp≤m. We prove that this continuous relaxation incurs zero relaxation gap, as its minimum is always attained at a binary vector (Prop. 3.1). This yields an exact closed-form solution via topm(g) in O(K) time.
Concretization and soundness then follow directly in Sec. 3.3. Because input bounds and parameter perturbations decouple additively in Eq. (10), minimizing them independently in Eq. (12) yields a provably sound lower bound.
To ground the methodology, Sec. 3.2 now walks through a concrete 4-parameter example (g=(−5.5,−3.2,−1.7,0) at m=2) demonstrating why continuous relaxation matches the integer optimum. Furthermore, Apdx. A provides a complete numeric trace across a two-neuron network, showing intermediate bounds and sensitivity values at every step.
Q2. Practical application of certificates
Safety-critical applications have safety specifications defining the required behavior over specified input ranges. Even millions of successful tests cannot establish the absence of violations. Our verifier takes these specifications and checks them entirely prior to deployment. A certificate guarantees the required behavior for every input in the specified region under the allowed parameter faults.
Yes, our sound bound propagation method covers every possible input within the region given by the specification, e.g., neighbor of a specific input in class “A”. The bounds can be conservative (loose), but they contain all possible behaviors within the region, and mBFV verifies all input in the defined input range all at once.
As noted above, the input ranges in the safety specification are verified entirely offline. Once a range is certified, every input satisfying its conditions preserves the required output property under the allowed faults. All verification cost is incurred offline, with zero additional computation during inference.