- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BAPS, a decision-based black-box attack for classifiers with randomized input-noise defenses. Repeated queries to the same input may return inconsistent hard labels. BAPS separates probability-boundary geometry estimation from reliable endpoint selection. It uses sequential paired comparisons around boundary anchors, a radius-constrained candidate archive, and independent 200-draw verification of the final output. Experiments on CIFAR-10 and ImageNet under Gaussian, uniform, and Laplace input noise report improved strict ASR over adapted SurFree/CGBA baselines and mixed results relative to an adapted PSJ baseline.
Critical strengths
- The proposed strict ASR criterion is an interesting evaluation perspective. Requiring the final adversarial example to achieve a high attack probability with an explicit confidence lower bound is informative, and could be useful for evaluating attacks against randomized defenses.
Critical weaknesses
-
The core technical novelty and query-efficiency advantage are not convincingly established. Many ingredients have clear precedents, such as low-frequency priors, local normal estimation, tangent moves, and probabilistic boundary search. The main distinctive choice is to spend repeated queries refining a small number of paired directional comparisons and to maintain a separate reliability archive. The paper does not provide a matched-budget comparison showing that this is better than using the same queries to explore more noisy directions as in HSJA/PSJ-style normal estimators.
-
The experimental evidence is too tied to the paper’s own strict-ASR protocol and is not broad enough to establish general effectiveness. Almost all attack results are reported only under the paper-specific choice of , 95% one-sided confidence, and 200 fresh verification draws, without standard ASR, final , distortion/border-distance, or sensitivity to these operating points. It is difficult to tell whether BAPS is generally a better attack or mainly better under its own evaluation criterion. In addition, Gaussian, uniform, and Laplace noise are all variants of independent additive pixel noise, so the paper does not sufficiently test qualitatively different randomized defenses such as random resize/padding, crop/resize, blur/compression, or other stochastic transformations.
Questions for authors
-
How often does the claimed geometry–reliability mismatch actually occur in practice? Is this decoupling genuinely necessary?
-
How sensitive is BAPS to its heuristic design choices? 15 DCT directions, DCT scales, h=0.5, 3–7 samples per side, momentum, proposal multipliers, probability/radius gates, and the 7+31 archive screening scheme, they are introduced with limited justification.
-
The paper explicitly changes PSJ’s process, and many ImageNet runs hit the imposed iteration cap. How much of the observed gap is due to these adaptations rather than the underlying attack method, and can results under a more native PSJ configuration be provided?
-
Under the stated confidence-sequence formula, , and a cap of only 3–7 samples per side, the advertised early-stopping condition appears unable to trigger at all?