- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes B-OPT, a Bayesian optimization framework for hard-label black-box adversarial attacks. It uses a Gaussian process to optimize the reciprocal boundary-distance objective. It further introduces PB-OPT, which incorporates surrogate-derived functional priors into Bayesian optimization through a decoupled forward–backward mechanism and adaptive prior calibration. The authors provide a regret bound for a simplified framework and evaluate their methods on CIFAR-10, ImageNet, CLIP, and defense models.
Critical strengths
- PB-OPT achieves competitive query efficiency against the selected baselines across CIFAR-10, ImageNet, CLIP, and adversarially defended models. The experiments also include component-wise ablation studies that help assess the contributions of different design choices.
Critical weaknesses
-
The proposed method largely combines existing techniques rather than introducing new mechanisms. Bayesian optimization for hard-label attacks has been explored in QE-DBA, functional priors in Gaussian processes have been studied in P-BO, and surrogate boundary-distance gradients based on implicit differentiation have already been derived in Prior-OPT. The paper does not adequately distinguish its contributions from these closely related works or establish why their combination constitutes a significant methodological advance.
-
The experimental evaluation overlooks several recent and competitive methods, particularly Prior-Adam-OPT, CORTA in [1], and GGS in [2]. Comparisons with strong surrogate-assisted and modern transfer-based attacks are important. Without these comparisons under appropriate and comparable settings, the claim of outperforming state-of-the-art methods is not sufficiently substantiated. [1] Li, S., Li, Z., Ma, X., Bai, X., Hu, P., Zhang, D., & Zhu, B. (2025). Consensus-Robust Transfer Attacks via Parameter and Representation Perturbations. Advances in Neural Information Processing Systems, 38 [2] Niu, Z., Xie, W., Song, S., Yu, Z., Liu, F., & Shen, L. (2025). Enhancing Adversarial Transferability by Balancing Exploration and Exploitation with Gradient-Guided Sampling. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 3885–3894.
Questions for authors
-
Why is gradient-based acquisition optimization necessary? Can you compare the proposed forward–backward mechanism with derivative-free alternatives, such as random search or evolutionary optimization, under comparable computational budgets?
-
What is the actual source of the performance improvement? Given the relatively modest performance of B-OPT, how much of PB-OPT's improvement comes from Bayesian optimization rather than surrogate guidance?
7 comments
Follow-up questions
Thank you for the additional experiments. I have follow-up questions.
-
My main concern is that PB-OPT primarily combines and adapts established ideas from QE-DBA, P-BO, and Prior-OPT. The rebuttal clarifies how the components fit together, but the regret analysis largely follows standard GP-UCB arguments and covers only a simplified algorithm. I remain unconvinced that the combination offers a sufficiently original contribution for ICLR.
-
Why does Prior-Adam-OPT underperform both GGS and PB-OPT by such a large margin at 500 queries, then improve sharply by 1,000 queries?
-
Please revise “outperforms 16 state-of-the-art hard-label attacks.” This statement pools surrogate-free methods such as HSJA, TA, and SurFree with surrogate-assisted methods. Please distinguish these settings and specify where PB-OPT improves over baselines with comparable surrogate access.
Follow-up response to Reviewer Yg1f
Q1+: Methodological Novelty.
A1+: We respectfully disagree that PB-OPT is merely a combination of existing techniques. This view overlooks a fundamental challenge: under hard-label settings, the target provides only discrete labels, making its true boundary-distance function non-differentiable, while BO must directly optimize normalized ray directions in a high-dimensional space. Existing techniques cannot be directly transferred to this setting.
(1) QE-DBA's BO is not directly applicable. QE-DBA performs BO in a low-dimensional perturbation parameter space (e.g., Perlin or wavelet parameters), whereas our method directly optimizes normalized ray directions in 3,072 dimensions for CIFAR-10 and 9,408 dimensions for ImageNet. Derivative-free candidate sampling can be effective in low-dimensional spaces but becomes inefficient in such high-dimensional spherical optimization. This requires gradient-based acquisition optimization, including tangent-space projection, rather than merely changing the search space. Our matched-budget experiments with random search and CEM further verify this limitation.
(2) P-BO's functional prior cannot be directly applied to hard-label settings. P-BO relies on differentiable logits, whereas our boundary distance () is obtained through discrete searches and has no directly accessible gradient. PB-OPT therefore introduces a decoupled forward–backward mechanism: the forward pass estimates surrogate boundary distances through parallel queries, while the backward pass uses implicit differentiation of surrogate C&W margins to provide substitute gradients. Without the backward mechanism, the scalar prior alone would lead to derivative-free acquisition optimization. Our prior-gradient ablation further confirms its importance.
(3) Prior-OPT's transfer prior and our global functional prior operate at different levels. Prior-OPT uses transfer information for local gradient estimation and direction sampling. In contrast, PB-OPT incorporates surrogate boundary information into the GP mean as a global functional prior, jointly calibrating it through learnable (\gamma) and (c) and GP hyperparameters. GP-UCB then balances global prior guidance and predictive uncertainty. Our prior-mean ablation and adaptive calibration results support the effectiveness of this design.
Overall, our contribution is a hard-label-compatible mechanism for high-dimensional global Bayesian ray search, addressing limitations that prevent the direct application of the three existing approaches.
Q2+: Query-Budget Behavior of Prior-Adam-OPT.
A2+: The sharp ASR increase of Prior-Adam-OPT is mainly explained by its substantial initialization overhead. Before actual ray-direction optimization, it requires approximately 200 queries to identify the best direction among 100 candidates, followed by approximately 200 queries for the initial gradient estimation.
Consequently, under a 500-query budget, approximately 400 queries are consumed by initialization, leaving only about 100 queries for Adam-based optimization. Under a 1,000-query budget, approximately 600 queries remain for iterative updates. This explains the sharp ASR increase from 4.43% to 60.22% on AT ResNet-50 and from 0% to 14.94% on MIMIR.
Thus, the low ASR at 500 queries primarily reflects the initialization overhead and insufficient optimization steps, rather than the performance of the subsequent Adam updates.
We also clarify that GGS is a transfer-based attack with a fixed perturbation budget, rather than a query-based attack. Its ASR is independent of the target-query budget, so its result at 500 queries should not be interpreted as a direct query-efficiency comparison.
Q3+: Experimental Comparisons.
A3+: We agree with the reviewer and will reorganize the experimental results into two separate tables to distinguish the resource settings:
Surrogate-free attacks: HSJA, TA, SurFree, and other methods that do not use surrogate models.
Surrogate-assisted attacks: SQBA, Prior-OPT, Prior-Sign-OPT, PARS-OPT, Prior-Adam-OPT, and PB-OPT.
This separation will make the comparison settings explicit and highlight PB-OPT's improvements over baselines with comparable surrogate access, particularly Prior-OPT, Prior-Sign-OPT, PARS-OPT, and Prior-Adam-OPT. We will report the corresponding surrogate architectures, target models, and query budgets to facilitate fair comparisons.
We will also revise the statement "outperforms 16 state-of-the-art hard-label attacks" to avoid implying that all methods share identical resource assumptions. Transfer-based methods such as GGS and CORTA will be clearly distinguished from query-based hard-label attacks and cannot be directly compared.
These revisions will provide a more transparent evaluation of PB-OPT under different resource settings.
Clarifications on Methodological Novelty, Experimental Comparisons, and Optimization Efficiency
Q1: The proposed method mainly combines existing techniques, without sufficiently clarifying its novelty and contributions.
A1: We adapt existing techniques to high-dimensional hard-label ray search, with key differences from three related works.
(1) QE-DBA: It performs BO in a low-dimensional perturbation parameter space (e.g., Perlin or wavelet parameters), whereas B-OPT/PB-OPT directly optimize normalized ray directions in the perturbation space (3,072 dimensions for CIFAR-10 and 9,408 for ImageNet).
(2) P-BO: It studies functional priors under soft-label settings with differentiable logits. In contrast, PB-OPT addresses hard-label settings, where boundary distances require non-differentiable discrete searches. We introduce a decoupled forward–backward mechanism: parallel surrogate queries estimate boundary distances, while implicit differentiation of surrogate C&W margins provides gradients. Learnable γ and c jointly calibrate the prior with GP hyperparameters.
(3) Prior-OPT: It uses transfer-based priors for local gradient estimation and direction sampling. In contrast, PB-OPT constructs a global functional prior from surrogate boundary distances and incorporates it into the GP mean, enabling GP-UCB to balance prior guidance and uncertainty.
Overall, we integrate calibrated surrogate boundary information into GP-based global optimization for high-dimensional hard-label attacks.
Q2: The evaluation omits several recent competitive methods, including Prior-Adam-OPT, CORTA, and GGS. Comparisons with these methods are needed to substantiate the state-of-the-art claims.
A2: We additionally compared PB-OPT with Prior-Adam-OPT and GGS on ImageNet AT ResNet-50 (ℓ∞ = 8/255) and MIMIR (ViT, ℓ∞ = 4/255), using an independently trained AT ResNet-50 (ℓ∞ = 4/255) as the surrogate model. All ASRs use the same L2 threshold 15.026.
Table 1: ASR (%) on AT ResNet-50.
Table 2: ASR (%) on MIMIR.
GGS and CORTA are transfer-based attacks with fixed L2 budgets, unlike our query-based hard-label attack. Their ASRs are independent of query budgets; thus, direct comparisons are not strictly fair.
Q3: Why is gradient-based acquisition optimization necessary, and how does it compare with derivative-free alternatives?
A3: Gradient-based optimization is crucial in our high-dimensional search space (9,408 dimensions for ImageNet). Under the unit-sphere constraint, the steepest ascent direction is the normalized tangent-space projection of the acquisition gradient. In contrast, for a uniformly sampled unit direction:
$ \mathbb E[(\nabla\alpha(\theta)^\top v)^2] =\frac{|\nabla\alpha(\theta)|_2^2}{d}. $
Thus, random-direction alignment decreases as 1/d, making derivative-free optimization increasingly sample-inefficient in high dimensions.
CEM iteratively selects elite candidates to update its sampling distribution but relies on sampling rather than direct gradient information.
We compare all three optimizers on 10 ImageNet images as follows.
Table 3: Acquisition optimization comparison (mean L2 distortion).
Q4: What is the actual source of PB-OPT's improvement: Bayesian optimization or surrogate guidance?
A4: PB-OPT's improvement comes from the synergy between BO and surrogate guidance.
First, B-OPT independently validates BO: on ImageNet Swin (Table 3), it achieves 28.78 L2 distortion at 2,000 queries, outperforming HSJA (45.82) and Prior-OPT (36.74).
Second, Table 4 demonstrates the prior's contribution. Under the same reciprocal objective, a random prior barely changes performance (28.78 vs. 28.69), whereas our Standard prior reduces distortion to 15.75 and increases ASR from 31.06% to 57.79%.
Furthermore, removing the GP acquisition gradient or prior gradient increases distortion to 27.50 or 28.92, respectively, compared with 15.75 for full PB-OPT.
These results confirm the complementary contributions of BO and informative surrogate priors: BO exploits target-model feedback, while the functional prior provides transferable global guidance. Their integration is essential to PB-OPT's performance.