acceptodds
Under review as a conference paper at ICLR 2027

Differentiable Boundary Priors for Bayesian Ray Search in Hard-Label Attacks

Abstract

Hard-label attacks, one of the most challenging settings in black-box adversarial attacks, rely solely on top-1 label feedback and thus suffer from extremely high query complexity. Existing approaches, such as OPT, formulate hard-label attacks as a zeroth-order ray search problem. Subsequent methods build on this formulation and seek better query efficiency through refined gradient estimation or surrogate priors, yet their improvements remain limited under the extremely sparse feedback of the hard-label setting. In this paper, we propose B-OPT, a Bayesian optimization framework that improves hard-label ray search by optimizing a reparameterized objective under extremely limited feedback. We further propose PB-OPT, which goes beyond local gradient priors by incorporating a global functional prior into the optimization process. To enable this in the hard-label setting, we design a decoupled forward–backward mechanism: the forward pass estimates the surrogate boundary distance via parallel queries, while the backward pass approximates its gradient using a substitute loss. Based on this design, PB-OPT incorporates the surrogate objective as a functional prior in the Gaussian-process mean, allowing Bayesian optimization to refine its posterior estimate using both the surrogate prior and observed target evaluations. We derive a regret bound showing that informative functional priors tighten optimization bounds. Extensive experiments on image classifiers and vision-language models demonstrate that our method significantly improves query efficiency and outperforms 16 state-of-the-art hard-label attacks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
hard-label attacks, Bayesian optimization, adversarial attacks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Yg1fOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes B-OPT, a Bayesian optimization framework for hard-label black-box adversarial attacks. It uses a Gaussian process to optimize the reciprocal boundary-distance objective. It further introduces PB-OPT, which incorporates surrogate-derived functional priors into Bayesian optimization through a decoupled forward–backward mechanism and adaptive prior calibration. The authors provide a regret bound for a simplified framework and evaluate their methods on CIFAR-10, ImageNet, CLIP, and defense models.

Critical strengths
  1. PB-OPT achieves competitive query efficiency against the selected baselines across CIFAR-10, ImageNet, CLIP, and adversarially defended models. The experiments also include component-wise ablation studies that help assess the contributions of different design choices.
Critical weaknesses
  1. The proposed method largely combines existing techniques rather than introducing new mechanisms. Bayesian optimization for hard-label attacks has been explored in QE-DBA, functional priors in Gaussian processes have been studied in P-BO, and surrogate boundary-distance gradients based on implicit differentiation have already been derived in Prior-OPT. The paper does not adequately distinguish its contributions from these closely related works or establish why their combination constitutes a significant methodological advance.

  2. The experimental evaluation overlooks several recent and competitive methods, particularly Prior-Adam-OPT, CORTA in [1], and GGS in [2]. Comparisons with strong surrogate-assisted and modern transfer-based attacks are important. Without these comparisons under appropriate and comparable settings, the claim of outperforming state-of-the-art methods is not sufficiently substantiated. [1] Li, S., Li, Z., Ma, X., Bai, X., Hu, P., Zhang, D., & Zhu, B. (2025). Consensus-Robust Transfer Attacks via Parameter and Representation Perturbations. Advances in Neural Information Processing Systems, 38 [2] Niu, Z., Xie, W., Song, S., Yu, Z., Liu, F., & Shen, L. (2025). Enhancing Adversarial Transferability by Balancing Exploration and Exploitation with Gradient-Guided Sampling. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 3885–3894.

Questions for authors
  1. Why is gradient-based acquisition optimization necessary? Can you compare the proposed forward–backward mechanism with derivative-free alternatives, such as random search or evolutionary optimization, under comparable computational budgets?

  2. What is the actual source of the performance improvement? Given the relatively modest performance of B-OPT, how much of PB-OPT's improvement comes from Bayesian optimization rather than surrogate guidance?

7 comments
AuthorsOpenReview ↗

Clarifications on Methodological Novelty, Experimental Comparisons, and Optimization Efficiency

Q1: The proposed method mainly combines existing techniques, without sufficiently clarifying its novelty and contributions.

A1: We adapt existing techniques to high-dimensional hard-label ray search, with key differences from three related works.

(1) QE-DBA: It performs BO in a low-dimensional perturbation parameter space (e.g., Perlin or wavelet parameters), whereas B-OPT/PB-OPT directly optimize normalized ray directions in the perturbation space (3,072 dimensions for CIFAR-10 and 9,408 for ImageNet).

(2) P-BO: It studies functional priors under soft-label settings with differentiable logits. In contrast, PB-OPT addresses hard-label settings, where boundary distances require non-differentiable discrete searches. We introduce a decoupled forward–backward mechanism: parallel surrogate queries estimate boundary distances, while implicit differentiation of surrogate C&W margins provides gradients. Learnable γ\gamma and cc jointly calibrate the prior with GP hyperparameters.

(3) Prior-OPT: It uses transfer-based priors for local gradient estimation and direction sampling. In contrast, PB-OPT constructs a global functional prior from surrogate boundary distances and incorporates it into the GP mean, enabling GP-UCB to balance prior guidance and uncertainty.

Overall, we integrate calibrated surrogate boundary information into GP-based global optimization for high-dimensional hard-label attacks.


Q2: The evaluation omits several recent competitive methods, including Prior-Adam-OPT, CORTA, and GGS. Comparisons with these methods are needed to substantiate the state-of-the-art claims.

A2: We additionally compared PB-OPT with Prior-Adam-OPT and GGS on ImageNet AT ResNet-50 (ℓ∞ = 8/255) and MIMIR (ViT, ℓ∞ = 4/255), using an independently trained AT ResNet-50 (ℓ∞ = 4/255) as the surrogate model. All ASRs use the same L2 threshold 15.02615.026.

Table 1: ASR (%) on AT ResNet-50.

Method500 queries1000 queries1500 queries2000 queries
Prior-Adam-OPT4.4360.2267.8669.83
GGS (Transfer)34.6134.6134.6134.61
PB-OPT (Ours)87.3288.6789.0489.29

Table 2: ASR (%) on MIMIR.

Method500 queries1000 queries1500 queries2000 queries
Prior-Adam-OPT0.0014.9422.3624.66
GGS (Transfer)13.2713.2713.2713.27
PB-OPT (Ours)56.1160.6164.3766.98

GGS and CORTA are transfer-based attacks with fixed L2 budgets, unlike our query-based hard-label attack. Their ASRs are independent of query budgets; thus, direct comparisons are not strictly fair.


Q3: Why is gradient-based acquisition optimization necessary, and how does it compare with derivative-free alternatives?

A3: Gradient-based optimization is crucial in our high-dimensional search space (9,408 dimensions for ImageNet). Under the unit-sphere constraint, the steepest ascent direction is the normalized tangent-space projection of the acquisition gradient. In contrast, for a uniformly sampled unit direction:

$ \mathbb E[(\nabla\alpha(\theta)^\top v)^2] =\frac{|\nabla\alpha(\theta)|_2^2}{d}. $

Thus, random-direction alignment decreases as 1/d1/\sqrt{d}, making derivative-free optimization increasingly sample-inefficient in high dimensions.

CEM iteratively selects elite candidates to update its sampling distribution but relies on sampling rather than direct gradient information.

We compare all three optimizers on 10 ImageNet images as follows.

Table 3: Acquisition optimization comparison (mean L2 distortion).

Optimizer400800120016002000
Gradient (Ours)13.6356.0405.1944.4484.028
Random Search42.17741.99741.57140.35840.188
CEM39.40337.07136.24136.21035.551

Q4: What is the actual source of PB-OPT's improvement: Bayesian optimization or surrogate guidance?

A4: PB-OPT's improvement comes from the synergy between BO and surrogate guidance.

First, B-OPT independently validates BO: on ImageNet Swin (Table 3), it achieves 28.78 L2 distortion at 2,000 queries, outperforming HSJA (45.82) and Prior-OPT (36.74).

Second, Table 4 demonstrates the prior's contribution. Under the same reciprocal objective, a random prior barely changes performance (28.78 vs. 28.69), whereas our Standard prior reduces distortion to 15.75 and increases ASR from 31.06% to 57.79%.

Furthermore, removing the GP acquisition gradient or prior gradient increases distortion to 27.50 or 28.92, respectively, compared with 15.75 for full PB-OPT.

These results confirm the complementary contributions of BO and informative surrogate priors: BO exploits target-model feedback, while the functional prior provides transferable global guidance. Their integration is essential to PB-OPT's performance.

Reviewer_Yg1fOpenReview ↗

Follow-up questions

Thank you for the additional experiments. I have follow-up questions.

  1. My main concern is that PB-OPT primarily combines and adapts established ideas from QE-DBA, P-BO, and Prior-OPT. The rebuttal clarifies how the components fit together, but the regret analysis largely follows standard GP-UCB arguments and covers only a simplified algorithm. I remain unconvinced that the combination offers a sufficiently original contribution for ICLR.

  2. Why does Prior-Adam-OPT underperform both GGS and PB-OPT by such a large margin at 500 queries, then improve sharply by 1,000 queries?

  3. Please revise “outperforms 16 state-of-the-art hard-label attacks.” This statement pools surrogate-free methods such as HSJA, TA, and SurFree with surrogate-assisted methods. Please distinguish these settings and specify where PB-OPT improves over baselines with comparable surrogate access.

AuthorsOpenReview ↗

Follow-up response to Reviewer Yg1f

Q1+: Methodological Novelty.

A1+: We respectfully disagree that PB-OPT is merely a combination of existing techniques. This view overlooks a fundamental challenge: under hard-label settings, the target provides only discrete labels, making its true boundary-distance function non-differentiable, while BO must directly optimize normalized ray directions in a high-dimensional space. Existing techniques cannot be directly transferred to this setting.

(1) QE-DBA's BO is not directly applicable. QE-DBA performs BO in a low-dimensional perturbation parameter space (e.g., Perlin or wavelet parameters), whereas our method directly optimizes normalized ray directions in 3,072 dimensions for CIFAR-10 and 9,408 dimensions for ImageNet. Derivative-free candidate sampling can be effective in low-dimensional spaces but becomes inefficient in such high-dimensional spherical optimization. This requires gradient-based acquisition optimization, including tangent-space projection, rather than merely changing the search space. Our matched-budget experiments with random search and CEM further verify this limitation.

(2) P-BO's functional prior cannot be directly applied to hard-label settings. P-BO relies on differentiable logits, whereas our boundary distance (g(θ)g(\theta)) is obtained through discrete searches and has no directly accessible gradient. PB-OPT therefore introduces a decoupled forward–backward mechanism: the forward pass estimates surrogate boundary distances through parallel queries, while the backward pass uses implicit differentiation of surrogate C&W margins to provide substitute gradients. Without the backward mechanism, the scalar prior alone would lead to derivative-free acquisition optimization. Our prior-gradient ablation further confirms its importance.

(3) Prior-OPT's transfer prior and our global functional prior operate at different levels. Prior-OPT uses transfer information for local gradient estimation and direction sampling. In contrast, PB-OPT incorporates surrogate boundary information into the GP mean as a global functional prior, jointly calibrating it through learnable (\gamma) and (c) and GP hyperparameters. GP-UCB then balances global prior guidance and predictive uncertainty. Our prior-mean ablation and adaptive calibration results support the effectiveness of this design.

Overall, our contribution is a hard-label-compatible mechanism for high-dimensional global Bayesian ray search, addressing limitations that prevent the direct application of the three existing approaches.

Q2+: Query-Budget Behavior of Prior-Adam-OPT.

A2+: The sharp ASR increase of Prior-Adam-OPT is mainly explained by its substantial initialization overhead. Before actual ray-direction optimization, it requires approximately 200 queries to identify the best direction among 100 candidates, followed by approximately 200 queries for the initial gradient estimation.

Consequently, under a 500-query budget, approximately 400 queries are consumed by initialization, leaving only about 100 queries for Adam-based optimization. Under a 1,000-query budget, approximately 600 queries remain for iterative updates. This explains the sharp ASR increase from 4.43% to 60.22% on AT ResNet-50 and from 0% to 14.94% on MIMIR.

Thus, the low ASR at 500 queries primarily reflects the initialization overhead and insufficient optimization steps, rather than the performance of the subsequent Adam updates.

We also clarify that GGS is a transfer-based attack with a fixed perturbation budget, rather than a query-based attack. Its ASR is independent of the target-query budget, so its result at 500 queries should not be interpreted as a direct query-efficiency comparison.

Q3+: Experimental Comparisons.

A3+: We agree with the reviewer and will reorganize the experimental results into two separate tables to distinguish the resource settings:

Surrogate-free attacks: HSJA, TA, SurFree, and other methods that do not use surrogate models.

Surrogate-assisted attacks: SQBA, Prior-OPT, Prior-Sign-OPT, PARS-OPT, Prior-Adam-OPT, and PB-OPT.

This separation will make the comparison settings explicit and highlight PB-OPT's improvements over baselines with comparable surrogate access, particularly Prior-OPT, Prior-Sign-OPT, PARS-OPT, and Prior-Adam-OPT. We will report the corresponding surrogate architectures, target models, and query budgets to facilitate fair comparisons.

We will also revise the statement "outperforms 16 state-of-the-art hard-label attacks" to avoid implying that all methods share identical resource assumptions. Transfer-based methods such as GGS and CORTA will be clearly distinguished from query-based hard-label attacks and cannot be directly compared.

These revisions will provide a more transparent evaluation of PB-OPT under different resource settings.

Reviewer_VMSuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes B-OPT and PB-OPT for hard-label adversarial attacks. B-OPT casts the OPT ray search as Bayesian optimization over normalized directions. The prior is computed by a decoupled forward-backward scheme. The forward pass estimates the surrogate boundary distance with parallel surrogate queries. The backward pass uses implicit differentiation of the surrogate C&W margin to get a substitute gradient. The prior is scaled by learnable γ and shifted by c, both fitted with the GP hyperparameters by marginal likelihood. The paper proves a regret bound for a simplified setting with a fixed prior. Experiments on CIFAR-10 and ImageNet, CLIP, AT, and MIMIR, against 16 hard-label baselines, show large gains in query efficiency.

Critical strengths
  1. The decoupled forward-backward design is valuable. Hard-label attacks only see top-1 labels, so the boundary distance is not differentiable. The paper uses the surrogate model's C&W margin and implicit differentiation to get a substitute gradient, while the target is only queried for labels. This is a clean way to combine a differentiable prior with a non-differentiable target.

  2. The empirical gains are large and consistent on the main benchmarks. On CIFAR-10 WRN, PB-OPT reaches 0.29 L2 at 2,000 queries, against 0.50 for HSJA/TA and 0.59 for Prior-OPT. On ImageNet Swin, it reaches 15.75 against 36.74 for Prior-OPT and 45.82 for HSJA.

Critical weaknesses
  1. The comparison is not apples-to-apples, and the surrogate is doing most of the work. PB-OPT uses a surrogate model trained on the same data; HSJA, TA, and SurFree do not. The paper's headline "outperforms 16 SOTA" mixes the two groups. The fair comparison is to the prior-based methods, which also use a surrogate. PB-OPT still wins there, but the margin is smaller. More importantly, B-OPT without the surrogate is worse than HSJA on CIFAR-10. So the gain comes from the surrogate prior, not from the BO framework.

  2. The theory is a standard GP-UCB regret bound with a prior mean, not a new result. The paper admits this, but then the theorem does not apply to the algorithm. The proof seems to be correct, but the result is not a contribution. A reviewer should ask for an empirical estimate of B_L, or a discussion of when the bound is non-vacuous.

Questions for authors
  1. How was the surrogate trained? On the target's training set or a disjoint set?

  2. In Table 3, the HSJA and G-TA ℓ∞ rows are identical at every budget? Is this a transcription error?

  3. On DenseNet-BC, PB-OPT is worse than SQBA (1.156 vs 0.386).

  4. The ablation says the "Standard" prior beats the "C&W Margin" prior, but the definitions are not clear.

1 comment
AuthorsOpenReview ↗

Clarifications on Methodology, Theory, and Experimental Results

Authors thank Reviewer VMSu.

Q1: The comparison mixes surrogate-free and surrogate-assisted methods, raising fairness concerns. Although PB-OPT outperforms prior-based baselines, B-OPT underperforms HSJA on CIFAR-10, suggesting that the gains mainly come from the surrogate prior rather than BO.

A1: We will separate surrogate-free and surrogate-assisted methods in the experimental tables and ensure that all surrogate-assisted baselines use the same surrogate architectures as our method for fair comparison.

As shown in Table 1, at 2,000 queries, PB-OPT reduces mean L2 distortion from 0.46 to 0.29 compared with SQBA on WRN-28-10. On ImageNet, Table 3 shows a reduction from 36.74 to 15.75 compared with Prior-OPT on Swin Transformer. PB-OPT also consistently outperforms baselines against AT and MIMIR defenses (Fig. 2).

Although B-OPT does not always outperform HSJA on CIFAR-10, it achieves substantially lower distortion on ImageNet (28.78 vs. 45.82; Table 3). Our contribution lies in integrating Bayesian optimization with surrogate-derived functional priors, rather than either component alone. Table 4 confirms their complementary contributions: removing the GP acquisition gradient or prior gradient increases mean L2 distortion from 15.75 to 27.50 or 28.92, respectively. These results demonstrate that the improvement stems from their effective integration rather than the surrogate prior alone.


Q2: The regret bound largely follows standard GP-UCB analysis, with limited theoretical novelty and unclear applicability to practical PB-OPT. An empirical estimate of BLB_L or an analysis of when the bound is non-vacuous is needed.

A2: Theorem 3.1 characterizes how functional priors affect Bayesian ray search. Without a prior, the complexity term is B0=∥∥fL∥∥kB_0=\|\| f_L\|\|_k; with a prior, it becomes BL=∥∥fL−f^∥∥kB_L=\|\| f_L-\hat f \|\|_k. An informative prior with BL<B0B_L<B_0 therefore yields a tighter regret bound:

$ \frac{R_T}{T}\leq B_L\sqrt{\frac{8\Gamma_T}{T\log(1+\sigma_n^{-2})}}+2\delta_L. $

The bound is informative when residual complexity, information gain, and boundary-estimation error are sufficiently small. This motivates our adaptive calibration of γ\gamma and cc, whose effectiveness is supported by Table 4 and Fig. 3(b). We will clarify the theoretical contribution and applicability conditions.

Q3: How was the surrogate trained? On the target's training set or a disjoint set?

A3: The surrogate and target models are independently trained on the same dataset's training split (CIFAR-10 or ImageNet), with separate parameters. The surrogate has no access to target parameters or gradients. PB-OPT further learns γ\gamma and cc from target-model observations to adaptively calibrate the prior across datasets and target models.

Q4: In Table 3, the HSJA and G-TA ℓ∞\ell_\infty rows are identical at every budget. Is this a transcription error?

A4: No. The identical values are due to rounding to two decimal places. The results with three-decimal precision are reported below.

Method400800120016002000
HSJA0.3510.3100.2880.2450.225
G-TA0.3520.3090.2900.2480.227

Q5: On DenseNet-BC, PB-OPT is worse than SQBA in CIFAR-10.

A5: We acknowledge this exception (1.156 vs. 0.386). However, PB-OPT achieves the best results on the other three CIFAR-10 architectures (Table 20): PyramidNet-272, WRN-28-10, and WRN-40-10, with L2 distortions of 0.414, 0.290, and 0.276, respectively. It also achieves the best results across all six ImageNet architectures (Tables 3 and 18; Figs. 7–8) and consistently outperforms baselines on AT and MIMIR (Fig. 2). DenseNet-BC highlights potential surrogate–target boundary misalignment. Our learnable γ\gamma and cc adaptively calibrate prior influence using target feedback, although they cannot eliminate all structural mismatches.

Q6: The ablation says the "Standard" prior beats the "C&W Margin" prior, but the definitions are unclear.

A6: The C&W Margin prior follows P-BO (Cheng et al., 2024), evaluating the surrogate C&W logit margin at x+rtθˉx+r_t\bar\theta, where θˉ\bar\theta is the normalized search direction and rtr_t is the current best target-model boundary distance. The radius remains fixed during each acquisition optimization and is updated when a better adversarial example is found. For untargeted attacks:

$ \hat f_{\mathrm{CW}}(\theta) =\max_{j\neq y}z_j^{(s)}(x+r_t\bar\theta) -z_y^{(s)}(x+r_t\bar\theta). $

In contrast, Standard denotes our proposed boundary-distance prior, obtained by explicitly searching for the surrogate boundary g^(θ)\hat g(\theta) and using 1/g^(θ)1/\hat g(\theta) (or −g^(θ)-\hat g(\theta)) as the GP prior, with gradients computed via implicit differentiation. Under the same −g(θ)-g(\theta) objective, Table 4 shows that Standard reduces L2 distortion from 20.72 to 17.70 and increases ASR from 37.39% to 49.25%.

Reviewer_H47nOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies query-efficient hard-label black-box adversarial attacks, where the attacker can only access the top-1 prediction of the target model. The authors propose two methods, B-OPT and PB-OPT, to improve the efficiency of ray-search-based attacks. B-OPT reformulates the conventional boundary-distance minimization objective as maximizing its reciprocal and employs Gaussian Process (GP)-based Bayesian optimization to search for effective perturbation directions in a reduced-dimensional space. Building upon B-OPT, PB-OPT incorporates a surrogate-derived functional prior into the GP model. To address the non-differentiability of the boundary-search objective, the authors introduce a decoupled forward–backward mechanism: the forward pass estimates surrogate boundary distances through parallel queries, while the backward pass uses surrogate margin gradients to guide acquisition optimization. The method also introduces learnable calibration parameters to adjust the influence of the functional prior. The proposed approach is promising, particularly in its integration of surrogate-derived functional information into hard-label Bayesian optimization. However, the extent to which the functional prior itself contributes to the reported improvements, the reliability of the approximate boundary gradients, and the applicability of the theoretical analysis to the practical algorithm require further clarification.

Critical strengths

[1] The paper addresses the high query complexity of hard-label black-box adversarial attacks, which is an important problem in adversarial robustness evaluation. [2] The integration of surrogate-derived functional priors with Gaussian Process-based ray search provides a principled way to exploit information from an accessible surrogate model while retaining feedback from the target model. The decoupled forward–backward mechanism is an interesting component of the proposed framework. [3] he experiments cover multiple datasets, target architectures, query budgets, and model settings, including robust classifiers and vision-language models. The reported results indicate that PB-OPT can improve both attack success rates and perturbation quality under constrained query budgets.

Critical weaknesses
  1. The ablation results suggest that removing the prior-gradient component substantially weakens PB-OPT, making it difficult to determine whether the main improvement comes from the global functional prior, gradient-based acquisition optimization, or their interaction.
  2. The theoretical derivation assumes suitable local regularity conditions and a non-degenerate boundary crossing. However, the practical method estimates boundary locations using finite-precision searches. The paper does not adequately quantify the resulting gradient approximation error or its effect on optimization stability.
  3. The regret bound is derived for a simplified setting with a fixed functional prior and idealized acquisition maximization. Practical PB-OPT uses adaptive prior calibration, approximate gradients, and finite-step acquisition optimization. The implications of these differences are not sufficiently established.
  4. Although PB-OPT performs well across several target architectures, the appendix reports a notable failure on DenseNet-BC, where PB-OPT underperforms SQBA. This raises concerns about the reliability of the surrogate prior when the surrogate and target decision boundaries are poorly aligned.
  5. Target-model query counts are important, but PB-OPT also incurs surrogate boundary-search, gradient-computation, and GP-optimization costs. Without wall-clock runtime and surrogate evaluation statistics, the practical efficiency of the method is difficult to assess comprehensively.
  6. The empirical results support the use of the reciprocal boundary-distance objective, but the paper does not sufficiently explain its advantages over alternative monotonic transformations or analyze its sensitivity to boundary-distance estimation errors.
Questions for authors

Please refer to the part on weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Concerns on Methodology, Theoretical Guarantees, and Experimental Evaluation

Q1: Removing the prior-gradient component substantially weakens PB-OPT, leaving the individual and joint contributions of the functional prior and acquisition optimization unclear.

A1: Table 4 already provides component-wise ablations. Under matched objective transformations, different priors (Random, C&W Margin, and Standard) demonstrate the importance of an informative functional prior. Furthermore, at 2,000 queries, removing the GP acquisition gradient increases mean L2 distortion from 15.75 to 27.50 and reduces ASR from 57.79% to 26.23%. Removing the prior gradient further increases distortion to 28.92 and reduces ASR to 29.75%. These results confirm that both components make complementary contributions to PB-OPT.


Q2: Finite-precision boundary estimation may introduce gradient errors and affect optimization stability, which are not sufficiently analyzed.

A2: We compared implicit C&W gradients with high-precision finite-difference (FD) references on 25 random rays (5 images × 5 rays). The surrogate boundary search introduces a +3.7% estimation bias. Nevertheless, implicit gradients achieve cosine similarities of 0.98 and 0.93 at the exact and estimated boundaries, respectively, whereas FD using the same search precision achieves only 0.66. This demonstrates the robustness of implicit differentiation against boundary quantization errors.


Q3: The theoretical regret bound does not account for adaptive prior calibration, approximate gradients, or finite-step acquisition optimization in practical PB-OPT.

A3: Theorem 3.1 is intended to characterize the effect of functional prior quality under a simplified GP-UCB setting. In practical PB-OPT, the prior is adaptively calibrated through learnable γ\gamma and cc, whose effectiveness is supported by Fig. 3(b) and Table 4. The implicit gradient approximation achieves cosine similarities of 0.98 and 0.93 against high-precision FD references. Finite-step acquisition optimization introduces an additional maximization gap not explicitly covered by the theorem. We will clarify the theorem's scope and discuss these practical approximations in the revised manuscript.


Q4: Although PB-OPT performs well across several target architectures, the appendix reports a failure on DenseNet-BC, where PB-OPT underperforms SQBA.

A4: We acknowledge this exception. However, PB-OPT achieves the best overall performance across all evaluated ImageNet architectures and three of four CIFAR-10 architectures (PyramidNet-272, WRN-28-10, and WRN-40-10). The DenseNet-BC result highlights potential surrogate–target boundary misalignment. PB-OPT mitigates this issue through adaptive calibration of γ\gamma and cc, using target-model observations to adjust the prior's influence. Although this cannot eliminate all structural mismatches, the results demonstrate its effectiveness across most evaluated architectures.


Q5: Computational overhead.

A5: We profiled PB-OPT on ImageNet using ResNet-101 as the target and ResNet-50 as the surrogate, with 2,000 target queries.

MetricResult
Surrogate evaluations/image13,959 batched calls (2.45M image-forwards)
Wall-clock time/image19.8 min

Further profiling attributes approximately 88% of runtime to surrogate boundary search, 10% to gradient computation, and 0.9% to GP fitting. Thus, surrogate boundary search is the primary bottleneck.

Nevertheless, in practical black-box attacks, reducing queries to the target model is often more critical, as target access may be limited or costly. In contrast, surrogate evaluations can be extensively parallelized and batched, and their runtime can be substantially reduced with additional GPU resources. Thus, PB-OPT trades scalable surrogate computation for improved target-query efficiency.


Q6: The advantages of the reciprocal boundary-distance objective over other monotonic transformations and its sensitivity to boundary estimation errors are not sufficiently analyzed.

A6: Table 4 already compares 1/g(θ)1/g(\theta) with the monotonic alternative −g(θ)-g(\theta) under the same optimization framework. At 2,000 queries, the reciprocal objective improves B-OPT from 31.01 to 28.78 mean L2 distortion and from 18.09% to 31.06% ASR. For PB-OPT, distortion decreases from 17.70 to 15.75, while ASR increases from 49.25% to 57.79%. Although both transformations share the same optimum, they induce different optimization landscapes, with the reciprocal transformation assigning greater sensitivity to smaller boundary distances.

Regarding estimation errors, suppose ∣g~−g∣≤δ|\tilde g-g|\leq\delta and g≥gmin⁡>δ>0g\geq g_{\min}>\delta>0. Then,

$ \left|\frac{1}{\tilde g}-\frac{1}{g}\right| \leq \frac{\delta}{g_{\min}(g_{\min}-\delta)}.$

Thus, the reciprocal transformation has bounded estimation error when the boundary distance is sufficiently far from zero. We will clarify this theoretical property and the results in the revised manuscript.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Request for Reviewer Feedback

Dear Reviewers and Area Chair,

We thank all reviewers for their careful reading and constructive feedback. We have added new experiments and provided detailed answers to each question, and we hope these additions address your concerns.

We would like to kindly note that the rebuttal and discussion period is now very close to its deadline, and we have not yet heard back from Reviewers H47n and VMSu regarding our responses. We would greatly appreciate it if you could take a moment to review them and share your thoughts. Your feedback is important to us and to the final assessment of this paper.

To briefly restate the main contributions of our work:

Bayesian optimization for hard-label attacks with functional priors. We propose B-OPT, a Bayesian ray-search framework for hard-label attacks, and further develop PB-OPT by incorporating surrogate-derived functional priors. PB-OPT uses a decoupled forward–backward design: hard-label queries provide boundary-aware objective values in the forward pass, while surrogate gradients supply optimization guidance in the backward pass, enabling efficient acquisition maximization despite the non-differentiability of the boundary objective.

Theoretical analysis. We derive a regret bound (Theorem 3.1) for a simplified PB-OPT, showing that informative priors reduce the discrepancy between the target objective and the prior (the residual complexity BLBL​), leading to tighter optimization bounds. This motivates our adaptive prior calibration with learnable γ and c.

State-of-the-art performance. Extensive experiments on ImageNet and CIFAR-10, as well as attacks against CLIP and robust defense models (MIMIR & AT), show that our method consistently outperforms 16 recent state-of-the-art hard-label attacks, achieving minimum distortion with only 2,000 queries.

We hope this summary helps clarify the value of our submission. We remain available for any further questions.

Best regards, The Authors

AuthorsOpenReview ↗

Request for Reviewer Feedback

Thank you again for your review. As the discussion period is approaching its deadline, I would like to kindly follow up on our rebuttal. We have carefully addressed your concerns and provided additional clarifications and experimental evidence. We would greatly appreciate it if you could take another look at our response and let us know if any concerns remain.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.