acceptodds
Under review as a conference paper at ICLR 2027

Rethinking the Use of Ensemble Updates in Adversarial Attacks

Abstract

Adversarial examples can mislead deep neural networks with subtle perturbations and often transfer across models, a phenomenon known as adversarial transferability. Model ensemble attacks improve the transferability of adversarial examples to unknown models by jointly using information from multiple surrogate models. Existing methods primarily focus on constructing effective attack updates and typically apply the resulting update rule throughout the attack. However, iterative attacks are stateful sequential optimization processes, in which the same update may have different effects on final transferability depending on when it is applied along the attack trajectory. To investigate this question, we define Attack Operation Value (AOV), which measures the final benefit of method-specific computation by comparing the final transfer effects of method-specific and base ensemble updates from the same attack state. Our analysis reveals substantial variation in the returns of computationally expensive updates across attack stages: their effectiveness at one stage does not imply that they should be applied throughout the attack trajectory. Experiments under a fixed compute budget further show that a small number of early expensive updates, followed by sufficient base ensemble optimization, can yield a more effective allocation of computation. Motivated by these findings, we propose Temporal Ensemble Scheduling (TES), which concentrates a limited number of method-specific updates at the beginning of the attack and uses inexpensive base ensemble updates for the remaining iterations. Experiments on images and 3D point clouds show that TES substantially reduces computational cost and runtime while maintaining competitive black-box transferability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Transferability, Ensemble Attacks, Temporal Scheduling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BeZaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Existing transfer-based ensemble attacks typically apply computationally expensive updates throughout the iterative attack, although the same update may contribute differently to terminal transferability at different attack stages. To address this issue, the paper proposes Temporal Ensemble Scheduling (TES). TES applies a limited number of expensive method-specific updates early in the attack and then switches to inexpensive base ensemble updates, while preserving the attack state and the original number of outer iterations. To analyze stage-dependent returns, the authors define Attack Operation Value (AOV), which measures the benefit of an update by comparing terminal transfer utility from the same incoming attack state under different updates followed by a common continuation rule. TES is applied to CWA, SMER, and SVRE and evaluated on image and point-cloud datasets. The experiments show that TES substantially reduces runtime while maintaining competitive transferability and, in some settings, outperforms the full schedule that applies method-specific updates at every iteration. Additional controls on scheduling, optimizer state, and parameter sensitivity support its effectiveness and help characterize its scope of applicability.

Critical strengths

1.While the proposed scheduling method is simple, its practical value is clear. Expensive ensemble updates need not be applied throughout the entire attack: a short early refinement prefix can substantially reduce runtime while maintaining or even improving transferability. For example, Table 5 reports ASR gains of approximately 3.27, 2.77, and 2.92 percentage points for CWA, SMER, and SVRE, respectively, on NIPS2017, together with speedups of approximately 43.3%, 74.4%, and 70.5%.. 2.This paper includes early, middle, and late placement comparisons with matched update counts, as well as uniform and random schedules, extended-base controls, short-refinement controls, different surrogate compositions, and attack-seed repetitions. The authors also appropriately distinguish the target panels used for mechanism analysis and method evaluation, and acknowledge settings in which TES underperforms Full. These controls make the contribution more convincing than a comparison between TES and the original attacks alone.

Critical weaknesses

1.The novelty of the contribution and the generality of the scheduling rule remain limited, and the separation between schedule selection and independent evaluation needs further clarification. AOV is a useful diagnostic contrast, but its value depends on the incoming state, intervention window, continuation policy, and evaluation targets. The paper does not yet establish a relationship that predicts when refinement is beneficial or how to select the prefix length. The default configuration, K = 0.2T, is chosen based on the CWA prefix-length experiments, and its effectiveness varies across settings. For example, Table 37 reports 72.90% ASR for the 20% prefix versus 76.48% for Full in point-cloud SMER at T = 30. In Table 39, image CWA with the M2 surrogate ensemble achieves 19.39% ASR with TES, below the 21.20% achieved by Base. 2.The existing budget-allocation experiments primarily match reference gradient-computation counts and do not fully establish the advantage of TES over extended-base and short-refinement controls under equal wall-clock budgets. Appendices D.5 and E clearly distinguish reference computation from runtime, but gradient counts do not capture all method-dependent costs, including additional forward evaluations and ensemble-weight updates. Some SVRE controls also use approximately rather than exactly matched reference counts. Consequently, the experiments convincingly demonstrate computational savings relative to the evaluated Full implementations, but the evidence remains insufficient to establish the best allocation under an equal wall-clock budget.

Questions for authors

1.Which samples, surrogate models, and target models were used to select K = 0.2T? Do these settings overlap with those used for the final method evaluation? Although the appendix distinguishes the model panels used for mechanism analysis and method evaluation, the independence of schedule selection from final evaluation needs further clarification. Can the authors verify whether this default configuration remains competitive on samples and target models that were not used for prefix selection? 2.Do the advantages of TES over extended-base and short-refinement controls persist under matched wall-clock budgets on an exclusive GPU? A representative comparison on image and point-cloud tasks would help support the conclusions about compute allocation. 3.Please clarify the source, label mapping, and train/test split of the evaluated ShapeNet HDF5 data, and explain how parity between the adapted attack implementations and their native implementations was checked.

Reviewer_dZNMOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the temporal allocation of optimization operations in ensemble-based transferable adversarial attacks. Existing ensemble attack methods typically execute computationally expensive method-specific updates throughout the entire attack process. The authors argue that iterative attacks are inherently state-dependent sequential optimization processes, meaning that the same operation may have different effects on final transferability depending on when it is applied. To investigate this phenomenon, the paper introduces Attack Operation Value, which quantifies the terminal contribution of an attack operation by comparing different updates from the same attack state under a shared continuation policy. Motivated by this analysis, the authors propose Temporal Ensemble Scheduling, which concentrates computationally expensive method-specific updates in the early attack iterations and subsequently switches to cheaper base ensemble updates.

Critical strengths
  1. The paper revisits ensemble-based adversarial attacks from the perspective of temporal operation allocation, offering a research perspective distinct from conventional model selection, gradient aggregation, and ensemble reweighting.
  2. The experimental evaluation is extensive, covering six datasets, including NIPS2017, ImageNet-Val, ImageNet-Sketch, ModelNet40, ScanObjectNN, and ShapeNet, together with diverse black-box target architectures, perturbation budgets, and defense configurations.
  3. The manuscript is generally well organized, progressing from the state-dependent nature of iterative attacks to the formal definition of AOV, the analysis of stage-dependent operation values, computational budget allocation, and the proposed TES strategy.
Critical weaknesses
  1. Limited methodological novelty and theoretical justification: Although AOV is introduced to characterize the terminal benefits of attack operations at different stages, it essentially provides an empirical estimate of the terminal performance difference between two update strategies under a shared continuation policy. It does not establish a new optimization principle or provide theoretical guarantees regarding scheduling optimality. More importantly, the connection between AOV and TES remains largely indirect. AOV is estimated through offline interventions and target-model evaluations, whereas TES ultimately adopts a fixed-ratio prefix schedule without explicitly using AOV estimates to adaptively determine operation selection or switching points.
  2. The fixed prefix ratio of K=0.2TK=0.2T raises concerns about the generalizability of TES. The experimental results indicate substantial variations in the optimal number of refinement updates across attack methods and modalities.
  3. The paper acknowledges the multi-stage model scheduling approach of Hu et al. (2022), which is an important related study. However, the discussion mainly emphasizes the distinction between model scheduling and operation scheduling, without sufficiently establishing the practical advantages of TES relative to existing scheduling principles through direct experimental comparisons. In addition, Composite Adversarial Attacks (Mao et al., AAAI 2021) formulates attack construction as a sequence of attack algorithms and employs multi-objective search to optimize attack strength and computational complexity. Although its primary setting differs from transferable ensemble attacks, it addresses closely related questions concerning attack-operation composition, execution order, and computational budget allocation.
Questions for authors

Please refer to the above weaknesses.

Reviewer_5WgHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether expensive ensemble refinement updates are equally useful throughout iterative transfer attacks. The authors introduce Attack Operation Value (AOV) to measure how the timing of refinement updates affects the final attack success rate (ASR). The experimental results suggest that refinement is most beneficial early in optimization, while continued refinement may even hurt transferability.

Based on this observation, the authors propose Temporal Ensemble Scheduling (TES), which applies refinement during the first 20% of iterations and uses cheaper base updates afterward. Experiments on image and point-cloud datasets show significant speedups, often with improved ASR on images, although results on point clouds are more mixed.

Critical strengths
  • TES is simple to implement and substantially reduces computational cost. The reported speedups are impressive, especially since they often come with improved ASR.
  • The paper includes multiple attack methods, datasets, and modalities, along with repeated trials, computational cost measurements, and several useful ablations.
  • The extended-base, short-refinement, and alternative scheduling experiments help rule out simple explanations for the improvements.
Critical weaknesses
  1. My main concern is whether the Full baselines are sufficiently tuned for the evaluated settings. For example, CWA uses inner hyperparameters originally designed for $ \epsilon = 16/255 $, while the main experiments use $ \epsilon = 4/255 $. Some Full results are surprisingly poor: CWA-M2 Full performs substantially worse than Base (13.52 vs. 21.20, Table 39), and SMER Full also falls below Base at (T = 40) (Fig. 9b).

These results raise the possibility that TES benefits from avoiding poorly tuned refinement updates rather than demonstrating a general advantage of early refinement. Moreover, TES's advantage diminishes at $ \epsilon = 8/255 $, while results at the standard ϵ=16/255\epsilon = 16/255 are missing.

  1. TES concentrates expensive refinement updates at the beginning, but the paper does not compare against using cheaper refinement throughout all iterations. For instance, SVRE could use fewer inner steps per iteration while maintaining a similar total computational budget. Without this comparison, it is unclear whether concentrating refinement early is actually better than distributing weaker refinement across the entire attack.

  2. AOV is a useful empirical measurement, but it does not explain why early refinement should improve transferability. The mechanism experiments are also largely limited to CWA and a few update positions. An alternative explanation is that continued refinement overfits the surrogate ensemble, while switching to base updates reduces this overfitting. Comparing surrogate-side losses or white-box ASR between Full and TES would help clarify the underlying mechanism.

  3. The default $ K = 0.2T $ is mainly motivated by CWA, but Fig. 8 shows that the preferred $ K $ varies across methods and modalities. Full also outperforms TES in some longer-horizon settings. A method for selecting $ K $ without target-model feedback would make TES more convincing and practically useful.

  4. Some terminologies and notations are not clearly explained. For example, what exactly does the "method-specific update" do, and what is the function $ U $?

Questions for authors
  1. What exactly does the "method-specific update" do, and what is the function $ U $?

  2. Were CWA, SMER, and SVRE hyperparameters retuned for the evaluated perturbation budgets and attack horizons? How do the authors explain cases where Full performs worse than Base?

  3. How does TES compare with applying lower-cost refinement at every iteration under the same computational budget?

  4. Can the authors provide results at $ \epsilon = 16/255 $, given the diminishing advantage of TES at $ \epsilon = 8/255 $?

  5. Can $ K $ be selected without target feedback, and do the mechanism findings hold under the same model configurations used in the main experiments?

  6. Table 1 shows that early loss-level updates can reduce transferability substantially, while the main CWA experiments demonstrate positive early-stage returns. Can the authors characterize what properties of a method-specific update determine whether its optimal placement should be early, middle, or late?

1 comment
AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear Reviewer 5WgH,

Thank you for your careful review and for raising questions about the experimental controls and the interpretation of our results. As the discussion period draws to a close, we would appreciate any feedback you have on our two-part response. If any of your questions remain open, please let us know and we would be happy to clarify.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.