Rethinking the Use of Ensemble Updates in Adversarial Attacks

Adversarial examples can mislead deep neural networks with subtle perturbations and often transfer across models, a phenomenon known as adversarial transferability. Model ensemble attacks improve the transferability of adversarial examples to unknown models by jointly using information from multiple surrogate models. Existing methods primarily focus on constructing effective attack updates and typically apply the resulting update rule throughout the attack. However, iterative attacks are stateful sequential optimization processes, in which the same update may have different effects on final transferability depending on when it is applied along the attack trajectory. To investigate this question, we define Attack Operation Value (AOV), which measures the final benefit of method-specific computation by comparing the final transfer effects of method-specific and base ensemble updates from the same attack state. Our analysis reveals substantial variation in the returns of computationally expensive updates across attack stages: their effectiveness at one stage does not imply that they should be applied throughout the attack trajectory. Experiments under a fixed compute budget further show that a small number of early expensive updates, followed by sufficient base ensemble optimization, can yield a more effective allocation of computation. Motivated by these findings, we propose Temporal Ensemble Scheduling (TES), which concentrates a limited number of method-specific updates at the beginning of the attack and uses inexpensive base ensemble updates for the remaining iterations. Experiments on images and 3D point clouds show that TES substantially reduces computational cost and runtime while maintaining competitive black-box transferability.