- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work examines the problem of developing a method to improve white-box attack success rate against defenders who use an array of defenses which are randomly selected when presented with an example. The central idea is to use white box model access to develop "skeleton key" inputs, which are inputs where the target model will produce a specific class label and all other models will produce a class that differs from the specific class output by the target model. Skeleton keys can be generated by a PGD-like algorithm where an input is perturbed in a way that encourages correct classification for the target model and incorrect classification for non-target models. By generating many skeleton keys for each target model, one can directly estimate the probability distribution with which the defender selects a given defense model by querying the defense with skeleton keys and identifying which model is used each time.
Critical strengths
- The overall method is sound: a PGD-like approach is used to stabilize the class of the target model and perturb the class of non-target models, which enables a simple and effective way for the attacker to leak information about the defense distribution.
- The results in Table 4 show that taking the skeleton key distribution into account can increase the effectiveness of adversarial attacks against randomized defenses using an ensemble of models.
Critical weaknesses
- The paper as written leaves me with some questions about exactly how the randomized defense is evaluated. I am assuming that 1000 APGD steps are use to create an adversary, and that the defense is shown this final adversary a single time to determine whether the input is classified correctly or incorrectly. This should be clarified directly in the main text. The exact evaluation strategy relates to the Single Query Use assumption: if the defense can be queried repeatedly, Skeleton Keys should not be necessary and it should be possible to continue to show the defender adversaries crafted with each individual model, which will cause robustness to collapse to at most the level of the weakest defense in the ensemble. The importance of the SQU assumption is not clearly outlined in the text but it matters for understanding the evaluation. The natural accuracy and robustness for this kind of random defense is stochastic and it seems better to include some kind of uncertainty measurements when presenting these metrics, or to present expected natural accuracy/robustness. Overall, there are some key areas relating to the randomness of this defense that are not laid out clearly in the text.
- The experimental scope of the method is relatively limited. The method only studies the case where the randomness comes from model selection, and only studies CIFAR-10. It would strengthen the paper to also investigate the possibility of using Skeleton Keys for random transformations such as BaRT, and to extend the results to other datasets.
- The results in Table 12 seem to cast doubt upon the claims that follow from Table 4. The benefit of the skeleton-key informed method is much less significant in Table 12 (except for Two defenses) than Table 4. This should be clarified: are skeleton keys more effective for certain defender configurations, and only provide a minor benefit for other configurations?
Questions for authors
- Can you provide a more detailed explanation of the evaluation protocol, how it relates to the SQU assumption, and how uncertainty across model selection impacts the metrics reported in Table 4?
- Can the method be extended beyond the scope of model ensembles to incorporate random transformations? Does it work on datasets beyond CIFAR-10?
- Can you provide more context about the less significant results in Table 12 vs. Table 4?
- The LLM statement required by ICLR 2027 looks like it is missing. Did you use LLMs when writing the paper, and if so how?
5 comments
Q2: “datasets beyond CIFAR-10” = We have evaluated our method on the Tiny-ImageNet dataset, which also shows promising results in terms of skeleton key yield rate and estimation problem. During the rebuttal period, for the Tiny-ImageNet experiments, we used three adversarially trained defense models (WRN-28-10, CaiT-XXS, and MambaVision-T). In the following table, we have outlined the clean accuracy and the robust accuracy of the three models: Table: Clean and robust accuracy of each of the three models on Tiny-ImageNet. Robust accuracy is evaluated under a maximum perturbation of .
| Model | Clean Accuracy | Robust Accuracy |
|---|---|---|
| WRN | 65.19% | 31.30% |
| CAIT | 52.82% | 30.47% |
| MAMBA | 71.05% | 40.14% |
| The comparison between the ordinary-input and skeleton key algorithm generated keys is outlined in the following table: | ||
| Table: Skeleton key generation algorithm evaluation for each of the three models on Tiny-ImageNet. All skeleton keys use 100 iterations and a maximum perturbation of . | ||
| Model | Valid | Non-Valid |
| :-- | :--: | :--: |
| WRN-28-10 | 605 | 9395 |
| CaiT-XXS | 397 | 9603 |
| MambaVision-T | 1477 | 8523 |
On Tiny-ImageNet, our algorithm generates a large number of valid skeleton keys for every target model, from 397 for CaiT-XXS up to 1477 for MambaVision-T, with yield rates of 3.97% to 14.77%. In each case, the generated keys far exceed the clean samples that already satisfy the skeleton key condition (2.0x to 4.5x), confirming that our method scales effectively to a larger and more complex dataset. The following table reports the estimated defense distributions and the resulting robust accuracies on Tiny-ImageNet.
Table: Estimate of and robust accuracy of randomized model ensemble under different attack settings on Tiny-ImageNet. JSD denotes Jensen–Shannon divergence computed using base-2 logarithms. R_U denotes robust accuracy when the attacker assumes a uniform probability distribution. R_S denotes the robust accuracy when the attacker uses the skeleton key estimated distribution . .
| Models | JSD | |||||
|---|---|---|---|---|---|---|
| WRN, CaiT, Mamba | [0.10, 0.00, 0.90] | [0.0812, 0.0001, 0.9187] | 48.20% | 35.90% | 12.30% | |
| WRN, CaiT, Mamba | [0.33, 0.33, 0.34] | [0.3342, 0.3143, 0.3515] | 42.80% | 42.80% | 0.00% | |
| WRN, CaiT, Mamba | [0.00, 0.00, 1.00] | [0.0000, 0.0001, 0.9999] | 48.40% | 33.10% | 15.30% | |
| WRN, CaiT, Mamba | [0.24, 0.24, 0.52] | [0.2477, 0.2303, 0.5220] | 44.00% | 42.30% | 1.70% |
Attack Analysis on Tiny-ImageNet: The above table shows that using the skeleton key estimated defense distribution also improves attack effectiveness over the uniform baseline on Tiny-ImageNet. For the two skewed defense configurations, robust accuracy drops from 48.20% to 35.90% and from 48.40% to 33.10%, reductions of 12.30 and 15.30 percentage points, respectively. When the defense distribution is close to uniform, the gain is small or vanishes (1.70 and 0.00 percentage points), because the uniform assumption already closely matches the true distribution. These results show that the estimated defense distribution gives the attacker the most useful information when the defender’s model selection is non-uniform, yielding up to a 15.30 percentage-point gain in attack success. These results demonstrate that our method generalizes beyond CIFAR-10 to a larger and more complex dataset. We will include the Tiny-ImageNet results in the revised manuscript.
[1] https://arxiv.org/abs/2207.03574 (Adversarial Robustness of Random Transformation Defenses) [2] https://arxiv.org/abs/2211.14669 (GaME paper)
W3, Q3: “context about the less significant results in Table 12 vs. Table 4”
= We have added the results for all the rows to make Table 4 and Table 12 consistent. Both the tables now have an equal number of rows. The numbers of Table 12 have now changed. Here are the updated numbers for our Table 12.
Table: Estimate of and robust accuracy of randomized model ensemble under different attack settings on CIFAR-10. R_U denotes robust accuracy when the attacker assumes a uniform probability distribution. R_S denotes the robust accuracy when the attacker uses the skeleton key estimated distribution . . Model abbreviations are: W = WRN-28-10, C = CaiT-XXS, V = ViT-L/16, M = MambaVision-T, D = DM-R18, and F = FAT-R18.
| Models | |||||
|---|---|---|---|---|---|
| W,C,V,M,D,F | [1, 0, 0, 0, 0, 0] | [0.9995, 0.0001, 0.0001, 0.0001, 0.0001, 0.0001] | 80.30% | 70.40% | 9.90% |
| W,C,V,M,D,F | [0.5, 0, 0.25, 0.25, 0, 0] | [0.5049, 0.0001, 0.2349, 0.2599, 0.0001, 0.0001] | 78.50% | 71.60% | 6.90% |
| W,C,V,M,D,F | [0.26, 0.07, 0.26, 0.17, 0.17, 0.07] | [0.2699, 0.0777, 0.2521, 0.1344, 0.1934, 0.0725] | 70.20% | 69.60% | 0.60% |
| W,C,V,M,D,F | [0.167, 0.167, 0.167, 0.167, 0.166, 0.166] | [0.1877, 0.1555, 0.1630, 0.1555, 0.1729, 0.1654] | 66.60% | 67.20% | −0.60% |
| C,V,M,D,F | [0, 0.4813, 0.5187, 0, 0] | [0.0001, 0.4824, 0.5173, 0.0001, 0.0001] | 75.90% | 65.90% | 10.00% |
| Here, for the CIFAR-10 results, in three of the five settings, skeleton keys reduce robust accuracy by 6.90–10.00 points. Only the two near-uniform settings show small gains, as expected. | |||||
| For the Tiny-ImageNet results, in two of the four settings, skeleton keys reduce robust accuracy by 12.30–15.30 points. The moderately non-uniform setting gives a smaller gain of 1.70 points, and the uniform setting gives none, as expected. | |||||
| We will clarify this in the revised manuscript. |
Q4: “Did you use LLMs when writing the paper, and if so how?” = We used LLM only to polish the writing, such as correcting grammar and spelling. We will update this in the revised manuscript.
We will make sure to address all the responses in the updated version. We thank the reviewer for their diligence and humbly ask them to reconsider their score for the paper accordingly.
We thank the reviewer for the careful and constructive feedback. We have addressed all the questions which improved the overall contributions of this work.
W1, Q1: “explanation of the evaluation protocol”, “how it relates to the SQU assumption”
= We thank the reviewer for this insightful comment. The reviewer is correct that the final adversarial example is shown a single time and our robustness is average over 1000 different APGD samples. The one caveat here is the defense can be queried multiple times, but our assumption is this occurs only with DIFFERENT samples. The same sample cannot be repeatedly used to query the defense. This also motivates our use of multiple skeleton keys (instead of a single skeleton key). The original assumption of SQU is to defeat memory based defense systems. E.g. if a defender keeps track of the inputs submitted by each user, it would be easy to identify malicious users as they would be repeating the same exact input every time in a sequence. This phenomena and corresponding defense was well documented in the literature:
[1] https://arxiv.org/abs/1907.05587 (Stateful Detection of Black-Box Adversarial Attacks)
[2] https://arxiv.org/abs/2006.14042 (Defense for Neural Networks against Query-Based Black-Box Attacks)
We will update the paper’s writing according to the reviewer’s feedback to make the relationship between the evaluation protocol and SQU assumption more clear.
Q2: “incorporate random transformations”
= BaRT is a different estimation problem which is expectation over transformation for a series of image processing techniques. Furthermore, BaRT is an older defense that has been shown as vulnerable to different techniques that use advanced estimation techniques. Papers that have already advanced attacks on BaRT to reduce robustness are listed here:
[1] https://arxiv.org/abs/2207.03574 (Demystifying the Adversarial Robustness of Random Transformation Defenses) [2]https://ieeexplore.ieee.org/document/11155075 (Momentum Iterative Method over Expectation)