- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a granularity-aware black-box transfer adversarial attack. The author refines existing cropping, hidden state alignment, and hidden state selection, respectively, with semantic scope construction, correspondence-aware full-patch alignment, and hierarchy-wise semantic alignment. The SSA emphasizes cropping within reasonable regions to control the abrupt perturbation brought by random cropping, while the CFPA provides a more robust alignment through computing the optimal transport in the hidden state. The experiment shows that this method achieves state-of-the-art results in almost all cases.
Critical strengths
-
This paper provides a method with clear motivation and explanation. The method itself is easy to follow and performs reasonably well. The semantic scope construction is novel, providing alignment in a restricted region with strong semantic meanings.
-
The authors provide extensive experiments across different victim model series, including both commercial models like GPT-5.4, GPT-5.6o, Gemini, and Claude Opus 5, and open-source series like Kimi, Qwen, and GLM, covering a wide range of victim models.
Critical weaknesses
- There is no ablation study on the hyperparameters introduced in this paper, although many are presented such as as well as cropping parmeters.
- Appendix A.5 admits that the cost of CFPA is extensive when trying to align a 1024 tokens, but they also did not provide a time comparison.
Questions for authors
I am concerned about potential model bias (leak) in the evaluation, as the SSC used GPT-6 Astra to label the semantic areas and GPT-4o to act as the judge, since they are both GPT series. It would be better to include a small additional setting using either a different judge or a different labeling model.
4 comments
Response to Reviewer d1Ut (part2)
W2: Running time and memory cost.
As suggested, we compare the running time and peak GPU memory for generating an adversarial example on a single NVIDIA RTX 4090, including model loading.
| Method | Iterations | Time (s) | Peak GPU memory (GB) |
|---|---|---|---|
| AnyAttack | 1 forward pass | 28.0 | 0.75 |
| M-Attack | 300 | 48.4 | 8.38 |
| FOA | 300 | 96.3 | 8.41 |
| M-Attack-v2 | 300 | 462.4 | 8.30 |
| MPCAttack | 300 | 48.6 | 10.43 |
| PRAF | 300 | 67.4 | 12.09 |
| GAST | 300 | 746.7 | 9.79 |
Most of GAST's extra runtime comes from exact Hungarian matching solved sequentially on the CPU via SciPy. This matching only pairs adversarial and target patches outside the computation graph, adding no extra loss terms or surrogate passes. Given the pairs, the alignment loss costs the same as aligning patches in a fixed order. The remaining overhead comes from the per-scope branches and the alignment of all intermediate layers. Even so, GAST's peak GPU memory (9.79 GB) stays below that of PRAF (12.09 GB) and MPCAttack (10.43 GB).
PRAF and MPCAttack use the same five surrogate encoders as GAST. To check that this extra cost does not explain our gains, we further ran PRAF and MPCAttack for more iterations to match GAST's total FLOPs. Since GAST's cost scales with the number of scopes, we match FLOPs per image pair and list the average iterations below. Each cell reports AvgSim / [email protected] (%) / [email protected] (%) / KMR (%) on the 100-pair benchmark.
| Method | Iterations | GPT-5.4 | Gemini-2.5 | Qwen3.8-Max | Average |
|---|---|---|---|---|---|
| MPCAttack | 300 | 0.415 / 45 / 25 / 47.9 | 0.558 / 69 / 51 / 65.8 | 0.593 / 76 / 58 / 74.6 | 0.522 / 63.3 / 44.7 / 62.8 |
| MPCAttack (FLOPs-matched) | 2,213 | 0.431 / 53 / 29 / 53.3 | 0.625 / 79 / 62 / 68.3 | 0.628 / 84 / 61 / 76.3 | 0.561 / 72.0 / 50.7 / 66.0 |
| PRAF | 300 | 0.553 / 69 / 49 / 63.2 | 0.610 / 79 / 61 / 65.7 | 0.668 / 87 / 71 / 75.0 | 0.610 / 78.3 / 60.3 / 68.0 |
| PRAF (FLOPs-matched) | 1,500 | 0.611 / 76 / 58 / 64.2 | 0.664 / 87 / 68 / 69.4 | 0.705 / 90 / 76 / 78.5 | 0.660 / 84.3 / 67.3 / 70.7 |
| GAST | 300 | 0.659 / 86 / 69 / 69.5 | 0.677 / 85 / 73 / 72.8 | 0.727 / 92 / 83 / 76.6 | 0.688 / 87.7 / 75.0 / 72.9 |
More iterations boost PRAF and MPCAttack by 7.0 and 6.0 points in average [email protected]. However, GAST still leads both FLOPs-matched baselines by at least 5 points in [email protected] on every victim, showing that compute budget alone does not explain the performance gap. We have added both analyses to Section 4.4 (Table 4) and Appendix A.5 (Tables 5 and 6) of the revised paper.
Summary. We have made the following revisions in response to the reviewer's comments.
- Section 4.5 and Appendices A.7 and A.8: results with two alternative concept extractors, including the non-GPT tagger RAM++, and with three non-OpenAI judges.
- Appendices A.4 and A.10: the momentum coefficient and the sensitivity study on the loss weights.
- Section 4.4 and Appendix A.5: running time, peak GPU memory, and comparison with FLOPs-matched baselines.
Across all of these analyses, GAST keeps a clear lead over the strongest baseline. Its gains therefore come from semantic-scope supervision rather than from GPT-family components, tuned hyperparameters, or extra computation.
We hope these results address the reviewer's concerns, and we would be glad to provide further details if needed.
Thank you for the detailed response and the additional experiments. The rebuttal has addressed most of my concerns, and I will maintain my score.
Thank you for your positive score and continued support for our work. We are glad that the additional experiments and clarifications have resolved your concerns.
Response to Reviewer d1Ut (part1)
We thank the reviewer for the positive review. Below we respond to the comments in Questions (Q) and Weaknesses (W). The new results on the concept extractor, the judge, hyperparameter sensitivity, and computational cost have been added to the revised paper (Sections 4.4 and 4.5, Appendices A.5, A.7, A.8, and A.10), with all changes marked in red.
Q: GPT family annotator and judge.
We followed this suggestion and replaced each of the two components separately.
Different annotator. We replaced GPT-6-Astra with RAM++, an open-source image tagger, and passed its tags directly to Grounding DINO without any verification by GPT, keeping the number of scopes close to the original. On the 100-pair benchmark (averaged over GPT-5.4, Gemini-2.5-Flash, and Qwen3.8-Max), [email protected] changes from 75.0% to 72.7% and AvgSim from 0.688 to 0.685, which remains well above PRAF (60.3% and 0.610). On the OpenAI victim GPT-5.4, [email protected] stays at 69% with RAM++, the same as with GPT-6-Astra. This shows that the gain of GAST comes mainly from the semantic scopes of SSC rather than from a specific extractor. We have added these results to Section 4.5 (Table 4) and Appendix A.7 (Table 9) of the revised paper.
Different judge. We rescored the same victim outputs used in the paper (GPT-5.4, Gemini-3-Flash, Qwen3-VL-32B-Instruct and Kimi K3) with three judges outside the OpenAI family, namely Claude Sonnet 5, Qwen3.7 Flash and DeepSeek 4.1 Flash. Under all four judges, GAST ranks first among the five methods on all four metrics and leads the second-best method, PRAF, by 13.0 to 15.7 points in [email protected]. In addition, the scoring reference for each victim is the caption that the victim itself produces for the clean target image (Section 4.1), so no GPT model supplies the reference for victims outside the GPT family. The gains of GAST therefore reflect a genuine transfer of target semantics rather than a preference of the GPT-4o judge. We have added these results to Section 4.5 (Table 4) and Appendix A.8 (Table 10) of the revised paper.
Changing either the annotator or the judge leaves the conclusions unchanged, so the gains do not appear to be tied to GPT family annotation or judging.
W1: Hyperparameter sensitivity.
For a fair comparison, GAST adopts the same hyperparameter values as prior work (Appendix A.4). The values of the loss weights (λc,λp)=(0.5,1.5) follow PRAF, the momentum coefficient μ=0.9 takes the same value as in M-Attack-v2, and the crop ranges are taken from M-Attack-v2 and PRAF. Since GAST changes what the patch term aligns, from the top 60% of patches in PRAF to all patches under exact correspondence, the inherited loss weights are not guaranteed to remain suitable. We therefore ran a sensitivity study on λc and λp with 12 settings around the default, which is shown in bold. Results are on the 100-pair benchmark and averaged over GPT-5.4, Gemini-2.5-Flash, and Qwen3.8-Max.
Removing the patch alignment term (λp=0) causes a sharp drop to 30.0% [email protected], showing that correspondence-aware patch alignment is indispensable. When λp is small, increasing λc lowers performance, since the global CLS term then dominates the patch term. The inherited default remains the best setting in AvgSim, [email protected] and [email protected], and the other settings with λp≥1.0 keep [email protected] between 68.0% and 71.0%, still 7.7 to 10.7 points above PRAF (60.3%). GAST therefore does not rely on a narrowly tuned choice of loss weights. We have added this sensitivity study to Appendix A.10 (Table 13) of the revised paper.
Unlike the loss weights, μ and the crop ranges govern the optimizer and view sampling rather than the alignment objective that GAST modifies, so we keep them at these inherited values in all experiments. The only crop that GAST changes is the target crop of each semantic branch, which is placed according to its semantic scope, and this choice is evaluated by the Random Scope ablation in Table 3(a). Placing this crop on the semantic scope raises average [email protected] by 8.2 points over random placement. We have added the value of μ to Appendix A.4 of the revised paper.