- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes FRA-Attack, a frequency-domain approach to improving the transferability of targeted adversarial attacks against closed-source MLLMs. The method consists of two components: (1) high-frequency DCT feature alignment, which aims to capture shared visual focus across different vision encoders, and (2) frequency-domain gradient regularization (FGR), which suppresses high-frequency surrogate-specific gradient noise to improve transferability. Experiments on 15 MLLMs demonstrate consistent improvements over existing attack methods, particularly on closed-source models such as GPT, Claude, and Gemini.
Critical strengths
- The proposed method is simple and effective, combining frequency-domain feature alignment and gradient regularization to improve attack transferability.
- The motivation behind the method is clear.
- Extensive experiments on 15 MLLMs demonstrate consistent improvements over strong baselines.
Critical weaknesses
First, the authors argue that spatial-domain local features fail to capture the intrinsic visual focus. However, it remains unclear whether high-frequency components can effectively capture this focus. Second, the authors propose a low-pass filter to suppress high-frequency gradients due to their low cross-model consistency. However, I am concerned that this may also remove useful information, especially since much of the gradient energy is concentrated in high-frequency components.
Although Table 6 provides additional analyses to support these claims, the experimental setups and measurement definitions are not sufficiently clear. I suggest providing more detailed explanations of how these experiments are conducted to help readers better understand and evaluate the proposed motivations.
Questions for authors
Can the author explain the Table 6 more?
6 comments
Weakness 2:
Thank you for raising this. FGR is not a hard low-pass cutoff. In fact, it attenuates smoothly with frequency and never zeroes out any band. To see what it actually removes, we compared gradients directly before and after filtering. In short, filtering mainly shrinks the update's magnitude and leaves its direction largely intact. The small part it does change is what makes the gradient transfer better.
The direction barely changes. On 100 images, filtered vs. unfiltered updates:
| Metric | Value |
|---|---|
| Cosine similarity | 0.94 |
| Pixels with the same update sign | 88% |
| Magnitude removed | ~70% |
About 70% of the magnitude is removed, but the direction hardly moves. High-frequency content goes down but never reaches zero.
| High-frequency share of energy | w/o FGR | w/ FGR |
|---|---|---|
| Per-step gradient | 0.308 | 0.074 |
| Final perturbation (300 steps) | 0.316 | 0.164 |
What we change moves in the direction of improved transfer. We use a leave-one-out setting: one surrogate is held out, the rest are aggregated, and we measure the cosine between the aggregated gradient and each surrogate's raw gradient.
| Cosine with | w/o FGR | w/ FGR | Relative | Paired 95% CI (Δ) |
|---|---|---|---|---|
| Held-out surrogate | 0.0152 | 0.0166 | +9% | [+0.0011, +0.0016] |
| Contributing surrogates | 0.533 | 0.489 | −8% | — |
FGR gives up a little fidelity to the ensemble members in exchange for agreement with a model outside the ensemble, and that trade is exactly what it is designed to make.
End-to-end results. After optimization, the embedding cosine between adversarial and target images on 3 unseen encoders rises from 0.566 to 0.581. In an otherwise identical ablation, FGR improves attack success rate by 5.0 points over no filtering, averaged over three closed-source victims.
Question:
Thank you for the suggestion. Below we explain what each row of Table 6 measures, how its control is built, and what the numbers indicate.
Rows 1–2 test the loss-side DCT alignment; rows 3–6 test the gradient-side low-pass filter FGR, on the gradient (rows 3–4) and on the final perturbation (rows 5–6). Each control changes one component only. Surrogates throughout: CLIP ViT-B/16, CLIP ViT-B/32, LAION ViT-G/14.
| Row | Measured | N | Control → Ours |
|---|---|---|---|
| 1. Focus-map agreement | Cross-surrogate correlation of maps rebuilt from the selected token-DCT coefficients | 100 images × 3 domains | Count-matched low-frequency: 0.244 → 0.537 (2.2×) |
| 2. Support agreement | Cross-surrogate agreement under one Grad-times-Input attribution | 100 target images | FOA cluster centers: 0.040 → 0.236 (5.9×) |
| 3. Shared gradient energy | Input-gradient energy along the cross-model mean, unrelated-gradient floor removed | 100 images | FGR off: 0.0092 → 0.0124 (+35%) |
| 4. Cosine to held-out surrogate | Filtered ensemble gradient vs. a held-out surrogate's raw gradient | 100 images | FGR off: 0.0152 → 0.0166 (+9%) |
| 5. High-band energy of δ | Perturbation energy at radial frequency d ≥ 0.5 | 1,000 pairs | FGR off: 0.316 → 0.164 (−48%) |
| 6. Target cosine, 3 unseen encoders | Adversarial–target cosine on non-surrogate encoders | 1,000 pairs | FGR off: 0.566 → 0.581 (+2.7%) |
Row 1: do the selected components mark locations that all encoders emphasize? The encoders are trained independently, so agreement means the emphasis comes from the image content. High-frequency selection agrees 2.2 times as well as low-frequency selection, whose components (the slowest variation along the token sequence, largely shared across patches and close to what [CLS] already carries) localize weakly. Our rule's indices overlap least across encoders (0.281 vs. 0.533), yet its maps agree best: the encoders share image locations, reached through encoder-specific frequency coordinates. The alignment target is thus anchored to image locations, which every victim also sees.
Row 2: how should the local target be defined? FOA-Attack's centers come from K-means on each surrogate's patch features and depend on initialization and on that surrogate's feature geometry; our support is selected by energy on a fixed DCT basis. Under the same attribution, the centers' maps are nearly uncorrelated across encoders (0.040), so each surrogate acts on different regions; our support agrees 5.9 times as well (top-20% IoU 0.132 → 0.201). This is the direct basis for replacing cluster centers with the DCT support.
Row 3: how does filtering change the gradient's composition? Here we apply a spatial 2D DCT to each surrogate's input gradient. C_adj is the energy of the cross-model mean spectrum over the mean energy of the individual spectra. With similar norms, it is roughly the mean pairwise cosine. So the unfiltered 0.0092 means the surrogate gradients are nearly orthogonal. Filtering raises this shared fraction by 35% relative. This matches our band-wise probe in the response to Reviewer 82Gz: cross-surrogate agreement falls from low to high frequency, and by the end of the run the shared fraction is 2.5%, 1.5% and 1.0% in the low, mid and high bands. FGR keeps the band where the surrogates agree most and attenuates the band where they agree least.
Row 4: does the change point toward models outside the ensemble? The filtered gradient of two surrogates is compared with the raw gradient of the third, which contributes nothing and stands in for an unseen victim. This cosine rises by 9% while the cosine to the contributing surrogates falls from 0.533 to 0.489: filtering weakens each member's specific directions, trading a little fidelity to the members for the agreement with outside models that transfer requires.
Row 5: does the change survive into the final perturbation? Filtering moves energy from the high band (−48%) to the low band (0.342 → 0.518) and lowers the spectral centroid from 0.373 to 0.278, with all 1,000 images moving the same way on all three quantities. The gradient-level change thus carries through 300 sign steps and ℓ∞ clipping into the perturbation that reaches the victim.
Row 6: does the perturbation also act on encoders outside the attack? CLIP-L/14@336, ImageNet ViT-B/16 and BLIP-Base are all outside the surrogate set, the latter two also outside the CLIP family. Filtering raises the adversarial–target cosine on them by 2.7%, so FGR's change also acts in representation spaces the attack never optimized. The measure also tracks transfer: the four complete methods (M-Attack, FOA-Attack, M-Attack-V2, ours) score 0.548, 0.560, 0.576 and 0.581, matching their ASR order in Table 1.
Weakness 1:
Thank you for this follow-up. The high-frequency components do capture this focus, and the claim can be measured step by step.
What the DCT removes and what it keeps. The 1D DCT over the token sequence removes the content shared by all patches and keeps each patch's deviation from it. On a 14×14 grid each row is 14 consecutive tokens, so atom k goes through k half-cycles from the top row to the bottom, and for k < 10 it changes by less than a third of a cycle within any row. The removed atoms (k < θ = 10) are therefore horizontal stripes. F0 is proportional to the mean of all patches, the global content [CLS] already carries; the other nine are the slow vertical changes of the row means, such as a sky–ground gradient. Removing them approximately subtracts from each patch the mean of its row. Background in natural images is largely layered by height (sky, horizon, ground, wall, tabletop), so it is shared along a row and absorbed by the row mean. The subject is local in the horizontal direction and stands out from its row. This is why the remaining components should concentrate on the subject, and the two measurements below test this directly.
Agreement across encoders. For each encoder and each image we zero the token-DCT coefficients that the selection rule did not choose, invert the transform, take the per-patch feature norm and normalize; this gives one focus map per encoder. If the maps of different encoders point to the same place, the content they emphasize is shared. Rebuilt from our high-frequency selection, the correlation across the three surrogates is 0.537, against 0.244 for a count-matched low-frequency selection and 0.009 for a random one. The spatial local feature sits in the same measurement: scoring FOA-Attack's cluster centers and our DCT support with the same patch-level gradient-times-input attribution gives 0.040 and 0.236 across the three surrogates, and 0.020 against 0.537 on each method's own native region map. The three surrogates use different patch grids (14×14, 7×7, and 16×16) and therefore different scan orders. Their selected coefficient indices overlap by only 0.281, yet the maps rebuilt from these differing indices are precisely the ones that agree most closely. What they share is a spatial emphasis: the reconstruction concentrates on the subject (Appendix A, Fig. 5).
Masking the highlighted regions. Agreement shows that the maps are reproducible across encoders, and the next question is whether the model actually reads these positions. We masked them and looked at the output. For each clean source image we replace the top 5%, 10% and 20% of the region highlighted by the focus map rebuilt under our rule with grey patches of equal area, ask a black-box victim to describe the image and measure how far that description moves from the one it gives for the unmasked image (one minus the cosine between the two text embeddings). Across three victims (GPT-4o, GPT-5.4, Claude-Opus-4.6) and three mask sizes, masking the high-frequency regions shifts the description more than masking low-frequency regions of the same area in all nine combinations,; at the 20% size the paired differences are 0.033, 0.065 and 0.114. Going from 5% to 10%, the shift for the low-frequency regions barely moves while the high-frequency shift keeps rising on all three victims.
Revision. The masking measurement will be added to the revision.