acceptodds
Under review as a conference paper at ICLR 2027

FRA-Attack: Frequency-Domain Regularized Alignment for Transferable Adversarial Attacks against Closed-Source MLLMs

Abstract

Multimodal large language models (MLLMs) remain vulnerable to transfer-based targeted attacks, where perturbations optimized on open-source surrogate encoders can generalize to closed-source MLLMs. A key challenge for improving adversarial transferability is to effectively capture the intrinsic visual focus shared across different models, such that perturbations align with transferable semantic cues rather than surrogate-specific behaviors. However, existing methods suffer from spatial-domain feature redundancy and surrogate-specific gradient signals, thereby hindering cross-model transferability. In this paper, we propose FRA-Attack, which addresses both challenges from a unified frequency-domain perspective. For feature alignment, a high-pass DCT objective concentrates on the high-frequency band of patch features that carries MLLMs' intrinsic visual focus, suppressing redundant global structures. For gradient optimization, we introduce a model-agnostic low-pass gradient regularizer on the frequency domain that modulates the surrogate gradient using only the geometric frequency coordinate, removing surrogate-specific high-frequency artifacts while preserving transferable low-frequency directions. Together, the two components form a unified frequency-domain treatment of transferability. Extensive experiments on flagship MLLMs across vendors show that FRA-Attack achieves superior cross-model transferability, particularly with state-of-the-art performance on GPT-5.4, Claude-Opus-4.6 and Gemini-3-flash.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Attacks, Multimodal Large Language Models, Frequency-Domain Feature Alignment, Gradient Regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HN7rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FRA-Attack, a frequency-domain approach to improving the transferability of targeted adversarial attacks against closed-source MLLMs. The method consists of two components: (1) high-frequency DCT feature alignment, which aims to capture shared visual focus across different vision encoders, and (2) frequency-domain gradient regularization (FGR), which suppresses high-frequency surrogate-specific gradient noise to improve transferability. Experiments on 15 MLLMs demonstrate consistent improvements over existing attack methods, particularly on closed-source models such as GPT, Claude, and Gemini.

Critical strengths
  1. The proposed method is simple and effective, combining frequency-domain feature alignment and gradient regularization to improve attack transferability.
  2. The motivation behind the method is clear.
  3. Extensive experiments on 15 MLLMs demonstrate consistent improvements over strong baselines.
Critical weaknesses

First, the authors argue that spatial-domain local features fail to capture the intrinsic visual focus. However, it remains unclear whether high-frequency components can effectively capture this focus. Second, the authors propose a low-pass filter to suppress high-frequency gradients due to their low cross-model consistency. However, I am concerned that this may also remove useful information, especially since much of the gradient energy is concentrated in high-frequency components.

Although Table 6 provides additional analyses to support these claims, the experimental setups and measurement definitions are not sufficiently clear. I suggest providing more detailed explanations of how these experiments are conducted to help readers better understand and evaluate the proposed motivations.

Questions for authors

Can the author explain the Table 6 more?

6 comments

Weakness 1:

Thank you for this follow-up. The high-frequency components do capture this focus, and the claim can be measured step by step.

What the DCT removes and what it keeps. The 1D DCT over the token sequence removes the content shared by all patches and keeps each patch's deviation from it. On a 14×14 grid each row is 14 consecutive tokens, so atom k goes through k half-cycles from the top row to the bottom, and for k < 10 it changes by less than a third of a cycle within any row. The removed atoms (k < θ = 10) are therefore horizontal stripes. F0F_0 is proportional to the mean of all patches, the global content [CLS] already carries; the other nine are the slow vertical changes of the row means, such as a sky–ground gradient. Removing them approximately subtracts from each patch the mean of its row. Background in natural images is largely layered by height (sky, horizon, ground, wall, tabletop), so it is shared along a row and absorbed by the row mean. The subject is local in the horizontal direction and stands out from its row. This is why the remaining components should concentrate on the subject, and the two measurements below test this directly.

Agreement across encoders. For each encoder and each image we zero the token-DCT coefficients that the selection rule did not choose, invert the transform, take the per-patch feature norm and normalize; this gives one focus map per encoder. If the maps of different encoders point to the same place, the content they emphasize is shared. Rebuilt from our high-frequency selection, the correlation across the three surrogates is 0.537, against 0.244 for a count-matched low-frequency selection and 0.009 for a random one. The spatial local feature sits in the same measurement: scoring FOA-Attack's cluster centers and our DCT support with the same patch-level gradient-times-input attribution gives 0.040 and 0.236 across the three surrogates, and 0.020 against 0.537 on each method's own native region map. The three surrogates use different patch grids (14×14, 7×7, and 16×16) and therefore different scan orders. Their selected coefficient indices overlap by only 0.281, yet the maps rebuilt from these differing indices are precisely the ones that agree most closely. What they share is a spatial emphasis: the reconstruction concentrates on the subject (Appendix A, Fig. 5).

Masking the highlighted regions. Agreement shows that the maps are reproducible across encoders, and the next question is whether the model actually reads these positions. We masked them and looked at the output. For each clean source image we replace the top 5%, 10% and 20% of the region highlighted by the focus map rebuilt under our rule with grey patches of equal area, ask a black-box victim to describe the image and measure how far that description moves from the one it gives for the unmasked image (one minus the cosine between the two text embeddings). Across three victims (GPT-4o, GPT-5.4, Claude-Opus-4.6) and three mask sizes, masking the high-frequency regions shifts the description more than masking low-frequency regions of the same area in all nine combinations,; at the 20% size the paired differences are 0.033, 0.065 and 0.114. Going from 5% to 10%, the shift for the low-frequency regions barely moves while the high-frequency shift keeps rising on all three victims.

Revision. The masking measurement will be added to the revision.

Weakness 2:

Thank you for raising this. FGR is not a hard low-pass cutoff. In fact, it attenuates smoothly with frequency and never zeroes out any band. To see what it actually removes, we compared gradients directly before and after filtering. In short, filtering mainly shrinks the update's magnitude and leaves its direction largely intact. The small part it does change is what makes the gradient transfer better.

The direction barely changes. On 100 images, filtered vs. unfiltered updates:

MetricValue
Cosine similarity0.94
Pixels with the same update sign88%
Magnitude removed~70%

About 70% of the magnitude is removed, but the direction hardly moves. High-frequency content goes down but never reaches zero.

High-frequency share of energyw/o FGRw/ FGR
Per-step gradient0.3080.074
Final perturbation (300 steps)0.3160.164

What we change moves in the direction of improved transfer. We use a leave-one-out setting: one surrogate is held out, the rest are aggregated, and we measure the cosine between the aggregated gradient and each surrogate's raw gradient.

Cosine withw/o FGRw/ FGRRelativePaired 95% CI (Δ)
Held-out surrogate0.01520.0166+9%[+0.0011, +0.0016]
Contributing surrogates0.5330.489−8%—

FGR gives up a little fidelity to the ensemble members in exchange for agreement with a model outside the ensemble, and that trade is exactly what it is designed to make.

End-to-end results. After optimization, the embedding cosine between adversarial and target images on 3 unseen encoders rises from 0.566 to 0.581. In an otherwise identical ablation, FGR improves attack success rate by 5.0 points over no filtering, averaged over three closed-source victims.

Question:

Thank you for the suggestion. Below we explain what each row of Table 6 measures, how its control is built, and what the numbers indicate.

Rows 1–2 test the loss-side DCT alignment; rows 3–6 test the gradient-side low-pass filter FGR, on the gradient (rows 3–4) and on the final perturbation (rows 5–6). Each control changes one component only. Surrogates throughout: CLIP ViT-B/16, CLIP ViT-B/32, LAION ViT-G/14.

RowMeasuredNControl → Ours
1. Focus-map agreementCross-surrogate correlation of maps rebuilt from the selected token-DCT coefficients100 images × 3 domainsCount-matched low-frequency: 0.244 → 0.537 (2.2×)
2. Support agreementCross-surrogate agreement under one Grad-times-Input attribution100 target imagesFOA cluster centers: 0.040 → 0.236 (5.9×)
3. Shared gradient energyInput-gradient energy along the cross-model mean, unrelated-gradient floor removed100 imagesFGR off: 0.0092 → 0.0124 (+35%)
4. Cosine to held-out surrogateFiltered ensemble gradient vs. a held-out surrogate's raw gradient100 imagesFGR off: 0.0152 → 0.0166 (+9%)
5. High-band energy of δPerturbation energy at radial frequency d ≥ 0.51,000 pairsFGR off: 0.316 → 0.164 (−48%)
6. Target cosine, 3 unseen encodersAdversarial–target cosine on non-surrogate encoders1,000 pairsFGR off: 0.566 → 0.581 (+2.7%)

Row 1: do the selected components mark locations that all encoders emphasize? The encoders are trained independently, so agreement means the emphasis comes from the image content. High-frequency selection agrees 2.2 times as well as low-frequency selection, whose components (the slowest variation along the token sequence, largely shared across patches and close to what [CLS] already carries) localize weakly. Our rule's indices overlap least across encoders (0.281 vs. 0.533), yet its maps agree best: the encoders share image locations, reached through encoder-specific frequency coordinates. The alignment target is thus anchored to image locations, which every victim also sees.

Row 2: how should the local target be defined? FOA-Attack's centers come from K-means on each surrogate's patch features and depend on initialization and on that surrogate's feature geometry; our support is selected by energy on a fixed DCT basis. Under the same attribution, the centers' maps are nearly uncorrelated across encoders (0.040), so each surrogate acts on different regions; our support agrees 5.9 times as well (top-20% IoU 0.132 → 0.201). This is the direct basis for replacing cluster centers with the DCT support.

Row 3: how does filtering change the gradient's composition? Here we apply a spatial 2D DCT to each surrogate's input gradient. C_adj is the energy of the cross-model mean spectrum over the mean energy of the individual spectra. With similar norms, it is roughly the mean pairwise cosine. So the unfiltered 0.0092 means the surrogate gradients are nearly orthogonal. Filtering raises this shared fraction by 35% relative. This matches our band-wise probe in the response to Reviewer 82Gz: cross-surrogate agreement falls from low to high frequency, and by the end of the run the shared fraction is 2.5%, 1.5% and 1.0% in the low, mid and high bands. FGR keeps the band where the surrogates agree most and attenuates the band where they agree least.

Row 4: does the change point toward models outside the ensemble? The filtered gradient of two surrogates is compared with the raw gradient of the third, which contributes nothing and stands in for an unseen victim. This cosine rises by 9% while the cosine to the contributing surrogates falls from 0.533 to 0.489: filtering weakens each member's specific directions, trading a little fidelity to the members for the agreement with outside models that transfer requires.

Row 5: does the change survive into the final perturbation? Filtering moves energy from the high band (−48%) to the low band (0.342 → 0.518) and lowers the spectral centroid from 0.373 to 0.278, with all 1,000 images moving the same way on all three quantities. The gradient-level change thus carries through 300 sign steps and ℓ∞ clipping into the perturbation that reaches the victim.

Row 6: does the perturbation also act on encoders outside the attack? CLIP-L/14@336, ImageNet ViT-B/16 and BLIP-Base are all outside the surrogate set, the latter two also outside the CLIP family. Filtering raises the adversarial–target cosine on them by 2.7%, so FGR's change also acts in representation spaces the attack never optimized. The measure also tracks transfer: the four complete methods (M-Attack, FOA-Attack, M-Attack-V2, ours) score 0.548, 0.560, 0.576 and 0.581, matching their ASR order in Table 1.

Reviewer_82GzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Proposes an approach for increased transferability of adversarial attacks in the context of multimodal language models. The attack focuses on use of directions that are not tied to a particular model and deliberate targeting of shared intrinsic focus.

Critical strengths

Multimodal language models are of high interest to the ICLR community.

Experimental results show improvements over baselines in terms of attack success rate, sometimes substantial.

Critical weaknesses

Evidence to support the limitations identified in 2.2 is not extensive enough.

There is already a vast literature on adversarial attacks. This is a well studied and not so fresh problem, need to convince the community that yet another attack is important.

Multiple hyperparameters required to be chosen for using the approach.

At a broad level, the use of frequency domain approaches in adversarial attacks is not a new idea.

Questions for authors

What are the theoretical reasons the frequency domain regularization is appropriate and effective?

6 comments

Weakness 1:

Thank you for your comment. We ran a dedicated measurement for each limitation.

Limitation 1: existing local terms are low-pass aggregates that duplicate the global term.

Setup: 500 images × 3 surrogates (CLIP ViT-B/16, ViT-B/32, LAION CLIP). We test whether the local terms of both families add information beyond the global term.

MeasurementB/16B/32LAION
Crop [CLS] vs. global [CLS], same image0.920.910.85
[CLS] cosine across different images (baseline)0.340.310.34
Residual of global token on crop subspace (same / cross image)0.26 / 0.910.25 / 0.920.41 / 0.94
Low-frequency share of [CLS] attention weights (white-noise level)0.22 (0.05)0.80 (0.20)0.68 (0.04)
  • Multi-scale [CLS] cropping (Li et al., 2025): crops of the same image capture 83–94% of the global token's energy, versus 12–17% for crops of other images. The per-crop [CLS] restates the global term instead of adding to it.
  • The cause is the aggregation itself. [CLS] attention puts 4–17× more energy on low frequencies than white noise, so each per-crop [CLS] is a low-frequency weighted average.
  • Patch clustering (Jia et al., 2025; K = 5): each center's aggregation weights carry 43–53% of their energy in the low band (white-noise level 4–20%), so centers average over large, smooth regions. Pairwise center cosine (0.40–0.54) is no higher than for random same-image patch pairs (0.29–0.62): clustering is no more separable than raw patches.
  • By contrast, token-wise frequency selection targets the focus (App. E.1): its focus map correlates at 0.537 across the three independently trained encoders, versus 0.244 for a count-matched low-frequency control and 0.009 for random selection.

Limitation 2: surrogate-specific noise still dominates after averaging, most of all at high frequencies.

Setup: 50 images × 300 steps, adaptive weighted averaging over the same three surrogates, snapshots every 50 steps.

We define ρ, the cross-encoder gradient cosine, as follows. For each image and each frequency band, we take each surrogate encoder's input gradient, apply the 2D DCT, keep the coefficients inside that band's radial mask, and flatten all three channels into a single vector per encoder. We then normalize each vector and compute the cosine between every pair of encoders, and average over the three pairs. A higher ρ means the encoders' gradients point in more similar directions within that band.

Measurement (t = 0 → 299)LowMidHigh
Cross-encoder gradient cosine ρ0.013 → 0.0250.003 → 0.0150.002 → 0.010
Shared-energy share after averaging (derived, final)7.00%4.40%2.90%
Band share of gradient energy0.23 → 0.180.59 → 0.550.18 → 0.27
Direction persistence: 1-step / 5-step cosine0.033 / 0.0080.021 / 0.0010.012 / 0.0004
  • The shared component decays with frequency. At every snapshot, ρ decreases monotonically from the low to the high band, and all three bands remain well above a mismatched-image baseline. The full curve ρ(d) decays smoothly. This regularity allows a fixed radial prior to replace surrogate-derived statistics.
  • The absolute level is small, as expected. Gradients of different models are known to be nearly orthogonal, while their adversarial subspaces still overlap. Writing each gradient as a shared part plus a surrogate-specific part, ρ equals the geometric mean of the two encoders' shared-energy fractions: 2.5% / 1.5% / 1.0% at the end.
  • Averaging helps, but equally in every band. It cuts surrogate-specific energy by ~1/3 in every band, nearly tripling the shared fraction (low band 2.5% → 7.0%), consistent with ensembles transferring better. It cannot change the ordering: 93–97% of the averaged gradient is still surrogate-specific, more so at higher frequencies. The high band's energy share also rises (0.18 → 0.27).
  • Plain averaging across surrogates does not remove this high-band noise. Over the course of the attack, it builds up. The reason is that the high-band component is not a fixed direction. It is re-injected at every step, and its direction is essentially gone after five steps (cosine 0.0004 between steps t and t+5). Under sign updates, every pixel moves by the same step size, wherever its gradient comes from. So this non-transferable content spends the perturbation budget just as much as the shared component does. Without filtering, the final perturbation still has a high-frequency share of 0.316.

Weakness 2:

Although the adversarial-example literature is extensive, it concerns image classification. The problem studied in this paper is different: targeted, image-to-image transfer attacks against closed-source MLLMs that are accessible only through an API. In this setting, success requires the victim to describe or answer about the adversarial image as it would about the target image. The attacker has neither victim gradients nor a usable query budget. The attack surface is likewise consequential, closed-source MLLMs increasingly receive image inputs in agentic and computer-use pipelines, and their robustness bears directly on user safety.

This difference carries measurable consequences. Under an identical protocol, four established methods from the classifier line (AttackVLM, AdvDiffVLM, SSA-CWA and AnyAttack) attain 0–1% ASR on all six closed-source standard victims. To our knowledge, only three works target closed-source MLLMs directly: M-Attack, FOA-Attack and M-Attack-V2. None of the three addresses the two conditions for transferability that we identify in Section 2.2: each aligns spatial-domain features (multi-crop features, patch clusters or stronger ensembles) and follows raw surrogate gradients. FRA-Attack changes what the alignment transports, namely the frequency band that carries visual focus shared across encoders, and adds a model-agnostic regularizer on the gradient side that this line has not used.

The improvement is substantial and stable across evaluation protocols. Against M-Attack-V2, the strongest prior method, ASR rises by 12.3 points on Claude-Opus-4.6 (64.5→76.8) and 10.0 points on Claude-Sonnet-4.6 (62.9→72.9). FRA-Attack ranks first on all nine closed-source victims and on five of six open-source victims spanning three vision-encoder families. The margin widens from +0.7 points at ε=4/255 to +11.3 points at ε=16/255, the ranking is preserved when the GPT-4o judge is replaced by two stricter independent judges, and FRA-Attack retains its lead under a VQA protocol scored by exact match.

Weakness 3:

Thank you for raising this. Our method adds only three hyperparameters: the DCT band cutoff θ, the number of components n taken from that band, and the exponent p of the radial attenuation. Every other value is either part of the attack setting shared with all methods in the comparison, or inherited from the prior transfer-attack recipe we build on.

ParameterValueRoleOrigin
θ10DCT cutoff for the alignment termours
n10components taken from the band, by energyours
p1.5exponent of the (1−d)p(1−d)^p gradient attenuationours
ε16/255ℓ∞ℓ_∞ budgetshared
N300iterationsshared
α1/255step sizeMI-FGSM
μ0.9momentum decayMI-FGSM
ensembleCLIP ViT-B/16, B/32, LAION ViT-g/14source of the gradientsprior work
wgw_g1.0weight of the [CLS] cosine termprior work
wlw_l0.2weight of the alignment termprior work
λ0.1entropic regularization of the alignment transportprior work
T1.0temperature of the ensemble weightingprior work

Weakness 4:

We thank the reviewer for this comment. Our method differs from prior frequency-domain attacks in the task it targets, in the regularities measured on that task, and in the design that follows from them.

The task is different. Prior frequency-domain attacks were mainly developed for and validated on CNN/ViT classification or detection (§4.3), where the spectrum serves label-flip transferability: shrinking query budgets (Guo et al., 2020), simulating model diversity via spectral augmentation (Long et al., 2022), keeping perturbations imperceptible (Luo et al., 2022) or confining them to critical bands (Wang et al., 2024), smoothing gradients for translation invariance (Dong et al., 2019), regularizing gradients in spectral subspaces (Guesmi et al., 2025), and reweighting gradient components by cross-model consistency (Gao et al., 2025; Fu et al., 2025). Our task is a targeted image-to-image attack on closed-source MLLMs: the victim exposes no logits, gradients or features, and success means the caption takes on the target image's semantics, not a label flip.

The placement and role of the spectrum also differ. Prior work transforms the input or the perturbation; FRA-Attack transforms two internal quantities: the patch embeddings that set the alignment target and the input gradient that sets the update. The design rests on measurements on this task. On the feature side, visual focus shared across independently trained encoders concentrates in the high-frequency band of the patch-token DCT (Appendix E.1). On the gradient side, the high-frequency band of the input gradient is the least consistent across encoders, i.e., surrogate-specific (§2.2). Together, the two measurements say which band to keep and which to filter. Appendix G (Table 20) compares the methods one by one.

Question : Thank you for the question. Frequency appears twice in our method: over patch features, and over the input gradient.

Feature side: the frequency domain captures each patch's deviation from the global content

The transform acts on an encoder's patch features, a 196×d matrix (a 14×14 grid in row-major order), along the token dimension, channel by channel. With the Type-II DCT basis on P = 196 patches, ψk(n)=cos[π(n+1/2)k/P]ψ_k(n) = cos[π(n + 1/2)k/P], the k-th component is Fk=∑nψk(n)⋅enF_k = \sum_{n} ψ_k(n)·e_n, and three identities make this the right place to separate global from local content:

  • F0=Σnψ0(n)⋅en=Σnen=PeˉF_0 = Σ_n ψ_0(n)·e_n = Σ_n e_n = P\bar{e}, so the DC component is exactly the mean patch embedding, the content every patch shares;
  • for every 1≤k≤P−11 \le k \le P-1, Σnψk(n)=0Σ_n ψ_k(n) = 0, so a patch-uniform signal (en≡ce_n ≡ c) gives Fk=0F_k = 0 exactly: no patch-uniform content enters the remaining components, and the separation is an identity, not an approximation;
  • within those components, 1P∑n[ψk(n+1)−ψk(n)]2=2sin⁡2πk2P\frac{1}{P}\sum_n[\psi_k(n+1)-\psi_k(n)]^2=2\sin^2\frac{\pi k}{2P} grows monotonically in k, so high k responds to the difference between neighbouring tokens, and the high band carries patch-to-patch variation — layout, boundaries, the edges that separate a subject from its surroundings — rather than the overall level.

We therefore retain only components beyond a cutoff in raster order, which excludes the image-wide mean and the slow top-to-bottom variation such as sky-to-ground. Inverting the selected components and plotting them by per-patch norm concentrates their energy on the image subject as one connected region (Fig. 5, Appendix A).

Spatial-domain local terms are redundant with the global term instead. For M-Attack, crops correlate with the global feature at 0.85 to 0.92 (cross-image baseline 0.31 to 0.34), so the local term largely measures what the global term already covers. FOA-Attack's cluster centres are weighted averages of correlated tokens and land in low frequencies too: only 25% to 39% of a centre's energy falls in the crop subspace, and its cosine with the nearest crop is 0.19 to 0.37. Both introduce a local term to add detail the global term lacks, but with neighbouring patches correlated to this degree, the local features collapse into low-frequency aggregates and the high-frequency detail is never explicitly aligned.

Gradient side: what averaging does, and how to modify the gradient

Our response to Limitation 2 shows that averaging suppresses surrogate-specific energy by a common factor across all bands, altering only the scale and not the ordering of shared fractions across bands. The averaged gradient remains surrogate-specific, especially at high frequency.

A frequency-weighted correction addresses this directly: FGR rescales the 2D DCT g^\hat{g} of the gradient coefficient-wise by a fixed profile w evaluated at radial frequency d, i.e. it applies IDCT(w⊙g^)IDCT(w ⊙ ĝ). The per-frequency optimal weight grows with the fraction of band energy shared across surrogates and saturates at 1, so any such weight should decrease with frequency, be continuous, and take values in [0,1]; these conditions mean the weight varies monotonically in frequency, treats neighbouring frequencies alike, and never amplifies a band. The last is exact: the DCT is orthogonal, so ⟨g,Mwg⟩=Σu,vw(d(u,v))⋅g^u,v2≥0⟨g, M_w g⟩ = Σ_{u,v} w(d(u,v))·\hat{g}^2_{u,v} ≥ 0 and ‖Mwg‖2≤‖g‖2‖M_w g‖_2 ≤ ‖g‖_2, hence the correction never points into the descent half-space of the raw gradient. Curves meeting the three conditions constitute the family of decay functions FGR uses (Section 3.3).

Empirical results

On 1000 pairs, 15 victims and 7 vendors, versus the strongest baseline M-Attack-V2 we lead by 12.3 points on Claude-Opus-4.6, 10.0 on Claude-Sonnet-4.6, and on all three thinking variants (+7.3, +7.3, +5.6); on open-weight models, five of six (+3.6, +8.0, +9.2, +5.0, +12.7), Gemma-3-27B being 5.2 lower. We lead at all four budgets from 4/255 to 32/255, widening with budget (+0.7, +7.7, +9.0, +11.3). A change of evaluation protocol preserves the ranking: 40.0% versus 37.1%, 30.4% and 29.2% on 300 VQA-style pairs with three untuned victims.

Reviewer_CpoCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FRA-Attack, a transfer-based targeted attack against multimodal large language models. The method has two main components: a DCT-based feature alignment strategy that selects high-frequency components from patch embeddings, and a frequency-domain gradient regularization (FGR) module that suppresses high-frequency gradients during optimization. The authors evaluate the method on 15 MLLMs, including several closed-source models, and report improvements over existing attacks. They also provide ablation studies and additional analyses to investigate the effects of the two components.

Critical strengths

1.Testing on 15 MLLMs from different model families provides useful evidence of transferability. The improvements over M-Attack-V2 on several closed-source models, particularly the Claude models, are notable.

2.The component ablations, cross-surrogate analyses, and additional evaluation with different judges make the experimental section more informative.

Critical weaknesses

1.The main concern is the assumption that high-frequency DCT components of patch embeddings capture the intrinsic visual focus shared across different models. The authors apply a 1D DCT to the flattened patch-token sequence. However, it is not clear why high-frequency variations along this sequence should necessarily correspond to semantically important regions in the original image. The result may also depend on how the patches are ordered. I noticed that Appendix E provides additional evidence through cross-surrogate agreement of reconstructed focus maps. This is useful, but I do not think agreement between feature maps alone is sufficient to establish that these components capture the semantic evidence actually used by the models. I have a similar concern about FGR. The proposed operation is essentially a frequency-dependent low-pass filter applied to input gradients. Since gradient smoothing has already been widely studied in transferable adversarial attacks, I would like to see a clearer distinction between FGR and existing approaches such as Gaussian smoothing or TI-FGSM. Without stronger evidence, it is difficult to tell whether the improvements come from the proposed interpretation of frequency-domain features or from relatively standard filtering operations.

2.The comparison with M-Attack-V2 deserves further attention. The authors disable its target-side retrieval mechanism to maintain a single-target setting. I understand the motivation for this choice, but it also means the comparison does not use the complete configuration of the published method. It would be helpful to report results with both configurations, or at least discuss more explicitly how this choice affects the conclusions. Another concern is the contribution of FGR across different models. In Table 4, removing FGR actually improves the result on Gemini-3-flash. More importantly, Table 11 shows that the average ASR across six open-weight models is almost identical with and without FGR (70.12% versus 70.13%). The authors acknowledge this limitation in the appendix, but I think it deserves more discussion in the main paper, given that FGR is presented as one of the main contributions. Finally, the evaluation relies primarily on caption similarity judged by an LLM. The additional judge experiments are helpful, but caption similarity does not always mean that the original visual semantics have been successfully replaced. The VQA experiment in Appendix F is a useful step toward addressing this issue, although its small scale and the absence of M-Attack-V2 limit what can be concluded from it.

Questions for authors

Q1. Could the authors explain why high-frequency components from a 1D DCT over flattened patch tokens should correspond to semantic visual focus? Have the authors considered comparing this approach with a 2D DCT over the original patch grid, or testing whether the results are sensitive to token ordering?

Q2. How much of the improvement from FGR comes from the specific proposed filter, rather than gradient smoothing in general? A direct comparison with Gaussian smoothing or TI-FGSM under matched settings would help clarify the contribution.

Q3. Could the authors provide results using the original M-Attack-V2 configuration with target-side retrieval enabled? This would make it easier to understand how the proposed method compares with the published baseline, while keeping the different attack settings explicit.

Q4. Table 11 suggests that FGR provides little overall benefit across the six open-weight models and can even reduce performance on some of them. What explains this behavior? Is there a way to determine when FGR is useful without relying on feedback from the victim model?

Q5. Could the authors extend the VQA-based evaluation to include M-Attack-V2 and a larger number of samples? I think this would help establish whether the improvements in caption similarity also translate into more reliable targeted attacks in downstream tasks.

4 comments

Q1: Thank you for this valuable question. To examine how the components selected by our 1D DCT relate to semantic visual focus, we ran further experiments and analyzed the mechanism in more depth. Results ([email protected], 100 pairs):

RuleGPT-5.4Claude-Opus-4.6Gemini-3-flashMacro
No selection (align all)40785156.3
raster1d (row-major; ours)54826968.3
1D column-major54786064
1D random permutation47826163.3
2D wedge (≈ raster cut)40796059.7
2D radial high-pass46745458
2D full-plane ranking41786360.7
2D horizontal band only43775357.7

(1) Mechanism.

In short, our 1D DCT filters out the content shared by all patches — the image-wide mean and slow vertical transitions such as sky–ground gradients, which [CLS] already carries — and keeps each patch's deviation from this shared content, which concentrates on the subject.

To see what each component means spatially, we map its 1D atom back onto the grid. Each row of the grid is 14 consecutive tokens. Atom k therefore goes through k half-cycles from the top row to the bottom row. For k < 10, it changes by less than a third of a cycle along any single row. These low atoms are horizontal stripes: nearly constant within a row and varying only with height. The components we remove (k < θ = 10) therefore make up the row-mean profile of the feature map. F₀ is proportional to the mean of all patches, the same global content that [CLS] carries. The other nine are vertical frequencies 1–9 of the row means. Removing them approximately subtracts from each patch the mean of its row. In natural images, background is largely layered by height (sky, horizon, ground, wall, tabletop), so it is shared along a row and absorbed by the row mean. The subject is local in the horizontal direction and stands out from its row. Our existing results already show quantitatively that the selected components concentrate on the subject and agree across encoders.

(2) Sensitivity to token order.

Token order matters. Row-major order is best (68.3). Column-major (64.0) and a fixed random permutation (63.3) score close to each other. Under row-major order, the removed components are the row means like large uniform regions layered by height (sky, ground, wall). Under column-major order, the removed components are the column means. A column usually crosses several of these layers, so its mean matches no region of the image. Removing it leaves the background's height-wise layering in the residual, where it competes with the subject for the selected components. Under a random order, the removed components are arbitrary mixtures of patches with no spatial meaning. Column-major and random orders therefore keep only the generic gain from selection.

To test our explanation, we rotated all images by 90°. On rotated images, column-major scores 54 and row-major 52.3. After rotation, the background is layered by width rather than height, so the column-major cut is now the one that removes it and leaves the subject. Rotation lowers all scores because the victims recognize rotated targets less reliably. For example, GPT-5.4's descriptions of 45 of the 100 rotated targets drift beyond normal variation, with outright errors such as a mouse described as a barcode scanner. We therefore compare only the gap between orders within each setting.

SettingOrderGPT-5.4Claude-Opus-4.6Gemini-3-flashMacro
Rotated 90°Row-major39704852.3
Rotated 90°Column-major38705454

(3) 2D DCT comparison.

We implemented four 2D variants on the 14×14 grid. All beat the no-selection baseline but fall short of every 1D order. We see two reasons. First, the radial and full-plane rules treat both axes alike, while the scene does not. A subject that fills a compact region of the grid is itself a low-frequency pattern in 2D, so these cuts remove part of the subject together with the background. The raster cut removes only content that is constant along rows. The subject is local in the horizontal direction, so the raster cut largely misses it. Second, the basis of the aligned components matters even when the cut is the same. The 2D wedge removes approximately the same components as the raster cut and still scores only 59.7. A 2D atom is a separable template over the whole grid, so its coefficient records where content sits at that scale, in both directions. Matching it means reproducing the target's spatial layout on the surrogate's grid, which is close to the rigid patch-by-patch match of the baseline. The raster harmonics k = 28m, by contrast, depend only on column sums and ignore where content sits vertically. They constrain what the target contains more than where it sits.

(4) Paper revisions. We will add the rule ablation and the four 2D comparisons to the appendix .

Q2: Thank you for this question. In short, the gain comes from FGR's specific weight, not from smoothing in general: under matched settings, no smoother we tested reproduces it, and most of them hurt.

Different assumptions. TI-FGSM convolves the gradient with a Gaussian to approximate attacking many translated copies of the image. It assumes the target attends to shifted discriminative regions. Gaussian smoothing more broadly assumes gradient noise is spatially fine-grained, so a smoother gradient is a more reliable one. Both are priors about images and models in general. These priors were designed and validated for CNN classifiers, where the attacker and the target share a fixed label space and similar architectural biases. Transfer to an unseen VLM is different: the target couples its own vision encoder with a projector and a language model, and it produces open-ended text rather than a class label. There is little reason to expect a prior tuned for CNN classifiers to carry over. FGR does not rely on one.

FGR starts from a claim about the ensemble: a gradient component transfers to an unseen VLM when the surrogates agree on its direction. We measured this agreement directly in frequency domain. Across our three surrogates, the share of gradient direction they have in common falls steadily with frequency, while the share of energy in the upper band rises (see our reply to 82Gz, W1). FGR's weight is built to follow that agreement. The aim is not a smoother gradient but one that keeps what surrogates share.

We choose frequency domain for two reasons. First, agreement varies monotonically with frequency, so frequency is where the weights should be set. Second, in the DCT basis, a per-frequency weight is applied exactly, so the curve itself is what we design. The agreement view tells us what that curve must look like: decreasing with frequency, non-negative everywhere and sloping down from DC.

We compare FGR with Gaussian smoothing over σ ∈ [0.5, 2.0], the TI-FGSM kernel, and box filters, with the following results.

OperatorGPT-5.4Claude-Opus-4.6Gemini-3-FlashAvg.
No regularization47.5775760.5
FGR (1−d)1.5(1−d)^{1.5}48.5846465.5
Gaussian σ=0.544.4845962.5
Gaussian σ=0.652815663
Gaussian σ=0.747766061
Gaussian σ=0.850815461.7
Gaussian σ=0.945.5805158.8
Gaussian σ=1.046.5745257.5
Gaussian σ=1.539.4653847.5
Gaussian σ=2.032502034
TI-FGSM kernel34511734
Box 3×324.2623540.4
Box 5×51024914.3

First, FGR is the only operator that beats no regularization on all three targets (+1.0 / +7.0 / +7.0, avg. +5.0). Under symmetric boundary handling, a Gaussian kernel is also a fixed per-frequency weight on the gradient, φ(d)=exp(−π2σ2d2)φ(d) = exp(−π^2σ^2d^2), yet no σσ we tested matches FGR. The best one (σ = 0.6) reaches 63.0, and every σ lowers success on at least one target.

Second, the CNN-era operators hurt, and each fails in its own way. Stronger Gaussians and the TI kernel cut the mid band, and transfer falls steadily, from 58.8 at σ = 0.9 to 34.0 at σ = 2.0, with TI also at 34.0. Box kernels have sinc responses that turn negative at higher frequencies, which flips the sign of those components. They score lowest (40.4 and 14.3) even though they remove less energy than the strong Gaussians, and the 5×5 box, whose negative lobe starts lower, does far worse.

Q3:

We fully reproduce the complete experimental configuration of M-Attack-V2. We run this comparison on the 100 source–target pairs and the retrieved auxiliary targets provided in the M-Attack-V2 repository, and enable target-side retrieval with the released defaults (use_retrieval=True, target_num=3, i.e., one fixed target and two retrieved auxiliary targets) and keep every other setting identical to our main comparison:

SettingGPT-5.4Claude-Opus-4.6Claude-Sonnet-4.6Gemini-3-flashClaude-Opus-4.6-thinkingGemini-3-flash-thinkingMean
M-Attack-V2 (single-target)47.5767060786165.4
M-Attack-V2 (retrieval, released config)44.4736856655660.4
FRA-Attack (ours)54.5848666806973.3

The results show that FRA-Attack leads on every victim under both configurations. Notably, enabling retrieval does not improve ASR on this panel. Our explanation is that the auxiliary targets merely provide additional retrieval terms rather than directly strengthening alignment to the primary target. When evaluated under a scoring protocol that credits only the primary-target caption, sharing the perturbation budget with two semantically similar but non-identical auxiliary images dilutes target-specific alignment. At the moderate ASR range covered by our panel, this dilution cost outweighs the stabilization benefit. We will add this table to the appendix in the revision.

Q4:

We thank the reviewer for this comment. The main paper does discuss the effect of FGR: Table 5 reports the full p sweep, which we summarize below. We agree that the link between Table 11 and Table 5 could be made clearer, and in the revision we will add a sentence in the main text pointing from one to the other.

The FGR comparison in Table 11 is read at a fixed p = 1.5. At this fixed value our component does have a negative effect on some models because different models vary in their sensitivity to p. Specifically, Qwen3-VL-8B/32B and Llama-11B-V are adversely affected at this value, whereas Gemma-3-27B is adversely affected regardless of the value of p. These two effects explain the negative entries in Table 11. To see whether this is a problem with FGR itself or just with the choice of p, we refer to the p sweep in Table 5 of the main paper.

In Table 5, five of the six open-weight victims improve at p = 1.0 (Qwen3-8B +3.0, Qwen3-32B +1.0, GLM-4.6V +7.8, Llama-3.2-11B-V +0.4, Kimi-K2.5 +5.0; Gemma-3-27B-it is negative). At that setting, the nine-victim mean is the highest in the table (66.2, against 63.0 without FGR), eight of the nine victims improve (+0.4 to +7.8), and five of them sit exactly at their own optimal exponent (Claude-Opus-4.6, Gemini-3-flash, both Qwen models, and Kimi-K2.5). The mean over the six open-weight columns is 69.8 against 67.7. From this perspective, our FGR does improve transferability.

Choosing the exponent does not require feedback from the victim. The design requirements behind FGR — the weight is determined by frequency alone, varies continuously with frequency, and never changes sign on any band — define a family of admissible instantiations and involve no victim-side information; the sweep then locates a plateau at p ∈ [0.5, 1.5], over which the nine-victim mean stays within 65.8–66.2 (against 63.0 without FGR). No per-victim tuning is therefore needed in practice: taking p = 1.0 captures most of the available gain.

Q5:

We extend the VQAv2 evaluation from 50 to 300 source–target pairs. The victim sees the adversarial image and answers the VQAv2 question attached to the target image. M-Attack-V2 runs with its retrieval disabled, since no images similar to the target are available for the VQA pairs. With the enlarged evaluation and M-Attack-V2 included, our method retains the lead.

MethodGPT-5.4Gemini-3.1-flashClaude-Opus-4.6Mean
Clean source (lower bd.)2595.313.1
Clean target (upper bd.)86.374.28682.2
M-Attack34.922.73029.2
FOA-Attack33.924.932.330.4
M-Attack-V238.332.740.337.1
FRA-Attack (ours)43.1324540

On GPT-5.4's lower bound. The clean-source lower bound in this column is 25.0%, clearly higher than those of the other victims (5.3–9.0%), and we examined it specifically and found the reason: when the asked object is not in the image, GPT-5.4 tends to output a common color phrase rather than stating that the object is not visible, and its floor successes are dominated by common color words ('white' alone accounts for about a third, while 'white' appears in the answer lists of 26% of the samples), so its guesses frequently land on the soft-accuracy pass line. This compresses the usable headroom of all methods on this victim equally and does not affect the between-method comparison.

Dear Reviewer CpoC,

Thank you once again for your time and valuable comments on our submission.

As the discussion phase is approaching its end, we would like to kindly confirm whether we have sufficiently addressed all of your concerns. We have also uploaded the revised manuscript. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know.

We sincerely look forward to your feedback.

Best Regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.