- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the role of spectral constraints in targeted transfer attacks against multimodal large language models. The authors incorporate three spectral shrinkage strategies—Hard Spectral Shrinkage, Uniform Spectral Shrinkage, and Target-Aware Spectral Shrinkage—into an existing surrogate-based attack framework, and evaluate their effects on attack success rate, perceptual distortion, and detectability.
The experimental results show that the improvement in transfer attack success achieved by the complete spectral configurations over the original attack configurations is primarily attributable to changes in the optimization framework rather than the spectral constraints themselves. Under specific experimental conditions, spectral shrinkage can reduce the perceptual distortion of adversarial perturbations and significantly lower the detection rates of a particular detector. The authors further investigate the factors influencing detection performance through amplitude-matched controls, effective-rank controls, and analyses of detector training coverage.
Critical strengths
S1. Relatively Comprehensive Ablation Studies.
The authors compare attack performance with and without spectral constraints under the same optimization framework, thereby distinguishing the effects of the optimization configuration from those of the spectral operations. The experimental results show that the improvement in transfer success rates primarily comes from changes in the optimization configuration rather than from HSS, USS, or TASS themselves. This is a valuable negative result that also helps avoid incorrectly attributing performance improvements to the spectral operations.
S2. Evaluation Beyond Attack Success Rates.
The paper evaluates transfer success rates, LPIPS, RMS, effective rank, and detection rates, providing a more comprehensive understanding of the trade-offs among different evaluation metrics in adversarial attacks.
S3. Meaningful Control Experiments.
The amplitude-matched experiments, effective-rank controls, and analyses of detector training coverage provide useful empirical evidence. In particular, the analysis of detector training coverage demonstrates that low detection rates do not necessarily imply that adversarial attacks are universally undetectable.
Critical weaknesses
W1. Lack of a Clear Connection Between Spectral Constraints and Targeted Transfer Attacks, with Insufficient Motivation
The paper focuses on targeted transfer attacks against MLLMs, but the spectral constraint operations employed do not adequately reflect the specific characteristics of this task.
Specifically, HSS and USS operate solely on the singular values of the perturbation matrix, without explicitly considering the target class or the relationship between surrogate and victim models. Although TASS incorporates gradients of the attack loss computed on surrogate models, it remains a generic gradient-projection-based weighted shrinkage strategy that does not explicitly model transferable attack directions shared across models or target-specific semantic features.
The paper does not first investigate the singular-value spectral characteristics of transferable adversarial perturbations, nor does it establish a relationship between singular directions and target-class semantics, cross-model gradient alignment, or transfer success rates.
W2. The Main Experimental Findings Focus on Detectability, but the Mechanistic Connection Among Spectral Structure, Transferability, and Detectability Remains Unclear
The most significant experimental finding is that spectral shrinkage can substantially reduce the detection rate of a particular detector while maintaining comparable attack success rates.
However, the paper does not provide a sufficient mechanistic explanation for this phenomenon.
Through amplitude-matched experiments, the authors show that simply scaling down existing perturbations cannot reproduce the effects of spectral shrinkage. Similarly, effective-rank control experiments suggest that reducing effective rank alone is insufficient to explain the observed changes in detection rates.
Although these results rule out some simple explanations, several important questions remain unanswered:
- Which singular values or singular directions are associated with cross-model transferability?
- What spatial statistical properties of adversarial perturbations are actually altered by spectral shrinkage?
- Why do these changes affect the detector's responses?
- Do the spectral structures that reduce detectability also help preserve the information necessary for targeted transfer?
W3. Lack of Fair Comparisons with Existing Structured-Constrained Attack Methods, Leaving the Necessity of SVD Unsubstantiated
The spectral shrinkage operations employed in this paper are closely related to established techniques. HSS corresponds to truncated SVD, USS corresponds to standard singular-value soft-thresholding, and TASS is a gradient-weighted thresholding variant.
The main detection experiments compare the spectral configurations against the original Omni-Attack, a no-rule configuration, and an L2-constrained configuration. However, the L2 baseline uses a different parameterization and optimization setup from the spectral configurations, making it difficult to isolate the contribution of spectral constraints.
Furthermore, the amplitude-matched experiments mainly construct control perturbations by scaling down previously optimized unconstrained perturbations, rather than re-optimizing the attacks under smaller energy budgets.
In addition, the DCT-based compaction experiments in the appendix achieve detection performance comparable to singular-value-based compaction, further suggesting that SVD may not be necessary to obtain low detection rates.
Questions for authors
Q1. Research Motivation and Task Specificity
Why does this paper specifically focus on targeted transfer attacks against MLLMs? Beyond adopting an existing surrogate-based attack framework, is there any theoretical or empirical evidence suggesting a particular connection between singular-value spectral structures and targeted transferability?
Q2. Spectral Structure and Transferability
Have the authors investigated how different singular directions affect the targeted attack loss, gradient alignment across surrogate models, and transfer success rates on unseen models? If spectral constraints do not improve transfer success rates, can they at least preserve attack-relevant information shared across models more effectively?
Q3. The Unique Contribution of Spectral Constraints
Can the authors provide comparisons with re-optimized L2-constrained, DCT-based, low-pass, and other structured-constrained attacks under exactly the same optimization framework? If these alternatives achieve similar distortion and detection rates, how should the core contribution of spectral constraints in this paper be understood?
Q4. Detection Mechanism
How exactly does spectral shrinkage affect the residual statistics used by the detector? Are there more direct and informative statistical measures than RMS and effective rank that could explain the differences in detection rates across different constraint methods?
Q5. Joint Evaluation of Generative MLLM Transfer and Detectability
Can the authors report both targeted attack success rates on generative MLLMs and detector true-positive rates using the same attack configurations and image samples, to demonstrate that the main findings hold under a unified experimental setting?
Q6. Generalizability Across Detectors
If different detector architectures are used, or if the detector is trained on attack distributions with varying perturbation magnitudes, does spectral shrinkage still provide a detection advantage over other structured constraint methods?
4 comments
Response to Reviewer Sqyo (Part 2/2)
W3 / Q3. The rules are close to established techniques, the baseline is not in the same framework, and DCT compaction does as well as SVD; if alternatives perform similarly, what is the contribution?
Relation to established techniques. We agree, and Section 2 says so: singular-value thresholding "is exactly our uniform rule", and TASS is its weighted form with weights taken from the attack gradient. The paper's claims rest on the controlled comparison, not on the novelty of the operators.
in the same framework, DCT, and the role of SVD. We re-optimised both constraints inside the same skeleton at the rules' own step, on all 500 pairs. CNN is a second, learned detector trained under the same protocol.
| Configuration (step 0.5, same skeleton) | Top-1 | LPIPS | TPR@5%, published detector | TPR@5%, CNN |
|---|---|---|---|---|
| USS | 0.864 | 0.186 | 0.086 | 0.13 |
| TASS | 0.849 | 0.183 | 0.086 | 0.12 |
| ball 7.33 | 0.875 | 0.175 | 0.094 | 0.33 |
| ball 8.2 | 0.884 | 0.204 | 0.150 | 0.63 |
| DCT low-pass | 0.718 | 0.227 | 0.050 | 0.07 |
| DCT low-pass | 0.806 | 0.238 | 0.054 | 0.07 |
(Intervals and paired differences: Tables A6 to A8 and App. D of the revised paper.)
- ball. Under the published detector the 7.33 ball matches USS in success and in detection, and the 8.2 ball is flagged more often. The learned detector flags the balls at the rules' energy about to times as often as the rules.
- DCT low-pass. It is flagged no more often than the rules, but it loses to points of success at a larger distortion.
- SVD is not necessary. The basis controls of App. D already reach this conclusion, and the new controls confirm it.
How to understand the contribution. It is the attribution: success gains belong to the optimisation skeleton (Table 2), distortion and detection move with the constraint (Tables 3 and 4), and the detection rate depends on the detector's training coverage (Table A5). The revised paper states the detection finding for energy-reducing structured constraints in general, not for the singular decomposition.
Q5. Can generative-victim success and detector TPR be reported on the same configurations and images?
We now report it (Table A11 of the revised paper), on the 200 images sent to the generative victims. Each victim cell gives targeted ASR (%) / Joint, where Joint is the fraction of the 200 images that name the target and are not flagged at 5% FPR.
| Configuration | TPR@5% | Qwen2.5 | Qwen3.8 | DeepSeek | GLM-5.3 | Gemini-3.8 | GPT-5.6 |
|---|---|---|---|---|---|---|---|
| Omni-Attack (unconstrained) | 0.985 | 36.5 / 0.000 | 43.5 / 0.005 | 40.0 / 0.000 | 35.0 / 0.000 | 32.0 / 0.000 | 26.5 / 0.000 |
| No rule (step 2.5) | 0.915 | 41.5 / 0.025 | 42.0 / 0.025 | 44.5 / 0.030 | 41.5 / 0.030 | 35.0 / 0.015 | 31.0 / 0.020 |
| HSS (step 2.5) | 0.285 | 37.0 / 0.300 | 47.5 / 0.335 | 45.0 / 0.335 | 40.0 / 0.300 | 39.5 / 0.295 | 29.0 / 0.205 |
| USS (step 2.5) | 0.420 | 42.0 / 0.250 | 45.0 / 0.250 | 41.5 / 0.245 | 39.0 / 0.260 | 37.0 / 0.215 | 27.5 / 0.150 |
| TASS (step 2.5) | 0.535 | 42.5 / 0.200 | 46.5 / 0.195 | 47.5 / 0.225 | 44.0 / 0.220 | 35.0 / 0.165 | 31.0 / 0.145 |
| TASS (step 0.5) | 0.100 | 32.0 / 0.290 | 43.0 / 0.400 | 41.0 / 0.385 | 35.0 / 0.315 | not sent | not sent |
ASR is similar across configurations; Joint differs because of detection. Every rule's Joint rate exceeds that of the unconstrained attack on every victim, with paired intervals excluding zero. The revised Section 5.6 reports this in place of the submitted statement that no setting was measured as both effective and unflagged on the same victims.
Q6. Does the detection advantage survive other detector architectures, or a detector trained on other perturbation magnitudes?
- Magnitudes. Table A5 trains the detector, at the same budget of attacked images, on the known attack at reduced amplitude. Adding it at quarter amplitude raises detection of the compacted configuration from to at a nominal 10% FPR, at almost no cost on the original attack. Detection is recovered, and the revised Section 5.6 states this explicitly.
- Architectures. A learned residual CNN trained under the same zero-day protocol (Table A6) reproduces the low rates of the soft rules ( to ) and the ordering among constraints. Both detectors read the luminance plane (Section 5.1), and the claim is made for that input.
- Against other structured constraints. Under the learned detector the rules are flagged less often than the same-skeleton ball. DCT low-pass is flagged slightly less often than the rules but loses to points of success (W3 / Q3).
Thank you for the detailed rebuttal and substantial additional experiments. I appreciate the authors' efforts and consider most of my concerns regarding experimental completeness to have been adequately addressed.
However, my core concern regarding scientific novelty remains unresolved. The new analyses do not establish a convincing, task-specific connection between SVD and targeted transferability. Similar observations hold for unconstrained attacks, and other structured constraints achieve comparable results. It therefore remains unclear what new scientific insight SVD provides in this setting.
Overall, while the experimental evidence is now comprehensive, my fundamental concerns about motivation, mechanistic understanding, and scientific contribution remain. This is why I am willing to raise my rating from 1 to 2 in recognition of the substantial improvements, but not to 3, as the central scientific concerns remain unresolved.
Response to Reviewer Sqyo
We thank Reviewer Sqyo for the follow-up, for raising the rating, and for recognising the added experiments.
In short, what SVD provides is not more transfer, but a way to make a transferable perturbation cheaper and harder to detect, together with an account of why this costs no success. The gain in success that accompanies such constraints belongs to the optimisation skeleton (Table 2).
We answer the three remaining points in turn, with two further analyses under the first.
1. Connection between SVD and targeted transfer. Beyond the component and gradient analyses of the revision, we ran two further analyses on the 50 scored images that ask what the leading singular directions encode. They are image-specific, not target-specific.
We re-attacked each image toward another image's target and compared the leading-k singular subspaces of pairs of perturbations (1 = identical):
| Pair of perturbations | k=8 | k=16 | k=64 |
|---|---|---|---|
| Same image, different target | 0.167 | 0.229 | 0.375 |
| Different image, same target | 0.139 | 0.209 | 0.364 |
| Different image, different target | 0.140 | 0.210 | 0.364 |
| Chance | 0.027 | 0.054 | 0.214 |
Sharing the target adds nothing across images (−0.001 [−0.006, +0.003] at k=8), and a perturbation added to a different clean image reaches its target on 0 of 50 images.
The connection is therefore specific. Transfer is carried by a few leading components of each perturbation: removing the top 8 lowers success from 93 to 34 points. That is why the spectrum can be shrunk without losing it (Table A9 of the revised paper).
2. Unconstrained attacks. They show the same leading structure, and this is what makes it a property of targeted transfer and not of our rules. The difference is in the trailing components. Without its top 64, the no-rule perturbation has no success and is still flagged on 0.56 (Table A9). Shrinkage removes that part, and the flagged rate falls from 0.94 to 0.12 at the same success (Table 3).
3. Other structured constraints. Under the published detector an energy ball is comparable, and the revision says so. The constraints are not equivalent otherwise:
- a DCT low-pass loses 6 to 15 points of success at the same low rate (Table A8 of the revised paper);
- a learned detector flags the balls at the rules' energy on 0.33 to 0.63, against 0.12 to 0.13 for the soft rules, on 500 pairs (Tables A6 and A7 of the revised paper).
We will add the two analyses to App. D. We hope this makes the role of SVD clear.
Response to Reviewer Sqyo (Part 1/2)
We thank Reviewer Sqyo for the thoughtful comments. Our point-by-point responses follow.
Generic rules, and why this setting. The rules are generic, and this is by design. The paper uses them to measure what a spectral constraint changes in this task once its effect is separated from the optimisation configuration and from the detector; a rule tailored to the task would not answer that question. Targeted transfer to MLLMs is a setting where that question is open: the strongest attacks spend the full ℓ∞ budget, the attacker cannot query the victim, so input screening matters, and prior work on structured perturbations reports success and low distortion jointly.
Spectral structure and transferability. The paper answers this at two levels.
The submitted Section 5.5 left this open. We ran two new analyses.
Component ablation. For each of the 50 scored images we rebuild the final perturbation from its top k singular components only (keep k), or from all the others (drop k), and read success and detection again.
(All k and all four configurations: Table A9 of the revised paper.)
What the detector reads. Across the 57 configurations with a zero-day rate, the RMS of the perturbation after the detector's own residual filters tracks TPR@5% with Spearman ρ=0.945, against 0.865 for plain RMS and 0.670 for effective rank. The detector's rate therefore follows high-frequency residual energy. Shrinkage lowers that energy by removing trailing components and keeps the leading ones that carry success. The rules thus preserve the shared attack information at lower cost, not more of it: the same success at about a third less LPIPS (Table 3).
This is a descriptive account of final images, not a causal test of the optimisation. The revised Section 5.5 and App. D state it this way.