acceptodds
Under review as a conference paper at ICLR 2027

Beyond First-Order Feature Alignment: Second-Order Subspace Alignment for Adversarial Attacks on Closed-Source MLLMs

Abstract

Multimodal large language models (MLLMs) are vulnerable to targeted adversarial attacks transferred from surrogate encoders to closed-source models. Recent feature-alignment based attacks exploit transformed views and fine-grained local representations, yet remain centered on first-order feature correspondence. We observe that subspace modeling offers a natural structural perspective for capturing second-order moment structure across related representations. Accordingly, we propose the Second-Order Subspace Alignment Attack (SOSA-Attack), which aligns augmentation-induced subspace descriptors derived from second-order moments at both global and patch levels and progressively fuses the resulting subspace alignment gradients with first-order feature alignment updates. Theoretically, we characterize SOSA as second-order distribution matching, proving an exact equivalence between second-order moment discrepancy and squared polynomial-kernel MMD, with corresponding bounds for the MaxExp subspace discrepancy. Experiments across diverse MLLMs and downstream tasks show that SOSA-Attack consistently outperforms prior targeted transfer attacks and achieves state-of-the-art performance.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Large Language Models, Adversarial Attacks, Robustness, Subspace Alignment, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MTyROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes SOSA-Attack, a targeted transferable attack against closed source MLLMs. Building on FOA-Attack, it constructs global and local representation sets with random image transformations. It computes uncentered moments of order two after scaling by the trace, then obtains subspace descriptors with a MaxExp spectral map. SOSA-Attack aligns structure of order two at both the image and patch levels between adversarial and target images. It also gradually combines the gradient from this alignment with the original gradient for first order feature alignment. The theory proves that, for empirical distributions induced by feature vectors scaled by the trace, the squared Frobenius norm of the difference between moments of order two is exactly equal to MMD with a polynomial kernel of degree two. It also gives upper and lower bounds relating the MaxExp subspace difference to this MMD. The experiments cover image captioning, VQA, and image classification. They test several closed source models, open source models, and MLLMs with enhanced reasoning.

Critical strengths
  1. The paper combines descriptors of global and local representations of order two with the existing objective for first order feature alignment. Norm balancing and gradual gradient fusion give the method one optimization procedure. The design is fairly complete.

  2. The paper evaluates image captioning, VQA, and image classification. It covers closed source models, open source models, and models with enhanced reasoning. The empirical analysis of cross model vulnerabilities in MLLMs has some value.

Critical weaknesses
  1. Appendix D.3 says that each sample is first attacked with Kc=3K_c=3, and the victim model is queried. If the attack fails, the authors run it again with Kc=5K_c=5 and keep the better result from the two runs. This creates an adaptive choice between two attempts based on feedback from the victim model. It may depart from a transfer attack that relies only on a surrogate model, and it can raise the final ASR. The paper does not report the actual average number of queries per sample or the total query budget. It also does not state whether all baselines use exactly the same query and optimization procedure. If this procedure is used for attack generation and result selection, the method should be explicitly described as a query-assisted adaptive black-box attack rather than a purely surrogate-model-based transfer attack.

  2. The paper does not report results for clean images, repeated experiments, confidence intervals, or tests of statistical significance. It also does not evaluate all baselines on the same complete set of 1000 samples. The theory establishes only that the difference between moments of order two has an exact relation to MMD with a polynomial kernel of degree two. This result does not guarantee transferability to a black box model. Although w/o MaxExp and FOA-Attack offer partial comparisons, the paper does not include comparisons under the same budget and implementation framework for raw moments of order two, CORAL, or a polynomial kernel of degree two. It also does not directly compare using only the second order branch of SOSA-Attack with using only the first order branch of FOA-Attack. If the procedure from Kc=3K_c=3 to Kc=5K_c=5, or the extra random views, is not matched for every baseline, the reported gains may partly come from extra attack attempts or extra computation. The experiments then do not establish that the gain comes from SOSA-Attack’s second order structure.

Questions for authors
  1. Please state how many queries to the victim model each sample actually requires in Appendix D.3. State whether the decision to move from Kc=3K_c=3 to Kc=5K_c=5 uses the victim model’s actual output. Report separate results for fixed Kc=3K_c=3, fixed Kc=5K_c=5, and the adaptive setting. Also state whether all baselines use the same number of queries, optimization iterations, and computational budget. If feedback from the victim model enters the attack or the choice of the final result, please define the setting as an adaptive black box attack that uses queries.

  2. The paper says that some earlier baselines are evaluated only on the first 100 samples, while recent baselines use all 1000 samples. Please evaluate every method on the same complete set of 1000 samples. Report results on clean images, repeated runs with multiple random seeds, confidence intervals, and tests of statistical significance. For GPT-4o scores, state the exact model version, temperature, decoding settings, control of randomness, and score calibration procedure.

  3. The main text explicitly says that the method uses uncentered moments of order two. Please confirm that the implementation follows this definition. State whether any extra feature normalization is applied before constructing the moments and give the exact value of ν\nu. Although w/o MaxExp and FOA-Attack offer partial comparisons, please also report comparisons under the same budget and implementation framework for raw moments of order two, CORAL, and a polynomial kernel of degree two. Include a direct comparison between using only the second order branch of SOSA-Attack and using only the first order branch of FOA-Attack. Please report the numbers of forward and backward passes, GPU memory usage, runtime, and API query cost for SOSA-Attack and each baseline.

2 comments
AuthorsOpenReview ↗

Thank you for your valuable review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1 / Q1. Victim feedback and query budget.

Thank you for asking where victim feedback enters the evaluation. SOSA-attack generate candidates using three CLIP surrogates with zero victim queries. At fixed Kc=3K_c=3 or 55, each method submits one candidate per image to each victim. Appendix D.3 uses the same retry rule for both methods. We score the victim's actual output on the Kc=3K_c=3 candidate and submit the precomputed Kc=5K_c=5 candidate only if the first attempt fails. For NN images and FF first-attempt failures, each victim receives N+FN+F candidate queries, averaging 1+F/N1+F/N per image. These counts exclude reference and judge calls.

The table reports mean ASR (%) and queries per image across six non-thinking models on 1,000 pairs, with FOA listed first and SOSA second in each cell. With one query per image at fixed Kc=3K_c=3, SOSA exceeded FOA by 15.1 percentage points on captioning, 5.2 on VQA and 11.9 on classification. Allowing a retry increased ASR for both methods. Because SOSA succeeded more often on the first submission, it required fewer retries on all three tasks. We also evaluated fixed Kc=5K_c=5 with one query per image on a separate 100-pair captioning comparison. FOA / SOSA achieved 42% / 53% ASR on GPT-5.4 and 58% / 74% on Claude Opus 4.6.

TaskFixed Kc=3K_c=3 ASRAdaptive ASRAdaptive queries/image
Captioning40.6 / 55.752.6 / 68.51.59 / 1.44
VQA38.7 / 43.947.9 / 54.01.61 / 1.56
Classification62.3 / 74.269.9 / 81.41.38 / 1.26

W2 / Q2. Baseline coverage, uncertainty and caption scoring.

Thank you for asking why sample sizes differ. Using the first 100 source–target pairs gave all four earlier baselines identical images and targets. In captioning, AttackVLM, AdvDiffVLM and AnyAttack achieved at most 1% ASR on each of the six closed-source models, and SSA-CWA achieved 1–12% (Table 1). Given these results, we expected the gap to recent methods to persist and stopped those runs at 100 pairs. We evaluated SOSA and its closest baselines (FOA, M-Attack and M-Attack-V2) on all 1,000 pairs.

For clean source images, we used the same caption prompt, target references and scoring threshold. Clean success was 0.1% on Gemini 2.5 Flash and Claude Sonnet 4.6 and 0.0% on the other four models. Mean similarity ranged from 0.009 to 0.014. We used the gpt-4o-2024-11-20 with the Appendix D.1 scoring prompt, temperature 0 and max_tokens=100. We adopted M-Attack's threshold (score ≥0.5\geq 0.5) without separate human calibration.

W2 / Q3. Second moments, ablations and computational cost.

Thank you for asking what the structural loss measures and how the two branches compare. We apply row-wise ℓ2\ell_2 normalization, compute M=F⊤F/nM=F^\top F/n without centering, and divide by tr(M)+ν\mathrm{tr}(M)+\nu, with ν=10−5\nu=10^{-5}. For the trace-scaled moments A,BA,B, Eq. (14) uses the identity of Li et al. [1].

D=∥A−B∥F2=MMDk2,k(u,v)=(u⊤v)2.D=\lVert A-B\rVert_F^2=\mathrm{MMD}_k^2,\quad k(u,v)=(u^\top v)^2.

We prove the MaxExp bound in Eq. (15).

LS=∥Ψρ(A)−Ψρ(B)∥F2,γ2D≤LS≤ρ2D.L_S=\lVert\Psi_\rho(A)-\Psi_\rho(B)\rVert_F^2,\quad \gamma^2D\leq L_S\leq\rho^2D.

Eq. (15) gives D≤LS/γ2D\leq L_S/\gamma^2 on the surrogate. We will revise Sec. 4.3 to clarify this implication. We evaluate transfer empirically through victim ASR.

All ablations used the first 100 captioning pairs, fixed Kc=3K_c=3, 300 steps and one submission per image per victim. The three structural controls retained the first-order term. After increasing the view count of FOA to 20, both ASR metrics improved. At 20 views, full SOSA exceeded FOA by a further 10 and 9 points.

GPT-5.4Claude Opus 4.6
MethodASR (%)AvgSimASR (%)AvgSim
FOA400.372660.506
FOA, 20 views490.412700.534
FOA + raw moment500.421650.520
FOA + CORAL400.382730.567
FOA + kernel (u⊤v+1)2(u^\top v+1)^2540.449680.538
Full SOSA590.489790.612

In separate timing runs on 100 captioning pairs, SOSA took 1.40–1.48 times as long as FOA at fixed KcK_c. Peak GPU memory was 7.79 GiB for FOA, 17.87 for 20-view FOA, 18.32 for the kernel control and 17.90 for the other variants. Feature inputs / gradient calls totaled 270,000 / 30,000 for FOA, 1,806,000 / 90,000 for second-order only and 1,476,000 / 90,000 for each remaining variant. These count individual image inputs and torch.autograd.grad calls. Batched forward-call totals and comparable compute records for earlier baselines are unavailable.

MethodKcK_cGPT-5.4 (min)Claude Opus 4.6 (min)
FOA3281278
FOA5287296
FOA3→5394375
SOSA3402412
SOSA5419415
SOSA3→5573526

References

[1] Y. Li, N. Wang, J. Liu and X. Hou. Demystifying Neural Style Transfer. IJCAI 2017, pp. 2230–2236.

Thank you for the response and the added experiments. The fixed Kc=3 results and the new structural ablations addressed part of my soundness concerns, so I raised soundness, but I am keeping my rating. In the main tables, you still need to separate single query transfer from the adaptive Kc=3 to Kc=5 retry that uses victim feedback. The rebuttal also leaves out repeated runs with significance tests and omits the second order only ablation row from the table, while SOSA adds extra memory and runtime cost over FOA.

Reviewer_vvzKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a second-order subspace alignment method for targeted attack, which aligns the hidden states of the surrogate model using second-order metrics. It computes the hidden states of the source image and the target image. Then the authors compute both global and patch-level alignment terms. With progressive gradient fusion, the method emphasizes first-order alignment in the beginning and then progressively moves to second-order alignment after a period of optimization. The authors also provide a theoretical analysis showing the connection between the proposed method and a second-order distribution matching method with a square polynomial kernel MMD. The experimental results show consistently better performance compared to other attack methods.

Critical strengths
  1. This paper introduces second-order alignment methods for multimodal large language model attacks, which have sufficient motivation and sound theoretical analysis.

  2. The experiment section includes attacks on different mainstream commercial model series (including GPT, Gemini, and Claude) and open-source series like Qwen, Gemma, and LLaVA. The experiment also includes a wide range of ablation studies about the hyperparameters.

Critical weaknesses
  1. The proposed method’s reported performance is effectively a `pass@2', achieved by testing multiple hyperparameter settings (cluster centers of 3 and 5) on the victim model. This comparison is unfair, as it tunes hyperparameters directly on the victim model, which means that the test protocol is a few-shot query adversarial attack for SOSA-Attack. There are no results for a single cluster center at 3 or a cluster center at 5 case in the main table.

    Though FOA Attack also provides such a `pass@2', it also provides a pass@1 by setting the cluster center to be fixed at 3. The authors are encouraged to use the same setting, e.g., a cluster center at 3 or 5, for a fair comparison. The pass@2 can be reported as scaling. However, the authors are also recommended to report the time consumption, following the FOA Attack.

  2. In the contribution, the author lists the proof that the second-order discrepancy is equivalent to a squared polynomial kernel MMD. However, this is actually already proved as a conclusion in Demystifying Neural Style Transfer [1].

    The author is recommended to cite this literature and instead restate their contribution as providing the lower and upper bounds brought by the Max-Exp.

[1] Li, Yanghao, et al. "Demystifying neural style transfer." arXiv preprint arXiv:1701.01036 (2017).

Questions for authors
  1. Can the authors provide the exact reasoning effort and other settings of the commercial APIs, including exact versions, temperature, thinking budget, and reasoning tokens? This would help the paper's reproducibility.

If the authors address all the weaknesses and the questions, I will consider raising my score.

5 comments
AuthorsOpenReview ↗

Thank you for your review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1: Fixed-cluster evaluation and runtime.

Thank you for highlighting the role of victim feedback in the comparison. As you point out, the FOA and SOSA entries in Table 1 of the submitted manuscript allow up to two submissions per image pair. Both methods optimize candidates on surrogate encoders. Under Appendix D.3, the score of the victim's caption at Kc=3K_c=3 determines whether a pair is retried at Kc=5K_c=5. We retain the higher score, so the reported ASR includes pairs that succeed on the second attempt. Across all 1,000 captioning pairs, allowing this retry raised SOSA's ASR by 14.5 percentage points on GPT-5.4 and 10.1 points on Claude Opus 4.6.

Following your suggestion, we compared the methods with one submission per pair at each fixed cluster count. For Kc=5K_c=5, we separately evaluated every one of the first 100 pairs, including those that had already succeeded at Kc=3K_c=3. As the table shows, SOSA also achieved higher single-submission ASR than FOA on both models at this setting. In separate timing runs on these 100 pairs, SOSA took 40–48% longer than FOA to generate candidates across the four fixed comparisons. In the revision, we will report single-submission ASR separately from ASR with an adaptive retry.

GPT-5.4Claude-Opus-4.6
MethodTimesASRAvgSimTimesASRAvgSim
FOA-attack ([3])-Image captioning281 mins42%0.369278 mins65%0.526
FOA-attack ([5])-Image captioning287 mins42%0.337296 mins58%0.492
FOA-attack ([3,5])-Image captioning394 mins56%0.428375 mins72%0.563
SOSA-Attack ([3])-Image captioning402 mins50%0.449412 mins74%0.607
SOSA-Attack ([5])-Image captioning419 mins53%0.460415 mins74%0.575
SOSA-Attack ([3,5])-Image captioning573 mins70%0.524526 mins84%0.643

W2. Attribution of Eq. (14) and the MaxExp bounds

We thank the reviewer for pointing out that Eq. (14) follows from the result of Li et al. [1] and for suggesting that we emphasize the MaxExp bounds. Applied to our trace-scaled empirical feature distributions, their result identifies the squared MMD for the kernel k(u,v)=(u⊤v)2k(u,v)=(u^\top v)^2 with the squared Frobenius distance between the uncentered second moments. In the revision we will cite Li et al. at Eq. (14) and in its derivation in Appendix B. We will revise the abstract, introduction, and third contribution to identify the two-sided bounds in Eqs. (15) and (17) as our theoretical contribution.

For a fixed surrogate encoder, consider either its global or its patch features. Let AA and BB denote the trace-scaled second moments of the adversarial and target features, respectively. Their squared distance before MaxExp is D=∥A−B∥F2D=\lVert A-B\rVert_F^2. The corresponding structural term optimized in Eq. (8) is

LS=∥Ψρ(A)−Ψρ(B)∥F2,Ψρ(A)=I−(I−A)ρ.L_S=\lVert\Psi_\rho(A)-\Psi_\rho(B)\rVert_F^2,\qquad \Psi_\rho(A)=I-(I-A)^\rho.

Trace scaling with ν>0\nu > 0 ensures that c=max⁡(∥A∥2,∥B∥2)<1c=\max(\lVert A\rVert_2,\lVert B\rVert_2) < 1. For integer ρ≥1\rho\ge1, Eq. (15) gives

γ2D≤LS≤ρ2D,γ=ρ(1−c)ρ−1>0.\gamma^2D\le L_S\le\rho^2D,\qquad \gamma=\rho(1-c)^{\rho-1} > 0.

The lower bound gives D≤LS/γ2D\le L_S/\gamma^2. Thus, on this spectral range, achieving LS≤εL_S\le\varepsilon guarantees that the squared distance between the underlying second moments is at most ε/γ2\varepsilon/\gamma^2. The upper bound limits the amplification of that squared distance to a factor of ρ2\rho^2. Eq. (17) extends both bounds to the weighted sum of the global and patch terms in Eq. (8), with lower-bound coefficient min⁡(γg,γp)2\min(\gamma_g,\gamma_p)^2.

Remark. For fixed ρ>1\rho > 1, γ\gamma approaches zero as cc approaches one, so the bound on DD becomes weaker. For every c<1c < 1, MaxExp preserves exact second-moment alignment: LS=0L_S=0 if and only if A=BA=B. When ρ=1\rho=1, both bounds hold with equality and LS=DL_S=D.

[1] Li et al., Demystifying Neural Style Transfer, IJCAI 2017.

Q1: Model versions and inference settings.

Thank you for asking about reproducibility. All victim models used temperature 1.0. Reasoning effort/budget were None/None for gpt-5.2-2025-12-11, gpt-5.4-2026-03-05, Claude-sonnet-4.6 (2026-02-17), Claude-opus-4.6 (2026-02-05), and Gemini-3-flash-preview (20251217). The same GPT-5.4, Claude Opus 4.6, and Gemini 3 Flash versions also ran with High effort. These runs and Gemini-2.5-flash (reasoning On) had a 3,200-token reasoning cap per response. The judge (gpt-4o-2024-11-20) used temperature 0, no reasoning mode or budget, and output limits of 100 tokens for captioning and 20 for VQA.

Thank you for the response. I appreciate the clarification on how the adaptive retry affected the reported success rates. The fixed-cluster results and runtime measurements address my main concern, and the API settings answer my reproducibility question.

Please update the revised manuscript to distinguish single-submission results from results with an adaptive retry. I would also appreciate the proposed revision to the theoretical contribution statement, attributing the equivalence in Eq. (14) to Li et al. and emphasizing the MaxExp bounds.

I will keep my current rating for now and remain open to raising it after reviewing the updated manuscript.

AuthorsOpenReview ↗

Rebutta follow up

Dear Reviewer vvzK,

Thank you very much for your careful review and constructive suggestions. We have now uploaded the revised manuscript. Your feedback has been valuable in guiding our revisions and improving the paper. We sincerely appreciate the time and care you have devoted to reviewing our work.

Best, Authors

Reviewer_JQ5iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SOSA-Attack employs second-order subspace alignment as a targeted transfer objective, extending alignment from individual feature correspondence to set-level structure. It constructs global- and patch-level descriptors from stochastic image views and progressively fuses second-order gradients with first-order feature alignment updates. Experiments on image captioning, VQA, and image classification across closed-source, open-source, and reasoning-enhanced MLLMs show that the method consistently outperforms existing targeted transfer attacks.

Critical strengths
  1. The proposed method was evaluated on multiple MLLMs, including closed-source models, open-source models, and reasoning-enhanced models. Experimental results demonstrate that SOSA-Attack consistently outperforms existing methods across these various models and tasks.
  2. Extending feature alignment from first-order feature alignment to second-order structural matching offers a valuable perspective.
  3. The study includes component ablations studies and hyperparameter sensitivity analysis.
Critical weaknesses
  1. A key ablation study is missing. While Table 4 shows that the complete SOSA-Attack outperforms the original FOA-Attack, it does not sufficiently demonstrate that the performance gain can be attributed specifically to the proposed second-order structural alignment mechanism. It may instead result from the use of more randomly augmented views, rather than the second-order alignment objective itself.
  2. Theorem 1 in Section 4.3 appears to be largely an application of the known properties of kernel methods. The proposed method seems to be an incremental extension of existing methods such as FOA-Attack and M-Attack, and the novelty of the approach requires further clarification.
  3. Regarding the design of "Progressive Gradient Fusion," why is dynamic weighting used for the first-order component, whereas the second-order component uses uniform averaging of the surrogate gradients?
  4. After the warm-up stage, in each optimization iteration, SOSA requires the processing of K=20 stochastic views, as well as the construction of both global and patch-level second-order moments. However, the paper does not provide a quantitative comparison of computational cost with existing methods such as FOA-Attack.
  5. Some of the experimental analysis is redundant, for instance, the paragraphs before and after Table 10 in the appendix repeat the same performance improvements and conclusions.
Questions for authors

Please refer to Points 1, 3, and 4 under Critical Weaknesses.

1 comment
AuthorsOpenReview ↗

Thank you for your review and suggestions. Below we respond to the comments in Weaknesses (W) and Questions (Q).

W1. Contribution of additional views and second-order alignment.

Thank you for asking whether additional stochastic views explain the improvement over FOA-Attack in Table 4. We tested this by evaluating FOA with 20 views while retaining its first-order objective. On image captioning, the additional views raised FOA's ASR by 9 percentage points on GPT-5.4 and 4 points on Claude Opus 4.6. With both methods using 20 views, full SOSA-attack exceeded FOA-attack by a further 10 and 9 percentage points, respectively, with higher AvgSim on both models. Thus, additional views account for part of the observed gain, and the advantage of the full SOSA-attack formulation persists at the same view count. We will report this result in the updated manuscript.

GPT-5.4Claude-Opus-4.6
MethodASRAvgSimASRAvgSim
FOA-Attack40%0.37266%0.506
FOA-Attack, 20 views49%0.41270%0.534
Raw second moment50%0.42165%0.520
CORAL40%0.38273%0.567
Polynomial kernel54%0.44968%0.538
Full SOSA-Attack59%0.48979%0.612

W2. Methodological contribution and interpretation of Theorem 1.

Thank you for asking how SOSA extends M-Attack and FOA. M-Attack matches features from transformed views, and FOA extends this matching to local feature correspondences. SOSA adds a set-level objective that compares the second moments of global and patch features across stochastic views. These moments capture pairwise products of feature coordinates averaged across the sampled representations.

We appreciate your observation that Eq. (14) follows from known kernel properties. Li et al. [1] established the identity between squared MMD under k(u,v)=(u⊤v)2k(u,v)=(u^\top v)^2 and the squared Frobenius distance between uncentered second moments. Our analysis relates this identity to the MaxExp loss LSL_S used in SOSA. Let A,BA,B be the trace-scaled second moments, D=∥A−B∥F2D=\lVert A-B\rVert_F^2, and c=max⁡(∥A∥2,∥B∥2)<1c=\max(\lVert A\rVert_2,\lVert B\rVert_2) < 1. For integer ρ≥1\rho\geq 1, Eq. (15) gives the bounds below. The lower bound implies that achieving LS≤ϵL_S\leq\epsilon bounds the remaining second-moment discrepancy by ϵ/γ2\epsilon/\gamma^2. Eq. (17) extends the bounds to the weighted global and patch terms. We will cite Li et al. at Eq. (14) and emphasize the MaxExp bounds in the abstract, introduction, and Section 4.3.

γ2D≤LS≤ρ2D,γ=ρ(1−c)ρ−1>0.\gamma^2 D \leq L_S \leq \rho^2 D,\qquad \gamma=\rho(1-c)^{\rho-1} > 0.

[1] Li et al., Demystifying Neural Style Transfer, IJCAI 2017.

W3. Weighting across surrogate encoders.

Thank you for asking why the two branches weight the surrogate encoders differently. In the first-order branch, we retain FOA’s dynamic weighting because each weight reflects the relative change in that encoder’s first-order alignment loss across iterations (Appendix C). Applying those same weights to the second-order branch would make an encoder’s contribution to structural alignment depend on its progress in first-order feature matching. We therefore give each encoder’s structural loss the same coefficient, which yields the average of the surrogate gradients in Eq. (9). The w/o gradient fusion variant in Table 4 retains both directions and applies them jointly from the first iteration. The full method begins with first-order alignment and then progressively introduces the structural gradient. On the same 100 captioning pairs, this change raised ASR from 49% to 55% on Gemini 3 Flash and from 58% to 69% on Claude Sonnet 4.6. Thus, progressive fusion improved ASR over direct joint updates by 6 and 11 percentage points, respectively.

W4. Computational cost relative to FOA.

Thank you for asking us to quantify the cost of the 20 views and the global and patch-level moment computations. We measured candidate generation time in separate runs on 100 captioning pairs with the cluster count fixed at Kc=3K_c=3 or Kc=5K_c=5. These measurements exclude victim API calls. Across the four fixed comparisons, full SOSA took 40–48% longer than original FOA to generate candidates.

Thank you for asking about the computational cost of SOSA. We measured candidate generation time on the same 100 image-captioning pairs, excluding victim API calls. At fixed Kc=3K_c=3 or Kc=5K_c=5, SOSA took 40–48% longer than FOA across the four comparisons. The [3,5] rows report the adaptive protocol separately.

MethodKcK_cGPT-5.4 (min)Claude Opus 4.6 (min)
FOA3281278
FOA5287296
FOA[3,5]394375
SOSA3402412
SOSA5419415
SOSA[3,5]573526

W5. Repetition around Table 10.

Thank you for pointing out the repetition in our discussion of Table 10, which we will address in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.