- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AM-GARD, an activation-steering-based defense method for recovering pre-attack responses in vision-language models (VLMs) against adversarial image perturbations. By investigating where interventions should be applied and which directions are effective for answer recovery, they observe that attack-detection directions are weakly aligned with answer-margin directions and that the latter achieve substantially better recovery under controlled interventions. Motivated by these findings, AM-GARD decouples intervention selection from answer correction using a learned intervention gate and input-specific answer-margin updates. The method employs local linearization to identify nearby answer decision boundaries and applies the resulting correction through activation steering without updating the backbone parameters. Experiments on Qwen2-VL-7B and LLaVA-1.5-7B across ten datasets demonstrate improvements in pre-attack answer recovery while largely preserving responses to clean inputs. Overall, the paper presents an interesting investigation of adversarial answer recovery from the perspective of internal representation geometry. The distinction between attack detection and answer correction is well motivated, and the controlled experiments provide useful evidence supporting this separation. However, several aspects of the theoretical motivation and experimental evaluation require further clarification, particularly regarding the relationship between local decision boundaries and the recovery of original answers.
Critical strengths
- Insightful motivation for decoupling detection and correction.The paper identifies an important distinction between detecting adversarial perturbations and recovering original responses. This observation motivates separating intervention selection from answer correction, providing a meaningful perspective on adversarial defense in VLMs.
- Effective answer recovery with minimal disruption to clean responses. The proposed method improves the recovery of pre-attack answers while largely preserving the model's original behavior on clean inputs. By combining selective intervention with input-specific answer-margin correction, it achieves a balance between adversarial recovery and clean-response preservation without modifying the backbone parameters.
Critical weaknesses
- Insufficient justification for nearest-boundary selection. The theoretical analysis only establishes the minimum displacement required to cross a local decision boundary, without explaining its relationship to the pre-attack answer. Consequently, it remains unclear whether selecting the nearest boundary systematically facilitates answer recovery or merely produces coincidental corrections.
- Limited generalizability of the proposed correction mechanism. The method primarily relies on first-token answer margins, which may not adequately capture the complexity of multi-token generation. Furthermore, its effectiveness under different adversarial objectives and attack distributions remains insufficiently established, raising concerns about its applicability to broader VLM defense scenarios.
Questions for authors
- Question 1: Justification of nearest-boundary selection: The proposed method assumes that the nearest local answer boundary is a meaningful correction target, but the relationship between the selected competitor and the original pre-attack answer remains unclear. Could the authors provide further theoretical or empirical analysis to verify whether nearest-boundary selection consistently aligns with the original answer, rather than merely leading to alternative outputs that happen to improve recovery.
- Question 2: Generalization to diverse attack mechanisms: The evaluation mainly focuses on margin-oriented adversarial perturbations, which may align with the proposed answer-margin correction strategy. It remains unclear whether AM-GARD can generalize to attacks that induce different representation shifts, such as patch-based attacks or universal perturbations. More evaluation under diverse attack settings would strengthen the robustness claims of the method.
9 comments
W1 / Q1: Nearest-Boundary Justification and Behavioral Recovery
We thank the reviewer for highlighting the missing connection between minimum-norm boundary crossing and recovery of the pre-attack response. We agree that reaching the nearest local decision boundary does not, by itself, establish that the boundary corresponds to the original answer.
We have therefore strengthened the analysis to distinguish geometric efficiency from recovery-relevant boundary identification and to evaluate their relationship to actual behavioral recovery.
1. Theoretical relationship between the nearest boundary and the pre-attack answer
Propositions 3.1 and 3.2 characterize the minimum-norm displacement required to cross a fixed linearized token-pair boundary and the effect of directional alignment on the required displacement.
However, these results alone do not establish whether the selected boundary corresponds to the pre-attack answer.
To address this gap, we introduce Proposition 3.3 (Correct-boundary selection condition) in Section 3.2, with its proof and a counterexample in Appendix C.4.
Let denote the attacked leading token, the pre-attack leading token, and another competitor. Define
m_c = z_y - z_c,$$ with margin gradients $$n_{ay} = \nabla_h(z_a-z_y),\qquad n_{ac} = \nabla_h(z_a-z_c).$$ Proposition 3.3 establishes that $$d_y < d_c \iff m_c > g_y\left( \frac{\lVert n_{ac}\rVert_2}{\lVert n_{ay}\rVert_2} - 1 \right).$$ When this condition holds for every other candidate and $y$ belongs to the candidate set, the nearest-boundary rule selects the boundary associated with the pre-attack token. This extends our theoretical analysis from how to reach a selected boundary efficiently to when that boundary is associated with the original answer. The accompanying counterexample also distinguishes this conditional result from an unconditional guarantee. ### 2. From candidate accessibility to genuine behavioral recovery We further examine whether the conditions relevant to correct-boundary selection arise in actual attacked representations. Our analysis follows three stages: **(A) Candidate accessibility.** Under the primary attack, the pre-attack leading token remains among the top four competitors for **96.6–100.0% of flipped inputs** across GQA, POPE, MMBench, and MathVista (Appendix H.1, Table 14). This establishes that the primary attack generally leaves a recovery-relevant candidate accessible across different answer formats. **(B) Correct-boundary selection.** In a separate five-seed diagnostic analysis, the nearest-boundary selector identifies the pre-attack token in **87.7% of available cases on GQA** and **86.3% on MMBench** (Appendix H.1, Table 15). **(C) Behavioral recovery.** More importantly, final response recovery differs substantially depending on whether the selected competitor matches the pre-attack token: | Selected competitor | GQA recovery | MMBench recovery | |---|---:|---:| | Pre-attack token | 89.9% | 93.4% | | Another competitor | 6.9% | 4.0% | These results demonstrate a strong empirical relationship between identifying the pre-attack boundary and recovering the original response, rather than merely producing an alternative answer. The conditional results are complemented by the controlled selector comparison in Section 6.3 (Table 3), which varies the competitor-selection rule while keeping the attacked inputs, intervention decisions, and correction settings fixed. Together, these analyses connect **boundary accessibility, target identification, and genuine response recovery** within the same correction framework.W1 / Q1: Recovery–Preservation Trade-offs and Attack-Strength Analysis (Continued)
3. Recovery and unintended harm: Why not simply select the highest-logit competitor?
We also investigate whether normalized boundary-distance selection offers benefits beyond increasing the frequency of answer changes.
Under the primary attack, Table 3 shows that normalized boundary-distance selection substantially improves recovery over random top- selection and achieves aggregate recovery comparable to highest-logit selection.
However, similar aggregate recovery does not imply identical intervention outcomes.
To examine this distinction, we added an analysis separating new recoveries from harmful changes to previously correct responses (Appendix H.1, Figure 11).
| Dataset | New recovery: Highest-logit → Normalized | New harm: Highest-logit → Normalized |
|---|---|---|
| GQA | 67% → 66% | 44% → 33% |
| POPE | 91% → 79% | 73% → 18% |
| MMBench | 86% → 84% | 93% → 80% |
| MathVista | 79% → 76% | 98% → 98% |
On GQA, normalized selection maintains nearly identical recovery while reducing harmful changes. On POPE, it substantially reduces harm, accompanied by a reduction in recovery.
These measurements use separate conditioning populations for recovery and harm and are therefore interpreted as complementary outcomes rather than a single net-effect score.
These findings provide an empirical reason to use normalized boundary-distance selection beyond its minimum-displacement interpretation. Unlike raw logit ranking, it considers both the answer margin and local gradient geometry when selecting an input-specific correction target. Proposition 3.3 characterizes when this criterion selects the pre-attack token's boundary, while Table 15 connects correct-target selection to actual recovery. The controlled comparison (Table 3, Figure 11) further shows competitive recovery with fewer harmful changes in several settings: on GQA, new recovery is nearly unchanged (66% versus 67%) while harm falls from 44% to 33%; on POPE, harm falls from 73% to 18%, accompanied by lower recovery (79% versus 91%). Thus, normalized boundary-distance selection offers a geometrically grounded recovery--preservation trade-off, rather than merely inducing alternative outputs or claiming uniform superiority over simpler selectors.
4. Attack-strength analysis: When does correct-boundary selection become difficult?
We further investigate how the relationship between the nearest boundary and the pre-attack answer changes with attack strength and objective (Section 6.3, Figure 6).
The analysis distinguishes three factors.
Attack-budget sweep (S1–S4). Increasing the optimization budget does not immediately eliminate the pre-attack token from the candidate set. Candidate accessibility remains relatively stable across the evaluated primary-attack budgets.
Attack-depth analysis. In contrast, candidate accessibility generally decreases as the pre-attack token's margin deficit increases. This indicates that the depth of the attacked answer relative to the original answer is more informative than optimization budget alone.
Matched-depth comparison. Even at comparable margin deficits, correct-boundary selection depends on the attack objective.
For , the observed correct-boundary selection rates are:
| Dataset | First-token margin attack | Sequence-NLL attack |
|---|---|---|
| GQA | 28% | 2% |
| POPE | 76% | 37% |
Thus, the accessibility and selection of a recovery-relevant boundary depend not only on attack depth but also on the answer-boundary structure induced by the attack objective.
This result provides an empirical characterization of when nearest-boundary selection remains useful and when its ability to identify the pre-attack answer becomes limited.
Revisions: We added Proposition 3.3 and its proof in Section 3.2 and Appendix C.4; strengthened the controlled selector and attack-strength analyses in Section 6.3 (Table 3 and Figure 6); and added candidate-accessibility, conditional-recovery, and recovery–harm diagnostics in Appendix H.1 (Tables 14–15 and Figure 11).
Response to Reviewer R3
We sincerely thank the reviewer for the thoughtful and constructive assessment of our work. We particularly appreciate the reviewer's recognition of our central design principle: combining selective intervention with input-specific answer-margin correction to balance recovery of adversarially altered responses and preservation of pre-attack behavior.
The reviewer's concerns address two fundamental questions underlying our approach: (1) why the nearest answer boundary should be relevant to recovering the pre-attack response, rather than merely inducing an alternative output, and (2) whether this correction principle extends beyond first-token margin attacks to different attack mechanisms and generation settings.
To address these questions, we substantially strengthened the manuscript along five directions:
Importantly, the additional attack analyses go beyond evaluating robustness under more attacks. By explicitly separating intervention selection from input-specific answer correction, AM-GARD provides a framework for diagnosing whether recovery is limited by intervention coverage, identification of a useful answer boundary, or the subsequent correction itself.
We address the two concerns below.