- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether source-role sensitivity can help audit image-induced errors in vision-language models. The authors propose Boundary Violation Rate, which compares the sensitivity of a reference continuation to visual and trusted-text value ablations. They further introduce SRBV-PGD to increase relative visual sensitivity and SRBV-stealth to disrupt the original answer while suppressing its visual-ablation gap. Experiments mainly use LLaVA-1.5-7B, with additional task and architecture checks.
Critical strengths
-
The paper addresses a meaningful question beyond attack success alone: whether internal sensitivity measurements distinguish harmful image influence from legitimate visual dependence.
-
Considering both exposure-increasing and exposure-suppressing attacks is useful. In particular, adaptive evaluation is necessary for assessing whether an internal diagnostic remains informative when explicitly targeted.
Critical weaknesses
-
The usefulness of BVR for error auditing remains insufficiently validated. Most measurements concern the fixed clean-image continuation, whereas behavioral success is assessed using the separately decoded adversarial answer. Consequently, low or high BVR does not directly characterize the erroneous decision. Moreover, increasing BVR through an objective designed to increase it provides limited evidence of its diagnostic value. A more convincing evaluation would test whether BVR distinguishes induced errors from correct responses with legitimate visual dependence, using attacks that do not optimize the metric and thresholds calibrated on held-out clean data. Although the authors explicitly acknowledge this gap, it remains central to the paper’s motivation.
-
The main stealth result uses a different final estimator. Standard exposed evaluations use the hybrid estimator with (k=8), while SRBV-stealth is evaluated using group-only ablation with (k=0). Since the hybrid estimator takes a maximum over whole-span and selected individual-token effects, low group-only BVR does not imply low hybrid BVR. The reported 100% yes/no reversal with zero BVR therefore establishes suppression of a particular reference-based measurement, rather than evasion of the same detector used elsewhere. Using an inexpensive optimization surrogate is reasonable, but the final detector should remain fixed across attacks.
-
The evaluated models are relatively old and restricted to a narrow parameter range. The main experiments use LLaVA-1.5-7B, and the additional models are Qwen2-VL-7B and InternVL2-8B. This limits confidence that the findings extend to more recent architectures or larger models. The cross-model comparisons also differ in continuation length, estimator, template, and visual-token configuration, making architecture effects difficult to isolate. More importantly, these checks do not reproduce the central stealth finding under a matched detector. Including a recent open-weight model and a controlled comparison across model sizes would substantially strengthen the evidence.
-
The measurement choices require stronger justification, and the methodological novelty appears limited. BVR combines value ablations, relative sensitivity thresholding, and corresponding PGD objectives. The theoretical results mainly describe algebraic properties of these definitions; they do not establish diagnostic validity. The sensitivity to the intervention is also substantial: Supplementary Table 20 reports increased ratios under value-zero but not under mean-ablation on the same attacked images. The acknowledged limitations of the mean-ablation implementation prevent a definitive comparison, leaving independent validation unresolved. Stronger justification is needed for the chosen intervention, aggregation, and threshold.
-
Several experimental comparisons are difficult to interpret quantitatively. Clean and attacked measurements often use different continuation lengths, and attack comparisons differ in sample subsets, initialization, or implementation completeness. The repeated VQAv2 runs also reuse the same example window. The authors appropriately present these results as descriptive, but this limits the evidence for the proposed attacks’ advantages over simpler objectives such as CE-PGD. A smaller, fully matched evaluation would be more persuasive than the current collection of heterogeneous comparisons.
Questions for authors
-
On examples that are initially answered correctly, how well does BVR distinguish attack-induced errors from responses that remain correct, including clean examples requiring strong visual dependence?
-
Does SRBV-stealth retain its reversal rate and low exposure when evaluated with the same hybrid (k=8) detector, continuation length, and fixed threshold used for other attacks?
-
Do the central findings persist on a recent open-weight VLM and across different model sizes under a unified evaluation protocol?