acceptodds
Withdrawn from ICLR 2027

Auditing Image-Induced Errors in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Small image perturbations can change vision-language model (VLM) answers, but pixel constraints and output changes alone do not establish whether an internal signal identifies image-induced errors. Such a signal must distinguish harmful perturbation effects from legitimate visual dependence; its usefulness against ordinary attacks must also be separated from its vulnerability to adaptive attacks. We study these questions through Boundary Violation Rate (BVR), which compares visual and trusted-text value-zero sensitivity. Attack optimization and final detection are specified separately: a cheaper optimization surrogate does not redefine the detector used to assess evasion. A matched-estimator analysis motivates SRBV-PGD as an exposed positive control, while SRBV-stealth suppresses reference-answer exposure during output disruption. Existing LLaVA-1.5/POPE experiments show 100% normalized yes/no reversals across 360 evaluations with near-zero group-only BVR. This establishes a limitation of that reference-based diagnostic, not evasion of the hybrid detector. The current evidence motivates matched actual-answer validation and fixed-detector evaluation, rather than establishing universal error attribution or reliable deployment-time detection.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Adversarial Robustness, Vision Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FZBCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether source-role sensitivity can help audit image-induced errors in vision-language models. The authors propose Boundary Violation Rate, which compares the sensitivity of a reference continuation to visual and trusted-text value ablations. They further introduce SRBV-PGD to increase relative visual sensitivity and SRBV-stealth to disrupt the original answer while suppressing its visual-ablation gap. Experiments mainly use LLaVA-1.5-7B, with additional task and architecture checks.

Critical strengths
  1. The paper addresses a meaningful question beyond attack success alone: whether internal sensitivity measurements distinguish harmful image influence from legitimate visual dependence.

  2. Considering both exposure-increasing and exposure-suppressing attacks is useful. In particular, adaptive evaluation is necessary for assessing whether an internal diagnostic remains informative when explicitly targeted.

Critical weaknesses
  1. The usefulness of BVR for error auditing remains insufficiently validated. Most measurements concern the fixed clean-image continuation, whereas behavioral success is assessed using the separately decoded adversarial answer. Consequently, low or high BVR does not directly characterize the erroneous decision. Moreover, increasing BVR through an objective designed to increase it provides limited evidence of its diagnostic value. A more convincing evaluation would test whether BVR distinguishes induced errors from correct responses with legitimate visual dependence, using attacks that do not optimize the metric and thresholds calibrated on held-out clean data. Although the authors explicitly acknowledge this gap, it remains central to the paper’s motivation.

  2. The main stealth result uses a different final estimator. Standard exposed evaluations use the hybrid estimator with (k=8), while SRBV-stealth is evaluated using group-only ablation with (k=0). Since the hybrid estimator takes a maximum over whole-span and selected individual-token effects, low group-only BVR does not imply low hybrid BVR. The reported 100% yes/no reversal with zero BVR therefore establishes suppression of a particular reference-based measurement, rather than evasion of the same detector used elsewhere. Using an inexpensive optimization surrogate is reasonable, but the final detector should remain fixed across attacks.

  3. The evaluated models are relatively old and restricted to a narrow parameter range. The main experiments use LLaVA-1.5-7B, and the additional models are Qwen2-VL-7B and InternVL2-8B. This limits confidence that the findings extend to more recent architectures or larger models. The cross-model comparisons also differ in continuation length, estimator, template, and visual-token configuration, making architecture effects difficult to isolate. More importantly, these checks do not reproduce the central stealth finding under a matched detector. Including a recent open-weight model and a controlled comparison across model sizes would substantially strengthen the evidence.

  4. The measurement choices require stronger justification, and the methodological novelty appears limited. BVR combines value ablations, relative sensitivity thresholding, and corresponding PGD objectives. The theoretical results mainly describe algebraic properties of these definitions; they do not establish diagnostic validity. The sensitivity to the intervention is also substantial: Supplementary Table 20 reports increased ratios under value-zero but not under mean-ablation on the same attacked images. The acknowledged limitations of the mean-ablation implementation prevent a definitive comparison, leaving independent validation unresolved. Stronger justification is needed for the chosen intervention, aggregation, and threshold.

  5. Several experimental comparisons are difficult to interpret quantitatively. Clean and attacked measurements often use different continuation lengths, and attack comparisons differ in sample subsets, initialization, or implementation completeness. The repeated VQAv2 runs also reuse the same example window. The authors appropriately present these results as descriptive, but this limits the evidence for the proposed attacks’ advantages over simpler objectives such as CE-PGD. A smaller, fully matched evaluation would be more persuasive than the current collection of heterogeneous comparisons.

Questions for authors
  1. On examples that are initially answered correctly, how well does BVR distinguish attack-induced errors from responses that remain correct, including clean examples requiring strong visual dependence?

  2. Does SRBV-stealth retain its reversal rate and low exposure when evaluated with the same hybrid (k=8) detector, continuation length, and fixed threshold used for other attacks?

  3. Do the central findings persist on a recent open-weight VLM and across different model sizes under a unified evaluation protocol?

Reviewer_fJVaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an auditing technique to identify when adversarial perturbation cause a VLM to alter the output and whether an adaptive attacker can suppress that. It introduces a metric Boundary Violation Rate (BVR) which measures the fraction of images which have higher visual sensitivity than text sensitivity. Visual sensitivity basically refers to the change in log prob of the answer with and without the image input. Similarly for text sensitivity.

Two attack approaches are proposed around this measure. a) SRBV-PGD increases visual sensitivity relative to text's while SRBV-stealth attack lowers the clean answer's likelihood while suppressing the visual ablation gap.

Evaluation is done mainly on LlaVa-1.5-7B with POPE and VQAv2 mainly. Additional experiments include ChartQA, DocVQA, TextVQA datasets as well as qwen2-vl and InternVL2 models.

SRBV-PGD flips all POPE answers with BVR(1) of about 0.2. SRBV-stealth achieves 100% yes/no reversals across POPE evaluations with near zero BVR. Paper conducts additional ablations covering defenses, pruning, prompt preambles and so on.

Critical strengths

Interesting idea to connect the attack success to both increasing and decreasing image sensitivity intentionally

Critical weaknesses

It is not clear why BVR is a useful audit signal. There is very little evidence that BVR identified image-induced errors. Also, why is BVR scoring the clean answer instead of the model's new answer with the attack?

It misses ordinary attacks such as the vanilla PGD which change 10-33% of answers with BVR(1) close to 0, so BVR is not really helping with any diagnostics here.

SRBV-PGD is optimized to raise BVR, so high BVR is expected. The authors concede its scores “alone cannot validate BVR.”

Similarly, SRBV-stealth is optimized to keep BVR low, and is then scored with the same simplified estimator it optimized against (k = 0), not the full detector (k = 8) used for every other attack.

Also, lower BVR doesn't mean safer. For example, Resize+crop keeps BVR(1)=0 but leads only 52% accuracy under attack.

Questions for authors

Errors from Attention-Hijacking, CAGE-EFD, and V-Attack are image-induced, yet BVR(1) = 0 for all of them. Do you read this as BVR failing to detect them? If not, why not?

Of the 10–33% of answers the foils change, how many go from correct to incorrect, and what is BVR (at α = 0.5 and α = 1) on just those samples?

Were the three stealth runs (360 evaluations) on different samples? Can the per-run results be recovered?

What are the other uses of BVR? It clearly cannot be used for detection purposes if you also require the clean images to compute it

Does BVR on the attacked answer (instead of the clean answer) separate attacked and clean samples?

Reviewer_HpRTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Boundary Violation Rate (BVR), a white-box diagnostic that compares visual-span and trusted-text-span sensitivity of a VLM's output under value-zero intervention. Sensitivity is measured as the absolute log-probability change of a fixed clean-image reference continuation y_ref. BVR is the fraction of samples whose visual sensitivity exceeds α times the trusted-text sensitivity. The paper introduces two attacks:

  • SRBV-PGD, which raises BVR as an "exposed positive control."
  • SRBV-stealth, which flips POPE yes/no answers while suppressing a group-only (k = 0) version of the visual gap.

The headline result is 100% yes/no reversals on LLaVA-1.5/POPE with group-only BVR(1) = 0. The paper also reports cross-task and cross-model measurements, defense stress tests, and a large supplement of diagnostics. By its own account, the evidence "does not complete" the validation chain the paper sets out.

Critical strengths
  1. Unusually explicit claim scoping. The paper states clearly what each result does not show: y_ref vs. y_adv, group-only vs. hybrid estimator, unmatched archives, pseudoreplication. It also separates attack surrogates from the final detector. That discipline is commendable.
  2. Some useful negative findings. Reduced BVR does not imply task robustness: blur, JPEG, and resize+crop reach BVR(1) ≈ 0 while outputs still change or accuracy collapses under BPDA/EOT. Mean-ablation also largely erases the value-zero exposure effect. Both are worth knowing for anyone using ablation-based source attribution.
Critical weaknesses
  1. The paper does not answer its own research question. The introduction sets out a four-step chain:

    1. An image perturbation must turn a correct answer into an incorrect one.
    2. The signal must be evaluated on the actual erroneous decision.
    3. An independent intervention must validate the signal.
    4. Adaptive evasion must be tested against a frozen detector.

    None of steps 2–4 is completed:

    • BVR is always computed on y_ref, never on y_adv.
    • There is no causal validation, and the activation-replacement check is absent.
    • The stealth result uses a k = 0 detector, while the exposed attacks use k = 8.

    The abstract concedes the headline number "establishes a limitation of that reference-based diagnostic, not evasion of the hybrid detector." What remains is a metric whose usefulness for detecting image-induced errors is never demonstrated.

  2. BVR does not detect attacks that do not optimize it, and the positive results are circular.

    • The attention, feature, and compression foils change 10–33% of outputs yet all have BVR(1) = 0.
    • On the oracle-reference ROC, SRBV-PGD and plain CE-PGD are indistinguishable (AUC 0.819 vs. 0.807).
    • High BVR appears mainly when the attack optimizes BVR itself (SRBV-PGD), and low BVR when the attack suppresses it (stealth). Showing that a differentiable quantity can be pushed up or down by gradient descent is expected and uninformative.
    • Detection also requires the clean-image continuation, an oracle that is unavailable at deployment.
    • The effect is intervention-specific. Under mean-ablation, only 33% of the same attacked images show increased ρ, versus 100% under value-zero.
  3. The evidence base is a post-hoc reanalysis of mismatched archives.

    • Clean baselines use K = 10, attacked results use K = 3, and matched-estimator Lemma 1 therefore cannot be instantiated.
    • Foils use n = 44–48, while SRBV-PGD uses n = 498.
    • The initialization rules differ between attacks.
    • The CE comparison is pseudoreplicated (15 reruns of one 334-example window).
    • Seed-level stealth files and sample IDs are missing.
    • The resize+crop RNG is unseeded, and its configuration fields are unused.

    The paper flags each of these issues, but flagging them does not fix them. The result is a long list of descriptive, non-comparable cells, presented with dense hedging that makes the paper very hard to read.

Questions for authors
  1. Can you evaluate SRBV-stealth under the same frozen k = 8 hybrid detector used for the other attacks, and report BVR computed on y_adv? (This would most change my assessment.)
  2. On examples that are correct on the clean image, does BVR distinguish ordinary attacks (CE-PGD, the foils) from clean inputs without the oracle reference? Please report results at a calibrated false-positive rate.
  3. Is there any independent intervention (e.g., activation patching) showing that elevated BVR corresponds to the image actually driving the erroneous answer?
  4. Can you rerun the core comparisons with matched n, K, k, and initialization so that Lemma 1 and the foil comparisons become valid?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.