acceptodds
Withdrawn from ICLR 2027

Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Representation-based safety defenses for vision-language models (VLMs) often learn to separate harmful from benign inputs, but separation alone does not establish what a risk score means or when it should be trusted. We introduce Counterfactual Neighborhood Risk Fields (CNRF), an evidence-grounded framework for selective VLM safety. CNRF constructs context-matched benign-harmful counterfactual pairs, uses their representation differences as capability-related risk directions, and estimates neighborhood support separately from risk. This separation prevents unfamiliarity from being conflated with harmfulness and enables selective intervention rather than blanket rejection. A victim-adaptive interface accommodates different safety observations across VLM architectures and routes selected requests for same-victim response review, which either releases acceptable candidates or produces constrained alternatives with reduced harmful capability. Across three VLMs and five multimodal jailbreak families, CNRF achieves 86.06-89.30% macro defense success rate while retaining substantial benign-task utility. Controlled experiments further show that query-matched references and preserved context-direction correspondence materially improve risk ranking in the low-false-positive regime. CNRF thus reframes representation-based VLM safety from learning harmfulness boundaries to constructing applicable evidence for selective risk decisions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-Language Models, AI Safety, Counterfactual Learning, Multimodal Jailbreak Defense

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9ewwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a central difficulty in VLM safety: blocking harmful requests without rejecting benign requests that share similar content. It constructs matched benign/harmful reference pairs, models representation differences as local risk directions, and separately estimates reference support. A model-specific screening interface routes requests to response review and constrained generation.

Experimental evaluation covers three VLMs, five attack families, and three benign benchmarks, with additional geometric and response-action controls.

Critical strengths

The matched-pair idea gives the method a plausible distinction from ordinary harmful/benign classification: it tries to measure a change in requested capability while holding context similar.

Separating reference support from estimated risk is also a useful way to frame uncertain safety decisions.

Evaluating benign utility alongside attack resistance is appropriate for the problem.

Critical weaknesses

The conceptual advance over the closest work needs to be clearer. [1] already addresses the confusion between maliciousness and distribution shift. CNRF adds counterfactual correspondence and selective review, but the evidence that these additions explain its gains is uneven across models. Gemma does not use the local field for routing, and InternVL does not consistently benefit from matched directions. Thus, three-model performance does not establish a broadly effective counterfactual mechanism.

The safety–utility conclusion needs better validation. Table 1's SUS values do not match the stated formula—for example, Qwen gives 72.94 on recalculation rather than 69.82. The appendix also says evaluation results inform operating-budget selection and MOSSBench informs the release rule. This leaves uncertainty about how the reported trade-off would generalize after all choices were frozen.

[1] Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring

Questions for authors
  1. After accounting for the model-specific controls in the appendix, what improvement can be attributed to counterfactual correspondence over RCS-style scoring and generic review? Please identify the evidence supporting the central contribution, even if it applies to fewer models.

  2. Which data informed operating budgets and release rules, and which were used only for final testing? Please reconcile the SUS values and state whether the corrected comparison preserves the conclusions. Evidence of an independent final test would materially improve the assessment.

Reviewer_7wuvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Counterfactual Neighborhood Risk Fields (CNRF), a representation-based framework for selective VLM safety defense. It constructs matched benign-harmful counterfactual pairs to estimate risk while separately measuring neighborhood support to avoid conflating unfamiliarity with harmfulness. The method enables selective intervention and response review rather than blanket rejection. Experiments across three VLMs and five jailbreak families demonstrate strong defense performance while preserving benign-task utility.

Critical strengths
  • The authors conduct extensive experiments on different models and compare lots of baselines.
  • The safety performance is good.
  • The idea of using counterfactual property to improve safety performance is interesting.
Critical weaknesses
  • Potential error in Figure 1.

    The sentence shown for option B in the "Overcautious" scenario appears to be incorrect, making the figure somewhat confusing.

  • Unclear motivation.

    Is there clear empirical evidence that directly classifying representations or using a linear probe cannot effectively distinguish harmful from benign requests? I find the motivation for the proposed method insufficiently justified.

  • Significant over-refusal.

    The proposed method also appears to suffer from substantial over-refusal. Its performance on MOSSBench drops considerably compared to the base model and is also notably worse than several other defense methods.

  • Insufficient evidence for the advantage of counterfactual representation differences.

    I don't think the paper sufficiently demonstrates that learning harmful–benign representation differences is more effective than directly training a linear probe classifier. Some methods following the latter approach appear to achieve better defense effectiveness and benign utility [1].

  • I didn't find the AI Use Statement in the paper.

[1] ReSaM: Representation-Level Safety Margin Alignment for Vision–Language Models

Questions for authors
  • Could the author provide the results on more challenging safety tasks? e.g., MSSBench [1], SIUO [2], MIS [3]. These benchmarks construct unsafe intention using benign inputs.

  • Could the author provide the inference overhead compared to base models and baselines?

[1]: Multimodal Situational Safety

[2]: Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Models

[3]: Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models

Reviewer_xnDHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CNRF builds capability directions from context-matched benign/harmful counterfactual pairs and aggregates nearby directions into a risk score, keeping a separate support score for evidence applicability: r0(z)=⟨z,vN(z)⟩−dN(z)r_0(z) = \langle z, v_N(z) \rangle - d_N(z). Requests flagged as risky go through same-victim response review, which either releases the original answer or generates a constrained alternative. Across 3 VLMs and 5 jailbreak families, CNRF reaches 86–89% macro DSR and the best safety–utility score.

Critical strengths
  1. The framing is novel: it asks what relationship reference evidence should preserve, and separates risk from applicability.

  2. The controlled ablations (matched vs. random/far neighbors, direction mismatch) clearly support the evidence design.

Critical weaknesses
  1. The pipeline is victim-specific. Gemma does not use the risk field at all (it uses an internal safety observer instead), and on Qwen the support score does not affect routing. Hyperparameters differ per victim, and MOSSBench, an evaluation benchmark, informs InternVL’s release rule. Full review performs about the same as fixed refusal (10.70 vs. 9.97 ASR).

  2. The evaluation is unreliable. A single Gemma-3-12B judge, which is also a victim and the rewriting model, scores everything, with no human validation. The detection baselines are near chance (46–51% AUROC on Qwen). The SUS values cannot be reproduced from the stated definition: SUS=100×(0.5 D+0.5 B)\mathrm{SUS} = 100 \times (0.5\,D + 0.5\,B). For example, Qwen CARE gives about 57.3 by this formula, but the paper reports 33.75. Table 4 also diverges from Table 1 without explanation.

Questions for authors
  1. How is SUS actually computed, and what explains the differences between Table 4 and Table 1 (a different judge, different prompts)?

  2. What results does a single CNRF configuration achieve on all three victims, with the same reader, the same routing operator, and no internal observer? On Gemma, which component of CNRF accounts for the gains?

  3. How well does the judge agree with human annotations? Do the conclusions hold with a judge outside the Gemma family, such as Llama Guard or a GPT-class model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.