- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses a central difficulty in VLM safety: blocking harmful requests without rejecting benign requests that share similar content. It constructs matched benign/harmful reference pairs, models representation differences as local risk directions, and separately estimates reference support. A model-specific screening interface routes requests to response review and constrained generation.
Experimental evaluation covers three VLMs, five attack families, and three benign benchmarks, with additional geometric and response-action controls.
Critical strengths
The matched-pair idea gives the method a plausible distinction from ordinary harmful/benign classification: it tries to measure a change in requested capability while holding context similar.
Separating reference support from estimated risk is also a useful way to frame uncertain safety decisions.
Evaluating benign utility alongside attack resistance is appropriate for the problem.
Critical weaknesses
The conceptual advance over the closest work needs to be clearer. [1] already addresses the confusion between maliciousness and distribution shift. CNRF adds counterfactual correspondence and selective review, but the evidence that these additions explain its gains is uneven across models. Gemma does not use the local field for routing, and InternVL does not consistently benefit from matched directions. Thus, three-model performance does not establish a broadly effective counterfactual mechanism.
The safety–utility conclusion needs better validation. Table 1's SUS values do not match the stated formula—for example, Qwen gives 72.94 on recalculation rather than 69.82. The appendix also says evaluation results inform operating-budget selection and MOSSBench informs the release rule. This leaves uncertainty about how the reported trade-off would generalize after all choices were frozen.
[1] Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoring
Questions for authors
-
After accounting for the model-specific controls in the appendix, what improvement can be attributed to counterfactual correspondence over RCS-style scoring and generic review? Please identify the evidence supporting the central contribution, even if it applies to fewer models.
-
Which data informed operating budgets and release rules, and which were used only for final testing? Please reconcile the SUS values and state whether the corrected comparison preserves the conclusions. Evidence of an independent final test would materially improve the assessment.