- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper uses OPSD to improve model safety alignment with respect to implicit risks. Specifically, it addresses two issues: (1) over-refusal caused by assuming malicious user intent, and (2) the high training overhead and data construction costs associated with DPO, PPO, and GRPO. The proposed solutions are the design of tailored privileged information and the adoption of OPSD, respectively.
Critical strengths
The paper provides a method for constructing privileged information tailored to implicit risks.
Critical weaknesses
- This work is essentially an application of OPSD to safety alignment for implicit risks, with the design of privileged information being its only novel component. Given the current state of research, I find this design quite intuitive and lacking in deeper insights and analysis. I therefore consider the contribution limited.
- In the experimental evaluation, I find it insufficient to report only a joint score for helpfulness and safety. Without separate results for these two dimensions, I cannot compare the proposed method with the baselines in terms of safety and helpfulness individually. This substantially hinders my assessment of the method.
- In the ablation study, “Ours w/o General-Task Anchors” performs worse than “Ours” on some benchmarks. My intuition is that removing general-task training data would allow the model to learn more thoroughly from the privileged-information data, thereby improving both safety and helpfulness.
- The experimental evaluation should report the false refusal rate separately to provide more convincing evidence of improvements in helpfulness.
Overall, I find the motivation sound, but the paper lacks deeper insights and more substantial innovation.
Questions for authors
See weaknesses.
5 comments
Response to W2 & W3
2. Response to W2: Separate Safety and Helpfulness Results
We thank the reviewer for this crucial suggestion. In the main text, we adopted the Joint Success metric (requiring both full safety compliance and constructive helpfulness in the same response) following the evaluation paradigm of Meerkat-VL [4], because evaluating safety or helpfulness in isolation can be misleading under implicit risks (e.g., a model can achieve high apparent safety via blanket refusals, or high helpfulness by ignoring subtle visual hazards). However, we fully agree that reporting both dimensions separately is essential for transparent comparison. Below, we provide the separate proportions of responses achieving full safety (maximum safety score of 3) and constructive helpfulness (helpfulness score of at least 2) across the three benchmark groups:
Table A: Proportion of Responses Achieving Full Safety (%, higher is better)
| Benchmark | Base | SPA-VL | VLGuard | TiS | SafeGRPO | Source-Privilege | Ours |
|---|---|---|---|---|---|---|---|
| SIUO+HoliSafe | 43.93 | 56.06 | 50.06 | 47.06 | 45.50 | 41.72 | 55.93 |
| BeaverTails-V | 32.00 | 45.50 | 46.00 | 33.00 | 32.00 | 35.50 | 59.00 |
| MOSSBench | 30.33 | 37.33 | 37.00 | 31.67 | 37.00 | 14.33 | 39.67 |
Table B: Proportion of Responses Achieving Constructive Helpfulness (%, higher is better)
| Benchmark | Base | SPA-VL | VLGuard | TiS | SafeGRPO | Source-Privilege | Ours |
|---|---|---|---|---|---|---|---|
| SIUO+HoliSafe | 81.62 | 79.27 | 42.63 | 79.53 | 82.27 | 73.27 | 83.44 |
| BeaverTails-V | 70.50 | 76.00 | 12.00 | 73.50 | 73.50 | 60.00 | 84.50 |
| MOSSBench | 82.67 | 83.00 | 11.00 | 76.33 | 84.67 | 61.67 | 85.33 |
Simultaneous Gains Without the Safety-Helpfulness Trade-off: Unlike baselines that sacrifice helpfulness for safety (for instance, VLGuard reaches 37.00%–50.06% on full safety but collapses to 11.00%–42.63% on helpfulness due to severe over-refusal), Ours achieves the highest full-safety rate on 2 out of 3 benchmarks (and is within 0.13% of SPA-VL on SIUO+HoliSafe) while simultaneously achieving the highest helpfulness rate across all 3 benchmarks.
3. Response to W3: Role of General-Task Anchors in Preserving Multimodal Capabilities
This phenomenon is explained by two well-established mechanisms in the multimodal alignment literature:
- Cross-Modal Implicit Safety Hinges on Fine-Grained Visual Perception: Unlike explicit textual toxicity, implicit risks in
SIUO[5],HoliSafe, andMSSBenchrequire the model to accurately perceive subtle visual details (such as recognizing a lit cigarette, background flames, or museum framing) before judging the safety boundary. As shown in Table 4, Removing general-task anchors leads to a noticeable degradation in general multimodal capabilities, as reflected by the decreases onMMStarandMME-RealWorld. This degradation may also contribute to the modest performance drops observed on some implicit-risk benchmarks. - Preventing Policy Collapse and Over-Refusal via Utility Regularization: Prior studies on VLM and LLM safety alignment—including VLGuard [8], MOSSBench [9], Safe RLHF-V [10], and Safety-Tuned LLaMAs [11]—consistently show that fine-tuning exclusively on safety-focused data shifts the output distribution toward overly conservative refusal templates and degrades helpfulness. In our framework, achieving Joint Success requires delivering concrete, constructive assistance within safe boundaries. General-task anchors act as a behavioral regularizer on ordinary helpfulness, preventing the student model from overfitting to cautionary templates and preserving the generative fluency needed to remain helpful.
Response to W4
4. Response to W4: Additional Analysis of False Refusals
We compute the False Refusal Rate (FRR, lower is better) as the proportion of benign inputs that the model refuses without a valid, evidence-based safety reason:
| Model | HoliSafe-SSS | MOSSBench | MSSBench-safe | Avg. |
|---|---|---|---|---|
| Base | 0.0% | 16.3% | 15.5% | 10.6% |
| SPA-VL | 0.0% | 13.6% | 14.5% | 9.4% |
| VLGuard | 5.0% | 83.0% | 30.0% | 39.3% |
| TiS | 0.0% | 18.0% | 16.5% | 11.5% |
| SafeGRPO | 0.0% | 14.6% | 16.5% | 10.4% |
| Source-Privilege OPSD | 2.6% | 53.0% | 25.0% | 26.9% |
| Intent-Privilege OPSD (Ours) | 0.0% | 14.0% | 14.0% | 9.3% |
- Overall False Refusal Analysis, outperforming even the unaligned Base model: While safety baselines such as
VLGuard(83.0%) andTiS(18.0%) exacerbate over-refusal onMOSSBench, Ours achieves the lowest average FRR (9.3%) and reduces false refusals below the unadapted Base model on bothMOSSBench(16.3% → 14.0%) andMSSBench-safe(15.5% → 14.0%). - Effect of Evidence-Grounded Privilege: Distilling from unverified source intent labels (
Source-Privilege OPSD) causes severe over-refusal (53.0% FRR onMOSSBench) due to speculative threat assumptions. Our Evidence-Grounded Privilege slashes FRR by 39.0 percentage points onMOSSBench(53.0% → 14.0%) and 11.0 points onMSSBench-safe(25.0% → 14.0%), directly confirming that preserving uncertainty and benign interpretations in safety supervision is essential for maintaining helpfulness.
Response to W1
We thank the reviewer for the constructive feedback. We report controlled comparisons on SIUO+HoliSafe, BeaverTails-V, and MOSSBench using the same Qwen3-VL-4B-Instruct backbone and training budget. All scores are Joint Success (%): full safety compliance (S=3) and constructive helpfulness (H≥2).
W1. Contributions: Why Evidence-Grounded Privilege Matters
Unlike verifiable math/code solutions, cross-modal risks are ambiguous: image and text may each be benign, and intent remains uncertain. Source Privilege can harm alignment by turning speculative motives into blanket refusals. We therefore introduce Evidence-Grounded Privilege (EG-Privilege), which supervises observable multimodal facts, epistemic uncertainty, and conditional output boundaries—not inferred motives or refusal instructions.
In the frozen-teacher diagnostic (manuscript Table 1), Source Privilege reduces Joint Success from 25.0% to 11.7% and increases first-sentence refusals from 12.9% to 45.0%. Distilling these annotations yields just 29.9%, below Base (34.7%) and No-Privilege OPSD (38.5%), especially on MOSSBench (12.6%). Errors stem from invented malicious motives and worst-case interpretations of ambiguity. Shuffling EG-Privilege drops performance from 46.5% to 38.4% (-8.1 points), confirming input-specific grounding is essential.
Schema and Pipeline Ablations
Our schema separates (i) grounded evidence, (ii) epistemic limits, and (iii) output effects, excluding hidden motives, safety labels, refusal directives, and reference answers. Matched-budget ablations show distinct contributions:
Even single-pass, unaudited EG-Privilege achieves 41.9% (47.0/45.0/33.6), above Base (34.7%) and Source Privilege (29.9%), validating the schema. Modality-isolated extraction and independent auditing yield another +4.6 points (46.5%) by limiting confirmation bias and filtering 14% schema/grounding violations in single-pass outputs.
Why On-Policy Token-Level Distillation?
Under matched backbone/budget, SFT and offline KD use the same 1,447 intent annotations.
Intent-SFT (35.7%) learns annotation metadata without calibrating responses. Offline KD (41.9%) lags ours by 6.7/6.5 points on SIUO+HoliSafe/BeaverTails-V: the student lacks privileged intent at inference, so off-policy teacher demonstrations induce exposure bias/trajectory mismatch [2,3]. OPSD instead queries the intent-conditioned teacher token by token on the student's own prefixes, guiding decisions where implicit risks arise.
Unlike SPA-VL (30,000 preference pairs) or SafeGRPO (8 rollouts/prompt, reward/cost models), ours uses 1,447 examples, one rollout/prompt, no reference responses or reward model, with ~5× lower wall-clock training time. At deployment, both are removed: no extra modules or reasoning tokens. We will add these analyses to revised Section 5.3.