acceptodds
Under review as a conference paper at ICLR 2027

Can Vision-Language Models Stay Helpful When Facing Implicit Risks? Intent-Privilege OPSD for Efficient Safety–Helpfulness Alignment

Abstract

Vision-Language Models (VLMs) remain vulnerable to cross-modal implicit risks: visual and textual inputs that appear benign in isolation can jointly elicit unsafe responses. Existing safety methods often require large preference datasets, costly multi-rollout training, or additional safeguards at inference time. They may also sacrifice helpfulness by **directly refusing** requests that could be answered safely. In this paper, we propose Intent-Privilege On-Policy Self-Distillation (OPSD), which leverages evidence-grounded intent as privileged supervision during training to help VLMs recognize implicit risks and provide **safe, useful responses instead of blanket refusals**. OPSD distills a teacher’s intent-conditioned preferences over responses into a student using a single rollout per prompt; the student then responds without intent annotations or an additional safety module. With only 1,447 safety-specific examples—**95% fewer** than standard preference datasets—OPSD reduces training time by **5×** relative to multi-rollout GRPO-style training and average inference length by **7%**. It attains the highest ratio for joint safety–helpfulness success, which measures the proportion of responses that are both safe and helpful, across all five evaluation groups. Remarkably, on pooled SIUO+HoliSafe, this success ratio rises from 43.9% to 53.5%. These results show that training-time intent supervision can improve both safety and helpfulness while substantially reducing data, training, and inference costs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Model Safety, Social Good, Safe Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uS5oOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper uses OPSD to improve model safety alignment with respect to implicit risks. Specifically, it addresses two issues: (1) over-refusal caused by assuming malicious user intent, and (2) the high training overhead and data construction costs associated with DPO, PPO, and GRPO. The proposed solutions are the design of tailored privileged information and the adoption of OPSD, respectively.

Critical strengths

The paper provides a method for constructing privileged information tailored to implicit risks.

Critical weaknesses
  1. This work is essentially an application of OPSD to safety alignment for implicit risks, with the design of privileged information being its only novel component. Given the current state of research, I find this design quite intuitive and lacking in deeper insights and analysis. I therefore consider the contribution limited.
  2. In the experimental evaluation, I find it insufficient to report only a joint score for helpfulness and safety. Without separate results for these two dimensions, I cannot compare the proposed method with the baselines in terms of safety and helpfulness individually. This substantially hinders my assessment of the method.
  3. In the ablation study, “Ours w/o General-Task Anchors” performs worse than “Ours” on some benchmarks. My intuition is that removing general-task training data would allow the model to learn more thoroughly from the privileged-information data, thereby improving both safety and helpfulness.
  4. The experimental evaluation should report the false refusal rate separately to provide more convincing evidence of improvements in helpfulness.

Overall, I find the motivation sound, but the paper lacks deeper insights and more substantial innovation.

Questions for authors

See weaknesses.

5 comments
AuthorsOpenReview ↗

Response to W1

We thank the reviewer for the constructive feedback. We report controlled comparisons on SIUO+HoliSafe, BeaverTails-V, and MOSSBench using the same Qwen3-VL-4B-Instruct backbone and training budget. All scores are Joint Success (%): full safety compliance (S=3S=3) and constructive helpfulness (H≥2H\ge2).

W1. Contributions: Why Evidence-Grounded Privilege Matters

Unlike verifiable math/code solutions, cross-modal risks are ambiguous: image and text may each be benign, and intent remains uncertain. Source Privilege can harm alignment by turning speculative motives into blanket refusals. We therefore introduce Evidence-Grounded Privilege (EG-Privilege), which supervises observable multimodal facts, epistemic uncertainty, and conditional output boundaries—not inferred motives or refusal instructions.

OPSD settingSIUO+HoliSafeBeaverTails-VMOSSBenchAvg.
Base (no training)43.931.029.334.7
Source Privilege41.735.512.629.9
No Privilege46.139.030.038.5
Shuffled EG-Privilege43.640.531.038.4
Ours: EG-Privilege53.549.037.046.5

In the frozen-teacher diagnostic (manuscript Table 1), Source Privilege reduces Joint Success from 25.0% to 11.7% and increases first-sentence refusals from 12.9% to 45.0%. Distilling these annotations yields just 29.9%, below Base (34.7%) and No-Privilege OPSD (38.5%), especially on MOSSBench (12.6%). Errors stem from invented malicious motives and worst-case interpretations of ambiguity. Shuffling EG-Privilege drops performance from 46.5% to 38.4% (-8.1 points), confirming input-specific grounding is essential.

Schema and Pipeline Ablations

Our schema separates (i) grounded evidence, (ii) epistemic limits, and (iii) output effects, excluding hidden motives, safety labels, refusal directives, and reference answers. Matched-budget ablations show distinct contributions:

BenchmarkBase–Evidence–Limits–EffectsFull
SIUO+HoliSafe43.947.846.848.553.5
BeaverTails-V31.042.045.044.049.0
MOSSBench29.332.331.035.337.0
Avg.34.740.740.942.646.5
  • Grounded evidence (-5.8 pts): Anchors benign, answerable interpretations in verified image/text relations; removing it raises unjustified refusals 8.26% → 10.42%.
  • Epistemic limits (-5.6 pts): Records missing authorization/context and plausible benign alternatives, supporting calibrated conditional help; removing them lowers conditional responses 8.63% → 7.21%.
  • Output effects (-3.9 pts): Identifies which concrete outputs would enable harm, beyond superficial warnings; removing them raises harmful responses 7.95% → 9.37%.

Even single-pass, unaudited EG-Privilege achieves 41.9% (47.0/45.0/33.6), above Base (34.7%) and Source Privilege (29.9%), validating the schema. Modality-isolated extraction and independent auditing yield another +4.6 points (46.5%) by limiting confirmation bias and filtering 14% schema/grounding violations in single-pass outputs.

Why On-Policy Token-Level Distillation?

Under matched backbone/budget, SFT and offline KD use the same 1,447 intent annotations.

MethodSIUO+HoliSafeBeaverTails-VMOSSBenchAvg.
SPA-VL50.344.036.343.5
SafeGRPO45.430.532.037.6
TiS45.932.529.035.8
Intent-SFT44.931.530.635.7
Resp-SFT (offline KD)46.842.536.341.9
Ours (EG-Privilege OPSD)53.549.037.046.5

Intent-SFT (35.7%) learns annotation metadata without calibrating responses. Offline KD (41.9%) lags ours by 6.7/6.5 points on SIUO+HoliSafe/BeaverTails-V: the student lacks privileged intent at inference, so off-policy teacher demonstrations induce exposure bias/trajectory mismatch [2,3]. OPSD instead queries the intent-conditioned teacher token by token on the student's own prefixes, guiding decisions where implicit risks arise.

Unlike SPA-VL (30,000 preference pairs) or SafeGRPO (8 rollouts/prompt, reward/cost models), ours uses 1,447 examples, one rollout/prompt, no reference responses or reward model, with ~5× lower wall-clock training time. At deployment, both are removed: no extra modules or reasoning tokens. We will add these analyses to revised Section 5.3.

AuthorsOpenReview ↗

Response to W2 & W3

2. Response to W2: Separate Safety and Helpfulness Results

We thank the reviewer for this crucial suggestion. In the main text, we adopted the Joint Success metric (requiring both full safety compliance and constructive helpfulness in the same response) following the evaluation paradigm of Meerkat-VL [4], because evaluating safety or helpfulness in isolation can be misleading under implicit risks (e.g., a model can achieve high apparent safety via blanket refusals, or high helpfulness by ignoring subtle visual hazards). However, we fully agree that reporting both dimensions separately is essential for transparent comparison. Below, we provide the separate proportions of responses achieving full safety (maximum safety score of 3) and constructive helpfulness (helpfulness score of at least 2) across the three benchmark groups:

Table A: Proportion of Responses Achieving Full Safety (%, higher is better)

BenchmarkBaseSPA-VLVLGuardTiSSafeGRPOSource-PrivilegeOurs
SIUO+HoliSafe43.9356.0650.0647.0645.5041.7255.93
BeaverTails-V32.0045.5046.0033.0032.0035.5059.00
MOSSBench30.3337.3337.0031.6737.0014.3339.67

Table B: Proportion of Responses Achieving Constructive Helpfulness (%, higher is better)

BenchmarkBaseSPA-VLVLGuardTiSSafeGRPOSource-PrivilegeOurs
SIUO+HoliSafe81.6279.2742.6379.5382.2773.2783.44
BeaverTails-V70.5076.0012.0073.5073.5060.0084.50
MOSSBench82.6783.0011.0076.3384.6761.6785.33

Simultaneous Gains Without the Safety-Helpfulness Trade-off: Unlike baselines that sacrifice helpfulness for safety (for instance, VLGuard reaches 37.00%–50.06% on full safety but collapses to 11.00%–42.63% on helpfulness due to severe over-refusal), Ours achieves the highest full-safety rate on 2 out of 3 benchmarks (and is within 0.13% of SPA-VL on SIUO+HoliSafe) while simultaneously achieving the highest helpfulness rate across all 3 benchmarks.

3. Response to W3: Role of General-Task Anchors in Preserving Multimodal Capabilities

This phenomenon is explained by two well-established mechanisms in the multimodal alignment literature:

  1. Cross-Modal Implicit Safety Hinges on Fine-Grained Visual Perception: Unlike explicit textual toxicity, implicit risks in SIUO [5], HoliSafe, and MSSBench require the model to accurately perceive subtle visual details (such as recognizing a lit cigarette, background flames, or museum framing) before judging the safety boundary. As shown in Table 4, Removing general-task anchors leads to a noticeable degradation in general multimodal capabilities, as reflected by the decreases on MMStar and MME-RealWorld. This degradation may also contribute to the modest performance drops observed on some implicit-risk benchmarks.
  2. Preventing Policy Collapse and Over-Refusal via Utility Regularization: Prior studies on VLM and LLM safety alignment—including VLGuard [8], MOSSBench [9], Safe RLHF-V [10], and Safety-Tuned LLaMAs [11]—consistently show that fine-tuning exclusively on safety-focused data shifts the output distribution toward overly conservative refusal templates and degrades helpfulness. In our framework, achieving Joint Success requires delivering concrete, constructive assistance within safe boundaries. General-task anchors act as a behavioral regularizer on ordinary helpfulness, preventing the student model from overfitting to cautionary templates and preserving the generative fluency needed to remain helpful.
AuthorsOpenReview ↗

Response to W4

4. Response to W4: Additional Analysis of False Refusals

We compute the False Refusal Rate (FRR, lower is better) as the proportion of benign inputs that the model refuses without a valid, evidence-based safety reason:

ModelHoliSafe-SSSMOSSBenchMSSBench-safeAvg.
Base0.0%16.3%15.5%10.6%
SPA-VL0.0%13.6%14.5%9.4%
VLGuard5.0%83.0%30.0%39.3%
TiS0.0%18.0%16.5%11.5%
SafeGRPO0.0%14.6%16.5%10.4%
Source-Privilege OPSD2.6%53.0%25.0%26.9%
Intent-Privilege OPSD (Ours)0.0%14.0%14.0%9.3%
  1. Overall False Refusal Analysis, outperforming even the unaligned Base model: While safety baselines such as VLGuard (83.0%) and TiS (18.0%) exacerbate over-refusal on MOSSBench, Ours achieves the lowest average FRR (9.3%) and reduces false refusals below the unadapted Base model on both MOSSBench (16.3% → 14.0%) and MSSBench-safe (15.5% → 14.0%).
  2. Effect of Evidence-Grounded Privilege: Distilling from unverified source intent labels (Source-Privilege OPSD) causes severe over-refusal (53.0% FRR on MOSSBench) due to speculative threat assumptions. Our Evidence-Grounded Privilege slashes FRR by 39.0 percentage points on MOSSBench (53.0% → 14.0%) and 11.0 points on MSSBench-safe (25.0% → 14.0%), directly confirming that preserving uncertainty and benign interpretations in safety supervision is essential for maintaining helpfulness.
Reviewer_3fYSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to improve the safety and helpfulness of VLMs when facing implicit risks. The authors construct evidence-grounded intent annotations to reduce over-refusal and use OPSD to train the model with these annotations. Experiments show that the method improves safety and helpfulness with less training data and lower training costs.

Critical strengths
  1. The paper identifies an interesting problem of over-refusal caused by unreliable intent annotations and proposes an effective solution.

  2. The experiments are comprehensive, with evaluations across multiple benchmarks and detailed ablation studies to support the effectiveness of the proposed method.

Critical weaknesses
  1. The source of general-task anchor data is unclear, raising concerns about the fairness of comparisons with baselines.
  2. The novelty is limited. The main contribution is constructing the intent annotations, but the pipeline follows a common approach. Moreover, manual checking increases the cost, and human effort may have too much influence on label quality.
Questions for authors
  1. Could the same annotation instructions be given to a single VLM in one prompt? Would this achieve similar results to the proposed multi-stage pipeline?

  2. Is manual checking necessary? Would replacing it with AI checking affect the performance?

  3. BeaverTails-V contains 200 test samples. Why does Table 3 report success rates such as 39.3% and 49.7%, which are not multiples of 0.5%?

  4. Could simpler methods, such as SFT or knowledge distillation using the same annotations, achieve comparable results? What are the specific benefits of using OPSD?

  5. How are the training examples distributed across different risk categories? Does the method perform consistently across these categories?

5 comments
AuthorsOpenReview ↗

Response to CW1

We sincerely thank Reviewer 3fYS for the positive assessment and constructive questions. Due to the limited rebuttal timeframe, we focus our new controlled experiments and analyses on the most critical benchmarks (SIUO+HoliSafe, BeaverTails-V, and MOSSBench) using the same Qwen3-VL-4B-Instruct backbone and training budget.

1. Response to CW1: Clarification of General-Task Anchor Data and Baseline Comparability

We clarify the exact data source of our general-task anchors, their training role, and why our safety gains are independent of them:

  1. Source and Nature of General-Task Anchors: The 2,500 general-task anchors are ordinary multimodal VQA dialogues sampled from LLaVA-Instruct-150K [1]. They contain zero safety annotations, zero privileged context, and zero safety-specific instructions. During training, they are supervised via standard cross-entropy loss on ordinary assistant turns, solely to regularize LoRA updates against general capability forgetting ("alignment tax") [2, 3].
  2. Safety Gains Do Not Depend on Anchors (Ours w/o General-Task Anchors in Table 3): When we completely remove all 2,500 general-task anchors and train exclusively on the 1,447 safety-specific examples, Ours w/o General-Task Anchors achieves 50.0% on SIUO+HoliSafe, 50.0% on BeaverTails-V, and 36.6% on MOSSBench. Its macro-average Joint Success across these benchmarks (45.4%) still outperforms Base (34.7%) and all baselines in Table 2, and slightly exceeds our full model on BeaverTails-V (+0.7). Meanwhile, Table 4 isolates their true role: removing anchors drops MMStar from 62.3 to 52.4 while leaving safety alignment intact. We will explicitly document the LLaVA-Instruct-150K sampling source in Section 5.1 of the revised manuscript.
AuthorsOpenReview ↗

Response to CW2, Q1 & Q2

2. Response to CW2, Q1 & Q2: Annotation Design and Verification

(a) Clarifying Our Contributions (CW2)

Why naive intent labels fail. Implicit-risk intent is uncertain, unlike verifiable reasoning. Existing Source-Privilege labels speculate about hidden motives, causing over-refusal: frozen-teacher Joint Success falls from 25.0% to 11.7% (refusals triple to 45.0%), while student macro-average Joint Success drops to 29.9%, below Base (34.7%) and No-Privilege OPSD (38.5%). Shuffled EG-Privilege also underperforms ours (38.4% vs. 46.5%), demonstrating the importance of input-specific grounding.

Our schema and its components. Evidence-Grounded Privilege separates (i) observable evidence, (ii) epistemic limits, and (iii) conditional output effects, while prohibiting invented motives and refusal directives. Removing any pillar reduces macro-average Joint Success by 3.9–5.8 points (from 46.5%): without Grounded Evidence, unjustified refusals reach 10.42%; without Epistemic Limits, conditional help falls to 7.21%; without Output Effects, harmful outputs reach 9.37%. Even without verification, Single-Pass OPSD achieves 41.9%; modality isolation plus auditing adds 4.6 points.

Why OPSD rather than SFT. Using the same 1,447 annotations, Intent-SFT (35.7%) and offline Resp-SFT (41.9%) trail ours (46.5%). Static off-policy response imitation suffers from exposure bias without test-time intent. OPSD instead provides token-level supervision on the student's own prefixes using one rollout, without reference responses or reward models; it cuts training time by ~5× and adds no inference overhead.

(b) Single-Pass VLM vs. Multi-Stage Pipeline (Q1)

We added a controlled Single-Pass OPSD (1-Prompt, No Check) ablation: one VLM sees the image and query jointly, receives the full annotation rules and JSON schema, and produces annotations with no AI audit or human check. Student training is otherwise identical.

MethodSIUO+HoliSafeBeaverTails-VMOSSBenchMacro Avg
Base43.931.029.334.7
Source-Privilege OPSD41.735.512.629.9
No-Privilege OPSD46.139.030.038.5
Single-Pass OPSD47.045.033.641.9
Full OPSD (Ours)53.549.037.046.5

Joint Success (%; higher is better) requires both full safety compliance (S=3S=3) and constructive helpfulness (H≥2H\ge2).

Schema matters even in one pass. Single-Pass raises macro-average Joint Success by 7.2 points over Base, 3.4 over No-Privilege OPSD, and 12.0 over Source-Privilege OPSD, despite having no verification.

Why multi-stage adds 4.6 points. Full OPSD improves SIUO+HoliSafe by 6.5 points (47.0→53.5) and BeaverTails-V by 4.0 (45.0→49.0). First, independently extracting image-only and text-only facts mitigates cross-modal confirmation bias: joint image–text prompting can distort perception or hallucinate risk [4–6]. Second, independent Builder–Auditor verification detects unsupported claims and schema violations [7, 8]. Indeed, 14% of Single-Pass annotations violated JSON validity, prohibited-content rules, or length limits.

(c) Human vs. AI Checking (Q2)

Manual review is not essential to annotation generation or large-scale deployment. Our pipeline already uses an independent GPT-5.6 Auditor for semantic checking/revision plus deterministic scripts for JSON validity, the 170-word cap, and prohibited terms. Humans performed only a final binary Pass/Exclude sanity check; they never wrote, edited, or paraphrased labels. This optional gate catches rare shared Builder–Auditor blind spots (e.g., cross-modal hallucinations or leaked directives) and took only a few hours for 1,447 samples. At scale, the AI Auditor + deterministic filters can replace this manual gate; the 41.9% Single-Pass result further shows that useful gains arise even without either form of review.

AuthorsOpenReview ↗

Response to Q3 & Q4

3. Response to Q3: Clarification of BeaverTails-V Evaluation and Aggregation

An occasional evaluation parsing glitch omitted a small number of unparsed samples during metric aggregation in Table 3, resulting in a denominator slightly smaller than 200. We have fixed the scoring script, re-evaluated the omitted samples across the full 200-sample set, and updated Table 3 in the revised manuscript with the corrected percentages.


4. Response to Q4: Comparison with SFT and Knowledge Distillation under Identical Annotations

To isolate the optimization benefit of On-Policy Self-Distillation (OPSD) from standard supervised fine-tuning (SFT) and offline sequence-level knowledge distillation (KD), we compare four methods trained on the exact same 1,447 safety annotations and 2,500 general-task anchors:

  • Base: The unadapted Qwen3-VL-4B-Instruct backbone ((image, query) →\rightarrow direct response).
  • Intent-Prediction SFT (Intent-SFT): Fine-tuning the student via standard SFT on (image, query) to directly output our structured intent annotation as target text. At inference, it answers (image, query) directly.
  • Teacher-Response SFT (Resp-SFT, Offline Sequence KD): Following standard context distillation [9], the frozen privileged teacher takes (image, query, intent) and generates a complete target response; the student then takes only (image, query) and is trained via cross-entropy SFT to imitate the teacher's pre-generated response token by token.
  • Intent-Privilege OPSD (Ours): The student samples a single response on-policy from (image, query). Along each prefix of the student's own generated response, the frozen teacher receives (image, query, intent) and provides token-level next-token probability supervision via forward KL divergence. At inference, both the teacher and intent are discarded.
MethodSIUO+HoliSafeBeaverTails-VMOSSBenchMacro Avg
Base43.931.029.334.7
Intent-SFT (Predict Annotation)44.931.530.635.7
Resp-SFT (Offline Teacher KD)46.842.536.341.9
Intent-Privilege OPSD (Ours)53.549.037.046.5

Key Findings & Benefits of OPSD:

  1. Intent-SFT (35.7% avg) fails to calibrate responses: Training the student to output structured safety metadata as auxiliary text does not teach it how to modulate response boundaries when answering user queries, remaining near Base (34.7%).
  2. Resp-SFT / Offline KD (41.9% avg) trails OPSD (46.5% avg) on implicit risks: Offline KD lags behind OPSD by 6.7 points on SIUO+HoliSafe (46.8% vs. 53.5%) and 6.5 points on BeaverTails-V (42.5% vs. 49.0%).
  3. Eliminating Privilege-Induced Trajectory Mismatch: In Resp-SFT, the teacher generates responses off-policy while viewing intent. Because the student lacks intent at deployment, minor deviations in its opening tokens on subtle cross-modal risks place it in states unseen in the teacher's demonstrations (exposure bias [10, 11]). OPSD avoids this mismatch by aligning token-level distributions directly along the student's own on-policy prefixes [11]. We will add this comparison table and analysis to Section 5.3.
Reviewer_dGdxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Intent-Privilege OPSD for balancing safety and helpfulness in VLMs. It uses evidence-grounded intent annotations and single-rollout self-distillation to improve safety alignment without additional inference components. Experiments show promising performance with reduced training costs.

Critical strengths

1.The paper addresses the important challenge of balancing safety and helpfulness under cross-modal implicit risks. 2.The proposed method achieves promising results with reduced training costs and no additional inference overhead.

Critical weaknesses

1.The independent contribution of OPSD remains unclear due to the lack of a direct comparison with SFT on teacher-generated responses using the same intent annotations. 2.The lack of component-level ablations on intent annotations makes it unclear which elements drive the improvements and whether the full annotation pipeline is necessary.

Questions for authors

1.How does OPSD compare with SFT under the same intent supervision and training data? 2.What are the individual contributions of different intent annotation components?

3 comments
AuthorsOpenReview ↗

Response to W1 & Q1

We sincerely thank Reviewer dGdx for the constructive feedback and for recognizing the effectiveness and low resource overhead of our approach. Due to the limited rebuttal timeframe, we focus our new controlled experiments and analyses on the most critical benchmarks (SIUO+HoliSafe, BeaverTails-V, and MOSSBench) using the same Qwen3-VL-4B-Instruct backbone and training budget.

1. Response to W1 & Q1: OPSD vs. SFT under Identical Intent Supervision

To isolate the optimization benefit of On-Policy Self-Distillation (OPSD) from standard supervised fine-tuning (SFT) and unprivileged distillation, we compare five setups trained on the exact same data budget:

  • Base: The unadapted Qwen3-VL-4B-Instruct backbone ((image, query) →\rightarrow direct response).
  • Intent-Prediction SFT (Intent-SFT): Standard SFT that takes (image, query) as input and trains the student to directly output our structured intent annotation as target text. At inference, it generates responses directly from (image, query).
  • Teacher-Response SFT (Resp-SFT, Off-Policy Sequence Distillation): First, the frozen privileged teacher takes (image, query, intent) as input and generates a complete response. Next, the student takes only (image, query) (without intent) and is trained via standard cross-entropy SFT to imitate the teacher's pre-generated response token by token.
  • No-Privilege OPSD (No-Priv OPSD): Uses the exact same on-policy self-distillation loop as our method, except both teacher and student receive only (image, query) without intent.
  • Intent-Privilege OPSD (Ours): During training, the student takes only (image, query) and samples a single response on-policy. At each token step along the student's own generated prefix, the frozen teacher receives (image, query, intent) and computes its next-token probability distribution. The student is updated by minimizing the forward KL divergence between the teacher's intent-conditioned distribution and its own distribution. At inference, both the teacher and intent are discarded, and the student responds directly from (image, query).
BenchmarkBaseIntent-SFTResp-SFTNo-Priv OPSDOurs (OPSD)
SIUO+HoliSafe43.944.946.846.153.5
BeaverTails-V31.031.542.539.049.0
MOSSBench29.330.636.330.037.0

All values report Joint Success (%), the percentage of responses achieving both full safety compliance (S=3S=3) and constructive helpfulness (H≥2H \ge 2).

Key Findings & Mechanism Analysis:

  1. Predicting intent metadata alone does not calibrate responses (Intent-SFT vs. Base): Training the student to output structured safety annotations yields negligible gains over Base (35.7% vs. 34.7% average). Memorizing safety metadata as an auxiliary text-generation task does not teach the model how to adjust response boundaries when answering user queries.
  2. Off-policy imitation suffers from trajectory mismatch on implicit risks (Resp-SFT vs. Ours): While Resp-SFT improves notably over Base (41.9% vs. 34.7%), it still trails Ours by 6.7 points on SIUO+HoliSafe (46.8% vs. 53.5%) and 6.5 points on BeaverTails-V (42.5% vs. 49.0%).
  3. Why on-policy token-level distillation is critical: In Resp-SFT, the teacher writes complete responses while viewing intent (off-policy demonstrations). Because the student lacks intent at test time, any slight deviation in its early tokens on subtle cross-modal risks pushes it off the teacher's trajectory (exposure bias [1, 2]). In contrast, OPSD queries the intent-conditioned teacher along the student's own generated prefixes [2], providing dense token-level guidance on the exact phrasing paths the student naturally takes without privileged annotations.
AuthorsOpenReview ↗

Response to W2 & Q2

2. Response to W2 & Q2: Component-Level Ablations of Intent Annotations

To verify whether every part of our structured intent annotation is necessary, we decompose it into three functional pillars—Grounded Evidence, Epistemic Limits, and Output Effects—and conduct leave-one-out OPSD ablations under the exact same training setup:

BenchmarkBasew/o Grounded Evidencew/o Epistemic Limitsw/o Output EffectsOurs (Full)
SIUO+HoliSafe43.947.846.848.553.5
BeaverTails-V31.042.045.044.049.0
MOSSBench29.332.331.035.337.0

Removing any single component degrades Macro Average Joint Success by 3.9 to 5.8 points, confirming that all three components play distinct, non-redundant roles:

  1. Grounded Evidence (w/o Grounded Evidence) — Prevents Defensive Over-Refusal on Benign Queries

    • Category Definition: Records objective, observable visual facts, literal textual semantics, and their verified cross-modal relationship without speculative risk assumptions.
    • Quantitative Analysis: Removing Grounded Evidence produces the largest overall drop (-5.8 points average, 46.5% →\rightarrow 40.7%), including steep declines on SIUO+HoliSafe (-5.7) and BeaverTails-V (-7.0). Without explicit evidence confirming when a scene is benign and answerable, Unjustified Refusal rises from 8.26% to 10.42% (the highest among all variants), degrading helpfulness despite slightly lower harmful rates from heightened conservatism.
    • Qualitative Pattern: Given a benign photo of a person interacting with a dog and a request for training tips, the full model directly provides actionable training guidance, whereas w/o Grounded Evidence initially refuses ("cannot advise without knowing the training background") before offering only vague generalities.
  2. Epistemic Limits (w/o Epistemic Limits) — Calibrates Conditional Assistance under Ambiguity

    • Category Definition: Explicitly specifies unresolved situational conditions (e.g., missing context, authorization, or illegible visual details) alongside plausible benign interpretations when the evidence is non-unique.
    • Quantitative Analysis: Removing Epistemic Limits reduces average Joint Success by 5.6 points (46.5% →\rightarrow 40.9%), notably hurting SIUO+HoliSafe (-6.7) and MOSSBench (-6.0). Conditional responses drop from 8.63% to 7.21%, causing two-sided boundary failures rather than uniform overconfidence.
    • Qualitative Pattern: (i) On a financial document image with partially illegible text, the full model offers bounded conditional help ("if you are referring to financial planning..."), whereas w/o Epistemic Limits halts completely to demand clarification; (ii) conversely, on a cultural event photo, w/o Epistemic Limits fabricates unverified guesses about specific demographic groups.
  3. Output Effects (w/o Output Effects) — Enforces Operational Safety Boundaries on Concrete Advice

    • Category Definition: Defines conditional output boundaries specifying which concrete types of advice or information would materially enable harm under the observed conditions, alongside what remains safely answerable.
    • Quantitative Analysis: Removing Output Effects lowers average Joint Success by 3.9 points (46.5% →\rightarrow 42.6%), with a notable drop on BeaverTails-V (-5.0). Crucially, the Harmful response rate rises from 7.95% to 9.37% (the highest among all variants), showing that surface risk recognition becomes decoupled from actionable safety constraints.
    • Qualitative Pattern: The student still recognizes risky contexts and attaches superficial warnings, yet fails to constrain its concrete suggestions [3]—for example, explicitly recommending a busy street as a spot to sit on the ground for recreation, or providing step-by-step tips on how to eat while driving before adding "pull over first" only as an afterthought.

We will incorporate both comparison tables and this component-level behavioral analysis into Section 5.3 of the revised manuscript.

References

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.