acceptodds
Under review as a conference paper at ICLR 2027

Seeing Through, Still Complying: Binding Multimodal Safety to the Underlying Intent

Abstract

Multimodal large language models (MLLMs) have made rapid progress in visual reasoning, enabling them to solve complex real-world tasks across images and text. This growing capability, however, also expands their attack surface. Recent multimodal jailbreaks conceal harmful requests by dispersing the intent across images, text, and task frames, leaving the model's own reasoning to reconstruct it. Modern MLLMs do reconstruct the intent, yet a gap remains between the intent they recover and the action they take. This gap is largest for the most dispersed attacks: with the model's own reconstructed intent held fixed, the attack frame alone suffices to elicit harmful content, and restating the intent in the prompt does not restore safety against the most dispersed attacks. These findings suggest that safety decisions should track the jointly formed intent rather than the surface frame through which it is presented. Guided by this principle, we construct BIND-5K, in which every harmful case is paired with benign counterparts that share either its image or its request. Its held-out benchmark scores each group of matched inputs as a single unit, granting credit only when every decision within it is correct. Unlike existing multimodal safety benchmarks that score each input independently, BIND-5K cannot be passed by uniformly refusing or answering all requests. On its training splits, we develop BIND-Align, which teaches the model to recover the joint intent before answering and optimizes its safety decision with a reward that explicitly couples behavior to that intent. Across four backbones, BIND-Align consistently balances resistance to dispersed attacks with helpful responses to benign requests, while defenses that perform strongly on attack-only benchmarks through broad refusal degrade substantially under our paired evaluation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multimodal large language models, multimodal safety, safety alignment

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WkzyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the intent–action gap in multimodal large language models, where models can recover harmful intentions hidden across images, text, and task frames but still comply with unsafe requests. The authors introduce BIND-5K, a dataset and benchmark comprising matched harmful and benign examples to evaluate intent-aware safety decisions. They further propose BIND-Align, which combines intent-first supervised fine-tuning with reward-based reinforcement learning. Experiments across four MLLM backbones demonstrate improved jailbreak resistance while largely preserving helpfulness on benign requests.

Critical strengths

1. An interesting perspective on multimodal safety failures. The paper highlights the intent–action gap in MLLMs, showing that models may comply with harmful requests even after recovering their underlying intent. The analysis provides useful insights into multimodal safety failures.

2. A meaningful benchmark design with promising results. BIND-5K uses counterfactually matched examples to jointly evaluate safety and helpfulness, discouraging blanket refusal. Experiments across multiple backbones demonstrate improved jailbreak resistance while largely preserving benign helpfulness.

Critical weaknesses

W1. The paper doesn't show what each type of training unit contributes. BIND-5K mixes four kinds of units (dispersed/frame-based, scene edits, same-image pairs, and joint-harm), each meant to teach something different. The training-data ablation in Table 2a changes how units are used (paired vs. one per unit, fewer units, SFT in place of RL), but never removes a unit type. So we can't tell which types drive VoTA defense, which keep the model from refusing benign MOSS requests, which help on BIND, or whether the mixture beats any single type.

W2. The gap-aware reward may help simply because it punishes harmful compliance harder. Dropping the gap penalty lowers VoTA defense from 83.3 to 76.8 (Table 2b). But dropping it changes two things at once: the reward no longer depends on recovery evidence, and the total penalty for harmful compliance shrinks. The "gap from intent" variant keeps the penalty size and changes only where the evidence is read from. It does worse (64.3), and the paper takes this to mean the effect comes from the evidence (lines 455-457). I agree it shows the evidence source matters. It does not show that conditioning on recovery beats a plain, stronger penalty, because the two variants can fire on different samples. The missing control is a penalty of comparable strength that ignores recovery altogether.

Questions for authors

Q1. Could you ablate the unit types, for example by leaving one type out at a time, while holding total training size and compute fixed? VoTA, MOSS, and BIND for each run would show whether the types really complement each other.

Q2. Could you add a baseline that adds a harmful-compliance penalty independent of recovery evidence? It should start from the same SFT checkpoint, use the same data and budget, and have its weight tuned as carefully as the gap penalty's. If the gap-aware reward still wins, that would directly support the intent-action binding claim.

Q3. The benchmark has 944 units (Appendix G), but Table 1 uses 91 of them: the ones that passed a validity check out of 114 drawn (lines 1100-1101). How does this subset compare with the full benchmark in topic, difficulty, and expected action (answer, adapt, decline)? With 91 units, a single unit moves accuracy by about 1.1 points, so unit-level confidence intervals would help. Could you also report results on a larger validated subset?

Q4. On Qwen3-VL-8B, GRPO lowers BIND accuracy from 54.9 to 44.0 while VoTA defense barely moves (83.3 to 84.5; Table 14). Which expected actions does the model get wrong after GRPO: answer, adapt, or decline? Is the drop larger than seed-to-seed variation? If it holds up, does it mean GRPO trades context-sensitive decisions for attack resistance?

6 comments
AuthorsOpenReview ↗

Q1: At a fixed budget, removing any unit type lowers attack defence

Thank you for the precise, constructive review.

Q1: Leave one unit type out at fixed size and compute, and test whether the types complement each other.

The types complement each other. Dispersed units teach intent recovery, scene and joint-harm units teach acting on it, and scene edits also teach context-sensitive answers. We retrained intent-first SFT on 4B with each unit type removed and up-sampled the remaining types, so every run has the same number of rows, batches and steps (full mixture: three seeds; each removal: two seeds).

SFT data (4B)VoTA defence ↑MOSS not refused ↑BIND ↑Intent recovered ↑
Full mixture60.3 (49.8–75.8)89.233.398.2
− dispersed / frame units9.0 (8.5–9.5)88.743.478.9
− scene edits47.7 (both seeds)90.224.297.9
− same-image pairs41.1 (34.3–47.8)91.238.595.8
− joint-harm units45.6 (44.3–46.8)91.238.597.8
  • Every type contributes to attack resistance. All eight removal runs fall below the lowest full-mixture run on VoTA defence.
  • Dispersed units drive recovery. Removing them gives by far the largest drop in intent recovery (98 → 79). This removal also halves the share of refusal targets in training, so its defence drop is not due to the format alone.
  • Recovering the intent does not by itself produce the action. Without scene edits or joint-harm units, recovery stays near 98%, but defence drops to 47.7 and 45.6. Their share of refusal targets is higher (scene) or about the same (joint-harm) as in the full mixture, so fewer refusal targets do not explain the drop. This is the intent–action gap.
  • Scene edits support context-sensitive answers. Without them, BIND is lowest (24.2).

Removing same-image or joint-harm units also shifts the model slightly towards answering: MOSS and BIND rise while defence falls. The full mixture gives the highest attack defence with comparable benign non-refusal.

We will add these runs, with the share of refusal targets in each, to the appendix.

AuthorsOpenReview ↗

Q2: The requested uniform-penalty control, from the same start with three seeds

Q2: Add a harmful-compliance penalty that ignores recovery evidence, from the same SFT checkpoint and budget.

We added it as you describe: same SFT checkpoint, RL data, steps, batch size and rollout budget, with −2.2 on every harmful compliance, the largest penalty BIND-Align applies. All other reward components, including the treatment of benign refusals, are unchanged, and neither weight was tuned on evaluation data. We re-ran every reward ablation with three seeds after re-checking the reward implementation; these runs replace the single-seed rows of Table 2b.

GRPO reward (4B, three-seed means)C_h ↑VoTA defence ↑MOSS not refused ↑S ↑Unseen carriers: over-refusal ↓
No gap penalty (−1.4)83.783.785.184.337.6
Safety-only (action reward alone)88.688.686.687.540.7
Gap-aware, but recovery read from the intent82.882.787.184.639.4
Uniform −2.2 (your control)82.282.089.485.530.1
Gap-aware (BIND-Align)82.682.588.785.429.0

C_h: defence on items whose task the model recovers in the neutral probe. Bold: best per column. CIs below are paired item bootstrap.

1. The uniform control and the gap-aware reward give similar results (C_h difference +0.4 [−1.7, 2.5]), so this control does not establish an additional benefit from conditioning the penalty on recovery evidence. An audit of the starting checkpoint, written down before these runs, is consistent with this: on its RL rollouts, 89.8% of harmful answers already state the recovered task in the answer (95% CI 87.1 to 92.3), so the two penalties give the same value to most of them. The 89.8% is itself a measurement of the intent–action gap: about nine in ten remaining failures state the task and then comply.

2. The variants trade attack defence against over-refusal. Safety-only gains 6 points of VoTA defence but refuses 11.6 points more benign inputs on unseen carriers [6.1, 17.2], with similar attack success there (3.3% vs 3.5%). The gap-aware and uniform rewards have the two lowest over-refusal rates (29.0% and 30.1%).

We will update Table 2b and the paragraph that attributes the gain to conditioning on recovery evidence.

AuthorsOpenReview ↗

Q3: On 161 new units validated in advance, the gain over base holds on every backbone

Q3: Comparison of the 91-unit subset with the full benchmark, and a larger validated subset.

We expanded the evaluated set from 91 to 252 validated units, including 161 new units whose validity was locked before generation.

1. Prospective validation and full gate coverage. We drew 202 more units with the same stratified sampler and applied the same blind three-vote gate. 161 passed (79.7%, vs 79.8% originally), and the valid list was locked before any model generated on them. We have also gated all 944 units: 769 pass. We will release this validated split with the gate votes and code. The model results below cover the 252 evaluated units.

2. The base → BIND-Align comparison extends to the new units:

Unit accuracy (%)4B8B32BInternVL3-8B
Original 9137.4 → 37.434.1 → 44.052.7 → 58.218.7 → 46.2
New 16131.7 → 39.841.6 → 49.155.3 → 60.219.9 → 46.0
Union 25233.7 → 38.938.9 → 47.254.4 → 59.519.4 → 46.0
Union 252: difference [95% CI]+5.2 [−2.0, 12.7]+8.3 [0.4, 16.3]+5.2 [−2.8, 13.1]+26.6 [19.4, 33.7]

CIs resample whole units and keep the pairing between methods; bold differences have CIs above zero. 32B base on the original 91 is 48/91 = 52.7; the 52.8 in Table 1 was a rounding error and is corrected in the revision.

On the new, prospectively validated units, BIND-Align is above base on all four backbones. On the 4B union, the refusal-based defences stay near zero (VLGuard 0.8, TiS 5.2, AdaShield 5.6), each 33 to 38 points below BIND-Align.

3. The gate changes the action and topic composition:

Composition (%)All 944Original 91New 161Union 252
Units whose correct action is decline16.79.912.411.5
Cells whose correct action is answer / adapt72.2 / 23.271.4 / 25.871.7 / 24.871.6 / 25.2
Topics covered (of 24)24192021

Units whose correct action is decline pass the gate less often than those requiring adaptation (66% vs 84%), so they make up 13.7% of all 769 validated units, close to the 11.5% of the evaluated union. The union still differs from the 944 in topic and action mix (p = 0.015 and 0.010). As a sensitivity check, reweighting the 91 to the full benchmark's action and unit-size mix changes no accuracy by more than 2.5 points.

4. Difficulty. The available checks show no detectable difference: on the original sample, base models score 35.7 on units that passed and 40.2 on units that did not (p = 0.61), and build-time measures (gate vote margins, edit retries, prompt length) do not differ between validated and unsampled units (all p > 0.12).

We will add the expanded results with accuracy and paired-difference CIs and the topic and action breakdowns, and distinguish the full pool, the validated pool and the evaluated subset.

Reviewer_QwKyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the intent–action gap in multimodal large language models under dispersed jailbreak attacks, where models can reconstruct harmful intent from visual and textual cues but still generate unsafe responses. The authors analyze this phenomenon through comparisons across attack families, controlled changes to task framing, and explicit intent-restatement experiments. To address this issue, the paper introduces BIND-5K, a dataset and benchmark consisting of matched counterfactual inputs that jointly evaluate safety decisions and appropriate responses to benign requests. The authors further propose BIND-Align, which combines intent-first supervised fine-tuning with gap-aware reinforcement learning to encourage safety decisions that follow the recovered intent. Experiments on four multimodal backbones evaluate attack resistance, benign-request handling, general capabilities, component ablations, and generalization to unseen attack carriers.

Critical strengths

I think this paper provides meaningful empirical evidence that task framing can substantially affect safety decisions even when the underlying harmful intent is recoverable. In particular, the within-VoTA framing experiments in Figure 2 and Appendix A offer a useful analysis beyond simply reporting attack success rates. These findings highlight an important practical vulnerability in modern multimodal models. BIND-5K evaluates matched inputs jointly, discouraging both indiscriminate compliance and blanket refusal. This provides a useful complement to existing instance-level safety evaluations. Moreover, BIND-Align achieves considerable improvements in the balance between jailbreak resistance and benign-request handling across four backbones (Table 1). These empirical results demonstrate the practical value of the proposed training and evaluation pipeline.

Critical weaknesses

W1. Insufficient novelty and evidence for the proposed intent–action gap. The distinction between harmfulness recognition and refusal behavior has been explored in prior work, and the conceptual advance of the proposed intent–action gap remains unclear. Moreover, intent recovery is primarily measured through separate neutral probes (Appendices A–C), which demonstrate recoverability but not necessarily recognition during the original generation. Although the framing experiments support a behavioral discrepancy, they do not establish a distinct underlying failure mechanism.

W2. Unclear independent contribution of gap-aware reinforcement learning. BIND-Align combines intent-first SFT with GRPO, but Figure 5 and Table 14 show that SFT accounts for most improvements. GRPO provides modest gains in the average balance score (81.9 to 83.6), while slightly reducing average BIND performance (46.7 to 46.4). Although Table 2(b) supports the proposed reward on one backbone, the single-model, single-seed ablations leave its generalizable contribution insufficiently established.

Questions for authors

Q1. Novelty of the intent–action gap. How does the proposed intent–action gap fundamentally differ from prior findings on the separation between harmfulness recognition and refusal behavior (e.g., Zhao et al., 2025 [1])? What new understanding does the paper establish beyond demonstrating this phenomenon in dispersed multimodal jailbreaks?

Q2. Evidence for intent recovery during unsafe generation. Intent recovery is largely measured through separate neutral probes (Appendices A–C). Can the authors provide stronger evidence that models recognize the harmful intent during the original unsafe generation? For instance, controlling dispersion and task framing for matched underlying requests would help distinguish the proposed gap from differences between probing and generation conditions.

Q3. Attribution of BIND-Align's performance gains. Figure 5 and Table 14 suggest that most improvements come from SFT, while GRPO sometimes reduces BIND performance. Can the authors explain these regressions and validate the gap-aware reward ablations across additional backbones or seeds? A controlled comparison under matched training data and budgets would help establish its independent contribution.

[1]. LLMs Encode Harmfulness and Refusal Separately. NeurIPS 2025.

6 comments
AuthorsOpenReview ↗

Q1: The gap concerns acting on a reconstructed task, not refusal alone

Thank you for the careful reading and for highlighting the framing analysis.

Q1: Difference between the intent–action gap and the harmfulness/refusal separation of Zhao et al. (2025).

Zhao et al. show that harmfulness and refusal are encoded separately, and that some jailbreaks suppress refusal while the harmfulness signal remains. We build on this distinction and study what it leaves open: whether the model's action follows a harmful task that it must first reconstruct across modalities. In dispersed attacks no single input states the request; the model composes it from the image, the text fragments and the game framing.

1. Making the recovered request explicit does not neutralise the attack. On items whose intent each model recovers, we give the model its own recovered request with the original image, either directly or appended to the original attack (Table 9, mean of five models):

(%)MM-SafetyBenchFigStepVoTA
Intent recovered798090
Harmful response: recovered request given directly131318
Harmful response: recovered request appended to the attack223494

The increase is much larger on VoTA (18% → 94%) than on the other two attacks. Compared with the original VoTA attack alone, appending the recovered request does not help: safe responses go from 7% to 6% (Table 6). Each model mostly does not carry out its own recovered request when it is stated alone, yet carries it out inside the attack.

2. The unsafe response itself states the hidden task. In 77% of successful VoTA attacks on four base models, the harmful answer states the hidden task, with no separate probe (details in Q2).

3. Restoring refusal alone does not close the gap. The model must answer, adapt or decline as each scene requires. VoTA measures attack resistance; BIND's matched units check this choice. Defences that mostly refuse score high on the first and near zero on the second (Table 1, mean of four backbones):

Mean of four backbones (%)VoTA defence ↑BIND ↑
Base29.235.8
AdaShield99.41.9
VLGuard90.70.3
TiS97.66.6
BIND-Align81.646.4

We will clarify this positioning relative to Zhao et al. in the related work section.

AuthorsOpenReview ↗

Q2: The original unsafe answers already state the hidden task

Q2: Evidence that the intent is recognised during the original unsafe generation, not only under separate probes.

The original generation shows it. We examined the original attack responses themselves and, for a reasoning model, the trace before each answer. No separate probe is involved.

1. The unsafe answers themselves state the hidden task. Across 5,365 successful VoTA attacks on the four base models of Table 1:

Among successful VoTA attacksShare (%)
The answer states the hidden task (judge)77.0
The answer contains actionable harmful content91.8
The task title is found by a judge-free lexical check77.4

The lexical check gives a similar aggregate rate without any judge.

2. Risk is explicitly acknowledged in traces before harmful answers. For the reasoning model Qwen3.8-27B on VoTA, the trace before a harmful answer names the activity in 100% of cases, states that it is harmful, illegal or against policy in 78%, and treats the game or story as the context to follow in 97% (Table 11). We use these as evidence of what the trace states, not as a causal account.

3. Reconstruction stays frequent across frame conditions, while attack success changes sharply. Table 5 keeps the image panels and text fragments fixed and varies the game frame and the requested detail (mean of five models):

VoTA conditionHidden task stated (%)Attack success (%)
No frame, brief answer725
Frame, brief answer8166
No frame, detailed answer9123
Frame, detailed answer8377

The gap is therefore observable during the original generation: task reconstruction appears in the unsafe answers themselves, and explicit risk acknowledgement also appears in the reasoning traces before harmful answers.

We will add this audit, per model, to the appendix.

AuthorsOpenReview ↗

Q3: The RL stage improves attack defence over fine-tuning in every seed

Q3: Contribution of GRPO beyond SFT, across seeds or backbones.

As the paper states (§4), fine-tuning provides most of the gain and GRPO adds to it on attack inputs. We re-ran every reward ablation with three seeds after re-checking the reward implementation; these runs replace the single-seed rows.

1. All three 4B GRPO seeds improve VoTA defence and S over the same SFT checkpoint, and MOSS non-refusal stays at the SFT level on average (new evaluation; VoTA scored with HarmBench):

4B (%)SFT (start)+ GRPO, mean (seed range)Change [95% CI]
VoTA defence ↑76.882.5 (78.8–87.2)+5.7 [2.4, 9.1]
MOSS not refused ↑88.788.7 (88.0–89.7)0.0 [−3.0, 2.9]
S ↑82.385.4 (83.3–87.6)+3.1 [0.9, 5.5]
BIND ↑29.737.7 (36.3–39.6)+8.1 [0.0, 16.5]

Ranges are across GRPO seeds; CIs are paired item bootstrap.

2. The reward ablations show a trade-off between attack defence and over-refusal (4B, same SFT checkpoint, data, steps and rollout budget; three-seed means; bold = best per column):

GRPO rewardVoTA defence ↑MOSS not refused ↑S ↑Unseen carriers: over-refusal ↓
No gap penalty83.785.184.337.6
Safety-only (action reward alone)88.686.687.540.7
Gap-aware, but recovery read from the intent82.787.184.639.4
Uniform penalty of matched strength82.089.485.530.1
Gap-aware (BIND-Align)82.588.785.429.0

Safety-only gains 6 points of VoTA defence but refuses 11.6 points more benign inputs on unseen carriers [6.1, 17.2], with similar attack success there (3.3% vs 3.5%). The gap-aware and uniform rewards perform on par: on the SFT checkpoint's rollouts, 89.8% of harmful answers already state the recovered task, so the two penalties act on largely the same outputs, and we do not attribute an additional benefit to conditioning the penalty on this evidence.

3. Across backbones, GRPO raises VoTA and S on average; the BIND decrease comes mainly from 8B (Table 14, one run per backbone):

BackboneVoTA changeS changeBIND: SFT → GRPO
Qwen3-VL-4B+10.0+5.329.7 → 37.4
Qwen3-VL-8B+1.2+0.254.9 → 44.0
Qwen3-VL-32B+5.5+3.059.3 → 58.2
InternVL3-8B−0.8−2.042.9 → 46.2
Mean+3.9+1.746.7 → 46.4

The 8B errors after GRPO are mainly missed adaptations on benign requests whose image shows a hazard, not added refusals. On 161 new, prospectively validated units, the 8B drop does not recur: GRPO − SFT = +1.2 [−6.8, 8.7].

In the revision, we will add the three-seed 4B results, with per-seed values in the appendix.

Reviewer_KUx2OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies an intent–action gap in multimodal jailbreaks, where MLLMs recognize harmful intentions but still comply with them. To address this issue, the authors propose BIND-5K, a benchmark of matched harmful and benign inputs, and BIND-Align, which combines intent-first fine-tuning with gap-aware GRPO. Experiments across four MLLMs demonstrate improved jailbreak resistance while preserving benign helpfulness.

Critical strengths
  1. Interesting motivation and problem formulation. The paper identifies an important failure mode in multimodal safety alignment. The analysis of the intent–action gap provides useful insights into why existing multimodal jailbreak attacks remain effective even against models with strong reasoning capabilities.
  2. Strong empirical performance. The proposed BIND-Align consistently improves defense performance across several MLLM backbones on several multimodal jailbreak benchmarks.
Critical weaknesses
  1. Insufficient comparison with data-matched safety alignment baselines. The paper compares BIND-Align with several existing safety defenses, including VLGuard, TiS, and SPA-VL. However, these training-based baselines use their own training datasets, while BIND-Align benefits from the newly constructed BIND-5K dataset. Therefore, the observed improvements may partially originate from the paired training data rather than the proposed intent-aware alignment strategy. Although the authors provide component ablations, a comparison in which existing alignment methods are trained on the same BIND-5K data would better isolate the methodological contribution.
  2. Inconsistent generalization to unseen carriers across backbones. The paper reports strong generalization to six unseen carrier families. However, the per-backbone results in Appendix Table 15 reveal substantial variation in benign helpfulness. For example, BIND-Align exhibits a 37.9% over-refusal rate on benign requests with unseen carriers for Qwen3-VL-4B, compared with 0.0% for Qwen3-VL-8B and 2.3% for InternVL3-8B. This considerable discrepancy suggests that the claimed generalization of intent-aware safety decisions may not be equally reliable across different backbone models.
  3. Missing analysis of inference overhead introduced by explicit intent reconstruction. BIND-Align requires the model to explicitly reconstruct the underlying intent before generating its final response. While this design improves safety alignment, it introduces additional generation tokens and potentially increases inference latency. The paper evaluates safety performance and general capabilities but does not report the computational overhead of the proposed intent-first inference process. Understanding the additional latency and token costs would help assess the practicality of deploying BIND-Align in real-world multimodal applications.
Questions for authors
  1. According to weakness 1, could the authors compare BIND-Align with existing safety alignment methods trained on the same BIND-5K dataset? This would help clarify whether the observed improvements primarily stem from the proposed intent-aware alignment strategy or the paired training data.
  2. According to weakness 2, could the authors explain the substantial variation in over-refusal rates across different backbones on unseen carriers, particularly the 37.9% over-refusal rate of Qwen3-VL-4B reported in Appendix Table 15? Further analysis would help establish the robustness and consistency of the claimed generalization capabilities.
  3. According to weakness 3, could the authors report the additional inference latency and token consumption introduced by explicit intent reconstruction compared with the original backbone models? This would help assess the computational efficiency and practical deployment costs of BIND-Align.
6 comments
AuthorsOpenReview ↗

Q1: Existing recipes trained on our data fall short, especially on unseen carriers

Thank you for the positive assessment and the concrete questions.

Q1: Compare with existing alignment methods trained on the same BIND-5K data.

We trained three baselines on BIND-5K data with Qwen3-VL-4B: answer-only SFT, and adaptations of TiS and of SPA-VL's DPO, started from the base model. (In Table 1, TiS and SPA-VL were trained on their own datasets.) All rows are evaluated on the same items; BIND-Align is the mean of three GRPO seeds.

Training on BIND-5K (4B)VoTA defence ↑MOSS not refused ↑S ↑Unseen carriers: attack success ↓
SFT without <intent> (answers only)32.773.345.23.8
TiS-style training60.773.366.469.7
SPA-VL-style DPO from the base model80.282.381.275.0
BIND-Align (3 seeds)82.588.785.43.5

Access to BIND-5K data alone does not reproduce BIND-Align.

  • Answer-only SFT falls short on dispersed attacks (32.7 vs 82.5), consistent with the paper's component ablation (31.3 without <intent>).
  • Strong VoTA defence does not guarantee transfer. SPA-VL-style DPO reaches 80.2 on VoTA but carries out 75.0% of harmful requests on carrier families absent from training, and TiS-style training 69.7%, against 3.5% for BIND-Align.
  • Only BIND-Align has both strong defence and low attack success on unseen carriers. Answer-only SFT also keeps attack success on unseen carriers low (3.8%) but defends only 32.7% of VoTA attacks; BIND-Align has the highest S (85.4) and 3.5%.

We will add this table to the revision and document how each recipe was adapted.

AuthorsOpenReview ↗

Q2: The 4B rate partly reflects seed variation and two carrier formats

Q2: 4B over-refusal on unseen carriers (37.9%) is high compared with the other backbones.

The 37.9% is the highest of three 4B seeds, and the refusals concentrate in two carrier formats. 4B nonetheless has the highest over-refusal of the four backbones, and the revision will state this.

1. Part of the 37.9% is seed variation. Three GRPO seeds from the same fine-tuned start give:

Qwen3-VL-4B, unseen carriers (%)Seed 1 (reported)Seed 2Seed 3Mean
Over-refusal ↓37.930.318.929.0
Attack success ↓4.54.51.53.5

The reported run is the highest of the three, and the lowest seed halves it. The other backbones refuse 0.0% (8B), 8.3% (32B) and 2.3% (InternVL3-8B) of benign twins, with attack success between 0.8% and 9.8%.

2. The 4B refusals concentrate in two carrier formats. Protocols and support tickets account for half of them, and 88% of these refusals still give partial help. The stated reasons target the format (e.g. writing a procedure "for an operator"), not the task.

3. Across backbones, BIND-Align refuses fewer benign inputs than TiS and VLGuard, with similarly low attack success. Means of Tables 1 and 15, one run per backbone for every method (including 37.9% for BIND-Align on 4B):

Mean of four backbones (%)MOSSBench: over-refusal ↓Unseen carriers: over-refusal ↓Unseen carriers: attack success ↓
Base9.70.075.9
SPA-VL10.90.073.1
VLGuard92.477.85.3
TiS65.315.97.0
BIND-Align14.212.14.9

On unseen carriers, Base and SPA-VL refuse no benign inputs but carry out about three quarters of the harmful ones.

4. The reward choice matters. On 4B, a safety-only reward reaches 40.7% over-refusal on unseen carriers, 11.6 points above the gap-aware reward [6.1, 17.2]; a uniform penalty of matched strength gives a similar 30.1%.

In the revision, we will add the three-seed 4B results (per-seed values in the appendix) and the carrier breakdown, and qualify the generalisation claim: attack success stays low on every backbone, while benign over-refusal depends on the backbone and is highest for 4B on unseen carriers.

AuthorsOpenReview ↗

Q3: No extra model call, and faster than the base model on safety inputs

Q3: Latency and token overhead of intent reconstruction.

Intent reconstruction requires no additional model call or prompt tokens. The <intent> block is generated within the same response and averages 96 tokens on the safety sets across four backbones. On these sets, because the answer that follows is shorter, the full response, intent included, averages 309 tokens versus 618 for Base.

Measured inference time for every method in Table 1 (Qwen3-VL-4B, one H100, batch 16, a fixed sample of each set; time per input amortised over the batch; ratio to Base in parentheses). The last column is the full MOSSBench result from Table 1 for the same checkpoints.

Method (4B)Model calls per inputVoTAMOSSBenchMOSS not refused (%, Table 1) ↑
Base10.754 s0.681 s87.7
ECSO2.0–2.31.554 s (2.06×)0.930 s (1.36×)89.0
AdaShield10.421 s (0.56×)0.260 s (0.38×)18.3
VLGuard10.686 s (0.91×)0.127 s (0.19×)8.3
TiS10.539 s (0.72×)0.368 s (0.54×)33.0
SPA-VL10.759 s (1.01×)0.683 s (1.00×)86.3
BIND-Align10.457 s (0.61×)0.403 s (0.59×)88.0

BIND-Align reduces amortised inference time by 39 to 41% on VoTA and MOSSBench while keeping the base model's MOSSBench non-refusal rate (88.0% vs 87.7%). The lower times of AdaShield, VLGuard and TiS come with much lower non-refusal rates (8.3 to 33.0%). ECSO keeps a high non-refusal rate but takes 1.36 to 2.06× the base time. We will add the full protocol and per-backbone token counts to the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.