- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the intent–action gap in multimodal large language models, where models can recover harmful intentions hidden across images, text, and task frames but still comply with unsafe requests. The authors introduce BIND-5K, a dataset and benchmark comprising matched harmful and benign examples to evaluate intent-aware safety decisions. They further propose BIND-Align, which combines intent-first supervised fine-tuning with reward-based reinforcement learning. Experiments across four MLLM backbones demonstrate improved jailbreak resistance while largely preserving helpfulness on benign requests.
Critical strengths
1. An interesting perspective on multimodal safety failures. The paper highlights the intent–action gap in MLLMs, showing that models may comply with harmful requests even after recovering their underlying intent. The analysis provides useful insights into multimodal safety failures.
2. A meaningful benchmark design with promising results. BIND-5K uses counterfactually matched examples to jointly evaluate safety and helpfulness, discouraging blanket refusal. Experiments across multiple backbones demonstrate improved jailbreak resistance while largely preserving benign helpfulness.
Critical weaknesses
W1. The paper doesn't show what each type of training unit contributes. BIND-5K mixes four kinds of units (dispersed/frame-based, scene edits, same-image pairs, and joint-harm), each meant to teach something different. The training-data ablation in Table 2a changes how units are used (paired vs. one per unit, fewer units, SFT in place of RL), but never removes a unit type. So we can't tell which types drive VoTA defense, which keep the model from refusing benign MOSS requests, which help on BIND, or whether the mixture beats any single type.
W2. The gap-aware reward may help simply because it punishes harmful compliance harder. Dropping the gap penalty lowers VoTA defense from 83.3 to 76.8 (Table 2b). But dropping it changes two things at once: the reward no longer depends on recovery evidence, and the total penalty for harmful compliance shrinks. The "gap from intent" variant keeps the penalty size and changes only where the evidence is read from. It does worse (64.3), and the paper takes this to mean the effect comes from the evidence (lines 455-457). I agree it shows the evidence source matters. It does not show that conditioning on recovery beats a plain, stronger penalty, because the two variants can fire on different samples. The missing control is a penalty of comparable strength that ignores recovery altogether.
Questions for authors
Q1. Could you ablate the unit types, for example by leaving one type out at a time, while holding total training size and compute fixed? VoTA, MOSS, and BIND for each run would show whether the types really complement each other.
Q2. Could you add a baseline that adds a harmful-compliance penalty independent of recovery evidence? It should start from the same SFT checkpoint, use the same data and budget, and have its weight tuned as carefully as the gap penalty's. If the gap-aware reward still wins, that would directly support the intent-action binding claim.
Q3. The benchmark has 944 units (Appendix G), but Table 1 uses 91 of them: the ones that passed a validity check out of 114 drawn (lines 1100-1101). How does this subset compare with the full benchmark in topic, difficulty, and expected action (answer, adapt, decline)? With 91 units, a single unit moves accuracy by about 1.1 points, so unit-level confidence intervals would help. Could you also report results on a larger validated subset?
Q4. On Qwen3-VL-8B, GRPO lowers BIND accuracy from 54.9 to 44.0 while VoTA defense barely moves (83.3 to 84.5; Table 14). Which expected actions does the model get wrong after GRPO: answer, adapt, or decline? Is the drop larger than seed-to-seed variation? If it holds up, does it mean GRPO trades context-sensitive decisions for attack resistance?
6 comments
Q2: The requested uniform-penalty control, from the same start with three seeds
Q2: Add a harmful-compliance penalty that ignores recovery evidence, from the same SFT checkpoint and budget.
We added it as you describe: same SFT checkpoint, RL data, steps, batch size and rollout budget, with −2.2 on every harmful compliance, the largest penalty BIND-Align applies. All other reward components, including the treatment of benign refusals, are unchanged, and neither weight was tuned on evaluation data. We re-ran every reward ablation with three seeds after re-checking the reward implementation; these runs replace the single-seed rows of Table 2b.
| GRPO reward (4B, three-seed means) | C_h ↑ | VoTA defence ↑ | MOSS not refused ↑ | S ↑ | Unseen carriers: over-refusal ↓ |
|---|---|---|---|---|---|
| No gap penalty (−1.4) | 83.7 | 83.7 | 85.1 | 84.3 | 37.6 |
| Safety-only (action reward alone) | 88.6 | 88.6 | 86.6 | 87.5 | 40.7 |
| Gap-aware, but recovery read from the intent | 82.8 | 82.7 | 87.1 | 84.6 | 39.4 |
| Uniform −2.2 (your control) | 82.2 | 82.0 | 89.4 | 85.5 | 30.1 |
| Gap-aware (BIND-Align) | 82.6 | 82.5 | 88.7 | 85.4 | 29.0 |
C_h: defence on items whose task the model recovers in the neutral probe. Bold: best per column. CIs below are paired item bootstrap.
1. The uniform control and the gap-aware reward give similar results (C_h difference +0.4 [−1.7, 2.5]), so this control does not establish an additional benefit from conditioning the penalty on recovery evidence. An audit of the starting checkpoint, written down before these runs, is consistent with this: on its RL rollouts, 89.8% of harmful answers already state the recovered task in the answer (95% CI 87.1 to 92.3), so the two penalties give the same value to most of them. The 89.8% is itself a measurement of the intent–action gap: about nine in ten remaining failures state the task and then comply.
2. The variants trade attack defence against over-refusal. Safety-only gains 6 points of VoTA defence but refuses 11.6 points more benign inputs on unseen carriers [6.1, 17.2], with similar attack success there (3.3% vs 3.5%). The gap-aware and uniform rewards have the two lowest over-refusal rates (29.0% and 30.1%).
We will update Table 2b and the paragraph that attributes the gain to conditioning on recovery evidence.
Q3: On 161 new units validated in advance, the gain over base holds on every backbone
Q3: Comparison of the 91-unit subset with the full benchmark, and a larger validated subset.
We expanded the evaluated set from 91 to 252 validated units, including 161 new units whose validity was locked before generation.
1. Prospective validation and full gate coverage. We drew 202 more units with the same stratified sampler and applied the same blind three-vote gate. 161 passed (79.7%, vs 79.8% originally), and the valid list was locked before any model generated on them. We have also gated all 944 units: 769 pass. We will release this validated split with the gate votes and code. The model results below cover the 252 evaluated units.
2. The base → BIND-Align comparison extends to the new units:
| Unit accuracy (%) | 4B | 8B | 32B | InternVL3-8B |
|---|---|---|---|---|
| Original 91 | 37.4 → 37.4 | 34.1 → 44.0 | 52.7 → 58.2 | 18.7 → 46.2 |
| New 161 | 31.7 → 39.8 | 41.6 → 49.1 | 55.3 → 60.2 | 19.9 → 46.0 |
| Union 252 | 33.7 → 38.9 | 38.9 → 47.2 | 54.4 → 59.5 | 19.4 → 46.0 |
| Union 252: difference [95% CI] | +5.2 [−2.0, 12.7] | +8.3 [0.4, 16.3] | +5.2 [−2.8, 13.1] | +26.6 [19.4, 33.7] |
CIs resample whole units and keep the pairing between methods; bold differences have CIs above zero. 32B base on the original 91 is 48/91 = 52.7; the 52.8 in Table 1 was a rounding error and is corrected in the revision.
On the new, prospectively validated units, BIND-Align is above base on all four backbones. On the 4B union, the refusal-based defences stay near zero (VLGuard 0.8, TiS 5.2, AdaShield 5.6), each 33 to 38 points below BIND-Align.
3. The gate changes the action and topic composition:
| Composition (%) | All 944 | Original 91 | New 161 | Union 252 |
|---|---|---|---|---|
| Units whose correct action is decline | 16.7 | 9.9 | 12.4 | 11.5 |
| Cells whose correct action is answer / adapt | 72.2 / 23.2 | 71.4 / 25.8 | 71.7 / 24.8 | 71.6 / 25.2 |
| Topics covered (of 24) | 24 | 19 | 20 | 21 |
Units whose correct action is decline pass the gate less often than those requiring adaptation (66% vs 84%), so they make up 13.7% of all 769 validated units, close to the 11.5% of the evaluated union. The union still differs from the 944 in topic and action mix (p = 0.015 and 0.010). As a sensitivity check, reweighting the 91 to the full benchmark's action and unit-size mix changes no accuracy by more than 2.5 points.
4. Difficulty. The available checks show no detectable difference: on the original sample, base models score 35.7 on units that passed and 40.2 on units that did not (p = 0.61), and build-time measures (gate vote margins, edit retries, prompt length) do not differ between validated and unsampled units (all p > 0.12).
We will add the expanded results with accuracy and paired-difference CIs and the topic and action breakdowns, and distinguish the full pool, the validated pool and the evaluated subset.
Q1: At a fixed budget, removing any unit type lowers attack defence
Thank you for the precise, constructive review.
The types complement each other. Dispersed units teach intent recovery, scene and joint-harm units teach acting on it, and scene edits also teach context-sensitive answers. We retrained intent-first SFT on 4B with each unit type removed and up-sampled the remaining types, so every run has the same number of rows, batches and steps (full mixture: three seeds; each removal: two seeds).
Removing same-image or joint-harm units also shifts the model slightly towards answering: MOSS and BIND rise while defence falls. The full mixture gives the highest attack defence with comparable benign non-refusal.
We will add these runs, with the share of refusal targets in each, to the appendix.