- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether an LVLM should execute a task visible in an image, seperating current-user authorization from conditional safety. VTID-BENCH contains 18,000 authorization examples organized into controlled U/AS/AU triads and 12,000 ordinary vision-language examples. VTID-SFT combines cause-specific response supervision, auxiliary authorization/safety heads, weak geometric constraints, and ordinary-response anchoring. Experiments cover prompting, three separately trained backbones, ablations, and external safety/capability tests. The strongest evidence concerns learning this constructed response policy and improving authorized-safe completion; broader authorization generalization remains less established.
Critical strengths
- I find the controlled separation of delegation and harmfulness useful: Section 4.1 varies authorization for a benign visual task and safety under the same delegating prompt, avoiding the assumption that every unauthorized instruction must itself be harmful. The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions already trains conditional compliance with aligned lower-priority instructions; VTID’s added value is a visual delegation benchmark with explicit safe-completion and unsafe-refusal contrasts, rather than introducing authority-sensitive instruction following from scratch.
- The gains extends beyond one backbone and include meaningful outcome changes. Table 2 raises Qwen/InternVL authorized-safe Exact from 33.33%/33.67% to 94.33%/89.78%; Table 20 reduces unauthorized execution-evidence counts from 597/602 to 42/10 out of 900. Three LLaVA seeds, paired group bootstraps, the matched-budget data control, and the blinded three-rater audit strengthen the evidence. External LLaVA safety improves while retaining 95.9–99.4% of the reported capability scores, providing useful checks against indiscriminate refusal.
Critical weaknesses
- Table 20 shows LLaVA’s U score rising from 0.0127 to 0.9520 while execution evidence rises from 6/900 to 12/900; Tables 21–22 recover 98.1% of the ablation gap by swapping explanation/refusal channels. Human validation supports this scoring policy, but behavioral claims need paired execution measures, as in Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?.
- Table 16 measures partial request-relevant content, and Figure 5 omits requested transcription after declining the embedded task. Clean-capability retention cannot establish original-task completion under injection; the already-cited Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection separates original-task and injected-task outcomes, providing a more direct evaluation template.
- Appendix A.3 splits groups and images, yet every split shares the same eight task labels, sixteen instruction families, and 128 paraphrase IDs. This supports instance-level generalization but leaves transfer to new families untested; §5.5’s external safety/capability suites do not evaluate visual delegation.
- Section 5.2’s generic safety baseline and Appendix D.7’s post-SFT DPO do not substitute for SecAlign: Defending Against Prompt Injection with Preference Optimization or StruQ: Defending Against Prompt Injection with Structured Queries, both omitted. Fair visual adaptations must preserve authorized tasks; without them, the advantage of VTID’s objective over targeted authority-training data remains unclear.
- Appendix E.3 releases annotations, outputs and 540 test images, but omits trained adapters, most test images, training/development images and the full renderer. Although disclosed, these omissions prevent independent inference or training: consistency checks on archived records cannot establish that the benchmark and reported models are reproducible.
Questions for authors
- Clarification/evidence: On pixel-identical paired U/AS inputs, how often does each model both withhold the unauthorized action and correctly execute it when delegated, independent of explanation wording? Does cause-specific supervision improve this rate over the collapsed control?
- Evaluation: What fraction of U responses fully satisfies the original request, particularly explicit transcription requests, and how does that compare with Raw Base? Can Table 16 be extended beyond partial request-relevant content?
- Generalization: Can you report a held-out instruction-family or independently authored visual-authorization evaluation, with the evalutor frozen? Please clarify the distinction between disjoint semantic-group identifiers and shared task-family semantics.
- Comparison: Would an authority-specific preference objective or structured instruction-tuning baseline retain the observed gains under matched data, backbone, and budget while allowing delegated visual tasks? This would materially clarify the method’s contribution.
- Reproducibility: Can the frozen adapters and full test images be made accessible during review, and can missing training images be reconstructed from a released renderer with the recorded seeds?
2 comments
Follow-up on our response to Reviewer B8hE
Dear Reviewer B8hE,
We would appreciate your feedback on the additional paired execution analyses, trained visual baselines, and reproducibility information in our response. If any of your main concerns remain unresolved, we would be grateful for clarification so that we can address them before the discussion closes.
Thank you for your time.
Best regards, The Authors
Response to Reviewer B8hE
We thank Reviewer B8hE for asking us to separate evaluator evidence from direct behavior, test outer-request utility, assess independent authorization transfer, and strengthen baselines and reproducibility. We agree that the original paper made state-score improvements easier to see than execution evidence; we report paired diagnostics and narrow the claims below.
Paired execution behavior. Each Unauthorized development row is paired with its Authorized-Safe counterpart using the same base image and visual task. This is a same-image counterfactual, not a pixel-identical pair.
The direct paired completion rates are 42.56% and 41.44%; thus, cause-specific supervision does not create a large literal-completion gain over this control. Its main contribution is to make the reason for non-execution explicit. When an embedded visual instruction is visible but not delegated, silently ignoring it is not the intended behavior: the model should acknowledge the observed instruction and explain that the current user has not authorized its execution. This makes the authorization boundary understandable to the user and separates missing delegation from safety refusal. U target-execution signals are 10/900 (S0) versus 4/900 (S2), with no strict AU dangerous-action signals in either condition. Restoring the missing-authorization, safety-reason, and refusal channels recovers 98.1% of the PCC gap.
We also evaluated 100 pixel-identical pairs from five external visual families with a frozen evaluator. Only the authorization wrapper changes.
This supports transfer of unauthorized-task suppression, but delegated-safe execution remains conservative because the external probe lacks VTID-BENCH’s controlled safety contrast.
Outer-request utility. On a strict external judge, answer-like U responses are 45/100 for Raw Base versus 7/100 for VTID-SFT; on authorized-safe inputs they are 48/99 versus 23/100. We also audited 60 shared U scenes; one row was unevaluable per model:
These measures count responses that preserve meaningful content from the user’s observation request, including partial answers, while keeping boundary-preserving refusals distinguishable from empty responses. They show broad continuity on most scenes, but do not establish full original-task completion under U; that remains an open limitation.
Generalization and stronger baselines. The native split prevents semantic-group, source-scene, and counterfactual-triad overlap, but reuses task-family labels and paraphrase IDs; it establishes group/instance rather than held-out-family generalization. The external five-family probe is an independent authorization stress test with the evaluator frozen.
We trained three visual adaptations from Raw LLaVA-1.5-7B with matched 21,000 source rows, 2,100 updates, and the frozen 2,700-example development split. Hierarchy-SFT uses an instruction-priority scaffold, StruQ-style SFT uses reserved input delimiters, and SecAlign-style DPO uses visual preference pairs. These are mechanism-level visual adaptations, not exact text-only reproductions.
VTID’s PCC advantages over Hierarchy-SFT and StruQ-style SFT are 0.0964 and 0.1265; direct completion differences are smaller and their paired intervals include zero. SecAlign-style DPO failed delegated-safe generation under this visual recipe. This pattern shows that preference-only visual adaptation transfers poorly to VTID's joint authorization-and-completion objective.
Reproducibility. The anonymous package contains a 540-image subset and selected outputs. A complete checksummed private package is staged with full authorization/base-image archives, the three adapters, per-example outputs, renderer/source code, manifests, seeds, and SHA-256 records. We can provide it through a venue-compatible private channel or supplementary upload.
These results support the narrower conclusion that VTID-SFT learns an authorization-sensitive boundary across backbones and suppresses unauthorized visual-task following beyond its native benchmark. We thank the reviewer for prompting these analyses and helping us clarify the evidence and scope of this conclusion.