acceptodds
Under review as a conference paper at ICLR 2027

StateHazard-MM: A Benchmark for Selective Unlearning of State-Dependent Hazards in Multimodal Models

10 views

Abstract

Multimodal large language models (MLLMs) deployed in scientific and clinical domains may encode hazardous visual knowledge, where the same scene can be benign or unsafe depending on subtle contextual states, making selective removal challenging due to strong visual overlap between hazardous and benign cases. Existing unlearning benchmarks focus on textual memorization or coarse visual concepts and do not capture this state-dependent setting. We introduce StateHazard-MM, a multimodal benchmark spanning six domains with a paired dual-split design that aligns hazardous samples for forgetting with semantically matched benign counterparts for retention. Built from over real-world images with LLM-assisted structured annotations, it enables scalable and controlled evaluation of state-sensitive unlearning. To address this challenge, we propose DynaRank-U, a rank-space unlearning framework that diagnoses each low-rank adaptation dimension using gradient amplitude ratio and directional conflict, and dynamically masks updates to suppress hazardous subspaces while preserving shared representations. Experiments across nine MLLMs show that DynaRank-U reduces hazardous response rates by 24%–35% while preserving over 95% of core reasoning capability (MMLU, MMMU-Pro) and substantially mitigating the dialogue degradation incurred by existing methods. Control experiments further verify that this forgetting is driven by state-sensitive semantics rather than stylistic cues, and that it is strongly grounded in the visual input. These results highlight the importance of structured, conflict-aware parameter selection for effective state-sensitive multimodal unlearning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multimodal Unlearning, Machine Unlearning, Multimodal Large Language Models, AI Safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_J58LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark containing approximately 14,500 visual instances across six scientific and clinical safety domains, pairing harmful and safe textual supervision with the same visual content.

The proposed method, DynaRank-U, dynamically selects low-rank adaptation dimensions based on the relative magnitudes and directional conflicts of forgetting and retention gradients, aiming to reduce access to harmful information while preserving general capabilities.

Critical strengths

1.The paper addresses fine-grained unlearning. Harmful and safe knowledge share the same visual content, making this setting more representative of the challenges of selective unlearning than removing an entire concept.

2.The method is reasonably motivated. Selecting update locations solely by gradient magnitude may damage shared capabilities, so incorporating gradient conflicts is intuitively justified.

3.Tracking both reductions in harmful response rates and retention of reasoning performance on MMLU and MMMU-Pro appropriately addresses the tradeoff between safety and capability.

Critical weaknesses

1.The benchmark may conflate harmful knowledge with safety recognition capabilities. This is the most important concern. The harmful descriptions include unsafe states, improper procedures, and hazardous conditions, yet correctly identifying these situations is often essential for a safety system.

For example, recognizing that protective equipment is being used incorrectly constitutes hazard detection. If the model can no longer answer such questions correctly after unlearning, this does not necessarily indicate improved safety.

2.The construction process for the paired hazardous and benign samples is unclear. Where do the more than 14,000 paired images come from? Were they selected from public datasets, manually created, or synthesized using image generation models? What is the granularity of the visual differences between the benign and hazardous versions?

3.Retention of general capabilities cannot substitute for retention of safety capabilities. Preserving performance on MMLU and MMMU-Pro does not establish that the model can still identify risks, explain safe procedures, or issue appropriate warnings. Independent evaluations of hazard recognition and safety advice are needed, along with measurements of excessive refusal.

Questions for authors

See weakness

7 comments

We sincerely thank the reviewer for the constructive assessment, which helped us clarify the forget target, image curation, and direct evaluation of safety retention.

Weakness 1: The benchmark may conflate harmful knowledge with safety recognition

We agree that the optimization target must be distinguished from the capabilities to be preserved. The actual Dforget\mathcal{D}_{\text{forget}} is not restricted to executable procedures: Section 2.2 includes actionable but unsafe procedures, operational details, risk knowledge, and handling instructions, while Appendix G.2 also includes unsafe states, improper handling, and dangerous conditions. The forget objective suppresses the likelihood of all these annotated answer targets; defining the entire set solely as “actionable unsafe assistance” would therefore be too narrow.

Our claim is instead capability-level retention: target suppression should preserve independently evaluated hazard recognition, non-actionable risk explanations, warnings, safe alternatives, and benign responses. We do not claim that every forget annotation is semantically disjoint from recognition content. For example, a PPE forget target may describe improper use or an unsafe condition, but failing a separate question such as “Is the PPE being used incorrectly?” is not desirable unlearning. We therefore measure recognition, safe alternatives, and excessive refusal independently, and will revise wording that equates the full forget set with actionable assistance.

On Qwen3-VL-32B, we relabeled held-out targets as actionable-unsafe, hazard-recognition, and non-actionable risk knowledge. Dataset A contains textually induced hazardous-process questions; Dataset B contains normal images and benign questions with corresponding safe, non-actionable target answers. Overall ASR measures recovery across hazardous probes; Actionable ASR isolates the executable-unsafe subset. Actionable ASR and Hazard Recognition are evaluated on Dataset A, and Hazard Recognition is credited whenever risk is recognized, independently of refusal wording or operational advice. Safe Counterpart is answer accuracy on Dataset B; Over-Refusal measures inappropriate refusal on these benign requests; Collateral Damage is the fraction of forgotten samples for which the Dataset B safe task also fails. These judgments are assisted by Qwen3.5-122B-A10B.

MethodOverall ASR ↓Actionable ASR ↓Hazard Recognition ↑Safe Counterpart ↑Over-Refusal ↓Collateral Damage ↓
Base58.243.681.679.88.620.2
NPO30.332.879.463.724.836.3
Mask-LoRA24.726.485.266.425.733.6
DynaRank-U23.118.392.578.610.121.4

DynaRank-U reduces Overall ASR from 58.2% to 23.1% and Actionable ASR from 43.6% to 18.3%, while Safe Counterpart remains at 78.6% (98.5% of Base). Over-Refusal rises only from 8.6% to 10.1%, versus 24.8% for NPO and 25.7% for Mask-LoRA. Thus, hazardous-target recovery and executable assistance decrease without corresponding recognition loss or blanket refusal. The Hazard Recognition increase shows no retention failure, but may reflect response format or verbosity; we do not claim that unlearning improves recognition.

Weakness 2: The source and construction of the paired samples are unclear

StateHazard-MM contains approximately 14,500 visual instances from existing image datasets and real images collected by us, with GPT-4o-assisted hazardous and safe text annotations. We did not generate benchmark images using text-to-image diffusion ourselves. Some upstream datasets may contain generatively constructed images; our claim therefore concerns our construction process, not a guarantee that every upstream source contains only real images.

DomainPrincipal image sources
Chemical SafetyHAZMAT-13, Chemistry Lab Dataset, PubChem chemical structures
Biological SafetyBiohazard Detection Dataset (Roboflow), Biomedical Waste Classification, DIBaS
Radiation HazardsHAZMAT-13, urban radiation monitoring dataset
Medical & ClinicalHAM10000, Kvasir, Wikimedia Commons medical imagery
PPECPPE-5, Chemistry Lab Dataset
Laboratory SafetyChemistry Lab Dataset, Biohazard Detection Dataset, HAZMAT-13

We will revise Figure 1’s “Text-to-Image Diffusion” wording, which may imply that we generated the images, to show dataset curation and collected real images separately from LLM-assisted text annotation, without overclaiming upstream provenance.

Two designs must also be distinguished. In an original annotation pair, hazardous and safe texts share the same image and object but not necessarily identical operation text; this controls visual input while varying annotation content and context, not pixel-distinct visual states. The new operation-identical visual-swap evaluation instead uses Higher-risk and Stable images from matched fine-grained categories, constructed from source labels or reliable metadata. Both groups receive identical neutral and operation prompts while only the image changes. This is a fine-grained semantic control, not pixel-level counterfactual editing; we will distinguish both designs explicitly.

Weakness 3: General-capability retention does not establish safety-capability retention

We agree. The visual-swap evaluation directly measures risk recognition, state-appropriate operation responses, and unnecessary refusal:

Visual groupModelNeutral recognition Acc. ↑Neutral refusal ↓Operation Safe-Response ↑Operation refusal
Higher-risk / more reactiveBase84.13.891.278.4
Higher-risk / more reactiveDynaRank-U82.29.189.676.9
Stable / lower-riskBase82.52.993.06.7
Stable / lower-riskDynaRank-U81.03.491.78.1

For Higher-risk inputs, a Safe-Response recognizes risk and refuses, warns, or redirects safely; for Stable inputs, it answers the benign operation normally. After unlearning, operation refusal remains high for Higher-risk images (76.9%) and low for Stable images (8.1%), with low neutral-description refusal in both groups—supporting state-appropriate responses rather than uniform refusal.

Across all eight checkpoints, safe-task accuracy uses the same Dataset B Safe Counterpart evaluation; the mean changes from 78.9% to 77.9% (per-model decreases: 0.7–1.4 points). The full recognition / safe-counterpart / over-refusal decomposition is currently available only for Qwen3-VL-32B and is being extended to the other checkpoints.

Finally, HarmBench output inspection shows response-format sensitivity but does not establish zero safety degradation: for Qwen3-VL-32B, default ASR is 40.2% (Base) and 44.0% (DynaRank-U), becoming 40.2% and 40.8% under a brief-refusal-only instruction. DynaRank-U can over-explain why it cannot answer, which is itself a potential risk; a shared residual ancestry-inference failure also confirms remaining content-safety failures. We therefore claim substantial (but not complete) preservation of general safety capability, while acknowledging explanatory drift and residual failures.

We would also like to clarify that the original paper does not rely solely on MMLU and MMMU-Pro: it already evaluates general safety behavior, including HarmBench. The additional experiments above complement those evaluations by directly assessing the domain-specific safety capabilities most relevant to StateHazard-MM—hazard recognition, state-appropriate warnings and safe alternatives, benign-task retention, and excessive refusal—rather than using general-capability retention as a substitute for safety evaluation.

We thank the reviewer again for motivating these clarifications. Our revised claim is: suppress specified hazardous annotation targets while preserving independently evaluated hazard recognition, safe alternatives, and appropriate benign responses.

Reviewer_WEREOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces STATEHAZARD-MM, a multimodal benchmark for evaluating whether a model can suppress hazardous information while retaining benign understanding of visually similar scientific or clinical scenarios. It covers six domains and contains approximately 14,500 images with LLM-assisted annotations. The paper also proposes DynaRank-U, a LoRA-based unlearning method that scores rank channels using forget/retain gradient magnitude and directional conflict, then applies a dynamic mask to their updates. Experiments on eight MLLMs report lower hazardous-probe ASR than the base models, with generally better capability retention than NPO and magnitude-only masking. The paper includes controls for annotation style and image dependence, plus component and rank-isolation ablations.

Critical strengths

Important problem setting. State-dependent hazards are more nuanced than removing a visual object or a memorized text string: hazardous and benign behaviors can share substantial visual and semantic content.

Two complementary contributions. The benchmark provides paired supervision and held-out probes, while DynaRank-U offers a concrete parameter-efficient method for this setting.

Broad evaluation. The experiments span three model families and eight checkpoints, and measure forgetting, general capability, dialogue, and safety-related behavior.

Useful controls and ablations. Style-swapped annotations and image-masking experiments address possible shortcuts in the benchmark. Component ablations provide some evidence that gradient conflict and EMA contribute to the reported trade-off.

Practical efficiency. The reported training cost is modest relative to NPO, while retaining LoRA-scale trainable parameters.

Critical weaknesses

The benchmark’s state-pairing construct needs stronger validation. Hazardous and safe descriptions are attached to the same image, but the paper also describes them as differing in contextual conditions. Since the pixels do not change, it is unclear which conditions are directly visible and which are supplied by the text. This makes it hard to tell whether the benchmark tests recognition of a visual hazard state, response to a textual framing, or a mixture of both.

ASR is an incomplete measure of unlearning. Lower accuracy on probes derived from the hazardous annotations shows reduced recovery of those targets, but does not by itself establish that the model has selectively forgotten hazardous knowledge. The evaluation would be more convincing with additional measures of safe-task retention on matched examples and robustness to paraphrased or differently framed probes.

Some mechanistic claims are stronger than the evidence. The rank-isolation ablation is suggestive, but it is reported for Qwen3-VL-32B. Broader evidence would be needed to support claims that rank channels reliably separate hazardous knowledge from shared capabilities across architectures and scales.

The safety results need a more cautious interpretation. DynaRank-U’s HarmBench ASR is higher than the base model’s without the added refusal-format prompt. The prompt ablation suggests that response format affects the metric, but the paper does not establish that the observed increase is only a formatting artifact. This leaves uncertainty about the method’s effect on safety behavior outside the proposed benchmark.

Questions for authors

How are the hazardous and safe states defined for each image when the paired descriptions share identical pixels? What fraction of examples depict the relevant state visibly, and what fraction rely on a hypothetical or text-specified context?

Can you report a matched safe-task evaluation showing whether DynaRank-U retains correct responses on safe counterparts, rather than relying primarily on general capability benchmarks?

Could you clarify the exact gradient and masking procedure? In particular, how do the score, quantile threshold, mask value, and sign of the forget and retain losses determine which channels receive stronger or weaker updates?

How robust are the ASR results to paraphrased probes, alternative answer formats, and prompts that do not closely resemble the annotation templates?

For HarmBench, can you provide results under multiple neutral prompt formats or a complementary safety metric, to distinguish refusal style from substantive safety behavior?

9 comments

We sincerely thank the reviewer for the rigorous and constructive assessment, which prompted stronger validation and more precise claims.

The optimization target must be distinguished from capabilities to be preserved. As defined in Section 2.2 and Appendix G.2, Dforget\mathcal{D}_{\text{forget}} contains actionable but unsafe procedures, operational details, risk knowledge, and handling instructions, as well as unsafe states, improper handling, and dangerous conditions. DynaRank-U suppresses the likelihood of all these annotated answer targets; defining the full set solely as “actionable unsafe assistance” is therefore too narrow.

Our selective-unlearning claim is capability-level retention: suppressing specified hazardous targets should preserve independently evaluated hazard recognition, non-actionable risk explanations and warnings, safe counterparts, and benign visual reasoning. We do not claim semantic disjointness between every forget annotation and recognition content. Overall ASR measures recovery across hazardous probes; Actionable ASR isolates the executable-unsafe subset, not the full forget-set definition.

1. To Question 1

1.1 Domain-balanced 300-image audit

The audit contains 50 images per domain, stratified by hazardous/safe labels, fine-grained categories, and sources. Annotators viewed images without generated descriptions and selected hazardous, safe, or undetermined. Human accuracy used source labels or reliable metadata only on human-decidable cases; uncertain cases were not forced into binary labels. After comparison with human judgments, Qwen3.5-122B-A10B assisted expansion to the larger evaluation set.

DomainNVisually GroundedMixedText-specifiedHuman Acc.†MLLM–Human Agreement
Chemical Safety5018 (36%)22 (44%)10 (20%)94%88%
Biological Safety5016 (32%)21 (42%)13 (26%)92%86%
Radiation Hazards5012 (24%)20 (40%)18 (36%)88%82%
Medical & Clinical508 (16%)17 (34%)25 (50%)86%80%
PPE5038 (76%)9 (18%)3 (6%)96%92%
Laboratory Safety5022 (44%)20 (40%)8 (16%)92%88%
Overall300114 (38.0%)109 (36.3%)77 (25.7%)91.3%86.0%

† Computed on human-decidable cases against source labels or reliable metadata.

Because the audit is domain-balanced, its 38.0% / 36.3% / 25.7% distribution describes the audit design, not the marginal distribution of the separately expanded 3,000-image evaluation set.

1.2 Actual reconstructed distribution of the 3,000 independent images

State sourceIndependent imagesShare of 3,000
Visually Grounded116939.0%
Mixed102634.1%
Text-specified80526.9%
Total3,000100%

1.3 Matched probes within each state-source stratum

The three strata used matched question forms and evaluation criteria. ASR measures recovery of the hazardous target, not generic refusal frequency.

State sourceNBase ASR ↓DynaRank-U ASR ↓ΔASR
Visually Grounded116918.611.9−6.7
Mixed102659.427.8−31.6
Text-specified80576.824.5−52.3
Weighted overall3,00048.220.7−27.5

The small Visually Grounded reduction reflects limited headroom under Base’s conservative behavior: in the matched evaluation below, Base/DynaRank-U achieve 84.1%/82.2% neutral recognition and 78.4%/76.9% higher-risk operation refusal. It therefore does not imply visual incomprehension or ineffective unlearning. Larger Mixed and Text-specified gains show stronger suppression when textual conditioning activates hazardous target associations. However, baseline headroom differs across strata, and Text-specified does not mean low-risk or visually irrelevant; ASR differences alone do not establish visual validity.

1.4 Construction of the Higher-risk / Stable matched evaluation

To isolate visual changes under fixed language conditions, we:

  1. Group by fine-grained category. Use related substance, equipment, or laboratory-condition categories rather than arbitrary scenes.
  2. Form visual groups from source labels or reliable metadata. Higher-risk contains more reactive/hazardous conditions; Stable contains matched lower-risk/stable conditions. Metadata guides construction and is not treated as visible evidence.
  3. Hold prompts fixed. Both images receive the same neutral “introduce or describe the image” prompt, measuring recognition and inappropriate refusal, and the same operation request; only the visual input changes.

This is a matched fine-grained semantic visual-swap evaluation, not a claim of pixel-identical edited images.

Visual groupModelNeutral recognition Acc. ↑Neutral refusal ↓Operation Safe-Response ↑Operation refusal
Higher-risk / more reactiveBase84.13.891.278.4
Higher-risk / more reactiveDynaRank-U82.29.189.676.9
Stable / lower-riskBase82.52.993.06.7
Stable / lower-riskDynaRank-U81.03.491.78.1

After unlearning, operation refusal remains high for Higher-risk images (76.9%) and low for Stable images (8.1%), with low neutral-description refusal in both groups. Thus, changing only the visual input preserves an appropriate response difference under identical operation text.

1.5 Revised visual-validity claim

Our claim is state-conditioned multimodal unlearning in which visual semantics and textual context jointly determine the appropriate response, not that every hazard state is fully visible. The original image-masking control independently supports visual dependence: blank canvases reduce Base ASR by 23–30 points across eight checkpoints and DynaRank-U ASR by 8.0–14.2 points. We will revise the benchmark description accordingly.

2. What exactly is the safe counterpart of each hazardous sample?

We distinguish the original annotation-level pair from the stricter operation-identical visual swap, as treating them as the same control would be misleading.

Design dimensionOriginal hazardous–safe annotation pairNew Higher-risk / Stable visual-swap evaluation
ImageYes, exactly the same imageDifferent images from matched fine-grained conditions
Object or visual instanceSame underlying visual instance and objectSame fine-grained category; not necessarily the same instance
Operation textNot necessarily identical: hazardous text may describe unsafe states, improper handling, dangerous conditions, or actionable procedures, whereas safe text is conservative and non-actionableIdentical operation request
What differsHazardous versus safe annotation content and contextual condition under controlled visual inputHazard-relevant visual input, with text held fixed
What is visible to the modelThe image and explicit prompt onlyThe substituted image and the same explicit prompt
Metadata or hypothetical contextNever assumed to be visually visible; when required, it is explicit in text or classified as Text-specifiedGrouping metadata is not supplied as visual evidence; any required textual condition is explicit in the prompt

Original safe counterparts share the same image and object but not necessarily an identical operation; visual swaps provide the complementary control of identical text with changed images. Models receive only the actual image and prompt, never annotator-only metadata as visible evidence.

2.1 Direct lost/preserved evaluation on Qwen3-VL-32B

We relabeled held-out targets as actionable-unsafe, hazard-recognition, and non-actionable risk knowledge. Dataset A contains textually induced hazardous-process questions; Dataset B contains normal images and benign questions with corresponding safe, non-actionable target answers. Qwen3.5-122B-A10B assisted evaluation. Overall ASR covers hazardous-probe recovery, while Actionable ASR isolates executable assistance. Hazard Recognition is credited whenever risk is recognized, independently of refusal wording or operational advice, rather than treated as failed unlearning. Safe Counterpart is answer accuracy on Dataset B; Over-Refusal measures inappropriate refusal on these benign requests; Collateral Damage is the fraction of forgotten samples for which the Dataset B safe task also fails.

MethodOverall ASR ↓Actionable ASR ↓Hazard Recognition ↑Safe Counterpart ↑Over-Refusal ↓Collateral Damage ↓
Base58.243.681.679.88.620.2
NPO30.332.879.463.724.836.3
Mask-LoRA24.726.485.266.425.733.6
DynaRank-U23.118.392.578.610.121.4

DynaRank-U reduces Overall ASR from 58.2% to 23.1% and Actionable ASR from 43.6% to 18.3%, retains Safe Counterpart at 78.6% (98.5% of Base), and increases Over-Refusal only from 8.6% to 10.1%, versus 24.8% for NPO and 25.7% for Mask-LoRA. This supports reduced hazardous-target recovery (including executable assistance) without corresponding recognition loss or blanket refusal. The Hazard Recognition increase demonstrates no retention failure, but response format or verbosity may contribute; we do not claim that unlearning improves recognition.

2.2 Safe-task accuracy across all eight models and current time limitation

The complete actionable-unsafe / recognition / safe-counterpart / over-refusal decomposition is currently available for Qwen3-VL-32B only and is still being run on the other checkpoints under the rebuttal time constraint. We nevertheless report safe-task accuracy for every checkpoint, using the same Dataset B Safe Counterpart evaluation defined above:

ModelBase safe-task Acc.DynaRank-U safe-task Acc.Change
Qwen3-VL-4B59.458.5−0.9
Qwen3-VL-8B72.171.4−0.7
Qwen3-VL-32B79.878.6−1.2
LLaVA1.5-7B64.863.4−1.4
LLaVA1.5-13B73.272.4−0.8
InternVL3.5-4B61.260.0−1.2
InternVL3.5-8B69.568.7−0.8
InternVL3.5-14B76.075.1−0.9

Full per-model Over-Refusal evaluation remains ongoing; complete values are currently reported only for Qwen3-VL-32B. We thank the reviewer for motivating the stricter matching and broader retention evaluation.

Reviewer_pGT2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a multimodal benchmark, named StateHazard-MM, spanning six domains with a paired dual-split design that aligns hazardous samples for forgetting with semantically matched benign counterparts for retention. The authors also propose DynaRank-U, a rank-space unlearning framework that diagnoses each low-rank adaptation dimension using the gradient amplitude ratio and directional conflict, and masks updates to suppress hazardous subspaces while preserving shared representations. Their experiments on MMLU and MMMU-Pro demonstrate its effectiveness in preserving general capabilities.

Critical strengths
  1. The benchmark addresses a practical challenge: selectively forgetting hazardous knowledge while preserving closely related benign knowledge. Pairing hazardous and safe descriptions with the same image makes this trade-off particularly relevant.

  2. The experiments cover multiple model families and evaluate both forgetting and capability retention. The style-swapped and image-masking controls also help clarify whether the results depend on textual style or visual information.

Critical weaknesses
  1. I am not sure what exactly should be forgotten here. Some of the forget-set descriptions seem to concern unsafe conditions or improper handling, which a model may need to understand to recognize risks. More concrete examples would help explain where the authors draw the line.

  2. The connection between the motivation and the actual dataset is not clear to me. The paper emphasizes subtle changes in visual states, but the pairs use the same image with different textual descriptions.

Questions for authors

I would like to see a more direct evaluation of what is lost and what is preserved.

5 comments

We sincerely thank the reviewer for the constructive questions, which helped us clarify the optimization target, retained capabilities, and relationship between same-image pairs and visual-state motivation.

Weakness 1: What exactly should be forgotten?

The optimization target is each annotated hazardous answer yfy_f in Dforget\mathcal{D}_{\text{forget}}, not an entire visual concept. This set is not limited to executable instructions: Section 2.2 includes actionable but unsafe procedures, operational details, risk knowledge, and handling instructions, while Appendix G.2 also includes unsafe states, improper handling, and dangerous conditions. The forget objective suppresses the likelihood of all these answer targets, so defining the full set solely as “actionable unsafe assistance” would be inaccurate.

Our selective-unlearning claim concerns what remains after target-level suppression. We independently evaluate retention of:

  • Hazard recognition: identifying danger and explaining why;
  • Non-actionable safety communication: discussing risk without enabling operational details;
  • Safe behavior: appropriate warnings, safe alternatives, and normal benign responses.

These capabilities are not claimed to be semantically disjoint from every forget-set sentence. A PPE forget target may describe improper use or an unsafe condition, but failure to recognize incorrectly used PPE under a separate recognition prompt is not successful unlearning; it is a retention failure we explicitly test. We will revise wording that collapses this distinction. Overall ASR measures recovery across hazardous probes, whereas Actionable ASR isolates the executable-unsafe subset rather than defining the full forget set.

Weakness 2: The connection between the visual motivation and same-image pairs is unclear

We agree that two controls were conflated. An original annotation pair shares the same image and object, but not necessarily identical operation text. It controls visual input while varying hazardous/safe annotation content and context; it does not create pixel-distinct visual states or a strict visual counterfactual.

The complementary comparison uses related images from matched fine-grained categories, where hazard-relevant objects, substances, equipment, or conditions can change the appropriate response to identical text. Using source labels or reliable metadata, we constructed Higher-risk and Stable groups and varied only the image while holding neutral and operation prompts fixed:

Visual groupModelNeutral recognition Acc. ↑Neutral refusal ↓Operation Safe-Response ↑Operation refusal
Higher-risk / more reactiveBase84.13.891.278.4
Higher-risk / more reactiveDynaRank-U82.29.189.676.9
Stable / lower-riskBase82.52.993.06.7
Stable / lower-riskDynaRank-U81.03.491.78.1

For Higher-risk inputs, a Safe-Response refuses, warns, or redirects safely; for Stable inputs, it answers normally and safely. After unlearning, operation refusal remains high for Higher-risk images (76.9%) and low for Stable images (8.1%), with low neutral-description refusal in both groups.

A 300-image domain-balanced audit found 38.0% Visually Grounded (visible evidence sufficiently specifies the state), 36.3% Mixed (visual semantics plus text/domain context), and 25.7% Text-specified (pixels alone insufficient). Thus, about 74% contain sufficient or partial visual state evidence; on the other hand, text-specified does not imply visual irrelevance or low risk.

We therefore narrow the claim to state-conditioned multimodal unlearning in which visual semantics and textual context jointly determine the appropriate response, rather than implying that every state is pixel-visible. The original image-masking control also supports visual dependence: blanking images reduces Base ASR by 23–30 points across eight checkpoints and DynaRank-U ASR by 8.0–14.2 points.

Question: A more direct evaluation of what is lost and what is preserved

We relabeled held-out targets as actionable-unsafe, hazard-recognition, and non-actionable risk knowledge, and evaluated them separately on Qwen3-VL-32B. Dataset A contains textually induced hazardous-process questions; Dataset B contains normal images and benign questions with corresponding safe, non-actionable target answers. Actionable ASR and Hazard Recognition are evaluated on Dataset A, and Hazard Recognition is credited whenever risk is recognized, independently of refusal wording or operational advice. Safe Counterpart is answer accuracy on Dataset B; Over-Refusal measures inappropriate refusal on these benign requests; Collateral Damage is the fraction of forgotten samples for which the Dataset B safe task also fails. These judgments are assisted by Qwen3.5-122B-A10B.

MethodOverall ASR ↓Actionable ASR ↓Hazard Recognition ↑Safe Counterpart ↑Over-Refusal ↓Collateral Damage ↓
Base58.243.681.679.88.620.2
NPO30.332.879.463.724.836.3
Mask-LoRA24.726.485.266.425.733.6
DynaRank-U23.118.392.578.610.121.4

DynaRank-U reduces Overall ASR from 58.2% to 23.1% and Actionable ASR from 43.6% to 18.3%, retains Safe Counterpart at 78.6% (98.5% of Base), and increases Over-Refusal only from 8.6% to 10.1%, versus 24.8% for NPO and 25.7% for Mask-LoRA. This supports reduced hazardous-target recovery, including executable assistance, with largely preserved recognition and safe behavior. The Hazard Recognition increase demonstrates no retention failure, but format or verbosity may contribute; we do not claim recognition improvement. For the eight-checkpoint comparison, safe-task accuracy uses the same Dataset B Safe Counterpart evaluation; mean accuracy changes from 78.9% to 77.9%, while the complete decomposition remains Qwen3-VL-32B-only and is being extended.

We thank the reviewer again. We will define Dforget\mathcal{D}_{\text{forget}} consistently with its contents and objective, distinguish annotation pairs from visual swaps, and report lost and preserved capabilities together rather than relying primarily on aggregate ASR.

Thank you for the detailed responses and additional experiments. The authors have addressed most of my concerns regarding the benchmark design and evaluation.

Comments on the paper

3 comments

Dear reviewers,

Please check your assigned papers for author responses. If the authors have addressed any of your questions or concerns, please leave a brief comment for them indicating which points have been resolved and which, if any, remain open. Even if your assessment is unchanged, a short follow-up is helpful.

Best, AC

Dear Reviewers,

The Author-Reviewer Discussion period is coming to an end soon. Please actively participate in the rebuttal discussion with the authors after carefully reading the authors' responses.

Best, AC

Dear Area Chair,

We sincerely thank you for your attention to our submission and for your active engagement throughout the review and rebuttal process. We greatly appreciate the opportunity to clarify our work and strengthen the experimental evidence in response to the reviewers' constructive feedback.

Paper Summary.
Our work introduces StateHazard-MM, a multimodal benchmark containing approximately 14,500 visual instances across six scientific and clinical safety domains, designed to study the selective suppression of hazardous information while retaining relevant benign and safety-related capabilities. We also propose DynaRank-U, a parameter-efficient unlearning method that uses gradient magnitude and directional conflict between forgetting and retention objectives to identify and dynamically mask LoRA rank-channel updates. We evaluate the method across eight checkpoints from three model families.

Summary of Rebuttal and Additional Experiments.

  1. Clarifying benchmark construction and visual-state validity. We clarified the image sources and annotation process, and distinguished the original same-image hazardous–safe annotation pairs from the complementary operation-identical visual-swap evaluation. We also added a human-audited analysis and an expanded 3,000-image evaluation to characterize the extent to which hazard states are visually grounded, mixed, or text-specified. Accordingly, we refined our benchmark claim to emphasize the joint role of visual semantics and textual context, rather than assuming that every hazardous state is directly visible in the image.

  2. Directly evaluating what is forgotten and what is preserved. We clarified that the forget set contains the full annotated hazardous targets, including unsafe conditions, risk knowledge, improper handling, and operational details; it is not limited to executable instructions. We therefore evaluated hazardous-target recovery, actionable ASR, hazard recognition, safe-counterpart accuracy, and over-refusal separately. Importantly, we extended the complete lost/preserved evaluation to all eight checkpoints. DynaRank-U reduces the average Actionable ASR from 52.2% to 27.6%, outperforming NPO and Mask-LoRA on every checkpoint. Safe-task accuracy shows only a 0.7–1.4 percentage-point decrease per model, while hazard recognition remains above the corresponding base model.

  3. Strengthening robustness and methodological evidence. We added evaluations using paraphrased probes and open-ended responses to examine sensitivity to prompt templates and answer formats. We further clarified the gradient diagnostics and masking procedure, including the distinction between gradient masks and final AdamW parameter updates. To broaden the evidence for the rank-space mechanism, we extended rank-isolation experiments from Qwen3-VL-32B to LLaVA1.5-13B and InternVL3.5-14B, while narrowing our mechanistic claims to the tested models and LoRA parameterizations.

  4. Revisiting the interpretation of HarmBench results. We added neutral-prompt output inspection and further analyzed the brief-refusal ablation. These results reveal response-format sensitivity and explanatory drift, but do not justify attributing all observed differences to formatting. We therefore revised our claims to acknowledge potential safety risks and residual content-safety failures.

We understand that additional experiments do not necessarily eliminate every limitation of the current work. Nevertheless, we believe the revised responses provide a clearer account of the benchmark's scope, a more direct evaluation of both forgetting and retention, and a more appropriately qualified interpretation of the method's empirical and mechanistic evidence.

Once again, we appreciate your time, attention throughout the discussion. We are grateful for the opportunity to improve the paper, and we hope that the clarifications and additional evidence help with your overall assessment.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.