- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates cross-pathway concept unlearning in Unified Multimodal Models (UMMs), where target concepts should be suppressed in both image generation and visual understanding while preserving non-target capabilities. The authors introduce UUBench, a diagnostic benchmark for evaluating joint forgetting and capability preservation, and propose UniForget, which combines generation-side weight editing with understanding-side LoRA optimization. Experiments on BAGEL-7B-MoT and Janus-Pro-7B demonstrate the limitations of existing single-pathway approaches and show improvements in joint forgetting.
Critical strengths
Meaningful problem formulation. The paper identifies an important limitation of existing unlearning approaches: forgetting a concept in one pathway does not necessarily lead to forgetting in the other, even within a unified model. This motivates the need for cross-pathway unlearning and provides useful insights into the behavior of UMMs.
Useful benchmark and evaluation protocol. UUBench evaluates forgetting in both generation and understanding, along with neighboring concept retention and capability preservation. In particular, distinguishing Unified Forget from Unified Strict Forget provides a more comprehensive evaluation than measuring target suppression alone.
Critical weaknesses
1. What contributes most to the improvement in joint forgetting? UniForget combines generation-side weight editing with understanding-side LoRA optimization. However, the current comparisons do not fully isolate the benefits of sequential coordination from those of the richer understanding-side training objectives. It would be helpful to compare independent and sequential training under matched supervision to better understand the source of the improvements.
2. Can the authors provide a more fine-grained analysis of cross-pathway forgetting transfer? A central motivation of this work is that forgetting in one pathway does not necessarily transfer to the other. Beyond the aggregate results, it would be interesting to examine how transfer varies across concept categories and model architectures. In particular, analyzing the relationship between architectural parameter sharing and cross-pathway transfer could provide valuable insights into the underlying behavior of UMMs.
3. How does the similarity between target and neighboring concepts affect unlearning? UUBench includes semantically related neighboring concepts to assess unintended forgetting. A more fine-grained analysis based on visual or semantic similarity would be useful to understand when neighboring concepts are most vulnerable to unlearning and how effectively UniForget preserves them.
4. How robust is joint forgetting to indirect concept references? Although the paper evaluates several prompt perturbations and attacks, it would be useful to further investigate scenarios in which the target concept is referenced indirectly rather than explicitly named. For example, can the model still generate or identify a target from its distinctive visual or semantic attributes? Such evaluations would help distinguish robust concept forgetting from suppression of explicit target identities.
Questions for authors
Please refer to the weaknesses.
3 comments
Q3. How does target–neighbor similarity affect unlearning?
We analyzed UUBENCH’s 60 predefined target–neighbor pairs. We used frozen DINOv2 to encode held-out images and computed cosine similarity between the mean image embeddings of each pair. For semantic similarity, we compared frozen E5 embeddings of fixed, name-free concept descriptions. We computed both scores before inspecting edited outputs and binned each measure separately into low, medium, and high similarity, with 20 pairs per bin.
Nbr Strict counts a paired record as preserved only if the generated image depicts the correct neighbor and meets the image quality criterion, and the understanding answer correctly identifies that same neighbor and meets the answer quality criterion. We swept edit strengths to match TIME + GD and UniForget as closely as possible on target forgetting and computed pair-level Spearman correlations between similarity and preservation loss.
| Similarity | Backbone | Method | Low ↑ | Medium ↑ | High ↑ | Spearman ρ |
|---|---|---|---|---|---|---|
| Visual | BAGEL | TIME + GD | 89.0 | 88.3 | 88.7 | 0.04 |
| Visual | BAGEL | UniForget | 91.0 | 90.3 | 90.7 | 0.02 |
| Visual | Janus | TIME + GD | 88.3 | 87.7 | 88.0 | −0.03 |
| Visual | Janus | UniForget | 90.3 | 89.7 | 90.0 | 0.01 |
| Semantic | BAGEL | TIME + GD | 98.0 | 89.0 | 79.0 | 0.54 |
| Semantic | BAGEL | UniForget | 98.7 | 91.0 | 82.3 | 0.45 |
| Semantic | Janus | TIME + GD | 98.0 | 88.0 | 78.0 | 0.58 |
| Semantic | Janus | UniForget | 98.7 | 90.0 | 81.3 | 0.48 |
The bin scores are percentages.
At comparable levels of target forgetting, preservation varies little with visual similarity and falls as semantic similarity increases. UniForget improves preservation in every bin, though the most semantically similar neighbors remain the hardest to preserve.
Q4. How robust is joint forgetting to indirect concept references?
In UUBENCH, DF requires a correct generic category, omission of the target name and its aliases, and no refusal. TU separately checks whether non-identity answers are accurate, specific, and informative. These criteria distinguish name suppression from uninformative answers. We added three tests to see whether target identities can still be recovered through indirect references.
- Numbered choice. For each target image, we randomly order four candidates from the same category, including the target, two close neighbors, and one additional distractor. We ask the model to return only the option number. Option Accuracy is the fraction of trials in which it selects the target. Wrong choices, invalid numbers, and refusals count as incorrect.
- Multilingual recognition. We use semantically equivalent prompts in Chinese, Spanish, and Japanese. We ask the model to identify the subject of the image in each language and use the standard translated name in generation prompts. We draw four independent samples per condition. GPT-5.6 Sol checks image validity and target recognizability, and checks whether the answer reveals the identity through a name, translation, transliteration, or equivalent expression. ML-S.Forget requires a valid image with no recognizable target and an answer that gives the correct generic category without identity leakage or refusal.
- Name-free descriptions. We sample three Gemini 3.5 descriptions per target covering appearance, function, and context. We use string matching and other checks to exclude names, aliases, translations, and obvious variants. GPT-5.6 Sol checks target recognizability in images generated from the descriptions alone, without knowing the editing method. For understanding, the model describes the category, visible features, and scene without proper nouns. The judge checks for identity-specific information that cannot be inferred from the image. For an Eiffel Tower image, “a tall iron tower” describes visible features, while “located in Paris” can reveal identity if the location is not visible. We draw four independent samples per condition. Desc-S.Forget requires a valid image with no recognizable target and a correct, informative answer without refusal, target naming, or nonvisual identity clues.
| Backbone | Method | Option Accuracy ↓ | ML-S.Forget ↑ | Desc-S.Forget ↑ |
|---|---|---|---|---|
| BAGEL | Base | 95.3 | 0.0 | 0.0 |
| BAGEL | TIME + GD | 72.0 | 20.0 | 17.8 |
| BAGEL | UniForget | 58.7 | 31.7 | 28.3 |
| Janus | Base | 96.0 | 0.0 | 0.0 |
| Janus | TIME + GD | 68.0 | 25.0 | 21.7 |
| Janus | UniForget | 53.3 | 38.3 | 34.4 |
All scores are percentages.
UniForget lowers numbered-choice accuracy and raises both strict forgetting scores relative to TIME + GD on both backbones. Together, these results indicate that UniForget makes target identities harder to recover under tests that go beyond English-name matching.
We sincerely appreciate the reviewer’s positive reassessment of our manuscript. Thank you for your thoughtful suggestions and support!
Thank you for pointing out these problems! Table 5 examines the understanding-side objectives through cumulative ablations. To isolate the contribution of sequential coordination, we added the matched-supervision comparison.
We hold the generation-side edit, understanding-side supervision bank, LoRA configuration, training objectives, number of training steps, and merge coefficient fixed. In independent composition, we train the understanding-side LoRA on the base checkpoint and merge its update into the generation-edited checkpoint. In sequential coordination (UniForget), we train the same LoRA directly on the generation-edited checkpoint and merge the update using the same coefficient.
With supervision and objectives held fixed, sequential coordination improves Unified Forget by 16.0 percentage points on BAGEL and 15.4 points on Janus. Unified Strict Forget also improves by 3.9 and 14.3 points, respectively, while TU decreases by 2.5 and 1.5 points. The results show that allowing the understanding-side update to adapt directly to the generation-edited checkpoint makes an independent contribution to UniForget’s gains
In preliminary experiments, several existing image-to-text unlearning methods achieved limited forgetting on the understanding side of the UMMs we tested. We reported GD and KL as representative baselines. Their low within-pathway forgetting rates leave open whether a more effective understanding-side intervention would transfer to generation.
For the reverse direction, the generation-only edits TIME achieve strong target forgetting, which lets us examine how much forgetting carries over to understanding. We grouped the 30 targets by concept category and measured TF and DF on the generation-only checkpoints.
Characters have the highest DF on both backbones, at 50.0% on BAGEL and 46.0% on Janus. The ordering of landmarks and objects reverses across models. Objects have higher DF on BAGEL (16.0% vs. 6.0%), while landmarks have higher DF on Janus (30.0% vs. 16.0%).
BAGEL shares the edited attention projections across pathways for text tokens, while its generation-side VAE latents use separate projections. Janus uses the same edited backbone projections in both pathways. Even with this sharing, Janus reaches only 30.7% overall DF despite 96.7% TF. Overall, these results suggest that cross-pathway transfer depends on several factors and cannot be explained by concept category or parameter sharing alone.