- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TVE-Fisher, a task-vector unlearning method guided by Fisher information. It combines static retain-side importance with dynamically estimated forget-side sensitivity to gate parameter updates during fine-tuning, then negates the resulting task vector to suppress targeted knowledge. Experiments on CLIP benchmarks and multimodal identity QA demonstrate an improved forgetting–retention trade-off over existing task-vector baselines.
Critical strengths
-
The paper derives a retention-aware parameter selection criterion from a local analysis of forgetting benefits and preservation costs.
-
Fisher-guided gating produces selective task vectors whose negation strength can be adjusted without adding inference-time modules.
-
Experiments cover multiple CLIP backbones, eight datasets, and multimodal identity QA, supported by component ablations and parameter-level visualizations.
Critical weaknesses
-
Task-vector negation is an established approach to unlearning, and the main contribution here lies in Fisher-guided parameter selection during task-vector construction. Selective parameter updates and Fisher-based importance estimation have also been extensively studied. The paper should more clearly distinguish its contribution from these approaches and establish the benefit of its specific gating rule over simpler parameter-selection strategies.
-
The MLLM evaluation is restricted to LLaVA-1.5-7B on CLEAR. Experiments on MLLMU-Bench and additional MLLM architectures would strengthen the evidence that the method generalizes beyond the predominantly CLIP-based evaluation.
-
The theoretical guarantees concern small, local parameter edits, whereas the implementation uses multiple AdamW updates, dynamically changing masks, and subsequent task-vector negation. These operations may move beyond the region where the local approximation is reliable. Further analysis is needed to establish how well the theoretical predictions describe the final edits, particularly whether retention costs remain approximately symmetric under negation.
Questions for authors
See weaknesses.
6 comments
Response to Reviewer yVkL (part 2)
Response to Reviewer yVkL (part 2)
2. Computational and storage overhead
Response:
Dynamic Fisher adds forget-Fisher refreshes while retaining auxiliary working storage. Within a matched training parameterization, all three variants use the same optimization-step budget and task-vector construction.
Let be the number of Fisher/gate coordinates, the optimizer-update count, the number of forget-Fisher estimates including initialization, and the costs of one retain/forget Fisher estimate. The accounting below concerns epoch-refreshed weight-space FFT, where vector and Fisher coordinates coincide.
Table R20. Operation and auxiliary-storage accounting (analysis accompanying Appendix B.5, Computational Costs).
| Operation | Standard task vector | Static Fisher | Dynamic Fisher |
|---|---|---|---|
| Fine-tuning forward/backward and optimizer updates | Same budget | Same budget | |
| Task-vector construction | |||
| Retain-Fisher estimation | — | ||
| Forget-Fisher estimation | — | ||
| Calibration, EMA, and mask construction | — | ||
| Applying gates to optimizer increments | — | ||
| Additional Fisher/mask working state | — | ||
| Optional saved Fisher/mask artifacts | — | for a fixed number of snapshots |
-
Estimation and gating. A diagonal Fisher estimate accumulates squared per-sample gradients: gradient evaluations of cost require computation. We cache raw retain Fisher; each refresh re-estimates forget Fisher, updates its EMA, and jointly calibrates the statistics to construct the mask. Gating requires no extra backward pass. Estimation requires extra passes unless the required per-sample gradients are already available; with reuse, only accumulation is additional. Squaring an averaged minibatch gradient is not equivalent to averaging squared per-sample gradients.
-
Refresh overhead. Relative to static Fisher, dynamic Fisher adds . With one estimate per epoch, and for epochs and updates per epoch; initialization plus a further refresh every epoch instead gives . Under fixed per-epoch budgets, relative overhead is and does not grow simply with the epoch count. Step-refreshed adapters require the actual refresh count and parameterization-specific projection/merge costs.
-
Storage. Both Fisher variants keep a constant number of diagonal vectors and update EMA recursively, giving auxiliary working storage without dense Fisher matrices or epoch-wise histories. This does not imply identical peak GPU memory. For adapters, denotes the actual Fisher/gate space; weight-space state need not scale with trainable adapter parameters. After merging the edit, inference requires no Fisher/mask state. Disk overhead separately depends on saved artifacts.
Table R21. Static versus dynamic Fisher accuracy (paper Table 16; Appendix E.3).
| Variant | Mean forget accuracy (%) ↓ | Mean retain accuracy (%) ↑ |
|---|---|---|
| Static Fisher | 16.43 | 60.37 |
| Dynamic Fisher | 15.86 | 60.90 |
Dynamic refresh improves mean forget accuracy by 0.57 pp and retain accuracy by 0.53 pp. Runtime and peak-memory measurements are in Appendix B.5, Table 6.
Response to Reviewer yVkL (part 3)
Response to Reviewer yVkL (part 3)
3. Broader MLLM evaluation
Reviewer comment:
The MLLM evaluation is restricted to LLaVA-1.5-7B on CLEAR. Experiments on MLLMU-Bench and additional MLLM architectures would strengthen the evidence that the method generalizes beyond the predominantly CLIP-based evaluation.
Response:
We added both: the evaluation now covers two architectures and two benchmarks—LLaVA-1.5-7B on CLEAR and MLLMU-Bench, and Qwen2.5-VL-7B on CLEAR—with consistent gains. Protocols are in Appendix B.4. Results appear in paper Table 3 (§4.2.1) and Tables 4–5 (Appendix B.4), with a summary in §4.2.1.
For CLEAR Identity QA, training uses image–caption pairs and evaluation uses identity questions, testing transfer of suppression across input formats.
Table R22. CLEAR Identity QA with LLaVA-1.5-7B (complete paper Table 3).
| Method | Forget ↓ | Retain ↑ |
|---|---|---|
| Pre-trained model | 54.73 | 47.57 |
| Standard Task Arithmetic-LoRA | 37.25 | 44.59 |
| Uniform Merge | 34.97 | 45.13 |
| TIES-Merging | 33.55 | 44.45 |
| EMR-Merging | 32.47 | 43.10 |
| Function Ensemble | 33.16 | 44.70 |
| MMUnlearner | 32.29 | 43.05 |
| Ours (F-TVE-LoRA) | 31.54 | 45.10 |
-
CLEAR, LLaVA-1.5-7B. F-TVE-LoRA achieves the lowest forget accuracy among the compared methods: 31.54%, versus 37.25% for TA-LoRA, while improving retain accuracy from 44.59% to 45.10%.
-
New architecture: Qwen2.5-VL-7B. F-TVE achieves the best Retain (0.56, versus ≤0.38), Real, and Forget Quality (0.90) among all evaluated unlearning methods in paper Table 4. It improves all four metrics over standard task arithmetic, as summarized below.
-
New benchmark: MLLMU-Bench. Across the 5%, 10%, and 15% forget splits, F-TVE achieves the lowest Test classification accuracy under paraphrased questions and transformed images. Averaged across splits, it gives the largest Test-set suppression (10.65 pp) and smallest retain loss (1.28 pp) relative to Vanilla among the compared methods. Retain accuracy stays within 1.67 pp of Vanilla in every split. Generation, cloze, residual-likelihood, and relearning results are discussed further in our response to Reviewer RVdH (W2).
Table R23. Additional architecture: CLEAR/Qwen2.5-VL-7B (selected rows of paper Table 4).
| Method | Real ↑ | Retain ↑ | Forget ↓ | Forget Quality ↑ |
|---|---|---|---|---|
| Non-lin. FT | 0.18 | 0.38 | 0.23 | 0.82 |
| F-TVE | 0.22 | 0.56 | 0.19 | 0.90 |
Table R24. Additional benchmark: MLLMU-Bench/LLaVA-1.5-7B classification (selected entries of paper Table 5).
| Forget split | Vanilla Test (%) | F-TVE Test (%) | Vanilla Retain (%) | F-TVE Retain (%) |
|---|---|---|---|---|
| 5% | 43.31 | 33.09 | 43.29 | 41.62 |
| 10% | 43.67 | 32.60 | 41.70 | 40.14 |
| 15% | 43.42 | 32.76 | 44.18 | 43.56 |
Response to Reviewer yVkL (part 1)
Response to Reviewer yVkL (part 1)
We thank the Reviewer for the insightful comments and address each point below with additional results and analysis.
1. Contribution and evidence for the gating rule
Reviewer comment:
Response:
Our contribution is a retain-aware selection score motivated by the retention-constrained unlearning objective. Adding the retain term alone lowers mean forget accuracy from 16.91% to 15.86% at comparable retention (60.87% → 60.90%).
Contribution and local rationale. Task-vector negation is the deployment step; our gate shapes the vector during construction by combining fixed retain importance with dynamically updated forget sensitivity. Under the local diagonal surrogate, forget gain is first-order, while retain cost is quadratic. For forget gradient gi and positive damped retain curvature hi, maximizing the first-order forget gain on support S under retain budget ε gives squared optimal gain
2ε∑i∈Shigi2.
This yields the ranking score gi2/hi: forget benefit discounted by retain cost (Proposition 2). Forget-only saliency, such as SalUn-style masks, omits retain cost; EWC uses retain importance as a regularization penalty, and Fisher merging uses it as averaging weights. Our gate uses calibrated Fisher statistics as proxies for the ratio. Proposition 3(iii) shows that, up to ties, each superlevel set of the order-preserving soft mask is an optimal support of its own size under the local diagonal surrogate.
Evidence against simpler alternatives. The component ablations below use ViT-B/32, averaged across eight forget tasks.
Table R19. Component ablations, summarized from paper Tables 14–17 (Appendices E.1–E.4).
Adding retain Fisher and dynamically refreshing forget Fisher each improve forgetting on all eight tasks. In paper Table 1, F-TVE-FFT reduces forget accuracy at comparable retention relative to TaLoS and attention-only tuning: 15.93 / 60.90, versus 17.65 / 60.86 and 17.83 / 60.91 (Forget / Retain, %). Against TaLoS, forget accuracy improves from 18.39% to 15.39% on B/16 and from 21.28% to 14.48% on L/14.
The separate mask ablation in Appendix C further supports the ranking rationale: order-preserving Mask (iii) achieves 17.54% forget accuracy, versus 18.66% and 18.17% for the two masks that do not preserve order. Together, these results isolate benefits beyond task-vector negation itself.