acceptodds
Under review as a conference paper at ICLR 2027

Retain-unlearn Gated Gradient Unlearning

Abstract

We propose TVE-Fisher, a simple and principled approach to unlearning in multimodal models that makes task-arithmetic editing selective. We cast unlearning as a retention-constrained local edit and observe an asymmetry of orders: forgetting grows linearly with the edit, whereas retention damage has no first-order term and is governed by Fisher information (FI). Retain curvature should therefore modulate the forget gradient. Under a diagonal geometry, this yields an optimal parameter-selection score, which we relax into an order-preserving soft mask with guarantees on optimality and robustness to estimation error. Concretely, TVE-Fisher computes retain FI statistics once, on a subset of observable retain data under the transductive protocol, to identify parameters critical to retained knowledge. It tracks forget FI statistics dynamically with an exponential moving average to capture the parameters currently relied upon for fitting the forget data. The resulting mask gates gradient updates during forget-side fine-tuning. We then extract a task vector from the difference between the initial and fine-tuned parameters and negate it; the same analysis shows that negation preserves the selected parameters and the retention cost while reversing the forget effect. Extensive experiments on standard vision–language model (VLM) and multimodal large language model (MLLM) unlearning benchmarks across multiple backbones show that TVE-Fisher achieves a stronger forgetting–retention trade-off than related baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Machine Unlearning, Multimodal Models, Fisher Information

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yVkLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TVE-Fisher, a task-vector unlearning method guided by Fisher information. It combines static retain-side importance with dynamically estimated forget-side sensitivity to gate parameter updates during fine-tuning, then negates the resulting task vector to suppress targeted knowledge. Experiments on CLIP benchmarks and multimodal identity QA demonstrate an improved forgetting–retention trade-off over existing task-vector baselines.

Critical strengths
  1. The paper derives a retention-aware parameter selection criterion from a local analysis of forgetting benefits and preservation costs.

  2. Fisher-guided gating produces selective task vectors whose negation strength can be adjusted without adding inference-time modules.

  3. Experiments cover multiple CLIP backbones, eight datasets, and multimodal identity QA, supported by component ablations and parameter-level visualizations.

Critical weaknesses
  1. Task-vector negation is an established approach to unlearning, and the main contribution here lies in Fisher-guided parameter selection during task-vector construction. Selective parameter updates and Fisher-based importance estimation have also been extensively studied. The paper should more clearly distinguish its contribution from these approaches and establish the benefit of its specific gating rule over simpler parameter-selection strategies.

  2. The MLLM evaluation is restricted to LLaVA-1.5-7B on CLEAR. Experiments on MLLMU-Bench and additional MLLM architectures would strengthen the evidence that the method generalizes beyond the predominantly CLIP-based evaluation.

  3. The theoretical guarantees concern small, local parameter edits, whereas the implementation uses multiple AdamW updates, dynamically changing masks, and subsequent task-vector negation. These operations may move beyond the region where the local approximation is reliable. Further analysis is needed to establish how well the theoretical predictions describe the final edits, particularly whether retention costs remain approximately symmetric under negation.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer yVkL (part 1)

Response to Reviewer yVkL (part 1)

We thank the Reviewer for the insightful comments and address each point below with additional results and analysis.

1. Contribution and evidence for the gating rule

Reviewer comment:

Task-vector negation is an established approach to unlearning, and the main contribution here lies in Fisher-guided parameter selection during task-vector construction. Selective parameter updates and Fisher-based importance estimation have also been extensively studied. The paper should more clearly distinguish its contribution from those approaches and establish the benefit of its specific gating rule over simpler parameter-selection strategies.

Response:

Our contribution is a retain-aware selection score motivated by the retention-constrained unlearning objective. Adding the retain term alone lowers mean forget accuracy from 16.91% to 15.86% at comparable retention (60.87% → 60.90%).

Contribution and local rationale. Task-vector negation is the deployment step; our gate shapes the vector during construction by combining fixed retain importance with dynamically updated forget sensitivity. Under the local diagonal surrogate, forget gain is first-order, while retain cost is quadratic. For forget gradient gig_i and positive damped retain curvature hih_i, maximizing the first-order forget gain on support SS under retain budget ε\varepsilon gives squared optimal gain

2ε∑i∈Sgi2hi.2\varepsilon\sum_{i\in S}\frac{g_i^2}{h_i}.

This yields the ranking score gi2/hig_i^2/h_i: forget benefit discounted by retain cost (Proposition 2). Forget-only saliency, such as SalUn-style masks, omits retain cost; EWC uses retain importance as a regularization penalty, and Fisher merging uses it as averaging weights. Our gate uses calibrated Fisher statistics as proxies for the ratio. Proposition 3(iii) shows that, up to ties, each superlevel set of the order-preserving soft mask is an optimal support of its own size under the local diagonal surrogate.

Evidence against simpler alternatives. The component ablations below use ViT-B/32, averaged across eight forget tasks.

Table R19. Component ablations, summarized from paper Tables 14–17 (Appendices E.1–E.4).

AblationControl Forget / Retain (%)F-TVE Forget / Retain (%)Forget reduction (pp)Retain change (pp)
Forget-only → forget + retain Fisher16.91 / 60.8715.86 / 60.901.05+0.03
Separate → joint normalization16.95 / 60.7915.85 / 60.901.10+0.11
Static → dynamic forget Fisher16.43 / 60.3715.86 / 60.900.57+0.53
Random → Fisher-guided mask23.30 / 60.5315.85 / 60.907.45+0.37

Adding retain Fisher and dynamically refreshing forget Fisher each improve forgetting on all eight tasks. In paper Table 1, F-TVE-FFT reduces forget accuracy at comparable retention relative to TaLoS and attention-only tuning: 15.93 / 60.90, versus 17.65 / 60.86 and 17.83 / 60.91 (Forget / Retain, %). Against TaLoS, forget accuracy improves from 18.39% to 15.39% on B/16 and from 21.28% to 14.48% on L/14.

The separate mask ablation in Appendix C further supports the ranking rationale: order-preserving Mask (iii) achieves 17.54% forget accuracy, versus 18.66% and 18.17% for the two masks that do not preserve order. Together, these results isolate benefits beyond task-vector negation itself.

AuthorsOpenReview ↗

Response to Reviewer yVkL (part 2)

Response to Reviewer yVkL (part 2)

2. Computational and storage overhead

Response:

Dynamic Fisher adds forget-Fisher refreshes while retaining O(P)O(P) auxiliary working storage. Within a matched training parameterization, all three variants use the same optimization-step budget and task-vector construction.

Let PP be the number of Fisher/gate coordinates, UU the optimizer-update count, RR the number of forget-Fisher estimates including initialization, and Cr,CfC_r,C_f the costs of one retain/forget Fisher estimate. The accounting below concerns epoch-refreshed weight-space FFT, where vector and Fisher coordinates coincide.

Table R20. Operation and auxiliary-storage accounting (analysis accompanying Appendix B.5, Computational Costs).

OperationStandard task vectorStatic FisherDynamic Fisher
Fine-tuning forward/backward and optimizer updatesCFTC_{\mathrm{FT}}Same budgetSame budget
Task-vector constructionO(P)O(P)O(P)O(P)O(P)O(P)
Retain-Fisher estimation—CrC_rCrC_r
Forget-Fisher estimation—CfC_fRCfR C_f
Calibration, EMA, and mask construction—O(P)O(P)O(RP)O(RP)
Applying gates to optimizer increments—O(UP)O(UP)O(UP)O(UP)
Additional Fisher/mask working state—O(P)O(P)O(P)O(P)
Optional saved Fisher/mask artifacts—O(P)O(P)O(P)O(P) for a fixed number of snapshots
  • Estimation and gating. A diagonal Fisher estimate accumulates squared per-sample gradients: BB gradient evaluations of cost CgC_g require O(B(Cg+P))O(B(C_g+P)) computation. We cache raw retain Fisher; each refresh re-estimates forget Fisher, updates its EMA, and jointly calibrates the statistics to construct the mask. Gating requires no extra backward pass. Estimation requires extra passes unless the required per-sample gradients are already available; with reuse, only accumulation is additional. Squaring an averaged minibatch gradient is not equivalent to averaging squared per-sample gradients.

  • Refresh overhead. Relative to static Fisher, dynamic Fisher adds O((R−1)(Cf+P))O((R-1)(C_f+P)). With one estimate per epoch, R=TR=T and U=TSU=TS for TT epochs and SS updates per epoch; initialization plus a further refresh every epoch instead gives R=T+1R=T+1. Under fixed per-epoch budgets, relative overhead is O((Cf+P)/Cepoch)O((C_f+P)/C_{\mathrm{epoch}}) and does not grow simply with the epoch count. Step-refreshed adapters require the actual refresh count and parameterization-specific projection/merge costs.

  • Storage. Both Fisher variants keep a constant number of diagonal vectors and update EMA recursively, giving O(P)O(P) auxiliary working storage without dense Fisher matrices or epoch-wise histories. This does not imply identical peak GPU memory. For adapters, PP denotes the actual Fisher/gate space; weight-space state need not scale with trainable adapter parameters. After merging the edit, inference requires no Fisher/mask state. Disk overhead separately depends on saved artifacts.

Table R21. Static versus dynamic Fisher accuracy (paper Table 16; Appendix E.3).

VariantMean forget accuracy (%) ↓Mean retain accuracy (%) ↑
Static Fisher16.4360.37
Dynamic Fisher15.8660.90

Dynamic refresh improves mean forget accuracy by 0.57 pp and retain accuracy by 0.53 pp. Runtime and peak-memory measurements are in Appendix B.5, Table 6.

AuthorsOpenReview ↗

Response to Reviewer yVkL (part 3)

Response to Reviewer yVkL (part 3)

3. Broader MLLM evaluation

Reviewer comment:

The MLLM evaluation is restricted to LLaVA-1.5-7B on CLEAR. Experiments on MLLMU-Bench and additional MLLM architectures would strengthen the evidence that the method generalizes beyond the predominantly CLIP-based evaluation.

Response:

We added both: the evaluation now covers two architectures and two benchmarks—LLaVA-1.5-7B on CLEAR and MLLMU-Bench, and Qwen2.5-VL-7B on CLEAR—with consistent gains. Protocols are in Appendix B.4. Results appear in paper Table 3 (§4.2.1) and Tables 4–5 (Appendix B.4), with a summary in §4.2.1.

For CLEAR Identity QA, training uses image–caption pairs and evaluation uses identity questions, testing transfer of suppression across input formats.

Table R22. CLEAR Identity QA with LLaVA-1.5-7B (complete paper Table 3).

MethodForget ↓Retain ↑
Pre-trained model54.7347.57
Standard Task Arithmetic-LoRA37.2544.59
Uniform Merge34.9745.13
TIES-Merging33.5544.45
EMR-Merging32.4743.10
Function Ensemble33.1644.70
MMUnlearner32.2943.05
Ours (F-TVE-LoRA)31.5445.10
  • CLEAR, LLaVA-1.5-7B. F-TVE-LoRA achieves the lowest forget accuracy among the compared methods: 31.54%, versus 37.25% for TA-LoRA, while improving retain accuracy from 44.59% to 45.10%.

  • New architecture: Qwen2.5-VL-7B. F-TVE achieves the best Retain (0.56, versus ≤0.38), Real, and Forget Quality (0.90) among all evaluated unlearning methods in paper Table 4. It improves all four metrics over standard task arithmetic, as summarized below.

  • New benchmark: MLLMU-Bench. Across the 5%, 10%, and 15% forget splits, F-TVE achieves the lowest Test classification accuracy under paraphrased questions and transformed images. Averaged across splits, it gives the largest Test-set suppression (10.65 pp) and smallest retain loss (1.28 pp) relative to Vanilla among the compared methods. Retain accuracy stays within 1.67 pp of Vanilla in every split. Generation, cloze, residual-likelihood, and relearning results are discussed further in our response to Reviewer RVdH (W2).

Table R23. Additional architecture: CLEAR/Qwen2.5-VL-7B (selected rows of paper Table 4).

MethodReal ↑Retain ↑Forget ↓Forget Quality ↑
Non-lin. FT0.180.380.230.82
F-TVE0.220.560.190.90

Table R24. Additional benchmark: MLLMU-Bench/LLaVA-1.5-7B classification (selected entries of paper Table 5).

Forget splitVanilla Test (%)F-TVE Test (%)Vanilla Retain (%)F-TVE Retain (%)
5%43.3133.0943.2941.62
10%43.6732.6041.7040.14
15%43.4232.7644.1843.56
Reviewer_RVdHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TVE-Fisher. TVE-Fisher combines static retain Fisher statistics with dynamic forget-gradient statistics to gate visual-branch fine-tuning, then negates the resulting task vector. A local constrained formulation motivates coordinate selection. Experiments across three CLIP backbones, eight forget datasets, and CLEAR show encouraging gains.

Critical strengths
  1. The local Fisher formulation provides a clear rationale for balancing forgetting against retention, and the authors explicitly discuss its approximation limits.
  2. Table 1 shows consistent average improvements.
  3. Appendices E–F include useful controls for forget-only Fisher, normalization, dynamic statistics, random masks, pseudo-labels, and retain-set size.
Critical weaknesses
  1. The theoretical selection rule is not sufficiently connected to actual updates. Propositions 2–3 concern optimal support selection, whereas the implementation applies continuous gates before AdamW. As Appendix A.9 acknowledges, fixed positive gates can cancel under idealized Adam normalization, so small mask values do not establish sparse updates. The error bounds also depend on conditions that are not quantified for the empirical Fisher and calibrated scores. Please validate the relationship between masks, actual updates, and retain KL.
  2. The unlearning claims should match the evaluation. Classification and Identity QA accuracy establish suppression of measured behaviors, but do not establish removal of information accessible through other prompts or readouts. Task-level suppression is a valid objective; broader knowledge-deletion claims would need targeted recovery tests and other model attack methods. ImageNet retention alone also does not establish preservation of general multimodal capabilities.
  3. The fixed-retain comparison needs clearer data splits. Appendix B.3 selects models at 95% of initial retain accuracy. For ViT-B/16, Table 1 implies a reference threshold of 64.923%, yet Table 5 reports 64.0% for several F-TVE-FFT instances. Please distinguish Fisher-estimation, labeled tuning, and final evaluation sets, and report per-task constraint satisfaction.
  4. More recent machine unlearning methods should be compared.
Questions for authors

See weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer RVdH (part 1)

Response to Reviewer RVdH (part 1)

We thank the Reviewer for the careful reading and valuable questions.

1. Theoretical selection rules, AdamW gating, and actual updates

Reviewer comment:

The theoretical selection rule is not sufficiently connected to actual updates. Propositions 2–3 concern optimal support selection, whereas the implementation applies continuous gates before AdamW. As Appendix A.9 acknowledges, fixed positive gates can cancel under idealized Adam normalization, so small mask values do not establish sparse updates. The error bounds also depend on conditions that are not quantified for the empirical Fisher and calibrated scores. Please validate the relationship between masks, actual updates, and retain KL.

Response:

The gate scales the realized AdamW update, so it is not normalized away. High-mask coordinates contain a disproportionate share of the final edit, and gated edits have lower retain KL than ungated edits at matched forget accuracy on all eight tasks.

AdamW and gate placement. The cancellation argument applies to pre-AdamW gradient gating, as incorrectly described in the submitted §3.3. The reported full-FT and LoRA runs instead gate the parameter increment after AdamW; the implementation and results are unchanged:

old = [p.detach().clone() for p in params]
optimizer.step()
with torch.no_grad():
    for p, p_old, mask in zip(params, old, masks):
        p.copy_(p_old + mask * (p - p_old))

Here, params contains weights for full FT and trainable factors for LoRA. Gating scales the complete increment, including decoupled weight decay, after normalization. §3.3, Algorithm 1, and Appendix A.9 now specify this ordering.

From coordinate scores to realized updates. Propositions 2–3 justify coordinate selection under the local diagonal surrogate. Continuous gating implements this preference through graded attenuation:

Δs,i=mt,iΔs,iAdamW.\Delta_{s,i}=m_{t,i}\Delta^{\mathrm{AdamW}}_{s,i}.

For any coordinate set SS with 0≤mt,i≤τ0\leq m_{t,i}\leq\tau,

\leq \tau^{2}\sum_{i\in S} \bigl(\Delta^{\mathrm{AdamW}}_{s,i}\bigr)^{2}.$$ A gate of 0.1 therefore leaves 1% of the candidate squared update. This exact per-step relation holds in weight space for full FT and factor space for LoRA; positive gates attenuate updates without establishing exact sparsity. **Final-edit and retain-KL diagnostics.** We rank coordinates within each parameter tensor using the initial or final mask, then measure their share of the *same final task vector's* squared norm (**Figure 8, §4.4; Appendix E.5**). **Table R7. Final task-vector energy captured by high-mask coordinates.** | Top mask coordinates | Initial mask: energy share | Final mask: energy share | Uniform reference | |---|---:|---:|---:| | 1% | 3.73% | 4.27% | 1.00% | | 5% | 19.07% | 23.89% | 5.00% | | 10% | 36.46% | 45.11% | 10.00% | The final mask's top 10% captures 45.11% of the edit energy, 4.5 times its uniform share. At matched forget accuracy, gated final edits also have lower retain KL on all eight tasks (**Figure 13, Appendix E.5**), directly measuring the drift addressed by the theory. **Ordering and Fisher-proxy checks.** Proposition 3(iii) establishes optimal supports at every threshold for masks preserving the score ordering under the local diagonal surrogate. Consistently, order-preserving Mask (iii) achieves 17.54% mean forget accuracy versus 18.66% and 18.17% for the other masks, at comparable retention (**Appendix C**). Further checks quantify proxy agreement and stability: - **Pseudo-label agreement:** soft-pseudo versus ground-truth retain Fisher has parameter-level Spearman/Pearson correlations of 0.974/0.977 and layer-level correlations of approximately 0.998 or higher on ViT-B/16 (**Appendix F.1**). - **Effect on unlearning:** hard pseudo-labels, used by F-TVE, yield mean forget accuracies of 17.28/15.42/14.92%, versus 17.54/16.70/16.07% with ground-truth labels on B/32, B/16, and L/14, respectively, at comparable retention in the label ablation (**Appendix F.1**). - **Training stability:** initial and final forget Fisher have tensor-level rank correlation 0.88 on SVHN with ViT-B/32 (**§4.4**). For calibration, **Appendix A.7-A.8** bounds the effective retain-denominator range below 2, guaranteeing at least a $2^{-|1-\lambda|/2}$ fraction of frontier value under its assumptions. Theorem 4 remains conditional on coordinate-wise score distortion in $[a,b]$; our empirical checks do not establish those constants.
AuthorsOpenReview ↗

Response to Reviewer RVdH (part 2a)

Response to Reviewer RVdH (part 2a)

2a. Scope of unlearning and retention claims.

Reviewer comment:

The unlearning claims should match the evaluation. Classification and Identity QA accuracy establish suppression of measured behaviors, but do not establish removal of information accessible through other prompts or readouts. Task-level suppression is a valid objective; broader knowledge-deletion claims would need targeted recovery tests and other model attack methods. ImageNet retention alone also does not establish preservation of general multimodal capabilities.

Response:

The evaluation below supports suppression beyond the training readout while preserving non-target performance beyond ImageNet. We add MLLMU-Bench with LLaVA-1.5-7B and CLEAR with Qwen2.5-VL-7B, covering transformed inputs, generation, cloze completion, and likelihood-based readouts (Tables 4–5; Appendix B.4).

  • Generalization of suppression. On MLLMU-Bench, F-TVE produces the largest average Test classification drop under paraphrased questions and transformed images (10.65 pp), and the lowest Forget ROUGE-L and cloze accuracy at every split.
  • Retention. F-TVE has the smallest average Retain classification loss (1.28 pp). Its Real Celebrity loss is smaller than Non-lin. FT and GA. NPO preserves more Real Celebrity accuracy but achieves substantially less Test suppression.
  • Recovery and broader preservation. Part 2b presents the second-architecture results, residual-likelihood and relearning diagnostics, and cross-task retention analysis. Our claims concern selective suppression under these evaluated protocols.

Table R8. MLLMU-Bench classification changes relative to Vanilla (percentage points), averaged over the 5%, 10%, and 15% forget splits (paper Table 5).

MethodTest change ↓Retain change ↑Real Celebrity change ↑
Non-lin. FT−8.07−3.97−10.04
GA−5.16−6.14−9.45
GD−4.01−4.39−9.50
KL−1.47−6.47−7.95
NPO−2.52−1.93−4.02
F-TVE (Ours)−10.65−1.28−5.22

Tables R9–R14 reproduce the split-specific results from paper Table 5; Tables R11–R14 continue in part 2b.

Table R9. MLLMU-Bench, 5% forget split: Forget and Test.

MethodForget Class. ↓Forget ROUGE ↓Forget Fact. ↓Forget Cloze ↓Test Class. ↓Test ROUGE ↓Test Fact. ↓Test Cloze ↓
Vanilla (before unlearning)50.050.6306.1419.5743.310.4233.9213.74
Non-lin. FT43.570.5163.2112.4835.170.3123.1712.32
GA42.960.5083.0613.0434.730.3082.959.71
GD43.170.5072.8012.3238.360.3043.069.54
KL44.450.5224.5115.1741.000.2973.6012.29
NPO44.050.5433.1817.3539.370.2883.0111.93
F-TVE (Ours)40.580.4902.9310.3933.090.2952.7410.30

Table R10. MLLMU-Bench, 5% forget split: Retain and Real Celebrity.

MethodRetain Class. ↑Retain ROUGE ↑Retain Fact. ↑Retain Cloze ↑Real Celebrity Class. ↑Real Celebrity ROUGE ↑Real Celebrity Fact. ↑Real Celebrity Cloze ↑
Vanilla (before unlearning)43.290.6685.9021.7650.880.4104.5411.27
Non-lin. FT38.510.5803.8214.7741.080.3472.788.62
GA37.290.4772.9714.8543.790.3182.755.63
GD39.210.5883.8613.3845.120.2832.625.81
KL35.550.4913.7916.7643.820.3332.929.14
NPO40.430.5943.9316.9846.950.3513.769.74
F-TVE (Ours)41.620.6044.1116.5445.350.3943.1610.17

See the continuation (part 2b) in the nested response below.

AuthorsOpenReview ↗

Response to Reviewer RVdH (part 2b)

Response to Reviewer RVdH (part 2b)

2b. Scope of unlearning / retention claims.

Table R11. MLLMU-Bench, 10% forget split: Forget and Test.

MethodForget Class. ↓Forget ROUGE ↓Forget Fact. ↓Forget Cloze ↓Test Class. ↓Test ROUGE ↓Test Fact. ↓Test Cloze ↓
Vanilla (before unlearning)48.570.5945.8920.8343.670.4024.1815.63
Non-lin. FT41.630.4493.1010.9435.710.3503.247.21
GA43.710.5163.6817.0641.790.3373.7710.02
GD41.890.5173.1815.0939.920.3363.469.87
KL45.020.5844.1818.3643.490.3352.8813.13
NPO47.360.5204.9917.7042.180.3843.6213.68
F-TVE (Ours)38.750.4032.968.3832.600.3222.986.39

Table R12. MLLMU-Bench, 10% forget split: Retain and Real Celebrity.

MethodRetain Class. ↑Retain ROUGE ↑Retain Fact. ↑Retain Cloze ↑Real Celebrity Class. ↑Real Celebrity ROUGE ↑Real Celebrity Fact. ↑Real Celebrity Cloze ↑
Vanilla (before unlearning)41.700.5794.7721.2150.880.4104.5411.27
Non-lin. FT38.550.4582.7114.5640.610.3413.038.70
GA36.580.4622.5814.0441.560.2652.365.85
GD38.470.4632.6612.2639.310.3282.686.30
KL35.230.4462.9214.8944.050.3992.229.02
NPO40.430.4764.2915.6547.190.3783.4110.12
F-TVE (Ours)40.140.5133.9717.0845.980.3853.579.56

Table R13. MLLMU-Bench, 15% forget split: Forget and Test.

MethodForget Class. ↓Forget ROUGE ↓Forget Fact. ↓Forget Cloze ↓Test Class. ↓Test ROUGE ↓Test Fact. ↓Test Cloze ↓
Vanilla (before unlearning)50.730.5675.7820.4643.420.3973.1015.76
Non-lin. FT44.040.4472.7811.9335.320.3182.2510.90
GA40.160.4853.4313.3538.410.2802.6111.42
GD43.390.5323.8315.0740.100.3642.4112.44
KL46.550.5284.3418.6141.500.3522.1213.56
NPO44.720.5134.2515.7241.290.3832.2413.85
F-TVE (Ours)40.630.4272.5110.6232.760.2871.899.44

Table R14. MLLMU-Bench, 15% forget split: Retain and Real Celebrity.

MethodRetain Class. ↑Retain ROUGE ↑Retain Fact. ↑Retain Cloze ↑Real Celebrity Class. ↑Real Celebrity ROUGE ↑Real Celebrity Fact. ↑Real Celebrity Cloze ↑
Vanilla (before unlearning)44.180.5974.8321.8750.880.4104.5411.27
Non-lin. FT40.190.5142.8015.0140.830.3032.708.45
GA36.880.4013.2614.3438.940.3312.686.29
GD38.310.4972.8416.9039.720.3742.767.53
KL38.980.4553.7617.2340.910.3382.458.75
NPO42.530.5243.5617.1446.440.3313.168.33
F-TVE (Ours)43.560.5563.9417.7945.650.3973.049.07

A second architecture / likelihood-based readouts. CLEAR/Qwen2.5-VL-7B aggregates answer overlap and likelihood in Real, Retain, and Forget; Forget Quality measures agreement with a retain-only gold reference (Appendix B.4).

Table R15. CLEAR with Qwen2.5-VL-7B (paper Table 4).

MethodReal ↑Retain ↑Forget ↓Forget Quality ↑
Gold0.290.700.111.00
Original0.300.710.450.88
GD0.190.130.030.52
GA0.160.090.010.67
KL0.140.050.000.62
NPO0.210.290.150.79
Non-lin. FT0.180.380.230.82
F-TVE (Ours)0.220.560.190.90

F-TVE has the highest Retain (0.56, vs. at most 0.38), Real (0.22), and Forget Quality (0.90) among the unlearning methods. It improves all 4 metrics over Non-lin. FT. GA, GD, and KL achieve lower Forget scores along severe Retain degradation to 0.13 or below.

Residual likelihood / relearning. On the MLLMU-Bench 10% split, we measure forget answers below the 95th-percentile Vanilla NLL threshold (0.31, rounded). We also relearn on half of each identity's QA pairs and evaluate recovery on the other half (Appendix F.3, Fig. 14–15).

Table R16. Residual likelihood / relearning.

DiagnosticF-TVENon-lin. FTOther references
Forget answers below the Vanilla NLL threshold25%40%Threshold 0.31 (rounded)
Recovery of the gap to Vanilla after 50 relearning steps63%71%GA, GD, KL, NPO: 76–87%; Gold: 9%

F-TVE leaves fewer low-NLL forget answers than Non-lin. FT and shows the least recovery among the evaluated unlearning methods under this protocol.

Preservation beyond ImageNet: We also evaluate each deployed CLIP edit on all 8 visual tasks and ImageNet. Target accuracy drops by 15.2–48.7 pp, compared with 0.9–3.1 pp on ImageNet; every off-target change is smaller than its corresponding target drop (Cross-task accuracy:§4.4, Fig. 11). Along unit-norm task vectors, forget-probe feature sensitivity exceeds retain-probe sensitivity on all eight tasks (§4.4, Fig. 9, JVP figure), supporting local selectivity at the feature level.

All results support selective suppression/preservation across the evaluated readouts, transformations, architectures, and recovery protocol (Appendices B.4 and F.3). They do not establish irreversible deletion or preservation of all multimodal capabilities.

Reviewer_zak9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TVE-Fisher, which constructs a soft mask using fixed retain-side Fisher statistics and dynamically updated unlearn-side Fisher statistics. The mask modulates gradients during fine-tuning, after which the resulting task vector is extracted and negated to improve the unlearning–retention trade-off. The authors evaluate the method across multiple CLIP backbones, eight visual datasets, and the CLEAR multimodal benchmark.

Critical strengths

The method explicitly considers both sensitivity to the unlearning target and preservation of retained capabilities when constructing task vectors, providing a clear design rationale. The experimental and ablation results indicate that Fisher-guided gating improves unlearning effectiveness at comparable retain-set accuracy.

Critical weaknesses

The paper lacks a comparison with the directly relevant Task Simplex Arithmetic method [1] under the same evaluation protocol. This limits the evidence for its effectiveness and efficiency advantages over that approach.

The paper does not compare training-time gating with applying a Fisher-based mask to the task vector after full fine-tuning. The benefit of applying the mask during training therefore remains unclear. Furthermore, the additional runtime and GPU memory overhead of re-estimating Fisher statistics each epoch are not sufficiently reported.

Questions for authors

Could you provide a comparison of the performance and computational costs of Task Simplex Arithmetic [1] and its distilled version? How much additional runtime and GPU memory does dynamic Fisher estimation require compared with standard fine-tuning and the static-Fisher variant?

[1] Dong, J., Zhu, H., Zhang, Y., Qu, X., Ong, Y.-S., and Koniusz, P. Machine Unlearning via Task Simplex Arithmetic. NeurIPS, 2025.

4 comments
AuthorsOpenReview ↗

Response to Reviewer zak9 (part 1)

Response to Reviewer zak9 (part 1)

We thank the Reviewer for valuable questions that help us improve our work.

1. Comparison with Task Simplex Arithmetic

Reviewer comment:

The paper lacks a comparison with the directly relevant Task Simplex Arithmetic method [1] under the same evaluation protocol.

Related question: Provide a comparison of the performance and computational costs of Task Simplex Arithmetic [1] and its distilled version.


Response:


Main findings:

  • F-TVE-FFT obtains lower average forget accuracy than both TSA variants on all three backbones: 15.93%, 15.39%, and 14.48%, compared with 17.86%, 16.80%, and 14.94% for TSA and 18.16%, 17.18%, and 15.25% for distilled TSA.
  • The reductions relative to TSA are 1.93, 1.41, and 0.46 percentage points, with retain-accuracy differences of −0.05, +0.16, and −0.11 points.


Comparison with TSA and its distilled version.

BackboneMethodForget ↓ / Retain ↑ (%)Time (min)Peak memory (GiB)
B/32TSA17.86 / 60.9584.014.0
TSA (distilled)18.16 / 60.9188.014.3
F-TVE-FFT15.93 / 60.9017.04.3
B/16TSA16.80 / 65.3088.020.9
TSA (distilled)17.18 / 65.3492.021.2
F-TVE-FFT15.39 / 65.4617.011.3
L/14TSA14.94 / 72.66268.047.9
TSA (distilled)15.25 / 72.64281.049.0
F-TVE-FFT14.48 / 72.5537.914.3

Mean over 8 forget tasks.

Effectiveness. On every backbone, F-TVE reaches lower forget accuracy than TSA (by 1.93, 1.41 and 0.46 pp) and than distilled TSA (by 2.23, 1.79 and 0.77 pp), while retain accuracy stays within ±0.16 pp.

Efficiency. F-TVE builds a single task vector from one fine-tuning run, whereas TSA trains and ensembles several. As a result, F-TVE is 4.9–7.1× faster than TSA (5.2–7.4× than distilled TSA) and needs 1.9–3.4× less peak memory.

Per-task behaviour. The largest gains appear on fine-grained and scene-level tasks such as Cars and RESISC45: on B/32, F-TVE reaches 16.5% on Cars against 38.1% for TSA, and 11.7% on RESISC45 against 19.5%. Full per-task tables are in App. B.6.


Moreover:

  • Table R1 below compares paradigms; Tables R2/R3 provide the per-task values directly on ViT-B/32 /ViT-B/16 (results for all three backbones are provided in Appendix B.6, Tables 8–10).
  • These comparisons are collected in Appendix B.6, Detailed Comparison; the non-distilled main results also appear in paper Table 1, with backbone-specific breakdowns in paper Table 2 and Appendix D, Tables 12–13.
  • Lower forget accuracy indicates stronger target suppression; retain accuracy measures preservation.



Table R1. Comparison of task-vector unlearning paradigms.

Task ArithmeticTask Simplex ArithmeticTVE-Fisher (ours)
Single fine-tuning for unlearning✓✗✓
Free extra inference cost✓✗✓
Retain-shaped edit direction✗✗✓
Coordinate-wise update scaling✗✗✓
Retention-constrained optimality✗✗✓



Table R2. ViT-B/32 per-dataset forget / retain accuracy (%).

MethodCarsDTDEuroSATGTSRBMNISTRESISC45SUN397SVHN
TSA38.1 / 60.927.8 / 61.14.1 / 60.31.6 / 61.21.5 / 61.219.5 / 61.047.5 / 60.92.8 / 61.0
TSA (Distillation)39.8 / 60.627.2 / 61.34.6 / 60.41.4 / 60.81.8 / 61.520.6 / 60.846.7 / 61.03.2 / 60.9
F-TVE-FFT16.5 / 60.223.6 / 60.210.1 / 62.41.5 / 61.09.5 / 61.611.7 / 60.248.0 / 60.26.7 / 61.5
F-TVE-LoRA16.8 / 60.224.3 / 60.211.7 / 62.32.3 / 60.59.9 / 62.312.4 / 60.248.0 / 60.27.0 / 60.6
F-TVE-XGBLoRA16.1 / 60.224.3 / 60.39.2 / 62.11.3 / 60.48.2 / 61.211.1 / 60.248.1 / 60.26.4 / 60.7



Table R3. ViT-B/16 per-dataset forget / retain accuracy (%).

MethodCarsDTDEuroSATGTSRBMNISTRESISC45SUN397SVHN
TSA31.8 / 65.623.8 / 65.02.2 / 65.50.9 / 65.41.1 / 65.421.1 / 65.151.5 / 65.22.0 / 65.2
TSA (Distillation)31.1 / 65.425.0 / 65.32.6 / 65.41.1 / 65.80.9 / 65.122.6 / 65.352.4 / 65.11.7 / 65.3
F-TVE-FFT17.6 / 64.919.5 / 65.07.9 / 65.61.6 / 65.69.1 / 66.014.1 / 65.247.9 / 65.05.4 / 66.1
F-TVE-LoRA18.2 / 65.124.7 / 65.58.5 / 66.02.7 / 65.010.5 / 66.515.8 / 65.348.5 / 64.95.2 / 66.2
F-TVE-XGBLoRA17.5 / 65.419.9 / 65.08.3 / 65.22.6 / 64.99.6 / 66.215.2 / 65.549.0 / 65.15.2 / 67.0
AuthorsOpenReview ↗

Response to Reviewer zak9 (part 2)

Response to Reviewer zak9 (part 2)

2. Runtime and memory overhead of Fisher estimation.

Reviewer comment:

Furthermore, the additional runtime and GPU memory overhead of re-estimating Fisher statistics each epoch are not sufficiently reported.

Related question:

How much additional runtime and GPU memory does dynamic Fisher estimation require compared with standard fine-tuning and the static-Fisher variant?

Response:

  • Appendix B.5, Computational Costs, contains the resource comparison reproduced in Table R6 below.
  • This table separates total runtime, peak GPU memory, Fisher-estimation time, and mask time for each training parameterization.
  • Each method is compared with its corresponding Non-lin FT baseline. Dynamic FFT refreshes Fisher statistics by epoch; the adapter variants use their configured step-based refresh schedule.



Table R6. Resource costs, reproduced from Appendix B.5 (Computational Costs). Time is in minutes and peak memory in MiB. Fisher and mask entries list B/32 / B/16 / L/14 times.

MethodB/32 time / peakB/16 time / peakL/14 time / peakFisherMask
Non-lin FT (FFT)9.8 / 36209.6 / 1082026.1 / 11910——
Non-lin FT (LoRA)9.7 / 24609.5 / 763027.0 / 8050——
Non-lin FT (XGBLoRA)9.7 / 24609.5 / 763023.2 / 8050——
F-TVE dynamic (FFT)17.0 / 438217.0 / 1157637.9 / 146025.40 / 5.51 / 9.621.72 / 1.75 / 1.85
F-TVE static (FFT)10.7 / 406010.5 / 1125027.8 / 134500.80 / 0.81 / 1.410.02 / 0.02 / 0.02
F-TVE dynamic (LoRA)14.0 / 589213.9 / 1065234.4 / 197124.12 / 4.21 / 7.200.18 / 0.16 / 0.21
F-TVE static (LoRA)10.7 / 589210.5 / 1065228.7 / 197120.94 / 0.96 / 1.670.01 / 0.01 / 0.01
F-TVE dynamic (XGBLoRA)14.8 / 555014.7 / 1004031.7 / 185804.68 / 4.77 / 8.050.40 / 0.40 / 0.40
F-TVE static (XGBLoRA)10.6 / 488010.4 / 883024.8 / 163400.88 / 0.89 / 1.570.01 / 0.01 / 0.01
TSA84 / 1435088 / 21360268 / 49030——
TSA (Distillation)88 / 1469092 / 21690281 / 50190——
  • Relative to matched Non-lin FT, dynamic F-TVE takes 1.45–1.77× the runtime for FFT, 1.27–1.46× for LoRA, and 1.37–1.55× for XGBLoRA.
  • Static Fisher reduces these factors to 1.06–1.11×.
  • Relative to static Fisher, dynamic refreshes add 6.3 / 6.5 / 10.1 minutes for FFT, 3.3 / 3.4 / 5.7 minutes for LoRA, and 4.2 / 4.3 / 6.9 minutes for XGBLoRA, in B/32 / B/16 / L/14 order.


Moreover:

  • Fisher estimation accounts for 5.40–9.62 minutes in dynamic FFT, 4.12–7.20 minutes in dynamic LoRA, and 4.68–8.05 minutes in dynamic XGBLoRA.
  • Mask processing adds 1.72–1.85, 0.16–0.21, and 0.40 minutes, respectively.
  • Fisher estimation is the dominant additional runtime component in these configurations. The additional computation also improves performance: the static–dynamic ablation in Appendix E.3, Table 16 shows that dynamic refresh lowers mean forget accuracy from 16.43% to 15.86% and raises retain accuracy from 60.37% to 60.90% on ViT-B/32.


Finally:

  • Peak-memory increases over matched Non-lin FT are 762 / 756 / 2,692 MiB for dynamic FFT, 3,432 / 3,022 / 11,662 MiB for dynamic LoRA, and 3,090 / 2,410 / 10,530 MiB for dynamic XGBLoRA.
  • Relative to static Fisher, the increases are 322 / 326 / 1,152 MiB, 0 / 0 / 0 MiB, and 670 / 1,210 / 2,240 MiB, respectively.
  • Equal static and dynamic LoRA peaks indicate that more frequent computation need not raise the peak working set. Fisher estimation and gating are required only during task-vector construction. Deployment uses a single edited checkpoint with no additional inference overhead.
AuthorsOpenReview ↗

Response to Reviewer zak9 (part 3)

Response to Reviewer zak9 (part 3)

3. Training-time vs post-hoc gating.

Reviewer comment:

The paper does not compare training-time gating with applying a Fisher-based mask to the task vector after full fine-tuning. The benefit of applying the masking during training therefore remains unclear.

Response:

Is gating during training better than applying a Fisher mask to the task vector after fine-tuning?
Yes, and by a large margin. Gating during training improves on task arithmetic 2.3–5.8× more than post-hoc Fisher masking does.

Mean Forget↓ / Retain↑ (%)B/32B/16L/14
Task Arithmetic (FFT)24.10 / 60.7521.00 / 65.3316.94 / 72.49
Post-hoc Fisher mask on the task vector22.62 / 60.7620.03 / 65.8015.89 / 72.76
F-TVE-FFT (training-time gating)15.93 / 60.9015.39 / 65.4614.48 / 72.55

Result:

  • Post-hoc masking improves on task arithmetic by only 1.0–1.5 pp.
  • Compared with post-hoc masking, training-time gating reduces forget accuracy by 6.69, 4.64, and 1.41 percentage points on B/32, B/16, and L/14. Both variants are evaluated under the same protocol.
  • Retain accuracy changes by +0.14, −0.34, and −0.21 points.
  • Post-hoc masking improves over Task Arithmetic, but does not recover the forgetting performance of training-time gating.

Why: A post-hoc mask can only rescale a direction that has already been learned. Gating during training changes the optimization path itself, so the forget task is learned through retain-safe coordinates from the start.

This supports the benefit of shaping the optimization trajectory during task-vector construction, rather than only rescaling the completed task vector.

Thanks for the response. As my socre is positive, I keep it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.