acceptodds
Under review as a conference paper at ICLR 2027

TASE: Ultra-Sparse Vector-Level Unlearning via Token-Anchored Surgical Editing

Abstract

Current approximate unlearning methods aim to suppress the influence of target information and preserve general capabilities. Achieving both objectives remains challenging since Gradient Ascent incurs substantial training cost and inconsistent retained utility, while parameter-efficient LoRA-based approaches still struggle to achieve strong forgetting without sacrificing retained utility. Building on our observation that sparse edits to only a small set of forget-associated token embeddings suffice to alter target outputs, we propose TASE, a token-anchored approximate unlearning method with extreme parameter efficiency. TASE exploits token distribution differences between the forget and retain sets to construct a small anchored token set that maximizes forget-sample coverage while minimizing retain-side interference. Embedding-only surgical editing then updates only the corresponding embedding rows, achieving targeted knowledge suppression and retained-utility preservation through ultra-sparse vector-level unlearning. The anchored-token sparse update strategy further supports token-based unlearning requests, reducing data exposure in remote model update settings. Experiments on BERT, Phi-1.5B, and Llama-2-7B across classification, TOFU, and MUSE demonstrate competitive forgetting and retained utility across encoder and decoder architectures. TASE modifies 2-5 orders of magnitude fewer parameters than existing approaches, while reducing peak GPU memory and GPU-hours by at least 30%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Machine Unlearning, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_q75QOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TASE enabling a very sparse method for LLM unlearning. The authors claim that they observe that editing only a few token embeddings can change a model’s outputs on the forget set. TASE tries to pick a few “anchor” tokens that are common in the forget set but rare in the retain set. It then applies gradient ascent only to the input embeddings of these tokens and keeps everything else frozen. They claim that editing needs no retain-set gradients and no reference model. The experiments show forgetting and retained utility are competitive with or better than the other baselines.

Critical strengths

The paper has several highlights.

The observation is simple and interesting. Editing only a few input embeddings is enough to strongly suppress target outputs in a 7B model.

The method is cost-efficient. It needs neither retain-set optimization nor a reference model. It also uses less GPU memory and time than the baselines.

The experiments have broad coverage. They include encoder and decoder models, three to four model families, and both Q&A and long-document unlearning. The paper also reports extra checks such as MMLU, WMDP-Cyber, prompt robustness, recovery attacks, membership inference and ablations.

Critical weaknesses

Several concerns about the paper.

First, the core assumption is a bit questionable. So far as we know, factual knowledge in LLMs is mainly stored in the MLP and attention layers, not in input embeddings. Editing the embeddings of anchor tokens may only stop the model from recognizing the entity in the prompt. Maybe the knowledge itself still remain in the weights.

Second, editing a token embedding affects every input that contains this token. This can be dangerous as other entities that share the same name fragments would also be affected. Unfortunately, retain-set metrics and MMLU cannot detect this damage.

Als, the results are not consistent across benchmarks. TASE works very well on TOFU, however, it is a bit weaker on other settings, such as Phi-1.5B forget10. The method may depend on benchmark properties.

Questions for authors
  1. Regarding the concern in the Weaknesses part, does TASE really remove knowledge, or only block recognition of the entity in the input?

  2. How does TASE perform when the target knowledge has no unique tokens? For example, "real people with common names".

  3. What happens to unrelated inputs that contain the anchor tokens?

  4. For evaluation, could you compare with LoRA-based and any other localized unlearning methods (such as the following listed ones)? -SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation -Who's Harry Potter? Approximate Unlearning in LLMs ...

5 comments
AuthorsOpenReview ↗

Regarding the concern in the Weaknesses part, does TASE really remove knowledge, or only block recognition of the entity in the input?

TASE reduces the influence of targeted information on generated outputs through sparse edits to token embedding rows. These edits propagate through the fixed Transformer layers, changing contextualized representations involved in answer generation.

Representation Propagation (Appendix A.4.5, Figure 5). We traced the effects of embedding edits across Transformer layers. The representation drift progressively increases in both anchor and answer regions, while the first-order analysis captures the induced changes with high fidelity at moderate perturbation scales. This indicates that localized embedding edits induce changes that propagate through the Transformer and influence downstream answer representations.

Cross-prompt Robustness (Appendix A.4.4, Table 11). We evaluated target-answer recovery across eight prompt formulations, including paraphrases, aliases, multilingual queries, adversarial extraction, keyword-free inputs, and compositional reasoning. Target-answer probabilities remain low across all formulations. Although adversarial extraction, keyword-free, and compositional queries exhibit some residual lexical overlap with target answers, their target-answer probabilities remain low, indicating that TASE maintains effective suppression of target-answer generation even under these indirect formulations. These results demonstrate that TASE's suppression extends beyond the original entity wording to alternative formulations of the targeted knowledge.

Together, these results show that TASE induces downstream representation changes and reduces target-information reproduction across different query formulations, supporting representation-mediated unlearning rather than input-string-specific recognition blocking.

AuthorsOpenReview ↗

How does TASE perform when the target knowledge has no unique tokens? For example, "real people with common names".

TASE's anchor selection relies on relative Forget–Retain frequency contrasts at the subword level, rather than requiring an exclusive whole-name token. Shared name fragments and other eligible subwords can still provide useful selection signals when their occurrence frequencies differ between Forget and Retain data.

Distinctiveness analysis. For each entity in TOFU, its 20 QA examples were compared against those of the other entities. D@5 measures the mean log(1 + frequency ratio) of the five highest-scoring eligible tokens. Based on anchor distinctiveness, we divide all entities into three groups:

Distinctiveness groupEntity countUnique QA Coverage
Low6756.3%
Medium6688.2%
High6795.1%

Here, Unique QA Coverage measures the proportion of an entity's QA examples containing at least one eligible token absent from the other 199 entities. Then, we constructed matched Low- and High-distinctiveness forgetting requests from the standard forget10 author pool. Each request contains six authors (120 Forget QA), with the remaining 194 authors (3,880 QA) serving as Retain data. Both requests share the same initialization. The Low request has 68.3% Unique QA Coverage, compared with 100% for High. Results are reported as mean ± sample SD:

GroupForget Coverage ↑Retain Exposure ↓FTR ↑F-RG ↓R-RG ↑MU ↑
Low96.1% ± 2.7%24.6% ± 1.8%0.709 ± 0.0170.306 ± 0.0730.896 ± 0.0220.601 ± 0.006
High100.0% ± 0.0%13.3% ± 0.2%0.719 ± 0.0340.253 ± 0.0810.980 ± 0.0050.613 ± 0.002

TASE remains effective as anchor distinctiveness decreases. In the Low group, where nearly one-third of the Forget QA contain no unique eligible token, TASE still achieves 96.1% Forget Coverage. Compared with the High group, lower distinctiveness increases Retain Exposure from 13.3% to 24.6% and moderately reduces R-RG from 0.980 to 0.896, while FTR and overall model utility remain close. These results show that relative subword-frequency contrasts continue to provide effective localization signals even when unique-token availability is substantially reduced.

QA-level zero-unique analysis. We further examined TOFU's token characteristics together with its unlearning behavior. Among the 640 standard-split Forget QA examples, 65 (10.2%) contain no unique eligible token relative to their designated standard-split Retain vocabulary. Thirty of these examples have corresponding controlled per-sample outputs, meaning that each QA was individually evaluated using outputs generated by the edited model under the same controlled Low/High setting. These zero-unique QA exhibit higher residual target-answer probability and ROUGE than the non-zero-unique examples in the same evaluation. Despite this increased difficulty, the Low-distinctiveness request containing such examples still achieves high Forget Coverage together with 0.896 R-RG and 0.601 MU. The results therefore show that TASE continues to achieve a favorable forgetting–retention balance when the forgetting request includes QA examples without unique eligible tokens.

Together, these results show that TASE does not rely on exclusive entity-name tokens. Relative subword-frequency differences remain sufficient to identify effective anchors under reduced token distinctiveness, enabling broad target coverage while preserving retained knowledge for shared-name and common-name targets.

AuthorsOpenReview ↗

What happens to unrelated inputs that contain the anchor tokens?

Unrelated inputs that contain edited anchor tokens can experience some degradation because they use the same modified embedding rows. TASE limits this effect by reducing Retain Exposure during anchor selection and constraining the embedding updates, allowing most retained knowledge to remain well preserved.

Matched collision–control analysis. We evaluated 153 anchor-exposed Retain QA and matched unexposed controls across 31 retained-entity clusters. For each pair, we compared the model change from Base to TASE and report the difference between exposed and control groups using 10,000 cluster-bootstrap resamples.

MetricDifference-in-differences95% CI
Exact Match−0.719[−0.878, −0.462]
ROUGE-L−0.464[−0.732, −0.204]
Gold-answer probability−0.576[−0.784, −0.351]
Gold-answer NLL+4.201[+1.549, +7.132]

The matched comparison shows that shared-anchor exposure reduces retained-answer recovery and Gold-answer probability while increasing answer NLL. Thus, the retention effect is concentrated on the subset of Retain inputs that actually contain edited anchors.

Per-sample retain analysis. We further examined 2,400 retained-example evaluations corresponding to 800 unique Retain QA from the existing anchor-distinctiveness experiments (Q2). Each evaluation was grouped according to whether the input contained at least one selected and edited anchor.

Retain evaluation groupEvaluation recordsGold-answer probability ↑Gold-answer NLL ↓ROUGE-L ↑
Anchor-hit4530.7211.9910.747
Anchor-free1,9470.9860.0250.982

Anchor-hit examples show lower retained-answer performance, whereas the much larger Anchor-free group preserves its original answers well. Retained-answer quality also decreases as anchor exposure increases.

As a result, although shared-anchor inputs can experience local degradation, TASE keeps such exposure limited and retains strong overall Retain performance. The selector controls which retained examples are exposed, while the constrained update controls how strongly the selected embedding rows are modified, together preserving most unrelated knowledge while maintaining effective forgetting.

Reviewer_sivBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TASE proposes a token-based efficient machine unlearning method. Instead of retraining the entire model or using LoRA, it only modifies the embedding vectors of tokens that are highly associated with the forget data, while achieving forgetting and preserving the model's other capabilities.

The authors evaluate TASE on BERT, Phi-1.5B, and Llama-2-7B using classification tasks, TOFU, and MUSE benchmarks. Experimental results show that TASE achieves competitive forgetting performance and retained utility across both encoder and decoder architectures. Compared with existing methods, TASE modifies 2–5 orders of magnitude fewer parameters and reduces peak GPU memory usage and GPU-hours by at least 30%.

Critical strengths

1. High parameter efficiency and low computational cost.

TASE avoids the large-scale parameter updates required by full-parameter unlearning methods. Experimental results show that the TASE significantly reduces the number of trainable parameters, as well as peak GPU memory usage and GPU-hours.

2: Extensive experimental evaluation across different model architectures and unlearning tasks.

The authors evaluate TASE not only on BERT-based classification tasks but also on generative models such as Llama-2-7B and Phi-1.5B, covering different unlearning benchmarks, including TOFU and MUSE. In addition, the paper provides further experiments and analyses on different forgetting ratios, prompt robustness, recovery robustness, and embedding update mechanisms. These evaluations provide relatively comprehensive experimental evidence to evaluate the performance and effectiveness of the proposed method.

Critical weaknesses

1. Limited Methodological Novelty.

TASE combines anchored token selection, sparse embedding updates, gradient normalization, and norm preservation. However, related methods such as Parameter-Efficient Token Embedding Editing for Clinical Class-Level Unlearning and Don't Forget Your Embeddings: Robust Knowledge Erasure via Precise Editing of Embeddings have already explored token-level selective unlearning and targeted embedding editing. The paper does not clearly establish how TASE differs from these approaches or which specific design choices contribute to its performance improvements.

Weakness 2: Insufficient Analysis of Discrepancies from Retraining.

Although TASE achieves promising forgetting and retained utility, its effectiveness relative to retraining remains unclear. In Appendix A.4.6, the Forget Quality p-values for forget01 (2.1564 × 10⁻⁵) and forget10 (0.007945) indicate statistically significant differences from the retraining reference. Furthermore, Table 2 reports a Privacy Leakage score of 26.74, raising concerns about residual information leakage. However, the paper does not sufficiently analyze these discrepancies or explain their implications for the effectiveness of unlearning.

Questions for authors
  1. What are the key methodological differences between TASE and existing embedding-based unlearning methods, such as STEU and EMBER? Which components of TASE contribute most significantly to its performance improvements?

  2. How do the authors explain the statistically significant differences from retraining observed in the Forget Quality evaluation (Appendix A.4.6)? Additionally, how should the Privacy Leakage score of 26.74 in Table 2 be interpreted in terms of residual information leakage and unlearning effectiveness?

7 comments
AuthorsOpenReview ↗

What are the key methodological differences between TASE and existing embedding-based unlearning methods, such as STEU and EMBER?

We compare TASE with STEU [1] and EMBER [2] along three aspects: anchor selection, selective optimization, and the editing interface/infrastructure.

ComponentSTEUEMBERTASE
Anchor selectionPMI-based token rankingSparse MF + concept-feature identificationForget–Retain Frequency Ratio / Fisher sensitivity
Selective optimizationSelected embeddings + classifier head; Forget/Retain optimizationConcept-feature subtraction from selected embeddingsSelected anchor embeddings only, with row-wise normalization and norm preservation
Editing interface/infrastructureFull Forget and Retain batchesSparse MF + LLM-assisted feature identificationKeyword-only mode supported; no Retain gradients or external feature-identification model

Overall, TASE localizes each unlearning request to a request-specific anchor set and restricts optimization to the corresponding embedding rows. Compared with STEU [1], this removes the need to update an additional classifier head or repeatedly optimize a Retain-side objective, while TASE's Keyword-only mode further allows editing from compact anchor-based inputs rather than the original Forget samples. Compared with EMBER [2], TASE directly obtains its editable support from token statistics or gradient sensitivity and derives the edit direction from the forget objective, avoiding EMBER's additional sparse matrix factorization and LLM-assisted feature-identification pipeline. Owing to its simple and general formulation, the same sparse editing mechanism is preserved across both encoder and decoder settings.

References below will be added to the revised version: [1] AIT HOU I, BORAD S, SHARMA H, et al. Parameter-efficient token embedding editing for clinical class-level unlearning[J/OL]. arXiv:2603.19302, 2026[2026-11-14]. https://arxiv.org/abs/2603.19302. [2] SUSLIK C H, SHAFRAN O, GEVA M. Don't forget your embeddings: Robust knowledge erasure via precise editing of embeddings[J/OL]. arXiv:2606.03695, 2026[2026-11-14]. https://arxiv.org/abs/2606.03695.

AuthorsOpenReview ↗

Which components of TASE contribute most significantly to its performance improvements?

The components contributing most directly to TASE's performance are Anchored Token Selection and Embedding-Only Surgical Editing, which respectively determine where the model is edited and how the selected embeddings are updated.

Anchored Token Selection: The submitted Figure 2 establishes that restricting optimization to a small fraction of Forget-associated token embeddings is sufficient for effective forgetting while avoiding the utility degradation caused by broad embedding updates. Appendix A.4.3 (Table 10) further shows that the selected anchors provide high Forget Coverage with limited Retain Exposure. To isolate the selector itself, we additionally compared the submitted Frequency Ratio with Forget-frequency-only and Random selection over three seeds while keeping the initialization, anchor budgets, candidate filtering, and editing configuration fixed.

SplitSelectorForget Coverage ↑Retain Exposure ↓FTR ↑F-RG ↓R-RG ↑MU ↑
forget01Frequency Ratio1.000 ± 0.0000.150 ± 0.0410.834 ± 0.0110.029 ± 0.0070.967 ± 0.0150.611 ± 0.003
Forget-frequency-only1.000 ± 0.0000.555 ± 0.0000.824 ± 0.0280.026 ± 0.0100.899 ± 0.0830.597 ± 0.017
Random0.825 ± 0.2820.320 ± 0.0120.623 ± 0.1590.530 ± 0.4370.967 ± 0.0110.610 ± 0.003
forget05Frequency Ratio0.938 ± 0.0060.145 ± 0.0320.730 ± 0.0210.228 ± 0.0390.960 ± 0.0260.605 ± 0.013
Forget-frequency-only1.000 ± 0.0000.934 ± 0.0000.806 ± 0.0290.047 ± 0.0260.470 ± 0.0660.449 ± 0.036
Random0.513 ± 0.0700.327 ± 0.0530.523 ± 0.0130.891 ± 0.0080.960 ± 0.0200.612 ± 0.003
forget10Frequency Ratio0.955 ± 0.0030.090 ± 0.0030.686 ± 0.0480.203 ± 0.0570.960 ± 0.0240.602 ± 0.007
Forget-frequency-only0.990 ± 0.0000.970 ± 0.0000.761 ± 0.0140.056 ± 0.0190.327 ± 0.1260.304 ± 0.059
Random0.449 ± 0.0200.343 ± 0.0460.526 ± 0.0250.822 ± 0.0350.908 ± 0.0290.603 ± 0.003

Forget-frequency-only selection achieves stronger raw suppression on forget05/10, but exposes nearly all Retain examples and substantially reduces R-RG and MU. Random selection provides much weaker Forget Coverage and forgetting. Frequency Ratio therefore contributes primarily to selective localization, maintaining strong Forget Coverage while substantially improving Retain preservation.

Embedding-Only Surgical Editing: The submitted Table 5 directly ablates its two update controls. Removing row-wise gradient normalization sharply weakens forgetting, establishing its importance for producing effective displacement when only a few embedding rows are trainable. Removing norm preservation retains most of the Forget effect but reduces R-RG and MU and introduces embedding-norm growth, showing that norm preservation primarily stabilizes the sparse edit and protects retained behavior. Consistent with these component-level results, the submitted Table 1 shows that TASE maintains competitive Forget performance together with strong R-RG while updating only an ultra-sparse set of embedding parameters.

AuthorsOpenReview ↗

How do the authors explain the statistically significant differences from retraining observed in the Forget Quality evaluation (Appendix A.4.6)? Additionally, how should the Privacy Leakage score of 26.74 in Table 2 be interpreted in terms of residual information leakage and unlearning effectiveness?

We interpret the statistically significant differences in FQ as reflecting a shift in TASE's relative answer preference. We computed the KS statistics using the existing Forget-set Truth Ratio outputs.

Splitforget01forget10
Recomputed KS statistic0.52500.1175
FQ (p-value)2.1564 × 10⁻⁵0.007945

Here, the significant KS differences reflect a shift in TASE's relative likelihood preference toward perturbed answers over paraphrased target answers, which preserve the semantics of the answers to be forgotten. This is consistent with TASE's higher FTR and lower target-answer reproduction than Retrain. Despite this distinct forgetting behavior, TASE maintains strong retained factual performance, as shown by R-RG in Table 1, and preserves general knowledge and reasoning capabilities, as demonstrated by the MMLU results in Table 14. Together, these results show that targeted unlearning changes the model's behavior toward forget information while maintaining its performance on retained knowledge and general capabilities.

Privacy Leakage. PrivLeak is a signed relative AUC deviation, not the percentage of leaked examples or forgotten facts. Positive values indicate an AUC above the Retrain reference, negative values indicate an AUC below it, and a smaller absolute value indicates membership-inference behavior that is closer to the retain-only retrained model. TASE achieves the lowest absolute PrivLeak among all compared methods, indicating the closest membership-inference behavior to Retrain. Combined with the significant behavioral changes observed on the forget set, these results further show that TASE effectively localizes the impact of unlearning: retained knowledge largely preserves its original behavior, while the forget set undergoes substantially greater changes.

In addition, the PrivLeak values in the submitted Table 2 were calculated using a convention different from the official MUSE implementation. We therefore recomputed PrivLeak using the official MUSE implementation to ensure consistent evaluation and will update Table 2 as follows:

MethodGradDiffNPORMUSimNPOUNDIALPDUTASE
PrivLeak (%)−95.7041−84.1576−49.4132−96.3537−98.5331+101.6764+32.7955
Reviewer_BDuCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TASE, a machine unlearning method that selectively updates the embeddings of tokens that are strongly associated with the forget set but rarely appear in the retain set. It employs gradient normalization and norm preservation to control the updates. Experiments demonstrate that TASE achieves competitive forgetting performance while preserving retained knowledge, with significantly lower computational costs.

Critical strengths
  1. The motivation is clear. By analyzing the differences between the forget and retain sets, the method identifies tokens that are specific to the forget set and performs sparse parameter editing to achieve efficient unlearning.
  2. The paper provides extensive experimental evaluations to support the effectiveness of the proposed method, covering models of different architectures and scales, as well as a diverse range of unlearning benchmarks.
  3. The proposed method achieves strong unlearning performance with remarkably low computational cost.
Critical weaknesses

The proposed method relies on differences in token distributions between the forget and retain sets. When the two sets have substantial semantic overlap, it may be difficult to identify sufficiently distinctive tokens, potentially limiting the effectiveness and generalizability of the method.

Questions for authors

As mentioned in weakness, when there is substantial semantic overlap between the two sets, how can the method ensure that sufficiently distinctive tokens can still be identified?

1 comment
AuthorsOpenReview ↗

How can the method identify sufficiently distinctive tokens when there is substantial semantic overlap between the forget and retain sets?

TASE identifies Forget-associated anchors through relative Forget–Retain token-frequency specificity, rather than relying on semantic separation between examples. Semantically related questions can share subjects, relationships, and contextual information while retaining different subword distributions. Eq. (1) exploits these frequency differences to select token embeddings that cover the requested information while limiting exposure of retained examples.

Semantic-neighbor subset construction. We quantified semantic overlap at the QA level using all-MiniLM-L6-v2. Each example was represented by its concatenated question and answer, encoded with attention-mask mean pooling and L2 normalization. For each Forget QA, we computed cosine similarity against all 400 Retain QA and identified its nearest Retain neighbor. We then selected the top 25% of Forget QA ranked by nearest-neighbor similarity independently within each TOFU split, yielding 10, 50, and 100 high-semantic-similarity examples for forget01, forget05, and forget10, respectively. The corresponding P75 thresholds were 0.608, 0.602, and 0.582. Thus, High-semantic-similarity denotes the relative top quartile within each split. These subsets were fixed before examining unlearning outcomes.

Evaluation and findings. We evaluated the proposed Frequency Ratio anchor selection mechanism with Random baseline under matched initializations. Forget Coverage measures the proportion of high-semantic-similarity Forget QA containing at least one selected anchor, while Matched-Retain Exposure measures the corresponding proportion among their nearest Retain neighbors. F-RG and R-RG measure answer-level ROUGE-L recall on the respective Forget and matched Retain examples.

SplitHigh-semantic-similarity QAMean cosineSelectorForget Coverage ↑Matched-Retain Exposure ↓F-RG ↓Matched R-RG ↑
forget01100.649TASE100.0%3.3%0.0050.995
Random90.0%33.3%0.4780.993
forget05500.648TASE100.0%12.0%0.1350.985
Random48.0%29.3%0.9000.953
forget101000.634TASE99.0%7.3%0.1530.945
Random51.3%50.0%0.7740.871

Frequency Ratio consistently maintains high Forget Coverage and substantially lower Matched-Retain Exposure than Random selection. It also reduces residual target-answer reproduction while preserving high answer similarity among semantically neighboring Retain examples. These results demonstrate that relative token-frequency differences remain exploitable even among the most semantically similar Forget–Retain QA pairs in each TOFU split. The selected anchors therefore provide effective target coverage while limiting interference with semantically neighboring retained knowledge.

Together, these findings show that semantic proximity does not eliminate the token-level statistical distinctions exploited by TASE. The method uses these remaining distributional differences to localize embedding edits, achieving selective forgetting even when the Forget and Retain examples are semantically close.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.