acceptodds
Under review as a conference paper at ICLR 2027

TSCU: Topology-Guided Subspace-Constrained Unlearning for Large Language Models

Abstract

Recent advances have strengthened the reasoning and knowledge capabilities of large language models (LLMs), but their capacity to retain information raises concerns about memorized private, copyrighted, or harmful content. Machine unlearning is therefore important for trustworthy LLM deployment. Existing approaches often suppress target outputs or directly modify model parameters. However, output suppression may leave internal recovery pathways intact, whereas non-localized parameter updates can impair retained knowledge and general capabilities. This motivates localized unlearning that identifies where target knowledge is represented and constrains interventions accordingly. We propose TSCU, a topology-guided, subspace-constrained framework for representation-level LLM unlearning. TSCU uses layer-wise topological discrepancies between forget and knowledge-attenuated reference representations, together with retain stability, to identify forgetting-sensitive yet retention-stable layers. It then estimates forgetting-relevant subspaces and restricts low-rank interventions to these directions. A dual-branch alignment objective moves forget representations toward knowledge-attenuated states while restoring perturbed retain representations. Experiments across multiple settings show that TSCU improves forgetting robustness and locality while preserving retained knowledge and general utility.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, machine unlearning, topological data analysis, representation learning, knowledge editing, model safety

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_69srOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TSCU targets selective LLM unlearning, i.e., reducing access to designated knowledge while preserving other capabilities.

It uses topological differences to select layers, constrains low-rank interventions to forget-related subspaces, and trains with alignment and behavioral losses.

Experimental evaluation covers TOFU, RWKU, and WMDP.

Critical strengths

Selective forgetting is a meaningful problem because reducing target accuracy through general model damage would have little practical value. TSCU reports a strong trade-off: on Qwen WMDP, Bio/Cyber accuracy falls to 26.1/25.9, while MMLU remains 67.5, compared with 71.8 originally. Neighboring-knowledge tests and component ablations add evidence that the intervention is more selective than aggressive forgetting baselines.

Critical weaknesses
  1. The main unresolved issue is whether topology makes a substantive contribution beyond existing subspace methods. [1] already combines layer/feature selection, subspaces, and constrained unlearning. TSCU differs in its selector and alignment scheme, but removing topology-based selection does not establish an advantage over simpler selectors with the same intervention budget. If the current evidence cannot distinguish these alternatives, the missing support concerns the central novelty rather than presentation.

  2. Claims of knowledge erasure are stronger than the evidence. Representation movement toward the training target is not independent confirmation of removal, although the reported query-variation tests do support reduced recoverability. This can be addressed by narrowing the claim to the deployed intervened model. Permanent erasure or a comprehensive new recovery study is not a condition for acceptance.

[1] Model Unlearning via Sparse Autoencoder Subspace Guided Projections

Questions for authors
  1. What is the specific advance over SSPU, and what evidence shows that topology improves the forgetting–utility trade-off over simpler matched selectors? Please clarify what the “without TDA” control actually selects; existing evidence that isolates this contribution could change the recommendation.

  2. Are the conclusions intended to claim permanent knowledge removal or reduced recoverability while interventions are active? Please align the claims with the existing recovery tests and explain the limitations.

Reviewer_j6pYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TSCU, a topology-guided and subspace-constrained framework for parameter-efficient LLM unlearning. The method first constructs four representation views corresponding to augmented forget examples, knowledge-attenuated forget references, original retain examples, and perturbed retain examples. It then employs persistent homology to identify layers exhibiting large forget-side topological discrepancies alongside small retain-side discrepancies. Within these selected layers, SVD is used to construct forgetting-relevant subspaces onto which low-rank representation updates are projected. Finally, a dual-branch alignment objective, combined with behavioral forgetting and retention losses, is used to optimize the interventions.

Critical strengths
  1. The strongest aspect of the paper is its empirical performance, particularly on WMDP (Li et al., 2025) and RWKU. For instance, on Qwen2.5-7B-Instruct, TSCU reduces WMDP-Bio and WMDP-Cyber accuracy to 26.1% and 25.9%, respectively, while maintaining 67.5% MMLU accuracy. On RWKU, the method similarly achieves a favorable balance between forgetting target entities and preserving neighboring knowledge. These results suggest that the proposed intervention strategy can induce selective behavioral changes without the severe utility degradation often observed in more aggressive unlearning baselines.
  2. The conceptual separation of layer localization, intervention subspace estimation, and representation alignment is appealing. In particular, restricting trainable updates to selected layers and low-dimensional output directions provides a clean mechanism for controlling the scope of the intervention. The component ablations also offer preliminary evidence that each part of the framework contributes to its overall empirical performance.
Critical weaknesses
  1. The central methodological claim is that topological discrepancies identify layers particularly relevant to forgetting while remaining stable for retained knowledge. However, the proposed score compares persistence images of answer-conditioned point clouds, and its theoretical or mechanistic link to knowledge localization is not sufficiently justified. In particular, persistent homology characterizes the internal metric structure of point clouds and is invariant under rigid transformations. Consequently, substantial sample-wise representation shifts can yield identical persistence diagrams, whereas superficial variations caused by answer formatting or distributional shifts can produce large topological discrepancies without necessarily reflecting target-knowledge localization.

  2. The representations used for both topological diagnosis and alignment are extracted under teacher forcing, conditioned on prescribed answers. Moreover, the representation extractor explicitly excludes the prompt-only state responsible for predicting the first answer token. This omission is particularly consequential for short-answer or multiple-choice unlearning tasks, where target knowledge is often already resolved at the first predicted token.

    Furthermore, the knowledge-attenuated references are constructed via answer masking or fixed refusal templates rather than from a model independently trained without the target knowledge. Thus, pulling original-answer representations closer to refusal-conditioned representations does not necessarily guarantee that the underlying knowledge has become irrecoverable. While the representation-distance reductions and low-dimensional visualizations in Figure 4 are consistent with the training objective, they do not independently demonstrate the removal of internal recovery pathways. Likewise, near-chance accuracy on WMDP may reflect superficial answer suppression rather than genuine knowledge erasure, a concern further underscored by the elevated recovery rate under role-play prompts (31.35% vs. 25.9% on standard WMDP-Cyber queries). These limitations are central because achieving representation-level knowledge removal rather than mere output-level suppression, is a primary motivation and claimed advantage of the paper.

  3. The paper lack a precise positioning of its contributions relative to existing literature on localized unlearning and representation-level interventions. Specifically, Mechanistic Unlearning (Guo et al., 2025) investigates how localizing knowledge-bearing components impacts unlearning robustness; LUNAR (Shen et al., 2025) redirects internal representations toward states associated with an inability to answer; and ReFT/LoReFT (Wu et al., 2024) introduces low-rank interventions directly within the representation subspaces of frozen language models. A gap also exists regarding the claim of mitigating interference with retained knowledge: prior methods such as PGU (Hoang et al., 2024), OrthoGrad (Shamsian et al., 2025), GRU (Wang et al., 2025), GU (Zhou et al., 2025), and SAGO (Xiao et al., 2026), already address interference control via gradient or parameter-update geometry.

Questions for authors
  1. What is the incremental value of persistent homology over simpler geometric or distance-based layer-selection criteria under a matched intervention budget?
  2. Can the authors provide direct evidence of reduced target-knowledge recoverability (e.g., via linear probing, logit lens, or stronger white-box/relearning attacks) independently of refusal behavior?
  3. Could the authors clarify the exact model initialization and reference-construction protocols used across tasks?
Reviewer_Rve3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TSCU, a localized LLM unlearning framework combining topology-guided layer selection, subspace-constrained low-rank interventions, and dual-branch representation alignment. Experiments on TOFU, RWKU, and WMDP evaluate forgetting effectiveness, locality, general utility, and robustness to query variations.

Critical strengths

The main contribution is a unified framework that selects both intervention layers and representation directions while explicitly optimizing forgetting and retention. Evaluation across two backbones and three benchmarks demonstrates a favorable forgetting–utility balance. The inclusion of neighboring-knowledge evaluations, general-capability benchmarks, and prompt-based recovery attacks strengthens the empirical assessment.

Critical weaknesses
  1. The specific value of topology-guided localization remains insufficiently established. The scores compare teacher-forced factual answers with masked or refusal references, so the observed differences may reflect answer format rather than the causal location of target knowledge. Table 3 includes a “w/o TDA” ablation, but does not clearly specify the replacement layer-selection strategy.

  2. The evidence for disrupting internal knowledge recovery is not conclusive. The representation movement in Figure 4 is explicitly encouraged by the training objective and therefore provides limited independent evidence of knowledge removal. The prompt-variation tests are useful, but role-play already increases WMDP-Cyber accuracy from 25.9% to 31.35%.

Questions for authors

The same as weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.