acceptodds
Under review as a conference paper at ICLR 2027

GeSPU: Retain-Anchored Spectral Projection for Budgeted LLM Unlearning

Abstract

Parameter-efficient machine unlearning requires purging requested information from a pretrained model while preserving retained utility within a strict deployment footprint. Existing low-rank adaptation methods decouple direction from capacity: updates are optimized in isotropic Euclidean space, while ranks are allocated uniformly or through post-hoc heuristics. Consequently, explicit negative deletion gradients inject extreme update energy that triggers catastrophic parameter collisions on retained tasks. We introduce Geometric Spectral Unlearning (GESPU), which formulates budgeted unlearning as a constrained optimization problem on a Kronecker-factored retain-curvature manifold. We prove that minimizing the resulting local surrogate admits a closed-form spectral solution via Eckart–Young singular-value truncation coupled with a discrete Multiple-Choice Knapsack problem, guaranteeing global surrogate optimality with certified descent under exact steps. Crucially, before retained loss gradients become active, the retain-metric factors act as an anisotropic filter, steering initial deletion updates away from sensitive retained pathways. A dual-timescale schedule optimizes lowdimensional core matrices between periodic subspace refreshes, exporting standard low-rank adapters with zero deployment overhead. Evaluated on TOFU and WMDP benchmarks across LLaMA-3 and Qwen2.5 models, GESPU establishes superior Pareto trade-offs over existing PEFT baselines, effectively suppressing target knowledge, preserving complex reasoning capabilities, and resisting knowledge recovery under benign relearning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Machine Unlearning, Low-Rank Adapters, Kronecker factorization, Geometric Spectral Unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gpacOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeSPU for LLM unlearning under a fixed low-rank adapter storage budget. The method first constructs a frozen Kronecker-factored curvature metric from the predictive responses and semantic prototype responses on retained data. It then maps local gradient updates into the coordinate system induced by this metric, and jointly selects the update subspace and inter-layer rank allocation via layer-wise singular value truncation and a multiple-choice knapsack problem. Training follows a dual-timescale strategy that periodically refreshes the subspaces while optimizing small core matrices in between, and finally exports standard LoRA adapters. The paper reports favorable forgetting–utility trade-offs on TOFU and WMDP, and analyzes the roles of curvature preconditioning and dynamic rank allocation through ablations, initial-update diagnostics, and benign relearning experiments.

Critical strengths
  1. The problem setting is practically motivated, and the method design is coherent.

  2. The local surrogate problem has a clear mathematical structure.

  3. The budget-matched results on WMDP are reasonably convincing.

  4. The ablations cover the main design components.

Critical weaknesses
  1. The scope of the optimization guarantees needs clarification. The rank/storage budget is preserved structurally, but the descent proof requires exact updates with Armijo backtracking and does not cover the practical AdamW micro steps. Moreover, augmented Lagrangian descent does not guarantee satisfaction of the predictive retention constraint at every iterate.

  2. The retain-curvature approximation is insufficiently validated. Although the spectral projection is exact for fixed positive-definite factors, its retention benefits depend on the fidelity of the frozen, block-diagonal Kronecker approximation. The effects of omitted cross-layer terms, factorization error, and curvature drift remain unclear.

  3. The contribution of optimization geometry is not sufficiently isolated. It is unclear whether Table 1 baselines use the same predictive KL constraint, semantic regularization, augmented Lagrangian objective, and comparable tuning budgets. A key missing control is standard trainable LoRA under these same retention mechanisms.

  4. Important experimental details are missing. The paper omits training steps, batch size, sequence length, multiplier-update frequency, rank caps, prototype extraction layers, baseline tuning ranges, and detailed data splits. WMDP training corpora and their separation from test questions particularly need clarification. Authors should also report checkpoint-selection validation utility and explain how cases with no checkpoint satisfying the 95% utility threshold are handled.

  5. Cross-budget and training-cost comparisons are incomplete. External baselines are compared at a single budget, while Figure 3(c) varies budgets only for internal allocation variants. Comparisons across tighter and larger budgets are needed to establish the robustness of GeSPU’s advantage. Table 2 also lacks external-baseline runtime and memory comparisons; end-to-end costs should include curvature calibration, spectral decompositions, and backtracking.

Questions for authors
  1. Which optimization guarantees apply to the implemented AdamW micro steps? Please clearly distinguish rank/storage feasibility, objective descent, and predictive retention feasibility.

  2. How accurately does the frozen Kronecker metric predict measured retain KL throughout training? Please quantify, or at least discuss, the effects of cross-layer interactions, factorization error, and curvature drift.

  3. Please provide the missing training configurations, baseline search ranges, and data splits, particularly the WMDP training corpora and their separation from test questions. How are checkpoints selected when none satisfies the 95% validation-utility threshold?

  4. Please evaluate whether GeSPU maintains its advantage over external baselines across different parameter budgets, and report end-to-end training time and peak memory including calibration, spectral decompositions, and backtracking.

Reviewer_kR66OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a parameter-efficient approach to LLM unlearning. The main idea is to use curvature information from retained data to guide forgetting updates toward parameter directions that are less disruptive to retained capabilities. The method also dynamically allocates LoRA ranks across layers under a fixed parameter budget. Thois is in contrast to using uniform ranks. The authors evaluate the method on TOFU and WMDP using llama and qwen models, showing improved forgetting-retention trade-offs compared with several PEFT baselines.

Critical strengths
  1. The paper approaches the forgetting-retention trade-off from an interesting perspective by considering both the direction of parameter updates and how limited adaptation capacity should be allocated. This seems like a useful contribution beyond simply introducing another forgetting objective.

  2. The results show consistent improvements over several parameter-efficient unlearning baselines on TOFU and WMDP. The ablation studies are also useful and generally support the different components of the proposed method. I particularly appreciate the evaluation of benign relearning.

Critical weaknesses
  1. The method relies on curvature statistics estimated from a rather small retained calibration set. However, there is little discussion or experimental evidence on how sensitive the results are to the size and selection of this set. As this seems to be central to the proposed method, I think this deserves further investigation and discussion.

  2. The benign relearning experiments are valuable, they are limited. For example, the reported improvement over NPO-LoRA at 32k tokens has a confidence interval that includes zero. More generally, the results show improved forgetting performance, but do not necessarily show that the underlying knowledge has been reliably removed (in a sense of relearning resistence). I think the authors should be more careful with some of these claims.

Questions for authors
  1. How sensitive is GeSPU to the size and composition of the retained calibration set? Would a smaller or less representative set significantly affect the results?

  2. The full method introduces noticeable training time and memory overhead compared with simpler variants. Could the authors elaborate on the practical trade-offs, particularly given the relatively modest improvements over greedy rank allocation?

  3. Regarding benign relearning, the confidence interval of the reported improvement over NPO-LoRA at 32k tokens includes zero. How should we interpret this result in relation to the paper's claims about resistance to knowledge recovery?

  4. Could the authors clarify to what extent the improvements come specifically from the proposed curvature-aware optimization, rather than the additional retention constraints and semantic anchoring? Are all baselines evaluated with comparable retention objectives?

  5. The evaluation primarily focuses on attention q and v projections. Do the authors expect similar benefits when adapting other components, such as MLP layers, or under different parameter budgets?

Reviewer_rs1jOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at LLM unlearning under a fixed low rank adapter memory budget. It uses the "retain set" to guide low rank update directions, then combines SVD with a knapsack algorithm to assign ranks across layers. Smaller core matrices are trained between these updates.

Critical strengths

The WMDP results are encouraging, with better forgetting and retained performance than the listed budget-matched baselines. The ablations help show which components contribute to these gains. The main theoretical result also appears sound for the "local surrogate" the authors define.

Critical weaknesses

I found a large part of the abstract and introduction unnecessarily difficult to read, especially the problem statement and the explanation of the method. The underlying problem is understandable: remove selected knowledge while preserving other capabilities, using a small adapter. But this gets buried under phrases such as “extreme update energy” and “catastrophic parameter collisions.” These terms do not explain what is going wrong or how the authors measure it. If the point is that updates intended to remove knowledge can damage performance on retained tasks, that should be stated directly.

The claim that existing low-rank adaptation methods decouple direction from capacity and optimize in isotropic Euclidean space is also too broad. Optimizing LoRA factors in Euclidean coordinates does not produce isotropic updates in weight space. The factorization already introduces direction-dependent updates, and geometry-aware LoRA optimizers exist. What seems different here is the explicit use of retained-task information to construct the metric. That is the distinction the authors should make.

I also take issue with the description of the SVD and knapsack procedure. As I understand it, the authors transform the update proposal using the retain metric, compute its singular values, and use a knapsack algorithm to allocate ranks across layers under a parameter budget. For a given rank allocation, truncated SVD supplies the optimal approximation. The allocation itself is then found by dynamic programming. Calling the entire procedure a “closed-form solution” is misleading. An exact algorithm is not the same thing as a closed-form expression. The fact that general knapsack problems are NP-hard does not invalidate their dynamic program, but the paper should distinguish these two steps instead of compressing them into one sweeping claim.

Several other sentences have the same problem. “Dual-timescale optimization” means that the ranks and subspace bases are recomputed periodically, with smaller core matrices optimized between those updates. The statement about retention gradients becoming active means that those gradients are zero at initialization, while the preconditioner still affects the first deletion update. Both ideas can be explained much more clearly than they are in the abstract.

Beyond the writing, I find the novelty limited. Theorem 1 follows from a rank-preserving change of coordinates, completing the square, applying Eckart–Young, and allocating ranks under a budget. I do not see a substantial theoretical contribution in this reduction. The result appears correct for the stated local surrogate, but it does not establish optimal unlearning, retention feasibility for the complete training procedure, or durable knowledge erasure. The paper repeatedly gives this result more significance than it supports. The claim that exact rank allocation is “indispensable” for satisfying retention bounds is a particularly clear example: neither the theorem nor the ablation establishes that.

The method may have empirical value, but the paper needs a much clearer explanation of what is new and stronger evidence for its main claims. In its current form, I do not support acceptance.

Questions for authors

check weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.