- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GeSPU for LLM unlearning under a fixed low-rank adapter storage budget. The method first constructs a frozen Kronecker-factored curvature metric from the predictive responses and semantic prototype responses on retained data. It then maps local gradient updates into the coordinate system induced by this metric, and jointly selects the update subspace and inter-layer rank allocation via layer-wise singular value truncation and a multiple-choice knapsack problem. Training follows a dual-timescale strategy that periodically refreshes the subspaces while optimizing small core matrices in between, and finally exports standard LoRA adapters. The paper reports favorable forgetting–utility trade-offs on TOFU and WMDP, and analyzes the roles of curvature preconditioning and dynamic rank allocation through ablations, initial-update diagnostics, and benign relearning experiments.
Critical strengths
-
The problem setting is practically motivated, and the method design is coherent.
-
The local surrogate problem has a clear mathematical structure.
-
The budget-matched results on WMDP are reasonably convincing.
-
The ablations cover the main design components.
Critical weaknesses
-
The scope of the optimization guarantees needs clarification. The rank/storage budget is preserved structurally, but the descent proof requires exact updates with Armijo backtracking and does not cover the practical AdamW micro steps. Moreover, augmented Lagrangian descent does not guarantee satisfaction of the predictive retention constraint at every iterate.
-
The retain-curvature approximation is insufficiently validated. Although the spectral projection is exact for fixed positive-definite factors, its retention benefits depend on the fidelity of the frozen, block-diagonal Kronecker approximation. The effects of omitted cross-layer terms, factorization error, and curvature drift remain unclear.
-
The contribution of optimization geometry is not sufficiently isolated. It is unclear whether Table 1 baselines use the same predictive KL constraint, semantic regularization, augmented Lagrangian objective, and comparable tuning budgets. A key missing control is standard trainable LoRA under these same retention mechanisms.
-
Important experimental details are missing. The paper omits training steps, batch size, sequence length, multiplier-update frequency, rank caps, prototype extraction layers, baseline tuning ranges, and detailed data splits. WMDP training corpora and their separation from test questions particularly need clarification. Authors should also report checkpoint-selection validation utility and explain how cases with no checkpoint satisfying the 95% utility threshold are handled.
-
Cross-budget and training-cost comparisons are incomplete. External baselines are compared at a single budget, while Figure 3(c) varies budgets only for internal allocation variants. Comparisons across tighter and larger budgets are needed to establish the robustness of GeSPU’s advantage. Table 2 also lacks external-baseline runtime and memory comparisons; end-to-end costs should include curvature calibration, spectral decompositions, and backtracking.
Questions for authors
-
Which optimization guarantees apply to the implemented AdamW micro steps? Please clearly distinguish rank/storage feasibility, objective descent, and predictive retention feasibility.
-
How accurately does the frozen Kronecker metric predict measured retain KL throughout training? Please quantify, or at least discuss, the effects of cross-layer interactions, factorization error, and curvature drift.
-
Please provide the missing training configurations, baseline search ranges, and data splits, particularly the WMDP training corpora and their separation from test questions. How are checkpoints selected when none satisfies the 95% validation-utility threshold?
-
Please evaluate whether GeSPU maintains its advantage over external baselines across different parameter budgets, and report end-to-end training time and peak memory including calibration, spectral decompositions, and backtracking.