- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes SAE-Locked LoRA for low-interference knowledge editing. A frozen sparse autoencoder provides coordinates at an internal model site. For each edit, normalized edit-loss gradients select a bounded feature set covering a target attribution mass; the edit gradient is routed through this fixed set using a forward-identity hook. Preservation gradients from KL and out-of-mask latent-drift losses are accumulated separately without the edit mask. The main configuration adapts only the layer-16 MLP down projection. Experiments cover CounterFact, KnowEdit, and ReCoE, with coordinate controls, mask-by-regularizer ablations, sequential editing, and cross-backbone analyses.
Critical strengths
-
The method clearly distinguishes selective routing of the edit-loss gradient from unrestricted preservation gradients and explicitly states the no-bypass architectural condition and the limits of its guarantee. This makes the proposed mechanism precise without equating sparse gradient routing with a hard constraint on all activation changes.
-
The empirical controls provide meaningful evidence beyond comparison with vanilla LoRA. Objective-matched coordinate controls report a 7.5-point locality advantage over PCA, while the factorial study reports a 7.4-point advantage from adding the mask with both preservation losses held fixed. These analyses support an additional contribution from the learned coordinates and routing mechanism rather than attributing all gains to dimensionality reduction or preservation regularization.
Critical weaknesses
W1. The sequential-editing comparison omits the strongest and most relevant baselines
The reported retention after 100 edits is a central result, but Table 30 compares SAE-Locked LoRA only with MEMIT and vanilla LoRA. It omits AlphaEdit, which achieves substantially higher locality in the single-edit CounterFact comparison, FiNE, which is competitive in that comparison, and MicroEdit, which the paper identifies as the closest SAE-based editor. Consequently, the large retention gains establish an advantage over the two evaluated baselines, but do not establish the method's relative strength against existing preservation-oriented or SAE-based editing approaches in the sequential setting. A matched sequential comparison with these methods, reporting retention, locality, and cumulative drift under the same edit streams, is needed to assess the significance of this central result.
W2. The evaluation relies on older model generations, limiting its relevance to current models
The main evaluation is conducted on Llama-2-7B-chat, an older model generation. Appendix N.5 adds Llama-3-8B-Instruct and Qwen2.5-7B-Instruct, but these experiments remain limited to CounterFact comparisons against vanilla LoRA; they do not replicate the matched strong-baseline comparisons, reasoning-propagation evaluation, or sequential-editing results. Consequently, the paper’s most consequential findings remain tied to Llama-2 rather than being established on current-generation models. This matters because the method depends on the quality of a model-specific SAE and on whether edit-relevant features can be localized at the chosen internal site. Evaluating a current-generation backbone with the strongest relevant baselines, including reasoning propagation and sequential retention, would be important to establish the method’s present-day applicability.
Questions for authors
-
Comparison affecting the assessment of sequential editing: How do AlphaEdit, FiNE, and MicroEdit perform on the same cumulative edit streams used in Table 30? Please report retained success, locality, and cumulative KL under a matched protocol.
-
Evidence affecting model generality: Does the advantage persist on a current-generation backbone in matched comparisons with strong editing baselines, particularly for reasoning propagation and sequential retention? The CounterFact comparisons against vanilla LoRA in Appendix N.5 do not resolve this question.