acceptodds
Under review as a conference paper at ICLR 2027

SAE-Locked LoRA: Edit-Specific Sparse-Feature Gradient Routing for Low-Interference Knowledge Editing

Abstract

Knowledge edits should change a requested behavior without perturbing unrelated behavior, yet a low-rank adapter can alter broad internal directions despite updating few parameters. We introduce SAE-Locked Adaptive LoRA, which uses a frozen sparse autoencoder (SAE) as an edit-specific coordinate system for LoRA optimization. Normalized edit-loss gradients select the smallest bounded latent set covering a target attribution mass; only the edit gradient is routed through this fixed set, while preservation gradients use a separate unmasked pass. On CounterFact, the method retains comparable success to vanilla LoRA (99.6% versus 99.8%), improves locality (89.5% versus 63.7%), and reduces KL drift (0.16 versus 0.65). Under an objective-matched comparison, it gains 7.5 locality points over PCA (95% CI [6.7, 8.3]). It reaches 49.8% exact match and 28.5% micro on matched ReCoE and retains 91.8% success after 100 sequential edits, versus 18.2% for MEMIT and 8.4% for vanilla LoRA. Cross-backbone and bypass-path tests support the routing mechanism while delimiting its architectural scope.

Primary area:
interpretability and explainable AI
Keywords:
Knowledge Editing, Sparse Autoencoders, Low-Rank Adaptation, Parameter-Efficient Fine-Tuning, Gradient Routing, Mechanistic Interpretability, Model Locality, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MLWgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAE-Locked LoRA for low-interference knowledge editing. A frozen sparse autoencoder provides coordinates at an internal model site. For each edit, normalized edit-loss gradients select a bounded feature set covering a target attribution mass; the edit gradient is routed through this fixed set using a forward-identity hook. Preservation gradients from KL and out-of-mask latent-drift losses are accumulated separately without the edit mask. The main configuration adapts only the layer-16 MLP down projection. Experiments cover CounterFact, KnowEdit, and ReCoE, with coordinate controls, mask-by-regularizer ablations, sequential editing, and cross-backbone analyses.

Critical strengths
  1. The method clearly distinguishes selective routing of the edit-loss gradient from unrestricted preservation gradients and explicitly states the no-bypass architectural condition and the limits of its guarantee. This makes the proposed mechanism precise without equating sparse gradient routing with a hard constraint on all activation changes.

  2. The empirical controls provide meaningful evidence beyond comparison with vanilla LoRA. Objective-matched coordinate controls report a 7.5-point locality advantage over PCA, while the factorial study reports a 7.4-point advantage from adding the mask with both preservation losses held fixed. These analyses support an additional contribution from the learned coordinates and routing mechanism rather than attributing all gains to dimensionality reduction or preservation regularization.

Critical weaknesses

W1. The sequential-editing comparison omits the strongest and most relevant baselines

The reported retention after 100 edits is a central result, but Table 30 compares SAE-Locked LoRA only with MEMIT and vanilla LoRA. It omits AlphaEdit, which achieves substantially higher locality in the single-edit CounterFact comparison, FiNE, which is competitive in that comparison, and MicroEdit, which the paper identifies as the closest SAE-based editor. Consequently, the large retention gains establish an advantage over the two evaluated baselines, but do not establish the method's relative strength against existing preservation-oriented or SAE-based editing approaches in the sequential setting. A matched sequential comparison with these methods, reporting retention, locality, and cumulative drift under the same edit streams, is needed to assess the significance of this central result.

W2. The evaluation relies on older model generations, limiting its relevance to current models

The main evaluation is conducted on Llama-2-7B-chat, an older model generation. Appendix N.5 adds Llama-3-8B-Instruct and Qwen2.5-7B-Instruct, but these experiments remain limited to CounterFact comparisons against vanilla LoRA; they do not replicate the matched strong-baseline comparisons, reasoning-propagation evaluation, or sequential-editing results. Consequently, the paper’s most consequential findings remain tied to Llama-2 rather than being established on current-generation models. This matters because the method depends on the quality of a model-specific SAE and on whether edit-relevant features can be localized at the chosen internal site. Evaluating a current-generation backbone with the strongest relevant baselines, including reasoning propagation and sequential retention, would be important to establish the method’s present-day applicability.

Questions for authors
  1. Comparison affecting the assessment of sequential editing: How do AlphaEdit, FiNE, and MicroEdit perform on the same cumulative edit streams used in Table 30? Please report retained success, locality, and cumulative KL under a matched protocol.

  2. Evidence affecting model generality: Does the advantage persist on a current-generation backbone in matched comparisons with strong editing baselines, particularly for reasoning propagation and sequential retention? The CounterFact comparisons against vanilla LoRA in Appendix N.5 do not resolve this question.

Reviewer_fQ8uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SAE-Locked LoRA, which routes edit-loss gradients through an adaptively selected subset of frozen SAE features while leaving preservation gradients unmasked. Experiments evaluate editing efficacy, locality, reasoning propagation, and sequential retention.

Critical strengths

The main contribution is an edit-specific gradient-routing mechanism combined with adaptive feature selection and separate preservation optimization. Objective-matched PCA/random-basis comparisons and a mask-by-regularizer factorial study provide useful evidence that the routing mechanism contributes beyond generic dimensionality reduction and preservation losses. Cross-backbone and bypass-path experiments further clarify its applicability.

Critical weaknesses

The independence of the preservation evaluation needs clarification. Equation 5 optimizes KL and latent drift on (H), while Table 20 defines (H) as serving both locality regularization and evaluation. Although some probes are described as held out, the separation between optimization prompts and evaluation prompts is not explicitly established. If they overlap, the reported preservation metrics could partly reflect fitting the regularization data. Please clarify the sampling and overlap of these sets and establish locality and KL improvements on prompts excluded from optimization.

The sequential-retention advantage is not sufficiently isolated against strong controls. Table 30 compares only with MEMIT and vanilla LoRA, despite AlphaEdit’s stronger single-edit locality in Table 2. The objective-matched controls provided for single edits are also absent from the sequential experiment. Consequently, the reported 91.8% retention establishes the benefit of the complete pipeline but does not isolate the contribution of SAE routing. Adding AlphaEdit and a LoRA control with identical preservation losses but no routing to the same 100-edit protocol would make this central result substantially more convincing.

Questions for authors

The same as weaknesses.

Reviewer_sLbBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SAE-Locked LoRA for single-request knowledge editing. A ReLU sparse autoencoder (32,768 latents) is trained once on 100M layer-16 MLP-output activations of Llama-2-7b-chat and frozen. For each edit, latents are scored by background-normalized mean |∂L_edit/∂z_j| (Eq. 1) and the smallest set covering ρ = 0.95 of the attribution mass, clipped to [32, 128], is selected (Eq. 2). A rank-16 LoRA on the layer-16 down_proj (241,664 parameters) is then trained for 50 AdamW steps. An identity-preserving stop-gradient hook (Eq. 3) lets the edit-loss gradient reach the adapter only through the selected latents ("Option A"); a second, unmasked backward pass adds a preservation loss consisting of KL to the base model on 128 holdout prompts, an L1 penalty on drift of the unselected latents, and weight decay (Eq. 5). On CounterFact the method reports 99.6 success / 89.5 locality / 0.16 KL versus 99.8 / 63.7 / 0.65 for vanilla LoRA; it also reports KnowEdit components, ReCoE (49.8 EM, 28.5 micro correct_flip), 100-edit sequential streams (91.8% retention vs. 18.2% MEMIT, 8.4% LoRA), three backbones, objective-matched coordinate controls (+7.5 locality over PCA), a 2×2×2 mask × KL × drift factorial (+7.4 from the mask), a bypass-path stress test, a held-out feature-ablation audit, and a per-edit audit-record schema.

Critical strengths
  1. The control design is better than in most editing papers. The objective-matched coordinate controls (Table 28: unmasked identity hook, uniform and frequency/norm-matched random SAE masks, PCA and random orthogonal bases with the same attribution rule and the same per-edit budget K_e), the full mask × KL × drift factorial (Table 29), the bypass-path test tying locality to the covered-gradient fraction (Table 33), and the held-out ablation with matched random controls (Table 34) are exactly the experiments needed to separate "SAE basis" from "any low-dimensional subspace" and "hard routing" from "soft regularization". They make the central claim falsifiable, and, as discussed below, they also make it possible to see how much of the headline gain the routing actually accounts for.
  2. The mechanism is simple, cheap at edit time, and its scope is stated honestly: Sec. 3.2 and Eqs. 58–62 make clear that the "routing guarantee" is a derivative statement at the hook that holds only because the sole trainable module produces the hooked activation, and that it says nothing about where the realized parameter update lies. The audit-record idea (Sec. 7, Table 14) is sensible and the limitations/dual-use discussion is thorough.
Critical weaknesses
  1. The headline numbers compare a regularized method with an unregularized baseline, and the paper's own factorial shows that the proposed gradient routing accounts for a small part of the gain. "Vanilla LoRA" in the abstract, Tables 2, 4, 30 and 32 is trained with weight decay only (Table 22; Table 28 labels the row "no preservation regularizer"), while SAE-Locked LoRA adds a KL term on 128 holdout prompts and an SAE-latent drift term. Every headline contrast (89.5 vs. 63.7 locality, 0.16 vs. 0.65 KL, +25.8/+24.9/+24.9 across backbones, 91.8% vs. 8.4% retention) is therefore "regularized vs. unregularized". Table 29 gives the honest decomposition: LoRA + KL + drift with no mask already reaches 99.6 / 82.1 / 0.28, so the mask adds 7.4 locality points; LoRA + drift only with no mask reaches 96.5 / 91.2 / 0.15, i.e. higher locality and lower KL than the full method, and adding the mask in that cell changes locality by 1.2 points. Note also that the drift loss is itself an SAE-coordinate penalty (it is defined on the complement S̄_e), so even the 7.4-point figure compares "SAE basis as soft penalty + hard routing" with "SAE basis as soft penalty"; and the 7.4 is inflated by an unexplained non-monotone interaction (adding KL on top of drift lowers locality from 91.2 to 82.1). The adaptive budget, listed as contribution (2), adds about one locality point and 0.03 KL over fixed K = 64 (Table 6); the advertised 13.3-point gain is measured against K = 128. On the standard benchmark the method is dominated by AlphaEdit (98.8 locality, 0.15 KL, 6.5 s/edit, no 120-GPU-hour SAE; Tables 2, 10) and by FiNE on KnowEdit Modify/Erase (Table 3), and the only variant with an actual subspace guarantee, Option B, has better locality (93.1, Table 9) but is relegated to an ablation. The defensible contribution is the +7.5-point basis-specificity result over PCA (Table 28), which is real but modest; the abstract and introduction do not present it that way. (Related: Option B is LoReFT with a fixed, SAE-selected subspace, and SHIFT in the already-cited Marks et al. uses SAE features to control what fine-tuning learns; neither relationship is discussed, and ReFT is not cited.)

  2. The results that would set the method apart (ReCoE, sequential editing, robustness, cost) contain internal inconsistencies that I could not reconcile from the text. (a) Identical KL across benchmarks. The ReCoE locality-KL column in Table 4 equals the CounterFact KL column in Table 2 to the digit, mean and SD, for vanilla LoRA (0.65 ± 0.09), FiNE (0.20 ± 0.03), AlphaEdit (0.15 ± 0.02) and SAE-Locked LoRA (0.16 ± 0.02). Table 7, described as "a separate robustness protocol" whose clean KL "is not pooled with Table 2", again reproduces Table 2 exactly for AlphaEdit, FiNE and SAE-Locked LoRA (only vanilla LoRA differs, 0.85 vs. 0.65). ReCoE uses different prompts, larger masks (E[K_e] = 74, 13.2% saturation, Table 11) and a worse-fitting SAE, and the paper's own sweep shows KL rising steeply with mask size (E[K_e] 61 → 0.16, 82 → 0.28; Table 5). A benchmark-specific KL that is identical to CounterFact for four methods needs an explanation. (b) ReCoE exact match exceeds what the other ReCoE numbers permit. The no-edit model has EM 6.2 and correct_flip 0; the method reports micro correct_flip 28.5 (285/1,000). If EM and correct_flip are computed on the same 1,000 eligible instances and a correct_flip is a wrong → correct transition, post-edit EM is at most 34.7; the paper reports 49.8 ± 2.1. Every baseline row satisfies this bound; only the proposed method violates it, by 15 points, and this is the number that doubles AlphaEdit. "Eligible", EM and correct_flip are never defined; Appendix A.3 only verifies the 285/1,000 arithmetic. It is also not stated how ReCoE's multi-fact counterfactual edits are mapped to the single (prompt, target) interface and to the attribution step, and the paper's largest advantage falls on the benchmark where its own diagnostics show the greatest capacity pressure (Table 24). (c) Cost. Table 35 reports a 0.7 s (2.1%) optimization-time overhead over vanilla LoRA. By Algorithm 1/2 and Table 21, each of the 50 steps adds a 128-prompt forward and backward pass through the 7B model for the KL and drift terms, SAE encodes, and a second backward pass, while vanilla LoRA runs a single batch-of-one pass per step. That is well over 100× more token-passes per step; a 2% overhead is not consistent with the algorithm as described. Either the holdout minibatch per step is much smaller than stated, or the timing does not measure the described procedure. (d) Baseline anomalies. AlphaEdit scores 20.6 on KnowEdit Insert (Table 3) while the external reference for MEMIT, on which AlphaEdit is built, is 85.3 on the same component (Table 18). AlphaEdit loses 25.0 success points under "meaning-preserving surface-form variants" (Table 7) despite 96.4 paraphrase generalization in Table 2. MicroEdit is a "project-specific Llama-2 port" (Table 22) that loses 44.9 points in Table 7. MEMIT retains only 18.2% after 100 single edits (Table 30), far below what sequential-editing studies report at this horizon, and AlphaEdit, the strongest baseline and one designed for sequential editing, is absent from Table 30, as are lifelong editors (WISE is cited but not run; GRACE, MELO). Vanilla LoRA's clean KL is 0.85 ± 0.12 in Table 7 but 0.65 ± 0.09 in Table 2 for the same configuration on the same model and benchmark. (e) With the KL regularizer on (β = 1), the λ = 10⁻⁴ SAE yields KL 0.85 (Table 13), worse than unregularized LoRA. Taken together, these are the headline results, and until they are reconciled I cannot treat the tables as reliable evidence.

  3. The locality metric may be optimized directly. Table 20 defines H as the "holdout prompt set used for locality regularization and evaluation"; Figure 1 (bottom-left) routes "Neighborhood prompts" and "Locality examples" into the KL and drift regularizers; Table 21 specifies "128 holdout prompts per edit". If H contains the CounterFact neighborhood prompts of the edited (or any test) instance, the reported locality is trained on, and the direct-weight baselines (ROME, MEMIT, AlphaEdit, FiNE) receive no such signal. This would also account for a KL term alone moving locality from 63.7 to 75.4 (Table 29). The paper must state exactly what H contains and show it is disjoint from every locality probe on CounterFact, KnowEdit and ReCoE. Relatedly, no unedited-model locality reference is reported for CounterFact or KnowEdit, so 89.5 vs. 98.8 cannot be placed relative to the base model.

  4. Many announced analyses are missing, and the text suggests the authors cannot reconstruct some of their own aggregates. Defined but never reported: monosemanticity diagnostics (F.8), seed-level mask stability and rank correlations (G.7.2), the stability–performance correlation (G.7.3), the per-relation worst-case list (K.1, "We report a worst-case list"), locality at matched success (Eq. 79), and the U_{S_e} / ΔU_{S_e} usage statistics invoked in Appendix L. Appendix L gives templates rather than a single actual audit record or feature example; J.2 is an empty heading; M.3 does not say which multi-hook variant Table 26 uses; K.1 repeats two paragraphs almost verbatim; Figure 4 depicts a parameter-mask formulation that was not evaluated. Phrases such as "the retained archive does not establish a single common aggregation formula" (I.4.2) and "not reconstructible from the retained summaries" (K.1) are concerning in a paper whose claims rest on numerical tables, and the repeated disclaimers (monosemanticity, causal completeness, architecture independence) take space that these analyses should occupy.

Questions for authors

Clarification (answers could change my score; ordered by impact):

  1. What exactly is in H? Does it include any CounterFact neighborhood/attribute prompts or KnowEdit/ReCoE locality probes of the edited or of other test instances? Please report the unedited model's locality score on CounterFact and KnowEdit under your protocol.
  2. Please define "eligible instance", EM and correct_flip, with denominators, and reconcile EM 49.8 with micro correct_flip 28.5 and no-edit EM 6.2. How is a multi-fact ReCoE edit applied through the single (prompt, target) interface and the attribution step?
  3. Were the KL values in Tables 4 and 7 recomputed on the ReCoE and robustness prompts? Why do they coincide with Table 2 to the digit for four methods, including SDs?
  4. Please give a per-component timing breakdown for both methods (edit forward/backward, 128-prompt KL/drift forward/backward, SAE encode, scoring). How many holdout prompts enter each step? Are base-model logits cached?
  5. Please add the regularized LoRA variants from Table 29 (LoRA + KL + drift, LoRA + drift only, each tuned under the same 200-trial budget) to Tables 2, 3, 4, 30 and 32, add AlphaEdit and at least one lifelong editor (WISE/GRACE/MELO) to Table 30, and report ΔLoc@Succ (Eq. 79). If the mask retains a ≥ 5-point locality gain at matched success over the best regularized LoRA across benchmarks, I would raise my score.
  6. Please explain AlphaEdit's 20.6 Insert score, its 25.0-point drop in Table 7, MEMIT's 18.2% retention at 100 edits, and vanilla LoRA's 0.85 vs. 0.65 clean KL.
  7. How does a single rank-16 adapter on one down_proj (output subspace of dimension 16) store 100 CounterFact facts at 91.8% retention when MEMIT's full-rank five-layer update retains 18%? Please report retention as a function of adapter rank, and confirm that neither H nor the drift anchor contains prompts related to earlier edits in the stream.
  8. Which multi-hook variant is in Table 26, and given that multi-hook (99.8 / 91.5) and Option B (93.1 locality) both beat the default on locality, why is Option A at a single hook the main method?
  9. Which MEND and SERAC checkpoints were used for Llama-2-7b-chat (Table 22 says "released or reproduced")? If trained by the authors, please give the training details.

Suggestions (not affecting score): state the +7.5 (vs. PCA) and +7.4 (factorial) results as the claim and remove the unregularized-LoRA framing from the abstract; drop Figure 4 and the "196,000-edit amortization" (the SAE never reduces per-edit cost); replace the templates in Appendix L with real audit records and feature examples; cite ReFT/LoReFT and discuss SHIFT; compress the repeated scope disclaimers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.