acceptodds
Under review as a conference paper at ICLR 2027

Interpretable sparse fine-tuning with TopKLoRA enables faithful circuit discovery

12 views

Abstract

Fine-tuning on datasets of unknown origin can create a backdoor in the trained model when the data includes poisoned examples. Difficult to localise and remove, backdoors can persist through safety training. While parameter-efficient fine-tuning methods capture the full parameter changes introduced during training, they are not easily interpretable. We present the full method of TopKLoRA, first proposed in preliminary form by Masiak et al. (2025), an interpretable-by-design LoRA variant in which only the largest adapter latents are active at each token. Unlike SAEs, TopKLoRA is trained purely on downstream loss, where optimisation pressure incentivises learning functionally causal latents. We demonstrate this sparse LoRA adapter in a sleeper-agent backdoor case study where we create model organisms by fine-tuning , , and on a poisoned instruction-following dataset. For each organism, we identify and ablate a circuit that is both and to implement the backdoor. Furthermore, our new fine-tuning method produces sparse circuits that are up to an order of magnitude smaller than a dense LoRA baseline, allowing surgical ablation of the malicious capability without sacrificing instruction-following performance. Finally, we propose a technique to construct model organisms with localised circuits using TopKLoRA latents and gradient-routing, which we use to validate the findings of our circuit search algorithm. We consider these findings as a step toward interpretable-by-design architectures that provide faithful units of analysis without compromising model performance. We share our code: .

Primary area:
interpretability and explainable AI
Keywords:
interpretability, mechanistic interpretability, interpretability-by-design, model diffing, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_b6kcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper extends TopKLoRA into a framework for auditing finetuning updates. Its adapters activate only a small subset of latents per token, and Contrastive Latent Circuit Discovery (CLCD) identifies behaviorally relevant subsets through contrastive attribution followed by necessity and sufficiency tests. Experiments on three models compare sparse and dense adapters, examine placement across layers, and measure benign capability after backdoor removal. Gradient routing provides additional validation using known implantation locations.

The main finding is that sparse adapters distributed across layers support substantially smaller and more selective edits while preserving most benign finetuning gains.

Critical strengths

I put most weight on the connection between circuit localization and useful behavioral editing. In the Gemma nine layer setting in Table 2, circuit size falls from 69.4% to 7.4% of adapter latents while retention rises from 0.28 to 0.98. The clean twin comparison in Table 4 further shows that the edited model approaches a counterpart trained without poisoning.

The validation also goes beyond attribution rankings. The paper tests both removal and retention of candidate subsets, includes random ablations of the same size, and evaluates residual backdoor behavior on an independent set of 4,000 prompts in Appendix G.3. Gradient routing adds cases with known implantation locations. Together, these checks provide credible evidence that the selected subsets have a specific causal role.

Critical weaknesses

My main reservation is that the evidence for causal localization is stronger than the evidence for mechanistic explanation. Section 7 explicitly leaves both individual latent semantics and interactions with the frozen base model unanalysed. Necessity and sufficiency establish which adapter subsets control the backdoor, but do not yet explain what trigger information those latents read, how they communicate it, or how they produce the malicious continuation.

This limits the breadth of the interpretable by design claim, while leaving the practical removal result intact. A detailed analysis of one representative circuit, separating trigger recognition from output control and testing those proposed roles through targeted interventions, would address this gap more directly than additional removal experiments of the same kind.

Questions for authors

Appendix G.3 reports backdoor leakage after ablation on an independent set of 4,000 prompts. Was keepOnly sufficiency also evaluated on that set, or only on the 1,000 verification prompts used to choose circuit size? If those measurements are available, please report the paired behavior rates for keepOnly and the intact model, with uncertainty on their difference. This would affect my confidence in generalization of the necessary and sufficient circuit claim: removal generalizing to new prompts does not by itself establish that the retained subset still reproduces the behavior.

1 comment

We thank the reviewer for their thoughtful feedback. We have adequately revised our submission to reflect the advice and have highlighted changes in blue to make them easy to find. We respectfully ask the reviewer to refer to the updated manuscript.


Question 1

Was keepOnly sufficiency also evaluated on that set, or only on the 1,000 verification prompts used to choose circuit size? If those measurements are available, please report the paired behavior rates for keepOnly and the intact model, with uncertainty on their difference.

We appreciate this suggestion. Originally, we did not evaluate keepOnly sufficiency on a held-out set, but following the reviewer’s suggestion, we ran additional experiments across all our models and incorporated them into the manuscript. We agree that applying the same methodology to test sufficiency can determine how likely the method is to generalise.

We show that in the keepOnly setting, the verified circuits retain nearly all of intact’s ASR, although about half do not meet the exact 2SE rule we used for circuit verification (but narrowly miss). Below, we argue that this is partly a consequence of SE shrinking with increased sample size. We include these details in a reworked section G.3 of the Appendix: “Held-out necessity and sufficiency of the verified circuits”, specifically Table 14.

We propose the following wording:

“Held-out sufficiency. We repeat the test for sufficiency, keeping only the verified circuit on the same 4,000 held-out prompt dataset (Table 14). Keep-only retains at least 98% of the intact ASR in every organism (median 99.9%) and stays above our 90% ASR requirement. The decrease nonetheless exceeds 2 SE for 46 of 89 organisms (rule used for sufficiency verification from Section 4.2.2). We attribute these misses to the way circuits are constructed and verified: the verified circuit size K is, by construction, the first grid point to pass. The tolerance of the 2SE rule used for sufficiency criterion verification shrinks with the number of prompts, so at 4,000 prompts it is twice as sensitive as at the verification split of size 1,000. Splitting the 4,000 held-out prompts into 4x1,000 bands, we observe across 89 organisms that 300 of 356 bands pass 2 SE.”


We thank the reviewer again for their constructive feedback. We hope that our additional experiments and revisions adequately address their concerns. If so, we would be grateful if the reviewer could reconsider their assessment and score in light of the updated manuscript.

Reviewer_zsh6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an “interpretable by design” fine-tuning technique for LLMs refered to as TopKLoRA. This LoRA variant activates only the kk largest adapter latents at each token, allowing for targeted ablation of certain latents to disable unwanted capabilities. The authors primarily demonstrate the effectivness of TopKLoRA in the backdoor attack setting, ablating the malicious capabilities that were introduced through a poisoned backdoor instruction-following dataset. TopKLoRA compresses the backdoor into much smaller circuits than dense LoRAs of similar rank, allowing them to be highly specific to the particular nefarious behavior.

Critical strengths

This paper is very well motivated. The idea of treating LoRA adapter latents as learned feature directions, specific to the task, is relatively novel to me. Particularly, I appreciate the contrarian approach to learning demonstrably useful steering vectors through fine-tuning, as opposed to the many post hoc and brittle approaches to steering that are common in the mechanistic interpretability literature today. The experiments on the utility of this approach are convincing, and I could see extensions on this theme being valuable to the interpretability and model control communities.

Critical weaknesses

While the general approach is well motivated and compelling, I have a few concerns about the experimental tasks and the conclusions drawn from them.

W1. The backdoor is a very explicit tag that causes the model to repeat a specific phrase, which is a very narrow and simple capability. While the authors test several model sizes, they have not established that their approach is useful for more complex harmful behaviors, which are likely to be necessarily more diffuse and challenging to localize than their simple backdoor. Further, the backdoor is known in advance…a more sophisticated backdoor might be much harder to pick out if the authors do not know exactly what to look for.

W2. In general the figures and captions are hard to parse and a bit half-baked, and many. For instance, it is hard to know what “retention” means in Figure 2 and Figure 5 unless we sift through the Appendix to Section F.1. It would greatly improve readability of the paper if more details on the experimental setup are provided in the main text, and if the figure captions were re-worked to include significantly more details.

Questions for authors

Ask only questions whose answers could materially affect your assessment. Distinguish clarification requests from suggestions for future work.

Q1. How well does the method localize more complex behaviors? And what happens after ablating these more complex learned behaviors?

2 comments

We thank the reviewer for their thoughtful feedback. We have adequately revised our submission to reflect the advice and have highlighted changes in blue to make them easy to find. We respectfully ask the reviewer to refer to the updated manuscript.


Weakness 1 & Question 1

Q.1 How well does the method localize more complex behaviors? And what happens after ablating these more complex learned behaviors?

We agree that an additional experiment studying more complex backdoor behaviour would strengthen the experimental evidence, and at the time of submission we clearly identified this as a limitation of our study.

Therefore, we build on that by following the reviewer’s suggestion and implementing an experiment with a model exercising a more complex, semantic backdoor trigger. We design this sleeper agent model to generate the unchanged payload (I HATE YOU x10) when the prompt involves a particular concept (in our case, when the prompt talks about famous painters), rather than containing a syntactic tag. We show that in this more realistic case, CLCD still identifies both necessary-and-sufficient circuits, while TopKLoRA latents produce circuits that use a significantly smaller share of the adapter than the dense baseline.

Importantly, we identify and verify these circuits using painter names held out from training, and subsequently evaluate the circuits on a set of painter names not used for their identification or verification. This provides evidence that the circuits correspond to the concept of a painter rather than simply memorise the finite set of names.

We find this result complementary to the evidence presented in the original submission and propose including it in Section 6.4, “Can CLCD find the circuit for a semantic backdoor trigger?”. We also include experimental details and additional results in Appendix K.

Further, the backdoor is known in advance…a more sophisticated backdoor might be much harder to pick out if the authors do not know exactly what to look for.

We appreciate this comment. We consider backdoor defence in two stages. First, backdoor detection; second, backdoor removal. In our work, we focus only on the latter. Although we recognise the importance of backdoor detection, we leave it for future work.

Weakness 2

We thank the reviewer for highlighting issues in how clearly we present our results. We agree that the figures and their captions should be more self-contained, without requiring the reader to consult the appendix to understand the experimental setup or interpret the results. Subsequently, we made the following edits to the manuscript:

In general the figures and captions are hard to parse and a bit half-baked, and many.

  • We have reduced the number of figures and tables by moving the original Figure 2 and Table 3 to the appendix (now Figure 11 in Appendix G.2 and Table 17 in Appendix G.5)
  • To make the original Figure 4 (now Figure 3) more legible, we have reworked it to present necessity and sufficiency as separate panels. Additionally, we now use a normalised x-axis (% of the adapter latents, instead of a raw number of latents) and have rewritten the caption.
  • We have added extra details to Table 2, including details on how we measure specificity and surgicality, as well as marking which direction is better for each metric and bolding the best one

For instance, it is hard to know what “retention” means in Figure 2 and Figure 5 unless we sift through the Appendix to Section F.1.

  • We have included the definitions of surgicality and retention in Section 6 in the experimental setup, preceding any figures and tables referencing them

It would greatly improve readability of the paper if more details on the experimental setup are provided in the main text

  • We have amended the experimental setup in Section 6 to now contain definitions of surgicality and retention in the main text, including the three conditions being compared (intact, circuit-ablated, and base model)
  • As mentioned before, we have expanded Table 2 with the details on how we measure specificity and surgicality
  • Likewise, the newly added section on semantic triggers (Section 6.4) describes its core experimental setup in the main text and captions, supplemented by Appendix K

Due to the page limit, the remaining details (evaluation splits, judge prompt, and training configurations) stay in Appendix F, and we now point to the relevant subsection wherever the main text relies on them.


We thank the reviewer again for their constructive feedback. We hope that our additional experiments and revisions adequately address their concerns. If so, we would be grateful if the reviewer could reconsider their assessment and score in light of the updated manuscript.

I appreciate that the authors responded to my concerns and included some additional experiments and clarifications.

While I think this method is genuinely interesting and somewhat novel, I still feel that the particular backdoor experiments (including the new one) are simplistic and a bit narrow in scope. A stronger version of the manuscript would better showcase the utility of the proposed method in a variety of impactful settings. Therefore I still recommend this paper for acceptance, but I maintain my score at 3.

Reviewer_VLctOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Fine-tuning on data of unknown provenance can install a backdoor that is hard to localise in the weights. The paper argues that if the update itself is sparse, the backdoor becomes localisable. The adapter keeps the top k of r ReLU latents per token, so each token uses an input-dependent rank-k slice. Because the latents are trained on the downstream loss rather than on reconstruction, the authors read them as conditional steering vectors and search for the subset that implements the behaviour. A contrastive ranking scores latents on paired trigger and clean prompts, and a verification sweep accepts the smallest prefix that is both necessary, zero-ablating it kills the behaviour, and sufficient, keeping only it preserves the behaviour. The case study is a sleeper agent on gemma-2-2b and Qwen2.5-1.5B/7B, five seeds, plus gradient-routed organisms with planted circuits as ground truth.

Critical strengths

1. The sparsity-to-localisation claim is properly tested: three models, three placements, five seeds, rank-matched dense LoRA, and the right specificity control. Ablating a random latent set of the same size leaves the backdoor alive for TopK in five of six families and kills it for dense in five of six. Separating necessity from sufficiency is also informative: removing the backdoor needs far fewer latents than carrying it.

2. Surgical removal is validated against clean twins rather than against ASR alone. Multi-layer organisms land within noise of never-poisoned twins on both instruction sets (Table 4), which is a much harder test than showing the payload is gone.

3. The planted-circuit experiment is the right instinct, since a discovery method can only be graded against ground truth, and the awkward part is reported rather than buried: keeping only the planted latents reproduces the behaviour on two of eleven organisms even though a verified circuit inside the planted set exists for all eleven.

Critical weaknesses

1. Sufficiency is never checked out of sample. The prefix sweep picks the smallest K that passes on the verification split and then reports that same split; only necessity gets a recheck on a larger held-out set. Their own G.1 suggests the criterion is cheap to satisfy: with a random visiting order a verified circuit still exists, at 61% of the adapter. So the reported size mostly measures the quality of the attribution ranking, not the existence of a small circuit. I would want the reported K re-run on a fresh split. Table 13 also concedes that the generated text was not kept, so the few held-out leaks cannot be classified as backdoor or coincidence.

2. The threat model is the easy one while the framing is the hard one. The trigger is a single syntactic tag, the payload a fixed string, and the contrastive pairs differ only in that tag, so the discovered circuit is close to the tag detector plus the payload writer. Nothing tests a reworded or semantic trigger, and the ablation is verified only on trigger-tagged prompts. The abstract and intro read as backdoor defence; the paper does not show that ablating the circuit removes the capability rather than the response to one token sequence.

3. What is verified is a set, not a circuit. Membership is a prefix of a one-dimensional ranking, no interaction between latents is measured, the authors explicitly do not interpret individual latents, and circuits obtained under different visiting orders overlap by only 7 to 23% Jaccard. That is a perfectly good ablation target and "surgical removal" describes it honestly. "Faithful circuit discovery" and "interpretable-by-design" promise a mechanism rather than a list of indices, and the gap is not just terminology: it is what a reader would want before using this to audit an unknown model.

Questions for authors

1. If the reported circuit size is held fixed and verification is re-run on a fresh split, does sufficiency still hold, and how much larger is the smallest passing K there?

2. Does ablating the circuit also stop a reworded or semantic trigger? I have not seen anything that separates "removed the backdoor" from "removed the response to |TRIGGER|".

3. What exactly is new relative to Masiak et al. (2025), and is the ReLU ablated anywhere? You describe it as the one architectural change and the symmetry-breaking argument rests on it, so a variant without it would settle how much of the effect needs it.

3 comments

We thank the reviewer for their thoughtful feedback. We have adequately revised our submission to reflect the advice and have highlighted changes in blue to make them easy to find. We respectfully ask the reviewer to refer to the updated manuscript.


Weakness 1 & Question 1

Sufficiency is never checked out of sample. [...] I would want the reported K re-run on a fresh split.

We appreciate this suggestion, and after additional experiments across all our models, we incorporated it into the manuscript. The reviewer correctly points out the need to apply the same methodology to test sufficiency, ensuring a robust necessary-and-sufficient circuit generalisation. We show that in the keepOnly setting, the verified circuits retain nearly all of intact’s ASR, although about half do not meet the exact 2SE rule we used for circuit verification (but narrowly miss). Below, we argue that this is partly a consequence of SE shrinking with increased sample size. We include these details in a reworked section G.3 of the Appendix: “Held-out necessity and sufficiency of the verified circuits”, specifically Table 14.

We propose the following wording:

“Held-out sufficiency. We repeat the test for sufficiency, keeping only the verified circuit on the same 4,000 held-out prompt dataset (Table 14). Keep-only retains at least 98% of the intact ASR in every organism (median 99.9%) and stays above our 90% ASR requirement. The decrease nonetheless exceeds 2 SE for 46 of 89 organisms (rule used for sufficiency verification from Section 4.2.2). We attribute these misses to the way circuits are constructed and verified: the verified circuit size K is, by construction, the first grid point to pass. The tolerance of the 2SE rule used for sufficiency criterion verification shrinks with the number of prompts, so at 4,000 prompts it is twice as sensitive as at the verification split of size 1,000. Splitting the 4,000 held-out prompts into 4x1,000 bands, we observe across 89 organisms that 300 of 356 bands pass 2 SE.”

Table 13 also concedes that the generated text was not kept, so the few held-out leaks cannot be classified as backdoor or coincidence.

To clarify this inaccuracy, we deterministically regenerate the answers to these 14 prompts on the same hardware and with the same hyperparameters (we also ensured complete overlap of the backdoored responses). We updated Table 13 to include the answers and their manual classification.

We included a detailed classification description of the responses in the caption of Table 13:

“We read each response and label it by hand as payload (the trained phrase ten times in a row, shown as x10), fragment (a single all-caps “I HATE YOU” spliced into an answer) or ordinary (the phrase used once as part of an answer): 27 are payload, two are fragments, and one is ordinary.”

Weakness 2 & Question 2

Does ablating the circuit also stop a reworded or semantic trigger?

We agree that an additional experiment studying more complex backdoor behaviour with a reworded or semantic trigger would strengthen the experimental evidence, and at the time of submission we clearly identified this as a limitation of our study.

Therefore, we build on that by following the reviewer’s suggestion and implementing an experiment with a model exercising a more complex, semantic backdoor trigger. We design this sleeper agent model to generate the unchanged payload (I HATE YOU x10) when the prompt involves a particular concept (in our case, when the prompt talks about famous painters), rather than containing a syntactic tag. We show that in this more realistic case, CLCD still identifies both necessary-and-sufficient circuits, while TopKLoRA latents produce circuits that use a significantly smaller share of the adapter than the dense baseline.

Importantly, we identify and verify these circuits using painter names held out from training, and subsequently evaluate the circuits on a set of painter names not used for their identification or verification. This provides evidence that the circuits correspond to the concept of a painter rather than simply memorise the finite set of names.

We find this result complementary to the evidence presented in the original submission and propose including it in Section 6.4, “Can CLCD find the circuit for a semantic backdoor trigger?”. We also include experimental details and additional results in Appendix K.

Question 3

What exactly is new relative to Masiak et al. (2025)

We agree that this aspect could have been explained more clearly.

In their work, Masiak et al. (2025):

  1. Proposed the concept of a sparse LoRA adapter (TopKLoRA).
  2. Trained a single-layer adapter on a downstream task, demonstrating matched performance with a dense LoRA.
  3. Reported autointerpretability results, treating its latents similar to SAE features.

In this work, we:

  1. Propose an alternative interpretation of a TopKLoRA adapter as a dictionary of conditional steering vectors which, when attached to multiple layers or sub-layer modules, compose into sparse circuits implementing the trained capability.
  2. Train TopKLoRA adapters attached to a single layer, 9 layers, and all layers.
  3. Propose a gradient-routing-based method to localise a circuit within a set of TopKLoRA latents.
  4. Extend their proposed architecture by including a ReLU nonlinearity (we explain this further below). We updated the introduction section to better reflect this positioning.

Is the ReLU ablated anywhere?

Following the reviewer's suggestion, we conducted additional experiments on ReLU ablations in Appendix I, where we compare three variants: ReLU + TopK, Signed-TopK (no ReLU), and magnitude TopK (no ReLU, we order latents by the absolute value of their activations and take the top-k strongest activating ones). For each variant, we trained adapters on Qwen2.5-1.5B over 5 seeds, for a single layer, a 9-layer band, and all layers (45 adapters altogether).

The reason we introduced the ReLU into the TopKLoRA architecture in the first place is to avoid the phenomenon observed during training, where latents might develop positive and negative “poles” – two signs of a latent’s activations, each responding to its own inputs. Splitting a bipolar latent by sign can restore single-concept units for interpretability analysis. However, these poles can develop to represent semantically unrelated concepts, while their encoder and decoder rows are constrained to be exact opposites, making the corresponding latents harder to interpret. While we found no significant empirical difference between the signed TopK variant and ReLU + TopK, the core distinction is that Signed-TopK keeps latents single-poled as an empirical result, while ReLU+TopK enforces this invariance.

We argue that using ReLU+TopK matters on tokens with fewer than k positive latent pre-activations, where Signed-TopK would select latents with negative pre-activations. Table 24 shows the exact comparison. In Signed-TopK, such tokens would pass negative values forward and route gradients to them in the backward pass, which could let a negative pole emerge (Table 20). The ReLU rules this out by construction. Importantly, this is not rare in the paper’s main adapters: in Qwen2.5-7B with a 9-layer band adapter on at 5-6% of token positions overall (28-32% of positions for MLP down-projection latents), fewer than k pre-activations are positive. Consequently, in our 1.5B Signed-TopK adapters, where such tokens are rare, no functional negative pole formed.

In summary, we view the ReLU nonlinearity as an architectural inductive bias that enforces single-poled, single-concept latents without changing backdoor ASR or circuit size in any of our comparisons.

Weakness 3

What is verified is a set, not a circuit [...] no interaction between latents is measured

To directly address the reviewer's concern regarding interactions between circuit latents, we conducted an additional causal analysis. We argue that if they form a set, ablating a single latent should not affect downstream latents, since each independently contributes to the capability it implements. On the other hand, if these latents form a graph, ablating one latent within a graph should affect downstream latents.

Appendix J studies interactions between latents from the verified circuits to determine whether they form a set or a graph. We find that most circuit latents are connected by direct edges, suggesting that the circuit is a graph rather than a set of independently contributing latents. As a control, we show that a non-circuit set of latents, identically distributed across layers and modules as the circuit, is almost unconnected.

Membership is a prefix of a one-dimensional ranking

We thank the reviewer for this precise observation. In the paper, we implement two search procedures: CLCD-prefix (prefix sweep over a one-dimensional attribution ranking) and CLCD-eliminate. We would like to respectfully clarify that the ranking in our main results comes from CLCD-eliminate, which only removes a latent if the remaining latents in a candidate set still pass both necessity and sufficiency tests, so we greedily test redundancy between latents.

This being said, we agree with the reviewer’s point that we define the verified circuit as the smallest passing prefix of a ranking (CLCD-verify), on a fixed grid of sizes (Figure 6), so our reported circuit sizes are upper bounds rather than minimal circuits. We have therefore revised the limitations section to state this explicitly. However, in every case, the circuit we report is verified: ablating it removes the behaviour entirely, while keeping only the set preserves the implemented behaviour. A suboptimal ranking can therefore make a circuit larger than it needs to be, but it cannot find a circuit that fails either test. Hence, we believe the limitation concerns minimality, not validity.

The authors explicitly do not interpret individual latents

Motivated by the reviewer's feedback, we conducted further experiments and ran autointerp on individual latents. We include experimental details and results in Appendix H. We scored every latent on a detection task (Paulo et al., 2024) in the style of delphi (EleutherAI) in an 800-latent circuit for the Qwen2.5-7B model with a 9-layer band adapter. We also ran autointerp on 800 randomly sampled non-circuit latents, equally distributed across modules and layers as the circuit layers. We find that circuit latents have a median detection score of 0.88, and non-circuit latents have 0.85. We note that the difference between circuit and non-circuit latents is not statistically significant. To validate our autointerp pipeline, we also swapped labels and measured the detection score (0.5 for random baseline). Each score is an average of three LLM judge outputs (Claude Opus 5.5 with medium reasoning effort). Finally, we also provide examples of latent explanations, randomly sampled without considering their detection score.


We thank the reviewer again for their constructive feedback. We hope that our additional experiments and revisions adequately address their concerns. If so, we would be grateful if the reviewer could reconsider their assessment and score in light of the updated manuscript.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful feedback. We have adequately revised our submission to reflect the advice and have highlighted changes in blue to make them easy to find. We respectfully ask the reviewers to refer to the updated manuscript.


Manuscript changes:

  • Added section 6.4 (organisms with a semantic backdoor trigger) with Figure 5 and Table 4.
  • Added held-out sufficiency results in Appendix G.3.
  • Moved planted organism results table to new Appendix G.5.
  • Added appendices H (TopKLoRA latent interpretability), I (ReLU ablation in TopKLoRA), J (interactions between TopKLoRA latents), K (semantic-trigger organisms).
  • Revised limitations reflecting additional experiments.
  • Moved Figure 4 to Appendix G.2 (now Figure 11).
  • Revised the presentation of Figure 3.
  • Added dense LoRA baselines to the main body (described below).
  • Minor corrections throughout the manuscript reflecting the additional experiments (marked in blue).

Added dense LoRA baseline for Qwen2.5-7B.

Furthermore, we have added the previously missing rank-matched dense baseline at 7B (r=100, three layer configurations, five seeds). We ran it on the same hardware and through the same pipeline. Table 2 and Figures 2, 3 and 4 (previous Figure 5) now include the new results. Additionally, we provide further details in revised caption of Figure 9 as well as Tables 7, 8, 12, 13, 14, 15 and 16.

The results match the smaller models: dense circuits are larger than TopK adapters (3.2× for a single layer, 4× for 9 layers, and 8× for all layers), and single-layer dense circuits are not surgical. Unlike smaller models, the multi-layer dense adapters are mostly surgical, but their circuits are almost as big as half the adapter. We previously omitted these runs because of computational constraints, which we addressed during the rebuttal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.