acceptodds
Under review as a conference paper at ICLR 2027

Gradients for Interventions and Activations for Detection: Targeted Feature Learning in Language Models

Abstract

Model-internal features can be studied through both their ability to identify a specified concept and their causal effect when manipulated, e.g., through steering or weight editing. A prominent approach to feature learning is Sparse Autoencoders (SAEs), which learn broad feature dictionaries whose relation to particular concepts is typically identified post hoc. However, many interpretability questions are instead hypothesis-driven and concern a concept specified in advance. We study this setting as *targeted feature learning*, where a single feature is constructed for such a predefined concept. We present a controlled comparison across three model signals (activation values, activation gradients, and parameter gradients) and two estimators (contrastive mean and a learned one-dimensional encoder-decoder), yielding six targeted methods, with CAA and GRADIEND as existing instances and four new methods covering the remaining combinations. We compare these methods against pretrained SAEs across 15 tasks and three language models, evaluating both detection and causal intervention. Across models, the strongest detection performance is achieved by contrastive activation value methods, whereas the strongest intervention performance is achieved by gradient-based methods. Overall, our results show that targeted feature quality depends jointly on the model signal and estimator, with detection and intervention capturing complementary properties.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, targeted feature learning, sparse autoencoders, activation steering, gradient-based methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vNcYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies "targeted feature learning" that aims at identifying features for a concept chosen in advance, rather than finding it post-hoc as typically done with SAE dictionaries. The authors analyze 3 model signals (activation values, activation gradients and parameter gradients) with 2 estimators. This gives six methods: CAA and GRADIEND, which already exist, plus four new ones. The methods are compared with pretrained SAEs on 15 task. The main finding is that activation-value methods are the best for identifying whether a concept is present in the model's representations, while gradient-based methods are the best for causally steering the model's behaviour toward or away from that concept. Overall, the authors conclude that feature quality depends on both the signal and the estimator, and that detection and intervention measure different properties.

Critical strengths
  • The authors conduct a systematic comparison that changes one thing at a time: what information the feature is built from, and how it is built. This analysis makes clear which choice drives the results, identifying the methods best at spotting a concept vs those that are best at changing the model's behaviour. I never saw this observation in previous literature and I think it should be taken into account in interpretability vs causal intervention work.
  • The paper also introduces a new middle-ground method called AGIEND that uses gradient information but steers the model the same way activation methods do, by nudging its hidden states. It still steers almost as well as the weight-editing method showing that the steering advantage comes from the gradient information itself, not from editing the weights. I find this an interesting finding of the analysis.
Critical weaknesses
  • Two of the three models are GPT-2 and Pythia-70M, which are small and old by today's standards, so Llama-3.1-8B is the only current-scale model. It is not clear to me why the authors decided to focus on these models rather than alternatives. Is the quality of SAEs good for these models?
  • The practical takeaway remains a bit vague to me. The paper shows that spotting and steering favour different methods, but it never turns this into a concrete recommendation. The authors suggest a hybrid method that would combine the strengths of both families, but they don't test it. Even a small experiment would show whether the trade-off can be overcome or is more fundamental, which would make the paper's message more actionable.
Questions for authors
  • Do you have an intuition or evidence for why gradient-based features change the model's behaviour more?
2 comments
AuthorsOpenReview ↗

Response to Reviewer vNcY

Thank you for your review. Please also see our general response summarizing the manuscript changes.

Weakness 1 (small, old models; SAE quality): The revised paper uses ten models instead of three, adding Gemma, Qwen, Mistral and DeepSeek models with up to 9B params. This broadens the model coverage across families and scale. The main findings hold across all ten. The additional models also include more recent SAE architectures and releases (e.g., Gemma Scope). The main findings remain consistent across the expanded evaluation. We further examine SAE performance on EMOTION in a new case study (Appendix C.7), showing that feature recovery varies across models and SAE suites.

(Note: The results were already computed during the review period, which made this fast and large extension of the results feasible.)

Weakness 2 (practical takeaway; hybrid untested): We have added explicit recommendations to the discussion (Section 6): CAA is particularly effective for detection, CAGA for activation space intervention with layer selection, and CGA for persistent model editing. We also evaluate the suggested hybrid approach (HYBRIEND) in a new appendix ablation (Appendix C.8). HYBRIEND combined activation value encoding with parameter gradient decoding and generally improves on the other IEND variants, but does not consistently outperform contrastive mean methods. Thus, combining the strengths of different signals is promising, although the trade-off is not fully resolved.

Question (why do gradient-based features change behaviour more?): Gradients literally point in the direction that changes the output the most by definition. Activation differences capture only how representations differ without necessarily indicating how to change the model's output. This makes gradients more directly aligned with the intervention objective. However, whether gradients aggregated across examples yield effective intervention directions is an empirical question. Our results support this intuition: CAGA (activation gradients) achieves roughly twice the intervention performance of CAA (activation values) with layer selection, despite both intervening in activation space. Thus, the advantage cannot be explained only by parameter space editing.

I thank the authors for their response and clarifications. I will maintain my score.

Reviewer_r6zCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies targeted feature learning in language models by crossing three signal modalities (activations, activation gradients, parameter gradients) with two estimators (a contrastive mean and a one-dimensional encoder–decoder, IEND). This yields four new methods (ACTIEND, AGIEND, CAGA, CGA) alongside the existing CAA and GRADIEND. All six are compared against pretrained SAEs on 15 tasks in five families (semantic, factual, grammatical, language, algorithmic) across Pythia-70M, GPT-2 and Llama-3.1-8B, evaluating both detection and intervention. Contrastive activation methods are best for detection, gradient-based methods are best for intervention, and SAEs fail to recover a feature on EMOTION.

Critical strengths

None

Critical weaknesses
  1. Novelty is weak, while classic targeted learning approaches are missing for a thorough comparison. The IEND estimator is the one from GRADIEND (Drechsel & Herbold, ICLR 2026), and the gender, race and religion tasks reuse GRADIEND's data. The new methods come from feeding other signals into this estimator and into CAA's contrastive mean. Other targeted methods are not compared. Linear probes are the standard detection baseline. ReFT-r1 (AxBench) learns one direction that is used for both detection and steering. The core finding, contrastive activations best at detection and a trained direction best at steering, was already partially reported in AxBench: diff-of-means is best at detection, and among representation-based methods a direction trained with the LM loss (ReFT-r1) is best at steering.
  2. Model and SAE choice. The only justification is SAE availability (l. 306), yet Gemma-2-2B/9B have Gemma Scope and Matryoshka SAEs, the latter targeting the absorption that motivates the paper (l. 66-68), and would match AxBench. Pythia-70M uses older ReLU SAEs.
Questions for authors
  1. How do a logistic probe and ReFT-r1 compare on these tasks?
  2. Why Llama-3.1-8B rather than Gemma-2? Does the SAE result, including EMOTION, change with Gemma Scope or Matryoshka SAEs?
  3. How do the gradient directions differ from attribution-patching or logit-difference directions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer r6zC

Thank you for your review. Please also see our general response summarizing the manuscript changes.

Question 1: (Logistic probe and ReFT-r1): We now report logistic probe and ReFT-r1 results on five models and eight tasks in the new Appendix C.8. Logistic probes achieve very strong detection outperforming CAA, but yield negligible causal intervention (<=1<=1%) on many models and tasks. ReFT-r1 performs poorly on our (compared to AxBench more restrictive) detection metric but achieves strong intervention on some tasks, although it is outperformed by CGA overall. These results extend our comparison beyond the controlled framework and further support the detection-intervention trade-off.

Question 2: (Model choice): The revised paper uses ten models instead of three, adding Gemma, Qwen, Mistral, and DeepSeek models with up to 9B params. This broadens the model coverage across families and scales, including more recent SAE releases such as Gemma Scope. The main findings hold across these models.

However, one SAE-related finding does change. Among the newly evaluated SAE suites of the new models, some contain features that detect EMOTION well (e.g., Qwen-3.5-2B), whereas others do not (e.g., Gemma-2-2B). We have revised Section 5 accordingly and now provide an additional analysis for this task in Appendix C.7. This variation is consistent with known properties of SAEs: the recovered features can depend on dictionary size through feature splitting and absorption [1], on SAE architecture and training setup [2], and even on the random seed of otherwise identical SAE training runs [3]. This further motivates our study of targeted feature learning: rather than relying on an untargeted SAE dictionary to contain a suitable latent for a predefined concept, targeted methods explicitly and more reliably learn a feature for that concept.

(Note: The results were already computed during the review period, which made this fast and large extension of the results feasible.)

Question 3: Comparison to attribution-patching or logit difference directions: Mathematically, our contrastive gradient difference DD is exactly the negative gradient of the log-odds margin m=logp(F)−logp(A)m=log p(F)-log p(A), i.e., D=−∇mD=-\nabla m. The difference from attribution patching is in aggregation and use. Attribution patching uses the activation gradients nablahmnabla_h m for per-instance circuit localization by weighting with activation difference (∇hm⋅(hF−hA)\nabla_h m \cdot (h^F - h^A)) to score component importance for circuit localization. In contrast, our gradient methods use the gradients themselves as signals to construct reusable feature directions across multiple factual-alternative contrasts (using either contrastive mean or IEND). We further evaluate parameter gradients ∇θm\nabla_\theta m in addition to the activation gradient direction ∇hm\nabla_h m used by attribution-patching.

Weakness 1 (Weak novelity and missing classical baselines): We would like to emphasize that reusing datasets from prior work does not undermine methodological novelity. Moreover, only three of our 15 tasks reuse data from GRADIEND, while we constructed datasets for the remaining twelve specifically for this study, partly adapting existing benchmarks. More importantly, our main contribution is the controlled generalization and comparison of targeted feature learning methods, introducing four new methods and systematically disentangling the effects of model signals and estimators. We also added now logistic probe and ReFT-r1 baselines in Appendix C.8, as discussed in Question 1.

Weakness 2 (model and SAE choice): We have substantially expanded the evaluation to ten models, covering additional model families and more recent SAE architectures. We also provide a more detailed analysis of the SAE results, as discussed in our response to Question 2.

[1] Bussmann et al. (2025). Learning Multi-Level Features with Matryoshka Sparse Autoencoders. ICML 2025.

[2] Karvonen et al. (2025). SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. ICML 2025.

[3] Paulo & Belrose (2026). Sparse Autoencoders Trained on the Same Data Learn Different Features. ICLR 2026.

Thank you for the detailed response and the substantial additions. The expansion to ten models with more recent SAEs resolves my concern about model choice, and the logistic probe and ReFT-r1 results answer my question about missing baselines.

My main concern remains, and it is close to Reviewer RJeG's. It is expected that directions obtained by optimizing directly on the target objective steer better than directions estimated from activations. Likewise, detecting concepts from activations is standard practice in probing, so it is not surprising that it works best here. The new probe results follow the same pattern: the logistic probe detects best but has low casual effect. Prior work already shows that detection performance does not predict causal effect. AxBench reports that "better classification does not directly lead to better steering", and Marks & Tegmark (2024) find that logistic-regression and diff-of-means directions classify similarly but differ in causal effect.

The systematic 3×23 \times 2 comparison and the benchmark are useful, and the paper documents this trade-off across more models and tasks than before. However, quantifying an expected result does not provide enough new insight for the main track. The work would fit well as a workshop paper. I therefore keep my rating.

AuthorsOpenReview ↗

Response to Reviewer r6zC

Thank you for your response. We appreciate your acknowledgment that the expanded evaluation and additional baselines address your earlier concerns with respect to the soundness of our results. We assume this means that there are no remaining methodological concerns with our work, but that we disagree about the (more subjective) novelty and impact.

We agree that AxBench and Marks & Tegmark (2024) demonstrate that detection performance does not necessarily predict intervention effectiveness. However, neither investigates gradient-derived intervention directions as we do. Moreover, AxBench shows that some methods are rather weak for steering (e.g., SAE) AND methods with similar detection performance can vary in intervention performance (e.g., Probe vs. Diff-Mean vs. ReFT-r1), which demonstrates the referenced lack of correlation. We instead establish clear factors (activation values vs. activation gradients) that guide what makes detection/intervention better while reducing the other capability, establishing this trade-off relationship. Another critical difference to the results reported by AxBench is that the paper reported poor performance at detection for gradient-based attribution methods (IG and IxG), whereas our gradient-based methods achieve good detection performance (especially CGA), although generally below CAA.

While gradients may intuitively outperform activation values for intervention, the magnitude of this advantage and the role of the estimator cannot be inferred from intuition alone. Our controlled comparison provides empirical answers to these questions. Importantly, CAGA substantially outperforms CAA for intervention despite both operating in activation space, demonstrating that the advantage comes from gradient information rather than parameter-space editing. In contrast to AxBench, where learned directions (ReFT-r1) outperform activation-based contrastive means for steering (DiffMean), we find that simple contrastive means of gradients generally match or outperform learned directions (IEND). Moreover, CGA also outperforms (the learned) ReFT-r1, the best-performing AxBench method usable for both detection and intervention, in our additional experiments.

We therefore disagree that our study merely quantifies an expected result. It introduces new methods and establishes practically relevant findings not demonstrated by the cited prior work.

We updated the paper (L127f, L456ff, L1711f) to more clearly distinguish our contributions from prior work.

Reviewer_RJeGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper performs a controlled study of different approaches of detecting a concept in LM computations. Using different signals (activation vs gradients of activation or params w.r.t an auxilary loss) and different estimators (contrastive mean vs IEND) the paper constructs different probes, and then evaluates their concept detection and intervention performance.

Critical strengths

I couldn't really think of any critical strengths. This is possibly because I am still unsure about the technical and conceptual contributions of this paper. I will update this sections after authors' response to my questions below.

Critical weaknesses
  • I am not really sure what motivated this study. The results seem kind of trivial in retrospect. Please see my Q1 in Questions For Authors section below.
  • This paper uses 2 kinds of estimators for the intervention and detection directions. Conceptually, both estimators should converge to the same direction, but the paper reports very different intervention scores. Further analysis is necessary to understand why this is happening. I have flagged this in the Questions For Authors section below.
  • Some parts of the paper are very hard to read. The authors often use unnecessary details and notations instead of intuitive explanations. An a simple example: in Lines 242-251, the paper is explaining "contrastive mean", which is a very simple concept and is already well-studied. And, I have never seen this being explained in such a complicated way using so many different notations.
Questions for authors
  1. Possibly this is a hindsight bias after reading your paper, but a lot of what you find seems kind of obvious. For example.
    • "Gradients for Interventions": Gradients literally tell you how to change the activation direction (or params) to match an objective. The intervention test is also measuring the same objective used to get the gradients. How is this surprising that the grad-based approach would perform well? Also, you get better intervention score with GRADIEND, which is IEND on parameters. Parameters have more numbers and thereby more numeric capacity compared to activations. IEND is literally training for flipping to a target concept. Combine them both and of course you will get better intervention scores compared to their other counterparts.
    • "Activations for Detection": Its contrast with "Gradients for Interventions" is confusing in the first place, as activations can have gradients as well.
      • The convention is also to train any kinds of probes on activations. How many papers have you seen that tried to detect concepts from parameter gradients?
      • For a well-trained LM, does detecting concepts from activation gradients also make sense? The purpose of detection is the detect concepts that are already present in the LM computations. Using gradients from another auxilary loss (CAGA) just seems like a very indirect and inefficient way over directly probing the activations (CAA).

Please correct me if I got any of the conceptual/technical details wrong. This is important as I am trying to understand what motivated this study in the first place. What did you expect to find different?

  1. The conceptual contribution of IEND is also very confusing. Your objective, how you pair up the (S, T) pairs, seems to be to always flip the target concept label. Both (Ai,Di)(A_i, D_i) and (Fi,−Di)(F_i, -D_i) pairings suggest that.

    • So, when you say "IEND learns to reconstruct" or is "equivalent to an SAE direction" it is confusing and in fact contradicting the training objective. SAEs actually try to reconstruct the native concept/representation (given male, preserve male), whereas you are trying to flip to a different concept class (given male, flip to female).
    • As far as I could understand, for concepts with categorical labels K>2K > 2, you learn nC2nC2 different IEND models for each pairing of categories. Given a concept color realized in 3 categories {red, green, blue}, an SAE direction is expected to learn the presence of a single label (e.g. red is present or not). You will, however, have 2 IEND models for the target red: green -> red and blue -> red. This is also very different from an SAE direction.
    • Lines 265-266: "We instead use both realizations as sources: (Ai,Di)(A_i,D_i) and (Fi,−Di)(F_i, −D_i)". The encoder activation tanh\mathrm{tanh} goes from [−1,1][-1, 1]. Do you expect activations for the data slices (Ai,Di)(A_i,D_i) and (Fi,−Di)(F_i, −D_i) to be opposite of each other? This is assuming the encoder and decoder are still single directions as depicted in the equation at the end of page 5 (you should number your equations).
  2. Across all the LMs, you report better intervention performance with IEND counterparts compared to contrastive mean, which also seem a bit conflicting. Assuming the signal is activation value: the contrastive mean is Diˉ=mean(hΩ(ctarg)−hΩ(ccurrent))\bar{D_i} = \mathrm{mean}(h_\Omega(c_\mathrm{targ}) - h_\Omega(c_\mathrm{current})). Given the IEND objective, the corresponding cross-label intervention direction dcid_{c_i} is expected to be approximately equal to Diˉ\bar{D_i}, as it was literally baked into the training objective. So, I am very confused how are you reporting very different intervention scores for them (in Figure 3).

    • How different are IEND decoder/intervention directions compared to Diˉ\bar{D_i}. Have you checked their cosine affinity or magnitude?
    • Have you considered the possibility that the IEND model is overfitting to the training pairs? What kinds of measure have you taken to validate this? i.e: testing on different prompt templates, different languages, etc.
4 comments
AuthorsOpenReview ↗

Response to Reviewer RJeG Questions

Thanks for your review. Please also see our general response summarizing all changes.

Question 1 (Motivation and expected findings):

  • "Hindsight bias:" We agree that gradients are intuitively promising for interventions, as they indicate how to change model behavior. Although GRADIEND has demonstrated effective gradient-based interventions, it remains unclear whether gradients outperform activation values or whether their advantage depends on parameter-space intervention. To our knowledge, these factors have not been systematically disentangled. This is precisely the motivation for our controlled 3x2 comparison, introducing 4 new methods to isolate the effects of signals and estimators. Crucially, our findings challenge the common use of activation value directions for both detection and intervention (e.g., steering): while activation values perform best at detection, gradient-based directions achieve substantially stronger interventions, even within activation space. Parameter gradient methods additionally enable persistent model rewriting without requiring intervention hooks at inference time. These findings go beyond what follows from the definition of gradients alone. We have revised the Abstract, Introduction, Results, and Discussion to better emphasize these contributions.
  • Parameter capacity: CAGA achieves roughly twice the intervention performance of CAA despite both operating in activation space. Thus, parameter-space capacity cannot explain this advantage, although it may contribute to the additional gains of parameter gradient methods.
  • "IEND is literally flipping to a target concept": Both contrastive mean and IEND use the same factual-alternative transitions, i.e., the IEND decoder targets are the same transitions used to compute contrastive means. Thus, learning directions that move between feature realizations is not specific to IEND.
  • Activation values for detection: We agree that using activation values for concept detection is well established. Our contribution is the systematic comparison with gradient signals, not activation-based detection itself. To avoid confusion, we have now changed the title to "... Activation Values for Detection...".
  • Gradients in well-trained LMs: Importantly, our gradient methods use alternative/counterfactual gradients, rather than relying on factual gradients, which tend toward zero as the model's predicted probability for the factual target approaches one (i.e., when the model is very confident). Alternative gradients remain informative even when the model is highly confident in its factual prediction, making them useful for concept detection in well-trained models.

Question 2:

  • IEND Reconstruction and SAE comparison: We do not claim that IEND is "equivalent to an SAE direction", which would indeed be incorrect. Our comparison in Section 3.4 concerns their shared scalar encoder-decoder structure, not their objective. IEND constructs target transitions from source signals, while SAEs reconstruct their inputs. We revised the term "reconstruction" to avoid confusion.
  • Multiple feature classes: The reviewer describes our pairwise construction, which indeed requires binom(K,2)binom(K, 2) models. However, we introduce a new one-sided construction (Sec. 3.2), which requires only one model per class, i.e., KK models. Our results show that one-sided is generally competitive with pairwise construction (App. C.5).
  • Encoder values: Yes, using both realizations with oppositely signed target transitions, we expect the encoder to distinguish classes through opposite signs. However, we do not expect perfectly symmetric values or saturation at +-1. Empirically, the separation works well.
  • We now number all equations as suggested.

Question 3:

  • Although both estimators use the same transitions, they need not learn identical directions. Contrastive mean weights transitions equally, whereas IEND learns a source-dependant scalar and a decoder that constructs the corresponding transitions. Its decoder direction therefore effectively weights transitions differently. Optimization choices may introduce further differences.
  • Cosine similarity: We now report IEND-contrastive mean cosine similarities in Appendix C.9. Alignment is often substantial for activation values and activation gradients (0.6-0.9), but lower for parameter gradients (0.2-0.4; Fig. 16). Higher alignment generally correlates with better detection and intervention, except that the intervention correlation is negligible for activation gradients (Fig. 17).
  • Overfitting: We cannot rule out overfitting entirely. However, IEND checkpoints are selected on validation data and evaluated on held-out test data, using the same splits as contrastive mean. Several tasks also test generalization to unseen target vocabularies, reducing the risk of memorizing training pairs.
AuthorsOpenReview ↗

Response to Reviewer RJeG Weaknesses

Weakness 3 (Presentation and readability): Our contrastive mean formulation is more general than the standard definition, as it must consistently orient transitions across pairwise and one-sided feature constructions, requiring additional notation. However, we agree that the explanation could be more intuitive. We have revised Section 3.4 to introduce the estimator directly before explaining the orientation conventions. We further improved readability in several places, including a concrete example in Section 3.2 and clarifying the IEND formulation and its relationship to SAE features in Section 3.4.

I thank the authors for their response to my questions. However, I still think a lot of the insights drawn from this work should have been intuitively obvious to an experienced AI/ML researcher. For example:

On response to my Q1

  • Gradients will have an advantage over activation values, especially when they are directly being optimized to change the target behavior. As I said in my original review, this is literally what gradients are.

    • I brought up the parameter capacity point to explain the difference between GRADIND vs CAGA (not CAGA vs CAA). GRADIND vs CAGA is indeed explained by parameter capacity.
    • "parameter-space capacity cannot explain this advantage" - No. It is explained by gradients!
  • "Both contrastive mean and IEND use the same factual-alternative transitions" - Even if the train set is the same, the contrastive mean doesn't tell us anything how to change the channelwise (or paramwise) values to match a target objective, gradients tell us that. And, when the grad objective is the target behavior, IEND will have a natural advantage.

Q2 Thanks for confirming that my understanding of your approach was mostly accurate.

Q3

  • "they need not learn identical directions" -- thanks for clarifying this. My question was more like a theoretical speculation anyways.
  • "Alignment is often substantial for activation values and activation gradients (0.6-0.9)" -- this is what I hoped to see. Checking cosine similarity in param space is a bit noisy, as highly-parameterized LMs can have so many computational redundancies. The primary signal the LM actually uses for a task, doesn't necessarily get boosted during optimization.

Again, I appreciate the authors response. However, I retain my original score, as this paper is more like a confirmation of intuitive expectations than a source of new insights.

I wish the authors best of luck with their future work.

AuthorsOpenReview ↗

Response to Reviewer RJeG

Thank you for your response. We read this as an acknowledgment that there are no remaining methodological concerns with our work, but that we disagree about the (more subjective) novelty and impact.

We agree that gradients are intuitively well-suited for intervention. However, this intuition does not establish the magnitude or consistency of their advantage across models and tasks. Importantly, if gradients provide such informative signals about the target behavior, this advantage should (or at least could) also apply to detection. However, our results show the opposite: not only is there no advantage, we even see that only looking at values without direction (i.e., plain activation values in CAA without their gradient direction) is a clearer and better signal for detection.

Thus, we first of all believe that while some of our results align with intuition, the empirical observations confirm this for intervention, but also show that for detection, this is indeed a drawback instead (possibly because detecting feature expression requires no direction). There is no such empirical evidence in the literature so far that untangles these effects across signal/intervention space (activations vs parameter). Given the widespread use of activation value directions for intervention and SAEs for detection, systematically quantifying these differences has practical relevance for our community.

We therefore respectfully disagree that our findings merely confirm intuitive expectations.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We thank all the reviewers for their reviews. We are happy that the reviewers generally see the merit of our work, but acknowledge the open questions that were raised. We have addressed the related concerns and substantially strengthened the manuscript through the following revisions (all changes are highlighted in blue in the manuscript):

  • Clarified contributions: We better emphasize our controlled 3x2 comparison, which generalizes IEND, introduces four new targeted feature learning methods, and systematically disentangles the effects of model signals and estimators. In particular, our results show that gradient-based features improve intervention even within activation space (CAGA using activation gradients), demonstrating that the gradient intervention advantage does not require parameter space editing.
  • Expanded evaluation: We increased the number of models from 3 to 10, covering more recent model families and SAE architectures. The expanded results further support our central finding that activation values favor detection, while gradients favor intervention.
  • Refined intervention evaluation: During revisions, we identified an inconsistency between the implementation of the language modeling preserving constraint (LMS) and its description in the paper. We updated the intervention results of the original models. The updated results provide a more accurate comparison of the estimators: contrastive mean now generally matches or outperforms IEND for gradient-based interventions, while our main findings remain unchanged.
  • Additional experiments: We added logistic probe and ReFT-r1 baselines, a hybrid IEND variant (Appendix C.8), a direction-alignment analysis (Appendix C.9), and a detailed EMOTION case study (Appendix C.7).
  • Clarity and practical implications: We improved the methodological explanations (Section 3), updated the title ("Activation Values for Detection") to enhance clarity, and added concrete recommendations to the discussion (Section 6).
AuthorsOpenReview ↗

Final Remarks to the Area Chair

The discussion with the reviewers has shown that our response has clarified the concerns with respect to the soundness of our results that Reviewers RJeG and r6zC raised. Since these overlap with the concern raised by vNcY (more and larger models), we assume that this concern is also resolved.

The remaining issue from RJeG and r6zC is whether the results are sufficiently novel. First of all, we want to highlight that vNcY judges the variations our study provides over related work as critical strength. Additionally, RJeG and r6zC are presumably experts with deep knowledge about steering and feature detection within models, explaining their intuitive grasp of the direction of the results. However, our results are not just a direction, but they show that the gap is (very!) large (Figure 1).

  • The error when using contrastive activation value means (CAA) for detection is ~4%, when using contrastive activation gradient means (CAGA), this increases to 12%, i.e., an increase in the error by a factor of ~3 (lower is better).
  • The difference in intervention capability (steering) is similar, but in the opposite direction: ~7% probability shift when using contrastive activation value means (CAA), and ~14% (CAGA) when using the contrastive activation gradient means, i.e., an increase in steering capability by a factor of ~2 (higher is better).

We highlight that no prior work (including AxBench) has studied this trade-off so far. Indeed, AxBench rather shows that some methods are rather weak in steering AND detection performance can vary in intervention performance, i.e., shows a lack of correlation. We instead establish clear factors (activation values vs. activation gradients) that guide what makes detection/intervention better while reducing the other capability. This point is highlighted as critical strength by Reviewer vNcY (“I never saw this observation in previous literature and I think it should be taken into account in interpretability vs causal intervention work.“).

This is just one aspect of our results, which additionally show that directly intervening in the (larger) parameter space through gradients (CGA) has additional advantages for intervention and reduces the drawback when using this for detection – which is also not studied in the literature.

Therefore, we ask the area chair to answer the following question when determining whether our paper should be accepted or not: Do the members of the machine learning community know about this trade-off and do they have an accurate grasp on the strength of this trade-off, considering that activation-based approaches such as CAA and the much weaker Sparse Auto Encoders (SAEs) are the most common methods for both detection and steering within the current literature?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.