- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work proposes an attribution-guided pruning framework that unifies circuit discovery and targeted behavioral correction in language models. The method uses Layer-wise Relevance Propagation (LRP) and gradient saliency to identify important neurons and weights. For circuit discovery, highly attributed components are retained to extract sparse task-specific subnetworks. For behavioral correction, attribution scores on undesired behavior prompts are contrasted with those on general text to identify and prune behavior-associated parameters. Experiments on IOI, toxicity, and repetition tasks show that the method can extract sparse circuits and suppress targeted behaviors while largely preserving performance on selected general-purpose benchmarks.
Critical strengths
- The paper connects circuit discovery and targeted behavioral correction through a unified parameter-level attribution and pruning framework, providing a practical approach to analyzing and intervening on model behaviors without fine-tuning.
- The authors evaluate multiple attribution methods, pruning granularities, and model checkpoints. They also assess behavioral changes alongside general utility and conduct necessity and sufficiency experiments to investigate the functional relevance of the identified components.
Critical weaknesses
- The methodological novelty remains somewhat limited, as the framework primarily combines existing attribution methods with parameter pruning. The distinction from existing attribution-based pruning and circuit discovery approaches is not sufficiently established.
- The behavioral correction results rely on narrowly selected prompts and limited evaluation metrics. In particular, some toxicity evaluation sets are very small, while general capability preservation is assessed using only WikiText2, HellaSwag, and ARC-Challenge. The generalization of the method to broader prompt distributions and other capabilities remains unclear.
Questions for authors
- The authors should provide more direct comparisons with existing circuit discovery and attribution-based pruning methods under matched pruning budgets and evaluation settings. This would help clarify the methodological novelty and establish whether the proposed framework offers consistent advantages over existing approaches.
- The authors should evaluate behavioral correction on larger, independently held-out datasets using additional behavioral and utility metrics. They should also test the sensitivity of the results to different reference datasets and behavior definitions to better establish the robustness and generalizability of the proposed method.
4 comments
Additional message to Reviewer kDDK: Weight amplification
We additionally tested whether our framework can support a different parameter
intervention: strengthening selected behavior-associated weights rather than
setting them to zero. This is a preliminary experiment, not part of the
submitted claims. On google/gemma-3-4b-it and Qwen/Qwen3-4B, we use LRP or
Gradient to score weights from 128 correctly solved training examples for exact
arithmetic and GSM8K. We multiply selected weights from down_proj,
gate_proj, or up_proj by coefficients in {1.01, 1.1, 1.2, 1.5, 2.0} and
select separately for each of three seeds on the development set. We retain
only settings satisfying the strict utility constraints: PPL <= 1.10x baseline
and HellaSwag/ARC-Challenge >= 95% of baseline. The table reports development
accuracy; layers and coefficients are ordered by seed 0/1/2.
| Model/task | Method | Original -> mean selected | Gain | Layer by seed | Coefficient by seed |
|---|---|---|---|---|---|
| Gemma Arithmetic | LRP | 92.58 -> 96.48 | +3.91 pp | down/down/down | 1.2/1.2/1.2 |
| Grad | 92.58 -> 95.51 | +2.93 pp | down/down/- | 2.0/1.5/- | |
| Gemma GSM8K | LRP | 88.67 -> 92.06 | +3.39 pp | gate/gate/gate | 1.1/1.2/1.2 |
| Grad | 88.67 -> 91.93 | +3.26 pp | up/up/down | 1.2/1.1/1.5 | |
| Qwen Arithmetic | LRP | 99.22 -> 100.00 | +0.78 pp | gate/gate/up | 2.0/1.5/2.0 |
| Grad | 99.22 -> 100.00 | +0.78 pp | gate/up/gate | 1.2/2.0/1.1 | |
| Qwen GSM8K | LRP | 94.53 -> 95.31 | +0.78 pp | gate/up/gate | 2.0/2.0/1.2 |
| Grad | 94.53 -> 95.44 | +0.91 pp | down/gate/gate | 1.1/2.0/1.2 |
All 23 available seed-specific selections satisfy the strict utility criteria; one Gemma Arithmetic Gradient seed is incomplete and is not included. Gemma shows the clearest gains, while Qwen starts near the ceiling and therefore has less room to improve. These development-set results provide initial evidence that our framework can guide amplification as well as removal on two math behaviors; they do not yet establish general capability improvement. Still, every reported model-task-method mean improves over its unamplified baseline while satisfying the strict utility criteria. The gains are modest, especially for Qwen whose initial accuracy is already 94.53-99.22%, but their consistent direction is an encouraging sign that targeted amplification merits broader study. We plan to extend this direction to other model families, including Llama, and tasks with lower initial accuracy.
General response to reviewer kDDK and AC
Thank you again for asking for broader circuit-discovery evidence and clearer comparisons. During the rebuttal, we clarified why scalar-parameter budgets do not directly match activation-graph edges and made the purpose of the shared-neuron RelP comparison explicit. We also completed 0-100% LeRF curves for exact arithmetic and GSM8K on Gemma and Qwen. LRP permits more removal than Gradient in 11/12 projection cases at 95% accuracy retention and 12/12 at 90%. In addition, we added the preliminary weight-amplification experiment above and a held-out second-dataset correction study. We believe these additions address the main concerns about evaluation beyond IOI, comparison scope, and broader applicability.
If any concern remains, we would be grateful for your feedback and the opportunity to discuss it further.
Thank you for the detailed response and additional experiments.
I appreciate the clarification of the distinction between parameter-level circuit discovery and activation-graph methods. The new experiments are particularly helpful and strengthen the evidence that the framework generalizes beyond IOI.
However, the RelP comparison mainly measures neuron overlap rather than comparative circuit faithfulness under matched pruning budgets. Thus, my concern regarding comparisons with existing methods remains unaddressed and I will maintain the current rating.
Clarification of distinction of our work from other circuit discovery methods, and new experiments on math circuits for arithmetic and GSM8K
Thank you for the constructive questions. We clarify both the direct comparison and the scope of our generalization evidence below.
Comparison with existing methods. Many circuit-discovery methods construct circuits over an activation-level computation graph. Our question is different: which learned parameters can be retained or removed to obtain a fixed subnetwork for the behavior? We therefore evaluate parameter interventions at both neuron and individual-weight granularity. The individual-weight scores arise directly from LRP's relevance decomposition to connections or from gradients with respect to weights; they are not formed by multiplying a neuron score by weight values. An activation-graph node or edge has no one-to-one correspondence with a scalar parameter, so matching ACDC or EAP to an individual-weight pruning budget would compare different intervention units.
We therefore compare with RelP at neuron granularity, where both methods share the same intervention unit and propagate relevance, although RelP introduces LRP into an activation-patching graph. On the same six seeded IOI sets and matched top-k budgets, 18.4-45.1% of the retained neurons overlap at 5% retention, versus 5% expected from two random top-5% sets. Thus, their overlap is well above the random expectation, but they do not produce the same circuit. This is the intended purpose of the comparison. We test intervention faithfulness separately within our parameter-space framework by comparing LRP, gradient saliency, and Wanda at matched pruning budgets using LeRF and MoRF. We will make these two comparison questions explicit in the revision.
Circuit discovery beyond IOI. We added exact arithmetic and GSM8K using
google/gemma-3-4b-itandQwen/Qwen3-4B. The arithmetic benchmark is synthetically generated and balances addition, subtraction, multiplication, and exact-integer division; we evaluate exact final-answer accuracy. GSM8K contains natural-language word problems that usually require several reasoning steps. It is the harder task in this pair, while arithmetic gives us a controlled setting between IOI and multi-step reasoning.For each completed model-task pair and attribution method, we use three seeded sets of 128 correctly solved training examples and attribute the extracted final-answer tokens. Evaluation uses separate, fixed sets of 500 arithmetic questions and 660 randomly selected GSM8K test questions. We apply unstructured LeRF independently to all weights of each MLP projection type, using matched 0-100% pruning rates in 5% steps. The complete curves with seed-wise min-max bands are the most informative presentation because they show both gradual degradation and sharp transitions. Due to the response-space limit, we summarize them below. Each entry is the largest removal rate before mean accuracy first falls below 95%/90% of its unpruned value. D/G/U denote down/gate/up projections.
As in the submitted IOI experiments, LRP supports sparser retained circuits within the targeted projection family than Gradient at matched budgets: across the 12 comparisons, its 95%/90%-accuracy removal limits are larger in 11/12 and 12/12 model-task-projection settings, respectively. The math circuits are less sparse than the IOI circuits, consistent with arithmetic and especially multi-step GSM8K relying on more distributed parameter support. Nevertheless, the results show substantial removable fractions beyond IOI, while also showing that circuit compactness depends on the task, model, and projection type. We will provide all curves in the revision/supplement rather than only selected layers.
Second-dataset correction evaluation. Please see our response to Reviewer 85fz for the reserved ToxiGen protocol and fixed-setting results. This study tests whether the framework can be adapted to a second toxicity dataset; it does not claim that one mask transfers across datasets or models.
Please also see our response to Reviewer EyoC for the
G,-T, andG-Tablation and the complete WinoGrande/PIQA evaluation. The contrast is best in five of six model-method cases, but this does not test replacement of C4 by a different general-reference corpus. We will add reference-corpus sensitivity as a remaining limitation.