acceptodds
Under review as a conference paper at ICLR 2027

Faithful Pruning: Preserving Predictions and Interchange Intervention Responses

Abstract

Post-training pruning usually makes a network smaller while preserving its outputs. A smaller network would also make mechanistic interpretability studies cheaper, but these studies depend on more than the outputs. They apply interchange interventions to internal states and measure how the outputs change, and a pruned network that matches the original's predictions can still respond differently to these interventions. We study feed-forward (FFN) channel pruning that aims to preserve both the predictions and these responses. Removed channels are replaced by constants that fold exactly into the layer's bias, so the pruned layer is physically smaller. We introduce a paired-response objective that penalizes the pruned network's output error without the intervention, its output error with the intervention, and its error in the change the intervention causes. A greedy selector fits this objective from a small calibration set and can prune a single layer or every FFN of a network. Against an otherwise identical objective without the paired term, pairing lowers intervention-response error by 7.8% to 19.6% in all six single-layer GPT-2 language settings we test, carries over unchanged to Pythia-410m, and persists when the retained weights are also trained. When every FFN of GPT-2 is pruned, pairing helps on pronoun continuation and gives mixed results on indirect object identification. Without the intervention, the effect of pairing on output error varies across settings.

Primary area:
interpretability and explainable AI
Keywords:
structured pruning, mechanistic interpretability, interchange interventions, causal abstraction, model compression, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ALg3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies structured pruning for mechanistic interpretability experiments. It proposes Paired-Response Reduction, which, in addition to errors on natural and intervened outputs, explicitly constrains the intervention-effect error between the pruned and original models. Experiments focus primarily on GPT-2 on the IOI and occupation-pronoun continuation tasks, with additional experiments on Pythia-410m and DeiT-Tiny. Across six single-layer language-model settings, the paired objective reduces intervention-effect MSE by 7.8% to 19.6%, although natural-input KL and some task metrics improve in some settings and deteriorate in others. Overall, the paper demonstrates that incorporating a specified intervention into the pruning objective can improve the fidelity of the pruned model's intervention responses, but it does not establish that the method can consistently preserve real task performance or the internal causal mechanism.

Critical strengths
  1. The paper addresses a practically meaningful problem. The authors appropriately distinguish ordinary predictive fidelity from intervention-response fidelity and point out that conventional pruning methods may alter a model's responses to internal interventions even when they preserve ordinary outputs. This issue is relevant to using pruned models to reduce the cost of mechanistic interpretability experiments.
  2. The experimental results are reasonably consistent on the core metric. Across all six single-layer GPT-2 language settings, the paired method reduces intervention-effect MSE, and NRE also improves in every setting.
Critical weaknesses
  1. Whole-model pruning lacks a complete evaluation of real task performance. The whole-model pruning experiments do not report IOI accuracy, full-vocabulary top-1 accuracy, or true-target NLL. It is therefore unclear whether large-scale pruning actually preserves normal task performance, making it difficult to determine how much real task performance is sacrificed to improve intervention fidelity.
  2. Across the six single-layer language settings, Natural KL improves in only two settings and deteriorates in the other four, with increases of more than 30% in some settings. Under 50% whole-model pruning, IOI Natural KL also worsens by 40.6%. This suggests that the paired objective may substantially sacrifice ordinary predictive fidelity.
  3. Experimental coverage is limited. The main conclusions are based on a single GPT-2 checkpoint and two relatively specialized language tasks; Pythia uses only one checkpoint, with the IOI experiment being exploratory; and the vision experiments show no significant improvement. The current evidence is insufficient to establish that the method generalizes to larger language models, different architectures, interventions at different levels, or more complex downstream tasks.
  4. The method depends on prespecified intervention types, locations, and donor distributions, but the paper evaluates it primarily under the same intervention settings used during calibration. It remains unclear whether the pruned model can generalize to unseen intervention locations, different types of donor inputs, or multiple simultaneous interventions.
Questions for authors
  1. Could the authors report two-candidate IOI accuracy, full-vocabulary top-1 accuracy, and true-target NLL for 25% and 50% whole-model pruning? In particular, under 50% whole-model pruning on IOI, Natural KL worsens by 40.6%. Does this change lead to a decline in actual task performance?
  2. Could the authors provide end-to-end inference time, GPU memory usage, and the actual runtime of intervention experiments? Is the resulting speedup sufficient to support the paper's claimed cost advantage for interpretability?
  3. In the IOI experiment with 25% whole-model pruning, the paired method's Effect MSE is essentially tied with that of the control, whereas Task MAE worsens from 0.232 to 0.345. How do the authors explain the substantial deterioration in the task-related error despite little change in the overall intervention-effect error?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ALg3

Thanks for the detailed questions. We've added the evaluations you asked for and looked into the discrepancies you pointed out.

(W1, Q1) Whole-model task performance. We agree this was missing. We evaluated the saved compiled models on the 256 held-out indirect object identification (IOI) pairs (dense two-candidate accuracy 1.000; Appendix Table 13). The results are now in Section 5.3 and Appendix Table 24. NLL is true-target negative log-likelihood, VBP is variance-based pruning, and FFN is a feed-forward layer.

IOI, every FFNDensePairedSame-data controlVBP
25%: two-candidate accuracy1.0001.0001.0001.000
25%: full-vocabulary top-10.9100.9450.9570.918
25%: NLL0.7210.6690.7520.816
50%: two-candidate accuracy1.0000.9730.9880.910
50%: full-vocabulary top-10.9100.7230.7730.418
50%: NLL0.7211.8031.3982.712

At 25%, paired NLL is 0.084 nats below the control (95% interval 0.070–0.097). At 50%, the 40.6% Kullback–Leibler (KL) increase does come with worse task performance: paired misclassifies 7 of 256 pairs, the control 3, and VBP 23. Paired NLL is 0.405 nats above the control (0.335–0.476).

(W2) Natural KL. You're right: KL rises in four of six single-layer settings and by 0.274 nats at 50% whole-model IOI. At 25% whole-model IOI, pairing lowers it by 31.6% (Section 5.3). The single-layer increases are at most 1.95×10−31.95\times10^{-3} nats and vary across refits: over five perturbations, the frozen selector's KL change ranges from −41.8% to +127.6% (Section 5.6). With trained retained weights, pairing lowers KL on both draws, by 46.4% and 12.9% (Table 1). That does not explain away the 50% result, which shows up in both accuracy and NLL above. Section 6 identifies an explicit KL constraint as a next step.

(W3) Coverage. We agree the experiments cover a limited set of tasks, and acknowledge this in Section 6. We chose IOI and pronoun continuation because their mechanisms are documented, giving each intervention a defined meaning (Wang et al., 2023; Vig et al., 2020). Pythia-410m adds a second architecture family with parallel attention and FFN branches. Its pronoun result passes the prespecified stability check; its IOI result is exploratory (Section 5.4).

(W4) Unseen locations, donors, and simultaneous interventions. We've now tested these without refitting (Section 5.7, Appendix Tables 21–22). Entries give the geometric mean of paired / same-data-control effect mean squared error (MSE), followed by the number of interventions where paired is lower.

Held-out family (count)One FFN, 25%Every FFN, 25%Every FFN, 50%
Other residual locations (6)0.916 (5/6)0.984 (3/6)0.999 (1/6)
Random-name donors (2)0.936 (2/2)0.966 (1/2)1.059 (1/2)
Single-head output patches (11)0.935 (10/11)0.887 (8/11)1.181 (1/11)
Two simultaneous patches (2)0.928 (2/2)0.799 (2/2)1.091 (0/2)
All 210.929 (19/21)0.912 (14/21)1.106 (3/21)

At 25%, the advantage carries over on aggregate. At 50% removal from every FFN, it does not. The general comment gives the intervals and patching-study results.

(Q2) Inference time, memory, and intervention runtime. We measured these directly (Section 5.7, Appendix Tables 23 and 26). Removing 25% and 50% of every FFN leaves 110.3 and 96.1 of GPT-2's 124.4 million parameters. A full forward pass with logits at every position is at most 1.21 and 1.39 times faster in FP32 (batch 128, length 20), with no gain at batch 1, where kernel launches dominate. Peak GPU memory at batch 128 and length 20 falls from 996 MiB to 941 and 887 MiB. The 211-pass IOI patching sweep, using only final-position logits, is 1.30 and 1.71 times faster (13.6 s and 10.4 s versus 17.8 s). Selection uses 128 dense calibration pairs; a whole-model run of all four methods took 608–715 s. At 25%, where transfer holds, the speedup is modest at this scale. We've revised the Introduction to lead with response fidelity and describe cost using these measurements.

(Q3) Tied effect MSE, higher task error. We looked into this discrepancy (Section 5.3, Appendix Table 25 and Figure 5). Effect MSE weights all 50,257 centered logits equally. Task mean absolute error (MAE) measures one direction: the indirect-object logit minus the subject logit. That direction carries 0.026% of paired effect-error energy and 0.011% of the control's, so equal effect MSE can hide a difference on the task contrast. Paired overstates the intervention's effect (mean signed error −0.264 versus −0.107 for the control; dense effect −0.896). The methods are close through block 2 (task MAE 0.187 and 0.189). The gap opens as blocks 3–6 are reduced, upstream of the block-8 port: after block 6, MAE is 0.326 versus 0.206.

Thank you for the response. I still have some concerns, e.g, the mismatch between Effect MSE and task-relevant fidelity. Your explanation suggests that a task-relevant logit direction can account for only a tiny fraction of the total effect-error energy, such that similar Effect MSE values may coexist with substantially different task MAE. Does this indicate that Effect MSE alone may be insufficient as a measure of intervention fidelity, since errors in task-relevant causal directions can be diluted across the full vocabulary?

AuthorsOpenReview ↗

Effect MSE and task-relevant fidelity

Thanks, this is a fair point, and our answer is yes. Effect MSE averages the error over all 50,257 centered logits, so an error concentrated in one task-relevant direction can be diluted, and two models can tie on effect MSE while differing on the task contrast, as at 25% whole-model IOI. We read effect MSE as a measure of whole-output response fidelity and task-contrast error as a separate, complementary check. The paper reports both in every setting, and a study that cares about a particular contrast should check it directly.

We also tested whether weighting the task direction in the objective fixes the gap (new Table 28 in Appendix K). We added γλ\gamma\lambda times the squared error of the intervention-induced IO-minus-subject contrast, computed exactly with one extra derivative probe per pair, and reran the 25% whole-model IOI chain. No weight closes the gap to the control's task MAE of 0.232. The best setting, γ=1\gamma=1, lowers task MAE from 0.337 (the γ=0\gamma=0 rerun) to 0.290, but natural KL rises from 36.7 to 74.6 (×10−3\times10^{-3}) and the held-out advantage is lost (1.14 times the control's effect error). The results also aren't monotone in γ\gamma: γ=4\gamma=4 gives 0.418. With greedy constant replacement applied block by block, a single output direction is hard to control through an objective weight alone, so reporting task-level metrics next to effect MSE remains necessary. Appendix K now says this.

Reviewer_ST11OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is focused on post-training pruning, or feed-forward channel pruning to be specific for network models. The pruning considers both the interactions between different parts of the network (unsupervised aspect) and the prediction performance of the network (supervised aspect). The evaluation is on GPT-2 with two language tasks: indirect object identification and pronoun continuation. The results were mainly compared to variance-based pruning (VBP) and causal mechanism reduction (CMR).

Critical strengths

The proposed method suggested a direction of post-training network pruning that targets on keeping the output performance and the responses to interventions.

Critical weaknesses

The presentation and the evaluation of the proposed method lack enough evidence to show the proposed method is indeed superior to other prior work. In fact, the explanations and discussion on the experiment results are quite limited. We would like to see how the author(s) show evidence through various experiments and explain why we obtain certain results.

Questions for authors

• Why was the evaluation focused on GPT-2, or language-related networks and tasks? The title and the methodology do not show any linkage to the language tasks. On the other hand, the author(s) show the result on ImageNet-100 from DeiT in Figure 2. Does it make sense to compare all the results together? What could be conclusion for Figure 2?

• When comparing to other methods such as VBP and CMR, we do not see the same compared criteria that were used in the prior work.

• How can CMR and VBP help for tasks like IOI and pronoun continuation? For instance, can we say the use of CMR in the proposed method suggests the superior performance on the pronoun continuation task?

• Why was GPT-2 chosen as the tested model? Can the proposed method be extended to larger models? Can the proposed method be scaled up to cover big models that we use frequently these days?

• Why did the author(s) choose indirect object identification IOI and pronoun continuation as the two target tasks for the evaluation? How many IOI and pronoun continuation tasks should we investigate and they are enough for a meaningful evaluation?

• What is the computation overhead for the proposed method? I did not find any discussion on this draft.

• Why does the comparison cover only VBP and CMR on Table 2? There is no comparison to other two groups of methods, from distillation and intervention-sensitive pruning?

• The contribution paragraph should be called “summary” or “summary and contributions” because not all the items refer to contributions.

• Most content in the Results Section is about the quantitative evaluation results while not much discussion was done. In a sense, we would like to see how the author(s) explain for each improving or worse result compared to prior work. On the other hand, the Discussion and Limitations Section looks too short and contains no detail comments on individual experiment series.

• Ablation study, such as those for parameter tuning was not included in this draft, any reasons?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ST11 (Part 1 of 2)

Thanks for the review. We've numbered your questions Q1–Q10 in the order of your bullets.

(W) Evidence and discussion of results. The comparison we rely on most is the same-data control: it sees exactly the same calibration pairs as our method, so the paired term is the only difference (Section 4). Adding that term lowers intervention-effect error in all six single-layer GPT-2 language settings, by 7.8% to 19.6%, with every 95% interval excluding zero (Section 5.1).

We've also added a comparison with prior pruning under interventions that weren't used to fit the models. Across 21 held-out interventions, paired effect error relative to variance-based pruning (VBP) is 0.634 with 25% removal from one feed-forward (FFN) layer and 0.658 with 25% removal from every FFN. Paired has lower error in 21/21 and 20/21 interventions, respectively (new Section 5.7). At 50% removal from every FFN, however, the advantage over the same-data control does not transfer (ratio 1.106). We discuss the individual results under Q9 below.

(Q1) Why language models, and what Figure 2 shows. We focused on language models because the IOI circuit (indirect object identification; Wang et al., 2023) and the gendered-pronoun mediation analysis (Vig et al., 2020) give us documented mechanisms and interventions with a clear meaning. The method itself is not specific to language, which is why we also included DeiT-Tiny on ImageNet-100.

Figure 2A uses the same relative scale across settings: the percentage reduction in effect error compared with the same-data control. Pairing lowers effect error in every single-layer language setting. Its effect on natural Kullback–Leibler (KL) divergence varies across settings (Figure 2B), and the vision gain is not distinguishable from zero.

(Q2) Comparison criteria. We report ordinary-performance metrics for every method in every single-block setting in Appendix Tables 12 and 14: two-candidate accuracy, argmax agreement with the dense model, negative log-likelihood, full-vocabulary top-1, and calibration error for vision. We've now added task metrics for whole-model pruning (Appendix Table 24), along with inference throughput and memory (Appendix Table 26). The main text focuses on intervention-response error because that's what the paired term targets, and ordinary-performance metrics don't measure it.

(Q3) Role of CMR and VBP. We use VBP and CMR-local (causal mechanism reduction) as baselines. Our method shares their exact compilation step: a removed channel is replaced by a constant, and its contribution is folded into the output bias (Section 3.1, Equation 2). It does not use either method for channel selection, so the pronoun result does not come from CMR. VBP ranks channels by activation variance, CMR-local adds outgoing-weight magnitude, and our selector uses output derivatives and the paired-response objective (Sections 2 and 3.3). For pronoun continuation, paired reduction has lower effect mean squared error (MSE) than CMR-local on both draws: 0.954 versus 2.316 and 0.843 versus 2.543, both ×10−3\times10^{-3} (Appendix Table 18).

(Q4) GPT-2 and larger models. We chose GPT-2 small because that's where the IOI circuit was identified, so the intervention has a known meaning. The selector uses a fixed number of backward passes per calibration pair, one per random output probe (M=16M=16 probes, n=128n=128 pairs; Section 3.4), and needs no retraining. Its cost therefore grows with model size as a backward pass does.

On GPT-2, single-block selection took 486 s of worker time and at most 0.60 GB of GPU memory. Pruning every FFN took 608–715 s for all four methods together (Section 4, Appendix K). We also used the same settings on Pythia-410m, a 24-block model from a second architecture family (Section 5.4). That's the largest model we've tested.

Our response continues in a reply to this comment.

AuthorsOpenReview ↗

Response to Reviewer ST11 (Part 2 of 2)

Continuing with Q5–Q10.

(Q5) Choice and number of tasks. We chose these two tasks because both have documented mechanisms, which gives the interventions a clear meaning. They also use different donors: a role-swapped donor patched at the second subject position for indirect object identification (IOI), and a donor differing in one occupation for pronoun continuation (Section 4). The revision also evaluates each pruned IOI model under 21 interventions that weren't used for selection (Section 5.7). We agree that more tasks would strengthen the evidence, and we've made that limitation explicit in Section 6.

(Q6) Computational overhead. The single-block study used 486 s of worker time and at most 0.60 GB of GPU memory on one RTX 4500 Ada GPU. These numbers were in Appendix C under "Numerical execution"; we've brought them into Section 4 and added measured whole-model costs in Appendix K. A whole-model run of all four methods for one task and budget took 608–715 s and at most 1.01 GB. With 25% and 50% of every feed-forward (FFN) layer removed, the compiled networks run a full forward pass at most 1.21 and 1.39 times faster than the dense model. The IOI patching sweep runs 1.30 and 1.71 times faster.

(Q7) Comparators in Table 2. Table 2 compares paired reduction with variance-based pruning (VBP) and the same-data control (E2E-augmented). Appendix Table 6 adds augmented local reconstruction, and CMR-local (causal mechanism reduction) appears in every single-block table (Appendix Tables 11–12). For distillation, Section 5.2 and Table 1 compare students whose retained weights are trained on the same 128 pairs; pairing helps in that setting too.

We haven't run a full causal-distillation model such as DIITO (Wu et al., 2022), and we've listed this as a limitation in Section 6. Circuit-discovery methods such as Edge Pruning and DiscoGP select edges or parameters that explain one task (Section 2). Their outputs are not direct matched-width comparators for this evaluation, which uses compiled smaller models.

(Q8) Name of the contributions paragraph. We now introduce the list as "Summary and contributions" in Section 1, as you suggested.

(Q9) Discussion of results. We discuss the results that go against pairing in Section 5, and the revision adds two more. Here are the main points.

The frozen selector's single-run Kullback–Leibler (KL) changes switch sign across refits (Section 5.6, Appendix Tables 8–9), so they don't establish a consistent trade-off. At the primary IOI setting, VBP has lower task error but 55% higher full-output effect mean squared error (MSE) than paired reduction (Appendix Table 11): it tracks the task contrast more closely and the full response less closely. Augmented reconstruction performs much worse on language; it fits each FFN layer's local outputs rather than the network's final outputs (Section 5.1).

The new analysis in Section 5.3 explains the higher IOI task error at 25% removal from every FFN (0.345 versus 0.232 for the control), despite tied effect MSE. The task direction accounts for just 0.026% of the paired model's effect-error energy, so it barely affects the aggregate error. In Section 5.7, we also show that the advantage does not transfer to held-out interventions at 50% removal from every FFN. These networks are already far from the dense model without an intervention: IOI top-1 accuracy is 0.723 for paired versus 0.910 for dense. Section 6 now discusses transfer, selection using one intervention family, the two-task scope, and the missing DIITO comparison.

(Q10) Ablations. We report these in the appendix and summarize them in the main text. Appendix Figure 3 sweeps the paired weight λ\lambda from 0 to 16 in the primary setting. As summarized at the end of Section 5.2, nearly every λ>0\lambda>0 lowers both effect MSE and KL relative to λ=0\lambda=0 on both draws. Appendix Tables 8–9 vary the sketch seed, calibration subset, and refinement step; pairing lowers effect MSE in all five variants (Section 5.6). Section 5.2 also trains the retained weights. We selected λ\lambda on diagnostic pairs using a rule fixed before test evaluation (Appendix C).

Reviewer_tMfHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a post-training pruning method for FFN channels that tries to keep not only the model’s outputs, but also how it responds to interchange interventions. Removed channels are replaced by constants and folded into the bias. On top of the usual output-matching loss, the authors add a term that penalizes the difference between the pruned model’s errors on intervened and natural inputs. Channels are selected greedily using output derivatives, one layer at a time. Experiments on GPT-2 (IOI and pronoun continuation), Pythia-410m and DeiT-Tiny show that the extra term reduces intervention-effect error compared with a same-data baseline, but its effect on ordinary fidelity is mixed.

Critical strengths
  1. The experiments are carefully controlled. The same-data baseline is well matched, the protocol is fixed in advance, and the authors report bootstrap intervals and stability checks.

  2. The authors are open about mixed and negative results, which I appreciate.

Critical weaknesses
  1. Limited applicability. The pruned model is calibrated on one task with one fixed intervention, and the main paper does not show that it still behaves like the original under any other intervention site, type or task. This matters because the paper’s stated goal is to use pruned models in place of the original for interpretability studies, which usually involve trying many interventions. I also don’t see how this saves cost in practice: building the calibration data already requires running the intervention on the dense model, and the main setting only removes about 1% of GPT-2’s parameters.

  2. Limited contribution relative to the gains. The core idea is to add the error difference to the loss. The selection procedure is built from existing pieces (constant replacement and folding as in VBP/CMR, Gauss–Newton/Taylor approximations, greedy selection and random projections). The improvements mainly show up on the quantity being optimized, and they are small in absolute terms. In all 25% single-layer settings, NRE is below 1% for both methods (Table 3, Section 6). Meanwhile, the other metrics do not improve consistently. KL rises in four of the six single-layer GPT-2 settings (Fig. 2B). In 25% whole-model IOI pruning, the paired method’s task MAE is 0.345, compared with 0.232 for the same-data baseline (Table 2). In the main IOI setting, even VBP has lower task MAE than the paired method (0.0536 vs. 0.0718, Section 5.5). In addition, there is no significant gain on the vision model (Fig. 2A).

Overall, my main concern is that the motivation does not hold up. The method is only shown to work for an intervention that is known in advance, and even in this narrow setting the gains are small and come with worse fidelity in several settings. The paper does not show a realistic case where the pruned model could reliably replace the original in an interpretability study. Since this is about the core premise rather than missing details, I don’t expect it to be resolved in the rebuttal.

Questions for authors

In practice, when would this approach make an interpretability study cheaper, given that the intervention must be known in advance and run on the dense model to build the calibration data?

3 comments
AuthorsOpenReview ↗

Response to Reviewer tMfH

Thanks for the detailed review. We've added the missing evaluations and clarified the scope of the claim.

(W1) Behavior under other interventions. We agree this was missing. We evaluated the saved models without refitting (Section 5.7, Appendix K). The 21 held-out indirect object identification (IOI) interventions cover other residual locations, random-name donors, 11 single-head output patches, and two simultaneous-patch interventions. The geometric-mean ratio of paired to same-data-control effect mean squared error (MSE) is 0.929 (95% interval 0.915–0.944; lower in 19/21) with 25% removal from one feed-forward (FFN) layer, and 0.912 (0.896–0.932; 14/21) with 25% removal from every FFN. Against variance-based pruning (VBP), the ratios are 0.634 and 0.658. With every FFN pruned at 25%, the gain comes from head and two-patch interventions; the methods are close at other residual ports (0.984). On pronoun continuation, the ratio at three other ports (0.819) is close to the calibrated one (0.811).

At 50% removal from every FFN, the advantage does not transfer: the ratio is 1.106 (1.086–1.124), with paired lower in only 3/21. We also reran the standard IOI patching study on each model. At 25% whole-model removal, head-map correlations with the dense model are 0.998 for both paired and control and 0.989 for VBP; all recover the dense model's ten most important heads. At 50%, the correlations are 0.941, 0.902, and 0.745.

(W1) Cost and the 1% figure. You're right that the single-layer settings remove only 0.95% of GPT-2 (Appendix Table 4). They isolate the objective's effect. The whole-model experiments remove 11.4% and 22.8% of all parameters (Section 5.3). We discuss measured costs below.

(W2) Contribution and size of the gains. We agree that the ingredients have precedents in causal distillation, Jacobian matching, and compensated pruning (Section 6). We don't claim priority for intervention-sensitive pruning (Section 2). Our point is that matching outputs does not ensure matching intervention responses: at matched width and data, paired reduction has lower effect error than VBP and the same-data control in all six single-layer language settings, with no additional data (Figure 2, Appendix Tables 11 and 18). The normalized response errors below 1% at 25% single-layer removal do reflect a setting where all methods stay close to the dense model. With whole-model pruning, where errors are larger, pairing lowers effect MSE in three of four settings: 18.9% and 6.0% on pronoun continuation, and 11.8% on IOI at 50%. The fourth is a tie (Section 5.3).

(W2) Other metrics. The single-layer Kullback–Leibler (KL) increases are at most 1.95×10−31.95\times10^{-3} nats, and the change in KL switches sign across refits (Section 5.6, Appendix Tables 8–9). When retained weights are trained, pairing lowers both effect MSE and KL on both draws (Table 1).

The 25% whole-model IOI task mean absolute error (MAE), 0.345 versus 0.232, is a real regression. Effect MSE weights all 50,257 centered logits equally, and the IO-minus-subject direction carries only 0.026% of paired effect-error energy. As blocks 3–6 are reduced, paired starts to overstate the intervention's effect on that contrast (Section 5.3, Appendix Figure 5). Two-candidate accuracy remains 1.000 for every method, while paired has the lowest true-target negative log-likelihood (0.669 versus 0.752 for the control). At the primary IOI setting, VBP's lower task MAE (0.0536 versus 0.0718) comes with 55% higher full-output effect MSE (0.803 versus 0.519, both ×10−3\times10^{-3}; Appendix Table 11). On DeiT-Tiny, the gain is not distinguishable from zero; we make no claim of a vision benefit (Figure 2A).

(Q) When would this make a study cheaper? The measurements suggest a study that reuses one moderately pruned model for many interventions. At 25% removal from every FFN, calibration on one intervention family (128 dense pairs) gives lower aggregate response error than the control across the 21 held-out interventions (ratio 0.912), recovers the dense model's ten most important heads, and runs the 211-pass patching sweep 1.30 times faster (13.6 s versus 17.8 s). The up-front cost is dense calibration plus selection: a whole-model run of all four methods took 608–715 s and at most 1.01 GB. Savings therefore require reuse across many sweeps, prompt sets, or donor distributions. At 50%, the sweep is 1.71 times faster, but transfer fails, so we would not use that model as a substitute. The savings at GPT-2 small scale are modest: a full forward pass gains at most 1.39 times. The revised Introduction leads with response fidelity, which remains relevant even when the cost savings are small.

Reviewer_tMfHOpenReview ↗

comments to authors' answer

Thanks for the authors' answer. They have addressed some of my concerns, but some remain.

  1. Limited applicability. I appreciated the additinoal experiments. However, they are still within the IOI tasks. I concerned about the cross-task generalisation capability.
  2. Cost saving. I appreciate your honest answers-modest speedup and depends on my factors. but it does not address my concern.
AuthorsOpenReview ↗

Cross-task generalization and cost

Thanks for the follow-up. We ran the cross-task test directly, without refitting (new Table 27 in Appendix K; the revised PDF is uploaded).

  • Across tasks, the advantage does not carry over. The pronoun-calibrated whole-model network has 1.42 times the same-data control's effect error on the 21 IOI interventions (95% interval 1.38–1.47), and the IOI-calibrated single-FFN model has 1.70 times the control's error at the pronoun ports. Paired still has lower error than VBP in most of these interventions (16 of 21 on IOI).
  • Calibrating on both tasks recovers it on both. With 64 IOI and 64 pronoun pairs (the same 128-pair budget) and the objective summed over both, paired reduction has 0.87 of the matched control's effect error on the 21 held-out IOI interventions (0.86–0.89; lower in 20 of 21) and 0.93 on the other pronoun ports (0.90–0.97; 3 of 3). On IOI, this mixed model also beats the IOI-only paired model at the calibrated port (0.78 of its effect error).

So the gain transfers across locations, donors, and heads within a task, but not across tasks. A study that uses several tasks should calibrate on all of them, which the objective supports without any change because it already sums over pairs. We've added this to Section 6.

On cost, we agree the measured speedup at GPT-2 small scale is modest. The revised paper rests its case on response fidelity: at matched width, standard pruning changes intervention responses unless the objective includes them, and including them fixes that within the calibrated tasks.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers (Part 1 of 2)

Thanks to all three reviewers for the helpful feedback. We study whether a pruned network that matches a model's outputs also preserves its responses to interchange interventions. Our method adds one term to a post-training pruning objective: a penalty on errors in the change caused by an intervention. At matched width and data, this lowers intervention-response error in all six single-layer GPT-2 language settings (Section 5.1).

The reviews pointed to three things we hadn't shown: responses to other interventions, measured cost, and whole-model task performance. We've now evaluated all three using the submission's saved compiled models, without refitting. The results are in the new Section 5.7 and Appendix K (Tables 21–26, Figures 4–5), with revisions marked in blue.

Transfer to held-out interventions (tMfH W1, ALg3 W4). Each indirect object identification (IOI) model was selected using one intervention: a role-swapped residual patch at the second subject position before block 8. We then tested 21 IOI interventions that were not used for selection: six residual patches (blocks 2, 4, 6, and 10, plus two other positions of block 8), two random-name donor sets, output patches of 11 IOI-circuit heads from Wang et al. (2023), and two pairs of simultaneous patches.

The table shows the geometric mean across interventions of paired effect mean squared error (MSE) divided by the comparator's; lower is better. Brackets give 95% bootstrap intervals for the control comparison, and parentheses count interventions where paired has lower error (Table 3a, Appendix Table 21). VBP is variance-based pruning; FFN is a feed-forward layer.

Pruned modelPaired / same-data controlPaired / VBP
IOI, one FFN, 25%0.929 [0.915, 0.944] (19/21)0.634 (21/21)
IOI, every FFN, 25%0.912 [0.896, 0.932] (14/21)0.658 (20/21)
IOI, every FFN, 50%1.106 [1.086, 1.124] (3/21)1.016 (12/21)
Pronoun, every FFN, 25% (3 other ports)0.819 [0.791, 0.846] (3/3)0.446 (3/3)

At 25% removal, the advantage carries over on average, both for one FFN and for every FFN. With every FFN pruned, the gains come from head patches (0.887) and two-patch interventions (0.799); the methods are nearly tied at the other residual ports (0.984). At 50% removal from every FFN, the advantage does not carry over. These models already differ substantially from the dense model without interventions: IOI top-1 accuracy is 0.723 for paired and 0.773 for the control, versus 0.910 for the dense model.

We also reran the standard IOI patching study (Wang et al., 2023), using each model's own clean activations in its own corrupted run, across 13 residual layers at five positions and all 144 heads (Section 5.7, Appendix Table 23). At 25% removal from every FFN, correlations with the dense head map are 0.998 for both paired and the control, and 0.989 for VBP. All three recover the dense model's ten most important heads. At 50%, correlations fall to 0.941, 0.902, and 0.745, with 9, 9, and 8 of those heads recovered.

Cost (tMfH W1 and Q, ALg3 Q2, ST11 Q6). Calibration runs the intervention on 128 dense-model pairs. Selection for the single-block study took 486 s of worker time and at most 0.60 GB of GPU memory (now in Section 4). A whole-model run of all four methods for one task and budget took 608–715 s and at most 1.01 GB. Single-layer pruning removes 0.95% of GPT-2 (Appendix Table 4); removing 25% or 50% of every FFN leaves 110.3 or 96.1 of its 124.4 million parameters.

The speedups are modest (Appendix Tables 23 and 26). The 211-pass IOI patching sweep ran 1.30 times faster at 25% removal and 1.71 times faster at 50% (13.6 s and 10.4 s, versus 17.8 s). A full forward pass with logits at every position is at most 1.21 and 1.39 times faster in FP32 at batch 128, with no gain at batch 1. We've revised the Introduction to lead with response fidelity: whether a pruned copy can stand in for the original in an intervention study. Cost remains a motivation, now supported by the measurements in Sections 4 and 5.7 and Appendix K.

Continued in a reply to this comment.

AuthorsOpenReview ↗

Response to all reviewers (Part 2 of 2)

Continued from the comment above.

Ordinary fidelity and task-level error (tMfH W2, ALg3 W1–W2, Q1, Q3). The reviewers are right about the reported increases in natural Kullback–Leibler (KL) divergence, and we've made the discussion more direct. In the frozen single-layer fits, KL rises in four of six language settings. The largest absolute increase is 1.95×10−31.95\times10^{-3} nats and the largest relative increase is 34.6% (Section 5.1).

These single-run differences vary considerably across refits. Across five perturbations of the sketch seed, calibration subset, and refinement step in the primary setting, the KL change ranges from −41.8% to +127.6%, while pairing lowers effect error in all five. The removed channel sets overlap with Jaccard index 0.50–0.54 (Section 5.6, Appendix Tables 8–9). When we train the retained feed-forward (FFN) weights on the same 128 pairs, pairing lowers both effect mean squared error (MSE), by 15.9% and 20.9%, and natural KL, by 46.4% and 12.9%, on the two draws (Section 5.2, Table 1).

At 50% removal from every FFN, the indirect object identification (IOI) natural KL increase is larger: 0.274 nats, or 40.6% (Section 5.3). This is accompanied by worse task performance. Here are the IOI task metrics on the 256 held-out pairs (Section 5.3, Appendix Table 24). NLL is true-target negative log-likelihood; VBP is variance-based pruning.

IOI, every FFNDensePairedSame-data controlVBP
25%: two-candidate accuracy1.0001.0001.0001.000
25%: full-vocabulary top-10.9100.9450.9570.918
25%: NLL0.7210.6690.7520.816
50%: two-candidate accuracy1.0000.9730.9880.910
50%: full-vocabulary top-10.9100.7230.7730.418
50%: NLL0.7211.8031.3982.712

At 25%, two-candidate accuracy stays at 1.000 for every method, and paired NLL is 0.084 nats below the control (95% interval 0.070–0.097). At 50%, paired misclassifies 7 of the 256 pairs, the control 3, and VBP 23. Paired NLL is 0.405 nats above the control (0.335–0.476).

We also looked into why, at 25% whole-model IOI pruning, paired and the control tie on effect MSE while paired has higher task-contrast mean absolute error (MAE): 0.345 versus 0.232 (Table 2). Effect MSE weights all 50,257 centered logits equally. The indirect-object-minus-subject direction accounts for only 0.026% of the paired model's effect-error energy (0.011% for the control), so it contributes very little to the objective (Section 5.3, Appendix Table 25 and Figure 5).

Paired overstates the intervention's effect on this contrast: its mean signed error is −0.264, versus −0.107 for the control, against a dense effect of −0.896. The two methods are close through block 2 (task MAE 0.187 and 0.189). The gap opens as blocks 3–6 are reduced, upstream of the patched port, reaching 0.326 versus 0.206 after block 6.

Changes in the revised PDF.

LocationChange
AbstractOne sentence on the 21 held-out interventions at 25% removal
Section 1Opening focused on response fidelity; one sentence on transfer; list introduced as "Summary and contributions."
Section 4Single-block selection cost (486 s, 0.60 GB)
Section 5.3Whole-model task performance; task-direction explanation of 0.345 versus 0.232
Section 5.7 (new)Transfer to held-out interventions (Table 3a) and the patching-study rerun
Section 6Transfer result; selection uses one intervention family; two tasks with documented mechanisms; no DIITO comparison
Section 7Conclusion updated for transfer
Appendix K (new)Tables 21–26: held-out intervention intervals, NRE under every intervention, patching agreement, whole-model task performance, task-direction decomposition, inference cost. Figures 4–5: head-patching maps, reduction-chain trajectory

NRE is normalized response error: summed squared effect error divided by dense effect energy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.