- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies structured pruning for mechanistic interpretability experiments. It proposes Paired-Response Reduction, which, in addition to errors on natural and intervened outputs, explicitly constrains the intervention-effect error between the pruned and original models. Experiments focus primarily on GPT-2 on the IOI and occupation-pronoun continuation tasks, with additional experiments on Pythia-410m and DeiT-Tiny. Across six single-layer language-model settings, the paired objective reduces intervention-effect MSE by 7.8% to 19.6%, although natural-input KL and some task metrics improve in some settings and deteriorate in others. Overall, the paper demonstrates that incorporating a specified intervention into the pruning objective can improve the fidelity of the pruned model's intervention responses, but it does not establish that the method can consistently preserve real task performance or the internal causal mechanism.
Critical strengths
- The paper addresses a practically meaningful problem. The authors appropriately distinguish ordinary predictive fidelity from intervention-response fidelity and point out that conventional pruning methods may alter a model's responses to internal interventions even when they preserve ordinary outputs. This issue is relevant to using pruned models to reduce the cost of mechanistic interpretability experiments.
- The experimental results are reasonably consistent on the core metric. Across all six single-layer GPT-2 language settings, the paired method reduces intervention-effect MSE, and NRE also improves in every setting.
Critical weaknesses
- Whole-model pruning lacks a complete evaluation of real task performance. The whole-model pruning experiments do not report IOI accuracy, full-vocabulary top-1 accuracy, or true-target NLL. It is therefore unclear whether large-scale pruning actually preserves normal task performance, making it difficult to determine how much real task performance is sacrificed to improve intervention fidelity.
- Across the six single-layer language settings, Natural KL improves in only two settings and deteriorates in the other four, with increases of more than 30% in some settings. Under 50% whole-model pruning, IOI Natural KL also worsens by 40.6%. This suggests that the paired objective may substantially sacrifice ordinary predictive fidelity.
- Experimental coverage is limited. The main conclusions are based on a single GPT-2 checkpoint and two relatively specialized language tasks; Pythia uses only one checkpoint, with the IOI experiment being exploratory; and the vision experiments show no significant improvement. The current evidence is insufficient to establish that the method generalizes to larger language models, different architectures, interventions at different levels, or more complex downstream tasks.
- The method depends on prespecified intervention types, locations, and donor distributions, but the paper evaluates it primarily under the same intervention settings used during calibration. It remains unclear whether the pruned model can generalize to unseen intervention locations, different types of donor inputs, or multiple simultaneous interventions.
Questions for authors
- Could the authors report two-candidate IOI accuracy, full-vocabulary top-1 accuracy, and true-target NLL for 25% and 50% whole-model pruning? In particular, under 50% whole-model pruning on IOI, Natural KL worsens by 40.6%. Does this change lead to a decline in actual task performance?
- Could the authors provide end-to-end inference time, GPU memory usage, and the actual runtime of intervention experiments? Is the resulting speedup sufficient to support the paper's claimed cost advantage for interpretability?
- In the IOI experiment with 25% whole-model pruning, the paired method's Effect MSE is essentially tied with that of the control, whereas Task MAE worsens from 0.232 to 0.345. How do the authors explain the substantial deterioration in the task-related error despite little change in the overall intervention-effect error?
3 comments
Thank you for the response. I still have some concerns, e.g, the mismatch between Effect MSE and task-relevant fidelity. Your explanation suggests that a task-relevant logit direction can account for only a tiny fraction of the total effect-error energy, such that similar Effect MSE values may coexist with substantially different task MAE. Does this indicate that Effect MSE alone may be insufficient as a measure of intervention fidelity, since errors in task-relevant causal directions can be diluted across the full vocabulary?
Effect MSE and task-relevant fidelity
Thanks, this is a fair point, and our answer is yes. Effect MSE averages the error over all 50,257 centered logits, so an error concentrated in one task-relevant direction can be diluted, and two models can tie on effect MSE while differing on the task contrast, as at 25% whole-model IOI. We read effect MSE as a measure of whole-output response fidelity and task-contrast error as a separate, complementary check. The paper reports both in every setting, and a study that cares about a particular contrast should check it directly.
We also tested whether weighting the task direction in the objective fixes the gap (new Table 28 in Appendix K). We added times the squared error of the intervention-induced IO-minus-subject contrast, computed exactly with one extra derivative probe per pair, and reran the 25% whole-model IOI chain. No weight closes the gap to the control's task MAE of 0.232. The best setting, , lowers task MAE from 0.337 (the rerun) to 0.290, but natural KL rises from 36.7 to 74.6 () and the held-out advantage is lost (1.14 times the control's effect error). The results also aren't monotone in : gives 0.418. With greedy constant replacement applied block by block, a single output direction is hard to control through an objective weight alone, so reporting task-level metrics next to effect MSE remains necessary. Appendix K now says this.
Response to Reviewer ALg3
Thanks for the detailed questions. We've added the evaluations you asked for and looked into the discrepancies you pointed out.
(W1, Q1) Whole-model task performance. We agree this was missing. We evaluated the saved compiled models on the 256 held-out indirect object identification (IOI) pairs (dense two-candidate accuracy 1.000; Appendix Table 13). The results are now in Section 5.3 and Appendix Table 24. NLL is true-target negative log-likelihood, VBP is variance-based pruning, and FFN is a feed-forward layer.
At 25%, paired NLL is 0.084 nats below the control (95% interval 0.070–0.097). At 50%, the 40.6% Kullback–Leibler (KL) increase does come with worse task performance: paired misclassifies 7 of 256 pairs, the control 3, and VBP 23. Paired NLL is 0.405 nats above the control (0.335–0.476).
(W2) Natural KL. You're right: KL rises in four of six single-layer settings and by 0.274 nats at 50% whole-model IOI. At 25% whole-model IOI, pairing lowers it by 31.6% (Section 5.3). The single-layer increases are at most 1.95×10−3 nats and vary across refits: over five perturbations, the frozen selector's KL change ranges from −41.8% to +127.6% (Section 5.6). With trained retained weights, pairing lowers KL on both draws, by 46.4% and 12.9% (Table 1). That does not explain away the 50% result, which shows up in both accuracy and NLL above. Section 6 identifies an explicit KL constraint as a next step.
(W3) Coverage. We agree the experiments cover a limited set of tasks, and acknowledge this in Section 6. We chose IOI and pronoun continuation because their mechanisms are documented, giving each intervention a defined meaning (Wang et al., 2023; Vig et al., 2020). Pythia-410m adds a second architecture family with parallel attention and FFN branches. Its pronoun result passes the prespecified stability check; its IOI result is exploratory (Section 5.4).
(W4) Unseen locations, donors, and simultaneous interventions. We've now tested these without refitting (Section 5.7, Appendix Tables 21–22). Entries give the geometric mean of paired / same-data-control effect mean squared error (MSE), followed by the number of interventions where paired is lower.
At 25%, the advantage carries over on aggregate. At 50% removal from every FFN, it does not. The general comment gives the intervals and patching-study results.
(Q2) Inference time, memory, and intervention runtime. We measured these directly (Section 5.7, Appendix Tables 23 and 26). Removing 25% and 50% of every FFN leaves 110.3 and 96.1 of GPT-2's 124.4 million parameters. A full forward pass with logits at every position is at most 1.21 and 1.39 times faster in FP32 (batch 128, length 20), with no gain at batch 1, where kernel launches dominate. Peak GPU memory at batch 128 and length 20 falls from 996 MiB to 941 and 887 MiB. The 211-pass IOI patching sweep, using only final-position logits, is 1.30 and 1.71 times faster (13.6 s and 10.4 s versus 17.8 s). Selection uses 128 dense calibration pairs; a whole-model run of all four methods took 608–715 s. At 25%, where transfer holds, the speedup is modest at this scale. We've revised the Introduction to lead with response fidelity and describe cost using these measurements.
(Q3) Tied effect MSE, higher task error. We looked into this discrepancy (Section 5.3, Appendix Table 25 and Figure 5). Effect MSE weights all 50,257 centered logits equally. Task mean absolute error (MAE) measures one direction: the indirect-object logit minus the subject logit. That direction carries 0.026% of paired effect-error energy and 0.011% of the control's, so equal effect MSE can hide a difference on the task contrast. Paired overstates the intervention's effect (mean signed error −0.264 versus −0.107 for the control; dense effect −0.896). The methods are close through block 2 (task MAE 0.187 and 0.189). The gap opens as blocks 3–6 are reduced, upstream of the block-8 port: after block 6, MAE is 0.326 versus 0.206.