acceptodds
Under review as a conference paper at ICLR 2027

Demystifying Variance in Circuit Discovery of Large Language Models

Abstract

Circuit discovery is a key technique in mechanistic interpretability to pinpoint the model components that are crucial for performing a given task. Although the current state-of-the-art method (EAP-IG) performs well on the metric of (un)faithfulness, it suffers from substantial variability. This includes _resampling variance_, where the circuit changes when we probe with a new batch of data from the same distribution; _phrasing variance_, where the discovered circuit shifts when the prompts are rephrased; and _sample-wise variance_, where a circuit with low population unfaithfulness exhibits large fluctuations in unfaithfulness across individual samples. This paper studies the roots of these variances. We demonstrate that conductance-based EAP (CEAP), a more mathematically principled alternative to EAP-IG, lessens resampling variance. We further show that phrasing variance arises because prompts expressed in different templates tend to activate different circuits with limited cross-template generalizability, and that weight sparsity can mitigate this limitation. Regarding sample-wise variance, we find that extremely poor unfaithfulness scores often arise from how the metric is normalized, rather than from large discrepancies between the circuit and the full model's behavior.

Primary area:
interpretability and explainable AI
Keywords:
interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_CGwwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies three sources of variability in circuit discovery: resampling the data, changing prompt templates, and differences in unfaithfulness across individual samples. It introduces CEAP, which uses conductance to score edges and improves resampling stability over EAP-IG. It also reports better cross-template circuit transfer in weight-sparse models and shows that extreme unfaithfulness can arise from a small normalization denominator.

Critical strengths

The distinction between the three sources of variability is useful. CEAP has a clear mathematical motivation, and the cross-template experiments test behavioral transfer rather than relying only on circuit overlap. The normalization analysis also provides a concrete reason to be cautious when comparing unfaithfulness across samples.

Critical weaknesses
  1. The mechanism proposed in Section 6 is not sufficiently established. The correlations support a relationship between the normalization denominator, attribution concentration, and unfaithfulness. However, I am less convinced that they establish selective contribution scaling as the underlying explanation. The conclusion that sample-wise variance is “largely benign” also seems too broad without quantifying how often high unfaithfulness still corresponds to meaningful behavioral errors.

  2. The sparsity results need an additional check. Increased circuit overlap provides supporting evidence, but the behavioral transfer comparison in Section 5.2 uses normalized unfaithfulness across models with different sparsity levels. Given the findings in Section 6, reporting unnormalized discrepancies and full-model task performance would help establish the magnitude and practical significance of this improvement.

Questions for authors
  1. Clarification: What evidence supports interpreting selective contribution scaling as an underlying mechanism rather than a description of the correlations in Figure 6? In Section 6.2, could you distinguish what follows from partition consistency from what remains an empirical hypothesis?

  2. Requested analysis: Beyond the examples and correlations in Section 6, could you quantify how often very high unfaithfulness coincides with a large unnormalized discrepancy or a changed task prediction? This would help determine how broadly the conclusion that sample-wise variance is “largely benign” applies.

  3. Requested comparison: Does the cross-template improvement in Section 5.2 remain when evaluated using unnormalized discrepancies? Please also report how full-model task performance and the magnitude of the normalization denominator vary with sparsity.

  4. Resampling robustness: Section 4 uses four resampled datasets. Do the stability gains persist with additional independent resamples? An uncertainty estimate that accounts for the shared circuits in the six pairwise comparisons would help assess the reliability of these gains.

4 comments
AuthorsOpenReview ↗

rebuttal 1

Thank you for your careful review. We appreciate that you found our contribution useful and clear. You raised a few concerns, which we will address below.

Materials relevant to our responses below are marked in the manuscript with your reviewer ID, CGww, in blue. Newly added text appears in green.

W1 & Q1: Evidence and Limitations of Selective Contribution Scaling

(W1) The mechanism proposed in Section 6 is not sufficiently established. The correlations support a relationship between the normalization denominator, attribution concentration, and unfaithfulness. However, I am less convinced that they establish selective contribution scaling as the underlying explanation. (Q1) Clarification: What evidence supports interpreting selective contribution scaling as an underlying mechanism rather than a description of the correlations in Figure 6?

We agree that selective contribution scaling should be presented as a tentative explanation rather than an established mechanism. We have revised the reasoning in Section 6.2 and explicitly call it a "possible explanation" in the main text (lines 480–481).

Selective contribution scaling refers to preferential growth of leading absolute scores relative to lower-ranked scores as ∣QG∣|Q_G| increases. We have now conducted experiments to examine this directly. We compare samples from the same template with ∣QGn1∣>∣QGn2∣|Q_G^{n_1}|>|Q_G^{n_2}|, independently rank their absolute scores, and compute the componentwise ratios rk=skn1/skn2r_k = s_k^{n_1}/s_k^{n_2}. If leading scores grow proportionally more than lower-ranked scores, these ratios should tend to decrease with rank kk. We measure this trend using Spearman rank correlation (SRC).

We restrict the analysis to the top 1000 scores to avoid the extreme tail, where dividing by tiny scores can amplify numerical errors. Within this range, the mean SRC is significantly negative in all six model–task setups, supporting this preferential growth (Appendix AC.2, Table 14, lines 3112–3149). Thus, selective contribution scaling remains a correlation-based explanation, not an established causal mechanism.

(Q1) In Section 6.2, could you distinguish what follows from partition consistency from what remains an empirical hypothesis?

Both selective contribution scaling and the sparse coding interpretation are empirical hypotheses.

Selective contribution scaling: The limitations of the above SRC-based evidence are discussed in Appendix AC.4 (lines 3327–3337). We prove that if rkr_k is non-increasing across the full score curve, then μn1≤μn2\mu^{n_1}\leq\mu^{n_2}, matching the direction of the first arrow in Equation (7) (Proposition AC.1, lines 3089–3096). However, a negative SRC is weaker than monotonicity and does not guarantee this ordering. Moreover, our analysis includes only the top 1000 scores, so the observed trend may not extend to the full curve.

Sparse coding: The connection to sparse coding is also a hypothesis. To support it, we examine edge-rank fluctuation (displacement) across samples and find that leading edges generally have smaller rank fluctuations than lower-ranked edges (Appendix AC.3, lines 3153–3166). Together with the preferential scaling at leading ranks, this motivates a tentative sparse-coding interpretation: differences in ∣QG∣|Q_G| may be attributed primarily to changes in the contributions of a relatively small, recurring set of edges.

In summary, selective contribution scaling is an empirical hypothesis supported by SRC, and sparse coding is a possible interpretation of this phenomenon. Neither follows from partition consistency. They provide a mental image for understanding the first arrow in Equation (7), rather than an established mechanism. We have also removed the discussion of partition consistency from Section 6.2, as its discussion is no longer necessary.

Thank you for this question. It prompted us to refine our reasoning and conduct the SRC and rank-displacement experiments, enriching the discussion.

W1 & Q2: Extreme Unfaithfulness and Unnormalized Discrepancies

(W1) The conclusion that sample-wise variance is “largely benign” also seems too broad without quantifying how often high unfaithfulness still corresponds to meaningful behavioral errors.

(Q2) Requested analysis: Beyond the examples and correlations in Section 6, could you quantify how often very high unfaithfulness coincides with a large unnormalized discrepancy or a changed task prediction? This would help determine how broadly the conclusion that sample-wise variance is “largely benign” applies.

We agree that "largely benign" was too broad and hard to quantify. We have replaced it in the conclusion with: "Sample-wise variance in unfaithfulness need not reflect comparable variation in absolute error, because normalization can amplify differences across samples." (Section 7, lines 516–517.)

AuthorsOpenReview ↗

rebuttal 2

We now measure how often samples with extreme unfaithfulness UU also have near-maximum unnormalized unfaithfulness U′U', compared with other samples for the same model, template, and circuit size. Here, near-maximum means at least 90% of the largest U′U' in that group. Full experimental details and results are provided in Appendix Y, Table 13 (lines 2579–2627).

Pooling together all the setups, only 22.4% (84/375) of cases with U>1U>1 and 26.7% (24/90) of cases with U>2U>2 have near-maximum U′U'. Notably, none of the 17 cases with U>5U>5 has near-maximum U′U'.

These results support the normalization-based explanation in the main text (lines 449–452): extreme unfaithfulness values need not indicate correspondingly extreme absolute discrepancies.

W2 & Q3: Weight Sparsity and the Effect of Normalization

(W2) The sparsity results need an additional check. Increased circuit overlap provides supporting evidence, but the behavioral transfer comparison in Section 5.2 uses normalized unfaithfulness across models with different sparsity levels. Given the findings in Section 6, reporting unnormalized discrepancies and full-model task performance would help establish the magnitude and practical significance of this improvement.

(Q3) Requested comparison: Does the cross-template improvement in Section 5.2 remain when evaluated using unnormalized discrepancies? Please also report how full-model task performance and the magnitude of the normalization denominator vary with sparsity.

We report the normalization denominator ∣QG∣|Q_G| in unfaithfulness (Appendix R, Figure 38a, lines 2325–2335). The medians and central percentile bands remain relatively stable and well away from zero even at lower sparsity (more nonzero parameters). This suggests that the larger cross-template deterioration in denser models is not primarily due to near-zero denominators inflating unfaithfulness.

We separately report task-choice accuracy as task performance: the fraction of clean prompts where the correct answer token or set receives greater probability mass than the competing token or set (Appendix R, Figure 38b, lines 2336–2340). On the same 1,000 samples per task, this accuracy is 100% for single-double-quote and else-elif at every sparsity level. For with-open, it ranges from 72.0% to 80.5%, without a monotonic trend across sparsity.

We also repeat the cross-circuit evaluation without normalization (Appendix S, Figure 39, lines 2379–2393). The unnormalized cross-template deterioration ΔUm′(n)\Delta U'_m(n) decreases with increasing weight sparsity, following the same overall trend as its normalized counterpart in Figure 3a. This provides a qualitative check that the observed generalizability improvement is not merely a normalization artifact.

Q4: Resampling Robustness and Uncertainty

Resampling robustness: Section 4 uses four resampled datasets. Do the stability gains persist with additional independent resamples? An uncertainty estimate that accounts for the shared circuits in the six pairwise comparisons would help assess the reliability of these gains.

Yes. We have expanded the experiment from four to twelve independently resampled datasets for every model–task setup. Instead of comparing all seed pairs, we now use six disjoint pairs: (0,1),(2,3),…,(10,11)(0,1),(2,3),\ldots,(10,11). Each circuit appears in only one pair, removing the shared-circuit dependence raised in your question (Appendix I, lines 1299–1309).

For the uncertainty estimate, we average the PJI differences over circuit sizes, templates, and model–task setups separately for each seed pair. We then compute a 95% confidence interval and perform a two-sided one-sample tt-test using these six independent averages. We give each model–task setup equal weight so that tasks with more templates do not dominate the result.

CEAP's mean PJI improvement over EAP-IG is 2.05×10−22.05\times10^{-2}, with a 95% confidence interval of [1.86,2.24]×10−2[1.86,2.24]\times10^{-2} and a Holm-adjusted p=1.35×10−6p=1.35\times10^{-6}. The improvement is also significant when each model–task–template combination receives equal weight. Thus, the average stability gain persists with additional independent resamples. Appendix I, Table 7 (lines 1350–1380), provides both analyses and corresponding results for unfaithfulness and RelP.


Thank you again for your thoughtful feedback. Your questions prompted us to clarify the limits of our interpretation and strengthen the empirical evidence with additional experiments and uncertainty estimates. We hope these revisions address your concerns. If so, we would be grateful if you would consider increasing your score. We would be happy to address any remaining questions.

Thank you for the detailed analysis and experiments. I find the additional experiments solve most of my concerns. And I think this paper propose some interesting improvement of some existing theoretical tools and apply them to circuit discovery. Since I have given a score of 3 and I will keep my score of weak acceptance.

Thank you so much for your support! We are especially grateful that you challenged our selective contribution scaling story. Your questions pushed us to think it through more carefully and develop a much clearer picture.

Reviewer_sDqGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new way to identify circuits in models (CEAP). This method is based on conductance where the importance of an edge is measured by jointly considering the effects of perturbing the input on the activation and also the activation on the output. Then, the paper describes several sources of variance in circuit discovery including variation across templates; a way to reduce variance by increasing weight sparsity; variance across samples.

Critical strengths

I think the findings in the paper are valuable to the mechanistic interpretability community and also point towards a series of benchmarks or metrics where one could develop and test new circuit discovery methods.

Critical weaknesses

The paper seems scattered and incomplete. It's unclear to me what the main contribution of the paper is or what the takeaway is to the reader. It seems that many of these findings have already been explored throughout the literature and this paper simply is acting as a compilation. It also feels like a lot of the variance analyses are conducted on very few circuit discovery methods and datasets. I doubt whether these findings would generalize. Moreover, the proposed method of CEAP is introduced, but then very sparsely evaluated and then not mentioned until the conclusion.

Questions for authors
  • I think I understand the point of Figure 2, but what does this plot look like when you make the colorscale actually scale linearly? (I think the current scaling is slightly misleading). Also, given that you estimated the circuit variable across samples even within same template was already large wouldn't it make more sense to instead plot statistical significance (i.e. p-value) in each of these boxes?
  • To what extent is this problem solved by simply doing a best-of-N over the templates. This seems like one obvious thing to examine along side of CEAP (WB26)
  • In Section 5.2, what is the intuition of why weight sparsity increases circuit generalization? Is it simply that there are fewer plausible circuits that the model could find?
4 comments
AuthorsOpenReview ↗

rebuttal 1

Thank you for your careful review and for recognizing the value of our findings to the mechanistic interpretability community. We address your concerns and questions below.

Materials relevant to our responses are marked in the manuscript with your reviewer ID, sDqG, in purple. Newly added text appears in green.

W1

The paper seems scattered and incomplete

We appreciate this feedback. The breadth is deliberate: circuit variability manifests in data resampling, template dependence, and even sample-wise metric readings. We provide a self-contained study of these complementary aspects, investigating how they can be understood and, where possible, mitigated.

W2

It's unclear to me what the main contribution of the paper is or what the takeaway is to the reader.

Main contribution: We explain sources of variability in gradient-based circuit discovery and develop ways to mitigate it when warranted.

Takeaways:

  1. CEAP yields more stable circuits under data resampling than the current mainstream method EAP-IG, while also improving unfaithfulness.

  2. Circuits discovered for one template may not generalize to other templates of the same task. But this limitation can be mitigated: increasing model weight sparsity improves generalization across templates of the same task.

  3. Extreme sample-wise unfaithfulness readings need not indicate large discrepancies between circuit and full-model behavior, as its normalization can amplify otherwise small errors.

W3

It seems that many of these findings have already been explored throughout the literature and this paper simply is acting as a compilation.

We build on prior observations of variance, but contribute new explanations and ways to mitigate it rather than merely compiling these observations. We see this as a strength: our work provides new explanations and methodological improvements in areas that prior work has established as important.

Component attribution and resampling variance. Hanna et al. [1] introduced EAP-IG, now a mainstream gradient-based method for LLM circuit discovery, and Méloux et al. [2] documented its instability under data resampling. IG was originally proposed for input attribution [4]. Later, conductance was adapted from IG for internal-component attribution [5], making itself a natural fit for scoring components in circuit discovery. Yet EAP-IG uses IG to score internal edges, leaving out the formulation specifically developed for internal attribution. We introduce CEAP, to our knowledge the first conductance-based method for edge-level LLM circuit discovery, and systematically compare the two attribution formulations.

Theoretically, we show that conductance recovers ground-truth component importance in the sum-of-branches setup, whereas IG does not guarantee the correct ordering (Theorem 3.4, line 187). We further show that CEAP can eliminate resampling variance while EAP-IG may remain unstable (Proposition H.2, lines 1217-1222). Empirically, compared with EAP-IG, CEAP achieves statistically significant improvements in Pairwise Jaccard Index (PJI) and unfaithfulness (Appendix I, lines 1350-1380). CEAP's higher PJI indicates greater agreement across data resampling, mitigating the resampling instability reported by Méloux et al. [2].

Template dependence. Méloux et al. [2] observed circuit differences under rephrasing. We show this template dependence is accompanied by poor cross-template circuit generalizability, even within the same task. We further show that increasing weight sparsity improves cross-template generalizability, revealing a previously unrecognized benefit of weight-sparse transformers [6] (Section 5.2, lines 344-354).

Sample-wise unfaithfulness. Miller et al. [3] highlighted large variation in faithfulness across samples, raising concerns about circuit reliability. Our findings convey a reassuring message: extreme unfaithfulness often arises from the normalization in its formula rather than large discrepancies between circuit and full-model behavior (Section 6, lines 408-412).

AuthorsOpenReview ↗

rebuttal 2

W4

It also feels like a lot of the variance analyses are conducted on very few circuit discovery methods and datasets. I doubt whether these findings would generalize.

Circuit discovery methods: The original submission focused primarily on CEAP, with a direct comparison against EAP-IG in Section 4. We have expanded our evaluation to additional methods. CEAP achieves statistically significant average improvements in PJI and unfaithfulness over both EAP-IG [1] and RelP [7] (Appendix I, lines 1350-1380). To complement the CEAP-based evidence of template dependence (Section 5.1, lines 286-292), we add an activation-patching case study showing that changing the template changes which heads matter (Appendix P, lines 2238-2258). Our weight-sparsity experiments now also cover both CEAP (Section 5.2, lines 344-354) and EAP-IG (Appendix U, lines 2454-2460), which show the same overall trend. We also examine the correlations related to samplewise variance with EAP-IG (Appendix AA, lines 2846-2848, and Appendix AB, lines 2891-2893), and find broadly consistent correlation patterns with the CEAP experiments (Appendix Z, lines 2632-2640). Finally, we clarify our scope in Section 2 (lines 80-84): we study circuit discovery in the model's original computational graph.

Datasets: We study six datasets spanning linguistic, numerical, and coding tasks. We use subject-verb agreement (SVA), indirect object identification (IOI), and greater-than to compare CEAP, EAP-IG, and RelP, and to investigate template dependence and sample-wise variance. For weight-sparse transformers, which specialize in coding, we construct three additional datasets: single-double-quote, else-elif, and with-open. Appendix E (lines 896-903 and Tables 1-6) details their construction and templates.

Overall, our observations generalize well across different circuit discovery methods and datasets.

W5

Moreover, the proposed method of CEAP is introduced, but then very sparsely evaluated and then not mentioned until the conclusion.

The original submission compared CEAP with EAP-IG across four models, three tasks, and four dataset resamples, evaluating both PJI and unfaithfulness.

We have now increased the number of independent resamples from four to 12 and added comparisons with RelP. CEAP achieves statistically significant average improvements in both PJI and unfaithfulness over both methods (Appendix I, lines 1299-1309 and 1350-1380).

The original submission also used CEAP for the later empirical analyses: template dependence (Section 5.1, lines 255-259), weight sparsity and cross-template generalizability (Section 5.2, lines 321-323), and sample-wise unfaithfulness variance (Section 6, lines 403-407). Thus, CEAP is used throughout the empirical study, not only in the initial method comparison with EAP-IG.

Q1

I think I understand the point of Figure 2, but what does this plot look like when you make the colorscale actually scale linearly? (I think the current scaling is slightly misleading).

We provide the same heatmaps with linear color scales in Appendix T (Figure 40, lines 2432-2448). The plots of Kendall's τb\tau_b, PJI, and ΔUm(n)\Delta U_m(n) still show clear blocks aligned with template boundaries. The observed template dependence does not rely on the colorscale. We have also statistically tested this template dependence, as discussed in our responses below.

Also, given that you estimated the circuit variable across samples even within same template was already large wouldn't it make more sense to instead plot statistical significance (i.e. p-value) in each of these boxes?

The heatmap values are pairwise measurements of similarity or cross-circuit deterioration, not hypothesis-test results. They do not, by themselves, have associated pp-values. We therefore directly test whether these quantities depend on template identity despite within-template variation. In Appendix O (lines 2125-2161, Tables 9-11), we use template-label permutation tests to assess whether low Kendall's τb\tau_b, low PJI, and high ΔUm(n)\Delta U_m(n) occur disproportionately often between samples from different templates.

All 27 tests across nine model-task setups are statistically significant, with Holm-adjusted p=0.0027p=0.0027 in every case. These results support systematic template dependence, even though circuits can also vary within the same template.

These tests provide formal statistical support for our claim that circuits are template-dependent. Thank you for this question.

AuthorsOpenReview ↗

rebuttal 3

Q2

To what extent is this problem solved by simply doing a best-of-N over the templates. This seems like one obvious thing to examine along side of CEAP (WB26)

We have now examined this (details in Appendix M, Figure 30, lines 2007-2048). We use 1,000 samples per setup across GPT-2 small and Pythia-160M on SVA, IOI, and greater-than. We apply the Best-of-NN strategy [8] over the templates.

Pooling cross-template pairs across these six model-task setups, Best-of-10 increases PJI for 62.2% of pairs. However, it reduces cross-circuit unfaithfulness deterioration ΔUm(n)\Delta U_m(n) for only 42.9% of pairs and increases it for the rest 57.1%. Thus, in these experiments, BoN improves circuit similarity for a majority of pairs but does not consistently reduce cross-circuit deterioration. Selecting a circuit that better fits one sample does not necessarily make it generalize better across templates.

Q3

In Section 5.2, what is the intuition of why weight sparsity increases circuit generalization? Is it simply that there are fewer plausible circuits that the model could find?

As noted at the beginning of Section 5.2 (lines 317-320), Gao et al. [6] found that weight sparsity produces smaller task circuits. We interpret this as weight sparsity putting pressure on the model to perform each task with a smaller circuit. Informally, a task circuit can be viewed as the union of circuits corresponding to different templates of that task. This motivates our hypothesis that the pressure exerted by weight sparsity to shrink the task circuit encourages greater overlap between its template-specific circuits, allowing their union to become smaller through shared components. We empirically observe that circuit overlap across templates of the same task increases with weight sparsity (Figure 3c, lines 324-341), supporting this hypothesis. This greater sharing of circuitry across templates potentially explains the improved cross-template generalizability.

The precise mechanism, however, is nontrivial. Our computational graph uses attention heads and MLPs as nodes (Figure 8, lines 764-768). Gao et al. impose the same fraction of nonzero parameters in each weight matrix, rather than directly removing whole heads or MLPs. Thus, weight sparsity is not equivalent to simply deleting nodes or edges in our computational graph.

One plausible explanation is that sparsity acts as a regularizer, constraining the network's capacity. With fewer nonzero parameters, the network may learn more parsimonious representations and reuse circuitry across templates instead of implementing separate template-specific solutions, leading to better cross-template generalizability.


Thank you again for your thoughtful feedback. Your suggestions have helped us broaden our investigation to additional circuit discovery methods and strengthen the empirical support for our conclusions. We hope these additions and clarifications address your concerns. If so, we would be grateful if you could reconsider your score. We would be happy to address any remaining questions.

References

[1] Hanna et al., Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. COLM 2024.

[2] Méloux et al., Mechanistic Interpretability as Statistical Estimation: A Variance Analysis. ICML 2026.

[3] Miller et al., Transformer Circuit Evaluation Metrics Are Not Robust. COLM 2024.

[4] Sundararajan et al., Axiomatic Attribution for Deep Networks. ICML 2017.

[5] Dhamdhere et al., How Important Is a Neuron? ICLR 2019.

[6] Gao et al., Weight-sparse Transformers Have Interpretable Circuits. ICML 2026.

[7] Jafari et al., RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patching. NeurIPS Mechanistic Interpretability Workshop 2025.

[8] Wu and Barez, Query Circuits: Explaining How Language Models Answer User Prompts. ICML 2026.

AuthorsOpenReview ↗

Follow-up on our rebuttal

Dear reviewer sDqG,

Thank you again for your time and feedback. As the discussion period ends in less than a day, we would greatly appreciate it if you could let us know whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.

Best regards,

The Authors

Reviewer_BWy8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at circuit discovery in LLMs, specifically EAP. They trace and explain sources of variance in circuits, find that all sources of variance have a big impact on what circuits EAP found and they propose conductance-based EAP, which seems to be a slightly more faithful incremental improvement on EAP.

Critical strengths

The paper provides a rigorous analysis of the different reasons for why EAP may find different circuits on the same tasks. I like that the paper tries to find and explain the root causes for problems rather than just suggesting a new method that beats benchmarks.

Critical weaknesses
  • Throughout the manuscript, the authors talk about "circuit discovery" but it seems their experiments and results are specifically for attention head / mlp component graphs discovered with EAP-IG. However, there are lots of other common approaches like ACDC, neuron or SAE feature-based circuits, Crosslayer Transcoder, MOLT, or other feature-based circuits. Can the authors either add more experiments showing that their results are broadly applicable or restrict their language to the special method used? In general, understanding how broadly applicable the author's findings are is really important to judge the impact/contribution of the paper.
    • The writing is not amazing. It's readable and doesn't contain too much AI slop. It could be improved, it often isn't precise enough, lacks a natural language description and intuition about the concepts introduced.
    • Some of the math could be more precise and improved, for example in the Background section: "the set of (competing) tokens that x1 encourages" is too imprecise; why do C and W have different letters, what's the probability of a set (ie P(C(x))), how is x different from x1 or x2 and why do we need 3 prompts when we only use clean and corrupted. If the terms "behavior" and "faithfulness" measure something specific, like a logit difference or similarly concrete things, they could be replaced by more concrete terms. The graph and faithfulness definitions would benefit from a natural language explanation.
    • Section 3 Conductance is especially hard to parse. It mixes related work with definitions, adds more jargon and definitions than necessary, and lacks a natural language description, motivation, and precision.
    • Section 4 ○ Usage of the Jaccard index as the only measure of "stability" is not well-motivated. It only measures how many components are in both circuits. ○ It's remains completely unclear to me how to interpret the results and how bad of an issue resampling actually is. If we look at those circuits, will we conclude a different high-level explanation? If we only strip away the difference, how much does behavior change? I.e. it could be the case that the components that aren't shared and that drive the Jaccard index low are relatively unimportant and close to noise. ○ It also remains unclear how big of an improvement CEAP is. There are no stats or proper explanation of errorbands, some of them seem to overlap, and overall the difference doesn't seem all too big to make a meaningful contribution. ○ I'd love to see concrete circuits, try to interpret them, and see if we'd draw different conclusions. Most of these weaknesses apply to section 5 and 6 too: The authors compute some metrics but don't investigate how their findings translate to actual interpretations and analyses of circuits.
Questions for authors
  • Why was the Jaccard index chosen as the sole metric to measure "stability"? Did you consider other metrics? Why should the Jaccard index be meaningful here?
    • Did you do any qualitative investigations into the circuits found? Ultimately, we are interested in the interpretation of the circuits, not solely quantitative measurements of circuit overlap. Did you find examples where the (high-level, interpretable, natural language) conclusion of how the model computes a specific behavior was different? What's the error band in Figure 1? In some plots, the error bands overlap, does this mean we conclude no significant difference here?
5 comments
AuthorsOpenReview ↗

rebuttal 1

Thank you very much for your careful review and constructive suggestions. We appreciate your recognition of our rigorous approach to the problem. We address your concerns and questions below.

Materials relevant to our responses below are marked in the manuscript with your reviewer ID, BWy8, in red. Newly added text appears in green.

W1

Throughout the manuscript, the authors talk about "circuit discovery" but it seems their experiments and results are specifically for attention head / mlp component graphs discovered with EAP-IG. However, there are lots of other common approaches like ACDC, neuron or SAE feature-based circuits, Crosslayer Transcoder, MOLT, or other feature-based circuits. Can the authors either add more experiments showing that their results are broadly applicable or restrict their language to the special method used? In general, understanding how broadly applicable the author's findings are is really important to judge the impact/contribution of the paper.

We intend our paper to focus on computational graphs whose internal nodes are attention heads and MLPs, consistent with the prior work we build on [1-3]. We now explicitly state this scope at the beginning of Section 2 (lines 80–84):

"We represent a transformer as a computational graph whose internal nodes are multilayer perceptrons and individual attention heads. Residual-stream addition is decomposed into directed edges, each carrying a source node's output to a downstream node (Figure 8). In this paper, we study circuit discovery methods that identify a subgraph (circuit) of the model's full computational graph that best reproduces the full model's behavior."

We also add the following discussion of future work in our final section (lines 534–537):

"Lastly, the scope of this manuscript is limited to circuit discovery in the model's original computational graph (Figure 8). It would be meaningful to explore whether our insights carry over to other circuit-discovery approaches, including circuits based on sparse autoencoders [4], transcoders [5], cross-layer transcoders [6], and sparse mixtures of linear transforms (MOLT) [7]."

We now also compare CEAP with RelP [8], a recently proposed circuit discovery method operating on the same computational graph granularity. RelP refines gradient-based attribution using heuristic backpropagation rules, rather than using integrated gradients. The comparisons in Appendix I (Table 7, lines 1363–1377) show that CEAP achieves statistically significant improvements over RelP in both circuit stability, measured by PJI, and unfaithfulness.

W2

The writing is not amazing. It's readable and doesn't contain too much AI slop. It could be improved, it often isn't precise enough, lacks a natural language description and intuition about the concepts introduced.

We address these concerns in our responses to your specific suggestions below.

W3

Some of the math could be more precise and improved, for example in the Background section: "the set of (competing) tokens that x1 encourages" is too imprecise; why do C and W have different letters,

Thanks for pointing this out. We now revise that sentence to (lines 95–98):

"C(x1)C(x_1) is the set of correct answer tokens for the clean input, while W(x0)W(x_0) is the set of competing answer tokens, which are wrong for the clean input x1x_1 but correct for the corrupted input x0x_0. In our SVA example, C(x1)C(x_1) is the set of singular verbs and W(x0)W(x_0) is the set of plural verbs."

CC and WW stand for Correct and Wrong, respectively.

what's the probability of a set (ie P(C(x)))

We now write, in lines 98–100:

"P(C(x1)∣x)\mathbb{P}(C(x_1)\mid x) is the sum of the probabilities that the model assigns to the tokens in C(x1)C(x_1) as the next token, given input xx. P(W(x0)∣x)\mathbb{P}(W(x_0)\mid x) is defined analogously for the competing answer tokens in W(x0)W(x_0)."

how is x different from x1 or x2 and why do we need 3 prompts when we only use clean and corrupted.

By x1x_1 and x2x_2, we take it that you meant x0x_0 and x1x_1.

x0x_0 and x1x_1 are the corrupted and clean prompts, which determine the token sets W(x0)W(x_0) and C(x1)C(x_1), respectively. These sets remain fixed, but their probabilities can be evaluated for any model input xx, as clarified in our previous answer. Thus, xx denotes the input at which we evaluate the metric, not a third reference prompt. During circuit discovery, we evaluate M(x;x0,x1)=P(C(x1)∣x)−P(W(x0)∣x)M(x;x_0,x_1)= \mathbb{P}(C(x_1)\mid x) - \mathbb{P}(W(x_0)\mid x) at xx's interpolated between x0x_0 and x1x_1 in the embedding space.

If the terms "behavior" and "faithfulness" measure something specific, like a logit difference or similarly concrete things, they could be replaced by more concrete terms.

We now explicitly define "behavior" (Equation (1), lines 103–106):

"We define the behavior of the full model and the selected circuit to be QΓ(x0,x1)≔MΓ(x1)−MΓ(x0)Q_\Gamma(x_0,x_1)\coloneqq M_\Gamma(x_1)-M_\Gamma(x_0), where Γ∈{G,G′}\Gamma\in\{G,G'\}."

AuthorsOpenReview ↗

rebuttal 2

We also give a natural language interpretation for "behavior" (lines 106–107):

"QΓ(x0,x1)Q_\Gamma(x_0,x_1) quantifies how much the metric prob diff changes as the input changes from the corrupted input x0x_0 to the clean input x1x_1."

We have defined faithfulness as Φ(G′,G;x0,x1)=QG′(x0,x1)QG(x0,x1)\Phi(G',G;x_0,x_1)=\frac{Q_{G'}(x_0,x_1)}{Q_G(x_0,x_1)} (lines 112–113), where GG is the full model's computational graph and G′G' is the selected circuit (lines 102–103). We will further clarify this concept with our natural language description of unfaithfulness in the next question.

The graph and faithfulness definitions would benefit from a natural language explanation.

As noted in one of our answers above, we have added a natural language explanation of the computational graph (lines 80–84). Figure 8 also illustrates this graph (Appendix A, lines 764–768).

As our paper primarily discusses unfaithfulness, we provide a natural language explanation for unfaithfulness. We first expand its definition in Equation (2) (lines 114–117):

=\frac{\lvert Q_{G'}(x_0,x_1)-Q_G(x_0,x_1)\rvert}{\lvert Q_G(x_0,x_1)\rvert}.$$ We explain this as follows (lines 118–120): *"The last equality in Equation (2) gives a straightforward interpretation: unfaithfulness measures the relative absolute deviation of the circuit's behavior from the full model's behavior."* ## W4 > Section 3 Conductance is especially hard to parse. It mixes related work with definitions, adds more jargon and definitions than necessary, and lacks a natural language description, motivation, and precision. The conductance formula can be difficult to interpret on its own. We have moved the intuition originally presented in Appendix D to immediately after the definitions of conductance and IG in Section 3 (lines 150–167). The revised passage provides a natural-language description of both scores as weighted sums of gradients, with explicit formulas specifying their different weights. It explains how IG can mismatch gradients with activation changes, motivating our comparison with conductance. Please refer to this revised passage for the full explanation. We have retained the references needed to contextualize our contribution, together with the definitions needed to state the theoretical result precisely. We hope that presenting the intuition earlier makes Section 3 easier to follow. ## W5 & Q1 Beyond PJI > (W5) Section 4 ○ Usage of the Jaccard index as the only measure of "stability" is not well-motivated. It only measures how many components are in both circuits. ○ It's remains completely unclear to me how to interpret the results and how bad of an issue resampling actually is. If we look at those circuits, will we conclude a different high-level explanation? If we only strip away the difference, how much does behavior change? I.e. it could be the case that the components that aren't shared and that drive the Jaccard index low are relatively unimportant and close to noise. > > (Q1) Why was the Jaccard index chosen as the sole metric to measure "stability"? Did you consider other metrics? Why should the Jaccard index be meaningful here? We use PJI following Méloux et al. [2], who studied the statistical properties of circuit discovery under data resampling. We aim to address the instability they identified. We agree that overlap alone cannot establish whether the differing edges matter for model behavior. To assess whether the selected circuits capture components important for reproducing the full model's behavior, we also compare their unfaithfulness. Figure 9(b) in Appendix I (lines 1326–1331) shows the distribution of $\Delta U$, defined as CEAP's unfaithfulness minus EAP-IG's, averaged over swept circuit sizes (lines 1305–1309). **CEAP reduces unfaithfulness in 72.6% of model-task-template-seed combinations. Giving each model–task setup equal weight, the one-sample Student's $t$-test in Table 7 (lines 1363–1377) shows a statistically significant mean unfaithfulness reduction of $3.11\times10^{-2}$ compared with EAP-IG, with a 95% confidence interval of $[2.95,3.28]\times10^{-2}$.** Thus, CEAP's improved stability accompanies better recovery of the full model's behavior. These results indicate that the differences between the circuits selected by CEAP and EAP-IG matter for behavioral recovery: CEAP's circuits tend to reproduce the full model's behavior better.
AuthorsOpenReview ↗

rebuttal 3

W5, W6 & Q2: Qualitative Mechanistic Investigation

(Q2) Did you do any qualitative investigations into the circuits found? Ultimately, we are interested in the interpretation of the circuits, not solely quantitative measurements of circuit overlap. Did you find examples where the (high-level, interpretable, natural language) conclusion of how the model computes a specific behavior was different?

(W5) I'd love to see concrete circuits, try to interpret them, and see if we'd draw different conclusions.

For the comparison of CEAP and EAP-IG (Sections 3 and 4), we have added a mechanistic case study in Appendix K (lines 1754–1759). Wang et al. [9] identified an IOI circuit containing 26 attention heads in GPT-2 small. We therefore use CEAP and EAP-IG to select the same number of heads for further interpretation. On one IOI template, repeating this selection across 12 independently resampled datasets, we find that CEAP selects head 9.6 every time, whereas EAP-IG never selects it.

Attention analysis and activation-patching tests on held-out prompts show that this head's behavior aligns well with the name-mover heads identified by Wang et al. [9] (lines 1761–1801, Figure 23). This illustrates how choosing CEAP or EAP-IG can affect which functional heads are included in the resulting interpretation.

In the same setup, we find no head showing the reverse pattern: selected by EAP-IG and omitted by CEAP across all 12 resampled datasets (lines 1808–1810).

(W6) Most of these weaknesses apply to section 5 and 6 too: The authors compute some metrics but don't investigate how their findings translate to actual interpretations and analyses of circuits.

To supplement our discussion in Section 5, we have added a mechanistic case study on GPT-2 small in Appendix P (lines 2216–2258, Figure 34) showing that ABBA and BABA prompts may rely on different circuits despite expressing the same task. This provides a concrete example to demonstrate that circuits are template-dependent, substantiating our claims in Section 5. Following Wang et al.'s [9] methodology, we identify head 0.10 as a fuzzy duplicate-token head and head 6.9 as an induction head. Patching tests show that, for the studied template, ABBA relies on head 0.10 but is barely affected by the patching of head 6.9. BABA exhibits the opposite pattern. When circuit components are selected based on their activation-patching effects, this case shows that even swapping two names can change which heads are retained in the circuit.

Section 6 addresses a quantitative question about the evaluation metric itself: what causes sample-wise unfaithfulness to take extreme values? We show that these extremes are largely driven by small normalization denominators, which amplify discrepancies between the circuit and the full model (lines 408–412). Interpreting individual components would not directly explain this effect of normalization.

W5 & Q2: Significance of CEAP Improvement

(W5) It also remains unclear how big of an improvement CEAP is. There are no stats or proper explanation of errorbands, some of them seem to overlap, and overall the difference doesn't seem all too big to make a meaningful contribution.

(Q2) What's the error band in Figure 1? In some plots, the error bands overlap, does this mean we conclude no significant difference here?

The error bands in the (previous) Figure 1 show standard deviations across seed pairs, not confidence intervals. We clarify this for the updated curves in Appendix J (lines 1386–1390). Their overlap does not imply that the difference is statistically insignificant. We now perform a two-sided one-sample Student's tt-test on CEAP-minus-EAP-IG PJI differences using six independently generated seed pairs. As reported in Appendix I (Table 7, lines 1363–1377), giving each model–task setup equal weight, CEAP achieves a statistically significant mean PJI improvement of 2.05×10−22.05\times10^{-2} over EAP-IG, with a 95% confidence interval of [1.86,2.24]×10−2[1.86,2.24]\times10^{-2}.


Thank you very much for your valuable suggestions and thorough review. We hope our revisions and additional experiments address your concerns about the scope of our contributions, clarity of presentation, mechanistic insights, and significance of the improvements. The additional experiments you suggested have helped ground our conclusions in more concrete empirical evidence. If these revisions resolve your concerns, we would be grateful if you could reconsider your score. We would be happy to address any remaining questions.

Reviewer_E3beOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper motivates the need for and introduces CEAP -- conductance-based Edge Attribution Patching. It provides theoretical as well as empirical evidence for why CEAP can outperform EAP when it comes to sampling variability, template variability of a set of inputs.

Critical strengths
  1. The paper motivates and studies the cause of different kinds of variances in circuit discovery -- this is an important topic since circuits are really important for interpretability research.
  2. The three kinds of variance that the authors have chosen to explain are important.
Critical weaknesses

(1) Results in section 4 (line 177) are cherry-picked to show the improvements of CEAP over EAP-IG. For eg. figure 1 shows a comparison of PJI yielded by CEAP and EAP-IG (using GPT-2 XL for a subset of SVA), where the results present evidence for CEAP having a higher PJI than EAP-IG at nearly all circuit sizes, but especially at small circuit sizes. However, while these results hold for all 6 pairwise comparisons for GPT-2-XL on IOI as well as SVA, they don't seem to hold on the greater-than task. They also largely don't seem to hold on any of the tasks in the case of the 2 Pythia models. For GPT-2, they hold somewhat on the SVA task but not on the other two tasks. This is even when PJI is measured between circuits of the same size.

(a) I think this should be plainly stated in the main paper for a fair presentation of the results, since reviewers are not required to read the Appendices. (b) This also weakens the very first claim in the paper around circuit stability under data resampling as well as the replication of results on multiple models. Even models w the same architecture, like GPT-2 and GPT-2-XL don't replicate the results.

(2) In section 5, I don't see comparisons between EAP-IG and CEAP. Is the improvement due to circuit sparsity or CEAP?

(3)

First, conductance is a more principled way of scoring compared to IG, as it satisfies additive order preservation and lowers resampling variance.

The paper does not prove this claim.

(4) Additive order preservation is a significant assumption when models still use LayerNorm -- the paper admits this, yet still claims that CEAP > EAP-IG.

Nits: -- Please increase font-size in all your images. They're extremely hard to read.

Questions for authors

(1)

For each template, we select the smallest swept circuit size at which the average unfaithfulness within each template falls below 0.09.

How was this threshold chosen?

(2)

The lack of improved cross-task transfer is not a drawback, since our aim is to understand or steer a given task rather than use its circuit for another.

Could you motivate why the cross-task transfer is necessary to reference in the paper if it's not directly relevant?

(3) Why does section 5 have so many metrics for motivating templated induced circuit variance as well as resolving it? Maybe some of these can be moved to the appendix?

6 comments
AuthorsOpenReview ↗

rebuttal 1

Thank you for your comprehensive review and constructive suggestions. We appreciate your recognition of the importance of our work. Below, we address your concerns and questions.

Materials relevant to our responses below are marked in the manuscript with your reviewer ID, E3be, in blue. Newly added text appears in green.

W1

Results in section 4 (line 177) are cherry-picked to show the improvements of CEAP over EAP-IG. For eg. figure 1 shows a comparison of PJI yielded by CEAP and EAP-IG (using GPT-2 XL for a subset of SVA), where the results present evidence for CEAP having a higher PJI than EAP-IG at nearly all circuit sizes, but especially at small circuit sizes. However, while these results hold for all 6 pairwise comparisons for GPT-2-XL on IOI as well as SVA, they don't seem to hold on the greater-than task. They also largely don't seem to hold on any of the tasks in the case of the 2 Pythia models. For GPT-2, they hold somewhat on the SVA task but not on the other two tasks. This is even when PJI is measured between circuits of the same size.

(a) I think this should be plainly stated in the main paper for a fair presentation of the results, since reviewers are not required to read the Appendices. (b) This also weakens the very first claim in the paper around circuit stability under data resampling as well as the replication of results on multiple models. Even models w the same architecture, like GPT-2 and GPT-2-XL don't replicate the results.

Thank you for raising this concern. We have revised Section 4 (lines 201–214) to present the distribution of ΔPJI\Delta\mathrm{PJI}, defined as CEAP’s PJI minus EAP-IG’s, averaged over the swept circuit sizes. This distribution covers all twelve model–task setups, rather than highlighting only one example as in the original version. ΔPJI\Delta\mathrm{PJI} is positive in 78.1% of cases, indicating that CEAP tends to improve circuit stability under data resampling. We have also expanded the experiments from 4 to 12 independently resampled datasets. These yield six disjoint seed pairs for PJI computation, replacing the original six pairs that shared datasets and were therefore not independent.

To assess differences across model–task setups, we perform a separate two-sided one-sample tt-test for each of the twelve setups below (four models × three tasks), with Holm correction across these tests. The results are reported in Table 8 of Appendix J.2 in the revised manuscript (lines 1728–1745) and reproduced below. Mean differences and unadjusted 95% confidence intervals are reported in units of 10−210^{-2}. The Result column indicates statistically significant improvements (Holm-adjusted p<0.05p<0.05) caused by CEAP over EAP-IG. No setup shows a statistically significant decrease in PJI with CEAP compared with EAP-IG.

ModelTaskMean ΔPJI\Delta\mathrm{PJI}Unadjusted 95% confidence intervalpp (Holm-adjusted)Result
GPT-2 XLSVA+6.32[+5.69, +6.95]1.61×10−51.61\times10^{-5}Improved
GPT-2 XLIOI+3.00[+2.88, +3.12]2.05×10−72.05\times10^{-7}Improved
GPT-2 XLGreater-than+1.34[+1.20, +1.48]1.64×10−51.64\times10^{-5}Improved
Pythia-2.8BSVA+2.70[+2.35, +3.05]4.15×10−54.15\times10^{-5}Improved
Pythia-2.8BIOI+1.58[+1.42, +1.74]1.64×10−51.64\times10^{-5}Improved
Pythia-2.8BGreater-than+0.50[+0.42, +0.58]9.38×10−59.38\times10^{-5}Improved
GPT-2 smallSVA+8.70[+8.13, +9.27]2.17×10−62.17\times10^{-6}Improved
GPT-2 smallIOI−0.48[−0.90, −0.06]0.130No significant difference
GPT-2 smallGreater-than−0.34[−0.79, +0.11]0.213No significant difference
Pythia-160MSVA+1.20[+0.84, +1.57]1.88×10−31.88\times10^{-3}Improved
Pythia-160MIOI−0.21[−0.57, +0.15]0.213No significant difference
Pythia-160MGreater-than+0.29[−0.02, +0.60]0.179No significant difference

Eight of the twelve setups show statistically significant improvements, including all six involving the larger models, GPT-2 XL and Pythia-2.8B. These improvements range from 0.50×10−20.50\times10^{-2} to 8.70×10−28.70\times10^{-2}. The remaining four setups show no statistically significant difference. Section 4 now explicitly reports these figures (lines 214–217).

In these experiments, several setups that you identified as showing no improvement, including GPT-2 XL on Greater-than and several Pythia setups, show statistically significant improvements with CEAP. Our original presentation may have made the differences difficult to assess because we did not clarify that the error bands show standard deviations rather than confidence intervals. We have now clarified this in Appendix J's introduction (lines 1386–1390) and figure captions.

AuthorsOpenReview ↗

rebuttal 2

You also noted that the PJI improvement is more pronounced for small circuits. This is true for some datasets. We point out that an advantage in the small-circuit regime is particularly relevant because circuit discovery aims to identify small accurate circuits, consistent with the minimality criterion [1]. Thus, a smaller difference at large circuit sizes does not negate the improvement in the small-circuit regime.

W2

In section 5, I don't see comparisons between EAP-IG and CEAP. Is the improvement due to circuit sparsity or CEAP?

We now compare the cross-template generalizability of CEAP and EAP-IG using statistical tests at all five weight-sparsity levels. We find no statistically significant difference in mean cross-circuit unfaithfulness deterioration ΔUm(n)\Delta U_m(n) at any sparsity level (Appendix V, Table 12, lines 2483–2506). Taken together with the improvement across sparsity levels for both methods (Appendix U, lines 2454–2460), these results indicate that weight sparsity, rather than the choice between CEAP and EAP-IG, is the main factor behind the improved cross-template generalizability.

W3

First, conductance is a more principled way of scoring compared to IG, as it satisfies additive order preservation and lowers resampling variance.

The paper does not prove this claim.

We support this claim both theoretically and empirically.

Theoretically, Appendix C proves that conductance satisfies additive order preservation, whereas IG does not (Theorem 3.4, line 187, with its proof in lines 788–831). Appendix H further proves that CEAP lowers resampling variance compared with EAP-IG in the sum-of-branches setting (Proposition H.2, lines 1217–1222).

Empirically, Figure 1 (lines 232–240) and Table 8 (Appendix J.2, lines 1728–1745), discussed in our response to W1 above, show that CEAP can reduce resampling variance in transformer circuit discovery. Figure 1 shows higher PJI for CEAP in 78.1% of model-task-template-seed-pair combinations. Table 8 shows statistically significant PJI improvements with CEAP in eight of twelve model-task setups. The remaining four show no statistically significant difference between CEAP and EAP-IG.

W4

Additive order preservation is a significant assumption when models still use LayerNorm -- the paper admits this, yet still claims that CEAP > EAP-IG.

For theoretical insight, we use a tractable setup (sum-of-branches functions) with accessible ground-truth edge importance to prove additive order preservation. We do not assume that transformer computations have this additive structure (Remark 3.5, lines 188–193). CEAP's advantage in transformer circuit discovery is supported empirically by the results discussed in W1 and W3.

Nits:

Please increase font-size in all your images. They're extremely hard to read.

Thank you for this suggestion. We have enlarged the text in the affected plots to improve readability.

Q1

For each template, we select the smallest swept circuit size at which the average unfaithfulness within each template falls below 0.09.

How was this threshold chosen?

We swept 100100 circuit sizes from 1010 to 53,07553{,}075 edges (15%15\% of the full graph). If the unfaithfulness threshold is too high, some templates may meet it with circuits smaller than our smallest swept size. Conversely, if it is too low, some templates may not meet it even at our largest swept size. Both cases degrade the accuracy of circuit-size estimation.

Our sweep range supports unfaithfulness thresholds in [0.05,0.13][0.05,0.13]. We choose the midpoint, 0.090.09, for the main-text presentation. The observed trend is not specific to this threshold: Appendix Q shows the same qualitative trend at both endpoints, 0.050.05 and 0.130.13 (Figures 35 and 37, captions at lines 2285 and 2320).

Q2

The lack of improved cross-task transfer is not a drawback, since our aim is to understand or steer a given task rather than use its circuit for another.

Could you motivate why the cross-task transfer is necessary to reference in the paper if it's not directly relevant?

We intended to clarify the difference in treating within-task and cross-task sample pairs. However, we agree that the sentence is unnecessary and have removed it to avoid confusion. Thank you for pointing this out.

AuthorsOpenReview ↗

rebuttal 3

Q3

Why does section 5 have so many metrics for motivating templated induced circuit variance as well as resolving it? Maybe some of these can be moved to the appendix?

We examine both circuit overlap and functional generalizability, which require different metrics. However, ranking similarity, measured by Kendall's τb\tau_b, provides limited additional insight beyond the circuit-overlap results. We have therefore moved its heatmaps and discussion to Appendix L (lines 1813–1819, Figures 24–29) to streamline the main text. Thank you for this suggestion.


Thank you again for your careful review and insightful suggestions. We hope the expanded experiments, statistical analyses, and clearer presentation address your concerns. If so, we would be grateful if you could reconsider your score. We would be happy to address any further questions.

References

[1] Wang et al., Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small. ICLR 2023.

Comments on the paper

4 comments
Area_Chair_jTGaOpenReview ↗

Paper likely to be rejected; follow discussion

Hi reviewers,

Thank you for your work reviewing this paper. The authors did not provide a response so far and since reviews are all negative, this paper is likely going to be rejected. However, the authors still have a chance to submit a response, so please keep following this and if they respond, please then go over the author response in detail and consider it.

Your AC

Dear Area Chair,

We have been working intensively on the rebuttal, including substantial additional experiments addressing the reviewers' concerns, which is why we are responding only now. We remain committed to the paper.

We would also like to kindly point out that our initial scores are not all negative, and we found all four reviews constructive and encouraging. We appreciate the reviewers' feedback and look forward to their replies.

Best regards, The Authors

Thank you for correcting my comment, authors. Indeed, we have one positive and three negative reviews so far.

Reviewers, the authors have now provided responses. Please go over them carefully and think whether you might want to change your evaluation in either direction. Please also go over the other reviews to see another opinion.

In any case, even if you remain with your prior assessment, please respond to the author response to acknowledge and explain why you change or keep your evaluation.

Thank you for helping move the discussion forward! With the discussion period nearing its end, we really appreciate your help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.