acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Circuit Evaluation: Do Circuits Explain Model Errors?

Abstract

Mechanistic interpretability (MI) aims to explain a model's behaviour through analyzing its internal computations; circuit-based explanations aim to isolate these computations with compact subnetworks validated by ablating the rest of the model. We show that circuits validated this way may fail to recover the underlying mechanism of the model's behaviour by closely reproducing its successful decisions while failing to account for most of its errors. Such explanations should account for the model's particular errors as well as its successes. We evaluate this requirement by measuring exact answer agreement separately on model successes and failures, across circuit sizes and ablation settings, on IOI, Docstring, and six model-task settings from the Mechanistic Interpretability Benchmark. We discover that many tested circuits closely replicate correct behaviour while missing most of the model's errors. On indirect object identification (IOI) for GPT-2 small, under mean ablation, the manual circuit and tested automated circuits, including one trained against the model's full output distribution, agree with the model on 97.3-99.5% of prompts it answers correctly but only 11.4-41.7% of errors. An IOI case study shows that lost errors are recoverable by restoring omitted attention-heads which raise error reproduction from 14.2% to 75.1% on a separate held-out set with 0.41 percentage point decrease on correct agreement, exceeding matched random extensions and scalar-biased control. Intervention traces show how omitted computations produce specific wrong answers for a reproducible subset of errors. In all, these findings show circuits can preserve task success without adequately explaining model's failures, and support exact error reproduction as a necessary, but not sufficient, test of circuit-based explanations of model behaviour.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, circuit discovery, circuit evaluation, mechanism recovery, model errors, causal interventions, ablation, indirect object identification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NCPQOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an analysis and ablation studies of six model-task settings from the Mechanistic Interpretability Benchmark to show that circuit-based explanations are insufficient to fully reproduce model behavior because they do not account for the model’s failures.

Critical strengths
  • The authors clearly state the research question: “To what extent do existing circuit-analysis methods account for naturally occurring model errors?”
  • They evaluate their approach using three different benchmarks: IOI, Docstring, and MIB.
  • They address an important issue in model interpretability: whether circuit-based explanations can fully explain model behavior, including its failures.
Critical weaknesses
  • Contributions: The paper’s main contribution is not entirely clear. The authors could better explain what distinguishes their work from previous studies.

  • Experiments and reproducibility: The implementation details are not sufficiently clear to ensure reproducibility. In particular, for results such as those reported in Table I, it is unclear which original models were evaluated.

  • My main concern is: since circuit-based explanations might capture only specific behaviors rather than the full range of a model's behavior, why can we not simply define a different input set, including only errors, when extracting circuits? Since models can behave differently across different input distributions, it would be natural to apply the same circuit-analysis methods directly to incorrect predictions and compare the resulting circuits with those associated with correct predictions, wouldn’t it? This could help identify the mechanisms responsible for model errors and clarify the limitations of existing circuit-based explanations. So, the problem might not be the circuit extraction itself, but rather what we want to extract. To me, it seems natural that circuits do not explain the whole behavior of the model; if that were the case, we could even use the circuits instead of the model. To summarize: If circuit-based explanations do not capture model errors, is this because the circuit extraction methods are insufficient, or because the targeted behaviors are only the ones in which the model is correct?

Questions for authors

I have included questions above.

Reviewer_JNKoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors look at circuits in LMs for well studied tasks, and find that reported circuits don't always explain model errors well. Rather, the circuits mainly only explain the correct, clean cases. Commonly used circuit evaluations can conceal a substantial failure to reproduce model errors

Critical strengths

The paper is well-written, clear, and makes a concise and easily understandable point. The scope of the experiments is well . I think this is clearly a well thought out and good paper investigating an interesting phenomenon (the relationship to model mistakes and divergence from model failures).

Critical weaknesses

I think the takeaways and impact of this paper is a little half-baked, although I think this is resolvable. The crux of the paper, as I understand it, is that model failures do not use the same circuit as model successes for the given task, and that's a critique on circuit evaluations. (1) is this really a problem? (2) And either although I'm not sure that this topic has been directly studied before, I think it is decently often observed in practice and I think the paper falls short by not providing a strong enough insight beyond the observation of the behavior. The paper makes the claim "Our contribution covers the systematic evidence about the ability of existing circuit-analysis methods to account for the full mechanism of models", but I think that falls a bit flat. Should a circuit for a task explain incorrect behaviors? I think it shouldn't. I think it's a feature of circuits that they explain only model successes. A generalizing algorithm should* only explain the successes. I think there is a good argument that this insight can be made *useful, but I don't think this paper currently demonstrates that. Maybe put another way: I don't expect model errors to have as clean circuits as correct predictions, but what can we do with that?

Missing quite a few of citations, especially on the IOI circuit. I can write some of these up, but I think if the authors used an LM to help them find additional related work they would find most of them

Questions for authors

N/A

Reviewer_EmskOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors identify and investigate a surprising and potentially disturbing behavior of mechanistic interpretability methods that discovering circuits within large neural networks: The discovered circuits tend to reproduce the successes of the full network but not the failures.

Critical strengths

The authors pose a remarkably simple and important question (“To what extent do existing circuit-analysis methods account for naturally occurring model errors?”). The answer (“Not well”), is both informative and disturbing. Discovered circuits should reproduce errors as well as successes. Indeed, based on concepts from the philosophy of science (e.g., Popper’s notion of falsifiability) error are perhaps more important than successes for assessing whether a circuit represents an effective model of a network’s reasoning.

The authors do detailed analysis of several known circuits.

Critical weaknesses

The authors provide relatively little intuition for why AerrA_{err} is typically so low. One plausible explanation is that correct answers result from the larger network activating the circuit (and the circuit doing its job), whereas errors result from the larger network activating some other circuit (which may be one of many). That would result in a situation in which correct answers are informative about a circuit whereas incorrect answers are far less informative. That is, it could be that correct answers all act the same, but each error is different in its own way.

As with many circuit analysis papers, the authors provide a huge amount of detail, both in the body of the paper and in supplementary materials. Due to this, the paper can be difficult to follow in parts. The authors would communicate better with readers if they provided a clearer and more intuitive narrative arc throughout the paper, supported by the details but not succumbing so much to them.

Questions for authors

At a high level, what are you hypotheses for why error are under-represented by the circuits discovered by most methods?

Why, precisely, is it important that errors be accurately reproduced by discovered circuits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.