acceptodds
Withdrawn from ICLR 2027

Are We Recovering Mechanisms? Objective-Level Recovery Gaps in Mechanistic Interpretability

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mechanistic interpretability aims to recover the internal computations responsible for model behavior. Progress in automated circuit discovery is often framed as a search problem: better attribution or optimization should identify better mechanisms. This assumes that the evaluation objective can recognize a better circuit once it is found. We show that intervention-defined faithfulness can instead prefer an equally sized circuit that reproduces the model's behavior less well, creating an objective-level recovery gap. Across four human-reference tasks and InterpBench, we compare validation faithfulness with behavior on held-out prompts under fixed ordinary resampling. The behavioral criterion is agreement with the intact model, including its mistakes, except on Greater-Than, where we use semantic accuracy. Controlled reference edits reveal misranking without any discovery algorithm, and outputs of EAP, EAP-IG, ACDC, and Edge-SP exhibit the same failure. Under resampling, KL misranks 9.4%–41.2% of candidate pairs across these methods on the human-reference tasks. We investigate context distortion as an explanation: replacing excluded signals changes the inputs on which retained components operate. Restoring selected signals from the recipient's intact-model execution repairs 96 of 100 persistent KL misrankings from the discovery pool on both validation and held-out prompts. The circuits and their original behavioral scores remain unchanged. These findings show why better discovery alone is insufficient when its objective rewards the wrong candidate.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, circuit discovery, circuit faithfulness, mechanism recovery, misranking, causal interventions, context distortion

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mLZ8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Circuit discovery is framed as a search problem: if the optimization or attribution is improved, we recover better circuits. The selection objective used is intervention-defined faithfulness, in which the non-circuit activations are replaced with donor activations, mean or zero, and the circuit is then expected to produce an output faithful to the intact model. This paper tests whether intervention-defined faithfulness, measured by FF (negative KL between the circuit and intact model output distributions on validation prompts), ranks equal-sized circuits the same way as a behavioral metric QQ (the circuit's top-1 task agreement with the intact model on held-out prompts). FF is computed under resampling, mean and zero interventions, while QQ is always computed under resampling and therefore stays fixed across interventions. Authors show that there is an objective-level recovery gap: FF can favor the lower-QQ circuit. They show this on four tasks (IOI, Greater-Than, Docstring, Acronym) and ten InterpBench tasks. Authors show that on equal-size edits of the reference circuits, misranking depends heavily on task and intervention, and that the same misranking appears among equal-size outputs of the discovery algorithms (ACDC, EAP, EAP-IG and Edge-SP). Authors attribute this gap to context distortion: the replacement signals alter the inputs to the retained circuit components, so they compute on inputs that differ from those in the intact model. To support this, they restore intact-model activations at selected excluded connections, which repairs 96 of 100 persistent misrankings on both validation and test prompts, while the circuits and their QQ values remain fixed.

Critical strengths
  • S1: the experimental design is clear such that all comparisons are size-matched, candidates, prompts and QQ are fixed and the reported measures appear reliable. However, this doesn't outweigh my concerns in W1 and W2 about how the results are interpreted and how well they support the paper's central claim.
Critical weaknesses
  • W1: most discovery algorithms do not actually use F precisely to select better circuits, so the relevance of the FF-QQ disagreement to improving the selection objective for circuit discovery is unclear, which is one of the central claims of the paper. ACDC and EAP/EAP-IG score individual edges by activation and attribution patching from task-specific contrastive prompts, which is different from the whole-circuit FF evaluated here. Based on this, I find the setup to be ill-motivated.

  • W2: I don't see how QQ under resampling is a valid ground truth. I appreciate that authors listed this as a limitation, but in my opinion, it is central to the paper's main argument of faithfulness-vs-behavior mismatch. (i) QQ is computed under resampling, so it carries the same context distortion attributed to FF, and I do not see why QQ is used as a reference, not FF. (ii) Fixing QQ to resampling also biases the comparison: the largest misrankings occur when FF and QQ use different interventions, whereas under matched resampling, misrankings to controlled edits are low (Table 3). These results only suggest that different interventions rank circuits differently, which is already known from Miller et al. (2024), "Transformer Circuit Faithfulness Metrics Are Not Robust", rather than FF failing to capture QQ. Measuring QQ under each intervention would separate these explanations, but the paper does not do this. (iii) Even in the matched resampling setting, the discovery misrankings involve small deficits (median is 1.0 and 0.11 pp on human and InterpBench tasks, Table 19). Based on (i--iii), the central claim is not adequately supported.

Questions for authors
  • Q1: How does FF-QQ disagreement precisely improve circuit discovery, if addressed?
  • Q2: What kind of prompts are being used as donors for the resampling intervention?
  • Q3: Does the misranking persist when QQ is measured under the same intervention as FF? (e.g., under mean and zero ablations)
  • Q4: How do misranking rates and deficits change with circuit size KK within a task, for example, when discovery outputs are extracted at several sparsity levels?
  • Q5: why is QQ the right criterion for a better circuit, rather than FF, and how much misranking remains among pairs whose output distributions differ only off the top-1 token?
Reviewer_Bf46OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks the question whether faithfulness (F) of a circuit as measured by KL divergence between the original vs intervened model (according to the circuit) can be at odds with the behaviour (Q) captured by the intervened model, as defined by the top-token agreement with the intact model on held-out prompts under fixed resample ablation. These "misrankings" can occur both for controlled edits of reference circuits and among outputs of EAP, EAP-IG, ACDC and Edge-SP, with KL misranking of 9.4–41.2% under resampling on human reference tasks. The authors attribute misranking to "context distortion" and report that restoring intact model activations can restore the rankings in 96% of the cases.

Critical strengths
  1. Careful experimental design with controls for size, prompts, etc

  2. The practical takeaway, according to me, is not to over rely on any metric in particular, nor to read too much into small differences in the metric scores.

Critical weaknesses
  1. Methodological issues with metrics are not new (Miller et al) and evaluating circuits is still quite qualitative. In this light, while the issues pointed out are interesting, they are not too unexpected. It is also unclear to me why the Q criterion is treated as the more "correct" one throughout the paper, as the KL operates over the full model distribution and can be said to more informative, although it may not correctly capture the behavior. It would be better to see which of the metrics picks out the circuit which has the highest overlap with the ground truth circuit, which are available in some synthetic benchmarks such as InterpBench. Without actually digging deeper into the mechanism itself, it is unclear what is going on.

  2. The context-restoration experiment effectively increases the size of the circuits by adding more information than is normally available to the circuit. But it is not clear if the restoration process adds the same number of edges to both circuits in a pair, which breaks the size control. The experiment therefore cannot distinguish "context distortion" from simple circuit incompleteness due to small size.

Questions for authors

If you could add experiments which use mechanism-level criterion and show with examples why some criterion (Q) is better than the other, that could be useful. Please report how often each metric (KL, LD, PD, Q) prefers the candidate with higher edge overlap with the ground-truth circuit in InterpBench.

Reviewer_xY8yOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors investigate whether circuits found in language models can have a high validation score but be unfaithful or don't explain behavior well.

Critical strengths

Red teaming interp techniques and describing their failures, especially unfaithfulness, is an important topic.

It's a great idea to use existing circuits from human suite and interp bench and ask how faithful they are.

Critical weaknesses
  • The writing appears AI generated throughout, including the abstract. This includes Claudisms and weird sentence structures that overall make the paper very hard to read and to understand.
  • From the abstract (and most of the paper), it is difficult to understand what the purpose of this paper is. What's the research question that's being investigated, what are the hypotheses?
  • The writing lacks detail and often invents jargon without explaining it. This contributes to the paper being hard to read: On the one hand, the writing is not concrete enough to understand what was done, on the other hand, there's a lot of nonstandard terminology that I don't understand. Some examples out of many just form the abstract: "controlled reference edits reveal misranking without any discovery algorithm", "candidate pairs", "context distortion", "recipient's intact-model execution".
  • Their measurement of faithfulness seems questionable. It is not obvious to me, why output KL should be called "faithfulness" at all, especially when the authors point is that this metric is not faithful. Similarly, I also don't understand why the "independent behavioral criterion" is the correct measurement of faithfulness, and if it is, why we weren't only using that in the first place.
  • There are only high-level quantitative experiments and no descriptions of individual circuits. This makes the experiments feel less grounded: It is hard to draw conclusions from the data alone with no reference to the actual circuit.
  • This carries over to the next weakness: There's no real conclusion from any of the experiments or the Discussion section. The latter only restates the results directly without any interpretation and discussion. This makes it hard for me to take away anything from the paper at all.
  • The big tables are hard to parse and should be replace by more informative plots.
Questions for authors
  • Why is the "independent test behavior" a better measure of faithfulness than the "validation score"? And if that's better, why isn't it used in the first place? How can we test faithfulness in the first place?
    • In table 2, why is KL sometimes zero and otherwise exceptionally high? What exactly do we measure here and how to interpret the results?
  • Can you show some concrete circuits (e.g. IOI) and qualitatively state what part of the circuit is unfaithful and how it should be interpreted instead?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.