- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents an analysis and ablation studies of six model-task settings from the Mechanistic Interpretability Benchmark to show that circuit-based explanations are insufficient to fully reproduce model behavior because they do not account for the model’s failures.
Critical strengths
- The authors clearly state the research question: “To what extent do existing circuit-analysis methods account for naturally occurring model errors?”
- They evaluate their approach using three different benchmarks: IOI, Docstring, and MIB.
- They address an important issue in model interpretability: whether circuit-based explanations can fully explain model behavior, including its failures.
Critical weaknesses
-
Contributions: The paper’s main contribution is not entirely clear. The authors could better explain what distinguishes their work from previous studies.
-
Experiments and reproducibility: The implementation details are not sufficiently clear to ensure reproducibility. In particular, for results such as those reported in Table I, it is unclear which original models were evaluated.
-
My main concern is: since circuit-based explanations might capture only specific behaviors rather than the full range of a model's behavior, why can we not simply define a different input set, including only errors, when extracting circuits? Since models can behave differently across different input distributions, it would be natural to apply the same circuit-analysis methods directly to incorrect predictions and compare the resulting circuits with those associated with correct predictions, wouldn’t it? This could help identify the mechanisms responsible for model errors and clarify the limitations of existing circuit-based explanations. So, the problem might not be the circuit extraction itself, but rather what we want to extract. To me, it seems natural that circuits do not explain the whole behavior of the model; if that were the case, we could even use the circuits instead of the model. To summarize: If circuit-based explanations do not capture model errors, is this because the circuit extraction methods are insufficient, or because the targeted behaviors are only the ones in which the model is correct?
Questions for authors
I have included questions above.