- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses a fundamental yet often overlooked assumption in patching-based circuit discovery: that the baseline corruption is fixed a priori. To tackle this, this paper introduce SEED, an alternating optimization method that iteratively picks counterfactual tokens and updates circuit edges via a straight-through surrogate. Experiments on 8 tasks show that SEED yields improvements in held-out faithfulness and completeness compared to fixed-corruption controls.
Critical strengths
- Questioning the fixed-corruption convention in activation patching is timely and thought-provoking.
- The paper covers 8 standard benchmarks across a wide range of budgets, evaluates both sufficiency (denoising) and necessity (noising), and sensibly tests everything on held-out benchmark corruptions to avoid trivial metric cheating.
Critical weaknesses
-
While the paper claims to rethink fixed corruptions, the contribuytion is still extremely constrained. The algorithm still requires human-designed reference templates, manually pre-selected editable slots (), and token-length matching. The most difficult parts of creating corrupted text—deciding how to intervene, and fixed token counts—are still solved by humans. In reality, SEED doesn't escape the fixed-corruption paradigm; it merely tunes a single token inside an already human-curated counterfactual slot.
-
In tasks like IOI, different corrupted text designs serve distinct causal purposes (e.g., change which name is direct and indirect object without changing the names or positions, like "John and Mary ... John" to "John and Mary ... Mary", is for the value and position of the indirect object). These fixed contrasts are intentionally crafted to isolate specific sub-mechanisms. Simply picking tokens that maximize a generic behavioral drop () risks collapsing these fine-grained causal inquiries.
-
In mechanistic interpretability, we generally view a "circuit" as an intrinsic property of the model's weights—the actual computational subgraph implementing a capability. If a circuit is real, it shouldn't be fragile to the specific optimization path. However, RQ3 shows that using SEED’s learned corruptions on other algorithms (like EAP / EAP-IG) often leads to negative transfer. The authors frame this as corruptions being "circuit-coupled best responses", but to me, this raises a red flag: does SEED actually uncover the model's true underlying mechanism, or is it just finding an ad-hoc subnetwork overfitted to a co-adapting prompt contrast?
-
The manuscript has a clear tendency to using AI tools in math and method derivation:
Theorem 3.2 is essentially a restatement of subset maximization. Theorem 4.1: Framing a simple scoring and ranking over discrete candidate tokens as a variational optimization feels like massive overkill. In practice, it's just a softmax score. Eq. 16 vs. Eq. 27: Introducing an abstract "recovery-aligned shaping utility" in the main text, only to reveal in the appendix that it is simply a smooth function.
Stripping away this artificial complexity would make the paper much more transparent and enjoyable to read.
Questions for authors
-
Following RQ3, if a discovered circuit cannot be reliably replicated under alternative optimizers using the same contrasts, why should we consider it the model's "true" mechanism rather than an artifact of SEED’s joint trajectory?
-
How would SEED scale if the editable coordinate is not a single token, but spans multiple tokens or disjoint sites across a long-context prompt or larger size model?
-
In Table 18, SEED incurs a ~1.46x wall-time overhead compared to SEED-EDGE (327s vs. 223s), why SEED-EDGE is faster than SEED?