- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates what the authors call the lottery circuit hypothesis. The authors demonstrate experimental results showing that task-functional circuits can emerge from zero-masking-based circuit discovery performed on random-weight transformers. The paper also shows that we can hide the trace of exact-zero-masking by refilling the zero-masked weights with newly generated random weights that approximate a Gaussian distribution together with the unmasked weights. A minimal toy theoretical modelling is provided to rigorize the results.
Critical strengths
(1) The paper performed extensive experiments to clarify and support their claims and results.
(2) Distributional refill after masking-based circuit discovery can retain useful behavior is, to some extent, an interesting observation.
Critical weaknesses
(1) The biggest deal-breaker for me is that the extensive analysis of the problem the authors study seems without a motivation that is compelling or justifiable. Lottery circuit hypothesis is not surprising given strong lottery ticket hypothesis, and task-functional circuits might be a result of the circuit discovery method itself is not news: there has already been works performing circuit discovery on random transformers. (See my questions) Distributional refilling, while an interesting insight, offers little practical value for circuit discovery researchers. Its setup is a narrow contrivance.
(2) The writing of this paper needs great improvement, with a lot of technical details missing and explanations of the experimental results often scattered far away from the results themselves. The use of terminologies that are not defined also causes a lot of confusion.
Questions for authors
At this point, I am inclined to reject this work, unless the authors resovle my doubts about their paper's motivation.
(1) The take-home message from the lottery circuit hypothesis is that the task-functional circuit could be a result of the circuit-discovery method being able to "learn" the task. There has already been at least one previous work [1] on this, which also performs circuit discovery on random-weight transformers. I wonder how your work furthers or differs from their insight.
(2) I do not see the practical value of distributional refilling. How do you expect this insight to change or guide future practices of circuit discovery or interpretability research in general?
(3) What is the fundamental difference between your lottery circuit hypothesis and strong lottery ticket hypothesis and supermasks? Both show how masking can learn for a specific task, right?
(4) A lot of contents are discursive and off-topic: Table 1 is a discussion on DiscoGP's property. Table 2 also feels uninformative. Why do we want to compare the circuit density of pretrained and randomised transformers? Figure 3 and 4 are properties of distributional refilling, which is not useful unless the previous question regarding the latter's motivation is properly addressed. Figure 5 fixes a problem on randomised transformer circuit discovery. I wonder how this can be useful. We almost never perform circuit discovery on randomised transformer, and there are a lot of other ways of randomisation.
Other minor suggestions/questions:
(5) You should define or clarify the following things:
- "task margin" on line 84.
- and on line 94-95. Moreover, the loss where these regularization coefficients show up is never defined.
- "refilling" and "completion" are used exchangeably, which is confusing.
- "selection"on line 265 appears abruptly.
(6) The transition between the two paragraphs on lines 201-215 needs to be clarified. They are different refilling schemes.
(7) The paragraph on line 210 requires major rewriting: it is the key technical detail of distributional refilling, but much of it is conveyed in rather abstruse language. The two-components reference distribution and the Gaussian bin-filling both warrants detailed discussion.
(8) A few other citations might be added for "variation among circuits" in the related work [2-4].
(9) Much of the circuit behavior is quantified using “accuracy” (Equation 3), defined as the fraction of examples with a positive task margin. This thresholded metric discards the magnitude of the margin and may therefore obscure behavioral differences between circuits. Would examining logit diff or prob diff, as discussed in [5], provide a more fine-grained characterization? Could the authors contextualize their choice of accuracy, including references to prior circuit-discovery work using this metric? This is not a major concern for my assessment. I just want a clarification.
I will consider improve my rating if the authors adequately address the above questions.
[1] Méloux et al. (2025). The Dead Salmons of AI Interpretability. arXiv:2512.18792.
[2] Rai et al. (2026). Data-driven Circuit Discovery for Interpretability of Language Models. arXiv:2605.09129.
[3] Wu et al. (2026). Demystifying Variance in Circuit Discovery of LLMs. arXiv:2606.16920.
[4] Cho et al. (2026). Shared Semantics, Divergent Mechanisms: Unsupervised Feature Discovery by Aligning Semantics and Mechanisms. ICML.
[5] Hanna et al. (2026). Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanisms. COLM.