- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tackles a genuinely interesting question: how much can complete low-order activation patching evidence actually tell us about the uniqueness of a minimum faithful circuit? Working in an additive model with maskable components and minimum faithful cardinality , the authors bound the number of minimum faithful circuits by . What's particularly nice is that they then construct a patched ReLU that attains this bound, in this example, going from order to order abruptly makes the full mask the uniquely faithful one. The paper doesn't stop there: it also provides a distributed construction with bounded local component complexity, tolerance-based stability results, an approximate-additivity extension, and a query lower bound for identifying an unknown member of the constructed candidate family. The validation here is purely mathematical, explicit constructions and proofs, no experiments on trained networks.
Critical strengths
-
What I found most convincing is how the paper combines an upper bound, an attaining construction, and an explicit next-order transition. The ambiguity survives complete evidence on the continuous low-order domain, so it cannot be waved away as insufficient sampling, a weak search procedure, or an incomplete verification step. This is exactly why the paper's distinction between faithfulness and unique identification feels genuinely useful rather than purely terminological.
-
The first construction concentrates a lot of cancellation in a single component, and honestly, without the distributed version I would have been much less convinced by the neural realization. By demonstrating exponential multiplicity with bounded local fan-in, parameter count, and response norm, the authors make it much harder to attribute the phenomenon to that particular modeling choice. That is a real strength of the paper.
Critical weaknesses
-
My main hesitation concerns the gap between these additive constructions and ordinary circuit ablations in a network with downstream dependencies. The paper is well aware of this and provides an approximate-additivity theorem, which points in exactly the right direction. That said, asking for a small uniform residual over all relevant patches and all masks is quite a strong requirement, and the paper stops short of establishing when this condition actually holds in a representative non-additive setting. To be clear, I do not believe a theory paper needs a transformer benchmark to be publishable. Even so, a concrete account of where this assumption can be certified in practice would make a substantial difference to the significance of the result.
-
Interestingly, the count can already be attained with constant additive components: simply take for and . In other words, the counting phenomenon itself is not specific to bounded-order activation patching. What feels more distinctive is that the paper realizes this count together with low-order equivalence and next-order separation. A related scope issue arises with robustness. Exact cancellation is generally destroyed by arbitrarily small perturbations, so the stability results understandably work with a positive tolerance, though the allowable per-component perturbations typically shrink as grows. The formal statements are mostly careful about this, but I would like to see the high-level discussion held to the same standard. Let me stress that these are limitations on the breadth of the contribution, not counterexamples to the stated theorems.
Questions for authors
-
Could you spell out how the cited Móricz–Nagy theorem and lemmas yield the integer zero-sum statement used in Lemma 23, including any relevant restrictions on their applicability? The universal bound rests on this step, and I would like that dependency to be readily auditable.
-
Does the case of the distributed construction also have the full mask as its unique order- faithful mask? My reading of the witness equations is that they force each block to be selected as a whole, and then matching the full response forces all blocks to be retained. If this is correct, I think it deserves an explicit corollary, since it would nicely strengthen the bounded-complexity part of the paper.
-
Is there a standard masking setting, beyond independent parallel branches or prescribed readout-edge groups, in which the required uniform additive-surrogate residual can already be bounded? An existing analytic example would really help me judge whether the approximate-additivity extension is mainly a continuity statement around the construction, or a useful route to less idealized circuit analyses.
4 comments
Thank you for adding the residual example. In Section 6, it establishes two minimum faithful circuits at positive tolerance, but does not demonstrate exponential multiplicity or the uniqueness transition at order .
In Theorems 10 and 14, the component responses are constant on , so faithfulness reduces to one scalar constraint. These constructions are valid, but their relevance to broader nonadditive networks remains unclear. Please keep this limitation explicit when discussing the residual example.
Overall, I keep weak reject recommendation but reduce my conf. to 2.
Thank you again for your careful review and further comments.
We agree that the scalar example alone does not establish exponential multiplicity or the transition at order . Theorem 19 and equation ~4 already establish their preservation under the stated residual bound. Following your comment, we have added a corollary to the existing Theorem 19 that provides an explicit residual network satisfying that bound.
-
Multiplicity and transition. For a positive integer , the corollary constructs consecutive residual branches from paired blocks. Under the stated patch-site assumptions, exactly masks of size are minimum faithful at patch order and tolerance . At order , the full mask is uniquely faithful at the same tolerance. The residual relative to the paired-block surrogate is uniformly at most over all masks and patches of order at most , satisfying equation (4). The corollary appears in Section 6, page 9, with its proof in Appendix D.1.2, pages 25–26.
-
Nonconstant responses. Your observation about the constructions in Theorems 10 and 14 is correct. In the added construction, every singleton response varies on the low-order patch domain. At an allowed patch, deleting the first branch changes the contribution of the second. Each branch uses nonzero local parameters, independently of . These conclusions concern the explicit constructed family under the stated assumptions. The scalar example retains its separate role as a certificate for two consecutive branches.
We accommodated the corollary within ten pages by shortening the residual calculations in Section 6 and consolidating few details in Appendix D.1.1. The existing mathematical results and their numbering remain unchanged.
We would be grateful for any further suggestions you may have on the revision.
Thank you for the additional construction. The new corollary in Section 6 and its proof in Appendix D.1.2 address my main remaining concerns. They establish exponential multiplicity and the transition at order in a nonadditive residual network with nonconstant singleton responses. I therefore raise my rating to Weak Accept. The lack of experiments on trained networks remains a limitation on practical relevance.
Thank you for your detailed and constructive review of our submission.
Q1. Zero-sum bound. For Lemma 23 we use Theorem 1.7 of Móricz and Nagy (arXiv:2411.04449v2). Remove the component responses that vanish on the allowed patch domain. Let N be the number remaining and q the maximum number whose sum vanishes there. Appendix C.4 constructs a linear functional on their rational span that takes rational values and is nonzero on every nonzero subset sum. Applying it and clearing denominators gives nonzero integers a1,…,aN with exactly the original zero-sum subsets. Let bi=i+2ai for 1≤i≤N. Each bi is irrational, and equality bi=bj forces ai=aj and then i=j. For every q-element index set T, the sum ∑i∈Tbi is rational exactly when ∑i∈Tai=0. For N/2<q<N, Theorem 1.7 bounds the number of such subsets by (q−1N−1). The reduction therefore needs no separate appeal to their Lemmas 2.1 and 2.2. For 1≤q≤N/2, we apply the Erdős–Ko–Rado theorem. Two disjoint maximum zero-sum omissions would have a larger zero-sum union. The family is therefore intersecting. If q=0, the empty set is the only possible omission among the remaining components. Your question led us to clarify both ranges after Theorem 11 (Section 5, p. 6). Appendix C.4 gives the reduction in the proof of Lemma 23 (pp. 18–19).
Q2. Paired blocks. Yes. In the paired-block construction of Theorem 14, let q be the number of blocks and p=2q the number of positive components. For the patch-order bound r in that theorem, the full mask is uniquely faithful at order r+1. The new corollary follows your argument. Each block has two positive components and one cancellation component. Moving from the zero patch to the order-(r+1) witness patch for positive component j changes the masked score by the retention indicator of j minus that of its cancellation component. The full score stays at q. Faithfulness forces these indicators to agree for every j. Each block must therefore be retained or omitted as a whole. Since each retained block contributes one, matching q requires all q blocks. Every proper mask misses the full score by a nonzero integer at zero or at a witness. Omitting one whole block gives error exactly one everywhere, proving the gap is sharp (Section 5, corollary after Theorem 14, p. 7, and Appendix C.6, pp. 20–21).
Q3. Residual branches. Veit et al. (2016, Section 4.1) delete individual residual branches while retaining the identity connections. We analyze all four masks on two consecutive branches under this operation. Let h be the patched input, u,v the branches, w the linear readout vector, and x1,x2 the binary retention indicators of the two branches. Let the additive surrogate match the empty and singleton masks. The masked score minus the surrogate score is x1x2w⊤[v(h+u(h))−v(h)]. If v has Euclidean Lipschitz constant at most Lv and ∥u(h)∥2≤U on all allowed patches, the absolute residual is at most ∥w∥2LvU for every mask and patch. We derive a bound from the weights for ReLU branches and an exact value for linear branches with affine patched input (Section 6, p. 9, and Appendix D.1.1, pp. 24–25).
W1. Residual certificate. Your suggestion prompted the scalar example. For h=z∈[0,1], w=1, u(h)=h/8, and v(h)=h/10, the uniform residual is 1/80. The larger surrogate singleton error is 10/80, and the surrogate empty-mask error is 18/80. For each mask, Theorem 19 bounds the change in error by 2/80. At tolerance 14/80, both singletons are faithful (error at most 12/80), and the empty mask is unfaithful (error at least 16/80). These bounds certify precisely the two singleton masks as minimum in a model where deleting the first branch changes the input to the second (Section 6, p. 9, lines 471–476).
W2. Counting and stability. Your constant-component example now follows Proposition 9. For n components and minimum faithful size 1≤k<n, its (kn−1) minimum circuits agree at every patch order. With m patching sites, 0≤r<m, and n−1≤(r+1m), the patched ReLU construction in Theorem 10 attains the same count through order r and separates every pair at order r+1 (Section 5, pp. 5–6, and Section 8, p. 10).
As you note, arbitrarily small perturbations can destroy exact cancellation. The abstract and Section 1 now specify positive tolerance for stability. At tolerance 0<ε<1, Theorem 17 preserves the constructed family when the component perturbations satisfy its uniform bound and Equation (3). For fixed ε and k=⌈n/2⌉, the bound on the uniform perturbation per component decreases as 1/n. The unperturbed next-order gap of one is 1/k of the full score (Section 5, pp. 7–8, and Section 8, p. 10).
We would be grateful for any further suggestions you may have on the revision.