- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies when benchmark scores can distinguish competing model mechanisms and when targeted probes are needed. It gives a bounded-score “blindness” result, query bounds in terms of pairwise probe separation , a greedy multicover procedure for selecting probes, and experiments on synthetic candidates, small trained transformers, and Pythia-410m checkpoints. The distinction between estimating performance and identifying behavior is useful, and gives it a simple quantitative framing.
Critical strengths
- The problem is clearly motivated, and the separation measure makes the role of probe design easy to interpret.
- The authors acknowledge that the Pythia analysis is conditional on its two chosen candidates.
Critical weaknesses
-
The paper defines as the empirical separation of a fixed probe set. But the noiseless proof of Theorem 4 uses the probability of missing a separating probe, and the lower bound in Theorem 5 scales as . Those arguments require probes sampled afresh from a distribution with separation probability . For a fixed designed set with positive empirical separation, every candidate pair is already separated; with noiseless observations, identification is exact and has no cost. This is a material issue because the query bounds are the central theoretical contribution.
-
The two Pythia candidates together account for about 43% of predictions, and the reference label is which candidate agrees more on 1,200 designed probes. This demonstrates candidate ranking, not that the model follows one of those mechanisms with high probability—the assumption behind the identification guarantee. The experiments are also small: 16 tiny transformers and seven checkpoints from one Pythia training trajectory.
-
The blindness bound is a direct coupling argument, while the identification and selection results draw on standard hypothesis-testing and submodular-cover ideas. The synthesis may be useful, but the paper does not yet make clear what new insight or capability it provides beyond applying those tools to mechanism probing.
Questions for authors
-
Could you state separate assumptions and results for fixed designs and random sampling?
-
Concerning the Pythia candidates, could you keep the distinction between candidate ranking and distinguishing competing model mechanisms explicit in the abstract and conclusions?
Response to Reviewer 51Qk
Thank you for the careful reading. Separating fixed designs from random sampling, as you suggested, produced the main theoretical result of the revision.
W1 / Q1. Separate assumptions and results for fixed designs and random sampling. The Δ−1 scaling is a random-sampling law, whereas on a fixed design noiseless identification is exact once every pair is separated. The revision states the two settings as separate theorems, each with matching upper and lower bounds, and the comparison shows that design changes the exponent itself:
The experiments that test the Δ−1 law (Figure 2, the random arms of Figures 3–4, and the natural queries for language models) draw queries at random, the setting of Theorem 5. At all 44 settings of Figure 2, the measured n⋆ lies inside the window of Theorem 5 and matches the exact value up to Monte Carlo error. The designed arms are fixed designs, governed by Theorem 4.
W3. What the synthesis adds. A designed separating query contributes the exponent D(ρ)=−logβ, which diverges as ρ→0, while a random query contributes ΔQ(1−β)≤ΔQ. Design therefore removes the factor ΔQ−1, and for reliable models also the dependence on δ. The gain is largest for reliable models and rarely disagreeing candidates, where benchmark scores are most readily trusted. Two further results are new: Corollary 6 covers candidates that explain only part of the behavior, and Proposition 7 gives a single submodular objective whose value certifies the error. In simulation the certificate is within a factor of 1.7 of the measured cost at every k up to 256.
W2 / Q2. Candidate ranking. We adopted the term in the abstract, Section 6.1 and the discussion, and Corollary 6 makes it formal: if the model matches the two candidates with probabilities a>b on separating queries and neither otherwise, ranking errs with probability at most (1−(a−b)2)c, with a matching lower bound. The guarantee therefore covers ranking without assuming that the candidates exhaust the behavior, and the paper reports coverage a+b for every model. In current models the two candidates cover 0.53–0.75 of the answers, more than in any final Pythia checkpoint (at most 0.50), coverage increases with size within each family, and answers derived from the prompt or corpus statistics account for 76–90% of the behavior (Figure 5c).
For the reference label, an independent set of 1,200 designed queries (separate random stream, final token unique in the prompt) gives the same label for 131 of the 131 checkpoints and models evaluated on it.
Internal mechanism (new, Section 6.1, Figure 5d; general response, point 4). The certified ranking is tied to the circuit that produces it. Removing the 10% of heads with the highest induction score reverses the ranking in all 18 models ablated, while as many random heads, which cost at least as much WikiText accuracy, leave it unchanged in 34 of 36 draws.
Scale of the experiments. In addition to the original 16 networks (Figure 4a), we trained 960 networks across three widths and five separations, and the zero-parameter prediction holds within 0.014 in every cell (Figure 4b). 14 Pythia runs join the original one (five sizes, both training sets, five seeds each at 160M and 410M), and the change from bigram to induction falls between steps 512 and 1000 in all 15. We also evaluated 12 current open models up to Qwen2.5-72B, using identical queries (Section 6.1, Figure 5c).