acceptodds
Under review as a conference paper at ICLR 2027

When AI Explains AI: Understanding and Mitigating the Interpretability Lottery

13 views

Abstract

Interpreting the internal representations of language models is crucial for understanding their behavior and guiding efforts to make them reliable and safe. Recent advances have enabled the discovery and interpretation of model features at scale. We study the interpretability lottery, in which a feature’s attributed meaning remains contingent on the evidence and inferential choices of the interpretation procedure after evaluation and selection. A single published description leaves this semantic uncertainty implicit. We ask how the uncertainty relates to a feature’s activating evidence and use, and how it can be mitigated. First, we probe an explain-then-score pipeline on Gemma Scope sparse autoencoder (SAE) features of Gemma-2-2B, rerunning it fifteen times on each of 150 features with the model, the SAE and the feature held fixed. No feature is near-unanimous across runs: the median feature receives about 9.5 effectively distinct labels, and automated judgments find substantial differences in meaning. Variation in held-out AUC does not track label instability, and label indices from different runs share less than a fifth of their top-ten retrieval results. Evidence selection produces the largest contrast in our factorial design, while context diversity correlates with instability (ρ ≈ 0.54). Second, we use these findings to test mitigations. With examples drawn at random, greedy decoding does not reduce median instability, and aggregating runs reduces score SD more than medoid-label instability against matched controls. Selecting the explainer’s top-k examples deterministically roughly halves instability at the same pipeline-run budget and holds on new features, yet even then most features remain above the near-unanimity threshold. These findings motivate treating a feature description as a semantic claim whose uncertainty and dependence on the selected evidence must be characterized alongside its evaluation score.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, sparse autoencoders, automated interpretability, reliability, reproducibility, measurement, evaluation, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D7QQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at the labels of features that received their meaning through autointerpretability. They show that this meaning can depend on choices in the pipeline, such as which activating examples are shown. The authors find high label instability for all features. Additionally, they show a weak association between label instability and score variability. Their results suggest that the example draw is a dominant driver of label instability. As mitigations, they test greedy decoding, more examples, top-k and aggregations, where top-k halves instability with minimal AUC change.

Critical strengths

The paper is well-motivated and tackles a relevant problem of using LLMs as judges. It is a critical issue: semantic uncertainty hidden by single-run autointerp labels. The paper shows the results for one pipeline and one model very well. A strong contribution is the repeated-run measurements (15x per feature), as well as the semantic vs score dissociation. The separation of semantic variation and score reliability is strong, as they measure each separately (Section 3.3). This avoids confusing stable scores with stable meanings. Section 4.5 ties their results to how practitioners use autointerpretability to label features, which shows the importance of their research.

Critical weaknesses

The main setting is quite narrow. Using a single-layer, single-source model and single (relatively small) SAE. Using a single judge and scorer. Holding one layer fixed can be motivated, but generalisability of results would require additional SAEs and the potential influence of the judge and the scorer.

Top-k strongest context seems to be likely to bias a label towards a high-activation context, rather than just the feature’s potentially broader meaning. The consequences of this interpretation should be mentioned more clearly

Greedy decoding is not tested under fixed examples, only when they are random, which weakens the mentioned outcomes about greedy decoding (see section 4.4). This contradicts the fact that in real pipelines, examples are often fixed and not random.

Probes are written based on the candidate label. This could push vague or general labels. The relationship between stable AUC and unstable semantics is not addressed deeply enough.

Questions for authors
  1. Could you expand your analysis to at least another SAE on a different model?
  2. Do you have any experiments with alternative explainers or scorer models? Maybe even small feature subsets, just to see if the instability is similar.
  3. What is the meaning of selection AUC in the appendix table 6?
  4. “Features whose contexts are more varied get less stable labels”, section 4.5. this is attributed to the context of the activating samples, which makes sense in the experiment itself. However, some features might activate on more diverging contexts. How could this be related to polysemanticity?
  5. Could you test greedy decoding under fixed examples? Like temperature 0 for the explainer under fixed top k examples? This would be a good addition to Table 2
  6. Contexts are chosen and then 5 candidate labels. How does the instability change with more contexts and candidates or maybe fewer? Is there any sensitivity related to the number you picked? Nice pipeline in B1.
3 comments

Thank you for the review.

W1/Q1: Another SAE and model (60 features each; Appendix C.2, Figure 9):

SettingRandomTop-kkReduction
Gemma-2-2B L12 16k (paper's SAE, 50 features)0.6060.3190.287
Gemma-2-2B L20 65k0.5220.3280.194
GPT-2 small L60.5280.3160.212

No feature is near-unanimous under random draw in either new setting. Without features whose median held-out AUC is at most 0.55 (36 and 47 remain), the reductions are 0.18 in both.

W1/Q2: Other explainers and scorers. An Opus 5.5 explainer lowers instability by 0.145 (random) and 0.093 (top-kk) but leaves no feature near-unanimous under random draw. With the Opus explainer held fixed, swapping the scorer to Opus 5.5 gave no reduction on 25 features (0.485 vs 0.474 under random draw, not distinguishable from zero). The lottery thus appears with three scorers and two explainers. Reselecting each run's label by AUC alone, which takes the steering judge out of selection, barely changes instability (median 0.604 vs 0.585).

W2: Top-kk bias toward high-activation contexts. We agree this is an important caveat, and have now measured it. A judge marked which of a feature's stored contexts (up to 40) each label describes. Compared with random-draw labels, top-kk labels cover the five strongest contexts far better (0.597 vs 0.377) but the remaining contexts worse (0.241 vs 0.308); a second judge agrees. Part of top-kk's stability therefore comes from describing the top of the activation distribution, as you suspected. For a single label the overall difference is modest (0.290 vs 0.316), but the labels from 15 runs together cover 67% under top-kk against 88% under random draw, so the loss is mainly across the collection of labels. Section 4.4 now states this trade-off.

W3/Q5: Greedy decoding with fixed examples. With top-kk examples and explainer temperature 0, instability falls from 0.319 to 0.073. 25 of 50 features fall below the near-unanimity threshold (instability below 0.05), and in 17 every run returns the identical label. A single label covers as much as under sampled top-kk (0.358 vs 0.366; false rate 0.049 vs 0.052), so greedy decoding removes variation without narrowing the label. Ten runs together cover less (49% vs 70%) because they yield fewer distinct labels (2.0 vs 7.8). The new experiments show that our greedy conclusion applied only to random examples. The revision reports this in Appendix C.1 and Figure 4b, and revises Finding 3 (not Table 2: different scorer).

W4: Probe-based scoring and vague labels. We measured vagueness as how often a label is judged to describe other features' contexts. Pipeline labels do so 6–7% of the time. In the two main experiments the selected label has a lower false rate than the best rejected candidate (by 0.015 and 0.011); on the pilot arms with the new scorer the difference is not distinguishable from zero. So we find no evidence that selection favors vaguer labels, though this does not isolate the AUC term alone. Stable AUC with unstable semantics follows from how AUC is computed: a label is tested on probes written for that same label, so two different labels can each score well.

Q3: Selection AUC. Selection AUC is the AUC of a candidate label on the probes the scorer wrote for that candidate during selection, and it carries weight 0.60 in the confidence score. Held-out AUC is computed on a fresh probe set written after selection, so it plays no part in choosing the label. The revised Table 6 caption defines both.

Q4: Polysemanticity. Varied contexts fit polysemanticity, but also a single feature that fires across topics; the feature in Figure 1 is compatible with either reading. If the draw exposed polysemanticity, runs shown similar contexts should get similar labels, most strongly for diverse features. Within features, context-set similarity and label similarity correlate only weakly (ρ=0.097\rho = 0.097, against 0.001 when labels are permuted), with no rise across diversity terciles (0.088, 0.106, 0.097; Appendix C.8).

Q6: Contexts and candidates (50 features; the baseline of 5 and 5 gives 0.606):

Contexts351020
Instability0.6220.6060.5030.438
Candidates1358
Instability0.4400.5650.6060.601

More contexts help steadily (partly because larger draws overlap), and fewer than 5 barely matters. Requesting one candidate lowers instability by 0.166. A replay that holds generation fixed attributes most of this to selection: publishing the explainer's first-listed candidate from the same five-candidate runs gives 0.484 against 0.606 for the pipeline's choice, and only 0.045 remains between that rule and the one-candidate arm. So instability is sensitive to both numbers, and our choice of 5 candidates sits on the plateau. Both sweeps and the replay are in Appendix C.4.

AuthorsOpenReview ↗

Follow-up on W4 and Q6: which selection rule gives better labels, judged on held-out text

We compared three rules on the same stored candidates: the explainer's first candidate, the candidate with the highest selection AUC, and the published rule. A blind judge marks which windows of held-out FineWeb text each label describes (Appendix C.9, Table 24). Paired differences in single-label coverage, Haiku (50 features) / Sonnet (25 features):

ArmComparisonHaikuSonnet
Random drawhighest AUC −- first+0.050+0.050 [0.027, 0.076]+0.039+0.039 [0.019, 0.062]
Random drawpublished −- highest AUC−0.025-0.025 [−0.040-0.040, −0.011-0.011]−0.027-0.027 [−0.049-0.049, −0.009-0.009]
Top-kkhighest AUC −- first+0.005+0.005 [−0.016-0.016, 0.025]+0.000+0.000 [−0.033-0.033, 0.033]

With random examples, the highest-AUC candidate describes more of the held-out text than either alternative. On W4: it also marks slightly more topically similar inactive text under Haiku (+0.032+0.032), but not under Sonnet (+0.005+0.005), so we find no consistent sign of vaguer labels. With top-kk examples, the rules give equally good labels, and the first candidate is far more stable. Candidates whose selection AUCs are within 0.02 of each other cover largely different held-out windows (Jaccard overlap 0.18 to 0.31), so a near-tie in score is not a near-tie in meaning.

Reviewer_D7QQOpenReview ↗

Reviewer response

Hi, thanks for the concise response.

W1 and W2 are fixed. Results of W2 are interesting.

W3. Thank you for testing this. Conclusions are interesting, “greedy decoding removes variation without narrowing the label”.

All other questions are answered. I will keep my score.

Reviewer_UTNtOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a pitfall of feature interpretation pipelines, where the exact same procedure assigns different semantic meanings to the same features across different runs. The paper identifies three sources of randomness in these pipelines: the examples shown to the explainer, an explainer that writes semantic meanings, and a scorer that writes new contexts and scores how well the semantic meanings align with the feature activations. Experiments vary which source of randomness is held fixed and how they are held fixed, and the paper proposes a metric to measure semantic instability. Testing on 150 features from Gemma Scope SAE where the same auto-interp pipeline is repeated 15 times, the paper finds that no feature has a unanimous meaning, with the median number of distinct meanings per feature being 9.5. The random evidence shown to the explainer contributes the most to semantic instability, and when deterministically selecting the top-k evidence, as current feature interpretation pipelines do, the semantic instability is halved, but still no feature has a unanimous meaning. The paper shows how feature descriptions jointly depend on both the features and the feature interpretation pipeline.

Critical strengths
  1. The authors identify an important issue they call the interpretability lottery with commonly used feature interpretation pipelines. They propose concrete metrics to determine semantic instability and near-unanimity and find insightful evidence on how different sources of randomness in the feature interpretability pipeline contribute to semantic instability.
  2. The experiments are thorough, comparing not only different components of the auto-interp pipeline but also different remedies for each component, interactions between different components, and direct downstream consequences in feature search and efficient monitoring signals of semantic instability.
Critical weaknesses
  1. It is questionable whether semantic stability is desirable without any guarantee of the correctness of the feature description. If the SAE decomposition truly finds monosemantic features, then it is possible that a correct semantic label should also be a stable one. However, if the SAE decomposition finds polysemantic features, or if the auto-interp pipeline constrains how semantic descriptions can be presented, a collection of diverse semantic labels might more accurately represent the feature than a single label that only captures the top-k evidence.
  2. The difference between this work and van der Ben et al., 2026, specifically on experiments that target score variation, is unclear.
Questions for authors
  1. Could the authors explain how the score variation experiments in this paper are different from those in van der Ben et al., 2026?
  2. Could the authors justify why stable semantic labels are desirable when their correctness is still difficult to verify? Alternatively, could the authors provide some evidence that in experiments where semantic label stability is halved, the fewer unique labels still cover the semantic meanings of all or most of the full 40 activating contexts?
  3. In Table 1, could the authors describe or interpret ICC(3, 1) and ICC(2, 1) before reporting their values?
  4. In Line 292 where the paper states that the pilot “set the sample size”, which sample size is it referring to?
  5. In Lines 309-311, the non near-unanimous label instability score is contrasted against the “moderate” score reliability. However, we don’t know which range of label instability score is equivalently considered “moderate”. Could the authors explain what this range should be?
  6. In Line 421, is there any specific challenge in testing the effect of greedy decoding with fixed examples? Alternatively, could the authors provide updated results including the effect of greedy decoding with fixed examples?
3 comments

Thank you for the review.

Q1/W2: Difference from van der Ben et al. (2026). They decompose the variance of auto-interp scores and feature rankings across pipeline choices, with the scorer held fixed. We study the selected label itself: we separate the three random steps inside one configuration (including the scorer), measure the semantic variation of the selected label, and test whether a feature's score variation tracks its label variation. It does not (on 141 features, we can rule out a linear correlation stronger than about 0.18 in either direction), so a stable score cannot certify a stable label. Both studies find the example draw to be a leading source of variation. The revision makes this comparison explicit in Section 2.

Q2/W1: Why stability, and do the fewer labels cover the 40 contexts? Repeatability matters when users expect a published label to make the same claim on a rerun: a label that changes from run to run cannot be relied on even if one version is correct. How much evidence a label describes is a separate question, which we now measure. A judge marked which of a feature's stored contexts (up to 40) each label describes, mixed with other features' contexts (the false rate). Haiku 5.5 judged all 150 main features and Sonnet 5.5 rejudged 100 (item agreement 90–91%, κ\kappa 0.66–0.67).

Haiku judge, 150 featuresRandomTop-kk
One label: all 40 contexts0.3160.290
One label: top 50.3770.597
One label: ranks 6–400.3080.241
Union of labels from 15 runs0.8780.672
False rate0.0710.058

The fewer labels cover the 40 contexts partly. A single label covers about 30% of the contexts under either policy. Top-kk labels fit the top 5 far better (+0.22+0.22) but rank 6–40 worse (−0.066-0.066). The labels from 15 runs together cover 67% under top-kk against 88% under random draw (61% against 87% with the Sonnet judge), so a collection of labels loses more than a single label does, and your point about a collection holds. Part of top-kk's stability comes from describing the top of the distribution. In the two main experiments, selected labels are also more specific than the best rejected candidate (false rate lower by 0.015 under random draw and 0.011 under top-kk); on the pilot arms with the new scorer, this difference is not distinguishable from zero. The revision reports this trade-off in Section 4.4 and Appendix C.6 and recommends reporting stability and coverage together.

Q3: ICC. ICC(3,1) is the share of score variance due to differences between features, after removing shifts shared by all features within a run (consistency). ICC(2,1) counts such shifts as error (absolute agreement). In Table 1, they agree within 0.001. The revision explains both, with reading bands, in Section 3.3, before Table 1.

Q4: "Set the sample size." This was wrong. The pilot was used to compare mitigations and to decide to run the full 150-feature list; 15 runs per feature were set by the compute budget. The revision corrects it (Section 4.1).

Q5: What instability is "moderate"? No convention exists, so we built two anchors with the same encoder and 15 labels per set: rewordings of one label (median 0.166, 90th percentile 0.338) and unrelated labels (median 0.892). Random-draw labels (median 0.604) sit 60% of the way toward unrelated labels, with 88% of features above the rewording 90th percentile; top-kk (0.328) sits 22% of the way, with 45%. This is a reference scale for the label metric, not an equivalence with an ICC band; the revision reports it in Appendix C.7 and cites it in Section 4.2.

Q6: Greedy decoding with fixed examples. There was no technical obstacle; we had simply not run it. On the 50 pilot features ×\times 10 runs:

ConditionMeanNear-unanimous (instability <0.05< 0.05)
Random 5 of up to 400.6061 / 50
Random 5 of the top 100.4642 / 50
Top-kk0.3191 / 50
Top-kk + temperature 00.07325 / 50

Greedy decoding cuts top-kk instability by 0.246, with 98% of features lower: with fixed examples, the explainer is the main remaining source, as our attribution predicted. In 17 of the 50 features, every run returns the identical label. We also measured coverage for this arm: a single label covers as much as under sampled top-kk (0.358 vs 0.366; false rate 0.049 vs 0.052), so greedy decoding removes variation without narrowing the label. Ten runs together cover less (49% vs 70%) because they yield fewer distinct labels (2.0 vs 7.8). The revision reports this in Appendices C.1 and C.6 and Figure 4b, cites it in Section 4.4 and revises Finding 3. It stays out of Table 2 because these runs use a different scorer.

AuthorsOpenReview ↗

Follow-up on Q2/Q6 and Q1: coverage on held-out text, and one qualification

Q2/Q6. We now judge labels on held-out text that no run could have shown. A judge marks windows of fresh FineWeb text in three activation strata, plus topically similar and random inactive windows, with matched run counts and two judges (Appendix C.9, Table 24). Paired differences in single-label coverage (Haiku / Sonnet):

ComparisonCoverageTop stratumUnion of a feature's labelsInstability
Top-kk −- random draw−0.055-0.055 / −0.048-0.048−0.067-0.067 / −0.061-0.061−0.194-0.194 / −0.160-0.160−0.273-0.273 / −0.285-0.285
Greedy −- sampled top-kk+0.009+0.009 / +0.010+0.010+0.004+0.004 / +0.019+0.019−0.149-0.149 / −0.105-0.105−0.260-0.260 / −0.228-0.228

On held-out text, the top-kk trade-off is somewhat larger than on the stored contexts. Top-kk labels cover less even of the strongest activations, and they mark less inactive text, so they are narrower rather than less accurate. Greedy decoding leaves single-label coverage unchanged, and the interval rules out a loss above about 0.02 under either judge. Its labels are reproducible without being narrower.

Q1 (a qualification). Our response said a feature's score variation does not track its label variation. This holds for the published pipeline. When the label is instead chosen by selection AUC, and all held-out AUCs come from one newer scorer, a weak positive association appears (r=0.18r = 0.18 and 0.220.22, intervals below 0.37; Appendix C.10). The revision says "at most weakly tracks". The point of the comparison stands: score variation explains at most about 14% of label instability, so a stable score cannot certify a stable label.

I thank the authors for clarifying and revising the differences between their experiments and those of van der Ben et al. (2026), adding a coverage metric alongside label stability, ICC metrics, the role of the pilot runs, a reference to interpret the range of label instability, and additional greedy-decoding experiments. All the revisions and answers address all my concerns. I have updated my score accordingly.

Reviewer_NGoxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the “interpretability lottery”: repeated automated interpretation of a fixed SAE feature can produce semantically different descriptions even after evaluation and selection. Experiments on Gemma-2-2B features examine label instability, score reliability, and feature retrieval. The authors find that score variation does not track semantic disagreement and that deterministic top-k example selection roughly halves label instability, with the improvement replicated on a disjoint feature set.

Critical strengths
  1. This paper serves as a useful investigation of semantic reliability. By holding the model, SAE, and feature fixed, the study identifies variation in the selected descriptions themselves. The retrieval analysis further demonstrates that this variation changes which features users encounter through semantic search.

  2. The writing and presentation are overall understandable and easy to follow.

Critical weaknesses
  1. The paper does not test whether enlarging the corpus used to collect feature activations or using more capable models to generate and evaluate descriptions substantially reduces instability. Consequently, it remains unclear how much the reported lottery is driven by the particular data and model choices of the tested pipeline. Without these comparisons, the paper does not sufficiently establish whether the instability persists as a substantial practical concern under stronger interpretation settings. Did you try these settings (larger corpora for activation collection and more capable explainer/scorer models, e.g., opus 5.5, gpt 6) before?

  2. No unsteered baseline is described (Appendix C), so the score may reflect ordinary prompt continuation rather than the feature’s causal effect. This weakens its justification for selecting descriptions.

Questions for authors

see weaknesses

4 comments

Thank you for the review. Both weaknesses led to new experiments (50 pilot features, 10 runs per arm). For W2, the revision documents the missing control and shows that the instability findings survive without the steering score.

W1: Larger corpus and stronger models. We had not tried either setting before submission, but we have now tested both. For the stronger model we used Claude Opus 5.5, since we did not have access to GPT-6.

Stronger models (Claude Opus 5.5; Appendix C.3, Figure 9):

  • Opus as explainer lowers instability from 0.606 to 0.461 under random draw and from 0.319 to 0.226 under top-kk. Still, 0 of 50 features are near-unanimous under random draw, and 2 of 50 under top-kk.
  • Opus as explainer+scorer (25 features) does not reduce instability: 0.485 vs 0.474 with the Haiku scorer, and 0.287 vs 0.245 under top-kk. Neither difference is distinguishable from zero.

Larger corpus. The paper's contexts come from Neuronpedia's pool. We streamed FineWeb through Gemma-2-2B and kept each feature's 40 strongest windows after 10M and after 200M tokens of the same stream, so the two pools differ only in how much text was searched (Appendix C.5).

  • Random draw: 0.593 (10M) vs 0.517 (200M), a drop of 0.075.
  • Top-kk: 0.296 vs 0.324 (49 features), so no gain.
  • Neuronpedia's pool does not differ detectably from our 10M pool (differences of 0.013 and 0.024).

Stronger settings shrink the lottery but do not remove it. Ranked by effect: top-kk examples 0.18–0.30, a stronger explainer 0.09–0.15, a larger corpus at most 0.08, and no reduction from the stronger scorer. Our strongest sampled configuration, the Opus explainer with top-kk examples, still leaves 48 of 50 features above the near-unanimity threshold, and the larger corpus adds nothing under top-kk.

W2: No unsteered baseline. The stored steering score mostly reflected ordinary prompt continuation, not the feature's causal effect. We generated unsteered continuations for all 41,600 steering prompts in both main experiments and judged them identically.

  • The intervention was nearly inert. It adds the unit-norm decoder direction at magnitude 1, against a median layer-12 residual norm of about 154 per token, so the push is 0.65% of the residual. On 300 sampled steering prompts, 76% of steered continuations are identical to the unsteered ones, and the rest differ only slightly.
  • About 93% of judge replies gave a number plus an explanation, which our parser rejected, falling back to a word-overlap score.
  • Steered and unsteered continuations score the same: 0.425 vs 0.426 under random draw (lift −0.0014-0.0014) and 0.427 vs 0.428 under top-kk (lift −0.0008-0.0008).

The instability findings, however, do not depend on this component:

  1. Reselecting every label by AUC alone changes 34% and 32% of labels, but median instability moves only from 0.604 to 0.585 (random draw) and from 0.328 to 0.285 (top-kk). The ordering of the four pilot arms (random draw, top 10, top-kk, top-kk with greedy decoding) is also unchanged.
  2. A corrected steering score (4×4\times the feature's maximum activation, an equal-norm control direction, the judge's own number) shows that Neuronpedia's labels get a specific lift of 0.108, against 0.021 for the pipeline's candidates (50 random-draw features). Reselecting with it gives the same instability as AUC-only selection (−0.030-0.030 vs −0.031-0.031), and the two rules pick the same label in 86% of runs.

The revision states that the confidence score does not validate labels causally (Appendix E), flags the steering rows of Table 6, and adds Appendix B.4 with the baseline, AUC-only selection and the corrected score.

AuthorsOpenReview ↗

Follow-up on W2: every retained claim rechecked under corrected selection

Our response rechecked the main contrast without the steering term. We have now rechecked every claim the paper retains (Appendix C.10, Table 25). We replayed two AUC-based rules over all stored candidates: AUC-only, and the highest selection AUC alone. The labels these rules publish had no held-out AUC, so every label was rescored with one scorer, the published labels included.

Claim (all held-out AUCs from one scorer)Published ruleAUC-onlyHighest AUC
Top-kk reduction in mean instability0.2730.2750.253
Largest source (one at a time and factorial)example drawexample drawexample draw
Mitigation rankingtop-kk > contexts > greedysamesame
Retrieval overlap@10 (chance 0.067)0.1860.2050.208
rr, held-out AUC SD vs instability0.130 [−0.032-0.032, 0.286]0.178 [0.017, 0.330]0.221 [0.062, 0.368]

The main contrast, the source attribution and the mitigation ranking hold under both corrected rules, and no mitigation is disqualified by the held-out AUC guard. Two claims change. Under the AUC-based rules, score variation shows a weak positive association with label instability, explaining at most about 14% of it. Retrieval overlap rises to about 0.21. The revision softens both statements accordingly.

Thanks for the response. I still have concerns on whether the instability persists as a substantial practical concern under stronger interpretation settings. The authors’ response shows that stronger explainers and larger corpora reduce the instability. As the explainers and corpora becomes more powerful and larger, the instability tends to achieve a practically negligible level, the paper’s claim that the lottery is a substantive problem would be substantially weakened. I keep my score.

AuthorsOpenReview ↗

The Practical Value of Measuring and Mitigating Interpretation Instability

We thank the reviewer NGox for the follow-up. The reviewer argues that stronger explainers and larger corpora reduce instability, and that further scaling could make it practically negligible. The concern, as we understand it, is that practitioners could eventually avoid the problem simply by using better models or more data, thus possibly reducing the value of studying the Interpretability Lottery problem, its causes, as well as developing specific remedies.

However, for this argument to weaken the paper’s practical contribution, the observed improvements must support a further conclusion: that the remaining semantic differences will become too small to materially affect how interpretations are understood or used. This is what “practically negligible” would mean here. Showing that an upgrade reduces instability establishes progress; it does not establish that continued upgrades will make the problem unimportant.

Specifically, we would argue for the following five relevant findings:

  • Scaling produces partial, configuration-dependent improvements. Tables 18 and 21 report mean instability:

    ChangeRandom evidenceFixed top-k evidence
    Haiku → Opus explainer0.606 → 0.4610.319 → 0.226
    10M → 200M corpus tokens0.593 → 0.5170.296 → 0.324
    Haiku → Opus scorer, with Opus explainer fixed0.474 → 0.4850.245 → 0.287

    The top-k corpus comparison uses 49 matched features; the scorer comparison uses 25. The stronger explainer helps. However, corpus expansion provides no detectable improvement under top-k, and the stronger scorer provides no detectable improvement under either policy. These results do not establish a general trend toward practical negligibility.

  • Evidence selection still matters with stronger models. With Opus as both explainer and scorer, changing the evidence policy alone reduces instability from 0.485 to 0.287. This benefit also replicates on a second SAE and a second source model (Tables 17–18). Strengthening the interpretation models therefore leaves a measurable effect of a pipeline choice that practitioners can control.

  • The study demonstrates consequences for feature retrieval. In the main study’s 150-feature index, repeated label indices share approximately two of their top-ten search results, including under corrected AUC-based selection (Section 4.5; Table 25). The same query therefore leads researchers to substantially different feature lists. Reviewer D7QQ specifically highlighted this connection to practical use. These findings establish downstream consequences within the evaluated setting.

  • Effective mitigation is part of the contribution. Fixed top-k examples with greedy decoding reduce instability from 0.319 to 0.073, with no measured loss in single-label coverage relative to sampled top-k (Tables 16 and 24). This result appears prominently in the abstract and Section 4.4. The paper identifies an effective control and evaluates its consequences; its contribution does not require instability to remain irreducible.

  • Reproducibility and semantic adequacy are evaluated separately. Candidate labels with nearly identical selection scores describe substantially different held-out passages (Appendix C.9). We therefore assess both stability and coverage, including the coverage trade-off from top-k selection. After these additions and the fixed-example greedy experiment, Reviewer UTNt explicitly confirmed: “All the revisions and answers address all my concerns.”

To summarize, further scaling may eventually reduce these effects to negligible levels. That remains an open empirical question. The observed improvements alone cannot tell us whether this will happen, under which configurations, or whether the resulting labels adequately represent the feature. Answering those questions requires additional evidence about the interpretations and their use.

We would argue that the possibility that future scaling might eventually make instability practically negligible does not establish that studying and mitigating the demonstrated problem now lacks practical significance. Using that possibility to discount the contribution would require explaining why the measured retrieval effects and tested controls have little practical value. A forecast of future improvement does not supply that explanation. We respectfully ask that the assessment distinguish the paper’s demonstrated findings from an unverified expectation about future systems.

We would kindly like to have all audiences carefully consider this nuance and evaluate our contributions fairly. We are happy to answer any remaining questions from the reviewer. Thanks so much!

Comments on the paper

2 comments

We thank the reviewers for careful reviews. Three requests came up more than once: broader settings (a second SAE and model, stronger explainers and scorers, a larger corpus), greedy decoding with fixed examples, and evidence that stable labels are also adequate labels. We ran about 11,000 new pipeline runs to answer them, and the revised PDF adds them in a new Appendix C (Figure 9 summarizes them). The lottery appears in every new setting, and top-kk example selection reduces it in every one.

Scorer. GPT-4o-mini was not available to us, so the new runs use Claude Haiku 5.5 as scorer, at the provider's default sampling (the paper's scorer ran at temperature 0.2). The explainer and the steering judge match the paper. Rerun with the new scorer on the 50 pilot features, the paper's two pilot conditions give median instability 0.665 under random draw (paper's pilot: 0.645) and 0.318 under top-kk (paper's pilot: 0.332). In the new runs, 169 of 10,900 (1.6%) were excluded because the scorer returned no usable probes, against 1 of 10,000 in the paper (Appendix C). All differences below are paired over features.

QuestionSettingRandom drawTop-kk
Second SAEGemma-2-2B, layer 20, 65k (60 features)0.5220.328
Second modelGPT-2 small, layer 6 (60 features)0.5280.316
Frontier explainerClaude Opus 5.5 (50 features)0.4610.226
Frontier explainer and scorerOpus 5.5 for both (25 features)0.4850.287
Larger corpusFineWeb, 10M vs 200M tokens0.593 vs 0.5170.296 vs 0.324 (49 shared features)
Greedy, fixed examplesExplainer temperature 00.073

Values are mean instability. The paper's SAE gives 0.606 and 0.319 on the 50 pilot features with the new scorer. Greedy decoding leaves what a single label covers unchanged (coverage 0.358 vs 0.366 for sampled top-kk), so its gain in reproducibility comes at no measured cost to the published label.

Corrections in the revision. From the new experiments, we found that the steering score was nearly inert and mostly a parser fallback (see the response to NGox). The revision documents this in Appendix B.4 and shows that the instability findings hold without it. We also corrected the description of the selection rule (candidates within 0.08 of the best confidence are re-ranked by a selection score that penalizes generic labels), the description of ablation, and the sentence saying the pilot "set the sample size", and we disclose the scorer change.

AuthorsOpenReview ↗

Follow-up: three further checks (revision Appendices C.9 and C.10)

We ran three further checks that the reviews point to, and added them to the revision. All use stored runs, so no pipeline run was repeated.

  1. Labels judged on held-out text (C.9). A judge marks which windows of fresh FineWeb text, never shown to any run, each label describes: top, mid and low activation strata for the 50 pilot features, plus topically similar and random inactive windows. Two judges are used (Haiku 5.5 on 50 features, Sonnet 5.5 on 25).
  2. Stability and coverage on matched runs (C.9). The same judgments for random draw, top-kk and top-kk with greedy decoding, using the same number of runs per feature in every arm.
  3. Every retained claim under corrected selection (C.10). Two AUC-based selection rules (AUC-only, and the highest selection AUC alone) are replayed over all stored candidates. Every label is rescored on held-out probes with one scorer, and each claim of the paper is recomputed. Where a run's own scorer was the rescoring scorer, rescoring reproduces the stored held-out AUC in all 1,945 runs.

What holds. Top-kk examples cut mean instability by 0.25 to 0.28 under every rule. The example draw remains the largest source, in both the one-at-a-time and the factorial designs. The mitigation ranking is unchanged, with top-kk first and none disqualified by the held-out AUC guard. Greedy decoding with top-kk examples leaves single-label coverage of held-out text unchanged (+0.009+0.009 [−0.011-0.011, 0.0300.030]).

What we soften. Under the AUC-based rules, a weak positive correlation appears between held-out AUC variation and label instability (r=0.18r = 0.18 and 0.220.22, intervals below 0.37). The paper now says score variation "at most weakly tracks" label instability, rather than "does not track". Retrieval overlap rises from 0.186 to about 0.21, so we now say "about a fifth" rather than "under a fifth". Neither change affects a conclusion that score variation still explains at most about 14% of label instability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.