- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper looks at the labels of features that received their meaning through autointerpretability. They show that this meaning can depend on choices in the pipeline, such as which activating examples are shown. The authors find high label instability for all features. Additionally, they show a weak association between label instability and score variability. Their results suggest that the example draw is a dominant driver of label instability. As mitigations, they test greedy decoding, more examples, top-k and aggregations, where top-k halves instability with minimal AUC change.
Critical strengths
The paper is well-motivated and tackles a relevant problem of using LLMs as judges. It is a critical issue: semantic uncertainty hidden by single-run autointerp labels. The paper shows the results for one pipeline and one model very well. A strong contribution is the repeated-run measurements (15x per feature), as well as the semantic vs score dissociation. The separation of semantic variation and score reliability is strong, as they measure each separately (Section 3.3). This avoids confusing stable scores with stable meanings. Section 4.5 ties their results to how practitioners use autointerpretability to label features, which shows the importance of their research.
Critical weaknesses
The main setting is quite narrow. Using a single-layer, single-source model and single (relatively small) SAE. Using a single judge and scorer. Holding one layer fixed can be motivated, but generalisability of results would require additional SAEs and the potential influence of the judge and the scorer.
Top-k strongest context seems to be likely to bias a label towards a high-activation context, rather than just the feature’s potentially broader meaning. The consequences of this interpretation should be mentioned more clearly
Greedy decoding is not tested under fixed examples, only when they are random, which weakens the mentioned outcomes about greedy decoding (see section 4.4). This contradicts the fact that in real pipelines, examples are often fixed and not random.
Probes are written based on the candidate label. This could push vague or general labels. The relationship between stable AUC and unstable semantics is not addressed deeply enough.
Questions for authors
- Could you expand your analysis to at least another SAE on a different model?
- Do you have any experiments with alternative explainers or scorer models? Maybe even small feature subsets, just to see if the instability is similar.
- What is the meaning of selection AUC in the appendix table 6?
- “Features whose contexts are more varied get less stable labels”, section 4.5. this is attributed to the context of the activating samples, which makes sense in the experiment itself. However, some features might activate on more diverging contexts. How could this be related to polysemanticity?
- Could you test greedy decoding under fixed examples? Like temperature 0 for the explainer under fixed top k examples? This would be a good addition to Table 2
- Contexts are chosen and then 5 candidate labels. How does the instability change with more contexts and candidates or maybe fewer? Is there any sensitivity related to the number you picked? Nice pipeline in B1.
3 comments
Follow-up on W4 and Q6: which selection rule gives better labels, judged on held-out text
We compared three rules on the same stored candidates: the explainer's first candidate, the candidate with the highest selection AUC, and the published rule. A blind judge marks which windows of held-out FineWeb text each label describes (Appendix C.9, Table 24). Paired differences in single-label coverage, Haiku (50 features) / Sonnet (25 features):
| Arm | Comparison | Haiku | Sonnet |
|---|---|---|---|
| Random draw | highest AUC first | [0.027, 0.076] | [0.019, 0.062] |
| Random draw | published highest AUC | [, ] | [, ] |
| Top- | highest AUC first | [, 0.025] | [, 0.033] |
With random examples, the highest-AUC candidate describes more of the held-out text than either alternative. On W4: it also marks slightly more topically similar inactive text under Haiku (), but not under Sonnet (), so we find no consistent sign of vaguer labels. With top- examples, the rules give equally good labels, and the first candidate is far more stable. Candidates whose selection AUCs are within 0.02 of each other cover largely different held-out windows (Jaccard overlap 0.18 to 0.31), so a near-tie in score is not a near-tie in meaning.
Reviewer response
Hi, thanks for the concise response.
W1 and W2 are fixed. Results of W2 are interesting.
W3. Thank you for testing this. Conclusions are interesting, “greedy decoding removes variation without narrowing the label”.
All other questions are answered. I will keep my score.
Thank you for the review.
W1/Q1: Another SAE and model (60 features each; Appendix C.2, Figure 9):
No feature is near-unanimous under random draw in either new setting. Without features whose median held-out AUC is at most 0.55 (36 and 47 remain), the reductions are 0.18 in both.
W1/Q2: Other explainers and scorers. An Opus 5.5 explainer lowers instability by 0.145 (random) and 0.093 (top-k) but leaves no feature near-unanimous under random draw. With the Opus explainer held fixed, swapping the scorer to Opus 5.5 gave no reduction on 25 features (0.485 vs 0.474 under random draw, not distinguishable from zero). The lottery thus appears with three scorers and two explainers. Reselecting each run's label by AUC alone, which takes the steering judge out of selection, barely changes instability (median 0.604 vs 0.585).
W2: Top-k bias toward high-activation contexts. We agree this is an important caveat, and have now measured it. A judge marked which of a feature's stored contexts (up to 40) each label describes. Compared with random-draw labels, top-k labels cover the five strongest contexts far better (0.597 vs 0.377) but the remaining contexts worse (0.241 vs 0.308); a second judge agrees. Part of top-k's stability therefore comes from describing the top of the activation distribution, as you suspected. For a single label the overall difference is modest (0.290 vs 0.316), but the labels from 15 runs together cover 67% under top-k against 88% under random draw, so the loss is mainly across the collection of labels. Section 4.4 now states this trade-off.
W3/Q5: Greedy decoding with fixed examples. With top-k examples and explainer temperature 0, instability falls from 0.319 to 0.073. 25 of 50 features fall below the near-unanimity threshold (instability below 0.05), and in 17 every run returns the identical label. A single label covers as much as under sampled top-k (0.358 vs 0.366; false rate 0.049 vs 0.052), so greedy decoding removes variation without narrowing the label. Ten runs together cover less (49% vs 70%) because they yield fewer distinct labels (2.0 vs 7.8). The new experiments show that our greedy conclusion applied only to random examples. The revision reports this in Appendix C.1 and Figure 4b, and revises Finding 3 (not Table 2: different scorer).
W4: Probe-based scoring and vague labels. We measured vagueness as how often a label is judged to describe other features' contexts. Pipeline labels do so 6–7% of the time. In the two main experiments the selected label has a lower false rate than the best rejected candidate (by 0.015 and 0.011); on the pilot arms with the new scorer the difference is not distinguishable from zero. So we find no evidence that selection favors vaguer labels, though this does not isolate the AUC term alone. Stable AUC with unstable semantics follows from how AUC is computed: a label is tested on probes written for that same label, so two different labels can each score well.
Q3: Selection AUC. Selection AUC is the AUC of a candidate label on the probes the scorer wrote for that candidate during selection, and it carries weight 0.60 in the confidence score. Held-out AUC is computed on a fresh probe set written after selection, so it plays no part in choosing the label. The revised Table 6 caption defines both.
Q4: Polysemanticity. Varied contexts fit polysemanticity, but also a single feature that fires across topics; the feature in Figure 1 is compatible with either reading. If the draw exposed polysemanticity, runs shown similar contexts should get similar labels, most strongly for diverse features. Within features, context-set similarity and label similarity correlate only weakly (ρ=0.097, against 0.001 when labels are permuted), with no rise across diversity terciles (0.088, 0.106, 0.097; Appendix C.8).
Q6: Contexts and candidates (50 features; the baseline of 5 and 5 gives 0.606):
More contexts help steadily (partly because larger draws overlap), and fewer than 5 barely matters. Requesting one candidate lowers instability by 0.166. A replay that holds generation fixed attributes most of this to selection: publishing the explainer's first-listed candidate from the same five-candidate runs gives 0.484 against 0.606 for the pipeline's choice, and only 0.045 remains between that rule and the one-candidate arm. So instability is sensitive to both numbers, and our choice of 5 candidates sits on the plateau. Both sweeps and the replay are in Appendix C.4.