- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies co-activation patterns in sparse autoencoders and how they relate to concept hierarchies, separating two questions that are often mixed together. The authors show that a common rule for choosing the parent mostly comes down to comparing how many features each word activates. They prove that co-activation alone cannot identify which concept is the parent. On WordNet pairs with Llama-3.1-8B, the predicted direction mainly follows word frequency rather than the true hierarchy. Still, hierarchical pairs share more features than unrelated ones. The authors conclude that feature sharing is a useful signal, but co-activation does not reliably reveal hierarchy direction.
Critical strengths
- Co-activation is often used to read concept hierarchies off SAE features. Showing that the inferred direction largely follows word frequency is a useful warning for the interpretability community.
- The authors compare against frequency-matched random pairs and related word pairs (that are not hierarchical), and report confidence intervals and threshold sensitivity. They also work on isolating where the frequency effect comes from by swapping the feature-selection mechanism while keeping the SAE weights fixed.
Critical weaknesses
- Clarity: the paper is very hard to follow. Figure 1 is meant to give the intuition early on, but it is hard to understand without definitions that only come later. The text is dense and heavily hedged, with many terms never explained in plain words ("non-tied pairs", "support rule", "slot agreement", "regime contrast"). The main message, that the inferred direction mostly follows word frequency (if I understand correctly!), is simple but gets lost. It is not even clearly understandable from the abstract.
- Limited and mixed results: almost all of the main evidence comes from one SAE, trained on Llama-3.1-8B at layer 16, with a short check on two other Llama layers. The authors also test GPT-2 and Gemma, but there the pattern doesn't hold.
Questions for authors
- Based on these findings, what should practitioners who want to extract concept hierarchies from SAEs do differently, beyond comparing against frequency baselines?
4 comments
Reply to rebuttal
I thank the authors for their effort in improving the readability of this work, and for taking care of the entire list of terms I had highlighted. I also appreciate that they highlighted the "practical takeaways" of the work, as I had requested. As these were the weaknesses that I pointed out and the authors made an effort to address them, I am bumping my grade. Nonetheless, I want to clarify that I still remain a bit uncertain wrt how assess this work properly, as it is a bit far from my expertise.
Thank You for Spotting Our Core Message: Clearer Notes on Readability, Models, and Practical Steps
Dear Reviewer mp73,
Thank you sincerely for your thoughtful review, for recognizing the value of our warning and controls, and above all for capturing our main message so clearly in your summary. Frankly, your two-sentence summary stated our core point more clearly than our original abstract did! Looking back at both our initial submission and our November 13 reply, we realized we still leaned too heavily on shorthand and symbols, so we wanted to follow up clearly on all three of your points, including a new practical workflow in our November 17 revision.
1. Cleaning up readability from page 1 (Weakness 1) Our mechanism comes down to one simple fraction check: if animal activates 3 features, dog activates 4, and 2 are shared, comparing 2/3 against 2/4 cancels out the 2 shared features on top and simply picks whichever word activates fewer total features as the parent. We rewrote the Abstract, Introduction, and Figure 1 caption so this intuition comes first without forward-looking symbols or hedged prose, and we explained every technical term you flagged right where it first appears (when a pair gives a directional verdict vs. a tie, set-based rules, how non-hierarchical control pairs are scored, and the accuracy gap between common-parent and common-child pairs).
2. Why GPT-2 and Gemma look different, yet tell one consistent story (Weakness 2) Separating two takeaways resolves what looked like mixed results across models:
- Universal across all three model families: On Llama-3.1-8B (layers 8/16/24, 32K/128K), GPT-2-small, and Gemma-2-9B, conditional coverage reduces to raw active-feature count within 0.1% (60.8% vs. 60.8%; 46.0% vs. 46.1%; 49.4% vs. 49.4%), and in none of the three models does co-activation recover hierarchy direction (46.0% on GPT-2, 49.4% on Gemma, and 52.4% balanced accuracy on Llama).
- Why the frequency tilt differs across training recipes:
- Llama Scope (JumpReLU) uses a variable per-token threshold, so rarer words activate more features and get flipped below common children (25.3% vs. 79.1%). Freezing those Llama weights and switching only the inference gate to TopK cuts that 53.8-point gap to 6.4 points.
- GPT-2 (OpenAI TopK) was trained end-to-end with TopK=32. Across contexts, common words light up more varied feature subsets and accumulate larger union sets, so direction still tracks frequency, just tilted the opposite way (36.1% vs. 64.9%), and still fails overall (46.0%).
- GemmaScope (131K) hits feature-set saturation: words activate roughly 800 features on average (16x Llama's ~50), drowning relative size differences in noise (49.4%, coin flip).
3. What practitioners should do differently, beyond frequency baselines (Question 1) Prompted by your question, revised Section 7 lays out a concrete Two-Stage "Skeleton + Orientation" Workflow:
- Stage 1 (Use co-activation only for an undirected concept map): Check average feature counts upfront to avoid Gemma-style ~800-feature saturation, apply TopK selection at inference time on variable-gate SAEs to remove per-token count skew, and filter pairs via excess overlap against frequency-matched random words (where real WordNet pairs show a strong signal, Cohen's d = 0.91).
- Stage 2 (Orient parent vs. child with an asymmetric signal): Across 9 static internal statistics we tested, none beat chance on common-child pairs. Instead, on a frozen LLM + SAE, run asymmetric causal interventions by suppressing A's enriched SAE features to test whether B drops downstream vs. the reverse (reaching 99.1% accuracy across 540 new noisy overcomplete SAE runs in Table 12). Or, when training SAEs, build in tree or orthogonality priors (80%-100% under low-to-moderate noise, as in Tree-SAE, MP-SAE, and TaxoSAE).
Thank you again for showing us where our writing got in the way of a crisp, useful message. We hope the revised manuscript is much easier and more rewarding to read.
Thank You Very Much!: A Quick Note on Our New Follow-Up Comment
Dear Reviewer mp73, Thank you so much for re-reading our revision, recognizing the readability improvements, and raising your score! !!We truly appreciate your openness and fairness.
Response to Reviewer mp73: Clarity Revisions, Cross-Model Scope, and Practitioner Guidelines
We thank Reviewer mp73 for recognizing the practical value of our warning and our experimental controls (frequency-matched and related non-hierarchical pairs, CIs, threshold sweeps, and fixed-weight gate isolation).
Your understanding is spot on: parent–child direction read off SAE co-activation tracks word frequency (79.1% when rcorpus<1 vs. 25.3% when rcorpus≥1), whereas shared feature count reflects genuine semantic relatedness (d=0.91 vs. frequency-matched controls).
1. W1: Clarity—Abstract, Figure 1, hedged prose, and plain-word definitions Thank you for the diagnostic examples; we revised all of them:
|S_P|,excess overlap,reciprocal coverage): both panels share core features{2,3}(teal) and active-count product (3×4=4×3=12), keeping chance-corrected overlap identical, while changing only unshared features (gray) flips active counts from (3,4) to (4,3) and reverses which concept has more of its features covered.2. W2: Concentration on Llama-3.1-8B L16 and behavior on GPT-2 / Gemma We clarify scope across three levels (§6.6, §7, App. B.4 Table 8):
3. Q1: What should practitioners do differently, beyond frequency baselines? Beyond frequency baselines (predicting the higher-frequency word scores 66.1% vs. 61.1% for M1; balanced accuracy 50.0% vs. 52.4%), §7 gives four practices: