- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes HSAE, which jointly trains 4 JumpReLU SAEs (2K to 16K latents) and links features of adjacent levels into a partial forest. It has 3 parts: (1) an L2 loss L_PC that asks each parent's decoder contribution to equal the sum of its children's contributions, (2) a random perturbation that replaces some non-root parents with their children during reconstruction, and (3) re-assignment of parents by encoder cosine every 5000 steps, with the bottom 20% of children left unassigned. On Gemma-2-2B layer 13 at L0 ~50, the 16K level keeps the variance explained of JumpReLU (0.710 vs 0.712) and gets the best scores on RAVEL, SCR and sparse probing. HSAE also has lower parent-child Hamming distance than post-hoc hierarchies (21.3 vs 36.1). An LLM judge rates 70.5% of 5000 sampled edges as "parent subsumes child", vs 60.5% for JumpReLU post-hoc and 67.1% for Tree SAE.
Critical strengths
-
The method is simple and each level remains a usable SAE. At matched L0 reconstruction does not drop, so the gains on RAVEL, SCR and sparse probing over framework-controlled baselines are useful if they hold across seeds. Also, the activation metrics show the same pattern on 3 layers of Gemma-2-2B, 1 layer of Qwen3-4B and 3 L0 values.
-
The paper is careful about what an edge means. It separates functional substitution, activation containment and semantic subsumption, it does not overclaim. Besides, the LLM judge is checked against human labels (kappa = 0.72).
Critical weaknesses
-
The evidence for a semantic hierarchy is weak. Hamming distance and P(P|C) closely track the loss, and Sec. 4.2 itself says they are not an independent test of a semantic hierarchy. L_PC directly penalizes a parent firing without children and a child firing without its parent, so these metrics should improve even on data with no hierarchy. Also, with 1 active child, L_PC is 0 only when d_child points in the same direction as d_parent. So the loss pushes children to copy the parent direction, not to specialize it (Park et al., ICLR 2025). The main support is then the LLM judge, but it has no reversed-order test and no random-pair baseline. Since child top examples likely overlap with the parent's, a "yes" may only reflect this overlap.
-
It is unclear where the SAEBench gains come from. These gains are the strongest result of the paper. But the ablations report only Hamming distance, plus variance explained in Fig. 5, and none of them reports RAVEL, SCR or sparse probing. Also, no result reports seeds or error bars, while the gaps over JumpReLU are small (0.02-0.04). So we cannot tell if the gains come from L_PC, the perturbation, the reassignment, or run-to-run noise. In addition, the paper should explain more clearly what it adds over Tree SAE, which also combines a reconstruction condition with dynamic parent assignment. I treat Tree SAE as concurrent work, so this is a clarity point, not a novelty penalty.
Questions for authors
- What is the parent-child decoder cosine distribution for HSAE vs post-hoc hierarchies? How many edges are near copies (cos > 0.9)?
- What is the judge's "yes" rate when parent and child are swapped? What is the rate for random same-level pairs and for sibling pairs?
- Can you report Table 1 over 3+ seeds, and also RAVEL, SCR and sparse probing for the ablations (constraint only, perturbation only, fixed tree)?
- How many parents have children in each method? Does the ranking change if Hamming distance is normalized by this number, or if the post-hoc baseline assigns parents by co-activation?
Clarification:
- Beyond the loss form and the 20% filter, what is the main difference from Tree SAE?
We thank the reviewer for their detailed review and concrete suggestions for strengthening the semantic evaluation and comparisons with related methods.
Q1
We provide the distribution of parent–child decoder cosine similarities in Appendix E.2 (Figure 12) of the revised version. For HSAE, 48.7% of pairs have a cosine similarity greater than 0.9. For JumpReLU SAE, this rate is 19.2%. However, these cosine > 0.9 pairs does not necessarily mean that they are near copies. For these pairs, the mean activation-support Jaccard similarity is only 0.6, and the child activates with a probability of approximately 62% when its parent is active. This is more consistent with the child being a more selective version of its parent feature than an identical copy.
W1/Q2
We provide additional results for reversed-order, random same-level, and sibling pairs in Appendix C.2 of the revised version. Random pairs (0.4%) and sibling pairs (23.2%) have much lower acceptance rates than pairs in the assigned direction (71.6%). The reversed-order acceptance rate remains relatively high (42.7%). We suspect that this partly reflects the limited set of 15 examples used for each judgment: the sampled parent and child examples may share substantial overlap.
Q3
We have updated Table 1 in the revised version to include the mean results over 4 training seeds and their standard deviations. HSAE gains a clear advantage in RAVEL (0.661 vs 0.636; HSAE SD: 0.005) and Sparse Probing (0.893 vs 0.874; HSAE SD: 0.004), while maintaining the lead in mean SCR (0.339 vs 0.332; HSAE SD: 0.013).
We have also added Table 3 in Appendix B.2 of the revised version to report the SAEBench results of the ablation studies. The explicit constraint loss is the most effective component of HSAE. This is consistent with the improvement in parent–child relationships.
Q4
For HSAE, 99.99% of features outside the final level have at least one child feature. The corresponding proportion for JumpReLU SAE is 99.6%. Sparsity encourages the coarse SAE to allocate its limited feature budget to useful directions, which are therefore likely to remain represented, possibly through more specialized features, in the larger dictionary.
The ranking remains unchanged when we normalize the Hamming distance: HSAE's normalized distance is still approximately 40% lower than that of JumpReLU SAE + post-hoc. Replacing the post-hoc assignment rule with co-activation does not improve the Hamming distance; instead, it makes it slightly worse (36.1 → 36.4). This is similar to the HSAE hierarchy-update ablation, in which different assignment methods yield similar hierarchy quality.
Clarification: Difference from Tree SAE
We appreciate the reviewer's recognition of Tree SAE as concurrent work. HSAE was developed slightly earlier within the same period, and we regard the two approaches as contemporaneous contributions.
Both methods allow the hierarchy to change during training, but HSAE revises the hierarchy more actively. Tree SAE's dynamic allocation mechanism only reallocates dead child features according to parent capacities. In contrast, HSAE periodically recomputes assignments across adjacent levels, allowing active as well as inactive features to change parents as their representations evolve. Thus, HSAE treats the hierarchy itself as an object of continued optimization, rather than limiting reassignment to dead features. This broader rematching mechanism is a core distinction beyond the loss formulation and the orphan filter.