From Atoms to Trees: Building a Structured Feature Forest with Hierarchical SAEs

Sparse autoencoders (SAEs) have proven effective for extracting monosemantic features from large language models (LLMs), yet these features are typically identified in isolation. Feature splitting and cross-feature organization suggest that SAE dictionaries may admit useful coarse-to-fine structure, although splitting alone does not establish a semantic hierarchy. To study such structure, we propose the **Hierarchical Sparse Autoencoder (HSAE)**, which jointly learns a series of SAEs and the parent-child relationships between their features. HSAE strengthens the alignment between parent and child features through two novel mechanisms: a structural constraint loss and a random perturbation of eligible non-root features. At matched $L_0$, HSAE retains nearly the same variance explained as JumpReLU while achieving the best RAVEL, SCR, and sparse-probing scores among the evaluated methods, including clear gains over the framework-controlled H-SAE and Tree SAE reimplementations. Across layers and models, HSAE improves cross-level activation alignment over post-hoc hierarchies built from independently trained SAEs, while case studies and large-scale hierarchy judgments show coarse-to-fine semantic distinctions. Finally, we demonstrate two applications of the learned hierarchy: reusing parent descriptions for feature interpretation and steering broad concepts with coarse-level features.