acceptodds
Under review as a conference paper at ICLR 2027

DIMENSIONALITY EXPANSION IMPROVES SPARSE FEATURE LEARNING IN NEURAL NETWORKS

Abstract

Sparse dictionary learning is the standard tool for decomposing neural network activations into interpretable features, and every deployed pipeline must choose an expansion factor E, the ratio of dictionary width to activation width. That choice is made by convention rather than by measurement, because on real activations the ground-truth features are unknown and recovery cannot be scored. We have built 1024 features in superposition in R64 under a heavy-tailed frequency law. Across 153 training runs, recovery rises monotonically with expansion, from mean max cosine similarity 0.374 at E = 1 to 0.703 at E = 32, a 9.7× increase in the fraction of features isolated at a cosine threshold of 0.9. Over the same range normalised MSE improves by less than half, so tuning width by reconstruction error substantially understates what expansion buys. The relationship is affine in log2 E (R2 = 0.981) and a random-allocation null model is rejected. The benefit is strongly regime-dependent: it is largest when few features are co-active (0.457 MMCS gained at k = 2) and nearly vanishes when the support is dense (0.092 at k = 32), so width cannot rescue a problem that has left the identifiable regime. Finally we introduce Progressive Dimensional Expansion (PDE), which grows into a target width while recycling duplicated and dead capacity. At matched compute PDE improves MMCS over fixed-width training by 0.0073 (paired t(9) = 4.98, p = 0.0008) and recovery rate by 15.0% relative, using 95% of the floating point operations. An ablation shows the gain comes from capacity recycling rather than from the growth schedule

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Sparse dictionary learning, Sparse autoencoders, Dimensionality expansion, Feature recovery, Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_44MvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how dictionary expansion affects sparse feature recovery in synthetic data and proposes progressive dimensional expansion with residual-guided atom insertion and capacity recycling for more efficient sparse auto-encoder training.

Critical strengths

This paper shows dimensionality expansion could improve the sparse feature learning.

Critical weaknesses
  1. The novelty of method is limited, increasing the complexity of the network could improve the performance is straightforward.
  2. The contributions are only derived from the simulation experiments, lack of theoretical analysis.
Questions for authors
  1. Only from the experiment, is it enough to prove your recovery conclusion? It is a lack of theoretical analysis.
  2. Why are the research questions, especially Q2, Q3, important to be answered? What are the limitations of the baselines corresponding to these questions?
  3. From the results, compared with TopK fixed, the advancement of PDE is limited. What is the benefit of the proposed method? What is the cost of the dimension expansion in the proposed method?
Reviewer_Q13iOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the expansion factor in sparse autoencoders (SAEs), which are used to decompose neural network activations into interpretable features. The expansion factor E is the ratio of dictionary width to activation width, and the authors note that it is usually chosen by convention because ground-truth features are unknown on real activations and recovery cannot be scored.

To make recovery measurable, the authors use a synthetic setup with known generating features (1024 features in 64 dimensions, drawn under a Zipf frequency law) and run 153 training runs. They report three main findings. First, feature recovery improves with expansion, and does so much more than reconstruction error: MMCS rises from 0.374 at E = 1 to 0.703 at E = 32, while normalised MSE improves by less than half. Second, the benefit of expansion shrinks as more features are active at once and as noise grows. Third, they propose Progressive Dimensional Expansion (PDE), which grows into a target width while recycling duplicated and dead atoms, and report a small improvement over fixed-width training at matched compute (+0.0073 MMCS, +15.0% relative recovery rate). An ablation attributes the gain to capacity recycling rather than to the growth schedule.

Critical strengths
  1. Typically, the expansion factor E is chosen by convention. The authors consider E as a single variable, keeping the data, optimiser, sparsity mechanism and training budget fixed, so any change in recovery can be attributed to dictionary width and activation width.
  2. Since the 1024 generating features are known, recovery can be measured directly instead of through proxies. This is something that cannot be done on real activations.
  3. From E = 1 to E = 32, MMCS rises from 0.374 to 0.703, and the fraction of features recovered at cosine ≥0.9\geq 0.9 rises from 4.2%4.2\% to 40.6%40.6\% (about 9.7×9.7\times, though from a small baseline). Normalised MSE improves by less than half (0.326 to 0.208). The ReLU-L1 row in Table 1 (best NMSE, zero recovery) reinforces the point. This is a useful, well-illustrated caution against tuning width by reconstruction error alone.
  4. The benefit of expansion shrinks as the support gets denser (MMCS gain of 0.457 at k = 2 versus 0.092 at k = 32) and as noise grows. This agrees with compressed-sensing intuition and gives practitioners a concrete limit on what width can do.
Critical weaknesses
  1. The dataset is tiny and synthetic (d = 64, random dictionary, independent features, and the SAE is told the true sparsity). The paper acknowledges this cost to realism, but the title and abstract are worded generally, and no evidence is given that the findings or the proposed method carry over to real activations. I am not in a position to judge how limiting this is, so I raise it as a concern for a more expert reviewer.
  2. There is a broken cross-reference in Section 2 (“Section ??”).
  3. The conclusion states that PDE “recovers more features per unit of compute than simply training wider.” Table 1 shows that, at the same budget, a narrower width-1024 model trained for longer outperforms both PDE and the fixed-width baseline at 2048 (MMCS 0.678 vs 0.628 and 0.620). Section 6.5 explains that PDE’s claim applies to a given target width rather than to the choice of width itself, but the conclusion does not carry that qualification, and as written the sentence reads as a stronger claim than the results support.
  4. The author has written “Removing progressive growth, that is, training at full width throughout, improves MMCS by −0.034-0.034” in the ablation study; however, in Fig 4 (left) it looks like removing the progressing growth improves MMCS by 0.0340.034. Possibly a typo.
  5. The ablation reports that residual-guided insertion is not significant and the coherence penalty has no effect. Removing progressive growth makes accuracy better even though it costs about 40%40\% more compute. PDE is presented as a three-component framework, but one component is inert, one is not significant, and one trades accuracy for compute.
  6. The algorithms (Section 4.4) get one sentence of explanation. The values of the PDE settings (start width, number of growth rounds, recycling interval, and so on) are not given in the paper. LauxL_{aux} in Algorithm 1 is never defined.
Questions for authors

Clarification requests

  1. Table 1 vs. conclusion. The width-1024 model beats PDE at width 2048 (0.678 vs 0.628). Section 6.5 explains that PDE’s claim applies to a given target width, but the conclusion omits this. Should the claim be limited accordingly?
  2. Which components matter. The ablation supports only capacity recycling; residual-guided insertion and the coherence penalty show no clear effect, and removing growth improves accuracy at higher cost. Would you consider narrowing the framing to capacity recycling, and revising the abstract accordingly?
  3. Sign convention. The text says removing growth “improves MMCS by −0.034,” but Figure 4 shows an increase of about 0.034. Is this a sign typo?
  4. Hyperparameters. What values were used for start width, growth rounds, growth horizon, recycling interval, duplicate threshold and recycling budget? How sensitive are the results to them?
  5. Definition of LauxL_{aux}. What is the explicit form of LauxL_{aux} in Algorithm 1? Also, could you add a short walkthrough of Algorithms 1 and 2?
  6. What does the "Section ??" reference in Section 2 point to?

Suggestions for future work

  1. Do the trends hold at larger d, with correlated features, or when the SAE’s sparsity differs from the true sparsity?
  2. Is there any evidence on real model activations? If not, please state this limitation in the abstract.
Reviewer_rpLDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how sparse autoencoder dictionary width affects recovery of known generating features. The synthetic experiments contains 1024 random features directions in 64dimensions, with heavy-tailed activation frequencies. Using TopK autoencoders, the authors measure mean maximum cosine similarity (MMCS), feature recovery at a cosine threshold of 0.9, and normalized reconstruction error. The paper also proposes Progressive Dimensional Expansion (PDE), combining progressive width growth, residual-guided initialization of new atoms, and recycling of dead or duplicate atoms. Across ten paired seeds, PDE produces a small improvement over a fixed-width TopK baseline at approximately matched compute.

Critical strengths

The ablations are informative, and the paper reports results that qualify its own method rather than leaving them out. Removing capacity recycling is the only ablation that significantly degrades MMCS (p = 0.0013). The coherence penalty shows no detectable benefit (p = 0.75). At the common compute budget, a narrower fixed-width TopK outperforms PDE (Table 1). This transparency makes the empirical record credible and shows which components merit further study, even though the same results weaken the case for PDE as proposed.

Critical weaknesses
  1. The experiments do not isolate the benefit of progressive growth from recycling. Section 6.4 finds that recycling is the only component whose removal significantly hurts (MMCS, p = 0.0013). However, the compute-matched TopK baseline shares only the AuxK dead-latent loss, not recycling. The no-growth ablation keeps recycling and outperforms PDE by 0.034 MMCS, but it uses 40% more compute. Neither comparison answers the central question: does growth beat a fixed-width model with identical recycling under the same budget? Without this control, the Section 6.3 claim that growing into a width beats training at it is unsupported; the evidence shows only that the full package beats one baseline. The control is needed even under the authors’ restriction to a prescribed final width, a restriction that is itself necessary because width-1,024 TopK beats PDE at the same budget (MMCS 0.678 vs. 0.628; recovery 0.382 vs. 0.258). Since the headline saving is only 5%, the FLOP count should also explicitly include residual profiling and the full decoder Gram matrix in Algorithm 2.

  2. The scaling and identifiability claims are not separated from metric artifacts and limitations of the learning procedure. MMCS takes a maximum over n atoms, so it rises with width even for an untrained dictionary. For independent uniform unit atoms in R^64, the spherical inner-product distribution alone gives an expected MMCS of 0.346 at n = 128 and 0.435 at n = 2,048. The trained k = 32 curve matches this baseline at both ends (0.344 vs. 0.346; 0.436 vs. 0.435), so in the dense regime this metric cannot distinguish the trained SAE from a random dictionary. A matched-width random baseline is needed to interpret both this curve and the log₂ E fit. The recovery-at-0.9 gains are unaffected, since random atoms essentially never reach that threshold. Separately, poor recovery by one TopK SAE does not show that the problem has become unidentifiable. Falling outside a sufficient recovery guarantee is not evidence of non-identifiability, and the experiments do not separate encoder, optimization, finite-training and noise effects from an information-theoretic limit. The claims in Section 6.2 and the abstract should be narrowed to the tested procedure or supported by a direct argument. This matters more because all evidence comes from one small synthetic dictionary family, and Section 6.5 describes the fixed 3,000-step sweep as a result “at convergence” without a convergence check.

Questions for authors
  1. At the same final width and total compute budget, how does PDE compare with fixed-width TopK using exactly the same recycling procedure?

  2. Can the authors include matched-width random dictionary baselines for MMCS and explain how the conclusions change after accounting for this baseline?

  3. What numerical growth/recycling settings and actual training-step counts were used for each comparisons? Does the FLOP budget include all residual profilling passes and full Gram-matrix calculations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.