acceptodds
Under review as a conference paper at ICLR 2027

MorphSAE: Reserved Sparse Slots without Automatic Component Recovery

Abstract

Understanding a neural network may require separating its activations into components with different roles. Standard TopK sparse autoencoders learn one sparse feature inventory, so they cannot ensure that every proposed group receives selected features. We introduce MorphSAE, an additive sparse autoencoder inspired by Morphological Component Analysis (MCA), a sparse-demixing method from the Compressed Sensing and sparse-representation literature. Each named dictionary has its own TopK budget, and the decoded components add to reconstruct the original activation. This design guarantees returned indices from every dictionary. In three matched comparisons, an optional cross-dictionary loss also reduced its sampled overlap statistic. In a controlled synthetic copy task, an explicit separator signal concentrates the task-relevant activation change in a controller dictionary. Compared with a selected conventional-SAE intervention, the combined named and supervised design improves target recovery while reducing collateral error. This demonstrates a useful named decomposition in this supervised synthetic task. A matched model with one shared budget performs similarly, so the quota alone does not explain the success. On natural activations, the tested quota provides no reliable reconstruction or unsupervised-specialization benefit. Because each decoded component remains in the original activation space, the architecture permits it to be passed into another MorphSAE. We hypothesize that future separator functions could support repeated decomposition into smaller, more interpretable dictionaries; this remains untested.

Primary area:
interpretability and explainable AI
Keywords:
Sparse Autoencoders, Mechanistic Interpretability, Dictionary Learning, Sparse Representation Learning, Morphological Component Analysis, Compressed Sensing, Multi-dictionary Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_F9LFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Inspired by morphological component analysis, morphoSAE splits an SE into named addictive dictionaries, each with its own top-K budget. The morphological component analysis adds an optional cross-dictionary cosine loss to reduce overlap between groups decoder rows.

Critical strengths

This paper talked in a very honest way, including a lot of declaration and disclosure about the performance. It also have good comparison designs, with careful separation of the allocation and geometries, reconstructions and functions for each of the parts. It included transparent statistics.

Critical weaknesses

However, it also have the issue for methodology has not been demonstrated of the benefit and the positive result does not supporting the paper's methodology which making the weak evidence basis and the efficiency of the methodology become a challenge. This paper is relatively harder to understand.

Questions for authors

In what concrete natural analysis would guaranteed returned indices change a conclusion? Can you show one, or a case of starvation under named-global at realistic budgets?

1 comment

We appreciate your recognition of the controls and honest reporting. Your request for a concrete use case helped clarify the contribution. We have revised the paper to include new matched experiments, simpler explanations and explicit limitations. All references below point to the revised paper.

W1 No demonstrated practical benefit or efficiency advantage

  • MorphSAE offers a way to study SAEs and LLM behavior: choose what a group should learn, protect its selection budget, and test what changing it does. This research use can matter despite reconstruction costs or information remaining elsewhere. AlignSAE already guides dedicated slots; our contribution tests whether reserved selections help under the same guidance (§§1–2, 6).
  • We added GPT-2 block 3 position correction. Eight learned features target activation changes caused by shifting position IDs while keeping words fixed. On 256 held-out articles and five paired seeds, MorphSAE recovers 91.85% versus 78.02% for equally guided named-global: +13.84 percentage points [95% interval: 12.10, 15.83], positive in every seed.
  • Both methods use the same groups, total budget and clean reference. We add decoded clean-minus-shifted coefficient differences, matching nonzero edits up to eight per token across 64 tokens. The same eight features are reused at every token. Recovery measures reduction in squared error of centered final-token prediction scores. A 100-unit error becomes 8.15 versus 21.98. This is neither accuracy nor a fraction of positional information.
  • This is a limited benefit. Position remains detectable elsewhere. Removing the controller leaves less of the position-shift effect for global. Factual recovery favors global: 50.27% versus 38.12% on 230/384 eligible contrasts. Global was still improving at the training cutoff.

See revised §§5.6–5.7, Tables 4–5 and Appendices D.3 and F.2–F.4.

We did not test deployment uses, other traits, runtime or memory savings or efficiency at smaller budgets: we compared learned selectors at a fixed budget. Simpler mean-difference or low-rank baselines were also not tested. Transferring correction patches between articles still gives 75.53% recovery for quota and 70.79% for global, making those alternatives important. Whether an SAE is needed remains open (Table 5; §6).

W2 The positive result does not identify the method's benefit

  • We agree that the original conventional-SAE comparison mixed structure, guidance, losses and intervention choices. It remains labeled confounded.
  • The new 72-run synthetic study matches groups, widths, initial weights, data, training and intervention rules within each objective. Only group-specific guidance gives a clear quota benefit: +0.923 percentage points [adjusted 98.33% interval: +0.604, +1.212]. Recovery restores the target-versus-competitor score gap, not accuracy. Edits consider all atoms and match changed-coefficient counts, capped at 32; active-code counts and patch norms are not matched.
  • The position study compares equally guided selectors with overlap loss and inactive-feature replacement off. Both receive 10,000 steps and the same two-learning-rate search. A separate norm-matching test retains the gain. This supports a benefit at that budget, not superiority after convergence: global was still improving.

See revised §§5.5, 5.7, Table 3 and Appendices C.1–C.3 and F.2–F.3 for the matched controls and remaining limits.

W3 Difficult presentation

We added definitions and a worked example before equations, including the quota's reconstruction cost. Section 5.7 explains the clean reference, eight reused features and recovery example. Table 5 explains which question each test answers. Appendix F provides formulas; F.2 specifies that SAE selection uses 32/128 development articles, while probes use all 128. These changes are in the revised paper.

Q1 What natural conclusion changes, or where is realistic starvation?

  • The position experiment changes a concrete conclusion about correction quality: at the tested budget, an eight-feature quota controller reverses a known position-ID effect more faithfully than an equally guided global controller. Both have total TopK 32, equal controller capacity and matched edit counts. It does not establish a new account of the model's native position mechanism.
  • Global returns 7.137 controller indices on average versus eight for quotas. This is partial underallocation, not evidence of complete group starvation. Selected indices can still have zero coefficients. Factual recall showed no first-branch allocation shortfall and favored global.
  • We did not isolate how much underallocation causes the correction gap: changing selection also changes what is learned. The experiment compares trained selectors; it does not prove a starvation mechanism.

See revised §§5.6–5.7 and Appendices D.3 and F.3.

Reviewer_71djOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This manuscript introduces the Morphological Sparse Autoencoder (MorphSAE), an additive dictionary-learning architecture inspired by Morphological Component Analysis (MCA) in compressed sensing. Standard sparse autoencoders (SAEs) typically enforce a single, global TopK activation constraint across a unified feature dictionary. In contrast, MorphSAE establishes dedicated, per-dictionary selection quotas (k_j), guaranteeing that every designated dictionary branch yields a fixed number of returned indices regardless of cross-group competition. The partial reconstructions from these sub-dictionaries are additively combined to reconstruct the network's activation vector.

Critical strengths
  1. Experimental Controls: The study cleanly isolates the effects of named group structures from the effects of per-group index quotas using the "named-global TopK" baseline.
  2. Methodological Honesty: The authors transparently report negative results alongside positive ones and avoid overclaiming capabilities. 3. High Reproducibility: The paper provides thorough tracking, including configuration hashes, pre-registration receipts, and execution completion matrices.
  3. Clear Theoretical Grounding: The paper clearly explains its roots in Morphological Component Analysis while defining where classical theory stops applying.
Critical weaknesses
  1. Incomplete Natural Language Evaluation Matrix: The natural language empirical program was halted after 45 of 64 planned execution runs.

  2. Confounded Interventions: In the synthetic copy experiment, the primary contrast against conventional SAEs blends multiple variables simultaneously, including dictionary structure, supervision targets, and intervention selection rules.

  3. Post-Hoc Evaluation Criteria: Several evaluation decisions, including the Stage 2 baseline package, natural composite thresholds, and the C2 transcript-dependency component clustering, were formulated post-hoc after inspecting initial results.

Questions for authors
  1. Convergence Stability: Since all 12 natural models stopped training before reaching a plateau, do you expect the reconstruction gap or lack of functional roles to change if they are trained to full convergence?

  2. Sparsity Matching: In the synthetic copy task, MorphSAE and named-global used different active feature counts (L_0). Does MorphSAE still beat conventional SAEs if intervention sparsity is strictly matched?

2 comments

We appreciate your recognition of the controls and transparency. We have revised the paper to include new comparisons, clearer explanations and remaining limitations. All references below point to the revised paper.

Presentation Assessment: Substantial Effort to Understand

We added definitions and a worked example (§3), a plain-language position explanation (§5.7), and a guide to each test (Table 5). We've also made smaller revisions in order to make the paper easier to digest.

W1 Incomplete natural evaluation matrix

  • All 64 models and 960 evaluations are complete. Affected runs use matched initial weights and hardware.
  • The completed comparison does not favor full MorphSAE. Its normalized reconstruction error (FVU; lower is better) is 0.023948 versus 0.021235 for conventional TopK. The paired difference is +0.002713 [95% interval: +0.000262, +0.007039]. All six seeds favor TopK; one poor run contributes much of the average gap.
  • Only 6/64 models plateaued. Finishing the matrix does not establish convergence.

See revised §4, §5.3 and Appendices A.3–A.4.

W2 Confounded synthetic interventions

  • We agree: the original comparison changed too many things to identify a quota effect. The revised paper labels it confounded.
  • We added 72 runs: three teachers × four SAE seeds × two selectors × three objectives. Within each objective, the methods share groups, widths, initial weights, data, training and intervention rules. Named-global uses the same groups and total budget, with global competition for selections.
  • Both methods edit the same number of nonzero coefficients, capped at 32. Only group-specific guidance shows a clear quota benefit: +0.923 percentage points [adjusted 98.33% interval: +0.604, +1.212]. Reconstruction-only and whole-output guidance show no clear advantage. Recovery measures restoration of the target-versus-competitor score gap, not accuracy. Active-code counts and patch norms were not matched.

See revised §5.5, Table 3 and Appendices C.1–C.3.

We also added guided GPT-2 position correction: 91.85% versus 78.02% recovery, a +13.84-point gain [95% interval: 12.10, 15.83], across five paired seeds and 256 held-out articles. Both methods use a clean reference and eight controller features, with matched nonzero edits up to eight per token across 64 tokens. Recovery measures reduction in squared error of centered final-token prediction scores; it is not accuracy or a fraction of positional information. Factual recovery and controller removal favor global, so this is a limited use case. See §§5.6–5.7 and Appendices D.3 and F.2–F.4.

W3 Post-hoc evaluation decisions

  • We agree. Baseline additions, functional thresholds, transcript grouping and synthetic uncertainty analysis were chosen with earlier results visible. Later runs cannot make those choices prospective.
  • We now label the affected findings exploratory. Earlier thresholds were not tested on fresh data: reruns reused evaluations, while position tests a different hypothesis. Zero of 12 complete passes does not mean every individual signal was absent.
  • Main-matrix analysis groups linked transcripts together. Continuation retains older grouping that may understate uncertainty; historical functional intervals are unchanged.
  • For position, the protocol and splits were fixed before SAE tuning and final outcomes, after earlier motivating results. This was a local freeze, not external preregistration.

See revised §§4, 5.4, 6 and Appendices A.1, A.4–A.6, B.1, C.2 and F.1–F.2.

Q1 Would convergence change reconstruction or functional roles?

Possibly. We extended 12 quota-only/global models from 150M to 500M repeated exposures, reusing the cache and restarting Adam identically. All met the plateau rule; the final reconstruction-error difference remains near zero: +0.000021 [95% interval: −0.000104, +0.000135].

This does not settle full-recipe convergence or later functional roles. Those tests were not run because continuation isolated quotas, with overlap loss and inactive-feature replacement off. Guided global was also still improving at 10,000 steps. See revised §§5.2–5.3, 5.7 and Appendix A.5.

Q2 Does the gain survive strictly matched sparsity?

The gain survives matching actual coefficient-edit counts. We did not match active code features (L0) or synthetic patch norms because the experiment controlled intervention size. That stricter question remains open; the separate position norm control does not settle it. See revised Table 3 and Appendices C.2 and F.3.

Reviewer_71djOpenReview ↗

Rebuttal Acknowledgment

Thank you to the authors for the remarkably thorough and transparent revision.

Completing the full 64-model natural matrix, running the 500M-exposure continuation, de-confounding the synthetic comparison (Table 3), explicitly labeling the exploratory post-hoc analyses, and adding the controlled factual and GPT-2 position experiments directly address all of my concerns. I particularly appreciate the intellectual honesty in reporting both the limits of quota-based selection (no reconstruction gain, unsupervised specialization failure, and lower factual recovery) and the specific conditional regime where guided slot reservation helps.

All of my methodological concerns have been resolved, and I am maintaining my positive rating in support of the paper.

Reviewer_eN6MOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

paper proposes morphsae, an additive sae that gives separate topk budgets to named dictionaries. it takes inspiration from morphological component analysis. architecture guarantees a fixed allocation of active features per dictionary and uses a cross-group loss to control geometric overlap. through synthetic and natural text experiments (pythia-160m) they show that while allocation and geometry can be controlled this doesnt automatically give unsupervised semantic separation. explicit supervision is needed to organize useful functional components.

Critical strengths

theoretical and empirical isolation of allocation geometry reconstruction and function is rigorous. by acknowledging that classical mca bounds dont trivially carry over to learned nonlinear encoders the authors avoid overclaiming identifiability. the named-global topk baseline is a clean way to ablate the effect of the quota versus the additive decoding.

Critical weaknesses

lack of a strong positive functional result in the unsupervised natural setting makes the immediate practical utility narrow. it currently serves better as an investigative probe than a replacement for standard saes. natural experiments are mostly restricted to a single block of pythia-160m and the models are noted as non-converged.

Questions for authors

have you considered whether specific data or sparsity regimes might allow the cross-group loss to guarantee functional separation in a nonlinear setting? did you try evaluating on much larger models to see if feature manifolds naturally decouple more cleanly?

1 comment

We appreciate your recognition of the matched control and careful theoretical limits. Your questions helped sharpen the contribution. We have revised the paper to include the new experiments, clearer explanations and remaining limitations. All references below point to the revised paper.

Presentation assessment: substantial effort to understand

We added definitions and a worked example before the equations. We also explain the position experiment in plain language and separate what each test measures. See §3, §5.7 and Table 5.

W1 Missing unsupervised benefit and narrow practical utility

  • The research goal is to choose what a dictionary group should learn, protect its selection budget, and test how changing it affects LLM behavior. This can be useful even with worse reconstruction or information remaining elsewhere. Guided SAEs already exist, including AlignSAE; our contribution tests whether reserved selections help under the same guidance (§§1–2, 6).
  • We agree that unsupervised natural-text usefulness remains unproven. None of 12 evaluations across nine trained states passed every role criterion. Some individual signals were present (§5.4; Appendix B.1).
  • We added a guided position experiment in GPT-2 block 3. Eight learned features target changes caused by shifting position IDs while keeping the words fixed. Across 256 held-out articles and five paired seeds, recovery is 91.85% versus 78.02% for equally guided named-global: +13.84 percentage points, with a 95% interval [12.10, 15.83]. Named-global has the same groups and total budget but lets all groups compete for selections.
  • Both methods use a clean reference and change equal numbers of nonzero coefficients: up to eight per token across 64 tokens, using the same eight features. Recovery measures the reduction in squared error of centered final-token prediction scores. It is not accuracy or the fraction of positional information stored in those features. The gain remains when the decoded changes have matched magnitudes.
  • The benefit is conditional: global was still improving at 10,000 steps. Position remains detectable elsewhere. Removing the controller leaves less of the position-shift effect for global. Factual recovery also favors global: 50.27% versus 38.12% on 230/384 eligible contrasts. These outcomes limit broader claims.

The revised paper gives the experiment and controls in §§5.6–5.7, Tables 4–5 and Appendices D.3 and F.2–F.4. Other traits, downstream uses and simpler mean-difference or low-rank baselines were not tested: we focused on the matched selector comparison. Their value remains open (§6).

W2 Narrow model and layer coverage

  • We added Pythia-160M block 7 and Pythia-1B block 7, with three paired seeds each and equal training/tuning opportunities. Neither shows a clear reconstruction advantage from quotas.
  • These additions measured reconstruction and selection counts. They did not test functional separation across models or layers. GPT-2 adds a family for guided intervention, but broad functional generalization remains untested.

See the revised §5.2, Table 2 and Appendix A.5.

W3 Nonconverged natural models

  • We extended 12 quota-only/global models from 150M to 500M training exposures, reusing the same data and restarting Adam identically. All met the stated plateau rule. The final reconstruction-error difference is +0.000021, with a 95% interval [−0.000104, +0.000135]: no clear advantage.
  • A plateau means slow change, not a proven optimum. We did not extend the full recipe or repeat functional tests because this continuation isolated the selection rule, without overlap loss or inactive-feature replacement. Only 6/64 original-budget models plateaued; none was full MorphSAE.

See the revised §§5.2–5.3 and Appendices A.3 and A.5 for the stopping rule and scope.

Q1 Can data/sparsity regimes guarantee functional separation?

We have not established such a regime or proof. Lower overlap alone cannot ensure that groups learn different intended roles. Classical MCA guarantees require assumptions about dictionaries and sparse representations that we have not established for learned nonlinear encoders. We did not run a targeted regime sweep: our tests separated the effects of quotas and overlap. Revised §2 gives a counterexample and explains what a stronger guarantee would require.

Q2 Do larger models decouple more cleanly?

We tested the larger Pythia-1B model, but measured reconstruction and allocation, not spontaneous functional separation. Guided GPT-2 teaches a role, so it cannot answer that question either. A functional scaling study was not run; the added sites addressed reconstruction coverage. See revised §5.2, Table 2 and §5.7.

Comments on the paper

1 comment

We thank the reviewers for their feedback. We completed the missing evaluations, made the intervention comparisons fairer, and added a guided position-correction experiment. The revised paper reports both favorable and unfavorable results. Our contribution is an interface for specifying a role for a feature group, protecting its selection budget, and testing whether it performs that role. Guidance supplies the target; held-out interventions assess the result. Broader generalization and complete concept isolation remain open (§§1–2, 5.7 and 6).

What changed in the paper, and where to find it.

  • Completed natural evaluation. We replaced the incomplete evaluation with all 64 models and 960 checkpoint/split evaluations. Section 4 and Table 1 distinguish the quota-only test from the full recipe. Section 5.3 reports the full recipe's reconstruction disadvantage against conventional TopK: +0.002713 FVU [+0.000262, +0.007039], where higher is worse. Appendices A.3–A.4 document the initialization/hardware corrections, completion and paired estimates.

  • Longer training. We added the 12-model continuation from 150M to 500M repeated training exposures. Section 5.2 and Table 2 report no clear quota-only reconstruction advantage after all 12 meet the operational plateau rule. Appendix A.5 gives the continuation procedure. Sections 5.3 and 6 explicitly distinguish this result from convergence of the full recipe, which remains unestablished.

  • Broader model and layer coverage. We added Pythia-160M block 7 and Pythia-1B block 7, with three paired seeds at each site. Section 5.2 and Table 2 report no clear reconstruction advantage at either site; Appendix A.5 gives their setup. We explicitly state that these additions test reconstruction and allocation, not functional separation across models.

  • Controlled synthetic interventions. We added 72 training runs that compare selectors under the same three supervision conditions and match the number of changed coefficients. Section 5.5 and Table 3 show a clear benefit only with explicit branch supervision: +0.923 pp [+0.604, +1.212], using an adjusted 98.33% interval. Appendices C.1–C.2 specify the controls and state that activation-code L0 and patch norms were not additionally matched. Appendix C.3 retains the older bundled comparison as historical evidence, rather than evidence identifying a quota effect.

  • Post-hoc choices and unsupervised findings. We revised Section 4 and Appendix A.6 to identify the result-aware baseline package, thresholds and transcript grouping. Section 5.4 and Appendix B.1 explain that 0/12 historical evaluations, covering nine trained states, passed the full role criterion; this does not mean every functional signal was absent. Section 6 states that corrected historical evaluations reuse inspected holdouts and remain exploratory.

  • Factual intervention and practical benefit. We added the corrected country-capital comparison in Section 5.6: MorphSAE recovers 38.12% of the target-margin effect versus 50.27% for named-global on 230/384 eligible contrasts. That section also reports no observed first-branch allocation shortfall in the monitored setting. Appendices D.1–D.3 disclose the replacement task, repairs, eligibility and allocation counts. The paper therefore does not claim a practical benefit from this experiment.

  • Guided position correction. We added Section 5.7 and Table 4, reporting 91.85% recovery versus 78.02% for equally guided global selection: +13.84 pp [12.10, 15.83] across five paired seeds. The section explains the eight atoms reused at each token, per-token edit budget, clean reference and squared-logit recovery definition beside the result; it also reports the secondary new-offset and norm-matched checks. Appendices F.1–F.3 give the data splits, guidance, training, patches and estimator.

  • What the position result does and does not establish. Section 5.7 and Table 5 separate correction, content swap, readout, removal and shuffled-patch tests. Appendix F.4 gives the supporting controls, including position remaining outside the controller and the removal result favoring global. Sections 5.7 and 6 retain the improving global baseline and untested simpler correction baselines. We explicitly state that recovery is neither accuracy nor the fraction of all positional information stored in eight atoms.

  • Clearer contribution and explanation. We revised the abstract and Sections 1, 6 and 7 to describe specifying a role, protecting its selection budget and testing its interventions, while leaving generalization and complete isolation open. Section 2 credits prior supervised dictionaries and AlignSAE, then identifies our comparison of reserved TopK budgets under common guidance. Section 3 now defines the terms and provides a worked numerical example before the equations; Table 5 explains the distinct experimental questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.