W1. The methodological contribution reduces to a known decoder orthogonality penalty. Stripped of terminology, the change to the SAE objective (Eqs. 1 and 3) is a penalty ||WD^T WD - I||F^2 on the decoder. Penalising the Gram matrix of a dictionary, including the cross-Gram between blocks that the paper calls basis separation, is long established in dictionary learning [1] and deep learning [2]. It has recently been studied for SAEs on LLMs [3], and orthogonal decoder initialisation was even available as a configuration flag in SAELens [4]. The other components also have direct precedents. Re-initialising dead latents towards poorly reconstructed directions corresponds to neuron resampling [5] and to atom replacement in dictionary learning [6]. Growing the dictionary during training corresponds to function-preserving network growth [7]. None of these is compared against, so the paper does not show what PARSE adds.
W2. The paper's description of current SAEs does not reflect the field. The problem is framed around SAEs trained with static L1 regularisation and warm-up schedules. The most widely used sparsifiers are instead TopK [8], BatchTopK [9] and Matryoshka SAEs [10], together with JumpReLU [11]. The paper mentions TopK and JumpReLU (Sections 1 and 4.5) only to dismiss them, and it compares exclusively against an L1 SAE. The motivation therefore concerns a largely superseded baseline and does not transfer to current practice.
W3. The dead-latent problem is presented as open although it is already addressed. Dead latents under L1 training are well known, and standard remedies exist, including neuron resampling [5], the AuxK loss of Gao et al. [8], and initialising the decoder as the transpose of the encoder [12]. None of these serves as a baseline. The paper supports the problem with Smith et al. [13], a post about SAEs underperforming linear probes on downstream tasks rather than about L1 schedules. A substantial fraction of latents also remains dead under PARSE itself (Table 2).
W4. The evaluation does not use the criteria by which interpretability methods are judged. SAEs are typically evaluated on faithfulness when the reconstruction is spliced into the model, on probing [14], on steering [15] and on causal interventions [16, 17], across several models and layers [18, 19]. The paper instead reports one layer of one model with reconstruction MSE, on which PARSE is clearly worse (Table 2). It adds the condition number of the decoder Gram matrix, L0, dead latents and anecdotal prompt completions. These geometric proxies do not measure interpretability. An untrained Gaussian dictionary of the same width has a lower condition number than every model in Table 2, and the mean cosine in Section 4.4 equals the value expected for random directions in 4096 dimensions.
W5. The standard SAE metrics and baselines are missing. Table 2 reports single points at different L0 values instead of the sparsity-fidelity frontier, so the comparison cannot be interpreted, and loss recovered is absent. The only baseline is a ReLU-L1 SAE without dead-latent treatment. The paper includes no TopK, BatchTopK, JumpReLU or Matryoshka SAEs with AuxK, no resampled ReLU SAEs and no SAEBench evaluation [12]. Training is short, and SAEBench, which the paper cites as justification, in fact reports continued improvement well beyond the budget used here. All results come from a single seed, although SAE features vary substantially across seeds [20].
W6. There is no related-work section. Positioning is limited to a few paragraphs in the introduction, and the bodies of work in W1 to W3 are not discussed. The theory and both toy models are deferred to an external document with the same title, so the paper is not self-contained and the contribution cannot be located relative to prior work.
W7. Central claims are assumptions presented as findings. The abstract states that static schedules force concepts into entangled representations and induce semantic hallucination, and that representation collapse is an artefact of static latent expansion. Neither statement is supported by a citation or an experiment. Semantic hallucination is also undefined for an SAE, whose output is a reconstruction rather than generated text. These premises motivate every component of PARSE, so the method inherits their lack of support.
W8. Vague, non-standard terminology makes the method impossible to follow precisely. The abstract states that orthogonal vectors are spawned when local semantic cross-talk exceeds the LLM's inherent noise floor. Semantic cross-talk is never defined, and spawning simply means appending latents. The noise floor turns out (Section 2.4) to be a cross-entropy that is compared with a Frobenius norm of residuals, two quantities in different units. Representation shock, geometric strain, matrix stiffness and alien minima are likewise used as explanations without definitions. Quantities that drive the algorithm (κ∞, L2,max, the update rule for Λ, γ, ρ and c) are not specified. As written, the claims cannot be tested.
W9. The computational claim is undefined. The abstract claims quadratic-to-linear scaling of the computational profile and an overhead reduction of over 90%, without stating which cost scales with which quantity. Section 2.6 shows that the claim is a count of entries in the orthogonality penalty, with no measured runtime or memory. The full penalty does not require the dh × dh Gram matrix in any case, because ||WD^T WD||F = ||WD WD^T||F allows it to be evaluated at a cost linear in dh.
W10. The submission does not comply with the ICLR 2027 guidelines. The Nomenclature section precedes the references and therefore belongs to the main text. The main text consequently runs onto page 10, beyond the 9-page limit that the guidelines state will be strictly enforced. The mandatory AI-use section is also absent, although the prose strongly suggests heavy use of generative AI.
W11. The reported runs do not implement the described method or test the superposition claim. Appendix A is presented as the code behind Table 2. Its loop optimises only Lrecon plus a fixed L1 term (lines 78 to 81), with no Lortho, no Phase A, no κ trigger and no Ebase stopping rule. Recycling modifies decoder columns but not encoder rows (line 117), which is the reverse of Section 2.3, and spawning uses plain Gaussian initialisation (lines 24 to 34). All runs also use fewer latents than the 4096-dimensional residual stream, whereas superposition requires an overcomplete dictionary. For an overcomplete dictionary, ||WD^T WD - I||F^2 cannot fall below (dh - din) dh / din. The claim that superposition is isolated under strict orthogonality is therefore untested by these experiments, and in the overcomplete regime it is unattainable.
References
[1] I. Ramirez, P. Sprechmann, G. Sapiro. Classification and Clustering via Dictionary Learning with Structured Incoherence and Shared Features. CVPR 2010.
[2] N. Bansal, X. Chen, Z. Wang. Can We Gain More from Orthogonality Regularizations in Training Deep Networks? NeurIPS 2018. arXiv:1810.09102.
[3] A. Korznikov et al. OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features. ICLR 2026. arXiv:2509.22033.
[4] J. Bloom, C. Tigges, A. Duong, D. Chanin. SAELens. https://github.com/decoderesearch/SAELens, 2024 (option decoder_orthogonal_init, versions 3 to 5).
[5] T. Bricken et al. Towards Monosemanticity: Decomposing Language Models with Dictionary Learning. Transformer Circuits Thread, 2023.
[6] J. Mairal, F. Bach, J. Ponce, G. Sapiro. Online Learning for Matrix Factorization and Sparse Coding. JMLR 11:19-60, 2010.
[7] U. Evci, B. van Merriënboer, T. Unterthiner, M. Vladymyrov, F. Pedregosa. GradMax: Growing Neural Networks using Gradient Information. ICLR 2022. arXiv:2201.05125.
[8] L. Gao et al. Scaling and Evaluating Sparse Autoencoders. ICLR 2025. arXiv:2406.04093.
[9] B. Bussmann, P. Leask, N. Nanda. BatchTopK Sparse Autoencoders. NeurIPS 2024 Workshop. arXiv:2412.06410.
[10] B. Bussmann, N. Nabeshima, A. Karvonen, N. Nanda. Learning Multi-Level Features with Matryoshka Sparse Autoencoders. ICML 2025. arXiv:2503.17547.
[11] S. Rajamanoharan et al. Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders. arXiv:2407.14435, 2024.
[12] A. Karvonen et al. SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability. ICML 2025. arXiv:2503.09532.
[13] L. Smith et al. Negative Results for SAEs on Downstream Tasks and Deprioritising SAE Research (GDM Mech Interp Team Progress Update #2). Alignment Forum, 2025.
[14] S. Kantamneni et al. Are Sparse Autoencoders Useful? A Case Study in Sparse Probing. ICML 2025. arXiv:2502.16681.
[15] Z. Wu et al. AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders. ICML 2025. arXiv:2501.17148.
[16] J. Huang, Z. Wu, C. Potts, M. Geva, A. Geiger. RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations. arXiv:2402.17700, 2024.
[17] S. Marks et al. Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. ICLR 2025 (oral). arXiv:2403.19647.
[18] T. Lieberum et al. Gemma Scope: Open Sparse Autoencoders Everywhere All at Once on Gemma 2. arXiv:2408.05147, 2024.
[19] Z. He et al. Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders. arXiv:2410.20526, 2024.
[20] G. Paulo, N. Belrose. Sparse Autoencoders Trained on the Same Data Learn Different Features. ICLR 2026. arXiv:2501.16615.