acceptodds
Under review as a conference paper at ICLR 2027

Emergent phases of superposition: from partial to full representation

Abstract

Large language models are thought to represent features by vectors in a hidden space of dimension given by the model's width. Superposition, in which more features are represented than the width by letting representation vectors overlap, is a leading account of how representation vectors are organized. However, how model width and data statistics determine the configuration of representation vectors and the resulting loss when the number of features and the width are large remains less understood. Here we show, in Anthropic's toy model of superposition, that increasing the width drives a continuous phase transition from a partial-representation phase, where only a subset of features receives appreciable representation vectors while the rest vanish, to a full-representation phase, where every feature is represented. Our theory via a partial random projection approximation predicts, and experiments confirm, that the critical width grows linearly with the number of active features up to a logarithmic factor. The loss scaling changes across the transition: below the critical width, the loss grows linearly with the number of active features and depends weakly on the width in a form set by data statistics; above it, the loss grows approximately quadratically with the number of active features and decays inversely with the width. Non-uniform firing probabilities delay the transition and lower the loss, as more frequent features occupy more space. Our results provide an account of how model width and data statistics jointly shape representations and loss, a step toward understanding representation scaling in large models.

Primary area:
learning theory
Keywords:
superposition, representation, phase transition, neural scaling laws

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mrikOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the toy model of superposition introduced by Elhage et al. It considers at what width scale do all individual features get represented. The paper shows empirically that there exists a phase transition after which all individual features are represented. The paper shows that this behavior arises in a random model, in which a subset of features are represented and the rest are not. (Then the question becomes the optimal number of features to represent). The paper shows this value to be about E log (n/E) where n is the total number of features and E is the average number of active features. This matches classical compressed sensing results.

Critical strengths

While many papers have demonstrated that superposition is possible when the width is on the scale of E log (n/E) (using the paper's notation), the paper illustrates interesting behavior prior to this cutoff point, where subsets of features are well-represented and the rest are not (as shown by Figure 2(a)). I think the paper could perhaps further emphasize this aspect of the result, which is comparatively novel. Naively, one might expect that even below the E log (n/E) threshold, the model is best off trying to represent every feature (even if poorly). This fact is more surprising than on first read of the paper.

The partial random projection model is elegant, and gives a simple way to reach this conclusion.

Critical weaknesses

The implications for further developing a theory of language models or the linear representation hypothesis are not obvious, and could be further developed.

The paper would also benefit from exploring what happens under alternative "architectures" such as PCA. What aspect of the model yields the phase transition behavior?

Questions for authors

How do you see the model's core implications beyond establishing the E log n critical threshold, particularly in relation to further developing theory of language models?

What are the analogs of the results for PCA? What are the crucial aspects of the model that yield the phase transition behavior?

2 comments

We thank the reviewer for the thoughtful review, and in particular for pointing out that the behavior below the threshold, where a subset of features is well represented and the rest are not, is the more surprising part of our results.

How do you see the model's core implications beyond establishing the Eˉlog⁡n\bar E \log n critical threshold: We thank the reviewer for this question, and for the suggestion to further emphasize the behavior below the threshold, which we will follow in the revised version. Beyond the value of m∗m^*, we see two implications for a theory of language models.

  1. Scaling laws hold in only one phase. The loss follows a power law in the width, L∝1/m\mathcal L\propto 1/m (Liu et al., 2025), only in the full-representation phase. Below $m^* $ it instead decreases approximately linearly, L≈Lunrep(1−κm/m∗)\mathcal L\approx\mathcal L_{\rm unrep}(1-\kappa m/m^* ), which is not a power law. A scaling law fitted across the two phases, or extrapolated from models below $m^* $ to models above it, would therefore give misleading exponents. One should identify which phase a family of models is in before applying or testing a scaling law.
  2. Language models may operate in the partial phase. The number of features in natural language is plausibly enormous, so a language model, especially a small one, may be below $m^* $. As the reviewer points out, one might expect the model to represent every feature poorly. However our results describe what happens there in an ideal setup, and give a different picture: the model represents a subset well and discards the rest, with the number of represented features growing in proportion to the width. This gives a picture of the capacity and limits of a model of finite width.

We will add further discussions about these points to the revised version. Testing these implications in language models requires feature-level measurements across model widths (e.g., with sparse autoencoders), which we leave to future work.

What are the analogs of the results for PCA? We read PCA as the linear version of the model (no ReLU), whose optimum is a principal-component projection of the covariance matrix Cov(x){\rm Cov }(\boldsymbol x) of the input x\boldsymbol x (Baldi & Hornik, 1989). When features differ in importance or frequency, it keeps the mm leading features and discards the rest (r=mr = m), as in Elhage et al. (2022). This is the linear analog of our non-uniform result that frequent features are represented first, but the transition from partial to full representation cannot happen at m<nm<n. For uniform features, Cov(x)∝I{\rm Cov }(\boldsymbol x) \propto \boldsymbol I, so every rank-mm projection is optimal and the linear model has no preferred number of represented features. Therefore, the phase transition cannot happen for uniform features either. We will add the discussion on PCA in the revised version.

What are the crucial aspects of the model that yield the phase transition behavior? The transition in our model requires three ingredients: 1) sparsity, so that interference grows with the number of active features Eˉ\bar E rather than nn; 2) a nonlinear threshold that filters interference on inactive features. In our toy model it is ReLU with bias, which is where log⁡n/Eˉ\log n/\bar E comes from; and 3) a width-dependent trade-off: the cost of representing a feature and the cost of discarding it depend differently on the width mm, so their balance shifts with mm and sets $m^* $.

Dear Reviewer mrik,

Thank you again for your time and effort in reviewing our paper.

With the discussion period closing in less than 24 hours (Nov. 18, AoE), we would like to follow up on our response to your questions on the implications of our results, the comparison with PCA, and the ingredients of the transition. Please let us know whether our replies answer them, or whether anything is still unclear; we would be happy to clarify.

If our responses address your concerns, we would be grateful if you could take them into account in your final assessment.

Best regards, The authors

Reviewer_Efr3OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Using a toy model of a single-layer neural network reconstructing random sparse feature vectors, this paper studied the phase-transition phenomenon of representable features, and the scaling behavior of the loss. The study focused on the regime where the number of features to represent is larger than hidden dimension, thus superposition has to happen in the hidden layer. As the hidden dimension increases, the model transition from representing some of the features and ignoring other, to a full representation. Around the phase transition point, the scaling of loss with the number of features to represent changed from a 1st order power law to 2nd order one. The paper also studied the impact of feature sparsity and imbalance on the phase-transition. To theoretically understand this phenomenon, the paper proposed a random projection-based approximation that explained a lot of the observations.

Critical strengths

The toy model being studied is interesting and relevant. Perceptron is the fundamental building block of all AI models. The regime being studied here, where number of features to represent is larger than the hidden dimension, is where most AI models operate, thus it has great practical and theoretical significance. Although simple, the toy model revealed many interesting phenomenon commonly known in practical ML, like the inability to learn some feature under underfitting, and the power-law scaling of loss with respect to model width (size). The experiments and theoretical analysis appear sound, and the visualization of results is elegant, clear and consistent throughout.

Critical weaknesses

My main concern is about how strongly the results support the phase-transition framing. The experiments appear to use a single value of n (n = 10,000), and the paper does not explain too much why this value was chosen.

Questions for authors

How do the location and sharpness of the transition change as n increases while Ē/n is held fixed?

3 comments

We sincerely thank the reviewer for the positive assessment, and for finding the problem relevant, the analysis sound and the visualizations clear.

The paper does not explain too much why this value was chosen: The choice balances the limit we study against computational cost. Our analysis concerns the regime 1≪Eˉ≪n1\ll \bar E\ll n, with the transition in strong superposition (m∗≪nm^* \ll n). With n=104n=10^4, the range Eˉ∈[8,128]\bar E\in [8,128] satisfies both conditions while keeping Eˉ\bar E large enough for the mean-field description (m∗m^* ranges from about 0.5% to 6% of nn). Larger nn is costly, especially as the paper requires many runs over m,Eˉ,σv,σEm, \bar E, \sigma _v, \sigma _E and ϵp\epsilon _p. So we treat n=104n=10^4 as a default setup and exhibit results of that. We will add the explanation of the choice of nn in the revised version.

Results do not depend on the choice of nn. See Q1 for the result at other nn.

Q1: At fixed Eˉ/n\bar E/n, the critical width $m^* $ grows linearly with nn, making m∗/nm^* /n a fixed number, see figure Fig. 26 (App. D.1). Within our resolution, the shape of r/nr/n vs $m/m^* $ is independent of nn; we do not study finite-size rounding near $m^* $. We will express these more clearly in the revised version.

Dear Reviewer Efr3,

Thank you again for your time and effort in reviewing our paper, and for your positive assessment.

With the discussion period closing in less than 24 hours (Nov. 18, AoE), we would like to follow up on our response. Results at other values of n with Eˉ/n\bar E/n fixed are now in Fig. 26 (Appendix D.1 of the revised PDF). Please let us know whether this answers your question, or whether anything is still unclear; we would be happy to clarify.

Best regards, The authors

Thank you very much for answering my questions! I will keep my score.

Reviewer_fWwQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies superposition, the way a neural network can store more features than it has dimensions by letting their representations overlap. It uses the toy model of Elhage et al. (2022), an autoencoder that maps nn sparse input features into a hidden layer of width mm and reconstructs them through a ReLU, and asks how the width and the statistics of the data determine which features the trained model represents and what loss it reaches when both nn and mm are large. The main finding is a transition as the width grows. Below a critical width, denoted m∗m^{\ast}, the model represents only part of the features, even though all features are statistically identical, and the remaining ones are effectively dropped; above m∗m^{\ast}, every feature is represented. The authors track this with the fraction r/nr/n, where rr is the effective number of represented features (a participation ratio of the norms of the representation vectors), using n=104n=10^4 features in all experiments. To explain the transition, they propose an analytical "partial random projection" approximation, which treats the represented features as random directions in the hidden layer and reproduces the measured rr and loss as functions of mm. From this approximation they derive that m∗m^{\ast} grows roughly like Eˉlog⁡(n/Eˉ)\bar E\log(n/\bar E), where Eˉ\bar E is the average number of features active in a single input, and that the loss scales differently with mm and Eˉ\bar E on the two sides of the transition. Finally, they show that when some features fire more often than others, the transition moves to larger widths and the loss decreases.

Critical strengths

The paper has the following key strengths:

  • The partial phase seems sharp. As mm grows, the norms sis_i of the representation vectors (one per feature ii) go from two well-separated groups to a single group, the pairs (si,bi)(s_i,b_i), where bib_i is the bias of the output unit of feature ii, form two clusters, and the results barely change across random seeds (Figs. 2, 10). The location of the transition also collapses onto one curve when the width is rescaled as m/(Eˉlog⁡(n/Eˉ))m/(\bar E\log(n/\bar E)), for Eˉ\bar E between 8 and 128 (Fig. 4b). Locating where this partial phase ends when nn is large, and without weight decay, goes beyond the small-model observations in prior work and bears on how models allocate capacity when there are many more features than dimensions.

  • The partial random projection approximation is transparent, and its numerical solution closely tracks the measured rr, the loss and the location of the transition as functions of mm (Figs. 3c-d, 4a, 11a). It also offers an intuitive mechanism: representing a feature costs interference ("crosstalk") with the other features, while dropping it costs its reconstruction error (L316-323), and it could be a useful tool for follow-up work. The appendix checks the approximation against two constrained versions of the model, against the measured distribution of the interference, and for finite-size effects (App. C), and the limitations are stated openly (L471-476).

Critical weaknesses

There seem to be some major concerns that require a careful pass:

  • The analytical derivation of m∗m^{\ast} relies on an assumption that fails near the transition, and the scaling law is tested at a single nn. The asymptotic analysis in App. A.4.1 assumes that the interference noise on each feature is small, so that the bias is small and the ReLU leaves active features essentially untouched (L889-896). In the paper's notation the bias is b≈−βσηb\approx-\beta\sigma_\eta (Eq. 67), where ση\sigma_\eta is the standard deviation of the interference noise and β\beta is the bias measured in units of that standard deviation, so the analysis requires βση\beta\sigma_\eta to be small. Let vˉ\bar v and σv2\sigma_v^2 denote the mean and variance of the activation strength of an active feature (the paper uses vˉ=1\bar v=1 and σv2=1/3\sigma_v^2=1/3). Inserting the paper's expressions for β\beta (Eq. 77) and ση\sigma_\eta (Eq. 10) at the predicted m∗m^{\ast} (Eq. 84) gives β2ση2=(vˉ2+σv2)2/(2σv2)=8/3\beta^2\sigma_\eta^2=(\bar v^2+\sigma_v^2)^2/(2\sigma_v^2)=8/3, so βση≈1.6\beta\sigma_\eta\approx 1.6, which is not small. The trained models agree: near the transition the represented features have biases between −1.4-1.4 and −1.7-1.7 (Fig. 2b), so the ReLU zeroes out a substantial fraction of the active features. Consistently, Eq. (84) predicts m∗→0m^{\ast}\to 0 as σv→0\sigma_v\to 0, whereas Fig. 11a shows that m∗m^{\ast} is largest at σv=0\sigma_v=0. The paper acknowledges this only in the appendix (L1007-1008, L1384-1386), while the main text refers to App. A.4.1 for the dependence of Eq. (15) on σv\sigma_v (L323). The remaining evidence for m∗∼Eˉlog⁡(n/Eˉ)m^{\ast}\sim\bar E\log(n/\bar E) is therefore numerical, and it comes from a single number of features: every reported result uses n=104n=10^4, although runs with nn from 1,000 to 65,536 were performed (L1282-1284). With nn fixed, the factor log⁡(n/Eˉ)\log(n/\bar E) changes by only about 1.6 times over the tested range Eˉ=8\bar E=8 to 128128, so the data cannot distinguish the claimed form from other slowly growing functions of Eˉ\bar E. Moreover, Fig. 2c has no width between m=200m=200 (where r/n≈0.6r/n\approx 0.6) and m=350m=350 (where r/n=1r/n=1), which is exactly where the transition occurs, and the sharpening of the transition for large systems (L191-195) is never shown.
  • The novelty is narrower than the abstract suggests. The two loss regimes match predictions of Cowsik et al. (2025), and the decay of the loss as 1/m1/m matches Liu et al. (2025), as the paper itself says (L346-349). The critical-width scaling Eˉlog⁡(n/Eˉ)\bar E\log(n/\bar E) was already proposed through the compressed-sensing bound (Sarkar & Deka, 2026; acknowledged at L459-460 and L478-480), and Elhage et al. (2022, "Uniform Superposition") observed that only a subset of equally important features is learned, with their number growing linearly in the width; this observation is not discussed. The abstract and the contribution paragraph (L78-88) nonetheless present these scalings as findings. In addition, the claim that "the loss follows the representation" (L87) is not supported by the paper's own controls: the equal-norm (EN) and random-projection (RP) models, which by construction cannot drop features (L1424-1425), show the same loss curve and "similar bending" (Fig. 12a, L1444-1445). The bend in the loss as a function of mm therefore seems to come from the trade-off between interference and dropping features rather than from the transition itself. As I read it, the genuinely new parts are the characterization of the partial phase at large nn and the partial random projection approximation, and the framing should make that explicit.

Additional, less decisive concerns:

  • The analysis of non-uniform firing probabilities is qualitative. The paper controls the non-uniformity with an exponent ϵp\epsilon_p (a larger ϵp\epsilon_p means more unequal firing probabilities). In Fig. 6b the predicted transition appears to move much less with ϵp\epsilon_p than the observed one, the loss correction (Eq. 19) is never compared quantitatively with measured losses, and the transition is only well defined for ϵp≲1\epsilon_p\lesssim 1 (L1749-1755).
  • Convergence of rr is not shown. All runs start with every representation norm equal to one (L1299-1301), and once the bias of a dropped feature becomes negative it receives no gradient (L865-867), so the partial configuration could partly reflect the optimization rather than the loss landscape.
  • Minor: σv\sigma_v is given as 1/31/3 at L1798 and in the captions of Figs. 24-25, but as 1/31/\sqrt{3} elsewhere. Please also double-check the third term in the numerator of Eq. (98), where FF is the auxiliary function of Eq. (48) and β0\beta_0 is the optimal β\beta defined in Eq. (97); I obtain −F′′′(β0/2)β02/(162)-F'''(\beta_0/\sqrt{2})\beta_0^2/(16\sqrt{2}), although this does not affect the conclusion at L1069-1070.
Questions for authors

I have the following questions for the authors:

  1. Can you report m∗m^{\ast} with an uncertainty estimate, and plot r/nr/n against m/m∗m/m^{\ast} for several values of nn with the ratio Eˉ/n\bar E/n held fixed, on a finer grid of widths around m∗m^{\ast}? App. B indicates that runs with nn from 1,000 to 65,536 exist. This would test both the log⁡(n/Eˉ)\log(n/\bar E) factor and the sharpening of the transition.

  2. Can the asymptotic analysis of App. A.4.1 be redone without assuming that βση\beta\sigma_\eta is small, taking into account that the ReLU zeroes out some active features? If not, which of Eqs. (13)-(15) and (83)-(84) should be read as heuristic, and could the main text say so?

  3. What exactly is new relative to Cowsik et al. (2025), Sarkar & Deka (2026), and the uniform-superposition results of Elhage et al. (2022)?

  4. How much lower is the loss of the unconstrained model than that of the equal-norm model, as a function of mm (Fig. 12a)? Is any feature of the loss curve specific to the partial phase?

  5. Is rr itself converged, and is the same partial configuration reached from other initializations (for example small norms, or the equal-norm solution)? A plot of rr over training near m∗m^{\ast} would help.

  6. Let LL be the loss of the trained model and LunrepL_{\mathrm{unrep}} the loss of a model that represents no feature (Eq. 7). Below m∗m^{\ast}, Eqs. (82)-(84) seem to imply L≈Lunrep(1−m/(2m∗))L\approx L_{\mathrm{unrep}}(1-m/(2m^{\ast})). Is this the "form set by data statistics" mentioned in the abstract? If so, please state it explicitly.

8 comments

We thank the reviewer for the very careful and detailed review, in particular for checking our derivation closely.

The sharpening of the transition for large systems (L191-195) is never shown: We agree the sentence is imprecise. We only mean that mm is discrete, which leads to a step-like rr-mm dependence; we do not claim, and do not study, how the transition sharpens with nn (finite-size rounding, critical exponents, etc.). We will modify L191-195 and employ an operational definition of m∗m^* (threshold mm that r/nr/n exceeds 0.99) in the revised version.

The claim that "the loss follows the representation" (L87) is not supported by the paper's own controls: We agree with the reviewer’s interpretation that the bend comes from the trade-off between interference and dropping features. We see this as consistent with our claim, since which features are dropped is exactly what the representation determines.

Below $m^* $, the EN and RP models cannot drop individual features. Instead, their shared norm decays appreciably below 1 (Fig. 12b), so effectively all features are dropped together. The unconstrained model drops only a subset, which gives a lower loss that is approximately linear in mm below m∗m^* (see Q4 & Q6). We will add a sentence after L87 clarifying that “the loss follows the representation” refers to which features are dropped.

The analysis of non-uniform firing probabilities is qualitative: We agree. As noted in App. A.5.3, the perturbation analysis requires ϵplog⁡n≪1\epsilon_p\log n \ll 1, which our experiments do not satisfy, so it captures only the signs of the responses and not their magnitudes (Fig. 25). As stated in App. A.5.3 and C.3, this part is intended to explain trends, not to give quantitative predictions. We will make it more explicit in the revised version.

σv\sigma _v is given as 1/3 at L1798 and in the captions of Figs. 24-25, but as 1/31/\sqrt{ 3 } elsewhere: This is not a typo. By chance we exhibit the result with σv=1/3\sigma _v=1/3 in that section, and the qualitative results are unchanged under different σv\sigma _v. In the revised version we will replace it by the result of σv=1/3\sigma _v=1/\sqrt{ 3 } for better consistency.

Please also double-check the third term in the numerator of Eq. (98): We thank the reviewer for pointing this out. It is a typo and the correct term is proportional to β02\beta_0 ^2. We will correct it in the revised version.

Q1:

  1. We define $m^* $ operationally as the threshold mm that r/nr/n exceeds 0.99. For Eˉ=50,n=10000,σv=1/3\bar E=50,n=10000,\sigma _v=1/\sqrt{ 3 }, the critical point $m^* =290\pm 10 $. Note that this error bar is a bound from the grid of simulation, not a standard deviation, since the standard deviation of $m^* $ with different seeds is negligible. The error bars of $m^* $ with other parameter setups will be reported in the revised version.
  2. At fixed Eˉ/n\bar E/n, the critical width depends linearly on nn, and the curves r/nr/n vs m/nm/n collapse as predicted, see Fig. 26 (App. D.1). It cannot be used to verify the log⁡n/Eˉ\log n/\bar E factor, since the factor is fixed for all curves.
  3. To test the dependence on n/Eˉn/\bar E, we plot r/nr/n against $m/m_{\rm pRP}^* $ for several (Eˉ,n)(\bar E, n), where $m_{\rm pRP}^* $ is the critical width predicted by the partial RP approximation (see Fig. 27 (App. D.1)). Although log⁡n/Eˉ\log n/\bar E varies by a factor of 2.2, the curves collapse, with the trained transition at $m^* \approx 0.8 m_{\rm pRP}^* $ for all (Eˉ,n)(\bar E,n). We agree with the reviewer that no accessible range of n/Eˉn/\bar E can by itself distinguish log⁡n/Eˉ\log n/\bar E from other slowly growing functions. Our claim for the logarithm rests on the theory: the partial RP approximation gives mpRP∗∝Eˉlog⁡n/Eˉm_{\rm pRP}^* \propto \bar E\log n/\bar E asymptotically (see the answer of Q2), and the trained models follow $m_{\rm pRP}^* $ up to a constant factor over the range we can simulate. We will state this explicitly in the revised version.

Q2 & The analytical derivation of m∗m^* relies on an assumption that fails near the transition

  1. The analytical derivation of $m^* $ can be done without assuming βση≪1\beta \sigma _\eta \ll 1. For bounded activation 0≤v0\le v, in the limit 1≪Eˉ≪n1\ll \bar E \ll n, we can show m∗=CEˉlog⁡(n/Eˉ)+h.o.t.m^* =C \bar E \log (n/\bar E)+{\rm h.o.t.}, so Eq. (13)-(15) are all valid in corresponding limits. But Eq. (83)-(84) should be read as heuristic, since the derivation of the prefactor 4σv2vˉ2+σv2{4\sigma _v^2\over \bar v^2+\sigma _v^2} in Eq. (84) relies on the assumption βση≪1\beta \sigma _\eta \ll 1. Detailed derivation can be found in App. D.5, and we will add it into the revised version.
  2. Without doing any extra approximation, we numerically solve Eq. (64). The numerical result at large nn supports the asymptotic expression m∗=CEˉlog⁡(n/Eˉ)m^* =C\bar E\log (n/\bar E). Detailed results can be found in App. D.2 (Figs. 28–30), and we will add it into the revised version.

(Continued.)

Q3 & The novelty is narrower than the abstract suggests

We thank the reviewer for this careful comparison, and in particular for pointing us to the uniform-superposition results of Elhage et al. (2022): we had overlooked their observation that the number of represented features grows linearly with the width. We agree that the abstract and the contribution paragraph (L78–88) present some scalings as findings although they are consistent with or anticipated by prior work. We will rewrite both in the revised version and expand the related-work section, including the results of Elhage et al. Relative to the works mentioned:

  1. Cowsik et al. (2025): in the following we refer to the arXiv version of this paper (arXiv:2410.12101v2). They restrict their analysis to permutation-symmetric solutions, in which all features have equal norms and biases (their Sec. 4.1–4.2). In our notation, they obtain the loss in two limits. As m/n→0m/n \to 0 at fixed Eˉ/n\bar E/n, L→Lunrep\mathcal L \to \mathcal L_{\rm unrep} to leading order (their Sec. 4.3.1). For m≫Eˉm \gg \bar E, they give an upper bound of order Eˉ2nmlog⁡nEˉ\frac{\bar E^2}{nm}\log\frac{n}{\bar E} (their Eq. (13)) and a lower bound of order Eˉ2nm\frac{\bar E^2}{nm} (their Sec. 4.3.3, App. D). Our results agree with theirs in both limits. The range of widths in between, which they give no analytical expression for, is the subject of our paper:
    1. Below m∗m^*, the learned representation is not permutation symmetric, even though the data are: a subset of features is discarded entirely. This partial-representation phase lies outside their ansatz.
    2. We locate its boundary m∗∼Eˉlog⁡(n/Eˉ)m^* \sim \bar E\log(n/\bar E) from the model’s loss (Q2), and confirm it in trained models across nn and Eˉ\bar E (Q1).
    3. We obtain the loss in this range, L≈Lunrep(1−κm/m∗)\mathcal L \approx \mathcal L_{\rm unrep}(1-\kappa m/m^*). Its linear decay is specific to the partial phase: the best symmetric (equal-norm) solution has higher loss (Q4 & Q6), and the partial configuration is reached from different initializations (Q5).
  2. Sarkar & Deka (2026) take the capacity of a width-mm layer from the compressed-sensing phase transition (their Sec. 3.2) and derive a lower bound on the loss from it, assuming the hidden layer acts as a random projection (their Theorem 1 and Limitation 5). The bound counts only discarded features; for uniform features it reads L≥Lunrep(1−m/m∗)\mathcal L \ge \mathcal L_{\rm unrep}(1-m/m^*). Our work differs in two aspects:
    1. We derive m∗m^* from the toy model’s own loss, without assuming a capacity: the transition is where representing all features becomes worthwhile, not where they become recoverable.
    2. We obtain the loss itself rather than a bound. Represented features carry interference, so L≈Lunrep(1−κm/m∗)\mathcal L \approx \mathcal L_{\rm unrep}(1-\kappa m/m^* ) with κ≈1/2\kappa\approx1/2, and the loss at $m^* $ is about half of Lunrep\mathcal L_{\rm unrep} rather than zero.
  3. Elhage et al. (2022) report three related results. (a) For equally important features with n≫mn\gg m, only part of the features is represented, and “doubling the number of hidden dimensions just doubles the number of features the model learns” (“Uniform Superposition”). (b) In small models (n=2n=2, m=1m=1), a feature can be not learned, represented in superposition, or given a dedicated dimension, with sharp transitions between these outcomes (“Superposition as a Phase Change”). (c) If the model recovers all inputs almost exactly and its embedding satisfies a restricted-isometry property, then m=Ω(Eˉlog⁡(n/Eˉ))m=\Omega(\bar E\log(n/\bar E)) (“Compressed Sensing”); how many features are actually stored is left open (“Open Questions”). Our work goes beyond these in two ways:
    1. The partial RP approximation is analytically tractable and explains (a): below $m^* $, $r/n\approx m/m^* $, so rr grows linearly in mm and, at fixed Eˉ/n\bar E/n, does not depend on nn.
    2. We derive the critical width m∗∼Eˉlog⁡(n/Eˉ)m^*\sim\bar E\log(n/\bar E) from the model’s loss, as the width at which representing all features becomes optimal, rather than from a recovery bound.

Besides, the 1/m1/m decay of the loss above $m^* $ is consistent with Liu et al. (2025), and we will present it as such rather than as a new finding. We agree with the reviewer’s reading that the genuinely new parts are the characterization of the partial-representation phase at large nn and the partial RP approximation; together with the derivation of $m^* $ within the model (Q2), these are the contributions the revised abstract will state.

(Continued.)

Q4 & Q6:

  1. In partial representation phase, the loss of equal-norm model is explicitly higher than the loss of the unconstrained model. In full representation phase, the losses are essentially the same. See Fig. 31 (App. D.3).
  2. The loss of unconstrained model in the partial representation phase can be approximated by L≈Lunrep(1−κm/m∗)\mathcal L\approx \mathcal L_{\rm unrep } (1-\kappa m/m^* ), where κ\kappa is a constant independent of mm. The detailed derivation can be found in App. D.5.1, Eq. (165). See Fig. 32 for the approximate linearity of the loss in partial representation phase. This linear form is the "form set by data statistics". We will make it clear in the revised version.
  3. As answered in the item for Q2, Eq. (84) is heuristic, so the coefficient κ\kappa in general is not precisely 1/21/2. Nevertheless, the expression L≈Lunrep(1−m/2m∗)\mathcal L\approx \mathcal L_{\rm unrep }(1-m/2m^* ) captures the linear trend, with a slightly different slope. See Fig. 32.
  4. In contrast, the loss of equal-norm model in the partial representation phase is not linear. So approximate linearity is a feature specific to partial representation phase.

Q5 & Convergence of rr is not shown: To ensure the convergence of rr and test the possibility of optimization artifact, we carried out two new experiments: 1) We set the initialization to be a random projection with vector norms 0.01. After training, this initialization gives r(m)r(m) curves similar to the original experiments. 2) We revive some of unrepresented features (rescale their norms to an appreciable value) in trained partially represented configurations, and train the configuration afterwards. After training, r(m)r(m) goes back to the values before revival. These results indicate that the partial representation is not an artifact of our initialization, nor of the vanishing gradients of dropped features. However, we agree that other optimizers or initializations might reach different local minima, and a characterization of the loss landscape remains open. Detailed results can be found in App. D.4 (Figs. 33–34), and we will add it into the revised version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised PDF: new Appendix D for the discussion

We thank all reviewers for their careful reading. We have uploaded a revised PDF. The only change is a new Appendix D, which contains the results referred to in our replies:

  • D.1: trained models at different nn and Eˉ\bar E (fWwQ Q1, Efr3 Q1)
  • D.2: numerical solution of the partial RP approximation at large nn (fWwQ Q2)
  • D.3: loss of the original and the equal-norm model below m∗m^\ast (fWwQ Q4 & Q6)
  • D.4: initialization and revival experiments (fWwQ Q5)
  • D.5: derivation of m∗∼Eˉlog⁡(n/Eˉ)m^\ast \sim \bar E\log(n/\bar E) without the small-bias assumption (fWwQ Q2)

The main text and Appendices A–C are unchanged. The other revisions promised in our replies (abstract and contribution paragraph, L87, L191–195, related work, discussion, choice of nn, Eq. (98), Figs. 24–25) will be made in the final version, where Appendix D will also be merged into the existing appendices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.