- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies FFN activation density, defined as a width-normalized participation ratio of activation magnitudes. Controlled TinyStories experiments vary the expansion ratio while holding the training recipe fixed, and measurements on eleven pretrained models extend the analysis. The paper expresses density as a functional of the empirical pre-activation distribution, tests a Gaussian moment model, and uses a zero-mean-row constraint to investigate the effect of pre-activation mean. It finds weak width dependence for its SwiGLU sweep, accurate moment-conditioned predictions for the controlled non-gated activations, and larger residuals for gated models. It also compares spectral participation ratio with entropy-based effective rank.
Critical strengths
-
Controlled experiments isolate a relevant architectural variable. Holding data order, model dimension, depth, and training schedule fixed makes the expansion-ratio sweep more interpretable than a comparison across unrelated pretrained models. Reporting seed variability, validation loss, and the hardware sensitivity of the slopes makes the empirical evidence easier to assess (Tables 4 and 10; Appendix K).
-
The intervention tests a specific mechanism. The zero-mean-row experiment goes beyond fitting observed activation statistics. The different predictions for homogeneous and non-homogeneous gates are useful controls: ReLU density returns close to its zero-mean prediction, while SwiGLU retains a positive slope. The paper also reports the gated residuals and failed initial prediction rather than presenting the Gaussian approximation as universally accurate.
Critical weaknesses
-
The measured density does not establish the claimed notion of FFN utilization. Equation 2 uses activation magnitudes but ignores the norms and directions of the output-projection columns appearing in Equation 1. Consequently, a unit with a large activation need not contribute proportionally to the FFN output, and cancellation is not represented. For a ReLU FFN, positive per-unit rescaling of the input weights and bias, compensated by inverse rescaling of the corresponding output column, preserves the network function but can change this density. The statistic is still useful descriptively, but the inference that utilization cannot inform width or exploitable sparsity is too broad without an output-aware or ablation-based check. Relatedly, disagreement between spectral participation ratio and entropy effective rank does not alone establish that the former underestimates a ground-truth effective dimension: the two measures weight a nonuniform spectrum differently (Section 3.1; Appendix A).
-
Width independence is more conditional than the headline conclusion suggests. The 7% change comes from small SwiGLU models trained on TinyStories. The paper’s unconstrained ReLU sweep shows a reduction from 0.1954 to 0.1569 as the ratio increases from 2 to 8, approximately 20% (Table 10). The 0.49% predictive error is an average over the two controlled non-gated arms, whereas pretrained SwiGLU errors reach 26–35%. Proposition 1 is an exact reformulation of the statistic; the invariance corollary additionally assumes convergence to a width-independent distribution and uniform integrability. The shared-component measurement addresses one dependence model, but does not establish all those conditions, particularly tail control in the presence of massive activations. These results support a useful conditional account, but not yet the general claim that width cannot affect utilization or available sparsity.
Questions for authors
-
Do the width trends persist when density is computed from the norms of per-unit output contributions, or when utilization is measured by the fraction of units that can be removed at a fixed loss increase? Such a check would directly address the link between Equation 2 and the efficiency conclusions.
-
How should the title and the claim that changing width cannot increase available sparsity be reconciled with the approximately 20% density decrease in the unconstrained ReLU sweep? Which conclusions are intended specifically for SwiGLU, and which are supported across activations?
-
Beyond the equicorrelated shared-component proxy, is there evidence that the relevant empirical moments and tail contributions stabilize across the controlled width sweep? Please distinguish direct support for the corollary’s assumptions from observations that are merely consistent with them.