- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether a model-growth direction found on finite training data still works on independent test data in normalized residual networks. This paper introduces the effective alignment dimension , built from the activation gradient at an insertion point. Theorem 1 gives the exact mean/variance of the train-vs-test gradient inner product and a finite-sample bound on misalignment. Theorem 2 plugs this into the function-preserving residual-expansion framework of Cheng et al. (2026). Experiments on width-controlled LLaMA-style Transformers, Pythia checkpoints, and ResNet-20 report that grows with width while misalignment drops, plus small-scale interventions predicting held-out loss changes.
Critical strengths
- Addresses a clear, practically relevant question—whether a trained growth direction transfers—that asymptotic width theory does not answer.
- The exact second-moment derivation is transparent, and the exact vs. simplified certificates are both disclosed.
- Experiments span three architectures/modalities with multiple seeds, and the intervention study attempts causal rather than purely correlational validation.
Critical weaknesses
W1. (Critical) Width expansion is never actually performed in this paper
The paper mainly focuses on "width growth" in this paper. However, as far as I can tell, no model is ever widened:
- Figs. 1–3 compare six models of different widths that are each trained from scratch, which is not an expansion.
- Fig. 4a inserts one residual layer and, on the stated path, only the additive bias is trained (). This "expansion" is not a width-scaling regime.
- Fig. 4b "sequentially adds residual structures" (up to ~20), which is depth growth, not width.
So the operation in the title is not instantiated by any experiment. The width evidence is correlation across independent runs, and the actual interventions are a bias insertion and a depth addition. This can't be fixed by re-framing; it needs a real function-preserving width-expansion run, e.g. expanding hidden dimension / ffn inner dimension during training as [1] proposed.
Otherwise, if the authors are focusing on simply scaling up model width, there lacks a thorough study of the relationship between width, alignment dimension and downstream performance.
[1] Yu et al. SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning, ICML 2026
W2. (Critical) The effect of on expansion is never tested
Even within the paper's framing, the causal claim "larger indicates more reliable/beneficial expansion" is not validated. Fig. 4 verifies the sign of the trial-level (close to a restatement of Theorems 1–2), not the population . There is no comparison across insertion points/models with different to see whether higher gives larger or more reliable gains; the three Pythia scales aren't linked to (trials are pooled); only infinitesimal steps are tested, not training to convergence; and no intervention is run on the flagship LLaMA-style models (everything is $\le$70M / CIFAR-scale).
W3. (Critical) The relevant regime / scope is unclear
Under iid sampling misalignment as , and Fig. 2 already shows at on small models. This circumstance is non-trivial and realistic, as the LLM pretraining is more of an overfitting paradigm (as we always want to include as much data as we can obtain for pretraining, which dramastically reduces the gap between training and testing). The paper does not discuss when the bound is meaningful or vacuous for large-scale pretraining, whether the intended setting is instead low-data / non-stationary / distribution-shift (fine-tuning, RL, narrow-domain growth), or how the certificate behaves under train/test shift.
W4. "Dimension" is not well justified
The standard participation ratio is , which equals for isotropic covariance. Here the numerator is the signal energy , making these signal-to-noise ratios rather than geometric dimensions. There is no insurance that . A formal justification or qualified terminology would help.
W5. Estimator rigor for is missing
The paper doesn't report how many activation-gradient samples are used to estimate , gives no confidence intervals for , and doesn't address finite-sample bias of the ratio . The Table 8 std-devs are across training seeds, not estimation error, so it's unclear whether adjacent-width differences are meaningful.
W6. No link to downstream outcomes or baseline predictors
is never correlated with downtream results (ResNet accuracy in Table 7, Pythia perplexity in Table 5 does not show the relationship with ), and there is no comparison against simple baselines like , training loss, parameter count, or , so the added value of the statistic isn't shown.
W7. Largely validated on its own terms
is defined and shown to co-vary with width, and used to predict the loss change of the insertion whose derivative defines it. What is missing is evidence of external utility—e.g., that high- points outperform low- points, or high- points are easier to conduct width expansion, etc. Otherwise, I cannot see the practical use of this indicator.
Questions for authors
See Weaknesses
4 comments
Rebuttal by Authors
W5
The paper uses 10,000 C4 sequences, each containing 2,048 tokens, for estimation.
To further evaluate estimation variability caused by data sampling, we have added ten independently sampled C4 subsets, each also containing 10,000 sequences of 2,048 tokens each. With the model checkpoint and measurement location fixed, we separately re-estimate the gradient statistics and . The results are as follows.
| Model | Sequences per subset | Independent repetitions | : mean ± SD | Approximate 95% interval for the mean of repeated estimates |
|---|---|---|---|---|
| Pythia-2.8B | 10,000 | 10 | 8.0684 ± 1.0354 | [7.3277, 8.8091] |
| Pythia-6.9B | 10,000 | 10 | 9.4837 ± 1.2485 | [8.5906, 10.3768] |
W6: Comparison with simple gradient statistics
We use the same six candidate locations and continued-training results under the same computational budget as in our response to Reviewer jTs7. Here, we compare simple pre-widening gradient statistics with the effective alignment dimension.
| Model | Layer | Mean squared gradient norm | Total noise | Test-loss gain | |
|---|---|---|---|---|---|
| 2.8B | 4 | 0.24732 | 0.09418 | 7.9503 | 0.07652 |
| 2.8B | 8 | 0.26049 | 0.09928 | 9.9884 | 0.12201 |
| 2.8B | 12 | 0.23054 | 0.08762 | 8.1138 | 0.05917 |
| 2.8B | 16 | 0.28789 | 0.10964 | 8.3310 | 0.08509 |
| 2.8B | 24 | 0.24297 | 0.09279 | 7.0314 | 0.06228 |
| 2.8B | 32 | 0.27213 | 0.10228 | 8.2551 | 0.11542 |
| 6.9B | 4 | 0.31798 | 0.11995 | 9.3989 | 0.08553 |
| 6.9B | 8 | 0.30412 | 0.11651 | 9.6191 | 0.09701 |
| 6.9B | 12 | 0.33009 | 0.12689 | 8.5850 | 0.07145 |
| 6.9B | 16 | 0.29705 | 0.11433 | 8.2087 | 0.04621 |
| 6.9B | 24 | 0.31687 | 0.11862 | 10.3062 | 0.10520 |
| 6.9B | 32 | 0.32452 | 0.12297 | 8.2612 | 0.06772 |
For Pythia-2.8B and Pythia-6.9B, maximum selects layers 8 and 24, respectively, achieving the highest mean test-loss gain among the six candidate locations in each model. In comparison, the maximum mean squared gradient norm selects layers 16 and 12, with gains of 0.08509 and 0.07145, respectively; minimum total noise selects layers 12 and 16, with gains of 0.05917 and 0.04621, respectively.
These results connect pre-widening with actual test performance after continued training under the same computational budget. They show that, among the evaluated candidate layers, , which combines signal strength with directional noise structure, selects widening locations with higher mean gains than rules based solely on overall gradient energy or total noise.
Furthermore, using the pre-widening signal-to-noise ratio reported in our response to Reviewer JvjR's Weaknesses 2,
we can directly construct a normalized-noise-scale baseline in the same activation-gradient space:
Here, we compute the score using the aggregate pre-widening values in the table. Minimizing this score is equivalent to selecting the location with the largest . The reported candidate-layer statistics and gains yield the following comparison.
| Selection strategy | 2.8B selected layer | 2.8B mean gain | 6.9B selected layer | 6.9B mean gain |
|---|---|---|---|---|
| Minimum normalized noise scale | 32 | 0.11542 | 24 | 0.10520 |
| Maximum | 8 | 0.12201 | 24 | 0.10520 |
This comparison is obtained by summarizing existing pre-widening statistics and continued-training results for the candidate locations; it does not involve additional continued-training experiments. Compared with this normalized-noise-scale baseline, maximum selects a location with a higher mean gain on 2.8B. On 6.9B, both strategies select the same location and obtain the same gain.
Follow up on rebuttal
Dear Reviewer xtQe,
We apologize for disturbing you. As the discussion deadline approaches, we are writing to follow up and ensure our previous response and revisions fully address your concerns.
We believe that these improvements have significantly strengthened the quality of the paper. We would be extremely grateful if you could take a moment to review our reply and re-evaluate our work. We would be happy to engage in further discussion if needed.
Best regards, The Authors
Ack of Rebuttal
Thank you for your reply. However, I am still questioning the real value of this paper. Even if the d_align has shown correlation with scaling up width in training, the core question still remain unresolved that, How does d_align guide the width expansion during training or simply scaling up models? How does current scaling benefit from this metric? What principle does this metric reveal for scaling up models, is it "the core improvement of scaling up width comes from d_align improvement"? What should we really do when we are conducting width scaling from the perspective of this paper?
I will maintain my score for now.
Rebuttal by Authors
Weaknesses 1 and Weaknesses 7
The central research question of this paper is how model width affects activation-gradient alignment and local directional transfer under finite sampling. To address this question, Figures 1–3 compare independently trained models of different widths while keeping depth, training data, and token budget fixed. This controlled design examines the relationships between width, gradient statistics, effective alignment dimension, and misalignment probability. Training the models independently from scratch is therefore consistent with our research objective. The residual-insertion experiments in Figure 4 examine the connection between local directional transfer in the theory and changes in held-out loss.
To further examine actual width growth, we have added local FFN widening experiments on Pythia-2.8B and Pythia-6.9B, using continued training without widening under the same computational budget as the control. The widening settings, layer-wise results, and comparisons of layer-selection strategies are provided in Tables R1 and R2 of our response to Reviewer jTs7. These experiments directly examine whether the pre-widening metric can help select locations for subsequent growth.
Weaknesses 2
The detailed layer-wise results and comparisons of layer-selection strategies are provided in Tables R1 and R2 of our response to Reviewer jTs7.
The metric and gains are positively associated across locations, and the pre-widening metric can assist practical location selection. This pattern also appears in models with larger parameter counts.
Weaknesses 3
Our primary application is the analysis of local directional transfer at a given checkpoint and hidden location in a pretrained model. In the current experiments, training-side and test-side gradient samples come from independently sampled C4 subsets. We use an evaluation setting in which both sides share the same data distribution, without deliberately introducing distribution shift. Thus, the relatively low misalignment rates in Figure 2 indicate that the evaluated models and locations have relatively stable local gradient directions under the current C4 setting.
Our focus is to quantitatively explain this stability and how it varies with model width and sampling budget. Even when misalignment rates are already low, comparing gradient statistics and misalignment probabilities across widths helps us understand which models have more reliable local directions under the same finite sampling budget, and what signal and noise structures underlie these differences.
For the Pythia widening experiments above, we set M=K=8 and compute Balign from the estimated statistics as follows. Each range covers the six candidate locations in the corresponding model and is not a confidence interval.
For further applications, please see our response to Reviewer jTs7.
W4
The terminology can be explained through a simple example. Consider a gradient vector with k independent coordinates, each with mean σ>0 and variance σ2, so that each coordinate has a GSNR of 1. Then,
μ=σ1k,Σ=σ2Ik,
and
\mu^\top\Sigma\mu=\operatorname{tr}(\Sigma^2)=k\sigma^4.$$ Substituting these quantities into our definitions gives $$d_{\parallel}=d_2=d_{\mathrm{align}}=k.$$ We therefore use "effective dimension" to compare this metric with the number of independent coordinates having unit GSNR. We are also considering renaming the metric "effective alignment number" to avoid potential misunderstanding of the term "dimension."