acceptodds
Under review as a conference paper at ICLR 2027

Width Scaling in Normalized Residual Networks:The Effective Alignment Dimension

Abstract

Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data. We study this problem for function-preserving residual expansion and introduce the effective alignment dimension, a measurable quantity describing the signal–noise geometry of activation gradients. By deriving the exact mean and variance of the inner product between independently estimated training and test gradients, we obtain a finite-sample upper bound on misalignment probability. The bound depends only on the effective alignment dimension and an effective sample size, requiring finite second moments and a nonzero population gradient, without covariance spectral assumptions or prescribed width-growth rates. We integrate this certificate into the train–test residual-expansion framework, yielding a high-probability condition for test-risk improvement. Experiments across width-controlled LLaMA-style Transformers, Pythia, and ResNet-20 show that wider models exhibit larger effective alignment dimensions and lower empirical misalignment. Direct residual interventions confirm that the alignment statistic predicts the sign and magnitude of held-out loss changes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
width.+scaling laws

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_xtQeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether a model-growth direction found on finite training data still works on independent test data in normalized residual networks. This paper introduces the effective alignment dimension dalign=min⁡(d∥,d2)d_{\text{align}}=\min(d_\parallel,d_2), built from the activation gradient at an insertion point. Theorem 1 gives the exact mean/variance of the train-vs-test gradient inner product and a finite-sample bound on misalignment. Theorem 2 plugs this into the function-preserving residual-expansion framework of Cheng et al. (2026). Experiments on width-controlled LLaMA-style Transformers, Pythia checkpoints, and ResNet-20 report that dalignd_{\text{align}} grows with width while misalignment drops, plus small-scale interventions predicting held-out loss changes.

Critical strengths
  • Addresses a clear, practically relevant question—whether a trained growth direction transfers—that asymptotic width theory does not answer.
  • The exact second-moment derivation is transparent, and the exact vs. simplified certificates are both disclosed.
  • Experiments span three architectures/modalities with multiple seeds, and the intervention study attempts causal rather than purely correlational validation.
Critical weaknesses

W1. (Critical) Width expansion is never actually performed in this paper

The paper mainly focuses on "width growth" in this paper. However, as far as I can tell, no model is ever widened:

  • Figs. 1–3 compare six models of different widths that are each trained from scratch, which is not an expansion.
  • Fig. 4a inserts one residual layer and, on the stated path, only the additive bias bb is trained (V(η)=0V(\eta)=0). This "expansion" is not a width-scaling regime.
  • Fig. 4b "sequentially adds residual structures" (up to ~20), which is depth growth, not width.

So the operation in the title is not instantiated by any experiment. The width evidence is correlation across independent runs, and the actual interventions are a bias insertion and a depth addition. This can't be fixed by re-framing; it needs a real function-preserving width-expansion run, e.g. expanding hidden dimension / ffn inner dimension during training as [1] proposed.

Otherwise, if the authors are focusing on simply scaling up model width, there lacks a thorough study of the relationship between width, alignment dimension and downstream performance.

[1] Yu et al. SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning, ICML 2026

W2. (Critical) The effect of dalignd_{\text{align}} on expansion is never tested

Even within the paper's framing, the causal claim "larger dalignd_{\text{align}} indicates more reliable/beneficial expansion" is not validated. Fig. 4 verifies the sign of the trial-level μM⊤gK\mu_M^\top g_K (close to a restatement of Theorems 1–2), not the population dalignd_{\text{align}}. There is no comparison across insertion points/models with different dalignd_{\text{align}} to see whether higher dalignd_{\text{align}} gives larger or more reliable gains; the three Pythia scales aren't linked to dalignd_{\text{align}} (trials are pooled); only infinitesimal steps are tested, not training to convergence; and no intervention is run on the flagship LLaMA-style models (everything is $\le$70M / CIFAR-scale).

W3. (Critical) The relevant regime / scope is unclear

Under iid sampling misalignment →0\to 0 as M,K→∞M,K\to\infty, and Fig. 2 already shows <0.1%<0.1\% at M=K=8M=K=8 on small models. This circumstance is non-trivial and realistic, as the LLM pretraining is more of an overfitting paradigm (as we always want to include as much data as we can obtain for pretraining, which dramastically reduces the gap between training and testing). The paper does not discuss when the bound is meaningful or vacuous for large-scale pretraining, whether the intended setting is instead low-data / non-stationary / distribution-shift (fine-tuning, RL, narrow-domain growth), or how the certificate behaves under train/test shift.

W4. "Dimension" is not well justified

The standard participation ratio is tr⁡(Σ)2/tr⁡(Σ2)\operatorname{tr}(\Sigma)^2/\operatorname{tr}(\Sigma^2), which equals NN for isotropic covariance. Here the numerator is the signal energy ∥μˉ∥4\|\bar\mu\|^4, making these signal-to-noise ratios rather than geometric dimensions. There is no insurance that ∥μˉ∥22∝O(N)\|\bar\mu\|_2^2 \propto O(N). A formal justification or qualified terminology would help.

W5. Estimator rigor for dalignd_{\text{align}} is missing

The paper doesn't report how many activation-gradient samples are used to estimate μˉ^/Σ^\hat{\bar\mu}/\hat\Sigma, gives no confidence intervals for d^align\hat d_{\text{align}}, and doesn't address finite-sample bias of the ratio d^2\hat d_2. The Table 8 std-devs are across training seeds, not estimation error, so it's unclear whether adjacent-width differences are meaningful.

W6. No link to downstream outcomes or baseline predictors

dalignd_{\text{align}} is never correlated with downtream results (ResNet accuracy in Table 7, Pythia perplexity in Table 5 does not show the relationship with dalignd_{\text{align}}), and there is no comparison against simple baselines like ∥q∥22\|q\|_2^2, training loss, parameter count, or tr⁡(Σ)\operatorname{tr}(\Sigma), so the added value of the statistic isn't shown.

W7. Largely validated on its own terms

dalignd_{\text{align}} is defined and shown to co-vary with width, and used to predict the loss change of the insertion whose derivative defines it. What is missing is evidence of external utility—e.g., that high-dalignd_{\text{align}} points outperform low-dalignd_{\text{align}} points, or high-dalignd_{\text{align}} points are easier to conduct width expansion, etc. Otherwise, I cannot see the practical use of this dalignd_{\text{align}} indicator.

Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Weaknesses 1 and Weaknesses 7

The central research question of this paper is how model width affects activation-gradient alignment and local directional transfer under finite sampling. To address this question, Figures 1–3 compare independently trained models of different widths while keeping depth, training data, and token budget fixed. This controlled design examines the relationships between width, gradient statistics, effective alignment dimension, and misalignment probability. Training the models independently from scratch is therefore consistent with our research objective. The residual-insertion experiments in Figure 4 examine the connection between local directional transfer in the theory and changes in held-out loss.

To further examine actual width growth, we have added local FFN widening experiments on Pythia-2.8B and Pythia-6.9B, using continued training without widening under the same computational budget as the control. The widening settings, layer-wise results, and comparisons of layer-selection strategies are provided in Tables R1 and R2 of our response to Reviewer jTs7. These experiments directly examine whether the pre-widening metric can help select locations for subsequent growth.

Weaknesses 2

The detailed layer-wise results and comparisons of layer-selection strategies are provided in Tables R1 and R2 of our response to Reviewer jTs7.

The metric and gains are positively associated across locations, and the pre-widening metric can assist practical location selection. This pattern also appears in models with larger parameter counts.

Weaknesses 3

Our primary application is the analysis of local directional transfer at a given checkpoint and hidden location in a pretrained model. In the current experiments, training-side and test-side gradient samples come from independently sampled C4 subsets. We use an evaluation setting in which both sides share the same data distribution, without deliberately introducing distribution shift. Thus, the relatively low misalignment rates in Figure 2 indicate that the evaluated models and locations have relatively stable local gradient directions under the current C4 setting.

Our focus is to quantitatively explain this stability and how it varies with model width and sampling budget. Even when misalignment rates are already low, comparing gradient statistics and misalignment probabilities across widths helps us understand which models have more reliable local directions under the same finite sampling budget, and what signal and noise structures underlie these differences.

For the Pythia widening experiments above, we set M=K=8M=K=8 and compute BalignB_{\mathrm{align}} from the estimated statistics as follows. Each range covers the six candidate locations in the corresponding model and is not a confidence interval.

ModelUpper-bound range before wideningUpper-bound range after widening and continued training
Pythia-2.8B2.594%–3.643%2.336%–3.270%
Pythia-6.9B2.515%–3.138%2.270%–2.825%

For further applications, please see our response to Reviewer jTs7.

W4

The terminology can be explained through a simple example. Consider a gradient vector with kk independent coordinates, each with mean σ>0\sigma > 0 and variance σ2\sigma^2, so that each coordinate has a GSNR of 1. Then,

μ=σ1k,Σ=σ2Ik,\mu=\sigma\mathbf{1}_k,\qquad \Sigma=\sigma^2I_k,

and

\mu^\top\Sigma\mu=\operatorname{tr}(\Sigma^2)=k\sigma^4.$$ Substituting these quantities into our definitions gives $$d_{\parallel}=d_2=d_{\mathrm{align}}=k.$$ We therefore use "effective dimension" to compare this metric with the number of independent coordinates having unit GSNR. We are also considering renaming the metric "effective alignment number" to avoid potential misunderstanding of the term "dimension."
AuthorsOpenReview ↗

Rebuttal by Authors

W5

The paper uses 10,000 C4 sequences, each containing 2,048 tokens, for estimation.

To further evaluate estimation variability caused by data sampling, we have added ten independently sampled C4 subsets, each also containing 10,000 sequences of 2,048 tokens each. With the model checkpoint and measurement location fixed, we separately re-estimate the gradient statistics and dalignd_{\mathrm{align}}. The results are as follows.

ModelSequences per subsetIndependent repetitionsdalignd_{\mathrm{align}}: mean ± SDApproximate 95% interval for the mean of repeated estimates
Pythia-2.8B10,000108.0684 ± 1.0354[7.3277, 8.8091]
Pythia-6.9B10,000109.4837 ± 1.2485[8.5906, 10.3768]

W6: Comparison with simple gradient statistics

We use the same six candidate locations and continued-training results under the same computational budget as in our response to Reviewer jTs7. Here, we compare simple pre-widening gradient statistics with the effective alignment dimension.

ModelLayerMean squared gradient norm GGTotal noise tr⁡(Σ)\operatorname{tr}(\Sigma)dalignd_{\mathrm{align}}Test-loss gain Δℓ(C)↑\Delta_\ell(C)\uparrow
2.8B40.247320.094187.95030.07652
2.8B80.260490.099289.98840.12201
2.8B120.230540.087628.11380.05917
2.8B160.287890.109648.33100.08509
2.8B240.242970.092797.03140.06228
2.8B320.272130.102288.25510.11542
6.9B40.317980.119959.39890.08553
6.9B80.304120.116519.61910.09701
6.9B120.330090.126898.58500.07145
6.9B160.297050.114338.20870.04621
6.9B240.316870.1186210.30620.10520
6.9B320.324520.122978.26120.06772

For Pythia-2.8B and Pythia-6.9B, maximum dalignd_{\mathrm{align}} selects layers 8 and 24, respectively, achieving the highest mean test-loss gain among the six candidate locations in each model. In comparison, the maximum mean squared gradient norm selects layers 16 and 12, with gains of 0.08509 and 0.07145, respectively; minimum total noise selects layers 12 and 16, with gains of 0.05917 and 0.04621, respectively.

These results connect pre-widening dalignd_{\mathrm{align}} with actual test performance after continued training under the same computational budget. They show that, among the evaluated candidate layers, dalignd_{\mathrm{align}}, which combines signal strength with directional noise structure, selects widening locations with higher mean gains than rules based solely on overall gradient energy or total noise.

Furthermore, using the pre-widening signal-to-noise ratio reported in our response to Reviewer JvjR's Weaknesses 2,

s=∣μ∣2tr⁡Σ,s=\frac{\\|\mu\\|^2}{\operatorname{tr}\Sigma},

we can directly construct a normalized-noise-scale baseline in the same activation-gradient space:

tr⁡Σ∣μ∣2=1s.\frac{\operatorname{tr}\Sigma}{\\|\mu\\|^2}=\frac{1}{s}.

Here, we compute the score 1/s1/s using the aggregate pre-widening ss values in the table. Minimizing this score is equivalent to selecting the location with the largest ss. The reported candidate-layer statistics and gains yield the following comparison.

Selection strategy2.8B selected layer2.8B mean gain6.9B selected layer6.9B mean gain
Minimum normalized noise scale 1/s1/s320.11542240.10520
Maximum dalignd_{\mathrm{align}}80.12201240.10520

This comparison is obtained by summarizing existing pre-widening statistics and continued-training results for the candidate locations; it does not involve additional continued-training experiments. Compared with this normalized-noise-scale baseline, maximum dalignd_{\mathrm{align}} selects a location with a higher mean gain on 2.8B. On 6.9B, both strategies select the same location and obtain the same gain.

AuthorsOpenReview ↗

Follow up on rebuttal

Dear Reviewer xtQe,

We apologize for disturbing you. As the discussion deadline approaches, we are writing to follow up and ensure our previous response and revisions fully address your concerns.

We believe that these improvements have significantly strengthened the quality of the paper. We would be extremely grateful if you could take a moment to review our reply and re-evaluate our work. We would be happy to engage in further discussion if needed.

Best regards, The Authors

Reviewer_xtQeOpenReview ↗

Ack of Rebuttal

Thank you for your reply. However, I am still questioning the real value of this paper. Even if the d_align has shown correlation with scaling up width in training, the core question still remain unresolved that, How does d_align guide the width expansion during training or simply scaling up models? How does current scaling benefit from this metric? What principle does this metric reveal for scaling up models, is it "the core improvement of scaling up width comes from d_align improvement"? What should we really do when we are conducting width scaling from the perspective of this paper?

I will maintain my score for now.

Reviewer_JvjROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the alignment between activation-gradients estimated from two different sample datasets. The authors derive variance identity used to obtain a misalignment bound, which is used as the effective alignment dimension. The main calculations appear sound, but the novelty and implications of this work is limited.

Critical strengths
  • The way the alignment bound was derived is intuitive and straightforward. The bound requires only finite second moments (which is expected), and seems to retain more information about gradient geometry than bounds in earlier works.
  • The experiments cover multiple setups (architectures, datasets, widths).
  • Some limitations are acknowledged (e.g. distinction between local improvement and improvement of the final trined model)
Critical weaknesses
  • Limited theoretical novelty: the main result (Thm1) follows from standard variance and concentration inequality calculations. Since this result is presented as the main contribution, it should include a significantly novel contribution. In the current form, it is just a simple application of standard results to an existing framework.
  • Impact on width scaling: The work does not explain why width scaling should improve the proposed alignment metric. It is unclear what the impact of this work on how width scaling is conducted.
  • Empirical validation: there are potentially some problems in the reported numbers. For instance, at width 512, the reported alignment dimension is 4. From Fig1b, for M=K=1, the formula gives Balign≈1/(1+4/3)=43B_{align} \approx 1/(1 + 4/3) = 43%, while in Fig 2a shows roughly 3333%. Not sure why there is this mismatch.
  • Writing: there are many sentences in the paper that are very loose. For instance, I’m not sure what “well trained reference model” means. Also, even in the abstract, we found sentences such as ”Existing theories of neural-network width characterize asymptotic limits, but provide limited guidance on whether an expansion direction identified from finite training data remains beneficial on unseen data”. It is unclear what “expansion direction” means. The authors mentioned that they used AI in writing, which might be the reason such “weird” wording appears in the paper.
Questions for authors

Please see weaknesses.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Weaknesses 1

We wish to clarify our contributions. To our knowledge, this work is the first to introduce the effective alignment dimension dalignd_{\mathrm{align}} into the analysis of local expansion in normalized residual networks and establish a finite-sample relationship whereby this dimension and the effective sample size jointly control the train-test activation-gradient misalignment probability.

Specifically, we introduce

d_{\mathrm{align}}=\min\left\\{\frac{\\|\mu\\|^4}{\mu^\top\Sigma\mu},\frac{\\|\mu\\|^4}{\operatorname{tr}(\Sigma^2)}\right\\},

unifying directional noise and covariance energy in an alignment characterization, and obtain

Pr⁡(μM⊤gK≤0)≤11+neffdalign.\Pr(\mu_M^\top g_K\le0)\le\frac{1}{1+n_{\mathrm{eff}}d_{\mathrm{align}}}.

This enables quantitative analysis of finite-sample directional-transfer reliability from model-specific gradient statistics and explains how width affects reliability through signal and noise structure, providing a new perspective on model training.

Our novelty lies in the effective alignment dimension, its finite-sample theoretical characterization, and the resulting explanation of the role of width. Using established probabilistic tools in the proof does not mean that prior work has already provided this specific metric and theoretical relationship.

Weaknesses 2

Section 4.2 and Appendix K provide a conditional width analysis: width changes the activation-gradient distribution, whose signal and noise structure determines alignment reliability. To explain this mechanism, we write

d∥=sa,d2=s2reff,dalign=min⁡d∥,d2,d_\parallel=\frac{s}{a},\qquad d_2=s^2r_{\mathrm{eff}},\qquad d_{\mathrm{align}}=\min\\{d_\parallel,d_2\\},

where

r_{\mathrm{eff}}=\frac{(\operatorname{tr}\Sigma)^2}{\operatorname{tr}(\Sigma^2)},\qquad a=\frac{\mu^\top\Sigma\mu}{\\|\mu\\|^2\operatorname{tr}\Sigma}.$$ If the overall signal-to-noise ratio remains unchanged or increases, a higher effective noise dimension helps increase $d_2$, while a lower noise fraction along the mean-gradient direction helps increase $d_\parallel$. This decomposition provides concrete criteria for assessing whether width changes improve alignment structure. Widening settings and test-loss gains under the same computational budget are reported in Tables R1 and R2 of our response to Reviewer jTs7. Here, we decompose the signal and noise structure to explain the statistical changes. **Table R2. Signal and noise statistics before and after local FFN widening and continued training.** Entries show before → after. | Model | Layer | $s$ | $r_{\mathrm{eff}}$ | $a$ | $d_{\parallel}$ | $d_2$ | |---|---:|---:|---:|---:|---:|---:| | 2.8B | 4 | 1.63 → 1.63 | 4.24 → 4.91 | 0.20 → 0.19 | 7.95 → 8.63 | 11.22 → 13.12 | | 2.8B | 8 | 1.62 → 1.62 | 4.69 → 5.59 | 0.16 → 0.15 | 9.99 → 11.18 | 12.36 → 14.72 | | 2.8B | 12 | 1.63 → 1.64 | 4.28 → 4.91 | 0.20 → 0.18 | 8.11 → 9.28 | 11.38 → 13.16 | | 2.8B | 16 | 1.63 → 1.61 | 4.88 → 5.89 | 0.20 → 0.17 | 8.33 → 9.40 | 12.89 → 15.30 | | 2.8B | 24 | 1.62 → 1.63 | 4.38 → 5.44 | 0.23 → 0.21 | 7.03 → 7.88 | 11.47 → 14.37 | | 2.8B | 32 | 1.66 → 1.66 | 4.45 → 5.17 | 0.20 → 0.18 | 8.26 → 9.27 | 12.26 → 14.18 | | 6.9B | 4 | 1.65 → 1.66 | 5.13 → 6.01 | 0.18 → 0.15 | 9.40 → 10.86 | 13.99 → 16.53 | | 6.9B | 8 | 1.61 → 1.62 | 5.91 → 6.86 | 0.17 → 0.15 | 9.62 → 11.02 | 15.32 → 17.99 | | 6.9B | 12 | 1.60 → 1.61 | 4.38 → 5.15 | 0.19 → 0.17 | 8.59 → 9.74 | 11.24 → 13.29 | | 6.9B | 16 | 1.60 → 1.62 | 3.90 → 4.47 | 0.19 → 0.17 | 8.21 → 9.34 | 9.98 → 11.77 | | 6.9B | 24 | 1.67 → 1.68 | 4.28 → 4.81 | 0.16 → 0.15 | 10.31 → 11.44 | 11.96 → 13.60 | | 6.9B | 32 | 1.64 → 1.64 | 5.92 → 6.89 | 0.20 → 0.18 | 8.26 → 9.16 | 15.90 → 18.55 | These results show a consistent pattern: - The effective noise dimension $r_{\mathrm{eff}}$ increases at every candidate layer, accompanying an increase in $d_2$. - The noise fraction $a$ along the mean-gradient direction decreases at every candidate layer, accompanying an increase in $d_\parallel$. The overall gradient signal-to-noise ratio $s$ changes little, while the fraction of gradient noise variance along the mean gradient direction, $a$, decreases. This decrease accounts for most of the increase in $d_{\parallel}=s/a$. The increase in $d_2=s^2r_{\mathrm{eff}}$ comes primarily from a higher effective rank of the gradient noise covariance.
AuthorsOpenReview ↗

Rebuttal by Authors

Weaknesses 3

Figures 1 and 2 both use 10,000 C4 sequences of 2048 tokens each. Each experiment uses a different, independently sampled subset. In Fig. 2, each sampling budget (M,K) also uses its own data pools, and we estimate the corresponding gradient statistics and d^align\hat{d}_{\mathrm{align}} separately for each budget.

Finite-sample estimates can differ across data pools. Therefore, the estimates reported in Fig. 1b or the related tables cannot be used directly to recompute the results in Fig. 2. We have rerun the relevant experiments and revised the corresponding parts of the manuscript.

Weaknesses 4

As we are not native English speakers, parts of the manuscript were translated with GPT assistance. We apologize for imprecise wording that made the original manuscript difficult to read and understand. We take responsibility for the accuracy of its content and presentation. We will carefully address the reviewer's concerns and check terminology, definitions, and sentence-level expression throughout the manuscript to improve its accuracy and readability.

AuthorsOpenReview ↗

Follow up on rebuttal

Dear Reviewer JvjR,

We apologize for disturbing you. As the discussion deadline approaches, we are writing to follow up and ensure our previous response and revisions fully address your concerns.

We believe that these improvements have significantly strengthened the quality of the paper. We would be extremely grateful if you could take a moment to review our reply and re-evaluate our work. We would be happy to engage in further discussion if needed.

Best regards, The Authors

Reviewer_JvjROpenReview ↗

Response

Thank you for the additional comments and experiments. However, most of concerns remain unaddressed.

  1. The work does not even address or explain how this new "metric" should help width scaling: how would you use this in practice? why? and how does it help exactly? Your comment (weakness 2) is very loose and doesn't explain any of this.
  2. Your comment regarding "weakness 3" suggests that your metric varies significantly across subsets. How is it a reliable metric then?
  3. Regarding AI generated text: while I understand that most authors are not native English speakers, it is not an excuse to heavily use AI in writing.
Reviewer_xZJfOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether a model expansion direction identified from training data can also improve performance on held-out data. It derives an upper bound on the probability that the training and test activation-gradient directions are misaligned, in terms of the proposed dalignd_{\mathrm{align}} and the sample sizes. Experiments find larger dalignd_{\mathrm{align}} in wider models. This offers an interesting perspective on model performance under width scaling.

Critical strengths
  1. The paper studies an interesting problem. It defines a measurable dalignd_{\mathrm{align}} and proves a bound relating it to the probability of train-test gradient misalignment. This provides an interesting perspective on model width scaling.
  2. To the best of my understanding, the proof is sound.
Critical weaknesses
  1. There remains a gap between the paper's setting and the broader claim that model performance improves as width increases.
  2. The paper does not establish that dalignd_{\mathrm{align}} increases with width in general. The observed trend is supported by experiments, while the theoretical result requires additional assumptions.
Questions for authors
  1. I noticed that d∥d_{\parallel}, one component of dalignd_{\mathrm{align}}, seems closely related to the GSNR in Liu et al. (2020). Could the authors clarify this connection more explicitly?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Weaknesses 1

Theorem 2 links local directional transfer to the expanded model's test performance. Under its stated conditions, the final test loss strictly improves when the initial expansion's test gain ΔRtest\Delta_R^{\mathrm{test}} plus the subsequent optimization gain ΔERM\Delta_{\mathrm{ERM}} exceeds the generalization error terms. Theorem 1 uses dalignd_{\mathrm{align}} to quantify the reliability of transfer to independent test data.

Appendix B reports validation loss, validation perplexity, and test accuracy across widths. In the controlled LLaMA-style models, increasing hidden width from 512 to 3072 reduces late-training validation PPL from approximately 29.0 to 17.8, while aggregate dalignd_{\mathrm{align}} increases from 3.944 to 9.523. For ResNet-20, increasing the width multiplier from 0.5×0.5\times to 4×4\times raises mean CIFAR-10 accuracy from 88.21% to 94.49% and CIFAR-100 accuracy from 58.25% to 75.67%; the gradient-statistics experiments also show increasing dalignd_{\mathrm{align}} with width. These results support an empirical association between width, improved alignment statistics, and practical performance in language modeling and image classification.

We also added local FFN widening and layer-selection experiments under a matched computational budget. In both models, the location selected by pre-widening dalignd_{\mathrm{align}} achieves the highest mean test-loss gain among six candidates. Settings and results are in Tables R1 and R2 of our response to Reviewer jTs7 .

Weaknesses 2

Theorem 1 holds for a fixed model and measurement location under independent sampling, finite second moments, and a nonzero population mean gradient. It requires no assumptions about how the gradient signal or covariance scales with width.

Section 4.2 provides a conditional cross-width analysis: width changes the activation-gradient mean and covariance, affecting dalignd_{\mathrm{align}} and hence the misalignment-probability bound. Prior work derives width-dependent rates under

\lambda_{\max}(\Sigma^{(N)})\le C_\lambda\tau_N^2,\qquad \operatorname{tr}(\Sigma^{(N)})\le C_{\mathrm{tr}}N\tau_N^2,$$ where the constants are independent of $N$ and $\tau_N^2 > 0$ is a variance scale that may vary with width. These conditions require sufficient mean-gradient signal accumulation while controlling the strongest noise direction and total noise. They imply $d_{\mathrm{align}}(N)\ge c_0N$, recovering the corresponding width-dependent misalignment bound. **Question 1** Let the activation gradient $q$ have population mean $\mu\ne0$ and covariance $\Sigma$, and let $u=\mu/\\|\mu\\|$. Then $$\operatorname{GSNR}(u^\top q) =\frac{[\mathbb{E}(u^\top q)]^2}{\operatorname{Var}(u^\top q)} =\frac{(u^\top\mu)^2}{u^\top\Sigma u} =\frac{\\|\mu\\|^4}{\mu^\top\Sigma\mu} =d_{\parallel}.$$ Thus, $d_{\parallel}$ is exactly the GSNR of the activation gradient projected onto its population mean direction. Two-sample alignment also depends on interactions between sampling-noise vectors. For independent sample means $\mu_M$ and $g_K$ from the same gradient distribution, $$\frac{\operatorname{Var}(\mu_M^\top g_K)}{\\|\mu\\|^4} =\left(\frac{1}{M}+\frac{1}{K}\right)\frac{1}{d_{\parallel}} +\frac{1}{MKd_2}.$$ Here $d_2=\\|\mu\\|^4/\operatorname{tr}(\Sigma^2)$. The first term captures noise along the mean-gradient direction; the second captures the interaction between the two sampling-noise vectors. Both components therefore enter the analysis, and $d_{\mathrm{align}}=\min\\{d_{\parallel},d_2\\}$ yields the simplified bound in Theorem 1. Table 9 illustrates their distinct behavior. Among six-layer Pythia models, the component means at width 128 are $d_{\parallel}=5.5915$ and $d_2=3.4390$, whereas at width 256 they are $d_{\parallel}=7.6445$ and $d_2=14.2963$. Their ordering differs across these models. These separately aggregated means determine neither the mean of the per-repetition minimum nor how often either component is smaller. The metric $d_{\mathrm{align}}$ must be computed per evaluation unit before aggregation, rather than taking the minimum of the two means in Table 9. Thus, $d_{\parallel}$ has a precise GSNR interpretation, while $d_{\mathrm{align}}$ additionally accounts for the covariance-energy contribution needed for two-sample directional-transfer analysis.
AuthorsOpenReview ↗

Follow up on rebuttal

Dear Reviewer xZJf,

We apologize for disturbing you. As the discussion deadline approaches, we are writing to follow up and ensure our previous response and revisions fully address your concerns.

We believe that these improvements have significantly strengthened the quality of the paper. We would be extremely grateful if you could take a moment to review our reply and re-evaluate our work. We would be happy to engage in further discussion if needed.

Best regards, The Authors

Thank you for the detailed response. I have no further questions and will maintain my current rating. I suggest explaining the relationship between dalignd_{\mathrm{align}} and GSNR more clearly in the next version of the paper.

Reviewer_jTs7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper explores theoretical aspects of width scaling in normalized residual networks in the context of function-preserving residual expansion. The main question is whether an expansion direction estimated from training data transfers to test data gradients. The proposed theorems derive the exact mean and variance of the inner product between train and test activation gradients, and bound misalignment in terms of an "effective alignment dimension" and an effective sample size. This yields high probability guarantee for local directional transfer. The authors demonstrate through experiments on LLMs and ConvNets that wider models exhibit larger alignment dim and lower misalignment at fixed sample sizes.

Critical strengths
  • Clean theory with few assumptions. The required estimates can computed from a trained model. This makes it a potentially practical diagnostic.
  • Controlled empirical design. Width is varied at fixed depth, data and token budget, the trend holds across three model families.
Critical weaknesses
  • I see very little practical value. d_align is only shown to correlate with misalignment, but is never used to make a decision (e.g. where/when to grow, or which width to choose), nor compared to simpler quantities such as gradient norm or noise scale. I think authors should explore ways to increase practical value (see below for suggestions).
Questions for authors

I would very much like to see some additional practical implications for you work. Some examples, which can be verified with additional experiments:

Can we better select optimal width of model for fixed compute? Can we create better model- growth recipes (e.g. progressive stacking / Net2Net), using d_align to decide when and where to grow during training? Can d_align be used to select the insertion layer? does inserting at the layer with the highest d_align yield a larger test-loss gain than other layers?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We explicitly demonstrate how dalignd_{\mathrm{align}} can inform practical growth decisions through additional layer-wise FFN widening experiments and comparisons of layer-selection strategies.

Selecting widening locations. We compare each widening choice with continued training without widening under the same computational budget CC, and define the test-loss gain as

Δℓ(C)=Lcontrol(C)−Lwidened layer ℓ(C).\Delta_\ell(C)=L_{\text{control}}(C)-L_{\text{widened layer }\ell}(C).

A positive Δℓ(C)\Delta_\ell(C) indicates an improvement over continued training without widening under the same computational budget. The experimental setup and continued-training hyperparameters are described later.

Table R1. Alignment dimensions and test-loss gains. Here d=dalignd=d_{\mathrm{align}}; "after" means after widening and continued training.

ModelLayerdd beforedd afterΔℓ(C)↑\Delta_\ell(C)\uparrow
2.8B47.9503 ± 0.32988.6256 ± 0.55630.07652 ± 0.00988
2.8B89.9884 ± 0.443211.1843 ± 1.07920.12201 ± 0.01475
2.8B128.1138 ± 0.35989.2765 ± 0.64180.05917 ± 0.01521
2.8B168.3310 ± 0.35079.3999 ± 0.34480.08509 ± 0.01502
2.8B247.0314 ± 0.23387.8759 ± 0.43090.06228 ± 0.01470
2.8B328.2551 ± 0.34049.2723 ± 0.46590.11542 ± 0.01127
6.9B49.3989 ± 0.394110.8578 ± 0.59810.08553 ± 0.02264
6.9B89.6191 ± 0.359011.0181 ± 0.49040.09701 ± 0.01820
6.9B128.5850 ± 0.68709.7361 ± 0.72280.07145 ± 0.02205
6.9B168.2087 ± 0.34289.3351 ± 0.65520.04621 ± 0.02765
6.9B2410.3062 ± 0.382611.4382 ± 0.21970.10520 ± 0.02055
6.9B328.2612 ± 0.41759.1633 ± 0.55120.06772 ± 0.01469

Selecting the largest mean pre-widening dd yields layer 8 for Pythia-2.8B and layer 24 for Pythia-6.9B. These locations achieve the highest mean test-loss gains among the six candidates in their respective models, supporting the use of this metric for selecting widening locations in advance.

Comparison with other selection strategies. All strategies use the same candidate layers and computational budget. Metric-based selection uses pre-widening statistics.

Table R2. Selected layers and mean test-loss gains.

Strategy2.8B layer2.8B gain6.9B layer6.9B gain
Maximum dd80.12201240.10520
Fixed final layer320.11542320.06772
Minimum dd240.06228160.04621

Maximum-dd selection yields higher mean gains than these baselines in both models. The gains at the highest- versus lowest-dd locations are 0.12201 versus 0.06228 for 2.8B and 0.10520 versus 0.04621 for 6.9B. These results directly address whether higher metric values identify more beneficial growth locations.

Comparison with simple gradient statistics. Maximum-dd selection also yields higher mean gains than maximum-gradient-energy and minimum-total-noise selection in both models. Compared with minimum normalized noise scale, it yields a higher mean gain on 2.8B; on 6.9B, both strategies select the same layer and obtain the same gain. The signal/noise decomposition is provided in our response to Reviewer JvjR, Weaknesses 2.

These results support dalignd_{\mathrm{align}} as a pre-widening selection signal for the two models, six candidate layers, and budget studied. They do not imply that maximum-dd selection always yields the largest gain in every model, nor do they guarantee a globally optimal width or optimal timing for widening.

Data distribution and potential applications. At a fixed checkpoint and measurement location, dalignd_{\mathrm{align}} depends on the gradient mean and covariance induced by the data distribution. Data sources, mixture proportions, and preprocessing may change signal strength and noise structure, affecting finite-sample directional transfer. This provides a measurable way to study interactions between data characteristics and model width and evaluate the reliability of local updates under different data configurations.

AuthorsOpenReview ↗

Rebuttal by Authors

Starting from the final publicly released checkpoints of Pythia-2.8B and Pythia-6.9B, we double the FFN intermediate dimension at only one candidate layer at a time, considering layers 4,8,12,16,24,32\\{4,8,12,16,24,32\\}. The intermediate dimension increases from 10,240 to 20,480 for Pythia-2.8B and from 16,384 to 32,768 for Pythia-6.9B, adding approximately 2% to the original model's total parameter count. All existing parameters are preserved at initialization. Following the initialization setting in Table 3 of our paper, the input weights of the newly added neurons are independently sampled from N(0,0.022)\mathcal{N}(0,0.02^2), while the output weights corresponding to these neurons are initialized to zero. Thus, the widened model preserves the original model's function before continued training begins.

We reinitialize the AdamW optimizer for both the widening group and the control group without widening, resetting the first-moment estimates, second-moment estimates, and optimizer step counters for all parameters to zero. Both groups use the same C4 training data stream and data order, with a global batch size of 480 sequences and a sequence length of 2,048 tokens. The learning rate increases linearly to 3×10−43\times10^{-4} over the first 100 optimization steps and then follows a cosine decay schedule to 1.5×10−71.5\times10^{-7}. Both groups use the same number of warmup steps, peak learning rate, minimum learning rate, and decay form. The duration of the cosine decay phase is determined separately for each group by the stopping step at which its computational budget is reached.

AuthorsOpenReview ↗

Follow up on rebuttal

Dear Reviewer jTs7,

We apologize for disturbing you. As the discussion deadline approaches, we are writing to follow up and ensure our previous response and revisions fully address your concerns.

We believe that these improvements have significantly strengthened the quality of the paper. We would be extremely grateful if you could take a moment to review our reply and re-evaluate our work. We would be happy to engage in further discussion if needed.

Best regards, The Authors

I thank the authors for their answer. Rebuttal directly addresses my main concern about practical value. That said, the new results does not yet establish stron practical value such as whether we can reliably choose growth timing, target width, or globally optimal growth recipes. I would encourage the authors to include these new results in the paper, clearly label them as preliminary evidence for layer-selection, and discuss the limitations.

Overall, the rebuttal substantially improves my assessment of the paper’s practical relevance. I will raise my evaluation.

Comments on the paper

1 comment

Dear Reviewers,

Thank you to those who have already engaged with the authors' rebuttal. As we approach the end of the discussion period, if you have not yet replied, please do so soon.

Thank you for your time and effort in contributing to the ICLR community.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.