- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the inductive biases of scaling model width and training time on a synthetic task with spurious features. They find that scaling primarily width encourages the use of the spurious feature shortcuts, while scaling training time results in finding the true solution (with lower loss).
Critical strengths
The inductive biases of different types of scaling is a significant and interesting question, and the use of synthetic settings like parity is an established and effective way to gain insight into such questions in deep learning.
Critical weaknesses
The experiments done by the paper are insufficient to support the claims made. They swept over only one parameter (eta) of the one setting (not testing different sized inputs or rule features), used only one architecture and no sweep of learning rate, batch size, depth, or weight decay. Claims about the effects of width and training time are not really useful without at least sweeping learning rates to be able to compare the results at the best learning rate for each width and training time. In many similar settings I have seen choice of learning rate (or small architectural choices, such as the type of layer norms) completely change the results of experiments. Such sweeps wouldn't require an onerous amount of compute for this setting, in my eyes one of the main benefits of a 'toy' synthetic setting such as this one is that orders of magnitude more training runs can be done at little cost of compute.
Additionally, even if the current experiments were better, the paper would still read more like a workshop paper, since it is rather single note, neither connecting to more practical settings than parity nor advancing the state of deep learning theory through further theoretical understanding.
Questions for authors
Clarification: Did you sweep hyperparameters besides width and training time? Potential Improvement: Sweeping over at least lr, doing so well enough to know that finding better lrs wouldn't change results.
Other potential improvements would be showing that the results carry over to more settings, including other synthetic settings but also more applied ones (especially interesting if it can be shown that the results apply to applications such as finetuning LLMs). Alternatively (or even better, additionally) theoretical understanding could help the paper, as long as it actually increases our understanding, such as being useful for making new specific verifiable (and non-trivial) predictions that could then be verified, leading evidence that the theory actually leads to real understanding.
4 comments
I appreciate the additional experiments, which strengthen the paper. Unfortunately, the paper is still sloppy, and a bit narrow, so I can only raise the score to weak reject. The result is interesting, but I believe that the paper is still missing either additional understanding or extension to more realistic settings.
As a more minor presentation note, the use of as a parameter of the setting is a bit awkward as is normally used to represent learning rate.
Additionally, the added AI use statement does not cover the required areas as mentioned here: https://iclr.cc/Conferences/2027/AIPolicyForAuthors
Thank you for reading the revision and indicating the score increase. We have uploaded a new PDF that responds to each point of your comment.
Notation. The disagreement rate is now written ε throughout the text, tables and figures, so η no longer appears in the paper and cannot be read as a learning rate. Learning rates are always given as numbers, such as 3×10⁻³. The change also unifies the notation with the introduction, which already used ε for the same quantity. Our earlier replies in this thread write η for the same rate.
Presentation. We did a thorough precision pass over the manuscript so that every count, average and step window carries its exact scope: the acquisition fractions in Section 7 (0.22 at d=128 against 0.86 at d=16) state the 10⁴-step window of that run explicitly (at 5000 steps Figure 5a gives 0.16 and 0.70), the 61,568 new networks and 49,024 three-bit parity networks state which grids they comprise (with 830 of the 836 non-exact networks computing partial parity), cell-level bounds specify ε-averaging, the best learning rate notes the tie at d=64, and the median-based remark in Section 7 is replaced by an exact count showing that all 2,588 rule-acquiring seeds at 3×10⁻⁴ and 3×10⁻³ first fell below m = −½.
Understanding. The step-size account of the width effect is now measured directly as well as tested. At the fixed rate 3×10⁻³ the root-mean-square change of a unit's input in one step grows from 2.7×10⁻³ at d=16 to 15.8×10⁻³ at d=128, close to d^0.86, and with the learning rate scaled as 3×10⁻³·32/d it differs by at most a factor 1.2 across widths (Section 5.1, Appendix E). At the fixed rate 3×10⁻³ the wider networks, which take the larger steps, leave the shortcut for the rule less often (Figure 5a), while at 3×10⁻⁴, where the steps are small at every width, the four widths have nearly the same acquisition-time distribution (Figure 5b). Scaling the learning rate as 1/d equalises the step and removes the width effect (Table 2). The Discussion now states separately what is measured, what is explained and tested by a prediction, and what is derived from measured premises (Proposition 1).
Practical settings. The step-size result bears directly on how width sweeps are run in practice. Under standard parametrisation with a fixed learning rate, wider models move toward the shortcut, whereas scaling the learning rate as 1/d, the rule that maximal-update parametrisation prescribes for hidden weights under Adam (Yang et al., 2022), removes the shift while keeping the benefit of more steps. Which hyperparameter-transfer rule is used therefore decides which algorithm the wider models learn, not only their loss. The probe construction that detects this, an evaluation set on which candidate algorithms disagree together with a probe of each network's own training exceptions, applies unchanged wherever the candidate algorithms can be named (Section 8).
AI use statement. Following the ICLR 2027 policy page you linked, we expanded the statement on page 11 so that it covers each required area: the required-disclosure categories (none used, with translation and dataset cleaning not applicable), the recommended-disclosure uses (language polishing and literature organisation), manual verification against the experimental records and original publications, and author responsibility for all content. If this resolves the policy concern, we would also be grateful if you could update the Official Review form when convenient, as it currently still displays the initial rating and ethics flag.
Thank you again for the detailed reviews, which led to the learning-rate sweep and to these measurements.
Rating on the Official Review form
In your 17 November comment you noted raising your score to weak reject, while the Official Review form above still shows the initial rating of 1. In case the edit did not save, could you update the rating field before the discussion window closes (Nov 18 AoE)? Thank you again for your time.
Thank you for the specific and actionable suggestions. We ran the learning-rate sweep you asked for at every width and training length, together with batch size, depth, weight decay and a second rule size; the results sharpen the paper's central claim and turn it into a tested prediction.
Every seed is trained under a budget-independent schedule (100 warm-up steps, then constant learning rate) at seven learning rates from 3e-5 to 3e-2, at every width, so that all budgets are read from the same run (Table 2, Appendix E); the one-cycle protocol of the main map is repeated at peak rates 3e-4, 1e-3, 3e-3 and 1e-2 (Appendix E). The grid brackets the optimum: above the best rate of each cell f falls monotonically with the learning rate, below it the rule is still being acquired when the budget ends.
Steps favour the rule everywhere: f is non-decreasing in the budget in every width column at every learning rate on both tasks (88 consecutive-budget comparisons on parity, 64 on recall).
At every fixed rate from 3e-4 to 3e-2, parity at d=128 lies below the best width in every cell where more than 10% of seeds acquire the rule at some width (e.g. 1e-3: 0.86 at d=64 vs 0.73 at d=128), and the same holds under one-cycle at 1e-3 and 1e-2. The optimal width shrinks as the rate grows: d=128 at 1e-4, 64 at 3e-4 and 1e-3, 32 at 3e-3, 16 at 1e-2.
The best learning rate per width and budget falls with width (3e-4 for d ≤ 32, 1e-4 for d ≥ 64 at 5000 steps), and at those rates f is 0.84, 0.86, 0.89 and 0.92 for d = 16–128. This is what carries the width effect at a fixed rate: a wider network at the same learning rate takes a larger effective step, since under standard parametrisation an Adam update moves a hidden unit's input in proportion to its fan-in.
This account predicts that scaling the learning rate as 1/d removes the width effect, and the experiments verify it. With lr = 3e-3·32/d, f at 5000 steps is 0.51/0.72/0.76/0.85 for d = 16/32/64/128, against 0.64 and 0.23 for d = 16 and 128 at the fixed 3e-3; with 1e-3·32/d it is 0.72/0.82/0.87/0.86 (Table 2).
At matched compute, that is at equal training FLOPs, the narrower network trained longer learns the rule at least as often in all 21 comparisons, at each of the seven learning rates and also when each width is trained at its own best learning rate (Section 5.1). This comparison trades width against steps at a fixed budget, and the longer-trained network is never behind.
We tested batch sizes 128 and 512, depth 4, weight decay 0 and 0.1, and a four-bit parity rule (Appendix E). In every variation f never decreases with the budget at any width, and at the final budget d=128 lies below the best width: at 5000 steps, depth 4 gives 0.61 (d=32) vs 0.18 (d=128); weight decay 0.1 gives 0.65 (d=64) vs 0.51; batch 128 and 512 give 0.46 and 0.89 (d=32) vs 0.11 and 0.50 (d=128). Four-bit parity is learned more slowly (after 10⁴ steps 0.14 at d=32, 0.02 at d=128), again with d=128 lowest.
The revised paper goes beyond a single note. It contains a resource map on two unrelated tasks; the step-size account of the width effect with a prediction that the experiments confirm, which is the kind of verifiable prediction you suggested; identification of the learned function on all 16 input classes with interventions and per-network memorisation probes, which establishes two discrete outcomes on parity and three on recall; and a parameter-free mixing law for the averaged accuracy. A practical consequence is that the learned algorithm in a width sweep depends on how the learning rate follows the width, which matters for hyperparameter transfer across model sizes (Section 8). The measurement itself carries over to applied settings wherever the candidate algorithms can be named and made to disagree on held-out inputs, as in the heuristic-versus-rule evaluations of fine-tuned language models discussed in Section 2.
On the hyperparameters swept, the submitted version varied width, step budget and η at learning rate 3e-3, batch size 256 and weight decay 0.01. The revision adds learning rate (7 values), schedule (2), batch size (3), depth (2), weight decay (3) and rule size (2), with 64 seeds per configuration and 61,568 new networks in total.
AI use statement (Ethics flag). Thank you for catching the missing mandatory AI use statement in our initial submission; we have added it on page 10 of the revised PDF.
We hope these results address your concerns.