acceptodds
Withdrawn from ICLR 2027

From Class Means to Margins: Predicting Hidden-Unit Placement in Width-One Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

At small output scale, logistic loss favors separation of class means; at large scale, it favors the worst-case margin. We study a width-one network with activation on symmetric inner-versus-outer intervals, the smallest setting in which these preferences require different hidden features. An independently re-checked certificate locates the placement switch, and a scaling reduction predicts its activation dependence with a first-order coefficient confirmed at finite parameter values. Registered prospective tests predict training crossings from conditional thresholds fixed before training. A first-order tracking law, derived post hoc, predicts each run's lag from the branch geometry and its measured ratio of output-scale growth to hidden relaxation. At an activation value with no prior crossing data, its registered test gives median per-run observed/predicted ratios of 1.07 under Adam and 1.06 under SGD; individual predictions are accurate under SGD, where every run is within 10%, but not under Adam, where 22% are. A registered learning-rate test finds that the lag does not vanish as the learning rate decreases. This quantitative predictive link is established for width-one sine networks that reach a branch and grow slowly relative to relaxation; outside this setting, registered predictions fail or remain unresolved, and agreement conditioned on the eventual branch is explanatory.

Primary area:
optimization
Keywords:
implicit bias, margin maximization, regularization path, feature learning dynamics, simplicity bias, output scale, timescale separation, computer-assisted proof, non-monotone activations

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_ahGBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies a width-one network N(x)=w2fa(w1x+b1)+b2N(x) = w_2 f_a(w_1 x + b_1) + b_2 with the non-monotone activation fa(t)=t+asin⁡tf_a(t) = t + a \sin t (a>1a > 1), trained with logistic loss to separate an inner interval from two symmetric outer intervals, a task no monotone width-one network can solve. At fixed output scale ss, the loss minimizer maximizes the class-mean gap at small ss, and the worst-case gap at large ss. On the paper task, only maximizing the worst-case gap leads to hidden-unit placement, i.e. whose output separates the two classes. The scale at which the minimizer switches is located with computer-assisted certificates, and its scaling as a→1a \rightarrow 1 is derived. During training with Adam or SGD, the unit becomes placed a few percent above this switch scale. The authors model this lag with a first-order slow-manifold correction, in which the lag is proportional to the ratio of output-scale growth to hidden-parameter relaxation, and evaluate it in preregistered tests, including one at an activation value without prior training data. The predictions hold in the median under both optimizers and per run under SGD, but not per run under Adam. Extensions to width two, other non-monotone activations (GELU, SiLU, Mish) and higher-dimensional inputs fail or remain unresolved.

Critical strengths

None

Critical weaknesses

In its current form, the manuscript is not suitable for publication as a conference paper. The main text is a condensed index of a 47-page appendix running from A to Y, and most claims can only be understood, let alone checked, by going through the appendices. The main text does not clearly state the question, the main results, or the scope, and as a reader I found it very difficult to evaluate the contributions from it. Many sentences report the verification status of results whose content has not been explained, and the notation reads as an internal log of tests rather than a presentation of findings. The paper also reports the outcome of every prediction made during the project. This is useful as research notebook, but it is not the role of a conference paper, and it leaves the reader to work out which results actually support the main claims. In its current form, I do not think the work can be fairly understood, evaluated, or used by an external reader.

This cannot be fixed by a revision within the review cycle. I encourage the authors to choose the few claims that matter, build the paper around them, and move the audit trail to supplementary material. They should also demonstrate that their results extend beyond their toy model. The extensions reported in the appendices fail or remain unresolved, so the paper currently offers no setting beyond a width-one sine network in which the results hold.

Questions for authors

None

Reviewer_fPvWOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This article studies a width-one network with activation $ f_a(t) = t + a \sin( t)$ on what the authors call ``symmetric inner-versus-outer intervals''. Beyond that point, I did not comprehend the rest of the abstract of that article, with many concepts and keywords thrown over, let alone the main text.

Critical strengths

I am not able to evaluate the strengths of this article because I could not understand the nature of the contributions nor the motivation of the study.

Critical weaknesses

I find this article completely unreadable, and it starts from the abstract that is unintelligible, keywords are thrown all over and it's unclear what is the core motivation of the study in the first place. The authors do not explain whether the problem and/or contributions are new: looking into the introduction reveals that there is no comparison with the literature, there is only a section about Related work that is also unintelligible.

Questions for authors

NA

Reviewer_LxE7OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies width 1 networks on a toy setting.

Critical strengths

None

Critical weaknesses
  • The paper is extremely hard to read. It's structured more like a research log report rather than an academic paper.
  • The studied setting is extremely simple and the utility is unclear.
Questions for authors

Could you explain the significance of the results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.