- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies a width-one network with the non-monotone activation (), trained with logistic loss to separate an inner interval from two symmetric outer intervals, a task no monotone width-one network can solve. At fixed output scale , the loss minimizer maximizes the class-mean gap at small , and the worst-case gap at large . On the paper task, only maximizing the worst-case gap leads to hidden-unit placement, i.e. whose output separates the two classes. The scale at which the minimizer switches is located with computer-assisted certificates, and its scaling as is derived. During training with Adam or SGD, the unit becomes placed a few percent above this switch scale. The authors model this lag with a first-order slow-manifold correction, in which the lag is proportional to the ratio of output-scale growth to hidden-parameter relaxation, and evaluate it in preregistered tests, including one at an activation value without prior training data. The predictions hold in the median under both optimizers and per run under SGD, but not per run under Adam. Extensions to width two, other non-monotone activations (GELU, SiLU, Mish) and higher-dimensional inputs fail or remain unresolved.
Critical strengths
None
Critical weaknesses
In its current form, the manuscript is not suitable for publication as a conference paper. The main text is a condensed index of a 47-page appendix running from A to Y, and most claims can only be understood, let alone checked, by going through the appendices. The main text does not clearly state the question, the main results, or the scope, and as a reader I found it very difficult to evaluate the contributions from it. Many sentences report the verification status of results whose content has not been explained, and the notation reads as an internal log of tests rather than a presentation of findings. The paper also reports the outcome of every prediction made during the project. This is useful as research notebook, but it is not the role of a conference paper, and it leaves the reader to work out which results actually support the main claims. In its current form, I do not think the work can be fairly understood, evaluated, or used by an external reader.
This cannot be fixed by a revision within the review cycle. I encourage the authors to choose the few claims that matter, build the paper around them, and move the audit trail to supplementary material. They should also demonstrate that their results extend beyond their toy model. The extensions reported in the appendices fail or remain unresolved, so the paper currently offers no setting beyond a width-one sine network in which the results hold.
Questions for authors
None