- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper characterizes fixed-batch, single-channel batch normalization as the solution of a convex orbit-selection problem. It connects this construction to scale invariance, gradient projection, zero-variance degeneracy, and shift absorption, then proposes a four-part account of normalization benefits. The backward-only intervention is promising, and the elementary variational identity is useful. However, the explanatory contribution is less established than the algebraic characterization, the empirical decomposition remains confounded, and several mathematical statements require qualification.
Critical strengths
Theorem 1 and Proposition 1 give transparent convex objectives whose minimizers recover centering and standard-deviation scaling. The joint location-scale formulation is particularly useful because its strict convexity and closed-form solution can be checked directly. This provides a coherent mathematical description of the operator.
Critical weaknesses
The divisor experiment does not isolate backward obliqueness. The backward ablations do not isolate geometric projection benefits from altered optimization. The writing style appears too AI-generative.
Questions for authors
- Divisor attribution. Does the range deficit persist when forward activations and their scale are held fixed while only the backward projection is changed?
- Optimization controls. Do the backward-ablation losses persist under matched weight norms and angular step sizes, and ordinary per-image augmentation?
Thank you for reading the paper. The report shows us where the exposition fails, and we answer each point.
"It is obvious from the definition of BN that this is what it does." We agree that Thm. 1 is elementary. The paper says its proof is "two one-dimensional convex minimizations". It also took too much space, and the revision cuts Sec. 2 by about half. Its role is to define the set of alternatives to BN: every other rule that picks one point from the same family {cz + a·1}. The results that do not follow from BN's definition are these.
Terminology. You are right that the coined terms obscure the text. "Payload ledger" was our name for a list of four candidate sources of BN's benefit. "Angular thermostat" referred to a known fact (Wan et al., 2021): with weight decay, the angle by which a scale-invariant weight vector rotates per step settles at about sqrt(2ηλ). Both terms are removed, and the revision uses plain words.
Corollary 3. In plain words: the normalization condition at layer ℓ involves only the outputs of earlier layers, so a single forward pass satisfies the conditions of all layers and no iteration across layers is needed. It is in the paper because methods that fix the same kind of redundancy in weight space, such as equi-normalization (Stock et al., ICLR 2019, raised by Reviewer K6hw), must iterate. The revision moves it next to that comparison.
What Figure 2 tests. Three separate things, one per panel.
We agree the figure does not serve the main results. In the revision the backward-ablation table takes its place.
Soundness. Every statement is proved in the paper and rechecked in float64 by the supplement's scripts, and the full-BN arm of our implementation matches PyTorch's BN to 1.8e-15. If a particular step looks wrong, please tell us which one.
We hope this makes clear what the paper proves beyond the definition of BN, and that you will consider updating your assessment.