acceptodds
Under review as a conference paper at ICLR 2027

Normalization as Gauge Fixing: A Variational Characterization of Batch Normalization

Abstract

Normalization layers appear in nearly every deep network, yet why they help is still unsettled: BN's original internal-covariate-shift explanation has been challenged, and the six later accounts each rest on measurements, yet none is load-bearing alone, since networks train competitively without a normalizer. Recent work has pinned down the geometry of what the operator produces, the centered sphere its outputs lie on, but not the problem that geometry solves. We supply that problem. Rescaling a neuron's incoming weights leaves a normalized network's function unchanged, so many parameter settings compute the same map and normalization picks one: gauge fixing, in the language of constrained mechanics. For one batch and one channel, batch normalization's two steps are the exact closed-form solution of a strictly convex selection problem over this redundancy, verified symbolically and numerically. Short proofs follow: why a zero-variance channel breaks the operator, what epsilon repairs, why depth needs no iteration, and when a centering folds into the next layer for free. The objective also says why this operator and not a near neighbour: other divisors remove the same redundancy and leave the forward pass unchanged, but only the standard deviation has a backward pass that distorts no direction, and the alternatives lose accuracy in proportion to their distortion, up to points. The same structure splits the benefit into four parts, one of which we measure cleanly: holding the forward pass identical and changing only the backward pass, the batch-coupled gradient terms are worth to accuracy points on CIFAR-10.

Primary area:
optimization
Keywords:
batch normalization, gauge fixing, variational characterization, scale invariance, optimization geometry, gradient dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_yMCHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper characterizes fixed-batch, single-channel batch normalization as the solution of a convex orbit-selection problem. It connects this construction to scale invariance, gradient projection, zero-variance degeneracy, and shift absorption, then proposes a four-part account of normalization benefits. The backward-only intervention is promising, and the elementary variational identity is useful. However, the explanatory contribution is less established than the algebraic characterization, the empirical decomposition remains confounded, and several mathematical statements require qualification.

Critical strengths

Theorem 1 and Proposition 1 give transparent convex objectives whose minimizers recover centering and standard-deviation scaling. The joint location-scale formulation is particularly useful because its strict convexity and closed-form solution can be checked directly. This provides a coherent mathematical description of the operator.

Critical weaknesses

The divisor experiment does not isolate backward obliqueness. The backward ablations do not isolate geometric projection benefits from altered optimization. The writing style appears too AI-generative.

Questions for authors
  1. Divisor attribution. Does the range deficit persist when forward activations and their scale are held fixed while only the backward projection is changed?
  2. Optimization controls. Do the backward-ablation losses persist under matched weight norms and angular step sizes, and ordinary per-image augmentation?
1 comment

Thank you for reading the paper. The report shows us where the exposition fails, and we answer each point.

"It is obvious from the definition of BN that this is what it does." We agree that Thm. 1 is elementary. The paper says its proof is "two one-dimensional convex minimizations". It also took too much space, and the revision cuts Sec. 2 by about half. Its role is to define the set of alternatives to BN: every other rule that picks one point from the same family {cz + a·1}. The results that do not follow from BN's definition are these.

  1. Uniqueness. Dividing by the range or by the mean absolute deviation, or centering by the median, removes the same redundancy and gives the same forward invariance. Only BN has a symmetric Jacobian (Prop. 2; extended to every smooth rule as Prop. 2′ in our reply to H5CK). We agree that computing the Jacobian is routine. The content is the "only if". Reviewer 8cbZ singled this result out as "particularly interesting".
  2. What a truncated backward pass does. Dropping the mean term of BN's gradient adds ∂L/∂β times the input mean to the weight gradient, exactly (Eq. 6). This predicts that the damage grows with the input mean. On digits the mean-ablated arm goes from 98.8% to 90.7%, 74.5% and 31.5% as the input mean is raised, while full BN stays at 98.5% (reply to 8cbZ).
  3. A boundary. Folding a fixed centering into the next layer's bias is exact to 1e-15 before a dense layer or a circular-padding convolution. It is off by 0.07 to 0.12 before a zero-padded convolution (Prop. 3).
  4. A measurement. With the forward pass identical in all arms, removing the batch-coupled gradient terms costs 7.8 and 23.2 points on CIFAR-10 (Sec. 4.3), and 6.4 and 8.0 points in a new control that matches weight norms and step sizes across arms. Reviewer H5CK calls this "a cleaner design than most prior ablations of BN", and Reviewer yMCH calls the intervention "promising".

Terminology. You are right that the coined terms obscure the text. "Payload ledger" was our name for a list of four candidate sources of BN's benefit. "Angular thermostat" referred to a known fact (Wan et al., 2021): with weight decay, the angle by which a scale-invariant weight vector rotates per step settles at about sqrt(2ηλ). Both terms are removed, and the revision uses plain words.

Corollary 3. In plain words: the normalization condition at layer ℓ involves only the outputs of earlier layers, so a single forward pass satisfies the conditions of all layers and no iteration across layers is needed. It is in the paper because methods that fix the same kind of redundancy in weight space, such as equi-normalization (Stock et al., ICLR 2019, raised by Reviewer K6hw), must iterate. The revision moves it next to that comparison.

What Figure 2 tests. Three separate things, one per panel.

  • (a) Whether an optimizer that whitens its update (Muon) closes the gap between BN and a BN-free network trained on the sphere. The gap falls from 0.559 to 0.026, but the baseline barely trains, so the comparison is confounded. It moves to the appendix.
  • (b) Prop. 3 at the operator level, in 15 cells: the bias correction is exact for dense layers and circular padding, and not for zero padding or across a ReLU.
  • (c) A control for Sec. 4.4: at fixed trained weights, changing only the statistics used at evaluation moves test accuracy in 12 of 12 cells, by 2.6 to 5.6 points.

We agree the figure does not serve the main results. In the revision the backward-ablation table takes its place.

Soundness. Every statement is proved in the paper and rechecked in float64 by the supplement's scripts, and the full-BN arm of our implementation matches PyTorch's BN to 1.8e-15. If a particular step looks wrong, please tell us which one.

We hope this makes clear what the paper proves beyond the definition of BN, and that you will consider updating your assessment.

Reviewer_K6hwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes to view Batch Normalization as gauge fixing: because rescaling/shifting a channel's pre-activations leaves the network function unchanged, BN can be read as picking one canonical representative per redundancy class.

Its core mathematical claim is that, for one fixed batch and one channel, BN's centering and scaling are the exact minimizers of simple strictly-convex objectives from which zero-variance failure, the role of epsilon, single-pass depth resolution, and shift-absorption conditions are derived.

Critical strengths

There is certainly some interest in better understanding the role of normailzation layer in neural networks.

Critical weaknesses

The writing of the paper makes it very difficult to read, with overcomplicated sentences at multiple places. Overall I have a hard time understanding the contribution of the paper. I may miss the key points, but in its current form I don't think that the paper is fit to the audience of ICLR.

Overall I am still uncertain of what to conclude after reading this paper. It lists six partial accounts in the Introduction: loss-smoothing, auto rate-tuning, preconditioning/whitening, signal propagation at init, length-direction decoupling, batch-noise regularization. In Section 5, despite the catchy title "The six explanations, placed", the paper downgrades the claim to: three follow as consequences, one loses the specific form it is usually given, two it can name but not close. It defers to Table 11, but this table is a bit difficult to read and I am not sure how to interpret any claim related to the status column. In any case, since these accounts are emphasized as soon as in the intro, I feel like that the paper should elaborate Section 5 and possibly include what we can really conclude in a clearer manner.

There has been multiple paper exploiting the reparametrization in different ways, see for instance the works focusing on RELU. It may be worth commenting. https://arxiv.org/pdf/1902.10416 https://arxiv.org/pdf/2602.19799

Questions for authors

Please address the weakness, in particular explaining more clearly what we can conclude from this paper.

2 comments

Thank you for the frank report. If a careful reader cannot say what the paper concludes, that is our failure in writing, and we address it first.

What can be concluded.

  1. BN's output solves a strictly convex selection problem over the family z → cz + a·1 of pre-activations that give the same network function. The problem has no solution when the channel is constant on the batch, and ε restores one.
  2. BN is the only such selection rule, up to its own γ and β, whose Jacobian is symmetric. Dividing by the range, the mean absolute deviation, or centering by the median removes the same redundancy and does not have this property.
  3. Dropping the batch-coupled terms of BN's gradient, with the forward pass unchanged, costs 7.8 and 23.2 points on CIFAR-10 with ResNet-8 (6.4 and 8.0 with step sizes matched across arms). A closed form says what each dropped term puts into the weight gradient, and its predictions hold in new tests (reply to 8cbZ).
  4. Folding a fixed centering into the next layer's bias is exact before a dense layer or a circular-padding convolution, and not before a zero-padded one.

What cannot be concluded: how much each of BN's benefits is worth, and whether the six earlier accounts reduce to one. The submitted text suggested more than this, and the revision does not.

Section 5 and the six accounts. You are right that the title promised more than the section delivers. The status of each, plainly:

  • Internal covariate shift: refuted by Santurkar et al. (2018). We add nothing.
  • Automatic rate tuning and length–direction decoupling: both are consequences of scale invariance, and both were known. We restate them.
  • Loss smoothing: partly derived. The 1/‖w‖ factor in the gradient norm follows from scale invariance and norm growth. The remainder is neither derived nor measured.
  • Whitening: per-channel BN is the diagonal case. We show that the orthogonality BN has cannot exist for full whitening (reply to H5CK). We do not measure what whitening is worth.
  • Batch-noise regularization: we tested one Gaussian noise model and it fails. We offer no replacement.
  • Signal propagation at initialization: our analysis says nothing about it.

The section is retitled "What the selection view says about earlier accounts", this list replaces the table, and the introduction no longer suggests a unification.

Related work on ReLU rescaling. Thank you for both references. They concern the same kind of redundancy on the weight side, and we will discuss them.

  • Equi-normalization (Stock et al., ICLR 2019) uses the rescaling symmetry of ReLU units, which exists without any normalization layer, and moves each unit to the minimum-norm point of its rescaling orbit by Sinkhorn-like iterations. In our terms that is the selection problem of Prop. 6 (App. C.8), where weight decay pulls along the same orbit. It must iterate because neighbouring units share weights; App. C.6 has a three-number example of that obstruction. BN's sites share no variables, which is why one forward pass suffices (Cor. 3).
  • Path-conditioned training (Lebeurrier et al., 2026) selects the rescaling by a criterion in path-lifting space instead of the minimum norm. It asks, for the weight-side symmetry, the question Prop. 2 asks for BN: which representative should one train on. The difference in one line: these works choose a representative in weight space by an explicit algorithm, while BN chooses one in activation space, in closed form, at every forward pass.

Writing. The revision has a plain contribution list, a notation table in Sec. 2, shorter sentences, and none of the coined terms. The Hamiltonian vocabulary of Sec. 3 moves to the appendix.

Soundness. The report does not name a specific concern. Every statement is proved in the paper, and the supplement recheck each identity in float64. If a particular step looks wrong, please point us to it and we will answer it directly.

We hope this makes the contribution clear, and that you will consider updating your assessment.

Thank you for the answers, which partly address my concern,.

However I have looked at the current version of the paper. The problem of readability remains and, in my humble opinion, does not meet the criterions of top-tier conference like ICLR. The paper would need to be re-written from scratch in a way that would make it pleasant to read by humans.

Reviewer_8cbZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a variational characterization of Batch Normalization (BN) as gauge fixing, showing that its centering and scaling operations solve convex optimization problems. It further connects BN to the Kempf–Ness framework and establishes that standard deviation is the unique divisor (up to positive scaling) whose backward Jacobian is a scaled orthogonal projection within the considered class. The theoretical results are supported by numerical verification and experiments on digits and CIFAR-10, including controlled backward-pass ablations.

Critical strengths
  1. Interesting theoretical perspective. Understanding why and how normalization improves optimization is an important problem. The gauge-fixing interpretation and convex optimization formulation provide a useful mathematical perspective on BN.

  2. Insightful characterization of backward geometry. Proposition 2 establishes the uniqueness of standard deviation in yielding an orthogonal backward projection, is particularly interesting. The connection to the Kempf–Ness framework further strengthens the theoretical contribution.

Critical weaknesses
  1. Limited connection between theoretical characterization and practical benefits. While the variational formulation is mathematically interesting, it remains unclear to what extent solving this gauge-fixing problem explains why normalization improves optimization, rather than providing an alternative characterization of the existing operator. The empirical evaluation is also limited to relatively small architectures and datasets, leaving its broader practical significance uncertain.

  2. Limited predictive understanding of backward ablations. Removing the mean and variance gradient terms has substantially different effects on digits and CIFAR-10. Although Appendix H.3 derives the corresponding gradient perturbations, the paper does not establish whether these quantities can predict performance differences across settings, or are there any insights that we can incorporate during pretraining on them.

Questions for authors
  1. Predictive value of the framework. Beyond providing a variational characterization of BN, what concrete predictions does the proposed framework make about optimization behavior? In particular, can the authors establish a clearer connection between the orthogonal-projection property in Proposition 2 and practical training performance?

  2. Understanding backward ablations. Can the author measurable quantities, such as input statistics or gradient perturbation norms, explain or predict why removing the mean and variance gradient terms has substantially different effects on digits and CIFAR-10?

  3. Generalization across tasks and architectures. Can the author provide more comparisons across different tasks and model architectures? And analyze in which cases it failed, and how it can be generalized?

  4. Positioning the theoretical contributions. Can the author bridge the theoretical contributions so that the reader can better understand why the results come and how do they relate to the prior results?

Additional Suggestions

  1. Presentation clarity. Several important definitions and technical explanations are deferred to the appendix, requiring frequent cross-referencing. Moving essential notation and intuitions into the main text would improve readability.

  2. Training-stage dependence. It would be interesting to discuss whether the proposed framework offers any insight into when normalization is most beneficial during training. For example, could normalization be applied only during early training or selectively enabled and disabled across training stages, rather than being used throughout training?

3 comments

Thank you for the encouraging remarks on Prop. 2, and for questions that pushed us to test predictions.

Q2. Can a measurable quantity predict the effect of the backward ablations?

Within a task, yes. The quantity is the leak of Eq. 6 relative to the true gradient. For the mean term it is proportional to the mean of the site's input. For the std term it does not depend on that mean. This gives three predictions.

1. The mean-ablated arm should degrade as the input mean grows, and recover when the input is centered. We add a constant c to the input of every block. Full BN is exactly invariant to c.

digits MLP, 5 seedsc = 0c = 1c = 3c = 10
gradient perturbation ‖g_abl − g‖/‖g‖ at init (cosine)1.4 (0.57)4.8 (0.21)12 (0.09)37 (0.04)
full BN (%)98.4 ± 0.798.5 ± 0.798.5 ± 0.698.6 ± 0.3
mean term dropped (%)98.8 ± 0.390.7 ± 2.274.5 ± 3.731.5 ± 7.6
std term dropped (%)98.6 ± 0.498.5 ± 0.598.5 ± 0.398.7 ± 0.2

Learning rates are selected per cell on validation. For the mean-ablated arm we extended the grid down to 0.001, so that every rate it selects is interior. On CIFAR-10 with a 4×512 BN-MLP (3 seeds), the mean-ablated arm scores 45.2 with centered input, 39.7 at native input, 19.6 at c = 1 and 14.5 at c = 3. Full BN stays between 45.0 and 45.2.

2. The std-ablated arm should not react to c. It does not (last row).

3. The leak is proportional to ∂L/∂β, so making γ and β learn ten times faster should shrink it. On the CIFAR-10 MLP this takes the mean-ablated arm from 39.7 to 44.2, against 44.4 for full BN under the same change.

The limit, stated plainly. Across tasks the leak size alone does not predict the loss. At native inputs the perturbation at initialization is about the same on digits (1.4, cosine 0.57) and on ResNet-8 with CIFAR-10 (1.3 to 1.6, cosine 0.5 to 0.7), yet digits loses nothing and CIFAR-10 loses 8 to 30 points, depending on the optimizer control. How much misalignment a task tolerates is a second factor, and we do not predict it. Digits at 98% tolerates a cosine of 0.57 and stops tolerating it near 0.2.

A practical use. If a method detaches or approximates BN's statistics in the backward pass, as small-batch and online variants do, centering the site's input first removes the mean leak identically. The mean-ablated gradient then equals the full gradient to 8e-16.

Q1. What does the framework predict, and how does Prop. 2 connect to training?

Two predictions about the backward pass, both now measured on ResNet-8 (3 seeds, details in our reply to yMCH).

  • Removing one of the two redundant directions from the gradient matters a great deal. With weight norms and angular steps matched across arms, dropping the std term costs 6.4 points [−6.6, −6.3] and dropping the mean term 8.0 [−14.5, −1.5].
  • The angle at which they are removed matters less. This is where Prop. 2 enters. We kept exact BN in the forward pass and replaced only the backward projection by the oblique one of the range divisor (s_max between 12 and 54 in this network). The loss is 1.6 points [−2.3, −1.0]. With s_max up to 3 (the mean-absolute-deviation divisor; both oblique projections in an MLP) we measure no loss.

So the link from Prop. 2 to training is real and modest, and smaller than our abstract suggested: the 7.7 points there included the effect of the range statistic in the forward pass, and we correct it.

On your first weakness: the selection view explains one mechanism by which BN helps optimization, the removal of the two redundant directions from the gradient, and it measures that mechanism. It does not account for all of BN's benefit, and the revision does not claim that it does.

(continued next)

Q3. More tasks and architectures; where does it fail?

We could not run larger models in the discussion window, and we say so. What we have is the backward ablation in four settings:

Settingfull BNmean term droppedstd term dropped
digits, MLP (5 seeds)98.398.998.6
CIFAR-10, 4×512 BN-MLP (3 seeds)45.039.745.1
CIFAR-10, ResNet-8, 60 epochs, batch-level augmentation (3 seeds)86.463.278.6
CIFAR-10, ResNet-8, 30 epochs, per-image augmentation (3 seeds)84.754.575.2
same, weight norms and angular steps matched across arms85.177.178.6

Where it fails to show an effect: the std term matters in the convolutional network and not in either MLP, and we do not have a tested explanation. The mean term matters wherever the site input has a mean and the task is not saturated. Beyond BN, Xu et al. (NeurIPS 2019) ran the same kind of ablation for layer normalization in transformers and also found that the derivatives of the statistics matter more than the forward normalization. We have no ImageNet-scale or transformer experiments of our own.

Q4. How do the results relate to prior ones? The revision opens with this table.

ResultStatus
BN's image is a centered sphere; γ, β change its radius and centerKnown: Sun et al. (2020); as a projection, Li et al. (2020)
⟨w, ∇L⟩ = 0, norm recursion, effective rate, equilibrium angleKnown: van Laarhoven (2017), Arora et al. (2019), Sun et al. (2020), Wan et al. (2021)
Radial component under momentumKnown in a more explicit form: Heo et al. (2021)
Detaching the statistics' derivatives as an ablationKnown for layer normalization: Xu et al. (2019)
Convex selection problem; no minimizer at zero variance; ε restores oneOurs, elementary
Uniqueness of BN among all selection rules (Prop. 2′)Ours
No orthogonal selection for whitening (reply to H5CK)Ours
Closed form for what each ablation leaks, with tested predictionsOurs
Shift absorption fails under zero paddingOurs (the boundary; the folding itself is standard)
Forward-identical backward ablation of BN, with matched-norm and forward-fixed controlsOurs

Presentation. Agreed. The notation table and the definitions of the four technical terms move from App. B into Sec. 2, and the intuition paragraphs stay next to each statement.

Training-stage dependence. An interesting question, and we can give one measurement. If the batch-coupled terms mattered only early, the gradient change caused by dropping them should shrink during training. In our ResNet-8 it does not shrink much: relative to the true gradient, dropping the mean term changes the gradient by 1.3 to 1.6 at initialization and 1.4 to 1.5 after 30 epochs, and dropping the std term by 0.6 to 0.9 and 0.3 to 0.4 (3 seeds). So we see no stage at which BN's backward terms become negligible. Whether the forward normalization can be switched off late is a separate question that we have not tested, and we will say so.

We hope the dose-response results show that the framework makes testable predictions, and that you will consider updating your assessment.

Reviewer_8cbZOpenReview ↗

Official Comment by Reviewer 8cbZ

We thank the authors for their detailed responses. The comparison table helps clarify the distinctions between this work and prior studies a lot. However, I would still appreciate some preliminary results on larger-scale models, as observations from MLPs and ResNet-8 on CIFAR-10 may not necessarily generalize to more complex architectures and tasks.

In particular, the finding that the gradient perturbation caused by dropping the batch-coupled terms does not substantially decrease during training on ResNet-8 raises further questions about whether this behavior holds across different models and under what conditions the perturbation might diminish. This further motivates a broader empirical analysis across architectures and tasks.

Regarding presentation, I would appreciate a concrete update to the revised PDF. Although the authors have described their planned revisions in the response, I have not yet seen these changes reflected in the manuscript. Incorporating more of the analysis into the main paper and clearly highlighting the revisions would help readers better understand and appreciate the contributions.

Given these remaining concerns, I will maintain my current score.

Reviewer_siTHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper attempts to provide an explanation for the success of batch normalization (BN) in deep learning through gauge fixing, by rewriting the BN operation in terms of a convex objective (termed "kempfness").

Critical strengths

The authors attempt to provide an explanation for an important operation in neural net training.

Critical weaknesses

In my opinion, this paper is not up to par for acceptance and I recommend strong rejection, based on the following:

  • From a technical point of view, it is not clear to me what the paper actually proves. As far as I could understand, the first 5 pages (up to section 3) are trying to show tht BN can be decomposed in mean centering and rescaling operations (as shown in Figure 1). In Theorem 1, the authors introduce a function FF whose minimizer is the rescaled output of BN, and they label this a Kempfness function. It is not clear to me what is the value of doing this, and why does it span such a large part of the paper, as it is obvious from the definition of BN that this is what it does. Similarly, the Jacobian of BN in proposition 1 is also a trivial calculation.

  • In general, the paper is extremely hard to read, and several sections are incoherent and burried in a lot of notation, definitions, and text that does not carry any content: "payload ledger", "angular thermostat" to name a few. Several of the theorems are also quite unreadable, and seem disconnected from the rest of the text (Corollary 3 for example)

  • I don't understand what the experiments in Figure 2 are testing?

Questions for authors

See weaknesses.

2 comments

Thank you for two precise questions. We ran both experiments. Protocol for all runs below: ResNet-8, CIFAR-10, B = 128, 30 epochs, SGD with momentum 0.9, cosine schedule, rate 0.1, seeds 2701 to 2703, and ordinary per-image crop and flip.

Q1. Does the range deficit persist when the forward activations and their scale are held fixed and only the backward projection changes?

Partly. A deficit of 1.6 points persists. The rest of the 6 to 8 points in Table 10 does not, so it comes from the range statistic in the forward pass. Your reading of that table was right.

Both arms run exact BN in the forward pass, so activations, their scale and the evaluation-time operator are identical at equal weights. Only the backward projection differs. We checked in float64 that each projection removes the same two directions (|⟨1, g⟩| and |⟨z − mean, g⟩| below 2e-14), and that the BN arm matches PyTorch's BN to 3e-15.

test accuracy (%)paired difference [95% CI]
BN84.72 ± 0.18—
BN forward, range projection backward83.11 ± 0.09−1.61 [−2.26, −0.96]

In this network the range projection has s_max between 12 and 54 (Table 10 printed 1.9 to 4.0, computed for B elements where a convolutional site has B·H·W; we correct it). In a BN-MLP, where it is 3.1, the same swap costs nothing measurable (44.69 ± 0.79 against 44.97 ± 0.42). We will state the consequence of Prop. 2 as this 1.6 points, and the abstract's "up to 7.7 points" is withdrawn.

Q2. Do the backward-ablation losses persist under matched weight norms and angular step sizes, and ordinary per-image augmentation?

Yes, both persist. For the mean term, most of the loss under plain SGD was the confound you named.

Matched optimizer. Every filter that feeds a BN site keeps its initial norm. At each step it rotates by the same angle θ_t in every arm, in the direction of that arm's momentum update projected on the tangent space. θ_t follows a cosine schedule from 0.01 rad, the angular step that plain BN takes in its first epoch under SGD (measured: 0.0095). So norms are identical across arms, angular steps are identical for every filter that receives a gradient, and the arms differ only in direction.

plain SGDmatched norms and angles
full BN84.72 ± 0.1885.09 ± 0.34
mean term dropped54.45 ± 11.20, −30.3 [−57.8, −2.7]77.09 ± 2.30, −8.0 [−14.5, −1.5]
std term dropped75.21 ± 0.60, −9.5 [−10.7, −8.3]78.64 ± 0.30, −6.4 [−6.6, −6.3]
  • Per-image augmentation does not remove the effect. The paper's figures, with batch-level augmentation and 60 epochs, were −23.2 and −7.8.
  • The arms did differ in norm and step size under plain SGD. The final mean filter norm is 0.96, 2.29 and 1.49 for the three rows, and the mean angular step 0.0044, 0.0031 and 0.0027.
  • With both matched, the std-term loss is 6.4 points with a tight interval. The mean-term loss falls from 30 to 8, so about three quarters of it was norm growth and step size. A directional part remains, with a wide interval.
  • One caveat. Norms agree to four digits in all arms, and the angular step follows the schedule exactly in the full and std-ablated arms. In the mean-ablated arm it is 0.85 of the schedule from epoch 5 on, because about 15% of its filters have zero momentum and stop moving.

The revision reports both columns and quotes the matched figures in the abstract.

"The empirical decomposition remains confounded." We agree. It is now a list of hypotheses, and the Muon and noise-model experiments move to the appendix.

"Several mathematical statements require qualification." We agree, and will make these:

  • ε. Projection and idempotence hold at ε = 0. At ε > 0 the Jacobian stays symmetric, with eigenvalue ε/(σ²+ε) along ŷ, and ⟨w, ∇L⟩ = γ·∂L/∂γ·ε/(σ²+ε) (full list in our reply to H5CK).
  • Prop. 2 needs B ≥ 3 and a divisor differentiable at the point.
  • App. A. BN's output is the orbit point on the level set ‖y‖² = B. It is not a minimum-norm point of the scaling orbit, whose infimum is 0. We remove the sentence that says otherwise.
  • Thm. 2 proves that the brackets close, not that the quotient is a smooth reduced space. The wording changes accordingly.
  • Prop. 3 concerns a fixed shift, not the batch mean of training-mode BN.

Writing style. The paper's AI-use statement discloses an AI-assisted first draft. The criticism is fair. We are rewriting the main text by hand: a plain contribution list, no coined vocabulary, and Table 6 in the main text.

We hope the two experiments answer your questions and that you will consider updating your assessment.

Reviewer_siTHOpenReview ↗

Reply

Thank you for the reply. Unfortunately I don't think these answers address the issues I raised with this work. I maintain my score and recommend rejection.

Reviewer_H5CKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reinterprets batch normalization as "gauge fixing": for one channel and one batch, BN's centering and scaling are shown to be the closed-form minimizers of two strictly convex 1-D problems (the latter a Kempf–Ness function) over the weight-rescaling redundancy. From this objective, the authors derive results on zero-variance channels, the role of ϵ\epsilon, shift absorption into the next layer, and the uniqueness of the standard deviation among degree-one divisors (its backward pass is an orthogonal projection). They also split BN's benefit into a four-part "payload ledger"; the main empirical validation is a ResNet-8/CIFAR-10 ablation that keeps the BN forward pass and removes backward-pass terms (7.8 and 23.2 accuracy-point drops), plus smaller supporting experiments.

Critical strengths

The paper writes BN's two steps as the closed-form minimizers of strictly convex 1-D problems (Thm 1, Prop 1), so zero-variance failure, the role of ϵ\epsilon, and one-pass depth (Cor 1–3) all follow from one objective. The identities are elementary and checked symbolically and in float64, which makes the main theoretical claims credible.

Sec 4.3 keeps the exact BN forward pass in every arm and removes only the batch-coupled gradient terms, which isolates the backward-pass contribution (7.8 and 23.2 accuracy points on CIFAR-10) from forward-pass effects. This is a cleaner design than most prior ablations of BN, and the authors are candid about its scope and limitations.

Critical weaknesses

The objective F(s)=e2sv−2BsF(s)=e^{2s}v-2Bs and its level BB are chosen so that the minimizer is BN's 1/σ1/\sigma; the paper itself notes that any level τ\tau gives a convex FτF_\tau, so the objective is not derived from a principle that independently selects BN. The geometry (centered sphere, radius, affine parameters as change of radius/center) is already in Sun et al. (2020), and shift absorption, norm recursion and orthogonality are acknowledged as prior or standard results. The remaining new content (Kempf–Ness/Hilbert–Mumford language, Prop 2) is not shown to matter for practice: the "uniqueness" of the std. dev. is not predictive of accuracy, since the MAD divisor (smax⁡≈1.25s_{\max}\approx1.25, i.e., not orthogonal) stays within one point of BN, and the predicted batch-size widening for the range divisor rests on two seeds and is called "suggestive". The six existing explanations are not unified either: only one is derived, and two are left open.

The central result (7.8 and 23.2 point drops from removing backward terms) comes from one architecture (ResNet-8), one dataset (CIFAR-10) and 2–3 seeds, and the authors state it is "the size of that leak at the input means this network has, not a constant of the operator". The four-arm interaction has a 95% interval of [−12.8,+39.0][-12.8,+39.0], several success criteria were set after seeing results (the 30–50% band; the learning-rate table after a diagnostic round), and the Muon result is described as only an upper bound with unresolved trainability confounds. Nothing is tested on LN, RMSNorm or transformers, yet the title and abstract speak of "normalization" and the "payload ledger" is presented as what any normalization-free design must supply. As a result, the quantitative "ledger" is not a reliable basis for the broad conclusions drawn.

Questions for authors

Cor 2 states that ϵ\epsilon "is exactly the coercification" of the variational problem, but Prop 5 and Sec 3 say ϵ>0\epsilon>0 does not repair the degenerate pairing and removes the exact scale invariance on which the identities (e.g., Prop 4) rely. Which of the paper's results still hold for BN as actually deployed (ϵ>0\epsilon>0), and which are only approximate?

The level BB is left open (any τ>0\tau>0 gives a convex FτF_\tau). What independent principle singles out this objective, as opposed to any convex function with the same minimizer 1/σ1/\sigma? Please name one result that cannot be obtained without the gauge-fixing formulation (e.g., from the known centered-sphere picture of Sun et al., 2020).

The MAD divisor is non-orthogonal (smax⁡≈1.25s_{\max}\approx1.25) yet stays within one point of BN. What does orthogonality of the backward pass actually predict about accuracy, and can the batch-size trend for the range divisor be confirmed with more than two seeds?

2 comments

Thank you for a detailed report and for three questions that each have a concrete answer.

Q1. Which results hold for BN as deployed (ε > 0)? We checked every statement in float64 at ε = 1e-5, 1e-3 and 1e-1.

Exact for every ε ≥ 0.

  • Thm. 1, Cor. 2, Prop. 1: BN_ε is the unique minimizer of the convex problem. This is where ε enters.
  • BN_ε(z) = ∇_z[B·sqrt(σ²+ε)], so its Jacobian is symmetric for every ε.
  • The backward decomposition (Eq. 4) and both leak formulas (Eq. 6): relative error below 4e-15 at all three values.
  • Invariance to shifts of z, Prop. 3, and the one-pass structure of Cor. 3.

Exact at ε = 0, and off by a known factor at ε > 0. These are the statements that use scale invariance.

  • ⟨w, ∇_w L⟩ = γ·∂L/∂γ·ε/(σ²+ε) per channel, instead of 0. This identity is new (float64 relative error 1e-10). The recursion of Prop. 4(ii) picks up the matching cross term.
  • The Jacobian's eigenvalue along ŷ is ε/(σ²+ε), in units of 1/s, instead of 0. It stays symmetric but is no longer exactly a projection.
  • BN_ε(BN_ε(z)) differs from BN_ε(z) by O(ε): 7e-6 at ε = 1e-5.

Size in practice. In our trained ResNet-8 at the default ε = 1e-5, the per-site median of ε/(σ²+ε) lies between 1e-5 and 1e-3 at the end of training, and its maximum over all channels is 2.5e-3 (3 seeds).

Not repaired by ε. The pairing of Prop. 5.

Cor. 2 and Prop. 5 do not conflict, but our wording invited that reading. Cor. 2 says the forward problem has a minimizer for every batch once ε > 0. Prop. 5 says the variance condition stops being transverse to the scale direction at v = 0, and adding a constant to the condition does not change that. So ε buys existence and pays with exact scale invariance, by the factor above. The revision states this list in Sec. 2 and replaces "is exactly the coercification" by "is what restores a minimizer".

Q2. What principle singles out this objective, given that the level is free?

You are right on both points: any τ > 0 gives a convex F_τ, and many functions share a minimizer. We do not claim F is canonical. The principle is a property of the selected set, and it needs no objective.

Prop. 2′ (all selection rules). Let B ≥ 3, and let N be a smooth map that is invariant under z → cz + a·1 (c > 0) and returns a point of the same family. Then DN(z) is a positive multiple of the projection onto the tangent space of the image of N along span{1, N(z)}. It is symmetric for all z if and only if N = γ·BN + β·1 for constants γ > 0 and β.

Proof sketch. A projection is symmetric exactly when its kernel is orthogonal to its range. So the image S is orthogonal to 1 and to y at each of its points y. Then mean(y) and ‖y‖² are constant on S, and S is a centered sphere. ∎

So the principle is that the selected set meets the redundant directions at right angles. It selects BN and leaves free exactly a radius and a center, which are the two parameters BN makes learnable. The level is undetermined because it is γ. Prop. 2 in the paper covered divisors only; Prop. 2′ covers every rule, median centering included (float64: asymmetry 0.82, against 1e-9 for BN).

"Name one result that cannot be obtained without the formulation."

(a) Prop. 2′. The centered-sphere picture of Sun et al. (2020) describes the image of BN, and Li et al. (NeurIPS 2020), whom we will cite, write BN as the projection onto that sphere. Neither can state uniqueness, because uniqueness compares BN with other ways of selecting from the same families, and those families are what the formulation adds.

(b) A consequence we had not included. Ask the same question for decorrelating normalization, where the redundancy mixes channels: Z → AZ with A in GL(C). The answer reverses. For C ≥ 2 no selected set is orthogonal to the orbits, even locally. The directions orthogonal to the orbit at Y are T = A(I − P_Y), with P_Y the projector onto the row space of Y. The bracket of two such fields has orbit component A₁(I − P_Y)A₂ᵀ − A₂(I − P_Y)A₁ᵀ, which is nonzero, so by Frobenius these directions have no integral surface. For per-channel scalings the same component is the diagonal of an antisymmetric matrix, hence zero. In float64 it is 4.5 for GL(2) and 2e-10 for the diagonal group. So per-channel BN is the case in which an orthogonal backward pass exists, and no whitening rule has one. This cannot be stated in the sphere picture.

(c) The leak formulas, with three predictions that we tested (Part 2 and our reply to 8cbZ).

(continued in Part 2)

Q3. What does orthogonality predict about accuracy? Can the range trend be confirmed with more seeds?

First a correction that bears on your point. Tables 9 and 10 listed s_max for B elements per channel. In a convolutional network a BN site averages over B·H·W elements. Measured in the trained ResNet-8 at B = 128, s_max is 1.3 to 1.4 for the mean-absolute-deviation divisor and 12 to 54 for the range divisor. So the first is close to orthogonal and the second is far from it.

Reviewer yMCH asked for the test that separates the backward pass from the forward statistic: exact BN in the forward pass, and only the projection in the backward pass replaced by the range divisor's. ResNet-8, CIFAR-10, B = 128, 30 epochs, 3 seeds:

test accuracy (%)paired difference [95% CI]
BN84.72 ± 0.18—
BN forward, range projection backward83.11 ± 0.09−1.61 [−2.26, −0.96]
range divisor in both passes (Table 10)−6.3 and −7.7

In a 4×512 BN-MLP on CIFAR-10, where s_max is 3.1 for the range projection, the same swap gives 44.69 ± 0.79 against 44.97 ± 0.42 for BN, and 44.97 ± 0.04 with the mean-absolute-deviation projection.

So the answer is graded, and smaller than our text suggested. An s_max of 1.3 to 3 costs nothing we can measure, which is why the mean-absolute-deviation divisor stays within a point of BN. An s_max of 12 to 54 costs 1.6 points. The rest of the range divisor's deficit in Table 10 comes from its forward statistic. We will present Prop. 2 as a characterization with this measured consequence, and the abstract will no longer say the alternatives lose accuracy "in proportion to their distortion, up to 7.7 points".

On the batch-size trend: we could not add seeds for the two-batch-size comparison in the discussion window. Two seeds do not support it, so we remove that sentence.

"One architecture, one dataset, 2–3 seeds", and "not a constant of the operator". We agree the numbers 7.8 and 23.2 are specific to ResNet-8. The sentence you quote is the claim we do make: the size of the effect is set by the input mean through Eq. 6, and that can be tested. On a CIFAR-10 BN-MLP the mean-ablated arm scores 45.2% when the site input is centered, 39.7% at its native mean, and 19.6% and 14.5% when the mean is raised by 1 and by 3, while full BN stays at 45.0 to 45.2%. The same happens on digits (table in our reply to 8cbZ).

The four-arm interaction. Agreed. With an interval of [−12.8, +39.0] it is not a finding, and it leaves the main text.

Criteria set after seeing results. Neither headline number depends on them. The 30–50% band belonged to the section-tracker experiment that the paper withdraws. The learning-rate table governs only the follow-up diagnostics of Sec. 4.4. Both are reported in the paper as weaker than a registration, and the revision drops the word "registered" for them.

Muon and the ledger. Agreed. The Muon result moves to the appendix as a stress test with its confounds, and we draw no conclusion from it. The four-part ledger is now a list of hypotheses, of which only the backward-pass part is measured.

Prior results. We agree that the sphere geometry, the orthogonality ⟨w, ∇L⟩ = 0, the norm recursion and bias folding are prior or standard. The revision opens with a table that separates them from what is new (it is in our reply to 8cbZ).

LN, RMSNorm, transformers. We ran no experiments on them, and the abstract will say BN. Two things do carry over. Thm. 1 and Prop. 2′ are statements about one vector, so they hold as written for layer, group and instance normalization, which apply the same map along other axes. For RMSNorm the same proof without centering shows z/rms(z) is the only scale-only rule with a symmetric Jacobian. And Xu et al. (NeurIPS 2019) ran the corresponding backward ablation for layer normalization in transformers and found that the derivatives of the mean and variance matter more than the forward normalization, which agrees with Sec. 4.3.

The six explanations. Agreed: two are restatements of scale invariance, one is partly derived, and the rest are open. Sec. 5 is retitled and says so (status list in our reply to K6hw).

We hope the ε list, Prop. 2′ and the forward-fixed test answer your three questions, and that you will consider updating your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.