- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a family of normalization methods that track statistics across batches, with BN’s training behavior recovered at a full-update endpoint. It analyzes how the assumed observation noise controls the tracker’s distance from BN and explains why underestimating this noise makes the tracker behave almost like BN. Experiments on ResNet-20 with CIFAR datasets test these predictions through noise measurements, changes to the noise estimate, and recalibration of inference statistics.
Critical strengths
-
The paper gives a useful explanation of why a seemingly more principled tracker can end up behaving like BN. The bounds and gain analysis turn this explanation into predictions that can be checked. I find this valuable even without a clear accuracy improvement over BN.
-
The experiments go beyond reporting accuracy. They measure the noise and change the noise model while keeping the update rule fixed. This provides meaningful support for the proposed explanation.
Critical weaknesses
-
The evidence is relatively limited to ResNet-20 and CIFAR datasets. Several key mechanism tests use only one or two retrained seeds, and the calibrated small-batch results use one seed. This makes it hard to judge how broadly the findings hold.
-
The experiments compare mainly with BN, GN, and LN, rather than other methods that track statistics across batches. Direct comparisons with methods such as Momentum BN or Online Normalization would help establish what the proposed analysis adds to existing work.
Questions for authors
-
How much of the noise-misspecification explanation applies to existing tracking normalizers, rather than only to the proposed gain rule?
-
Are results available for the key noise-calibration and recalibration tests across more seeds or on a larger network? Such evidence could materially strengthen my assessment.
2 comments
Thank for the detailed explainations. My concerns have mostly be addressed.
Thank you for stating exactly what would strengthen your assessment. We aimed our new runs at it.
Q2 and W1: more seeds. Every key test now has 5 seeds. The 60 training runs are new, on a new machine, and reproduce Table 1 within seed variation.
Larger network. The ResNet-18 runs did not finish in the discussion window on our shared GPU. We commit to them for the camera-ready version.
What the present data say about scale: the noise excess grows with the feature map. At B = 2 on corrupted streams, true noise / r_n is:
At every resolution this corresponds to 2 to 24 effectively independent values per image. We therefore expect a network with larger feature maps to be further from the per-position model.
Q1: how much applies to existing tracking normalizers?
W2: Momentum BN and Online Normalization. We could not retrain both in the discussion window. We ran the comparison that isolates the ingredient they share with us, a running average of the statistics, on the same five checkpoints and streams, with momentum chosen on held-out corruptions. On stationary streams it matches the best fixed gain, and with the correct noise our gain reaches the same accuracy without a validation set. Both methods will be retrained for the camera-ready version, and Table 13 will compare them mechanism by mechanism.
We hope this five-seed evidence strengthens your assessment, and thank you again.