- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies why structural re-parameterization improves accuracy. This work focuses on the simplest instance: two identical 3×3 branches with per-branch BN normalization and a shared scale. This paper argues that the benefit is an optimization effect rather than added capacity, and that it is concentrated at the first and last layers. They identify two mechanisms. At the last layer, the branches collapse, which amounts to a learning-rate boost (×4 on the BN scale, ×2 on the shift). At the first layer, the branches stay apart and their difference aligns with the top eigenvector of the input covariance. This reduces the effective learning rate along that dominant direction. A single-branch network with these two learning-rate modifications reproduces the gain of the paired network.
Critical strengths
- The paper gives a concrete, mechanistic account of what re-parameterization branches do during training, going beyond the explanations usually offered in the literature (e.g., implicit ensembles). Viewing the extra branch as an implicit preconditioner provides useful intuition.
- The explanation is tested constructively: applying the identified learning-rate modifications to a single branch recovers the benefit of the paired network.
- This reviewer could not verify every detail due to limited background on the topic (and much of the derivation is in the appendix) but the analysis appears sound overall.
- The findings may help in designing architectures and training procedures. Replacing training-time branches with targeted learning-rate adjustments might help reducing training time.
Critical weaknesses
- The paper relies heavily on informal, idiosyncratic terminology ("hot/cool," "open/closed," "parked," "arm," "cell"), much of it used before or without definition. Replacing these with standard terms (e.g., "effective learning rate" and "configuration") and defining the remaining ones before first use would substantially improve readability.
- The introduction does not clearly state the research problem or the prior work that motivates it. It moves from defining the block directly to results.
- The analysis covers a single block shape, two identical 3×3 branches with a shared scale, chosen because it is simpler to analyze. This raises the question of how representative the studied block is.
Questions for authors
- The open/closed regimes appear to be empirical observations. Is there a principled criterion that predicts which layers open or close? Would it extend to other block shapes (e.g., 1×1/identity branches) or other normalization layers?
- Given the limitations the authors acknowledge, what do the authors consider the core contribution that generalizes beyond this specific configuration?