acceptodds
Under review as a conference paper at ICLR 2027

Structural re-parameterization is not magic

Abstract

Some vision models are trained with several parallel kernels per layer, each with its own BatchNorm, summed during training and folded into a single kernel at inference, a technique known as structural re-parameterization. It has been reported to give a modest accuracy benefit, explained so far by implicit ensembles, diversity and "training-time overparameterization". We study a simple instance, two identical branches, and take it apart. We attribute the accuracy benefit to two simple mechanisms, previously unidentified. The mechanisms are optimization effects: a coordinate specific learning rate boost at the last layer and a directional preconditioning effect at the first layer. The boost acts on the scale coordinate and helps to grow the logit scale at the end of training. The preconditioning cools the learning rate of the highest variance mode of the input, making the system better conditioned. We show this by dissecting the model one layer at a time, deriving the modified step in closed form, and seeing that applying the corresponding optimization modifications to a single branch reproduces the observed benefit. We conduct our study in a small VGG like network on CIFAR and then show that the two mechanisms carry over to production architectures on ImageNet.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
structural re-parameterization, batch normalization, learning rate, weight decay, preconditioning, training dynamics, over-parameterization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KGrhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies why structural re-parameterization improves accuracy. This work focuses on the simplest instance: two identical 3×3 branches with per-branch BN normalization and a shared scale. This paper argues that the benefit is an optimization effect rather than added capacity, and that it is concentrated at the first and last layers. They identify two mechanisms. At the last layer, the branches collapse, which amounts to a learning-rate boost (×4 on the BN scale, ×2 on the shift). At the first layer, the branches stay apart and their difference aligns with the top eigenvector of the input covariance. This reduces the effective learning rate along that dominant direction. A single-branch network with these two learning-rate modifications reproduces the gain of the paired network.

Critical strengths
  1. The paper gives a concrete, mechanistic account of what re-parameterization branches do during training, going beyond the explanations usually offered in the literature (e.g., implicit ensembles). Viewing the extra branch as an implicit preconditioner provides useful intuition.
  2. The explanation is tested constructively: applying the identified learning-rate modifications to a single branch recovers the benefit of the paired network.
  3. This reviewer could not verify every detail due to limited background on the topic (and much of the derivation is in the appendix) but the analysis appears sound overall.
  4. The findings may help in designing architectures and training procedures. Replacing training-time branches with targeted learning-rate adjustments might help reducing training time.
Critical weaknesses
  1. The paper relies heavily on informal, idiosyncratic terminology ("hot/cool," "open/closed," "parked," "arm," "cell"), much of it used before or without definition. Replacing these with standard terms (e.g., "effective learning rate" and "configuration") and defining the remaining ones before first use would substantially improve readability.
  2. The introduction does not clearly state the research problem or the prior work that motivates it. It moves from defining the block directly to results.
  3. The analysis covers a single block shape, two identical 3×3 branches with a shared scale, chosen because it is simpler to analyze. This raises the question of how representative the studied block is.
Questions for authors
  1. The open/closed regimes appear to be empirical observations. Is there a principled criterion that predicts which layers open or close? Would it extend to other block shapes (e.g., 1×1/identity branches) or other normalization layers?
  2. Given the limitations the authors acknowledge, what do the authors consider the core contribution that generalizes beyond this specific configuration?
Reviewer_pbnZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper presents a study of the properties of structural re-parameterization (SRP) in CNNs, i.e. training multiple arms in one layer with independent batch-norms and merging them after training. SRP is known to provide modest accuracy raises. The analysis is restricted to a very specific setting using two identical CNN arms.

The paper concludes to have found novel insights in the underlying mechanics of SRP which help to understand it's causes beyond the existing explanations of implicit ensembles, diversity and “training-time overparameterization”

Critical strengths

None

Critical weaknesses

Unfortunately, the paper suffers from multiple problems:

a) readability: the paper is hard to follow, not in technical sense but due to the fact that the reader is provided with very little introduction, hard jumps between sections and an bombardment of facts that make it hard to see the "big picture" of the "why" and "what". The purpose of the study is not well motivated (what do we hope to gain by this?) and the results are not well discussed (What have we learned and how does that help us?)

b) potential impact: the potential impact of gaining further understanding of an outdated technique (who actually uses VGG style CNNs with Bach-Norm anymore?) that provided marginally better results is very limited

c) generalization: the authors write in their limitations section: "We study two identical 3×3 branches with per branch BatchNorm and not RepVGG’s 3×3 with a 1×1 and an identity, or MobileOne’s four branches. The account is one instance taken apart, not a general theory." - this is exactly the point: there is no theoretical or empirical indication that the findings will generalize beyond the very special setting investigated here.

Questions for authors

A) did you actually investigate the statistics of the two independent Batch-Norms? How does their difference effect the outcome? Does this depend on initialization? I would have expected to investigate the correlation of the alignment between the norm parameters and the final effects.

Reviewer_dk3dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Structural re-parameterization is architectural modifications of a base CNN network that leave the network functionally equivalent yet improves training. The authors examine a particular form of branched reparameterization with two identical convolutional branches sharing BatchNorm and identifies two mechanisms for the observed training improvements.

Critical strengths
  • Interesting empirical observations that reparameterizing first and last layers yield majority of the training gain which could be useful to practitioners if the trend continues for other reparameterizaiton on other models.
  • Novel analysis of BatchNorm reparameterization giving some insight into how one class of reparameterizations may effect optimization.
Critical weaknesses
  • Results are focus on training accuracy improvements and not on held-out accuracy which is what deep learning practitioners who use reparameterizaiton care about. Both training and test gains are small for ImageNet.
  • Relevant prior work is missing. “Reparameterization through Spatial Gradient Scaling” connect branched reparameterization and modified gradient updates [Detkov et al., 2023]. There should be more emphasis on explaining the effect of BatchNorm setting. Furthermore, the above paper reports measurements over multiple seeds contradicting the statement: “…to our knowledge, the first multi-seed measurement…”
  • Empirical experiments are not extensive with no hyper parameter search and short 40 epoch training on imagenet. Its not clear whether gains are due to originally misconfigured hyper parameters.
Questions for authors

How general is the observation that reparameterizing only the first and last layers recovers most of the gain?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.