- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks when the EMA teacher in self-distillation is actually needed. The authors record the outputs of the student and teacher on a fixed set of images at every step and analyze them in the frequency domain. In BYOL, they find that the student's trajectory is dominated by a fast oscillation caused by the optimizer's momentum, and that the teacher mainly helps by filtering out this oscillation. Consistent with this, removing optimizer momentum (at a matched effective step size) makes the teacher nearly unnecessary: on CIFAR-100, dropping the teacher costs 11.5 linear-probe points with standard momentum but only 0.7 without it, with similar trends on other datasets and on ImageNet-1k.
Critical strengths
The paper asks a clear and useful question and turns it into a testable hypothesis. The experiments go beyond correlation: a short-memory two-tap teacher that cancels only the oscillation recovers most of the EMA teacher's benefit, the cost of removing the teacher grows steadily with the optimizer's memory, and the authors rule out a pure denoising explanation with a clean argument.
Critical weaknesses
- The mechanism applies only to BYOL, and the paper does not explain why.
- The statistical evidence is thin, especially beyond CIFAR-100.
Questions for authors
- Does SimSiam's student show the same oscillation? If so, why does it not hurt teacher-free training there?
- At β = 0.5 there is still an oscillation, yet removing the teacher costs almost nothing. What determines whether the oscillation is harmful?
- Can you report the absolute ImageNet-1k accuracies, ideally with more seeds or a longer budget?
- How does teacher-free BYOL without momentum compare with SimSiam under the same setup?
4 comments
Reply to Reviewer 5vzK, Part2
W2: are the differences statistically meaningful? Yes, at the level of paired gaps, and the revision adds the runs that show it. We read them as paired gaps rather than as intervals fitted to small cells.
CIFAR-100 momentum sweep - fifteen paired gaps. Of the fifteen, the seven at lie at or below 1.23, while the eight at lie at or above 4.04, so the two groups do not overlap. All fifteen, per seed: 0.40, 1.01, 0.74 and 0.62 at ; 1.04, 0.32 and 1.23 at ; 4.04, 4.98 and 4.98 at ; 12.69 and 10.28 at ; 19.84, 21.38 and 18.04 at . The cell means and spreads behind them are in Table 21 in the revised submission (Table 13 in the original).
| teacher | no teacher | |||
|---|---|---|---|---|
| 0 | 4 | 63.27 0.23 | 62.57 0.10 | 0.69 |
| 0.5 | 3 | 63.35 0.54 | 62.49 0.25 | 0.86 |
| 0.8 | 3 | 64.01 0.16 | 59.35 0.69 | 4.67 |
| 0.9 | 2 | 63.27 0.07 | 51.78 1.77 | 11.49 |
| 0.95 | 3 | 63.77 0.25 | 44.01 1.44 | 19.75 |
Beyond CIFAR-100. We conducted additional experiments and results are shown below. The more comprehensive results are included in Table 22. Here, every cell carries two seeds, with each gap read inside one seed.
| dataset | =0.9 | =0 |
|---|---|---|
| CIFAR-10 | 5.07, 4.54 | -0.27, 1.05 |
| Tiny-ImageNet | 8.84, 10.16 | 1.11, 0.62 |
| ImageNet-1k | 15.95, 16.81 | 2.37, 2.20 |
On all three, every = 0.9 gap exceeds every = 0 gap on the same dataset, with no overlap. The teacher-trained arms are also nearly invariant where the teacher-free arms are not.
Q3: ImageNet-1k absolute accuracies. Added, with a second seed in all four cells (Table 22 in the revision with per-seed absolutes and kNN in App. E):
| with teacher | without teacher | gap, per seed | |
|---|---|---|---|
| 0.9 | 59.33, 59.32 | 43.38, 42.50 | 15.95, 16.81 |
| 0 | 59.79, 59.80 | 57.43, 57.60 | 2.37, 2.20 |
This is a ResNet-50 at 40 epochs at batch 512, =0.97. The level sits below published ResNet-50 numbers because of the 40-epoch budget, which is why we read the pair qualitatively rather than as an absolute. Its teacher-free runs stay above collapse, though they are the closest to it of any setting we ran.
Due to the time limits, the four runs we added went to a second seed in every cell rather than a longer budget for one of them, because repeating each cell settles more at this size than one longer run would.
Q4: teacher-free BYOL w/o momentum against SimSiam under the same setup. SimSiam at is more accurate: 63.85 against 62.57 for teacher-free BYOL. The 63.85 is the mean over the four seeds in Table 30 of the revision, while 62.57 comes from Table 2's four-seed mean.
The two are different networks, with different projectors, output normalization, and predictors, so the comparison prices two recipes rather than testing what a teacher does. SimSiam’s role in the paper is as a control on the mechanism, not a baseline. It is the framework with no EMA teacher in its standard form; we added one and found its value essentially unchanged by the optimizer’s momentum, 1.33 points against 1.06, where BYOL’s moves from 11.49 to 0.69. That is what shows the momentum dependence is specific to a framework whose teacher is compensating, rather than a generic property of EMA teachers. This is also explained in the response to W1 in Part 1.
response
I thank the authors for the thorough response and the additional experiments. The band-share analysis clearly explains why the oscillation at β = 0.5 is harmless, and the extra seeds make the results much more convincing. However, some concerns remain.
First, the Barlow Twins results make the scope clearer, but each excluded framework is explained by a different reason. These explanations are reasonable but largely post hoc, so the mechanism still holds only for BYOL.
Second, I understand that SimSiam serves as a control rather than a baseline. Still, SimSiam outperforms teacher-free BYOL under the same setup, so the practical benefit of removing the teacher is limited.
Third, the ImageNet-1k results remain preliminary given the short training budget, and removing the teacher still costs over two points even without momentum.
Overall, I think this is a correct and interesting analysis paper with a fairly narrow impact, so I believe weak accept is an appropriate rating.
Followup of 5vzK's Reply
We thank the reviewer for reading the additional runs; the following is the follow-up reply:
On the post-hoc concern. The paper gives three general criteria, and they are conditions under which an EMA teacher can matter at all: (1) training survives without the teacher; (2) a large share of the teacher-free student's motion lies in the 20--200-step band the teacher attenuates; and (3) nothing in the loss already holds the target steady. All three can be read before the is run: the first two from one teacher-free run, and the third from the framework's loss (Table 11 in the revised submission; new, with no counterpart in the raw submission).
BYOL is the one framework meeting all three, which is why its teacher is worth a great deal with the optimizer's momentum and almost nothing without it. Each of the other four fails one of them: SimSiam's motion does not lie in the band, MoCo v3's negatives and Barlow Twins' covariance penalty each already hold the target steady, and DINO does not survive without its teacher, which is the one case the account does not cover. Barlow Twins is the test rather than an illustration, because it passes the first two and its teacher is still not worth keeping. The criteria are stated in terms of a framework's dynamics and its loss rather than BYOL, which is what makes them portable to a framework we did not run.
On SimSiam. We agree. We are not presenting teacher-free BYOL as a competitor. It is part of a trade-off question which is conditional: The trade is worth taking when memory or parameter count is scarce at this resolution and when compute is scarce at larger ones, and not worth taking when accuracy matters most. This is included in App. F that is substantially expanded.
On long budget ImageNet-1k. We finally finished the 200-epoch training, with two seeds in all four cells. It strongly supports the mechanism we present:
| with teacher | without teacher | gap, per seed | |
|---|---|---|---|
| 0.9 | 61.02, 61.15 | 48.25, 48.22 | 12.77, 12.93 |
| 0 | 62.51, 62.60 | 61.13, 61.15 | 1.38, 1.45 |
For easy comparison, we copied the 40-epoch results provided in the first-round rebuttal:
| with teacher | without teacher | gap, per seed | |
|---|---|---|---|
| 0.9 | 59.33, 59.32 | 43.38, 42.50 | 15.95, 16.81 |
| 0 | 59.79, 59.80 | 57.43, 57.60 | 2.37, 2.20 |
Reply to Reviewer 5vzK, Part1
Thank you for the review and suggestions. We provide a point-by-point response below with additional experiments and results.
W1: why the mechanism applies only to BYOL. In the original submission, we applied the mechanism to BYOL, SimSiam, MoCo v3, and DINO. The revision adds, Barlow Twins, giving five. We report BYOL as the main result because it is the one framework where the teacher is worth a lot with the optimizer's momentum and little without it. Our claim is that the optimizer's momentum should matter where the teacher is compensating and nothing else is. That singles out BYOL. The other four frameworks are ruled out in three distinct ways, (1) something else already holds the target steady, (2) the teacher is doing a different job, or (3) the reading does not reproduce. Hence they represent boundary cases.
Barlow Twins is the sharpest of the five, which is why we added it. It is the one framework that meets two conditions — its teacher-free student does not collapse, and about two thirds of its motion lies in the band — and whose teacher is still not worth keeping. Its covariance penalty constrains what the teacher would otherwise constrain, so the harm is already prevented and adding a teacher costs accuracy: −3.94 points at β = 0.9 and −1.70 at β = 0.
We have added this table as Table 11 (App. C) in the revision.
Q1: does SimSiam's student carry the same oscillation, and if so why does it not hurt teacher-free training? The teacher-free β = 0.9 run does hold a peak that survives every admissible search window, at ~11 steps that is below the 20–200-step range the teacher acts on. But that series does not reproduce between the two halves of the probe set, so we report it weakly and do not claim it is the same oscillation. The momentum sweep below does show that motion faster than the band does little harm with or without a teacher.
The revision extends SimSiam’s four-cell comparison to four seeds. The following results are included in Table 30 of the revision.
The means are 1.33 points with the optimizer's momentum and 1.06 without, against BYOL's 11.49 and 0.69. Removing the momentum therefore barely changes what SimSiam's teacher does, so that teacher is not compensating for the momentum, and teacher-free SimSiam is not hurt by it.
Q2: at β=0.5 the student oscillates, yet removing the teacher costs little. What the cost follows is not whether an oscillation exists but how much of the student's motion lies where the teacher acts. We measured that share across the whole momentum sweep, which is new in this revision. At β=0.5, the oscillation is fast, about six steps, and so sits below the 20-200 step range the teacher's action matters over. The EMA already suppresses it strongly there. In the run without the teacher, only about a fifth of the student's motion lies at 20--200 steps, against about three quarters at β=0.9:
The share column is added to Table 12, a new table in the revised submission. The costs come from the Δ column of Table 2, which is numbered the same in the revised and the original submission.