acceptodds
Under review as a conference paper at ICLR 2027

Two Momenta: A Training-Dynamics View of the EMA Teacher

Abstract

Self-distillation trains a student against a teacher that is an exponential moving average (EMA) of its own weights. Weight averaging is commonly interpreted as a filter: slow changes pass to the teacher largely intact, whereas fast changes are attenuated by an amount determined by the EMA momentum. In self-distillation, however, the filtered trajectory feeds back into the optimization process that generates it. We therefore analyze the EMA teacher explicitly as a filter and ask what it attenuates at each timescale. We answer that question on behavior rather than weights, from frequency-domain measurements of what both networks compute on a fixed set of images at every step. The student trajectory contains a dominant narrow oscillation, one to two orders of magnitude faster than anything the averaging can follow, so the teacher accumulates the student rather than averaging recent copies of it. That oscillation is a resonance of the student's other momentum, the optimizer's: a curvature fitted at one momentum predicts how its period shifts at three others, and the two momenta are coupled through the attenuation of this mode. Removing optimizer momentum at a matched effective step eliminates the identifiable resonance and sharply reduces the teacher's benefit. On CIFAR-100, removing the teacher costs 11.5 linear-probe points at the standard momentum and 0.7 without optimizer momentum. This reduction persists across three additional datasets, including ImageNet-1k, and at twice the training budget; removing the teacher also halves the model's forward FLOPs per training step.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Knowledge Distillation, Koopman, Dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5vzKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks when the EMA teacher in self-distillation is actually needed. The authors record the outputs of the student and teacher on a fixed set of images at every step and analyze them in the frequency domain. In BYOL, they find that the student's trajectory is dominated by a fast oscillation caused by the optimizer's momentum, and that the teacher mainly helps by filtering out this oscillation. Consistent with this, removing optimizer momentum (at a matched effective step size) makes the teacher nearly unnecessary: on CIFAR-100, dropping the teacher costs 11.5 linear-probe points with standard momentum but only 0.7 without it, with similar trends on other datasets and on ImageNet-1k.

Critical strengths

The paper asks a clear and useful question and turns it into a testable hypothesis. The experiments go beyond correlation: a short-memory two-tap teacher that cancels only the oscillation recovers most of the EMA teacher's benefit, the cost of removing the teacher grows steadily with the optimizer's memory, and the authors rule out a pure denoising explanation with a clean argument.

Critical weaknesses
  • The mechanism applies only to BYOL, and the paper does not explain why.
  • The statistical evidence is thin, especially beyond CIFAR-100.
Questions for authors
  • Does SimSiam's student show the same oscillation? If so, why does it not hurt teacher-free training there?
  • At β = 0.5 there is still an oscillation, yet removing the teacher costs almost nothing. What determines whether the oscillation is harmful?
  • Can you report the absolute ImageNet-1k accuracies, ideally with more seeds or a longer budget?
  • How does teacher-free BYOL without momentum compare with SimSiam under the same setup?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer 5vzK, Part1

Thank you for the review and suggestions. We provide a point-by-point response below with additional experiments and results.

W1: why the mechanism applies only to BYOL. In the original submission, we applied the mechanism to BYOL, SimSiam, MoCo v3, and DINO. The revision adds, Barlow Twins, giving five. We report BYOL as the main result because it is the one framework where the teacher is worth a lot with the optimizer's momentum and little without it. Our claim is that the optimizer's momentum should matter where the teacher is compensating and nothing else is. That singles out BYOL. The other four frameworks are ruled out in three distinct ways, (1) something else already holds the target steady, (2) the teacher is doing a different job, or (3) the reading does not reproduce. Hence they represent boundary cases.

Barlow Twins is the sharpest of the five, which is why we added it. It is the one framework that meets two conditions — its teacher-free student does not collapse, and about two thirds of its motion lies in the band — and whose teacher is still not worth keeping. Its covariance penalty constrains what the teacher would otherwise constrain, so the harm is already prevented and adding a teacher costs accuracy: −3.94 points at β\beta = 0.9 and −1.70 at β\beta = 0.

frameworkteacher's value, β=0.9→β=0\beta=0.9 \to \beta=0why
BYOL11.49 →\to 0.69nothing in its loss holds the target steady, and three quarters of its motion sits where the teacher filters
SimSiam (EMA teacher added)1.33 →\to 1.06its motion does not sit there: the peak is below the band, at 11 steps, and the in-band share does not reproduce between the probe halves
MoCo v30.44 →\to 0.17its negatives already hold the target steady
Barlow Twins (EMA teacher added)−3.94→−1.70-3.94 \to -1.70its loss already constrains the output covariance, although its motion does sit there
DINOcollapses at bothtraining does not survive without the teacher

We have added this table as Table 11 (App. C) in the revision.

Q1: does SimSiam's student carry the same oscillation, and if so why does it not hurt teacher-free training? The teacher-free β\beta = 0.9 run does hold a peak that survives every admissible search window, at ~11 steps that is below the 20–200-step range the teacher acts on. But that series does not reproduce between the two halves of the probe set, so we report it weakly and do not claim it is the same oscillation. The momentum sweep below does show that motion faster than the band does little harm with or without a teacher.

The revision extends SimSiam’s four-cell comparison to four seeds. The following results are included in Table 30 of the revision.

seedteacher, β=0.9\beta=0.9noneΔ\Deltateacher, β=0\beta=0noneΔ\Delta
164.1462.741.4065.1163.851.26
264.8263.411.4164.8563.801.05
364.4863.271.2164.5463.730.81
464.9263.631.2965.1364.011.12

The means are 1.33 points with the optimizer's momentum and 1.06 without, against BYOL's 11.49 and 0.69. Removing the momentum therefore barely changes what SimSiam's teacher does, so that teacher is not compensating for the momentum, and teacher-free SimSiam is not hurt by it.

Q2: at β=0.5\beta=0.5 the student oscillates, yet removing the teacher costs little. What the cost follows is not whether an oscillation exists but how much of the student's motion lies where the teacher acts. We measured that share across the whole momentum sweep, which is new in this revision. At β=0.5\beta=0.5, the oscillation is fast, about six steps, and so sits below the 20-200 step range the teacher's action matters over. The EMA already suppresses it strongly there. In the run without the teacher, only about a fifth of the student's motion lies at 20--200 steps, against about three quarters at β=0.9\beta=0.9:

β\betashare of the teacher-free student's motion at 20--200 stepscost of removing the teacher
05.7%0.69
0.520.7%0.86
0.869.6%4.67
0.974.6%11.49
0.9590.6%19.75

The share column is added to Table 12, a new table in the revised submission. The costs come from the Δ\Delta column of Table 2, which is numbered the same in the revised and the original submission.

AuthorsOpenReview ↗

Reply to Reviewer 5vzK, Part2

W2: are the differences statistically meaningful? Yes, at the level of paired gaps, and the revision adds the runs that show it. We read them as paired gaps rather than as intervals fitted to small cells.

CIFAR-100 momentum sweep - fifteen paired gaps. Of the fifteen, the seven at β≤0.5\beta\le0.5 lie at or below 1.23, while the eight at β≥0.8\beta\ge0.8 lie at or above 4.04, so the two groups do not overlap. All fifteen, per seed: 0.40, 1.01, 0.74 and 0.62 at β=0\beta=0; 1.04, 0.32 and 1.23 at β=0.5\beta=0.5; 4.04, 4.98 and 4.98 at β=0.8\beta=0.8; 12.69 and 10.28 at β=0.9\beta=0.9; 19.84, 21.38 and 18.04 at β=0.95\beta=0.95. The cell means and spreads behind them are in Table 21 in the revised submission (Table 13 in the original).

β\betannteacherno teacherΔ\Delta
0463.27 ±\pm 0.2362.57 ±\pm 0.100.69
0.5363.35 ±\pm 0.5462.49 ±\pm 0.250.86
0.8364.01 ±\pm 0.1659.35 ±\pm 0.694.67
0.9263.27 ±\pm 0.0751.78 ±\pm 1.7711.49
0.95363.77 ±\pm 0.2544.01 ±\pm 1.4419.75

Beyond CIFAR-100. We conducted additional experiments and results are shown below. The more comprehensive results are included in Table 22. Here, every cell carries two seeds, with each gap read inside one seed.

datasetβ\beta=0.9β\beta=0
CIFAR-105.07, 4.54-0.27, 1.05
Tiny-ImageNet8.84, 10.161.11, 0.62
ImageNet-1k15.95, 16.812.37, 2.20

On all three, every β\beta = 0.9 gap exceeds every β\beta = 0 gap on the same dataset, with no overlap. The teacher-trained arms are also nearly invariant where the teacher-free arms are not.

Q3: ImageNet-1k absolute accuracies. Added, with a second seed in all four cells (Table 22 in the revision with per-seed absolutes and kNN in App. E):

β\betawith teacherwithout teachergap, per seed
0.959.33, 59.3243.38, 42.5015.95, 16.81
059.79, 59.8057.43, 57.602.37, 2.20

This is a ResNet-50 at 40 epochs at batch 512, λ\lambda=0.97. The level sits below published ResNet-50 numbers because of the 40-epoch budget, which is why we read the pair qualitatively rather than as an absolute. Its teacher-free runs stay above collapse, though they are the closest to it of any setting we ran.

Due to the time limits, the four runs we added went to a second seed in every cell rather than a longer budget for one of them, because repeating each cell settles more at this size than one longer run would.

Q4: teacher-free BYOL w/o momentum against SimSiam under the same setup. SimSiam at β=0\beta=0 is more accurate: 63.85 against 62.57 for teacher-free BYOL. The 63.85 is the mean over the four seeds in Table 30 of the revision, while 62.57 comes from Table 2's four-seed mean.

The two are different networks, with different projectors, output normalization, and predictors, so the comparison prices two recipes rather than testing what a teacher does. SimSiam’s role in the paper is as a control on the mechanism, not a baseline. It is the framework with no EMA teacher in its standard form; we added one and found its value essentially unchanged by the optimizer’s momentum, 1.33 points against 1.06, where BYOL’s moves from 11.49 to 0.69. That is what shows the momentum dependence is specific to a framework whose teacher is compensating, rather than a generic property of EMA teachers. This is also explained in the response to W1 in Part 1.

Reviewer_5vzKOpenReview ↗

response

I thank the authors for the thorough response and the additional experiments. The band-share analysis clearly explains why the oscillation at β = 0.5 is harmless, and the extra seeds make the results much more convincing. However, some concerns remain.

First, the Barlow Twins results make the scope clearer, but each excluded framework is explained by a different reason. These explanations are reasonable but largely post hoc, so the mechanism still holds only for BYOL.

Second, I understand that SimSiam serves as a control rather than a baseline. Still, SimSiam outperforms teacher-free BYOL under the same setup, so the practical benefit of removing the teacher is limited.

Third, the ImageNet-1k results remain preliminary given the short training budget, and removing the teacher still costs over two points even without momentum.

Overall, I think this is a correct and interesting analysis paper with a fairly narrow impact, so I believe weak accept is an appropriate rating.

AuthorsOpenReview ↗

Followup of 5vzK's Reply

We thank the reviewer for reading the additional runs; the following is the follow-up reply:

On the post-hoc concern. The paper gives three general criteria, and they are conditions under which an EMA teacher can matter at all: (1) training survives without the teacher; (2) a large share of the teacher-free student's motion lies in the 20--200-step band the teacher attenuates; and (3) nothing in the loss already holds the target steady. All three can be read before the 2×22\times2 is run: the first two from one teacher-free run, and the third from the framework's loss (Table 11 in the revised submission; new, with no counterpart in the raw submission).

BYOL is the one framework meeting all three, which is why its teacher is worth a great deal with the optimizer's momentum and almost nothing without it. Each of the other four fails one of them: SimSiam's motion does not lie in the band, MoCo v3's negatives and Barlow Twins' covariance penalty each already hold the target steady, and DINO does not survive without its teacher, which is the one case the account does not cover. Barlow Twins is the test rather than an illustration, because it passes the first two and its teacher is still not worth keeping. The criteria are stated in terms of a framework's dynamics and its loss rather than BYOL, which is what makes them portable to a framework we did not run.

On SimSiam. We agree. We are not presenting teacher-free BYOL as a competitor. It is part of a trade-off question which is conditional: The trade is worth taking when memory or parameter count is scarce at this resolution and when compute is scarce at larger ones, and not worth taking when accuracy matters most. This is included in App. F that is substantially expanded.

On long budget ImageNet-1k. We finally finished the 200-epoch training, with two seeds in all four cells. It strongly supports the mechanism we present:

β\betawith teacherwithout teachergap, per seed
0.961.02, 61.1548.25, 48.2212.77, 12.93
062.51, 62.6061.13, 61.151.38, 1.45

For easy comparison, we copied the 40-epoch results provided in the first-round rebuttal:

β\betawith teacherwithout teachergap, per seed
0.959.33, 59.3243.38, 42.5015.95, 16.81
059.79, 59.8057.43, 57.602.37, 2.20
Reviewer_vXMKOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper analyzes the mechanism of Exponential Moving Average (EMA) teachers in self-supervised self-distillation through the lens of signal processing (frequency-domain). The authors identify the EMA teacher as a one-pole low-pass filter of the student network and demonstrate that their analytical framework closely predicts their empirical observations. Using their framework the authors find that the need for low-pass filtering in BYOL is primarily to suppress weight oscillations caused by the student optimizer's momentum. They show that if momentum is disabled (beta = 0), the EMA teacher can be removed entirely. Finally they show that oscillation suppression comes at the cost of lag (age of student it represents). While oscillation suppression is important, lag can hurt accuracy. To address this, they propose two-tap filter method to suppress oscillations while reducing teacher lag.

Critical strengths
  • The authors present novel insights on why and when EMA teachers are needed in BYOL by applying signal processing perspective to training dynamics. Further they test how these findings generalize across other frameworks.
  • The authors validate that their theoretical framework predicts empirical observations well, supporting their theoretical perspective and insights.
Critical weaknesses
  • In particular, AdamW, or more recently, Muon, have become standard optimizers, where setting beta = 0 is rarely done in practice. Furthermore, disabling momentum performs worse overall even in the paper's own experiments (e.g., Table 2: CIFAR-100 drops from 64.18% to 62.57% and Table 3: ViT-Tiny drops from 55.25% to 52.28% ). It would be very interesting to explicitly discuss the practical trade-off here: does the memory/compute savings of removing the EMA teacher justify the performance cost of training with zero or lower momentum (such as beta = 0.5, which also shows a small performance drop)? (Minor note: Ordering the table rows monotonically by beta would make it much easier to read these trends)
  • For Table 2 and Table 3, it would be helpful to include multi-seed experiments with standard errors or 95% confidence intervals, particularly because the performance differences between beta = 0 and beta = 0.5 are relatively small, error bars would allow readers to better interpret the statistical significance of the performance drops.
Questions for authors
  • How would the authors' framework generalize to standard modern optimizers, such as AdamW or Muon?
  • Do the authors recommend the two-tap filter as a practical method to reduce teacher lag, or is it primarily an analytical tool?
  • Would the authors be able to run multi-seed results (with error bars) for the smaller-scale experiments in Tables 2 and 3?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer vXMK, Part1

Thank you for the review. We have conducted additional experiments as suggested.

W1(a) and Q1, Q3: How does the framework generalize to modern optimizers (AdamW or Muon)? We redo the core experiments replacing LARS with AdamW (still using BYOL, ResNet-18, CIFAR-100), comparing three β1\beta_1 settings (first moment) with each setting showing results on linear probe from two seeds. We confirm that the claim we made based on SGD and LARS generalizes to modern optimizers like AdamW.

β1\beta_1teacher, s42teacher-free, s42Δ\Deltateacher, s43teacher-free, s43Δ\Delta
0.960.9353.497.4461.3153.557.76
0.560.1759.430.7460.2158.721.49
059.8859.120.7660.4757.742.73

The teacher’s benefit tracks β1\beta_1: Δ=7.6±0.2\Delta = 7.6 \pm 0.2 at β1=0.9\beta_1 = 0.9, falling to 1.1±0.51.1 \pm 0.5 at β1=0.5\beta_1 = 0.5, and 1.7±1.41.7 \pm 1.4 at β1=0\beta_1 = 0. Since our main results in the original submission use SGD and LARS, this isolates the momentum buffer rather than the update rule as the operative component.

Note that the shrinking Δ\Delta reflects a student that no longer needs the teacher rather than a failure to train, and no teacher-free arm collapses — stable ranks stay between 1.611 and 1.683, against 1.000 for a collapsed cell (See Table 32). Also note that the two-seed spread at low β1\beta_1 is comparable to the residual Δ\Delta; the β1=0.9\beta_1 = 0.9 contrast is well outside it.

Because the diagnostic is defined on recorded behavior and refers to no update rule, it applies to any optimizer, Muon included. We have added Table 32 and a new section (App. I) in the revision with more comprehensive results and in-depth discussion.

W1(b): is the trade-off worth it? It depends on which resource is scarce.

What removal buys: when measured on the copy-free implementation (i.e., implementing without a second copy of the network since we do not need an EMA teacher), removing the teacher takes the parameters held in training from 30.8M to 16.4M, peak memory from 3033 to 2589 MiB, and the model step gets 1.30×1.30\times faster. How much of that reaches the clock depends on the pipeline: on CIFAR-100 the input path costs about as much as the model step, so the actual end-to-end gain is only 2.2%.

What removal costs: the accuracy cost is 0.69 linear-probe points against a teacher-trained run at the same momentum, and 1.33 against the standard scheduled-λ\lambda recipe. The latter is a difference between two recipes' means, not a gap inside a seed. Under AdamW, the same route costs 2.69 points against the standard β1=0.9\beta_1=0.9 run. SimSiam is more accurate at about the same memory and step time.

The two sides move oppositely with scale: at ImageNet-1k, the compute saving grows to 1.33x per model step, with forward FLOPs halving at both shapes, while the memory saving shrinks from 14.6% to 6.1%. At that resolution, a 197 ms model step dominates its own training step, so most of the saving survives to the clock. What improves with scale is compute.

So the answer to the trade-off question is conditional: The trade is worth taking when memory or parameter count is scarce at this resolution and when compute is scarce at larger ones, and not worth taking when the last point of accuracy matters most. This is included in App. F that is substantially expanded.

On the drops you cite. Note that the 64.18 quote was the β\beta = 0.8 cell at a single seed in the original submission. At three seeds it is 64.01, and it is a sweep cell rather than the standard recipe (i.e., the scheduled-λ\lambda recipe’s 63.90 over ten runs). The drops being cited in both cases can be more informative if we decompose them. On CIFAR-100 (Table 2), 64.01 →\rightarrow 63.27 is 0.74 from removing the optimizer’s momentum with the teacher still in place, then 63.27 →\rightarrow 62.57 is 0.69 from removing the teacher at β=0\beta = 0. On the ViT-Tiny (Table 3b), under AdamW, 55.25 →\rightarrow 53.24 is 2.01 from removing the momentum with the teacher in place, then 53.24 →\rightarrow 52.28 is 0.96 from removing the teacher. So in both cases, the drops cited by the reviewer is a sum of two things that the paper treats differently. We claim only the second leg where the teacher becomes nearly free once momentum is gone (0.69, 0.96).

W1(c): reordering the rows in Tables 2 & 3. Fixed and the pattern does flow better.

AuthorsOpenReview ↗

Reply to Reviewer vXMK, Part2

W2 and Q3: seeds and error bars. Every cell now carries its seed count and its spread. The sweep is Table 21 (App. E) in the revised submission that replaces Table 13 in the original submission. Linear probe, mean ±\pm the spread over that cell's repeats, with Δ\Delta the mean paired gap:

β\betannteacherno teacherΔ\Delta
0463.27 ±\pm 0.2362.57 ±\pm 0.100.69
0.5363.35 ±\pm 0.5462.49 ±\pm 0.250.86
0.8364.01 ±\pm 0.1659.35 ±\pm 0.694.67
0.9263.27 ±\pm 0.0751.78 ±\pm 1.7711.49
0.95363.77 ±\pm 0.2544.01 ±\pm 1.4419.75

Q2: what the two-tap teacher is for. It is an analytical tool, not a method we recommend. Keeping only five steps of the student's history lets us ask whether the teacher's benefit needs long averaging, and with the optimizer untouched, it recovers 0.967 of what the EMA teacher is worth. That is what separates suppression from staleness.

Reviewer_RH24OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the self-supervised methods in which a student network is trained using teacher with EMA weights of those of the student. The paper's main focus is on the interaction between optimizer momentum and EMA teacher momentum. The authors analyze training dynamics through frequency domain measurements of network's output observables and propose that a major role of the EMA teacher is to suppress oscillations generated by optimizer momentum. They observe that removing optimizer momentum significantly reduces the performance degradation caused by removing the EMA teacher, and propose a teacher-free BYOL training recipe.

Critical strengths
  1. The paper's main observation that EMA teacher benefit in self-supervised settings like BYOL may strongly depend on optimizer's momentum is interesting and potentially valuable.
Critical weaknesses

1. Paper contribution and significance. While the paper presents several interesting empirical observations, I am not convinced that the authors sufficiently establish their broader scientific or practical significance. In particular, it remains unclear what new behavior the proposed interpretation allows us to predict beyond the investigated settings, under what general conditions the findings should be expected to hold, or what advantages they offer practitioners compared with existing approaches. The authors emphasize three contributions in the introduction, and I have concerns about each of them.

First, the authors claim to identify a role of the EMA teacher and characterize the boundary of the proposed mechanism. I find the resonance-suppression hypothesis plausible, and the authors provide meaningful empirical evidence supporting it. However, the evidence does not conclusively establish the proposed mechanism, and no substantial new theoretical analysis is developed to characterize its applicability. More importantly, I am not convinced by the claim that the authors characterize its boundary. Section 5.1 provides examples of frameworks in which removing optimizer momentum makes the teacher nearly unnecessary, as well as examples where it does not. Although the authors discuss possible reasons for these differences, they do not establish general conditions determining when the proposed mechanism is important or when the teacher can safely be removed. Consequently, I find the predictive value of this characterization limited: it remains unclear how researchers could apply these findings to a new self-supervised learning framework without performing similar experiments.

Second, the authors measure student to teacher transfer in the frequency domain and demonstrate close agreement with the classical EMA transfer function. While this is not mathematically automatic for nonlinear network's output observables, I am not convinced that the significance of this empirical observation is sufficiently established. The paper offers only a brief first-order approximation as an explanation, without developing the mathematical argument or specifying conditions under which the approximation should be expected to hold. More importantly, it is unclear what broader understanding or useful predictions this correspondence provides beyond the particular experiments presented.

Third, the authors propose teacher-free BYOL training recipe. While the teacher-free BYOL configuration reduces model computation, parameter count, and memory usage, the measured end-to-end speedup is only 2.2% in the reported setup. Moreover, the authors' own experiments in Appendix F show that SimSiam achieves higher accuracy at lower or nearly the same step cost than their teacher-free BYOL configuration. I also find the comparison extremely limited, only two algorithm families compared in table 15 on a single dataset. Thus, the paper does not convincingly establish a clear advantage over existing methods to be interested to practitioners.

2. Insufficient clarity and experimental details documentation. The paper is hard to follow and it appears to omit important details as well lacking notation explanations or clear definitions. For instance, the paper's main focus is on student-teacher dynamical system, but this dynamical system have never been introduced explicitly. We can only see a dynamical equation for teacher weights, but what about the student? If the student dynamics is optimizer dependent, that should be explicitly stated as well as the training objective should also be discussed. Otherwise, the setting itself feels too abstract to me. Also, H(ω)H(\omega) was only discussed briefly without clear mathematical definition in terms of Fourier components, so it is quite difficult to see why this quantity is important or interesting. Moreover, the authors do not sufficiently distinguish between weight-space dynamics and the dynamics of nonlinear network-output observables. Equations (2) and (3) describe classical filtering properties of EMA in parameter space, whereas the experiments concern derived observables of network behavior. The transition between these mathematically different objects deserves more discussion.

Apart from that, maybe more importantly, I found not sufficient information about what do the authors actually represent in their plots. The authors discuss several observables in Section 3, such as the logarithm of leading eigenvalue and cosine similarity. However, in Figures 1a, 1b and 2b it is unclear which observables are plotted, I found no information about that in figure captions, main text and appendix. For Figure 2a it seems that appendix G answers that logarithm of leading eigenvalue is plotted, but it reports λˉ=0.99599\bar{\lambda} = 0.99599, but plots on fig 2a are done with λ=0.9\lambda=0.9 and λ=0.99\lambda=0.99, so the explanations are insuficient and confusing. This insufficient documentation is a critical gap for this kind of paper because the manuscript's central scientific claims rely primarily on empirical evidence.

Questions for authors
  1. Can the authors specify precisely which observables are used to obtain the measurements in Figures 1a, 1b, and 2b? For the figure 2b it is also not clear how did the authors obtain scalar quantities out of the frequency dependend ones.
  2. Can the authors provide more direct evidence that suppression of the identified oscillatory mode, rather than other changes in optimization dynamics, explains the teacher's benefit? In particular, how does downstream performance depend on the delay used in the two-tap teacher, and does targeting the measured resonance outperform suppressing other frequencies?
  3. What practical advantage does the proposed teacher-free BYOL configuration offer over existing approaches, particularly SimSiam, which performs better in the authors' own computational comparison?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer RH24, Part 1

Thank you for the detailed reading. We provide our point-to-point response below.

W1(i): where the mechanism applies. The revision states the condition explicitly in Sec. 5.1: “The account says which framework that should be, through three parts: training survives without the teacher; a large share of its teacher-free motion lies in the 20–200-step band the teacher attenuates; and nothing else in the loss already holds the target steady.” The third part is what the account turns on — a teacher can matter where the first two hold, but it is worth keeping only where nothing else already does its job. We ran the same four-cell comparison in five frameworks:

frameworkteacher's value, β=0.9→β=0\beta=0.9 \to \beta=0why
BYOL11.49 →\to 0.69nothing in its loss holds the target steady, and three quarters of its motion sits where the teacher filters
SimSiam, EMA teacher added1.33 →\to 1.06its motion does not sit there: the peak is below the band, at 11 steps, and the in-band share does not reproduce between the probe halves
MoCo v30.44 →\to 0.17its negatives already hold the target steady
Barlow Twins, EMA teacher added−3.94→−1.70-3.94 \to -1.70its loss already constrains the output covariance, although its motion does sit there
DINOcollapses at bothtraining does not survive without the teacher

That is Table 11 in the revised submission. The four rows other than Barlow Twins also appear in Table 4, where DINO's row gives its two removal costs, 51.48 and 45.63, rather than a collapse; the four seeds behind SimSiam's 1.33 and 1.06 are Table 30 (new in the revised submission).

So the boundary can be read from a framework's design plus two teacher-free runs rather than from a full ablation.

W1(ii): what the filter measurement is for. It establishes that the teacher is the filter its definition says it is, on behavior and not only on weights. The measured gain follows H(ω)H(\omega) with no fitted parameter, departing from it by 1.1% at the median for λ=0.90\lambda=0.90 and 1.7% for λ=0.99\lambda=0.99 across the 46 bins of that band (Table 27 in the revised submission). Inverting the measurement then returns the λ\lambda the run used. See Table 17 in the revised submission (Table 9 in the original submission). Two things follow that we could not otherwise say:

  • It is not noise averaging. At a matched effective step, removing the optimizer's momentum raises the per-step noise, so an averaging teacher should matter more. However, it is worth 0.69 points there, against 11.49 with momentum. The filter view explains why that extra noise is invisible to it: momentum low-passes the injected noise with a one-pole filter of the same form as the teacher’s, so the extra noise at β = 0 lies below the band the teacher acts on.
  • It gives a rule. Holding the teacher's attenuation fixed requires 1−λ1-\lambda to shrink as the learning rate does, which is what BYOL's reference schedule already does.

W1(iii) and Q3: significance, and SimSiam. We agree, and we have modified the claim. SimSiam at β=0\beta=0 is more accurate, 63.85 over four seeds against 62.57 for teacher-free BYOL at the same β\beta (Table 30 in the revision):

seedteacher, β=0.9\beta=0.9noneΔ\Deltateacher, β=0\beta=0noneΔ\Delta
164.1462.741.4065.1163.851.26
264.8263.411.4164.8563.801.05
364.4863.271.2164.5463.730.81
464.9263.631.2965.1364.011.12

We no longer present teacher-free BYOL as an advantage over existing methods. Contribution (iii) is now what the mechanism tells a practitioner inside one framework: in BYOL with the optimizer's momentum, keep the teacher; at β≤0.5\beta\le0.5 it can be dropped. Dropping it takes the parameters held in training from 30.8M to 16.4M, peak memory from 3033 to 2589 MiB, and the model step to 1.30×1.30\times faster, measured on the same update implemented without the second copy of the network (App. F in the revised submission), for an accuracy cost of 0.69 points at β=0\beta=0, 0.86 at β=0.5\beta=0.5, or 1.33 against the standard scheduled-λ\lambda recipe. End-to-end, the time gain is 2.2% in our setup, because there the input pipeline costs about as much as the model step.

The evidence for the mechanism is the within-network four-cell across four datasets (Table 3a in both submissions). The cost table prices two families on one dataset (Table 25 in the revision, Table 15 in the original submission), which is narrow, so we draw no ranking from it.

AuthorsOpenReview ↗

Reply to Reviewer RH24, Part 2

W2(a): what the student--teacher system is. The teacher is an exponential moving average of the student's weights and receives no gradient:

θT(t)=λθT(t−1)+(1−λ)θS(t)\theta_T(t) = \lambda\theta_T(t-1) + (1-\lambda)\theta_S(t).

The student is the only network that is trained, with heavy-ball momentum β\beta and learning rate η\eta:

θS(t)=θS(t−1)−ηg(t)+β(θS(t−1)−θS(t−2))\theta_S(t) = \theta_S(t-1) - \eta g(t) + \beta (\theta_S(t-1) - \theta_S(t-2)), with g(t)=∇θSL(θS(t−1),θT(t−1))g(t) = \nabla_{\theta_S}\mathcal{L}(\theta_S(t-1), \theta_T(t-1)).

L\mathcal{L} is BYOL's loss: the negative cosine similarity between the student predictor's output on one augmented view and the teacher projector's on another, summed over both orderings of the views, with a stop-gradient on the teacher branch. These two recursions are the two momenta of the title, and only student network carries a gradient.

W2(b): what H(ω)H(\omega) is. It is the discrete-time transfer function of the EMA recursion above,

H(ω)=(1−λ)/(1−λe−iω)H(\omega) = (1-\lambda)/(1 - \lambda e^{-i\omega}),

so ∣H(ω)∣|H(\omega)| is the factor by which a component of period T=2π/ωT = 2\pi/\omega is attenuated on its way into the teacher. Its phase arg⁡H(ω)\arg H(\omega) is that component's lag. We estimate it from the recorded series as H^=Sxy/Sxx\hat H = S_{xy}/S_{xx}, the cross-spectrum of student and teacher divided by the student's auto-spectrum (App. C in the revised submission).

The other frequency-domain quantity the paper uses is the in-band share. It is the fraction of the increment spectrum’s power falling in bins whose periods lie between 20 and 200 steps, ∑BSxx/∑allSxx\sum_B S_{xx} / \sum_{\text{all}} S_{xx}, computed on a teacher-free run’s log⁡eig1\log\mathrm{eig}_1 series. It matters because it is what orders the teacher’s value across momentum settings, and because it is readable from a single teacher-free run without repeating our ablations.

W2(c): weights against outputs. The EMA is exact on weights and only approximate on outputs. A first-order expansion carries HH to an output observable under two conditions: the observable has to depend on the weights alone, and the averaging window has to be short against the rate at which the network's Jacobian changes. We now state both in Sec. 3 and test them in App. C of the revised submission, which also says where the match holds and where it stops. The same concern governs which observables enter at all: a split-half coherence check on disjoint halves of the probe set retains log⁡eig1\log\mathrm{eig}_1 and the student–teacher agreement and discards the next four eigenvalues (Table 13). App. C also says where the match holds and where it stops.

W2(d) and Q1: which observable, and which run each panel shows. The observable is log⁡eig1\log\mathrm{eig}_1, the log of the leading eigenvalue of the covariance of the projector's outputs on a fixed set of 512 CIFAR-100 images, recorded at every step. Figures 1a and 1b plot it directly, while Figure 2b summarizes it as one amplitude ratio, which the second correction below defines. All three are Figures 1a, 1b and 2b in the original submission too, with the same plotted points, and a table of panel sources (See the new Table 7 added in the revised submission) now names the recorded run behind every panel.

Two corrections here:

  • Figure 2a in both submissions. Its companion table is Table 26 in the revised submission and Table 16 in the original submission. That table described a scheduled-λ\lambda run, not the two fixed-λ\lambda runs that are plotted. The plotted runs are now given.
  • Figure 2b: its scalar was described loosely. It is the amplitude ratio of the student's motion to the teacher's, (∑BSxx/∑BSyy)1/2(\sum_B S_{xx}/\sum_B S_{yy})^{1/2}, taken over the bins that bracket the resonance. The caption now reflects this change.

Beyond these four items, we have gone through the manuscript for the notation problems your reading exposes: the weight trajectory and the function trajectory are now distinguished at every use, the contributions paragraph has been rewritten, and the panel-source table above closes the figure documentation gap.

AuthorsOpenReview ↗

Reply to Reviewer RH24, Part 3

We took Q2 in the following: more direct evidence that suppression, rather than another change in the dynamics, explains the benefit; how downstream performance depends on the delay in the two-tap teacher; and whether targeting the measured resonance outperforms suppressing other frequencies.

Q2(a) Direct evidence. Two interventions that change different things agree. Replacing the EMA with a teacher that keeps only five steps of the student's history, with the optimizer untouched, recovers 0.967 of the EMA teacher's benefit. Removing the optimizer’s momentum instead, with the teacher untouched, takes the cost of dropping the teacher from 11.49 points to 0.69 (Table 2, both submissions). One changes the target and the other the optimizer: the first filters the oscillation, the second never excites it.

The third isolates suppression within the teacher’s own design. At comparable short memory, a λ\lambda = 0.5 EMA — which barely attenuates the band — recovers 0.55, while the two-tap at D = 11, which does attenuate there, recovers 0.89 on the probe and 0.94 on the kNN, despite its longer mean lag (5.5 steps against 1). Conversely, pairs matched on attenuation but differing in lag, (D = 8, D = 14) and (D = 3, D = 25), give the shorter lag 0.16 and 0.14 more recovery. Suppression in the band is what buys recovery; lag is what costs it; and the two-tap design lets them be set independently, which a first-order EMA cannot.

Q2(b) How the benefit depends on the delay. We swept the delay D of that short-memory teacher over eight values (Table 24 in the revised submission).

Dits zero at periodmean lagrecovered (seed 42)
361.50.695
5102.50.967
8164.00.962
11225.50.915
255012.50.559
5010025.00.635

Recovery is highest strictly inside the sweep, at neither end, and the kNN ranks the eight delays at Spearman ρ = 0.976 with the probe, so the shape is not an artifact of the fitted probe.

Q2(c) Does aiming at the resonance help? The account predicts it should not: a teacher needs short lag and has to act somewhere on the range the motion occupies, and nothing in it says that removing more of that range recovers more. That is what we find. D = 5 through 11 all recover most of the benefit and their differences sit inside run-to-run variation (the recipe’s ten-run spread is 0.39 points), so nulling the 22-step resonance exactly at D = 11 is not better — indeed it is not the maximum. A plain EMA at λ\lambda = 0.85, lag-matched to D = 11, recovers 1.03 while attenuating the resonance only 2.02× rather than nulling it.

What does matter is acting on the range at all: D = 3, whose zero sits at 6 steps and leaves the 14–40-step range the resonance occupies nearly untouched (1.10× at 22), recovers only 0.695. Recovery also falls once the lag grows (D ≥\ge 25). We state in App. E the two readings the sweep does not support: the deepest notch is not the best teacher, and the in-band power removed rises monotonically with D while recovery does not.

I thank the authors for their detailed responses and substantial revisions. I appreciate the additional experiments and clarifications, which meaningfully address several of my concerns, particularly regarding the experimental documentation and supporting evidence.

However, my main concern about the paper's scientific significance remains. While the empirical observations are interesting, I am still not convinced that the proposed interpretation provides sufficiently generalizable or predictive scientific insights beyond the investigated settings. Furthermore, despite the revisions, I continue to find the manuscript difficult to follow, with extensive technical and numerical details that obscure the main scientific message.

Therefore, I maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Reply to All Reviewers

We thank the reviewers for their insightful comments. Here, we summarize our claim and additional experiments conducted in response to the comments.

The claim. In a self-distillation framework, the optimizer's heavy-ball momentum introduces oscillation into the student, and the EMA teacher acts as a low-pass filter. Removing the momentum removes the oscillation at its source and leaves the teacher with little to do. This is shown in Table 2 of the original submission: on CIFAR-100, at the standard setup (β=0.9\beta=0.9 with optimizer momentum), removing the teacher costs 11.49 linear-probe points. However, if we set β=0\beta=0, which removes the optimizer's momentum, removing the teacher only costs 0.69 linear-probe points. Put it another way, the teacher's contribution depends on the optimizer's momentum: when momentum puts a large share of the student's motion on the 20--200-step timescales, the teacher's filtering matters most; otherwise, training can survive without the teacher.

Highlights of additional experiments conducted.

  • We showed the claim generalizes to modern optimizers and added experiments and results using AdamW (vXMK).
  • We added a fifth framework, Barlow Twins, to provide more boundary scenarios on how teacher might contribute to training stability (5vzK, RH24).
  • We conducted more experiments to run multi-seed results using ImageNet-1k (5vzK, vXMK).
  • We added a more comprehensive experiment that conducts a two-tap delay swept over eight values (RH24).

We also clarified the contribution and elaborated on the student--teacher system, the filter H(ω)H(\omega), and the observable behind every panel (RH24).

Changes since submission are marked in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.