- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies an early phase in JEPA training where the loss drops quickly but the learned representations are no better than those of a random encoder. The authors call this the staleness phase. They attribute it to two effects: the EMA keeps the target encoder too close to the encoder, and the predictor learns much faster than the encoder and fits the targets with a near-trivial solution. They derive an upper bound linking this coupling to the EMA rate and the masking ratio, and propose adding noise to the encoder weights early in training. Experiments on images and time series show small gains in downstream accuracy.
Critical strengths
The phenomenon is interesting and practically relevant, and the paper describes it clearly with useful diagnostics such as gradient norms, effective rank and parameter gaps. The proposed fix is simple, cheap and easy to apply.
Critical weaknesses
- The theory is shallow and its logic does not hold. The main bound is essentially a triangle inequality applied to a single-layer MLP, so it says little about real JEPA models. More importantly, the paper uses an upper bound in the wrong direction. A small bound does imply strong coupling, but a growing bound does not imply that coupling weakens, so the bound cannot explain when the model escapes the staleness phase. The analysis also contradicts itself on masking: the bound implies that a higher masking ratio weakens coupling, yet the text claims it delays escape and later recommends high masking early on. Finally, the predictor half of the mechanism has no theoretical analysis at all.
- The method's evaluation is weak and does not support the proposed mechanism. The paper never measures whether noise injection actually shortens the staleness phase, which is the key link between the theory and the method. The staleness phase covers only a small fraction of training, so shortening it should not produce lasting gains, especially since the authors themselves state that the long-term dynamics are unchanged. The improvements are mostly small, with no reported variance across seeds. The ablation lacks a no-noise baseline and does not show the consistent sweet spot the authors describe. Simple alternatives that target the same problem, such as initializing the target independently, lowering the EMA rate early in training, or slowing down the predictor, are not compared.
Questions for authors
- Does noise injection actually shorten the staleness phase? Can you report its duration with and without noise, across several seeds?
- Given that a slower EMA shortens staleness, how do you reconcile this with the claim that the EMA causes it?
- How does noise injection compare with simply initializing the target encoder independently from the encoder?
- If the noise is absorbed within a few hundred steps and long-term dynamics are unchanged, where does the final accuracy gain come from?