acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Staleness Phase: Understanding and Accelerating JEPA Training

Abstract

The exponential moving average (EMA) mechanism, introduced to stabilize self-supervised learning, coincides with a surprising failure mode in Joint-Embedding Predictive Architectures (JEPAs): during early training, the reconstruction loss plummets while learned representations remain useless for downstream tasks. We call this the “staleness” phase. It arises from two interacting effects: the EMA keeps the target encoder tightly coupled to the main encoder in parameter space, and the predictor, which trains much faster than the encoder, fits the targets of the still uninformative encoder, minimizing the objective while starving the encoder of gradient. We derive an upper bound showing that the coupling depends on the interplay between momentum dynamics and augmentation strategy, and that escape requires the encoder's updates to outpace the EMA's smoothing. This reframes staleness not as an objective flaw but as a transient regime with predictable onset and recovery. The analysis suggests a simple fix: perturbing encoder weights at the start of training breaks the initial symmetry between encoder and target, while the perturbation is quickly absorbed by the EMA and vanishes as training matures. Experiments on images and time-series support the analysis and show that noise injection accelerates recovery and improves final performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-Embedding Predictive Architecture (JEPA), Exponential Moving Average (EMA)

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1SB9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an early phase in JEPA training where the loss drops quickly but the learned representations are no better than those of a random encoder. The authors call this the staleness phase. They attribute it to two effects: the EMA keeps the target encoder too close to the encoder, and the predictor learns much faster than the encoder and fits the targets with a near-trivial solution. They derive an upper bound linking this coupling to the EMA rate and the masking ratio, and propose adding noise to the encoder weights early in training. Experiments on images and time series show small gains in downstream accuracy.

Critical strengths

The phenomenon is interesting and practically relevant, and the paper describes it clearly with useful diagnostics such as gradient norms, effective rank and parameter gaps. The proposed fix is simple, cheap and easy to apply.

Critical weaknesses
  • The theory is shallow and its logic does not hold. The main bound is essentially a triangle inequality applied to a single-layer MLP, so it says little about real JEPA models. More importantly, the paper uses an upper bound in the wrong direction. A small bound does imply strong coupling, but a growing bound does not imply that coupling weakens, so the bound cannot explain when the model escapes the staleness phase. The analysis also contradicts itself on masking: the bound implies that a higher masking ratio weakens coupling, yet the text claims it delays escape and later recommends high masking early on. Finally, the predictor half of the mechanism has no theoretical analysis at all.
  • The method's evaluation is weak and does not support the proposed mechanism. The paper never measures whether noise injection actually shortens the staleness phase, which is the key link between the theory and the method. The staleness phase covers only a small fraction of training, so shortening it should not produce lasting gains, especially since the authors themselves state that the long-term dynamics are unchanged. The improvements are mostly small, with no reported variance across seeds. The ablation lacks a no-noise baseline and does not show the consistent sweet spot the authors describe. Simple alternatives that target the same problem, such as initializing the target independently, lowering the EMA rate early in training, or slowing down the predictor, are not compared.
Questions for authors
  • Does noise injection actually shorten the staleness phase? Can you report its duration with and without noise, across several seeds?
  • Given that a slower EMA shortens staleness, how do you reconcile this with the claim that the EMA causes it?
  • How does noise injection compare with simply initializing the target encoder independently from the encoder?
  • If the noise is absorbed within a few hundred steps and long-term dynamics are unchanged, where does the final accuracy gain come from?
Reviewer_vUHrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates the early training dynamics of joint-embedding predictive architectures (JEPA). The authors identify a "staleness phase" during which the training loss decreases rapidly while the learned representations remain uninformative for downstream tasks. They attribute this phenomenon to two interacting mechanisms: the close coupling between the student and target encoders through EMA, and the predictor learning faster than the encoder, allowing it to minimize the prediction loss while starving the encoder of gradients. To mitigate this, they propose injecting noise into the student encoder's weights during early training, breaking the initial coupling and accelerating recovery from the staleness phase.

Critical strengths
  1. The paper investigates the early training dynamics of JEPA, particularly the interaction between the encoder and predictor. Understanding these dynamics is important for improving representation learning and downstream task performance.
Critical weaknesses
  1. The paper's main contribution is identifying and characterizing a "staleness phase" during early training, in which the encoder receives limited gradients. However, the authors do not provide sufficiently convincing evidence that this phase is necessarily detrimental to the quality of the final representations used for downstream tasks. It is possible that this phase is simply a natural consequence of the optimization dynamics, where the predictor must first adapt before the encoder can learn meaningful representations. Without demonstrating that shortening this phase leads to consistent and substantial improvements in final downstream performance, the practical significance of the proposed intervention remains unclear.

  2. The empirical evaluation provides insufficient evidence to support the practical benefits of the proposed intervention. The authors primarily demonstrate the existence of the staleness phase by examining the initial training iterations on CIFAR-10. However, CIFAR-10 images have an original resolution of only 32×32 pixels, and the authors do not clearly specify the input resolution used in these experiments. If the images are processed at their native resolution, a ViT-B/16 would operate on only four patches, making these experiments poorly representative of standard I-JEPA training. Even if the images are upsampled, their relatively limited visual complexity raises questions about whether the observed dynamics generalize to larger datasets. This makes CIFAR-10 a questionable choice for studying the training dynamics of a ViT-based architecture such as I-JEPA. The paper would benefit from demonstrating the staleness phase on ImageNet-1K in the main paper, including the evolution of linear-probe accuracy during early training. Additionally, reporting the complete training trajectory, potentially over the full 600 epochs, would provide stronger evidence of how the proposed intervention affects convergence and final representation quality. Furthermore, the downstream evaluation is primarily conducted on relatively small-scale datasets and a subset of ImageNet (ImageNet-100). The authors report results for a ViT-B architecture, which appears to rely on their own independently trained I-JEPA baseline, as the official I-JEPA repository only provides pretrained checkpoints for ViT-H and larger architectures. The paper would have benefited from training a ViT-H model for 300 epochs on ImageNet-1K and comparing it directly against the officially released I-JEPA checkpoint. Alternatively, the authors could have evaluated their ViT-B baseline using the same evaluation protocol and benchmark suite as the original I-JEPA paper to establish the reliability and competitiveness of their baseline implementation. Additionally, there are substantial discrepancies between the TS-JEPA baseline results reported in this paper and those reported in the original TS-JEPA work. These discrepancies raise concerns regarding the consistency of the experimental setup and the fairness of the comparisons. The authors should clarify the source of these differences and ensure that the proposed method is evaluated against appropriately reproduced and competitive baselines.

Overall, I find the empirical evidence supporting the benefits of the proposed noise-injection strategy for final downstream performance insufficient. More comprehensive experiments on larger-scale datasets, comparisons against established baselines, and analyses of the complete training dynamics would considerably strengthen the paper's claims.

Questions for authors
  1. Did you train the I-JEPA ViT-B baseline following I-JEPA's protocol for 600 epochs on ImageNet-1k?
  2. If yes, can you please complement with linear probe results on ImageNet-1k for the final checkpoint aswell as every 100th epoch of training?
  3. Could you please clarify the substantial discrepancy between your reported TS-JEPA baseline accuracy on FaultDetectionA (95.64%) and the result reported in the original TS-JEPA paper (85.8 ± 0.1%)? Additionally, I suggest reporting classification accuracy with standard deviations over 10 runs, following the evaluation protocol of the original TS-JEPA paper, to establish the statistical reliability of the reported improvements.
  4. Can the authors quantify the relationship between the duration of the staleness phase and final downstream performance across different noise-injection configurations and random seeds?
Reviewer_1875OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies an early “staleness” phase in EMA-based Joint-Embedding Predictive Architectures, during which the prediction loss decreases rapidly while downstream representation quality stagnates or deteriorates. The authors attribute this behavior to encoder–target coupling and rapid predictor adaptation, which allegedly reduces the learning signal received by the encoder. They derive an upper bound on encoder–target output discrepancy and propose injecting weight noise during early training to disrupt the initial coupling. Validation includes detailed diagnostics on CIFAR-10, transfer evaluations following ImageNet-1K/ImageNet-100 pretraining, and time-series classification experiments.

Critical strengths
  1. The discrepancy between decreasing prediction loss and poor representation quality is practically relevant. The analysis combines downstream accuracy, effective rank, parameter discrepancy, and gradient measurements. Particularly useful is the frozen-encoder experiment, which demonstrates that predictor optimization alone can reproduce much of the initial loss reduction.

  2. Early weight-noise injection requires no additional network or change to the training objective. The reported improvements across image and time-series tasks suggest potential practical value beyond a single dataset, although their reliability and impact on training efficiency require stronger validation.

Critical weaknesses
  1. The evaluation lacks a directly comparable benchmark against the original I-JEPA. Although the paper reports ImageNet-1K pretraining, it does not evaluate the baseline and proposed method using the original I-JEPA’s main ImageNet-1K linear-evaluation benchmark. The original paper provides a ViT-B/16 result, making a comparison at a similar model scale particularly relevant.

  2. The causal explanation and its distinction from existing EMA-based self-supervised learning remain insufficiently established. EMA teachers are also used in MoCo, DINO, and BYOL, with BYOL being particularly relevant because it includes an asymmetric predictor. Their different objectives and collapse-prevention mechanisms mean that sharing EMA does not automatically imply the same failure mode. The paper needs to identify which ingredients make the proposed mechanism applicable to JEPA and distinguish its contribution from prior analyses of EMA–predictor learning dynamics.

Questions for authors
  1. Why is the standard ImageNet-1K linear-evaluation benchmark from the original I-JEPA omitted despite ImageNet-1K pretraining being performed? Does the reported “Orig.” baseline reproduce the original paper’s performance under matched settings, and does noise injection improve that baseline?

  2. Does the defined staleness phase also occur in BYOL, MoCo, or DINO? Which differences in their objectives or architectures determine whether the proposed mechanism applies, and what distinguishes this explanation from previous analyses of EMA–predictor learning dynamics?

  3. Under a matched training protocol, how much does noise injection reduce the steps or computation required to reach a fixed accuracy target, with variability across seeds? Does its advantage persist against baselines with tuned EMA schedules and predictor learning rates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.