acceptodds
Under review as a conference paper at ICLR 2027

Beyond Spread: Distinct Organizational Pathways in Predictive JEPA

Abstract

Existing analyses of geometric regularization in predictive JEPA often emphasize endpoint properties such as feature spread, isotropy, or collapse avoidance. We argue that these endpoint summaries are too coarse to explain how different regularizers shape predictive representations during training. Motivated by observations from a shell-based neighborhood regularization strategy, we compare shell-based neighborhood regularization (shell_context_knn), whitening decorrelation, VICReg-style regularization (vicreg_like), and a repaired SIGReg-like Gaussian matching configuration (sigreg_like) in a controlled PatchJEPA setting. We observe consistent differences in linear probe accuracy, effective rank, and trajectory dynamics across methods. In particular, shell and whitening reach similar endpoint probe performance through visibly different training trajectories, while SIGReg exhibits stronger context-prediction coupling but weaker structural expansion and lower final probe. We further show that these pathway differences remain visible in a joint rank-coupling state space, persist under low-cost hyperparameter perturbations, and respond sharply to mid-training intervention. When the regularizer is switched at step 100, context-prediction coupling rapidly follows the new regularizer style, whereas final probe remains only partially plastic. These patterns motivate an organizational view of regularization in which fast coupling variables and slower predictive-structural variables separate in time. In our controlled setting, these pathways provide a diagnostic observational framework that complements spread-based summaries for understanding downstream behavior.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-Embedding Predictive Architecture, Self-Supervised Learning, Representation Learning, Representation Geometry, Regularization, Learning Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uppvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Focusing on the fact that existing regularization analyses for predictive JEPA mostly depend on end‑point representation statistics, this paper argues such static end‑point measures cannot fully explain how regularizers shape learned representations. In a controlled PatchJEPA setting, it investigates shell‑based neighborhood regularization, whitening decorrelation, VICReg‑style regularization and repaired SIGReg Gaussian‑matching, with linear‑probe accuracy, effective rank and context‑prediction coupling as observables to propose the organizational‑pathway perspective. Experiments reveal different regularizers induce distinct representational organizational pathways, where context‑prediction coupling is a fast, readily‑redirectable variable, whereas final predictive performance and representation structure are slow variables with memory inertia. Rather than rejecting spread‑based analysis, this work presents organizational pathways as a complementary diagnostic framework for predictive‑JEPA representation formation. Its conclusions stem from small‑scale controlled experiments, with no universally optimal regularizer or formal theory offered.

Critical strengths

This paper proposes the perspective of organizational pathways which shows that end‑point representation metrics alone cannot fully evaluate regularization effects, and similar final performance can stem from drastically different training dynamics.

Critical weaknesses
  1. Conclusions are purely empirical without rigorous mathematical proofs for the underlying mechanisms. Metrics such as effective rank and context‑prediction coupling serve only as proxy metric rather than direct causal measurements of intrinsic representation‑learning mechanisms. Moreover, the metric comparison between shell‑based neighborhood regularization and whitening decorrelation is not definitive when accounting for randomness across different seeds.
  2. Although the paper repeatedly emphasizes that it focuses solely on the organizational dynamics during representation formation, its value would be greatly diminished if its findings cannot offer practical guidance for mitigating representation collapse. In other words, supplementary evidence demonstrating the engineering utility of its results is highly desirable.
Questions for authors
  1. All core experiments are limited to the small‑scale controlled Patch‑JEPA setting. Validation experiments on large‑scale real‑image datasets are suggested to verify whether observed phenomena such as organizational pathways and timescale separation still hold, so as to strengthen the external persuasiveness of conclusions.
  2. Relatively rigorous mathematical analysis is recommended to explain the intrinsic mechanisms behind distinct pathways induced by different regularizers and the timescale‑separation effect of representations, which would further improve the credibility of experimental results.
  3. It is suggested to add positive experimental attempts showing how the insights can guide representation‑collapse mitigation or regularizer design, clarifying the practical engineering utility of this work.
Reviewer_JnbuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines how different regularizers shape representation learning dynamics in predictive JEPA. The authors claim that using only endpoint geometric statistics are insufficient to characterize these processes. The authors compare shell-based, whitening, VICReg-style, and SIGReg-like regularization, finding that methods with similar final linear-probe accuracy can have different training trajectories in effective rank and context-prediction correlation.

Critical strengths
  • The paper examines how different regularizers influence representation learning in JEPA, an important question for understanding self-supervised learning.
  • The authors compare multiple regularization methods and examine both final representation quality and training dynamics.
  • The paper includes robustness checks and regularizer-switching experiments to investigate how training dynamics respond to changes in regularization.
Critical weaknesses
  • The main observation that different regularizers produce different training trajectories is unsurprising. The paper does not establish a novel mechanism or methodological contribution beyond these empirical observations.
  • The proposed "organizational pathways" are largely descriptive, without demonstrating how they improve our understanding or practical use of JEPA. In particular, the significance of the context-prediction coupling metric remains unclear.
  • Experiments are restricted to a small, controlled PatchJEPA setting, with limited downstream evaluation to contextualize the reported performance.
Questions for authors
  • The figures are very hard to read and have overlapping text.
  • What is the concrete insight enabled by the proposed organizational pathways that cannot be obtained from conventional training or endpoint metrics?
  • What is the precise definition of ctx_pred_paircorr, and why is this an informative measure of representation organization?
  • In Table A1, differences in final probe accuracy are small relative to the reported standard deviations. What evidence supports the conclusion that predictive quality retains memory of the earlier regularizer?
Reviewer_Nn9WOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the effects of different regularization methods on the representation learning process in predictive JEPA. It introduces an analytical perspective termed Organizational Pathways, which characterizes the formation of representations by tracking the dynamics of Linear Probe performance, Effective Rank, and Context-Prediction Coupling throughout training. The experiments reveal that although Shell and Whitening achieve similar final performance, they exhibit distinct training trajectories. In contrast, SIGReg-like regularization demonstrates strong predictive coupling but relatively low effective rank and downstream performance. By switching regularizers midway through training, the authors further show that Coupling responds rapidly to changes in the regularization objective, whereas final predictive performance retains some influence from earlier training stages. These findings highlight the different timescales at which key variables evolve during the organization of learned representations.

Critical strengths
  1. A relatively novel research perspective. Rather than focusing solely on final representation metrics such as Spread and Effective Rank, this work investigates how different regularization methods influence the dynamic formation of JEPA representations throughout training.

  2. A relatively comprehensive experimental design. The authors compare multiple regularization methods and provide evidence for the existence of distinct organizational pathways through analyses of training trajectories, Dynamic Time Warping (DTW) distances, experiments with multiple random seeds, and hyperparameter perturbations.

  3. The inclusion of intervention experiments. By switching regularizers midway through training, the authors demonstrate that Coupling responds rapidly to changes in the regularization objective, while final predictive performance exhibits a certain degree of dependence on training history. These findings provide empirical evidence for investigating the multi-timescale dynamics of representation learning.

Critical weaknesses
  1. Lack of theoretical support. The paper primarily relies on empirical observations to explain the training behaviors of different regularizers, without providing rigorous theoretical derivations to explain the underlying mechanisms governing the formation of organizational pathways or their relationship with downstream performance.

  2. Unclear practical significance of the core contribution. Although the paper demonstrates that different regularizers exhibit distinct training trajectories, it does not further establish whether these trajectories can predict final performance, guide the design of regularization methods, or improve model performance.

  3. Limited experimental scale and convincingness. The experiments are primarily conducted in small-scale PatchJEPA settings, with limited validation on larger datasets and models. Moreover, the differences in final Linear Probe performance observed in the regularizer-switching experiments are relatively small, making the evidence insufficient to fully support the claim that final performance depends on training history.

  4. Insufficient validation of the definition of organizational pathways. The authors primarily characterize organizational pathways using Effective Rank and Coupling, but do not establish whether these metrics can comprehensively capture the organization of learned representations. Furthermore, the paper does not sufficiently investigate whether the observed differences in organizational pathways correspond to distinct underlying learning mechanisms.

Questions for authors
  1. What is the unique value of Organizational Pathways? The paper demonstrates that different regularizers exhibit distinct training trajectories. However, beyond describing the training process, can these differences provide additional predictive power or mechanistic insights? For example, can early-stage training trajectories be used to predict final Linear Probe performance?
  2. Why are Effective Rank and Context-Prediction Coupling chosen to define Organizational Pathways? Are these two metrics sufficient to characterize the organizational structure of learned representations? Would the same classification of organizational pathways still emerge if alternative geometric metrics were used?
  3. Are the regularizer-switching experiments sufficient to support the claim of training history dependence? The differences in final Linear Probe performance after switching are relatively small, while substantial variations are observed across random seeds. Could the authors provide statistical significance tests and investigate the effects of switching regularizers at different stages of training?
  4. Can more direct experimental evidence be provided to support the proposed mechanistic explanations of different regularizers? The authors attribute the rapid coupling observed with Whitening to global covariance constraints and the gradual organization observed with Shell to local neighborhood adjustments. Could these explanations be validated through gradient analysis, covariance spectrum evolution, or additional intervention experiments?
  5. Can Organizational Pathways effectively guide model training? Could the authors leverage the observed pathway characteristics to design new regularization strategies or training schedules and demonstrate improvements over fixed regularizers? The current Whitening Annealing experiments do not sufficiently establish the practical benefits of the proposed framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.