- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper tries to understand in which conditions would a JEPA model collapse during training (i.e., returns a trivial mapping of latent state representation). The paper tries to study different penalty/regularisation for the loss for T-JEPA models, and also at different levels of training precision, finding different amounts of collapse during training. It also provides some simple fix to the issue (like using a variance floor).
Critical strengths
- There are some attempts to tackle a problem which can potentially improve the understanding of JEPA models. This would be useful as the JEPA models gain more popularity and usage in practice.
Critical weaknesses
-
The writing is hard to follow in general, and many parts require several re-reads to understand. Some of my examples for this are:
- In the first two sections, I feel it is not so immediately clear to follow with what the work is exploring or doing (especially if the reader is already not familiar with the area). The experimental setup for the paper remains unclear even after reading this point.
- In the proceeding sections, the results are not organised in a manner that can be easily/logically followed, but rather as scattered findings.
- Much of the results seem to be relegated to the appendix which may affect the flow of the writing, since I would have to check with the appendix to see the results that support the statement made in the paper. It is likely that not all the parts in the paper is required (for example, the very specific details about which seeds give what characteristics), and that some of the details can be placed in the appendix in favour of some more representative results.
- While there are some attempts to summarise the results, it still remains unclear from reading as to what is the main outcome of the work, and in which context it may be useful. I feel this would be useful to state earlier on, and with some empirical demonstration that can be useful in a wider context for JEPA.
I believe major editing to the writing would be required to make the paper more presentable and understandable (potentially beyond what would be possible during the rebuttal period).
-
While JEPA is a useful model to study, the paper seems to make little attempt to frame their finding as to why it is useful in practice. I feel there are limited discussion (e.g., in Section 7) as to implications of the work to practical JEPA training, but this discussion is limited. It would benefit from a more practical discussed use cases or from a better set of experiments/demonstrations to how the finding can be used. As of now, the findings seem unclear as to what it is, and how it can be useful in practice. This point also somewhat links back to the comment about the writing from the above, that the paper seems to make it remain unclear.
-
For the results specifically: experiments are presented in a way that seems doubtful to its generalisability (t seems to fixate itself on specific seeds of the training, and is unclear as to whether the findings are true outside the settings provided in the paper), while the theoretical results are unclear as how closely related they would be to the practical T-JEPA training scenarios for them to be applicable.
Other minor typos (not consequential to my decision)
- Line 98: indicator variable \mathbb{1} rendered incorrectly (I think need to use \mathbbm{1} or something)
Questions for authors
-
What is the motivation behind the work provided, and how can the findings of the paper be useful for people using or training T-JEPA models in practice? What proposed changes might be possible to the conventional T-JEPA training currently?
-
Why does the paper study T-JEPA models specifically? Would the findings be generalisable to other variants of JEPA models as well?