- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work aims to improve the JEPA representation space by decomposing it into state and action regions, each loosely trained to be informative of the current state and the action executed. They propose the Branch-JEPA, which partitions the feature space in State and Act and define a residual branch that is based on the transformer patch embeddings and a CLS branch. Branch-JEPA leads to improvement of robustness to image noise as improved success rate.
Critical strengths
Extensive empirical evaluation to establish performance, including simulator tasks and ablation studies.
Critical weaknesses
- The writing does not hold up to academic standards. The language is very hard to read and descriptions are incomplete (e.g. 4.1 setting and overview). In general, the manuscript lacks clear flow of arguments and conclusions and often connections are missing. The connection of BranchJEPA design with distribution spread as mentioned is not clear (other than being an SSL method). The overall motivation is not clear.
- Motivation section is not convincing. Fig.1a shows the widely established fact that SSL losses like SIGReg exist to prevent collapse - it is not clear how this is related to the noise experiment. The precise conclusion from the noise experiment that would support the motivation of BranchJEPA is also not clear. Moreover, it is not clear why specifically action prediction in JEPA space being robust to noise is important for a specific task or aspect. The authors claim that the experiments motivate the model design to "expose action-associated change" as an important requirement but it is not clear how that conclusion was reached with the experiments provided.
- The action alignment loss entirely depends on the distance between two consecutive action embeddings z_{t+1}-z{t}. The contrastive loss simply learns to distinguish an action and does not guarantee a representation geometry that represents understanding how the action physically affects the environment. Moreover, not considering the state in which the action starts its execution is another important limitation, as how the action is executed is often dependent on the current state. q and k remain undefined in Sec 4.3 which breaks the connection between the displacement delta and the loss.
- The State vectors suggest to be specialized to the state of the environment but there is no loss or mechanism that enforces that or disentagles those features from action. Therefore, the State may not be specialized and could just contain the information already available in the CLS/patches.
- The technical contribution is rather low, as it boils down to an action alignment loss on part of the features. According to Tab.2 it can be replaced with a simple regression loss with minimal accuracy difference.
- The ablation provided does not prove that the action alignment helps. The extra branches and layers simply provide more model capacity that can increase the performance and by choosing not to add the residuals in the ablation this capacity is removed from the model. A better ablation would be to retrain without the action alignment loss to show if it has an effect on performance.
Questions for authors
- How is the gaussian noise parameter 0.05 chosen in the context of the scale of the latent/image space? What is the precise conclusion from the noise experiment that leads to in support of the contributions? Why should the readout be expected to be robust to noise? Is action prediction robustness to Gaussian noise a key problem to solving certain tasks?
- What is the precise proof that Act/State separation itself is the source of improvement?
- In what way is the representation space enhanced? Is the action encoded in a meaningful way in terms of geometry between states? How does that compare with the baseline?