- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes InnerJEPA, an auxiliary training objective for video multimodal models. The method masks entire visual time slices and requires the causal language model to reconstruct their corresponding frozen visual features. The authors find that direct cosine regression leads to severe representation collapse, and further introduce target centering, removal of the dominant direction, and variance and covariance regularization. During inference, all auxiliary modules are removed, so no additional inference cost is introduced.
Critical strengths
1、The method is simple and practical. InnerJEPA directly reuses the existing frozen visual encoder and causal language model, adding only auxiliary modules during training that can be fully removed at inference time. 2、The paper clearly shows that direct cosine regression produces predictions with very low effective rank and highly parallel representations, and quantifies this phenomenon using effective rank and pairwise cosine similarity. This analysis itself provides useful insight.
Critical weaknesses
1、The current experiments do not clearly separate the benefit of latent prediction itself from that of masking. In InnerJEPA, the CE loss is computed on the same masked sequence. From the current description, it is unclear whether the SFT baseline uses exactly the same visual masking with only removed. If the baseline uses unmasked inputs, masking itself becomes an additional training intervention and may contribute to the performance gain. Therefore, a masked-SFT baseline with the same masking but without the reconstruction objective is necessary for proper attribution. 2、The anti-collapse design has not been directly linked to downstream performance improvements. The current ablations clearly show that different regularization designs significantly change the geometry of the predicted representations, but the paper does not report downstream QA or grounding results for these main ablation settings. The current evidence therefore does not show that reducing representation collapse itself is responsible for the downstream improvement.
Questions for authors
1、Does the SFT baseline use exactly the same visual masking as InnerJEPA, with only removed? If not, could the authors add this masked-SFT control? 2、Could the authors report downstream QA and grounding results for the main collapse ablations in Table 4? In particular, a comparison between cosine-only and the full method would help verify whether the degree of representation collapse actually corresponds to downstream performance.
2 comments
Official Comment by Reviewer zxvN
Thank you for clarifying the SFT setup and revising the claims. Two experiments are still needed to address my concerns.
- Masked-SFT control. Use the same visual masking policy and learned mask token as InnerJEPA, set , and keep all other training and evaluation settings fixed. Report QA accuracy and temporal-grounding mIoU alongside the results in Tables 1 and 2.
- Downstream evaluation of the collapse ablation. Compare cosine-only and the full objective on QA and temporal grounding, using matched training steps, learning-rate schedules, and evaluation settings.
I also noticed that the entire abstract and much of the method section are marked in blue. Could you briefly summarize the substantive changes and clarify whether any of the edits reflect changes to the training procedure or implementation? If so, which experiments were rerun with those changes?
Response to Reviewer zxvN
Thank you for identifying the two controls most directly relevant to interpreting our results. We have revised the manuscript to distinguish the demonstrated benefits of the complete recipe from component-level attribution.
R1. Does SFT use identical masking? No. The reported SFT baseline uses unmasked inputs; InnerJEPA computes CE on masked inputs and adds the visual objective. Section 5.1 now states this difference explicitly.
The existing comparison shows that the complete recipe improves the reported 0.8B SFT reference across all four QA benchmarks, with gains of 1.07-3.09 percentage points, and across four of five grounding benchmarks. It does not determine how much of this improvement comes from masking alone. The appropriate additional control is masked-SFT with the same masking policy and λ=0, holding the remaining training and evaluation settings fixed. No result for that control is included in the current evidence.
R2. Does preventing collapse improve downstream performance? The current ablations establish an effect on prediction geometry: at the matched step-3000 horizon, cosine-only predictions have effective rank 14.44 and pairwise cosine 0.908, compared with 253.84 and 0.179 for the full objective. This demonstrates that the regularized objective avoids the nearly parallel predictions observed in the cosine-only diagnostic.
However, these measurements do not establish that the geometry change causes the benchmark improvement. We have clarified in Sections 3 and 5.3 that the downstream mask-ratio and loss-weight experiments vary different factors and cannot replace evaluation of the collapse-ablation checkpoints. A training-budget-matched cosine-only versus full-objective comparison on QA and grounding is needed to test that connection.
The revised manuscript therefore preserves two supported findings: the complete recipe improves the reported 0.8B downstream comparison, and the full objective produces less collapsed predictions in the diagnostic setting. It no longer treats the causal connection between these findings as established.