acceptodds
Withdrawn from ICLR 2027

InnerJEPA: Masked Visual Latent Prediction for Video Understanding and Generative Transfer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video multimodal large language models (MLLMs) are typically instruction-tuned with answer-token supervision, leaving visual prediction implicit. We introduce InnerJEPA, a lightweight auxiliary objective that turns an existing video MLLM into a masked visual latent predictor. Its frozen vision encoder supplies clean target features, while its causal language model predicts masked visual representations through a small projection head. Masking is applied after vision encoding, allowing inputs and targets to share a single encoder pass. The mask embedding and prediction head are discarded at inference. However, naive cosine regression admits a shortcut in the anisotropic target space: predictions become nearly parallel despite decreasing reconstruction loss. InnerJEPA combines target centring, leading-direction removal, and variance–covariance regularisation to counter this degeneracy. In our representation diagnostic, these changes increase prediction effective rank from 14 to 332 and reduce mean pairwise cosine similarity from 0.908 to 0.160. In a controlled Qwen3.5-0.8B comparison, InnerJEPA improves schedule-matched supervised fine-tuning by 1.07, 1.72, 2.45, and 3.09 percentage points on MVBench, TempCompass, NExT-QA, and STAR, respectively, averaging a 2.09-point gain. It also improves four of five temporal-grounding benchmarks, with a 2.34-point gain in mean mIoU. Beyond video understanding, a matched text-to-image transfer experiment using identical alignment-MLP and DiT-LoRA adaptation settings raises CLIP score from 0.2329 to 0.2449 on 553 images generated from GenEval prompts. A blinded pilot on a fixed 20-prompt subset favours InnerJEPA over SFT on 14 prompts, with five ties. These results support collapse-resistant masked latent prediction as useful auxiliary supervision for video understanding and provide evidence of transfer to image generation without adding inference-time components to the video MLLM.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-Embedding Predictive Architectures, Masked Representation Modeling, Representation Collapse, Feature-Space Anisotropy, Self-Supervised Video Representation Learning, Causal Latent Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zxvNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes InnerJEPA, an auxiliary training objective for video multimodal models. The method masks entire visual time slices and requires the causal language model to reconstruct their corresponding frozen visual features. The authors find that direct cosine regression leads to severe representation collapse, and further introduce target centering, removal of the dominant direction, and variance and covariance regularization. During inference, all auxiliary modules are removed, so no additional inference cost is introduced.

Critical strengths

1、The method is simple and practical. InnerJEPA directly reuses the existing frozen visual encoder and causal language model, adding only auxiliary modules during training that can be fully removed at inference time. 2、The paper clearly shows that direct cosine regression produces predictions with very low effective rank and highly parallel representations, and quantifies this phenomenon using effective rank and pairwise cosine similarity. This analysis itself provides useful insight.

Critical weaknesses

1、The current experiments do not clearly separate the benefit of latent prediction itself from that of masking. In InnerJEPA, the CE loss is computed on the same masked sequence. From the current description, it is unclear whether the SFT baseline uses exactly the same visual masking with only LJEPA\mathcal{L}_{\mathrm{JEPA}} removed. If the baseline uses unmasked inputs, masking itself becomes an additional training intervention and may contribute to the performance gain. Therefore, a masked-SFT baseline with the same masking but without the reconstruction objective is necessary for proper attribution. 2、The anti-collapse design has not been directly linked to downstream performance improvements. The current ablations clearly show that different regularization designs significantly change the geometry of the predicted representations, but the paper does not report downstream QA or grounding results for these main ablation settings. The current evidence therefore does not show that reducing representation collapse itself is responsible for the downstream improvement.

Questions for authors

1、Does the SFT baseline use exactly the same visual masking as InnerJEPA, with only LJEPA\mathcal{L}_{\mathrm{JEPA}} removed? If not, could the authors add this masked-SFT control? 2、Could the authors report downstream QA and grounding results for the main collapse ablations in Table 4? In particular, a comparison between cosine-only and the full method would help verify whether the degree of representation collapse actually corresponds to downstream performance.

2 comments
AuthorsOpenReview ↗

Response to Reviewer zxvN

Thank you for identifying the two controls most directly relevant to interpreting our results. We have revised the manuscript to distinguish the demonstrated benefits of the complete recipe from component-level attribution.

R1. Does SFT use identical masking? No. The reported SFT baseline uses unmasked inputs; InnerJEPA computes CE on masked inputs and adds the visual objective. Section 5.1 now states this difference explicitly.

The existing comparison shows that the complete recipe improves the reported 0.8B SFT reference across all four QA benchmarks, with gains of 1.07-3.09 percentage points, and across four of five grounding benchmarks. It does not determine how much of this improvement comes from masking alone. The appropriate additional control is masked-SFT with the same masking policy and λ=0\lambda=0, holding the remaining training and evaluation settings fixed. No result for that control is included in the current evidence.

R2. Does preventing collapse improve downstream performance? The current ablations establish an effect on prediction geometry: at the matched step-3000 horizon, cosine-only predictions have effective rank 14.44 and pairwise cosine 0.908, compared with 253.84 and 0.179 for the full objective. This demonstrates that the regularized objective avoids the nearly parallel predictions observed in the cosine-only diagnostic.

However, these measurements do not establish that the geometry change causes the benchmark improvement. We have clarified in Sections 3 and 5.3 that the downstream mask-ratio and loss-weight experiments vary different factors and cannot replace evaluation of the collapse-ablation checkpoints. A training-budget-matched cosine-only versus full-objective comparison on QA and grounding is needed to test that connection.

The revised manuscript therefore preserves two supported findings: the complete recipe improves the reported 0.8B downstream comparison, and the full objective produces less collapsed predictions in the diagnostic setting. It no longer treats the causal connection between these findings as established.

Reviewer_zxvNOpenReview ↗

Official Comment by Reviewer zxvN

Thank you for clarifying the SFT setup and revising the claims. Two experiments are still needed to address my concerns.

  1. Masked-SFT control. Use the same visual masking policy and learned mask token as InnerJEPA, set λ=0\lambda=0, and keep all other training and evaluation settings fixed. Report QA accuracy and temporal-grounding mIoU alongside the results in Tables 1 and 2.
  2. Downstream evaluation of the collapse ablation. Compare cosine-only and the full objective on QA and temporal grounding, using matched training steps, learning-rate schedules, and evaluation settings.

I also noticed that the entire abstract and much of the method section are marked in blue. Could you briefly summarize the substantive changes and clarify whether any of the edits reflect changes to the training procedure or implementation? If so, which experiments were rerun with those changes?

Reviewer_ucmmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper describes a method for improving VLM performance using video feature self-prediction. Video input frames are masked in VLM input, and masked frames' features used as training targets in an augmented objective. Since features can share dominant directions, mean-centering and first pc removal within microbatches is performed to measure feature similarity. There is also discussion on increasing effective rank of predictions using covariance-based aux objectives, though its connection to downstream performance is not established. The resulting system performs better than matched baselines in video VQA on 0.8B-scale models.

Critical strengths
  • Uses feature comparison to boost performance in answering tasks

  • 0.8B model shows gains in multiple QA benchmarks, 2B model mixed but gains in majority of benchmarks

  • Does not introduce additional inference computation

Critical weaknesses
  • Relation between effective rank and downstream task performance is not established. This in particular is a large gap, as much of the argument from the paper is around rank and variance/covariance, separate from the feature comparison loss. Nothing links these yet.

  • The paper is not very clear in several parts, often using concepts without introducing or defining them, for example much of sec 3 (prediction matrix Y, the "diagnostic run") and 5.5 text-to-image descriptions (see below).

  • Largest model is 2B, and starting to show more limited gains scaling to this size compared to the 0.8B results.

  • Not compared to frame-wise autoregression on targets, nor to masking-only with lambda=0 (masking without feature targets). Both of these would distinguish effects of feature loss from masking

Questions for authors
  • The goal of the system is not clear reading the abstract or intro --- I only understood it reading sec 4

  • "For a centred prediction matrix Y" -- what is this the prediction matrix of exactly? The def here is too abstract

  • l.132 "On the Qwen3.5-0.8B diagnostic run" --- this refers to "the diagnostic run" without explaining its setup. maybe it matches some of the sec 4 expers described later?

  • section 3 is does not have enough context: What is the experiment being performed here? What is its setup, data, inputs/outputs and evals concretely?

  • fig 1: the direction of flow and placement of comparisons, losses and acts hop around between top and bottom, making it difficult to follow

  • l.189: "Section 3 quantifies what each term removes": I don't see this alignment yet from the sec 3 measurements. I can see the relation with z-hat here and the cosine direction argument in sec 3. But I don't see what relates to the other two terms, particularly the third term cov(Y).

  • how many training examples were used in the dataset and over the course of training, and how were these drawn/mixed from the source datasets?

  • table 4 shows rank increase comes mostly from the covariance terms in the loss, but doesn't show downstream task performance for these ablations. So the paper doesn't establish a relation between rank and task performance. In particular, the covariance terms do not include the target features, and so might just encourage amplifying last-layer noise that is not otherwise used or doesn't help downstream tasks.

  • sec 5.5 text-to-image: I don't really understand this section yet. It seems that the three conditions of vlm models (base, sft, innerjepa) are used as source features for text-to-image conditioning in DiT. But what data is used for tuning, what is input to the VLMs, and what is extracted for conditioning features?

  • l.370 "553 images from geneval prompts" -- this seems like a rather small eval set especially for 5% relative performance difference. what is the variance of the observations and significance of this difference?

  • how does this compare to causal frame-wise block autoregressive feature targets for the llm? that is, instead of masking, just use the causal mask to autoregress features

  • table 5: could also compare to lambda=0 (masking only, no feature loss). this would distinguish effect of predicting feature targets from masking regularization only

1 comment
AuthorsOpenReview ↗

Response to Reviewer ucmm

Thank you for the detailed questions. We have addressed the presentation issues with explicit definitions and procedural descriptions, and revised the claims where the current experiments do not establish the proposed explanation.

R1. Goal, prediction matrix, and diagnostic setup. The abstract and introduction now state the goal directly: to test whether training-only visual latent prediction complements answer-token supervision in a video MLLM. The method masks encoded visual time slices and predicts their clean frozen features from the causal LM's masked-slot states.

Section 3 now includes a dedicated "Diagnostic setup" paragraph. Each row of Y∈RN×dY\in\mathbb{R}^{N\times d} is a prediction-head output for one masked visual slot; its target is the corresponding clean feature. NN counts masked tokens rather than videos, and d=1024d=1024. The paragraph identifies the 0.8B setting, frozen vision tower, joint CE and visual objectives, and the step-3000 comparison. We also explain that effective rank is computed on centered predictions, whereas pairwise cosine measures uncentered directions.

R2. Figure 1 and the roles of the loss terms. We have rewritten the caption to give a sequential reading of the diagram: encode the clean video once, retain detached targets, replace selected visual slots, process the corrupted sequence through the causal LM, and apply the visual and language heads at their respective positions.

Section 4 now distinguishes the loss terms precisely: cosine regression aligns predictions with conditioned targets; variance discourages constant coordinates; covariance discourages redundant coordinates. We removed the statement implying that Section 3 independently validates the downstream role of every term.

R3. Rank versus downstream performance. Your noise-amplification concern is valid: high effective rank alone does not imply useful semantic information. The existing diagnostics nevertheless identify a concrete failure mode of naive cosine regression: nearly parallel predictions despite auxiliary optimization. At step 3000, cosine-only rank is 14.44, compared with 253.84 for the full objective. We now limit this conclusion to prediction geometry. Establishing its connection to task performance requires downstream evaluation of the same ablation checkpoints; the mask-ratio and loss-weight experiments do not provide that evidence.

R4. Masking-only and frame-wise AR controls. The reported SFT baseline uses unmasked inputs. Consequently, the full-recipe comparison does not separate input corruption from visual supervision. We explicitly identify same-mask SFT with λ=0\lambda=0 as the required control. Frame-wise causal feature autoregression is also a relevant alternative: it predicts subsequent frame features rather than reconstructing selected masked slots. We do not currently have a matched comparison establishing superiority over that objective.

R5. Data and text-to-image protocol. Section 5.1 identifies the five training sources, but the exact filtered counts, mixture weights, and consumed examples are not yet fully documented; we do not infer them from nominal dataset sizes. For generation, Section 5.5 and the appendix now explain that a text prompt enters a frozen source VLM, its last-layer text states are mapped token-wise through AlignMLP, and the mapped states condition the DiT. Each source checkpoint receives separately trained adapters under the reported matched adaptation protocol. The executed image-text training manifest remains necessary to complete the data specification.

R6. Generation uncertainty and 2B scaling. The CLIP difference is +0.0120 over 553 generated images. We describe it as a higher observed mean, not an established significance result. The separate 20-prompt pilot does not supply uncertainty estimates for that comparison. We also explicitly report the mixed 2B behavior: mean QA changes by -0.09 points and mean grounding by +1.28 points relative to 2B SFT. Thus, we do not claim that the 0.8B gains consistently scale to 2B.

Reviewer_reZ2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes InnerJEPA, a training-only masked visual latent prediction objective for video multimodal large language models. The method masks complete temporal slices of visual tokens and uses the causal language model’s hidden states to predict features from the frozen vision encoder. To address collapse caused by anisotropy in the frozen target space, the authors combine target centering, removal of the leading principal direction, cosine regression, and VICReg-style variance/covariance regularization. The auxiliary head and mask token are discarded at inference, so the method introduces no inference-time computation. On Qwen3.5-0.8B, the method is compared with schedule-matched SFT. The authors report average gains of 2.09 accuracy points over four video-QA benchmarks and 2.34 mIoU points over five temporal-grounding benchmarks. They also evaluate transfer to text-to-image generation through a separately trained alignment MLP and DiT-LoRA adapter, reporting an increase in CLIP score from 0.2329 to 0.2449. Additional experiments analyze prediction effective rank, masking and loss-weight sensitivity, and frame-order perturbations.

Critical strengths
  1. InnerJEPA reuses the existing frozen vision encoder and causal language model, adding only a mask vector and a small prediction head during training. Removing these components at inference is practically attractive because it avoids additional inference latency or architectural complexity.

  2. The analysis shows that cosine-only prediction can produce nearly parallel outputs despite decreasing the auxiliary loss. The combination of effective-rank and pairwise-cosine diagnostics is a useful way to distinguish loss reduction from genuinely diverse latent predictions.

Critical weaknesses
  1. Could the authors report results from multiple independently trained SFT and InnerJEPA models, including confidence intervals or statistical tests, rather than averaging evaluation seeds for one fixed checkpoint?

  2. Which components of InnerJEPA—masking, target centering, PC1 removal, cosine loss, and variance/covariance regularization—are essential for the reported gains?

  3. How does InnerJEPA compare with existing latent reconstruction or visual alignment methods, such as latent denoising and visual embedding distillation, under the same backbone, data, and compute budget?

  4. Since shuffling the frames changes performance by only 0.19 points, what evidence shows that InnerJEPA improves temporal reasoning rather than mainly learning appearance or unordered event information?

Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer reZ2

Thank you for highlighting the practical simplicity of InnerJEPA and the value of its collapse diagnostics. We have revised the manuscript to make the supporting evidence and its scope more explicit.

R1. Component contributions and the evidence for effectiveness. The complete 0.8B recipe improves schedule-matched SFT on all four QA benchmarks, by 1.07, 1.72, 2.45, and 3.09 percentage points, and on four of five grounding benchmarks. These results support the usefulness of the overall recipe. Separately, the matched-step diagnostics show that cosine-only predictions have effective rank 14.44 and pairwise cosine 0.908, whereas the full objective reaches 253.84 and 0.179 at step 3000.

These observations answer different questions: downstream performance and prediction geometry. We have clarified this distinction in Sections 3 and 5.3. In particular, our SFT baseline is unmasked, so the comparison does not isolate latent prediction from masking. A same-mask, zero-visual-loss control and downstream evaluation of the component-removal variants are still needed to establish which components drive the gains. We do not present the geometry measurements as a substitute for those controls.

R2. Training variability and statistical uncertainty. The reported averages use three evaluation seeds for one fixed checkpoint per arm, not three independent training runs. We now state this explicitly in Section 5.1 and the statistics appendix. The consistent direction of the 0.8B benchmark gains is descriptive evidence; it does not establish robustness across training seeds or statistical significance.

R3. Alternative latent-reconstruction methods. Our contribution is an auxiliary objective implemented within the existing frozen vision encoder and causal LM, with no additional video-understanding modules at inference. The current comparisons evaluate this recipe against SFT, rather than establish superiority over latent denoising or visual distillation. We have clarified that public-model scores provide context and do not replace same-backbone, same-data, same-budget objective comparisons.

R4. Temporal reasoning. The order-perturbation experiment shows smaller degradation under reversal, but the +0.19-point change under shuffling does not demonstrate stronger order reasoning. Section 5.6 now distinguishes robustness to these perturbations from sensitivity to temporally necessary information. The QA and grounding improvements remain observed task-level results; their temporal mechanism is not established by this probe.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Manuscript Revisions

We thank all three reviewers for their careful reading and constructive feedback. We have revised the manuscript to clarify the method, make the experimental protocols easier to follow, and align our claims with the available evidence. Revised passages are highlighted in blue. This comment provides a roadmap to the changes; our separate responses to Reviewers reZ2, ucmm, and zxvN address their questions individually.

1. Clearer motivation and method (Abstract, Introduction, Section 4, and Figure 1). We now state the goal directly: to test whether training-only masked visual latent prediction complements answer-token supervision in a video MLLM. We explain the data flow sequentially, from clean frozen visual targets and masked input slots to the causal LM, prediction head, and two losses. We clarify which parameters are updated and which auxiliary components are removed at inference.

2. Concrete collapse diagnostics (Section 3 and Section 5.3). A new diagnostic-setup paragraph defines the prediction matrix: one row per masked-slot prediction, with 1024 feature dimensions, in the 0.8B diagnostic setting. We distinguish centered effective rank from uncentered pairwise cosine and explain the respective roles of target conditioning, cosine alignment, and variance/covariance regularization. The step-3000 comparison documents a geometry change (rank 14.44 to 253.84; pairwise cosine 0.908 to 0.179), not proof that higher rank causes better task performance.

3. More explicit experimental scope (Section 5.1, Sections 5.3-5.4, and the appendix). We clarify that the reported SFT baseline is unmasked and that the reported three-seed averages evaluate one fixed checkpoint per arm, rather than three independently trained models. We distinguish the collapse-component diagnostics from downstream mask-ratio/loss-weight sensitivity experiments and separate the main 0.8B results from the 2B results.

4. Reproducible description of generative transfer (Section 5.5 and the appendix). We specify the conditioning path: prompt text enters a frozen source VLM, its text hidden states pass through AlignMLP, and the mapped features condition the DiT. Base, SFT, and InnerJEPA use the same reported alignment-MLP/DiT-LoRA adaptation protocol, with separately learned adapters. We separate the CLIP comparison on 553 GenEval-prompt generations from the exploratory 20-prompt pilot and qualitative examples. The CLIP increase from 0.2329 to 0.2449 is an observed mean difference, not a significance result or GenEval compositional accuracy.

5. Narrower interpretation of temporal evidence (Section 5.6 and Discussion). We distinguish robustness to frame-order perturbations from evidence of order-sensitive reasoning. In particular, the +0.19-point change under shuffling does not establish stronger temporal reasoning. We also explicitly acknowledge the mixed 2B results rather than claim consistent scaling gains.

What the current evidence supports. The complete 0.8B recipe improves schedule-matched SFT by 2.09 points in mean video-QA accuracy and 2.34 points in mean grounding mIoU; the diagnostic objective produces less collapsed predictions; and the generation probe provides exploratory transfer evidence. These are distinct observations. We do not claim that the current experiments establish a causal link between them.

What remains to be established. The revisions above clarify existing evidence; they do not add results for the requested same-mask SFT control with zero visual loss, downstream evaluation of the collapse-component ablations, matched alternative visual objectives, or independent training-seed replications. Exact executed data-mixture counts and paired uncertainty estimates also remain necessary for a complete account. We have made these boundaries explicit in the manuscript and reviewer-specific replies.

We appreciate the reviewers' feedback, which has helped us sharpen both the presentation and the scope of the conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.