- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LeAVJEPA, an audiovisual extension of LeJEPA using a shared early-fusion transformer. Two joint AV views define a mean projected representation, and modality-specific local views are trained to match this representation. SIGReg prevents collapse (like in LeJEPA), allowing training without an EMA teacher, cross-modal predictor, reconstruction decoder, or contrastive negatives. The main empirical contribution is a controlled study showing that modality-specific local views substantially improve representation quality relative to joint, masked, and other view constructions. The paper evaluates frozen classification/probing, end-to-end fine-tuning, cross-modal retrieval, and qualitative spatial structure in the learned representations.
Critical strengths
-
The insight with regards to view construction is interesting. In my opinion, the strongest result is the comparison in Figure 4 and Table A4. With the backbone and reported training schedule held fixed, modality-specific local views give substantial improvements over joint local views: 47.1% versus 24.7% attentive-probe accuracy. The agreement between linear and attentive evaluation strengthens the reliability of these results.
-
The formulation is simple, which is a strength. The same encoder can process audio, video, and joint inputs, and learns useful representations without a teacher, predictor, decoder, or negatives (which can complicate training or conceptual understanding). Joint input improves over either modality alone in the frozen evaluations. Although the method is not the strongest out of all considered, establishing a functional audiovisual baseline with fewer training components can be useful, and the paper is generally honest and upfront about its limitations.
Critical weaknesses
- The experiments do not clearly show what LeAVJEPA contributes beyond previous work on cross-modal learning and the existing LeJEPA framework.
The teacher-free objective and SIGReg are inherited from LeJEPA. Learning from partial modalities using joint audiovisual targets has also been done before, including the self-supervised formulation of Unified Speech Recognition (Haliassos et al., NeurIPS 2024) (though its token-level teacher–student training differs from this method).
The ablation shows that using audio-only and video-only views is beneficial, but it does not establish whether matching these views to a joint audio-video representation is important. A missing baseline would directly align audio-only and video-only embeddings using the same SIGReg regularisation, without the joint views. The “no local views” experiment does not answer this question as it removes the unimodal observations. Furthermore, gradients flow through the joint centre, so the joint representation is also pulled towards the partial representations rather than acting as a richer "target". The paper offers little evidence about the benefits or limitations of that choice relative to an alternative, where the richer joint view is treated more like a target (e.g., using a stop-gradient so that it's not incentivised to "lose" information by getting pulled to the less rich unimodal views). As a result, the main conceptual contribution is somewhat insufficiently established.
- Though the method is simple, this is not accompanied by a convincing accuracy-cost trade-off. The method remains substantially below the strongest reported alternatives across the main evaluations. For example, its ViT-L reaches 36.03 audiovisual mAP on frozen AudioSet-20K versus 42.90 for MJEPA ViT-L, and 91.3% versus 95.2% on ESC-50. Fine-tuned VGGSound performance is also below the reported baselines. While SoTA results are not necessary for a useful contribution, the paper does not quantify a corresponding efficiency advantage under a setting that would make these losses an attractive trade-off.
Questions for authors
-
Evidence concerning the joint centre. What evidence shows that joint-centre matching is better than an audio-video invariance objective with SIGReg and the same shared backbone?
-
Evidence concerning the simplicity claim. Do you have ablations with a teacher-based alternative, use of predictor or stop-gradient? More ablations showing that these components are unnecessary would strengthen some of the paper's claims.
-
Clarification of the localisation experiment. Is Figure 2 produced by a self-supervised-only checkpoint or after supervised fine-tuning, and how are the attention maps constructed? Is there an audio-mismatch experiment which would show that the localisation changes when the accompanying sound is unrelated?
5 comments
W1. The experiments do not clearly show what LeAVJEPA contributes beyond previous work on cross-modal learning and the existing LeJEPA framework.
LeJEPA provides a stable objective but does not say what to align when the input has two modalities. Our contribution is showing that this choice matters as much as the objective. With the loss held fixed, the comparison above shows that cross-modal learning is much weaker unless single-modality views are matched to a joint audio-visual target. Prior methods that use joint targets, such as USR, rely on a teacher. We show that view construction alone is sufficient and reaches results comparable to far more complex methods with fewer components and less compute. Nothing in this recipe is specific to audio and video, so we expect it to be a useful starting point for future multimodal models. We have added USR (Haliassos et al., 2024) to the related work (Sec. 2).
W2. Though the method is simple, this is not accompanied by a convincing accuracy-cost trade-off.
Our ViT-B now matches MJEPA ViT-B on AS-20k with 3.3× fewer GPU-hours, and our ViT-L leads MJEPA on ESC-50 with 2.7× fewer GPU-hours (Tables A–B). At matched training FLOPs the two reach similar accuracy: MJEPA ViT-L stopped at 102% of our ViT-L training FLOPs gets 42.31 AS-20k AV mAP against our 42.01, and we get there in about half the GPU-hours (420 against 864 H100 GPU-hours) (App. C.5).
Q3. Is Figure 2 produced by a self-supervised-only checkpoint or after supervised fine-tuning, and how are the attention maps constructed? Is there an audio-mismatch experiment which would show that the localisation changes when the accompanying sound is unrelated?
Figure 2 is from a self-supervised-only ViT-B checkpoint (50 epochs of AudioSet-2M, no labels, no fine-tuning). Each map is the last-block attention, averaged over heads, from the audio tokens time-aligned with the displayed frame to that frame's video tokens. It is upsampled for display, with no rollout or thresholding. The revised Fig. 2 caption states that the maps come from the last block of the frozen, self-supervised ViT-B without fine-tuning, and the figure uses our updated checkpoint.
Audio-mismatch test (App. C.6, Fig. A12): We ran our model on AVSBench (S4 single-source, MS3 multi-source). A light segmentation head is trained on frozen video tokens, then tested with the clip's own audio, audio from a different recording, and silence. The encoder is never fine-tuned, so any change comes from how the audio shapes the video tokens.
| Test audio (mIoU) | S4, ViT-B | S4, ViT-L | MS3, ViT-B | MS3, ViT-L |
|---|---|---|---|---|
| Matched | 67.7 | 67.1 | 46.4 | 45.4 |
| Mismatched | 63.4 | 64.7 | 40.9 | 42.3 |
| Silent | 65.1 | 66.2 | 40.1 | 40.8 |
| Matched − mismatched | +4.3 | +2.4 | +5.5 | +3.2 |
Wrong audio lowers mIoU in every setting by up to 5.5. Loudness-matching the swapped audio gives the same drop. On S4, wrong audio is worse than silence, so the video tokens depend on which sound is playing. This is learned with no labels and no localisation objective.
We thank the reviewer again. The central question, whether the joint centre matters, now has a direct answer, the cost–accuracy trade-off is quantified, and the audio-mismatch test shows the learned video tokens depend on the paired audio. We hope these results establish the contribution the reviewer found insufficiently supported, and that the reviewer will consider raising the score.
We thank the reviewer for their patience. The ablations that were still running when we posted our response have finished.
W1 / Q2. Do you have ablations with a teacher-based alternative, use of predictor or stop-gradient?
We ran all three. Each row adds one component to our setup and changes nothing else: ViT-B, AudioSet-2M, 15 epochs, and the same frozen AS-20k probe.
| Model | A | V | AV |
|---|---|---|---|
| Ours | 33.77 | 18.16 | 37.85 |
| + stop-gradient on the joint target | 1.51 | 0.84 | 1.24 |
| + predictor | 1.27 | 0.77 | 1.06 |
| + EMA teacher | 21.99 | 10.12 | 15.61 |
| + EMA teacher + predictor | 12.81 | 7.32 | 9.67 |
The predictors are MLPs from the audio-only and video-only views to the joint target. The EMA teacher provides the joint target with momentum 0.999-1, as in MJEPA, and the EMA rows evaluate the teacher.
We thank the reviewer for a thoughtful review and for engaging with the conceptual question at the heart of the paper. We are glad the reviewer found the view-construction insight interesting and the simplicity of the formulation a strength. Reviewer iQU7 likewise described LeAVJEPA as the first demonstration that LeJEPA carries over to paired audio and video, and Reviewer M3an praised the controlled local-view ablation. The reviewer's two main questions, whether the joint centre matters and whether there is a favourable cost–accuracy trade-off, are now both answered with new experiments.
Since submission, we have improved the training setup in three ways. The method itself is unchanged: the same shared encoder, objective (invariance + SIGReg) and λ, and still no teacher, predictor, decoder or negatives.
We pretrained on AudioSet-2M for 50 epochs at ViT-B and ViT-L. All results below use this setup, except the two ViT-L fine-tuning results in Table A, which start from the epoch-45 checkpoint of the same pretraining run. Some of the experiments the reviewers requested are still running. Each one is marked in the corresponding answer below, and we will post those results as a follow-up comment in this thread as soon as they finish.
Table A. New results (Tables 2 and 4 in the revised paper).
LeAVJEPA now matches MJEPA on AS-20k (ViT-B) and on ESC-50.
Table B. Training cost (same H100, same per-GPU batch).
We train about 3× faster than MJEPA, with less memory and about a third of the GPU-hours.
We ran this comparison directly, and joint-centre matching is better by a large margin. We trained exactly the proposed baseline: the same encoder, SIGReg, augmentations and audio-only / video-only views, with the two aligned directly and no joint view. Matched setup: ViT-B, AudioSet-2M, 15 epochs, and the same frozen AS-20k probe (Table A4 in the revised paper).
Removing the joint centre costs 9.8 AV mAP and degrades both single modalities. The joint view is therefore the key ingredient, not just the presence of unimodal views.
Two results already bear on whether these components are needed. MJEPA uses an EMA teacher, seven predictors and nine loss terms; we use one encoder, two loss terms and one hyperparameter, λ, and now match MJEPA ViT-B on AS-20k (Table A). Omitting them is also what makes training 3.5× faster than MJEPA (Table B), since there is no teacher forward pass and no predictor over the dropped tokens.
We are currently running evaluations for the stop-gradient, EMA-teacher and predictor arms and will report them when they finish.