acceptodds
Under review as a conference paper at ICLR 2027

LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning

Abstract

Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes audio, video, and joint audio-video inputs. Modality dropout treats a missing modality as another view of the same event, making cross-modal alignment implicit in the objective. The model aligns global embeddings with modality-specific local embeddings, and SIGReg prevents representational collapse. A controlled ablation identifies modality dropout as the key mechanism for audio-visual alignment. Despite the architectural simplicity, LeAVJEPA reaches 36.0 mAP on AudioSet-20K and 91.3% accuracy on ESC-50 under frozen evaluation. After fine-tuning, it reaches 61.1% accuracy on VGGSound, and its embeddings support zero-shot audio-visual retrieval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised learning, Audio-visual learning, Multimodality, Joint-Embedding Predictive Architectures

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9j9MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LeAVJEPA, an audiovisual extension of LeJEPA using a shared early-fusion transformer. Two joint AV views define a mean projected representation, and modality-specific local views are trained to match this representation. SIGReg prevents collapse (like in LeJEPA), allowing training without an EMA teacher, cross-modal predictor, reconstruction decoder, or contrastive negatives. The main empirical contribution is a controlled study showing that modality-specific local views substantially improve representation quality relative to joint, masked, and other view constructions. The paper evaluates frozen classification/probing, end-to-end fine-tuning, cross-modal retrieval, and qualitative spatial structure in the learned representations.

Critical strengths
  1. The insight with regards to view construction is interesting. In my opinion, the strongest result is the comparison in Figure 4 and Table A4. With the backbone and reported training schedule held fixed, modality-specific local views give substantial improvements over joint local views: 47.1% versus 24.7% attentive-probe accuracy. The agreement between linear and attentive evaluation strengthens the reliability of these results.

  2. The formulation is simple, which is a strength. The same encoder can process audio, video, and joint inputs, and learns useful representations without a teacher, predictor, decoder, or negatives (which can complicate training or conceptual understanding). Joint input improves over either modality alone in the frozen evaluations. Although the method is not the strongest out of all considered, establishing a functional audiovisual baseline with fewer training components can be useful, and the paper is generally honest and upfront about its limitations.

Critical weaknesses
  1. The experiments do not clearly show what LeAVJEPA contributes beyond previous work on cross-modal learning and the existing LeJEPA framework.

The teacher-free objective and SIGReg are inherited from LeJEPA. Learning from partial modalities using joint audiovisual targets has also been done before, including the self-supervised formulation of Unified Speech Recognition (Haliassos et al., NeurIPS 2024) (though its token-level teacher–student training differs from this method).

The ablation shows that using audio-only and video-only views is beneficial, but it does not establish whether matching these views to a joint audio-video representation is important. A missing baseline would directly align audio-only and video-only embeddings using the same SIGReg regularisation, without the joint views. The “no local views” experiment does not answer this question as it removes the unimodal observations. Furthermore, gradients flow through the joint centre, so the joint representation is also pulled towards the partial representations rather than acting as a richer "target". The paper offers little evidence about the benefits or limitations of that choice relative to an alternative, where the richer joint view is treated more like a target (e.g., using a stop-gradient so that it's not incentivised to "lose" information by getting pulled to the less rich unimodal views). As a result, the main conceptual contribution is somewhat insufficiently established.

  1. Though the method is simple, this is not accompanied by a convincing accuracy-cost trade-off. The method remains substantially below the strongest reported alternatives across the main evaluations. For example, its ViT-L reaches 36.03 audiovisual mAP on frozen AudioSet-20K versus 42.90 for MJEPA ViT-L, and 91.3% versus 95.2% on ESC-50. Fine-tuned VGGSound performance is also below the reported baselines. While SoTA results are not necessary for a useful contribution, the paper does not quantify a corresponding efficiency advantage under a setting that would make these losses an attractive trade-off.
Questions for authors
  1. Evidence concerning the joint centre. What evidence shows that joint-centre matching is better than an audio-video invariance objective with SIGReg and the same shared backbone?

  2. Evidence concerning the simplicity claim. Do you have ablations with a teacher-based alternative, use of predictor or stop-gradient? More ablations showing that these components are unnecessary would strengthen some of the paper's claims.

  3. Clarification of the localisation experiment. Is Figure 2 produced by a self-supervised-only checkpoint or after supervised fine-tuning, and how are the attention maps constructed? Is there an audio-mismatch experiment which would show that the localisation changes when the accompanying sound is unrelated?

5 comments

We thank the reviewer for a thoughtful review and for engaging with the conceptual question at the heart of the paper. We are glad the reviewer found the view-construction insight interesting and the simplicity of the formulation a strength. Reviewer iQU7 likewise described LeAVJEPA as the first demonstration that LeJEPA carries over to paired audio and video, and Reviewer M3an praised the controlled local-view ablation. The reviewer's two main questions, whether the joint centre matters and whether there is a favourable cost–accuracy trade-off, are now both answered with new experiments.

Since submission, we have improved the training setup in three ways. The method itself is unchanged: the same shared encoder, objective (invariance + SIGReg) and λ, and still no teacher, predictor, decoder or negatives.

  1. More local views: 12 audio-only and video-only local views instead of 2, with one global view instead of 2.
  2. Augmentations: crops, colour jitter, blur, etc. on video; gain, noise, reverb, etc. on audio.
  3. Token dropping: a random subset of input tokens is dropped in each view (50% of video and 25% of audio in the global view, 50% in the local views), halving the tokens the encoder processes.

We pretrained on AudioSet-2M for 50 epochs at ViT-B and ViT-L. All results below use this setup, except the two ViT-L fine-tuning results in Table A, which start from the epoch-45 checkpoint of the same pretraining run. Some of the experiments the reviewers requested are still running. Each one is marked in the corresponding answer below, and we will post those results as a follow-up comment in this thread as soon as they finish.

Table A. New results (Tables 2 and 4 in the revised paper).

LeAVJEPA now matches MJEPA on AS-20k (ViT-B) and on ESC-50.

BenchmarkPreviousOurs ViT-BOurs ViT-LReference
AS-20k frozen, AV mAP36.03 (L)40.4542.01MJEPA B 40.68 / L 42.90
AS-20k fine-tuned, AV mAP29.6 (B)43.4144.53CAV-MAE 42.0
ESC-50 frozen, acc91.3 (L)94.6295.45MJEPA L 95.2
VGGSound fine-tuned61.1 (L)62.7364.17AV-MAE 64.2

Table B. Training cost (same H100, same per-GPU batch).

We train about 3× faster than MJEPA, with less memory and about a third of the GPU-hours.

Samples / sPeak memory (GB)GPU-h, full schedule
Ours ViT-B152.828.1174
MJEPA ViT-B44.257.4579
Ours ViT-L63.239.6420
MJEPA ViT-L22.349.11,149

W1 / Q1. What evidence shows that joint-centre matching is better than an audio-video invariance objective with SIGReg and the same shared backbone?

We ran this comparison directly, and joint-centre matching is better by a large margin. We trained exactly the proposed baseline: the same encoder, SIGReg, augmentations and audio-only / video-only views, with the two aligned directly and no joint view. Matched setup: ViT-B, AudioSet-2M, 15 epochs, and the same frozen AS-20k probe (Table A4 in the revised paper).

TargetAVAV
Joint AV centre (ours)33.7718.1637.85
Direct A ↔ V alignment28.7612.9528.03
Joint centre, but local views keep both modalities31.7210.8934.90

Removing the joint centre costs 9.8 AV mAP and degrades both single modalities. The joint view is therefore the key ingredient, not just the presence of unimodal views.

W1 / Q2. Do you have ablations with a teacher-based alternative, use of predictor or stop-gradient?

Two results already bear on whether these components are needed. MJEPA uses an EMA teacher, seven predictors and nine loss terms; we use one encoder, two loss terms and one hyperparameter, λ, and now match MJEPA ViT-B on AS-20k (Table A). Omitting them is also what makes training 3.5× faster than MJEPA (Table B), since there is no teacher forward pass and no predictor over the dropped tokens.

We are currently running evaluations for the stop-gradient, EMA-teacher and predictor arms and will report them when they finish.

W1. The experiments do not clearly show what LeAVJEPA contributes beyond previous work on cross-modal learning and the existing LeJEPA framework.

LeJEPA provides a stable objective but does not say what to align when the input has two modalities. Our contribution is showing that this choice matters as much as the objective. With the loss held fixed, the comparison above shows that cross-modal learning is much weaker unless single-modality views are matched to a joint audio-visual target. Prior methods that use joint targets, such as USR, rely on a teacher. We show that view construction alone is sufficient and reaches results comparable to far more complex methods with fewer components and less compute. Nothing in this recipe is specific to audio and video, so we expect it to be a useful starting point for future multimodal models. We have added USR (Haliassos et al., 2024) to the related work (Sec. 2).

W2. Though the method is simple, this is not accompanied by a convincing accuracy-cost trade-off.

Our ViT-B now matches MJEPA ViT-B on AS-20k with 3.3× fewer GPU-hours, and our ViT-L leads MJEPA on ESC-50 with 2.7× fewer GPU-hours (Tables A–B). At matched training FLOPs the two reach similar accuracy: MJEPA ViT-L stopped at 102% of our ViT-L training FLOPs gets 42.31 AS-20k AV mAP against our 42.01, and we get there in about half the GPU-hours (420 against 864 H100 GPU-hours) (App. C.5).

Q3. Is Figure 2 produced by a self-supervised-only checkpoint or after supervised fine-tuning, and how are the attention maps constructed? Is there an audio-mismatch experiment which would show that the localisation changes when the accompanying sound is unrelated?

Figure 2 is from a self-supervised-only ViT-B checkpoint (50 epochs of AudioSet-2M, no labels, no fine-tuning). Each map is the last-block attention, averaged over heads, from the audio tokens time-aligned with the displayed frame to that frame's video tokens. It is upsampled for display, with no rollout or thresholding. The revised Fig. 2 caption states that the maps come from the last block of the frozen, self-supervised ViT-B without fine-tuning, and the figure uses our updated checkpoint.

Audio-mismatch test (App. C.6, Fig. A12): We ran our model on AVSBench (S4 single-source, MS3 multi-source). A light segmentation head is trained on frozen video tokens, then tested with the clip's own audio, audio from a different recording, and silence. The encoder is never fine-tuned, so any change comes from how the audio shapes the video tokens.

Test audio (mIoU)S4, ViT-BS4, ViT-LMS3, ViT-BMS3, ViT-L
Matched67.767.146.445.4
Mismatched63.464.740.942.3
Silent65.166.240.140.8
Matched − mismatched+4.3+2.4+5.5+3.2

Wrong audio lowers mIoU in every setting by up to 5.5. Loudness-matching the swapped audio gives the same drop. On S4, wrong audio is worse than silence, so the video tokens depend on which sound is playing. This is learned with no labels and no localisation objective.


We thank the reviewer again. The central question, whether the joint centre matters, now has a direct answer, the cost–accuracy trade-off is quantified, and the audio-mismatch test shows the learned video tokens depend on the paired audio. We hope these results establish the contribution the reviewer found insufficiently supported, and that the reviewer will consider raising the score.

We thank the reviewer for their patience. The ablations that were still running when we posted our response have finished.


W1 / Q2. Do you have ablations with a teacher-based alternative, use of predictor or stop-gradient?

We ran all three. Each row adds one component to our setup and changes nothing else: ViT-B, AudioSet-2M, 15 epochs, and the same frozen AS-20k probe.

ModelAVAV
Ours33.7718.1637.85
+ stop-gradient on the joint target1.510.841.24
+ predictor1.270.771.06
+ EMA teacher21.9910.1215.61
+ EMA teacher + predictor12.817.329.67

The predictors are MLPs from the audio-only and video-only views to the joint target. The EMA teacher provides the joint target with momentum 0.999-1, as in MJEPA, and the EMA rows evaluate the teacher.

Reviewer_iQU7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LeAVJEPA, which extends the LeJEPA objective (an invariance loss plus SIGReg) to audio-visual self-supervised learning. A single early-fusion ViT encodes audio-only, video-only, and joint inputs. Two global views contain both modalities; local views are made audio-only or video-only through modality dropout, and all view embeddings are pulled toward the mean of the global embeddings of the same clip. The method uses no EMA teacher, predictor, reconstruction decoder, or contrastive negatives. Based on a view-composition ablation on VGGSound, the authors argue that modality-specific local views are the key ingredient for cross-modal learning. They report frozen attentive-probe results on AudioSet-20K and ESC-50, zero-shot audio-visual retrieval on the CAV-MAE AudioSet subset, and end-to-end fine-tuning on VGGSound. The results are below prior work, and the paper positions the method as a much simpler alternative that remains competitive.

Critical strengths
  1. One shared encoder and two loss terms are enough to avoid collapse without a teacher, predictor, or stop-gradient, and the λ=0 and λ=1 runs make the role of each term clear. As far as I know, this is the first demonstration that the LeJEPA recipe carries over to paired audio and video, which is a useful data point for the community.

  2. The paper draws attention to view composition as a design variable in multimodal joint-embedding learning. With the encoder, loss, and budget held fixed, the choice of local views changes probe accuracy substantially, and paired audio and video embeddings become retrievable well above chance without any explicit cross-modal objective.

Critical weaknesses
  1. The performance gap is large, and the benefit of the simplification is not quantified.

The most direct comparison is MJEPA, which also uses a single shared encoder, AudioSet-only pretraining, and the same frozen protocol, and differs mainly in having an EMA teacher and a predictor. At ViT-B scale, LeAVJEPA is about 10 mAP lower on AudioSet-20K with audio-video input (30.98 vs. 40.68), and the ViT-L model (36.03) is still below MJEPA ViT-B. Zero-shot retrieval with ViT-B is below CAV-MAE in both directions, and after fine-tuning on VGGSound, the best model is 3 to 6 points below the listed baselines. Since the main argument of the paper is simplicity, I would expect it to show what the simplification buys, for example wall-clock time, memory, throughput, training stability, or tuning effort compared with MJEPA. No such comparison is given. The reported cost (about 192 hours on 8 H200 GPUs for ViT-B) does not suggest a cheap method either, since every step encodes four views of up to 1969 tokens.

On novelty, the method is essentially the LeJEPA objective combined with modality dropout, which has been used before (e.g., AV-HuBERT, which the authors cite). A single shared ViT for audio and video is also not new. AVSiam uses one, and it appears in Table 4 but not in Table 1 or in the related-work discussion of shared encoders, so the novelty of the shared-encoder design is somewhat overstated.

  1. The evidence for the central claim (that modality-specific views are the key mechanism) has confounds, and the experimental reporting is inconsistent in several places.

(a) The joint-view baselines may be weak for reasons unrelated to view composition. The global views use only a centre crop and horizontal flip for video and a clean spectrogram for audio, and the two 8-second crops come from the same 10-second clip, so the global views overlap heavily. The paper does not specify which audio augmentations, if any, are applied to the joint local views. If the audio in every view is close to identical, the single-modality shortcut described in Sec. 4.1 comes almost for free, and the gap in Table A4 may reflect weak augmentation rather than a fundamental limitation of joint views. The no-local-view control (10.3% linear top-1) is close to an untrained baseline, which is consistent with this reading. The shortcut explanation itself is also not tested directly, for example by probing the joint-view model with audio-only and video-only inputs.

(b) The hyperparameter studies were run in a different regime from the main results. The λ and K sweeps use one GPU with a per-GPU batch of 32, whereas the main runs use effective batches of 320 and 1024. The SIGReg test statistic depends on the number of samples it is computed over, so the conclusion that the method is insensitive to λ at or below 0.05 does not necessarily hold at the batch sizes used for the reported results. The K sweep covers only 7 epochs.

(c) The ablation settings are described inconsistently. Sec. 3.1 describes the local views as clean audio-only or video-only views. Sec. C.5, however, describes the ablation baseline as using modality dropout together with tube and frequency/time masking, and C.5.3 removes "modality dropout and masking" jointly. It is therefore unclear whether the baseline in Fig. A12 to A14 is the method described in Sec. 3. C.5.3 also cannot separate the effect of dropping a modality from that of masking.

(d) Several reported results raise questions about the experimental setup:

  • The headline 36.03 mAP comes from a ViT-L checkpoint trained for 31 epochs (Sec. C.3.1), while Table A1 reports ViT-L training to epoch 66.
  • Fine-tuning on AudioSet-20K gives lower mAP than the frozen probe for every input type (Table A3 vs. Table 2).
  • Sec. C.3.1 mentions modality dropout 0.25 during VGGSound fine-tuning, which does not appear in Table A2.
  • Fig. A6 shows 14 fine-tuning epochs, while Table A2 lists 20.
  • All results are single runs, and a refit of the headline AudioSet-20K probe alone changes it by about 0.5 mAP.
Questions for authors

Clarifications:

  1. Which audio and video augmentations are applied to the local views in each arm of Table A4, in particular the "joint" and "no local views" arms? Is the audio in these arms augmented beyond the temporal crop?

  2. Is the baseline in Sec. C.5 the method described in Sec. 3.1, or a variant with masking? Were the λ=0 and λ=1 runs in Fig. 4 (right) trained under the same configuration as the view ablations in Fig. 4 (left)?

  3. On AudioSet-20K, the fine-tuned ViT-B (A-V 29.6, A 23.7, V 10.3) is below the frozen attentive probe (30.98, 27.79, 14.82). What explains this? The paper does not describe the AudioSet-20 K fine-tuning recipe. Was it tuned?

  4. Does the VGGSound fine-tuning in Table 4 use modality dropout (0.25, as mentioned in Sec. C.3.1)? If so, please add it to Table A2 and clarify the number of epochs, which differs between Table A2 and Fig. A6.

Suggestions that would affect my assessment:

  1. A joint-view baseline with standard audio augmentation (e.g., SpecAugment on the local views), and a direct test of the shortcut explanation, such as probing the joint-view model with audio-only and video-only inputs.

  2. A λ sweep at the batch size used for the main AudioSet runs, and a longer K sweep.

  3. A cost comparison with MJEPA (wall-clock time, memory, throughput), since simplicity is the main argument for the method.

4 comments

We thank the reviewer for an exceptionally careful and constructive review. We are glad the reviewer sees LeAVJEPA as the first demonstration that the LeJEPA recipe carries over to paired audio and video, and that it highlights view composition as a design variable for multimodal joint-embedding learning. Reviewer 9j9M likewise called the view-construction comparison the strongest result and the simple formulation a strength, and Reviewer M3an praised the controlled local-view ablation. Your suggestions (Q5–Q7) directly shaped the experiments below.

Since submission, we have improved the training setup in three ways. The method itself is unchanged: the same shared encoder, objective (invariance + SIGReg) and λ, and still no teacher, predictor, decoder or negatives.

  1. More local views: 12 audio-only and video-only local views instead of 2, with one global view instead of 2.
  2. Augmentations: crops, colour jitter, blur, etc. on video; gain, noise, reverb, etc. on audio.
  3. Token dropping: a random subset of input tokens is dropped in each view (50% of video and 25% of audio in the global view, 50% in the local views), halving the tokens the encoder processes.

We pretrained on AudioSet-2M for 50 epochs at ViT-B and ViT-L. All results below use this setup, except the two ViT-L fine-tuning results in Table A, which start from the epoch-45 checkpoint of the same pretraining run. Some of the experiments the reviewers requested are still running. Each one is marked in the corresponding answer below, and we will post those results as a follow-up comment in this thread as soon as they finish.

Table A. New results (Tables 2 and 4 in the revised paper).

LeAVJEPA now matches MJEPA on AS-20k (ViT-B) and on ESC-50.

BenchmarkPreviousOurs ViT-BOurs ViT-LReference
AS-20k frozen, AV mAP36.03 (L)40.4542.01MJEPA B 40.68 / L 42.90
AS-20k fine-tuned, AV mAP29.6 (B)43.4144.53CAV-MAE 42.0
ESC-50 frozen, acc91.3 (L)94.6295.45MJEPA L 95.2
VGGSound fine-tuned61.1 (L)62.7364.17AV-MAE 64.2

Table B. Training cost (same H100, same per-GPU batch).

We train about 3× faster than MJEPA, with less memory and about a third of the GPU-hours.

Samples / sPeak memory (GB)GPU-h, full schedule
Ours ViT-B152.828.1174
MJEPA ViT-B44.257.4579
Ours ViT-L63.239.6420
MJEPA ViT-L22.349.11,149

W1 / Q7. A cost comparison with MJEPA (wall-clock time, memory, throughput), since simplicity is the main argument for the method.

Table B gives this comparison: 3.5× (ViT-B) and 2.8× (ViT-L) higher throughput than MJEPA and about one third of the GPU-hours, at MJEPA-level accuracy. The simplification also leaves far fewer components to train and tune. MJEPA needs an EMA teacher, seven predictors and nine separately weighted losses, while we use one encoder, two loss terms and one hyperparameter, λ. Being teacher-free is also what makes token dropping cheap. At matched training FLOPs, MJEPA ViT-L reaches 42.31 mAP against our 42.01, and we get there in about half the GPU-hours (App. C.5).

W1. On novelty, the method is essentially the LeJEPA objective combined with modality dropout, which has been used before (e.g., AV-HuBERT, which the authors cite). A single shared ViT for audio and video is also not new.

The components exist in prior work, and our contribution is showing how they must be combined. LeJEPA provides a stable objective but does not say what to align when the input has two modalities, and modality dropout does not say what the single-modality views should be matched to. With the loss held fixed, our ablations show that cross-modal learning is much weaker unless they are matched to a joint audio-visual target. Getting this right is enough to reach MJEPA-level accuracy at about one third of the cost, with one encoder and no teacher or predictor. Nothing in this recipe is specific to audio and video, so we expect it to be a useful starting point for future multimodal models. We have added AVSiam to Table 1 and to the related-work discussion of shared encoders (Sec. 2).

W2(a) / Q5. A joint-view baseline with standard audio augmentation (e.g., SpecAugment on the local views), and a direct test of the shortcut explanation, such as probing the joint-view model with audio-only and video-only inputs.

We ran both in a matched setup: ViT-B, AudioSet-2M, 15 epochs, identical augmentations, and the same frozen AS-20k probe (Table A4 in the revised paper):

Local viewsAVAV
Audio-only / video-only (ours)33.7718.1637.85
Joint31.7210.8934.90

The joint arm differs only in keeping both modalities in the local views. Its audio score barely moves, but video falls by 40%: the model matches views through audio and stops relying on video. Strong augmentations do not remove this shortcut, so dropping a modality is still necessary. Retrieval shows the same effect: with joint local views, direct-cosine R@1 falls by 76% (A→V) and 97% (V→A).

W2(b) / Q6. A λ sweep at the batch size used for the main AudioSet runs, and a longer K sweep.

We reran the λ sweep on AudioSet-2M (ViT-B, 15 epochs) with the updated setup at batch 512, the batch of all reported results. We have updated this in the paper as well (now Fig. A7, App. C.3).

λAVAV
0.02533.8518.3037.81
0.05 (default)33.7718.1637.85
0.1032.5815.5135.83
0.2023.759.8527.37

Frozen AS-20k mAP; ViT-B, 15 epochs, only λ changed.

The main runs now use K = 12 for 50 epochs. We are currently running the K sweep (K ∈ {2, 6, 12}) and will report it when it finishes.

W2(c) / Q2. Is the baseline in Sec. C.5 the method described in Sec. 3.1, or a variant with masking? Were the λ=0 and λ=1 runs in Fig. 4 (right) trained under the same configuration as the view ablations in Fig. 4 (left)?

The revised paper adds the matched grid shown under W2(a) (Table A4 in the revised paper, App. C.5). It uses one documented base configuration with no tube masking and changes one factor per arm; the λ sweep uses the same base.

Q1. Which audio and video augmentations are applied to the local views in each arm of Table A4, in particular the "joint" and "no local views" arms? Is the audio in these arms augmented beyond the temporal crop?

In the new ablations, every arm uses the identical augmentation stack. The joint arm's local views receive the same audio and video augmentations as ours (Table A1). The augmentations of the submitted arms are now stated in App. C.3.

W2(d) / Q3. On AudioSet-20K, the fine-tuned ViT-B (A-V 29.6, A 23.7, V 10.3) is below the frozen attentive probe (30.98, 27.79, 14.82). What explains this? The paper does not describe the AudioSet-20 K fine-tuning recipe. Was it tuned?

That fine-tuning recipe was untuned. With an MJEPA-style head (15 epochs, tuned backbone learning rate, hold-out selection), fine-tuning now exceeds the frozen probe at both scales (Table A). We have updated the paper to include these details (Table 2, App. B.2).

W2(d). The headline 36.03 mAP comes from a ViT-L checkpoint trained for 31 epochs (Sec. C.3.1), while Table A1 reports ViT-L training to epoch 66.

All new AS-20k frozen-probe results use the final epoch-50 checkpoint, the final layer and the released MJEPA frozen protocol, which is the same protocol behind the MJEPA numbers we compare against (Sec. 4.2, Table A1).

W2(d) / Q4. Does the VGGSound fine-tuning in Table 4 use modality dropout (0.25, as mentioned in Sec. C.3.1)? If so, please add it to Table A2 and clarify the number of epochs, which differs between Table A2 and Fig. A6.

The new VGGSound fine-tuning uses modality dropout 0.25, a backbone learning-rate multiplier of 0.1, mixup 0.2, SpecAugment with both an attentive and linear probe. We have updated the paper to reflect these details (Table A2, Fig. A3).

W2(d). All results are single runs, and a refit of the headline AudioSet-20K probe alone changes it by about 0.5 mAP.

Single pretraining runs are common in this literature because each run is long: one ViT-B run takes about 174 GPU-hours (Table B). Our conclusions rest on effects far larger than the ~0.5 mAP variation the reviewer notes. They also hold across pretraining datasets (VGGSound and AudioSet-2M), model sizes (ViT-B and ViT-L) and four benchmarks (Table A).


We are grateful for this review. Q5–Q7 led directly to the strongest new evidence in this rebuttal. With the updated setup matching MJEPA at a fraction of the cost, we hope the reviewer will consider the main concern resolved and will consider raising the score.

We thank the reviewer for their patience. The K sweep that was still running when we posted our response has finished.


W2(b) / Q6. A λ sweep at the batch size used for the main AudioSet runs, and a longer K sweep.

We swept the number of local views with the updated setup at batch size 512: ViT-B, AudioSet-2M, 15 epochs, and the same frozen AS-20k probe. Only K changes, and the local views alternate between audio-only and video-only, so K = 2 is one of each.

KAVAV
229.5613.5733.68
633.3016.7637.32
12 (default)33.7718.1637.85

Accuracy rises with K for every input type. Most of the gain comes between K = 2 and K = 6, and video benefits the most from the additional views. We have added the table to the appendix in the paper.

With the discussion period drawing to a close, we haven't yet heard back and would appreciate your assessment of our response. As a summary: our rebuttal addresses the questions on the λ and K sweeps, the view construction, and the contribution relative to LeJEPA and MJEPA. LeAVJEPA is on par with MJEPA under frozen evaluation while training about 3× faster, and the paper has been revised accordingly.

Reviewer_M3anOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors apply the LeJEPA objective (Balestriero & LeCun, 2025) to audio-visual SSL using a single early-fusion ViT (Sec. 3). For each 10s clip, two 8s joint global views and two single-modality local views (one audio-only, one video-only via modality dropout) are pulled toward the mean of the global embeddings via MSE, while SIGReg regularizes embeddings toward an isotropic Gaussian without an EMA teacher, predictor, decoder, or negatives. A VGGSound ablation (Sec. 4.1, Fig. 4, Table A4) shows that single-modality local views are critical for cross-modal learning (43.9% linear top-1 vs. 10–14% with joint or no local views). Pretrained on AudioSet-2M, ViT-L reaches 36.03 mAP on frozen AudioSet-20K, 91.3% on ESC-50 (Table 2), ~19% R@1 on zero-shot retrieval (Table 3), and 61.1% when fine-tuned on VGGSound (Table 4).

Critical strengths
  1. I liked the controlled ablation on local views in Section 4.1 (Fig. 4, Table A4). Keeping both modalities in all views gives near-zero invariance loss (0.003–0.004) and unit embedding std (~0.985) yet stays at audio-only accuracy (10.3–13.7% linear top-1), whereas dropping one modality per local view jumps to 43.9% linear / 47.1% attentive, showing that avoiding collapse does not guarantee cross-modal learning.

  2. The setup is a clean, simple reference point (one encoder, one projector, and one loss weight λ=0.05\lambda = 0.05, Sec. 3.2–3.3) that still supports audio, video, and joint tasks as well as above-chance zero-shot retrieval (Table 3). I also appreciate that the authors clearly separate frozen probes, fine-tuning, zero-shot retrieval, and the trained InfoNCE head (Secs. 3.4, 4.2–4.4).

Critical weaknesses
  1. The technical novelty over recent work is narrow. The objective and default λ\lambda come directly from LeJEPA (Sec. 3.3), treating a dropped modality as a view has many precedents (ModDrop; AV-HuBERT / u-HuBERT; Shvetsova et al., 2022; Recasens et al., 2021), and MJEPA (Teotia et al., 2026) already feeds audio-only, video-only, and joint inputs to a shared encoder to predict joint targets from single modalities, yet there is no ablation isolating the effect of removing MJEPA's teacher/predictor or comparing SIGReg to a simpler VICReg regularizer.

  2. The comparison tables overstate the architectural differences and omit relevant baselines. Table 1 marks AV-MAE (Georgescu et al., 2023) as lacking a shared encoder even though that paper studies weight-shared and early-fusion encoders, AVSiam (Lin & Bertasius, 2024), another single shared ViT, is in Table 4 but left out of Tables 1 and 3, and Table 4 omits from-scratch baselines without ImageNet init (from CAV-MAE and AV-MAE) that score higher than LeAVJEPA.

  3. In the VGGSound view ablation (Sec. 4.1, Table A4), the two global views are 8s crops of a 10s clip with only a center crop + flip on video and a clean spectrogram on audio, so they overlap heavily. Because the authors do not compare modality dropout against standard multi-crop local views (shorter temporal and smaller spatial crops keeping both modalities) or stronger global augmentations, it is unclear if dropping a modality is still necessary when views are less redundant.

  4. We do not see a practical advantage in compute, memory, or robustness to make up for the drop in accuracy across benchmarks (Tables 2–4). Because the encoder runs on four unmasked views per clip (Sec. 3.2) rather than the small fraction of tokens used in MAE methods, pretraining is still expensive (Table A1), yet the authors do not compare training speed, memory, or missing-modality robustness at test time.

  5. The fine-tuning and probing protocols have several inconsistencies. On AudioSet-20K, end-to-end fine-tuning of ViT-B (Table A3) scores worse than the frozen probe on the same backbone (Table 2), and Table 2's ViT-L result uses an epoch-31 checkpoint with a 4-layer probe (App. C.3.1, taking the better of two fits) rather than the 66-epoch run in Table A1 and the 1-layer probe described in App. A.2.

  6. A few claims in the text do not match the appendix figures. Section 4.1 claims limited sensitivity to λ\lambda, whereas Appendix C.4.1 (Fig. A10) shows a sharp collapse in probe accuracy at λ=0.10\lambda = 0.10 (using training-split probes on short runs), and the t-SNE/PCA plots in Appendix C.2 (Figs. A7–A8) use fine-tuned rather than pretrained features.

Questions for authors
  1. Since MJEPA uses a similar single-to-joint shared encoder setup, can you isolate how much of the gap in Table 2 comes from removing the EMA teacher and predictor (e.g., by adding a small predictor to LeAVJEPA), and how does SIGReg compare against a tuned VICReg regularizer with the same views and budget?

  2. How does LeAVJEPA compare in Tables 1, 3, and 4 once you include AV-MAE's shared-encoder variants, AVSiam's zero-shot AudioSet retrieval, and the non-ImageNet from-scratch baselines from CAV-MAE and AV-MAE?

  3. Given the heavy overlap between the two 8s global views in Section 4.1 (Table A4), how does modality dropout compare against standard multi-crop local views (shorter temporal and smaller spatial crops keeping both modalities) or stronger global augmentations, and do the gains also hold for single-modality transfer and retrieval?

  4. Can you compare training throughput, peak memory, or matched-compute accuracy against MJEPA, CAV-MAE, or AVSiam, and test whether the model is more robust when a modality is missing or noisy at test time?

  5. Why does Table 2 use the epoch-31 ViT-L checkpoint and a 4-layer probe (App. C.3.1) instead of epoch 66 (Table A1) and the 1-layer probe (App. A.2), which probe was used for ViT-B in Table 2, why does AudioSet-20K fine-tuning (Table A3) score below frozen probing (Table 2), and how do you reconcile Section 4.1's low-sensitivity claim with the collapse at λ=0.10\lambda = 0.10 in Figure A10?

5 comments

We thank the reviewer for a detailed review and for the specific, actionable requests. We are glad the reviewer liked the controlled ablation on local views and found the method clean and simple. Below we address the three main concerns with new experiments: practical advantage (W4), confounds in the view ablation (W3) and protocol consistency (W5–W6).

Since submission, we have improved the training setup in three ways. The method itself is unchanged: the same shared encoder, objective (invariance + SIGReg) and λ, and still no teacher, predictor, decoder or negatives.

  1. More local views: 12 audio-only and video-only local views instead of 2, with one global view instead of 2.
  2. Augmentations: crops, colour jitter, blur, etc. on video; gain, noise, reverb, etc. on audio.
  3. Token dropping: a random subset of input tokens is dropped in each view (50% of video and 25% of audio in the global view, 50% in the local views), halving the tokens the encoder processes.

We pretrained on AudioSet-2M for 50 epochs at ViT-B and ViT-L. All results below use this setup, except the two ViT-L fine-tuning results in Table A, which start from the epoch-45 checkpoint of the same pretraining run. Some of the experiments the reviewers requested are still running. Each one is marked in the corresponding answer below, and we will post those results as a follow-up comment in this thread as soon as they finish.

Table A. New results (Tables 2 and 4 in the revised paper).

LeAVJEPA now matches MJEPA on AS-20k (ViT-B) and on ESC-50.

BenchmarkPreviousOurs ViT-BOurs ViT-LReference
AS-20k frozen, AV mAP36.03 (L)40.4542.01MJEPA B 40.68 / L 42.90
AS-20k fine-tuned, AV mAP29.6 (B)43.4144.53CAV-MAE 42.0
ESC-50 frozen, acc91.3 (L)94.6295.45MJEPA L 95.2
VGGSound fine-tuned61.1 (L)62.7364.17AV-MAE 64.2

Table B. Training cost (same H100, same per-GPU batch).

We train about 3× faster than MJEPA, with less memory and about a third of the GPU-hours.

Samples / sPeak memory (GB)GPU-h, full schedule
Ours ViT-B152.828.1174
MJEPA ViT-B44.257.4579
Ours ViT-L63.239.6420
MJEPA ViT-L22.349.11,149

W4 / Q4. Can you compare training throughput, peak memory, or matched-compute accuracy against MJEPA, CAV-MAE, or AVSiam, and test whether the model is more robust when a modality is missing or noisy at test time?

There is now a clear practical advantage. We train 3.5× (ViT-B) and 2.8× (ViT-L) faster than MJEPA with about one third of the GPU-hours (Table B), while matching MJEPA ViT-B on AS-20k (Table A). The concern in W4 about encoding four unmasked views no longer applies, because we now drop tokens (Sec. 3.2; Tables A4–A5 in the revised paper):

ViT-BTokens / clipSamples / sPeak memory (GB)AS-20k AV mAP (15 epochs)
No token drop4,35792.549.738.86
Token drop (ours)2,285152.828.137.85

Token dropping makes training 1.65× faster and uses 43% less memory, at a cost of 1.0 AS-20k AV mAP in the same 15-epoch setup. Token dropping makes the pretraining task harder, and 15 epochs is early in our 50-epoch schedule.

At matched training FLOPs the two reach similar accuracy: MJEPA ViT-L stopped at 102% of our ViT-L training FLOPs gets 42.31 AS-20k AV mAP against our 42.01, and we get there in about half the GPU-hours (420 against 864 H100 GPU-hours) (App. C.5).

We evaluated our model and MJEPAs under the audio-visual corruptions of AVRobustBench with a frozen probe - MJEPA ViT-L is more robust than our ViT-L (33.64 against 26.68 mean accuracy).

W3 / Q3. Given the heavy overlap between the two 8s global views in Section 4.1 (Table A4), how does modality dropout compare against standard multi-crop local views (shorter temporal and smaller spatial crops keeping both modalities) or stronger global augmentations, and do the gains also hold for single-modality transfer and retrieval?

Modality dropout remains necessary against multi-crop local views with strong augmentations. Matched setup: ViT-B, AudioSet-2M, 15 epochs, identical augmentations, and the same frozen AS-20k probe (Table A4 in the revised paper).

ViewsAVAV
12 audio-only / video-only locals (ours)33.7718.1637.85
12 joint multi-crop locals31.7210.8934.90

The joint arm uses the same short, small multi-crop views and the same strong augmentations, and differs only in keeping both modalities. Dropping a modality is still necessary, and the gain holds for single-modality transfer, most strongly for video.

The gain also holds for retrieval. Same two checkpoints, 1,722-clip AudioSet pool, R@1 (%):

ViewsDirect cosine: A→VV→APost-hoc InfoNCE: A→VV→A
12 audio-only / video-only locals (ours)5.875.4617.7717.48
12 joint multi-crop locals1.390.179.9310.28

The InfoNCE head is trained on frozen features, 300 clips per class.

We are still running the variant with 2 globals + 2 locals and strong augmentations, as well as the originally submitted setup, and will report both when they finish.

W1 / Q1. Since MJEPA uses a similar single-to-joint shared encoder setup, can you isolate how much of the gap in Table 2 comes from removing the EMA teacher and predictor (e.g., by adding a small predictor to LeAVJEPA), and how does SIGReg compare against a tuned VICReg regularizer with the same views and budget?

MJEPAOurs
Networksencoder, EMA teacher, 24-layer predictor, 6 cross-modal predictorsencoder, projector
Loss9 prediction termsInvariance + SIGReg
Collapse preventionEMA teacher, stop-gradient, momentum scheduleSIGReg
Objective hyperparameters9 loss weights, EMA schedule, predictor size, 2 mask generatorsλ
ViT-L throughput (samples / s)22.363.2

MJEPA prevents collapse with an EMA teacher, a stop-gradient and a momentum schedule, and trains a 24-layer predictor and six cross-modal predictors on nine separately weighted prediction losses. LeJEPA argues that such heuristics make JEPA training brittle, and replaces them with SIGReg, which is stable across hyperparameters and has a single trade-off hyperparameter. We keep that property: one encoder, two loss terms and one hyperparameter, λ. Removing the teacher and predictors is also what makes our training 2.8–3.5× faster (Table B), at no loss of accuracy at ViT-B. The view design also matters beyond modality dropout. In the same matched setup, aligning audio and video directly, without the joint centre, costs 9.8 AV mAP (37.85 → 28.03, Table A4 in the revised paper).

We are currently running evaluations for the added-predictor, added-EMA-teacher and tuned-VICReg arms and will report them when they finish.

W2 / Q2. How does LeAVJEPA compare in Tables 1, 3, and 4 once you include AV-MAE's shared-encoder variants, AVSiam's zero-shot AudioSet retrieval, and the non-ImageNet from-scratch baselines from CAV-MAE and AV-MAE?

AV-MAE's main model uses mid-fusion, so it is not truly a single shared encoder and Table 1 keeps that entry; we now credit its shared-encoder variants in the related work (Sec. 2). We have added AVSiam to Tables 1 and 3. Table 4 now includes the from-scratch CAV-MAE and AV-MAE baselines. Our ViT-L is now on par with AV-MAE without ImageNet initialisation.

W5 / Q5. Why does Table 2 use the epoch-31 ViT-L checkpoint and a 4-layer probe (App. C.3.1) instead of epoch 66 (Table A1) and the 1-layer probe (App. A.2), which probe was used for ViT-B in Table 2, why does AudioSet-20K fine-tuning (Table A3) score below frozen probing (Table 2)?

In the revised paper every AS-20k frozen-probe result for both ViT-B and ViT-L uses the final epoch-50 checkpoint. The probe follows the released MJEPA frozen protocol behind the MJEPA numbers we compare against (Table 2, Sec. 4.2). AS-20k fine-tuning now exceeds the frozen probe (Table A).

W6 / Q5. Section 4.1 claims limited sensitivity to λ, whereas Appendix C.4.1 (Fig. A10) shows a sharp collapse in probe accuracy at λ = 0.10 (using training-split probes on short runs), and the t-SNE/PCA plots in Appendix C.2 (Figs. A7–A8) use fine-tuned rather than pretrained features.

We reran the λ sweep on AudioSet-2M (ViT-B, 15 epochs) with the updated setup at batch size 512, the batch of all reported results. We have updated this figure in the paper with these results (now Fig. A7, App. C.3) and reworded the sensitivity claim in Sec. 4.1.

λAVAV
0.02533.8518.3037.81
0.05 (default)33.7718.1637.85
0.1032.5815.5135.83
0.2023.759.8527.37

The t-SNE and PCA plots (now Figs. A4–A5) are from fine-tuned models, which are simply fine-tuned on more data with a linear layer. We will update the paper with comparisons between the fine-tuned and frozen backbones.


With these changes, LeAVJEPA now matches MJEPA ViT-B on AS-20k and leads on ESC-50 while training about 3× more cheaply, and the controls the reviewer requested confirm that modality-specific views and the joint centre drive the result. We hope the reviewer will reconsider their assessment in light of these results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request for Rebuttal Feedback

Thank you again for your thoughtful reviews and constructive feedback. We would greatly appreciate it if you could take a moment to review our rebuttal and let us know whether it adequately addresses your concerns or if any questions remain. We would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.