acceptodds
Under review as a conference paper at ICLR 2027

MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos

Abstract

Audio-visual models improve egocentric action recognition by exploiting complementary cues, yet typically assume that both streams remain available at inference. Existing missing-modality methods operate on trimmed, single-event clips in which a stream is entirely present or absent, whereas real sensors fail and recover within long, untrimmed observations. We redefine egocentric modality missingness as temporally localized sensor outages within untrimmed, multi-event observations, with whole-clip absence as the limiting case. We introduce MacJEPA, a missing-modality-robust Masked-context query JEPA that recognizes visual actions and acoustic events from supplied interval queries over audio-visual context. Window-local modality dropout simulates these sensor outages during training. MacJEPA further repurposes masking in JEPA from a self-supervised pretext into a supervised robustness objective, aligning masked and clean latent representations of both multimodal content tokens and the task-conditioned queries. All objectives are optimized jointly with recognition in a single stage, requiring no test-time adaptation. Across Epic-Kitchens-100 and Epic-Sounds, a single checkpoint remains competitive under complete input and consistently surpasses published missing-modality baselines when either the dominant or auxiliary stream is removed. MacJEPA thus unifies strong full-input recognition with temporal missing-modality robustness in a single model operating on untrimmed multi-event videos.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-visual recognition, missing-modality robustness, untrimmed egocentric videos, Joint-Embedding Predictive Architectures, multimodal learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RWULOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses the challenge of missing modality during inference for long untrimmed videos. Previous work only considered the missing modality challenge for full duration of trimmed videos. The core motivation is that sensors are likely to experience outage during deployment but that could be for a short period of time during the untrimmed video. Further, within the same video, you could experience outage of different modalities at different intervals rather than one modality for the whole duration of the clip. The authors propose a variation of JEPA that uses window local modality dropout that assigns a binary value for masking the features of a specific modality for each window in the non-overlapping windows partitioned from the original observation. Further, masked online representations of both content tokens and task queries are aligned with clean targets from an EMA teacher, trained jointly with classification. They test their model and compare it to baselines on epic-kitchens and epic-sounds

Critical strengths
  • The motivation behind the problem is sound and relaxes several assumptions that might not hold in real-world deployment.
  • Well written and easy to follow
  • The choice of typed and timed absence tokens helps minimise the design while being able to mask either modality within the same architecture.
  • One model for both Epic-Kitchen and epic sounds that does not require adaption during test-time. This gives it a practical advantage during deployment
  • Competitive performance when compared with model’s trained without missing modalities as shown on table 2
Critical weaknesses
  • Joint outage is excluded from the design where in the real world there could be an outage with both the visual and audio sensors at the same time.
  • Supervision on unrecoverable targets. Epic-Kitchens and Epic-Sounds annotations are independent and many acoustic events have no visual counterpart. Figure 1 shows Scrub/Wipe co-occurring with the unrelated Close Kettle. When window-local dropout removes the audio under such an event, the sound query is still trained with cross-entropy on its clean label. Query JEPA also pushes its state toward the clean teacher’s. The model is then asked to predict information that is absent from the input.
  • Unexplored allocation between modalities. Training always splits the outage probability equally (π/2 each), and Appendix B.1 varies only the total π. For example, privacy muting could be more likely than camera failure
  • Substituting features for an interval of a modality during the typed timed absence process during training is only an approximation of what could happen during deployment. During deployment, there could be no data at all e.g. all silence in audio or all black or frozen frames.
  • Train and test outages share the same artificial setup. Real outages have arbitrary durations and start times. Long failures span window boundaries in ways the training sampler never produces.
Questions for authors
  • What fraction of training sound and action queries have all their supporting evidence removed by window-local dropout, e.g. an audio-only event with no overlapping visual action? Does excluding these queries from cross-entropy and query JEPA, or down-weighting them, change clean or robust accuracy?
  • What happens if both modalities are missing? Equivalent to both sensors failing during deployment? It is understandable how you justified not dropping both modalities in L215-217 but that does not rule out that it could happen during deployment for which you are trying to make the model more robust. How does adding a small joint dropout probability during training change the results? Have you tried asymmetric splits (e.g. 0.7π / 0.3π in either direction)? What impact would this have on results and conclusions in terms of robustness to missing one modality over the other?
  • How does substitution of features compare with deleting missing tokens via attention mask? How does the model perform on outages that differ from training e.g. random start times not aligned to the window grid, variable durations?
5 comments
AuthorsOpenReview ↗

Response to Reviewer RWUL Part (1/2)

W2, Q1: “Supervision on unrecoverable targets.”

We appreciate the reviewer for raising this distinction. In Figure 1, Close Kettle is not a semantic counterpart to Scrub/Wipe, and a local audio outage may remove its decisive cue. However, the sound query attends to the full untrimmed observation and may use neighboring audio, visual motion, and broader context. Independent annotations imply unpaired labels and not necessarily absent evidence. If no informative evidence survives, cross-entropy and query JEPA cannot recover it. Instead, they learn the label distribution and clean readout conditioned on the remaining context. The fraction of truly unsupported queries is not identifiable because the annotations specify events, not their supporting evidence. Treating no overlapping visual label as unrecoverable would therefore discard potentially useful temporal or visual cues. We consequently do not exclude or down-weight queries using this proxy and will clarify this distinction in the revision.

Q2.1: “What happens if both modalities are missing?”

We thank the reviewer for raising this important boundary case. MacJEPA remains well-defined under a deployment-time local joint outage: both streams are replaced by their typed, time-stamped absence tokens, while each query can draw on surviving evidence elsewhere in the observation. If both modalities are unavailable for the full observation, no event-specific sensor evidence remains, and predictions are necessarily limited to learned class priors. We will clarify this behavior and recovery limit explicitly in the revised manuscript.

W4: “During deployment, there could be no data at all.”

We thank the reviewer for prompting us to clarify the deployment interface. As defined in Sec. 3.3 and Appendix A.2, MacJEPA does not require black frames or silence to resemble its learned mask vector. Once an interval is marked unavailable by sensor-health metadata or an upstream fault detector, its missing feature is substituted with the learned modality-specific absence vector while retaining its timestamp. The same mechanism therefore represents absent, black, silent, or frozen input after detection, independent of the raw failure signature. This typed, timed interface allows either stream to be handled by the same model without test-time adaptation. We will state this more explicitly in our revised manuscript.

Q3.1: “How does substitution compare with deleting missing tokens?”

We appreciate this question because it highlights the purpose of typed, timed absence. Canonical JEPAs remove target tokens from the context encoder and train a predictor to infer their latent representations. Applying that design here would shift outage handling toward the train-only predictor and prevent the deployed encoder and queries from explicitly identifying the missing modality and interval. MacJEPA instead substitutes only the unavailable content component with learned mask tokens while preserving the token position, time encoding, and modality identity. Content and query attention can then recognize the outage and reroute toward surviving temporal or cross-modal evidence; the predictor is discarded after training. Our choice of substitution therefore follows the objective of teaching the deployed encoder rather than the predictor to recognize and handle missing input.

We hope these explanations address the reviewer’s concerns and questions quoted above and clarify the relevant design choices. We will make these distinctions explicit in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer RWUL Part (2/2)

W1, Q2.2: Joint-outage training.

We appreciate the reviewer for asking how including simultaneous sensor outages during training affects robustness. We evaluate the requested training ablation against our MacJEPA checkpoint while keeping the architecture, curriculum, and optimization fixed. Let PVP_V, PAP_A, PVAP_{VA}, and P0P_0 denote the per-window probabilities of dropping video only, audio only, both modalities, or neither, respectively. Standard MacJEPA uses (PV,PA,PVA,P0)=(π/2,π/2,0,1−π)(P_V,P_A,P_{VA},P_0)=(\pi/2,\pi/2,0,1-\pi), with π=0.7\pi=0.7. The joint-outage variant adds the fourth state and uses (π/3,π/3,π/3,1−π)(\pi/3,\pi/3,\pi/3,1-\pi). Both models are evaluated under our nested per-window dominant modality missingness protocol; values are top-1 action accuracy for Epic-Kitchens (EK) and sound accuracy for Epic-Sound (ES) (%).

Training samplerEK-V 0EK-V 25EK-V 50EK-V 75EK-V 100ES-A 0ES-A 25ES-A 50ES-A 75ES-A 100
With joint-outage state53.052.451.546.317.556.655.753.951.849.3
MacJEPA54.053.752.247.319.558.156.054.352.150.0

Joint-outage training lowers clean and missing-input accuracy throughout. At fixed π\pi, it reduces exposure to single-modality outages and adds windows with no sensor evidence, weakening the clean–robustness balance discussed in Sec. 4.2. These results support our two-state dropout design for the dominant sensor failure setting studied in the paper. We include the protocol details and results in Appendix Sec. B.6.1 and Table 11 of the revised manuscript.

W3, Q2.3: Training-time asymmetric modality dropout.

We thank the reviewer for prompting this test. We keep π=0.7\pi=0.7 and all other settings fixed, and compare MacJEPA's (PV,PA)=(π/2,π/2)(P_V,P_A)=(\pi/2,\pi/2) with (3π/4,π/4)(3\pi/4,\pi/4) and (π/4,3π/4)(\pi/4,3\pi/4). Evaluation uses nested per-window dominant-modality missingness protocol.

Training dropout allocationEK-V 0EK-V 25EK-V 50EK-V 75EK-V 100ES-A 0ES-A 25ES-A 50ES-A 75ES-A 100
Audio-drop-heavy: PA=3π/4P_A=3\pi/453.852.350.144.417.855.856.254.851.049.1
Video-drop-heavy: PV=3π/4P_V=3\pi/452.853.553.148.919.257.753.251.850.446.2
MacJEPA54.053.752.247.319.558.156.054.352.150.0

Increasing exposure to one missing modality improves selected intermediate rates for that failure but reduces clean accuracy and robustness when the complementary modality disappears. The specialized models perform better under partial removal than clean input, indicating adaptation to their favored dropout distribution. Our balanced training provides the strongest clean and complete-loss endpoints on both tasks and avoids this cross-modality trade-off, supporting our choice not to favor either sensor. The experiment and protocol are detailed in Appendix Sec. B.6.2 and Table 12 of the revised manuscript.

W5, Q3.2: Off-grid and variable-duration test-time outages.

We appreciate the reviewer for motivating this stress test. We evaluate the paper's MacJEPA checkpoint under these variable-duration, window-off-grid test-time outages, without retraining or test-time adaptation. At each rate, the original nested sampler selects kk missing four-row windows, we preserve the resulting 4k4k-row budget but redistribute it into random off-grid segments: short 1--2 rows (0.53--1.07 s), long 6--12 rows (3.2--6.4 s), or mixed 1--18 rows (0.53--9.6 s). Reusing the random order across rates preserves nested missingness. The unchanged 0% and 100% endpoints are omitted.

Test-time outageEK-V 25EK-V 50EK-V 75ES-A 25ES-A 50ES-A 75
Training-matched fixed windows53.752.247.356.054.352.1
Short, variable53.652.148.857.754.352.4
Long, variable53.551.945.357.554.051.7
Mixed, variable53.751.946.457.454.251.9

Performance remains stable when the outage starts, and durations differ from training. Short outages match or exceed the fixed-window results, while longer sustained failures are harder but remain comparable. These additional results show that the fixed-window-trained MacJEPA checkpoint generalizes to out-of-distribution outage geometries, unaligned to the training time window grid, during test time without any adaptation. The revised manuscript reports this experiment and its protocol details in Appendix Sec. B.7 and Table 13.

We sincerely appreciate the reviewer's additional ablation and stress test suggestions, which have substantially strengthened our practical evaluation. We hope these results address the reviewer's concerns and welcome any further questions.

AuthorsOpenReview ↗

Looking Forward to Discussion

Dear Reviewer RWUL,

Thank you for your time and effort in reviewing our paper.

As the discussion period approaches its conclusion, we kindly invite you to consider our response and revised manuscript. We would be happy to discuss any remaining aspects of our paper that need further clarification.

Once again, we sincerely appreciate your valuable feedback, insights, and suggestions.

Reviewer_4KcNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper defines an egocentric modality-missingness problem in untrimmed, multi-event audio-visual observations. To address this problem, the authors introduce MacJEPA to align latent representations of multimodal content tokens with task-conditioned queries. The proposed method demonstrates competitive performance on the EPIC-KITCHENS-100 and EPIC-SOUNDS datasets.

Critical strengths

This paper reformulates egocentric modality missingness as an interval property of multi-event, untrimmed observations. Since this better reflects real-world sensor failures, the problem formulation is reasonable and meaningful. Following this formulation, the paper proposes a well-engineered pipeline and demonstrates improvements on Epic-Kitchens and Epic-Sounds.

Critical weaknesses

1. The biggest concern is the novelty of this work. While the authors claim to introduce a new JEPA architecture, the proposed method appears to use a JEPA-like architecture to construct a loss function rather than as a latent feature encoder. Therefore, introducing a JEPA-like loss seems more like borrowing an existing concept than proposing something truly new. The experiments also reveal that the largest performance gain comes from dropout, while the JEPA loss provides only moderate improvements, further raising questions about the proposed design.

2. The overall idea is also quite similar to Mean Teacher [1]. However, the authors didn’t compare their model with these related methods, nor do they compare it with loss functions operating on reconstructed multimodal signals rather than latent representations. Therefore, the effectiveness of the overall design is not adequately validated.

References:

[1] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results." Advances in neural information processing systems 30 (2017).

Questions for authors
  1. Regarding Weakness 1, please cite and compare with previous work and clearly state the differences beyond those arising from the tasks themselves.

  2. Regarding Weakness 2, please report comparison results using loss functions that operate on reconstructed signals.

  3. The paper also claims that "Limiting the predictor instead encourages stability in the task-conditioned readouts," but provides no experimental evidence to support this claim. Could the authors report results using more complex predictors, such as attention-based predictors or predictors with deeper layers?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 4KcN Part (1/2)

W1--W2/Q1: Novelty claim related to prior JEPAs and Mean Teacher.

We thank the reviewer for highlighting that our contribution boundary could be stated more clearly. We want to clarify that in Sec. 1. Introduction, we describe MacJEPA as repurposing JEPA from a latent self-supervised feature learning pretext into a supervised missing-modality robustness objective, and not as a new general JEPA architecture or EMA mechanism.

Instead, our core novelty lies in redefining egocentric modality missingness as modalities failing and recovering within untrimmed, multi-event egocentric videos, thus reflecting real-world deployment, and in repurposing JEPA to learn supervised robustness to these localized outages. Canonical JEPA models like V-JEPA 2 [1] and MJEPA[2] use masking for self-supervised feature learning. MacJEPA instead uses masking to represent time-localized sensor failures within untrimmed, multi-event videos. MacJEPA therefore demonstrates that JEPA-style masked-to-clean latent prediction can extend beyond transferable-feature pretraining and serve as an objective for task-supervised training for missing-modality robustness. As described in Sec. 3.4, our query JEPA loss extension directly constrains each interval-conditioned classifier readout, thus extending JEPA consistency from self-supervised feature representation to the task-supervised prediction representation under missing-modality constraints.

Dropout yielding the largest initial gain is expected since it introduces the failure condition from which the model must learn to recover. However, as discussed and shown in Sec. 4.2 and Table 1 respectively, dropout alone reduces clean accuracy, whereas content and query JEPA improve both clean and missing-modality recognition. JEPA thus addresses the non-trivial clean--robustness trade-off reported in prior missing-modality training work [4], demonstrating more than a marginal robustness increment.

MacJEPA shares EMA consistency with Mean Teacher [3], but the objectives serve different purposes. Mean Teacher aligns class predictions under stochastic perturbations for semi-supervised learning. MacJEPA instead maps structured multimodal missing modality inputs to clean latent content and query targets while jointly optimizing supervised recognition. The distinction therefore concerns model corruption semantics, prediction targets, loss support, and training purpose.

We hope this clarifies that our novelty claim is not a new JEPA architecture but a novel temporal-missingness formulation and protocol together with the repurposing of JEPA for supervised robustness objective, distinct from Mean Teacher’s semi-supervised prediction consistency. We will make this scope and comparison explicit in the revised manuscript.

References

[1] [Assran et al., ''V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning'', 2025.]

[2] [Teotia et al., ''MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning'', 2026.]

[3] [Tarvainen, Antti, and Valpola, ''Mean Teachers Are Better Role Models'', 2017.]

[4] [Ramazanova et al., ''Exploring Missing Modality in Multimodal Egocentric Datasets'', 2025.]

AuthorsOpenReview ↗

Response to Reviewer 4KcN Part (2/2)

Q2: Latent targets versus modality-specific reconstruction.

We thank the reviewer for suggesting this direct comparison. For this controlled experiment, we retain the online encoder, window-local dropout, masking, etc., but remove the EMA teacher, predictor, and JEPA losses. Two modality-specific decoders reconstruct the masked visual and audio input features from the shared multimodal encoder states using MSE, computed only at dropped positions, thus targeting the heterogeneous modalities in their respective input spaces. We evaluate all models using nested per-window dominant-modality missingness protocol; values are top-1 action accuracy for Epic-Kitchens (EK) and sound accuracy for Epic-Sound (ES) (%).

ObjectiveEK-V 0EK-V 25EK-V 50EK-V 75EK-V 100ES-A 0ES-A 25ES-A 50ES-A 75ES-A 100
Clean-only encoder + CE53.651.645.727.84.156.555.153.545.226.3
Window-local dropout + CE52.951.748.744.016.355.554.453.851.046.7
Dropout + masked feature reconstruction + CE52.952.651.345.718.255.955.653.950.048.9
MacJEPA54.053.752.247.319.558.156.054.352.150.0

Under missing modality constraints, reconstruction provides modest, inconsistent gains beyond dropout, while remaining below clean-only training under complete input and underperforming MacJEPA at every missing rate. It therefore does not resolve the clean--robustness trade-off. These results support aligning missing and clean representations in a shared latent space instead of requiring a shared encoder to reconstruct heterogeneous modality-specific features. The revised manuscript reports this experiment in Appendix Sec. B.1 and Table 5.

Q3: Effect of predictor capacity.

We thank the reviewer for requesting direct evidence for this design choice. We keep the complete MacJEPA protocol fixed and replace only its lightweight MLP predictor with either a deeper four-linear-layer predictor with 384-D hidden layers or a four-block, 384-D attention predictor. We evaluate all models using nested per-window dominant-modality missingness protocol; values are top-1 action and sound accuracy (%).

PredictorEK-V 0EK-V 25EK-V 50EK-V 75EK-V 100ES-A 0ES-A 25ES-A 50ES-A 75ES-A 100
Four-block attention53.052.651.145.416.556.855.953.851.748.5
Deep MLP bottleneck53.453.051.846.817.957.156.054.751.949.2
Lightweight MLP (MacJEPA)54.053.752.247.319.558.156.054.352.150.0

The lightweight predictor leads or ties across nearly all conditions, with the deeper MLP providing only an isolated intermediate-rate improvement on Epic-Sound; the attention predictor is consistently weaker. This supports our claim in Sec. 3.4 and Appendix Sec. A.2 that limiting the train-only predictor places greater pressure on the online encoder and query readouts to preserve outage-invariant evidence, rather than allowing the predictor to absorb cross-token fusion, thus encouraging stability in the task-conditioned queries. The experiment and protocol are detailed in Appendix Sec. B.4 and Table 9 of the revised manuscript.

We sincerely appreciate the reviewer for motivating these controlled experiments, which have substantially strengthened our evaluation. We hope these additional results address the reviewer’s concerns, and we welcome any further questions.

AuthorsOpenReview ↗

Looking Forward to Discussion

Dear Reviewer 4KcN,

Thank you for your time and effort in reviewing our paper.

With the discussion period nearing its conclusion, we kindly invite you to consider our response and revised manuscript. We would be happy to discuss any remaining aspects of our paper that need further clarification.

Once again, we sincerely appreciate your valuable feedback, insights, and suggestions.

Reviewer_1G8YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MacJEPA addresses audio-visual recognition under temporally localized modality outages in untrimmed egocentric videos. It combines window-local modality dropout with supervised alignment of corrupted content and interval-query representations to clean EMA-teacher targets. Evaluations on EPIC-KITCHENS-100 and EPIC-SOUNDS show improved missing-modality robustness using a single checkpoint without test-time adaptation.

Critical strengths
  • The paper addresses a practically relevant problem: maintaining audio-visual recognition under temporally varying modality availability in untrimmed videos.
  • The unified framework combines robustness learning with task supervision, allowing a single model to handle complete and incomplete inputs without test-time adaptation.
  • The reported results demonstrate useful robustness gains while maintaining competitive clean-input performance, with component ablations supporting the proposed design.
Critical weaknesses
  • The main components of the proposed framework have clear precedents: temporal interval querying in TIM (Chalk et al., 2024), missing-modality training in MMT (Ramazanova et al., 2025), and masked-to-clean representation prediction in data2vec and I-JEPA (Baevski et al., 2022; Assran et al., 2023). The conceptual advance of extending these ideas to window-local outages and task-conditioned queries needs stronger justification.
  • Limited coverage of relevant baselines. Missing-modality comparisons are restricted to MMT and MiDl, while the proposed local-outage setting is evaluated mainly through component ablations. Relevant approaches such as KARMMA, discussed in the related work, are not evaluated. Broader comparisons under the proposed setting would strengthen the evidence for the claimed robustness advantage.
Questions for authors
  • What distinguishes the proposed content–query JEPA objective from standard teacher–student representation distillation? Can a comparison using the same architecture and modality-dropout strategy demonstrate its additional benefit?
  • KARMMA is discussed in the related work but excluded from the experiments. How does MacJEPA compare with KARMMA under the proposed local-outage setting, or what prevents a meaningful comparison?
  • EPIC-KITCHENS-100 and EPIC-SOUNDS share the same underlying videos. Do the robustness gains persist on an independent dataset with different environments and activities?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 1G8Y Part (1/2)

W1: Relation to prior work.

We thank the reviewer for prompting us to clarify the justification for this conceptual advance. The individual ingredients have conceptual precedents. TIM introduces interval queries over untrimmed audio-visual context but assumes continuous modality availability. MMT replaces a whole missing stream within an action-centered clip and does not represent local disappearance and recovery in untrimmed observations. data2vec applies masked self-supervision separately to speech, vision, or text, while I-JEPA learns self-supervised image representations; both learn transferable unimodal features using masked-to-clean representation, but neither addresses supervised robustness to missing multimodal sensors. In real-world egocentric deployment, one local sensor outage removes different degrees of evidence from overlapping events, requiring the model to identify which stream is absent when and preserve each interval-conditioned decision from its surviving evidence. To represent such missingness, we introduce time-indexed multimodal signal absence into TIM's multi-event untrimmed setting. Typed, timed masks encode the local sensor outages, thereby replacing MMT's sample-global absence. Content JEPA stabilizes the globally contextualized evidence, while query JEPA aligns the task-specific representation used for classification under missing modality constraints. Our content-query JEPA therefore extends I-JEPA and data2vec's masked-to-clean latent prediction from a transferable-feature pretraining objective to task-supervised training under missing-modality constraints, enabling MacJEPA to preserve recognition accuracy under local sensor outages. MacJEPA thus unifies and advances these precedents around a previously untested egocentric missingness problem motivated by real-world sensor failures.

W2: Baseline coverage.

We agree that KARMMA should be included, and we add its results in part 2 of our response in the following comment. MMT and MiDl assign each sample one global modality state, while KARMMA also removes entire modalities and varies their dropout probability across samples. Their published missing rates measure how many samples lose a stream, whereas the missing rates in our nested per-window temporal missingness protocol defined in Sec 4.1 of our paper measure a stream's local absence within each untrimmed observation. Applying these methods to the baselines would require per-window reformulation and retraining, yielding new variants rather than evaluations of the published methods; we therefore compare them under their matched observation-level protocol. We hope this clarification has successfully addressed the reviewer's baseline coverage concerns.

AuthorsOpenReview ↗

Response to Reviewer 1G8Y Part (2/2)

Q1: MacJEPA vs. Teacher–Student Representation Distillation.

We appreciate the reviewer for suggesting this comparison. Standard representation distillation first trains and freezes a clean teacher, then trains a separate student. MacJEPA instead jointly trains the online encoder and evolving EMA teacher in one stage. For a controlled comparison, we trained our encoder for recognition on unmasked inputs (H1 ablation from Table 1), froze it as the teacher, replaced MacJEPA’s EMA teacher and content–query JEPA with standard representation distillation, and retained the window-local modality dropout and the encoder architecture as the student. Top-1 action accuracy (%) following our nested per-window temporal missingness protocol under dominant modality loss, i.e., video for Epic-Kitchens (EK) and audio for Epic-Sounds (ES), is reported.

MethodEK-V 0EK-V 25EK-V 50EK-V 75EK-V 100ES-A 0ES-A 25ES-A 50ES-A 75ES-A 100
Frozen clean teacher53.651.645.727.84.156.555.153.545.226.3
Two-stage teacher–student distillation53.352.951.846.918.956.355.554.151.347.6
MacJEPA54.053.752.247.319.558.156.054.352.150.0

The two-stage student improves robustness, but its clean accuracy falls below the clean teacher on both EK and ES by 0.3 and 0.2 points, respectively. MacJEPA exceeds the distilled student at every missing rate, including complete dominant-stream loss by 0.6 and 2.4 points, while also improving clean accuracy by 0.7 and 1.8 points. Thus, this controlled comparison confirms that preserving clean recognition while gaining missing-modality robustness is non-trivial, and content–query JEPA provides benefits beyond standard two-stage representation distillation. The revised manuscript reports this experiment in Appendix Sec. B.1 and Table 5.

Q2: Comparison with KARMMA.

We thank the reviewer for identifying this omission. KARMMA recognizes only visual actions from trimmed, single action-centered clips, with each modality removed for the entire clip. Its published protocol therefore provides modality-subset endpoints but no temporally local outage setting. Evaluating local outages would require window-indexed corruption and retraining, producing a new KARMMA variant. We instead compare the published Epic-Kitchens-100 action endpoints for shared modality subsets below. KARMMA additionally uses optical flow (F), which MacJEPA does not require.

ModelV+AVAV+F+A
KARMMA40.838.88.1743.0
MacJEPA54.052.819.5—

MacJEPA exceeds KARMMA by 13.2, 14.0, and 11.3 points with V+A, V only, and A only, respectively. Simultaneously, the same MacJEPA checkpoint supports local outages and jointly recognizes visual actions and acoustic events in multi-event untrimmed observations. This supplies the requested KARMMA comparison while clarifying why its published formulation has no directly comparable local-outage sweep. This experiment is detailed in Appendix Sec. B.2 of the revised manuscript.

Q3: Generalization to an Independent Dataset.

We extended MacJEPA to Ego4D, which covers diverse daily activities and in-the-wild environments beyond EPIC’s kitchen setting. Since Ego4D has no official action-recognition benchmark, we utilize the Ego4D-AR dataset used by MMT and construct untrimmed observations from its source videos for training and evaluation. Following prior work, we report top-1 noun accuracy since the derived verb classes are highly imbalanced. Ego4D-AR also naturally lacks audio in 27% of its test clips, so the protocol-matched comparison uses 50%, 75%, and 100% missing audio, following the observation-level missingness protocol defined in Sec 4.1.

Method5075100
MiDl27.123.416.6
MMT35.233.532.0
MacJEPA48.248.148.0

MacJEPA exceeds the strongest baseline by 13.0, 14.6, and 16.0 points at 50%, 75%, and 100% observation-level missing audio, respectively. We additionally evaluate top-1 noun accuracy under dominant-modality (video) removal using our nested per-window temporal missingness protocol:

ModelV drop 0V drop 25V drop 50V drop 75V drop 100
MacJEPA48.247.846.541.515.5

These results show that MacJEPA’s robustness persists on independent Ego4D videos spanning broader environments and activities. We have added this experiment and its protocol details in Appendix Sec. B.3 of the revised manuscript.

We sincerely hope these additional experiments answer the reviewer’s questions. We thank the reviewer for these constructive suggestions, which have substantially strengthened our evaluation; and we are happy to address any further questions.

AuthorsOpenReview ↗

Looking Forward to Discussion

Dear Reviewer 1G8Y,

Thank you for your time and effort in reviewing our paper.

As the discussion period approaches its conclusion, we kindly invite you to consider our response and revised manuscript. We would be happy to discuss any remaining aspects of our paper that need further clarification.

Once again, we sincerely appreciate your valuable feedback, insights, and suggestions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Revision Summary

We thank all reviewers for their time and effort in evaluating our paper. Their questions helped us articulate our contribution and novelty more precisely, while their requested experiments and ablations substantially strengthened our evaluation.

All three reviewers recognize the novelty and practical value of our problem formulation, which models local sensor outages in untrimmed egocentric videos. MacJEPA advances egocentric modality missingness from an observation-level binary state on trimmed single-event clips to a temporal property of untrimmed, multi-event observations. In this setting, sensors may disappear and recover locally, affecting overlapping events differently. The reviewers unanimously acknowledge our problem formulation as a meaningful advance that better reflects real-world sensor failures. The reviewers further recognize that our proposed novel framework with typed, timed masking and jointly supervised repurposed JEPA objective allows one checkpoint to remain competitive on complete inputs, withstand localized missing inputs, and jointly recognize visual actions and acoustic events without test-time adaptation.

We have successfully addressed each reviewer’s remaining questions as follows.

Reviewer 1G8Y. The reviewer requested stronger justification for our conceptual advancement over precursory ideas like TIM, MMT, data2vec, and I-JEPA, together with broader comparisons and independent-dataset evidence. We clarified that MacJEPA advances these ideas for a previously untested missingness problem formulation by integrating interval-conditioned recognition with typed, time-localized multimodal outages and extending masked-to-clean alignment from self-supervised feature learning to a task-supervised robustness objective. Furthermore, we outline how existing published missing-modality baselines use whole-stream clip-level removal and cannot be directly evaluated under our local outage setting without retraining and redesign. We added the requested controlled teacher–student distillation comparison, KARMMA endpoint comparison, and Ego4D evaluation. Together, the results validate the benefits of content–query JEPA beyond conventional distillation, establish substantial gains over KARMMA at shared modality endpoints, and demonstrate that MacJEPA’s robustness generalizes beyond the shared EPIC videos.

Reviewer 4KcN. The reviewer recognizes that our meaningful reformulation of egocentric missingness better reflects real-world sensor failures. To alleviate the raised novelty concern, we cited our manuscript and clarified that the paper does not claim to introduce a new general JEPA or EMA mechanism. We outlined that our core novelty is the egocentric temporal modality missingness formulation, together with the repurposing of JEPA's latent masked-to-clean alignment from self-supervised feature learning to task-supervised missing-modality robustness training. Dropout provides necessary outage exposure; content-query JEPA restores clean accuracy and improves missing-input recognition. We distinguished MacJEPA’s supervised content–query alignment under localized modality outages from the semi-supervised prediction consistency of Mean Teacher. The requested modality-specific reconstruction experiment and predictor ablations were added. Their results support latent alignment over signal reconstruction and validate our lightweight predictor design.

Reviewer RWUL. The reviewer raised deployment-oriented questions about unrecoverable supervision, joint failures, asymmetric outage frequencies, raw failure signatures, token deletion, and unseen outage timing. We addressed each concern through targeted clarification and added the requested controlled experiments. We clarified how, under local sensor outages, our typed, timed substitution preserves recoverable temporal and cross-modal evidence while also representing any detected failure type. We evaluated joint-outage training, asymmetric training, and off-grid variable-duration test-time outages. These results support our balanced sampler and show that the fixed MacJEPA checkpoint generalizes to unseen outage geometries during evaluation without any test-time adaptation.

The revised manuscript now includes all requested experiments and their protocols:

  • Teacher–student distillation and masked-feature reconstruction: Appendix Sec. B.1, Table 5.
  • KARMMA endpoint comparison: Appendix Sec. B.2, Table 6.
  • Ego4D evaluation: Appendix Sec. B.3, Tables 7–8.
  • Deep and attention-based predictor ablations: Appendix Sec. B.4, Table 9.
  • Joint-outage and asymmetric-dropout training: Appendix Secs. B.6.1–B.6.2, Tables 11–12.
  • Unseen, variable-duration test-time outages: Appendix Sec. B.7, Table 13.

We again thank the reviewers for their valuable feedback and insights. We sincerely hope our clarifications, additional experiments, and revisions address every concern and make the paper’s contribution and supporting evidence clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.