- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work addresses the challenge of missing modality during inference for long untrimmed videos. Previous work only considered the missing modality challenge for full duration of trimmed videos. The core motivation is that sensors are likely to experience outage during deployment but that could be for a short period of time during the untrimmed video. Further, within the same video, you could experience outage of different modalities at different intervals rather than one modality for the whole duration of the clip. The authors propose a variation of JEPA that uses window local modality dropout that assigns a binary value for masking the features of a specific modality for each window in the non-overlapping windows partitioned from the original observation. Further, masked online representations of both content tokens and task queries are aligned with clean targets from an EMA teacher, trained jointly with classification. They test their model and compare it to baselines on epic-kitchens and epic-sounds
Critical strengths
- The motivation behind the problem is sound and relaxes several assumptions that might not hold in real-world deployment.
- Well written and easy to follow
- The choice of typed and timed absence tokens helps minimise the design while being able to mask either modality within the same architecture.
- One model for both Epic-Kitchen and epic sounds that does not require adaption during test-time. This gives it a practical advantage during deployment
- Competitive performance when compared with model’s trained without missing modalities as shown on table 2
Critical weaknesses
- Joint outage is excluded from the design where in the real world there could be an outage with both the visual and audio sensors at the same time.
- Supervision on unrecoverable targets. Epic-Kitchens and Epic-Sounds annotations are independent and many acoustic events have no visual counterpart. Figure 1 shows Scrub/Wipe co-occurring with the unrelated Close Kettle. When window-local dropout removes the audio under such an event, the sound query is still trained with cross-entropy on its clean label. Query JEPA also pushes its state toward the clean teacher’s. The model is then asked to predict information that is absent from the input.
- Unexplored allocation between modalities. Training always splits the outage probability equally (π/2 each), and Appendix B.1 varies only the total π. For example, privacy muting could be more likely than camera failure
- Substituting features for an interval of a modality during the typed timed absence process during training is only an approximation of what could happen during deployment. During deployment, there could be no data at all e.g. all silence in audio or all black or frozen frames.
- Train and test outages share the same artificial setup. Real outages have arbitrary durations and start times. Long failures span window boundaries in ways the training sampler never produces.
Questions for authors
- What fraction of training sound and action queries have all their supporting evidence removed by window-local dropout, e.g. an audio-only event with no overlapping visual action? Does excluding these queries from cross-entropy and query JEPA, or down-weighting them, change clean or robust accuracy?
- What happens if both modalities are missing? Equivalent to both sensors failing during deployment? It is understandable how you justified not dropping both modalities in L215-217 but that does not rule out that it could happen during deployment for which you are trying to make the model more robust. How does adding a small joint dropout probability during training change the results? Have you tried asymmetric splits (e.g. 0.7π / 0.3π in either direction)? What impact would this have on results and conclusions in terms of robustness to missing one modality over the other?
- How does substitution of features compare with deleting missing tokens via attention mask? How does the model perform on outages that differ from training e.g. random start times not aligned to the window grid, variable durations?
5 comments
Response to Reviewer RWUL Part (2/2)
W1, Q2.2: Joint-outage training.
We appreciate the reviewer for asking how including simultaneous sensor outages during training affects robustness. We evaluate the requested training ablation against our MacJEPA checkpoint while keeping the architecture, curriculum, and optimization fixed. Let , , , and denote the per-window probabilities of dropping video only, audio only, both modalities, or neither, respectively. Standard MacJEPA uses , with . The joint-outage variant adds the fourth state and uses . Both models are evaluated under our nested per-window dominant modality missingness protocol; values are top-1 action accuracy for Epic-Kitchens (EK) and sound accuracy for Epic-Sound (ES) (%).
| Training sampler | EK-V 0 | EK-V 25 | EK-V 50 | EK-V 75 | EK-V 100 | ES-A 0 | ES-A 25 | ES-A 50 | ES-A 75 | ES-A 100 |
|---|---|---|---|---|---|---|---|---|---|---|
| With joint-outage state | 53.0 | 52.4 | 51.5 | 46.3 | 17.5 | 56.6 | 55.7 | 53.9 | 51.8 | 49.3 |
| MacJEPA | 54.0 | 53.7 | 52.2 | 47.3 | 19.5 | 58.1 | 56.0 | 54.3 | 52.1 | 50.0 |
Joint-outage training lowers clean and missing-input accuracy throughout. At fixed , it reduces exposure to single-modality outages and adds windows with no sensor evidence, weakening the clean–robustness balance discussed in Sec. 4.2. These results support our two-state dropout design for the dominant sensor failure setting studied in the paper. We include the protocol details and results in Appendix Sec. B.6.1 and Table 11 of the revised manuscript.
W3, Q2.3: Training-time asymmetric modality dropout.
We thank the reviewer for prompting this test. We keep and all other settings fixed, and compare MacJEPA's with and . Evaluation uses nested per-window dominant-modality missingness protocol.
| Training dropout allocation | EK-V 0 | EK-V 25 | EK-V 50 | EK-V 75 | EK-V 100 | ES-A 0 | ES-A 25 | ES-A 50 | ES-A 75 | ES-A 100 |
|---|---|---|---|---|---|---|---|---|---|---|
| Audio-drop-heavy: | 53.8 | 52.3 | 50.1 | 44.4 | 17.8 | 55.8 | 56.2 | 54.8 | 51.0 | 49.1 |
| Video-drop-heavy: | 52.8 | 53.5 | 53.1 | 48.9 | 19.2 | 57.7 | 53.2 | 51.8 | 50.4 | 46.2 |
| MacJEPA | 54.0 | 53.7 | 52.2 | 47.3 | 19.5 | 58.1 | 56.0 | 54.3 | 52.1 | 50.0 |
Increasing exposure to one missing modality improves selected intermediate rates for that failure but reduces clean accuracy and robustness when the complementary modality disappears. The specialized models perform better under partial removal than clean input, indicating adaptation to their favored dropout distribution. Our balanced training provides the strongest clean and complete-loss endpoints on both tasks and avoids this cross-modality trade-off, supporting our choice not to favor either sensor. The experiment and protocol are detailed in Appendix Sec. B.6.2 and Table 12 of the revised manuscript.
W5, Q3.2: Off-grid and variable-duration test-time outages.
We appreciate the reviewer for motivating this stress test. We evaluate the paper's MacJEPA checkpoint under these variable-duration, window-off-grid test-time outages, without retraining or test-time adaptation. At each rate, the original nested sampler selects missing four-row windows, we preserve the resulting -row budget but redistribute it into random off-grid segments: short 1--2 rows (0.53--1.07 s), long 6--12 rows (3.2--6.4 s), or mixed 1--18 rows (0.53--9.6 s). Reusing the random order across rates preserves nested missingness. The unchanged 0% and 100% endpoints are omitted.
| Test-time outage | EK-V 25 | EK-V 50 | EK-V 75 | ES-A 25 | ES-A 50 | ES-A 75 |
|---|---|---|---|---|---|---|
| Training-matched fixed windows | 53.7 | 52.2 | 47.3 | 56.0 | 54.3 | 52.1 |
| Short, variable | 53.6 | 52.1 | 48.8 | 57.7 | 54.3 | 52.4 |
| Long, variable | 53.5 | 51.9 | 45.3 | 57.5 | 54.0 | 51.7 |
| Mixed, variable | 53.7 | 51.9 | 46.4 | 57.4 | 54.2 | 51.9 |
Performance remains stable when the outage starts, and durations differ from training. Short outages match or exceed the fixed-window results, while longer sustained failures are harder but remain comparable. These additional results show that the fixed-window-trained MacJEPA checkpoint generalizes to out-of-distribution outage geometries, unaligned to the training time window grid, during test time without any adaptation. The revised manuscript reports this experiment and its protocol details in Appendix Sec. B.7 and Table 13.
We sincerely appreciate the reviewer's additional ablation and stress test suggestions, which have substantially strengthened our practical evaluation. We hope these results address the reviewer's concerns and welcome any further questions.
Looking Forward to Discussion
Dear Reviewer RWUL,
Thank you for your time and effort in reviewing our paper.
As the discussion period approaches its conclusion, we kindly invite you to consider our response and revised manuscript. We would be happy to discuss any remaining aspects of our paper that need further clarification.
Once again, we sincerely appreciate your valuable feedback, insights, and suggestions.
Response to Reviewer RWUL Part (1/2)
W2, Q1: “Supervision on unrecoverable targets.”
We appreciate the reviewer for raising this distinction. In Figure 1, Close Kettle is not a semantic counterpart to Scrub/Wipe, and a local audio outage may remove its decisive cue. However, the sound query attends to the full untrimmed observation and may use neighboring audio, visual motion, and broader context. Independent annotations imply unpaired labels and not necessarily absent evidence. If no informative evidence survives, cross-entropy and query JEPA cannot recover it. Instead, they learn the label distribution and clean readout conditioned on the remaining context. The fraction of truly unsupported queries is not identifiable because the annotations specify events, not their supporting evidence. Treating no overlapping visual label as unrecoverable would therefore discard potentially useful temporal or visual cues. We consequently do not exclude or down-weight queries using this proxy and will clarify this distinction in the revision.
Q2.1: “What happens if both modalities are missing?”
We thank the reviewer for raising this important boundary case. MacJEPA remains well-defined under a deployment-time local joint outage: both streams are replaced by their typed, time-stamped absence tokens, while each query can draw on surviving evidence elsewhere in the observation. If both modalities are unavailable for the full observation, no event-specific sensor evidence remains, and predictions are necessarily limited to learned class priors. We will clarify this behavior and recovery limit explicitly in the revised manuscript.
W4: “During deployment, there could be no data at all.”
We thank the reviewer for prompting us to clarify the deployment interface. As defined in Sec. 3.3 and Appendix A.2, MacJEPA does not require black frames or silence to resemble its learned mask vector. Once an interval is marked unavailable by sensor-health metadata or an upstream fault detector, its missing feature is substituted with the learned modality-specific absence vector while retaining its timestamp. The same mechanism therefore represents absent, black, silent, or frozen input after detection, independent of the raw failure signature. This typed, timed interface allows either stream to be handled by the same model without test-time adaptation. We will state this more explicitly in our revised manuscript.
Q3.1: “How does substitution compare with deleting missing tokens?”
We appreciate this question because it highlights the purpose of typed, timed absence. Canonical JEPAs remove target tokens from the context encoder and train a predictor to infer their latent representations. Applying that design here would shift outage handling toward the train-only predictor and prevent the deployed encoder and queries from explicitly identifying the missing modality and interval. MacJEPA instead substitutes only the unavailable content component with learned mask tokens while preserving the token position, time encoding, and modality identity. Content and query attention can then recognize the outage and reroute toward surviving temporal or cross-modal evidence; the predictor is discarded after training. Our choice of substitution therefore follows the objective of teaching the deployed encoder rather than the predictor to recognize and handle missing input.
We hope these explanations address the reviewer’s concerns and questions quoted above and clarify the relevant design choices. We will make these distinctions explicit in the revised manuscript.