acceptodds
Under review as a conference paper at ICLR 2027

LOOM: Low-latency Object Onset Modeling for Egocentric Audio-Visual Anticipation

Abstract

For wearable assistive systems designed for people with visual impairments, timely alerting users to surrounding hazards, critical events, and key objects is paramount to ensuring safe navigation and independent mobility. We formulate egocentric audio-visual near-body intrusion anticipation: predicting whether an object will newly enter a region extending 0.40m beyond the body within three seconds, using historical monocular RGB and simulated two-channel audio. We introduce LOOM, a lightweight framework combining visual motion and expansion statistics, a learned acoustic sequence model, development-selected probability fusion, and stateful warning generation. We construct ReactionGap, comprising 150 simulated episodes across ten motion families, with fixed wearer-centered viewpoints, synchronized audio-visual observations, and geometric trajectories. Its entering and non-entering scenes provide 3,900 labeled queries for training and template-disjoint evaluation of future-entry prediction and sequential warnings. Across three seeds, LOOM achieves pooled average precision of 0.5438, compared with 0.5234 for vision and 0.4776 for audio. Median computation takes 8.8ms, compared with 54.1–72.4ms for the evaluated temporal and fusion adaptations. Warning replay yields one-second recall of 0.4178 at 0.7289 non-useful alarms per episode; vision retains stronger time-matched discrimination. These results highlight a critical discrepancy between pure model prediction and actionable warning performance, establishing ReactionGap as a standardized benchmark to advance robust, real-time wearable assistance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-visual learning, Visual impairment assistance, Egocentric perception, Near-body intrusion anticipation, Low-latency multimodal fusion, Synthetic datasets

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2S7pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LOOM, a lightweight framework for egocentric audio-visual near-body intrusion anticipation. The method combines visual motion statistics, a CNN-GRU audio model, probability fusion, and a stateful warning mechanism. The authors also introduce ReactionGap, a simulated dataset containing 150 episodes with 3,900 prediction queries. Experiments show that LOOM achieves competitive prediction performance with low inference latency.

Critical strengths
  1. The paper addresses a meaningful problem in wearable assistance. It focuses on anticipating future intrusion and generating timely warnings. The evaluation of warning lead time and false alarm burden is particularly useful.
  2. The paper is generally well organized, and the motivation and overall pipeline are clearly presented. The experiments include several baselines, ablation studies, and statistical analyses. The reported inference latency of 8.8 ms also demonstrates the efficiency of the proposed framework.
Critical weaknesses
  1. The contribution of audio-visual fusion is not sufficiently demonstrated. In Table 1, LOOM only slightly improves pooled AP over the vision-only baseline (0.5438 vs. 0.5234), and the confidence interval reported in Sec. 4.2 includes zero. Moreover, the vision-only model achieves higher time-matched AUC (0.7793 vs. 0.7257, Table 1) and conditional AUC (0.7526 vs. 0.6921, Table 13). Although the authors acknowledge these limitations, it remains unclear under what conditions audio provides useful complementary information. I would like to see more direct evidence supporting the contribution of the audio branch.
  2. The dataset is relatively simplified. As shown in Appendix Fig. 2, the scenes contain low-detail object models and largely textureless backgrounds, which may make approaching objects easier to distinguish than in real-world scenarios. Besides, all episodes use stationary cameras, while the proposed visual features rely heavily on frame differences and expansion statistics. It is unclear whether the model can handle ego-motion, complex backgrounds, or multiple moving objects. I am concerned that the reported performance may partly depend on these simplified simulation conditions.
  3. There may be some dataset-specific biases. As discussed in the appendix, the two subsets use different rendering and audio generation pipelines and have different positive sample ratios. Although the authors provide conditional-AUC analysis, it is still unclear how much these differences affect the reported performance.
  4. Some experimental comparisons could be strengthened. The paper lacks direct comparisons with simple task-specific methods based on visual looming or time-to-contact estimation. Besides, the warning evaluation only reports selected operating points, making it difficult to compare different methods under the same false alarm burden.
Questions for authors
  1. Since all episodes use stationary cameras, how well would LOOM handle ego-motion such as walking or head movements? Could the authors provide additional analysis or controlled experiments to clarify the robustness of the proposed visual features under camera motion?
  2. Under what conditions does audio actually improve intrusion prediction? Could the authors provide some quantitative analysis or failure cases comparing LOOM with the vision-only model?
  3. Could the authors further explain how the differences between the two data generation pipelines affect the results, particularly the audio-visual fusion performance?
  4. Could the authors provide warning recall versus alarm burden curves for LOOM and the vision-only baseline? Does LOOM still perform better under the same alarm burden?
Reviewer_hxPsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the system predicts whether an object will enter the 0.40-meter region around the body within the next 3 seconds, based on monocular RGB video and two-channel audio from the past 2.5 seconds. The proposed method, LOOM, combines statistical features representing changes and magnification in the video, acoustic prediction using a CNN–GRU, probability-weighted fusion, and state-based alarm processing. Predictive performance, lead time of alerts, and the number of false alarms are evaluated using ReactionGap, a dataset consisting of 150 simulation episodes. The LOOM method, which combines audio and video, slightly outperformed the video-only method in terms of the AP calculated across all test examples (0.5438 vs. 0.5234). On the other hand, when using the TM-AUC metric—which standardizes the elapsed time from the start of the video for comparison—the video-only method achieved a higher score (0.7793 vs. 0.7257).

Critical strengths
  • The prediction target is clearly defined and linked to time-series evaluation. The system targets the prediction of new geometric intrusions and is constrained by the requirement that input data be available prior to the prediction time. This goes beyond simple proximity object detection and becomes a challenge focused on evaluating early warnings.
  • The usefulness of alerts is evaluated independently. The design, which distinguishes between false alarms, duplicate alarms, and delayed alarms and replays alerts while accounting for processing time, identifies issues that cannot be detected by AP alone on a per-query basis.
Critical weaknesses
  • The AP difference compared to vision alone was 0.0204, with a 95% confidence interval of [−0.0623, 0.0662], which includes zero. The AUC, with time points aligned, was 0.7793 for vision alone compared to 0.7257 for LOOM, and vision alone outperformed LOOM in all nine fold–seed comparisons. Based on the results in Table 1 (p. 6), it does not appear possible to conclusively state that “fusion enhances discriminative ability.”
  • The rendering and audio generation conditions differ between subsets A and B, resulting in positive rates of 40.48% and 22.37%, respectively. Furthermore, since there are no positive examples from subset B in the validation sets for any of the folds, the selection of fusion weights and alert policies may be biased toward certain generation conditions.
  • The dataset consists of 150 episodes, with a fixed camera and simplified graphics. Visual features do not include camera motion compensation. It remains unconfirmed whether a wearable system that involves walking and head movements can distinguish between changes in the overall background and changes in approaching objects.
Questions for authors
  • Will LOOM's advantages be maintained even if the weights and alert policy are selected using a validation set that contains multiple templates and includes a sufficient number of positive and negative examples from both subsets?
  • What are the conditions under which audio alerts correctly provided relief, and conversely, under which conditions the accuracy of visual-only alerts was compromised? Even when the number of false alarms is kept the same, does fusion improve alarm performance?
Reviewer_zWAgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies egocentric audio-visual near-body intrusion anticipation: predicting whether an object will newly enter a region extending 0.40 m beyond the body within the next three seconds, using 2.5 seconds of historical monocular RGB and two-channel audio. The proposed LOOM system combines handcrafted visual motion and expansion statistics, a CNN–GRU acoustic predictor, development-selected probability fusion, and a stateful warning policy.

The authors also introduce ReactionGap, comprising 150 simulated episodes, 3,900 overlapping queries, and 75 entry events, evaluated using template-disjoint folds and three training seeds. LOOM achieves pooled AP of 0.5438 and median computation time of 8.8 ms on an RTX 4090. Its one-second warning recall is 0.4178 at 0.7289 non-useful alarms per episode. However, the visual-only branch achieves stronger time-matched and subset-conditioned discrimination, and the improvements over vision in pooled AP and warning recall have confidence intervals that include zero.

Critical strengths

1.A useful task formulation connected to event-level evaluation. The paper distinguishes future boundary entry from current occupancy and evaluates delivered warnings using lead time, false alarms, duplicates, and late alarms. Including computation delay in warning replay makes the evaluation more informative than query-level classification alone. This connection between prediction and warning utility is the strongest contribution.

2.Transparent analysis of uncertainty and dataset shortcuts. The paper reports template-group bootstrap intervals, per-fold and per-seed results, time-matched evaluation, and supplementary subset-conditioned diagnostics. It explicitly acknowledges that the visual-only model performs better under conditional discrimination and that generation-subset membership carries label information. These analyses make the empirical findings substantially easier to assess.

Critical weaknesses

1.The benefit of adding audio is unclear. LOOM slightly improves pooled AP over the visual-only model, but the confidence interval includes zero. The visual-only model performs better on time-matched and subset-conditioned AUC, while warning recall is very similar. These results do not show a clear advantage for fusion, which also increases computation time.

2.The evaluation is limited to 150 simulated episodes with stationary cameras. The two data subsets differ in rendering, audio generation, and label distribution. Although the paper examines these differences, their effects remain difficult to separate. Each development set also uses only one template, with all positive queries coming from Subset A. This makes it hard to judge how well model selection and warning policies generalize.

Questions for authors

1.How sensitive are the results to the development split? Does using multiple development templates, with positive examples from both subsets, change the selected fusion weights or warning performance?

2.In which cases does audio help detect events that the visual-only model misses? How often does fusion recover a missed warning, and how often does it lose a warning that vision alone would provide?

3.How do LOOM and the visual-only model compare across the three development alarm budgets (0.25, 0.5, and 1)? Is the small recall advantage consistent across these settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.