acceptodds
Under review as a conference paper at ICLR 2027

EgoMM: Towards Multimodal Egocentric Intelligence

Abstract

Wearables capture synchronized video, audio, and inertial signals, yet existing egocentric understanding systems remain largely centered on vision. When cameras are occluded, poorly lit, or directed away from an event, complementary evidence can remain in sound and motion. Learning from this evidence requires models that integrate wearable signals and evaluations that distinguish their contributions. We present EgoMM, connecting benchmarking, multimodal learning, and long-horizon memory around this goal. We establish EgoMMU, to our knowledge the first benchmark to assess egocentric language models across visual, acoustic, and inertial tasks with explicit modality requirements. Its five task families form thirteen task-modality combinations, supported by construction-time checks and controlled ablations. We then develop EgoMind, which connects all three modalities to a vision-language backbone through staged sensor pretraining and modality-matched instruction tuning. At each of four model sizes, a single checkpoint accepts any non-empty subset of the streams. Pretraining improves all five task families at every size, alongside competitive performance across external egocentric benchmarks. Finally, we build EgoBuddy to organize multimodal observations into hierarchical text memory for long-horizon egocentric understanding. Video-audio evaluations examine retrieval and question answering under progressively smaller visual frame budgets. Together, these components advance egocentric understanding toward a richer account of what is "seen" and "heard", and how the wearers "move".

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Vision, Multimodal Large Language Models, Video Question Answering, Narration, Temporal Grounding, Multi-Sensor Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bfx5OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents EgoMM for egocentric understanding from video, audio and head-mounted IMU. It contributes (1) EgoMMU, a benchmark of 3,762 LLM-authored items on 30-s clips, with five task families forming 13 task–modality columns; (2) EgoMind, a Qwen3.5-based model (1B–9B) with audio and IMU pathways, rule-based IMU "tag text", and LoRA instruction tuning on 1.75M generated items; and (3) EgoBuddy, a hierarchical text memory evaluated on one EgoLife-QA participant. Reported results: EgoMind-9B outperforms Gemini 3.1 Pro on IMU MCQ (66.2 vs. 45.4), sensor pretraining improves all task families at every size (Table 24), and EgoMind performs competitively on external egocentric benchmarks.

Critical strengths
  1. Modality-specific evaluation.

Splitting tasks by the modality that should supply the evidence, and testing this by withholding modalities (Table 23, E.2), is more informative than aggregate audio-visual scores. The finding that vision-only models match audio-visual models on sound-source attribution (Table 21) is a useful caution.

Why it matters: this is the most reusable part of the paper.

  1. Scale of the resources.

A ~973-hour harmonized corpus, a model family at four sizes, an evaluation of more than 60 models and variants, and detailed prompts (App. F) and IMU processing (A.3–A.4).

Why it matters: if released, these would be valuable, and the level of detail is what makes the issues below checkable.

Critical weaknesses
  1. The IMU results may come from leakage of the answer key through the model input.

What first caught my attention was Fig. 18 (p.73): for an IMU fact-check item, EgoMind-9B’s justification is word for word the same as the reference, down to the span “0:00–0:18”. In the next example it gives “roughly 17 and 29 seconds”, which none of the baselines shown does. Looking at how the benchmark is built, this is not surprising. The IMU references are generated from rule-based motion descriptions (B.3.2), the same thresholds are used for annotation and evaluation (A.4.4), and EgoMind receives tag text produced by these rules as part of its input (Eq. 2). The baselines get neither the IMU signal nor this text. The paper reports no tags-only or signal-only results, and no baselines given the tags; the authors themselves point out the first two gaps in H.1. Until these are reported, I cannot read the main IMU result as evidence that the model understands motion.

Why it matters: the main IMU result cannot currently be read as evidence of motion understanding.

  1. Possible train–test leakage, and a pretraining ablation that does not look right.

Table 8 lists EgoBlind and EgoPolice as “val” data, and the sizes match the evaluated sets almost exactly (1.3K items on 1.2K clips vs. 1,283 questions on 1,185 clips; 12.0K items vs. the 12,000 released EgoPolice questions, from which 1,993 are evaluated; C.1.5–C.1.6). A.2 then calls the 1.75M total, which includes these rows, “the training split”, and C.2.3 defines an EgoBlind grader “for systems trained against a designated reference”. Taken together, it looks as if test items of both benchmarks may have been used in training. The same question applies to EgoLife: only the first day of each participant is held out (B.3.1), but EgoBuddy is evaluated on that participant’s week-long recordings. Separately, Table 24 is hard to explain. After instruction tuning, the random-init arm scores almost exactly like the untouched Qwen3.5 backbone (9B MCQ 40.1 vs. 39.8 when I average the backbone’s V/A/I columns in Table 20, as Table 24 does for EgoMind; at 2B, MCQ, OQA and TG are identical). I suspect an evaluation problem, such as the trained weights not being loaded. If these concerns hold, the external-benchmark and memory results, and the claim that “Pretraining improves all five task families at every size”, would not be reliable.

Why it matters: the external-benchmark results and the claim that "Pretraining improves all five task families at every size" may not hold.

Questions for authors

Questions marked [Clarification] ask for information the authors should already have; [Additional analysis] asks for new results that could change my assessment.

  1. [Additional analysis] Please report the IMU columns with tags only, with signal tokens only (tag text removed), and for Qwen3.5-9B and Gemini 3.1 Pro (via its API) with the same tag text added to the prompt. If the signal-only model keeps most of the gain and the tag-given baselines stay well below EgoMind, my main concern is resolved.
  2. [Clarification] What does "val" mean in Table 8? Were any EgoBlind or EgoPolice items, or the evaluated EgoLife-QA participant's other recording days, used for training or model selection? Which systems were scored with the designated-reference EgoBlind grader (C.2.3)?
  3. [Clarification] For Table 24, please give per-column results of the random-init arms and confirm that their trained weights were loaded at evaluation.

Suggestions (these do not affect my rating)

  • Temper the "first attempt" claim (Sec. 1): EgoLife already combined a benchmark, an audio-visual model and a memory system (Tables 14 and 18), and the memory experiments here do not use IMU (D.5.1). Please also discuss AnyMAL (EMNLP 2024 Industry Track), EgoVideoIMU-MAE (ECCV 2024) and EgoLM (CVPR 2025).
  • State what will be released and under which licenses (G.1, H.2).
  • Fix the Table 8 clip totals (≈129.2K summed vs. 116.7K overall) and the Table 5 "Average" column, and report confidence intervals (193–401 items per column).

Overall.

Combining vision, audio and IMU for egocentric understanding is not a new direction; the paper's own Table 14 already lists MMG-Ego4D (CVPR 2023) with all three modalities. The value of this submission therefore rests on its empirical results, and my rating reflects serious concerns that these results may be affected by two kinds of leakage: through the IMU input, and between training and test data. Resolving these concerns would require the analyses requested in Questions 1–3.

6 comments
AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer bfx5,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best,

The Authors of Submission 4625

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer bfx5,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 4625

AuthorsOpenReview ↗

Follow-Up to Your Review (Third Reminder)

Dear Reviewer bfx5,

We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.

As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.

Thank you again for your time and effort devoted in this review!

Best,

The Authors of Submission 4625

Reviewer_s1wROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents EgoMM, combining multimodal training data, evaluation, and long-horizon memory for egocentric understanding. It constructs instruction-tuning annotations from existing wearable recordings and introduces EgoMMU, a benchmark covering five task families with explicit visual, acoustic, and inertial evidence requirements. EgoMind connects audio and IMU pathways to a vision-language backbone through staged pretraining and modality-matched instruction tuning. EgoBuddy uses the resulting descriptions in a hierarchical text-memory system. Experiments evaluate benchmark performance, modality dependence, sensor pretraining, and memory-based question answering under different visual frame budgets.

Critical strengths
  1. The work addresses a relevant limitation of vision-centric egocentric evaluation by organizing learning and evaluation around visual, acoustic, and inertial evidence. The annotated training resources and modality-specific benchmark provide a useful basis for studying these capabilities.

  2. Section 4.2 directly addresses the motivating concern that benchmark performance may reflect contextual guesses rather than use of the intended sensor evidence. Keeping the model and evaluation items fixed while withholding individual modalities provides useful evidence of modality dependence.

Critical weaknesses
  1. The sources of EgoMind’s performance gains in Tables 2–4 are not sufficiently isolated. Cross-model comparisons differ in backbone, training data, and available inputs, while comparisons against the Qwen3.5 backbone measure the combined effect of the proposed adaptation. The pretraining and modality-removal ablations are useful, but an important ambiguity remains in the IMU pathway: rule-based motion descriptions inform reference construction, and motion tags are also supplied as model inputs. The IMU-Off ablation removes both signal tokens and tags, while the pretraining comparisons retain tags. Without a tag-only baseline, it remains unclear how much the learned IMU encoder contributes beyond the provided motion descriptions.

  2. EgoBuddy’s connection to the paper’s central motivation and its distinct contribution are unclear. Although the framework supports IMU input, the reported memory experiments use only video and audio, leaving the value of inertial information for long-horizon understanding untested. The system follows an established hierarchical text-memory and retrieval pipeline, and the experiments primarily evaluate the effect of EgoMind-generated descriptions. This supports a downstream application of EgoMind more clearly than an independent methodological contribution.

Questions for authors
  1. How does EgoMind perform when retaining the motion tags but omitting the learned IMU signal tokens, and conversely when retaining signal tokens without tags? This comparison would help establish the encoder’s contribution.

  2. Does EgoMMU include questions that cannot be answered from either modality alone and require combining evidence from multiple modalities? For example, in a scene with several people producing different sounds, asking “What sound does the person wearing the red coat make?” would require identifying the person visually and associating that person with the corresponding sound. If such questions are included, how are they identified, and do the reported ablations evaluate this subset separately under single- and multi-modality inputs?

  3. Table 18 lists EgoBuddy as supporting video, audio, and IMU, whereas the reported memory experiments use only video and audio. Why was IMU excluded, and is there evidence that it improves long-horizon retrieval or question answering?

  4. What is EgoBuddy’s distinct methodological contribution beyond integrating EgoMind-generated descriptions into an established hierarchical memory and retrieval pipeline?

Presentation suggestions: The connection between Figures 3–4 and the corresponding discussion is not immediately clear. More explicit references and explanations of what each figure demonstrates would improve readability. Moreover, presenting the key modality-dependence comparisons directly in Section 4.2, rather than deferring them to the appendix, would make this central evidence easier to assess.

6 comments
AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer s1wR,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best,

The Authors of Submission 4625

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer s1wR,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 4625

AuthorsOpenReview ↗

Follow-Up to Your Review (Third Reminder)

Dear Reviewer s1wR,

We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.

As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.

Thank you again for your time and effort devoted in this review!

Best,

The Authors of Submission 4625

Reviewer_JKRAOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents EgoMM, combining a benchmark, a multimodal model, and a memory application for egocentric understanding using vision, audio, and IMU. EgoMMU contains 3,762 items from 1,196 clips across five task families and thirteen task–modality combinations. EgoMind connects audio and IMU pathways to a vision-language backbone through staged sensor pretraining and modality-matched instruction tuning, supporting any non-empty subset of available streams. EgoBuddy organizes clip descriptions into hierarchical text memory.

Evaluation includes benchmark comparisons, modality withholding, pretraining ablations, external tasks, and reduced-frame memory experiments. The work is substantial, having 99 pages in total, but the independent contributions of learned IMU representations, cross-modal reasoning, and the memory framework need clearer validation.

Critical strengths
  1. A useful resource for evaluating modality-specific egocentric capabilities. EgoMMU contains 3,762 items from 1,196 clips, supported by recording-group splits, shortcut screening, and human verification. The same-checkpoint ablations provide relevant evidence: for the 9B model, removing audio reduces audio MCQ accuracy from 67.3 to 36.7, while removing vision reduces visual MCQ accuracy from 70.3 to 31.2 (Table 23). These results support the benchmark’s ability to distinguish capabilities that aggregate multimodal scores can obscure.

  2. Systematic empirical support for staged sensor pretraining. Under matched instruction-tuning settings, pretrained initialization improves all five task families at all four model sizes. At 9B, average MCQ performance increases from 40.1 to 67.9 and temporal grounding from 11.4 to 23.8 (Table 24). EgoSchema accuracy also improves from 67.1 for the backbone to 72.1 for EgoMind (Table 4), providing evidence beyond the proposed benchmark, although external gains are not uniform.

Critical weaknesses
  1. The incremental contributions of learned representations and joint modeling remain unclear. EgoMind receives both learned IMU tokens and rule-based tags, while several task references rely on related motion descriptions. IMU-Off removes both (Table 23). Figure 9 supports signal dependence, but does not isolate the learned pathway’s downstream benefit beyond tags. Likewise, modality withholding and narration results demonstrate complementary information, but do not establish an advantage over independent perception followed by text-level fusion. Matched tag/token controls and fusion baselines on jointly dependent questions are needed.

  2. The memory component’s independent contribution and the overall contribution boundaries need clearer presentation. Table 6 changes the captioner within a shared downstream pipeline, primarily evaluating perception-record quality rather than the memory mechanism itself. The video–audio-only, single-participant, non-strictly-causal evaluation limits conclusions about tri-modal memory and online assistance. The main text should consolidate each component’s contribution, decisive controls, and evaluation boundaries, allowing readers to understand the most valuable findings without repeatedly navigating the extensive appendices.

Questions for authors
  1. How many retained benchmark items require joint VA, VI, AI, or VAI evidence, and how are they mapped to the reported columns? On these subsets, how do combined inputs compare with individual modalities and independently generated unimodal descriptions followed by text-level fusion?

  2. Under matched training settings, how do tags-only, learned-tokens-only, and combined configurations perform on the IMU tasks?

  3. Please specify the initialization, trainable modules, training duration, and data mixture for EgoMind† in Table 3. Which benchmarks use the base checkpoint, and which use an additionally fine-tuned variant?

  4. How well do fact-check explanation scores agree with human assessments that have access to the media? Does the grader reliably penalize correct-verdict responses containing unsupported details, and do the principal model rankings remain stable?

5 comments
AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer JKRA,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best,

The Authors of Submission 4625

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer JKRA,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 4625

AuthorsOpenReview ↗

Follow-Up to Your Review (Third Reminder)

Dear Reviewer JKRA,

We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.

As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.

Thank you again for your time and effort devoted in this review!

Best,

The Authors of Submission 4625

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank our reviewers for the thoughtful reviews and for recognizing the value of evaluating sight, sound, and motion separately. We appreciate the suggestions for testing when models benefit from combining these signals and for clarifying what our memory experiments establish.


We are glad that our reviewers recognize this work from the following perspectives:

  • Reviewer JKRA described EgoMMU as a “useful resource for evaluating modality-specific egocentric capabilities” and highlighted the “systematic evidence for staged sensor pretraining across model sizes”.
  • Reviewer s1wR valued the “annotated training resources” and highlighted how the “ablations test which sensors matter by holding the model and evaluation items fixed”.
  • Reviewer bfx5 called modality-specific evaluation “the most reusable part of the paper” and highlighted the “scale of the resources and the detailed documentation of prompts and IMU processing”.

Main Contributions

EgoMM brings wearable motion into egocentric language understanding as a source of evidence alongside video and audio. We connect learning from these signals with controlled evaluation and study how clip descriptions support long-horizon memory.

  • EgoMMU makes modality dependence testable. Across five task families, our benchmark identifies the visual, acoustic, and inertial evidence each item is designed to require, including combinations of these signals. Researchers can then vary the supplied modalities while holding the checkpoint and evaluation items fixed to test whether performance depends on the intended evidence.
  • EgoMind learns from raw wearable IMU. Our model combines learned representations of wearer motion with motion tags, synchronized video, and audio in an egocentric language model. At each of four model sizes, a single checkpoint can process any nonempty subset of the three streams, making it possible to evaluate the same model as sensor availability changes.
  • EgoBuddy evaluates how clip descriptions support long-horizon memory. We compare descriptions from EgoMind and EgoGPT at matched frame budgets in a shared video–audio pipeline. The study measures how the choice of captioner affects retrieval and question answering for one EgoLife-QA participant.

In line with the ICLR 2027 author guidelines for the discussion period, we have revised the manuscript in response to your suggestions.

We highlighted the changes in blue to make them easy to find. Table and figure numbers refer to the submitted manuscript unless we explicitly refer to the revised manuscript.


Additional Results & Experiments

  • Joint evidence and text fusion (JKRA, s1wR). Of the 852 items (22.6%) annotated as requiring joint evidence, 482 are open-ended questions and 370 are narration items. Across the three joint OQA subsets, direct multimodal inputs scored 7.5–9.8 points above the best single modality and 2.0–3.1 points above text fusion. On narration, using all three streams scored 3.8 points above text fusion.
  • Learned IMU tokens and motion tags (JKRA, s1wR, bfx5). In the matched-training comparison, learned tokens scored higher on fact checking, open-ended QA, and temporal grounding, while motion tags scored higher on multiple choice. Adding the same motion tags to Qwen3.5-9B and Gemini 3.1 Pro improved both models across all four IMU columns.
  • Human fact-check ratings (JKRA). For Gemini 3.1 Pro, the automatic grader and three raters with access to the source media had a mean quadratic weighted κ of 0.84. The grader awarded 4 or 5 out of 5 to 74.0% of the responses that gave the correct verdict and were flagged for unsupported details.
  • Random initialization and IMU ablations (bfx5). We verified that evaluation loaded the trained weights from the corresponding random-initialization runs and report scores for all 13 columns at each model size. For EgoMind-9B, the conservative pointwise 95% bounds for withholding IMU exclude zero on MCQ-I and FC-I under the assumptions in our response to Reviewer bfx5, Q7.

Other Revisions & Modifications

  • We have expanded the related work and clarified the scope of the video–audio memory experiments, including how they use observations relative to the question time.
  • We have clarified the dataset splits and the extra fine-tuning used for EgoMind†, and report the standard and adapted checkpoints separately on EgoAVU-Bench.
  • We have corrected the example in Figure 18, reconciled the training and evaluation clip counts in Table 8, and clarified the averaging rule in Table 5. The corrected 30-frame EgoBuddy average is 49.9%.
  • We have outlined a release plan for the EgoMMU annotations, split manifests, and evaluation code, subject to institutional approval and the terms of the source datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.