- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents EgoMM for egocentric understanding from video, audio and head-mounted IMU. It contributes (1) EgoMMU, a benchmark of 3,762 LLM-authored items on 30-s clips, with five task families forming 13 task–modality columns; (2) EgoMind, a Qwen3.5-based model (1B–9B) with audio and IMU pathways, rule-based IMU "tag text", and LoRA instruction tuning on 1.75M generated items; and (3) EgoBuddy, a hierarchical text memory evaluated on one EgoLife-QA participant. Reported results: EgoMind-9B outperforms Gemini 3.1 Pro on IMU MCQ (66.2 vs. 45.4), sensor pretraining improves all task families at every size (Table 24), and EgoMind performs competitively on external egocentric benchmarks.
Critical strengths
- Modality-specific evaluation.
Splitting tasks by the modality that should supply the evidence, and testing this by withholding modalities (Table 23, E.2), is more informative than aggregate audio-visual scores. The finding that vision-only models match audio-visual models on sound-source attribution (Table 21) is a useful caution.
Why it matters: this is the most reusable part of the paper.
- Scale of the resources.
A ~973-hour harmonized corpus, a model family at four sizes, an evaluation of more than 60 models and variants, and detailed prompts (App. F) and IMU processing (A.3–A.4).
Why it matters: if released, these would be valuable, and the level of detail is what makes the issues below checkable.
Critical weaknesses
- The IMU results may come from leakage of the answer key through the model input.
What first caught my attention was Fig. 18 (p.73): for an IMU fact-check item, EgoMind-9B’s justification is word for word the same as the reference, down to the span “0:00–0:18”. In the next example it gives “roughly 17 and 29 seconds”, which none of the baselines shown does. Looking at how the benchmark is built, this is not surprising. The IMU references are generated from rule-based motion descriptions (B.3.2), the same thresholds are used for annotation and evaluation (A.4.4), and EgoMind receives tag text produced by these rules as part of its input (Eq. 2). The baselines get neither the IMU signal nor this text. The paper reports no tags-only or signal-only results, and no baselines given the tags; the authors themselves point out the first two gaps in H.1. Until these are reported, I cannot read the main IMU result as evidence that the model understands motion.
Why it matters: the main IMU result cannot currently be read as evidence of motion understanding.
- Possible train–test leakage, and a pretraining ablation that does not look right.
Table 8 lists EgoBlind and EgoPolice as “val” data, and the sizes match the evaluated sets almost exactly (1.3K items on 1.2K clips vs. 1,283 questions on 1,185 clips; 12.0K items vs. the 12,000 released EgoPolice questions, from which 1,993 are evaluated; C.1.5–C.1.6). A.2 then calls the 1.75M total, which includes these rows, “the training split”, and C.2.3 defines an EgoBlind grader “for systems trained against a designated reference”. Taken together, it looks as if test items of both benchmarks may have been used in training. The same question applies to EgoLife: only the first day of each participant is held out (B.3.1), but EgoBuddy is evaluated on that participant’s week-long recordings. Separately, Table 24 is hard to explain. After instruction tuning, the random-init arm scores almost exactly like the untouched Qwen3.5 backbone (9B MCQ 40.1 vs. 39.8 when I average the backbone’s V/A/I columns in Table 20, as Table 24 does for EgoMind; at 2B, MCQ, OQA and TG are identical). I suspect an evaluation problem, such as the trained weights not being loaded. If these concerns hold, the external-benchmark and memory results, and the claim that “Pretraining improves all five task families at every size”, would not be reliable.
Why it matters: the external-benchmark results and the claim that "Pretraining improves all five task families at every size" may not hold.
Questions for authors
Questions marked [Clarification] ask for information the authors should already have; [Additional analysis] asks for new results that could change my assessment.
- [Additional analysis] Please report the IMU columns with tags only, with signal tokens only (tag text removed), and for Qwen3.5-9B and Gemini 3.1 Pro (via its API) with the same tag text added to the prompt. If the signal-only model keeps most of the gain and the tag-given baselines stay well below EgoMind, my main concern is resolved.
- [Clarification] What does "val" mean in Table 8? Were any EgoBlind or EgoPolice items, or the evaluated EgoLife-QA participant's other recording days, used for training or model selection? Which systems were scored with the designated-reference EgoBlind grader (C.2.3)?
- [Clarification] For Table 24, please give per-column results of the random-init arms and confirm that their trained weights were loaded at evaluation.
Suggestions (these do not affect my rating)
- Temper the "first attempt" claim (Sec. 1): EgoLife already combined a benchmark, an audio-visual model and a memory system (Tables 14 and 18), and the memory experiments here do not use IMU (D.5.1). Please also discuss AnyMAL (EMNLP 2024 Industry Track), EgoVideoIMU-MAE (ECCV 2024) and EgoLM (CVPR 2025).
- State what will be released and under which licenses (G.1, H.2).
- Fix the Table 8 clip totals (≈129.2K summed vs. 116.7K overall) and the Table 5 "Average" column, and report confidence intervals (193–401 items per column).
Overall.
Combining vision, audio and IMU for egocentric understanding is not a new direction; the paper's own Table 14 already lists MMG-Ego4D (CVPR 2023) with all three modalities. The value of this submission therefore rests on its empirical results, and my rating reflects serious concerns that these results may be affected by two kinds of leakage: through the IMU input, and between training and test data. Resolving these concerns would require the analyses requested in Questions 1–3.
6 comments
Follow-Up to Your Review (Second Reminder)
Dear Reviewer bfx5,
Thank you again for the time and care you put into reviewing our work!
We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.
If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.
Thanks again for your support!
Sincerely yours,
The Authors of Submission 4625
Follow-Up to Your Review (Third Reminder)
Dear Reviewer bfx5,
We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.
As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.
Thank you again for your time and effort devoted in this review!
Best,
The Authors of Submission 4625
Follow-Up to Your Review
Dear Reviewer
bfx5,Thank you again for the time and effort you put into reviewing our work!
We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.
Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.
Thank you!
Best,
The Authors of Submission
4625