- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
What: The paper introduces a benchmark consisting of 1,632 multiple-choice questions spanning 12 tasks across three capability axes (body configuration, motion dynamics, and temporal Reasoning) to evaluate Multimodal Large Language Models (MLLMs) on fine-grained human body-motion understanding from egocentric video.
How: To construct the benchmark, the authors processed egocentric video corpora (Nymeria and Ego-Exo4D) using an LLM-driven question generation pipeline based on detailed motion narrations, stripped out text-only language shortcuts via a "blind-language" filter, conducted human quality review, and created SOMA-Train, a supplementary dataset containing aligned video, 3D body pose, and text supervision samples.
Results: The authors evaluate 37 MLLMs, all of which underperform compared to humans. Ablations demonstrate that neither increasing temporal frame sampling nor enabling test-time "thinking" overcomes this bottleneck.
Critical strengths
Annotation quality is critical for a dataset, and the authors clearly had that in mind. I appreciate the thoroughness in putting together this dataset, including the "blind-language" filtering and human verification pipeline.
Critical weaknesses
To me, the main weakness of this paper is that
-
Roughly two-thirds of the benchmark re-applies body-motion phrasing to standard video QA concepts like action ordering, repetition counting, or next-action prediction that existing datasets already evaluate extensively.
-
The novel tasks often ask about occluded or out-of-frame body parts, where even human accuracy drops near random chance because the required visual cues are physically unobservable from a head camera.
As a result, the benchmark ends up overwhelmingly testing either what has already been measured elsewhere or what is naturally ambiguous to perceive from a single first-person perspective.
Questions for authors
-
Did the authors run a text-only "blind" baseline on the final 1,632-question benchmark set to verify that the 586 human-edited and 101 manually added questions did not reintroduce text-based shortcuts?
-
Could the authors provide insights into what potentially allows models like Qwen3.6-27B to outperform specialized egocentric models? To me, this analysis would be as important as reporting the result itself.
-
Could an anonymous preview link (like an anonymous GitHub) for the dataset and evaluation code be made available during peer review? I generally prefer that to a promise to provide that in the camera-ready version (besides, preparing the assets now makes the open-sourcing easier once this paper is accepted).
-
[Advice disguised as question] Given that "SOMA" is an overloaded term across AI and robotics literature, have the authors considered a more distinct title (maybe EgoSOMA?) to avoid search collisions and maximize the dataset visibility?
7 comments
W1: Relation to existing video QA tasks
We thank the reviewer for this comment. We would like to highlight that, apart from the Body Configuration tasks, which are not covered by any existing egocentric benchmark, the remaining SOMA tasks share familiar names with existing tasks but evaluate a different capability.
In existing egocentric benchmarks, these tasks mainly focus on objects and actions. For instance, counting questions in EgoThink [1] and EgoTempo [2] mostly focus on objects (e.g., "How many pots are there on my right?"), whereas counting questions in SOMA target repeated movements of the wearer's own body (e.g., "How many times does the person punch with the left hand toward the opposite side?"). Similarly, sequence questions in EgoTempo [2] and EgoSchema [3] tend to focus on the order of objects the person interacts with or the main steps of an activity (e.g., "What is the sequence of objects the person interacts with?" or "What is the primary sequence of actions performed by c throughout the video…"), while ordering in SOMA also focuses on the order in which the wearer moves (e.g., "After the person briefly bends down to take the placemat from the chair, what is the sequence of their next two turns?"). Answering the questions in SOMA therefore requires perceiving the wearer's body over time.
Related questions are evaluated by existing body-motion benchmarks (e.g. BABEL-QA [4] and MotionBench [5]), however, these benchmarks are built on third-person video or motion capture data without RGB, where the body is directly observed. As shown in Table 1, no existing benchmark combines egocentric video with body-centric questions, and SOMA fills exactly this gap between the two lines of work.
[1] EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models, Cheng et al., CVPR 2024.
[2] Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos, Plizzari et al., CVPR 2025.
[3] EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding, Mangalam et al., NeurIPS 2023.
[4] Motion Question Answering via Modular Motion Programs, Endo et al., ICML 2023.
[5] MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models, Hong et al., CVPR 2025.
W2: Tasks with occluded or out-of-frame body parts
We would like to clarify that human accuracy is close to chance on only one task, Next Action Prediction (34.5%), where the answer is an action that has not yet occurred in the clip and needs to be fully inferred. For the remaining tasks, we include body parts that are partially occluded or out of view as a design choice. Inferring the wearer's body from indirect cues, such as the height and motion of the camera or partial views of the limbs, is precisely what SOMA aims to evaluate, since a wearable assistant must reason about the wearer's posture and movement without seeing their full body. Our goal is therefore to compare how well humans and models make this inference.
To make this explicit, we have now grouped the tasks by observability into four groups: directly observable perception, temporal tasks with visible evidence, hidden-state inference, and future prediction.
| Group | Tasks | #Q | Human | Models | Gap |
|---|---|---|---|---|---|
| Directly observable perception | HU, BC, RC, OMD | 593 | 86.1 | 45.3 | 40.8 |
| Temporal tasks with visible evidence | AAI, CO, MEI, RER | 496 | 65.2 | 34.3 | 30.9 |
| Hidden-state inference | BP, CBS, MD | 379 | 60.3 | 36.6 | 23.7 |
| Future prediction | NAP | 164 | 34.5 | 27.8 | 6.8 |
When the evidence is directly visible, humans are highly accurate, for instance on Hand Use (90.5%) and Repetition Counting (88.9%). When the questions require temporal tracking of fine-grained actions or body movements, which are easy to miss because they are often very brief and accompanied by rapid head movements in egocentric video, the task becomes harder for humans as well (65.2%). When the questions ask about parts of the body that are rarely visible from the egocentric view, and the answer must be inferred, human accuracy is naturally lower but still well above chance, as on Body Posture (61.9%) and Concurrent Body-Part State (50.3%), and higher than model performance.
Q2: Could the authors provide insights into what potentially allows models like Qwen3.6-27B to outperform specialized egocentric models?
We thank the reviewer for raising this interesting question.
We first note that the egocentric models build on smaller and older backbones, namely 3–7B models from earlier generations (LLaVA-OneVision, Qwen2-VL and Qwen2.5-VL), and even the general-purpose Qwen3.5-4B (43.1%) outperforms all three of them. To isolate the effect of egocentric specialization, we have now evaluated each egocentric model against the base model it was fine-tuned from, under the same protocol.
In summary, egocentric fine-tuning brings little to no gain on SOMA (-0.8 points on average), even though the same models improve over their base models on existing egocentric benchmarks such as EgoSchema and EgoTaskQA. We believe that one likely explanation is their training data, which targets actions, objects, and planning rather than the wearer’s body.
Consistent with this, the largest average drops occur on Body Posture (-6.7 points) and Concurrent Body-Part State (-6.9 points), which are precisely the whole-body capabilities that SOMA targets and existing egocentric benchmarks do not capture. We will update the manuscript to show these insights.