acceptodds
Under review as a conference paper at ICLR 2027

SOMA: A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding

13 views

Abstract

Perceiving human body motion is critical for multimodal systems that understand and assist people. Yet current evaluations of Multimodal Large Language Models (MLLMs) largely focus on objects, scenes, and coarse actions, leaving their ability to reason about the body itself poorly assessed. This is especially challenging from the egocentric perspective, in which the body is only partially visible and motion must be inferred from occluded and temporally distributed cues. We introduce SOMA, a benchmark for fine-grained body-motion understanding from egocentric video, comprising 1,632 human-verified questions across three capability axes: body configuration, motion dynamics, and temporal reasoning. We evaluate 37 leading closed-source, open-source, and specialized egocentric MLLMs, complemented by an extensive human evaluation. We find that humans also find this task difficult from video alone (68.5% accuracy), yet clearly outperform the strongest MLLM (Qwen3.6-27B), which reaches 50.0% zero-shot accuracy. Our error analysis shows that MLLMs often misread the wearer's own body, misjudge events in time, and are prone to biases such as favoring the right hand or under-counting repetitions. To narrow the gap to human performance and support further progress, we introduce a training dataset of aligned video, body motion, and language supervision, and a multimodal baseline that jointly learns from these modalities. By identifying common failure modes and missing capabilities in current MLLMs, we hope SOMA will inspire future research on body-motion understanding and multimodal reasoning. We will release the full benchmark, training data, and code.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Video question answering, MLLM, VLM, Human behavior understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AqL1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

What: The paper introduces a benchmark consisting of 1,632 multiple-choice questions spanning 12 tasks across three capability axes (body configuration, motion dynamics, and temporal Reasoning) to evaluate Multimodal Large Language Models (MLLMs) on fine-grained human body-motion understanding from egocentric video.

How: To construct the benchmark, the authors processed egocentric video corpora (Nymeria and Ego-Exo4D) using an LLM-driven question generation pipeline based on detailed motion narrations, stripped out text-only language shortcuts via a "blind-language" filter, conducted human quality review, and created SOMA-Train, a supplementary dataset containing aligned video, 3D body pose, and text supervision samples.

Results: The authors evaluate 37 MLLMs, all of which underperform compared to humans. Ablations demonstrate that neither increasing temporal frame sampling nor enabling test-time "thinking" overcomes this bottleneck.

Critical strengths

Annotation quality is critical for a dataset, and the authors clearly had that in mind. I appreciate the thoroughness in putting together this dataset, including the "blind-language" filtering and human verification pipeline.

Critical weaknesses

To me, the main weakness of this paper is that

  1. Roughly two-thirds of the benchmark re-applies body-motion phrasing to standard video QA concepts like action ordering, repetition counting, or next-action prediction that existing datasets already evaluate extensively.

  2. The novel tasks often ask about occluded or out-of-frame body parts, where even human accuracy drops near random chance because the required visual cues are physically unobservable from a head camera.

As a result, the benchmark ends up overwhelmingly testing either what has already been measured elsewhere or what is naturally ambiguous to perceive from a single first-person perspective.

Questions for authors
  1. Did the authors run a text-only "blind" baseline on the final 1,632-question benchmark set to verify that the 586 human-edited and 101 manually added questions did not reintroduce text-based shortcuts?

  2. Could the authors provide insights into what potentially allows models like Qwen3.6-27B to outperform specialized egocentric models? To me, this analysis would be as important as reporting the result itself.

  3. Could an anonymous preview link (like an anonymous GitHub) for the dataset and evaluation code be made available during peer review? I generally prefer that to a promise to provide that in the camera-ready version (besides, preparing the assets now makes the open-sourcing easier once this paper is accepted).

  4. [Advice disguised as question] Given that "SOMA" is an overloaded term across AI and robotics literature, have the authors considered a more distinct title (maybe EgoSOMA?) to avoid search collisions and maximize the dataset visibility?

7 comments
AuthorsOpenReview ↗

Q2: Could the authors provide insights into what potentially allows models like Qwen3.6-27B to outperform specialized egocentric models?

We thank the reviewer for raising this interesting question.

We first note that the egocentric models build on smaller and older backbones, namely 3–7B models from earlier generations (LLaVA-OneVision, Qwen2-VL and Qwen2.5-VL), and even the general-purpose Qwen3.5-4B (43.1%) outperforms all three of them. To isolate the effect of egocentric specialization, we have now evaluated each egocentric model against the base model it was fine-tuned from, under the same protocol.

Egocentric modelBase modelBaseEgocentricΔ
EgoGPT-7BLLaVA-OV-Qwen2-7B31.930.8-1.1
EgoVLM-3BQwen2.5-VL-3B-Instruct35.733.3-2.3
EgoThinker-7BQwen2-VL-7B-Instruct34.835.8+1.0
Mean-0.8

In summary, egocentric fine-tuning brings little to no gain on SOMA (-0.8 points on average), even though the same models improve over their base models on existing egocentric benchmarks such as EgoSchema and EgoTaskQA. We believe that one likely explanation is their training data, which targets actions, objects, and planning rather than the wearer’s body.

Consistent with this, the largest average drops occur on Body Posture (-6.7 points) and Concurrent Body-Part State (-6.9 points), which are precisely the whole-body capabilities that SOMA targets and existing egocentric benchmarks do not capture. We will update the manuscript to show these insights.

AuthorsOpenReview ↗

W1: Relation to existing video QA tasks

We thank the reviewer for this comment. We would like to highlight that, apart from the Body Configuration tasks, which are not covered by any existing egocentric benchmark, the remaining SOMA tasks share familiar names with existing tasks but evaluate a different capability.

In existing egocentric benchmarks, these tasks mainly focus on objects and actions. For instance, counting questions in EgoThink [1] and EgoTempo [2] mostly focus on objects (e.g., "How many pots are there on my right?"), whereas counting questions in SOMA target repeated movements of the wearer's own body (e.g., "How many times does the person punch with the left hand toward the opposite side?"). Similarly, sequence questions in EgoTempo [2] and EgoSchema [3] tend to focus on the order of objects the person interacts with or the main steps of an activity (e.g., "What is the sequence of objects the person interacts with?" or "What is the primary sequence of actions performed by c throughout the video…"), while ordering in SOMA also focuses on the order in which the wearer moves (e.g., "After the person briefly bends down to take the placemat from the chair, what is the sequence of their next two turns?"). Answering the questions in SOMA therefore requires perceiving the wearer's body over time.

Related questions are evaluated by existing body-motion benchmarks (e.g. BABEL-QA [4] and MotionBench [5]), however, these benchmarks are built on third-person video or motion capture data without RGB, where the body is directly observed. As shown in Table 1, no existing benchmark combines egocentric video with body-centric questions, and SOMA fills exactly this gap between the two lines of work.

[1] EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Models, Cheng et al., CVPR 2024.

[2] Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos, Plizzari et al., CVPR 2025.

[3] EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding, Mangalam et al., NeurIPS 2023.

[4] Motion Question Answering via Modular Motion Programs, Endo et al., ICML 2023.

[5] MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models, Hong et al., CVPR 2025.

AuthorsOpenReview ↗

W2: Tasks with occluded or out-of-frame body parts

We would like to clarify that human accuracy is close to chance on only one task, Next Action Prediction (34.5%), where the answer is an action that has not yet occurred in the clip and needs to be fully inferred. For the remaining tasks, we include body parts that are partially occluded or out of view as a design choice. Inferring the wearer's body from indirect cues, such as the height and motion of the camera or partial views of the limbs, is precisely what SOMA aims to evaluate, since a wearable assistant must reason about the wearer's posture and movement without seeing their full body. Our goal is therefore to compare how well humans and models make this inference.

To make this explicit, we have now grouped the tasks by observability into four groups: directly observable perception, temporal tasks with visible evidence, hidden-state inference, and future prediction.

GroupTasks#QHumanModelsGap
Directly observable perceptionHU, BC, RC, OMD59386.145.340.8
Temporal tasks with visible evidenceAAI, CO, MEI, RER49665.234.330.9
Hidden-state inferenceBP, CBS, MD37960.336.623.7
Future predictionNAP16434.527.86.8

When the evidence is directly visible, humans are highly accurate, for instance on Hand Use (90.5%) and Repetition Counting (88.9%). When the questions require temporal tracking of fine-grained actions or body movements, which are easy to miss because they are often very brief and accompanied by rapid head movements in egocentric video, the task becomes harder for humans as well (65.2%). When the questions ask about parts of the body that are rarely visible from the egocentric view, and the answer must be inferred, human accuracy is naturally lower but still well above chance, as on Body Posture (61.9%) and Concurrent Body-Part State (50.3%), and higher than model performance.

Reviewer_3B38OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SOMA, a benchmark for fine-grained understanding of human motion in egocentric videos. The authors point out that existing egocentric video benchmarks mainly focus on coarse-grained reasoning, while relatively little attention has been paid to whether models can understand the fine-grained body-motion understanding. To address this gap, SOMA evaluates three capability dimensions: Body Configuration, Motion Dynamics, and Temporal Reasoning, covering 12 task categories and a total of 1,632 questions. The authors also adopt blind-language filtering to reduce potential shortcuts in the benchmark. 37 models are evaluated on SOMA, and the experimental results show that current models still exhibit clear limitations in fine-grained body-motion understanding.

Critical strengths
  1. The authors formulate a fine-grained problem, which evaluate whether models can understand the body-motion from an egocentric video. Compared with existing models and benchmarks that mainly focus on more global understanding, this formulation is meaningful and potentially important.
  2. The use of blind-language filtering helps reduce the possibility of linguistic shortcuts in the benchmark.
  3. The work also provides the fine-grained SOMA-Train dataset. If released publicly, it could serve as a useful training dataset for future research on video-motion understanding.
Critical weaknesses
  1. In my opinion, the most critical aspect of this problem is how to construct convincing fine-grained body-motion annotations. However, the fine-grained body-motion annotations in this benchmark are largely inherited from Nymeria and Ego-Exo4D, and SOMA mainly reorganizes existing open-source datasets. As a result, the work avoids the difficult process of collecting new fine-grained annotations, which makes the contribution of this benchmark relatively limited.
  2. Converting existing annotations from prior datasets into standardized multiple-choice questions is relatively straightforward and does not appear to involve substantial technical effort.
  3. Although fine-tuning MLLMs on SOMA-Train is necessary, the reported results are not sufficiently convincing. Even after incorporating both video and motion information, the experiments do not clearly demonstrate that SOMA-Train can be used to train a substantially stronger body-motion MLLM.
Questions for authors
  1. According to the experimental results, the improvement brought by SOMA-Train to body-motion understanding appears to be rather limited. Is this mainly because the current model architecture of injecting motion information into the MLLM is ineffective, or because SOMA-Train itself has limited effectiveness in improving body-motion understanding?
  2. The proposed overall pipeline of constructing SOMA appears relatively straightforward. What is the key novelty of SOMA compared with Nymeria and Ego-Exo4D? I would appreciate it if the authors could clarify the specific innovations and contributions beyond reorganizing and reformulating existing annotations.

If the authors can address the issues I raised in the weaknesses and questions, I would consider increasing my score.

4 comments
AuthorsOpenReview ↗

W1, W2 & Q2: Contribution and construction of SOMA

W1 & Q2: Contribution beyond Nymeria and Ego-Exo4D

We thank the reviewer for this comment. Collecting egocentric video and annotating fine-grained semantics are highly resource-intensive, and Nymeria and Ego-Exo4D are indeed essential contributions. While these datasets are valuable resources, our work builds on them to ask a research question that neither the original papers nor the benchmarks built on them have investigated, namely whether MLLMs can infer the camera wearer's body configuration, fine-grained motion dynamics, and temporal motion structure directly from egocentric video. The existing datasets do not constitute such an evaluation by themselves, and turning them into one required several additions beyond the source data.

First, SOMA is designed as a multimodal benchmark that pairs egocentric video with the aligned 3D body motion of the wearer. Such motion was not fully available in the source data, as the existing EE4D-Motion [1] fits are provided at only 10 fps and cover only part of the recordings. We therefore reconstructed the wearer's 3D body motion from the multi-view recordings at 20 fps, which raised the average motion coverage of each Ego-Exo4D clip in SOMA from 37.9% to 96.6% (Appendix E.2). This allows SOMA to evaluate a broader range of models, including emerging models that combine egocentric video with body motion or rely on body motion alone.

Next, some questions cannot be derived from the source annotations at all. For instance, the narrations do not provide exact repetition counts, so all Repetition Counting questions were created and counted manually from the video and the synchronized 3D body pose (Appendix B.3). We describe the extensive work behind the remaining questions in our response to W2.

More broadly, reusing valuable video corpora to study previously unmeasured capabilities is an established benchmark practice, as exemplified by EgoSchema [2], HourVideo [3], EgoTempo [4], and EgoAVU [5], which build on Ego4D, and by ExAct [6] and EgoExoBench [7], which build on Ego-Exo4D, to name a few. SOMA similarly converts rich existing recordings into a new diagnostic resource and reveals a substantial capability gap between humans and MLLMs, as well as specific failure modes that could not be observed through the original datasets themselves.

W2: Effort of converting annotations into questions

We respectfully disagree with this assessment. Converting the source annotations for SOMA into a reliable benchmark is far from straightforward. To do so, we (i) defined a new taxonomy of 12 tasks across three capability axes, (ii) identified task-relevant and nontrivial video segments from multiple annotation streams, (iii) designed task-specific questions and hard distractors, with dedicated prompts, in-context examples, and distractor strategies for each task refined over many iterations (Appendix B.1), (iv) ensured that answers depend on visual motion evidence rather than annotation conventions or language priors, and (v) ran psychophysics evaluations with five subjects.

Our pipeline produced 3,928 candidate questions, and the four-model blind-language filter removed 1,801 of them (45.9%). Every remaining question was then manually inspected against the video and a synchronized 3D body-pose rendering, with 586 questions modified and 596 deleted (Appendix B.4). Five evaluators independently answered all 1,632 questions, requiring 465 hours of work, to establish a human baseline against which model performance can be interpreted. Our benchmark is therefore the result of a carefully designed, task-specific question generation pipeline and manual curation, rather than a direct reformatting of existing annotations.

[1] UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generation, Patel et al., ICCV 2025.

[2] EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding, Mangalam et al., NeurIPS 2023.

[3] HourVideo: 1-Hour Video-Language Understanding, Chandrasegaran et al., NeurIPS 2024.

[4] Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videos, Plizzari et al., CVPR 2025.

[5] EgoAVU: Egocentric Audio-Visual Understanding, Seth et al., CVPR 2026.

[6] ExAct: A Video-Language Benchmark for Expert Action Analysis, Yi et al., NeurIPS 2025.

[7] EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs, He et al., NeurIPS 2025.

AuthorsOpenReview ↗

W3 & Q1: Effectiveness of SOMA-Train and motion integration

We thank the reviewer for this question. We designed these experiments to test whether body motion provides information beyond the egocentric video, rather than optimizing for state-of-the-art performance. For this reason, we used a vanilla architecture following the LLaVA [1]/MotionLLM [2] design, where motion is only lightly aligned, using a frozen motion encoder and a projector trained on Nymeria narrations.

To test whether this was the bottleneck, and whether better motion integration leads to stronger models, we kept the architecture, the SOMA-Train questions, and the question-answering hyperparameters fixed, and replaced the light alignment with a two-stage full alignment. In the first stage, we retrained the motion encoder and aligned it with language on motion–text data. In the second stage, we trained the model to narrate the wearer’s body from video and motion. Finally, we fine-tuned on SOMA-Train, mixing the question-answering data with motion captioning and narration. For each fine-tuned model, we report its accuracy with both inputs, and with motion, video, or all inputs withheld at inference.

Qwen3-VL-8BVideo + MotionVideo onlyMotion onlyGain from motionBlind
Zero-shotN/A35.4N/AN/A25.4
Light alignment + SOMA-Train42.441.029.1+1.425.8
Full alignment + SOMA-Train45.643.236.0+2.427.2

With full alignment, fine-tuning on SOMA-Train raises accuracy from 35.4% to 45.6%, closing about a third of the gap to human accuracy on the same questions. The gain from adding motion remains concentrated on Body Configuration, with the largest improvements on Body Posture (+16.4) and Concurrent Body-Part State (+10.0).

At the same time, SOMA-Train is designed so that the model remains close to chance without any perceptual input (27.2%), confirming that these gains come from video and motion rather than from answer patterns in the training data, and that SOMA-Train supports learning body-motion understanding.

We provide these additional baselines as validation of the benchmark's design, and expect models designed specifically for egocentric video and body motion, and architectures that integrate the two more tightly, to benefit more from SOMA and SOMA-Train.

[1] Visual Instruction Tuning, Liu et al., NeurIPS 2023.

[2] MotionLLM: Understanding Human Behaviors from Human Motions and Videos, Chen et al., TPAMI 2025.

Thank you to the author for the response. After carefully reviewing the additional information and experiment results, most of my questions have been addressed, so I’ve decided to raised my score from 2 to 3.

We are glad that the additional information and experiments have addressed the reviewer’s questions, and we sincerely thank them again for their time and for their thoughtful feedback throughout the discussion.

Reviewer_S21eOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SOMA is a benchmark for fine-grained body-motion understanding from egocentric video, covering body configuration, motion dynamics, and temporal reasoning. Evaluations reveal a substantial gap between current MLLMs and humans, with models struggling to interpret the wearer’s body, track event timing, and count repetitions. This paper also introduces aligned video–motion–language training data and a multimodal baseline to support further progress.

Critical strengths
  • The paper addresses problems often overlooked in existing works. For a first-person perspective assistant, we need to understand the user’s posture and movement. Simply recognizing objects or activities sometimes is insufficient. It is valuable to evaluate these capabilities.
  • This paper conducts error analysis and combines video and body-motion signals to further improve the performance, showing us more possibilities about this dataset.
  • The evaluation set underwent human verification, and the experiments cover a broad range of settings.
  • There is also a training dataset, though the quality is not human-verified.
Critical weaknesses
  • For a dataset constructed through automatic or semi-automatic annotation, approximately 100 hours is a relatively small scale.

  • In this paper human evaluators also make substantial errors on this benchmark (68.5% average accuracy). This raises the question of whether some tasks are insufficiently well defined, inherently ambiguous, or too difficult to answer from the available visual evidence. These difficulties may also affect dataset construction and introduce annotation errors. Further evidence of annotation reliability and video-based answerability would strengthen the benchmark.

  • The benchmark combines directly observable tasks or understanding tasks, such as identifying the active hand or counting repetitions, with tasks requiring inference about invisible body parts or prediction of future actions. While the former generally have verifiable answers, the latter may admit multiple reasonable answers given the video. Although per-task results are reported, an additional breakdown by observability and answer determinacy would help distinguish direct perception from hidden-state inference and future prediction.

  • Some tasks may remain solvable through scene context and associated objects without requiring detailed understanding of the user's body. For example, next-action prediction in a food-preparation scenario could potentially rely on familiar activity sequences rather than detailed understanding of the wearer’s body motion. Additional controls are needed to establish whether performance reflects body-motion understanding rather than contextual shortcuts.

Questions for authors
  • How was video-based answerability verified, particularly for questions about body parts outside the field of view? Since annotation reviewers had access to synchronized 3D poses, how did you ensure that the correct answer could also be determined from the video alone?
  • Given the relatively low human accuracy on some tasks, could you clarify how annotation errors, ambiguity, and insufficient visual evidence were distinguished from genuine task difficulty? Were independent annotations and inter-annotator agreement assessed?
  • Would you consider reporting directly observable tasks separately from hidden-state inference and future-action prediction? For the latter, how can uncertainty be addressed?
4 comments
AuthorsOpenReview ↗

W1: Scale of SOMA-Train

We thank the reviewer for this comment. We would like to note that 110.2 hours is in fact a substantial scale for data that pairs egocentric video with synchronized 3D body motion. Such data requires motion capture or multi-view reconstruction, and only a few datasets provide it.

However, even within these datasets, the required annotations do not cover all recordings. Body motion in EE4D-Motion [1] is only partially available and at 10 fps, so we reconstructed it at 20 fps, covering on average 96.3% of each SOMA-Train segment compared with 48.1% for the existing fits (Appendix E.2). Motion narrations and atomic action annotations are also not provided for all videos. In addition, we focused on segments in which the body is actively moving, ideally with several body parts engaged at once, and avoided repetitive questions.

Even with these constraints, SOMA-Train still provides 28,596 question–answer pairs over 110.2 hours, which is comparable to or larger than existing video and motion question-answering training sets and, to our knowledge, the first to pair egocentric video with synchronized 3D body motion and question–answer supervision.

Training SetHoursQuestionsEgocentric3D Body Motion
NExT-QA [2]~4734,132NoNo
EgoIT-99K [3]43.299.5K (with captions)YesNo
EgoTaskQA [4]~20~24KYesNo
BABEL-QA [5]~2.51,800N/A (motion only)Yes
SOMA-Train110.228,596YesYes

Moreover, since the generation pipeline of the training dataset is fully automatic, it can be directly applied to additional egocentric recordings with body motion narrations as they become available.

[1] UniEgoMotion: A Unified Model for Egocentric Motion Reconstruction, Forecasting, and Generation, Patel et al., ICCV 2025.

[2] NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions, Xiao et al., CVPR 2021.

[3] EgoLife: Towards Egocentric Life Assistant, Yang et al., CVPR 2025.

[4] EgoTaskQA: Understanding Human Tasks in Egocentric Videos, Jia et al., NeurIPS 2022.

[5] Motion Question Answering via Modular Motion Programs, Endo et al., ICML 2023.

AuthorsOpenReview ↗

W3 & Q3: Breakdown by observability

We thank the reviewer for this interesting suggestion and we agree that comparing humans and models under different levels of visibility is a valuable way to understand the challenges specific to the egocentric perspective. Following the reviewer’s distinction, we split the tasks into four groups: directly observable perception, temporal tasks with visible evidence, hidden-state inference, and future prediction.

GroupTasks#QHumanModelsGap
Directly observable perceptionHU, BC, RC, OMD59386.145.340.8
Temporal tasks with visible evidenceAAI, CO, MEI, RER49665.234.330.9
Hidden-state inferenceBP, CBS, MD37960.336.623.7
Future predictionNAP16434.527.86.8

This breakdown leads to four clear observations. When the evidence is directly visible, humans answer the questions with high accuracy (86.1%), while models score 40.8 points lower, even though these questions rely mostly on perception. When the questions require temporal tracking of fine-grained actions or body movements, which are easy to miss because they are often very brief and accompanied by rapid head movements in egocentric video, the task becomes harder for humans as well (65.2%). However, the gap to models remains rather large (30.9 points). For hidden-state inference, where the questions ask about parts of the body that are rarely visible from the egocentric view, both humans and models must therefore infer the state or movement of the body from indirect cues, such as the height and motion of the camera, and humans use these cues better than models (60.3% vs. 36.6%). Finally, for future prediction, where the answer needs to be fully inferred, the gap nearly closes (6.8 points).

AuthorsOpenReview ↗

W2, Q1 & Q2: Annotation reliability and answerability from video

W2 & Q2: Annotation reliability and agreement

We thank the reviewer for this comment. Annotation reliability is ensured at several levels. First, the ground truth in the benchmark itself is derived from high-quality human annotations coming from two widely used datasets. In Nymeria, the motion-language descriptions are obtained by annotators describing the motion in context while watching synchronized playback of the egocentric view, the third-person view, and the motion rendering. Similarly, in Ego-Exo4D, annotators are presented with four or five fixed-position exocentric cameras in addition to the egocentric view. In both cases, annotators could therefore see the wearer's body from external viewpoints, which overcomes the limited visibility of the egocentric view. These datasets are also widely used to build other benchmarks, such as ExAct [1] and EgoExoBench [2] on Ego-Exo4D and EgoEverything [3] on Nymeria.

Second, as the final step of our pipeline, every question was human-verified against the video and the synchronized 3D body pose to catch potential errors. This review led to corrections in 586 questions (for instance, in their wording, answer options, or segment boundaries) and to the removal of 596 questions that were potentially ambiguous or had more than one plausible answer.

In addition, the five evaluators, who were not involved in constructing the benchmark, could flag any question they found confusing and provide a reason. Such cases were rare, and every flagged question was either corrected or removed in the final benchmark. To assess the consistency of their answers, we further computed the inter-annotator agreement (Fleiss' κ\kappa) for each group. Agreement is substantial on directly observable tasks (κ=0.74\kappa = 0.74) and decreases as the answer becomes less visible and requires more inference (κ=0.40\kappa = 0.40 for hidden-state inference), consistent with the accuracy trend which we reported in our response to W3 & Q3. Furthermore, we note that annotation errors would instead cause evaluators to agree on an answer other than the ground truth, which we do not observe. We also note that the human evaluators do have access to all frames, but do not have access to the 3D motion.

Q1: Video-based answerability

We would like to clarify that our goal with SOMA is precisely to go beyond what is directly visible and to evaluate how well humans and models understand the wearer’s body from the egocentric view, where the body is often only partially visible and its state must be inferred by combining partial visual evidence. Questions about out-of-view body parts are therefore an intended part of SOMA, and the relevant question is whether they can be answered by inference from the video rather than by direct observation.

While the 3D body pose was used during construction as an additional reference to verify that the ground truth was correct, the five evaluators answered all questions from the video alone, without access to the 3D body pose or any annotations. When the body was not visible, they were instructed to rely on indirect cues such as camera motion, viewpoint changes, and hand–object interactions (Appendix C.1). Even on hidden-state inference (see our response to W3 & Q3), they reach 60.3%, well above the 25% chance level, showing that these questions can be answered from the video by inference.

This is also why we designed SOMA as a multimodal benchmark that pairs the video with the aligned 3D body motion of the wearer. Similar to how humans complement vision with a sense of their own posture and movement, the 3D body motion offers future models a direct signal for the parts of the body that the camera does not directly capture.

[1] ExAct: A Video-Language Benchmark for Expert Action Analysis, Yi et al., NeurIPS 2025.

[2] EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMs, He et al., NeurIPS 2025.

[3] EgoEverything: A Benchmark for Human Behavior–Inspired Long-Context Egocentric Video Understanding in AR Environment, Tang et al., ECCV 2026.

AuthorsOpenReview ↗

W4: Controls for contextual shortcuts

We thank the reviewer for this comment. We have taken these potential shortcuts into account when designing the benchmark, and our results confirm that the scene and object context alone is not sufficient. Specifically, to support this quantitatively we analyzed the performance of models with a single input frame, in which the scene and objects are visible but the body motion cannot be inferred from the static image. In this setting, the average accuracy across the evaluated models is 29.8% (Figure 4A), and specifically for Next Action Prediction it is 23.2%, showing that the scene and object context alone does not provide a shortcut for predicting the next action.

This follows from how the benchmark was designed. First, the distractors were constructed with task-specific strategies, so that the answer options within a question share the same scene, objects, or action, and differ mainly in how the body moves (e.g., which hand is used or in which direction the arm moves) or are equally plausible steps of the same activity. We provide per-task explanations in Appendix B.1.

Second, our pipeline includes the blind-language filter (Section 3.2), which is a crucial step that we designed specifically to mitigate such shortcuts. This step filtered questions that text-only models could answer from such priors and removed 45.9% of the initially generated questions (Appendix B.4).

Furthermore, we evaluated the performance of "blind" models on the final 1,632 questions, which received only the question and answer options without any visual input. All models score near chance, and even Qwen3.6-27B, the strongest model on SOMA, reaches only 29.2% without video, compared with 50.0% with video, confirming that the questions cannot be answered from text alone.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank all reviewers for their time and thoughtful feedback, which has helped us improve our work. We are glad that the reviewers recognized the following strengths of our work:

  • Motivation: The reviewers noted that the paper addresses problems often overlooked in existing work and evaluates capabilities that are valuable for first-person assistants (S21e), and found its fine-grained formulation meaningful and potentially important compared with benchmarks focused on more global understanding (3B38).
  • Benchmark quality: The reviewers appreciated our thoroughness regarding annotation quality, including the blind-language filtering and human verification (AqL1, 3B38, S21e).
  • Experiments and analysis: The reviewers found that the experiments cover a broad range of settings, and that the error analysis and the combination of video and body motion show further possibilities of the dataset (S21e).
  • Training data: The reviewers noted that SOMA-Train can serve as a useful resource for future research on video–motion understanding (3B38, S21e).

We have uploaded a revised manuscript, with all changes highlighted in blue. The main revisions are:

  • Breakdown of human and model accuracy by observability (Sec. 5.1, Appendix D.6): AqL1, S21e
  • Comparison of egocentric models with their base models (Sec. 5.1, Appendix D.7): AqL1
  • Blind baseline on the final benchmark (Sec. 3.2, Appendix B.5): AqL1, S21e
  • Discussion of the information available during question creation vs. evaluation (Sec. 6, Appendix D.6): AqL1, S21e
  • Relation to temporal tasks in existing benchmarks (Sec. 2): AqL1
  • Clarification that human evaluators answered from video alone (Sec. 4): S21e
  • Single-frame control for scene and object shortcuts (Sec. 5.2): S21e
  • Video–motion model with full motion–language alignment (Appendix E.3, E.4): 3B38
  • Anonymous repository with the full benchmark and evaluation code (Abstract, Reproducibility Statement): AqL1

We provide detailed responses to each reviewer in the individual threads, and we are happy to answer any further questions. We thank the reviewers again for their time and useful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.