acceptodds
Under review as a conference paper at ICLR 2027

EgoOmniWorld: Unifying Egocentric Understanding via Ego–Action–World Interactions

Abstract

Egocentric video understanding has rapidly expanded into a diverse landscape of benchmarks targeting different capabilities, modalities, and evaluation formats. While each direction captures an important aspect of egocentric understanding, these capabilities are often studied in isolation, despite being tightly coupled in real-world interaction. We argue that a defining property of egocentric video provides a natural unifying principle: the wearer perceives the world, forms goals, takes actions, changes the world, and observes the resulting feedback to guide subsequent behavior, forming a closed Ego–Action–World interaction loop. Based on this view, we introduce EgoOmniWorld, a unified benchmark that organizes egocentric understanding through four complementary interaction structures: Node, capturing ego, action, and world states; Edge, capturing directed relations among them; Chain, capturing linked dependencies across multiple relations; and Loop, capturing feedback-closed interactions. We further develop an interaction-first, evidence-grounded data pipeline that extracts structured interactions from egocentric videos, grounds them in visual and auditory evidence, and generates questions according to their interaction structure. Rather than treating multimodality and dialogue form as separate task families, the pipeline naturally determines the required modality and question-answering format according to the evidence and dependencies involved. EgoOmniWorld contains 24,166 QA groups, including 20,284 training and 3,882 test samples, spanning diverse interaction scopes, modality requirements, and single- and multi-turn settings. Experiments on representative open- and closed-source multimodal large language models (MLLMs) reveal substantial and uneven limitations across Ego–Action–World interactions. Models trained on EgoOmniWorld consistently improve in-domain interaction understanding and generalize to existing egocentric benchmarks. We hope EgoOmniWorld provides a unified foundation for systematically evaluating and advancing egocentric video understanding. Data, code, and models will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Reasoning, Multi-Turn Question Answering, Action–Intention–Goal Reasoning, Embodied AI, Benchmark Evaluation, Response Consistency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_3eADOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EgoOmniWorld, a benchmark for egocentric video understanding organized around the Ego–Action–World interaction loop: the wearer perceives the world, forms goals, acts, observes the resulting change, and uses that feedback to adjust subsequent behavior. The loop is instantiated as three node types (Ego, Action, World) and six directed edges (Goal Planning, Intent Attribution, Action Effects, Action Grounding, Intent Formation, World Perception), composed into four structural scopes: Node (individual states), Edge (one directed relation), Chain (linked dependencies across relations), and Loop (feedback-closed cycles). Modality requirements (visual / auditory / audio-visual) and dialogue form (single- / multi-turn) are treated as consequences of the evidence and dependency structure rather than as independent task axes.

An interaction-first, evidence-grounded pipeline over Ego4D (1,000 cluster-stratified videos covering 159/160 narration clusters) and HD-EPIC extracts local interaction records, grounds them in visual and auditory evidence, aggregates them into episodes with dependency links, and only then generates QA under evidence-first constraints with generic quality checks, category-specific structural filtering, constrained repair, and revalidation. The result is 24,166 QA groups (20,284 train / 3,882 test): 41.5 / 39.4 / 10.3 / 8.9% Node / Edge / Chain / Loop; 46.9% visual-only, 15.4% audio-only, 37.7% audio-visual; 82.1% single-turn. Eleven omni-modal MLLMs are evaluated: the best closed-source model (Gemini 3.8 Flash) reaches 61.51%, the best open-source base model 48.15% (Qwen3-Omni-Thinking). Training on the training split raises Qwen2.5-Omni-7B from 36.42% to 47.36% (LoRA) and 48.34% (FFT), and transfers zero-shot to EgoSound (+8.08), EgoToM (+2.27), and EgoTempo (+4.40), although several EgoTempo categories drop by 6–12 points.

Critical strengths

S1. Ego–Action–World as a substantive organizing principle. It converts modality and dialogue form into consequences of the interaction process and derives enforceable construction constraints from this: a modality counts as required only if removing it removes information needed to answer (C.1), each level has its own structural filtering criteria (C.5), and constrained repair may not introduce facts outside the existing evidence pool (C.7). The payoff is Figure 5, where mapping EgoSound, EgoToM, and EgoTempo back onto this taxonomy shows that none of them covers Chain or Loop questions, i.e. that existing benchmarks do not evaluate cross-interaction dependencies or feedback-driven behavioral adjustment. That diagnosis is directly useful for deciding what the field should build next.

S2. Scale, evidence grounding, and usability as supervision. EgoOmniWorld combines scale (24,166 QA groups, 159/160 cluster coverage, 37.7% requiring joint audio-visual evidence, 17.9% multi-turn) with per-item grounding, retaining video and event identifiers, temporal intervals, evidence descriptions, and provenance mappings (Appendix G). Its evidence-first ordering — reconstruct structured interactions first, then generate questions against already-established evidence and target structure — avoids the failure mode of question-first pipelines that produce ungrounded items. Providing 20,284 training samples and demonstrating transfer to EgoSound (+8.08) makes it a source of supervision rather than only a test set.

Critical weaknesses

W1. Data correctness is validated only by LLMs, and the human review carries no numbers. Extraction, evidence grounding, question generation, validation, and repair are all performed by LLMs, and Appendix D reports neither how many samples were inspected nor the accept / needs revision / reject / uncertain distribution nor inter-annotator agreement, stating explicitly that review coverage is itself unknown. It is also unstated whether any of the 920 samples that "retain pending/rejected provenance" and were kept in the released data (24,166 = 23,246 + 920) entered the test split. Because the same model family generates the data, judges the answers, and is fine-tuned on the result, the procedure is closed on itself, and without an acceptance rate the claim that all 24,166 items are evidence-grounded cannot be checked externally; this is my main reason for not rating higher.

W2. All headline numbers rest on an unvalidated, same-family judge. The judge is a locally hosted, text-only Qwen3.8-27B with no access to the video, and the paper reports no agreement with human judgments (no accuracy, no κ) and no cross-check with a non-Qwen judge. The risk is concrete: four evaluated systems are Qwen-family models (Qwen2.5-Omni, Qwen3-Omni, Qwen3.5-Omni-Plus, Qwen3.8-Omni-Flash), so same-family preference would bias both the closed-source comparison and the training gains, all the more because reference answers are long open-ended descriptions where judges are known to be lenient. Until human–judge agreement and a cross-family correlation are reported, I cannot treat the absolute values in Tables 2 and 3 as reliable.

W3. Each of the two central empirical claims lacks the one control that would establish it. The modality-necessity claim has no behavioral support: C.6 describes the audits but reports no counts, and the direct test of audio-visual versus visual-only input on the A+V subset is missing, leaving "37.7% require joint audio-visual evidence" unverified. Likewise, the training gains (36.42 → 47.36 / 48.34) are in-domain gains on a test split produced by the same pipeline, and without the same LoRA run on a comparable volume of EgoTaskQA / EgoLifeQA / EgoSound data they cannot be distinguished from adaptation to answer style and templates. Two further issues reduce confidence in the reported numbers: the Qwen3-Omni-Thinking LoRA result uses an epoch-2 checkpoint while others appear to use later ones with no selection protocol, and per-category conclusions rest on very small test sets (Goal Planning, n = 40) with no confidence intervals.

Questions for authors

Q1 (W1). What are the quantitative results of the human review: how many samples were inspected, what fraction of the test set does this represent, and what is the distribution of accept / needs revision / reject / uncertain (with inter-annotator agreement if multiple reviewers were used)? Relatedly, are any of the 920 samples that "retain pending/rejected provenance" present in the test split, and is the train/test split disjoint at the video level or only at the clip level?

Q2 (W2). Can you report agreement (accuracy and κ) between the Qwen3.8-27B judge and human judgments on roughly 200 samples, and run a non-Qwen judge over the same batch to report the correlation between the two judges? If the two judges diverge materially in absolute accuracy, how should the conclusions of Tables 2 and 3 be revised?

Q3 (W3). Can you provide the two missing controls: (a) accuracy on the A+V subset with the audio channel removed, and (b) the same LoRA trained on a comparable volume of EgoTaskQA / EgoLifeQA / EgoSound training data and evaluated on the EgoOmniWorld test set, against the +10.94 reported here? If either control does not support the current claim, how would the paper's main conclusion change?

3 comments
AuthorsOpenReview ↗

Part1: Comment on W1&Q1

We thank the reviewer for the careful assessment. Below, we provide human-review outcomes, judge comparisons, modality and training controls, and clarifications of the evaluation protocol.

W1&Q1: How reliable is the data based on human review, sample provenance, and train–test separation?

We clarify the question-correctness audit, recovered-item provenance, and model families below.

(1) 40% of the test set receive human correctness review.

We manually reviewed 1,552 VQA samples, covering 40% of the test set; the training data were not manually reviewed. The review was conducted by a single annotator who was at least a master’s student. Each sample received one of two labels: Pass or Requires Revision. Of the reviewed samples, 1,463 (94.3%) passed without modification, while the remaining 89 (5.7%) contained errors or imperfections and were corrected using the annotation interface described in Appendix D.

(2) Fifteen of the 920 recovered items appear in the final test set.

Approximately 1,000 items were flagged as uncertain or rejected during the automated audit. Subsequent human review and revision recovered 920 items for inclusion in the released dataset. Their retained pending/rejected provenance records the earlier automated outcome.

Of these 920 items, 15 are included in the final test set, and 40% of the test set were covered by the full question-correctness audit, resulting in an accuracy of 94% .

We split the source videos into disjoint training and test sets before data generation, ensuring that no video contributes samples to both splits.

(3) Data generation and answer judging use different model families.

The data engine uses Gemini 3.1 Pro for multimodal information extraction and GPT-5.6 Terra for information integration, QA generation, and subsequent auditing. The answer judge is Qwen3.8-27B. This separation clarifies that generation and judging do not use the same family, although it does not independently establish data correctness or judge reliability.

AuthorsOpenReview ↗

Part2 : Comment on W2&Q2

W2&Q2: Does the text-only, same-family Qwen judge introduce bias in model comparisons and reported fine-tuning gains, and how well does it agree with humans and a non-Qwen judge?

We provide matched-response comparisons and additional training evidence beyond the Qwen family.

(1) SALMONN2 7B also benefits from training on our data.

We trained SALMONN2 7B for two epochs and evaluated it on the same test set.

SALMONN2 7B conditionMean score (1–5)Accuracy (%)Mean-score change vs. before trainingAccuracy change vs. before training (pp)
Before training2.05835.28——
After two epochs2.77643.66+0.718+8.38

This extends the observed training benefit beyond Qwen-family models

(2) Judges from different model families and human evaluators produce similar scores and preserve the model ranking.

We sampled 500 questions and collected responses from Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Qwen3-Omni-Thinking. The same responses were evaluated by the judges below.

Each model column reports mean score (1–5) / accuracy (%).

JudgeQwen2.5-Omni-3B: score / accuracyQwen2.5-Omni-7B: score / accuracyQwen3-Omni-Thinking: score / accuracy
Qwen3.8-27B (original)1.91 / 30.22.34 / 35.82.56 / 37.6
GPT-6.1 Terra1.93 / 31.02.32 / 34.12.51 / 39.4
GPT-6 Sol1.92 / 30.62.38 / 33.92.49 / 38.5
Human assessment2.03 / 30.02.25 / 33.42.46 / 39.0

All settings yield:

Qwen3-Omni-Thinking > Qwen2.5-Omni-7B > Qwen2.5-Omni-3B.

(3) Adding video to the judge has a modest effect on aggregate scores.

We used Qwen3-Omni-Instruct, which can process both video and audio, to judge the responses of Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Qwen3-Omni-Thinking. The judge assigned similar scores when given only the question and answer text and when additionally given the original video with its audio, suggesting that the text-only setting is sufficient for scoring in this experiment.

JudgeInput available to judgeQwen2.5-Omni-3B: score / accuracyQwen2.5-Omni-7B: score / accuracyQwen3-Omni-Thinking: score / accuracy
Qwen3-Omni-InstructText only1.98 / 29.42.29 / 34.62.48 / 38.2
Qwen3-Omni-InstructText + video2.03 / 31.22.35 / 35.02.52 / 38.8
Evaluated modelMean-score change from adding video to Qwen3-Omni-InstructAccuracy change from adding video (pp)
Qwen2.5-Omni-3B+0.05+1.8
Qwen2.5-Omni-7B+0.06+0.4
Qwen3-Omni-Thinking+0.04+0.6
AuthorsOpenReview ↗

Part3: Comment on W3&Q3

W3&Q3: Do audio ablation and matched external-data training support the main claims?

(1) Adding audio improves accuracy by 10.30 pp on the A+V subset.

We evaluated Qwen2.5-Omni-7B on the A+V subset with and without audio.

Input conditionaccuracy (%)
Video only27.83
Audio + video38.13
Improvement from adding audio+10.30 pp

The higher performance with audio supports its contribution on this subset. This aggregate comparison does not establish that every annotated A+V question requires both modalities or independently validate the exact 37.7% prevalence claim.

(2) Training gains extend beyond the in-domain test split, while the external-data control shows negative transfer.

As reported in Table 3, training on EgoOmniWorld improves zero-shot performance on EgoSound, EgoToM, and EgoTempo, providing evidence of transfer beyond the generated test split.

We further fine-tuned Qwen2.5-Omni-7B on 20k EgoTaskQA training examples for two epochs, using the same training configuration as our LoRA experiment. Both models were evaluated on the same EgoOmniWorld test set.

Training conditionAccuracy (%)Accuracy change vs. base (pp)
qwen2.5omni-7b trained on EgoOmniWorld trainning set 20k data47.36+10.94
qwen2.5omni-7b trained on EgoTaskQA 20k data29.86−6.56
qwen2.5omni-7b wo finetune36.42—

Under the tested configuration, EgoOmniWorld supervision yields substantially higher accuracy than the EgoTaskQA control. This shows that the observed gains are not reproduced by an arbitrary addition of egocentric QA supervision, although differences in data distribution and answer style remain possible contributors.

(3) All trained models use the epoch-2 checkpoint.

All trained models, including Qwen3-Omni-Thinking, are evaluated using the checkpoint at the end of epoch 2. The apparent discrepancy arose from incomplete reporting of this common rule.

Revision. We have explicitly stated the checkpoint-selection protocol in Section 4.1 (“Experimental Setup”).

(4) Goal Planning accuracy averages 30%, with a reported 95% interval of 27.1%–32.9%.

Goal Planning contains relatively few items because future behavior must be sufficiently predictable from the available evidence without making the answer trivial. We repeated the Qwen2.5 evaluation five times on this category.

Question categoryNumber of runsMean accuracy (%)Reported 95% confidence interval (%)
Goal Planning53027.1%~32.9%
Reviewer_U9UjOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

EgoOmniWorld organizes egocentric video understanding around Ego–Action–World interactions, using Node, Edge, Chain, and Loop to describe different kinds of interaction structure. The authors build 24,166 QA groups from Ego4D and HD-EPIC, evaluate a range of multimodal models, and use the data for fine-tuning. The trained models improve on EgoOmniWorld, and the Qwen2.5-Omni model also shows positive aggregate transfer to three existing benchmarks.

Critical strengths
  1. The distinction between Chain and Loop is useful. Following a sequence of dependencies and recognizing feedback that changes later behavior are different things to test, and the taxonomy gives these questions a clearer place in the evaluation. Retaining evidence intervals and provenance helps connect the categories to the underlying video.

  2. The training results add practical value. There are substantial gains on the new benchmark, and the positive aggregate transfer to EgoSound, EgoToM, and EgoTempo suggests that the data are useful beyond their own test set. The category breakdowns are also worth keeping: they show that some abilities improve while others do not.

Critical weaknesses
  1. The central claim about composed and feedback-based reasoning needs a more direct test. The paper defines Chain and Loop carefully and describes structural checks, text-only shortcut tests, and modality audits. However, it does not report how model performance changes when required intermediate evidence or feedback is withheld, or compare the composed questions with their standalone constituents. Category-wise answer accuracy alone does not establish that a model must follow the annotated dependencies to answer correctly. A question might still be answered from a local cue or language prior. Without quantitative shortcut results or reported human-validation outcomes for these structures, the benchmark’s most distinctive capability claim remains uncertain.

  2. The fine-tuning comparison is mainly before versus after adding egocentric QA supervision. There is no size- and training-budget-matched baseline using conventional QA generated from the same source videos, so the benefit of the interaction-first construction is not isolated. The results also give mixed support for the distinctive structures: Qwen3-Omni-Thinking with LoRA improves overall but loses accuracy on both Chain and Loop (Table 2). Moreover, the three transfer benchmarks contain no explicit Chain or Loop questions under the authors’ mapping (Section 4.3). Their positive aggregate results therefore provide limited independent evidence for the composed and feedback reasoning that motivates this benchmark.

Questions for authors

How do Chain/Loop scores change when a required bridge event or feedback segment is removed, and how do they compare with performance on the standalone constituent questions? What were the quantitative outcomes of the shortcut and modality audits, and how often did human inspection confirm the assigned structures? Can a same-source, size- and training-budget-matched direct-QA baseline achieve comparable in-domain and transfer gains?

2 comments
AuthorsOpenReview ↗

Part1: Comment on W1&Q1

We thank the reviewer for the detailed comments. Below, we address dependency testing, training controls, LoRA configuration, and the scope of the transfer evidence.

W1 & Q1: Evidence dependence, compositional reasoning, modality audits, and structural validity of Chain/Loop questions

(1) Evidence dependence: Removing required evidence substantially reduces accuracy.

We randomly sampled 500 multi-turn QA groups and compared full-input inference with two ablated conditions: removal of the corresponding required video evidence and text-only inference. Changes are measured relative to full-input inference at the same turn.

Inference conditionTurn 1 accuracy (%)Turn 1 change (pp)Turn 2 accuracy (%)Turn 2 change (pp)Turn 3 accuracy (%)Turn 3 change (pp)
Full-input multi-turn inference42.28—41.40—40.27—
Required video evidence removed38.65−3.6334.78−6.6226.54−13.73
Text-only inference22.04−20.2423.80−17.6025.47−14.80

Text-only inference reduces accuracy at every turn, indicating that both earlier and later questions are difficult to answer without perceptual evidence. Removing specific visual evidence has a progressively larger effect on later turns: earlier questions depend on fewer of the removed evidence segments, whereas later questions additionally rely on earlier evidence through intermediate dependencies. This cumulative dependence is consistent with the increasing accuracy drop across turns.

(2) compositional reasoning: Standalone and combined formulations yield lower accuracy.

We further compared the original multi-turn questions with two alternative formulations: combined single-turn QA, which incorporates the first-turn conditions into the final question, and standalone constituent questions.

Question formulationTurn 1 accuracy (%)Turn 2 accuracy (%)Turn 3/final accuracy (%)Final-question accuracy change vs. original multi-turn QA (pp)
Original multi-turn QA42.2841.4040.27—
Combined single-turn QA——32.98−7.29
Standalone constituent questions42.2838.1335.80−4.47

Both alternative formulations yield lower final-question accuracy. Together, these results support the relevance of the annotated evidence and the utility of multi-turn context

(3) modality audits: Modality removal reduces accuracy

We randomly sampled 500 questions and compared accuracy before and after removing their annotated modality or modalities.

Annotated evidence modalityAccuracy before modality removal (%)Accuracy after modality removal (%)Accuracy drop (pp)
Video (V)40.7826.7913.99
Audio + video (A+V)44.7027.3217.38
Audio (A)35.7113.5322.18

These results support dependence on the annotated modalities. The text-only results above additionally show lower accuracy at every turn.

(4) structural validity: All final test samples have been manually reviewed and verified for structural correctness.

Human inspection of all 3,882 structural annotations confirmed 2,945 original assignments and corrected 937. We highlight that this construction manually check was done before the initial submission. These corrections concern the suitability of the assigned question structures.

Revision. In Appendix L (“Modality Dependence and Structural Annotation Audit”), we have reported both audit outcomes, clarified the removal procedure for each modality group, and distinguished corrections to structural annotations from corrections to question content.

AuthorsOpenReview ↗

Part2: Comment on W2&Q1

W2 & Q1: Matched conventional-QA controls, Chain/Loop degradation under LoRA, and Chain/Loop coverage and the value of structural supervision.

(1) Matched conventional-QA controls: EgoTaskQA fine-tuning reduces accuracy by 6.56 pp under the tested configuration.

We compared Qwen2.5-Omni-7B fine-tuned on approximately 20k EgoOmniWorld examples with the same model fine-tuned on 20k EgoTaskQA examples, using the same LoRA configuration and training for two epochs. All models were evaluated on the complete EgoOmniWorld test set.

Training conditionAccuracy (%)Change vs. base (pp)
Base model without fine-tuning36.42—
Fine-tuned on EgoOmniWorld (~20k examples) (ours)47.36+10.94
Fine-tuned on EgoTaskQA (20k examples)29.86−6.56

EgoOmniWorld fine-tuning improves accuracy by 10.94 percentage points, whereas EgoTaskQA fine-tuning reduces it by 6.56 points. This comparison shows that additional egocentric QA supervision does not automatically yield the observed gains and supports the effectiveness of EgoOmniWorld as a training resource.

Revision. We have added this control to Appendix K (“Alternative-Supervision Control with EgoTaskQA”) and clarified the scope of the conclusions it supports.

(2) Chain/Loop degradation under LoRA: The expanded LoRA configuration achieves consistent improvements on Chain and Loop types.

Thanks for your careful review. We clarify that the original LoRA configuration uses the LLaMA-Factory target setting all, rank 8, and scaling factor 16. To investigate whether adaptation capacity contributes to the better results, we retrained Qwen3-Omni-Thinking using rank 64, scaling factor 128 for the same 2 epochs.

Question structureBase Qwen3-Omni-Thinking ModelAccuracy after expanded LoRA (%) (Qwen3-Omni-Thinking -LoRA)
Chain53.2355.04
Loop55.4755.72
Node42.0249.35
Edge46.6854.88

With more trainable parameters in the expanded LoRA configuration, training improves accuracy over the base model across all four interaction structures, including Chain and Loop. This suggests that adaptation capacity may contribute to the declines observed under the original configuration. Moreover, Table 2 shows improvements across all four structures for Qwen2.5-Omni-7B under both LoRA and full fine-tuning, and for Qwen3-Omni-Thinking under full fine-tuning. Together, these results demonstrate that the Chain/Loop degradation is specific to certain adaptation settings and does not, by itself, indicate an inherent limitation of the EgoOmniWorld training data.

(3) Chain/Loop coverage and the value of structural supervision.

A. Distinctive coverage beyond existing benchmarks. The absence of explicit Chain and Loop questions in the three transfer benchmarks (EgoTempo, EgoToM, EgoSound) reflects their task scope rather than a limitation of EgoOmniWorld. Indeed, this coverage gap motivates our Ego–Action–World framework: EgoOmniWorld explicitly evaluates composed dependencies and feedback-based interactions that these benchmarks do not systematically assess. We therefore use the external benchmarks to evaluate broader transfer, rather than as direct validation of Chain/Loop reasoning.

B. In-domain gains on Chain and Loop. Table 2 directly evaluates training effectiveness on EgoOmniWorld’s Chain and Loop questions. Qwen2.5-Omni-7B improves on both structures under LoRA and full fine-tuning, while Qwen3-Omni-Thinking improves on both under full fine-tuning. Together with the expanded-LoRA results above, these findings support the value of EgoOmniWorld supervision for improving performance on its distinctive structures, while showing that gains depend on the adaptation configuration.

C. Value of Chain/Loop supervision for broader transfer. To examine whether Chain/Loop examples also benefit broader egocentric understanding, we trained Qwen2.5-Omni-7B with and without these examples and compared transfer performance. Below table shows that removing Chain/Loop examples reduces accuracy on all three benchmarks, suggesting that these examples contribute to broader transfer.

Training conditionEgoSound accuracy (%)EgoToM accuracy (%)EgoTempo accuracy (%)
Trained on full EgoOmniWorld data39.4158.2224.80
Trained without Chain/Loop examples38.6556.8322.97
Change from removing Chain/Loop (pp)−0.76−1.39−1.83
Reviewer_o8dYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the Ego–Action–World loop framework to systematically understand different aspects of egocentric scenes. EgoOmniWorld contains a total of 24,166 QA groups. The experimental results demonstrate that EgoOmniWorld covers diverse capability scopes and modality requirements, and that models fine-tuned on EgoOmniWorld exhibit zero-shot transfer capabilities on EgoSound, EgoTempo, and EgoToM.

Critical strengths
  1. The authors categorize Ego–Action–World interactions into Node, Edge, Chain, and Loop structures, and develop an interaction-egocentric data engine to structure these interactions into evidence-grounded VQA.

  2. Extensive in-domain and cross-benchmark experiments demonstrate that the EgoOmniWorld evaluation framework can systematically support the evaluation needs of egocentric video understanding. Its comprehensive question taxonomy can also help further diagnose model failures.

Critical weaknesses
  1. Human evaluations are missing to assess the difficulty of the evaluation questions, their correctness, and the upper bound of achievable performance. Given the five-point scoring scale, where scores of 4 and 5 are considered correct, would converting the results into accuracy better reflect performance differences between models on open-ended questions? Most models achieve mean scores between 2 and 3, while scores below 4 are considered incorrect. Does this indicate that the open-ended questions are too difficult for models to answer reliably?

  2. I notice that World→Ego and Action→Ego are categorized as Intent Formation and Intent Attribution, respectively. Do these two categories contain substantial overlap in terms of the intention-related questions they evaluate? For the Chain- and Loop-structured questions generated by the authors, would introducing a Chain-of-Thought (CoT) mechanism in multi-turn evaluation and extending the context length provide a more objective assessment of differences in model reasoning capabilities?

Questions for authors

QA pairs generated by LLM can easily suffer from answer leakage in the question wording. Have the authors conducted text-only evaluations to assess the quality of the question design and investigate the extent to which models can reason based solely on the textual context, without relying on visual or auditory evidence?

3 comments
AuthorsOpenReview ↗

Part1: Comment about W1

We thank the reviewer for the constructive feedback. Below, we clarify benchmark difficulty, the distinction between intention-related categories, multi-turn prompting, and dependence on perceptual evidence.

W1: Human review, human performance, and accuracy-based reporting clarify benchmark quality and difficulty

Thank you for raising these questions. We address question correctness, human performance, scoring metrics, and the effect of answer format separately below.

W1-Q1: Are the questions correct and answerable, and how well do humans perform?

Question correctness. We manually reviewed 40% of the test set and found that more than 90% of the reviewed VQA samples were correct as initially generated. We corrected erroneous or imperfect samples using the annotation interface described in Appendix D. These findings support the quality of the reviewed subset, while the corrections further improve its reliability for evaluation. Answerability and human performance. To assess answerability and provide a human-performance reference, we randomly sampled 500 questions and collected human answers, which were evaluated using the same metrics as model responses. Human participants achieved a mean score of 4.26/5 and 89.4% accuracy, indicating that they could successfully answer a substantial majority of the sampled questions. We have added these results to Table 2 in the revised manuscript.

Question structureHuman mean score (1–5)Human accuracy (%)
Node4.0984.56
Edge4.3390.36
Chain4.2788.58
Loop4.1385.12
Overall4.2689.4

W1-Q2: Would accuracy better reflect performance differences than mean scores?

We agree that accuracy offers a more direct measure of answer correctness. Accordingly, our original submission already reported both accuracy and mean score. Accuracy treats scores of 4–5 as correct, while the mean score on the 1–5 scale captures differences in partial correctness and completeness. The two metrics therefore provide complementary views of model performance. LLM-based scoring is a practical approach to evaluating open-ended responses and is also used in benchmarks such as EgoSound (Zhu et al., CVPR 2026) and EgoAVU (Seth et al., CVPR 2026). To assess whether scores of 4–5 reliably indicate correct answers, we randomly sampled 100 responses assigned these scores and manually reviewed them. 96% were judged correct, and the distinction between scores of 4 and 5 primarily reflected completeness rather than substantive correctness. For example, “To answer the phone” and “To answer the incoming phone call after hearing it ring” both correctly answer “Why does the person stop vacuuming?”, although the latter provides additional supporting detail. These findings support treating scores of 4–5 as correct.

W1-Q3: Do mean scores between 2 and 3 indicate that the open-ended questions are too difficult?

Thank you for raising this concern. Mean scores between 2 and 3 indicate that current models frequently produce incorrect answers, but do not by themselves imply that the questions are excessively difficult or unanswerable. Human participants achieve a mean score of 4.26/5 and 89.4% accuracy, indicating that the sampled questions are largely answerable. Moreover, model performance varies substantially across subcategories, with cross-model mean scores ranging from 1.95 to 3.87 and accuracies ranging from 22.99 to 67.54, revealing uneven capabilities rather than uniformly low performance.

Question subcategoryAbbreviationCross-model mean score (1–5)Cross-model Acc
Action → IntentAI3.8767.54%
Ego StateES3.0156.10%
State → ActionSA2.9254.94%
World → IntentWI2.6753.50%
World StateWS2.4045.96%
Goal PredictionGP2.2741.25%
Action → EffectAE2.1735.44%
Action DescriptionAD1.9629.49%
Egocentric PerceptionEP1.9522.99%

To examine the effect of answer format, we additionally converted 500 randomly sampled questions into multiple-choice questions and evaluated Qwen2.5-Omni-7B on the same items under both formats.

Evaluation formatAccuracy (%)
Open-ended QA37.4
Multiple-choice QA35.8

Converting the questions to multiple choice did not improve accuracy in this experiment. Together with the strong human performance and variation in model performance across subcategories, these results suggest that the open-ended format alone does not explain the low scores. Rather, they highlight substantial room for improvement in models’ egocentric understanding capabilities.

AuthorsOpenReview ↗

Part2: Comment about W2

W2: Overlap between intention categories and the effects of CoT prompting and context length on reasoning evaluation

W2-Q1: Do Intent Formation and Intent Attribution substantially overlap?

Thank you for this question. Although both categories concern intentions and may involve related intent content, they evaluate distinct reasoning relations. We clarify this distinction through their definitions and an empirical classification study.

(1) Distinct definitions and reasoning targets.

World → Ego (Intent Formation/Adjustment) examines how an external world state or event leads the agent to form, revise, or redirect an intention. In contrast, Action → Ego (Intent Attribution) infers an underlying intention from the agent’s observed actions.

For example:

World → Ego: “While vacuuming, I hear my phone ringing. How does this change my original plan to continue cleaning?”

Answer: “I temporarily interrupt cleaning and prepare to answer the phone.”

Action → Ego: “I turn off the vacuum cleaner, pick up my phone, and tap the answer button. What does this sequence of actions indicate that I intend to do?”

Answer: “I intend to answer the incoming call and speak with the caller.”

The first question examines an intention change triggered by an external event; the second asks what intention is indicated by an observed action sequence. Importantly, intention changes in World → Ego questions must also be supported by subsequent behavioral or verbal evidence, rather than inferred from the external event alone. Thus, the distinction lies in the relation being evaluated, even when the underlying intention is similar.

(2) Empirical distinguishability. To quantify whether this distinction can be consistently recognized, we evaluated both an LLM classifier and human classifiers on their ability to assign questions to the intended category.

Classification methodCategory classification accuracy (%)
LLM classifier (Qwen3.8)94.90
Human classifiers98.1

The high classification accuracies support the operational distinction between the two categories, while allowing that individual examples may involve related intentions.

W2-Q2: Would CoT prompting and longer context enable a more objective assessment of Chain and Loop reasoning?

Thank you for this suggestion. To examine the effect of explicit CoT prompting, we randomly sampled 500 multi-turn QA groups and evaluated three models under standard and CoT prompting conditions.

ModelAccuracy with standard prompting (%)Accuracy with CoT prompting (%)Change (pp)
Qwen2.5-Omni-3B37.8835.43−2.45
Qwen2.5-Omni-7B42.4641.00−1.46
Qwen3-Omni-Thinking54.2156.21+2.00

Explicit CoT prompting improves Qwen3-Omni-Thinking’s accuracy by 2.00 percentage points, while reducing accuracy for the two Qwen2.5-Omni models by 1.46–2.45 points. These results show that explicit CoT prompting does not consistently improve performance across the evaluated models. One possible explanation is that step-by-step prompting alone may be insufficient for models to identify and integrate the evidence required to resolve the dependencies and feedback relations captured by our Chain and Loop questions, which are grounded in Ego–Action–World interaction sequences.

Importantly, the model ranking remains unchanged, supporting the stability of the relative comparison across these two prompting conditions on the sampled subset.

AuthorsOpenReview ↗

Part3: Comment on Q1

Q1: Potential Answer Leakage and Reliance on Textual Context

Thank you for raising this concern. We address it through both leakage checks during data construction and an additional text-only evaluation.

(1) Leakage checks during data construction. Our data engine includes a text-only screening step (Sec. 3.2, Step 4). A text-only model is given the question and its reference answer and asked to assess whether the answer can be inferred from the textual information alone, without access to the video or audio. This screening aims to identify questions that may be answerable without perceptual evidence.

(2) Text-only evaluation. To assess how much performance can be explained by textual context alone, we randomly selected 500 qa and additionally evaluated Qwen2.5-Omni-7B under audio-visual and text-only input conditions.

Input conditionAccuracy (%)Change vs. audio-visual (pp)
Audio-visual39.4—
Text only17.2−22.2

Removing audio-visual evidence reduces accuracy by 22.2 percentage points. This substantial gap supports the importance of perceptual evidence and indicates that textual cues alone are insufficient for many questions.

Revision. We have added these results to Appendix J (“Text-Only Evaluation and Answer Leakage”) and clarified the complementary roles of blind testing and leakage review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.