- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EgoOmniWorld, a benchmark for egocentric video understanding organized around the Ego–Action–World interaction loop: the wearer perceives the world, forms goals, acts, observes the resulting change, and uses that feedback to adjust subsequent behavior. The loop is instantiated as three node types (Ego, Action, World) and six directed edges (Goal Planning, Intent Attribution, Action Effects, Action Grounding, Intent Formation, World Perception), composed into four structural scopes: Node (individual states), Edge (one directed relation), Chain (linked dependencies across relations), and Loop (feedback-closed cycles). Modality requirements (visual / auditory / audio-visual) and dialogue form (single- / multi-turn) are treated as consequences of the evidence and dependency structure rather than as independent task axes.
An interaction-first, evidence-grounded pipeline over Ego4D (1,000 cluster-stratified videos covering 159/160 narration clusters) and HD-EPIC extracts local interaction records, grounds them in visual and auditory evidence, aggregates them into episodes with dependency links, and only then generates QA under evidence-first constraints with generic quality checks, category-specific structural filtering, constrained repair, and revalidation. The result is 24,166 QA groups (20,284 train / 3,882 test): 41.5 / 39.4 / 10.3 / 8.9% Node / Edge / Chain / Loop; 46.9% visual-only, 15.4% audio-only, 37.7% audio-visual; 82.1% single-turn. Eleven omni-modal MLLMs are evaluated: the best closed-source model (Gemini 3.8 Flash) reaches 61.51%, the best open-source base model 48.15% (Qwen3-Omni-Thinking). Training on the training split raises Qwen2.5-Omni-7B from 36.42% to 47.36% (LoRA) and 48.34% (FFT), and transfers zero-shot to EgoSound (+8.08), EgoToM (+2.27), and EgoTempo (+4.40), although several EgoTempo categories drop by 6–12 points.
Critical strengths
S1. Ego–Action–World as a substantive organizing principle. It converts modality and dialogue form into consequences of the interaction process and derives enforceable construction constraints from this: a modality counts as required only if removing it removes information needed to answer (C.1), each level has its own structural filtering criteria (C.5), and constrained repair may not introduce facts outside the existing evidence pool (C.7). The payoff is Figure 5, where mapping EgoSound, EgoToM, and EgoTempo back onto this taxonomy shows that none of them covers Chain or Loop questions, i.e. that existing benchmarks do not evaluate cross-interaction dependencies or feedback-driven behavioral adjustment. That diagnosis is directly useful for deciding what the field should build next.
S2. Scale, evidence grounding, and usability as supervision. EgoOmniWorld combines scale (24,166 QA groups, 159/160 cluster coverage, 37.7% requiring joint audio-visual evidence, 17.9% multi-turn) with per-item grounding, retaining video and event identifiers, temporal intervals, evidence descriptions, and provenance mappings (Appendix G). Its evidence-first ordering — reconstruct structured interactions first, then generate questions against already-established evidence and target structure — avoids the failure mode of question-first pipelines that produce ungrounded items. Providing 20,284 training samples and demonstrating transfer to EgoSound (+8.08) makes it a source of supervision rather than only a test set.
Critical weaknesses
W1. Data correctness is validated only by LLMs, and the human review carries no numbers. Extraction, evidence grounding, question generation, validation, and repair are all performed by LLMs, and Appendix D reports neither how many samples were inspected nor the accept / needs revision / reject / uncertain distribution nor inter-annotator agreement, stating explicitly that review coverage is itself unknown. It is also unstated whether any of the 920 samples that "retain pending/rejected provenance" and were kept in the released data (24,166 = 23,246 + 920) entered the test split. Because the same model family generates the data, judges the answers, and is fine-tuned on the result, the procedure is closed on itself, and without an acceptance rate the claim that all 24,166 items are evidence-grounded cannot be checked externally; this is my main reason for not rating higher.
W2. All headline numbers rest on an unvalidated, same-family judge. The judge is a locally hosted, text-only Qwen3.8-27B with no access to the video, and the paper reports no agreement with human judgments (no accuracy, no κ) and no cross-check with a non-Qwen judge. The risk is concrete: four evaluated systems are Qwen-family models (Qwen2.5-Omni, Qwen3-Omni, Qwen3.5-Omni-Plus, Qwen3.8-Omni-Flash), so same-family preference would bias both the closed-source comparison and the training gains, all the more because reference answers are long open-ended descriptions where judges are known to be lenient. Until human–judge agreement and a cross-family correlation are reported, I cannot treat the absolute values in Tables 2 and 3 as reliable.
W3. Each of the two central empirical claims lacks the one control that would establish it. The modality-necessity claim has no behavioral support: C.6 describes the audits but reports no counts, and the direct test of audio-visual versus visual-only input on the A+V subset is missing, leaving "37.7% require joint audio-visual evidence" unverified. Likewise, the training gains (36.42 → 47.36 / 48.34) are in-domain gains on a test split produced by the same pipeline, and without the same LoRA run on a comparable volume of EgoTaskQA / EgoLifeQA / EgoSound data they cannot be distinguished from adaptation to answer style and templates. Two further issues reduce confidence in the reported numbers: the Qwen3-Omni-Thinking LoRA result uses an epoch-2 checkpoint while others appear to use later ones with no selection protocol, and per-category conclusions rest on very small test sets (Goal Planning, n = 40) with no confidence intervals.
Questions for authors
Q1 (W1). What are the quantitative results of the human review: how many samples were inspected, what fraction of the test set does this represent, and what is the distribution of accept / needs revision / reject / uncertain (with inter-annotator agreement if multiple reviewers were used)? Relatedly, are any of the 920 samples that "retain pending/rejected provenance" present in the test split, and is the train/test split disjoint at the video level or only at the clip level?
Q2 (W2). Can you report agreement (accuracy and κ) between the Qwen3.8-27B judge and human judgments on roughly 200 samples, and run a non-Qwen judge over the same batch to report the correlation between the two judges? If the two judges diverge materially in absolute accuracy, how should the conclusions of Tables 2 and 3 be revised?
Q3 (W3). Can you provide the two missing controls: (a) accuracy on the A+V subset with the audio channel removed, and (b) the same LoRA trained on a comparable volume of EgoTaskQA / EgoLifeQA / EgoSound training data and evaluated on the EgoOmniWorld test set, against the +10.94 reported here? If either control does not support the current claim, how would the paper's main conclusion change?
3 comments
Part2 : Comment on W2&Q2
W2&Q2: Does the text-only, same-family Qwen judge introduce bias in model comparisons and reported fine-tuning gains, and how well does it agree with humans and a non-Qwen judge?
We provide matched-response comparisons and additional training evidence beyond the Qwen family.
(1) SALMONN2 7B also benefits from training on our data.
We trained SALMONN2 7B for two epochs and evaluated it on the same test set.
| SALMONN2 7B condition | Mean score (1–5) | Accuracy (%) | Mean-score change vs. before training | Accuracy change vs. before training (pp) |
|---|---|---|---|---|
| Before training | 2.058 | 35.28 | — | — |
| After two epochs | 2.776 | 43.66 | +0.718 | +8.38 |
This extends the observed training benefit beyond Qwen-family models
(2) Judges from different model families and human evaluators produce similar scores and preserve the model ranking.
We sampled 500 questions and collected responses from Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Qwen3-Omni-Thinking. The same responses were evaluated by the judges below.
Each model column reports mean score (1–5) / accuracy (%).
| Judge | Qwen2.5-Omni-3B: score / accuracy | Qwen2.5-Omni-7B: score / accuracy | Qwen3-Omni-Thinking: score / accuracy |
|---|---|---|---|
| Qwen3.8-27B (original) | 1.91 / 30.2 | 2.34 / 35.8 | 2.56 / 37.6 |
| GPT-6.1 Terra | 1.93 / 31.0 | 2.32 / 34.1 | 2.51 / 39.4 |
| GPT-6 Sol | 1.92 / 30.6 | 2.38 / 33.9 | 2.49 / 38.5 |
| Human assessment | 2.03 / 30.0 | 2.25 / 33.4 | 2.46 / 39.0 |
All settings yield:
Qwen3-Omni-Thinking > Qwen2.5-Omni-7B > Qwen2.5-Omni-3B.
(3) Adding video to the judge has a modest effect on aggregate scores.
We used Qwen3-Omni-Instruct, which can process both video and audio, to judge the responses of Qwen2.5-Omni-3B, Qwen2.5-Omni-7B, and Qwen3-Omni-Thinking. The judge assigned similar scores when given only the question and answer text and when additionally given the original video with its audio, suggesting that the text-only setting is sufficient for scoring in this experiment.
| Judge | Input available to judge | Qwen2.5-Omni-3B: score / accuracy | Qwen2.5-Omni-7B: score / accuracy | Qwen3-Omni-Thinking: score / accuracy |
|---|---|---|---|---|
| Qwen3-Omni-Instruct | Text only | 1.98 / 29.4 | 2.29 / 34.6 | 2.48 / 38.2 |
| Qwen3-Omni-Instruct | Text + video | 2.03 / 31.2 | 2.35 / 35.0 | 2.52 / 38.8 |
| Evaluated model | Mean-score change from adding video to Qwen3-Omni-Instruct | Accuracy change from adding video (pp) |
|---|---|---|
| Qwen2.5-Omni-3B | +0.05 | +1.8 |
| Qwen2.5-Omni-7B | +0.06 | +0.4 |
| Qwen3-Omni-Thinking | +0.04 | +0.6 |
Part3: Comment on W3&Q3
W3&Q3: Do audio ablation and matched external-data training support the main claims?
(1) Adding audio improves accuracy by 10.30 pp on the A+V subset.
We evaluated Qwen2.5-Omni-7B on the A+V subset with and without audio.
| Input condition | accuracy (%) |
|---|---|
| Video only | 27.83 |
| Audio + video | 38.13 |
| Improvement from adding audio | +10.30 pp |
The higher performance with audio supports its contribution on this subset. This aggregate comparison does not establish that every annotated A+V question requires both modalities or independently validate the exact 37.7% prevalence claim.
(2) Training gains extend beyond the in-domain test split, while the external-data control shows negative transfer.
As reported in Table 3, training on EgoOmniWorld improves zero-shot performance on EgoSound, EgoToM, and EgoTempo, providing evidence of transfer beyond the generated test split.
We further fine-tuned Qwen2.5-Omni-7B on 20k EgoTaskQA training examples for two epochs, using the same training configuration as our LoRA experiment. Both models were evaluated on the same EgoOmniWorld test set.
| Training condition | Accuracy (%) | Accuracy change vs. base (pp) |
|---|---|---|
| qwen2.5omni-7b trained on EgoOmniWorld trainning set 20k data | 47.36 | +10.94 |
| qwen2.5omni-7b trained on EgoTaskQA 20k data | 29.86 | −6.56 |
| qwen2.5omni-7b wo finetune | 36.42 | — |
Under the tested configuration, EgoOmniWorld supervision yields substantially higher accuracy than the EgoTaskQA control. This shows that the observed gains are not reproduced by an arbitrary addition of egocentric QA supervision, although differences in data distribution and answer style remain possible contributors.
(3) All trained models use the epoch-2 checkpoint.
All trained models, including Qwen3-Omni-Thinking, are evaluated using the checkpoint at the end of epoch 2. The apparent discrepancy arose from incomplete reporting of this common rule.
Revision. We have explicitly stated the checkpoint-selection protocol in Section 4.1 (“Experimental Setup”).
(4) Goal Planning accuracy averages 30%, with a reported 95% interval of 27.1%–32.9%.
Goal Planning contains relatively few items because future behavior must be sufficiently predictable from the available evidence without making the answer trivial. We repeated the Qwen2.5 evaluation five times on this category.
| Question category | Number of runs | Mean accuracy (%) | Reported 95% confidence interval (%) |
|---|---|---|---|
| Goal Planning | 5 | 30 | 27.1%~32.9% |
Part1: Comment on W1&Q1
We thank the reviewer for the careful assessment. Below, we provide human-review outcomes, judge comparisons, modality and training controls, and clarifications of the evaluation protocol.
W1&Q1: How reliable is the data based on human review, sample provenance, and train–test separation?
We clarify the question-correctness audit, recovered-item provenance, and model families below.
(1) 40% of the test set receive human correctness review.
We manually reviewed 1,552 VQA samples, covering 40% of the test set; the training data were not manually reviewed. The review was conducted by a single annotator who was at least a master’s student. Each sample received one of two labels: Pass or Requires Revision. Of the reviewed samples, 1,463 (94.3%) passed without modification, while the remaining 89 (5.7%) contained errors or imperfections and were corrected using the annotation interface described in Appendix D.
(2) Fifteen of the 920 recovered items appear in the final test set.
Approximately 1,000 items were flagged as uncertain or rejected during the automated audit. Subsequent human review and revision recovered 920 items for inclusion in the released dataset. Their retained pending/rejected provenance records the earlier automated outcome.
Of these 920 items, 15 are included in the final test set, and 40% of the test set were covered by the full question-correctness audit, resulting in an accuracy of 94% .
We split the source videos into disjoint training and test sets before data generation, ensuring that no video contributes samples to both splits.
(3) Data generation and answer judging use different model families.
The data engine uses Gemini 3.1 Pro for multimodal information extraction and GPT-5.6 Terra for information integration, QA generation, and subsequent auditing. The answer judge is Qwen3.8-27B. This separation clarifies that generation and judging do not use the same family, although it does not independently establish data correctness or judge reliability.