- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MultiView-EB, a benchmark testing whether MLLMs can combine synchronized egocentric and exocentric video of real human activities. It is built from Ego-Exo4D and ACE-Data-0 with a semi-automatic pipeline and human verification, and contains 2,497 questions from 658 takes across eight tasks in three groups:
- cross-view perception: object, HOI and task correspondence;
- planning: next-step and long-horizon, 4-way multiple choice;
- hand-trajectory prediction: grounding, goal-conditioned and instruction-conditioned.
Outputs include time-stamped boxes, intervals and 2D waypoints, scored with geometric metrics.
The authors evaluate 13 general MLLMs and 8 embodied-brain models.
- GPT-6-astra leads every task, but embodied-brain models largely fail at cross-view grounding.
- In a matched ego-only vs. ego+exo ablation on planning, adding the exocentric frame improves accuracy by only +2.1 / +0.5 points on average and hurts four of eight models. This holds even though items were filtered so that the decisive state is visible only exocentrically.
- Failure analysis attributes planning errors mainly to "progress miscalibration", and correspondence errors to binding rather than tracking.
- Fine-tuning Qwen3.5-4B on a 38K corpus (MV-EmData) improves all three capability dimensions.
Critical strengths
- A clean, measurable research question. The matched ego-only vs. ego+exo protocol with paired McNemar tests (Table 3) makes "does the second view help?" directly testable.
- Diagnostic task design. Factorial planning options (correct action and location, same action at the wrong location, two action distractors) separate action errors from location errors. This yields a concrete finding: next-step errors repeat completed steps, while long-horizon errors skip prerequisites (Sec. 4.4).
- Grounded outputs. Boxes, intervals and waypoints with geometric metrics, rather than multiple choice only. The binding-vs-tracking probe on InternVL3.5-38B (0.66 → 0.38 → 0.016) is a useful diagnostic.
- Broad, current model coverage (21 models) and a detailed appendix: prompts, construction funnel, decoding configuration.
Critical weaknesses
- The small exo gain does not isolate "failure to use exo" (Q1). The paper attributes the +2.1 / +0.5 gain to models failing to exploit evidence that, by construction, is visible only exocentrically (Sec. 4.3). Yet ego-only accuracy is already high (e.g., GPT-6-astra 0.535 / 0.750 vs. 0.25 chance; Table 3), so many items appear solvable without the exo frame. A small Δ is then expected even for a model that uses exo well, and the necessity filter that would rule this out is not validated. 2 No ego-only evaluation of the fine-tuned model (Q2). The ego-only vs. ego+exo ablation covers only eight zero-shot general models (Table 3); MV-Emb is reported only under ego+exo (Table 16). So it is unknown whether fine-tuning on MV-EmData teaches the model to use exocentric evidence, or simply improves ego-side planning and output format.
Questions for authors
- What is Δ(ego+exo − ego) restricted to items each model answers wrongly ego-only, or to items whose multi-view necessity has been verified by humans? How precise is the necessity check?
- What are the ego-only and ego+exo planning accuracies of Qwen3.5-4B-MV-Emb? Does fine-tuning increase the exocentric gain?