acceptodds
Withdrawn from ICLR 2027

MultiView-EB: Benchmarking Ego–Exo Embodied Understanding of MLLMs in Real-World Scenarios

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied agents rely on egocentric observations for perception and decision-making, yet limited fields of view and frequent occlusions can conceal task-relevant objects, state changes, and spatial relations. Synchronized exocentric views offer complementary evidence, but existing benchmarks focus on single-view reasoning or static cross-view perception. Whether multimodal large language models can integrate both views over time to improve egocentric decisions remains unclear. We introduce MultiView-EB, a benchmark for cross-view spatiotemporal understanding in real-world embodied activities. Built from synchronized recordings of human activities, it contains 2,497 samples across nine activity domains, obtained through semi-automated annotation and human verification. Eight tasks assess cross-view fine-grained perception, task planning, and trajectory prediction through open-ended, multiple-choice, and spatial prediction outputs. We evaluate thirteen general-purpose MLLMs and eight embodied-brain models and analyze performance differences across egocentric and combined-view inputs. On the planning tasks, where input conditions can be matched, adding an exocentric view yields an average gain of only 0.5–2.1 points over egocentric input and degrades accuracy for four of the eight models tested. Additional views thus do not automatically improve embodied reasoning: current models struggle to establish cross-view correspondences, align temporal dynamics, and selectively use complementary evidence for planning. Finally, we introduce MV-EmData-38K, on which fine-tuning improves performance on all eight tasks. MultiView-EB provides a systematic testbed for embodied models capable of reasoning across views.

Primary area:
datasets and benchmarks
Keywords:
large vision language models, multi-view understanding, embodied understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rbfFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MultiView-EB, a benchmark testing whether MLLMs can combine synchronized egocentric and exocentric video of real human activities. It is built from Ego-Exo4D and ACE-Data-0 with a semi-automatic pipeline and human verification, and contains 2,497 questions from 658 takes across eight tasks in three groups:

  • cross-view perception: object, HOI and task correspondence;
  • planning: next-step and long-horizon, 4-way multiple choice;
  • hand-trajectory prediction: grounding, goal-conditioned and instruction-conditioned.

Outputs include time-stamped boxes, intervals and 2D waypoints, scored with geometric metrics.

The authors evaluate 13 general MLLMs and 8 embodied-brain models.

  • GPT-6-astra leads every task, but embodied-brain models largely fail at cross-view grounding.
  • In a matched ego-only vs. ego+exo ablation on planning, adding the exocentric frame improves accuracy by only +2.1 / +0.5 points on average and hurts four of eight models. This holds even though items were filtered so that the decisive state is visible only exocentrically.
  • Failure analysis attributes planning errors mainly to "progress miscalibration", and correspondence errors to binding rather than tracking.
  • Fine-tuning Qwen3.5-4B on a 38K corpus (MV-EmData) improves all three capability dimensions.
Critical strengths
  1. A clean, measurable research question. The matched ego-only vs. ego+exo protocol with paired McNemar tests (Table 3) makes "does the second view help?" directly testable.
  2. Diagnostic task design. Factorial planning options (correct action and location, same action at the wrong location, two action distractors) separate action errors from location errors. This yields a concrete finding: next-step errors repeat completed steps, while long-horizon errors skip prerequisites (Sec. 4.4).
  3. Grounded outputs. Boxes, intervals and waypoints with geometric metrics, rather than multiple choice only. The binding-vs-tracking probe on InternVL3.5-38B (0.66 → 0.38 → 0.016) is a useful diagnostic.
  4. Broad, current model coverage (21 models) and a detailed appendix: prompts, construction funnel, decoding configuration.
Critical weaknesses
  1. The small exo gain does not isolate "failure to use exo" (Q1). The paper attributes the +2.1 / +0.5 gain to models failing to exploit evidence that, by construction, is visible only exocentrically (Sec. 4.3). Yet ego-only accuracy is already high (e.g., GPT-6-astra 0.535 / 0.750 vs. 0.25 chance; Table 3), so many items appear solvable without the exo frame. A small Δ is then expected even for a model that uses exo well, and the necessity filter that would rule this out is not validated. 2 No ego-only evaluation of the fine-tuned model (Q2). The ego-only vs. ego+exo ablation covers only eight zero-shot general models (Table 3); MV-Emb is reported only under ego+exo (Table 16). So it is unknown whether fine-tuning on MV-EmData teaches the model to use exocentric evidence, or simply improves ego-side planning and output format.
Questions for authors
  1. What is Δ(ego+exo − ego) restricted to items each model answers wrongly ego-only, or to items whose multi-view necessity has been verified by humans? How precise is the necessity check?
  2. What are the ego-only and ego+exo planning accuracies of Qwen3.5-4B-MV-Emb? Does fine-tuning increase the exocentric gain?
Reviewer_re1bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a new synchronized ego-exo video benchmark for evaluating cross-view embodied understanding. It contains 2,497 question–answer pairs across eight tasks covering cross-view perception, planning, and trajectory grounding/prediction. The authors evaluate multiple general-purpose (both open-sourced and Proprietary) and other embodied models, and analyse the effect of adding exocentric observations to egocentric planning inputs. In addition, a new fine-tuning dataset MV-EmData-38K is introduced, demonstrating improvements from fine-tuning Qwen3.5-4B.

Critical strengths
  1. The proposed benchmark combines semantic decisions with bounding boxes, temporal intervals, and hand trajectories. Those modalties are often missing in most existing ego-exo benchmarks. The authors also provide a broader evaluation than multiple-choice questions alone.

  2. The authors include comparison of single-view input vs multi-view input on Planning task, and add paired significance tests, and task-specific failure analyses. Those are helpful for understanding MLLM's current shortcomings in tackling ego-exo understanding tasks.

  3. State-of-the-art Proprietary models like Claude-Sonnet-5, GPT-6-astra, and Gemini-3.8-flash are evaluated, demonstrating superior performance than general open-sourced and embodied-brain models.

Critical weaknesses
  1. The necessity of joint ego–exo reasoning is not sufficiently established for the planning tasks. The “multi-view necessity” check verifies that a decisive state is invisible egocentrically but visible exocentrically. However, this does not establish that both views are necessary: the exocentric frame, together with the goal and answer options, might already suffice. Table 3 compares ego-only with ego+exo but does not include an exo-only baseline. Moreover, the strongest model achieves 75.0% long-horizon accuracy under ego-only input, suggesting that visual invisibility does not necessarily imply that the answer cannot be inferred from other cues.

  2. The viewpoint ablation does not adequately distinguish failure to use exocentric evidence from failure to interpret or integrate it. Section 4.3 concludes that the exocentric frame is “largely unused,” but a small aggregate accuracy gain does not uniquely support this interpretation. Models might ignore the frame, misread its contents, fail to align it with the egocentric history, or use it in ways that help some samples while harming others. The existing failure analyses are informative but do not resolve these alternatives.

  3. Near-floor spatial scores have limited diagnostic value without more systematic performance decomposition. Several open models achieve near-zero task-correspondence box F1, and in some cases larger model performs worse than smaller models from the same family. This might not guide the development of open-sourced MLLMs. In addition, it is not very clear whether the low performance is the shortcomings of open-sourced MLLMs, or the task difficulty. Reporting format-validity rates, conditional performance on valid outputs, and separate semantic and spatial measures would improve interpretability.

  4. Related to Q3, Human performance under the same observable inputs, together with inter-annotator agreement, would also better calibrate task difficulty; annotation audits alone do not establish a human performance reference.

  5. The fine-tuning results do not yet establish improved cross-view integration or broader transfer. Improvements on tasks sharing the training corpus’s input structure and answer schemas demonstrate useful adaptation, but they do not isolate whether the model has become better at exploiting complementary views. (i) Repeating the ego-only/exo-only/ego+exo comparison after fine-tuning would directly test this claim. (ii) reporting the generalization performance on other ego-exo benchmarks might bring more insights.

  6. Minor presentation issues: Figure 1 uses very small text and does not consistently distinguish a single input image from a representative frame of a video. Explicit modality labels would make the task settings clearer. Figure 5 is cluttered by full model names and crossing label connectors; retaining all data points while selectively labeling representative models would improve readability.

Questions for authors
  1. Are both views necessary for the planning tasks and what supports the interpretation that exocentric evidence is “largely unused”?
  2. Does fine-tuning improve the use of complementary views?
  3. Could you explain a bit more about these concepts: “Hidden event”“decision point” and “current exo observation”.
Reviewer_S4anOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for evaluating how MLLMs understand real-world activities from synchronized egocentric and exocentric views. It contains 2,497 questions across eight tasks covering cross-view perception, planning, and trajectory prediction. The authors evaluate 21 models and find that most struggle with precise spatial grounding. Interestingly, adding an exocentric view provides little improvement in planning accuracy. The paper also introduces a 38K-sample training dataset, which improves performance across all eight tasks after fine-tuning.

Critical strengths

I think the benchmark addresses an important gap in embodied understanding. Its combination of synchronized real-world videos and spatially grounded tasks is particularly useful. The trajectory prediction tasks also go beyond the multiple-choice evaluations common in existing benchmarks.

The experimental analysis provides several interesting findings. I especially appreciate the controlled ego-only comparison, which shows that additional visual evidence does not necessarily improve planning. The detailed failure analysis also offers useful insights into where current models struggle.

Critical weaknesses

I am not fully convinced that the planning evaluation establishes the value of the additional exocentric view. Although the authors verify that relevant information is visible only exocentrically, this does not necessarily mean that the information is sufficient to determine the correct action. A human evaluation under both input conditions would help validate this assumption.

I have some concerns about the spatial evaluation protocol. Appendix E.4 selects the coordinate interpretation that minimizes distance to the ground truth. This effectively uses the correct answer to resolve ambiguous model outputs and could inflate performance. The comparisons are also affected by substantial differences in output-format compliance across models, making some of the reported gaps difficult to interpret.

Questions for authors

Could the authors report human performance under ego-only and ego+exo conditions? This would help establish whether the additional view provides meaningful information for planning.

How would the spatial results change if coordinate formats were determined without accessing ground truth?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.