acceptodds
Under review as a conference paper at ICLR 2027

Does Local Video Understanding Transfer Across Encounters? The EgoGears Benchmark

Abstract

Embodied systems must make knowledge acquired during one encounter usable in another despite changes in viewpoint, motion, and illumination. Yet aggregate cross-video accuracy conflates failures of local perception with failures to preserve observation identity, establish correspondence, and compose evidence, obscuring whether local video understanding actually transfers. We introduce EgoGears, a complementary single- and multi-video benchmark designed to diagnose this tran- sition. It contains 567 single-video and 1, 487 multi-video questions derived from 126 human-collected egocentric recordings covering 39 outdoor routes. Repeated traversals across movement speeds and lighting conditions ground comparisons in shared physical environments; 531 questions require alignment across inde- pendent recordings. Single-video questions measure the local visual, spatial, and motion evidence available to a model, while multi-video questions test whether evidence remains bound to the correct observation and can be composed into con- sistent route relationships. We report 29 single-video and 31 multi-video MLLM configurations across six model families in the main leaderboard. Among the 20 configurations evaluated comparably on both splits, every model performs worse on multi-video questions, with a mean decrease of 22.5 percentage points. The gap is not explained simply by additional videos or recording boundaries; the cen- tral bottlenecks are observation–evidence binding and ordered route-state track- ing. EgoGears provides a diagnostic testbed for understanding when, why, and how local video understanding fails to transfer across encounters.

Primary area:
datasets and benchmarks
Keywords:
Multi-modal large language models, cross-video comparison, video question answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6TniOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the EgoGears benchmark, which aims to evaluate MLLM outdoor embodied AI understanding on both single- and multi-video questions. The authors evaluate on 29 single-video and 31 multi-video configurations across 6 MLLM families. For single-video comparisons, reasoning improves performance. There seems to be fairly careful preparation of the data in the benchmark and analysis of the results.

Critical strengths
  • Very controlled study on combining human egocentric motion, repeated outdoor traversals, and complementary single- and multi-video evaluation

The study is very specific in what it is targeting and external variables were controlled as much as possible in the data collection and benchmark creation phases. Multiple speeds, ensuring landmarks are visible and shared across clips, having humans verify Gemini produced options/questions/captions, etc.

  • Rigorous analysis of open source models and some Gemini variants on the benchmark

I had an initial worry that with Gemini helping with question creation, captioning, and more that the benchmark would be biased towards Gemini models. The gain that Gemini saw cannot be explained by it being part of the question creation process. Using strict accuracy, Jaccard for partial credit, and chance normalized accuracy also made sense and it was interesting how reasoning only sometimes helped models get to the correct answer.

Critical weaknesses
  • Gemini 3.8-Flash seems fairly close to saturating the more difficult part of the benchmark (i.e. Multi-video)

I appreciate that the authors have already had a discussion about this in the paper (i.e. Appendix C.6), but the fact that Gemini 3.8-Flash is already at ~80% accuracy hurts the longevity of this benchmark, which seems well motivated for spurring outdoor embodied AI and video understanding. This hints that the difficulty of the benchmark might not be enough.

  • Other API providers not evaluated (e.g. OpenAI, Anthropic, etc.)

I understand the monetary cost with having already spent $30K on Gemini, but it is hard to say that just evaluating Gemini and open source models shows to the community that we are a long way away from being able to effectively reason on egocentric video (either single or multiple videos).

Questions for authors
  • Is there a reason why approximately 700 questions were selected across the 9 categories? Was not exactly sure what the motivation of that was.
  • Is there a reason why the Gemini models were not evaluated on the single-video questions?
5 comments
AuthorsOpenReview ↗

Response to W1

Weakness1

Approximately 80% Multi accuracy may indicate insufficient difficulty and limited benchmark longevity.

Overall accuracy masks substantial route-reasoning difficulty. On multi-step turns, Gemini-3.8-Flash scores only 33.3%; Sol and Terra achieve 37.9% and 41.4%, despite exceeding 93% on spatial consistency. These task categories were fixed during dataset construction. EgoGears therefore retains clear headroom and supports tracking progress in ordered route understanding.

Multi taskGemini 3.8-FlashGemini 3.1-ProGemini 2.5-ProSolTerraLuna
Spatial consistency88.6%67.9%62.1%93.6%94.3%82.9%
Event-phase association71.8%59.2%50.0%65.5%66.9%58.5%
Multi-step turns33.3%29.9%16.1%37.9%41.4%25.3%
AuthorsOpenReview ↗

Response to W2

Weakness2

Other API providers were not evaluated.

We have added complete evaluations of three OpenAI configurations on all 567 Single and 1,487 Multi questions, broadening API-provider coverage beyond Gemini. Their exact-set accuracies are shown below.

ModelSingle (N=567)Multi (N=1,487)
GPT-6.1-Sol77.60%80.70%
GPT-6.1-Terra78.84%81.78%
GPT-6-Luna73.02%72.36%
AuthorsOpenReview ↗

Response to Q1

Question1

Why approximately 700 questions across nine categories?

Approximately 700 Single questions were a practical annotation and evaluation target. Quotas across nine categories preserved task diversity and reserved examples for smaller categories. The quotas total 700, but OCR supplied only 19 eligible candidates for a quota of 21, yielding 698 candidates before human review and 567 released questions afterward.

Reviewer_sTCROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EgoGears, a benchmark for local video understanding and reasoning across egocentric observations of repeated outdoor routes. It comprises 567 Single and 1,487 Multi questions, covering variations in lighting and movement speed. Evaluations of diverse multimodal models, together with scoring and annotation robustness checks, reveal persistent difficulties in combining observations and tracking route structure.

Critical strengths
  1. A useful evaluation resource for repeated real-world observations. The combination of repeated outdoor routes, variations in lighting and movement speed, and local and relational questions from the same recording pool provides a valuable setting for studying spatial understanding across encounters. This concrete dataset contribution is the main reason I view the paper favorably.

  2. Informative empirical comparisons with meaningful robustness checks. Multi performance is consistently lower across 20 comparable configurations, and the gap persists under answer-format stratification and Jaccard scoring; the human-confirmed subset also largely preserves model rankings. These checks strengthen the case for evaluating reasoning across observations separately from local video understanding.

Critical weaknesses
  1. The central diagnostic claim is not fully isolated by the evaluation design. Single and Multi use separate question sets rather than matched local prerequisites and corresponding relational questions. The useful answer-format and scoring controls in Table 5 therefore do not establish that the gap primarily reflects failures to combine already-correct local evidence. Accumulated perception errors and differences in task difficulty remain alternative explanations, while insufficiently documented frame budgets and output failure rates further complicate attribution. This limits the evidence for identifying evidence binding and ordered route-state tracking as the central bottlenecks.

  2. The conceptual advance over closely related benchmarks is modest. CrossVid [1] already studies integration and comparison across videos, MMSI-Video-Bench [2] covers spatial reasoning and cross-video correspondence, and SAW-Bench [3] includes egocentric route reasoning. EgoGears adds a useful combination of repeated outdoor traversals and recording-condition variation, but the broader difficulties it highlights overlap with these findings. A more precise comparison of task coverage and newly enabled analyses would help establish the significance of its incremental contribution.

References

[1] Jingyao Li et al. CrossVid: A Comprehensive Benchmark for Evaluating Cross-Video Reasoning in Multimodal Large Language Models. 2025. Link

[2] Jingli Lin et al. MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence. 2025. Link

[3] Chuhan Li et al. SAW-Bench: Learning Situated Awareness in the Real World. 2026. Link

Questions for authors
  1. Diagnostic evidence. Is there an analysis that verifies the local facts required for a Multi question before assessing the final relational answer? If not, a small paired diagnostic subset would directly strengthen the bottleneck claim; alternatively, could the authors qualify this claim to reflect the current descriptive evidence?

  2. Evaluation protocol clarification. What frame-sampling policy, per-question frame budget, and allocation across clips were used? Please also specify output/reasoning budgets and report the proportions of truncated or invalid responses for each configuration, since these factors affect the interpretation of the performance gaps.

  3. Final-label verification clarification. In Appendices C.5–C.6, were repaired or adjudicated answers independently reconfirmed by human annotators? How does the 445-question human-confirmed subset relate to the other audit categories, and how were disagreements between human reviewers and model-based checks resolved?

5 comments
AuthorsOpenReview ↗

Response to W1/Q1

Weakness1 / Question1

Diagnostic evidence, local-prerequisite verification, and confounds in attributing the Single–Multi gap.

Shared-footage analysis. To investigate whether Multi-video QA failures persist despite successful local question answering, we construct a shared-footage subset linking 211 Multi questions to 164 released Single questions through 260 visual correspondences.

By aligning Single and Multi questions using shared video footage and conditioning on correctly answered matched Single questions, we reduce potential confounding factors related to video content and local question-answering performance. Crucially, even when all available matched Single questions are answered correctly, models still fail on 39.4–80.2% of the corresponding valid Multi responses across five open-model configurations (Table shown below). These findings further reinforce our central conclusion: even with shared underlying video evidence and successful local question answering, models continue to struggle substantially with Multi-video QA. This highlights the persistent challenges of multi-video understanding beyond those captured by isolated Single-video questions.

ModelSingle (%)Multi (%)Multi wrong after matched Single correct
Qwen3-VL-235B Thinking69.544.168/134 (50.7%)
Qwen3-VL-30B Thinking61.021.897/121 (80.2%)
Gemma-4-31B Thinking73.860.256/142 (39.4%)
GLM-4.6V Thinking58.538.445/90 (50.0%)
Kimi-VL Thinking47.616.145/62 (72.6%)
AuthorsOpenReview ↗

Response to W2

Weakness2

The conceptual advance over CrossVid, MMSI-Video-Bench and SAW-Bench is modest; clarify task coverage and newly enabled analyses.

Distinct contributions. We have added MMSI-Video-Bench and SAW-Bench to Table 1, which now compares task coverage and acquisition design with CrossVid and other related benchmarks. Unlike existing benchmarks, EgoGears builds its Single and Multi-video questions from the same real-world egocentric recording pool (126 recordings, 39 routes, 2,054 questions). This design enables two key analyses: (1) Shared-footage analysis, revealing that models still struggle with Multi-video QA even when matched Single questions are answered correctly; and (2) Controlled condition analysis, examining robustness to speed and illumination changes while holding the route and wearer fixed. Together, these capabilities distinguish EgoGears from prior benchmarks by enabling systematic investigation of the gap between local understanding and multi-observation reasoning.

AuthorsOpenReview ↗

Response to Q2

Question2

Specify frame sampling/allocation, visual/output budgets, and truncated or invalid-response rates for each configuration.

Visual budget. Each question uses 64 total frames, allocated proportionally to clip duration with a four-frame minimum per clip. Frames are sampled chronologically within each clip; width is capped at 480 pixels with aspect ratio preserved.

Output budget. Historical adapter defaults are 2,048 tokens for Direct/API and 16,384 for Thinking. The supplementary examples below use 16,384 tokens; 32,768 is a length-retry cap, which was not triggered.

ConfigurationSplitNTruncatedInvalid
Gemini-3.8-FlashSingle5670/567 (0.00%)0/567 (0.00%)
GPT-6.1-SolMulti1,4870/1487 (0.00%)0/1487 (0.00%)
GPT-6.1-TerraMulti1,4870/1487 (0.00%)0/1487 (0.00%)
GPT-6-LunaMulti1,4870/1487 (0.00%)1/1487 (0.07%)

Truncation requires an explicit length-termination flag; invalid answers include malformed JSON or wrong selection count. Both count as incorrect in the full split denominator. Full available audits are provided in the revised manuscript's appendix.

Reviewer_4NaPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EgoGears, an outdoor egocentric video QA benchmark intended to examine whether local video understanding supports reasoning across separate encounters. The contribution comprises repeated-route data collection, a question-generation and verification pipeline. The authors report lower multi-video accuracy across 20 comparable configurations and attribute the difficulties primarily to observation–evidence binding and ordered route-state tracking.

Critical strengths

The repeated-route acquisition design is the clearest substantive strength. Recording shared physical environments under different movement speeds, lighting conditions, and travel directions provides a useful setting for evaluating place correspondence and route understanding.

Critical weaknesses
  1. The experiments do not adequately establish the central claim about transferring local understanding across encounters.
  2. The reliability of the final benchmark labels is insufficiently demonstrated.
  3. The presentation also needs substantial improvement. Text in Figures 1–2 and the qualitative examples in Figures 4–9 is excessively small, particularly the legends, answer options, and model predictions.
Questions for authors

How many independent human reviewers verified each final answer, especially after repair or model-assisted adjudication? Were revised items approved again by humans, and what agreement or residual-error measurements are available?

4 comments
AuthorsOpenReview ↗

Response to W1

Weakness1

The experiments do not adequately establish transfer of local understanding across encounters.

To investigate whether multi-video failures persist despite successful local visual understanding, we construct a shared-footage subset comprising 260 visual correspondences between Single and Multi video questions. Crucially, even when models correctly answer all available matched Single questions, they still fail on 39.4–80.2% of the corresponding valid Multi responses across five open-model configurations (see the table below). This substantial conditional error rate highlights a critical gap between local question-answering success and the ability to solve more complex questions involving shared visual evidence. In particular, correctly answering individual questions does not guarantee that models can effectively utilize the corresponding evidence to solve Multi questions. While the matched Single questions cover only a subset of the prerequisites for each Multi question, and some correspondences involve within-recording cases, these limitations do not diminish the observed discrepancy between local and Multi question-answering performance. Rather than attributing all failures exclusively to cross-video reasoning, we interpret these results as evidence that successful local understanding alone is insufficient to ensure reliable Multi-video QA. Rather than attributing all failures exclusively to cross-video reasoning, our findings demonstrate that successful local question answering alone does not guarantee reliable performance on Multi-video QA.

ModelSingle (%)Multi (%)Multi wrong after matched Single correct
Qwen3-VL-235B Thinking69.544.168/134 (50.7%)
Qwen3-VL-30B Thinking61.021.897/121 (80.2%)
Gemma-4-31B Thinking73.860.256/142 (39.4%)
GLM-4.6V Thinking58.538.445/90 (50.0%)
Kimi-VL Thinking47.616.145/62 (72.6%)
AuthorsOpenReview ↗

Response to W2/Q1

Weakness2 / question1

Final-label reliability, human verification after repair, and agreement/error measurements.

After generation, each item underwent human review, model rechecking/repair, and final human approval. A ten-person annotation team conducted the initial review; one additional reviewer checked all released Multi items in their final form, including the 504 repaired and 277 rekeyed items. Revised items were therefore explicitly human-reapproved after modification.

AuthorsOpenReview ↗

Response to W3

Weakness3

The presentation also needs substantial improvement. Text in Figures 1–2 and the qualitative examples in Figures 4–9 is excessively small, particularly the legends, answer options, and model predictions.

We have enlarged the legends, answer options, and model predictions in Figures 1–2 and the qualitative examples in Figures 4–9. These changes have been incorporated into the revised manuscript, with the corresponding revisions marked in red.

AuthorsOpenReview ↗

Response to Reviewer 4NaP

Dear Reviewer 4NaP:

Thank you again for your thoughtful feedback on our submission. As the discussion period draws to a close, we would be grateful if you could let us know whether our responses have addressed your concerns or if there are any points we could clarify further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.