- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces the EgoGears benchmark, which aims to evaluate MLLM outdoor embodied AI understanding on both single- and multi-video questions. The authors evaluate on 29 single-video and 31 multi-video configurations across 6 MLLM families. For single-video comparisons, reasoning improves performance. There seems to be fairly careful preparation of the data in the benchmark and analysis of the results.
Critical strengths
- Very controlled study on combining human egocentric motion, repeated outdoor traversals, and complementary single- and multi-video evaluation
The study is very specific in what it is targeting and external variables were controlled as much as possible in the data collection and benchmark creation phases. Multiple speeds, ensuring landmarks are visible and shared across clips, having humans verify Gemini produced options/questions/captions, etc.
- Rigorous analysis of open source models and some Gemini variants on the benchmark
I had an initial worry that with Gemini helping with question creation, captioning, and more that the benchmark would be biased towards Gemini models. The gain that Gemini saw cannot be explained by it being part of the question creation process. Using strict accuracy, Jaccard for partial credit, and chance normalized accuracy also made sense and it was interesting how reasoning only sometimes helped models get to the correct answer.
Critical weaknesses
- Gemini 3.8-Flash seems fairly close to saturating the more difficult part of the benchmark (i.e. Multi-video)
I appreciate that the authors have already had a discussion about this in the paper (i.e. Appendix C.6), but the fact that Gemini 3.8-Flash is already at ~80% accuracy hurts the longevity of this benchmark, which seems well motivated for spurring outdoor embodied AI and video understanding. This hints that the difficulty of the benchmark might not be enough.
- Other API providers not evaluated (e.g. OpenAI, Anthropic, etc.)
I understand the monetary cost with having already spent $30K on Gemini, but it is hard to say that just evaluating Gemini and open source models shows to the community that we are a long way away from being able to effectively reason on egocentric video (either single or multiple videos).
Questions for authors
- Is there a reason why approximately 700 questions were selected across the 9 categories? Was not exactly sure what the motivation of that was.
- Is there a reason why the Gemini models were not evaluated on the single-video questions?
5 comments
Response to W2
Weakness2
Other API providers were not evaluated.
We have added complete evaluations of three OpenAI configurations on all 567 Single and 1,487 Multi questions, broadening API-provider coverage beyond Gemini. Their exact-set accuracies are shown below.
| Model | Single (N=567) | Multi (N=1,487) |
|---|---|---|
| GPT-6.1-Sol | 77.60% | 80.70% |
| GPT-6.1-Terra | 78.84% | 81.78% |
| GPT-6-Luna | 73.02% | 72.36% |
Response to Q1
Question1
Why approximately 700 questions across nine categories?
Approximately 700 Single questions were a practical annotation and evaluation target. Quotas across nine categories preserved task diversity and reserved examples for smaller categories. The quotas total 700, but OCR supplied only 19 eligible candidates for a quota of 21, yielding 698 candidates before human review and 567 released questions afterward.
Response to W1
Weakness1
Approximately 80% Multi accuracy may indicate insufficient difficulty and limited benchmark longevity.
Overall accuracy masks substantial route-reasoning difficulty. On multi-step turns, Gemini-3.8-Flash scores only 33.3%; Sol and Terra achieve 37.9% and 41.4%, despite exceeding 93% on spatial consistency. These task categories were fixed during dataset construction. EgoGears therefore retains clear headroom and supports tracking progress in ordered route understanding.