- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents infinity-THOR, a framework that includes a dataset and a QA benchmark for long-context embodied reasoning, built upon AI2-THOR. The proposed embodied QA task, Needle(s) in the Embodied Haystack, requires recalling and reasoning over scattered clues among long trajectories. Different algorithmic designs, such as architectural designs and training strategies, were tested and analyzed.
Critical strengths
- The proposed benchmark and QA questions are meaningful and real challenges for embodied AI. "Needle(s) in the Haystack" is a nice analogy for the sparse information and memory challenges in long-horizon EAI tasks.
- There is considerable effort put into the task designs and data/questions generation.
- The architecture/training study yields insightful, actionable findings
Critical weaknesses
- Positioning of this work in related work: although Table 1 is a nice summary that shows the advantage of this benchmark, it is very outdated. There are also real-world benchmarks (like HourVideo) that are relevant. Please update. Also, the notion of "steps" is a bit unclear; are they simulation steps? Are they the same across all benchmarks?
- Stronger, SOTA VLMs need to be evaluated; I understand that you might be limited by resources, but at least a small subset of static tasks should be evaluated.
- In Appendix C Alg 2: "Initialize context with GT actions up to Pi−1" which initializes context with ground-truth history before every plan and stops at the first failure, so errors never compound? This makes me wonder about the results reported in Table 4 and Fig. 8; please clarify.
- The effort trying to tie this to low-level manipulation (ManipulaTHOR) is nice, but it is half-baked, and it's unclear what we can learn from this.
Minor:
- VSI-bench is not a "photorealistic" benchmark; it is a benchmark with real images.
- In-text citation format issues throughout the text.
Questions for authors
See comments above.
2 comments
Rebuttal received, thanks for the responses.
We thank the reviewer for finding the benchmark meaningful and the training study actionable.
W1: Table 1 outdated; real-world benchmarks such as HourVideo; definition of "steps".
Thank you. We have updated Table 1 in two ways.
We also discuss LongVideoBench, which retrieves a referred segment from hour-long web videos, in the multimodal Needle-in-a-Haystack paragraph. It is not egocentric or embodied, so we did not add it to the embodied comparison table.
"Steps" are environment steps: one discrete agent action and its resulting observation. The video benchmarks have no actions, so their Avg steps and GT-action entries are not applicable. We now add a column marking real video vs. simulation and report video duration for the video rows, rather than mixing units, and the caption says so. The comparison shows the gap ∞-THOR fills: real-video benchmarks offer long horizons but no interaction or action labels, while interactive embodied benchmarks offer actions but much shorter horizons.
W2: Stronger SOTA VLMs. Please see R1 in the General Response. We evaluated Gemini-3.1-Pro and Qwen3.5-9B on the full NiEH/NsiEH test sets. Gemini-3.1-Pro reaches only 43.9% with images alone, so the benchmark remains challenging for frontier models.
W3: Algorithm 2, does initializing with GT history mean errors never compound? Clarification of Table 4 and Fig. 8.
Thank you for the careful reading. Your understanding is correct, and this is by our design.
Table 4 evaluates the final long-horizon task of each trajectory. Fig. 8 reports full-episode execution, plan by plan. Both use the same protocol:
The protocol is designed to isolate long-context understanding: can the model act correctly when given a long, correct history of hundreds of steps? Without the GT reset, no current policy reaches the final stages of a trajectory. Since evaluation also stops at the first failure, models would fail early and condition only on a short history of its own. Long-history conditions would then never be reached, and a fair comparison across context lengths and extension methods (Table 4, Fig. 8a–c) would not be possible. Even with this favorable setup, success is low (e.g., Put ≤ 2.2%, R3), which shows that acting on long-context history is itself a major bottleneck. Fully autonomous execution, where errors compound, is strictly harder and an important direction for future work. In the revision, we will state this protocol and its rationale explicitly in Sec. 5.2 and App. C.2.
W4: ManipulaTHOR integration is half-baked. We agree the current results do not support strong conclusions on low-level manipulations on ManipulaTHOR. Its purpose is to show that ∞-THOR trajectories can be executed with low-level arm control as well, not to benchmark manipulation policies. In the revision, we will state its limitations and narrow this contribution as one of capability demonstrations.
Minor: Thank you for catching details. We will describe VSI-Bench as a benchmark of real-world videos rather than "photorealistic", and fix the in-text citation formatting throughout.