acceptodds
Under review as a conference paper at ICLR 2027

Beyond Needle(s) in the Embodied Haystack: Environment, Architecture, and Training Considerations for Long Context Reasoning

Abstract

We introduce -THOR, a new framework for long-horizon embodied tasks that advances long-context understanding in embodied AI. -THOR provides: (1) a generation framework for synthesizing scalable, reproducible, and unlimited long-horizon trajectories; (2) a novel QA task, Needle(s) in the Embodied Haystack, where scattered clues across extended trajectories test agents’ long-context reasoning ability; and (3) a long-horizon dataset and benchmark suite featuring complex tasks that span hundreds of environment steps, each paired with ground-truth action sequences. To enable this capability, we explore architectural adaptations, including interleaved Goal-State-Action and Memory-Augmented Goal-State modeling, alongside training strategies—context extension techniques and Context Parallelism—to equip VLM-based agents for extreme long-context reasoning and interaction. Experiments highlight the challenges of our benchmark and provide insights into training strategies and model behaviors under long-horizon conditions. Furthermore, we demonstrate sim-to-real transfer: our QA dataset alone improves photorealistic benchmark performance by up to +11.2%, and we successfully integrate with low-level manipulation controls. Our work provides a foundation for the next generation of embodied AI systems capable of robust, long-term reasoning and planning.

Primary area:
datasets and benchmarks
Keywords:
Long-horizon embodided task, Long-context reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZEtCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents infinity-THOR, a framework that includes a dataset and a QA benchmark for long-context embodied reasoning, built upon AI2-THOR. The proposed embodied QA task, Needle(s) in the Embodied Haystack, requires recalling and reasoning over scattered clues among long trajectories. Different algorithmic designs, such as architectural designs and training strategies, were tested and analyzed.

Critical strengths
  • The proposed benchmark and QA questions are meaningful and real challenges for embodied AI. "Needle(s) in the Haystack" is a nice analogy for the sparse information and memory challenges in long-horizon EAI tasks.
  • There is considerable effort put into the task designs and data/questions generation.
  • The architecture/training study yields insightful, actionable findings
Critical weaknesses
  • Positioning of this work in related work: although Table 1 is a nice summary that shows the advantage of this benchmark, it is very outdated. There are also real-world benchmarks (like HourVideo) that are relevant. Please update. Also, the notion of "steps" is a bit unclear; are they simulation steps? Are they the same across all benchmarks?
  • Stronger, SOTA VLMs need to be evaluated; I understand that you might be limited by resources, but at least a small subset of static tasks should be evaluated.
  • In Appendix C Alg 2: "Initialize context with GT actions up to Pi−1" which initializes context with ground-truth history before every plan and stops at the first failure, so errors never compound? This makes me wonder about the results reported in Table 4 and Fig. 8; please clarify.
  • The effort trying to tie this to low-level manipulation (ManipulaTHOR) is nice, but it is half-baked, and it's unclear what we can learn from this.

Minor:

  • VSI-bench is not a "photorealistic" benchmark; it is a benchmark with real images.
  • In-text citation format issues throughout the text.
Questions for authors

See comments above.

2 comments

We thank the reviewer for finding the benchmark meaningful and the training study actionable.

W1: Table 1 outdated; real-world benchmarks such as HourVideo; definition of "steps".

Thank you. We have updated Table 1 in two ways.

  • Long egocentric video benchmarks: HourVideo (20–120 min Ego4D videos, including navigation questions) and EgoSchema (3-min Ego4D clips).
  • Recent embodied-memory benchmarks: OpenEQA (episodic-memory QA over real scans) and FindingDory (memory-dependent tasks for embodied agents in Habitat).

We also discuss LongVideoBench, which retrieves a referred segment from hour-long web videos, in the multimodal Needle-in-a-Haystack paragraph. It is not egocentric or embodied, so we did not add it to the embodied comparison table.

"Steps" are environment steps: one discrete agent action and its resulting observation. The video benchmarks have no actions, so their Avg steps and GT-action entries are not applicable. We now add a column marking real video vs. simulation and report video duration for the video rows, rather than mixing units, and the caption says so. The comparison shows the gap ∞-THOR fills: real-video benchmarks offer long horizons but no interaction or action labels, while interactive embodied benchmarks offer actions but much shorter horizons.

W2: Stronger SOTA VLMs. Please see R1 in the General Response. We evaluated Gemini-3.1-Pro and Qwen3.5-9B on the full NiEH/NsiEH test sets. Gemini-3.1-Pro reaches only 43.9% with images alone, so the benchmark remains challenging for frontier models.

W3: Algorithm 2, does initializing with GT history mean errors never compound? Clarification of Table 4 and Fig. 8.

Thank you for the careful reading. Your understanding is correct, and this is by our design.

Table 4 evaluates the final long-horizon task of each trajectory. Fig. 8 reports full-episode execution, plan by plan. Both use the same protocol:

  1. The context is initialized with the ground-truth history before every plan.
  2. Evaluation stops at the first failure.
  3. Errors therefore do not compound across plans.

The protocol is designed to isolate long-context understanding: can the model act correctly when given a long, correct history of hundreds of steps? Without the GT reset, no current policy reaches the final stages of a trajectory. Since evaluation also stops at the first failure, models would fail early and condition only on a short history of its own. Long-history conditions would then never be reached, and a fair comparison across context lengths and extension methods (Table 4, Fig. 8a–c) would not be possible. Even with this favorable setup, success is low (e.g., Put ≤ 2.2%, R3), which shows that acting on long-context history is itself a major bottleneck. Fully autonomous execution, where errors compound, is strictly harder and an important direction for future work. In the revision, we will state this protocol and its rationale explicitly in Sec. 5.2 and App. C.2.

W4: ManipulaTHOR integration is half-baked. We agree the current results do not support strong conclusions on low-level manipulations on ManipulaTHOR. Its purpose is to show that ∞-THOR trajectories can be executed with low-level arm control as well, not to benchmark manipulation policies. In the revision, we will state its limitations and narrow this contribution as one of capability demonstrations.

Minor: Thank you for catching details. We will describe VSI-Bench as a benchmark of real-world videos rather than "photorealistic", and fix the in-text citation formatting throughout.

Rebuttal received, thanks for the responses.

Reviewer_yKUqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ∞-THOR, a framework that generates long-horizon embodied trajectories by chaining planner-generated tasks in AI2-THOR and constructs single- and multi-evidence QA pairs from replay metadata. Using this benchmark, the authors systematically study interleaved goal–state–action inputs, memory representations, and context-extension strategies across static QA and online interactive evaluation, supplemented by transfer and policy-integration experiments on VSI-Bench and ManipulaTHOR.

Critical strengths

This paper tackles an important challenge in embodied AI by unifying long-horizon interaction histories, temporally grounded QA, and online decision-making within a single evaluation framework. The replayable trajectories and action demonstrations make ∞-THOR a valuable resource for studying how long-term memory supports both temporal understanding and embodied control. Furthermore, the systematic comparisons across history representations and context lengths offer practical empirical insights for future system design. Beyond evaluation, fine-tuning on the synthetic QA data boosts the overall VSI-Bench score from 36.7 to 38.8, providing encouraging evidence of sim-to-real transfer to real-world video QA and underscoring the utility of the generated training data.

Critical weaknesses
  1. The contribution of distant history to online task performance warrants more direct validation. Section 3.2 introduces long-horizon dependencies by sampling objects from early and late trajectory segments, yet it remains unclear how far recent observations alone suffice. Comparing full versus truncated recent history on a representative subset under the same model would better isolate the value of distant memory.

  2. A brief failure-mode analysis would make the online evaluation more diagnostic. In Table 4, early-stage failures rarely leave agents able to attempt Put, leaving the relative bottlenecks across memory retrieval, planning, and execution unclear. Categorizing a sample of existing failed rollouts and reporting stage-wise reach rates—without additional training—would clarify the primary bottlenecks and better guide future work.

  3. The effects of history representation versus additional semantic inputs could be better disentangled. In Table 3, interleaved inputs outperform image-only baselines but also include goal and action text, confounding history organization with extra linguistic cues. Further discussion or analysis clarifying the relative role of these language priors would help better interpret the observed improvements.

  4. An optional comparison with a recent state-of-the-art open-source VLM would strengthen the evaluation. Evaluating a stronger recent open-source VLM on a representative static QA subset—under identical input formats and context budgets without additional fine-tuning—would help verify whether the reported trends generalize to more capable backbones.

Questions for authors

Please refer to the Critical Weaknesses above.

2 comments

We thank the reviewer for the constructive and specific suggestions.

W1: Full vs. truncated recent history. Thank you for this suggestion, please see R3 in the General Response. On the full test set with the same Qwen3.5-9B backbone, a model limited to recent history (32K) reaches 7.4%, versus 15.2% with longer history (128K), and 4.8% with no history. Most of the gain therefore comes from distant history, consistent with the task design, which samples target objects from the first and last 20% of the episode.

W2: Failure-mode analysis with stage-wise reach rates. The stage-wise success rates in R3 already show the bottleneck: with long history, Go-to succeeds 28.5% of the time, but only 5.3% of episodes complete Pick and 2.2% complete Put. Navigation to the remembered location is the dominant failure point. We agree this makes Table 4 more diagnostic and will add it to Sec. 5.2.

W3: History representation vs. language cues. Please see R2 in the General Response. The new action-only row (g) separates the two. Language cues alone recover much of the interleaved gain on single-evidence questions, but images add 8–14 points for Gemini, with the larger drop on multi-evidence. The per-subset split shows where vision is necessary. We will revise Sec. 5.1 to attribute the interleaved gains to both sources rather than to history organization alone.

W4: Recent open-source VLM. Please see R1 in the General Response. We evaluated Qwen3.5-9B on the full test set under the same input formats, without fine-tuning. The trends hold: retrieval (d, e) underperforms full trajectories (a), and structured text-bearing histories (b, c, f) beat image-only (a), especially on multi-evidence questions.

Reviewer_yKUqOpenReview ↗

Comments on the author responses

Thanks for the responses, and the original rating is kept.

Reviewer_VoYBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ∞-THOR, a framework built on AI2-THOR for generating and evaluating very long embodied tasks. It chains many planned subtasks into trajectories of several hundred steps, which can exceed one million tokens when given to a VLM. Based on these trajectories, the paper builds NiEH, a question answering benchmark where the answer depends on one or several events scattered across the trajectory. It also adds an interactive task at the end of each trajectory that requires remembering objects seen much earlier. The paper compares feeding the full history to the model with memory-based alternatives, tests several context extension methods and finetunes models on long trajectories with context parallelism. Current VLMs struggle once the context grows beyond their pretraining length. The best model completes only 11.5% of the interactive tasks.

Critical strengths
  1. The paper presents a well-motivated benchmark, clearly identifying long-horizon memory as a key gap in current embodied AI evaluation and extending the needle-in-a-haystack setting to long, interactive embodied trajectories with multi-evidence questions.
  2. It provides a valuable contribution to the community by offering a scalable framework that generates trajectories of arbitrary length together with ground-truth actions and qa pairs. It thereby establishes a useful testbed for future research on memory and long-context reasoning in embodied agents.
Critical weaknesses
  1. The evaluation may not isolate long-context visual reasoning. While the paper emphasizes visual memory over long trajectories, many NiEH questions, such as what object was sliced or how many times an object was moved, can be answered directly from the action history. Since the interleaved model receives these actions as text, its large improvement over the image-only setting may come from reading action strings rather than recalling visual observations. The online evaluation raises a similar concern, as the best success rate is only 11.5% and there is no baseline without history, making it difficult to tell whether these tasks truly require long-term memory.
  2. The experimental findings are largely expected and the evaluated models are outdated. The training study mainly applies existing context extension techniques, and its conclusions, such as performance degrading beyond the pretraining length, offer limited new insight. Meanwhile, the evaluated models, including Qwen2.5-VL and Gemini 2.0 Flash, are well behind current VLMs with much longer context windows, which may limit the relevance of the conclusions. The sim2real claim is also based on a 2.1% improvement on VSI-Bench from a single run.
Questions for authors
  1. Could you report NiEH accuracy when the model only sees the action text or the images?
  2. Could you clarify whether the text memory is built from simulator metadata and how an agent would obtain it at test time?
  3. Could you evaluate some more recent VLMs on NiEH and clarify whether test scenes overlap with training scenes?
2 comments

We thank the reviewer for recognizing the benchmark as well-motivated and as a useful testbed for memory in embodied agents.

W1/Q1: Does NiEH isolate visual reasoning? Action-only vs. image-only accuracy. Please see R2 in the General Response. Image-only is row (a), and we added action-only as row (g). Action-only falls between image-only and interleaved for both new models. We appreciate your point on answerability: some questions can be answered from action strings, and we now quantify this: 81% of single-evidence questions are recoverable from actions in principle, 159 of those are ambiguous without visual observations, and 78 require vision. We will report accuracy separately on the action-derivable and vision-required subsets, release per-question tags, and revise our claims accordingly. We do not claim that interleaved gains come purely from visual recall.

W1: No baseline without history in the online evaluation. We added this baseline; please see R3. With Qwen3.5-9B, an agent without history reaches 4.8% total success, versus 15.2% with long interleaved history. Truncating to recent history (32K) recovers only 7.4%. The online task therefore genuinely depends on distant memory: the final subgoal targets objects sampled from the first and last 20% of the episode (Sec. 3.2). The low absolute numbers reflect how hard the task is, not a lack of memory dependence.

W2: Findings expected; outdated models. Please see R1. The new models sharpen rather than overturn our conclusions. Even Gemini-3.1-Pro, with a context window that fits every trajectory, reaches only 43.9% from images alone. That is below a 9B open model, so longer context windows alone do not solve long visual histories. On the training side, our contribution is the data and evaluation infrastructure: arbitrarily long trajectories with ground-truth actions and QA, which let the community study these effects at >1M tokens in an embodied setting. The context-extension study is one use of that infrastructure.

W2: Sim2real claim from a single run. We agree that a single run is weak evidence, so we trained a second seed:

VSI-Bench taskQwen2.5-VL-7BOurs (seed 42)Ours (seed 142)
Object Appearance Order34.846.0 (+11.2)44.3 (+9.5)
Object Abs Distance21.123.0 (+1.9)26.3 (+5.2)
Object Counting41.340.7 (−0.6)43.6 (+2.3)
Object Rel Distance38.339.2 (+0.9)38.6 (+0.3)
Object Size Estimation49.950.7 (+0.8)51.8 (+1.9)
Room Size Estimation36.637.8 (+1.2)37.4 (+0.8)
Route Planning30.930.2 (−0.7)29.3 (−1.6)
Object Rel Direction40.943.3 (+2.4)45.1 (+4.2)
Overall36.738.8 (+2.1)39.6 (+2.9)

Across the two seeds, the overall score is 39.2 ± 0.5 (+2.5). The largest gains are consistent across seeds and fall on the task closest to our temporal-recall training data, Object Appearance Order (+11.2 / +9.5), and on Object Rel Direction (+2.4 / +4.2). Route Planning drops slightly in both. We report the two-seed mean in the revision and describe the result as preliminary evidence of sim-to-real transfer.

Q2: How is text memory built, and how would an agent obtain it at test time? The text memory is built from simulator metadata, following ALFWorld's textual environments (e.g., "You are in the middle of a room. Looking quickly around you, you see a drawer 2, a shelf 5, …"). An agent would not have this at test time without a perception module such as an object detector or captioner. Row (c) therefore uses privileged information and should be read as an upper-bound reference for a perfect perception-to-text system, not as a deployable baseline. We will state this explicitly in Sec. 4 and the Table 3/4 captions.

Q3: Scene overlap between training and test. The NiEH QA results (Table 3 and the new rows) are zero-shot: none of these models is trained on ∞-THOR, so overlap does not affect them. For the fine-tuned online evaluation (Table 4), test trajectories use floorplans that also appear in training, with different task sequences and object placements. This is analogous to the "seen" split in ALFRED. We will state the split protocol explicitly.

Thank you for the detailed rebuttal. My concerns have been addressed and will raise my score to 3.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response (to all reviewers)

We thank all reviewers for their careful reading. Reviewers found the benchmark well-motivated (VoYB, ZEtC), the framework a valuable and scalable resource (VoYB, yKUq), and the architecture/training study actionable (ZEtC, yKUq). The main shared requests were (1) evaluating recent VLMs, and (2) disentangling visual memory from the language cues (goal/action text) in the interleaved input. We address both with new experiments below.

R1. Recent VLMs (all reviewers). We evaluated Qwen3.5-9B (open) and Gemini-3.1-Pro (closed) on the full NiEH/NsiEH test sets, with the same prompts, input formats, and GPT-4o judge as Table 3. Gemini uses low-resolution image tokens (280 tokens/image), which fits up to ~3,600 images, so every trajectory fits in context without frame subsampling.

MethodNiEH Qwen3.5-9BNiEH Gemini-3.1-ProNsiEH Qwen3.5-9BNsiEH Gemini-3.1-Pro
(a) Full trajectory (Image-only)53.0743.9233.5943.20
(b) Interleaved Goal-State-Action58.3288.6552.7469.29
(c) Memory-Augmented (Text)60.9863.1264.0148.11
(d) Memory-Augmented (Image, Top-10)43.9832.3430.0430.33
(e) Memory-Augmented (Image, Top-20)51.8836.9834.0138.41
(f) Memory-Augmented (Text + Image, Top-20)56.2445.4459.2962.61
(g) Action-only (goal + action text, no images)56.3080.3643.7255.04

We summarize takeaways with new experiments:

  • The benchmark is far from solved by current models. With images alone, Gemini-3.1-Pro reaches only 43.9% (single) / 43.2% (multi), below a 9B open model on single-evidence. Long visual histories remain the bottleneck even for frontier models with million-token contexts.
  • The original trends hold on newer backbones. Image retrieval (d, e) underperforms the full trajectory (a) for both models, and structured history (b, c, f) beats image-only (a) on multi-evidence questions for both models.
  • Multi-evidence questions stay hard. The best NsiEH score is 69.3%, versus 88.7% on NiEH.

R2. Visual memory vs. language cues (VoYB, yKUq). Row (g) removes all images and keeps only goal and action text. The result lands between image-only (a) and interleaved (b) for both models. Removing the images costs 8.3/14.2 points for Gemini and 2.0/9.0 for Qwen (single/multi), and the drop is larger on multi-evidence questions. So the action text carries much of the signal, but images add information that text alone does not provide.

We are upfront about why the action text is informative. NiEH asks about events the agent itself caused ("Where is X now?", "Which object did you put on Y?"), and the action log records those events. Parsing the PutObject actions, 670/829 (81%) of single-evidence questions could in principle be answered from actions alone. However, 159 of these are ambiguous: several objects were put on the same receptacle, so the action log names candidates but only the observations identify the correct one. Another 78 questions (counting objects on a receptacle, listing what is on a receptacle) are not recoverable from actions at all. For example, for "What items are on the drawer? — key, phone, watch, lamp", none of the items was placed by the agent.

In the revision we will (i) add row (g) and this split to Table 3, (ii) tag each question as action-derivable or vision-required in the released benchmark so future work can report both, and (iii) revise the text to say that NiEH measures long-context recall over multimodal histories, with images and text as complementary sources.

AuthorsOpenReview ↗

General Response (to all reviewers) (cont.)

R3. Online evaluation with Qwen3.5-9B, including no-history and truncated-history baselines (VoYB, yKUq).

We additionally trained Qwen3.5-9B with the same Table 4 protocol and added a no-history baseline: the agent sees only the current observation and the goal. The 32K and 128K-context model keeps only the most recent steps (truncated-history baseline).

Method (Qwen3.5-9B)Go toPick upPutTotal
No history8.41.60.04.8
Memory-Augmented (Text)16.83.51.06.1
Memory-Augmented (Image, Top-20)8.20.00.03.4
Interleaved G-S-A, 32K22.83.51.07.4
Interleaved G-S-A, 128K28.55.32.215.2

No history baseline never goes further after the first subgoal, ending up 0% success rate on Put tasks. Due to the limited rebuttal timeline, we report results from an intermediate fine-tuning checkpoint. However, we found that performance keeps improving with further training, and we will update these numbers before the end of the discussion period.

Final online results (update to R3). We report results from the fully trained Qwen3.5-9B checkpoints. Values in parentheses are the intermediate-checkpoint numbers reported earlier (in the previous comment).

Method (Qwen3.5-9B)Go toPick upPutTotal
No history8.41.60.04.8
Memory-Augmented (Text)16.83.51.06.1
Memory-Augmented (Image, Top-20)8.20.00.03.4
Interleaved G-S-A, 32K23.5 (22.8)4.3 (3.5)1.2 (1.0)7.8 (7.4)
Interleaved G-S-A, 128K33.1 (28.5)6.5 (5.3)2.5 (2.2)16.6 (15.2)

As further training improves both interleaved models, we keep the conclusions from the papers. We will include these final numbers in the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.