acceptodds
Under review as a conference paper at ICLR 2027

HiREx: Hierarchical Retrieval and Active Observation over Visual Memory for Embodied Exploration and Reasoning

Abstract

Embodied exploration and reasoning is the capability to explore an environment to answer questions or reach specified goals. Recent zero-shot methods build a scene representation from observations, over which a vision-language model (VLM) reasons to answer and to decide where to explore. However, existing representations fall short. 3D scene graphs abstract the scene into objects and relations, discarding visual evidence before the task is given. Image-set methods preserve the evidence but retrieve it through detected classes, inheriting their errors. Both leave exploration separate from retrieved evidence. To address these issues, we propose HiREx, a scene memory that organizes the image set under room, view, and object layers, on which both retrieval and exploration operate. Views are selected to preserve object co-visibility, rooms index the views as semantic anchors, and object crops supply fine-grained details. Given a task, retrieval narrows the evidence through the layers, and the level at which evidence is found activates exploration options at that level. Across question-answering and goal-navigation benchmarks, our method outperforms all baselines and shows that memory, retrieval, and exploration each contribute to embodied exploration and reasoning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Question Answering, Goal-Oriented Navigation, Scene Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ogbnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue that 3D scene graphs abstract the scene into objects and relations, discarding visual evidence; whereas image-set methods preserve evidence but retrieve it through detected classes, inheriting their errors. And both categories separate exploration from evidence retrieval.

To this end, they propose HiRex, which is a scene memory (memorizing the scenes in 3-layers, namely rooms, view, and objects), which enables both retrieval and exploration operations.

Finally, the authors show the superiority of HiRex against the baselines.

Critical strengths
  • This paper touches an interesting and key problem in Embodied AI: Exploration and Reasoning
  • The HiRex is a hierarchical retrieval and exploration framework bringing in the strengths of both scene graph- and image-set-based solutions
  • The experiments show strong performance against multiple benchmarks
Critical weaknesses
  • Computation Cost: The inference time and computation of HiRex could be a challenge; i.e., the proposed 3-layered framework for retrieval and exploration could be expensive.

  • Dynamic Environments: The authors don't discuss dynamic and evolving environments and how efficient their framework is such settings.

  • Partially Observable Settings: The paper and the experiments lack partially observable settings which are common settings for embodied tasks

  • I wish the authors invested more time in polishing their content. Some paragraphs were not easy to read or follow. It would've been great if they could use examples for their explanations.

Questions for authors
  • Sec 3.1 Object Layer: How do you capture relations between objects?
  • Lines 258-259: Could you explain how you decide whether an evidence is sufficient?
  • Lines 322-323: Why did you separate termination from exploration?
  • Lines 364-365: I'm wondering why the GPT-4o models were used, but not the latest models?
  • How do you compare this work with ActiveVOO?
    • Liu, Xiaotian, et al. "ActiveVOO: Value of observation guided active knowledge acquisition for open-world embodied lifted regression planning." Advances in Neural Information Processing Systems 38 (2026): 106621-106647.
6 comments
AuthorsOpenReview ↗

Response to Reviewer ogbn(1/2)

Summary of our response.

  • HiREx addresses partial observability, as the agent starts in an unseen environment and moves to observe what it still lacks, and we make this setting explicit in the revision (W1).
  • HiREx targets static scenes as the evaluated benchmarks do, and the revision states this and discusses how HiREx can extend to dynamic environments (W2).
  • HiREx adds a modest 16% to 3D-Mem's per-step time in exchange for higher accuracy and exploration efficiency (W3).
  • We added examples to the method description (Sec. 3.1 with a new figure and video, and Sec. 3.3) and clarified how HiREx captures object relations, decides when to terminate, and differs from ActiveVOO (W4, Q1–Q5).

<Line numbers (L) refer to the revised paper unless stated otherwise.>

W1. Does HiREx address partially observable settings, which are common in embodied tasks?

Yes, HiREx addresses partial observability in embodied question answering and goal navigation, where the agent obtains missing information by moving and observing.

Our problem definition requires the agent to reason over accumulated observations and to move for information it still lacks (L34–35, L41–42). In A-EQA and GOAT-Bench, the agent starts in an unseen environment without a prior map and receives only its own egocentric observations at each step.

The revision states the unseen-environment setting explicitly in Sec. 3 and Sec. 4.1.

Information that becomes observable only through interaction, such as objects inside closed containers, belongs to a different setting, which we noted as future work in the conclusion.

W2. The authors don't discuss how efficient their framework is in dynamic and evolving environments.

Sorry to miss the discussion. We added the key points of this discussion to Appendix C in the revision. HiREx assumes static scenes, as do all three benchmarks we evaluate (EM-EQA, A-EQA, and GOAT-Bench). Its memory does not record when objects and views were observed, so retrieval and exploration may miss information that changes over time. The object co-visibility that guides view selection also assumes a static scene. Extending HiREx to dynamic environments is an important future direction.

A natural extension is to keep the earlier views alongside the new ones when objects move, each with its observation time, while views of the unchanged object groups stay as they are. This keeps past states while limiting memory growth. Since memory is already updated locally around newly observed and re-observed objects (Algorithm 2), the extension would need to process only the parts where a change is observed. Retrieval could also consider observation times, and exploration could add an option to re-observe whether stored evidence is still valid.

W3. HiREx's inference time could be a challenge.

HiREx adds 16% to the per-step time of the closely related 3D-Mem (Appendix B.3), mainly for additional room retrieval and view retrieval with task-view matching. In return, it improves LLM-Match by 7.9 and LLM-Match SPL by 11.6 on A-EQA, and SR by 7.2 and SPL by 2.4 on GOAT-Bench.

W4. Examples would make the explanations easier to follow.

Thank you for the good suggestion.

In the revision, we added an example to the "View layer" paragraph of Sec. 3.1 showing how the view layer is built by the object clustering criterion and the view assignment to clusters. A new figure (Fig. 16) and a video in the supplementary material (hirex_view_layer_construction_example.mp4) illustrate it step by step. To keep the figure numbers cited in our responses, we placed Fig. 16 at the end of the appendix.

We also added an example to Sec. 3.3, showing with an example question how the suitable exploration option depends on how far the relevant information has been found (Fig. 3).

If you have other ideas for examples, please feel free to suggest them to us.

AuthorsOpenReview ↗

Response to Reviewer ogbn(2/2)

Q1. How does HiREx capture relations between objects?

HiREx captures relations between objects by storing images that show them together. The view layer selects views so that diverse groups of objects observed together are kept (L196–200, L212–213), and when a query arrives, the VLM reads the relations it needs directly from them. This keeps every relation visible in the image available, including ones hard to label in advance, whereas explicit relations such as “on” or “next to” are fixed before any query and may not include the relation a later query needs.

Q2. How does HiREx decide whether gathered evidence is sufficient to terminate exploration? (L268-269)

HiREx queries the VLM to decide whether to stop or continue exploration, given the task and the retrieved views (Sec. 3.4 — Termination check). The prompt is shown in Fig. 14(a) and Fig. 15(a). We added a reference to Sec. 3.4 at L268–269.

Q3. Why did you separate VLM’s termination decision from exploration?

Because once the agent decides to continue, the exploration query can require the VLM to select one option, since the agent has already decided to gather more information. Also, keeping termination separate also aims to avoid letting the available options sway it, such as stopping early because no option looks promising, or continuing because an option looks related to the query even when the gathered evidence suffices.

Q4. Why was GPT-4o used for the VLM in HiREx, but not the latest models?

It is for a fair comparison with competitive baselines, whose reported results and official implementations use GPT-4o. HGR (ECCV26’) does so for a fair comparison despite newer models. Our gains are not specific to GPT-4o; Our method also outperforms all baselines with an open VLM on EM-EQA (Table 1).

Q5. How do you compare this work with ActiveVOO?

The two methods differ in two key ways, although both direct exploration by the task.

  1. They differ in what their exploration decisions start from. HiREx chooses among exploration options built from what it has already observed, whereas ActiveVOO first decides which objects to look for and then finds them through observation.
  2. Unlike ActiveVOO, which prioritizes knowledge relevant to the current task, HiREx maintains a task-agnostic memory that can be reused for future tasks, as in the later subtasks of GOAT-Bench.
  3. (Minor) Targeting different tasks, EQA and goal navigation for HiREx and embodied planning for ActiveVOO.
AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for reviewing our paper and for the constructive feedback.

As the discussion period ends in about two days, we wanted to kindly check whether our response has addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.

Reviewer_KjDNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HiREx, a hierarchical visual memory for embodied exploration and reasoning. Instead of representing the environment mainly through abstract object nodes, HiREx organizes observations into room, view, and object layers. The view layer retains images selected to preserve object co-visibility, rooms provide semantic anchors for retrieval, and the object layer stores detailed crops. Given a task, HiREx first restricts retrieval using relevant rooms and then retrieves relevant views using object classes and task-view similarity. The retrieved hierarchy is additionally used to determine possible exploration actions, including room exploration and view refinement. The experiments cover EM-EQA, A-EQA, and GOAT-Bench. Ablations analyze the proposed view construction, retrieval stages, and exploration options.

Critical strengths

I think the strongest contribution is the attempt to use one hierarchical representation for both retrieving existing evidence and deciding where additional evidence should be collected. Room retrieval can activate room exploration, while view retrieval can activate viewpoint refinement. This is a clean and intuitive formulation of memory-guided active perception.

The authors do more than present the final model. They ablate room retrieval, similarity-based retrieval, view construction, room exploration, and view refinement. This makes it easier to understand where the improvements within HiREx come from.

Critical weaknesses

Important closely related work appears to be missing, making novelty and empirical positioning hard to assess This is my main concern. R4 is extremely relevant to this paper. It proposes a training-free persistent 4D memory for embodied VLM reasoning, performs structured retrieval over semantic, spatial, and temporal information, and evaluates directly on OpenEQA EM-EQA and A-EQA. (R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space)

Again, I cannot tell whether these numbers are fully apples-to-apples, and I would not reject simply because another paper reports higher scores. But I think the authors need to explain this comparison. Right now, using 3D-Mem as one of the central reference points makes the experimental positioning look incomplete when a substantially stronger and highly related memory/retrieval method exists. SNOW also looks relevant to the representation side. It constructs a persistent 4D scene graph combining visual semantics, geometry, and temporal information and exposes this representation to VLMs for downstream reasoning. I would like the authors to explain how HiREx's observation-centric room-view-object hierarchy differs from this type of structured 4D scene representation, and what information is gained or lost by retaining images rather than building explicit spatio-temporal object representations. This missing discussion makes the novelty claim much harder for me to assess.

The paper argues that an important contribution is that the level at which evidence is retrieved determines which exploration options are activated. However, Table 5 mainly removes room exploration and view refinement separately. I would like to see the more direct baseline where all exploration options are always available, but are not activated based on hierarchical retrieval. That would tell us whether the gain really comes from retrieval-conditioned exploration or simply from introducing stronger room/view exploration primitives.

Questions for authors
  1. Could the authors clarify how HiREx is conceptually different from R4, and why R4 is not included in the related work or experimental comparison? Are the reported OpenEQA results directly comparable? If not, what are the concrete differences in evaluation protocol? If they are comparable, I believe a direct comparison is needed.

  2. SNOW builds an explicit 4D scene graph containing semantic, geometric, and temporal object information that can be queried by VLMs. How should I understand the advantage of the proposed observation-centric room-view-object hierarchy relative to this kind of persistent 4D scene-graph representation? In particular, which reasoning capabilities are enabled by retaining hierarchical visual observations that cannot already be supported by explicit semantic/spatial/temporal object memory, and vice versa? I think this clarification is important for establishing the actual novelty of the representation.

  3. Is hierarchical activation itself responsible for the exploration gain? Have the authors compared against a variant in which room exploration, view refinement, and frontier exploration are all always available to the VLM, rather than being activated according to the level of retrieved evidence? This would more directly test the central claim that the retrieval hierarchy itself provides a useful signal for structuring exploration.

  4. Can the authors provide uncertainty estimates or repeated-run results for A-EQA and GOAT-Bench? Given the evaluation sizes and VLM-based decisions, this would help establish how robust the reported differences are.

3 comments
AuthorsOpenReview ↗

Response to Reviewer KjDN(1/2)

Summary of our response.

  • We added R4 to the related work, clarified how HiREx differs from it, and explained why the reported OpenEQA results are not directly comparable (W1, Q1).
  • We clarified how HiREx’s room–view–object hierarchy differs from explicit 4D scene graphs, including the reasoning gained and lost (W2, Q2).
  • An additional experiment shows that offering exploration options based on hierarchical retrieval, a core contribution of HiREx, yields higher exploration efficiency than simply offering more options (W3, Q3).
  • Per-run results show that our improvements over the best baselines hold in every run (Q4).

Note on OpenEQA scoring. gpt-4-1106-preview, the OpenEQA scorer, was removed from the OpenAI API on October 23, 2026. All new OpenEQA results in this response are therefore scored with gpt-5.6-sol, the substitute that OpenAI designates (https://developers.openai.com/api/docs/deprecations#past-deprecations). Our improvements over prior work hold under the original scorer used in the submission, and the new analyses compare variants of our method under gpt-5.6-sol.

W1-1 & Q1-1.

W1-1. Important closely related work—R4 [1]—appears to be missing.

Thank you for the suggestion. We added R4 as important related work (Related Work — Scene Representation for Embodied Agents), which keeps object-level records of semantic, spatial, and temporal attributes in a persistent 4D memory and retrieves them for VLM reasoning without training.

[1] R4: Retrieval-Augmented Reasoning for Vision-Language Models in 4D Spatio-Temporal Space, CVPR 2026.

Q1-1. Clarify how HiREx is conceptually different from R4:

The biggest difference is the exploration strategy. HiREx explicitly constructs and activates exploration options based on hierarchical retrieval, whereas R4 does not specify how retrieved information guides exploration actions.

Also, HiREx keeps a compact set of observed images (views) that show different combinations of objects together, and the VLM reads the evidence a query needs directly from the retrieved views. In contrast, R4 extracts object-centric descriptions from previously observed images without preserving them, which may discard essential information the query later needs (e.g., background, or relations visible when several objects are seen together). The example in Q2-2 shows a case where such information is needed.

W1-2 & Q1-2.

W1-2. Why is 3D-Mem the primary experimental baseline despite R4's higher reported performance?

We use 3D-Mem as a primary baseline for our detailed analyses, while our main evaluations also include more recent approaches, SplatReasoner (ECCV’26) on EM-EQA and HGR (ECCV’26) on A-EQA and GOAT-Bench. HiREx outperforms them under the same evaluation setting (Tables 1–3). 3D-Mem suits these analyses because it shares HiREx’s high-level framework, which enables intuitive comparisons of frame efficiency, qualitative results, and failure cases across memory, retrieval, and exploration (Figs. 4–5, Appendix B.5). R4 has not released its code, so we cannot run it on the same episodes and conditions for these analyses (see Q1-2).

Q1-2. Are HiREx and R4's OpenEQA results directly comparable?

No, R4's results are not directly comparable to HiREx's under controlled settings, despite being reported on the same benchmarks, for two reasons.

First, R4’s OpenEQA evaluation is difficult to reproduce faithfully. Its official code has not been released, and essential implementation details remain unclear.

In particular, R4 identifies its retrieval-augmented reasoning loop as its core contribution, but does not report the detailed prompts (answerability check, retrieval handbook, and final reasoning) that produce its results on both EM-EQA and A-EQA.

Moreover, A-EQA requires the agent to explore an unseen environment to gather information, yet R4 describes no exploration policy. It states only that its retrieval framework can also support navigation decisions and reports neither the action space nor how it chooses the next target. Its A-EQA results therefore cannot be compared with ours, since both LLM-Match and LLM-Match SPL depend on what the agent observes along its path.

Second, a controlled accuracy-efficiency comparison is difficult because VLM processing requirements differ.

HiREx evaluates accuracy with compact VLM inputs, averaging 2.7 retrieved frames per question on EM-EQA (Table 1), and reports performance across frame budgets (Fig. 4).

In contrast, R4 prompts a VLM for every segmented object in each frame during memory construction, so its VLM processing grows with the number of observed objects, which is large in cluttered indoor scenes such as those in OpenEQA. Its accuracy therefore reflects a different amount of VLM processing from ours.

If the reviewer finds it helpful, we will add R4’s reported EM-EQA and A-EQA results to Tables 1 and 2 as footnotes, together with these differences.

AuthorsOpenReview ↗

Response to Reviewer KjDN(2/2)

W2 & Q2.

W2-1 & Q2-1. How does HiREx’s observation-centric room–view–object hierarchy differ from persistent 4D scene graphs?

Unlike 4D scene graphs, HiREx retains visual information beyond individual objects, capturing relationships among objects seen together as observed so the VLM can read the relations each query requires directly from the views. While 4D scene graphs add temporal information to object nodes, the visual information each node holds remains about its own object (e.g., patch tokens [1], crops [2]), and inter-object relations, if any, are abstracted into forms such as geometric relations or predicate labels. These relations are chosen before any query, so the relation a later query needs may not be among them.

Beyond holding these views, the room and view layers of HiREx organize them for retrieval and exploration, narrowing the candidate views by room and determining which exploration options are offered. We added this comparison to Sec. 2 of the revision.

[1] SNOW: Spatio-Temporal Scene Understanding with World Knowledge for Open-World Embodied Reasoning, ECCV 2026.

[2] ROSE: Real-Time Open-World Scene Understanding from Monocular Video via Compact Multimodal 4D Scene Graphs, ECCV 2026.

W2-2 & Q2-2. What information and reasoning capabilities are gained or lost by retaining hierarchical visual observations rather than explicit spatio-temporal object memory?

Gained: HiREx can reason about information beyond individual objects that wasn't anticipated when the memory was built, such as how objects are arranged and the context around them, since the VLM reads it from the retained views after the query arrives.

For example,

“If I sit in the armchair in the living room, what animal can I touch with my right hand?” (Fig. 8)

requires the armchair’s facing direction and the arrangement of objects around it. Object-level geometry, such as centroids and extents, does not encode this, whereas a view that captures the armchair with its neighbors lets the VLM read it, and HiREx answers this question correctly.

Lost: Reasoning about temporal changes and metric units is harder. HiREx assumes static scenes, as do the benchmarks we evaluate. It does not record temporal information about objects, so it cannot answer queries about motion or change. Distances and sizes are not given to the VLM explicitly, although the VLM can still estimate them from the retained images.

Overall, embodied questions require reasoning well beyond metric quantities, as OpenEQA also covers object recognition, attributes, states, functions, and commonsense in addition to spatial understanding. Retained observations preserve the visual evidence such reasoning draws on, without deciding in advance which evidence a query will need.

W3 & Q3. Where does the exploration gain come from: offering options based on the retrieved evidence (a key claimed contribution) or simply introducing new options?

Offering options based on the retrieved evidence yields higher exploration efficiency than simply introducing new options. Compared with Always-on, which simply introduces new options at every step regardless of retrieval, HiREx raises LLM-Match SPL by 3.1 at comparable LLM-Match, larger than the drop from removing either new option from Always-on (2.2 and 0.5). The new options also contribute, mainly view refinement, as removing it from Always-on lowers both metrics.

MethodLLM-Match↑LLM-Match SPL↑
Always-on w/o view refinement57.144.5
Always-on w/o room exploration58.046.2
Always-on58.746.7
HiREx59.049.8

Tested on the same 184 A-EQA questions. Always-on builds room exploration for the room the agent is in and view refinement on the views nearest to the agent, matching the number of retrieved views. All other components are unchanged. Scored with gpt-5.6-sol (see Note on OpenEQA scoring). We replaced Table 5 of the revision with these results.

Q4. Can the authors provide uncertainty estimates or repeated-run results for A-EQA and GOAT-Bench?

Here, we present the per-run results for Tables 2 and 3. The numbers in these tables are averages over three runs with different random seeds, which we did not state in the paper. Every run outperforms the best baseline on all four metrics, so the reported differences hold across runs.

A-EQAGOAT
LLM-Match↑LLM-Match SPL↑SR↑SPL↑
1 (seed: 77)57.649.477.353.3
2 (seed: 0)58.648.476.252.6
3 (seed: 1234)56.248.176.253.4
Avg ±\pm std57.5 ±\pm 1.2148.6 ±\pm 0.6876.6 ±\pm 0.6453.1 ±\pm 0.44
Best baseline52.039.869.450.7

We scored these results with the original scorer, as in the submission, since we obtained them before the scorer was removed. In the revised paper, we state the number of runs in Sec. 4.1.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you again for reviewing our paper and for the constructive feedback.

As the discussion period ends in about two days, we wanted to kindly check whether our response has addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.

Reviewer_Fxu5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HiREx, an observation-centric memory framework for Embodied Question Answering (EQA) and multimodal goal navigation. The method organizes observations into a room–view–object hierarchy, using child-aware co-visibility clustering and representative view selection to preserve contextual information. It then retrieves task-relevant evidence through room filtering, category-based retrieval, and image–text similarity matching. When the retrieved evidence is insufficient, the system activates room exploration and view refinement based on the retrieval results, forming a closed-loop process together with frontier exploration. The paper reports substantial overall improvements across three task settings. However, the distinctions from closely related methods and the independent contributions of its core mechanisms still need to be more clearly established.

Critical strengths

1.The research motivation is clear, focusing on preserving visual evidence, retrieving task-relevant information, and actively gathering additional observations when the available evidence is insufficient. The method integrates co-visibility-driven view construction, hierarchical retrieval, and multi-level exploration strategies into a unified closed-loop framework, with each component clearly corresponding to a specific problem it aims to address. 2.The experimental evaluation is relatively comprehensive, covering episodic-memory EQA, active EQA, and multimodal goal navigation, along with component ablations, comparisons of retrieval strategies, runtime analysis, and failure case analysis.

Critical weaknesses

1.The distinction from prior work is not sufficiently precise. In particular, GraphEQA already selects task-relevant images using SigLIP-based text–image similarity, which is not adequately reflected in the discussion in Sec. 2 that groups prior image access mechanisms as dependent on detected objects. Although HiREx introduces specific view-construction and retrieval-conditioned exploration mechanisms, the paper should more clearly distinguish these contributions from existing visual-memory retrieval and hierarchical planning approaches. 2.Table 5 demonstrates the value of the additional exploration options by removing either room exploration or view refinement. However, these ablations also change the system's action space, making it difficult to determine whether the improvements come from a richer set of actions or from the rule that activates specific actions based on the retrieval hierarchy. 3. The co-visibility comparison in Sec. 4.3 uses different memory sizes: HiREx retains 15% of observations and covers 75% of co-visible class pairs, whereas the baseline retains 11% of observations and achieves 50% coverage. Therefore, this comparison cannot distinguish whether the improvement in coverage comes from more effective view selection or simply from retaining more observations. Although Fig. 4 evaluates QA performance with comparable numbers of VLM input frames, it does not control for the size of the stored memory.

Questions for authors

1.How does HiREx prevent incorrect retrieval results from further misleading exploration? For example, if the system retrieves the wrong room or misses a critical view, could activating exploration options based on these retrieval results restrict subsequent search? 2.Does the performance improvement from view construction mainly come from retaining more images or from preserving more useful visual information? In Sec. 4.3, HiREx and the baseline retain different proportions of observations. Under the same memory budget, can HiREx preserve more evidence essential for answering questions? Furthermore, is there direct experimental evidence supporting a relationship between higher coverage of co-visible class pairs and improved QA accuracy?

6 comments
AuthorsOpenReview ↗

Response to Reviewer Fxu5 (1/3)

Summary of our response.

  • We revised the description of GraphEQA and clarified how HiREx differs from visual-memory retrieval and hierarchical planning methods (W1).
  • We added a comparison with offering all options regardless of retrieval (Always-on), where activating options based on hierarchical retrieval, a core contribution of HiREx, raises exploration efficiency the most (W2, Q1).
  • At a comparable memory size, our view layer keeps more co-visible class pairs with higher accuracy, and a controlled experiment shows that higher pair coverage comes with higher accuracy (W3, Q2).

Note on OpenEQA scoring. gpt-4-1106-preview, the OpenEQA scorer, was removed from the OpenAI API on October 23, 2026. All new OpenEQA results in this response are therefore scored with gpt-5.6-sol, the substitute that OpenAI designates (https://developers.openai.com/api/docs/deprecations#past-deprecations). Our improvements over prior work hold under the original scorer used in the submission, and the new analyses compare variants of our method under gpt-5.6-sol.

<Line numbers (L) refer to the revised paper unless stated otherwise.>

W1-1. The distinction from prior work is not sufficiently precise. GraphEQA already selects task-relevant images by text–image similarity, unlike Sec. 2’s description of prior image access mechanisms as dependent on detected objects.

Thank you for pointing this out. L127–130 of the submission grouped GraphEQA with methods that access images through detected objects, and we revised them. HiREx differs from prior methods by combining both ways of accessing stored images. Prior methods use either text–image similarity alone, as in GraphEQA, or detected objects, as in 3D-Mem, SplatReasoner, and MSGNav. HiREx uses both as complementary retrieval paths and splits the view budget between them, which outperforms either path alone at every allocation we tested (Fig. 7, Appendix B.2).

W1-2. How do HiREx’s view construction and retrieval-conditioned exploration differ from existing visual-memory retrieval and hierarchical planning approaches, respectively?

HiREx’s view construction vs. visual-memory retrieval. GraphEQA keeps only the images most similar to the question and discards the rest. 3D-Mem aims to cover every observed object with a minimum number of images. MSGNav keeps every image in which a nearby object pair appears, and SplatReasoner builds a 3D Gaussian scene from all frames to extend stored views with novel view synthesis. HiREx keeps views independently of the task (L244), aims to preserve diverse groups of co-observed objects rather than covering each object once (L196-198), and stores only a compact subset of the observations (L211-213).

HiREx’s retrieval-conditioned exploration vs. hierarchical planning. GraphEQA reasons over a hierarchical scene graph to choose an object or frontier node to visit. HOV-SG selects one node per layer by CLIP similarity and plans a path to it. In both, the hierarchy picks a target among existing nodes. In HiREx, retrieval selects several relevant rooms and views, and each adds an exploration option at its level. A separate VLM decision then compares all activated options, including frontiers, each presented visually. Hierarchical retrieval thus constructs the exploration candidates, and the VLM decides among them. (L293-300)

We revised Sec. 2 to state both distinctions.

W2. Do the exploration gains come from a richer set of actions or from activating actions based on the retrieval hierarchy?

Both, and activating actions based on the retrieval hierarchy, a core contribution of HiREx, raises exploration efficiency the most among the compared settings. It improves LLM-Match SPL by 3.1 over offering all options regardless of retrieval (Always-on), larger than the drop from removing either option from Always-on (2.2 and 0.5). The richer set of actions also contributes, mainly through view refinement, as removing it from Always-on lowers both metrics.

MethodLLM-Match↑LLM-Match SPL↑
always-on w/o view refinement57.144.5
always-on w/o room exploration58.046.2
always-on58.746.7
HiREx59.049.8

Tested on the same 184 A-EQA questions. Always-on builds room exploration for the room the agent is in and view refinement on the views nearest to the agent, matching the number of retrieved views. All other components are unchanged. Scored with gpt-5.6-sol (see Note on OpenEQA scoring). We replaced Table 5 of the revision with these results.

AuthorsOpenReview ↗

Response to Reviewer Fxu5 (2/3)

W3 & Q2

W3. The comparison of co-visible class pairs kept in memory (Sec. 4.3) cannot distinguish whether the higher coverage comes from more effective view selection or simply from retaining more observations.

We agree, and therefore compared the two at a comparable memory size. The higher coverage comes from more effective view selection, as our view selection still retains more co-visible class pairs than the baseline at the matched memory size (Table A). To match the memory size, we lowered the similarity threshold η\eta of our view layer (Algorithm 1, line 9) from 0.85 to 0.7. Since η\eta only decides whether a group’s view is distinct enough from those already kept (L212–213), lowering it keeps fewer views while the selection rule stays the same.

Q2-1. Does the performance improvement from view construction come simply from retaining more images?

No. Our view layer achieves higher LLM-Match than the baseline at a memory size matched to the baseline, even while retrieving fewer images for the VLM (Table A). The improvement thus comes from which visual information it keeps, not simply from storing more images. We added this comparison to Appendix B.7 (Table 10), with a summary in Sec. 4.3.

Table A.

VIew selectionStored views (% of observations)Co-visible class pair coverage (%)LLM-Match↑Avg. frames (retrieved images)
Baseline (prior strategy)11.450.163.82.64
Ours (η\eta=0.7)11.671.767.62.34

EM-EQA 300-question subset. Both memories use the same retrieval as the view construction ablation in Table 4. Avg. Frames is the average number of retrieved images given to the VLM per question. Coverage is the fraction of distinct class pairs seen together in any observation that are also seen together in at least one stored view. For token cost, these runs use gpt-6-luna as the VLM. Scored with gpt-5.6-sol (see Note on OpenEQA scoring).

Q2-2. Is there direct experimental evidence that higher coverage of co-visible class pairs improves QA accuracy?

Yes. At a fixed memory size, LLM-Match rises with co-visible class-pair coverage, from 55.9 at 49% to 65.3 at 94% (Table B). To vary the coverage, the memory of size B consists of ⌊ρB⌋ images greedily selected to cover the most co-visible class pairs and B − ⌊ρB⌋ images selected at random, with B set to 11.4% of the observations in each episode as in the baseline.

Table B.

ρ\rhocoverage of co-visible class-pair(%)LLM-Match↑Avg. Frames (VLM input)
04955.92.22
0.257260.12.40
0.58661.42.44
0.759063.02.43
19465.32.47

EM-EQA 300-question subset with the same episodes, retrieval, and definition of coverage as Table A. For token cost, these runs use gpt-6-luna as the VLM. Scored with gpt-5.6-sol (see Note on OpenEQA scoring).

Q2-3. Can HiREx preserve more evidence essential for answering questions under the same memory budget?

Yes, HiREx preserves more evidence essential for answering questions. Whether a memory keeps more of such evidence can be judged by whether it leads to more correct answers under the same memory size, retrieval, and answering VLM. Our view layer answers more questions correctly than the baseline while using fewer retrieved images (Table A), and also more than a memory selected to cover as many co-visible class pairs as possible (ρ = 1 in Table B), although its own pair coverage is lower.

AuthorsOpenReview ↗

Response to Reviewer Fxu5 (3/3)

Q1. How does HiREx prevent incorrect retrieval results from further misleading exploration?

Incorrect retrieval could mislead exploration in two ways. A falsely retrieved room or view could steer the agent toward an irrelevant region (false positive), and a missed one could leave the relevant region unexplored (false negative). HiREx mitigates both by treating retrieval results as exploration options rather than commands, and by retrieving again at every step.

False positive) HiREx does not force the agent to explore a retrieved region. The VLM decides among all activated options, which always include frontier exploration. Each option is given to the VLM marked on the views that observe its location, rather than as a list of retrieved rooms or views, so the VLM judges from what these views show whether the option is relevant.

False negative) HiREx runs retrieval at every step over the updated memory, so evidence missed at one step may be retrieved later as new observations are added, while frontier exploration continues.

Two results support robustness to retrieval errors. First, exploration efficiency, measured by LLM-Match SPL, stays close to the original when retrieval becomes less reliable. Replacing the LLM for room or view retrieval with a smaller open model, which can both retrieve irrelevant regions and miss relevant ones, keeps LLM-Match SPL close to the original (Table 6). Second, activating options by retrieval improves LLM-Match SPL overall despite its possible errors. Compared with offering options at every step regardless of retrieval (Always-on in W2), HiREx reaches higher LLM-Match SPL with comparable LLM-Match.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Gentle Reminder as the Discussion Phase Nears Completion

Dear Reviewers,

As the discussion phase concludes, we thank all reviewers for the time and care devoted to our paper. Your questions led to new analyses and clarifications, which are now part of the revision (changes in blue).

For reviewers who have not yet had a chance to participate in the discussion, we would be sincerely grateful if you could let us know whether our responses have addressed your concerns.

The main revisions are as follows.

  • Analyses (Sec. 4.3, Table 5, Appendix B.7): compared HiREx with offering all exploration options regardless of retrieval (Always-on), and compared our view selection with the prior strategy at a matched memory size.
  • Related work (Sec. 2): revised the description of GraphEQA and added hierarchical planning methods, R4, and 4D scene graphs.
  • Problem setting (Sec. 3, Sec. 4.1, Appendix C): made explicit that the agent explores a static, unseen environment, and discussed how HiREx can extend to dynamic environments.
  • Examples (Sec. 3.1, Sec. 3.3, Appendix E): to make the method easier to follow, added examples of the view-layer construction (Fig. 16 and a supplementary video) and of how exploration options depend on the retrieved evidence.

We hope these revisions clarify the paper and address the reviewers' concerns. If any questions remain, we would be glad to address them before the discussion period ends.

Thank you again for your time and effort in reviewing our paper.

Best regards,

Authors of HiREx

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.