- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors argue that 3D scene graphs abstract the scene into objects and relations, discarding visual evidence; whereas image-set methods preserve evidence but retrieve it through detected classes, inheriting their errors. And both categories separate exploration from evidence retrieval.
To this end, they propose HiRex, which is a scene memory (memorizing the scenes in 3-layers, namely rooms, view, and objects), which enables both retrieval and exploration operations.
Finally, the authors show the superiority of HiRex against the baselines.
Critical strengths
- This paper touches an interesting and key problem in Embodied AI: Exploration and Reasoning
- The HiRex is a hierarchical retrieval and exploration framework bringing in the strengths of both scene graph- and image-set-based solutions
- The experiments show strong performance against multiple benchmarks
Critical weaknesses
-
Computation Cost: The inference time and computation of HiRex could be a challenge; i.e., the proposed 3-layered framework for retrieval and exploration could be expensive.
-
Dynamic Environments: The authors don't discuss dynamic and evolving environments and how efficient their framework is such settings.
-
Partially Observable Settings: The paper and the experiments lack partially observable settings which are common settings for embodied tasks
-
I wish the authors invested more time in polishing their content. Some paragraphs were not easy to read or follow. It would've been great if they could use examples for their explanations.
Questions for authors
- Sec 3.1 Object Layer: How do you capture relations between objects?
- Lines 258-259: Could you explain how you decide whether an evidence is sufficient?
- Lines 322-323: Why did you separate termination from exploration?
- Lines 364-365: I'm wondering why the GPT-4o models were used, but not the latest models?
- How do you compare this work with ActiveVOO?
- Liu, Xiaotian, et al. "ActiveVOO: Value of observation guided active knowledge acquisition for open-world embodied lifted regression planning." Advances in Neural Information Processing Systems 38 (2026): 106621-106647.
6 comments
Response to Reviewer ogbn(2/2)
Q1. How does HiREx capture relations between objects?
HiREx captures relations between objects by storing images that show them together. The view layer selects views so that diverse groups of objects observed together are kept (L196–200, L212–213), and when a query arrives, the VLM reads the relations it needs directly from them. This keeps every relation visible in the image available, including ones hard to label in advance, whereas explicit relations such as “on” or “next to” are fixed before any query and may not include the relation a later query needs.
Q2. How does HiREx decide whether gathered evidence is sufficient to terminate exploration? (L268-269)
HiREx queries the VLM to decide whether to stop or continue exploration, given the task and the retrieved views (Sec. 3.4 — Termination check). The prompt is shown in Fig. 14(a) and Fig. 15(a). We added a reference to Sec. 3.4 at L268–269.
Q3. Why did you separate VLM’s termination decision from exploration?
Because once the agent decides to continue, the exploration query can require the VLM to select one option, since the agent has already decided to gather more information. Also, keeping termination separate also aims to avoid letting the available options sway it, such as stopping early because no option looks promising, or continuing because an option looks related to the query even when the gathered evidence suffices.
Q4. Why was GPT-4o used for the VLM in HiREx, but not the latest models?
It is for a fair comparison with competitive baselines, whose reported results and official implementations use GPT-4o. HGR (ECCV26’) does so for a fair comparison despite newer models. Our gains are not specific to GPT-4o; Our method also outperforms all baselines with an open VLM on EM-EQA (Table 1).
Q5. How do you compare this work with ActiveVOO?
The two methods differ in two key ways, although both direct exploration by the task.
- They differ in what their exploration decisions start from. HiREx chooses among exploration options built from what it has already observed, whereas ActiveVOO first decides which objects to look for and then finds them through observation.
- Unlike ActiveVOO, which prioritizes knowledge relevant to the current task, HiREx maintains a task-agnostic memory that can be reused for future tasks, as in the later subtasks of GOAT-Bench.
- (Minor) Targeting different tasks, EQA and goal navigation for HiREx and embodied planning for ActiveVOO.
Official Comment by Authors
Thank you again for reviewing our paper and for the constructive feedback.
As the discussion period ends in about two days, we wanted to kindly check whether our response has addressed your concerns. If you have any remaining questions or would like further clarification, please let us know. We would be happy to address them before the deadline.
Response to Reviewer ogbn(1/2)
Summary of our response.
<Line numbers (L) refer to the revised paper unless stated otherwise.>
W1. Does HiREx address partially observable settings, which are common in embodied tasks?
Yes, HiREx addresses partial observability in embodied question answering and goal navigation, where the agent obtains missing information by moving and observing.
Our problem definition requires the agent to reason over accumulated observations and to move for information it still lacks (L34–35, L41–42). In A-EQA and GOAT-Bench, the agent starts in an unseen environment without a prior map and receives only its own egocentric observations at each step.
The revision states the unseen-environment setting explicitly in Sec. 3 and Sec. 4.1.
Information that becomes observable only through interaction, such as objects inside closed containers, belongs to a different setting, which we noted as future work in the conclusion.
W2. The authors don't discuss how efficient their framework is in dynamic and evolving environments.
Sorry to miss the discussion. We added the key points of this discussion to Appendix C in the revision. HiREx assumes static scenes, as do all three benchmarks we evaluate (EM-EQA, A-EQA, and GOAT-Bench). Its memory does not record when objects and views were observed, so retrieval and exploration may miss information that changes over time. The object co-visibility that guides view selection also assumes a static scene. Extending HiREx to dynamic environments is an important future direction.
A natural extension is to keep the earlier views alongside the new ones when objects move, each with its observation time, while views of the unchanged object groups stay as they are. This keeps past states while limiting memory growth. Since memory is already updated locally around newly observed and re-observed objects (Algorithm 2), the extension would need to process only the parts where a change is observed. Retrieval could also consider observation times, and exploration could add an option to re-observe whether stored evidence is still valid.
W3. HiREx's inference time could be a challenge.
HiREx adds 16% to the per-step time of the closely related 3D-Mem (Appendix B.3), mainly for additional room retrieval and view retrieval with task-view matching. In return, it improves LLM-Match by 7.9 and LLM-Match SPL by 11.6 on A-EQA, and SR by 7.2 and SPL by 2.4 on GOAT-Bench.
W4. Examples would make the explanations easier to follow.
Thank you for the good suggestion.
In the revision, we added an example to the "View layer" paragraph of Sec. 3.1 showing how the view layer is built by the object clustering criterion and the view assignment to clusters. A new figure (Fig. 16) and a video in the supplementary material (hirex_view_layer_construction_example.mp4) illustrate it step by step. To keep the figure numbers cited in our responses, we placed Fig. 16 at the end of the appendix.
We also added an example to Sec. 3.3, showing with an example question how the suitable exploration option depends on how far the relevant information has been found (Fig. 3).
If you have other ideas for examples, please feel free to suggest them to us.