acceptodds
Under review as a conference paper at ICLR 2027

RWMNav: Relational World Model for Compositional Zero Shot Object Navigation

Abstract

Zero shot object navigation requires an agent to leverage open vocabulary semantic knowledge to search for unseen target categories in unfamiliar environments. However, existing approaches typically represent navigation goals as a single object category, while natural language instructions in real world settings often specify the intended target instance through its relations with surrounding entities. Such relational constraints require the agent not only to recognize the target category, but also to reason about the compositional structure formed by the target, reference entities, and their relations, even before these entities become directly observable. To address this challenge, we propose RWMNav, a Relational World Model for compositional zero shot object navigation. RWMNav parses the language instruction into a relational goal graph and incrementally constructs an environmental relation graph from RGB-D observations. It then treats candidate frontiers as high level exploration actions and uses a training free relational world model instantiated with a frozen vision language model to predict the future relational states that may emerge along different exploration directions, including potential regions, entities, and their relations. The predicted relation graphs are matched with the relational goal graph to estimate goal compatibility, which is further combined with exploration utility and navigation cost for frontier selection. As exploration proceeds, predicted relational hypotheses are verified and corrected using newly acquired observations, forming a closed loop process of relational prediction, goal matching, exploration, and state update. Experiments on HM3D and MP3D demonstrate that RWMNav consistently improves relational navigation performance in terms of Relational Success Rate and Relational Success weighted by Path Length.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Zero-Shot Object Navigation, Relational World Model, Long-horizon Reasoning, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_QpJcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new task setting, relational ObjectNav, and designs a navigation framework built around future relation graph prediction and goal graph matching. The authors validate the effectiveness of the framework through a newly constructed relational benchmark, comparisons with state-of-the-art methods, module ablations, and experiments on different relation types.

Critical strengths
  1. The relational ObjectNav setting addresses a meaningful problem. The paper extends conventional ObjectNav from "finding any instance of a given category" to "finding a specific instance that satisfies relational constraints". This setting is closer to the needs of real-world language-guided navigation, and it also makes relational reasoning a necessary capability.
  2. The method fits the task well, and the experimental results support its effectiveness. RWMNav designs a complete pipeline around the relational goal, future relation prediction, and goal graph matching, and outperforms existing methods on multiple benchmarks. The ablation study also shows that these relational modeling components make a clear contribution to performance.
Critical weaknesses
  1. The core pipeline mainly combines existing techniques, including scene graph reasoning, VLM-based future state prediction, and frontier selection, and applies them to relational ObjectNav. Within this framework, the future relation graph appears to be a natural extension of existing representations rather than a fundamentally new world-modeling or planning mechanism. Furthermore, the novelty of the task setting itself is also limited. Although the specific task formulation differs from prior work, related navigation studies, such as HRON and SOON, have already explored finding specific object instances using language and relational constraints. Therefore, the paper's contributions in terms of both task formulation and methodological design remain relatively limited.
  2. This paper shows that having future relational reasoning is better than removing the module entirely, but it lacks a direct comparison with simpler future value prediction or with direct VLM frontier scoring. It therefore remains unclear whether the performance gain actually comes from representing the future as a graph, or from the relational goal itself together with stronger adaptation to the task.
Questions for authors
  1. How were the existing category-level ObjectNav methods adapted to the relational setting of this paper? In particular, how do they handle the relational instructions, target-instance determination, and the stopping condition? The answer to this question will directly affect my assessment of the fairness of the SOTA comparison.
  2. How much additional benefit does explicitly constructing the future relation graph actually bring compared with simpler future prediction? For example, using the same VLM, the same relational instruction, and the current environment context, what would the performance be if the VLM directly scored the candidate frontiers, without constructing and matching a future relation graph? This comparison will directly affect my assessment of the contribution of the core method.
1 comment

W1: Novelty of RWMNav’s task formulation and methodology

Thank you for the comment. HRON and SOON study navigation to instances specified by relational or linguistic descriptions. Building on this work, we ask a more specific question: what environmental information should an agent predict to guide exploration when the target and its reference objects have not yet been sufficiently observed?

We develop a geometrically verifiable evaluation protocol for compositional relational navigation. It links linguistic relational constraints to a unique target instance in the 3D scene and enables consistent evaluation across HM3D and MP3D. It assesses whether an agent finds the instance satisfying these relations.

To our knowledge, RWMNav is the first framework for object-instance navigation with compositional relational goals that combines frontier-conditioned future relation graph prediction with relational goal graph matching to select exploration directions, without requiring task-specific navigation training. Although HRON and SOON study navigation to relationally specified targets, their learned navigation methods do not use this per-frontier prediction and matching mechanism. RWMNav uses a frozen vision-language model to predict potential entities and relations beyond each candidate frontier, then ranks directions by the compatibility of predicted configurations with the relational goal. Thus, even when two directions may contain potential targets of the same category, the agent can distinguish them based on the predicted relations between those targets and their reference entities.

W2: Do future relation graphs offer additional benefits over simpler prediction and direct VLM frontier scoring?

Thank you for the suggestion. We have added a comparison with direct VLM frontier scoring. Both methods use the same VLM (Qwen2.5-VL-Instruct), relational goals, current environmental context, and task adaptation procedure. The direct scoring variant replaces future relation graph construction and matching with direct VLM estimates of the expected value of exploring each candidate frontier for achieving the goal.

MethodRSR ↑RSPL ↑
Direct frontier scoring50.123.7
RWMNav61.030.3

Compared with direct scoring, RWMNav improves RSR and RSPL by 10.9 and 6.6 percentage points, respectively. The comparison shows that the additional gains come from explicitly representing future states as relation graphs and matching them to the relational goal graph to guide exploration. We will include these results in the revised manuscript.

Q1: How are category-level ObjectNav baselines adapted for relational instructions, target instance selection, and STOP decisions?

Thank you for pointing this out. All baselines receive the same relational instructions and retain their original exploration strategies. Interfaces that accept only object categories use the target category for search, while those supporting full language input use the complete instruction. A shared verifier checks all relational constraints. Thus, although the methods differ in how they explore, they use the same criteria to verify the intended target instance.

When a candidate target is detected, the verifier uses actual observations to check whether all relations hold for the same target instance under a consistent assignment of reference objects. The agent continues exploring if the evidence is insufficient. It stops only after relational verification succeeds and the minimum Euclidean distance from its horizontal position to the horizontal footprint of the target’s bounding box is at most 0.2 m. Ground truth target annotations are used only for offline data generation and evaluation, and do not inform online navigation decisions. This prevents the original category-based stopping rules from introducing an additional source of bias while preserving each method’s exploration strategy.

Q2: How much does explicit future relation graph modeling improve performance over direct VLM frontier scoring?

Thank you for the suggestion. We compare the two methods using the same VLM, full relational instructions, and current environmental context, while keeping the task adaptation procedure unchanged. Direct VLM scoring without constructing or matching future relation graphs achieves an RSR of 50.1 and an RSPL of 23.7, compared with 61.0 and 30.3 for the full RWMNav. Explicitly predicting future relation graphs and matching them to the goal graph therefore improves RSR and RSPL by 10.9 and 6.6 percentage points, respectively, over direct scoring. These gains reflect higher navigation success and better path efficiency for relational goals.

If our response addresses your concerns, we would be grateful if you could consider revising your score upward. Thank you again for your time and consideration.

Reviewer_P26kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RWMNav addresses relational zero-shot object navigation by parsing instructions into goal graphs and maintaining an observed scene graph from RGB-D inputs. A frozen VLM predicts region–object–relation graphs beyond candidate frontiers. Frontier selection combines goal-graph compatibility, exploration utility, and travel cost. Evaluation uses geometrically generated relational episodes on HM3Dv1, HM3Dv2, and MP3D, with component ablations and results by relation type.

Critical strengths
  1. Table 1 reports RSR and RSPL improvements over SG-Nav across three evaluation sets. Table 2 reports decreases when future-state prediction or goal matching is removed. These results provide evidence for the combined pipeline under the reported protocol.

  2. The evaluation assigns a designated target instance using geometric relational constraints, rather than accepting any instance of the target category. This aligns the success criterion with the intended instance-disambiguation task.

Critical weaknesses

Critical Weaknesses

  1. The authors combine relational goal graphs with VLM-based prediction of unexplored scene structure. However, [1] already studies relation-specified navigation goals, and [2] already uses VLM-based scene-graph completion to guide frontier exploration. Their settings differ from this submission, but these connections are not discussed, leaving the additional contribution insufficiently delineated. Comparing against [2] and an entity/region-only prediction variant, with perception and final target verification held fixed, would help establish the contribution of predicting relational edges.

[1] Lingelbach, Michael et al. “Task-Driven Graph Attention for Hierarchical Relational Object Navigation.” 2023 IEEE International Conference on Robotics and Automation (ICRA) (2023): 886-893.

[2] Hu, Yue, et al. "Imaginative world modeling with scene graphs for embodied agent navigation." arXiv preprint arXiv:2508.06990 (2025).

  1. The authors generate relational episodes through geometric predicates and evaluate existing navigation methods on the resulting instructions. However, Appendix A omits numerical predicate thresholds and reference frames for directional relations, while the baseline adaptations for relational instance selection and stopping are not described. These choices can change episode labels and success outcomes, limiting interpretation of Table 1. Please specify the predicates and baseline adaptations. The rollout prompt and graph-matching computations in Appendix A.2–A.3 also need operational definitions to reproduce frontier rankings.
Questions for authors
  1. How does each baseline use the relational instruction and verify the intended target instance before stopping? What distance definition is used for the 0.2 m success threshold?

  2. Can you provide the rollout backbone and prompt, graph-matching computations and weights, and the geometric thresholds/reference frames used to construct directional relations?

  3. With perception, final target verification, backbone, and VLM-call budget held fixed, how does relational rollout compare with direct frontier scoring and prediction without relational edges?

1 comment

W1 & Q3: Contribution and controlled comparisons

Thank you for these comments. HRON [1] learns policies for relational goals; SGImagineNav [2] uses training-free hierarchical scene prediction for exploration. RWMNav matches future entity-relation configurations predicted for each frontier to a compositional goal graph, using a frozen VLM without navigation training. Predicted references and relations distinguish directions likely to contain the same target category.

All variants share the VLM, full instructions, goal parsing, observations, perception, verification, and planning. Direct scoring replaces graph prediction and matching with VLM frontier scores. Region/entity completion predicts no relation edges and uses entity coverage alone. Scores are normalized to [0,1][0,1] and enter the same frontier formula. Each round considers up to four frontiers, one call per frontier, and at most 1,024 output tokens per call, with identical filtering and replanning. HM3Dv2 results follow.

MethodRSR ↑RSPL ↑
Direct frontier scoring50.123.7
Region/entity completion54.224.6
RWMNav61.030.3

RWMNav gains 10.9 RSR and 6.6 RSPL percentage points over direct scoring. The region/entity comparison further supports the benefit of predicting and matching relational edges under the same budget constraints.

W2 & Q1: Baseline adaptation and STOP

Baselines share relational instructions and episodes, retaining their exploration strategies. Category-only interfaces search using the target category; language-capable interfaces use the full instruction. A shared verifier checks all constraints using actual observations, requiring one target instance and a consistent assignment of references. Insufficient evidence leads to continued exploration.

STOP requires successful verification and a minimum Euclidean distance of at most 0.2 m from the agent's horizontal position to the horizontal footprint of the target's bounding box. Unsuccessful episodes reaching 500 steps count as failures. Ground truth is used only for offline episode generation and evaluation, not online decisions.

W2 & Q2: Prediction, matching, and geometric definitions

We use frozen Qwen2.5-VL-7B-Instruct, with the observed graph, local map, frontier position, and path distance. The prompt is

Predict regions, objects, and relations potentially observable beyond this frontier from the observed graph and local map. Return only JSON with regions(id,type), objects(id,category,region_id), and relations(type,args). Keep existing IDs; give new entities unique IDs. Include all referenced nodes. For between, list the target ID and two reference IDs. Predictions remain temporary and must not enter the observed graph.

For each category-compatible instance mapping ϕ\phi, relation types, directions, and endpoints are checked under the same mapping. With nn goal nodes and mm constraints, nϕ,mϕn_\phi,m_\phi count matched nodes and satisfied constraints.

Srel(f)=max⁡ϕ(0.5nϕn+0.5mϕm).S_{\mathrm{rel}}(f)=\max_\phi\left(0.5\frac{n_\phi}{n}+0.5\frac{m_\phi}{m}\right).

S(f)=0.6Srel(f)+0.3Uˉ(f)−0.1Dˉ(f).S(f)=0.6S_{\mathrm{rel}}(f)+0.3\bar U(f)-0.1\bar D(f).

Missing nodes and associated constraints score zero. Between jointly checks the target and two distinct references. Partial matches rank frontiers; final verification requires every constraint to hold under one consistent mapping on the observed graph.

U(f)U(f) is unknown area within 3 m of the frontier, unoccluded by known obstacles, estimated by grid ray casting. D(f)D(f) is traversable path length to the frontier on the current map. Each is normalized by its maximum over current frontiers; a zero maximum gives zero.

HM3D/MP3D instructions use fixed templates grounded in annotated 3D geometry and retain only uniquely identified targets. Near/beside requires the same annotated region ID and minimum horizontal box-footprint distance below 1 m. Inside requires the target center inside the reference region and at least 80% of the target bounding-box volume within it. Between requires distinct reference centers, a target-center projection strictly inside their connecting segment (0<γ<10<\gamma<1), and perpendicular distance below 1 m.

Directional relations use a fixed frame defined by the agent's initial position and heading. Left/right correspond to negative/positive target-reference displacement along the initial rightward axis; front/behind correspond to positive/negative displacement along the initial forward axis. No extra threshold is used; zero displacement satisfies neither relation on that axis. Episode generation and online verification use the same frame, so agent rotations do not change the relational goal.

Reviewer_ekhJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

RWMNav addresses training-free navigation to object instances specified by relational instructions. It predicts relation graphs for candidate frontiers using a frozen VLM and matches them with a goal graph to guide exploration. Experiments on relational HM3D and MP3D episodes report improved success and path efficiency.

Critical strengths
  1. Well-motivated problem. Using relational constraints to identify a specific target addresses a meaningful limitation of category-level object navigation, particularly in environments containing multiple similar objects.

  2. The proposed design is reasonable. Separating observed relations from predictions limits unsupported hypotheses in persistent memory.

  3. Promising empirical results. Results improve across all three evaluation settings, with component ablations providing supporting evidence.

Critical weaknesses
  1. Novelty relative to close prior work is unclear. The contribution relative to predictive scene graphs in SGImagineNav [1] and relational navigation in HRON [2] requires clearer positioning and comparison, and UniGoal [3] already constructs goal and scene graphs and uses their matching states to guide training-free exploration and target verification.

  2. Core operations are insufficiently specified. Section 4.4 and Appendix A.3 do not define how entity correspondence, relation consistency, or multiple constraints are computed. Prediction prompts, scoring weights, and the target-verification and stopping procedures also require clarification.

[1] Hu, Yue, et al. "Imaginative world modeling with scene graphs for embodied agent navigation."

[2] Lingelbach, Michael et al. “Task-Driven Graph Attention for Hierarchical Relational Object Navigation.”

[3] Yin, Hang et al. “UniGoal: Towards Universal Zero-shot Goal-oriented Navigation.”

Questions for authors
  1. How are graph correspondences established, and how does matching ensure that all constraints hold for the same target instance?

  2. Does predicting explicit relations outperform simpler frontier scoring or scene-completion alternatives under comparable inference budgets?

3 comments

W1: Novelty beyond SGImagineNav, HRON, and UniGoal

Thank you for this comment. Prior work uses relational descriptions, scene prediction, and graph matching. We ask what to predict to guide exploration before the target and its reference objects are sufficiently observed.

Compared with hierarchical scene completion or matching an online scene graph, RWMNav predicts future entities and relations for each candidate frontier and matches them to the relational goal graph. This compares directions against the full relational goal before entering unknown regions. For “the cabinet beside the bed,” predicted relations distinguish directions that may contain cabinets. This combination guides exploration without additional navigation training. We will clarify this contribution in the revision.

W2: Insufficient implementation details

We use frozen Qwen2.5-VL-7B-Instruct once per frontier, using the observed graph, local map, frontier position, and path distance.

Given the observed graph, local map, and frontier, predict regions, objects, and relations potentially observable beyond it. Return only JSON containing regions(id, type), objects(id, category, region_id), and relations(type, args). Preserve existing IDs and assign unique IDs to new entities. Include all nodes referenced by relations. For between, list the target ID followed by the two reference IDs. Predictions are temporary hypotheses and must not be added to the observed graph.

Candidate mappings ϕ\phi use canonical category labels. Relation types, directions, and connected instances are checked under the same mapping. With nn goal nodes and mm constraints, nϕn_\phi and mϕm_\phi count matched nodes and satisfied constraints.

0.5\frac{n_\phi}{n}+0.5\frac{m_\phi}{m} \right).$$ **Missing nodes and their associated constraints score zero.** The frontier score is $$S(f)=0.6S_{\mathrm{rel}}(f)+0.3\bar U(f)-0.1\bar D(f).$$ Grid ray casting estimates $U(f)$, the unknown area within 3 m of the frontier not occluded by known obstacles. $D(f)$ is the traversable path length to the frontier on the current map. Each is divided by its maximum across current frontiers, or set to zero if the maximum is zero. Prediction scores **only rank exploration directions**. Before STOP, **all constraints must pass verification using actual observations**, and the minimum Euclidean distance from the agent’s horizontal position to the horizontal footprint of the target’s bounding box must be **at most 0.2 m**. With insufficient evidence, exploration continues; reaching the step limit without success counts as failure. **Q1: Graph correspondence and instance consistency** Each entity has a **distinct instance ID**. Category matches define candidate correspondences without merging objects. Under each mapping $\phi$, **the target and repeated reference nodes remain bound to the same instances across all constraints**. Two chairs separately satisfying “left of the table” and “near the sofa” cannot form one valid target. `between` jointly checks the target and two distinct reference instances. **Weighted prediction scoring allows partial matches** for exploration ranking, without ensuring all constraints hold. Final verification finds a new mapping on the observed graph, requiring **all constraints to hold simultaneously for the same target under one consistent mapping**. Missing or unknown relations fail verification, and predicted hypotheses cannot directly justify STOP. **Q2: Relation prediction vs. simpler baselines under comparable budgets** All settings share **the same VLM, full relational instructions, goal parsing, observations, perception, final verification, and planning**. Direct scoring replaces future graph matching with VLM frontier compatibility scores. Region/entity completion predicts regions and objects **without future relation edges**. Compatibility scores are normalized to $[0,1]$ and used in the same frontier scoring formula. Each round evaluates **up to four frontiers**, with **one model call each** and **at most 1,024 generated tokens per call**. Candidate filtering and replanning are identical. | Method | RSR ↑ | RSPL ↑ | | --- | ---: | ---: | | Direct frontier scoring | 50.1 | 23.7 | | Region/entity scene completion | 54.2 | 24.6 | | **Full RWMNav** | **61.0** | **30.3** | RWMNav improves RSR and RSPL over direct scoring by **10.9 and 6.6 percentage points**, respectively. Both comparisons show that **future relation prediction and goal matching** improve success and path efficiency **under the same budget constraints**, beyond region/entity completion alone. If our response addresses your concerns, we would be grateful if you could consider revising your score upward. Thank you again for your time and consideration.

Thank you for your clarifications, which have addressed most of my concerns. If possible, could you provide a revised version of your paper with futher clarification on your contributions? I will raise my score. As a minor presentation suggestion, some text labels in the pipeline figure have shadows while others do not; it would be nice to make this styling consistent.

Thank you for your positive feedback and for reconsidering your score. We have uploaded a revised manuscript that incorporates the clarifications discussed in our rebuttal.

In particular, we have clarified our contributions relative to SGImagineNav, HRON, and UniGoal in the Introduction and Related Work. We have also expanded the descriptions of graph correspondence, consistent instance matching, frontier evaluation, VLM inference, and target verification, and included additional controlled comparisons with simpler frontier scoring and scene completion methods.

We also appreciate your suggestion regarding the pipeline figure and have made the text styling consistent throughout the figure.

Thank you again for your constructive comments, which have helped us improve the clarity and presentation of our work.

Reviewer_tTaCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors addresses compositional zero-shot object navigation (ObjectNav) problem --- where a natural language instruction specifies a target object instance through its spatial or semantic relations with surrounding reference entities (for example, "find the sofa next to the chair" or "find the cabinet beside the bed"). Unlike standard category-level ObjectNav—where reaching any instance of the target category suffices—relational ObjectNav requires the agent to disambiguate and reach the specific target instance that satisfies all relational constraints under partial observability. The authors propose RWMNav (Relational World Model for Navigation), a training-free, closed-loop framework built on frozen foundation models: (a) Relational Goal Parsing and Spatial-Relational State Construction (b) Frontier-Conditioned Future Relational Rollout (c) Relational Goal Matching and Frontier Selection.

Critical strengths

Transitioning zero-shot ObjectNav from category-level stopping ("find any chair") to relation-disambiguated instance search ("find the chair in front of the window") addresses a critical gap in real-world instruction following, particularly in environments cluttered with similar objects.

Isolating the persistent environmental relation graph from the VLM's frontier-conditioned hypotheses prevents speculative hallucinations from corrupting the agent's long-term memory.

RWMNav shows consistent RSR and RSPL improvements across three distinct benchmark datasets, backed by component ablations.

Critical weaknesses

How does the paper fair wrt HRON (Lingelbach et al., ICRA 2023) and the SOON benchmark is unclear.

The paper does not show how graph matching (S_ent and S_relation) is actually computed, how partial matches are scored, or how online 3D spatial relations are extracted from noisy RGB-D data.

How benchmarks were adapted for this task is not clear. Lack of a direct VLM-scoring baseline make it difficult to assess the true methodological contribution.

The paper writing needs to be consistent wrt flow from a section to another. This can be done wrt a storyline stitching perhaps.

Questions for authors

How were category-level baselines (ESC, VoroNav, VLFM, SG-Nav) adapted to process relational instructions and decide when to trigger the STOP action?

Please explicitly clarify the differences in task formulation and methodology between RWMNav and existing relational/imaginative navigation frameworks.

What is the RSR if the VLM only predicts future regions and entities, but goal matching relies solely on entity presence?

Can some example prompt be given wrt fps level of image sent with prompt and the text template - how much performance is dependent on the good prompt selection for VLM?

How does the system penalize predicted graphs that contain the target but lack the correct reference entity?

2 comments

W1 & Q2: Relation to existing work 

Thank you for this comment. HRON and SOON have studied navigation to relationally specified object instances, while SGImagineNav uses scene prediction to guide exploration. Building on these studies, RWMNav focuses on predicting future entity relations for each candidate frontier and matching them against the complete relational goal graph. Unlike learned navigation policies, RWMNav uses a frozen VLM without additional navigation training. This enables the agent to distinguish exploration directions based not only on potential target categories but also on the predicted relations between targets and their reference objects.

W2 & Q5: Graph matching and online relation extraction

We establish candidate entity mappings based on canonical categories and evaluate all constraints under one consistent instance assignment. Given a goal graph with (N) nodes and (M) relations, let (n_\phi) and (m_\phi) denote the matched nodes and satisfied relations under mapping (\phi).

S_{\mathrm{ent}}(\phi)=\frac{n_\phi}{N}, \qquad S_{\mathrm{relation}}(\phi)=\frac{m_\phi}{M}. ]$$ $$[ S_{\mathrm{rel}}(f)= \max_\phi\left[ 0.5S_{\mathrm{ent}}(\phi)+ 0.5S_{\mathrm{relation}}(\phi) \right]. ]$$ Missing entities and their associated relations contribute zero. For example, if a goal contains a cabinet, a bed, and a beside relation, predicting only the cabinet gives a score of 0.25, predicting both entities without the relation gives 0.5, and a complete match gives 1. Online relations are extracted from RGB D observations. GroundingDINO and Mobile SAM provide entity detections and masks, while depth information is used to construct 3D point clouds. Multi view observations are projected into a unified world coordinate system using agent poses. Geometric predicates are then applied to the resulting entity locations and bounding boxes. Only relations supported by actual observations are stored in persistent memory. Predicted relations remain temporary hypotheses and are verified using subsequent observations. **W3: Benchmark adaptation and direct VLM comparison** Relational episodes are generated from HM3D/MP3D 3D annotations using geometric predicates and fixed language templates. We retain only instructions that uniquely identify one target instance and evaluate all methods on identical episode splits. We further compare direct VLM frontier scoring and region/entity-only prediction against RWMNav. All variants use the same VLM, instructions, perception, final verification, and planning pipeline. | Method | RSR ↑ | RSPL ↑ | | --- | ---: | ---: | | Direct VLM scoring | 50.1 | 23.7 | | Region/entity prediction | 54.2 | 24.6 | | **RWMNav** | **61.0** | **30.3** | RWMNav improves RSR/RSPL by **10.9/6.6 percentage points** over direct scoring and **6.8/5.7 points** over region/entity prediction, demonstrating the additional benefit of explicit relational prediction and matching. **W4: Manuscript organization** We will improve the flow of the manuscript by following a consistent pipeline from relational goal construction to observed scene representation, future relation prediction, goal matching, and online verification. We will also clarify the connections between these stages and maintain consistent notation throughout the paper.

Q1: Adaptation of category-level baselines and STOP

All baselines retain their original exploration strategies. Category-only interfaces use the target category, while language-capable interfaces receive the full relational instruction. A shared verifier checks all constraints using actual observations and requires the same target instance and consistent reference assignments. Insufficient evidence leads to continued exploration.

STOP requires successful relational verification and a minimum Euclidean distance of at most 0.2 m from the agent's horizontal position to the target bounding-box footprint. Episodes reaching 500 steps without success are counted as failures.

Q2: Task formulation and methodological differences

HRON and SOON study relationally described target navigation, while SGImagineNav predicts hierarchical scene structures to guide exploration. RWMNav addresses a more specific challenge by predicting future relational configurations for individual frontiers and matching them against the complete relational goal. This allows the agent to distinguish exploration directions containing similar object categories but different target-reference configurations, without additional navigation training.

Q3: RSR with entity-only prediction

When the VLM predicts only regions and entities, and matching relies solely on entity presence, the HM3Dv2 RSR is 54.2%, with an RSPL of 24.6%. Full RWMNav achieves 61.0% RSR and 30.3% RSPL. Thus, explicitly predicting and matching relational edges provides additional navigation benefits beyond entity presence alone.

Q4: Image inputs and prompt sensitivity

We use frozen Qwen2.5-VL-7B-Instruct. Visual inputs consist of 512 × 512 cropped and resized local RGB images, including both current and historical observations. Each candidate frontier receives one VLM call, with a maximum of 1,024 generated tokens. Images are supplied as discrete observations:

Our fixed prompt is:

Given the current and historical RGB observations, observed relation graph, local map, and candidate frontier, predict regions, objects, and relations potentially observable beyond this frontier. Return only JSON containing regions, objects, and relations. Preserve existing entity IDs and assign unique IDs to new entities. Include all referenced nodes. Predictions are temporary hypotheses and must not be added to the observed graph.

To further examine the sensitivity of RWMNav to prompt wording, we evaluate three semantically equivalent formulations of the relational rollout prompt: the original prompt, a concise reformulation, and an instruction-oriented reformulation. Throughout this experiment, we keep the frozen Qwen2.5-VL-7B-Instruct model, visual observations, observation history, candidate frontiers, structured JSON output schema, and inference settings unchanged, varying only the natural-language formulation of the prompt. Each candidate frontier is evaluated independently using one VLM call, with the maximum output length fixed at 1,024 tokens:

PromptRSRRSPL
Original61.030.3
Concise reformulation59.322.9
Instruction-oriented reformulation60.527.8

These results suggest that RWMNav does not rely critically on a single carefully worded prompt.

Q5: Missing reference entities

Missing reference entities reduce both entity coverage and relation satisfaction. For a goal containing a cabinet, a bed, and a beside relation, predicting only the cabinet gives an entity score of 0.5 and a relation score of 0. The combined matching score is therefore 0.25, compared with 0.5 when both entities exist but their relation is incorrect, and 1 for a complete match. No additional penalty is needed because missing nodes and their associated constraints already contribute zero.

Partial matches only guide frontier ranking. Final verification requires all relations to hold for the same target instance under actual observations. Predicted hypotheses alone cannot trigger STOP.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.