acceptodds
Under review as a conference paper at ICLR 2027

Mutual Guidance Diffusion for Object Navigation

Abstract

Modular object-goal navigation (ObjectNav) typically constructs a semantic map and then plans from that map as a fixed condition. Under partial observability, however, route selection and beliefs about unobserved space are coupled: a route determines which future regions matter, while plausible future layouts constrain which routes are executable. We introduce MGNav, a dual-branch diffusion framework that jointly generates future waypoints and waypoint-aligned semantic layouts. At every denoising step, gated bidirectional cross-attention lets evolving route hypotheses constrain layout imagination and lets imagined free space, obstacles, and semantic context refine the route. This interaction targets a mismatch between separately optimized perception and planning modules rather than merely adding an auxiliary prediction task. Experiments on Gibson and Matterport3D show consistent gains over a trajectory-diffusion baseline and over controlled concatenation and ungated cross-attention alternatives. We also state the additional trajectory and layout supervision required by our method and analyze the assumptions and failure modes of coupled generation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Navigation, Diffusion Models, World Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_b5DaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MGNav, which jointly generates future trajectories and along-path semantic layouts using two diffusion branches, leveraging a gated bidirectional cross-attention mechanism to enable mutual guidance between the two during the denoising process. The model is trained with supervision using expert trajectories and ground-truth semantic layouts, accomplishing ObjectNav via a local planner and continuous replanning. The authors evaluate navigation performance on Gibson and MP3D, and analyze the contributions of individual modules through ablation studies.

Critical strengths

1.The motivation for simultaneously predicting layout and trajectory is well-founded. During navigation, deciding where to move requires considering what might lie ahead, while choosing a specific path determines which areas require attention. By generating trajectory and semantic layouts synchronously to guide each other, this paper enables path planning to leverage predictions of the surrounding environment along the way. 2.The experimental evaluation is relatively systematic, providing support for the overall effectiveness of the method.

Critical weaknesses

1.The current ablation demonstrates the benefits of the layout branch and information interaction, but fails to clarify how the predicted layout in "Semantic Layout Branch only" aids trajectory generation. Consequently, it remains difficult to discern whether the performance gains stem primarily from "allowing the trajectory to reference the predicted layout" or from the emphasized "iterative mutual adjustment between layout and trajectory during denoising." 2.Consistency between predicted layouts and trajectories does not mean that they match the real environment. Incorrect layout predictions may misguide trajectory generation. However, the paper does not directly evaluate layout prediction accuracy in unexplored regions. It also does not analyze whether gating can reduce the impact of incorrect layouts. The reliability of this mutual guidance therefore needs further validation. 3.SGM already uses predicted semantic layouts to support navigation, while UWM jointly models future observations and actions. This paper introduces waypoint-aligned semantic layouts and gated interaction to jointly denoise layouts and trajectories. This design has practical value, but its additional contribution mainly lies in layout representation and information exchange for ObjectNav. The advance over existing joint generation approaches is therefore relatively limited.

Questions for authors

1.During training, each semantic layout is centered on a ground-truth waypoint. During inference, waypoint positions change during denoising. How does the model keep each layout aligned with its waypoint? Does it explicitly adjust the layout position, or learn this alignment through cross-attention? 2.What are the inputs to the gates? Do they mainly depend on the denoising step, or also on the current layout and trajectory predictions? How does the model decide whether to trust information from the other branch?

3 comments
AuthorsOpenReview ↗

Response to Reviewer b5Da (Part 1/2)

We sincerely thank Reviewer b5Da for the valuable comments. We are glad that the reviewer finds our motivation for simultaneously predicting layout and trajectory well-founded and our experimental evaluation systematic. We address each concern below, and all revisions are marked in red in the updated PDF.

Weakness 3: "SGM already uses predicted semantic layouts ... UWM jointly models future observations and actions ... The advance over existing joint generation approaches is therefore relatively limited"

Thank you for highlighting these relevant works. We acknowledge the contributions of SGM, which is cited and compared in our paper (L141, Table 1), and of UWM [1], which we add to the related work of the revised PDF. SGM imagines the unobserved map with a self-supervised generative model and then predicts a goal on it, so the imagined layout is fixed before planning. UWM couples robot actions with future camera frames in one diffusion model for manipulation, with the frames indexed by time. However, neither lets the route decide where the scene must be imagined while the imagined scene decides which routes are executable. In contrast, we take the study of imagination for navigation one step further and resolve exactly this two-way dependence. The above works form the basis of our research, whereas ours is not their combination.

As the reviewer notes, our motivation is to let planning and imagination guide each other, and this coupling, rather than the layout representation or the exchange operator, is our main contribution. To our knowledge, this study takes the first step towards joint future imagination for modular ObjectNav. Through an in-depth analysis of modular ObjectNav methods,

① we reveal a circular dependence that the perceive-then-plan pipeline cannot represent: choosing a route requires the layout along it, while knowing which unobserved layout matters requires the route (L157–161). Existing methods, SGM included, break the circle by fixing the layout before planning (L121–123).

② We support this finding with a precise formulation: the route and its layouts should be sampled together from p(ξ, L | C) (Eq. 1), whereas imagining first keeps only p(L | C) p(ξ | L, C), and predicting both without exchange keeps p(ξ | C) p(L | C), the assumption that breaks under partial observability (L196–200).

③ We locate the gain: changing only how the two branches exchange information raises SR from 77.1 to 80.3 and SPL from 42.6 to 45.5 (Table 3), as detailed in our response to Weakness 1 below.

④ We design MGNav accordingly: the waypoint-aligned layouts let the route decide where to imagine, and the gated exchange lets the imagination decide which routes survive. Both are the means of this coupling, not its substance.

Treating where to go and what lies there as one problem, rather than two, is the core spirit of our work.

Weakness 1: "... whether the performance gains stem primarily from 'allowing the trajectory to reference the predicted layout' or from the emphasized 'iterative mutual adjustment' ..."

Thank you for this precise question, which follows the logic of our diagnostic analysis.

① In "+ Semantic Layout Branch only", the route never reads the imagined layout at inference. The two branches exchange no states during denoising; the layout affects the route only during training, through the shared context encoder (Eq. 2) and the geometric term, which pushes predicted waypoints out of imagined obstacles (Eq. 8). The revised caption of Table 2 states this configuration.

② Exchanging states during denoising is what turns the imagined layout into guidance. Table 3 keeps both branches and changes only how they exchange information: SR rises from 77.1 without exchange to 78.0 with concatenation and 79.2 with cross-attention, and the gate, which decides at each denoising step how much each branch trusts the other, raises it further to 80.3. Since every exchange variant in Table 3 is bidirectional and repeated at every step, these rows measure iterative exchange as a whole; in Eq. 1, referencing a finished layout corresponds to the one-way factorization p(L | C) p(ξ | L, C), which keeps only one direction of the dependence that mutual refinement keeps in full.

[1] Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. In RSS, 2025.

AuthorsOpenReview ↗

Response to Reviewer b5Da (Part 2/2)

Weakness 2: "Consistency between predicted layouts and trajectories does not mean that they match the real environment ..."

Thank you for the valuable comment. We fully agree that consistency is not correctness; our paper states the risk that a confident but incorrect layout may reinforce an infeasible route (L472–474) and analyzes it as a failure mode (App. F, L912–917). MGNav bounds this risk in three ways:

① Imagined layouts never enter the map. They only shape the route, and the map is updated exclusively from real observations (L362–363), so an incorrect layout cannot propagate into later planning.

② Only a near-term subgoal is executed. The agent follows ŵ_{k_g} for five environment steps and then replans from the updated map (L358–361, L386–387), so an incorrect layout is corrected as soon as its region is observed.

③ The gate can down-weight unhelpful evidence. It is computed from the current states of both branches, and a closed gate reduces each branch to an ordinary single-variable diffusion (L250–252). With the same operator, the gated exchange outperforms the ungated one by 1.1 SR and 1.2 SPL (Table 3).

Q1. Alignment between layouts and waypoints. It is learned, with no explicit re-positioning. Each layout is defined in the frame centered at its own waypoint (L189–192), so as ŵ_k moves during denoising, the k-th patch moves with it by construction; its content is updated through the trajectory-to-layout attention, which gives each layout token the current waypoint hypotheses at every step (Eq. 4).

Q2. Inputs of the gate. The gate is Gξτ=σ(Wξ[Hξτ;Hˉξτ;π(τ)]+bξ)G_\xi^{\tau}=\sigma(W_\xi[H_\xi^{\tau};\bar H_\xi^{\tau};\pi(\tau)]+b_\xi), computed for each token and channel, and GLτG_L^{\tau} is defined symmetrically. It therefore depends on both the denoising step and the current trajectory and layout states. Trust is learned end to end from the denoising losses: cross-branch features that do not reduce the noise-prediction error are down-weighted. The full specification is added to Sec. 3.3 and App. C.4.

Thank you again for the insightful comments. If there are any remaining questions, we are more than happy to address them.

Reviewer_b5DaOpenReview ↗

Mutual Guidance Diffusion for Object Navigation

Thank you for the detailed clarifications. The response resolves my questions about some implementation details. However, the experimental support for the central contribution remains incomplete. The current ablations support the benefit of information exchange between the two branches. They do not isolate the additional benefit of bidirectional, iterative guidance over using predicted layouts to guide trajectory generation. The described measures may help reduce the impact of incorrect layouts. However, direct evidence of layout prediction accuracy and whether gating mitigates errors from incorrect layouts is still missing. I therefore maintain my original rating.

Reviewer_DuiWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MGNav, an object-goal navigation framework that jointly predicts future waypoints and semantic scene layouts. The framework introduces a mutual guidance mechanism to facilitate interaction between the two prediction tasks, aiming to improve the consistency between trajectory planning and scene understanding. Multiple training objectives supervise the predictions and regularize their relationship.

Critical strengths

The paper provides relatively comprehensive ablation studies that systematically evaluate the contributions of its key components, including the semantic layout branch, mutual guidance mechanism, and training objectives. The results consistently demonstrate improvements across SR, SPL, and DTS.

Critical weaknesses

(1)Insufficient comparison with recent state-of-the-art methods. The experimental evaluation lacks comparisons with several recent competitive ObjectNav methods on both Gibson and MP3D. On Gibson, the unified framework proposed by Wang et al. [R1] reports higher SR (84.4% vs. 80.3%) and SPL (48.2% vs. 45.5%) than MGNav. On MP3D, WMNav [R2] reports an SR of 45.4% and an SPL of 17.2%, while CogNav [R3] reports an SR of 46.6% and an SPL of 16.1%. Ref: [R1] Hongze Wang, Boyang Sun, Jiaxu Xing, Fan Yang, Marco Hutter, Dhruv Shah, Davide Scaramuzza, and Marc Pollefeys. What Matters in RL-Based Methods for Object-Goal Navigation? An Empirical Study and A Unified Framework. In European Conference on Computer Vision (ECCV), 2026. [R2] Dujun Nie, Xianda Guo, Yiqun Duan, Ruijun Zhang, and Long Chen. WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025. [R3] Yihan Cao, Jiazhao Zhang, Zhinan Yu, Shuzhen Liu, Zheng Qin, Qin Zou, Bo Du, and Kai Xu. CogNav: Cognitive Process Modeling for Object Goal Navigation with LLMs. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025. (2)Incomplete ablation study of the training objectives. The proposed framework employs five loss terms, but Table 5 only evaluates the trajectory loss, the addition of the layout loss, and the full objective. The individual contributions of the reconstruction, geometric, and smoothness losses are not investigated. In particular, the geometric loss is intended to couple trajectory planning with imagined scene layouts, making its individual contribution especially important for validating the proposed joint reasoning mechanism.

Questions for authors

Q1. The selection criteria for the baseline methods are not clear. Given the recent progress in ObjectNav, it would be helpful to compare MGNav with more recent state-of-the-art methods under consistent evaluation settings. Q2. The proposed framework contains several interacting components, while the current ablation studies only examine a subset of them. The contributions of the remaining components are therefore unclear. In particular, it would be useful to investigate whether joint prediction offers advantages over independently predicting future waypoints and semantic layouts. Q3. A more detailed analysis of failure cases would help assess the robustness of the proposed method. For example, under what conditions does joint prediction fail, and how do errors in semantic layout prediction influence subsequent navigation decisions?

3 comments
AuthorsOpenReview ↗

Response to Reviewer DuiW (Part 1/2)

We sincerely thank Reviewer DuiW for the valuable comments. We are glad that the reviewer finds our ablation studies comprehensive and recognizes that our results consistently demonstrate improvements across SR, SPL and DTS. We address each concern below, and all revisions are marked in red in the updated PDF.

Weakness 1 & Question 1: "Insufficient comparison with recent state-of-the-art methods ... The selection criteria for the baseline methods are not clear"

Thank you for the suggested methods, which we add to Table 1 of the revised PDF, together with T-Diff and Diffusion-as-Reasoning (DAR), two diffusion-based modular methods discussed in our related work (L137–138).

We would first like to clarify the role of Table 1 in our work. Our main contribution lies in revealing and resolving a circular dependence in modular ObjectNav: choosing a route requires the layout along it, while knowing which unobserved layout matters requires the route (L157–161). Existing methods break this circle by fixing the layout before planning (L121–123); we formalize it as one joint distribution (Eq. 1) and resolve it by refining the route and the layout together during denoising. Changing only how the two are coupled raises SR from 77.1 to 80.3 and SPL from 42.6 to 45.5 (Table 3). The benchmark comparison is therefore a minor contribution, and our paper presents Table 1 as evidence of competitive performance rather than a state-of-the-art claim (L398–399).

Selection criteria. Table 1 follows the supervised modular protocol of PONI and SGM, i.e., Gibson with 1,000 validation episodes and MP3D with 2,195 validation episodes (L370–374), and reports values from the respective papers, with an "Extra sup." column for training signals beyond online interaction (L324–327). We now add the paradigm and the external knowledge of each method:

MethodParadigm / external knowledgeGibson SR / SPL / DTSMP3D SR / SPL / DTS
T-Diff (NeurIPS 2024)Modular, trajectory diffusion79.6 / 44.9 / 1.0039.6 / 15.2 / 5.16
DAR (arXiv 2024)Modular, map diffusion, LLM object-room knowledge81.2 / 45.3 / 1.1241.4 / 15.2 / 4.82
Wang et al. [R1]Modular RL, Gibson-finetuned detector, test-time strategies84.4 / 48.2 / 0.74–
WMNav [R2]Training-free, VLM-based world model–45.4 / 17.2 / –
CogNav [R3]Training-free, LLM/VLM queries–46.6 / 16.1 / –
MGNav (ours)Modular, mutual-guidance diffusion80.3 / 45.5 / 0.9840.2 / 15.5 / 5.05

Within this protocol, MGNav achieves the best SR, SPL and DTS on Gibson among the modular methods that use neither external LLM knowledge nor test-time strategies (vs. T-Diff 79.6 / 44.9 / 1.00), and is on par with PEANUT on MP3D (40.5 / 15.8, Table 1). The higher numbers of the added methods come from components that are orthogonal to the planner. ① [R1] concludes that perception quality and test-time strategies are the decisive drivers, while policy improvements yield marginal gains; its Gibson-finetuned detector raises SR from 77.7 to 83.8, and its untrapping helper raises the continuous-action SR from 81.3 to 84.4. ② WMNav and CogNav are training-free systems that query large vision-language or language models during navigation; CogNav reports about 15 cognitive-state, 23 room-type and 230 scene-graph queries per episode. ③ DAR conditions map diffusion on LLM-based object-room knowledge. MGNav changes only the planning module, so these components are complementary to it rather than competing with it.

Under consistent settings, the fair test is to change only the planner, which is what our controlled study does: all variants in Tables 2–5 share the mapping module, the local planner and the data of the main-result setting (L418–419).

AuthorsOpenReview ↗

Response to Reviewer DuiW (Part 2/2)

Weakness 2: "Incomplete ablation study of the training objectives ... the geometric loss is intended to couple trajectory planning with imagined scene layouts ..."

Thank you for the suggestion. We clarify how each term is evaluated, and how the joint reasoning mechanism is validated:

① The joint reasoning mechanism is validated independently of the geometric loss. Table 3 keeps the training objective of the main-result setting and changes only how the two branches exchange information (L418–419, L430–431): SR rises from 77.1 without exchange to 80.3 with gated cross-attention. The coupling therefore does not rest on L_geo, which complements it during training by tying the route to the decoded obstacles (Eq. 8).

② L_rec is isolated in Table 4. Adding it to latent layout prediction raises SR from 78.7 to 79.5 and SPL from 43.9 to 44.7 (L455–456), by keeping the latent decodable, which L_geo requires (L299–301).

③ L_geo and L_smooth are evaluated together. Table 5 adds L_rec, L_geo and L_smooth to the objective at once (L459–462); with L_rec isolated in Table 4, the remaining pair is not separated in the current experiments, while ① isolates the gain of the coupling itself.

Q2. Joint vs. independent prediction. The "None" row of Table 3 predicts waypoints and layouts independently, with the same branches and setting but no exchange. Joint prediction improves SR from 77.1 to 80.3, SPL from 42.6 to 45.5 and DTS from 1.13 to 0.98.

Q3. Failure analysis. App. F analyzes five failure modes (L859–917): highly cluttered scenes, severe occlusion, out-of-distribution layouts, errors from perception and mapping, and mutual reinforcement of incorrect hypotheses. Joint prediction fails mainly in the last case, when both branches are uncertain and agree on a wrong hypothesis in occluded space (L912–917). Layout errors influence navigation only through the route: imagined layouts are never written into the map (L362–363), only a near-term subgoal is executed before replanning (L358–361), and a closed gate falls back to single-variable diffusion (L250–252). We move this analysis into Sec. 4.3 of the revised PDF.

Thank you again for the insightful comments. If there are any remaining questions, we are more than happy to address them.

Reviewer_DuiWOpenReview ↗

Mutual Guidance Diffusion for Object Navigation

Thank you for the clarifications.

My main concern remains that the central methodological claim is insufficiently supported. The experiments do not adequately isolate the contribution of the proposed joint diffusion formulation beyond coupling existing modules.

Comparisons with recent methods such as PLMD [1] and SSMG-Nav [2] are also needed. The authors should clarify the advantages of their method, particularly where it does not improve upon state-of-the-art performance.

I therefore maintain my original rating.

References [1] Plug-and-Play Label Map Diffusion for Universal Goal-Oriented Navigation. ICML, 2026. [2] SSMG-Nav: Enhancing Lifelong Object Navigation with Semantic Skeleton Memory Graph. ICRA, 2026.

Reviewer_3JCnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MGNav, a Mutual Guidance Diffusion framework for Object Goal Navigation (ObjectNav), aiming to address the limitations of conventional modular navigation approaches that follow a fixed perception-then-planning pipeline.

The key observation is that route planning and reasoning about unobserved scene structures are inherently coupled under partial observability. The selected route determines which future regions are relevant, while the anticipated spatial structures constrain which routes are feasible.

To address this issue, the authors introduce a dual-branch diffusion framework that jointly generates future waypoint trajectories and waypoint-aligned future semantic layouts.

The two branches interact through gated bidirectional cross-attention at every diffusion denoising step. Specifically, the layout-to-trajectory interaction provides spatial constraints, such as traversability and obstacle information, for route generation, while the trajectory-to-layout interaction guides future scene imagination toward regions relevant to the evolving route hypothesis.

Additionally, the authors introduce a geometric consistency loss to penalize trajectories that pass through predicted occupied regions.

Critical strengths
  1. The proposed framework combines trajectory diffusion, latent semantic layout diffusion, bidirectional cross-attention, gated residual fusion, and geometric consistency constraints. These components are generally well organized and have clear functional roles within the overall framework.

  2. The paper explicitly acknowledges its dependence on simulator-provided supervision, including expert trajectories and ground-truth future layouts. It also discusses potential error propagation between the two branches, additional inference costs, and the limited evaluation scope. This makes the limitations of the proposed approach relatively clear.

Critical weaknesses
  1. The methodological novelty is not sufficiently established. MGNav essentially combines several existing techniques, including trajectory diffusion, semantic layout diffusion, bidirectional cross-attention, gated residual fusion, and geometric consistency regularization. The main methodological contribution lies in introducing two diffusion branches that generate future trajectories and corresponding semantic layouts, respectively, and allowing them to exchange intermediate features through bidirectional gated cross-attention throughout the denoising process. Although this design is reasonable, the paper does not sufficiently establish what fundamentally distinguishes it from existing joint generative models or multimodal feature fusion approaches. In particular:
  • The proposed bidirectional guidance mechanism relies on cross-attention between two Transformer branches, which is a relatively common approach for multimodal feature interaction. The paper does not sufficiently explain what fundamentally distinguishes this interaction from conventional cross-attention-based feature fusion.
  • The paper argues that bidirectional interaction is more effective than one-way conditional generation or late fusion. However, it remains unclear why information exchange at every denoising step is necessary, rather than performing fusion at a few selected stages.
  • The distinction from existing diffusion-based navigation methods, such as T-Diff, Diffusion-as-Reasoning, and joint diffusion-based navigation, needs to be further clarified.

The key difference appears to be the joint denoising of future trajectories and waypoint-aligned semantic layouts. However, the paper should provide a stronger methodological justification for why this particular formulation offers advantages beyond combining two generative branches with cross-attention. The authors should provide more rigorous comparisons with one-way conditional diffusion, late fusion, and existing joint generative approaches, and clarify the unique benefits of bidirectional guidance throughout the denoising process.

  1. The technical formulation of the Mutual Guidance mechanism is insufficiently detailed.Equations (4) and (5) describe the core interaction mechanism of MGNav. However, several important implementation details remain unclear.

First, the implementation of the gated fusion mechanism is not sufficiently specified.The authors define learnable sigmoid gates that adaptively control information exchange between the two branches at different denoising stages. Second, the spatial correspondence between waypoint tokens and layout tokens is not sufficiently explained. Third, the probabilistic formulation of joint generation could be more rigorous.

  1. The geometric consistency constraints have several limitations. The intention is to penalize predicted waypoints that fall within occupied regions of the imagined semantic layouts. However, this formulation raises two concerns. First, checking obstacle occupancy only at discrete waypoint locations does not guarantee collision-free trajectories. Second, the definition of the smoothness loss is questionable. Although the authors describe this as a first-order smoothness prior, the loss actually penalizes the distance between consecutive waypoints rather than changes in trajectory direction or curvature. It is therefore closer to a step-length regularization term than a conventional trajectory smoothness constraint.
Questions for authors
  1. What is the fundamental methodological novelty of MGNav beyond combining two diffusion branches with bidirectional gated cross-attention? How does it differ from existing joint diffusion and multimodal conditional generation methods?
  2. Since the future semantic layouts are constructed around ground-truth waypoints during training, how does the layout branch maintain spatial alignment with generated waypoints at inference time, particularly when the predicted trajectory deviates from the expert trajectory?
  3. Why does the geometric consistency loss only evaluate obstacle occupancy at discrete waypoints rather than along trajectory segments? How does the proposed smoothness loss prevent trajectory collapse or overly short waypoint displacements?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3JCn (Part 1/3)

We would like to sincerely thank Reviewer 3JCn for the constructive comments on our work.

We take every comment seriously and provide point-by-point responses below. All revisions are marked in red in the updated PDF. If there are any remaining questions, we are more than happy to address them.

Weakness 1 & Question 1: "The methodological novelty is not sufficiently established ... MGNav essentially combines several existing techniques ..."

Thank you for highlighting these relevant works. We acknowledge the contributions of trajectory diffusion (T-Diff), semantic-map diffusion (Diffusion-as-Reasoning) and joint diffusion-based navigation (Niedoba et al.), all of which are cited in our paper (L137–141). T-Diff diffuses the route on a given semantic map, and Diffusion-as-Reasoning diffuses the unobserved map before a goal is chosen; each treats the other quantity as fixed (L137–139). Niedoba et al. jointly diffuse the trajectories of interacting agents on a given map, without a scene variable. However, they all generate either the route or the scene while holding the other fixed. Cross-attention fusion, likewise, describes how two feature streams are combined, not which variables of the navigation problem depend on each other.

In contrast, we take the study of map-based planning one step further: we study the circular dependence between the route and the unobserved layout along it, which the perceive-then-plan pipeline cuts. The above works form the basis of our research, whereas ours is not their combination.

To clarify, the operators that MGNav builds on, i.e., trajectory diffusion, cross-attention and gated fusion, are not the main focus and contribution of our work, and we did not claim them as our own. Instead, to our knowledge, this study takes the first step towards joint future imagination for modular ObjectNav, in which the route and the layout along it are generated together and refine each other, instead of planning on a map that is fixed once perceived. We are inspired by a common failure mode: since the target usually lies outside the explored area, a route that looks feasible on the current map crosses unobserved space it never considered, and the agent backtracks, oscillates or gets trapped at dead ends once that space is revealed (L33–34, L155–159). Through an in-depth analysis of modular ObjectNav methods,

① we reveal that this failure stems from a circular dependence that the perceive-then-plan pipeline cannot represent: choosing a route requires the layout along it, while knowing which unobserved layout matters requires the route (L157–161). Existing methods break the circle by fixing the layout before planning, as the observed map or an imagined one, so the imagination is blind to the route and the planner cannot question the imagination (L121–123, App. H).

② We support this finding with a precise formulation. Our core insight is that the circle is a joint distribution, not a pipeline: the route and the layouts along it should be sampled together from p(ξ, L | C) (Eq. 1). Existing designs each cut a direction of it, keeping p(ξ | C) (T-Diff), p(L | C) p(ξ | L, C) (SGM, Diffusion-as-Reasoning) or p(ξ | C) p(L | C), the very assumption that breaks under partial observability (L196–200). Iterative denoising resolves the circle instead of cutting it: both hypotheses start coarse and are refined together, each conditioned on the current state of the other at every step while both remain editable (L73–77, L254–258).

③ We locate the gain with a diagnostic experiment: changing only how the branches exchange information, SR rises monotonically as the coupling tightens, from 77.1 without exchange to 78.0 with concatenation, 79.2 with cross-attention and 80.3 with gated cross-attention (Table 3). The gain therefore comes from resolving the circle, not from imagining more. Based on this,

④ we design MGNav carefully, with two choices dictated by this analysis: layouts are cropped in the frames of their waypoints, so the route decides where to imagine (L189–192), and the two branches exchange states through gated cross-attention at every denoising step, so the imagined free space and obstacles decide which routes survive (Eqs. 4–5). The layout representation and the exchange operator are thus the means of this coupling, not its substance.

Given such fresh insights and remarkable merits, MGNav demonstrates sound motivation, a principled formulation and a solid solution, far beyond incremental innovation.

AuthorsOpenReview ↗

Response to Reviewer 3JCn (Part 2/3)

Weakness 1 (continued): "... why information exchange at every denoising step is necessary, rather than performing fusion at a few selected stages ... more rigorous comparisons with one-way conditional diffusion, late fusion, and existing joint generative approaches"

Thank you for this insightful question. Exchanging at every step is the general case, and fusion at a few selected stages is one of its special cases:

① A route can only be revised by evidence that exists while it is being generated. Diffusion refines a coarse hypothesis into a final one (L73–77). If the layout enters only at a few stages, the remaining steps refine the route without it and refine the layout without the route; exchanging at every step lets a route hypothesis be revised many times before it is finalized (L256–258).

② The gate learns which steps to use, instead of fixing them by hand. The exchange enters as a gated residual, and a closed gate reduces that step to an ordinary single-variable update (L250–252). Fusion at a few selected stages is thus one hand-chosen setting of our gates, whereas MGNav learns the setting from data. This control is measured: with the same cross-attention, adding the gate raises SR from 79.2 to 80.3 and SPL from 44.3 to 45.5 (Table 3).

③ Every-step exchange adds no parameters. The same denoiser is applied at every step, so one exchange block serves all of them.

Regarding the comparisons, the designs named by the reviewer correspond to the factorizations of Eq. 1 in our response above: late fusion and independent prediction keep p(ξ | C) p(L | C), and one-way conditional diffusion keeps p(L | C) p(ξ | L, C). Table 3 measures the factorized end of this range ("None", 77.1 SR) against increasingly tight couplings, up to 80.3. Existing joint generative models, such as unified world-action diffusion [1], denoise actions together with future camera frames indexed by time; MGNav instead generates top-down layouts indexed by waypoints and cropped in each waypoint's frame, so the region to be imagined is defined by the route itself (L189–192), and the decoded obstacles enter the geometric term (L299–301).

Weakness 2: "The technical formulation of the Mutual Guidance mechanism is insufficiently detailed ..."

Thank you for the suggestion. We give the full specification below and add it to Sec. 3.3 and App. C.4 of the revised PDF:

① Gated fusion. The gate is Gξτ=σ(Wξ[Hξτ;Hˉξτ;π(τ)]+bξ)G_\xi^{\tau}=\sigma(W_\xi[H_\xi^{\tau};\bar H_\xi^{\tau};\pi(\tau)]+b_\xi), computed for each token and channel, and GLτG_L^{\tau} is defined symmetrically. It thus depends both on the denoising stage, through π(τ), and on the current hypotheses of the two branches.

② Spatial correspondence between waypoint and layout tokens. The k-th layout is a patch cropped in the frame centered at the k-th waypoint (L189–192, L288–291), so the k-th layout token describes the surroundings of the k-th waypoint by construction. Attention between the two sequences is not restricted to the same index, because a waypoint approaching a doorway may need the layout of the next room (L239–242).

③ Probabilistic formulation. Both variables are noised independently at a shared step τ (Eq. 6, L310–311), and each reverse step is a product of two Gaussians whose means read both noisy variables:

pθ(ξτ−1,Zτ−1∣ξτ,Zτ,C)=N(ξτ−1;μθξ(ξτ,Zτ,τ,ht),στ2I)N(Zτ−1;μθL(ξτ,Zτ,τ,ht),στ2I)p_\theta(\xi^{\tau-1},Z^{\tau-1}\mid\xi^{\tau},Z^{\tau},C)=\mathcal{N}\big(\xi^{\tau-1};\mu_\theta^{\xi}(\xi^{\tau},Z^{\tau},\tau,h_t),\sigma_\tau^2I\big)\mathcal{N}\big(Z^{\tau-1};\mu_\theta^{L}(\xi^{\tau},Z^{\tau},\tau,h_t),\sigma_\tau^2I\big)

With a shared τ, the two losses of Eq. 7 form the standard denoising objective of the joint variable (ξ, Z), up to the weights λ1 and λ2, so MGNav is a diffusion model of p(ξ, L | C) in Eq. 1, with L decoded from Z. A closed gate makes each mean read only its own variable and recovers the factorized model p(ξ | C) p(L | C). The revised Sec. 3.3 states this derivation.

[1] Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. In RSS, 2025.

AuthorsOpenReview ↗

Response to Reviewer 3JCn (Part 3/3)

Weakness 3 & Question 3: "The geometric consistency constraints have several limitations ..."

We agree with the reviewer's reading of both terms, and thank you for the suggestion.

① L_geo is a training-time prior, not a collision guarantee. It couples the two branches during training by pushing predicted waypoints out of imagined obstacles (Eq. 8, L320–322). Occupancy is queried at the waypoints because each layout is defined around its own waypoint (L288–291); sampling interpolated points along each segment is a direct extension, which we describe in App. D.2. At execution, only the near-term subgoal ŵ_{k_g} is passed to the local planner, and the route is replanned every five steps from the updated map (L358–361, L386–387). The revised Sec. 3.6 states: "L_geo penalizes imagined obstacles at the predicted waypoints; it does not guarantee a collision-free path."

② L_smooth is a step-length regularizer. We agree that it penalizes consecutive displacements rather than changes in direction, as App. D.3 already notes (L848–851), and the revised PDF renames it a step-length regularizer in Sec. 3.6 and App. D.3. It does not collapse the route: L_traj regresses every waypoint to the expert one, which fixes the scale of the displacements, and zero-length steps appear only where the expert path itself ends, as padding (L266–267).

We hope this response also addresses Question 3, i.e., why occupancy is queried at the waypoints and what keeps the steps from shrinking.

Q2. Layout–waypoint alignment at inference. Alignment is learned rather than enforced by re-cropping. ① The k-th layout is expressed in the frame of the k-th waypoint, so as ŵ_k moves during denoising, the k-th patch moves with it by construction (L189–192). ② At every denoising step, the trajectory-to-layout attention gives each layout token the current waypoint hypotheses, so its content follows the route (Eq. 4). ③ When a generated route departs from the expert one, a misaligned layout can bias the route but cannot corrupt the map, since imagined layouts are never written into it (L362–363), and receding-horizon replanning corrects the route with new observations (L358–361). The revised Limitations state: "Training layouts are cropped at expert waypoints, so routes far from the expert path are supervised only indirectly."

Thank you again for the insightful comments. If there are any remaining questions, we are more than happy to address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their valuable comments and suggestions!

We have updated the manuscript with the following revisions based on the reviewers' suggestions. All revisions in the updated version are highlighted in red:

  1. Add more comparisons:
    • Add T-Diff, Diffusion-as-Reasoning, the unified framework of Wang et al., WMNav and CogNav to Table 1, with the paradigm and external knowledge of each method (DuiW-W1/Q1, 3JCn-W1; Table 1).
  2. Add more discussion and details:
    • Clarify the circular dependence between route planning and layout imagination, and how one-way conditioning, late fusion and independent prediction factorize Eq. 1 (3JCn-W1/Q1, b5Da-W1/W3, DuiW-Q2; Sec. 1, Sec. 3.1).
    • Formalize the joint reverse process, and specify the gate and the correspondence between waypoint and layout tokens (3JCn-W2/Q2, b5Da-Q1/Q2; Sec. 3.3, App. C.4).
    • Clarify the role of the geometric term and rename the smoothness term a step-length regularizer (3JCn-W3/Q3; Sec. 3.6, App. D).
    • State the configuration of each ablation row (b5Da-W1, DuiW-W2/Q2; Tables 2–5).
    • Discuss unified world-action models (b5Da-W3, 3JCn-W1; Sec. 2).
    • Move the failure analysis into the main text and extend the Limitations (b5Da-W2, DuiW-Q3, 3JCn-Q2; Sec. 4.3, Sec. 5).

Thanks again for all the effort and time, and we look forward to further discussions if there are any more questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.