- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MGNav, which jointly generates future trajectories and along-path semantic layouts using two diffusion branches, leveraging a gated bidirectional cross-attention mechanism to enable mutual guidance between the two during the denoising process. The model is trained with supervision using expert trajectories and ground-truth semantic layouts, accomplishing ObjectNav via a local planner and continuous replanning. The authors evaluate navigation performance on Gibson and MP3D, and analyze the contributions of individual modules through ablation studies.
Critical strengths
1.The motivation for simultaneously predicting layout and trajectory is well-founded. During navigation, deciding where to move requires considering what might lie ahead, while choosing a specific path determines which areas require attention. By generating trajectory and semantic layouts synchronously to guide each other, this paper enables path planning to leverage predictions of the surrounding environment along the way. 2.The experimental evaluation is relatively systematic, providing support for the overall effectiveness of the method.
Critical weaknesses
1.The current ablation demonstrates the benefits of the layout branch and information interaction, but fails to clarify how the predicted layout in "Semantic Layout Branch only" aids trajectory generation. Consequently, it remains difficult to discern whether the performance gains stem primarily from "allowing the trajectory to reference the predicted layout" or from the emphasized "iterative mutual adjustment between layout and trajectory during denoising." 2.Consistency between predicted layouts and trajectories does not mean that they match the real environment. Incorrect layout predictions may misguide trajectory generation. However, the paper does not directly evaluate layout prediction accuracy in unexplored regions. It also does not analyze whether gating can reduce the impact of incorrect layouts. The reliability of this mutual guidance therefore needs further validation. 3.SGM already uses predicted semantic layouts to support navigation, while UWM jointly models future observations and actions. This paper introduces waypoint-aligned semantic layouts and gated interaction to jointly denoise layouts and trajectories. This design has practical value, but its additional contribution mainly lies in layout representation and information exchange for ObjectNav. The advance over existing joint generation approaches is therefore relatively limited.
Questions for authors
1.During training, each semantic layout is centered on a ground-truth waypoint. During inference, waypoint positions change during denoising. How does the model keep each layout aligned with its waypoint? Does it explicitly adjust the layout position, or learn this alignment through cross-attention? 2.What are the inputs to the gates? Do they mainly depend on the denoising step, or also on the current layout and trajectory predictions? How does the model decide whether to trust information from the other branch?
3 comments
Response to Reviewer b5Da (Part 2/2)
Weakness 2: "Consistency between predicted layouts and trajectories does not mean that they match the real environment ..."
Thank you for the valuable comment. We fully agree that consistency is not correctness; our paper states the risk that a confident but incorrect layout may reinforce an infeasible route (L472–474) and analyzes it as a failure mode (App. F, L912–917). MGNav bounds this risk in three ways:
① Imagined layouts never enter the map. They only shape the route, and the map is updated exclusively from real observations (L362–363), so an incorrect layout cannot propagate into later planning.
② Only a near-term subgoal is executed. The agent follows ŵ_{k_g} for five environment steps and then replans from the updated map (L358–361, L386–387), so an incorrect layout is corrected as soon as its region is observed.
③ The gate can down-weight unhelpful evidence. It is computed from the current states of both branches, and a closed gate reduces each branch to an ordinary single-variable diffusion (L250–252). With the same operator, the gated exchange outperforms the ungated one by 1.1 SR and 1.2 SPL (Table 3).
Q1. Alignment between layouts and waypoints. It is learned, with no explicit re-positioning. Each layout is defined in the frame centered at its own waypoint (L189–192), so as ŵ_k moves during denoising, the k-th patch moves with it by construction; its content is updated through the trajectory-to-layout attention, which gives each layout token the current waypoint hypotheses at every step (Eq. 4).
Q2. Inputs of the gate. The gate is , computed for each token and channel, and is defined symmetrically. It therefore depends on both the denoising step and the current trajectory and layout states. Trust is learned end to end from the denoising losses: cross-branch features that do not reduce the noise-prediction error are down-weighted. The full specification is added to Sec. 3.3 and App. C.4.
Thank you again for the insightful comments. If there are any remaining questions, we are more than happy to address them.
Mutual Guidance Diffusion for Object Navigation
Thank you for the detailed clarifications. The response resolves my questions about some implementation details. However, the experimental support for the central contribution remains incomplete. The current ablations support the benefit of information exchange between the two branches. They do not isolate the additional benefit of bidirectional, iterative guidance over using predicted layouts to guide trajectory generation. The described measures may help reduce the impact of incorrect layouts. However, direct evidence of layout prediction accuracy and whether gating mitigates errors from incorrect layouts is still missing. I therefore maintain my original rating.
Response to Reviewer b5Da (Part 1/2)
We sincerely thank Reviewer b5Da for the valuable comments. We are glad that the reviewer finds our motivation for simultaneously predicting layout and trajectory well-founded and our experimental evaluation systematic. We address each concern below, and all revisions are marked in red in the updated PDF.
Thank you for highlighting these relevant works. We acknowledge the contributions of SGM, which is cited and compared in our paper (L141, Table 1), and of UWM [1], which we add to the related work of the revised PDF. SGM imagines the unobserved map with a self-supervised generative model and then predicts a goal on it, so the imagined layout is fixed before planning. UWM couples robot actions with future camera frames in one diffusion model for manipulation, with the frames indexed by time. However, neither lets the route decide where the scene must be imagined while the imagined scene decides which routes are executable. In contrast, we take the study of imagination for navigation one step further and resolve exactly this two-way dependence. The above works form the basis of our research, whereas ours is not their combination.
As the reviewer notes, our motivation is to let planning and imagination guide each other, and this coupling, rather than the layout representation or the exchange operator, is our main contribution. To our knowledge, this study takes the first step towards joint future imagination for modular ObjectNav. Through an in-depth analysis of modular ObjectNav methods,
① we reveal a circular dependence that the perceive-then-plan pipeline cannot represent: choosing a route requires the layout along it, while knowing which unobserved layout matters requires the route (L157–161). Existing methods, SGM included, break the circle by fixing the layout before planning (L121–123).
② We support this finding with a precise formulation: the route and its layouts should be sampled together from p(ξ, L | C) (Eq. 1), whereas imagining first keeps only p(L | C) p(ξ | L, C), and predicting both without exchange keeps p(ξ | C) p(L | C), the assumption that breaks under partial observability (L196–200).
③ We locate the gain: changing only how the two branches exchange information raises SR from 77.1 to 80.3 and SPL from 42.6 to 45.5 (Table 3), as detailed in our response to Weakness 1 below.
④ We design MGNav accordingly: the waypoint-aligned layouts let the route decide where to imagine, and the gated exchange lets the imagination decide which routes survive. Both are the means of this coupling, not its substance.
Treating where to go and what lies there as one problem, rather than two, is the core spirit of our work.
Thank you for this precise question, which follows the logic of our diagnostic analysis.
① In "+ Semantic Layout Branch only", the route never reads the imagined layout at inference. The two branches exchange no states during denoising; the layout affects the route only during training, through the shared context encoder (Eq. 2) and the geometric term, which pushes predicted waypoints out of imagined obstacles (Eq. 8). The revised caption of Table 2 states this configuration.
② Exchanging states during denoising is what turns the imagined layout into guidance. Table 3 keeps both branches and changes only how they exchange information: SR rises from 77.1 without exchange to 78.0 with concatenation and 79.2 with cross-attention, and the gate, which decides at each denoising step how much each branch trusts the other, raises it further to 80.3. Since every exchange variant in Table 3 is bidirectional and repeated at every step, these rows measure iterative exchange as a whole; in Eq. 1, referencing a finished layout corresponds to the one-way factorization p(L | C) p(ξ | L, C), which keeps only one direction of the dependence that mutual refinement keeps in full.
[1] Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. In RSS, 2025.