- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MultINav-Bench, a benchmark for interactive "object-goal navigation" that encompasses tasks such as opening doors, manipulating containers, and performing sequential interactions. It also proposes MultINav-Agent, a framework based on a vision-language model (VLM) that utilizes a persistent semantic interaction graph for action selection and state updates. While this approach demonstrates significant improvements over the enhanced baseline models, I have concerns regarding its level of innovation, the design of the benchmark, its reliance on "ground-truth perception," and the lack of sufficient validation in real-world environments.
Critical strengths
This work extends traditional "ObjectGoal navigation" to more complex scenarios—such as those involving doors and containers—by accounting for how interactive behaviors alter spatial connectivity and the visibility of target objects.
MultINav-Bench comprises 3,000 task episodes across 639 home environments, encompassing interactive tasks that involve passageways, containers, and a variety of other objects. Its automated data generation pipeline and annotations regarding "interaction necessity" provide a valuable evaluation resource for research in interactive navigation.
Compared to an adapted MoMa-LLM baseline model, MultINav-Agent improves the overall success rate from 10.53% to 41.63%. Furthermore, the study includes ablation experiments that offer in-depth insights into key components such as interaction memory, action selection, and state verification.
Critical weaknesses
The proposed framework primarily integrates existing techniques, including semantic scene graphs, action selection based on vision-language models (VLMs), and interaction-aware planning. Similar concepts have already been explored in works such as MoMa-LLM, CoINS, and previous graph-based navigation methods. The methodological innovation beyond these existing studies remains unclear.
In real-world scenarios, multiple instances of a target category may exist, some of which could be accessed without interaction. However, the framework mandates interaction for all tasks categorized as "Container" or "Mixed," with only "Channel" tasks allowing for non-interactive access. It is unclear whether the agent can avoid unnecessary interactions when other, directly accessible instances of the target are available.
The primary experiments utilize ground-truth data for object identity, category, and spatial positioning, which significantly simplifies the perception and localization processes. The framework’s performance under conditions involving realistic visual perception errors remains unknown, limiting the practical significance of the findings.
Real-world demonstrations rely on human assistance to open doors and refrigerators, while the simulation environment employs simplified interaction actuators (incorporating kinematic fallback mechanisms). Consequently, the framework’s effectiveness in scenarios requiring fully autonomous physical interaction has not been verified.
In the primary interaction evaluation, the framework is compared only against a modified version of MoMa-LLM. Furthermore, replacing the VLM-based selection strategy with a greedy selection strategy results in a decrease in success rate (SR) of only 1 percentage point; similarly, simplifying the object memory structure to a flat storage format leads to a drop of only 2 percentage points. These results fail to provide a compelling demonstration of the specific contributions of the individual modules.
Questions for authors
see weaknesses
2 comments
The authors’ rebuttal addressed some of my concerns, and after considering the other reviewers’ comments, I maintain my original score.
Thank you for the thorough review. We expanded unnecessary-container controls and clarified the scope of our perception and physical-execution claims. Line numbers below refer to the revised PDF.
Q1: Methodological contribution
Scene graphs, VLM-based selection, and interaction-aware planning are established ideas; we do not claim them as novel in isolation. Our contribution is their integration for ObjectGoal navigation: interaction states condition graph relations and candidates; candidates are constrained by route feasibility and observed evidence; and post-interaction observations plus execution feedback update beliefs before replanning. The benchmark separately labels required and unnecessary interactions across channel, container, and mixed tasks.
Revision: Related Work L134-137 and Sec. 4.2-4.3 L300-330.
Q2: Unnecessary interactions
The benchmark includes 1,000 Unnecessary episodes: 500 channel and 500 container controls. Container cases cover open containers (150), external targets (150), and accessible alternatives beside concealed candidates (200). Ours reaches 68.70% SR with 0.651 mean attempts; MoMa-LLM reaches 54.80% with 0.632.
Across all 4,000 episodes, Ours reaches 48.55% SR vs. 35.38% (+13.18 points). The gap remains on required tasks, including Container (46.50% vs. 28.40%) and Mixed (35.40% vs. 17.40%).
Revision: Appendix B.1 L773-783, Sec. 5.2 L367-394, and Table 3 L378-389.
Q3: Perception scope
Our primary comparison uses restricted-GT observations for currently visible objects. This reduces perception and instance-association confounds; it does not establish robustness to realistic perception errors. Hidden objects, joint metadata, and oracle plans are withheld.
The Go2 demonstration uses YOLO-based detection and segmentation with robot pose inputs. It is qualitative, not a quantitative robustness evaluation. We will evaluate this setting more systematically, including higher-fidelity simulation, in future work.
Revision: Appendix A L738-741 describes the Go2 perception setup; Sec. 5.1 L340-346 states the restricted-GT evaluation scope.
Q4: Physical interaction
The Go2 demonstration currently uses human assistance for door and refrigerator opening; fully autonomous real-world manipulation has not been demonstrated. In simulation, both methods use the same force-driven executor without kinematic fallback. Body collisions and unsuitable approach positions can still cause failures.
Our focus is the high-level loop: when to interact, which object to choose, and how to update beliefs after execution. Evaluating autonomous real-world manipulation remains future work.
Revision: Sec. 5.5 L479-519 and Appendix G L1128-1135.
Q5: Baseline and ablations
We corrected implementation issues and reran every ablation on Mixed1000 under identical conditions. Full improves over Flat Object Memory by 12.60 SR points and 15.66 task-level IP points, while cutting attempts by 72.05%. Against Nearest Greedy, Full gains 4.70, 5.00, and 4.75 points in SR, ISR, and task-level IP. Perception-Only Update reaches only 8.50% SR.
Revision: Sec. 5.3 L442-470, Table 4 L432-440, and Appendix B.5 L828-848.
We have expanded unnecessary-container evaluation, stated the perception and physical-execution limits, and strengthened the ablations. Thank you for helping us position this work more precisely.