acceptodds
Under review as a conference paper at ICLR 2027

Interactive Navigation Beyond Reachability: Benchmark and Agentic Framework

Abstract

Existing embodied navigation tasks typically assume that traversability, topology, and target observation conditions remain fixed throughout navigation. In real environments, however, an agent can alter the navigation state through physical interaction to affect reachability, target visibility/accessibility, and navigation efficiency. We extend the scope of interactive navigation towards comprehensive state-changing environments, where the agent reasons beyond mere reachability to determine whether interaction is needed, select a task-relevant interaction object, and exploit the resulting change to continue toward the goal. To this end, we introduce MultINav-Bench, a 3K-episode ObjectGoal benchmark that pairs channel interactions (reachability) with container interactions (target visibility/accessibility), includes interaction-necessity annotations, and evaluates both navigation success and interaction-process effectiveness. Beyond the frozen ObjectGoal benchmark, the same interaction-state representation supports our implementation of PointGoal and grounded InstructionGoal tasks, as well as light and full data-collection modes. We further propose MultINav Agent, a three-stage agentic framework that provides semantic interaction guidance through a state-conditioned interaction graph: semantic interactive perception grounds rooms, interaction states, and expected effects; a vision-language model selects the semantically optimal interaction-navigation action; and execution outcomes verify state transitions, update graph relations, and trigger replanning. We evaluate MultINav Agent against non-interactive and naive-interaction baselines on task success, navigation efficiency, and interaction-process effectiveness.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Interactive Navigation, Benchmark, Agentic Framework

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nBGROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MultINav-Bench, a benchmark for interactive "object-goal navigation" that encompasses tasks such as opening doors, manipulating containers, and performing sequential interactions. It also proposes MultINav-Agent, a framework based on a vision-language model (VLM) that utilizes a persistent semantic interaction graph for action selection and state updates. While this approach demonstrates significant improvements over the enhanced baseline models, I have concerns regarding its level of innovation, the design of the benchmark, its reliance on "ground-truth perception," and the lack of sufficient validation in real-world environments.

Critical strengths

This work extends traditional "ObjectGoal navigation" to more complex scenarios—such as those involving doors and containers—by accounting for how interactive behaviors alter spatial connectivity and the visibility of target objects.

MultINav-Bench comprises 3,000 task episodes across 639 home environments, encompassing interactive tasks that involve passageways, containers, and a variety of other objects. Its automated data generation pipeline and annotations regarding "interaction necessity" provide a valuable evaluation resource for research in interactive navigation.

Compared to an adapted MoMa-LLM baseline model, MultINav-Agent improves the overall success rate from 10.53% to 41.63%. Furthermore, the study includes ablation experiments that offer in-depth insights into key components such as interaction memory, action selection, and state verification.

Critical weaknesses

The proposed framework primarily integrates existing techniques, including semantic scene graphs, action selection based on vision-language models (VLMs), and interaction-aware planning. Similar concepts have already been explored in works such as MoMa-LLM, CoINS, and previous graph-based navigation methods. The methodological innovation beyond these existing studies remains unclear.

In real-world scenarios, multiple instances of a target category may exist, some of which could be accessed without interaction. However, the framework mandates interaction for all tasks categorized as "Container" or "Mixed," with only "Channel" tasks allowing for non-interactive access. It is unclear whether the agent can avoid unnecessary interactions when other, directly accessible instances of the target are available.

The primary experiments utilize ground-truth data for object identity, category, and spatial positioning, which significantly simplifies the perception and localization processes. The framework’s performance under conditions involving realistic visual perception errors remains unknown, limiting the practical significance of the findings.

Real-world demonstrations rely on human assistance to open doors and refrigerators, while the simulation environment employs simplified interaction actuators (incorporating kinematic fallback mechanisms). Consequently, the framework’s effectiveness in scenarios requiring fully autonomous physical interaction has not been verified.

In the primary interaction evaluation, the framework is compared only against a modified version of MoMa-LLM. Furthermore, replacing the VLM-based selection strategy with a greedy selection strategy results in a decrease in success rate (SR) of only 1 percentage point; similarly, simplifying the object memory structure to a flat storage format leads to a drop of only 2 percentage points. These results fail to provide a compelling demonstration of the specific contributions of the individual modules.

Questions for authors

see weaknesses

2 comments

Thank you for the thorough review. We expanded unnecessary-container controls and clarified the scope of our perception and physical-execution claims. Line numbers below refer to the revised PDF.

Q1: Methodological contribution

Scene graphs, VLM-based selection, and interaction-aware planning are established ideas; we do not claim them as novel in isolation. Our contribution is their integration for ObjectGoal navigation: interaction states condition graph relations and candidates; candidates are constrained by route feasibility and observed evidence; and post-interaction observations plus execution feedback update beliefs before replanning. The benchmark separately labels required and unnecessary interactions across channel, container, and mixed tasks.

Revision: Related Work L134-137 and Sec. 4.2-4.3 L300-330.

Q2: Unnecessary interactions

The benchmark includes 1,000 Unnecessary episodes: 500 channel and 500 container controls. Container cases cover open containers (150), external targets (150), and accessible alternatives beside concealed candidates (200). Ours reaches 68.70% SR with 0.651 mean attempts; MoMa-LLM reaches 54.80% with 0.632.

Across all 4,000 episodes, Ours reaches 48.55% SR vs. 35.38% (+13.18 points). The gap remains on required tasks, including Container (46.50% vs. 28.40%) and Mixed (35.40% vs. 17.40%).

Revision: Appendix B.1 L773-783, Sec. 5.2 L367-394, and Table 3 L378-389.

Q3: Perception scope

Our primary comparison uses restricted-GT observations for currently visible objects. This reduces perception and instance-association confounds; it does not establish robustness to realistic perception errors. Hidden objects, joint metadata, and oracle plans are withheld.

The Go2 demonstration uses YOLO-based detection and segmentation with robot pose inputs. It is qualitative, not a quantitative robustness evaluation. We will evaluate this setting more systematically, including higher-fidelity simulation, in future work.

Revision: Appendix A L738-741 describes the Go2 perception setup; Sec. 5.1 L340-346 states the restricted-GT evaluation scope.

Q4: Physical interaction

The Go2 demonstration currently uses human assistance for door and refrigerator opening; fully autonomous real-world manipulation has not been demonstrated. In simulation, both methods use the same force-driven executor without kinematic fallback. Body collisions and unsuitable approach positions can still cause failures.

Our focus is the high-level loop: when to interact, which object to choose, and how to update beliefs after execution. Evaluating autonomous real-world manipulation remains future work.

Revision: Sec. 5.5 L479-519 and Appendix G L1128-1135.

Q5: Baseline and ablations

We corrected implementation issues and reran every ablation on Mixed1000 under identical conditions. Full improves over Flat Object Memory by 12.60 SR points and 15.66 task-level IP points, while cutting attempts by 72.05%. Against Nearest Greedy, Full gains 4.70, 5.00, and 4.75 points in SR, ISR, and task-level IP. Perception-Only Update reaches only 8.50% SR.

Revision: Sec. 5.3 L442-470, Table 4 L432-440, and Appendix B.5 L828-848.


We have expanded unnecessary-container evaluation, stated the perception and physical-execution limits, and strengthened the ablations. Thank you for helping us position this work more precisely.

The authors’ rebuttal addressed some of my concerns, and after considering the other reviewers’ comments, I maintain my original score.

Reviewer_8xH6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MultINav-Bench, a 3000-episode ObjectGoal benchmark across 639 houses, covering channel, container, and mixed interactions. It also proposes MultINav-Agent, which combines a state-conditioned scene graph, VLM-based action selection, and post-interaction verification. With restricted-GT perception and a shared interaction executor, it achieves 41.63% success versus 10.53% for an adapted MoMa-LLM baseline.

Critical strengths
  1. The benchmark provides a useful structure for studying interactive object search. Its task groups, reference interaction annotations, and unnecessary-interaction controls support more detailed analysis than overall success alone.
  2. The main comparison shares the perception interface, high-level model, and interaction executor. The substantial gains across task groups suggest that the proposed system contains useful design choices.
Critical weaknesses
  1. The distinction from MoMa-LLM needs to be clearer. MoMa-LLM already addresses door opening and container search using dynamic scene graphs, object states, interaction history, and feedback-based replanning. The controlled benchmark is a clearer contribution than the broad task formulation. For the method, I would like a concrete account of which effect representations, candidate constraints, and verification rules are added.

  2. The experiments show that the complete system performs better, but do not establish which proposed components explain the gain. On the 200-episode ablation subset, Full achieves 23% success, compared with 21% for flat memory and 22% for greedy selection; greedy selection also has better SPL and Cost.

Questions for authors
  1. What are the exact differences from the MoMa-LLM adapter in candidate generation, navigation, state updates, and termination?
  2. What explains the Container ISR gap of 2.0% versus 46.7%? A breakdown of selection, execution, and subsequent navigation failures would help.
  3. How often does each method use kinematic completion, and does the performance advantage persist without it?
2 comments

Thank you for pushing us to strengthen the evidence. We clarified the differences from MoMa-LLM, reran the ablations on Mixed1000, and added a Container failure breakdown. Line numbers below refer to the revised PDF.

Q1: Differences from MoMa-LLM

The methods differ in candidate generation, navigation, and belief updates. The MoMa-LLM adapter selects among navigate, go_to_and_open, explore, and done over named rooms and objects, then converts the command to navigation waypoints or a public opening request. MultINav-Agent generates entity-specific interaction candidates with an entity, operation, approach viewpoint, current state, and expected effect, then ranks the eligible candidates.

The adapter uses Voronoi room topology and 2D occupancy-grid A*. MultINav-Agent uses closed-loop ROS move_base with arrival checks and recovery. The adapter maintains an occupancy map, a room-object graph, recent action outcomes, and an object-state belief updated by opening results; MultINav-Agent reconciles post-interaction observations with execution feedback to update entity states, confidence, room connectivity, and later candidates.

Both methods use the same model, restricted-GT interface, executor, post-interaction observations, rollout limit, and evaluator. A completion claim or stop signal is not success; SR requires all necessary effects in one valid plan plus visibility and distance for the same legal target.

Revision: Sec. 4.2-4.3 L300-330, Sec. 5.1 L347-354, Appendix E L976-989, and Appendix G L1140-1161 (adapter interface, navigation, and episode limits).

W2: Ablation evidence

We corrected implementation issues and reran all variants on Mixed1000, using identical episodes, perception interface, and executor.

VariantSR ↑SPL ↑ISR ↑IP_class ↑IP_task ↑Attempts ↓
Full (Ours)35.400.163357.1548.6943.382.609
Flat Object Memory22.800.120749.2332.4527.729.334
Nearest Greedy31.100.166752.1544.1438.632.725
Perception-Only Update8.500.037840.7545.3238.452.369

Full improves over Flat from 22.80% to 35.40% SR and from 27.72% to 43.38% task-level IP, while cutting attempts by 72.05%. Against Nearest Greedy, Full improves SR, ISR, and task-level IP by 4.70, 5.00, and 4.75 points; Greedy has slightly higher SPL (0.1667 vs. 0.1633). Perception-Only Update drops to 8.50% SR.

Revision: Sec. 5.3 L442-470, Table 4 L432-440, and Appendix B.5 L828-848 specify each ablation.

Q2: Container results

The earlier 10.30% SR / 2.00% ISR came from a superseded evaluation affected by perception leakage. After correction, MoMa-LLM obtains 28.40% SR / 31.90% ISR, while Ours obtains 46.50% / 50.00%.

We added the requested stage breakdown (Appendix Table 8). Each of the 1,000 Container episodes per method is assigned to exactly one outcome stage:

Container stage (1,000 episodes)OursMoMa-LLM
Selection/approach: required step not attempted416643
Execution: required action failed8438
Subsequent navigation: effects complete, task failed3535
Success465284

The baseline leaves a required step unattempted in 227 more episodes. Physical execution failure is not the main source of the gap: Ours has 46 more execution failures. Failures after all required effects, such as terminal visibility or distance, are equal at 35. Across all 3,000 required episodes, 280 of Ours' 388 additional successes come from fewer unattempted episodes (Figure 4). The first stage combines selection, exploration, and approach navigation; we do not split it further.

Revision: Sec. 5.2 L408-424 and Figure 4 summarize the breakdown; Appendix F Table 8 L1092-1102 gives the full counts.

Q3: Kinematic completion

Neither method used kinematic completion. Both used the force-driven executor, so fallback usage was 0, and the reported advantage is obtained without it. Collisions or unsuitable approach positions can still cause failures.

Revision: Appendix G L1128-1135 describes the shared force-driven executor without kinematic completion.


These revisions clarify the MoMa-LLM comparison, provide full-split ablations, and localize the Container gap with failure-stage counts. We appreciate your feedback.

Thank you for the clarifications and additional experiments. My concerns about the novelty and the attribution of gains relative to MoMa-LLM are not fully resolved. I therefore maintain my score.

Reviewer_KUteOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies object navigation where the agent needs to change the environment to reach or find a target. This includes opening doors to enter rooms and opening containers to reveal objects. The authors introduce MultINav Bench with 3000 episodes across 639 houses. It covers door interactions, container interactions, and tasks that require both. Their agent maintains an interaction graph and uses a VLM to select actions. The graph is updated using observations and execution feedback. The method achieves 41.63% success compared to 10.53% for an adapted MoMa-LLM baseline. Both methods use the same restricted ground-truth perception interface and interaction executor. The paper also includes ablations and a real-world demonstration with human-assisted opening actions.

Critical strengths
  1. The benchmark addresses a useful problem. Opening a door changes where the agent can go. Opening a container changes what it can see. Evaluating both within the same task makes sense. The mixed tasks are particularly useful since they require the agent to handle these interactions in sequence. The benchmark also separates final task success from intermediate interaction progress.

  2. The improvement over the adapted MoMa LLM baseline is substantial. Overall success increases from 10.53% to 41.63%. Mixed-task success increases from 1.70% to 24.10%. The methods share the same perception interface, model backbone, and interaction executor. The state update ablation is also useful. Success drops from 23% to 5.5% with perception only updates. This supports the importance of execution feedback.

Critical weaknesses
  1. The graph structure and effect guided semantic selection are central parts of the proposed method. However, their gains are small in Table 4. Full achieves 23% success compared to 21% for Flat Object Memory and 22% for Greedy Selection. On 200 episodes, these differences are only four and two additional successes. No uncertainty estimates are reported. Greedy Selection also achieves better SPL at 0.134 versus 0.107 and lower cost at 0.933 versus 0.961. Flat Object Memory achieves higher ISR at 43.8% versus 41.8%. These results do not establish a clear overall advantage for the more complex design. The graph does improve IP, but the limitations of that metric make this less conclusive.

The large drop with Perception only Update is useful evidence for execution feedback. However, it does not establish that the proposed graph representation is needed to use that feedback effectively. The exact information retained by each ablation is also not sufficiently specified. This makes it difficult to judge whether the comparisons isolate the intended components.

The main interactive comparison uses only one adapted baseline. CoINS is described as the closest comparator but is not evaluated. Sharing the backbone and executor is a positive aspect of the comparison. Still, the results do not isolate the contribution of the proposed interaction reasoning from other differences in navigation and exploration. For example the Channel SR improves by 36.20 percentage points while Channel ISR improves by only 5.81 points. This suggests that a substantial part of the success gap involves behavior beyond completing the required interactions. The paper acknowledges this limitation, but stronger controlled evidence is still needed to support the method contribution.

  1. All container and mixed episodes are labeled as requiring interaction. However, Appendix E allows terminal success without completing the annotated required effects. For the proposed method, 450 container episodes succeed but only 435 of those successes complete all required effects. Similarly, 241 mixed episodes succeed but only 223 of those successes complete all required effects. These cases may have valid explanations. Partial openings or alternative ways of observing the target could satisfy the goal. The issue is that these explanations are not analyzed. It remains unclear how closely the annotations capture actual necessity rather than the requirements of a particular reference plan. A baseline with explicit interaction actions disabled would help establish this.

The unnecessary-interaction controls are also limited. All 346 such episodes are channel tasks. There are no corresponding container controls. As a result, the benchmark provides limited evidence that the agent can decide when opening a container is unnecessary. Required and unnecessary results are not reported separately in the main table either.

The interaction metrics do not fully address this gap. IP gives credit for a physical effect on an object of the annotated class. The object does not need to be the relevant instance. Reopening an object after closure can also receive credit again. A higher IP therefore does not necessarily mean that the agent selects interactions that help find the target. ISR measures annotated effect completion but does not independently check whether the effect enables passage or exposes the target. These are valid diagnostic measures, but they are weaker evidence for purposeful interaction than the framing suggests. More direct results on unnecessary attempts and task-relevant effects are needed.

Questions for authors

Can the authors please respond to these questions:

  1. How often does each method succeed after completing all, some, or none of the annotated required effects? Please report this separately for each task type. What explains successes without full effect completion? How much success remains if explicit interaction actions are disabled?

  2. Can you report results separately for required and unnecessary episodes? In the unnecessary subset, how often does each method interact anyway? How do you evaluate whether container interactions are unnecessary given that all container episodes require interaction?

  3. What exactly changes in the three ablations? Does Flat Object Memory retain the same state information and execution feedback? What rule does Greedy Selection use? What information is removed in Perception-only update? Paired uncertainty estimates would help assess the small success differences in Table 4.

  4. Why was CoINS not included? How was the MoMaLLM adaptation validated for container interactions? Its container SR is 10.30% while its ISR is 2.00%. Please explain this gap. Can the authors also clarify whether the methods use the same navigation components and rollout limits.

  5. Can the authors provide a measure of interaction relevance beyond category-level IP? For example, how often does an interaction actually enable traversal toward the goal or expose the target? This would help interpret the graph variant's IP improvement over Flat Object Memory.

  6. Which executor produced the reported results? Appendix F describes a kinematic completion fallback. The supplied force_interaction.py explicitly states that it does not use a qpos success fallback. Please explain this difference and report how often the fallback was used if it was enabled. I could not locate the evaluated agent implementations or the full frozen benchmark manifest in the archive.

3 comments

Thank you for the careful review. We revised the success criterion, added unnecessary-interaction controls for channels and containers, and reran the ablations on the complete Mixed1000 split. Line numbers below refer to the revised PDF.

Q1: Success and required effects

A required episode is successful only when all necessary effects in at least one valid plan are verified and the same legal target is visible and within the distance threshold at termination. Partial completion across different plans no longer counts.

Appendix Table 9 reports SR by necessary-effect completion:

MethodScopeNonePartialAll
OursMixed0/2490/358354/393 (90.08%)
OursRequired0/1,1440/3581,255/1,498 (83.78%)
MoMa-LLMMixed0/3590/443174/198 (87.88%)
MoMa-LLMRequired0/1,4100/448867/1,142 (75.92%)

Thus no success comes from partial completion. Disabling interaction gives 0% required-task SR by definition; this is not a separate policy run.

Revision: Sec. 3 L192-199 and Appendix E L976-989 define success; Sec. 5.2 L426-431 and Appendix Table 9 L1103-1112 report this analysis.

Q2: Unnecessary-interaction controls

The benchmark now has 1,000 Unnecessary episodes: 500 channel and 500 container controls. Container controls place the target in an open container (150), outside a closed container (150), or provide an accessible alternative beside a concealed candidate (200). Ours reaches 68.70% SR with 0.651 attempts; MoMa-LLM reaches 54.80% SR with 0.632 attempts.

On all 4,000 episodes, Ours reaches 48.55% SR vs. 35.38% (+13.18 points). The gap remains on required splits: Container 46.50% vs. 28.40% and Mixed 35.40% vs. 17.40%.

Revision: Appendix B.1 L773-783 describes the controls; Sec. 5.2 L367-394 and Table 3 L378-389 report results.

Q3: Ablations

We corrected implementation issues and reran all variants on Mixed1000 (1,000 episodes, 369 houses) with the same episodes, perception interface, and executor. Flat Object Memory removes explicit relations, interaction states, and operation history. Nearest Greedy keeps the graph, state memory, and candidate filtering but picks the nearest candidate. Perception-Only Update disables direct updates from execution outcomes.

VariantSR ↑SPL ↑ISR ↑IP_class ↑IP_task ↑Attempts ↓
Full (Ours)35.400.163357.1548.6943.382.609
Flat Object Memory22.800.120749.2332.4527.729.334
Nearest Greedy31.100.166752.1544.1438.632.725
Perception-Only Update8.500.037840.7545.3238.452.369

Full improves SR over Flat by 12.60 points and cuts attempts by 72.05%. Against Nearest Greedy, it improves SR, ISR, and task-level IP by 4.70, 5.00, and 4.75 points, while Greedy has slightly higher SPL.

Revision: Sec. 5.3 L442-470, Table 4 L432-440, and Appendix B.5 L828-848.

Q4: Baseline scope and corrected results

CoINS addresses door-mediated free-space and connectivity in PointGoal, not container search; its unreleased code prevented a controlled comparison. We therefore use a MolmoSpaces adaptation of MoMa-LLM, which supports both doors and containers.

The earlier MoMa-LLM Container result (10.30% SR, 2.00% ISR) was affected by perception leakage. After correction, MoMa-LLM obtains 28.40% SR and 31.90% ISR; Ours obtains 46.50% and 50.00%. Appendix Table 8 shows that a required step is left unattempted in 643 of 1,000 baseline Container episodes, compared with 416 for Ours.

Revision: Related Work L134-137, Sec. 5.2 L408-424, Appendix F Table 8 L1092-1102, and Appendix G L1140-1161.

Q5: Task-relevant interaction

We added task-level IP. It credits a necessary object only after all of its required effects in the selected plan are verified; wrong instances, non-required objects, and repeated completions receive no credit. Overall task-level IP is 41.65% (Ours) vs. 38.10% (MoMa-LLM).

Revision: Appendix E L1034-1049 defines IP_task; Table 3 L378-389 reports it.

Q6: Interaction executor and fallback

Both methods used the shared force-driven executor without kinematic completion, so fallback usage was 0. Failures can still result from body collisions or poor approach positions.

Revision: Appendix G L1128-1135 describes the shared force-driven executor without kinematic completion.


These revisions make success stricter, add container-side unnecessary controls, and test the main components on the full Mixed split. Thank you for helping us improve the evaluation.

Thank you for the response. Some of my concerns are addressed. However, the following points remain unresolved:

  1. There are gaps in the perception leakage and it seems there are implementation issues with limited clarity on which methods were affected and whether all reported methods were rerun under the corrected setup. There is a substantial change in the baseline which needs explanation
  2. Flat object memory removes interaction states and operation history with explicit relations. A comparison that preserves the same state info and feedback would help understand the benefit of graph structure. Without this it is difficult to analyse the improvement
  3. Following up on my original question: your new rule counts a task as successful if the agent completes the interaction but does that agent actually need those interactions to see the target? No such results are added by disabling interactions and checking if the model still reaches the goal?

I appreciate the additional results but since my concerns are not fully addressed, I will retain my original score. I believe the paper will benefit from another round of revision

Thank you for the follow-up.

1. Leakage, fixes, and the baseline change

All reported results, including MoMa-LLM and every ablation, were rerun under the corrected setup with the same local VLM backend (no timeouts), executor, restricted-GT interface, and stricter evaluator.

IssueAffectedCorrection
Scene-level perception leakageAll methodsOccluded objects are no longer observable
Success did not require all necessary effectsAll methodsSR now requires them, so ISR ≥ SR
Fixed interaction pose (original MoMa-LLM code)MoMa-LLM (doors, containers)Pose clears the joint's swept region
Contents assumed known after openingMoMa-LLMOnly objects seen after opening are added
Local VLM timeoutsOursRemoved

Container (10.30/2.00 → 28.40/31.90 SR/ISR). With the fixed pose, the robot body blocked the opening motion, so containers were rarely opened (2.00% ISR); the same issue lowered Channel results. Meanwhile, leakage exposed some targets inside closed containers, and the old evaluator did not require the opening, so these counted as successes (10.30% SR). Fixing the pose lets MoMa-LLM actually open containers, raising ISR and valid SR; the leakage fix and stricter evaluator remove the invalid successes, and ISR ≥ SR now holds for every method and split.

Mixed (1.70/25.37 → 17.40/42.02). The original adapter marked a container's contents as known after opening and made no post-opening observation. Under restricted-GT, the target was never registered, giving high ISR but near-zero SR. Registering only objects seen after opening resolves this.

MethodSplitOld SROld ISRNew SRNew ISR
MoMa-LLMChannel*19.6054.2840.9062.35
MoMa-LLMContainer10.302.0028.4031.90
MoMa-LLMMixed1.7025.3717.4042.02
OursChannel*55.8060.0943.6060.50
OursContainer45.0046.7046.5050.00
OursMixed24.1045.0735.4057.15

*Not directly comparable: the submission's Channel split included 346 unnecessary episodes.

The fixes strengthen the baseline: the gap narrows from 31.10 to 12.93 SR points (41.83% vs. 28.90% on required episodes).

2. Relations vs. interaction-state memory

We added Stateful Flat Memory on Mixed1000 and renamed Flat Object Memory to Stateless Flat Memory. Both remove graph edges, room nodes, and relational fields (membership, connectivity, containment) and give the planner a relation-free object list; the stateful variant keeps each object's interaction state, failure information, and recent operation outcomes.

VariantSRSPLISRIP_taskAttempts
Full35.400.163357.1543.382.609
Stateful Flat29.500.174756.7237.343.349
Stateless Flat22.800.120749.2327.729.334

State and feedback account for 6.70 of the 12.60-point SR gap, nearly all of the ISR gain, and most of the attempt reduction. Relations add 5.90 SR and 6.04 IP_task points with fewer attempts at similar ISR: they help choose which objects to operate on and turn progress into success. Relation-derived candidates and prompt fields are removed together with the edges, so the gap measures the relational planning interface as a whole.

Revision: Sec. 5.3 L443-480, Table 4 L432-441, and Appendix B.5 L829-863 and E L1097-1109.

3. Are interactions needed to see the target?

Yes. Episodes are collected on the occupancy map with ground-truth trajectories and interactions, and all 3,000 required episodes were verified after collection: with every interaction disabled, we scanned all reachable poses, and the target is visible within the success distance in 0/3,000 episodes. This is a property of the scenes, not of the evaluator: even if interaction completion were not checked, an interaction-disabled agent could not succeed on any required episode. It can succeed only on the 1,000 Unnecessary episodes, which have zero-interaction routes by construction.

Revision: Sec. 3 L218-220 and Appendix B.7 L875-894.


Thank you for the continued engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.