acceptodds
Under review as a conference paper at ICLR 2027

Which to Visit, When to Stop: Autonomous Multi-Object Search with Interactive 3D Memory

Abstract

Existing embodied navigation benchmarks support diverse goal modalities and long-horizon tasks, but typically prescribe goal sequences or task lengths, leaving two central decisions underexplored: which to visit and when to stop. We introduce MOST-Bench, a benchmark for autonomous search in embodied navigation, where agents must select target instances, organize visits, track progress, and determine when the search is complete. MOST-Bench comprises two complementary tasks: Many-Object Navigation (ManyON), which requires visiting a specified number of distinct matching instances, and All-Object Navigation (AllON), which requires finding all matching instances and deciding when to stop without knowing the target count. The benchmark contains 750 episodes across 36 HM3D scenes, covering five task–modality combinations with 2–10 targets per episode. We also introduce MOST-Nav, a training-free baseline that combines vision-language model decision-making with interactive 3D memory reconstructed from RGB observations. We evaluate seven training-based and training-free frameworks on task completion, reporting accuracy, and target coverage. The best-performing method achieves an overall success rate below 18%, highlighting the difficulty of autonomous search and motivating further research on coordinating exploration, instance-level memory, and search termination.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied navigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_pbm5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MOST-Bench, a benchmark for autonomous multi-object search. Agents must choose which target instances to visit, in what order, and when the search is complete. In ManyON the agent must find and report a given number of distinct instances with no prescribed visit order. In AllON the target count is hidden, and the agent must find every instance that satisfies the goal and then issue FINISH. The benchmark has 750 episodes over 36 HM3D scenes with category and attribute goals, plus image goals in ManyON. The paper also proposes MOST-Nav, a training-free reference baseline in which a VLM makes decisions over a 3D memory built from RGB observations with VGGT-SLAM. Seven methods are compared on task completion, reporting accuracy, and target coverage.

Critical strengths
  1. The problem setting is a useful one. Separating ManyON, where the required count is given, from AllON, where it is hidden, lets the benchmark test instance selection, visit order, and progress tracking on one side and the decision to stop an exhaustive search on the other.
  2. The evaluation covers both trained policies and training-free agents, and reports results by goal modality and by target count. Reporting precision, recall, F1, and full coverage next to SR makes the gap between finding targets and completing the task visible.
  3. MOST-Nav is a concrete reference baseline. It lets the VLM query a spatial representation built from RGB observations, retrieve past frames, ground a chosen target in 3D, and check whether two candidates are the same instance, all inside its decision loop.
Critical weaknesses

W1. The baseline adapters are not validated enough to interpret the AllON differences. On AllON, the full coverage rates of 3D-Mem, SCOPE, and MOST-Nav are 21.3%, 22.0%, and 18.3%. Among episodes with full coverage, roughly 30%, 30%, and 100% also end with FINISH (Table 2 and Figure 6(b)). Termination therefore carries much of the SR comparison, so the reader needs a way to check that the reporting and termination mechanisms added to the existing methods work as intended. The released 3D-Mem adapter depends on external code both to associate report history with current candidate IDs and to handle FINISH, and the version used in the experiments is not available for inspection. The SCOPE adapter is not provided at all. If previous reports cannot be reliably matched to current candidates, duplicate reports and unnecessary revisits would affect precision and could also decide whether the agent terminates within its budget. The current results do not separate these possible adapter effects from the exploration and termination abilities of the underlying methods.

W2. The action budgets have not been shown to be comparable. 3D-Mem and SCOPE get 80 native steps, while MOST-Nav gets 800 primitive actions. Keeping the budget ratio from earlier GOAT evaluations does not by itself give the methods equal room to explore. The released 3D-Mem adapter documentation specifies at most 1 m of movement per native step, and turns, extra observations, and reports are counted differently from the primitive action protocol. In AllON the agent may keep exploring after it has found every target in order to decide whether to stop, so the budget matters there in particular. A comparison under a shared travel-distance limit, or results across several budgets, would show how sensitive the conclusions are to this setting.

W3. The experiments do not isolate the contributions of MOST-Nav’s components. MOST-Nav combines interactive 3D memory, selective RGB retrieval with target localization, visual duplicate checking, and termination decisions based on spatial evidence. No ablation separates these parts. The evidence that the proposed memory design is what improves AllON SR and precision is therefore thin.

W4.** The cost of MOST-Nav’s VLM-memory interaction and online reconstruction is not reported.** Section 4.2 and Figure 3 allow up to 15 rounds of memory queries per decision, but the paper gives no average number of rounds, no VLM call or token counts per episode, and no runtime. It also does not report memory use for online VGGT-SLAM reconstruction or for storing RGB observations and geometry. Without these numbers it is hard to judge what the reported performance costs during a long search.

Questions for authors
  1. Please report the Repeated Target Selection Rate (RTSR) of 3D-Mem, SCOPE, and MOST-Nav on AllON, with its exact definition, and break invalid reports down into duplicates and other errors. How are object IDs from previous reports matched to current candidates in the 3D-Mem and SCOPE adapters used in the experiments?
  2. If 3D-Mem and SCOPE have been run with larger step budgets, how do AllON SR, full coverage, and the share of full-coverage episodes that end with FINISH change?
  3. How much do interactive 3D memory retrieval, visual duplicate checking, and the spatial information used for termination each contribute to AllON SR and precision? Ablations that remove or replace these components under the same exploration and reporting settings would help.
9 comments
AuthorsOpenReview ↗

Author Response (1/3)

We thank the reviewer for the detailed comments and provide our responses below. We have revised the manuscript and prepared an updated PDF, with changes addressing shared concerns highlighted in blue and those specific to your comments in burnt orange.

1. Baseline Adapter Validation: Instance Matching, Duplicate Reports, and Termination (W1/Q1)

Thank you for highlighting the importance of validating the baseline adapters. We have added analyses of repeated reports, error breakdowns, and FINISH events on AllON, distinguishing benchmark-level duplicate reporting from each method’s internal target selection.

Evaluation protocol and RTSR. Report correctness is determined solely by the evaluator by matching the reporting position against ground-truth (GT) instances, independently of adapter-side ID association. For each episode, let cic_i denote the number of times GT target ii is matched by a report, and let qiq_i denote its final quota (qi=1q_i=1 by default; qi=nq_i=n when nn targets are jointly covered by a single report). We define the Repeated Target Selection Rate (RTSR) as:

RTSR=∑imax⁡(0, ci−qi)∑ici,\mathrm{RTSR}=\frac{\sum_i \max(0,\,c_i-q_i)}{\sum_i c_i},

with RTSR=0\mathrm{RTSR}=0 if ∑ici=0\sum_i c_i=0. For example, reporting target A five times and target B once yields (5−1)/(5+1)=4/6(5-1)/(5+1)=4/6. We macro-average RTSR across the 300 AllON category/attribute episodes. Using the quota at reporting time, we further partition false positives (FPs) into duplicate FPs (re-reports exceeding the quota) and other FPs (reports with no nearby target, excluded from RTSR).

MethodMacro-averaged RTSRDuplicate FPsOther FPsTotal FPs
3D-Mem52.52%11,0721,14912,221
SCOPE37.24%5,1324695,601
MOST-Nav19.69%7314411,172

Event-level adapter checks. For every report, the adapter’s runtime judgment agrees with the evaluator’s scoring. All explicit FINISH events—57/300 episodes for 3D-Mem and 70/300 for SCOPE—agree with the episode-end records, confirming that the termination interface is reachable and correctly recorded. We traced each duplicate report in 3D-Mem and SCOPE to its source and found two cases. In the first, the method re-selects a native object ID that is already listed in the report history shown to the model. Because the matching ID was explicitly available, these duplicates do not stem from a failure to associate past reports with current candidates. In the second, the same physical instance reappears under a new native ID, reflecting instance fragmentation within the method's own memory. In both cases, the duplicates originate from the methods' native target selection and instance tracking rather than from the adapter interface.

Report-history matching. 3D-Mem and SCOPE share a unified adapter that converts a MOST-Bench episode into sequential search rounds with a shared episode state and step budget, leaving their native navigation, mapping, and planning unchanged. Scene memory persists across rounds. After each report, the selected candidate’s native object ID, category, and reporting position are appended to the report history and included in subsequent prompts for candidate comparison. To prevent GT leakage, this history omits correctness labels, duplicate flags, and remaining target counts. Previously reported IDs are not forcibly masked. The model autonomously issues FINISH based on its observations, memory, and report history.

MOST-Nav’s termination mechanism. When VGGT-SLAM detects no frontiers for an extended period at a high step count, MOST-Nav prompts the VLM to synthesize accumulated spatial evidence and decide whether to issue FINISH. Its high FINISH rate at full coverage relies on explicitly modeling explored regions and detecting unexplored areas. This mechanism can also prompt premature termination before all targets are found; Point 3 quantifies this effect through an ablation. The triggers use only agent-observable states, without GT leakage, and the final decision remains with the VLM.

AuthorsOpenReview ↗

Author Response (2/3)

2. Comparability of Action Budgets and Their Effects on AllON Coverage and Success Rates (W2/Q2)

Thank you for raising this concern. MOST-Nav shares the same primitive-action output interface and 800-action budget as MTU3D. For 3D-Mem and SCOPE, we follow their native GOAT evaluation convention of 80 navigation iterations, each of which can execute a trajectory segment. The 1 m per-step movement limit described in our released 3D-Mem adapter documentation is inherited from the original 3D-Mem settings (https://github.com/UMass-Embodied-AGI/3D-Mem.git) rather than introduced by our adaptation. To compare these action granularities beyond nominal step counts, we measured travel distance for all budget-exhausted episodes in the evaluation set.

Table R1. Travel distance on method-specific subsets of budget-exhausted episodes.

MethodBudget capEpisodesMean distance (m)Median distance (m)
3D-Mem80 navigation iterations311≥ 60.9≥ 70.9
SCOPE80 navigation iterations311≥ 69.0≥ 75.6
MTU3D800 primitive actions38765.176.4
MOST-Nav800 primitive actions42278.880.3

The nominal tenfold budget difference does not translate into a tenfold difference in physical travel. Even the conservative median distances for 3D-Mem and SCOPE reach 70.9 m and 75.6 m, respectively, compared with 80.3 m for MOST-Nav. These observations provide empirical context for the chosen settings: the methods operate on a similar physical travel scale under their native interfaces.

We further examined whether failures to terminate occurred because full coverage was reached immediately before budget exhaustion. The AllON traces show that this is often not the case: 3D-Mem and SCOPE achieve full coverage in 64 and 66 episodes, respectively, but issue FINISH in only 19 and 20 of them.

Table R2. AllON coverage and termination diagnostics. Post-coverage steps are measured in native navigation iterations among full-coverage episodes without FINISH.

MethodAllON SRFull coverageFINISH given full coverageMedian post-coverage steps without FINISH
3D-Mem6.30%21.3% (64/300)29.7% (19/64)50
SCOPE6.70%22.0% (66/300)30.3% (20/66)44

These agents frequently continue for substantial periods after finding every target without recognizing completion. Among full-coverage episodes without FINISH, 82.2% for 3D-Mem and 78.3% for SCOPE continue for at least 20 additional native navigation iterations. Together with the travel-distance analysis, these results indicate that the observed gap between success rate and full coverage involves a termination bottleneck that cannot be explained solely by the nominal difference in budget caps. This reflects a central challenge evaluated by our benchmark: which instances to visit and when to stop.

AuthorsOpenReview ↗

Author Response (3/3)

3. Ablations: Memory Retrieval, Visual Deduplication, and Spatial Termination (W3/Q3)

We evaluated three ablations on the same 108 episodes, sampling one episode per difficulty tier from each of the 36 scenes.

Table R3. Ablation results (% unless reported as counts).

MetricFullw/o RGB memory retrievalw/o visual deduplicationw/o spatial termination mechanism
SR ↑13.94.613.01.9
Precision ↑57.551.046.853.1
Recall ↑42.632.639.039.3
Duplicate-report rate ↓6.96.827.011.3
Full-coverage episodes (count) ↑1561411
FINISH rate given full coverage ↑100.083.3100.018.2
FINISH rate given incomplete coverage ↓36.634.342.614.4

Each component addresses a distinct failure mode. Removing RGB memory retrieval reduces recall by 10.0 percentage points. Removing visual deduplication increases the duplicate-report rate from 6.9% to 27.0% and reduces precision by 10.7 percentage points. Without the spatial termination mechanism, 9 of the 11 full-coverage episodes fail to issue FINISH, reducing SR to 1.9%.

The spatial termination mechanism also increases the FINISH rate among episodes with incomplete coverage from 14.4% to 36.6%, revealing a trade-off between completion signaling and premature termination. These ablations demonstrate complementary benefits for target coverage, reporting accuracy, and completion signaling, and show why SR should be interpreted alongside these diagnostics.

4. Computational Costs: VLM Calls, Runtime, and Memory Usage (W4)

We provide measurements of VLM interactions, runtime, memory usage, and storage below. The limit of 15 memory-query rounds is a maximum interaction budget; in practice, the measured workload averages 1.92 VLM calls per decision on AllON and 2.05 on ManyON.

Table R4. MOST-Nav’s per-episode computational costs. Values are means; GPU figures are averages of per-episode peak allocated and reserved memory.

MetricAllONManyON
End-to-end runtime (min)58.250.5
Decision cycles89.575.8
Decision VLM calls170153
VLM calls per decision1.922.05
Input / output tokens2.30M / 10.3k2.21M / 9.1k
Stored RGB frames557486
RGB disk storage (GB)0.820.65
Local server peak GPU memory, allocated / reserved (GB)9.2 / 10.09.1 / 9.9
Server RAM, RSS (GB)9.58.5
Geometry within server RSS (GB)5.04.3
Agent RAM, RSS (GB)4.83.8

Online reconstruction accounts for a measurable but relatively small share of runtime. An AllON episode builds an average of 35.7 VGGT submaps, taking 2.5 minutes including optimization, or approximately 4% of total runtime. Decision VLM latency totals 6.6 minutes (11%), server-side captioning takes 2.7 minutes (5%), and SAM segmentation takes 0.85 minutes (1.5%). Approximately 76% of runtime is attributable to the remaining pipeline and execution overhead, including simulation, frame transfer, mapping and planning, logging, and resource contention during concurrent execution.

These wall-clock costs reflect the full evaluation setup. Stored RGB observations consume disk space, while geometry accounts for approximately half of server RAM usage. GPU measurements cover the local reconstruction and segmentation server.

Reviewer_8HAwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MOST-Bench for multi-object search, with tasks requiring agents either to visit a known quota of objects or to find all matching objects when the count is hidden. It also presents MOST-Nav, a training-free VLM agent with interactive 3D memory. The benchmark contains 750 episodes across 36 HM3D scenes. MOST-Nav achieves 17.73% overall SR, with 18.33% SR on AllON.

Critical strengths
  1. ManyON and AllON distinguish quota-based search from exhaustive search with an unknown target count. Reporting SR alongside precision, recall, and F1 helps separate task completion from partial discovery and reporting errors.
  2. The benchmark analysis identifies a meaningful failure pattern: finding many targets does not ensure that an agent can report them accurately or decide when to stop.
Critical weaknesses
  1. The evaluated methods do not all receive comparable budgets: some use 800 primitive-action steps, while 3D-Mem and SCOPE use an 80-step segment budget following their own evaluation protocol. This makes aggregate performance comparisons difficult to interpret.
  2. The benchmark is built from 36 HM3D validation scenes, and the paper does not establish how well its task distribution or results generalize to other environments. MOST-Nav’s low success rates establish task difficulty more clearly than they establish a strong solution to it.
Questions for authors
  1. Can you provide comparisons under a common movement or interaction budget, or otherwise quantify how the differing budget definitions affect the method ranking?
  2. How sensitive are the task distribution and method results to the chosen HM3D scenes, particularly for the attribute-based and exhaustive-search tasks?
4 comments
AuthorsOpenReview ↗

Author Response (1/2): Evaluation Budgets and Scene-Level Robustness

We thank the reviewer for the thoughtful feedback and provide our responses below. We have revised the manuscript and prepared an updated PDF, with changes addressing shared concerns highlighted in blue and those specific to your comments in purple.

1. Consistency of Evaluation Budgets and the Impact of Budget Differences on Method Rankings (w1/q1)

Thank you for raising this question. 3D-Mem and SCOPE are capped at 80 navigation segments, and MOST-Nav and MTU3D at 800 primitive steps. To compare these caps in physical terms, we measured the trajectory length of every budget-exhausted episode by summing the Euclidean distances between the start and end of each segment or step. Because 3D-Mem and SCOPE follow Habitat pathfinder paths within each segment, their measurements are lower bounds on actual travel.

Table R1. Travel distance on each method’s budget-exhausted episodes.

MethodBudget capBudget-exhausted episodesMean distance (m)Median distance (m)
3D-Mem80 segments311≥ 60.9≥ 70.9
SCOPE80 segments311≥ 69.0≥ 75.6
MTU3D800 primitive steps38765.176.4
MOST-Nav800 primitive steps42278.880.3

The two caps allow comparable amounts of travel. Even SCOPE’s lower-bound estimates are only 12.4% below MOST-Nav in mean distance and 5.9% below it in median distance. Actual travel also depends on each method’s navigation behavior and allocation of actions: under the same 800-step cap, MOST-Nav travels 78.8 m on average, compared with 65.1 m for MTU3D. These measurements provide a physical calibration of the budget choices and support their practical comparability. We will include this analysis in the revised manuscript.

2. Effects of Scene Selection on Task Distribution, Method Performance, and Generalization Across Environments (w2/q2)

Thank you for raising this question. Our complete automated benchmark-construction pipeline is publicly available at our anonymous repository. It requires only semantically annotated scenes and a navigation mesh, and can therefore be applied to the 145 semantically annotated HM3D training scenes, as well as other datasets with these inputs. To support evaluation on held-out environments and preserve the official training/validation separation, we construct the current benchmark exclusively from the 36 HM3D validation scenes.

To assess sensitivity to scene selection, we conducted scene-level bootstrap analysis with 3,000 resamples and leave-one-scene-out (LOSO) analysis, examining both the AllON task distribution and method performance, with particular attention to attribute goals.

The AllON task distribution is stable under scene resampling. The easy, medium, and hard tiers each account for approximately one-third of episodes, with 95% bootstrap confidence intervals of [0.27, 0.40], [0.28, 0.39], and [0.26, 0.41], respectively. Attribute goals account for 50% of episodes, with a 95% confidence interval of [0.43, 0.57], and occur in 35 of the 36 scenes, indicating broad representation of attribute-based exhaustive-search tasks.

The main AllON results are also robust to scene composition (Table R2). On the full AllON set, MOST-Nav ranks first in both SR and F1 in all 3,000 bootstrap resamples and all 36 LOSO splits. Removing any single scene leaves its SR within 16.4–19.1% and F1 within 46.1–47.9%. Its advantage is particularly consistent for attribute goals: MOST-Nav achieves 51.3% F1, compared with 20.2% for 3D-Mem and 26.5% for SCOPE, and ranks first in both metrics in every bootstrap resample and LOSO split. For category goals, it ranks first in 93.1% of bootstrap resamples for SR and 99.4% for F1, and in every LOSO split for both metrics.

These analyses support the stability of the task distribution and MOST-Nav’s AllON advantage within HM3D. Generalization to other scene datasets remains to be evaluated; the released construction pipeline enables this evaluation beyond the current 36 scenes.

Table R2. Scene sensitivity of AllON results (%; brackets show 95% scene-bootstrap confidence intervals).

SubsetMetric3D-MemSCOPEMOST-NavMOST-Nav ranks first (bootstrap / LOSO)
CategorySR8.7 [3.4, 14.6]10.0 [4.8, 15.6]16.0 [8.8, 24.2]93.1% / 35 of 35
F130.7 [26.6, 35.0]35.4 [31.0, 39.4]42.7 [36.6, 48.5]99.4% / 35 of 35
AttributeSR4.0 [1.1, 7.5]3.3 [0.7, 6.7]20.7 [13.5, 28.3]100% / 35 of 35
F120.2 [16.8, 23.9]26.5 [21.6, 31.5]51.3 [45.7, 56.6]100% / 35 of 35
AllSR6.3 [2.9, 10.3]6.7 [3.4, 10.2]18.3 [12.2, 24.6]100% / 36 of 36
F125.4 [22.5, 28.3]30.9 [27.4, 34.7]47.0 [42.4, 51.7]100% / 36 of 36
AuthorsOpenReview ↗

Author Response (2/2): Task Difficulty and MOST-Nav’s Effectiveness

3. Interpreting MOST-Nav’s Low Success Rate: Task Difficulty and Method Effectiveness (w2)

Thank you for raising this concern. We interpret MOST-Nav’s low absolute success rate from two complementary perspectives.

First, MOST-Bench evaluates a distinct and demanding search problem. Existing sequential navigation benchmarks, such as GOAT-Bench, supply successive goals and prescribe their order. ManyON instead requires finding a specified number of distinct matching instances, with the agent choosing their visit order and, when additional candidates exist, their subset. AllON requires finding every matching instance without knowing the target count and explicitly deciding when the search is complete. Success therefore depends jointly on navigation, memory, instance tracking, target selection, and—in AllON—search termination. MOST-Nav’s overall recall of 51.08%, alongside its success rate of 17.73%, illustrates the challenge of moving from recovering a substantial fraction of the targets to reliably completing the entire search objective.

Second, MOST-Nav provides a competitive reference baseline with simpler sensing requirements. Training-free baselines such as 3D-Mem and SCOPE use depth observations and camera poses for geometric reconstruction. MOST-Nav instead estimates poses and constructs interactive 3D memory directly from RGB using VGGT-SLAM. Using Qwen3.8-Flash across the training-free methods and evaluating on the same MOST-Bench episodes, MOST-Nav achieves the highest overall SR and F1 under the evaluated configurations despite requiring fewer sensor modalities for reconstruction. These results support the effectiveness of the complete framework and its value as an accessible starting point for future development, while leaving substantial room to improve autonomous search and reliable task completion.

Thank you for the detailed reply. The task proposed in the paper is valuable and interesting. However, as a benchmark paper, its overall scale and baseline results are very limited. It also makes little contribution at the algorithm level. I still think it is difficult to meet the ICLR acceptance standard. Therefore I will keep my score.

Thank you for your follow-up and for recognizing the value of the proposed tasks. We appreciate your concerns about benchmark scale and algorithmic novelty, and would like to clarify our contributions.

At the problem level, ManyON and AllON introduce search objectives that have received limited attention in existing navigation benchmarks. Unlike prescribed sequential goals in GOAT-Bench [1], ManyON requires agents to select and visit a specified number of distinct matching instances, while AllON requires exhaustive search with an unknown target count and an explicit completion decision. These settings expose a distinct challenge: finding targets does not necessarily imply knowing when the search is complete.

At the method level, MOST-Nav integrates RGB-based VGGT-SLAM reconstruction [2] with interactive 3D memory, selective RGB retrieval, visual instance deduplication, and spatial evidence for termination. Compared with training-free methods such as 3D-Mem [3] and SCOPE [4], it constructs geometric memory without depth observations or simulator-provided camera poses. Our ablations demonstrate complementary contributions to coverage, reporting precision, and completion judgment. We view this as a useful methodological integration accompanying the benchmark contribution.

We acknowledge the current evaluation scale and remaining performance limitations. At the same time, we would like to emphasize that we systematically evaluate representative mainstream training-based and training-free navigation methods, and that MOST-Nav achieves higher overall SR and F1 under the evaluated configurations while requiring fewer sensor inputs for geometric reconstruction. Your feedback has helped us clarify these boundaries, and we sincerely appreciate your time and consideration.

References

[1] Khanna, M., et al. (2024). GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigation.

[2] Maggio, D., Lim, H., and Carlone, L. (2025). VGGT-SLAM: Dense RGB SLAM Optimized on the SL(4) Manifold. NeurIPS.

[3] Yang, Y., et al. (2025). 3D-Mem: 3D Scene Memory for Embodied Exploration and Reasoning. CVPR.

[4] Wang, N., et al. (2026). Expand Your SCOPE: Semantic Cognition over Potential-Based Exploration for Embodied Visual Navigation. AAAI.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Author Response (1/3): Scientific Novelty and Fairness of Baseline Comparisons

We appreciate your thoughtful feedback. In the revised PDF, shared changes are blue and yours are teal.

1. Scientific Novelty of MOST-Bench and Its Distinction from Existing Navigation Benchmarks (w5/q1)

MOST-Bench studies autonomous multi-object search beyond executing a prescribed sequence of navigation goals. For example, one GOAT-Bench episode asks an agent to sequentially navigate to a recliner chair specified by category, an oven specified by an image, and “the white book on the coffee table in the living room” specified by language. Although this setting evaluates multimodal navigation and memory reuse over long horizons, the benchmark supplies each successive goal and prescribes their order.

MOST-Bench instead gives the agent a set-level search objective. ManyON requires finding a specified number of distinct matching instances, allowing the agent to choose their visit order and, when additional candidates exist, which instances to visit. AllON requires finding every matching instance without revealing the target count, followed by an explicit completion decision. The scientific question is how an agent uses accumulated observations to select targets, track distinct discoveries, and determine whether further search is necessary under partial observability. These decisions about which instances to visit and when to stop introduce responsibilities and success criteria that are not captured by simply extending a prescribed sequence of single-goal navigation tasks.

2. Fairness of Baseline Comparisons: Interface Adaptation, Sensor Inputs, and Budget Alignment (w1/w4)

We agree that these factors should be accounted for when interpreting the baseline comparisons.

Interface adaptation. Evaluating existing methods on this search problem requires adapting their task interfaces. We preserve their core algorithmic frameworks while adding support for multi-object reporting and task termination. The adapted baselines provide reference points for how publicly available methods perform under these adaptations. We also provide an explicitly defined Habitat-based evaluation interface to support methods designed directly for autonomous multi-object search.

Sensor inputs. The training-free baselines, including 3D-Mem and SCOPE, use depth observations and camera poses to construct and update their geometric representations. MOST-Nav uses VGGT-SLAM to estimate camera poses and reconstruct interactive 3D memory directly from the RGB stream. To reduce variation attributable to the decision-making backbone, we use Qwen3.8-Flash as the shared VLM across the training-free methods and evaluate them on the same MOST-Bench episodes constructed from HM3D validation scenes. Under the evaluated configurations, MOST-Nav achieves higher overall SR and F1 while requiring fewer sensor modalities for reconstruction. These results support the effectiveness of the complete VGGT-based navigation and memory framework.

Interaction budgets. We use 80 native navigation iterations for 3D-Mem and SCOPE, following their GOAT-Bench evaluation convention. Each iteration can execute a trajectory segment rather than an individual simulator action. NaVid, UniNaVid, InternVLA-N1, and MTU3D count their budgets in discrete Habitat interaction steps and use an 800-step limit in the extended-budget evaluations. MOST-Nav uses the same action granularity and step-counting unit as these methods, so we also assign it an 800-step budget.

To compare these budgets in physical terms, we measured the travel distance of every budget-exhausted episode for 3D-Mem, SCOPE, MTU3D, and MOST-Nav. Distance is computed by summing the Euclidean distances between consecutive recorded positions. For 3D-Mem and SCOPE, these positions are segment endpoints; because the agents follow Habitat pathfinder paths within each segment, their measurements are lower bounds on actual travel.

Table R1. Travel distance on each method’s budget-exhausted episodes.

MethodBudget capBudget-exhausted episodesMean distance (m)Median distance (m)
3D-Mem80 navigation iterations311≥ 60.9≥ 70.9
SCOPE80 navigation iterations311≥ 69.0≥ 75.6
MTU3D800 primitive steps38765.176.4
MOST-Nav800 primitive steps42278.880.3

The tenfold difference in budget counts does not correspond to a tenfold difference in physical travel. Even the conservative median distances for 3D-Mem and SCOPE reach 70.9 m and 75.6 m, respectively, compared with 80.3 m for MOST-Nav. Actual travel also reflects each method’s navigation behavior and allocation of actions: under the same 800-step cap, MTU3D travels 65.1 m on average, compared with 78.8 m for MOST-Nav. These measurements show that the budgets support travel on a similar physical scale and provide empirical support for their practical comparability.

AuthorsOpenReview ↗

Author Response (2/3): Statistical Reliability, Scene Generalization, and Reachability Filtering

3. Statistical Reliability and Scene-Level Generalization (w2)

Thank you for this suggestion. The 36-scene evaluation follows the official HM3DSem split, also used by GOAT-Bench, and supports fair comparison on held-out environments. All methods use the same 750 episodes for matched task comparisons. Our complete automated construction pipeline is publicly available and can generate tasks in the 145 semantically annotated HM3D training scenes, as well as other datasets with semantic annotations and navigation meshes.

We assess statistical uncertainty using 3,000 scene-level bootstrap resamples and leave-one-scene-out (LOSO) analysis, accounting for dependence among episodes within each scene. Table R2 reports 95% confidence intervals. MOST-Nav ranks first in both SR and F1 on the full AllON set in every bootstrap resample and all 36 LOSO splits. Removing any single scene leaves its SR within 16.4–19.1% and F1 within 46.1–47.9%. Its advantage is particularly consistent for attribute-based exhaustive search, with 51.3% F1 versus 20.2–26.5% for the baselines and first-place rankings in every resample and LOSO split.

These results show that the main comparative findings remain stable under changes in scene composition within HM3D. Moreover, MOST-Nav operates in a training-free, zero-shot setting without benchmark-specific training or fine-tuning, while the released construction pipeline supports further cross-dataset evaluation.

Table R2. Scene sensitivity of AllON results (%; brackets indicate 95% scene-bootstrap confidence intervals). The last column reports MOST-Nav’s first-place frequency across bootstrap resamples / LOSO splits.

SubsetMetric3D-MemSCOPEMOST-NavMOST-Nav ranks first
CategorySR8.7 [3.4, 14.6]10.0 [4.8, 15.6]16.0 [8.8, 24.2]93.1% / 35 of 35
F130.7 [26.6, 35.0]35.4 [31.0, 39.4]42.7 [36.6, 48.5]99.4% / 35 of 35
AttributeSR4.0 [1.1, 7.5]3.3 [0.7, 6.7]20.7 [13.5, 28.3]100% / 35 of 35
F120.2 [16.8, 23.9]26.5 [21.6, 31.5]51.3 [45.7, 56.6]100% / 35 of 35
AllSR6.3 [2.9, 10.3]6.7 [3.4, 10.2]18.3 [12.2, 24.6]100% / 36 of 36
F125.4 [22.5, 28.3]30.9 [27.4, 34.7]47.0 [42.4, 51.7]100% / 36 of 36

4. Impact and Sensitivity of Reachability Filtering (w3)

Thank you for this suggestion. The filter targets physical reachability only, excluding scene–category pairs containing an instance with no valid navigation point on the navmesh. It does not filter by visibility. Instances that are occluded, located deep inside rooms, or visible only after extensive exploration are retained, preserving the partial-observability challenge in AllON. This filtering is necessary because AllON requires reaching all matching instances; an unreachable target would make success unattainable.

The filter affects very few cases. It is applied once to raw scene–category pairs, before attribute derivation and query composition. Of the 651 raw pairs containing at least two instances, only 3 (0.46%) were removed because an instance lacked a valid navigation point. Attribute and composite queries are derived from retained pairs, so downstream exclusions are limited to queries that would otherwise depend on these three removed pairs.

Exhaustive search remains challenging after filtering. Although every retained target is reachable by construction, the three strongest methods achieve only 52.7–56.1% AllON recall, leaving roughly half of the targets undiscovered on average. Most of their episodes (58.3–71.3%) end with partial coverage. Across increasing target-count tiers, MOST-Nav’s AllON SR decreases from 30.0% to 15.0% and 10.0%, while recall remains within 52.5–55.2%. These results show that ensuring reachability does not remove the difficulty of finding every matching instance under partial observability—the challenge AllON is designed to evaluate.

AuthorsOpenReview ↗

Author Response (3/3): Diagnostic Evaluation and Component Ablations

5. Disentangling Target Selection, Visit Ordering, and Termination Judgment (q2)

Thank you for this suggestion. MOST-Bench provides separate diagnostics for target selection, visit ordering, and termination judgment, enabling a more detailed interpretation of end-to-end performance.

Target selection. The benchmark includes 180 ManyON episodes with more matching candidates than required targets, allowing agents to choose which target combination to visit. We compare the route cost of the selected combination with that of the optimal combination within the discovered candidate pool. MOST-Nav achieves a mean selection efficiency of 0.784 and a median of 1.0.

Visit ordering. Ordering efficiency (OE) fixes the selected target set and compares the shortest route under optimal ordering with the shortest route following the agent’s observed order. It is neither multiplied by a success indicator nor conditioned on FINISH, allowing it to be measured separately from final task success.

Table R3. Ordering efficiency with 95% confidence intervals.

MethodMean OE [95% CI]
MOST-Nav0.574 [0.38, 0.77]
3D-Mem0.688 [0.50, 0.85]
SCOPE0.760 [0.58, 0.92]
MTU3D0.714 [0.50, 0.89]
NaVid0.713 [0.50, 0.90]
UniNaVid0.680 [0.50, 0.84]
InternVLA-N10.738 [0.51, 0.94]

OE is a conditional process statistic, averaged over each method’s episodes with at least two discovered targets and valid route data.

Termination judgment. We separately report full coverage and FINISH conditional on full coverage, distinguishing “finding every target” from “recognizing that the search is complete.” For the current AllON records of 3D-Mem and SCOPE, SR decomposes as:

SR = Full-coverage rate × P(FINISH | Full coverage)

Their full-coverage rates are 21.3% and 22.0%, while their FINISH rates conditional on full coverage are 29.7% and 30.3%, yielding SRs of 6.3% and 6.7%, respectively. This decomposition separates the contributions of coverage and termination judgment to task success.

Together, these diagnostics help identify the source of improvements without attributing every SR gain to stronger search ability. MOST-Nav’s component contributions are further examined through the ablations below.

6. Contributions of MOST-Nav’s Core Components and Ablation Analysis (q3)

We evaluated three ablations on the same 108 episodes, sampling one episode per difficulty tier from each of the 36 scenes.

Table R4. Ablation results (% unless reported as counts).

MetricFullw/o RGB memory retrievalw/o visual deduplicationw/o spatial termination evidence
SR ↑13.94.613.01.9
Precision ↑57.551.046.853.1
Recall ↑42.632.639.039.3
Duplicate-report rate ↓6.96.827.011.3
Full-coverage episodes (count) ↑1561411
FINISH rate given full coverage ↑100.083.3100.018.2

Precision and recall are episode-level averages. The duplicate-report rate is the number of duplicate false positives divided by the total number of reports.

The components make distinct contributions. Removing RGB memory retrieval reduces recall by 10.0 percentage points. Removing visual deduplication increases the duplicate-report rate from 6.9% to 27.0% and lowers precision by 10.7 percentage points. Without spatial termination evidence, 9 of 11 full-coverage episodes fail to issue FINISH, reducing SR to 1.9%. These ablations demonstrate complementary benefits for target coverage, report accuracy, and completion signaling, and show why SR should be interpreted alongside these diagnostics.

Thank you again for your thoughtful feedback. We briefly summarize your main concerns and our responses below:

  • Novelty: We clarified that MOST-Bench studies autonomous target selection, visit ordering, and completion judgment, extending beyond navigation through prescribed goal sequences.
  • Comparison fairness: We detailed interface adaptations, sensor inputs, and budget conventions, and added travel-distance measurements to contextualize the comparisons. We interpret the results as comparisons of complete systems.
  • Statistical reliability: We added scene-level bootstrap confidence intervals and leave-one-scene-out analysis, showing stable AllON rankings within HM3D.
  • Reachability filtering: We clarified that filtering removes physically unreachable targets while retaining occluded ones, affects only 0.46% of raw scene–category pairs, and preserves substantial exhaustive-search difficulty.
  • Diagnostics and component contributions: We added separate analyses of target selection, ordering, and termination, alongside ablations demonstrating the contributions of memory retrieval, deduplication, and stopping evidence.

We hope these clarifications and additional analyses address your concerns. Are there any remaining questions or points you would like us to clarify further? We would be happy to provide additional explanations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.