- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MOST-Bench, a benchmark for autonomous multi-object search. Agents must choose which target instances to visit, in what order, and when the search is complete. In ManyON the agent must find and report a given number of distinct instances with no prescribed visit order. In AllON the target count is hidden, and the agent must find every instance that satisfies the goal and then issue FINISH. The benchmark has 750 episodes over 36 HM3D scenes with category and attribute goals, plus image goals in ManyON. The paper also proposes MOST-Nav, a training-free reference baseline in which a VLM makes decisions over a 3D memory built from RGB observations with VGGT-SLAM. Seven methods are compared on task completion, reporting accuracy, and target coverage.
Critical strengths
- The problem setting is a useful one. Separating ManyON, where the required count is given, from AllON, where it is hidden, lets the benchmark test instance selection, visit order, and progress tracking on one side and the decision to stop an exhaustive search on the other.
- The evaluation covers both trained policies and training-free agents, and reports results by goal modality and by target count. Reporting precision, recall, F1, and full coverage next to SR makes the gap between finding targets and completing the task visible.
- MOST-Nav is a concrete reference baseline. It lets the VLM query a spatial representation built from RGB observations, retrieve past frames, ground a chosen target in 3D, and check whether two candidates are the same instance, all inside its decision loop.
Critical weaknesses
W1. The baseline adapters are not validated enough to interpret the AllON differences. On AllON, the full coverage rates of 3D-Mem, SCOPE, and MOST-Nav are 21.3%, 22.0%, and 18.3%. Among episodes with full coverage, roughly 30%, 30%, and 100% also end with FINISH (Table 2 and Figure 6(b)). Termination therefore carries much of the SR comparison, so the reader needs a way to check that the reporting and termination mechanisms added to the existing methods work as intended. The released 3D-Mem adapter depends on external code both to associate report history with current candidate IDs and to handle FINISH, and the version used in the experiments is not available for inspection. The SCOPE adapter is not provided at all. If previous reports cannot be reliably matched to current candidates, duplicate reports and unnecessary revisits would affect precision and could also decide whether the agent terminates within its budget. The current results do not separate these possible adapter effects from the exploration and termination abilities of the underlying methods.
W2. The action budgets have not been shown to be comparable. 3D-Mem and SCOPE get 80 native steps, while MOST-Nav gets 800 primitive actions. Keeping the budget ratio from earlier GOAT evaluations does not by itself give the methods equal room to explore. The released 3D-Mem adapter documentation specifies at most 1 m of movement per native step, and turns, extra observations, and reports are counted differently from the primitive action protocol. In AllON the agent may keep exploring after it has found every target in order to decide whether to stop, so the budget matters there in particular. A comparison under a shared travel-distance limit, or results across several budgets, would show how sensitive the conclusions are to this setting.
W3. The experiments do not isolate the contributions of MOST-Nav’s components. MOST-Nav combines interactive 3D memory, selective RGB retrieval with target localization, visual duplicate checking, and termination decisions based on spatial evidence. No ablation separates these parts. The evidence that the proposed memory design is what improves AllON SR and precision is therefore thin.
W4.** The cost of MOST-Nav’s VLM-memory interaction and online reconstruction is not reported.** Section 4.2 and Figure 3 allow up to 15 rounds of memory queries per decision, but the paper gives no average number of rounds, no VLM call or token counts per episode, and no runtime. It also does not report memory use for online VGGT-SLAM reconstruction or for storing RGB observations and geometry. Without these numbers it is hard to judge what the reported performance costs during a long search.
Questions for authors
- Please report the Repeated Target Selection Rate (RTSR) of 3D-Mem, SCOPE, and MOST-Nav on AllON, with its exact definition, and break invalid reports down into duplicates and other errors. How are object IDs from previous reports matched to current candidates in the 3D-Mem and SCOPE adapters used in the experiments?
- If 3D-Mem and SCOPE have been run with larger step budgets, how do AllON SR, full coverage, and the share of full-coverage episodes that end with FINISH change?
- How much do interactive 3D memory retrieval, visual duplicate checking, and the spatial information used for termination each contribute to AllON SR and precision? Ablations that remove or replace these components under the same exploration and reporting settings would help.
9 comments
Author Response (2/3)
2. Comparability of Action Budgets and Their Effects on AllON Coverage and Success Rates (W2/Q2)
Thank you for raising this concern. MOST-Nav shares the same primitive-action output interface and 800-action budget as MTU3D. For 3D-Mem and SCOPE, we follow their native GOAT evaluation convention of 80 navigation iterations, each of which can execute a trajectory segment. The 1 m per-step movement limit described in our released 3D-Mem adapter documentation is inherited from the original 3D-Mem settings (https://github.com/UMass-Embodied-AGI/3D-Mem.git) rather than introduced by our adaptation. To compare these action granularities beyond nominal step counts, we measured travel distance for all budget-exhausted episodes in the evaluation set.
Table R1. Travel distance on method-specific subsets of budget-exhausted episodes.
| Method | Budget cap | Episodes | Mean distance (m) | Median distance (m) |
|---|---|---|---|---|
| 3D-Mem | 80 navigation iterations | 311 | ≥ 60.9 | ≥ 70.9 |
| SCOPE | 80 navigation iterations | 311 | ≥ 69.0 | ≥ 75.6 |
| MTU3D | 800 primitive actions | 387 | 65.1 | 76.4 |
| MOST-Nav | 800 primitive actions | 422 | 78.8 | 80.3 |
The nominal tenfold budget difference does not translate into a tenfold difference in physical travel. Even the conservative median distances for 3D-Mem and SCOPE reach 70.9 m and 75.6 m, respectively, compared with 80.3 m for MOST-Nav. These observations provide empirical context for the chosen settings: the methods operate on a similar physical travel scale under their native interfaces.
We further examined whether failures to terminate occurred because full coverage was reached immediately before budget exhaustion. The AllON traces show that this is often not the case: 3D-Mem and SCOPE achieve full coverage in 64 and 66 episodes, respectively, but issue FINISH in only 19 and 20 of them.
Table R2. AllON coverage and termination diagnostics. Post-coverage steps are measured in native navigation iterations among full-coverage episodes without FINISH.
| Method | AllON SR | Full coverage | FINISH given full coverage | Median post-coverage steps without FINISH |
|---|---|---|---|---|
| 3D-Mem | 6.30% | 21.3% (64/300) | 29.7% (19/64) | 50 |
| SCOPE | 6.70% | 22.0% (66/300) | 30.3% (20/66) | 44 |
These agents frequently continue for substantial periods after finding every target without recognizing completion. Among full-coverage episodes without FINISH, 82.2% for 3D-Mem and 78.3% for SCOPE continue for at least 20 additional native navigation iterations. Together with the travel-distance analysis, these results indicate that the observed gap between success rate and full coverage involves a termination bottleneck that cannot be explained solely by the nominal difference in budget caps. This reflects a central challenge evaluated by our benchmark: which instances to visit and when to stop.
Author Response (3/3)
3. Ablations: Memory Retrieval, Visual Deduplication, and Spatial Termination (W3/Q3)
We evaluated three ablations on the same 108 episodes, sampling one episode per difficulty tier from each of the 36 scenes.
Table R3. Ablation results (% unless reported as counts).
| Metric | Full | w/o RGB memory retrieval | w/o visual deduplication | w/o spatial termination mechanism |
|---|---|---|---|---|
| SR ↑ | 13.9 | 4.6 | 13.0 | 1.9 |
| Precision ↑ | 57.5 | 51.0 | 46.8 | 53.1 |
| Recall ↑ | 42.6 | 32.6 | 39.0 | 39.3 |
| Duplicate-report rate ↓ | 6.9 | 6.8 | 27.0 | 11.3 |
| Full-coverage episodes (count) ↑ | 15 | 6 | 14 | 11 |
FINISH rate given full coverage ↑ | 100.0 | 83.3 | 100.0 | 18.2 |
FINISH rate given incomplete coverage ↓ | 36.6 | 34.3 | 42.6 | 14.4 |
Each component addresses a distinct failure mode. Removing RGB memory retrieval reduces recall by 10.0 percentage points. Removing visual deduplication increases the duplicate-report rate from 6.9% to 27.0% and reduces precision by 10.7 percentage points. Without the spatial termination mechanism, 9 of the 11 full-coverage episodes fail to issue FINISH, reducing SR to 1.9%.
The spatial termination mechanism also increases the FINISH rate among episodes with incomplete coverage from 14.4% to 36.6%, revealing a trade-off between completion signaling and premature termination. These ablations demonstrate complementary benefits for target coverage, reporting accuracy, and completion signaling, and show why SR should be interpreted alongside these diagnostics.
4. Computational Costs: VLM Calls, Runtime, and Memory Usage (W4)
We provide measurements of VLM interactions, runtime, memory usage, and storage below. The limit of 15 memory-query rounds is a maximum interaction budget; in practice, the measured workload averages 1.92 VLM calls per decision on AllON and 2.05 on ManyON.
Table R4. MOST-Nav’s per-episode computational costs. Values are means; GPU figures are averages of per-episode peak allocated and reserved memory.
| Metric | AllON | ManyON |
|---|---|---|
| End-to-end runtime (min) | 58.2 | 50.5 |
| Decision cycles | 89.5 | 75.8 |
| Decision VLM calls | 170 | 153 |
| VLM calls per decision | 1.92 | 2.05 |
| Input / output tokens | 2.30M / 10.3k | 2.21M / 9.1k |
| Stored RGB frames | 557 | 486 |
| RGB disk storage (GB) | 0.82 | 0.65 |
| Local server peak GPU memory, allocated / reserved (GB) | 9.2 / 10.0 | 9.1 / 9.9 |
| Server RAM, RSS (GB) | 9.5 | 8.5 |
| Geometry within server RSS (GB) | 5.0 | 4.3 |
| Agent RAM, RSS (GB) | 4.8 | 3.8 |
Online reconstruction accounts for a measurable but relatively small share of runtime. An AllON episode builds an average of 35.7 VGGT submaps, taking 2.5 minutes including optimization, or approximately 4% of total runtime. Decision VLM latency totals 6.6 minutes (11%), server-side captioning takes 2.7 minutes (5%), and SAM segmentation takes 0.85 minutes (1.5%). Approximately 76% of runtime is attributable to the remaining pipeline and execution overhead, including simulation, frame transfer, mapping and planning, logging, and resource contention during concurrent execution.
These wall-clock costs reflect the full evaluation setup. Stored RGB observations consume disk space, while geometry accounts for approximately half of server RAM usage. GPU measurements cover the local reconstruction and segmentation server.
Author Response (1/3)
We thank the reviewer for the detailed comments and provide our responses below. We have revised the manuscript and prepared an updated PDF, with changes addressing shared concerns highlighted in blue and those specific to your comments in burnt orange.
1. Baseline Adapter Validation: Instance Matching, Duplicate Reports, and Termination (W1/Q1)
Thank you for highlighting the importance of validating the baseline adapters. We have added analyses of repeated reports, error breakdowns, and
FINISHevents on AllON, distinguishing benchmark-level duplicate reporting from each method’s internal target selection.Evaluation protocol and RTSR. Report correctness is determined solely by the evaluator by matching the reporting position against ground-truth (GT) instances, independently of adapter-side ID association. For each episode, let ci denote the number of times GT target i is matched by a report, and let qi denote its final quota (qi=1 by default; qi=n when n targets are jointly covered by a single report). We define the Repeated Target Selection Rate (RTSR) as:
RTSR=∑ici∑imax(0,ci−qi),
with RTSR=0 if ∑ici=0. For example, reporting target A five times and target B once yields (5−1)/(5+1)=4/6. We macro-average RTSR across the 300 AllON category/attribute episodes. Using the quota at reporting time, we further partition false positives (FPs) into duplicate FPs (re-reports exceeding the quota) and other FPs (reports with no nearby target, excluded from RTSR).
Event-level adapter checks. For every report, the adapter’s runtime judgment agrees with the evaluator’s scoring. All explicit
FINISHevents—57/300 episodes for 3D-Mem and 70/300 for SCOPE—agree with the episode-end records, confirming that the termination interface is reachable and correctly recorded. We traced each duplicate report in 3D-Mem and SCOPE to its source and found two cases. In the first, the method re-selects a native object ID that is already listed in the report history shown to the model. Because the matching ID was explicitly available, these duplicates do not stem from a failure to associate past reports with current candidates. In the second, the same physical instance reappears under a new native ID, reflecting instance fragmentation within the method's own memory. In both cases, the duplicates originate from the methods' native target selection and instance tracking rather than from the adapter interface.Report-history matching. 3D-Mem and SCOPE share a unified adapter that converts a MOST-Bench episode into sequential search rounds with a shared episode state and step budget, leaving their native navigation, mapping, and planning unchanged. Scene memory persists across rounds. After each report, the selected candidate’s native object ID, category, and reporting position are appended to the report history and included in subsequent prompts for candidate comparison. To prevent GT leakage, this history omits correctness labels, duplicate flags, and remaining target counts. Previously reported IDs are not forcibly masked. The model autonomously issues
FINISHbased on its observations, memory, and report history.MOST-Nav’s termination mechanism. When VGGT-SLAM detects no frontiers for an extended period at a high step count, MOST-Nav prompts the VLM to synthesize accumulated spatial evidence and decide whether to issue
FINISH. Its highFINISHrate at full coverage relies on explicitly modeling explored regions and detecting unexplored areas. This mechanism can also prompt premature termination before all targets are found; Point 3 quantifies this effect through an ablation. The triggers use only agent-observable states, without GT leakage, and the final decision remains with the VLM.