acceptodds
Under review as a conference paper at ICLR 2027

Situating Memory for Embodied Navigation

Abstract

Memory-based navigation requires an agent to relate stored environmental knowledge to its current observation. When navigation fails, however, task success alone cannot distinguish missing knowledge from errors in using it. We introduce SCOPE, a controlled evaluation framework for studying this distinction across 128 navigation tasks in 64 indoor scenes. We vary frozen scene memory and online episode memory while holding the navigation policy and execution pipeline fixed, and then replace native scene memory with audited ground-truth scene knowledge. The strongest ground-truth condition succeeds on 93 of 128 tasks. Among its 35 residual failures, trace-based audits attribute 24 to current localization, goal localization, or egocentric correspondence. One-shot corrections at diagnosed checkpoints recover 15 episodes, compared with five under matched sham interventions, providing behavioral evidence that the identified errors are consequential. Motivated by these findings, we introduce Experience Skeleton Memory (ESM), which uses simulator-provided poses to organize executed trajectories and express historical observations relative to the current view. With the same ground-truth scene memory, ESM achieves 85.94% success on the full suite, compared with 72.66% for the original structured episode memory. Together, these results identify situated correspondence between memory and current observations as a key navigation bottleneck and show that explicitly maintaining this correspondence can substantially improve closed-loop navigation under accurate pose information.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied Navigation, Multimodal Agents, Spatial Memory, Episode Memory, Spatial Grounding, Embodied AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dyopOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SCOPE examines navigation failures that persist despite accurate scene knowledge. Ground-truth memory replacements and checkpoint interventions distinguish missing knowledge from failures to relate that knowledge to the current situation. Experience Skeleton Memory (ESM) then combines executed experience with simulator geometry. On 128 tasks, targeted corrections recover ten more episodes than sham interventions, and ESM raises success from 93 to 110 episodes.

Critical strengths

S1. Interventions support the diagnosis. Targeted corrections are compared with both unmodified replay and same-channel sham interventions (Sections 4.2-4.3; Appendix F.4). The ten additional recoveries over sham provide evidence that some failures concern the use of available knowledge.

S2. The accounting is transparent. The paper reports paired outcomes, scene-clustered uncertainty and ESM's five regressions alongside its 22 recoveries (Tables 1-3). It also states its reliance on privileged geometry and avoids treating verbal rationales as direct evidence of internal reasoning.

Critical weaknesses

W1. ESM's design-specific contribution is unresolved. ESM adds pose-derived directions, depth-based associations, route history and revisit judgments together (Section 4.4; Appendix G). Frame and TCoT lack the same correspondence information. The gain supports this combined intervention, but does not show what the skeleton representation contributes beyond the added geometric information.

W2. The remedy is evaluated on the failures that motivated it. The full-suite test includes the same 35 failures used to develop ESM (Section 5.5; Appendix E.6). Counting the other 93 tasks measures regressions but supplies no independent test of the remedy. The second-policy study repeats memory replacement, not the diagnosis-to-ESM chain.

Assessment. I recommend weak rejection (2/4). The controlled diagnosis is useful, but the evidence leaves both ESM's distinct contribution and the independence of its validation unresolved.

Questions for authors

Q1 (W1). Which reported comparison holds pose, depth and camera correspondence fixed while changing the memory representation? If none does, please distinguish the demonstrated benefit of the combined geometric intervention from a benefit specific to ESM's skeleton design.

Q2 (W2). Which reported ESM results were obtained after its interface and parameters were frozen, on tasks not used during diagnosis or development? Please also state the rule used to select the executions in Appendix H when multiple attempts existed.

3 comments
AuthorsOpenReview ↗

Response to Reviewer dyop (1/3): Same geometry, different memory organization

W1/Q1: "ESM adds pose-derived directions, depth-based associations, route history and revisit judgments together … The gain supports this combined intervention, but does not show what the skeleton representation contributes beyond the added geometric information." "Which reported comparison holds pose, depth and camera correspondence fixed while changing the memory representation?"

A1: We thank the reviewer for this precise question. Table R1 is exactly this comparison: pose, depth, and camera are identical, and only the memory organization changes. Following the suggestion, we removed ESM's components one at a time (including aggregating observation points into skeleton nodes) and re-implemented Mem2Ego, which also uses pose. As shown in Table R1, with identical required environment information ESM is 11.72 pp above Mem2Ego (95% CI [+3.12, +20.31], excluding 0); removing any component lowers SR, and removing more lowers it further; observation-point memory alone (73.44%) is only 3.13 pp above the pose-free GT_JOINT (70.31%). Individual CIs are wide, but every removal points the same way. With identical required environment information, the gain comes from ESM's organization, not from more geometric information.

Table R1: Results with identical required environment information (pose, depth, and camera), varying only memory organization (128 tasks). Components: observation-point memory (each observation bound to where it was made), aggregation (repeatedly visited points merged into skeleton nodes), correction (RELABEL), revisit reasoning (direction-by-direction reasoning about what has been explored; backtracking must be justified). GT_JOINT uses no pose. For comparability with the new conditions, ESM and GT_JOINT use concurrent re-runs under the frozen configuration (manuscript runs: GT_JOINT 72.66%, ESM 85.94%). 95% CIs are scene-clustered paired bootstrap.

ConditionPose / depth / cameraObs.-point memoryAggregationCorrectionRevisit reasoningSR ↑SPL ↑ΔSR vs. ESM [95% CI]
ESM (baseline)✓✓✓✓✓82.030.640–
ESM w/o revisit reasoning✓✓✓✓✗77.340.598−4.69 [−14.84, +5.47]
ESM w/o correction✓✓✓✗✓78.120.619−3.91 [−12.50, +4.69]
ESM w/o aggregation✓✓✗✓✓75.000.573−7.03 [−17.19, +3.12]
Obs.-point memory only✓✓✗✗✓73.440.545−8.59 [−17.97, +0.78]
Mem2Ego✓––––70.310.518−11.72 [−20.31, −3.12]
GT_JOINT✗✗✗✗✗70.310.549−11.72 [−20.31, −3.91]
AuthorsOpenReview ↗

Response to Reviewer dyop (2/3): Breakdown by task family and failure type

A1 (cont.): We further break down the same runs by task family and failure category. As shown in Table R2, the gap concentrates on the multi-stage and disambiguation families: on multi-stage tasks ESM reaches 88.89% versus 55.56% for Mem2Ego, and removing any component lowers it; ESM has the highest SPL in all four families. As shown in Table R3, attributing each method's own failures, ESM reduces failures in the first three categories from 29 to 13, while other failures stay about the same. With the same geometric information, a different organization changes the outcome exactly on the diagnosed failures.

Table R2: SR (%) / SPL by task family; number of tasks in parentheses; bold marks where ESM is highest in the column (incl. ties).

ConditionDirect (37)Disambiguation (42)Search & collection (31)Multi-stage (18)
ESM81.08 / 0.64785.71 / 0.65774.19 / 0.61788.89 / 0.625
ESM w/o revisit reasoning78.38 / 0.58876.19 / 0.59977.42 / 0.60977.78 / 0.600
ESM w/o correction81.08 / 0.62276.19 / 0.63174.19 / 0.61183.33 / 0.599
ESM w/o aggregation75.68 / 0.57273.81 / 0.58170.97 / 0.56183.33 / 0.580
Obs.-point memory only72.97 / 0.54873.81 / 0.55877.42 / 0.53966.67 / 0.521
Mem2Ego75.68 / 0.56666.67 / 0.51577.42 / 0.54755.56 / 0.378
GT_JOINT72.97 / 0.62878.57 / 0.56067.74 / 0.53650.00 / 0.383

Table R3: Attribution of each method's own failures by the manuscript's failure categories (number of failures among 128 tasks, with the share in parentheses; annotators blind to the method). The first three categories are all failures to relate memory to the current situation.

Failure categoryMem2Ego (baseline)ESM
Current Localization11 (8.59%)4 (3.13%)
Goal Localization2 (1.56%)2 (1.56%)
Egocentric Correspondence16 (12.50%)7 (5.47%)
Other9 (7.03%)10 (7.81%)
First three combined29 (22.66%)13 (10.16%)
All failures38 (29.69%)23 (17.97%)
AuthorsOpenReview ↗

Response to Reviewer dyop (3/3): Independent validation and run selection

W2/Q2: "The full-suite test includes the same 35 failures used to develop ESM … Which reported ESM results were obtained after its interface and parameters were frozen, on tasks not used during diagnosis or development? Please also state the rule used to select the executions in Appendix H when multiple attempts existed."

A2: We thank the reviewer for the careful reading and constructive comments, and we agree: ESM's design was motivated by the diagnosis of these 35 failures; mindful of independence, we developed and debugged it on separate development scenes, but the manuscript's evaluation still included these 35 tasks. We therefore add the following.

(1) Frozen ESM's gain is significant on new scenes, GOAT-Bench, and a second policy. Following the suggestion, we froze ESM and evaluated it on new scenes and GOAT-Bench, neither used in diagnosis or development, and also ran a second policy, GPT-6 Luna, on the new scenes; unlike the manuscript's second-policy study, which only replaced memory, this runs the full ESM. As shown in Table R4, ESM improves on GT_JOINT by 12.50, 8.40, and 10.94 pp, all with CIs excluding 0, consistent with the manuscript.

(2) No run was selected by outcome: each task–condition pair was scored once. Development scenes are fully separate from evaluation scenes, and development runs are never selected; in the formal evaluation nothing was re-run: 89 task–condition pairs interrupted by execution-framework issues (mainly failed network requests) were resumed from the interruption point after the fix, all in the manuscript's baseline conditions M0–M3, none involving ESM or GT_JOINT. In Appendix H, "selected" denotes the single formal run of each task–condition pair; ESM's full-set result consists of its 35 residual-set runs and 93 further runs, all with the same code. After freezing, a full re-run keeps the gain significant (Table R5).

We will add the new results from all our replies to the final version.

Table R4: Frozen ESM on tasks and policies not used in diagnosis or development. New scenes: 64 tasks in 32 new HM3D scenes, disjoint from the original 64, built with the original pipeline and family proportions and fixed before any ESM run; GOAT-Bench: 8 val-unseen scenes, 16 episodes, 119 subtasks. ΔSR is relative to GT_JOINT; 95% CIs are scene-clustered paired bootstrap.

Task setPolicyGT_JOINT SRESM SRΔSR vs. GT_JOINT [95% CI]GT_JOINT SPLESM SPLESM-only / GT_JOINT-only successes
New scenesoriginal62.5075.00+12.50 [+1.56, +23.44]0.5030.59610 / 2
GOAT-Benchoriginal67.2375.63+8.40 [+1.56, +17.39]0.5630.58312 / 2
New scenesGPT-6 Luna50.0060.94+10.94 [+3.12, +18.75]0.3390.41810 / 3

Table R5: ESM in the manuscript run and in the independent re-run under the frozen configuration (128 tasks). 21 tasks changed outcome between runs (8 solved only in the re-run, 13 only in the manuscript run), reflecting run-to-run variability of the model calls; GT_JOINT, re-run under the same configuration, gets 90/128 (70.31%; manuscript 93/128); each ΔSR is relative to GT_JOINT from the same run; all new comparisons use the re-run.

RunESM successesESM SRESM SPLΔSR vs. GT_JOINT [95% CI]
Manuscript110/12885.940.612+13.28 [+6.25, +21.09]
Independent re-run105/12882.030.640+11.72 [+3.91, +20.31]
Reviewer_nUpYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SCOPE, a controllable evaluation framework designed to diagnose memory-related failures in embodied navigation. It also proposes "Experience Skeleton Memory" (ESM), which enhances navigation performance by explicitly aligning past navigational experiences with current egocentric observations.

Critical strengths

This paper explores how to distinguish between "insufficient environmental knowledge" and "failure to effectively utilize existing memory." Through controlled comparative memory analysis and checkpoint-based interventions, the study provides valuable diagnostic insights.

The results for ESM are effective: the success rate on the full task set increased from 72.66% to 85.94%. Furthermore, among the remaining failure cases, it also outperformed both Frame and TCoT.

Critical weaknesses

Although SCOPE encompasses various task types, most of its tasks remain confined to traditional indoor navigation settings—specifically those focused on reaching objects or specific areas. The paper fails to clearly justify the need for a new benchmark, rather than simply applying the proposed evaluation scheme to existing benchmarks (such as GOAT-Bench) or established ObjectNav tasks based on the HM3D dataset. Evaluating ESM on existing benchmarks would have better demonstrated its generalizability.

The paper lacks direct comparisons with similar memory-based navigation approaches—most notably Mem2Ego, which also explicitly addresses the alignment between global memory and egocentric memory. Consequently, the specific advantages of ESM over existing methods remain unclear.

ESM utilizes metric poses, camera geometry, and depth information—all provided by the simulator—to maintain spatial correspondences. Therefore, the observed performance gains may stem, in part, from precise localization rather than solely from the design of the memory mechanism. The absence of experiments with physical robots, combined with a lack of robust evaluation against localization noise, weakens the evidence for the method's feasibility in practical, real-world applications.

Questions for authors

see weaknesses

5 comments
AuthorsOpenReview ↗

Response to Reviewer nUpY (1/4): Why SCOPE

W1: "Although SCOPE encompasses various task types, most of its tasks remain confined to traditional indoor navigation settings … The paper fails to clearly justify the need for a new benchmark, rather than simply applying the proposed evaluation scheme to existing benchmarks (such as GOAT-Bench) or established ObjectNav tasks based on the HM3D dataset. Evaluating ESM on existing benchmarks would have better demonstrated its generalizability."

A1: We thank the reviewer for suggesting the comparison with existing benchmarks. We built dedicated tasks because existing benchmarks mostly do not require the agent to draw on memory repeatedly, so problems in memory use rarely surface; SCOPE's diagnostic protocol itself does not depend on these tasks and transfers to existing benchmarks. As shown in Table R1, every SCOPE task type requires remembering something specific within a step budget, while ObjectNav, MultiON, and GOAT together cover only three; as shown in Table R2, scene memory lifts the memory-dependent disambiguation and search & collection families by 40.48 and 22.58 pp, but direct navigation, closest to existing benchmarks, by only 5.41 pp. Keeping the indoor reaching format is deliberate: a fixed success criterion lets differences be attributed to memory. Moreover, success rate shows that an agent failed, not where; SCOPE's diagnostic protocol varies memory under a fixed policy and then swaps in ground-truth knowledge to locate the failing step; it does not depend on the task set, and we applied it unchanged to GOAT-Bench (next message).

Table R1: What each task type requires remembering within the step budget, and coverage by existing benchmarks (✓ = includes this task type).

Task and exampleWhat must be rememberedObjectNavMultiONGOATSCOPE
Collection: "Find all bedrooms."how many bedrooms there are, which were already found✓
Return: "the dining area, then return to the study"where the agent came from, how to go back✓
Candidate search: "the monitor ... may be in a study or bedroom"which studies and bedrooms were already searched✓
Relation: "the bedroom connected to the storage room"where the storage room is, which bedroom connects to it✓
Region: "Go to the laundry room."which room is the laundry room✓
Multi-goal: "the bathroom on the half-level below the kitchen, then ..."floor structure and progress so far✓✓✓
Attribute: "the long, multi-seat sofa in the corridor"which of several sofas matches✓✓
Object: "Go to the stove."where the stove was seen, how to get there✓✓✓✓

Table R2: SR (%) / SPL without memory (M0) and with scene memory (M5), by task family (SR from manuscript Tables 11–12; SPL computed from the same runs). SCOPE scenes have 5–24 annotated regions each (median 13), and each task allows only 30–60 decisions.

Task family (#)No memory (M0)Scene memory (M5)Gain
Direct (37)62.16 / 0.38767.57 / 0.569+5.41 / +0.182
Disambiguation (42)38.10 / 0.26878.57 / 0.613+40.48 / +0.345
Search & collection (31)32.26 / 0.17354.84 / 0.415+22.58 / +0.242
Multi-stage (18)27.78 / 0.15850.00 / 0.369+22.22 / +0.211
All (128)42.19 / 0.26465.63 / 0.518+23.44 / +0.254
AuthorsOpenReview ↗

Response to Reviewer nUpY (2/4): Results on GOAT-Bench

A1 (cont.): Following the suggestion, we carried the main-table setting over to GOAT-Bench unchanged, without retuning ESM. As shown in Table R3, 32.77% of subtasks still fail even with GT scene knowledge (27.34% on SCOPE in the manuscript), so the gap persists. But memory matters much less on GOAT: SR rises from 57.98% (no memory) to 63.87% (native memory) and 67.23% (GT_JOINT), the same ordering as on SCOPE, yet only 9.24 pp in total versus 30.47 pp on SCOPE in the manuscript, which is why we built dedicated tasks. ESM has the highest subtask SR, improving on GT_JOINT and Mem2Ego by 8.40 and 14.29 pp (95% CI vs. Mem2Ego: [+6.72, +21.88]), and is highest on all three goal types, including ObjectNav-style category goals (84.09% vs. 72.73%). Both the diagnostic protocol and ESM transfer directly to an existing benchmark.

Table R3: Results on GOAT-Bench (8 val-unseen scenes and 16 episodes chosen by a fixed rule, 119 subtasks, no scene overlap with SCOPE; success = stopping within 1 m of the goal, as in GOAT-Bench). M7's native scene memory is built from per-scene pre-task experience with the same pipeline as on SCOPE. 95% CIs are scene-clustered paired bootstrap; the last three columns give subtask SR by goal type (number of subtasks in parentheses).

ConditionScene mem.Episode mem.Subtask SR ↑SPL ↑ΔSR vs. GT_JOINT [95% CI]Category (44)Language (40)Image (35)
GT_JOINT (baseline)GT struct.struct.67.230.563–72.7355.0074.29
M0nonenone57.980.429−9.24 [−24.14, +5.17]68.1842.5062.86
M7native struct.struct.63.870.484−3.36 [−18.49, +11.11]72.7345.0074.29
Mem2EgoGT struct.Mem2Ego61.340.463−5.88 [−18.40, +7.96]70.4547.5065.71
ESMGT struct.ESM75.630.583+8.40 [+1.56, +17.39]84.0965.0077.14
AuthorsOpenReview ↗

Response to Reviewer nUpY (3/4): Comparison with Mem2Ego

W2: "The paper lacks direct comparisons with similar memory-based navigation approaches—most notably Mem2Ego, which also explicitly addresses the alignment between global memory and egocentric memory. Consequently, the specific advantages of ESM over existing methods remain unclear."

A2: We thank the reviewer for pointing to Mem2Ego. ESM stores what the agent saw where it actually stood, so it helps the agent determine where it is, the main failure our diagnosis identified; ESM is 11.72 and 14.29 pp above Mem2Ego on SCOPE and GOAT-Bench. Table R4 compares the two mechanisms: Mem2Ego's landmarks come from distant views; ESM's three components let the agent form continuous spatial judgments, keep duplicate records from piling up, and correct errors. For example, if the agent mistakes a distant room for a kitchen, Mem2Ego's landmark keeps that description, whereas ESM assigns semantics only after approaching and can correct a wrong label.

Following the suggestion, we re-implemented Mem2Ego with identical required environment information (pose, depth, and camera). As shown in Tables R5 and R3, both gaps have CIs excluding 0 ([+3.12, +20.31] and [+6.72, +21.88]); as shown in Table R6, attributing each method's own failures shows that ESM reduces current-localization, goal-localization, and egocentric-correspondence failures from 29 to 13 while other failures stay about the same: the gap comes mainly from these three types, which these designs target.

Table R4: Memory mechanisms of Mem2Ego and ESM.

ComponentMem2EgoESMPurpose
What is recordeddescriptions of distant candidatesobservations where the agent stoodnot misled by distant views
Egocentric mappingmarks candidates and visited positions (without what was seen)marks past observation points with what was seencontinuous spatial judgments
Spatial aggregationlandmarks kept separaterevisited places merged into nodesrecords do not pile up
Correctionunchanged once writtenwrong labels can be fixederrors can be corrected

Table R5: Results on SCOPE's 128 tasks. Mem2Ego has no official code; we re-implemented its memory mechanism from the paper, with everything else identical to ESM. In all our tables, ESM and GT_JOINT are concurrent re-runs under the frozen configuration (manuscript runs: 85.94% and 72.66%). 95% CIs are scene-clustered paired bootstrap.

MethodSR ↑SPL ↑ΔSR vs. Mem2Ego [95% CI]
Mem2Ego (baseline)70.310.518–
ESM82.030.640+11.72 [+3.12, +20.31]

Table R6: Attribution of each method's own failures by the manuscript's failure categories (annotators blind to the method). The first three categories are all failures to relate memory to the current situation. Entries are numbers of failed tasks, with the share of all 128 tasks in parentheses.

Failure categoryMem2Ego (baseline)ESM
Current Localization11 (8.59%)4 (3.13%)
Goal Localization2 (1.56%)2 (1.56%)
Egocentric Correspondence16 (12.50%)7 (5.47%)
Other9 (7.03%)10 (7.81%)
First three combined29 (22.66%)13 (10.16%)
All failures38 (29.69%)23 (17.97%)
Reviewer_45GxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SCOPE, a controlled framework for studying why embodied navigation agents fail despite having environmental memory. By varying scene and episode memory and analyzing failures under ground-truth scene knowledge, the authors identify memory-to-observation correspondence as a major bottleneck. They further propose Experience Skeleton Memory (ESM), which uses geometric trajectory information to align past observations with the agent's current viewpoint (Sec. 4.4).

Critical strengths
  1. Systematic failure diagnosis. The paper distinguishes missing environmental knowledge from errors in using available memory. The analysis attributes 24/35 residual failures to localization or egocentric correspondence (Fig. 4a), while targeted corrections recover 15/35 episodes compared with 5/35 under sham interventions (Fig. 4b).

  2. Meaningful navigation improvements. Scene memory substantially improves navigation success (Table 1). ESM further increases success from 72.66% to 85.94% under ground-truth scene knowledge (Table 3), demonstrating the potential benefit of explicit geometric correspondence.

Critical weaknesses
  1. Reliance on privileged simulator information. ESM uses simulator-provided metric poses and is evaluated with ground-truth scene memory. The paper does not evaluate noisy or estimated poses, so I’m not sure whether this approach would work in realistic settings with imperfect localization.

I do not work in navigation or embodiment. So I prefer to provide a clean signal rather than mentioning any generic weaknesses.

Questions for authors

Please see the weaknesses list.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 45Gx (1/2): Noisy poses, visual odometry, and a real robot

W: "Reliance on privileged simulator information. ESM uses simulator-provided metric poses and is evaluated with ground-truth scene memory. The paper does not evaluate noisy or estimated poses, so I'm not sure whether this approach would work in realistic settings with imperfect localization."

A1: We thank the reviewer for the positive assessment and for this clear question. The short answer is yes: on a real robot with LiDAR odometry and no ground-truth inputs, ESM completes 12/20 tasks versus 8/20; in simulation, its gain shrinks gradually as pose error grows and stays no worse than the pose-free baseline. Pose is a standard input in robot navigation (odometry or SLAM); ESM uses it only to organize memory, and the policy never sees coordinates. Ground-truth scene memory is only a diagnostic condition that rules out missing knowledge.

Following the suggestion, we added distance-accumulating drift to the poses ESM reads and also used visual-odometry poses. As shown in Table R1, success declines gradually with pose drift: ESM keeps most of its gain under low drift (+10.16 pp) and stays no worse than the baseline under high drift and visual odometry (+3.91, +5.47 pp). ESM tolerates error because shifted memories are corrected by new observations on revisits, and distant targets only need a rough direction. We also ran 20 closed-loop tasks on a DEEP Robotics Lynx M20 robot in two real indoor environments, with LiDAR-odometry poses and scene memory from a pre-task walk-through, i.e., both privileged inputs relaxed, with no ground truth. As shown in Table R2, ESM is higher in both environments. Its extra successes are on round trips, collection, and room-by-room checks: rooms in offices and warehouses look alike, so these tasks require remembering where the robot has been from accumulated observations, which is what ESM provides.

We will add these results to the final version. We summarize the other new results of this rebuttal in the next comment.

Table R1: Results under different pose sources (128 tasks, GT scene memory). The baseline uses the same scene memory but neither ESM nor pose, so pose error does not affect it; the executor is unchanged in all runs. For comparability with the new conditions, ESM and GT_JOINT use concurrent re-runs under the frozen configuration (manuscript runs: GT_JOINT 72.66%, ESM 85.94%). Drift accumulates with distance traveled, as in real odometry; visual odometry: RGB-D ORB-PnP VO; ATE is the measured trajectory error. 95% CIs are scene-clustered paired bootstrap.

ConditionMeasured ATE (m)SR ↑SPL ↑ΔSR vs. baseline [95% CI]
Baseline (GT_JOINT, no pose)–70.310.549–
ESM, simulator pose082.030.640+11.72 [+3.91, +20.31]
ESM, low drift0.0880.470.627+10.16 [−0.78, +21.09]
ESM, medium drift0.2877.340.600+7.03 [−2.34, +16.41]
ESM, high drift0.5574.220.572+3.91 [−6.25, +14.06]
ESM, visual odometry0.3475.780.576+5.47 [−5.47, +16.41]

Table R2: Closed-loop results on the physical robot, by task type. Two environments: an office floor (corridor, 4 offices, 3 meeting rooms, pantry, restroom; 10 regions) and a warehouse & showroom (corridor, 3 storerooms, showroom, mailroom, restroom; 7 regions), 10 tasks each. Four onboard RGB-D cameras give the four egocentric views used in simulation, with depth; executor: ROS 2 Nav2 with an MPPI controller. The baseline is structured episode memory, with the same scene memory (from a pre-task walk-through), start poses, and executor as ESM. Success: a human verifies the agent's claimed result. Per environment: office, baseline 3 vs. ESM 5; warehouse, 5 vs. 7. Per task: ESM-only successes 5, baseline-only 1, both succeed 7, both fail 7.

Task typeExample taskTasksBaseline successesESM successes
Unique-target searchFind the only white canvas bag655
Round tripOffice 1 → meeting room 3 → office 1413
Collect all instancesFind 2 different hand trucks612
Room-by-room checkFind which of 4 offices has a filing cabinet201
TraversalVisit each of the 3 storerooms at least once211
Total–20812
AuthorsOpenReview ↗

Response to Reviewer 45Gx (2/2): Other new results

Supplement: other new results in this rebuttal. ESM also outperforms Mem2Ego, a comparable method, and transfers to the external GOAT-Bench without retuning. SCOPE's tasks truly depend on memory, and its evaluation locates where a failure happens; this is how we found that even with complete and correct scene knowledge, the agent still fails on 27% of tasks in the manuscript, mostly (24/35) when relating memory to the current situation. Existing pose-based memory methods such as Mem2Ego mainly serve exploration; ESM targets this step with observation-point memory, aggregation, and correction. As shown in Table R3, under identical conditions ESM is 11.72 pp above Mem2Ego, and removing any design lowers success; as shown in Table R4, on the external GOAT-Bench, ESM, without retuning, also outperforms GT_JOINT and Mem2Ego (+8.40 and +14.29 pp, both CIs excluding 0).

Table R3: Success rate after removing each ESM design, and comparison with Mem2Ego (128 tasks, identical required environment information). For comparability with the new conditions, ESM and GT_JOINT use concurrent re-runs under the frozen configuration (manuscript runs: GT_JOINT 72.66%, ESM 85.94%).

SettingSuccess rateDifference vs. full ESM
Full ESM (baseline)82.03–
Without correction78.12−3.91
Without aggregation75.00−7.03
Observation-point memory only (no aggregation, no correction)73.44−8.59
Mem2Ego70.31−11.72
Without ESM (GT_JOINT, no pose)70.31−11.72

Table R4: Subtask success rate on GOAT-Bench (8 val-unseen scenes, 16 episodes, 119 subtasks; ESM not retuned).

MethodSubtask success rate
GT_JOINT67.23
Mem2Ego61.34
ESM75.63

Thank you for the rebuttal and additional experiments. The evaluations with noisy and estimated poses, along with the real-robot experiments, address my concerns.

As mentioned in my review, embodied navigation is outside my area of expertise. Nevertheless, I will maintain my positive assessment of this work, with a confidence score of 2.

Thank you for the follow-up and for maintaining your positive assessment. We have added the noisy-pose, visual-odometry, and real-robot results to the revised PDF (Section 5.5 and Table 4 in the main text; Appendix I.3–I.4).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank all reviewers for their constructive comments. We have uploaded a revised PDF that incorporates the new results from our replies (new content is marked in blue):

  • Main text. Section 5.5 adds a paragraph and Table 4: the matched-input comparison with Mem2Ego; frozen evaluations on unseen scenes, GOAT-Bench, and a second policy; robustness to pose drift and visual odometry; and the physical-robot study (Reviewers dyop, nUpY, 45Gx). Section 3.2 explains why dedicated tasks are needed (Reviewer nUpY). The abstract and conclusion are updated accordingly.
  • Appendix I (new), Robustness and Independent Validation of ESM. I.1 matched-input ablations, per-family results, the Mem2Ego mechanism comparison, and failure attribution (Reviewers dyop, nUpY); I.2 frozen evaluation on unseen scenes, a second policy, and GOAT-Bench (Reviewers dyop, nUpY); I.3 localization error (Reviewers 45Gx, nUpY); I.4 physical-robot setup, results, and a qualitative round-trip example (Reviewers 45Gx, nUpY).
  • Appendix A.5 (new). Comparison with existing benchmarks (Reviewer nUpY).
  • Appendix E.7 (new). The re-run under the frozen configuration (Reviewer dyop).

The original results in the main paper are unchanged; the new 128-task comparisons use concurrent re-runs under the frozen configuration, as stated in Section 5.5 and Appendix E.7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.