- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SCOPE examines navigation failures that persist despite accurate scene knowledge. Ground-truth memory replacements and checkpoint interventions distinguish missing knowledge from failures to relate that knowledge to the current situation. Experience Skeleton Memory (ESM) then combines executed experience with simulator geometry. On 128 tasks, targeted corrections recover ten more episodes than sham interventions, and ESM raises success from 93 to 110 episodes.
Critical strengths
S1. Interventions support the diagnosis. Targeted corrections are compared with both unmodified replay and same-channel sham interventions (Sections 4.2-4.3; Appendix F.4). The ten additional recoveries over sham provide evidence that some failures concern the use of available knowledge.
S2. The accounting is transparent. The paper reports paired outcomes, scene-clustered uncertainty and ESM's five regressions alongside its 22 recoveries (Tables 1-3). It also states its reliance on privileged geometry and avoids treating verbal rationales as direct evidence of internal reasoning.
Critical weaknesses
W1. ESM's design-specific contribution is unresolved. ESM adds pose-derived directions, depth-based associations, route history and revisit judgments together (Section 4.4; Appendix G). Frame and TCoT lack the same correspondence information. The gain supports this combined intervention, but does not show what the skeleton representation contributes beyond the added geometric information.
W2. The remedy is evaluated on the failures that motivated it. The full-suite test includes the same 35 failures used to develop ESM (Section 5.5; Appendix E.6). Counting the other 93 tasks measures regressions but supplies no independent test of the remedy. The second-policy study repeats memory replacement, not the diagnosis-to-ESM chain.
Assessment. I recommend weak rejection (2/4). The controlled diagnosis is useful, but the evidence leaves both ESM's distinct contribution and the independence of its validation unresolved.
Questions for authors
Q1 (W1). Which reported comparison holds pose, depth and camera correspondence fixed while changing the memory representation? If none does, please distinguish the demonstrated benefit of the combined geometric intervention from a benefit specific to ESM's skeleton design.
Q2 (W2). Which reported ESM results were obtained after its interface and parameters were frozen, on tasks not used during diagnosis or development? Please also state the rule used to select the executions in Appendix H when multiple attempts existed.
3 comments
Response to Reviewer dyop (2/3): Breakdown by task family and failure type
A1 (cont.): We further break down the same runs by task family and failure category. As shown in Table R2, the gap concentrates on the multi-stage and disambiguation families: on multi-stage tasks ESM reaches 88.89% versus 55.56% for Mem2Ego, and removing any component lowers it; ESM has the highest SPL in all four families. As shown in Table R3, attributing each method's own failures, ESM reduces failures in the first three categories from 29 to 13, while other failures stay about the same. With the same geometric information, a different organization changes the outcome exactly on the diagnosed failures.
Table R2: SR (%) / SPL by task family; number of tasks in parentheses; bold marks where ESM is highest in the column (incl. ties).
| Condition | Direct (37) | Disambiguation (42) | Search & collection (31) | Multi-stage (18) |
|---|---|---|---|---|
| ESM | 81.08 / 0.647 | 85.71 / 0.657 | 74.19 / 0.617 | 88.89 / 0.625 |
| ESM w/o revisit reasoning | 78.38 / 0.588 | 76.19 / 0.599 | 77.42 / 0.609 | 77.78 / 0.600 |
| ESM w/o correction | 81.08 / 0.622 | 76.19 / 0.631 | 74.19 / 0.611 | 83.33 / 0.599 |
| ESM w/o aggregation | 75.68 / 0.572 | 73.81 / 0.581 | 70.97 / 0.561 | 83.33 / 0.580 |
| Obs.-point memory only | 72.97 / 0.548 | 73.81 / 0.558 | 77.42 / 0.539 | 66.67 / 0.521 |
| Mem2Ego | 75.68 / 0.566 | 66.67 / 0.515 | 77.42 / 0.547 | 55.56 / 0.378 |
| GT_JOINT | 72.97 / 0.628 | 78.57 / 0.560 | 67.74 / 0.536 | 50.00 / 0.383 |
Table R3: Attribution of each method's own failures by the manuscript's failure categories (number of failures among 128 tasks, with the share in parentheses; annotators blind to the method). The first three categories are all failures to relate memory to the current situation.
| Failure category | Mem2Ego (baseline) | ESM |
|---|---|---|
| Current Localization | 11 (8.59%) | 4 (3.13%) |
| Goal Localization | 2 (1.56%) | 2 (1.56%) |
| Egocentric Correspondence | 16 (12.50%) | 7 (5.47%) |
| Other | 9 (7.03%) | 10 (7.81%) |
| First three combined | 29 (22.66%) | 13 (10.16%) |
| All failures | 38 (29.69%) | 23 (17.97%) |
Response to Reviewer dyop (3/3): Independent validation and run selection
W2/Q2: "The full-suite test includes the same 35 failures used to develop ESM … Which reported ESM results were obtained after its interface and parameters were frozen, on tasks not used during diagnosis or development? Please also state the rule used to select the executions in Appendix H when multiple attempts existed."
A2: We thank the reviewer for the careful reading and constructive comments, and we agree: ESM's design was motivated by the diagnosis of these 35 failures; mindful of independence, we developed and debugged it on separate development scenes, but the manuscript's evaluation still included these 35 tasks. We therefore add the following.
(1) Frozen ESM's gain is significant on new scenes, GOAT-Bench, and a second policy. Following the suggestion, we froze ESM and evaluated it on new scenes and GOAT-Bench, neither used in diagnosis or development, and also ran a second policy, GPT-6 Luna, on the new scenes; unlike the manuscript's second-policy study, which only replaced memory, this runs the full ESM. As shown in Table R4, ESM improves on GT_JOINT by 12.50, 8.40, and 10.94 pp, all with CIs excluding 0, consistent with the manuscript.
(2) No run was selected by outcome: each task–condition pair was scored once. Development scenes are fully separate from evaluation scenes, and development runs are never selected; in the formal evaluation nothing was re-run: 89 task–condition pairs interrupted by execution-framework issues (mainly failed network requests) were resumed from the interruption point after the fix, all in the manuscript's baseline conditions M0–M3, none involving ESM or GT_JOINT. In Appendix H, "selected" denotes the single formal run of each task–condition pair; ESM's full-set result consists of its 35 residual-set runs and 93 further runs, all with the same code. After freezing, a full re-run keeps the gain significant (Table R5).
We will add the new results from all our replies to the final version.
Table R4: Frozen ESM on tasks and policies not used in diagnosis or development. New scenes: 64 tasks in 32 new HM3D scenes, disjoint from the original 64, built with the original pipeline and family proportions and fixed before any ESM run; GOAT-Bench: 8 val-unseen scenes, 16 episodes, 119 subtasks. ΔSR is relative to GT_JOINT; 95% CIs are scene-clustered paired bootstrap.
| Task set | Policy | GT_JOINT SR | ESM SR | ΔSR vs. GT_JOINT [95% CI] | GT_JOINT SPL | ESM SPL | ESM-only / GT_JOINT-only successes |
|---|---|---|---|---|---|---|---|
| New scenes | original | 62.50 | 75.00 | +12.50 [+1.56, +23.44] | 0.503 | 0.596 | 10 / 2 |
| GOAT-Bench | original | 67.23 | 75.63 | +8.40 [+1.56, +17.39] | 0.563 | 0.583 | 12 / 2 |
| New scenes | GPT-6 Luna | 50.00 | 60.94 | +10.94 [+3.12, +18.75] | 0.339 | 0.418 | 10 / 3 |
Table R5: ESM in the manuscript run and in the independent re-run under the frozen configuration (128 tasks). 21 tasks changed outcome between runs (8 solved only in the re-run, 13 only in the manuscript run), reflecting run-to-run variability of the model calls; GT_JOINT, re-run under the same configuration, gets 90/128 (70.31%; manuscript 93/128); each ΔSR is relative to GT_JOINT from the same run; all new comparisons use the re-run.
| Run | ESM successes | ESM SR | ESM SPL | ΔSR vs. GT_JOINT [95% CI] |
|---|---|---|---|---|
| Manuscript | 110/128 | 85.94 | 0.612 | +13.28 [+6.25, +21.09] |
| Independent re-run | 105/128 | 82.03 | 0.640 | +11.72 [+3.91, +20.31] |
Response to Reviewer dyop (1/3): Same geometry, different memory organization
A1: We thank the reviewer for this precise question. Table R1 is exactly this comparison: pose, depth, and camera are identical, and only the memory organization changes. Following the suggestion, we removed ESM's components one at a time (including aggregating observation points into skeleton nodes) and re-implemented Mem2Ego, which also uses pose. As shown in Table R1, with identical required environment information ESM is 11.72 pp above Mem2Ego (95% CI [+3.12, +20.31], excluding 0); removing any component lowers SR, and removing more lowers it further; observation-point memory alone (73.44%) is only 3.13 pp above the pose-free GT_JOINT (70.31%). Individual CIs are wide, but every removal points the same way. With identical required environment information, the gain comes from ESM's organization, not from more geometric information.
Table R1: Results with identical required environment information (pose, depth, and camera), varying only memory organization (128 tasks). Components: observation-point memory (each observation bound to where it was made), aggregation (repeatedly visited points merged into skeleton nodes), correction (RELABEL), revisit reasoning (direction-by-direction reasoning about what has been explored; backtracking must be justified). GT_JOINT uses no pose. For comparability with the new conditions, ESM and GT_JOINT use concurrent re-runs under the frozen configuration (manuscript runs: GT_JOINT 72.66%, ESM 85.94%). 95% CIs are scene-clustered paired bootstrap.