- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies mapping-time visual poisoning of hierarchical embodied memory. CueClimb constructs scene-grounded text cues, optimizes cue bundles using leaf and consolidated-summary relevance, and selects them across auxiliary contexts to redirect subsequent coordinate retrieval. Experiments on HM3D report higher attack success than the tested baselines, with frozen attacks transferring across victim configurations and query formulations.
Critical strengths
- The mapping-time setting highlights a meaningful persistence risk: misleading visual observations can affect later retrieval through stored memory. CueClimb operates with local observations and target semantics, using auxiliary scenes disjoint from the evaluation scenes.
- The ablations of the LCA scoring term and cross-context deployment selection show lower attack success when either component is removed, providing evidence that both choices contribute within the evaluated system.
- The evaluation extends beyond the main embedding-based router to LLM-based selection, alternative memory constructions, and unseen query formulations. The printed-label experiment also provides concrete evidence of physical feasibility.
Critical weaknesses
-
SceneTAP-Adapted is insufficiently specified to assess the main comparative claim. The original SceneTAP generates text content, determines its position, and integrates it into images, yet the paper does not explain which components are retained or how its generation and search budgets compare with CueClimb. Matching the two-cue deployment budget leaves these factors unresolved. Although the component ablations are useful, the headline gain would be more convincing with a fully specified adaptation and a comparison under matched attack-generation budgets.
-
Memory is rebuilt after poisoning, and clustering depends on text descriptions, so clean and poisoned ancestor summaries may cover different groups of observations. The higher-level similarity gains, LCA scoring ablation, and physical summary examples provide evidence of semantic retention, but ancestor correspondence is not explained. Reporting whether target-path grouping changes, and how clean and poisoned ancestors are aligned, would clarify the layer-wise comparison. If grouping changes materially, repeating the analysis with the clean tree structure held fixed would help isolate retention from regrouping.
-
Cue construction is tied to local images, while the main experiments use a fixed protocol for collecting mapping observations. The physical example demonstrates feasibility through recaptured observations, but does not show whether fixed cues remain effective when the victim observes different viewpoints or fewer cues. This matters because consistent evidence across observations is central to the proposed consolidation mechanism. Testing fixed cues under independently sampled mapping views would address this gap.
-
The benign-query cohort excludes all attack-target locations and does not specifically evaluate legitimate functions supported mainly by signage or sparse evidence. It therefore remains unclear whether Strong suppresses misleading claims while preserving valid information in the cases most vulnerable to its filtering rules.
Questions for authors
see weakness
4 comments
Response by reviewer W9Fj
This response clarifies the baseline adaptation and budgets and addresses the regrouping concern. However, the experiment with independently collected observations still lacks clean ASR and SceneTAP-Adapted results on the same observations. Without these controls, the reported 30.0% ASR does not establish how much poisoning increases ASR or whether CueClimb still outperforms SceneTAP-Adapted in this setting.
Thank you for pointing out the missing controls. We evaluated the clean condition and SceneTAP-Adapted on the same 30 case–seed pairs as CueClimb (10 cases × 3 seeds), following the same observation collection protocol. We also checked SceneTAP-Adapted on the original observations for those 10 cases and found that it matched CueClimb at 50.0% ASR.
| Condition | ASR |
|---|---|
| Clean (shared) | 1/30 (3.3%) |
| SceneTAP-Adapted | 7/30 (23.3%) |
| CueClimb | 9/30 (30.0%) |
CueClimb was more effective than SceneTAP-Adapted under independent observation collection, achieving a 6.7 percentage point higher ASR (30.0% vs. 23.3%).
Dear Reviewer W9Fj,
Thank you again for taking the time to review our paper. Your feedback has been very helpful to us. With less than 5 hours left in the discussion period, we hope you don’t mind us following up once more. We would greatly appreciate even a brief note on whether our response and revisions have addressed your concerns.
Best regards, The Authors
We appreciate the reviewer's insightful suggestions and questions. Below is our clarification:
Q1: SceneTAP-Adapted: Adaptation Details and Matched Attack-Generation Budgets
(1) Models and Budget: CueClimb achieves an 18.2-point ASR advantage over SceneTAP-Adapted (46.5% vs. 28.3%) under matched attack-generation budgets. Both methods use GPT-5.5 for candidate generation and GPT-4o for search evaluation. The table summarizes 20 randomly sampled cases, where CueClimb uses ~0.19M fewer generation tokens.
(2) Baseline Adaptation: The specific adaptations are as follows:
These details are now added in Appendix A.
Q2: Ancestor Correspondence and Target-Path Grouping Changes
(1) Correspondence: Target-path grouping changes in 95 of 99 cases, so clean and poisoned ancestors do not correspond one-to-one. We therefore align comparisons by path position from the target observation leaf upward. The layer-wise gains in the paper thus also reflect regrouping.
(2) Fixed Clean Structure: With the clean tree structure held fixed, semantic gains persist at all ancestor levels and ASR rises from 46.5% to 53.5%, supporting cross-level semantic retention as an attack mechanism independent of regrouping. To isolate this effect, we keep the clean tree’s nodes, groupings, and parent–child relations fixed, replace only the attacked leaf descriptions, and regenerate summaries bottom-up. Other settings follow Table 3 (GPT-5.6-Terra, P0).
These results are now added in Appendices C and K.
Q3: Fixed-Cue Effectiveness under Independently Sampled Mapping Views
Fixed cues remain effective under independently sampled mapping views. We reevaluate 10 randomly sampled cases from 5 HM3D scenes under three random seeds per case (30 evaluations), with cue positions, sizes, and orientations fixed in 3D and no pose selection. Each seed changes the starting point, collected views, and cue visibility. Fresh data collection for every case–seed pair is costly, limiting scale within the rebuttal period.
As shown below, under uncontrolled viewpoints and visibility, ASR reaches 30.0%, retaining 60% of the original 50.0% (five successes and five failures). Attacks succeed under all three seeds, including cases with only one fully readable cue. Randomness in observation collection is an aspect of the deployment diversity noted in our limitations, and we will explicitly discuss this factor in the revised limitations section. We thank you for this suggestion, which led us to directly test the cross-observation consistency that consolidation relies on.
These results are now added in Appendix L.
Q4: Strong’s Utility on Legitimate Queries Most Vulnerable to Its Filtering
Strong maintains comparable utility to no defense on legitimate queries vulnerable to filtering (76.7% vs. 73.3%). We add 30 independently generated queries that may be wrongly filtered by Strong, 10 per category; the third category covers exactly the attack-target locations excluded from the original benign set. Humans judge whether the returned location satisfies the query.
As shown below, even on these hard queries, Strong differs from no defense by at most one case per category. Together with Strong reducing ASR from 46.5% to 6.1% (Table 4), this shows that Strong suppresses misleading claims while preserving legitimate functions.
These results are now added in Appendix J.