acceptodds
Under review as a conference paper at ICLR 2027

Local Cues, Global Misrouting: Poisoning Hierarchical Embodied Memory through Recursive Consolidation

Abstract

Long-horizon embodied agents usually store extensive visual observations, motivating the use of hierarchical memory. Typically, local visual observations are recursively consolidated into higher-level summaries that guide coarse-to-fine retrieval. However, the same consolidation process can also preserve misleading visual cues in higher-level summaries. By exploiting these cues, an attacker biases subsequent retrieval and thereby misdirects the agent to a specified wrong location. Despite growing work on poisoning in RAG and agent memories, poisoning through recursive consolidation remains underexplored. To address this gap, we develop CueClimb, a visual poisoning method that operates during environment mapping. CueClimb constructs sparse, scene-grounded visual cues and searches for compact bundles whose semantic influence survives recursive consolidation. It uses only local observations and auxiliary consolidation contexts, without access to the full victim scene or subsequent queries. Across 99 target-query cases in 23 HM3D scenes, CueClimb achieves a 46.5% ASR, compared with 28.3% for the strongest baseline. The attacks transfer across different models, consolidation prompts and hierarchical memory constructions. These findings expose recursive consolidation as a critical attack surface, underscoring the need for trustworthy embodied memory.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Memory poisoning, Embodied AI security, Hierarchical memory, Recursive consolidation, Hierarchical retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_W9FjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies mapping-time visual poisoning of hierarchical embodied memory. CueClimb constructs scene-grounded text cues, optimizes cue bundles using leaf and consolidated-summary relevance, and selects them across auxiliary contexts to redirect subsequent coordinate retrieval. Experiments on HM3D report higher attack success than the tested baselines, with frozen attacks transferring across victim configurations and query formulations.

Critical strengths
  • The mapping-time setting highlights a meaningful persistence risk: misleading visual observations can affect later retrieval through stored memory. CueClimb operates with local observations and target semantics, using auxiliary scenes disjoint from the evaluation scenes.
  • The ablations of the LCA scoring term and cross-context deployment selection show lower attack success when either component is removed, providing evidence that both choices contribute within the evaluated system.
  • The evaluation extends beyond the main embedding-based router to LLM-based selection, alternative memory constructions, and unseen query formulations. The printed-label experiment also provides concrete evidence of physical feasibility.
Critical weaknesses
  • SceneTAP-Adapted is insufficiently specified to assess the main comparative claim. The original SceneTAP generates text content, determines its position, and integrates it into images, yet the paper does not explain which components are retained or how its generation and search budgets compare with CueClimb. Matching the two-cue deployment budget leaves these factors unresolved. Although the component ablations are useful, the headline gain would be more convincing with a fully specified adaptation and a comparison under matched attack-generation budgets.

  • Memory is rebuilt after poisoning, and clustering depends on text descriptions, so clean and poisoned ancestor summaries may cover different groups of observations. The higher-level similarity gains, LCA scoring ablation, and physical summary examples provide evidence of semantic retention, but ancestor correspondence is not explained. Reporting whether target-path grouping changes, and how clean and poisoned ancestors are aligned, would clarify the layer-wise comparison. If grouping changes materially, repeating the analysis with the clean tree structure held fixed would help isolate retention from regrouping.

  • Cue construction is tied to local images, while the main experiments use a fixed protocol for collecting mapping observations. The physical example demonstrates feasibility through recaptured observations, but does not show whether fixed cues remain effective when the victim observes different viewpoints or fewer cues. This matters because consistent evidence across observations is central to the proposed consolidation mechanism. Testing fixed cues under independently sampled mapping views would address this gap.

  • The benign-query cohort excludes all attack-target locations and does not specifically evaluate legitimate functions supported mainly by signage or sparse evidence. It therefore remains unclear whether Strong suppresses misleading claims while preserving valid information in the cases most vulnerable to its filtering rules.

Questions for authors

see weakness

4 comments

We appreciate the reviewer's insightful suggestions and questions. Below is our clarification:

Q1: SceneTAP-Adapted: Adaptation Details and Matched Attack-Generation Budgets

(1) Models and Budget: CueClimb achieves an 18.2-point ASR advantage over SceneTAP-Adapted (46.5% vs. 28.3%) under matched attack-generation budgets. Both methods use GPT-5.5 for candidate generation and GPT-4o for search evaluation. The table summarizes 20 randomly sampled cases, where CueClimb uses ~0.19M fewer generation tokens.

ResourceSceneTAP-AdaptedCueClimb
Candidates1,0251,025
Two-cue combination evaluations2,7792,779
Deployed cues / case22
Generation requests (model calls)3,2733,265
Generation tokens~8.09M~7.90M

(2) Baseline Adaptation: The specific adaptations are as follows:

  • Text generation: Scene analysis, text design, and the plan→adjust step are kept. The VQA objective is replaced by the same target semantics CueClimb uses, so attacker knowledge is identical; text length is unified to 4–5 words.
  • Placement: SAM/SoM regions and placement planning are retained on the same valid surfaces as CueClimb.
  • Image blending: TextDiffuser-2 operates on 2D images and does not directly support real-time rendering in Habitat. Both methods therefore share a 3D label renderer, also controlling for rendering quality.
  • Combination search: SceneTAP-Adapted uses random combination search, with candidate counts and combination-evaluation budgets matched to CueClimb per case.

These details are now added in Appendix A.

Q2: Ancestor Correspondence and Target-Path Grouping Changes

(1) Correspondence: Target-path grouping changes in 95 of 99 cases, so clean and poisoned ancestors do not correspond one-to-one. We therefore align comparisons by path position from the target observation leaf upward. The layer-wise gains in the paper thus also reflect regrouping.

(2) Fixed Clean Structure: With the clean tree structure held fixed, semantic gains persist at all ancestor levels and ASR rises from 46.5% to 53.5%, supporting cross-level semantic retention as an attack mechanism independent of regrouping. To isolate this effect, we keep the clean tree’s nodes, groupings, and parent–child relations fixed, replace only the attacked leaf descriptions, and regenerate summaries bottom-up. Other settings follow Table 3 (GPT-5.6-Terra, P0).

Semantic gain / ASRRe-clusteringFixed clean structure
Leaf+0.2247+0.2247
Local ancestor+0.1306+0.1591
Region/floor ancestor+0.0598+0.0784
Root+0.0324+0.0405
Clean ASR → poisoned ASR1.0% → 46.5%1.0% → 53.5%

These results are now added in Appendices C and K.

Q3: Fixed-Cue Effectiveness under Independently Sampled Mapping Views

Fixed cues remain effective under independently sampled mapping views. We reevaluate 10 randomly sampled cases from 5 HM3D scenes under three random seeds per case (30 evaluations), with cue positions, sizes, and orientations fixed in 3D and no pose selection. Each seed changes the starting point, collected views, and cue visibility. Fresh data collection for every case–seed pair is costly, limiting scale within the rebuttal period.

As shown below, under uncontrolled viewpoints and visibility, ASR reaches 30.0%, retaining 60% of the original 50.0% (five successes and five failures). Attacks succeed under all three seeds, including cases with only one fully readable cue. Randomness in observation collection is an aspect of the deployment diversity noted in our limitations, and we will explicitly discuss this factor in the revised limitations section. We thank you for this suggestion, which led us to directly test the cross-observation consistency that consolidation relies on.

Label readabilityGroupsASR
Both fully readable2128.6%
Only one fully readable933.3%
Overall3030.0%

These results are now added in Appendix L.

Q4: Strong’s Utility on Legitimate Queries Most Vulnerable to Its Filtering

Strong maintains comparable utility to no defense on legitimate queries vulnerable to filtering (76.7% vs. 73.3%). We add 30 independently generated queries that may be wrongly filtered by Strong, 10 per category; the third category covers exactly the attack-target locations excluded from the original benign set. Humans judge whether the returned location satisfies the query.

As shown below, even on these hard queries, Strong differs from no defense by at most one case per category. Together with Strong reducing ASR from 46.5% to 6.1% (Table 4), this shows that Strong suppresses misleading claims while preserving legitimate functions.

Query categoryNo defenseStrong
Legitimate signage text7/106/10
Sparse legitimate cues8/109/10
True function of attack target location7/108/10
Overall73.3%76.7%

These results are now added in Appendix J.

Reviewer_W9FjOpenReview ↗

Response by reviewer W9Fj

This response clarifies the baseline adaptation and budgets and addresses the regrouping concern. However, the experiment with independently collected observations still lacks clean ASR and SceneTAP-Adapted results on the same observations. Without these controls, the reported 30.0% ASR does not establish how much poisoning increases ASR or whether CueClimb still outperforms SceneTAP-Adapted in this setting.

Thank you for pointing out the missing controls. We evaluated the clean condition and SceneTAP-Adapted on the same 30 case–seed pairs as CueClimb (10 cases × 3 seeds), following the same observation collection protocol. We also checked SceneTAP-Adapted on the original observations for those 10 cases and found that it matched CueClimb at 50.0% ASR.

ConditionASR
Clean (shared)1/30 (3.3%)
SceneTAP-Adapted7/30 (23.3%)
CueClimb9/30 (30.0%)

CueClimb was more effective than SceneTAP-Adapted under independent observation collection, achieving a 6.7 percentage point higher ASR (30.0% vs. 23.3%).

Dear Reviewer W9Fj,

Thank you again for taking the time to review our paper. Your feedback has been very helpful to us. With less than 5 hours left in the discussion period, we hope you don’t mind us following up once more. We would greatly appreciate even a brief note on whether our response and revisions have addressed your concerns.

Best regards, The Authors

Reviewer_z9K4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies mapping-time visual poisoning of hierarchical embodied memory. The attacker places a small number of scene-grounded textual cues near a designated incorrect destination, aiming to make their semantics survive recursive summarization and redirect subsequent coordinate retrieval.

The proposed method, CueClimb, combines atomic cue construction, local bundle search using leaf-level and lowest-common-ancestor semantic scores, and deployment selection across auxiliary contexts from disjoint scenes. Attack construction uses local observations and a target semantic concept, without the exact future query or the full victim scene.

On 99 query–target cases across 23 HM3D scenes, CueClimb achieves 46.5% attack success, compared with 28.3% for SceneTAP-Adapted. The paper also provides component ablations, cross-level analyses, transfer experiments, LLM-based retrieval evaluations, unseen-query reformulations, and a physical printed-label demonstration. Recursive-summary constraints substantially reduce the success of the evaluated frozen attacks. The empirical scope is destination-coordinate retrieval rather than complete navigation execution.

Critical strengths

1.An important security problem with clear relevance to embodied agents. The paper highlights how untrusted visual information can enter persistent memory and influence subsequent decisions. This is a meaningful concern for agents that rely on accumulated observations, and the study draws attention to an important vulnerability in hierarchical memory construction.

2.A focused approach with useful empirical insights. The proposed method is well aligned with the identified problem, focusing on how misleading information is retained and propagated during recursive consolidation. The complementary evaluations provide useful insight into this failure mode and help inform the design of more trustworthy embodied memory systems.

Critical weaknesses
  1. The generality of the reported attack success rate remains unclear.

    Although the evaluation queries are fixed before attack generation, they are constructed around predefined target semantics. The query-reformulation experiment provides useful evidence of robustness to alternative wording, but preserves the same underlying task intents. Consequently, the results establish effectiveness on the constructed query–target cases, while leaving unclear how much success depends on their particular formulation and selection. An evaluation using independently collected, task-grounded queries within the same predeclared target scope would help determine whether the reported effectiveness extends to realistic user requests.

  2. Attack reliability under independently collected visual observations is insufficiently established.

    The threat model assumes that the attacker cannot control the victim’s data collection. However, the main experimental description does not clearly establish how the attack performs when mapping observations are collected independently after cue placement, or how cue consistency across viewpoints is maintained. The physical demonstration supports feasibility, but does not quantify robustness to changes in viewing angle, distance, visibility, or observation coverage. These conditions matter because successful deployment requires the cues to be captured, interpreted, and retained in memory. The reported ASR therefore does not yet establish reliability across realistic variations in observation conditions.

  3. The source of the improvement over the strongest baseline is not sufficiently isolated.

    The paper matches the final cue budget, but does not describe SceneTAP-Adapted’s candidate generation, search procedure, and evaluation resources as clearly as those of CueClimb. It is also unclear whether the local cue-combination ablation retains the same number of deployed cues. These ambiguities make it difficult to determine how much of the improvement comes specifically from consolidation-aware optimization, rather than candidate quality, search effort, or deployment budget. Clearer implementation details and controlled comparisons are needed to substantiate the incremental contribution of the proposed method.

Questions for authors
  1. Query construction and evaluation independence.
    The paper states that evaluation queries are fixed before attack generation. Beyond this safeguard, how are the target concepts and corresponding queries generated and selected, and how do they reflect realistic user needs? Are there results on independently formulated, task-grounded queries within the same target scope, beyond reformulations of the original queries? Such evidence would help assess how strongly the reported ASR depends on the benchmark’s query construction.

  2. Visual deployment and observation robustness.
    In the main simulation experiments, are the cues anchored to fixed 3D surfaces and rendered consistently across viewpoints, or inserted separately into individual observation images? With the cues frozen, how does attack success change when mapping observations are collected independently of those used during attack construction? Results under changed viewpoints or visibility conditions would help establish the reliability of the attack under the stated data-collection constraints.

  3. Baseline and ablation comparability.
    What generation models, candidate-search procedures, and model-call budgets are used for SceneTAP-Adapted, and how do they compare with CueClimb? Does the “w/o local cue-combination” ablation still deploy exactly two cues, and how are those cues selected? Does the reported advantage persist under comparable deployment and search budgets?

3 comments

We thank the reviewer for the thoughtful suggestions.

Q1: Query Construction, Evaluation Independence, and ASR Generality

(1) Generation and Selection: For each scene, we select one of six functional need categories, propose a target function based on scene content near the destination, and check its plausibility. Queries are rewritten case by case from the template question, preserving functional constraints. We exclude cases where the destination already satisfies the query or the clean system already returns it.

(2) Real Needs: Our queries express practical indoor needs when users know their task but not the relevant room or service location. E.g., a staff member with faulty equipment asks “Where would staff go when equipment needs fixing?”; the target concept “Equipment repair support” is the function that resolves it. This follows demand-driven navigation and user-demand instruction generation [1-3].

(3) Independent Queries: CueClimb remains effective on independently formulated, task-grounded queries within the same target scope. Following NavRAG [3], we assign each of the 99 cases a role and task scenario. An independent LLM generates three colloquial questions per scenario (297 queries) without ever seeing the original query. E.g.:

  • Original: “Where would someone ask for maintenance support for equipment?”
  • Scenario: A resident’s vacuum has stopped working while they are cleaning.
  • New: “My vacuum’s stopped working—where should I go to get it looked at?”

On these queries, which better reflect users’ practical needs, CueClimb retains over 90% of its original ASR (46.5% → 42.1%) and has the highest retention of originally successful attacks (74.6%).

MethodOriginal ASRIndependent ASRRetention
Query-Copy Literal15.2%12.1%62.2%
SceneTAP-Adapted28.3%30.3%69.0%
CueClimb46.5%42.1%74.6%

We have added these details to Appendices B and F.

References: [1] Find What You Want; [2] Embodied-RAG; [3] NavRAG

Q2: Visual Deployment and Reliability under Independent Observations

(1) Rendering: In all reported evaluations, cues are bound to fixed 3D surfaces and rendered consistently across viewpoints. 2D overlays serve only as a low-cost proxy during optimization.

(2) Observation Robustness: Attacks remain effective under independently collected visual observations. We reevaluate 10 randomly sampled cases from 5 HM3D scenes using three random seeds per case (30 evaluations), with cue positions, sizes, and orientations fixed in 3D. Each seed changes the starting point, collected views, and cue visibility, without pose selection. Fresh collection per case–seed pair is costly, limiting scale within the rebuttal period.

ASR reaches 30.0%, versus 50.0% originally (five successes and five failures). Successful attacks occur under all three seeds, including cases with only one fully readable cue. Collection randomness is part of the deployment diversity noted in our limitations; we will discuss it explicitly in the revision. We thank you for prompting this test of cross-observation consistency.

Label readabilityGroupsASR
Both fully readable2128.6%
Only one fully readable933.3%
Overall3030.0%

We have added these results to Appendices A and L.

Q3: Baseline and Ablation Comparability and Attribution of Improvements

(1) Models and Budget: CueClimb outperforms SceneTAP-Adapted by 18.2 ASR points under matched attack-generation budgets. Both methods use GPT-5.5 for candidate generation and GPT-4o for search evaluation. The table covers 20 randomly sampled cases. Both reach 100% leaf Rank-1 (Fig. 4), so the gain arises in consolidation, not from candidate quality.

ResourceSceneTAP-AdaptedCueClimb
Candidates1,0251,025
Two-cue combination evaluations2,7792,779
Deployed cues / case22
Generation requests3,2733,265
Generation tokens~8.09M~7.90M

(2) Baseline Adaptation:

  • Text generation: Scene analysis, text design, and the plan→adjust step are kept. The VQA objective is replaced by the same target semantics CueClimb uses, so attacker knowledge is identical; text length is unified to 4–5 words.
  • Placement: SAM/SoM regions and placement planning are retained on the same valid surfaces as CueClimb.
  • Image blending: TextDiffuser-2 is 2D-only, so both methods share a 3D label renderer, also controlling for rendering quality.
  • Combination search: Random combination search, with candidate counts and combination-evaluation budgets matched to CueClimb per case.

(3) Ablation: "w/o joint combination scoring" still deploys two cues and removes only the combination search. It keeps the same atomic cue pool, scores every valid two-cue combination by the sum of its single-cue scores, keeps the top 5, and applies the same cross-context selection as the full method.

We have added these details to Appendix A.

Follow-up to Q2: Additional controls for observation robustness

We evaluated the clean condition and SceneTAP-Adapted on the same 30 case–seed pairs reported in Q2 (10 cases × 3 seeds), following the same observation collection protocol as in the CueClimb evaluation. On the original observations for these same 10 cases, both methods achieved 50.0% ASR (5/10).

ConditionASR
Clean (shared)1/30 (3.3%)
SceneTAP-Adapted7/30 (23.3%)
CueClimb9/30 (30.0%)

CueClimb remains more effective than SceneTAP-Adapted in this evaluation, achieving an ASR of 30.0% versus 23.3%, a 6.7 percentage point advantage.

Dear Reviewer z9K4,

Thank you again for taking the time to review our paper. Your feedback has been very helpful to us. With less than 5 hours left in the discussion period, we hope you don’t mind us following up once more. We would greatly appreciate even a brief note on whether our response and revisions have addressed your concerns.

Best regards, The Authors

Reviewer_zMSiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies poisoning of hierarchical embodied memory through visual cues encountered during mapping. CueClimb constructs locally plausible textual cues, optimizes cue combinations using leaf and ancestor semantics, and selects a deployment using auxiliary contexts from disjoint scenes. The attacker is given a target semantic concept rather than the exact future query. Evaluation measures retrieval of an attacker-designated wrong destination, rather than completed robot navigation. On 99 cases from 23 HM3D scenes, CueClimb reports 46.5% full-path attack success versus 28.3% for SceneTAP-Adapted. Additional experiments examine component ablations, frozen-attack transfer, query reformulations, defenses, and a controlled physical demonstration.

Critical strengths
  1. The paper identifies a consequential failure mode: locally observed misleading semantics can survive recursive consolidation and influence later global retrieval. The evidence connects the failure to memory hierarchy rather than merely showing that a captioner can read adversarial text. CueClimb and SceneTAP-Adapted both attain 100% leaf Rank-1, yet CueClimb retains more success at higher levels (Figure 4). Removing the ancestor-summary term reduces ASR from 46.5% to 39.4%, and removing cross-context selection reduces it to 38.4% (Table 2).

  2. The validation extends beyond the main embedding-based router. Appendix E evaluates LLM selection procedures adapted from the original retrieval systems, with 52.5% versus 41.4% ASR on the main memories. Frozen attacks also transfer across victim configurations and new query formulations; Appendix H demonstrates the mechanism with physically placed labels. These checks make the threat more credible, while the defense experiments provide actionable evidence that consolidation rules matter.

Critical weaknesses
  1. The comparison against the strongest adapted baseline needs clearer resource accounting. Equal cue count and comparable access to the target concept are established, but Appendix A specifies a substantial candidate-search and auxiliary-evaluation budget for CueClimb without documenting an equivalent budget for SceneTAP-Adapted. Consequently, the 18.2-point advantage cannot yet be cleanly attributed to consolidation-aware design rather than differences in optimization effort or adaptation. This does not negate the vulnerability demonstration, but it limits confidence in the claimed method advantage.

  2. The defense utility result has a narrower interpretation than the main table initially suggests. Appendix B.3 retains 111 benign queries from 644 candidates after clean-success filtering and deduplication, spanning seven scenes; thus the 100% original clean utility is a selected-cohort baseline. The 96.4% strong-defense result measures retention on that cohort, rather than absolute utility on a representative query distribution. This is disclosed in the appendix, but the denominator and selection should be explicit beside Table 4, and broader utility accounting would strengthen the practical defense conclusion.

Questions for authors
  1. Clarification: What candidate-generation, captioning/summarization, and selection budgets were used for SceneTAP-Adapted, and which parts of its original procedure were retained? Please provide a comparable resource breakdown and, if already available, a matched-budget result. A scene-clustered paired interval for the main CueClimb-minus-SceneTAP ASR difference would also help interpret the advantage; Appendix F's interval addresses a different contrast.

  2. Clarification or additional analysis: Can Table 4 explicitly identify its clean-success-selected cohort and distinguish the 111-query clean denominator from the 679 query–attack pairs? If broader results are available, how does Strong affect utility before clean-success filtering, with duplicate queries handled consistently? This would determine how broadly the utility-preservation claim can be interpreted.

2 comments

We thank the reviewer for the constructive feedback and thoughtful questions. We address each point below:

Q1: SceneTAP-Adapted Resource Accounting and Scene-Clustered Paired Interval

(1) Models and Budget: CueClimb outperforms SceneTAP-Adapted under matched search budgets and comparable generation costs. Both methods use GPT-5.5 for candidate generation and GPT-4o for search evaluation. The table summarizes 20 randomly sampled cases; even including cross-context deployment selection, CueClimb uses ~0.19M fewer generation tokens.

ResourceSceneTAP-AdaptedCueClimb
Candidates1,0251,025
Two-cue combination evaluations2,7792,779
Deployed cues / case22
Generation requests (model calls)3,2733,265
Generation tokens~8.09M~7.90M

(2) Baseline Adaptation: The specific adaptations are as follows:

  • Text generation: Scene analysis, text design, and the plan→adjust step are kept. The VQA objective is replaced by the same target semantics CueClimb uses, so attacker knowledge is identical; text length is unified to 4–5 words.
  • Placement: SAM/SoM regions and placement planning are retained on the same valid surfaces as CueClimb.
  • Image blending: TextDiffuser-2 operates on 2D images and does not directly support real-time rendering in Habitat. Both methods therefore share a 3D label renderer, also controlling for rendering quality.
  • Combination search: SceneTAP-Adapted uses random combination search, with candidate counts and combination-evaluation budgets matched to CueClimb per case.

(3) Paired Statistics: Scene-clustered paired analysis supports a statistically significant ASR advantage for CueClimb. On the same 99 cases across 23 scenes, CueClimb achieves an 18.2-point advantage (46.5% vs. 28.3%).

  • Per case: only CueClimb succeeds in 27 cases, only SceneTAP-Adapted in 9, both in 19.
  • Scene-clustered 95% CI of the difference: [8.6, 27.8] points.
  • Tests: scene-level permutation p = 0.006; case-level exact McNemar p = 0.0039.
  • Per scene: CueClimb leads in 12 scenes, ties in 9, and trails in 2.

These details are now added in Section 5.2 and Appendix A.

Q2: Clean-Success-Selected Cohort and Broader Defense Utility

(1) Denominators and Filtering in Table 4: As you correctly note, Table 4 measures utility retention on a clean-success-selected cohort. We will make the selection and denominators explicit in its caption: "This set is obtained from 644 candidates by filtering for original clean retrieval success and within-scene deduplication; the denominator of clean utility is 111 queries, and that of poisoned utility is 679 query–attack pairs."

(2) Utility beyond the Filtered Set: Strong maintains comparable utility to no defense on an additional query set without clean-success filtering. We independently generated 30 queries that are particularly challenging for the defense, 10 per category, with humans judging whether the returned location satisfies the query. As shown below, Strong achieves 76.7% utility versus 73.3% for no defense, with at most a one-case difference per category, providing additional evidence of utility preservation beyond the selected cohort.

Query categoryNo defenseStrong
Legitimate signage text7/106/10
Sparse legitimate cues8/109/10
True function of attack target location7/108/10
Overall73.3%76.7%

These details and results are now added in the Table 4 caption, Section 5.6, and Appendix J.

Dear Reviewer zMSi,

Thank you again for taking the time to review our paper. Your feedback has been very helpful to us. With less than 5 hours left in the discussion period, we hope you don’t mind us following up once more. We would greatly appreciate even a brief note on whether our response and revisions have addressed your concerns.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.