- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates item-side security risks in agentic recommender systems, focusing on whether attackers can manipulate recommendation rankings by modifying publicly available item information, such as titles, descriptions, and reviews. To examine this problem, the authors introduce AttackRecBench, a benchmark designed to systematically compare different attack strategies under consistent evaluation settings. The framework assesses attack effectiveness from two perspectives: the promotion of targeted items and the preservation of overall recommendation quality. The experiments span three data sources, Amazon, Goodreads, and Yelp, and consider multiple attack techniques, recommendation workflows, and underlying language models. In addition, the paper evaluates several lightweight defense mechanisms. The findings suggest that prompt injection is particularly effective under the main experimental setting, although attack performance and defense effectiveness vary depending on the specific configuration.
Critical strengths
- The paper investigates target-item promotion through modifications to public content, without requiring access to user histories, victim model internals, or victim feedback. The experimental results demonstrate that attackers can significantly influence recommendation outcomes even with limited control. This reveals potential security vulnerabilities in real-world recommender systems and highlights the practical significance of the study.
- The paper jointly evaluates defense effects on target-item promotion and recommendation quality, showing that security benefits and utility costs vary across attacks. This provides empirical evidence about security–utility trade-offs and makes the benchmark useful for comparing and selecting defense strategies.
Critical weaknesses
- The paper states: “An agent that ignores useful evidence may appear robust even though it is simply a weak recommender.” Yet Figure 2(b) reports relative changes without complete absolute clean/attacked recommendation quality. AdaptiveTool’s 11.6% source exposure and small attack effects could reflect either effective selective retrieval or insufficient evidence use. MACRec also uses a fixed six-stage workflow without Searcher, Interpreter, or dynamic retrieval, while evidence access and call budgets differ across architectures. Capability validation and mechanism controls are therefore needed to interpret these security–utility differences.
- The paper defines memory poisoning as attacked content entering writable working memory, yet Table 3 provides no WMP-specific memory-carry rate, leaving unclear how many WMP executions satisfy this definition. Both memory architectures also retain the original attacked evidence during ranking, so target promotion alone cannot establish the effect of memory contamination. Moreover, the evaluated memory is task-local and excludes existing cross-user collaborative and retrieval-based designs.
Questions for authors
- Could the authors provide absolute clean/attacked NDCG@3 and ER@3 by architecture and dataset for Figure 2(b), alongside actual model and tool calls, to clarify whether AdaptiveTool’s small attack effects coexist with competitive recommendation quality?
- Which attacks and denominators underlie Table 3’s Memory metric, and how is memory carry identified? For MemoryAgent and Agent4Rec, please report WMP execution and memory-carry counts, clarifying whether the retained content includes the attacker-induced favorable association rather than ordinary item information.
- Could the authors report AdaptiveTool’s paired clean/attacked access rates to the same target source, and distinguish unexposed executions that submit rankings before exhausting the tool budget from those that exhaust it? This would clarify how retrieval decisions and budget constraints relate to the low exposure rate.
4 comments
PART II: Response to Reviewer 5eLA — Paired Target Source Access Rate and Budget Exhaustion Analysis (Q2)
Q2: Paired target source access rate and tool budget exhaustion analysis for AdaptiveTool.
A2: Following your suggestion, we re-ran AdaptiveTool under paired clean and attacked conditions on the 300-task subset across all 11 attacks (3,299 valid attacked runs) and analyzed the retrieval logs.
1. Paired Target Source Access Rate For the 9 description-based attacks, AdaptiveTool accessed the target item's description in 3.7% (11/300) of clean runs and 3.5% (95/2,699) of attacked runs. Across all 2,699 (task × attack) pairs, 68 accessed the target description only when clean, 64 only when attacked, 31 in both, and 2,536 in neither. Injected text does not alter AdaptiveTool's propensity to inspect target descriptions.
For the remaining two attacks: LP modifies item titles, which are always visible in the initial candidate list (300/300 attacked runs exposed). PRI modifies reviews; the injected review appeared in tool outputs in 8 of 300 runs (2.7%, with the target reaching top-3 in 5 of those 8) and did not appear in the remaining 292 runs (where the target reached top-3 in 41; given the small sample, we report these counts descriptively).
2. Termination Reasons and Budget Breakdown for Unexposed Runs (87.8%) Across the 3,299 attacked runs, the target source was exposed in 403 runs (12.2%, close to the 11.6% full-dataset rate in Table 3: 300 LP, 95 description-based, and 8 PRI). Among the 2,896 runs (87.8%) where the target remained unexposed:
- 98.3% (2,848 runs) voluntarily called
submit_rankingbefore exhausting the 10-call tool budget, using ~2.7 data-tool calls on average; - Only 1.7% (48 runs) exhausted all 10 tool calls before final submission, and zero runs hit the 30-turn model interaction limit.
Thus, unexposed runs happen mostly because the agent stops retrieving early, not because it runs out of budget. Importantly, this does not mean AdaptiveTool is inherently robust to attacks. Across the 9 description-based attacks, ER@3 is 40.0% (38/95) when the target description is read versus 13.3% (346/2,604) when unread (an observational comparison). When we force candidate descriptions into context, prompt-injection attacks spike in ER@3: DRI rises from 13.7% to 78.3%, COI from 13.3% to 76.3%, and BSI from 14.7% to 40.0%, whereas the decision-bias attack BPI (authority/brand-prestige cues) barely moves (13.7% vs. 13.0% standard). The low overall attack effect therefore comes from reduced exposure via selective evidence access, not robustness against prompt injection.
PART III: Response to Reviewer 5eLA — Memory Metric Definitions, WMP Carryover, and Counterfactual Check (Q3)
Q3: Table 3 memory metric definitions, WMP-specific carryover rate, and counterfactual memory-only check.
A3: Thank you for highlighting these three issues regarding Table 3's memory metrics. We fully agree that WMP-specific carryover rates were not isolated, retaining original attacked evidence during final ranking prevented separating memory pollution from direct evidence exposure, and our evaluation covers task-local working memory. We address each below.
1. Table 3 Denominators and Carryover Criteria
The Memory column in Table 3 (MemoryAgent 0.7%, Agent4Rec 2.6%) uses total attacked runs across all 11 attacks per architecture as its denominator (Appendix D.1) and applies only to MemoryAgent and Agent4Rec, which maintain explicit writable memory slots (marked – for the other three architectures). Following Appendix D.1, "memory carryover" is logged whenever content from an attacked field is written into a working-memory slot. Because this structural criterion does not distinguish attacker-induced endorsements from benign item attributes, we added a content-level inspection below.
2. WMP-Specific Execution and Carryover Statistics (Appendix D.3, Table 5) Re-running WMP alone on the 300-task subset (Appendix D.3, Table 5), memory carryover occurs in 11/300 runs (3.7%) for MemoryAgent and 10/299 runs (3.3%) for Agent4Rec. Because these rates use WMP-only runs as the denominator whereas Table 3 pools all 11 attacks, the two sets of numbers are not directly comparable.
To inspect what was actually written across all 21 carryover cases, we checked both verbatim copying (any contiguous 5-word sequence from the attack text appearing verbatim in memory: 5/11 for MemoryAgent, 1/10 for Agent4Rec) and semantic adoption via unified manual annotation (whether memory explicitly names the target item and adopts attacker-induced claims such as "strong match," "reliable," or "prioritize"). Under manual review, adversarial content carryover holds in 10/11 cases for MemoryAgent but only 1/10 for Agent4Rec (the other 9 logged entries for Agent4Rec never mention the target item and contain only generic user-preference summaries).
3. Counterfactual Memory-Only Check As noted in Appendix D.1, descriptive carryover rates do not by themselves prove that memory pollution drives ranking shifts. To isolate the memory pathway, we ran a counterfactual check where, after memory is written, final ranking retains only the poisoned memory while restoring all item evidence to clean text.
- MemoryAgent (300-task subset): With poisoned memory alone, target ER@3 under WMP reaches 20.1% (vs. 12.8% clean baseline and 33.3% standard attack), showing that the memory pathway alone produces a measurable exposure gain, though smaller than the full attack.
- Agent4Rec (60 item cold-start tasks): Because the remaining 240 tasks rarely trigger the two-stage "write memory → re-rank" flow, the counterfactual is only meaningful on these 60 tasks. Here, WMP ER@3 is 11.7% (clean), 16.9% (standard attack), 8.5% (clean evidence + poisoned memory), and 11.7% (fresh-session control). Paired differences are statistically indistinguishable from noise (memory-only vs. control: −3.4, 95% CI [−10.2, +1.7]; standard attack vs. clean: +5.1, 95% CI [−1.7, +13.6]; n ≈ 59–60). We find no discernible causal effect through the memory pathway in Agent4Rec and make no claim on whether memory is its primary attack channel.
- Scope boundary: Our study evaluates task-local working memory; cross-user collaborative memory and retrieval-based long-term memory are outside our scope, and we explicitly refrain from extrapolating our findings to those settings.
Dear Reviewer 5eLA,
Thank you very much again for your careful reading and the precise, diagnostic questions — they have substantially improved the paper.
As the discussion period draws to a close today, we wanted to check whether any of your questions remain unaddressed, or whether anything in our responses or the revised manuscript has raised new confusion. If so, please let us know — we will respond and resolve it right away, while there is still time.
Thank you again for your time and consideration.
PART I: Response to Reviewer 5eLA — Absolute Metrics, Call Budgets, and Architectural Controls (Q1)
Thank you for recognizing our black-box threat model (which requires no access to user histories, model parameters, or feedback) and our empirical analysis of lightweight defense trade-offs. We organize our response across three parts (Part I: Q1; Part II: Q2; Part III: Q3) based on our full execution logs (Appendix B.2) and new controlled experiments.
Q1: Absolute performance metrics corresponding to Figure 2(b), actual call budgets, and architectural capability controls.
A1: Thank you for raising these important points, which highlight two key issues: testing the "weak-recommender" explanation via absolute metrics and actual call counts, and clarifying capability controls across architectures.
1. Absolute Performance and Actual Call Overhead (Appendix D.2, Table 4) In Appendix D.2 of the revision, we added Table 4 reporting absolute Clean/Attacked NDCG@3, absolute Clean/Attacked ER@3, and executed model calls (MC) and data-tool calls (TC) across all five architectures on Amazon, Goodreads, and Yelp (under configured limits of 10 tool calls and 30 model turns).
2. Architectural Capability and Mechanism Control Boundaries We agree that these implementation details belong in the main paper and have moved them from Appendix B.1 into Sections 4.1.2 and 4.2.2. Specifically,
MACRec-styleadapts MACRec to our closed-set candidate reranking setting via a fixed six-stage workflow (manager plan → user analyst → item analyst → manager draft → reflector → manager final). We intentionally exclude the upstream framework's Searcher, Interpreter, and dynamic retrieval loops for two reasons: (i) in closed-set reranking where candidate metadata is already structured and bounded, external search/code-execution tools are unnecessary, and (ii) keepingMACRec-styleon full-context prefetching isolates the effect of multi-role deliberation and reflection without confounding it with on-demand tool retrieval (tested separately in AdaptiveTool and Agent4Rec). Accordingly, we explicitly label this variantMACRec-style, avoid extrapolating to the full upstream framework with dynamic Searcher enabled, and qualify cross-architecture differences as reflecting the joint effect of evidence access channels and state/collaboration budgets rather than a single isolated factor.