- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes “on-policy attacks” on LLM ranking agents. A listing truthfully disclaims a fictitious concept, and planted web content links that concept to harm. An agent that searches for the concept may then rank the disclaiming item higher. The authors evaluate 12 models on four listing datasets under claim-only, in-context snippet, and mocked search conditions. They also report search intent, reasoning analyses, and a platform-policy survey.
Critical strengths
- The attack setting is interesting.
- The evaluation is comprehensive.
Critical weaknesses
- The realistic attack is not shown end to end. There is no real search engine, results page, page fetching, or live planted pages. The search condition is a fixed mocked query that returns one hand-written snippet, so retrieval of the planted content is assumed rather than demonstrated.
- The setup encourages search. The system prompts tell the agent to verify the ingredients or provisions involved, which pushes it to search for exactly the injected concept.
- No defenses are evaluated. The paper argues that existing defenses would fail but tests none of them, not even simple source-assessment prompting. This makes the severity of the vulnerability hard to judge.
- The targets (e.g., Opus 4.8) are older than the annotators the paper itself uses (Opus 5, Opus 5.5). Testing current models would strengthen the claims.
Questions for authors
See weaknesses above
2 comments
Thank you again for taking the time to review our work. We know this is a busy period. Since the discussion closes tomorrow (November 18), we wanted to briefly follow up on our rebuttal. In response to your review, we:
-
W1:
- Clarified practical considerations in conducting end-to-end experiments on the web: see general comment 1/3
- Performed a live end-to-end run using Gemini 3.1 Pro: see bottom of BXyK comment
-
W2: Shared grounded motivation for our search intent queries and ran additional experiments using a domain-agnostic prompt: see general comment 2/3
-
W3 + W4: Added additional defense and newer model experiments: see general comment 3/3
We hope these address your concerns. If so, we’d appreciate it if you’d consider updating your score. If anything is still unclear, we’re happy to clarify before the deadline.
We thank the reviewer for taking the time to review our work, for finding our setting interesting, and for recognizing the scope of our evaluation.
W1: on realism
We address the realism of our attack recipe in the general comment. We clarify that the backstory snippets are automatically generated: see Appendix C for considerations and Appendix I for steps and prompts used, e.g., Table 19, Figure 24, Figure 25, and Table 21.
W2: on search intent
We address search intent with additional ablations in the general comment, including a domain-agnostic prompt and native web-search tools.
W3: on defenses
We provide additional source-assessment prompting experiments in the general comment. We note that a motivated attacker could optimize both their description and online presence toward specific agents and use cases, making it exceedingly difficult to defend against. As we note in Section 6 “Recommendations for platform providers“, while it may not be feasible to check every listing for their likelihood to steer consumer agents, new guidelines and the threat of spot checks may suffice.
W4: on testing newer models
Per the reviewer’s request, we added additional ablations for Claude Opus 5.5 in the general comment, Table C. We note that this newer Claude Opus version remains steerable, although less so than 4.8.
We hope these clarifications and new results address the reviewer’s concerns, and would be glad to discuss further