acceptodds
Under review as a conference paper at ICLR 2027

HAE-GEO: Benchmarking Deep-Search Agents under Hierarchical Web Evidence Poisoning

Abstract

Search-augmented LLM agents increasingly mediate consumer decisions, making them targets of Generative Engine Optimization (GEO) poisoning: fabricated reviews, rankings, and endorsements published online so that generative search systems retrieve and trust them. Existing benchmarks measure whether manipulated content is retrieved or endorsed, but not how agents handle poisoned evidence after exposure. We introduce HAE-GEO, a benchmark that holds a fabricated brand and its false claim fixed while escalating how the claim is presented, from direct assertion (L1) to contextual camouflage (L2) and apparent corroboration across multiple pages (L3), within a controlled corpus of 72,039 clean pages and 770 poisoned pages per level. Agents interact through a Search–Scrape interface that separates seeing a search result from reading the full page; each trajectory is mapped to five observable states (exposure, adoption, verification, recovery, endorsement) and scored with a six-dimensional rubric. Across ten agents, we find that (1) poison recognition declines from L1 to L3, and the decline persists even when page type and length are matched—a pattern we call the corroboration trap; (2) agents rarely obtain independent evidence after adopting a fake brand, while defense prompting increases both verification attempts and strict evidence acquisition; (3) iterative search lowers fake-brand endorsement without consistently improving recognition, whereas explicit reasoning improves both; and (4) defense prompting raises false rejection of real but little-known brands from 11.4% to 18.0%. Evaluating Web agents therefore requires examining how they verify evidence, not only what they recommend. HAE-GEO provides a diagnostic testbed for building agents that verify, rather than merely avoid, suspicious evidence.

Primary area:
datasets and benchmarks
Keywords:
Deep-search agents, Generative Engine Optimization (GEO), Web evidence poisoning, Trajectory-aware evaluation, LLM agent safety, Evidence verification and recovery, Benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BXtoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what search agents do after reading fabricated promotional content about products. The authors build a closed Chinese-language web environment with about 72k real pages, 8 product categories and 154 made-up brands. For each made-up brand and false claim, they write poisoned pages at three levels: a bare advertisement, the claim embedded in a review, and supporting “evidence” spread across several pages that appear to back each other up. Search returns only titles, and the agent has to open a page to read it, so the authors can log which poisoned pages each agent read, whether it tried to verify the brand, and whether it recommended it in the end. They run 10 agents on 240 queries under a standard prompt and a cautionary prompt, and use 64 queries on real but little-known brands to measure false rejection. They find that recognition gets worse as the poisoning looks more corroborated, and that agents rarely go back to verify a fake brand once they have accepted it. Extra search rounds reduce endorsement without improving recognition, while explicit reasoning improves both. The cautionary prompt also raises false rejection of real brands from 11.4% to 18.0%.

Critical strengths
  1. The design holds the brand and the false claim fixed and changes only how the claim is presented, so the three levels can be compared directly. Even with page type and length matched (App. A.10), L3 still raises the fooled rate by about 13 points over L1, which I find fairly convincing.

  2. Page text is only visible after Scrape, so the authors can separate “never saw the poisoned page” from “saw it and wasn’t persuaded.” This is what lets them show that extra search rounds lower the fooled rate without making agents any better at spotting fake evidence (Sec. 5.3).

  3. The control set of real, little-known brands shows what the defense costs: with the cautionary prompt, false rejection rises from 11.4% to 18.0%, and it rises for every model. Most work in this area reports only the drop in attack success.

Critical weaknesses
  1. Whether an agent counts as “fooled” depends on whether the brand was made up, not on whether the evidence it saw was enough to tell. If an L3 fake brand looks as well supported as a real one, recommending it is a reasonable call. The real brands in the control were also chosen because they have several independent sources, so FR and FRR are measured on problems of very different difficulty. I would like to see a human baseline with the same tools, or real brands wrapped in the same L1–L3 promotional pages.

  2. The fake brands have no trace at all in the clean corpus, and it is not clear whether the environment contains anything like business-registration or trademark records. The low verification yield may therefore say more about the environment than about the agents. The claim that agents rarely re-examine after adoption also rests on only 57 trajectories.

  3. L3 changes several things at once, including cross-page corroboration and how official individual pages look. It is not yet clear that corroboration is what drives the effect the paper calls the “corroboration trap.” A simple test would be to put the same content on a single page and compare it with spreading the content across several pages.

Questions for authors

see weaknesses

1 comment

Thank you for the thoughtful response and the additional experiments. However, my core concern remains: it is still unclear whether the fabricated evidence is distinguishable from genuine evidence in this setting, and thus whether the results reflect a real weakness of the agents. I will therefore keep my current score.

Reviewer_Hf5gOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes HAE-GEO, a benchmark for evaluating deep-search agents under Web-evidence poisoning in product-recommendation tasks. It considers three poisoning levels: direct assertion (L1), contextual camouflage (L2), and evidence-enhanced corroboration (L3). In addition, the authors use an LLM-based judge to evaluate agent trajectories along six dimensions. Unlike prior work that measures only final attack success, HAE-GEO decomposes each trajectory into exposure, adoption, verification, recovery, and endorsement. The authors evaluate ten proprietary and open-source LLMs. Their main findings are that increasingly corroborated evidence degrades evidence-risk handling and poison recognition, while a defense prompt increases verification but does not consistently translate verification into recovery.

Critical strengths

S1. The paper studies an important and timely threat: adversarial Web content can influence search-augmented agents without relying on explicit prompt injection. This threat is highly relevant to recommendation and deep-research agents.

S2. The trajectory-aware evaluation provides a more fine-grained analysis of evidence-poisoning attacks against deep-search agents. Separating exposure, adoption, verification, recovery, and final endorsement provides substantially more diagnostic information than reporting attack success or final endorsement alone.

Critical weaknesses

[Threat Model Mismatch] The threat model conflates two distinct settings. In a substantial subset of queries, the user directly names a fabricated brand, for example, by asking the agent to compare a real brand with one or more fake brands. This seeds the target into the agent’s search process: the agent will naturally search for evidence about the named brand and may treat it as a legitimate candidate before encountering poisoned Web content. This differs from the more realistic discovery-style GEO threat model, in which a user asks for a product recommendation without mentioning the fake brand, and poisoned Web content causes the agent to discover and recommend it.

[Confounded Evaluation Results] Because the user query already includes the fabricated brand, the agent’s endorsement of that brand may simply result from an attempt to be helpful: it searches for evidence about the user-provided candidate and answers the comparison question based on the retrieved evidence. Consequently, it is difficult to attribute the endorsement solely to Web poisoning in the deep-search agent. The observed effect may instead reflect a combination of Web poisoning and the user’s framing of the fabricated brand as a legitimate candidate.

Questions for authors

Q1. Do all user queries include the fake-brand name? If not, what proportion of queries do not mention the fake brand? For this subset, please report the corresponding results, including M6, L1/L2/L3 FR, M1, M2, and M3.

Q2. The paper mentions eight attack page types. Could you provide results broken down by attack page type? This analysis would help clarify whether the effectiveness of the poisoning attack varies across different types of poisoned content.

Reviewer_7mwtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces HAE-GEO, a controlled benchmark for evaluating how deep-search agents respond to poisoned web evidence promoting fabricated brands. It holds the target brand, false claim, and poisoning budget fixed while varying evidence presentation across three levels:

  • L1 — Direct assertion: The false claim is stated directly.
  • L2 — Contextual camouflage: The claim is embedded in a plausible buying guide or review.
  • L3 — Apparent corroboration: Ratings, reports, and certifications distributed across seemingly independent pages appear to support the false claim.

The main contributions are the hierarchical search environment and a trajectory-aware evaluation protocol that tracks exposure, adoption, verification, recovery, and endorsement. These are complemented by six semantic scoring dimensions and a control measuring false rejection of legitimate, lesser-known brands.

Experiments across ten models show lower average poison-recognition scores at L3 than at L1, while defense prompting improves resistance but increases false rejection of legitimate brands. Separate comparisons on two models show that iterative search reduces fake-brand endorsement relative to a static baseline that receives the top-ranked page contents from a single search and answers immediately, without consistently improving poison recognition. Validation includes page-type- and length-matched controls, retrieval-opportunity analyses, bootstrap confidence intervals, and agreement studies comparing the rule-based states and LLM judge scores with human annotations.

Critical strengths
  1. The L1–L3 comparison provides a structured analysis of vulnerability to different evidence presentations. By holding the fabricated brand, false claim, and poisoning budget fixed, the benchmark compares direct assertion (L1), contextual camouflage (L2), and apparent cross-page corroboration (L3). This goes beyond reporting attack success under a single condition and helps identify which forms of evidence presentation make manipulation harder for models to recognize.

  2. The evaluation examines verification behavior and defense costs beyond final recommendations. It tracks adoption, verification, and recovery while also measuring false rejection of legitimate, lesser-known brands. This helps assess whether reduced fake-brand endorsement accompanies better evidence recognition and whether it comes at the cost of rejecting legitimate brands.

Critical weaknesses
  1. The fabricated-brand setting limits what the benchmark establishes about evidence verification. Because every attack target is nonexistent, verifying brand existence may help avoid the attack without assessing the truth of individual product claims. The paper does not establish whether its findings extend to false performance or certification claims about real brands, where existence checks are insufficient and recommending the brand is not necessarily an error. This limits the interpretation of benchmark performance as a measure of broader resistance to manipulated web evidence.

  2. Independent reproduction of the reported results remains limited. The review supplement provides illustrative samples and a runnable demonstration, but not the complete experimental corpora or original retrieval backend. It also provides a language-adapted evaluator without clearly establishing its correspondence to the frozen evaluation configuration used for the reported results. The post-acceptance release commitment includes the poisoned corpora, retrieval checkpoint, and clean-corpus URL manifest, but does not explicitly include the frozen trajectory parser and rule vocabulary used for the reported results or guarantee access to the complete clean-page bodies. Although the paper reports internal validation, these gaps make it difficult to independently verify the measurements and reuse the benchmark under the reported conditions, which is consequential for a benchmark contribution.

Questions for authors
  1. The paper acknowledges that withholding a recommendation after finding no reliable evidence can be appropriate even when no strict independent evidence is acquired. Can the reported analysis separately identify how often this occurs?

  2. The Defense prompt already encourages checking official websites and legal-entity records. Have you evaluated a baseline that makes an explicit brand-existence check mandatory before recommendation? If not, I suggest this comparison as an additional experiment to assess the value of a dedicated verification step.

  3. False claims about real brands cannot be resolved through existence checks alone. Could you discuss the extent to which the findings from fabricated-brand poisoning might apply to this setting?

  4. The released evaluator uses an English rule vocabulary and is described as language-adapted. Could you clarify whether the release plan, which lists the poisoned corpora, retrieval checkpoint, and clean-corpus URL manifest, also includes the frozen trajectory parser and rule vocabulary used for the reported results?

  5. Because access to clean-page bodies is not guaranteed, the URL manifest alone may not allow reconstruction of the original retrieval environment. Could you clarify which headline results an independent researcher would be able to reproduce using the planned release?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

For clarity, we refer to Reviewers 7mwt, Hf5g, and BXto as R1, R2, and R3, respectively.

We sincerely thank all reviewers for their thoughtful and constructive feedback. We have revised the manuscript with additional experiments, analyses, and clarifications, including new results addressing R1’s core methodological concerns and R2’s follow-up questions. Below, we summarize our contributions and updates.


Core Contributions of Our Work

  1. Controlled Hierarchical Benchmark: HAE-GEO holds fabricated targets, false claims, and poisoning budgets fixed while varying evidence packaging across three levels. Code and benchmark resources are available through our anonymous repository.
  2. Trajectory-Aware Evaluation: We track exposure, adoption, verification, recovery, and endorsement, complemented by six semantic dimensions and a legitimate-brand control.
  3. Systematic Empirical Analysis: Experiments across ten agents reveal the corroboration trap, distinguish reduced endorsement from improved recognition, and quantify the over-refusal cost of defense prompting.

Updates to Experimental Results during Rebuttal

  • Main Text and Appendix D.1: Added results separating queries with and without fabricated-brand names (R2).
  • Main Text and Appendix D.2: Added an identical-content single-page versus multi-page control (R3).
  • Main Text and Appendix D.3: Added a real-brand false-claim pilot covering 60 queries and two models (R1, R3).
  • Main Text and Appendix D.4: Added a dedicated brand-existence-check baseline and its tool costs (R1).
  • Appendices D.5–D.6: Added page-type breakdowns and a target-specific source-coverage audit (R2, R3).
  • Appendix D.7: Added counts of suspicion warnings, insufficient-evidence statements, and recommendation refusals under Clean and L1–L3 for queries naming fabricated brands (R2, follow-up).
  • Appendix D.8: Added unconditional endorsement rates for both query subsets and paired Clean–poisoned differences with normal-approximation 95% confidence intervals (R2, follow-up).
  • Appendix D.9: Added supplementary ablation and diagnostic experiments to address R1’s concerns regarding causal attribution and evaluation methodology (R1).

Updates to In-Depth Discussions during Rebuttal

  • Main Text: Clarified query framing, entity-level versus claim-level verification, and the interpretation of FR and SVY (R1–R3).
  • Main Text and Appendix D.2: Refined cross-page corroboration as a contributing factor rather than the sole explanation of the hierarchy effect (R3).
  • Reproducibility Statement and Appendix A.5: Added the anonymous code link and clarified the frozen evaluator, released resources, and reproduction limits (R1).
  • Appendices D.7–D.8: Distinguished protective language from successful resistance, clarified UER versus FR denominators, and refined attribution beyond “user framing alone” (R2, follow-up).
  • Appendix E: Updated limitations concerning matched comparisons, evidence availability, pilot scope, and the pending human baseline (R1, R3).
  • Main Text & Appendix B: Expanded discussions on causal identification and confounding factors raised by R1, together with careful caveats for interpreting benchmark outcomes (R1).

All additions and revisions are highlighted in blue for ease of reference. The latest revision incorporates our responses to R1’s methodological questions and R2’s November 14 follow-up. Our manuscript is updated on Nov 16, AOE time.

We remain grateful for the reviewers’ feedback and welcome further discussion.

AuthorsOpenReview ↗

General Response*

Dear Area Chair,

We posted our responses to the reviewers’ comments on 13 Nov AOE time. As of today, we have not yet received follow-up from reviewers 7mwt,and BXto

With the discussion deadline approaching, could you please send them a reminder to review our responses and engage in the discussion? We would appreciate the opportunity to clarify any remaining concerns before the discussion closes.

Thank you for your time and help.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.