acceptodds
Under review as a conference paper at ICLR 2027

Beyond Semantic Matching: Evidence-Grounded Multi-Agent Product Deep Search

Abstract

Modern product search excels at semantic matching, but realistic shopping queries often demand complex, multi-condition requirements—spanning appear- ance, specifications, compatibility, price, and user reviews—that holistic relevance scores alone cannot reliably guarantee. We introduce Product Deep Search, a benchmark and paradigm for evidence-grounded product retrieval. Candidate products are represented through structured product-page evidence, and models use tools to retrieve supporting or contradictory evidence across heterogeneous sources. To solve this problem, we propose PDeepSearch, a multi-agent frame- work that turns semantic search into evidence-grounded decision making. Start- ing from candidates produced by multimodal retrieval and reranking, PInspect independently examines each product with evidence-access tools and constructs condition-level profiles of support, uncertainty, and contradiction; PDecide then compares these profiles across candidates to make the final selection. The frame- work can be instantiated with both API-based and open-weight models, and con- sistently improves strong zero-shot backbones by 2.43–7.29 Hit@1 points. We further propose PDeepSearch-9B, a product search agent trained by synthesiz- ing inspection and decision trajectories. PDeepSearch-9B achieves 69.43 Hit@1, outperforming larger open-weight models, including Qwen3.5-27B (63.71) and Gemma-3-27B-IT (62.86), while approaching competitive top-tier models like Gemini-3.5-Flash and Kimi-K2.6. Further analysis shows that evidence verifica- tion acts as an effective correction layer over semantic reranking, with the largest absolute gains observed on queries combining a reference image with visual, de- tail, and review-grounded conditions. These results highlight evidence-grounded verification as a key capability beyond semantic matching for complex product search, and show that it can be effectively learned by compact models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Product Retrieval, Multi-Condition Search, Multi-Agent

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_GLaEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Product Deep Search, a 700-query benchmark of multi-condition shopping requests over ~80K Amazon products. In addition, the authors also propose a method, PDeepSearch, which assigns one tool-using LLM "Inspector" to each of the reranker's top-10 candidates; each Inspector labels every query condition as exact match, mixed, unknown or contradicted. A Decision agent then picks one product from the ten profiles. The authors also report gains over a single-agent baseline for five backbones, and a fine-tuned 9B model that reaches 69.43 Hit@1.

Critical strengths
  • The new benchmark is built to require condition-level checks, with proper leakage controls. In particular, a benchmark that considers evidence checking over embedding similarity is a reusable contribution to the community.

  • The empirical results demonstrate that per-condition verification fixes errors in the traditional semantic re-ranker in practice.

Critical weaknesses
  • From the experiments in the paper, it is not clear the gain of the proposed workflow over Base is attributed to the workflow or due to additional LLM calls. Further it is not clear if the system instruction for the Base is properly optimized/tuned for this task. For example, if the system instruction for the Base approach also include more specific about the high-level workflow, does that improve Base a lot?

  • Lack of proper related baselines in the experiments and disussions such as

  1. Yilun Zhu et al. Hint-Augmented Re-ranking: Efficient Product Search using LLM-BasedQuery Decomposition. IJCNLP-AACL 2025
  2. Jiangyuan Wang et al. ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents.
Questions for authors
  • For each query, please also report the candidates opened, tool calls and tokens for Base and PDeepSearch
  • Could also have the results for the Base required to open all ten candidates, with and without the condition decomposition
2 comments
AuthorsOpenReview ↗

Response to reviewer GLaE (1/2)

Dear Reviewer GLaE

We thank the reviewer for the valuable suggestions. We have conducted additional ablations and controlled experiments to further clarify the effectiveness of our workflow.

W1: Whether PDeepSearch’s gains stem from its workflow or additional computation, and whether stronger workflow instructions improve Base.

Response: Thank you for this valuable suggestion. We agree that the original comparison does not fully disentangle workflow design from additional inference computation.

We would like to clarify that our original Base includes task-specific instructions to evaluate requirements, inspect product evidence, and consider contradictions (Appendix C.1, Figure 15), but does not enforce exhaustive inspection or explicit condition decomposition. We therefore conduct additional component ablations and token-budget-controlled experiments.

1. Component Ablation

We progressively strengthen Base with (1) exhaustive inspection of all ten candidates and (2) explicit condition decomposition. We then compare the resulting single-agent workflow with PDeepSearch. All settings use the same reranked top-10 candidates and evidence-access tools.

Table A: Component ablation on Qwen3.5-27B.

SettingDecomp.Inspect 10Separate AgentsHit@1 (%)Avg. CallsAvg. Tokens
Original BaseNoNoNo57.8617.025.0K
Base + AllNoYesNo58.4336.351.2K
Single-agent full workflowYesYesNo60.1440.960.5K
PDeepSearchYesYesYes63.7137.548.3K

Table B: Component ablation on Gemini-3.5-Flash.

SettingDecomp.Inspect 10Separate AgentsHit@1 (%)Avg. CallsAvg. Tokens
Original BaseNoNoNo69.7115.019.0K
Base + AllNoYesNo70.8634.241.5K
Single-agent full workflowYesYesNo71.4337.648.9K
PDeepSearchYesYesYes75.7135.038.2K

The single-agent baseline uses the same decomposed conditions, evidence-profile format, and verification criteria as PDeepSearch, with the key difference being that inspection and final selection are performed within a single shared agent context. Strengthening the single-agent workflow progressively improves Hit@1 from 57.86% to 60.14% on Qwen3.5-27B and from 69.71% to 71.43% on Gemini-3.5-Flash.

Compared with this strengthened single-agent workflow, PDeepSearch further improves Hit@1 by 3.57 and 4.28 percentage points, respectively, while reducing average token consumption by 20.2% and 21.9% and requiring fewer tool calls. These results support the additional benefits of independent candidate inspection and multi-agent organization beyond stronger single-agent instructions.

2. Token Budget-Controlled Experiments

To control for inference computation, we compare both workflows on Qwen3.5-27B under identical global token limits of 40K, 60K, and 80K per query. Each limit covers the entire workflow, with tokens aggregated across all agents in PDeepSearch.

Table C: Global token budget ablation on Qwen3.5-27B.

Max TokensArchitectureHit@1 (%)Avg. TokensAvg. Calls
40KSingle-agent52.2937.86K27.0
40KPDeepSearch56.0031.43K21.5
60KSingle-agent58.7152.86K36.8
60KPDeepSearch61.5743.22K29.5
80KSingle-agent60.1460.5K40.9
80KPDeepSearch63.7148.3K37.5

PDeepSearch improves Hit@1 by 3.71, 2.86, and 3.57 percentage points under the 40K, 60K, and 80K token limits, respectively, while using 17.0%, 18.2%, and 20.2% fewer tokens and fewer tool calls.

Overall, stronger workflow instructions improve Base, while PDeepSearch achieves further accuracy gains with lower inference costs. These results support the benefits of multi-agent organization beyond additional computation. We will incorporate these findings into the revised paper.

AuthorsOpenReview ↗

Response to reviewer GLaE (2/2)

W2: Insufficient comparison with relevant baselines and related work.

Response: Thank you for pointing out these relevant works. We agree that their relationship to our work should be clarified.

Hint-Augmented Re-ranking focuses on decomposing implicit user preferences, particularly superlative expressions, into structured attribute hints to improve product retrieval and reranking. In contrast, PDeepSearch verifies multiple user-specified conditions against heterogeneous product evidence and uses condition-level evidence profiles for final product selection.

ShoppingBench evaluates shopping agents on end-to-end tasks involving diverse user intents, including budget management, voucher application, and multi-item purchasing. Our benchmark instead focuses on evidence-grounded product retrieval, requiring models to verify multiple conditions using product images, specifications, prices, and reviews.

Direct numerical comparisons may be difficult due to differences in task formulation and evaluation protocols. Our experiments therefore evaluate retrieval, reranking, and agentic baselines under a shared product corpus, candidate pool, and evidence-access setting. We will expand the Related Work section to clarify these distinctions.

Q1: Per-query candidate inspection and inference costs for Base and PDeepSearch.

Response: Thank you for this suggestion. We measure the number of distinct candidates opened, tool calls, and token consumption at the query level. Table D reports their averages across queries for the original Base and PDeepSearch.

Table D: Average candidate inspection and inference costs per query.

BackboneSettingAvg. Candidates OpenedAvg. Tool CallsAvg. Tokens
Qwen3.5-27BOriginal Base5.217.025.0K
Qwen3.5-27BPDeepSearch10.037.548.3K
Gemini-3.5-FlashOriginal Base4.615.019.0K
Gemini-3.5-FlashPDeepSearch10.035.038.2K

The original Base opens an average of 5.2 and 4.6 distinct candidates on Qwen3.5-27B and Gemini-3.5-Flash, respectively, whereas PDeepSearch inspects all ten candidates. This broader inspection increases average tool calls and token consumption relative to the original Base.

However, the controlled comparisons in W1 show that PDeepSearch achieves higher accuracy with fewer tokens and tool calls than the strengthened single-agent workflow under matched inspection and verification protocols. These advantages also persist under identical global token limits.

Q2: Base performance with exhaustive candidate inspection, with and without condition decomposition.

Response: Thank you for this suggestion. We directly evaluate both settings in the component ablations reported in W1 (Tables A and B).

Requiring Base to inspect all ten candidates without explicit condition decomposition improves Hit@1 from 57.86% to 58.43% on Qwen3.5-27B and from 69.71% to 70.86% on Gemini-3.5-Flash. Adding explicit condition decomposition further improves Hit@1 to 60.14% and 71.43%, respectively.

These results directly quantify the contributions of exhaustive inspection and explicit condition decomposition to the single-agent baseline.

Reviewer_EKGsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of product search and makes the observation that prior product search benchmarks largely focus on semantic retrieval and neglect the many other important aspects of a product search query such as visual appearance, price and user reviews. In response, the authors contribute a new benchmark called Product Deep Search derived from the public Amazon 2023 reviews dataset. In addition, the authors propose a multi-agent framework for product search called PDeepSearch. Finally, the authors introduce a product search agent called PDeepSearch-9B that achieves superior performance on the Product Deep Search benchmark than larger open weight models and comes close to the performance of frontier proprietary models.

Critical strengths
  1. The paper identifies a gap in the literature in the fact that existing product search benchmarks largely focus on semantic similarity and neglect other important aspects of a product search journey such as visual appearance and price and proposes a new benchmark to bridge this gap.
Critical weaknesses
  1. The contribution of the PDeepSearch multi-agent framework is scientifically thin on substance because it does not introduce any fundamentally new ideas in how to architect agentic systems but rather applies existing knowledge around tool calling LLMs for the new benchmark the authors create.
Questions for authors
  1. In Table 1, it looks like most methods achieve the same recall@30 value of 87.14 which seems strange to me. Does this mean that the benchmark saturates at a recall value before 100% or that all of these models happen to score the same after reranking?

  2. Can the authors discuss additional related work on agentic systems for e-commerce and explain how their work is different from these other efforts (e.g. https://cdn.amazon.science/6d/35/93d079804efbaa1dac900cffdeaf/scipub-approval152129-41828767-a-functionalitygrounded-benchmark-for-evaluating-web-agents-in-ecommerce-domains.pdf, https://arxiv.org/pdf/2606.17698)?

1 comment
AuthorsOpenReview ↗

Response to reviewer EKGs

Dear Reviewer EKGs

We thank the reviewer for the constructive feedback. We appreciate the opportunity to clarify our scientific contributions and relationship to existing work.

W1: Scientific contribution of the PDeepSearch multi-agent framework.

Response: Thank you for your thoughtful feedback. We appreciate the opportunity to further clarify the intended contributions of PDeepSearch. While our framework builds upon established agentic design principles, we believe its scientific contributions lie in three complementary aspects:

  1. Benchmark: We introduce Product Deep Search to systematically evaluate multi-condition evidence verification, a capability largely overlooked by existing product search benchmarks.
  2. Methodology: We formulate product selection as explicit condition-level verification of support, uncertainty, and contradiction, followed by evidence-grounded cross-candidate decision making, rather than relying solely on holistic semantic matching.
  3. Empirical findings: We demonstrate that this approach effectively corrects reranker errors and that the verification workflow can be learned by a compact 9B model through trajectory supervision.

We hope these clarifications help better convey our intended contribution: a structured approach to evidence-grounded product selection, together with empirical evidence of its effectiveness. We will make this positioning clearer in the revised paper.

Q1: Why do multiple methods achieve the same Recall@30 of 87.14%?

Response: Thank you for pointing this out. We apologize that this was not sufficiently clear in the original presentation.

All reranking methods in Table 1 operate on the same top-30 candidate pool retrieved by Qwen3-VL-Embedding-8B. Since reranking only changes the order of these candidates without modifying the candidate set, Recall@30 remains identical at 87.14% across these methods. Thus, 87.14% reflects the Recall@30 of the shared upstream retriever rather than a saturation limit of the benchmark. We will clarify this in the revised paper.

Q2: Relationship to existing e-commerce agent research

Response: We thank the reviewer for bringing these relevant works to our attention. We agree that discussing them would help better position our contributions within the broader literature on e-commerce agents.

Amazon-Bench vs. Ours: Amazon-Bench evaluates web agents across diverse e-commerce operations, including account management, wishlist management, and gift card settings, with an emphasis on task completion and operational safety. Our work focuses more specifically on evidence-grounded product retrieval and selection under multiple user-specified conditions.

EComAgentBench vs. Ours: EComAgentBench studies long-horizon shopping agents that discover user requirements across queries, hidden profiles, and clarifications. It also considers attribute- and review-based evidence through requirement-level evaluation rubrics. Our work shares an interest in reliable evidence verification but focuses on a different setting: explicit multi-condition product queries and structured per-candidate evidence profiles that capture support, uncertainty, and contradiction to guide cross-candidate selection.

We view these works as complementary to our research, addressing different aspects of reliable e-commerce agents. We appreciate the reviewer’s suggestions and will incorporate these references and distinctions into the revised Related Work section.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Response to reviewer 6mhg (1/2)

Dear Reviewer 6mhg

We thank the reviewer for the insightful feedback. We have conducted additional controlled experiments and a human audit to further validate our workflow and evidence verification quality.

W1: Contribution of agent organization beyond exhaustive inspection and additional computation.

Response: Thank you for this important observation. We agree that the original comparison evaluates the complete pipeline rather than isolating the contribution of agent organization. We clarify that our original Base already includes explicit requirement-checking and evidence-verification guidance (Appendix C.1, Figure 15) and uses the same candidate pool and evidence-access tools as PDeepSearch.

To directly address this concern, we conduct additional component ablations and token-budget-controlled experiments.

1. Component Ablation

We progressively strengthen the single-agent baseline with (1) exhaustive inspection of all ten candidates and (2) explicit condition decomposition, and then compare the resulting workflow with PDeepSearch.

Table A: Component ablation (Hit@1, %).

SettingQwen3.5-27BGemini-3.5-Flash
Original Base57.8669.71
Base + All58.4370.86
Single-agent full workflow60.1471.43
PDeepSearch63.7175.71

Here, Base + All requires inspection of all ten candidates, while the Single-agent full workflow additionally introduces explicit condition decomposition. The single-agent baseline uses the same decomposed conditions, evidence-profile format, and verification criteria as PDeepSearch, with the key difference being that inspection and final selection are performed within a single shared agent context.

Table B: Controlled comparison of agent organization.

BackboneSingle-agent FullPDeepSearchGainAvg. Calls (Single / Ours)Avg. Tokens (Single / Ours)
Qwen3.5-27B60.1463.71+3.5740.9 / 37.560.5K / 48.3K
Gemini-3.5-Flash71.4375.71+4.2837.6 / 35.048.9K / 38.2K

With matched inspection protocols, condition decomposition, and evidence-verification criteria, PDeepSearch achieves 3.57 and 4.28 percentage-point improvements, respectively, while reducing average token consumption by 20.2% and 21.9% and requiring fewer tool calls. These results support the additional benefits of independent candidate inspection and multi-agent organization beyond stronger single-agent workflow instructions.

2. Computation Budget-Controlled Experiments

We further compare both architectures under identical global token limits (40K/60K/80K) on Qwen3.5-27B. Each limit applies to the complete workflow, with token usage aggregated across all agents.

Table C: Global token budget ablation on Qwen3.5-27B.

Max TokensArchitectureHit@1 (%)Avg. TokensAvg. Calls
40KSingle-agent52.2937.86K27.0
40KPDeepSearch56.0031.43K21.5
60KSingle-agent58.7152.86K36.8
60KPDeepSearch61.5743.22K29.5
80KSingle-agent60.1460.5K40.9
80KPDeepSearch63.7148.3K37.5

PDeepSearch consistently outperforms the single-agent counterpart, achieving gains of 3.71, 2.86, and 3.57 percentage points under the 40K, 60K, and 80K limits, respectively, while consuming 17.0%, 18.2%, and 20.2% fewer tokens on average. It also requires fewer tool calls across all three budget settings.

Overall, these results show that PDeepSearch's improvements are not solely attributable to exhaustive inspection or larger inference token budgets. Independent candidate inspection and multi-agent organization provide additional accuracy and efficiency benefits beyond the strengthened single-agent workflow. We will incorporate these experiments into the revised paper.

AuthorsOpenReview ↗

Response to reviewer 6mhg (2/2)

W2: Reliability of intermediate Inspector evidence profiles.

Response: Thank you for highlighting this important distinction. We agree that Hit@1 and Judge@1 evaluate final product selection rather than the correctness of intermediate Inspector outputs.

To directly assess intermediate verification quality, we conduct a stratified human audit of 250 Inspector evidence profiles, covering 1,261 condition-level judgments generated by Gemini-3.5-Flash with PDeepSearch. The audit evaluates three aspects:

  1. Condition status correctness: Whether each condition is correctly classified as exact match, mixed, unknown, or contradicted.
  2. Evidence faithfulness: Whether the extracted supporting and contradictory evidence is faithfully grounded in the original product-page content.
  3. Evidence relevance: Whether the supporting or contradictory evidence is relevant to the corresponding query condition.

The audited profiles are stratified by query-level prediction outcomes, explicitly including both successful corrections and harmful overrides.

Table D: Human evaluation of Inspector evidence profiles.

Prediction OutcomeAudited ProfilesFully ValidValidity (%)
Kept correct958892.6
Correct → Wrong201155.0
Wrong → Correct706288.6
Still wrong654772.3
Overall25020883.2

A profile is considered fully valid only when all its condition statuses are correct and all associated supporting and contradictory evidence is faithful and relevant. The overall validity rate is calculated over the stratified audit sample.

Table E: Fine-grained evidence verification quality.

Evaluation MetricCorrect / TotalAccuracy (%)
Condition status accuracy1,172 / 1,26192.9
Evidence faithfulness1,584 / 1,62097.8
Evidence relevance1,532 / 1,62094.6

Condition status accuracy is evaluated over 1,261 condition-level judgments, while evidence faithfulness and relevance are evaluated over 1,620 individual evidence items.

Across the stratified audit sample, 83.2% of profiles are fully valid, with 92.9% condition status accuracy, 97.8% evidence faithfulness, and 94.6% evidence relevance. The audit also covers both successful corrections and harmful overrides, with fully valid rates of 88.6% and 55.0%, respectively.

These results directly evaluate intermediate evidence-verification quality, complementing the final selection metrics reported in the paper. We will include the audit protocol and results in the revised paper.

Q1: Comparison with a single-agent full workflow under comparable inference budgets.

Response: Thank you for this suggestion. As detailed in W1 (Tables A–C), we compare PDeepSearch against a strengthened single-agent workflow with the same condition decomposition, exhaustive inspection, and evidence-verification criteria.

PDeepSearch achieves 3.57 and 4.28 percentage-point improvements on Qwen3.5-27B and Gemini-3.5-Flash, respectively, while using 20.2% and 21.9% fewer tokens and requiring fewer tool calls. These advantages persist under identical global token limits of 40K, 60K, and 80K on Qwen3.5-27B.

These results support the benefits of the proposed multi-agent organization beyond exhaustive inspection and additional inference computation.

Q2: Human audit of Inspector profiles and training trajectory filtering criteria.

Response: Thank you for these thoughtful questions. We appreciate the opportunity to further clarify the reliability of Inspector evidence profiles and the quality-filtering criteria used for training trajectories.

1. Inspector Profile Quality

As detailed in W2 (Tables D and E), we conducted a stratified human audit of 250 Inspector profiles, including successful corrections and harmful overrides. The audit achieves 83.2% fully valid profiles, 92.9% condition status accuracy, 97.8% evidence faithfulness, and 94.6% evidence relevance. The fully valid rates for successful corrections and harmful overrides are 88.6% and 55.0%, respectively.

2. Training Trajectory Filtering

We apply structural validity filtering to retain trajectories with complete and well-formed Inspector and Decision outputs suitable for supervised fine-tuning. Neither final selection correctness nor intermediate evidence correctness is used as a filtering criterion. We will clarify these criteria in the revised paper.

Thanks for the response. Could you clarify how you chose the 250 profiles for the audit—how many queries they covered and which candidates were selected from each query? It would also help to know how the annotation was done and how disagreements were resolved. Could you also share the strengthened single-agent prompt? For the token limits, what tokens were counted, how was the budget split across agents, and how were runs handled and scored if they ran out of budget before making a final choice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.