- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Product Deep Search, a 700-query benchmark of multi-condition shopping requests over ~80K Amazon products. In addition, the authors also propose a method, PDeepSearch, which assigns one tool-using LLM "Inspector" to each of the reranker's top-10 candidates; each Inspector labels every query condition as exact match, mixed, unknown or contradicted. A Decision agent then picks one product from the ten profiles. The authors also report gains over a single-agent baseline for five backbones, and a fine-tuned 9B model that reaches 69.43 Hit@1.
Critical strengths
-
The new benchmark is built to require condition-level checks, with proper leakage controls. In particular, a benchmark that considers evidence checking over embedding similarity is a reusable contribution to the community.
-
The empirical results demonstrate that per-condition verification fixes errors in the traditional semantic re-ranker in practice.
Critical weaknesses
-
From the experiments in the paper, it is not clear the gain of the proposed workflow over Base is attributed to the workflow or due to additional LLM calls. Further it is not clear if the system instruction for the Base is properly optimized/tuned for this task. For example, if the system instruction for the Base approach also include more specific about the high-level workflow, does that improve Base a lot?
-
Lack of proper related baselines in the experiments and disussions such as
- Yilun Zhu et al. Hint-Augmented Re-ranking: Efficient Product Search using LLM-BasedQuery Decomposition. IJCNLP-AACL 2025
- Jiangyuan Wang et al. ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agents.
Questions for authors
- For each query, please also report the candidates opened, tool calls and tokens for Base and PDeepSearch
- Could also have the results for the Base required to open all ten candidates, with and without the condition decomposition
2 comments
Response to reviewer GLaE (2/2)
W2: Insufficient comparison with relevant baselines and related work.
Response: Thank you for pointing out these relevant works. We agree that their relationship to our work should be clarified.
Hint-Augmented Re-ranking focuses on decomposing implicit user preferences, particularly superlative expressions, into structured attribute hints to improve product retrieval and reranking. In contrast, PDeepSearch verifies multiple user-specified conditions against heterogeneous product evidence and uses condition-level evidence profiles for final product selection.
ShoppingBench evaluates shopping agents on end-to-end tasks involving diverse user intents, including budget management, voucher application, and multi-item purchasing. Our benchmark instead focuses on evidence-grounded product retrieval, requiring models to verify multiple conditions using product images, specifications, prices, and reviews.
Direct numerical comparisons may be difficult due to differences in task formulation and evaluation protocols. Our experiments therefore evaluate retrieval, reranking, and agentic baselines under a shared product corpus, candidate pool, and evidence-access setting. We will expand the Related Work section to clarify these distinctions.
Q1: Per-query candidate inspection and inference costs for Base and PDeepSearch.
Response: Thank you for this suggestion. We measure the number of distinct candidates opened, tool calls, and token consumption at the query level. Table D reports their averages across queries for the original Base and PDeepSearch.
Table D: Average candidate inspection and inference costs per query.
| Backbone | Setting | Avg. Candidates Opened | Avg. Tool Calls | Avg. Tokens |
|---|---|---|---|---|
| Qwen3.5-27B | Original Base | 5.2 | 17.0 | 25.0K |
| Qwen3.5-27B | PDeepSearch | 10.0 | 37.5 | 48.3K |
| Gemini-3.5-Flash | Original Base | 4.6 | 15.0 | 19.0K |
| Gemini-3.5-Flash | PDeepSearch | 10.0 | 35.0 | 38.2K |
The original Base opens an average of 5.2 and 4.6 distinct candidates on Qwen3.5-27B and Gemini-3.5-Flash, respectively, whereas PDeepSearch inspects all ten candidates. This broader inspection increases average tool calls and token consumption relative to the original Base.
However, the controlled comparisons in W1 show that PDeepSearch achieves higher accuracy with fewer tokens and tool calls than the strengthened single-agent workflow under matched inspection and verification protocols. These advantages also persist under identical global token limits.
Q2: Base performance with exhaustive candidate inspection, with and without condition decomposition.
Response: Thank you for this suggestion. We directly evaluate both settings in the component ablations reported in W1 (Tables A and B).
Requiring Base to inspect all ten candidates without explicit condition decomposition improves Hit@1 from 57.86% to 58.43% on Qwen3.5-27B and from 69.71% to 70.86% on Gemini-3.5-Flash. Adding explicit condition decomposition further improves Hit@1 to 60.14% and 71.43%, respectively.
These results directly quantify the contributions of exhaustive inspection and explicit condition decomposition to the single-agent baseline.
Response to reviewer GLaE (1/2)
Dear Reviewer GLaE
We thank the reviewer for the valuable suggestions. We have conducted additional ablations and controlled experiments to further clarify the effectiveness of our workflow.
W1: Whether PDeepSearch’s gains stem from its workflow or additional computation, and whether stronger workflow instructions improve Base.
Response: Thank you for this valuable suggestion. We agree that the original comparison does not fully disentangle workflow design from additional inference computation.
We would like to clarify that our original Base includes task-specific instructions to evaluate requirements, inspect product evidence, and consider contradictions (Appendix C.1, Figure 15), but does not enforce exhaustive inspection or explicit condition decomposition. We therefore conduct additional component ablations and token-budget-controlled experiments.
1. Component Ablation
We progressively strengthen Base with (1) exhaustive inspection of all ten candidates and (2) explicit condition decomposition. We then compare the resulting single-agent workflow with PDeepSearch. All settings use the same reranked top-10 candidates and evidence-access tools.
Table A: Component ablation on Qwen3.5-27B.
Table B: Component ablation on Gemini-3.5-Flash.
The single-agent baseline uses the same decomposed conditions, evidence-profile format, and verification criteria as PDeepSearch, with the key difference being that inspection and final selection are performed within a single shared agent context. Strengthening the single-agent workflow progressively improves Hit@1 from 57.86% to 60.14% on Qwen3.5-27B and from 69.71% to 71.43% on Gemini-3.5-Flash.
Compared with this strengthened single-agent workflow, PDeepSearch further improves Hit@1 by 3.57 and 4.28 percentage points, respectively, while reducing average token consumption by 20.2% and 21.9% and requiring fewer tool calls. These results support the additional benefits of independent candidate inspection and multi-agent organization beyond stronger single-agent instructions.
2. Token Budget-Controlled Experiments
To control for inference computation, we compare both workflows on Qwen3.5-27B under identical global token limits of 40K, 60K, and 80K per query. Each limit covers the entire workflow, with tokens aggregated across all agents in PDeepSearch.
Table C: Global token budget ablation on Qwen3.5-27B.
PDeepSearch improves Hit@1 by 3.71, 2.86, and 3.57 percentage points under the 40K, 60K, and 80K token limits, respectively, while using 17.0%, 18.2%, and 20.2% fewer tokens and fewer tool calls.
Overall, stronger workflow instructions improve Base, while PDeepSearch achieves further accuracy gains with lower inference costs. These results support the benefits of multi-agent organization beyond additional computation. We will incorporate these findings into the revised paper.