- Rating
- 4(+2)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Layered Dimension-Conditioned Query Optimization (LDCO), a user-side LLM framework for recommendation under target-domain cold start and underspecified queries. LDCO constructs domain-level and global profiles from source-domain histories, predicts the target domain, selects relevant source profiles, maps supported preferences to target-product attributes, and generates a personalized retrieval query. The query-generation prompt is optimized offline using an MCTS-style search with Hybrid Hit@100 as feedback. An LLM subsequently reranks the retrieved candidates using the generated query and user profile. The method is evaluated on a user-disjoint benchmark derived from Amazon Reviews 2023 and through live Amazon retrieval evaluated by an LLM judge.
Critical strengths
The paper studies a practically relevant interface-constrained setting in which a user-side agent can personalize retrieval without modifying a platform’s index or ranking model. The separation between query rewriting, which changes candidate membership, and reranking, which changes candidate order, is clearly motivated and reflected in the evaluation. The proposed evidence-selection and dimension-mapping stages also provide a sensible safeguard against unrestricted cross-domain preference transfer. The manuscript clearly documents the main pipeline, separates prompt-optimization and test users, freezes the selected prompt before testing, reports the seed and optimized prompts, and provides useful latency and API-cost measurements. The reported gains are sizable, particularly for the Electronics target domain.
Critical weaknesses
-
The evaluation does not establish that LDCO improves recommendations for actual cold-start users. Every offline fuzzy query is generated from the title, description, and review of one five-star target item, after which that same ASIN is treated as the sole relevant result. Therefore, plausible substitutes are counted as errors, and the task primarily measures recovery of an item that was used to synthesize the request rather than relevance for naturally issued queries. Moreover, the split is only stated to be user-disjoint: no target-ASIN or product-intent overlap analysis is reported, even though the optimized prompt explicitly contains product-specific rules for cases such as an iPad Air 2 case, a 1-inch curling iron, and a surge suppressor learned during optimization. The live experiment does not resolve this limitation because its labels come entirely from GPT-4.1, while the intervention being evaluated is itself an LLM reranker ordering products according to a closely aligned query/profile relevance criterion. Without human preference labels or validation of the judge, the reported gains cannot distinguish improved user relevance from agreement between two LLM-based scoring procedures.
-
The experiments do not show that the proposed architecture is preferable to existing or simpler alternatives. The comparison consists of the original fuzzy query and internal variants; no competing personalized query-rewriting, cross-domain recommendation, or user-side recommendation method is evaluated. The ablations also remove Layers 1–3 jointly rather than testing target classification, source selection, and dimension mapping separately, and they do not compare the hierarchical profile against domain-only or global-only profiles. Consequently, the results support the complete pipeline as one engineered system, but they do not substantiate the claimed contribution of its individual layers or show that MCTS-optimized LDCO is responsible for the gains relative to reasonable alternative pipelines.
Questions for authors
- Are target ASINs, product types, and fuzzy-query intents disjoint between the 568 prompt-optimization users and the 142 test users? What are the corresponding overlap rates, and how does the method perform when target items and intents observed during prompt optimization are excluded from testing?
- How is the “fixed reference profile” used by GPT-4.1 constructed, and what exact judging prompt and decision rules are used? Has this judge been compared with blinded human assessments of whether the returned products fit the stated request and user preferences?
- How does LDCO compare under the same retriever and execution model with a direct profile-conditioned rewrite, a non-MCTS prompt-optimization method, and representative personalized rewriting or user-side recommendation approaches? Can the three LDCO layers and the two profile levels be ablated individually?
3 comments
I thank you for addressing my concerns adequately and I decide to raise my rating. Thank you for your effort.
Thanks a lot for your valuable time in reviewing this paper and for raising the evaluation score. Your comments have significantly helped us to improve the quality of this work.
A Response to Reviewer 4Hk8
Thank you for recognizing the user-side setting's practical value. Your questions prompted a split and Judge audit, additional comparisons, and dimension-context analysis. We hope the following reponses could address your concerns. The main changes and improvements are smmarized as follows.
Main additions and corrections
1. Evaluation scope and independence of the split
Offline single-ASIN evaluation measures recovery of a known positive with a five-star rating. Acceptable substitutes receive no credit. The improvement therefore supports target-item recovery, without establishing real-user satisfaction.
The split audit gives:
\text{Quantity} & \text{Audited value} \\\\ \hline \text{Optimization: cases / users} & \text{568 / 525} \\\\ \text{Test: cases / users} & \text{142 / 140} \\\\ \text{Total distinct users} & 651 \\\\ \text{Shared users} & 14 \\\\ \text{Test cases with shared ASIN} & 5 \\\\ \text{Shared exact query strings} & 16 \end{array}$$ The shared users have different target domains: eight Electronics-to-Beauty and six Beauty-to-Electronics cases. No user-ID/target-domain pair repeats. The 16 shared query strings occur in 18 test cases. Product-type and semantic-intent disjointness have not been established. We apologize for the inaccurate user-disjoint description and will correct the manuscript. Performance after excluding these overlaps remains unmeasured. Different target domains do not eliminate dependence through shared users, so the current results do not establish strict split independence. **2. What reference and rubric does the Judge use?** The released Judge receives **target-product and candidate-product information**. The manuscript's description of a fixed reference profile and original-query input is inaccurate and needs correction. The configured model is GPT-4.1 (`gpt-4.1-2025-04-14`, temperature 0.1). Its rubric assigns: - **3:** a near match in primary function and critical purchase attributes; - **2:** a practical substitute with the same primary function and no critical incompatibility; - **1:** related, but not a reliable substitute; - **0:** irrelevant or incompatible. Grades of at least 2 count as relevant for the loose binary metric. Category or brand agreement alone is insufficient. The Judge uses only supplied product information; the full rubric is in `llm-product-judge/prompt_template.txt`. This accepts alternatives beyond the original ASIN. **3. Direct rewriting, MCTS, and profile contribution** The added LLM4CS-REW-style adaptation generates one query from the request and profile. With GPT-4o-mini, Hybrid retrieval, and no reranking, it obtains Hit/MRR/NDCG@100 of **0.2746/0.0346/0.0784**, versus **0.4155/0.0577/0.1236** for LDCO. This is an adapted rewriting reference. Table 2's profile-free configuration obtains **0.3453/0.0458/0.0993** without reranking. Profile conditioning improves all three point estimates. MCTS optimizes prompts offline;. **4. Dimension-context ablations and harmful transfer** We added the following ablations without reranking: $$\begin{array}{lrrr} \text{Dimension context} & \text{Hit@100} & \text{MRR@100} & \text{NDCG@100} \\\\ \hline \text{All groups} & 0.4155 & 0.0577 & 0.1236 \\\\ \text{No target-type dimensions} & 0.3803 & 0.0613 & 0.1197 \\\\ \text{No cross-type bridge} & 0.3944 & 0.0585 & 0.1204 \\\\ \text{No domain-pair mapping} & 0.4085 & 0.0611 & 0.1238 \end{array}$$ The complete context has the highest Hit@100, but some reduced contexts improve MRR. Routing and source selection are retained. Your questions helped us correct and clarify the evaluation. We welcome further discussion of your main concerns. If this analysis and corrected interpretation sufficiently address them, would you consider reassessing the paper and increasing the score accordingly? Thank you very much!