- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to efficiently retrieve evidence for explorations of items in the recommendation tasks. Specifically, the paper considers an agentic recommendation setting where agents retrieves so called "evidence" about each items to input them as a prompt for some LLM-based recommendation models (e.g., OneRec).The paper proposes a gating mechanism to determine if further evidence retrieval is necessary for provided inquiry to reduce the cost of evidence retrieval. The experiments are conducted on several recommendation datasets, against variety of baselines for recommendation.
Critical strengths
-
The motivation of considering retrieval-cost aware agentic use for recommendation task makes sense, and sufficiently explained in the paper.
-
The experiment results is good, it clearly demonstrates the benefits of the proposed method combined on top of the base model, which is OneRec. The choice of recommendation models compared are also representative choice. (While there are also some limitation, see Weaknesses).
Critical weaknesses
I feel that there might be several LLM-generated hallucinations in the paper, which makes the paper not easy to follow. I will detail specific points below.
(1) Abstract and Introductions (and problem formulation) are somewhat related, but provides different first impression in the topic. When I look at Figure 1 and read the Introduction, it somehow focuses on tool use, and it provides me the first impression that the action space of the decision problem is "which tools and how many times we should use" for some recommendation task.
Then, in the problem formulation section specifies action to be "(joint set of) interest focus, evidence mixing, retrieval-source allocation", although the explanation about what these variables actually refer to was never explained in the paper. This already sounds a bit contradictory, but if I assume that retrieval-source allocation means selection over which tool to use, then I can barely see connection to the Intro.
However, it is quite confusing that later in Section 4 (i.e., in the proposed method section), the paper writes "A recommender can rank only items that retrieval makes available", and this sounds like the first stage model is performing candidate item retrieval to pass to late-stage decision model, and this time, the paper does not consider tool choose as the action space.
I feel that probably LLMs or authors are somehow mixing the two different problem, "candidate item retrieval" and "tool selection". If they understand that these are distinct topics, then, the presentation of the paper matters. In any case, the fundamental problem setting is really unclear from the manuscript.
(2) Some important details are not explained. For example, (2-1) how the gating function is trained, (2-2) how the utility function (or value function) is defined for "joint action" over "interest focus, evidence mixing, retrieval-source allocation" in right after the equation, (2-3) what is "training target" in this setting
(3) Related work picks representative papers from each field, but the connection to the proposed method is not really explained. For instance, "Learning from limited feedback" cites many counterfactual learning paper from logged data, but counterfactual learning methods are not used in the proposed method, and why these papers are specifically discussed is questionable.
(4) Some other minor writing issues; no citation for OneRec in the experiment section, sudden introduction of Proposition 1 without any connection sentence, to name a few.
Other than the above issues in writing, following points are counted as weaknesses.
-
The paper only reports mean of the performance, and does not report variance-related quantity in the experiment. And if I did not overlook, it appears that the paper report the results with only a single random seed or single split of train/test data. This is not reliable performance reporting.
-
If the paper considers effective tool use as the main contribution, related work should be more detailed in this point, not just listing some representative papers for tool-use or agentic recommendation. The idea of each paper, and its connection to the proposed method should be explained. (And this is also important for evaluating the novelty of the paper.)
Questions for authors
Please write rebuttals with author(s)' own words, as LLM-generated sentence may be hallucinated and is hard to follow.
-
How are the action space defined?
-
How is the value function learned for the evidence retrieval? How is the reward defined?
-
How does the proposed method learn the gating function?
2 comments
Thank you for the detailed response. The rebuttal comments are useful, however, provided the quality of the manuscript, I will maintain my current evaluation score.
Response to Reviewer t8Lj
We are very sorry that the original text blurred tool use and candidate expansion, which could make ECB look like a choice among recommendation tools. We have rewritten the setup using the actual sequence of operations.
(lambda, rho, omega), controlling interest focus, evidence mixing, and retrieval allocation across dense, lexical, and sequential sources.Table 1: NDCG@5, mean ± sample SD over three training seeds
Table 2: Candidate coverage, final hits, and pool sizes
The retained pool can exceed the 20-item displayed shortlist. For these reference runs, final pool size = initial retained-pool size + 20 × mean upgrades.