- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper tackles an interesting topic about what parts can be replaced by LLM in a recommendation system. The paper proposed a framework with a planning agent that determines how recommendation evidence should be coordinated around the existing candidate backbone, and a ranking agent to refine the inherited ranking structure. Experiments are conducted across diverse recommendation domains demonstrate that Agent-R consistently improves both candidate coverage and ranking quality.
Critical strengths
The paper is well-organized and easy to follow. The topic discussed is very practical and could have potential real-world impact. The experiment is detailed and informative.
Critical weaknesses
One of the major issues I noticed is that in Table 1 and Table 2 the performance of Agent-R on Industrial dataset is quite different. I wonder if that is typo or any misunderstanding of the experiment results.
Another of my naive concern is that I agree on the framework design, but I wonder why is LLM necessary to work as ranking agent and planning agent. Any additional impact does LLM bring to the system comparing to non-LLM setting?
Questions for authors
Why is LLM necessary to work as ranking agent and planning agent? Any additional impact does LLM bring to the system comparing to non-LLM setting?
5 comments
Reply to W2 - Why LLM Agents? Understanding Their Contributions Beyond Conventional Recommendation
We sincerely appreciate your thoughtful question. We agree that the value of introducing LLM agents should be demonstrated beyond simply adding more powerful models. Our motivation is not to replace conventional recommenders with LLMs, but to assign contextual intelligence to specific decision processes while retaining existing recommendation structures. To clarify these contributions, we conducted several additional controlled experiments.
1. The complete Agent-R framework provides significant benefits beyond standalone LLM fine-tuning.
We independently fine-tuned a MAS-independent Qwen2.5-7B ranker using the same backbone, LoRA architecture, and listwise training objective, but without agent-generated memory, planning, or multi-route coordination (Appendix B.10, Table 10).
| Dataset | Fine-tuned LLM Ranker | Agent-R | Δ NDCG@10 [95% CI] |
|---|---|---|---|
| Software | 0.0797 | 0.0851 | +0.0054 [0.0034, 0.0075] |
| Industrial | 0.0119 | 0.0229 | +0.0111 [0.0080, 0.0142] |
| KuaiRec | 0.0567 | 0.0615 | +0.0048 [0.0020, 0.0076] |
| MicroLens | 0.0177 | 0.0199 | +0.0022 [0.0013, 0.0032] |
All four improvements are statistically significant. This demonstrates the additional value of the complete multi-agent recommendation pipeline beyond independently fine-tuning an LLM ranker. The comparison measures the combined effects of candidate adaptation and structured contextual modeling, rather than the isolated contribution of any single LLM component.
2. LLM-generated memory and contextual preference refinement contribute complementary information.
The Memory Agent constructs reusable short-term and long-term preference representations from user histories. Removing this component reduces NDCG@10 from 0.0851 to 0.0836 on Software and from 0.0229 to 0.0205 on Industrial, with statistically significant differences.
Meanwhile, contextual residual refinement improves ranking beyond the inherited recommendation prior. For example, NDCG@10 increases from 0.0155 to 0.0615 on KuaiRec and from 0.0142 to 0.0199 on MicroLens relative to Prior Only.
Our additional non-LLM residual comparisons further clarify that the structured semantic context and residual-learning interface are important, while the 7B LLM ranking backbone is not universally indispensable. Lightweight text-based rankers can also effectively leverage these representations. Thus, we distinguish the contribution of contextual preference modeling from claims of LLM-specific ranking superiority.
3. The Planning Agent provides constrained, user-conditioned evidence coordination.
Unlike conventional fixed-weight fusion, the Planning Agent generates per-user adjustments over heterogeneous recommendation evidence, while a deterministic executor remains responsible for candidate selection.
To isolate its contribution, we additionally compare against budget-matched non-LLM expansion strategies (Appendix B.11):
| Dataset | Base Only R@50 | Static Uniform | Static DEV-Tuned | LLM Planner |
|---|---|---|---|---|
| Software | 0.2980 | 0.3163 | 0.3191 | 0.3185 |
| Industrial | 0.0800 | 0.0947 | 0.0946 | 0.0950 |
| KuaiRec | 0.1203 | 0.1322 | 0.1315 | 0.1308 |
| MicroLens | 0.0726 | 0.0932 | 0.0937 | 0.0934 |
Candidate adaptation significantly improves Recall@50 across all four datasets. The LLM Planner additionally outperforms uniform expansion on Software (+0.0022, 95% CI [0.0005, 0.0039]) and achieves performance comparable to DEV-tuned static policies across all datasets, while generating user-specific adjustments without supervised planner-policy training. We acknowledge that its incremental accuracy advantage over optimized static policies is not universal.
Taken together, these results clarify why we introduce LLM agents and where their value lies. LLMs provide flexible semantic preference representations and user-conditioned coordination within a constrained recommendation pipeline, while specialized recommenders and deterministic execution retain responsibility for reliable item-level decisions. Importantly, our additional controls show that effective agency allocation does not require every component to be implemented by a large LLM.
Reply to Question - Why Use LLMs for Planning and Ranking?
We sincerely appreciate your thoughtful question. The key motivation behind Agent-R is not that every recommendation decision requires an LLM, but that LLMs can provide complementary contextual intelligence where conventional recommendation structures remain insufficient. Our framework therefore assigns LLMs constrained responsibilities rather than replacing specialized recommendation models.
Specifically, the two agents address different limitations:
| Component | Role of the LLM | Existing Recommendation Intelligence Preserved |
|---|---|---|
| Planning Agent | Interprets heterogeneous evidence and generates user-specific coordination policies | Retrieval routes, calibrated evidence, and deterministic item selection |
| Ranking Agent | Models contextual user–item compatibility using preference memory and item semantics | Original ranking prior and bounded residual intervention |
First, the Planning Agent provides flexible evidence coordination without requiring supervised policy training. Unlike fixed-weight fusion, it generates user-conditioned adjustments over multiple recommendation sources. Our new budget-matched experiments (Appendix B.11) show that LLM planning achieves coverage comparable to DEV-tuned static policies across all four datasets, with a significant improvement over uniform expansion on Software (+0.0022 Recall@50, 95% CI [0.0005, 0.0039]). This demonstrates a practical advantage of LLM-based coordination, although we do not claim it universally outperforms optimized static policies.
Second, the Ranking Agent leverages semantic preference information beyond conventional ranking scores. For instance, Agent-R improves NDCG@10 over Prior Only from 0.0155 to 0.0615 on KuaiRec and from 0.0142 to 0.0199 on MicroLens. LLM-generated preference memory also provides additional contextual information, with significant improvements when included on Software and Industrial.
Importantly, our additional non-LLM controls reveal that the residual ranking mechanism can also be implemented using smaller models. We view this as a useful clarification: the value of Agent-R lies in its structured coordination of candidate evidence and contextual preference information, not in requiring an LLM for every computation. The Ranking Agent is therefore a flexible component within the broader agency allocation framework.
As further demonstrated in Appendix B.10 (Table 10), the complete Agent-R framework significantly outperforms an independently fine-tuned, MAS-independent LLM ranker across all four datasets, confirming that its benefits extend beyond standalone LLM fine-tuning.
In essence, Agent-R addresses not only whether LLMs can improve recommendation, but where their capabilities should be integrated with existing recommendation intelligence. This is precisely the motivation behind our Agency Allocation formulation.
We sincerely appreciate your constructive feedback and encouragement throughout the review. Your suggestions have helped us substantially improve the rigor, completeness, and positioning of our paper, particularly in clarifying the distinct roles of LLM agents and conventional recommendation components.
We hope that our detailed responses, additional experimental evidence, and revised manuscript have adequately addressed your concerns and further strengthened your confidence in the soundness and practical value of our work. We would be very happy to provide further clarification or address any remaining questions, so please feel free to let us know.
We would also be very grateful if you could consider raise your assessment and rating of our paper in light of these additional experiments and revisions. Thank you again for your time, thoughtful consideration, and valuable support.
Response to Reviewer nqiC
We sincerely thank you for your careful review, encouraging assessment, and thoughtful suggestions. Your questions regarding experimental consistency and the necessity of LLM agents have motivated us to further examine the contributions of individual components and strengthen our empirical evidence. In response, we have carefully revised the manuscript and conducted additional controlled experiments and analyses. We hope the following responses address your concerns and provide a clearer and more convincing understanding of the contributions of Agent-R.
Reply to Weakness
Reply to W1 - Correction of Industrial Results
We sincerely thank you for carefully identifying this inconsistency. After examining our experimental records, we confirmed that the discrepancy originated from an inadvertent dataset-to-result mapping error during result consolidation, where results from another dataset were mistakenly reported as Industrial in Table 1. The intended benchmark is Amazon Industrial, and we apologize for this oversight.
To fully resolve this issue, we re-evaluated Agent-R and its ablations on Amazon Industrial and updated all affected results in the revised manuscript. The corrected results are:
The updated results are now fully consistent across Tables 1, 2, 5, and 6. Importantly, Agent-R continues to outperform the reported fusion baselines on all four Industrial metrics, preserving our main experimental conclusions.
We have also updated the manuscript PDF and anonymous code repository with data, including the corrected evaluation pipeline and benchmark data, to ensure that these results can be independently reproduced.
We sincerely appreciate your careful review, which helped us improve the accuracy and reproducibility of our experimental evaluation.