- Rating
- 3
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces FashionArena, a large-scale, production-grounded environment and benchmark for training and evaluating fashion shopping agents. The framework includes approximately 1.6 million real products, six shopping tools, and tasks derived from real user behavior logs. To reduce the gap between offline and online environments, the authors propose a response replay mechanism that retrieves cached results from production search services. The benchmark provides 20K training tasks and 2K evaluation tasks, together with human annotations of product suitability and tool-use quality. Experiments across multiple LLMs demonstrate strong correlations between offline and online evaluation results. Furthermore, SFT followed by GRPO improves shopping agent performance, with the improvements transferring to online tools.
Critical strengths
Practical motivation and large-scale benchmark. The paper addresses an important problem in shopping agent research: the discrepancy between offline training environments and real-world shopping services. FashionArena provides a large-scale environment with diverse products and realistic shopping scenarios.
Comprehensive dataset and evaluation framework. The benchmark includes multimodal shopping requests, multiple executable tools, and human annotations of both recommendation outcomes and tool-use trajectories. These resources could be valuable for future research on shopping agents.
Extensive experimental evaluation. The authors evaluate multiple proprietary and open-weight LLMs and investigate the consistency between offline and online environments. The online transfer experiments further demonstrate the potential of FashionArena for training shopping agents.
Critical weaknesses
Unclear availability and reproducibility of the benchmark. Although the abstract states that the environment, datasets, and training code are released, the reproducibility statement indicates that these resources will be made publicly available in the future. The manuscript does not provide an accessible repository or dataset link. Since the primary contribution is a benchmark and training environment, publicly accessible resources are essential for assessing reproducibility and enabling further research. The authors should provide the relevant links and clarify which components can be reproduced without access to the private production services.
Questions for authors
During the rebuttal period, I encourage the authors to provide a demonstration of FashionArena, ideally including a video or an interactive demo showing how shopping agents interact with the environment, invoke tools, and complete recommendation tasks. Additionally, a demonstration of the offline response replay mechanism and its consistency with the online environment would help reviewers better understand the practical functionality and contributions of the proposed benchmark.
2 comments
Thanks for your rebuttal. My concerns have been addressed. I will keep my score.
Response to Reviewer QRaD
W1 — Resource availability and reproducibility
We make a promise to publicly release the datasets, offline environment, complete cache used in the reported experiments, and training/evaluation code upon acceptance of the paper.
Thank you for pointing out the inconsistent release statements. We will update the abstract and the reproducibility statement to reflect this release plan. The released resources will enable offline training and evaluation without access to the private production service. Live online evaluations will require access to that service.
Q1 — Demonstration
We have provided two videos in our anonymous repository.