- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ShopLife-Bench evaluates whether shopping assistants can maintain changing user profiles across sessions, use remembered preferences to select products, and serve multiple users without mixing their information. It combines structured profiles, tasks requiring historical information, and user feedback policies derived from real e-commerce conversations. Each evaluation group includes two task users and one distractor across three sessions, with profiles and orders persisting after conversation history is cleared. Evaluation results highlight difficulties in cross-session memory maintenance and multi-user task execution.
Critical strengths
-
The paper proposes a practical evaluation setting that combines multiple requirements and components that simulate shopping behavior.
-
The benchmark construction makes successful shopping decisions meaningful evidence of memory use.
Critical weaknesses
-
The novelty of the benchmark may be limited. There are existing works covered important aspects of the proposed benchmark and none were discussed in the paper. For example, Shopping Companion evaluates cross-session preference memory in shopping. GroupMemBench evaluates multi-user memory. ShopLife-Bench may contribute a useful combination but the individual capabilities are not new.
-
The low perfect-group scores may need careful interpretation. Combining many requirements across users and sessions naturally leads to a strict requirement and lower score. The paper can provide more insights by running ablations on each component/requirements to illustrate the main bottlenecks.
-
The realism is questionable. Specifically, the simulator restricts patience thresholds to 1/3 errors, while the thresholds cover only 56% of the estimated real distribution.
Questions for authors
See weakness.