acceptodds
Under review as a conference paper at ICLR 2027

ShopLife-Bench: A Benchmark for Multi-Session Multi-User Agent Interaction in E-Commerce Environments

Abstract

Existing benchmarks for conversational AI agents mostly simulate a single user in a single session. They do not capture multi-user state management, the dynamic evolution of user behavior, or cross-session task decomposition and feedback. As a result, they cannot cover the complex, path-dependent behavior of real users across sessions. We propose ShopLife-Bench, a benchmark for multi-session, multi-user agent interaction in e-commerce environments, with four key contributions: 1) Our user simulator does not model internal mental states. Instead, it distills users' reactions to agent errors and their exit patterns from over ten thousand real e-commerce conversations, and builds a reproducible behavior policy driven by accumulated errors. 2) We restructure user profiles into structured, addressable atomic entries, so that every addition, update, and deletion can be verified item by item. We also design tasks in which the current request alone cannot uniquely determine the target product; the correct choice requires profiles maintained from earlier sessions. These tasks jointly evaluate profile maintenance and personalized use. 3) Cross-session tasks are built around the same user's product decisions and fall into three types: dependent, independent, and interleaved. Explicit statements, implicit needs, and noise are added on top. 4) Each evaluation session includes two task users and one distractor user who only asks about products or chats. This setting tests the agent's ability to attribute identities, switch between tasks, and isolate user states. Experiments on six representative models show that after shifting to multi-user concurrency, profile recall and order accuracy generally drop; even the best model reaches only order accuracy and a zero-error rate. Cross-user state isolation and cross-session consistency remain critical bottlenecks for current agents. Code and evaluation data are available at https://anonymous.4open.science/r/ShopLife-Bench.

Primary area:
datasets and benchmarks
Keywords:
AI agents, benchmarks, multi-session task

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ai2COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ShopLife-Bench evaluates whether shopping assistants can maintain changing user profiles across sessions, use remembered preferences to select products, and serve multiple users without mixing their information. It combines structured profiles, tasks requiring historical information, and user feedback policies derived from real e-commerce conversations. Each evaluation group includes two task users and one distractor across three sessions, with profiles and orders persisting after conversation history is cleared. Evaluation results highlight difficulties in cross-session memory maintenance and multi-user task execution.

Critical strengths
  • The paper proposes a practical evaluation setting that combines multiple requirements and components that simulate shopping behavior.

  • The benchmark construction makes successful shopping decisions meaningful evidence of memory use.

Critical weaknesses
  • The novelty of the benchmark may be limited. There are existing works covered important aspects of the proposed benchmark and none were discussed in the paper. For example, Shopping Companion evaluates cross-session preference memory in shopping. GroupMemBench evaluates multi-user memory. ShopLife-Bench may contribute a useful combination but the individual capabilities are not new.

  • The low perfect-group scores may need careful interpretation. Combining many requirements across users and sessions naturally leads to a strict requirement and lower score. The paper can provide more insights by running ablations on each component/requirements to illustrate the main bottlenecks.

  • The realism is questionable. Specifically, the simulator restricts patience thresholds to 1/3 errors, while the thresholds cover only 56% of the estimated real distribution.

Questions for authors

See weakness.

Reviewer_qaUrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ShopLife-Bench evaluates e-commerce agents across multiple sessions and users. In the evaluated dataset, each instance contains three sessions with two evaluated users and one distractor interacting in a shared conversation. The benchmark combines persistent structured profiles, shopping tasks involving historical user facts and preferences, error-dependent user behavior, and tool-mediated ordering. Evaluation covers profile maintenance, order correctness, tool compliance, retention, and aggregate zero-error rates. Experiments with six models reveal difficulties in completing these tasks reliably, which the paper interprets as evidence of limitations in cross-user state isolation and cross-session consistency.

Critical strengths
  1. The task design connects historical information to consequential actions. For history-dependent purchase tasks, the ambiguity criterion in Equation (4) requires current-session constraints to leave multiple candidates while historical profile information resolves the choice. This is a useful design principle that connects profile maintenance and use to shopping outcomes.

  2. The environment distinguishes conversational responses from executed state changes. Structured profiles, explicit update operations, and tool-mediated orders make it possible to inspect whether an agent actually persists information and completes an action. Releasing the implementation and task data enables detailed examination of these mechanisms.

Critical weaknesses
  1. The released evaluation implementation has correctness issues affecting central metrics. AI-assisted offline diagnostic checks against the released repository reproduced two counterexamples using native tools and released task data. First, score_profile_updates.py does not require agreement in sentiment: an available candidate recording a negative preference for white desktop peripherals receives full credit where the gold specifies a positive preference. Second, in a constructed case containing two correct orders and an additional order for the distractor using a real catalog item, the session correctly fails the order-count check, but the aggregate summary still counts the group as zero-error despite only two of three sessions passing. Additionally, saved orders lack an explicit buyer identifier, and evaluation preferentially infers ownership from the gold product mapping, limiting independent verification of cross-user order attribution.

These counterexamples concern the released implementation. Without the exact evaluation revision and original trajectories, I cannot establish their prevalence in the reported experiments or quantify their effect on the tables. Until this is clarified, my confidence in the affected profile and group-level zero-error results remains limited.

  1. The interpretation of the results as evidence of cross-user isolation bottlenecks needs stronger support. The component metrics and error analyses are useful, and Sections 5.3 and 6.2 appropriately acknowledge that causal attribution requires controlled experiments and is limited by differences in evaluation units and interaction budgets. Nevertheless, the abstract and introduction make stronger capability-level claims. The group-level zero-error rate measures joint reliability across users, sessions, and checks; its magnitude alone does not identify the contribution of cross-user interference.

There is also a concrete inconsistency: the introduction states that user-level zero-error rates decline for every model, whereas Tables 2 and 5 report single-user versus multi-user user-level rates of 13.0% versus 16.5% for Kimi and 25.0% versus 26.0% for DeepSeek. These point estimates do not establish an improvement, but they do not support a universal decline.

Finally, the benchmark explicitly evaluates a shared-context protocol, instantiated as an e-commerce livestream. This is a reasonable setting, but generalization to shopping assistants with independently routed conversations is not evaluated. The conclusions should distinguish these application settings.

Questions for authors
  1. Which exact code revision and configuration generated the reported tables? Were the sentiment-matching and aggregation issues described above present in that version, and what is the effect of correcting them on the affected metrics? How is buyer identity independently verified when an order contains the correct product? For reproducibility, please specify the candidate-update mode, dated model identifiers, and reasoning settings. Section 3.1 also states that prior in-session tool events are available, whereas the released harness appears to retain tool observations only within the current reply-generation loop; which behavior was used in the experiments?

  2. Could you present the single-user and multi-user comparison consistently using the same user-level success definition and clarify the per-user interaction budgets? A small matched-budget comparison would help quantify the concurrency effect; alternatively, please narrow the causal interpretation of the existing comparison.

  3. Is group 9/user002 intended as a cold-start personalization/control case or as a history-dependent purchase task? Its black extension-cord preference is present initially, none of the three released task scripts discloses that colour, and all three gold update lists contain only noop. The final three-metre request leaves both black and white variants available, while only black is accepted. Please clarify how this case relates to the cold-start filtering and earlier-session deposition rules in Appendix C.3, and how common this pattern is.

Reviewer_qP9vOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents ShopLife-Bench, a benchmark for multi-session, multi-user agent interaction in e-commerce environments.

Critical strengths

-- ShopLife-Bench is built on a 12k+ task corpus, making the content rich

Critical weaknesses

-- It seemed to be too narrowly scoped. I think the evaluation scheme needs significant restructuring to meet the bar of ICLR requirement.

-- The text was hard to follow and seems too verbose. It needs more refinement to be clear and to the point. I did not understand the main contribution very well.

Questions for authors

-- 'We evaluate six representative models on the same 100 instances', how did you pick 100 instances? what was the process of selecting them?

-- any competitive method to show comparison with? The current results seem to be mostly ablation studies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.