- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents a method SimPersona for simulating users with LLM agents in e-commerce applications. The central question is, how to simulate users that realistically mimic the real-world user distribution, which existing text-based persona grounding methods fail in terms of distribution matching and stability. The methodology involves learning user-shop codes with VQ-VAE, and condition the agent & model on the learned user representation codes. The authors propose various algorithmic engineering efforts to fit the framework better to the goal of online applications, such as feature engineering, defining extra behavior axes, etc.
The authors experiment on their own datasets in terms of 4 metrics: how the learned codes maintain user-shop semantic meanings, how the learned codes help mimic the per-shop conversion rate distribution, how the learned codes help shape individual agentic behaviors in terms of purchase/add-to-cart, etc., and how the learned codes help agents accurately complete shopping tasks, where the proposed SimPersona show good potentials in real-world applications and outperform text-based persona methods.
Critical strengths
-
Real-world problem. The authors tackle a real-world problem: designing a user simulator that not only shows desired behaviors, but also shows the behaviors with a similar distribution as the online users. More importantly, this paper aims to address the problem at the industrial level, aiming for online applications. This makes the problem more complex with considerably more design considerations.
-
Practical data engineering with application-specific insights. I appreciate the paper not only aiming to tackle a real-world problem, but also tackling it in a rigorous and application-oriented manner. For example, the feature engineering of the scalar ones, the funnel, the contrastive learning objective with careful design of positive and negative samples, the three behavioral axes of engage, explore and purchase, the two-step fine-tuning with different goals, etc. These designs are not fancy but are critical in real-world LLM-based algorithm engineering, and I appreciate these efforts.
-
Evaluation is done with real-world application insights. First, in terms of evaluating the code, the authors apply metrics related to the funnel and behavior axes, which provide far more semantic meanings than common clustering metrics. The authors also evaluate the trained models in terms of reconstructing actual user behavior distributions, and most importantly, on a sufficiently large and unseen test dataset.
Critical weaknesses
-
Lacking illustrative examples throughout the paper. This paper is dense in terms of engineering insights but notably missing on illustrative examples (i.e. concrete, or at least very similar in terms of detailedness instead of a template). Therefore, I think this paper may benefit from some illustrative examples to showcase the generated data. For example, for Section 3.4, it would be good to show some concrete tasks; for the goals (intent-neutral and non-neutral ones), some examples would also be appropriate.
-
Large-scale results in Sec. 3.2 and 3.3 lack more concrete metrics, such as prices. Intuitively, I think the purchase/add-to-cart rates would be more informative if the actual item information is presented in more details, such as prices, product category, etc., but no such information is present throughout the two sections. This is especially concerning, because in the input channel, view/cart/purchase embeddings, as well as scalar dollar values are present, but no evaluations are present in terms of the exact products the agent interacted with.
-
The metrics about the three behavioral axes in Section 3.1 are somewhat self-referential, mostly because k-means never takes this in mind, and the VQ-VAE is trained with this objective. This is a minor point but can be enhanced with some more discussion.
-
In the paper (Section 2.3), the authors train a Qwen3-14B model for the agent, but in Table 4, the authors compare with a Qwen3.5-9B model trained with textual persona grounding. This leads to a potentially unfair comparison and should be justified.
-
The "average buyer" issue may still exist given only 256 codes. Since the authors experiment with at most 8.37M users, but only 256 codes are present, each code may carry on average 30K users and a median of 2K users (reported in the ethics). Therefore, it is very likely that the "average user effect" may still exist given this scale. Therefore, a discussion on the within-token variance (e.g. depending on goals and shops) may be appropriate to demonstrate that the codes won't become anchors to bound the agent's behavior.
Questions for authors
Please see the 5 weaknesses above.
8 comments
Continue Answer Weakness 1
W1- Example agent call
System message
You are simulating the behavior of a potential human buyer in an ecommerce website. You must answer with a JSON object matching the schema with fields `shouldEnd`, `action`, and `reasoning`. The action contains a `nodeSelector`, `method`, `description`, and `arguments`.
User message.
You are simulating the behavior of a human buyer in an ecommerce website.
You have the following main goal:
<MainGoal>
From the homepage at [store], use the search bar to search for [product].
View the [product] product page, then add it to the cart, then end the session.
</MainGoal>
Your browsing and purchase behavior is highly influenced by your buyer profile.
<BuyerProfile>
<|special_token_51|>
</BuyerProfile>
You log your progress and thoughts along the way. Here is what you have logged so far:
<ProgressLog>
- Started by navigating to [store].
- I clicked the search link to open the search bar.
- I typed [product] into the search bar.
- I clicked the [product] result from the search suggestions.
- I landed on the [product] product page.
</ProgressLog>
<current_page_info>
You are currently on [store]/products/[product].
Here is the accessibility tree of the page, identifying each element with an ID:
<accessibility_tree>
[anonymized ]
</accessibility_tree>
</current_page_info>
<task>
1. Reason about the main goal you are trying to achieve.
2. Review the browsing session <ProgressLog>.
3. Review the current URL and accessibility tree.
4. Generate your output according to the JSON schema provided by the system.
</task>
Assistant response.
{
"shouldEnd": false,
"action": {
"nodeSelector": "0-3136",
"method": "click",
"description": "Click on the ADD TO CART button to add [product] to the shopping cart.",
"arguments": []
},
"reasoning": "I have successfully reached the product page for [product]. Now, I am adding it to my cart as the next step in my goal before I finish my session."
}
The next progress log records that the click succeeded and that the cart contains one item. The following model call then terminates the session:
{
"shouldEnd": true,
"action": null,
"reasoning": "I have successfully searched for [product], viewed its product page, and added it to my cart. Since I have completed all the steps of my goal, I am now ending the session."
}
The episode contains five agent decisions: four browser actions and one terminal decision. The product page is viewed, one item is added to the cart, and the evaluator marks the task as successful. This example illustrates the format of the generated browser actions used in Section 3.4; the aggregate success rates are reported in the quantitative evaluation.
C. Behavioral simulation example. The examples in Section B show deterministic instruction-following tasks, where the agent is explicitly told what product to search for and what endpoint to reach. Behavioral simulation is different. In this setting, the agent receives only a persona token and a short session goal, not a step-by-step instruction. The agent then decides how to navigate the store.
The following example is a recorded SimPersona behavioral-simulation rollout from the main analysis run. The goal is category-level, rather than a deterministic instruction:
Goal: “You are here to buy
[category].”
Persona token:<|special_token_183|>
The agent is not told to search for a specific product or to follow a fixed action sequence. Nevertheless, it generates the following trajectory:
| Step | Generated action |
|---|---|
| 1 | Open the search bar. |
| 2 | Type [product] into the search bar. |
| 3 | Open the [product] product page. |
| 4 | Add the product to cart. |
| 5 | Click checkout. |
| 6 | End the session after reaching checkout. |
The recorded outcome is: six decisions, five successful browser actions, one cart item, cart value [price], checkout reached, and termination label GOAL_REACHED. Reaching checkout does not imply completed payment; the live-store simulations stop before any real financial transaction.
A representative action from this rollout is:
{
"shouldEnd": false,
"action": {
"nodeSelector": "[node_id]",
"method": "click",
"description": "Click on the Add to cart button to add [product] to my shopping cart.",
"arguments": []
},
"reasoning": "I have found the [product] I was looking for, and now I am adding it to my cart so I can proceed to the checkout as planned."
}
This example illustrates the key distinction between instruction following and behavioral simulation. In Section 3.4, the task tells the agent exactly which product to search for and when to stop. In behavioral simulation, the agent receives a persona token and a short intent, then independently chooses the search query, product page, cart action, checkout action, and stopping point. We added these illustrative examples to Appendix L.
Answer Weakness 2
W2- We thank the reviewer for this helpful suggestion. We agree that adding product- and price-level diagnostics makes the large-scale behavioral results more concrete, so we performed the requested analysis.
First, we checked the most direct question: when SimPersona adds to cart or reaches checkout, is the product from the intended product family? Our analysis in the table below shows that category agreement is above 98% for both cart and checkout. “Matched” means at least one carted product belongs to the requested product family; “classifiable” means the product category could be identified.
| Event | Matched / classifiable | Match rate | Matched / all event episodes |
|---|---|---|---|
| Cart | 5,971 / 6,080 | 98.21% | 90.74% |
| Checkout | 3,410 / 3,468 | 98.33% | 90.93% |
Thus, SimPersona is not matching conversion rates by carting arbitrary merchandise; the products are usually in the intended product family.
Second, we asked whether SimPersona carts products from similar price ranges as humans. Because shops differ substantially in currency, product vertical, and catalog scale, we compare prices within each shop rather than using raw global prices. We rank products within each shop by price and divide them into four local price tiers:
| Within-shop product-price percentile | Human cart share | SimPersona cart share | Agent − human |
|---|---|---|---|
| 0–25 | 22.17% | 19.58% | -2.59 pp |
| 25–50 | 25.75% | 21.37% | -4.38 pp |
| 50–75 | 28.86% | 33.07% | +4.21 pp |
| 75–100 | 23.22% | 25.98% | +2.76 pp |
The aggregate differences are small: SimPersona is within roughly 2.6--4.4 percentage points of the human distribution in every price tier. This indicates that SimPersona covers the same broad within-shop price range as human carts. This percentile-based comparison is important because raw dollar values are highly heterogeneous. Across the catalog products used to define the buckets, the standard deviation increases from 148.52 dollars in the lowest quartile to 7,477.72 dollars in the highest quartile. Thus, raw dollar comparisons can be dominated by a small number of very high-priced catalog items, while within-shop percentile tiers provide a more stable comparison across heterogeneous storefronts. Under this metric, the mean human--SimPersona price-tier overlap across shops is 62.17%, where 100% would mean identical price-tier distributions.
Third, we compared product-category distributions beyond the simple category-match check. At the fine-grained assigned catalog-category level, SimPersona achieves 60.46% overlap with human cart-category distributions and 63.44% overlap with human view-category distributions. When categories are grouped into broader families using the first two taxonomy levels, overlap increases to 78.00% for cart and 79.31% for view:
| Category definition | Action | Mean overlap |
|---|---|---|
| Fine-grained assigned catalog category | Cart | 60.46% |
| Fine-grained assigned catalog category | View | 63.44% |
| Coarse category: first two taxonomy levels | Cart | 78.00% |
| Coarse category: first two taxonomy levels | View | 79.31% |
This shows that SimPersona's product interactions are close to human interactions at the broader category-family level, while the finer taxonomy naturally produces lower overlap because it distinguishes more specific product types.
Finally, we compared product semantics using archived product embeddings. For each shop, we compute a centroid of products carted by humans and measure how close SimPersona-carted products are to that centroid. SimPersona carted products have high similarity to the human-carted product centroid 0.858.
These analyzes are added to Appendix J.4 of the revised manuscript.
Answer to Weakness 1
W1- We thank the reviewer for this helpful suggestion. We agree that the paper is dense and benefits from concrete examples of the objects used in the pipeline. Below we provide representative examples of: (i) intent prompts, (ii) deterministic instruction-following tasks, (iii) generated rollouts, and (iv) behavioral-simulation trajectories. Store names, product names, URLs, prices, and other identifying details are anonymized.
A. Intent/goal examples. The behavioral-simulation goals are short intent prompts. They specify the product category and session-level intent, but they do not prescribe a sequence of browser actions. The persona token is supplied separately from the goal text:
The buy-oriented and browse-oriented prompts are used in the standard persona-conditioned simulation setting. The outcome-neutral prompt is used in the neutral-intent ablation in Appendix J.1. It keeps the category of interest but removes the explicit conversion direction. Thus, in the neutral case, the agent is told the topic of interest but is not told whether it should browse, add to cart, proceed to checkout, or stop.
This distinction is important: the goal describes the \emph{session intent}, while the persona token describes the \emph{buyer profile}. SimPersona conditions on both.
B. Section 3.4 task and rollout examples: The deterministic tasks in Section 3.4 are different from the behavioral-simulation goals above. They explicitly prescribe product-level browser actions and stopping conditions. They are used to test whether the agent can execute multi-step shopping instructions on live storefronts.
For example, Section 3.4 includes tasks of the following form:
These examples show that Section 3.4 evaluates instruction-following ability: the task tells the agent what product to find, what action to perform, and when to stop. This is different from the behavioral-simulation setting, where the agent receives only a persona token and a broad session intent.
A generated rollout for a cart-style task has the following structure. In this example, the agent receives a learned persona token and a deterministic cart task:
[product A]into the search bar.[product A]result to open the product page.To make the prompt format concrete, below is one structurally complete agent-call example from the same episode, with the long accessibility tree abbreviated. This is the fourth decision in the trajectory, after the agent has already searched for the product and opened the product page.