- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces UserArena, a benchmark for long-horizon shopping user simulation that supports both single-session and cross-session evaluation across five levels of behavioral fidelity. It also proposes On-Policy Trajectory Distillation, which uses trajectory-level intent and token-level teacher supervision to improve user simulation. Experiments show that OPTD outperforms baselines on multiple fidelity metrics and more closely reproduces real-world recommendation-policy effects in a production A/B test.
Critical strengths
-
Valuable dataset. UserArena provides a large-scale real-world dataset with rich user profiles, visual observations, and interaction histories across multiple sessions. Both its scale and detailed behavioral information could make it a valuable resource for the community.
-
Comprehensive evaluation. The benchmark supports multiple levels of evaluation, from next-action prediction to sequential behavior and population-level outcomes. The validation against a real production A/B test is particularly valuable for assessing the practical usefulness of user simulators.
Critical weaknesses
-
Limited individual-level evaluation. Although the paper proposes a multi-level evaluation framework, I have concerns about whether the metrics adequately capture individual-level user simulation quality. L1 evaluates next-action prediction, which makes sense, and L3 evaluates behavioral differences across user groups. However, if I understand correctly, L2 and L4 primarily compare aggregate statistics between real and simulated populations, rather than computing fidelity for each individual user and then averaging across users (please correct me if I am mistaken). This makes the evaluation less informative about individual user behavior. For example, a simulator could achieve a perfect terminal-state distribution score (T-JSD = 0) even if it predicts purchases for entirely different users or different products than those actually purchased. Similarly, trajectory error (TE) only compares average session lengths, meaning that individual trajectories could differ substantially while still achieving a perfect score. A model that learns population-level behavioral distributions could therefore achieve strong L2/L4 scores without accurately simulating individual users, which is particularly concerning given the paper's emphasis on personalized, long-horizon user simulation.
-
Reliability of trajectory-level intent supervision. Although the ablation study shows that trajectory-level intent is useful, I have concerns about how reliable the extracted intent is and whether it could introduce bias into the teacher's supervision. In the cross-session setting, intent is summarized from multiple sessions, but user preferences and intentions may change over time. A user may not have certain intentions in earlier sessions, and even within a single shopping session, their intent may shift as they explore different products. Therefore, a single intent summary extracted from the complete trajectory may not accurately reflect the user's intent at every step. It would be helpful to further validate the quality of the extracted intents and investigate how OPTD handles evolving user intentions.
Questions for authors
-
Session definition. How are sessions defined in the dataset, and how is the "End Session" action determined? Is it based on a time interval, an explicit user action, or some other criteria? I ask because session boundaries may affect the validity of the extracted intents. For example, if sessions are split based on inactivity thresholds, a single shopping intent might span multiple sessions, while a user could also develop different intents within the same session. It would be helpful to clarify how sessions are segmented and whether this affects intent extraction and evaluation.
-
User profile construction and effectiveness. How are user profiles obtained? The appendix lists several user attributes, but it is unclear whether these are self-reported by users, inferred from their behavior, or generated by an LLM. Additionally, have the authors conducted any ablation studies on the contribution of user profiles? Given that the listed attributes are relatively simple, I wonder how informative they are for predicting user behavior and how much they actually contribute to simulation performance. A comparison with and without user profiles would be helpful.
5 comments
Reply to Reviewer 8fUA (Part 2)
W2. Contribution of complete-trajectory intent.
Reply: Thank you for raising this concern. First, we would like to clarify how OPTD uses the extracted intent. The intent is a structured summary of preferences, concerns, information needs, and conditional decision tendencies, providing behavioral context for the teacher. Although the original intent record is shared across sessions, the teacher also conditions on the current observation, interaction history, and student-generated prefix. This allows the teacher’s guidance to vary with the current interaction context. A per-token KL budget limits the influence of intent, while the grounding loss anchors supervision to recorded behavior. Together, these mechanisms allow context-dependent guidance while constraining the influence of the trajectory-level summary.
To reduce step-specific answer leakage, the extraction and validation rules exclude concrete actions, specific products and identifiers, exact prices, trajectory outcomes, future-page content, and supporting step indices from the intent summary. These rules are designed to keep the supervision focused on high-level behavioral context.
To further examine whether a shared trajectory-level summary overlooks changes in user intent, we compared three alternatives: session-level intent from each complete session, centered-window intent from the current decision and up to four decisions on either side within the same session, and historical-prefix intent from interactions strictly before the current action. Session-level and centered-window summaries provide more temporally localized context, while historical-prefix intent excludes subsequent interactions from intent extraction. These comparisons assess how the temporal scope of intent supervision affects behavioral fidelity. The table reports cross-session results.
Method F1 ↑ Acc. ↑ DM-JSD ↓ TE ↓ PF ↑ ΔCTR ↓ ΔACR ↓ T-JSD ↓ OPTD 0.3456 0.2969 0.0284 0.72 0.8929 0.0357 0.1399 0.0077 Session-level intent 0.3434 0.2922 0.0353 0.43 0.9002 0.0357 0.1306 0.0127 Centered-window intent (±4) 0.3431 0.2909 0.0327 0.48 0.9000 0.0338 0.1294 0.0133 Historical prefix 0.3444 0.2965 0.0307 0.58 0.8986 0.0359 0.0812 0.0125
As shown in the table, OPTD achieves the highest F1 and Acc. and the lowest DM-JSD and T-JSD among the compared extraction scopes. One possible explanation is that complete trajectories help distinguish persistent preferences from temporary exploration and reveal recurring tendencies across sessions. With this broader context, intent inference may draw on more complete and consistent evidence, supporting coherent teacher supervision across interactions. More temporally localized or historical-only summaries also offer benefits: session-level intent achieves the lowest TE and highest PF, centered-window intent achieves the lowest ΔCTR, and historical-prefix intent achieves the lowest ΔACR. These results suggest that the temporal scope of intent supervision affects different aspects of behavioral fidelity, with complete-trajectory context benefiting local prediction, decision-mix fidelity, and terminal outcomes, and alternative scopes improving selected behavioral statistics.
Taken together, none of the alternative extraction scopes consistently outperforms OPTD across the evaluated dimensions, supporting complete-trajectory intent as an effective source of teacher guidance. OPTD’s contribution lies in translating this behavioral context into bounded, behaviorally grounded supervision on student-generated responses. We have added these comparisons to Appendix C.4 of the revised manuscript.
Reply to Reviewer 8fUA (Part 3)
Q1. Session definition and intent extraction.
Reply: Thank you for raising this question. A session starts when a user enters the shopping scenario in the app and ends when the user leaves it, rather than being split by an inactivity threshold. We mark the end of each recorded session with an “End Session” action.
For intent extraction, we do not assume that sessions and shopping goals correspond one-to-one. In the cross-session setting, we extract a structured summary from the user’s complete training trajectory. It captures multiple preferences, concerns, information needs, and conditional decision tendencies, rather than imposing one fixed goal on every session. The teacher uses this context alongside the current state and interaction history to guide each decision. This preserves relevant evidence when a goal spans multiple sessions.
Session boundaries determine trajectory lengths and termination points. We use users’ actual entry into and exit from the shopping scenario to define these boundaries. Our evaluation measures how closely simulated interactions and stopping behavior match the recorded behavior, rather than whether a shopping goal has been completed. This behavioral evaluation therefore does not require a one-to-one correspondence between sessions and shopping goals. We have clarified the session definition in Appendix A.2 of the revised manuscript.
Q2. User profile construction and effectiveness.
Reply: Thank you for raising this question. We use the profile information (e.g., age, gender, and city) reported by users on the platform as static profile attributes. These attributes are neither inferred from their behavior nor generated by an LLM. Recent clicks, purchases, and interaction histories come from recorded user behavior. To assess the contribution of these static attributes, we conducted an ablation removing only these attributes while retaining recent clicks, purchases, and interaction history. The table reports single-session results.
Method F1 ↑ Acc. ↑ DM-JSD ↓ TE ↓ PF ↑ ΔCTR ↓ ΔACR ↓ T-JSD ↓ OPTD 0.4178 0.3204 0.0156 1.72 0.8692 0.0026 0.0430 0.0387 w/o static profile 0.4101 0.3116 0.0175 1.92 0.8787 0.0018 0.0379 0.0419
These results suggest that age, gender, and city provide limited additional benefit when behavioral history is retained. These static attributes may not capture specific shopping habits, whereas recent clicks, purchases, and interactions may provide more direct evidence of user preferences and decision patterns. We have clarified that user-profile attributes are self-reported in Appendix A.2 of the revised manuscript.
Thank you again for your thoughtful questions. We hope the additional evaluations and clarifications address your concerns.
Reply to Reviewer 8fUA (Part 1)
Dear Reviewer 8fUA,
Thank you for your suggestions on individual-level evaluation and the reliability of intent supervision. We address these points and your questions about sessions and user profiles below.
W1. Per-user evaluation.