- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge of fine-grained, multimodal trajectory generation for modeling online user behavior by proposing the SimTrace framework. The approach begins by selecting representative sessions from real-world user activity logs, which are then standardized and anonymized to extract user personas and intents. Next, an autonomous agent, powered by programming capabilities, constructs a simulated sandbox environment that mirrors the original website. Finally, a large-model-based agent, equipped with a dual-layer planning and verification mechanism, interacts with this simulated environment based on the real-world trajectories. This process generates synthetic clickstreams that encompass user actions, web page code, and screenshots. The authors also release a synthetic dataset spanning multiple e-commerce platforms and languages, and they validate the effectiveness of the generated data by evaluating its behavioral fidelity, its utility in training models for downstream tasks, and its capacity for data augmentation in low-resource scenarios.
Critical strengths
-
The problem addressed by the authors is significant and holds substantial practical value. Real-world user interaction data characterized by fine granularity and web-based observations is both scarce and subject to privacy constraints. Thus, the use of synthetic data to alleviate this bottleneck directly targets a critical pain point in the field.
-
The paper is well-structured and clearly written, making for a smooth reading experience. Furthermore, the authors have made their complete codebase and datasets publicly available, covering everything from log processing and the construction of simulated websites to trajectory generation. They also provide a pragmatic analysis of real-world challenges encountered during the generation process, such as granularity mismatch and interface drift.
Critical weaknesses
-
The methodology section consists almost entirely of long, descriptive paragraphs, lacking both theoretical analysis and precise mathematical definitions. Whether it concerns the formal definition of the problem, the selection of representative sessions, the validation via simulated websites, or the planning and verification mechanisms for the dual-layer agent framework, the text remains purely qualitative. It fails to provide clear mathematical formulations or structured algorithmic workflows; the proposed method should be presented in a more organized and formal manner.
-
The argument and evaluation regarding whether synthetic data can adequately simulate real human behavior are insufficient. Human browsing and decision-making behaviors are highly complex, and automated metrics, such as distribution divergence, sequence distance, and semantic similarity, cannot capture the full picture. More critically, since the synthetic trajectories are generated by reconstructing them based on specific reference sessions, using these metrics to assess their similarity to the original trajectories or using paired synthetic data for downstream training, largely measures only the fidelity of the reconstruction. It does not truly demonstrate that the model has captured the underlying logic of complex human behavior. Furthermore, the experiments on data augmentation in low-resource settings lack comparisons with other synthetic data generation methods under equivalent conditions.
-
The paper posits that synthetic data can serve as a substitute for sensitive logs, allowing for external data sharing; however, it lacks an empirical assessment of its privacy-protection efficacy. Given that synthetic trajectories are generated by referencing individual real-world sessions, higher fidelity in reconstruction increases the likelihood that the original users' behavioral characteristics are preserved. Simple measures, such as hashing, attribute abstraction, or the replacement of sensitive content, cannot entirely eliminate the risk of exposing user behavioral privacy, yet the paper provides no empirical validation to address this concern.
Questions for authors
-
It is recommended to further structure the Methodology section. The current long, text-heavy descriptions should be replaced or supplemented with formal problem definitions, mathematical formulations for key modules, or algorithmic workflows. This would make the design logic of the proposed framework clearer and more rigorous.
-
Existing statistical, sequential, and semantic metrics only capture surface-level distributional similarities and struggle to comprehensively evaluate complex human interaction behaviors. If the framework were tasked with generating new trajectories on a website based solely on user personas and intents, without relying on existing, ground-truth reference trajectories, would it still maintain behavioral fidelity? What additional dimensions (e.g., human evaluation or in-depth behavioral pattern analysis) could the authors explore to further demonstrate that the synthetic data truly approximates authentic human behavior?
-
In the context of downstream tasks or low-resource scenarios, how would the performance compare if synthetic data generated by other baseline methods were used for data augmentation instead? Furthermore, how can it be proven that any observed improvements in downstream performance are specifically attributable to the proposed environment cloning and two-layer verification mechanism, rather than simply resulting from an increase in the total number of training samples?
-
Given that the synthetic data is generated by referencing individual real-world logs and is intended for public release as an open-source dataset, it is advisable to include a quantitative or empirical assessment of privacy protection. Specifically, the authors should clarify how they ensure that the generated trajectories do not compromise user privacy or expose sensitive information belonging to the original businesses---risks that could arise if the synthetic data too closely resembles the original records.
7 comments
Reviewer S8Fr Response Pt2
| Method | Metric ↑ | TRTR | TSTR | Δ | 95% CI for Δ |
|---|---|---|---|---|---|
| Purchase Prediction | |||||
| Qwen3.5-4B SFT | F1 | 65.78 | 61.9 | -3.88 | [−11.22, +3.64] |
| Session-based Recommendation | |||||
| RAIN | MRR@5 | 12.76 | 10.62 | -2.14 | [−3.73, −0.54] |
| RAIN | HR@5 | 21.04 | 17.51 | -3.53 | [−5.72, −1.27] |
| RAIN | MRR@10 | 13.6 | 11.55 | -2.05 | [−3.62, −0.47] |
| RAIN | HR@10 | 27.71 | 24.24 | -3.47 | [−5.86, −1.07] |
| DIMO | MRR@5 | 17.74 | 16.72 | -1.02 | [−2.71, +0.67] |
| DIMO | HR@5 | 32.35 | 30.88 | -1.47 | [−3.95, +1.02] |
| DIMO | MRR@10 | 18.69 | 17.5 | -1.19 | [−3.44, -0.15] |
| DIMO | HR@10 | 39.6 | 36.57 | -3.03 | [−5.51, −0.64] |
Reference:
[1] Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Yang Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, and Dakuo Wang. UXAgent: An LLM agent-based usability testing framework for web design. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2025. doi: 10.1145/3706599.3719729.
Q4: Given that the synthetic data is generated by referencing individual real-world logs and is intended for public release as an open-source dataset, it is advisable to include a quantitative or empirical assessment of privacy protection. Specifically, the authors should clarify how they ensure that the generated trajectories do not compromise user privacy or expose sensitive information belonging to the original businesses---risks that could arise if the synthetic data too closely resembles the original records.
Answer: Thank you for raising this concern. We agree that identifier pseudonymization and price bucketing alone leave open the possibility of linking a synthetic trajectory to its real reference. Based on your suggestion, we therefore conducted additional experiments to evaluate disclosure risk in SimTrace using two methods: membership inference and source-session re-identification. Both of the results suggest limited disclosure risk. We will add the results of both experiments to the paper.
Here are the details:
Membership inference: It assesses whether a candidate real session served as a reference for generating a synthetic trajectory. Following similarity-based membership inference attacks (MIA) [1,2], the attacker compares the candidate with every released synthetic trajectory and assigns it the highest pairwise similarity score. The attacker predicts membership when that score meets a decision threshold. Each pairwise score is a weighted average of action similarity, computed as one minus the normalized Levenshtein distance between the ordered action sequences, and context similarity, computed as the cosine similarity between embeddings of product handles and descriptions. We vary the weights and test alternative context representations to identify the strongest attack, which represents the highest disclosure risk.
We report (1) ROC-AUC [2,3,4], where values near 0.5 indicate the attack cannot distinguish whether data used for generating synthetic data or not; (2) precision [1,3], where values smaller than 0.5 indicate that the attack is no more successful than a random guess; and (3) TPR at 1% FPR [3, 4], where lower values indicate attacker cannot confidently identify any members of the synthetic dataset. For an empirical evaluation, we estimate 95% confidence intervals by bootstrapping users.
The sweep yields a maximum ROC-AUC of 0.5404 (95% CI 0.5224–0.5648), indicating modest discrimination between reference and nonreference sessions. For context, the Synth-SERL assessment uses an ROC-AUC of 0.55 or below as its Pass status and values above 0.55 through 0.60 as its Conditional band for releasing synthetic household energy data [7]. Our point estimate falls below the Pass threshold, providing a reference point for interpreting the magnitude of the observed discrimination.
At the nominal 1% FPR, the attack achieves 57.14% precision (95% CI 40.54%–73.53%). This interval includes the 50% random-guess baseline. The attack achieved 1.32% TPR (95% CI 0.86% - 2.03%), which indicates an uncertainty about the attack’s advantage at that operating point. For comparison, Zhang et al. report maximum precisions of 55% and 64% for two released synthetic health datasets [1]. Ward et al. report TPRs of 1.3%–5.3% at 1% FPR across attack methods on the synthetic tabular benchmark [8]. Our precision and TPR fall near the lower ends of these reported values. This pattern would suggest limited membership leakage and provide support for releasing SimTrace from a membership-inference perspective.
| Action / Context Weight | ROC-AUC | Precision | TPR at 1% FPR |
|---|---|---|---|
| 1 / 0 | 0.5345 | 50.00% | 0.99% |
| 0.8 / 0.2 | 0.5404 | 55.88% | 1.26% |
| 0.6 / 0.4 | 0.5386 | 57.14% | 1.32% |
| 0.5 / 0.5 | 0.5295 | 55.88% | 1.26% |
| 0.4 / 0.6 | 0.5264 | 55.88% | 1.26% |
| 0.2 / 0.8 | 0.5157 | 55.88% | 1.26% |
| 0 / 1 | 0.5089 | 46.43% | 0.86% |
Reviewer S8Fr Response Pt3
Re-identification through source-session linkage: It assesses whether an attacker can recover the exact real session used to generate a synthetic trajectory. Following the trajectory-linkage protocol of Jin et al. [5], the attacker treats each released synthetic trajectory as a query and ranks candidate real sessions. We evaluate two attacks. The similarity ranker [5] orders candidates by their similarity to the query in action sequence and context. The retriever + reranker uses the same similarity score to retrieve the top 20 candidates, then applies an LLM to reorder the shortlist, adapting the LLM reranking approach of Gangi Reddy et al. [6]. We report Top@1 and Top@5 [5], the fractions of queries for which the exact generation reference appears first or among the first five results, respectively.
We evaluate source-session linkage under two levels of attacker knowledge. In the first setting, the attacker searches only the 1,513 real sessions used as generation references. This is the more favorable setting for the attacker because every candidate is known to have served as a reference. In the second, more practical setting, the attacker knows the source shop and the relevant one-month window and searches all real sessions in that shop’s interaction log for the period.
The first setting yields the highest observed identification rates for the tested attacks, with the stronger attack reaching 0.925% Top@1 and 2.115% Top@5. In the more practical setting, the attacker searches the one-month shop log without knowing which sessions served as generation references. Both attacks recover fewer exact references, with the stronger attack reaching 0.330% Top@1 and 1.322% Top@5. The recovery will become harder if the attacker had less precise information about the shop or time window. Therefore it suggest limited re-identification risk.
| Candidate pool | Similarity ranker Top@1 / Top@5 | Retriever + reranker Top@1 / Top@5 |
|---|---|---|
| 1,513 ground-truth real sessions | 0.793% / 1.851% | 0.925% / 2.115% |
| 1-month real sessions from the known source shop | 0.198% / 0.727% | 0.330% / 1.322% |
Taken together, the results of both experiments suggest that SimTrace poses minimal disclosure risk under the conditions tested.
References
[1] Ziqi Zhang, Chao Yan, and Bradley A. Malin. 2022. “Membership Inference Attacks Against Synthetic Health Data.” Journal of Biomedical Informatics, 125, Article 103977. PDF
[2] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. 2020. “GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models.” In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 343–362. PDF
[3] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. 2022. “Membership Inference Attacks From First Principles.” In Proceedings of the 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. PDF
[4] Myeongseob Ko, Ming Jin, Chenguang Wang, and Ruoxi Jia. 2023. “Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study.” In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4871–4881. PDF
[5] Fengmei Jin, Wen Hua, Matteo Francia, Pingfu Chao, Maria E. Orlowska, and Xiaofang Zhou. 2023. “A Survey and Experimental Study on Privacy-Preserving Trajectory Data Publishing.” IEEE Transactions on Knowledge and Data Engineering, 35(6), pp. 5577–5596. PDF
[6] Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. 2024. “FIRST: Faster Improved Listwise Reranking with Single Token Decoding.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8642–8652. Article
[7] Minnie Ashdown, Tadj Oreszczyn, and Eoghan McKenna. 2026. Membership Inference Risk in Synthetic Household Energy Data: A Privacy Evaluation Framework for Synth-SERL. University College London. PDF
[8] Joshua Ward, Chi-Hua Wang, and Guang Cheng. 2026. Privacy Auditing Synthetic Data Release through Local Likelihood Attacks. arXiv:2508.21146v2. PDF
W1: The improvement for the methodology section.
Answer: See response to Question 1.
W2: The evaluation regarding whether synthetic data can simulate real human behavior are insufficient.
Answer: See response to Question 2 and 3.
W3: The paper lacks assessment of its privacy-protection efficacy.
Answer: See response to Question 4.
Reviewer S8Fr Response
Q1: It is recommended to further structure the Methodology section. The current long, text-heavy descriptions should be replaced or supplemented with formal problem definitions, mathematical formulations for key modules, or algorithmic workflows. This would make the design logic of the proposed framework clearer and more rigorous.
Answer: Thank you for this helpful suggestion. We will revise the Methodology section to improve readability and make the design of SimTrace easier to follow. We will begin with a formal problem definition that specifies the inputs (real session logs and the source website) and outputs (a runnable twin environment and synthetic trajectories that pair each action with its corresponding web observations). We will also add concise pseudocode for environment construction, following the format of the session-selection algorithm in Appendix A, to show how website exploration, twin generation, and iterative verification fit together.
Q2: Existing statistical, sequential, and semantic metrics only capture surface-level distributional similarities and struggle to comprehensively evaluate complex human interaction behaviors. If the framework were tasked with generating new trajectories on a website based solely on user personas and intents, without relying on existing, ground-truth reference trajectories, would it still maintain behavioral fidelity? What additional dimensions (e.g., human evaluation or in-depth behavioral pattern analysis) could the authors explore to further demonstrate that the synthetic data truly approximates authentic human behavior?
Answer: Thank you for raising this question. The Persona baseline in Table 4 generates trajectories from personas and intents without access to the reference action sequences. It shows lower behavioral fidelity overall than SimTrace, highlighting the value of reference guidance in our setting.
We agree that human behavior is complex and how to simulate it and what metrics to assess its fidelity remain open questions. Shoppers with the same intent can take several plausible paths, and no single metric captures the full range of those behaviors. We therefore use two common retail tasks to assess whether the synthetic trajectories preserve specific behavioral signals. Purchase prediction tests whether the interaction history contains signals about a shopper’s eventual purchase decision. Session-based recommendation tests whether prior interactions help predict which product the shopper will interact with next. Models trained on synthetic sessions achieve comparable performance to those trained on real sessions on both tasks. This provides complementary evidence that the synthetic trajectories preserve predictive signals about shopping behavior beyond the distributional similarities captured by our fidelity metrics. We will clarify the scope of these findings in the paper and discuss psychologically grounded evaluation as future work.
Q3: In the context of downstream tasks or low-resource scenarios, how would the performance compare if synthetic data generated by other baseline methods were used for data augmentation instead? Furthermore, how can it be proven that any observed improvements in downstream performance are specifically attributable to the proposed environment cloning and two-layer verification mechanism, rather than simply resulting from an increase in the total number of training samples?
Answer: To examine whether SimTrace’s gains in fidelity translate into better downstream performance, we conduct downstream experiments using synthetic data generated by the strongest baseline: UXAgent [1]. We found that our method is better than its performance.
Besides, UXAgent's planning and reflection pipeline essentially corresponds to SimTrace’s outer loop. Therefore, the gap in downstream performance suggests the contribution of SimTrace’s two-layer verification mechanism.
The table below shows UXAgent’s results for purchase prediction and session-based recommendation under the TRTR/TSTR setup described in the paper. All nine TSTR scores are lower than their corresponding TRTR scores, and most of the 95% confidence intervals for these differences exclude zero. For those metrics, we can reject the null hypothesis of equal performance between the two training conditions. In contrast, all confidence intervals from the SimTrace method include zero. These comparisons suggest that SimTrace’s synthetic trajectories better preserve the training signal needed for downstream tasks.