acceptodds
Under review as a conference paper at ICLR 2027

SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modeling

Abstract

Virtual clients offer a cost-effective approach to support applications such as A/B testing, recommender system development, and interface evaluation. However, building them requires access to large-scale, semantically faithful, fine-grained online user trajectories. These data are difficult to obtain because proprietary logs are subject to privacy restrictions and small businesses often lack sufficient traffic. Consequently, existing public datasets either abstract away fine-grained user interaction details or preserve rich context but remain platform-specific and small-scale. To address this gap, we propose SimTrace, a framework that generates faithful, fine-grained synthetic multimodal clickstreams through a computer-use client agent that is grounded in real user trajectories and the given web environment. SimTrace anonymizes real interactions and constructs a simulated twin of the given web environment, then uses both to generate synthetic interaction trajectories. Each action is paired with its corresponding web observations and user context, yielding a shareable alternative to confidential logs for developing computer-use agent-style virtual clients. We apply SimTrace to an e-commerce setting and evaluate both its fidelity and downstream utility. SimTrace outperforms competing baselines on 7 out of 8 fidelity metrics. Models trained on synthetic data achieve performance comparable to those trained on real data on downstream tasks such as purchase prediction and recommendation. For next action prediction task, augmenting real data with synthetic data further improves accuracy by 11.0% relative to training on real data alone. We release SimTrace as an open-source package to facilitate research on online user behavior modeling.

Primary area:
datasets and benchmarks
Keywords:
user simulator, data generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S8FrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenge of fine-grained, multimodal trajectory generation for modeling online user behavior by proposing the SimTrace framework. The approach begins by selecting representative sessions from real-world user activity logs, which are then standardized and anonymized to extract user personas and intents. Next, an autonomous agent, powered by programming capabilities, constructs a simulated sandbox environment that mirrors the original website. Finally, a large-model-based agent, equipped with a dual-layer planning and verification mechanism, interacts with this simulated environment based on the real-world trajectories. This process generates synthetic clickstreams that encompass user actions, web page code, and screenshots. The authors also release a synthetic dataset spanning multiple e-commerce platforms and languages, and they validate the effectiveness of the generated data by evaluating its behavioral fidelity, its utility in training models for downstream tasks, and its capacity for data augmentation in low-resource scenarios.

Critical strengths
  1. The problem addressed by the authors is significant and holds substantial practical value. Real-world user interaction data characterized by fine granularity and web-based observations is both scarce and subject to privacy constraints. Thus, the use of synthetic data to alleviate this bottleneck directly targets a critical pain point in the field.

  2. The paper is well-structured and clearly written, making for a smooth reading experience. Furthermore, the authors have made their complete codebase and datasets publicly available, covering everything from log processing and the construction of simulated websites to trajectory generation. They also provide a pragmatic analysis of real-world challenges encountered during the generation process, such as granularity mismatch and interface drift.

Critical weaknesses
  1. The methodology section consists almost entirely of long, descriptive paragraphs, lacking both theoretical analysis and precise mathematical definitions. Whether it concerns the formal definition of the problem, the selection of representative sessions, the validation via simulated websites, or the planning and verification mechanisms for the dual-layer agent framework, the text remains purely qualitative. It fails to provide clear mathematical formulations or structured algorithmic workflows; the proposed method should be presented in a more organized and formal manner.

  2. The argument and evaluation regarding whether synthetic data can adequately simulate real human behavior are insufficient. Human browsing and decision-making behaviors are highly complex, and automated metrics, such as distribution divergence, sequence distance, and semantic similarity, cannot capture the full picture. More critically, since the synthetic trajectories are generated by reconstructing them based on specific reference sessions, using these metrics to assess their similarity to the original trajectories or using paired synthetic data for downstream training, largely measures only the fidelity of the reconstruction. It does not truly demonstrate that the model has captured the underlying logic of complex human behavior. Furthermore, the experiments on data augmentation in low-resource settings lack comparisons with other synthetic data generation methods under equivalent conditions.

  3. The paper posits that synthetic data can serve as a substitute for sensitive logs, allowing for external data sharing; however, it lacks an empirical assessment of its privacy-protection efficacy. Given that synthetic trajectories are generated by referencing individual real-world sessions, higher fidelity in reconstruction increases the likelihood that the original users' behavioral characteristics are preserved. Simple measures, such as hashing, attribute abstraction, or the replacement of sensitive content, cannot entirely eliminate the risk of exposing user behavioral privacy, yet the paper provides no empirical validation to address this concern.

Questions for authors
  1. It is recommended to further structure the Methodology section. The current long, text-heavy descriptions should be replaced or supplemented with formal problem definitions, mathematical formulations for key modules, or algorithmic workflows. This would make the design logic of the proposed framework clearer and more rigorous.

  2. Existing statistical, sequential, and semantic metrics only capture surface-level distributional similarities and struggle to comprehensively evaluate complex human interaction behaviors. If the framework were tasked with generating new trajectories on a website based solely on user personas and intents, without relying on existing, ground-truth reference trajectories, would it still maintain behavioral fidelity? What additional dimensions (e.g., human evaluation or in-depth behavioral pattern analysis) could the authors explore to further demonstrate that the synthetic data truly approximates authentic human behavior?

  3. In the context of downstream tasks or low-resource scenarios, how would the performance compare if synthetic data generated by other baseline methods were used for data augmentation instead? Furthermore, how can it be proven that any observed improvements in downstream performance are specifically attributable to the proposed environment cloning and two-layer verification mechanism, rather than simply resulting from an increase in the total number of training samples?

  4. Given that the synthetic data is generated by referencing individual real-world logs and is intended for public release as an open-source dataset, it is advisable to include a quantitative or empirical assessment of privacy protection. Specifically, the authors should clarify how they ensure that the generated trajectories do not compromise user privacy or expose sensitive information belonging to the original businesses---risks that could arise if the synthetic data too closely resembles the original records.

7 comments
AuthorsOpenReview ↗

Reviewer S8Fr Response

Q1: It is recommended to further structure the Methodology section. The current long, text-heavy descriptions should be replaced or supplemented with formal problem definitions, mathematical formulations for key modules, or algorithmic workflows. This would make the design logic of the proposed framework clearer and more rigorous.

Answer: Thank you for this helpful suggestion. We will revise the Methodology section to improve readability and make the design of SimTrace easier to follow. We will begin with a formal problem definition that specifies the inputs (real session logs and the source website) and outputs (a runnable twin environment and synthetic trajectories that pair each action with its corresponding web observations). We will also add concise pseudocode for environment construction, following the format of the session-selection algorithm in Appendix A, to show how website exploration, twin generation, and iterative verification fit together.

Q2: Existing statistical, sequential, and semantic metrics only capture surface-level distributional similarities and struggle to comprehensively evaluate complex human interaction behaviors. If the framework were tasked with generating new trajectories on a website based solely on user personas and intents, without relying on existing, ground-truth reference trajectories, would it still maintain behavioral fidelity? What additional dimensions (e.g., human evaluation or in-depth behavioral pattern analysis) could the authors explore to further demonstrate that the synthetic data truly approximates authentic human behavior?

Answer: Thank you for raising this question. The Persona baseline in Table 4 generates trajectories from personas and intents without access to the reference action sequences. It shows lower behavioral fidelity overall than SimTrace, highlighting the value of reference guidance in our setting.

We agree that human behavior is complex and how to simulate it and what metrics to assess its fidelity remain open questions. Shoppers with the same intent can take several plausible paths, and no single metric captures the full range of those behaviors. We therefore use two common retail tasks to assess whether the synthetic trajectories preserve specific behavioral signals. Purchase prediction tests whether the interaction history contains signals about a shopper’s eventual purchase decision. Session-based recommendation tests whether prior interactions help predict which product the shopper will interact with next. Models trained on synthetic sessions achieve comparable performance to those trained on real sessions on both tasks. This provides complementary evidence that the synthetic trajectories preserve predictive signals about shopping behavior beyond the distributional similarities captured by our fidelity metrics. We will clarify the scope of these findings in the paper and discuss psychologically grounded evaluation as future work.

Q3: In the context of downstream tasks or low-resource scenarios, how would the performance compare if synthetic data generated by other baseline methods were used for data augmentation instead? Furthermore, how can it be proven that any observed improvements in downstream performance are specifically attributable to the proposed environment cloning and two-layer verification mechanism, rather than simply resulting from an increase in the total number of training samples?

Answer: To examine whether SimTrace’s gains in fidelity translate into better downstream performance, we conduct downstream experiments using synthetic data generated by the strongest baseline: UXAgent [1]. We found that our method is better than its performance.

Besides, UXAgent's planning and reflection pipeline essentially corresponds to SimTrace’s outer loop. Therefore, the gap in downstream performance suggests the contribution of SimTrace’s two-layer verification mechanism.

The table below shows UXAgent’s results for purchase prediction and session-based recommendation under the TRTR/TSTR setup described in the paper. All nine TSTR scores are lower than their corresponding TRTR scores, and most of the 95% confidence intervals for these differences exclude zero. For those metrics, we can reject the null hypothesis of equal performance between the two training conditions. In contrast, all confidence intervals from the SimTrace method include zero. These comparisons suggest that SimTrace’s synthetic trajectories better preserve the training signal needed for downstream tasks.

AuthorsOpenReview ↗

Reviewer S8Fr Response Pt2

MethodMetric ↑TRTRTSTRΔ95% CI for Δ
Purchase Prediction
Qwen3.5-4B SFTF165.7861.9-3.88[−11.22, +3.64]
Session-based Recommendation
RAINMRR@512.7610.62-2.14[−3.73, −0.54]
RAINHR@521.0417.51-3.53[−5.72, −1.27]
RAINMRR@1013.611.55-2.05[−3.62, −0.47]
RAINHR@1027.7124.24-3.47[−5.86, −1.07]
DIMOMRR@517.7416.72-1.02[−2.71, +0.67]
DIMOHR@532.3530.88-1.47[−3.95, +1.02]
DIMOMRR@1018.6917.5-1.19[−3.44, -0.15]
DIMOHR@1039.636.57-3.03[−5.51, −0.64]

Reference:

[1] Yuxuan Lu, Bingsheng Yao, Hansu Gu, Jing Huang, Jessie Wang, Yang Li, Jiri Gesi, Qi He, Toby Jia-Jun Li, and Dakuo Wang. UXAgent: An LLM agent-based usability testing framework for web design. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, 2025. doi: 10.1145/3706599.3719729.

Q4: Given that the synthetic data is generated by referencing individual real-world logs and is intended for public release as an open-source dataset, it is advisable to include a quantitative or empirical assessment of privacy protection. Specifically, the authors should clarify how they ensure that the generated trajectories do not compromise user privacy or expose sensitive information belonging to the original businesses---risks that could arise if the synthetic data too closely resembles the original records.

Answer: Thank you for raising this concern. We agree that identifier pseudonymization and price bucketing alone leave open the possibility of linking a synthetic trajectory to its real reference. Based on your suggestion, we therefore conducted additional experiments to evaluate disclosure risk in SimTrace using two methods: membership inference and source-session re-identification. Both of the results suggest limited disclosure risk. We will add the results of both experiments to the paper.

Here are the details:

Membership inference: It assesses whether a candidate real session served as a reference for generating a synthetic trajectory. Following similarity-based membership inference attacks (MIA) [1,2], the attacker compares the candidate with every released synthetic trajectory and assigns it the highest pairwise similarity score. The attacker predicts membership when that score meets a decision threshold. Each pairwise score is a weighted average of action similarity, computed as one minus the normalized Levenshtein distance between the ordered action sequences, and context similarity, computed as the cosine similarity between embeddings of product handles and descriptions. We vary the weights and test alternative context representations to identify the strongest attack, which represents the highest disclosure risk.

We report (1) ROC-AUC [2,3,4], where values near 0.5 indicate the attack cannot distinguish whether data used for generating synthetic data or not; (2) precision [1,3], where values smaller than 0.5 indicate that the attack is no more successful than a random guess; and (3) TPR at 1% FPR [3, 4], where lower values indicate attacker cannot confidently identify any members of the synthetic dataset. For an empirical evaluation, we estimate 95% confidence intervals by bootstrapping users.

The sweep yields a maximum ROC-AUC of 0.5404 (95% CI 0.5224–0.5648), indicating modest discrimination between reference and nonreference sessions. For context, the Synth-SERL assessment uses an ROC-AUC of 0.55 or below as its Pass status and values above 0.55 through 0.60 as its Conditional band for releasing synthetic household energy data [7]. Our point estimate falls below the Pass threshold, providing a reference point for interpreting the magnitude of the observed discrimination.

At the nominal 1% FPR, the attack achieves 57.14% precision (95% CI 40.54%–73.53%). This interval includes the 50% random-guess baseline. The attack achieved 1.32% TPR (95% CI 0.86% - 2.03%), which indicates an uncertainty about the attack’s advantage at that operating point. For comparison, Zhang et al. report maximum precisions of 55% and 64% for two released synthetic health datasets [1]. Ward et al. report TPRs of 1.3%–5.3% at 1% FPR across attack methods on the synthetic tabular benchmark [8]. Our precision and TPR fall near the lower ends of these reported values. This pattern would suggest limited membership leakage and provide support for releasing SimTrace from a membership-inference perspective.

Action / Context WeightROC-AUCPrecisionTPR at 1% FPR
1 / 00.534550.00%0.99%
0.8 / 0.20.540455.88%1.26%
0.6 / 0.40.538657.14%1.32%
0.5 / 0.50.529555.88%1.26%
0.4 / 0.60.526455.88%1.26%
0.2 / 0.80.515755.88%1.26%
0 / 10.508946.43%0.86%
AuthorsOpenReview ↗

Reviewer S8Fr Response Pt3

Re-identification through source-session linkage: It assesses whether an attacker can recover the exact real session used to generate a synthetic trajectory. Following the trajectory-linkage protocol of Jin et al. [5], the attacker treats each released synthetic trajectory as a query and ranks candidate real sessions. We evaluate two attacks. The similarity ranker [5] orders candidates by their similarity to the query in action sequence and context. The retriever + reranker uses the same similarity score to retrieve the top 20 candidates, then applies an LLM to reorder the shortlist, adapting the LLM reranking approach of Gangi Reddy et al. [6]. We report Top@1 and Top@5 [5], the fractions of queries for which the exact generation reference appears first or among the first five results, respectively.

We evaluate source-session linkage under two levels of attacker knowledge. In the first setting, the attacker searches only the 1,513 real sessions used as generation references. This is the more favorable setting for the attacker because every candidate is known to have served as a reference. In the second, more practical setting, the attacker knows the source shop and the relevant one-month window and searches all real sessions in that shop’s interaction log for the period.

The first setting yields the highest observed identification rates for the tested attacks, with the stronger attack reaching 0.925% Top@1 and 2.115% Top@5. In the more practical setting, the attacker searches the one-month shop log without knowing which sessions served as generation references. Both attacks recover fewer exact references, with the stronger attack reaching 0.330% Top@1 and 1.322% Top@5. The recovery will become harder if the attacker had less precise information about the shop or time window. Therefore it suggest limited re-identification risk.

Candidate poolSimilarity ranker Top@1 / Top@5Retriever + reranker Top@1 / Top@5
1,513 ground-truth real sessions0.793% / 1.851%0.925% / 2.115%
1-month real sessions from the known source shop0.198% / 0.727%0.330% / 1.322%
 

Taken together, the results of both experiments suggest that SimTrace poses minimal disclosure risk under the conditions tested.

   

References

[1] Ziqi Zhang, Chao Yan, and Bradley A. Malin. 2022. “Membership Inference Attacks Against Synthetic Health Data.” Journal of Biomedical Informatics, 125, Article 103977. PDF

[2] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. 2020. “GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models.” In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 343–362. PDF

[3] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. 2022. “Membership Inference Attacks From First Principles.” In Proceedings of the 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. PDF

[4] Myeongseob Ko, Ming Jin, Chenguang Wang, and Ruoxi Jia. 2023. “Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study.” In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4871–4881. PDF

[5] Fengmei Jin, Wen Hua, Matteo Francia, Pingfu Chao, Maria E. Orlowska, and Xiaofang Zhou. 2023. “A Survey and Experimental Study on Privacy-Preserving Trajectory Data Publishing.” IEEE Transactions on Knowledge and Data Engineering, 35(6), pp. 5577–5596. PDF

[6] Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. 2024. “FIRST: Faster Improved Listwise Reranking with Single Token Decoding.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8642–8652. Article

[7] Minnie Ashdown, Tadj Oreszczyn, and Eoghan McKenna. 2026. Membership Inference Risk in Synthetic Household Energy Data: A Privacy Evaluation Framework for Synth-SERL. University College London. PDF

[8] Joshua Ward, Chi-Hua Wang, and Guang Cheng. 2026. Privacy Auditing Synthetic Data Release through Local Likelihood Attacks. arXiv:2508.21146v2. PDF

W1: The improvement for the methodology section.

Answer: See response to Question 1.

W2: The evaluation regarding whether synthetic data can simulate real human behavior are insufficient.

Answer: See response to Question 2 and 3.

W3: The paper lacks assessment of its privacy-protection efficacy.

Answer: See response to Question 4.

Reviewer_zqFtOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the lack of fine-grained, environment-grounded user interaction data for building computer-use virtual clients, which are used in applications such as A/B testing, recommender systems, and interface evaluation. Such applications require trajectories that pair each user action with its corresponding web observation, yet existing public datasets either abstract away interaction-level details (e.g., item-ID sequences) or preserve rich context only at limited scale and on a single platform. Meanwhile, proprietary logs are restricted by privacy regulations, and recent LLM-based user simulators tend to optimize for plausible task completion, producing step-by-step behavior that deviates substantially from real user sessions. To address this gap, the authors propose SimTrace, a framework for generating faithful, fine-grained, multimodal synthetic clickstreams.

Critical strengths
  1. Clear motivation. Fine-grained multimodal trajectories are both scarce and privacy-sensitive, and that’s a real problem worth solving.
  2. Well-targeted design. The paper names two concrete failure modes, granularity mismatch and interface drift. The outer reflection loop handles the first and the inner verification loop handles the second.
  3. Reusable beyond the dataset. The paper also releases the twin-sandbox pipeline and an open-source package, so the approach can carry over to other websites.
Critical weaknesses
  1. The data is claimed as shareable, but the only protections are HMAC hashing and price bucketing. Since each synthetic trajectory closely follows a real session, re-identification is a real risk, and there’s no membership-inference or leakage analysis.
  2. Both visual fidelity and rationale alignment are scored by an LLM, with no human evaluation or check that the LLM scores agree with human ratings.
  3. The RL section lacks a clear connection to the paper’s claim of synthetic high quality data.
  4. The paper reports statistics for the synthetic dataset, but not for the real logs it is built from (e.g., source stores, number of sessions, and how many were selected as references). In Table 3, only 901 of the 1,513 sessions end with Terminate. It is unclear whether the remaining sessions were cut off by the action budget.
Questions for authors

See weakness

7 comments
AuthorsOpenReview ↗

Reviewer zqFt Response

W1: The data is claimed as shareable, but the only protections are HMAC hashing and price bucketing. Since each synthetic trajectory closely follows a real session, re-identification is a real risk, and there’s no membership-inference or leakage analysis.

Answer: Thank you for raising this concern. We agree that identifier pseudonymization and price bucketing alone leave open the possibility of linking a synthetic trajectory to its real reference. Based on your suggestion, we therefore conducted additional experiments to evaluate disclosure risk in SimTrace using two methods: membership inference and source-session re-identification. Both of the results suggest limited disclosure risk. We will add the results of both experiments to the paper.

Here are the details:

Membership inference: It assesses whether a candidate real session served as a reference for generating a synthetic trajectory. Following similarity-based membership inference attacks (MIA) [1,2], the attacker compares the candidate with every released synthetic trajectory and assigns it the highest pairwise similarity score. The attacker predicts membership when that score meets a decision threshold. Each pairwise score is a weighted average of action similarity, computed as one minus the normalized Levenshtein distance between the ordered action sequences, and context similarity, computed as the cosine similarity between embeddings of product handles and descriptions. We vary the weights and test alternative context representations to identify the strongest attack, which represents the highest disclosure risk.

We report (1) ROC-AUC [2,3,4], where values near 0.5 indicate the attack cannot distinguish whether data used for generating synthetic data or not; (2) precision [1,3], where values smaller than 0.5 indicate that the attack is no more successful than a random guess; and (3) TPR at 1% FPR [3, 4], where lower values indicate attacker cannot confidently identify any members of the synthetic dataset. For an empirical evaluation, we estimate 95% confidence intervals by bootstrapping users.

The sweep yields a maximum ROC-AUC of 0.5404 (95% CI 0.5224–0.5648), indicating modest discrimination between reference and nonreference sessions. For context, the Synth-SERL assessment uses an ROC-AUC of 0.55 or below as its Pass status and values above 0.55 through 0.60 as its Conditional band for releasing synthetic household energy data [7]. Our point estimate falls below the Pass threshold, providing a reference point for interpreting the magnitude of the observed discrimination.

At the nominal 1% FPR, the attack achieves 57.14% precision (95% CI 40.54%–73.53%). This interval includes the 50% random-guess baseline. The attack achieved 1.32% TPR (95% CI 0.86% - 2.03%), which indicates an uncertainty about the attack’s advantage at that operating point. For comparison, Zhang et al. report maximum precisions of 55% and 64% for two released synthetic health datasets [1]. Ward et al. report TPRs of 1.3%–5.3% at 1% FPR across attack methods on the synthetic tabular benchmark [8]. Our precision and TPR fall near the lower ends of these reported values. This pattern would suggest limited membership leakage and provide support for releasing SimTrace from a membership-inference perspective.

Action / Context WeightROC-AUCPrecisionTPR at 1% FPR
1 / 00.534550.00%0.99%
0.8 / 0.20.540455.88%1.26%
0.6 / 0.40.538657.14%1.32%
0.5 / 0.50.529555.88%1.26%
0.4 / 0.60.526455.88%1.26%
0.2 / 0.80.515755.88%1.26%
0 / 10.508946.43%0.86%
AuthorsOpenReview ↗

Reviewer zqFt Response Pt2

Re-identification through source-session linkage: It assesses whether an attacker can recover the exact real session used to generate a synthetic trajectory. Following the trajectory-linkage protocol of Jin et al. [5], the attacker treats each released synthetic trajectory as a query and ranks candidate real sessions. We evaluate two attacks. The similarity ranker [5] orders candidates by their similarity to the query in action sequence and context. The retriever + reranker uses the same similarity score to retrieve the top 20 candidates, then applies an LLM to reorder the shortlist, adapting the LLM reranking approach of Gangi Reddy et al. [6]. We report Top@1 and Top@5 [5], the fractions of queries for which the exact generation reference appears first or among the first five results, respectively.

We evaluate source-session linkage under two levels of attacker knowledge. In the first setting, the attacker searches only the 1,513 real sessions used as generation references. This is the more favorable setting for the attacker because every candidate is known to have served as a reference. In the second, more practical setting, the attacker knows the source shop and the relevant one-month window and searches all real sessions in that shop’s interaction log for the period.

The first setting yields the highest observed identification rates for the tested attacks, with the stronger attack reaching 0.925% Top@1 and 2.115% Top@5. In the more practical setting, the attacker searches the one-month shop log without knowing which sessions served as generation references. Both attacks recover fewer exact references, with the stronger attack reaching 0.330% Top@1 and 1.322% Top@5. The recovery will become harder if the attacker had less precise information about the shop or time window. Therefore it suggest limited re-identification risk.

Candidate poolSimilarity ranker Top@1 / Top@5Retriever + reranker Top@1 / Top@5
1,513 ground-truth real sessions0.793% / 1.851%0.925% / 2.115%
1-month real sessions from the known source shop0.198% / 0.727%0.330% / 1.322%
 

Taken together, the results of both experiments suggest that SimTrace poses minimal disclosure risk under the conditions tested.

   

References

[1] Ziqi Zhang, Chao Yan, and Bradley A. Malin. 2022. “Membership Inference Attacks Against Synthetic Health Data.” Journal of Biomedical Informatics, 125, Article 103977. PDF

[2] Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. 2020. “GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models.” In Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 343–362. PDF

[3] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. 2022. “Membership Inference Attacks From First Principles.” In Proceedings of the 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897–1914. PDF

[4] Myeongseob Ko, Ming Jin, Chenguang Wang, and Ruoxi Jia. 2023. “Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Study.” In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4871–4881. PDF

[5] Fengmei Jin, Wen Hua, Matteo Francia, Pingfu Chao, Maria E. Orlowska, and Xiaofang Zhou. 2023. “A Survey and Experimental Study on Privacy-Preserving Trajectory Data Publishing.” IEEE Transactions on Knowledge and Data Engineering, 35(6), pp. 5577–5596. PDF

[6] Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. 2024. “FIRST: Faster Improved Listwise Reranking with Single Token Decoding.” In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8642–8652. Article

[7] Minnie Ashdown, Tadj Oreszczyn, and Eoghan McKenna. 2026. Membership Inference Risk in Synthetic Household Energy Data: A Privacy Evaluation Framework for Synth-SERL. University College London. PDF

[8] Joshua Ward, Chi-Hua Wang, and Guang Cheng. 2026. Privacy Auditing Synthetic Data Release through Local Likelihood Attacks. arXiv:2508.21146v2. PDF

AuthorsOpenReview ↗

Reviewer zqFt Response Pt3

W2: Both visual fidelity and rationale alignment are scored by an LLM, with no human evaluation or check that the LLM scores agree with human ratings.

Answer: We acknowledge the importance of assessing whether LLM-based scores agree with human judgments. For visual fidelity, we involved humans in evaluating the appearance of websites scored by the LLM-based verifier. We used their judgments to refine the verifier prompt so that its scores better reflected human evaluations. We will include this analysis in the paper.

For rationale alignment, the LLM’s role is limited to inferring the shopping rationale for each real and synthetic session. We compute the alignment score as the cosine similarity between the embeddings of the two inferred rationales. During development, we manually inspected the generated rationales and refined the prompt to keep them grounded in the session evidence. We will clarify this process in the paper.

W3: The RL section lacks a clear connection to the paper’s claim of synthetic high quality data.

Answer: The contribution of our work includes both a method for generating synthetic GUI trajectories and a training approach that uses them to build computer-use agents for user modeling. The synthetic trajectories provide initial SFT, and the subsequent RL stage uses real sessions to adapt the model to the target environment. This RL stage gives us a better user model which in term will generate better synthetic trajectory data.

W4: The paper reports statistics for the synthetic dataset, but not for the real logs it is built from (e.g., source stores, number of sessions, and how many were selected as references). In Table 3, only 901 of the 1,513 sessions end with Terminate. It is unclear whether the remaining sessions were cut off by the action budget.

Answer: Thank you for the question! We will include statistics of the real logs in the paper. SimTrace used 3,000 unique real sessions from five stores as generation references. Of the 3,000 generation runs, 15 failed to complete due to API or parsing errors, 1,443 failed the realism or trajectory consistency checks, and 29 were excluded because they contained invalid action types. The remaining 1,513 synthetic sessions make up the released SimTrace dataset.

As shown in Table 3, 901 of the 1,513 released sessions ended with a Terminate action. Of the remaining 612 sessions, 310 ended when the agent clicked checkout, and 302 reached the action budget. We will clarify this breakdown in the paper.

Ethics Concerns

Answer: Thank you for raising these concerns. As described in the Ethics Statement, the source logs came from five participating stores and were collected and processed under a data-use agreement that users explicitly opted into. We also obtained consent from the store owners to use their websites to construct the simulated twin environments. We address the re-identification concern and membership inference risk in our response to Weakness 1.

We will revise the Ethics Statement to make the provenance of the logs, the scope of user consent and store-owner permission explicit. We will also include the privacy risk assessment in the paper. Thank you again for identifying these details.

Reviewer_X1S7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SIMTRACE, a data synthesis framework for generating user multimodal user interaction trajectories for LLM-based recommendation. SIMTRACE constructs a sandbox environment uses LLM agent to generate synthetic interaction sessions based on real activity logs. Experiments shows that the synthetic trajectories achieve high fidelity and improve downstream performance.

Critical strengths
  • The motivation is clear and makes sense. Existing data for LLM-based recommendation rarely provide semantically faithful trajectories at scale.
  • The evaluation is comprehensive. Experiments evaluate both the fidelity of the synthetic data and the performance of training models with the synthetic data.
Critical weaknesses
  • The motivation is to address the data scarity of detailed trajectories. So I think the experiment should provide the result using large-scale synthetic data, not just the results with 1:1 synthetic-real ratio. The current results shows that the synthetic data is comparable to real data, but not that SIMTRACE can scale the training up.
  • The code repo contains the Contact info of the author. This may violate the double-blind rule.
  • Table 2 mentioned that the synthetic datasets contains 3 tasks, but not all tasks are evaluated. It also contains multple platform, which is not presented in the experimental results.
Questions for authors

Why is the JSD of SIMTRACE in Table 6 is 0?

3 comments
AuthorsOpenReview ↗

Reviewer X1S7 Response

W1: The motivation is to address the data scarity of detailed trajectories. So I think the experiment should provide the result using large-scale synthetic data, not just the results with 1:1 synthetic-real ratio. The current results shows that the synthetic data is comparable to real data, but not that SIMTRACE can scale the training up.

Answer: We added a synthetic-only scaling experiment. We find that using more synthetic data would generally improve the results. Specifically, increasing the synthetic data from 1x to 5x improved F1 for purchase prediction and MRR@5 and HR@5 for session-based recommendation.

As shown in the table below, most metrics improved as the synthetic training set grew from 1× to 2× to 5×, although some scores declined at individual steps. For purchase prediction, Qwen3.5-4B achieved F1 scores of 65.52, 64.03, and 66.13 at 1×, 2×, and 5×, respectively. F1 peaked at 5× after dipping at 2×. For session-based recommendation with DIMO, increasing the training set from 2× to 5× yielded small gains in MRR@5 and HR@5, while MRR@10 and HR@10 declined. These results show that larger synthetic training sets can generally improve downstream performance, though the amount of data that yields the best results may vary by task and model.

MethodMetric ↑Real-onlySynthetic 1×Synthetic 2×Synthetic 5×
Purchase Prediction
Qwen3.5-4B SFTF164.3665.52 (+1.16, [−3.00, +5.00])64.03 (−0.33, [−1.63, +1.02])66.13 (+1.77, [−1.18, +3.64])
Session-based Recommendation
RAINMRR@512.4712.58 (+0.11, [−1.59, +1.70])12.81 (+0.34, [−1.43, +1.79])14.23 (+1.76, [−0.15, +3.11])
RAINHR@521.2520.85 (−0.40, [−3.20, +2.27])22.26 (+1.01, [−1.74, +3.69])24.77 (+3.52, [+0.85, +6.20])
RAINMRR@1013.213.64 (+0.44, [−1.24, +2.03])14.09 (+0.89, [−0.60, +2.50])15.53 (+2.33, [+1.01, +4.09])
RAINHR@1026.7129.12 (+2.41, [−0.71, +5.40])29.89 (+3.18, [+0.31, +5.93])32.53 (+5.82, [+3.00, +8.41])
DIMOMRR@517.8416.49 (−1.35, [−3.67, +1.16])16.59 (−1.25, [−3.85, +0.48])17.01 (−0.83, [−3.44, +0.97])
DIMOHR@531.8330.27 (−1.56, [−5.05, +2.28])30.96 (−0.87, [−4.21, +2.50])31.06 (−0.77, [−4.13, +2.48])
DIMOMRR@1018.9118.34 (−0.57, [−2.85, +2.41])18.79 (-0.12, [−1.60, +2.14])18.16 (−0.75, [−2.50, +1.75])
DIMOHR@1039.8640.46 (+0.60, [−2.83, +3.95])44.14 (+4.28, [+0.50, +7.77])40.33 (+0.47, [−3.35, +3.97])

Note: Each synthetic column reports the absolute score, followed by the difference from the real-only baseline (Δ) and its 95% confidence interval, both in percentage points.

W2: The code repo contains the Contact info of the author. This may violate the double-blind rule.

Answer: Thank you for flagging this oversight. We apologize for accidentally leaving author contact information in the anonymized repository. We have removed that information.

W3: Table 2 mentioned that the synthetic datasets contains 3 tasks, but not all tasks are evaluated. It also contains multple platform, which is not presented in the experimental results.

Answer: Thank you for highlighting the value of supporting multiple tasks and platforms in the dataset. We did evaluate all three tasks. Section 4 Table 5 and Figure 2 present the results for all three tasks. For purchase prediction and next-action prediction, we used synthetic data from all platforms, covering all five stores. Appendix F.3 illustrates the value of using data from multiple platforms by showing how this diversity reduces the overprediction of frequent target elements. We will make the mapping from Table 2 to the experiments explicit and clarify which stores contributed data to each evaluation.

Q1: Why is the JSD of SIMTRACE in Table 6 is 0?

Answer: Outcome JSD compares the aggregate proportions of browse-only, cart-abandonment, and checkout sessions in the real and synthetic data. If these proportions match, the divergence is zero even when individual action sequences differ. Table 6 captures those sequence differences separately, with a trajectory Levenshtein distance of 0.239.

Thank you again for your review! We appreciate your time and helpful suggestions! Kind reminder: as the discussion period closes on November 18, we wanted to check whether our response addresses your questions. Following your suggestions, we added a synthetic-only experiment comparing downstream performance at three training scale: 1×, 2×, and 5× (W1). We also clarified where the paper reports results for all three tasks and how the five stores contributed to the evaluations. If our response addresses your concerns, we’d appreciate it if you would consider updating your assessment. We’re happy to clarify anything that remains open.

Dear Reviewer X1S7,

We sincerely appreciate your review and feedback. As the discussion deadline approaches today, we would like to follow up once more.

If possible, we would be grateful if you could let us know whether our responses and additional experiments have addressed your main concerns. We would be happy to provide further clarification before the discussion period closes.

Thank you again for your time and thoughtful feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.