- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ShopMind uses a large language model to simulate users' shopping behavior. It records simulated users' beliefs, uncertainties, and purchase intentions, simulates several possible next actions, and then selects one. The paper also trains an MSR model on shopping behavior and mental-state descriptions generated by the large language model. MSR infers users' mental states from shopping records and is used for consumption value measurement and CTR prediction.
Critical strengths
- The method considers differences between users, as well as changes in the same user's thoughts during shopping.
- The ablation experiments examine the roles of four main components.
- The trained MSR model improves scores on consumption value measurement and CTR prediction. It also performs better than directly using the same base model, without MSR training, to generate mental-state descriptions.
Critical weaknesses
- There is still a lack of direct validation of whether the generated mental-state descriptions reflect real users' thoughts.
MSR learns from mental-state descriptions generated by ShopMind itself. The paper uses results on consumption value measurement and CTR prediction to show that these descriptions are useful. However, it does not sufficiently show whether they reflect what real users were thinking at the time.
- The paper does not clearly state which models each method uses or how many model calls it makes.
The main text says that ShopMind uses DeepSeek-V4-Pro, while the appendix says that the comparisons mainly use Qwen2.5-7B-Instruct. These statements seem inconsistent. In addition, the paper does not compare the total number of model calls or the running costs of the methods.
- The gains on some metrics are small.
For example, on ShopCART, ShopMind's Macro-F1 is 49.14, compared with 48.40 for AlignUSER, a difference of only 0.74 points.
- The code and model files needed to reproduce the experiments are incomplete.
The public repository does not explain how to obtain the Behavior Critic's model weights or provide instructions for training it. It also lacks the MSR training code, making it difficult to reproduce the main results with the available materials.
Questions for authors
-
Beyond improved prediction scores, what evidence shows that the mental-state descriptions generated by MSR reflect what real users were thinking at the time?
-
Which models does each method actually use, and how many model calls does each method make?
-
Why are the gains on some metrics small? Are these gains stable across multiple runs?
-
Will you provide the Behavior Critic's model weights and code for MSR training and related steps?
3 comments
Response to Reviewer xCMX–Section2
Weakness 3 & Question 3: Performance Gains
We agree that some individual improvements are modest, such as the 0.74-point Macro-F1 gain over AlignUSER on ShopCART. However, ShopMind aims to improve both individual consistency and behavioral realism, so its performance should be assessed across multiple dimensions. On ShopCART, ShopMind also improves Exact Accuracy by 3.27 percentage points, while reducing Transition JS and Action JS by 14.6% and 15.2%, respectively, over the strongest baselines. Similar gains appear on other datasets: OPeRA Top-3 Accuracy increases from 47.89 to 60.85, and Taobao Transition JS decreases from 16.72 to 12.18. These results show that ShopMind's improvements extend beyond individual metrics, particularly in long-horizon behavioral realism across datasets.
Weakness 4 & Question 4: Reproducibility
We have already made the core implementation and detailed experimental configurations available in our anonymous repository to support reproducibility. While some materials are not yet included, we will release the remaining resources, together with the necessary training instructions, upon acceptance of the paper to further support reproducibility.
Follow-up on Our Rebuttal
Dear Reviewer,
Thank you again for your valuable feedback. We have provided detailed responses and additional experimental results to address your concerns. We would greatly appreciate it if you could review our rebuttal at your convenience and let us know whether any concerns remain. We sincerely hope that our responses and additional results may help alleviate your concerns and provide a basis for reconsidering your initial assessment. If you have any further questions or suggestions, we would be more than happy to provide additional clarifications. Thank you for your time and consideration.
Response to Reviewer xCMX–Section1
We sincerely thank the reviewer for the constructive feedback and for recognizing ShopMind's personalized modeling, ablation studies, and the downstream utility of MSR. We address each concern below.
Weakness 1 & Question 1: Validation of Mental States
First, consumers' mental states are inherently latent and difficult to observe. Real-world e-commerce logs typically record behavioral interactions without continuous self-reports of beliefs, uncertainties, or intentions, making it difficult to obtain ground-truth mental states. This motivates our use of ShopMind to generate behavior–mental-state pairs and train MSR to reconstruct latent mental states from observed behavior.
Second, we evaluate mental states through behavioral consistency and downstream utility. During data construction, multi-stage quality control and human review assess whether mental states are supported by observed evidence, consistent with user behavior, and coherent over time. On real user data, MSR-reconstructed mental states improve Primary Accuracy from 0.6518 to 0.7868 in consumption value measurement and AUC from 0.6441 to 0.7156 in CTR prediction with DeepSeek-V4-Pro.
Moreover, our Zero-shot Mental State baseline shows that directly inferring mental states with a general-purpose LLM provides smaller gains or even degrades performance. This suggests that the improvements are not simply due to adding mental-state descriptions, but benefit from ShopMind-generated supervision.
We acknowledge that these results demonstrate behavioral consistency and predictive utility, rather than direct agreement with users' subjective experiences. Such validation requires collecting self-reported mental states during real shopping sessions. We will clarify this limitation in the revised manuscript and consider human-grounded validation in future work.
Our contribution is not to claim precise recovery of consumers' actual thoughts, but to explicitly model evolving mental states and demonstrate their value for realistic behavior simulation and downstream prediction.
Weakness 2 & Question 2: Model Configurations and Inference Cost
The main text and appendix refer to different model configurations. ShopMind uses DeepSeek-V4-Pro for simulation, Qwen3-4B-Instruct-2507 for the Behavior Critic, and Qwen3-8B for MSR. The Qwen2.5-7B-Instruct in Appendix D.2 is used for baseline comparisons. We acknowledge that Appendix D.3 is unclear and will clarify the backbone and training settings for each method. To examine the impact of backbone differences, we additionally evaluate ShopMind with Qwen2.5-7B-Instruct on Taobao, using the same backbone as the baselines.
Under the same backbone, ShopMind achieves the best performance on three of the four metrics, reducing Trans. JS, Len. Err., and Action JS by 8.0%, 1.8%, and 7.4%, respectively, over the strongest baselines. It also achieves the second-best Buy Err.
Regarding inference cost, ShopMind's multi-stage decision process introduces additional overhead. On OPeRA and ShopCART, Mental-World Rollout is disabled, avoiding the cost of future trajectory simulation. When enabled, rollout considers K = 4 candidates, each expanded for up to L = 2 steps, yielding at most eight simulated rollout steps per decision. Actual model calls depend on state updates, early termination, and behavior evaluation. Appendix B further shows that increasing L from 0 to 2 reduces Transition JS from 22.63 to 12.18 on Taobao, while longer rollouts provide no further gains. We will clarify these configurations and discuss the trade-off between performance and inference cost in the revised manuscript.