- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes PsyBer-Agent, a user simulator for pre-deployment recommender evaluation. Its main components are: 1) Psy-Engine, which maintains three evolving, model-defined latent states: cognitive reserve, emotional satisfaction, and self-congruence. State transitions incorporate user personas, memory, and item features. 2) A psycho-behavioral alignment framework that replays logged interactions through Psy-Engine to construct model-defined reference trajectories and uses Unbalanced Gromov–Wasserstein (UGW) alignment to derive soft action supervision. 3) A lightweight fast policy that handles most decisions, with LLM/VLM fallback when needed.
WebSim and the PsyBer Benchmark, are proposed to evaluate behavioral fidelity, model-defined latent-state consistency, robustness, and scalability, by comparing the simulated user behavior and real user behavior.
Critical strengths
-
The problem is important, and the modeling direction is reasonable. The cost of user simulation is high. To get stat-sig results, A/B testing with large population is needed to test a recommendation strategy change.
-
Behavior generation is constrained by observed interactions. UGW with uniform initialization, and prior-guided UGW. Decision Accuracy increases from 0.242 for Direct BC to 0.278 for prior-guided UGW, providing preliminary evidence for the alignment procedure.
-
The controlled baseline comparison shows measurable improvements. In Table 4, all simulators receive the same user histories, item metadata, and candidate pages. Compared with DyTA4Rec, PsyBer-Agent improves Decision Accuracy from 0.254 to 0.275 and reduces JSD from 0.335 to 0.286. These results support its relative advantage under the evaluated protocol.
Critical weaknesses
-
Long-horizon behavioral fidelity remains insufficiently validated, and the reported results show substantial degradation. Long-horizon closed-loop simulation is a central motivation, but the supporting evidence is limited: The human–simulator agreement study covers only 10 browsing rounds. In modern recommendation system, ultra-long sequence learning is the central problem, a typical user is usually represented by a sequence of O(10k). The sequence length in this work is limited.
-
Evaluation dataset is limited. The current data is built from 2 scenario: shopping and web browsing, while the modern recommendation system plays a more important role in social media with high frequency scenario like short-form video and news feeds, where catching user's interests are more important. Also, the key claim from the paper like "underlying user dynamics such as fatigue, affect, and shifting preferences", are easier to capture. In [1,2], such dataset is provided so it's interesting to check if the proposed approach can generalize.
[1] Fostiropoulos, Iordanis, et al. "GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification." arXiv preprint arXiv:2603.29112 (2026).
[2] Jiang, Bowen, et al. "PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks." arXiv preprint arXiv:2608.21381 (2026).
Questions for authors
-
With websim, is it possible to build long user history sequence? or is it better to utilize real-user distributed dataset like [1] and [2]?
-
Which scenario is better to study user dynamics such as fatigue, affect, and shifting preferences?
[1] Fostiropoulos, Iordanis, et al. "GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification." arXiv preprint arXiv:2603.29112 (2026).
[2] Jiang, Bowen, et al. "PersonaMem-v3: Toward Omni-Platform Personal Intelligence for Holistic User Understanding, Recommendation, and Agentic Tasks." arXiv preprint arXiv:2608.21381 (2026).
4 comments
Thanks for the update
Thanks for running the evaluation. The results are a bit confusing: when seq len increased from 10 to 100, the decision accuracy drops. This is a bit contradictory to the scaling law - model improves as the user behavior sequence gets longer because the model knows more context about the user
Thank you for pointing this out. In our table, 10, 50, and 100 refer to the number of evaluated steps. The results do not compare predictions for the same target using progressively longer user histories. Longer evaluations include later decisions, so average accuracy may decline even if more history improves prediction for a fixed target. Both PsyBer-Agent and the calibrated baseline show this decline.
Thanks for following up
The results do not compare predictions for the same target using progressively longer user histories
Got it, "same target using progressively longer user histories" is a standard evaluation protocol for recsys research so a table with this will be more persuasive. Given the current status of this manuscript, I tend to keep my original ratings
Dear Reviewer,
Thank you for your thoughtful feedback. We have carefully addressed the concerns raised in our response. If our clarifications have addressed your concerns, we would be grateful if you would consider updating your evaluation accordingly. Please let us know if any further clarification would be helpful.