- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents GEMS, a framework for generating human-like behaviors in interactive 3D environments by jointly modeling goal-directed and habitual behavior control. The cognitive module consists of a Habitual Controller that retrieves context-triggered actions from personalized habit memories, a Goal-directed Controller that evaluates candidate actions using an LLM-based world model, and an Arbiter that coordinates the two controllers based on action values, habit strengths, individual preferences, and psychological states. A keyframe-guided motion generation module then translates the selected behaviors into 3D human motions and object interactions. The framework is evaluated through comparisons with existing human-like agent methods, human preference studies, and ablation experiments.
Critical strengths
Interesting research motivation and coherent framework. The paper addresses an interesting aspect of human-like agent simulation by explicitly incorporating habitual behaviors alongside goal-directed planning. The proposed framework provides a coherent integration of context-triggered habit retrieval, goal-directed reasoning, and behavior arbitration, offering a meaningful perspective on generating more diverse and believable agent behaviors.
Comprehensive experimental evaluation. The paper provides extensive comparisons with existing human-like agent frameworks, including efforts to improve baseline comparability, as well as detailed ablation studies covering the major components. Particularly, the human preference studies provide valuable evidence that incorporating habitual control improves perceived human-likeness. The additional analyses of psychological states, habit ratios, and arbitration mechanisms further support the effectiveness of the proposed framework.
Critical weaknesses
Limited justification of the proposed dual-control architecture. Although the paper provides extensive ablation studies demonstrating the effectiveness of its individual components, the proposed cognitive framework primarily relies on predefined habit memories, context-based retrieval, and heuristic arbitration. It remains unclear to what extent this particular dual-control architecture offers advantages over simpler context-conditioned behavior generation approaches.
Insufficient temporal grounding and validation of psychological state dynamics. The framework maintains dynamically updated psychological states, including stress, depletion, and cognitive load, which directly influence the arbitration between habitual and goal-directed control. However, these states are estimated by an LLM, and their evolution is not clearly grounded in elapsed time, activity duration, or behavioral intensity. Although the paper includes daily-life simulations and evaluates the effects of predefined psychological states, it remains unclear whether the resulting state trajectories realistically reflect human behavior over time.
Questions for authors
The human study evaluates only 10 behavioral samples per method in each setting. How are these samples selected, and to what extent are they representative of the diversity of goals, personas, and habitual contexts considered in the paper? Would the observed human preference trends remain consistent with a larger and more diverse set of behavioral samples?
2 comments
Prompt of Simpler LLM
You are {agent_name}, a person with the following persona: {persona}. Your biography is: {biography}.
At this moment, your current goal is {intent_text}, and your memories of recent experiences are: {memory_text}.
You are currently in {spatial_name}, and your current spatial belief is: {spatial_text}.
Your last action was {last_action}, and its outcome was {event}.
Your current psychological state is {psychological_state}.
You are now considering what you would actually do next.
[Task]
Taking into account your current goal, habitual tendencies, psychological state, and situational context, generate {sample_window} candidate actions that {agent_name} might realistically perform next.
Candidate actions may include behaviors that directly advance the current goal, as well as habitual behaviors that naturally arise in the current context.
Finally, select the single most appropriate action based on the character's persona, current situation, and recent experiences.
[Action Generation Principles]
- Habitual behaviors must be generated based on the task setting and current context. Refer to [Definition of Habitual Behavior] for details.
- Each candidate action must describe exactly one concrete action. Do not combine multiple actions into a single candidate.
- Do not invent rooms, furniture, objects, or tools that are not present in the spatial belief.
[Definition of Habitual Behavior]
{introduce_of_habitual_action}
[1] Humanoid Agents: Platform for Simulating Human-like Generative Agents, Wang et al., EMNLP 2023
[2] Beyond Self-Interest: Modeling Social-Oriented Motivation for Human-like Multi-Agent Interactions, Lin et al., AAMAS 2026
[3] Simulating Human-like Daily Activities with Desire-Driven Autonomy, Wang et al., ICLR 2025
[4] VBench: Comprehensive Benchmark Suite for Video Generative Models, Huang et al., CVPR 2024
[5] Treating Stimuli as a Random Factor in Social Psychology, Judd et al., JPSP 2012
We appreciate your acknowledgment of the originality of our work and the quality of the presentation. We are confident that the following responses will effectively address the concerns raised in your comments.
To further justify our mutiple controller design, we consider two simpler context-conditioned behavior generation baselines:
Following our ablation protocol, Simpler LLM is evaluated over 15 independent runs. The quantitative evaluation is summarized in the table below:
These results demonstrate that GEMS substantially outperforms simpler context-conditioned behavior generation approaches.
We thank the reviewer for the observation regarding LLM-based psychological state updates. We acknowledge the challenges of fully modeling human psychological dynamics and have discussed this limitation in the revised manuscript.
However, we would like to clarify that LLM-based psychological state estimation and updates have been explored in prior work [1, 2, 3]. Our primary contribution lies in exploring how to effectively incorporate habitual behaviors to generate more believable behaviors, which is largely independent of the specific implementation of psychological state updates. The substantial improvements achieved by GEMS empirically demonstrate the effectiveness of our proposed framework.
Sample selection. Following [4], we randomly selected 10 of the 30 goals and sampled one behavior per goal from each of six sources, forming a fixed pool of 60 samples per setting. Each questionnaire compared all 15 method pairs, with a randomly selected goal for each pair. Non-human methods shared the same initialization, while human samples were matched by goal only.
Diversity of goals, personas, and habitual contexts. The original study covered goal diversity through random sampling. To further explore the effects of diversity in personas, habitual contexts, and habitual ratios on human subjective evaluations, we recruited 13 additional participants to evaluate pairwise preferences for GEMS-generated behaviors across three personas: Adrian (screenwriter, ρ=0.5), Elena (nurse, ρ=0.6), and Michael (forest ranger, ρ=0.4), under their corresponding habitual contexts. The Human preference comparison across 3 personas is summarized in the table below (each entry compares the win rate of the method on the vertical axis against that on the horizontal axis, with diagonal values fixed at 0.5.):
The results suggest that persona diversity has a limited impact on human judgments within the evaluated settings.
Preference consistency with larger samples. Following [5], we bootstrapped participants and goals of Human study in Appendix.B. GEMS achieved preference rates of 73.2% (95% CI: 59.9%–85.4%) for videos and 67.5% (95% CI: 54.3%–79.8%) for behavior sequences over four non-human baselines, with both confidence intervals above 50%. Leave-one-goal-out analysis further showed stable rates of 71.8%–74.8% and 66.0%–68.9%, respectively. These results suggest that the overall preference trends are likely to remain stable with larger samples drawn from the same evaluation distribution.