acceptodds
Under review as a conference paper at ICLR 2027

Generative Embodied Multiple Behavior Control Systems for Human-like Agents

Abstract

Building autonomous agents that reproduce human behavior in realistic 3D environments has been a longstanding objective in AI. Cognitive neuroscience commonly believes that human behaviors are controlled by multiple control systems including goal-directed and habitual. However, existing human-like agent frameworks primarily focus on goal-directed system and habitual system is largely overlooked. In this paper, we address this gap by proposing GEMS, a Generative Embodied Multiple Behavior Control System that jointly models goal-directed and habitual behaviors for generating believable human-like behavior. GEMS employs a cognitive module to generate natural language behaviors and a motion generation module to construct them in 3D environment. The cognitive module comprises a Habitual Controller that retrieves habitual actions from habit memory in response to environmental stimuli, a Goal-directed Controller that proposes goal-directed actions for current goal, and an Arbiter that governs their influence and selects the final action, supporting diverse and naturalistic behavior patterns generation. To construct these behaviors, the motion generation module adopts a Text-to-Motion model to generate versatile and expressive 3D human–scene interactions without relying on predefined actions. Extensive evaluations, human and ablation studies demonstrate that human-likeness performance is substantially improved by GEMS. The efficacy of GEMS indicates the benefits of leveraging habitual behavior and multiple behavior control system coordination for believable embodied human-like agents design. The code is available at https://anonymous.4open.science/r/review-video-82f4/README.md.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Human behavior simulation, 3D Humanoid, Human behavior control mechanism

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zc63OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GEMS, a framework for generating human-like behaviors in interactive 3D environments by jointly modeling goal-directed and habitual behavior control. The cognitive module consists of a Habitual Controller that retrieves context-triggered actions from personalized habit memories, a Goal-directed Controller that evaluates candidate actions using an LLM-based world model, and an Arbiter that coordinates the two controllers based on action values, habit strengths, individual preferences, and psychological states. A keyframe-guided motion generation module then translates the selected behaviors into 3D human motions and object interactions. The framework is evaluated through comparisons with existing human-like agent methods, human preference studies, and ablation experiments.

Critical strengths

Interesting research motivation and coherent framework. The paper addresses an interesting aspect of human-like agent simulation by explicitly incorporating habitual behaviors alongside goal-directed planning. The proposed framework provides a coherent integration of context-triggered habit retrieval, goal-directed reasoning, and behavior arbitration, offering a meaningful perspective on generating more diverse and believable agent behaviors.

Comprehensive experimental evaluation. The paper provides extensive comparisons with existing human-like agent frameworks, including efforts to improve baseline comparability, as well as detailed ablation studies covering the major components. Particularly, the human preference studies provide valuable evidence that incorporating habitual control improves perceived human-likeness. The additional analyses of psychological states, habit ratios, and arbitration mechanisms further support the effectiveness of the proposed framework.

Critical weaknesses

Limited justification of the proposed dual-control architecture. Although the paper provides extensive ablation studies demonstrating the effectiveness of its individual components, the proposed cognitive framework primarily relies on predefined habit memories, context-based retrieval, and heuristic arbitration. It remains unclear to what extent this particular dual-control architecture offers advantages over simpler context-conditioned behavior generation approaches.

Insufficient temporal grounding and validation of psychological state dynamics. The framework maintains dynamically updated psychological states, including stress, depletion, and cognitive load, which directly influence the arbitration between habitual and goal-directed control. However, these states are estimated by an LLM, and their evolution is not clearly grounded in elapsed time, activity duration, or behavioral intensity. Although the paper includes daily-life simulations and evaluates the effects of predefined psychological states, it remains unclear whether the resulting state trajectories realistically reflect human behavior over time.

Questions for authors

The human study evaluates only 10 behavioral samples per method in each setting. How are these samples selected, and to what extent are they representative of the diversity of goals, personas, and habitual contexts considered in the paper? Would the observed human preference trends remain consistent with a larger and more diverse set of behavioral samples?

2 comments

We appreciate your acknowledgment of the originality of our work and the quality of the presentation. We are confident that the following responses will effectively address the concerns raised in your comments.

W1: Limited justification of dual-control system.

To further justify our mutiple controller design, we consider two simpler context-conditioned behavior generation baselines:

  1. Simpler LLM: Uses a single LLM to generate and select goal-directed or habitual behaviors given the same contextual information as GEMS. The prompt can find in the botton.
  2. ACTOR†: Extends ACTOR with our habit memory, using an LLM for context-aware habit selection and controller artribution. You can understand as a simple “multiple controller” schema.

Following our ablation protocol, Simpler LLM is evaluated over 15 independent runs. The quantitative evaluation is summarized in the table below:

MethodNat. ↑Coh. ↑PA ↑Mean ↑
Simpler LLM57.0577.6179.3371.33
ACTOR47.7786.1877.6270.52
ACTOR†52.7985.8578.2772.30
GEMS (Ours)90.3189.9684.3588.21

These results demonstrate that GEMS substantially outperforms simpler context-conditioned behavior generation approaches.

W2: Insufficient of LLM based psychological state dynamics

We thank the reviewer for the observation regarding LLM-based psychological state updates. We acknowledge the challenges of fully modeling human psychological dynamics and have discussed this limitation in the revised manuscript.

However, we would like to clarify that LLM-based psychological state estimation and updates have been explored in prior work [1, 2, 3]. Our primary contribution lies in exploring how to effectively incorporate habitual behaviors to generate more believable behaviors, which is largely independent of the specific implementation of psychological state updates. The substantial improvements achieved by GEMS empirically demonstrate the effectiveness of our proposed framework.

Q1: Sample Selection, Representativeness, and Preference Consistency

Sample selection. Following [4], we randomly selected 10 of the 30 goals and sampled one behavior per goal from each of six sources, forming a fixed pool of 60 samples per setting. Each questionnaire compared all 15 method pairs, with a randomly selected goal for each pair. Non-human methods shared the same initialization, while human samples were matched by goal only.

Diversity of goals, personas, and habitual contexts. The original study covered goal diversity through random sampling. To further explore the effects of diversity in personas, habitual contexts, and habitual ratios on human subjective evaluations, we recruited 13 additional participants to evaluate pairwise preferences for GEMS-generated behaviors across three personas: Adrian (screenwriter, ρ=0.5\rho=0.5), Elena (nurse, ρ=0.6\rho=0.6), and Michael (forest ranger, ρ=0.4\rho=0.4), under their corresponding habitual contexts. The Human preference comparison across 3 personas is summarized in the table below (each entry compares the win rate of the method on the vertical axis against that on the horizontal axis, with diagonal values fixed at 0.5.):

AdrianElenaMichael
Adrian--61.5%46.2%
Elena38.5%--61.5%
Michael53.8%38.5%--

The results suggest that persona diversity has a limited impact on human judgments within the evaluated settings.

Preference consistency with larger samples. Following [5], we bootstrapped participants and goals of Human study in Appendix.B. GEMS achieved preference rates of 73.2% (95% CI: 59.9%–85.4%) for videos and 67.5% (95% CI: 54.3%–79.8%) for behavior sequences over four non-human baselines, with both confidence intervals above 50%. Leave-one-goal-out analysis further showed stable rates of 71.8%–74.8% and 66.0%–68.9%, respectively. These results suggest that the overall preference trends are likely to remain stable with larger samples drawn from the same evaluation distribution.

Prompt of Simpler LLM


You are {agent_name}, a person with the following persona: {persona}. Your biography is: {biography}.
At this moment, your current goal is {intent_text}, and your memories of recent experiences are: {memory_text}.
You are currently in {spatial_name}, and your current spatial belief is: {spatial_text}.
Your last action was {last_action}, and its outcome was {event}.
Your current psychological state is {psychological_state}.
You are now considering what you would actually do next.

[Task]
Taking into account your current goal, habitual tendencies, psychological state, and situational context, generate {sample_window} candidate actions that {agent_name} might realistically perform next.
Candidate actions may include behaviors that directly advance the current goal, as well as habitual behaviors that naturally arise in the current context.
Finally, select the single most appropriate action based on the character's persona, current situation, and recent experiences.

[Action Generation Principles]
- Habitual behaviors must be generated based on the task setting and current context. Refer to [Definition of Habitual Behavior] for details.
- Each candidate action must describe exactly one concrete action. Do not combine multiple actions into a single candidate.
- Do not invent rooms, furniture, objects, or tools that are not present in the spatial belief.

[Definition of Habitual Behavior]
{introduce_of_habitual_action}

[1] Humanoid Agents: Platform for Simulating Human-like Generative Agents, Wang et al., EMNLP 2023

[2] Beyond Self-Interest: Modeling Social-Oriented Motivation for Human-like Multi-Agent Interactions, Lin et al., AAMAS 2026

[3] Simulating Human-like Daily Activities with Desire-Driven Autonomy, Wang et al., ICLR 2025

[4] VBench: Comprehensive Benchmark Suite for Video Generative Models, Huang et al., CVPR 2024

[5] Treating Stimuli as a Random Factor in Social Psychology, Judd et al., JPSP 2012

Reviewer_RKdDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of human-like behavior generation in 3D environments and proposes GEMS (Generative Embodied Multiple Behavior Control System). The framework integrates goal-directed and habit-driven behaviors, using a Habitual Controller, a Goal-directed Controller, and an Arbiter to select actions. It further employs a keyframe-constrained motion generation module to translate natural-language behaviors into 3D human–environment interactions.

Critical strengths

The authors conduct quantitative experiments, component ablation studies, and human preference evaluations across multiple daily-life tasks. The results demonstrate that the proposed method offers certain advantages in behavioral naturalness, task execution, and motion generation.

Critical weaknesses
  1. The novelty of the proposed method is not sufficiently clear. The main contribution of this work lies in integrating habit-driven control, goal-directed control, and action arbitration into a human-like agent, while further incorporating a keyframe-guided motion generation module, which have been extensively studied.
  2. The evaluation protocol may be mismatched with the original design objectives of X-VirtualHome. AgentSense is designed to generate behaviorally diverse data driven by persona diversity. When GEMS evaluates X-VirtualHome under a protocol consisting of a single persona and 30 shared goals, the low Naturalness (Nat.), Coherence (Coh.), and Personality Alignment (PA) scores reports nothing.
  3. The paper does not adequately explain the implementation of the internal action-mapping and simulator-vocabulary alignment pipeline for the pipeline-based baseline X-VirtualHome. It should explicitly state in Table 1 whether all baseline results were reproduced under the unified evaluation protocol of this paper or directly cited from the original publications.
  4. Baseline selection and the comparability of motion execution metrics require further improvement. The paper uses step success rate (SSR) and goal success rate (GSR) to evaluate motion execution. These metrics are affected by the number of sub-actions generated by each method, differences in task decomposition, and variations in success criteria. Therefore, even when methods are evaluated on the same high-level goals, differences in these metrics may not fully reflect their underlying motion generation capabilities. Furthermore, although the paper discusses related work on scene-aware text-to-motion generation, it lacks sufficiently comprehensive experimental comparisons with directly relevant methods such as TST-Motion.
  5. The paper does not provide a sufficiently detailed analysis of average inference time per trajectory, the computational overhead of individual modules, or model-call costs. Furthermore, implementation details concerning keyframe constraint generation, coordinate transformations, motion stitching, and relevant inference configurations require clarification.
  6. The already-established habits design does not model the formation, reinforcement, attenuation, or evolution of habits in real individuals over extended periods of daily activity. Therefore, the ability of GEMS to simulate long-term behavioral patterns remains insufficiently validated.
  7. The mathematical design is intuitive, but this can not verify that they have directly comparable behavioral meanings because the goal values and habit strengths are derived from different estimation procedures. Furthermore, the psychological-state intervention function is parameterized primarily based on behavioral effects reported in previous studies, but the correspondence between its specific functional form and actual human control strategies has not been sufficiently verfied.
Questions for authors

None.

2 comments

Thank you for your detailed and thoughtful review. Your feedback highlights key areas for improvement, and we have carefully considered each point to enhance our work. Below, we address your concerns and questions in detail.

W1: novelty is not sufficiently clear.

Thank you for your concern regarding the novelty of our work. We would like to clarify the following:

The contribution of this work is to propose a cognitive neuroscience-inspired framework to explore the benefits of habitual behaviors, which is important but have been largely overlooked in prior research. As the Reviewer zc63 mentioned, This is a Interesting research motivation and offering a meaningful perspective on generating more diverse and believable agent behaviors.

Although habitual behaviors and goal-directed behaviors have been extensively studied in cognitive neuroscience and human-like behavior generation, respectively, to the best of our knowledge, no existing 3D human-like behavior generation framework supports the generation of both habitual and goal-directed behaviors.

W2: The evaluation protocol may be mismatched

Thank you for your concern regarding the comparison with X-VirtualHome. We would like to clarify the following points.

First, X-VirtualHome simulates human daily activities in a virtual smart-home environment, aligning with our evaluation setting.

As stated in its paper, X-VirtualHome aims at "producing diverse and structured sensor data that capture the heterogeneity of human behavior". Therefore, X-VirtualHome considers not only persona diversity but also the diversity of action trajectories.

Second, Nat. and Coh. evaluate the naturalness and coherence of generated behavior sequences, respectively, and are persona-agnostic metrics.

Third, to address the reviewer's concern about persona variations, we conducted additional experiments using three personas from Appendix C: Michael (forest ranger), Elena (nurse), and Adrian (screenwriter). We adapted X-VirtualHome's persona inputs using its Persona Generation Prompt and followed our ablation study evaluation protocol.

MethodNat. ↑Coh. ↑PA ↑Mean. ↑
X-VirtualHome (Michael)42.0182.4074.7066.37
X-VirtualHome (Adrian)42.6884.9079.8869.15
X-VirtualHome (Elena)41.6485.3076.5867.84
GEMS (Ours) (Michael)89.4788.4581.1686.36
GEMS (Ours) (Adrian)91.2391.6087.9590.26
GEMS (Ours) (Elena)90.8889.0185.7488.54

The results show that persona variations have limited impact on Nat. and Coh., while GEMS consistently outperforms X-VirtualHome across all three personas, demonstrating its robustness to persona variations.

W3: Baseline implementation

We thank the reviewer for pointing out. All baseline results were obtained by rerunning the methods under our unified evaluation protocol.

For X-VirtualHome, we used its official open-source implementation and followed its original pipeline.

We have revised the caption of Tab. 1 to explicitly clarify the sources of the baseline results and how they were reproduced.

W4: Baseline selection of scene-aware text-to-motion methods and Motion metrics.

Baseline selection: In Tab. 2c, we have already compared our method with both typical and advanced scene-aware text-to-motion generation methods, including HOI-from-HLI, and InfBaGel.

Motion metrics: SSR and GSR are widely adopted in prior works [1, 2]. While these metrics may be influenced by action sequence differences, Tab. 1 additionally reports Action Score (Act.) and Quality Score (Qual.), measuring alignment with target behavior descriptions and motion quality, respectively. These metrics assess performance across action difficulty, and granularity, as well as motion quality without depending on overall sequence length.

W5: Computational overhead and implementation details of 3D generation module

Computational overhead: Thank you for this concern. Our work focuses on non-time-sensitive behavior simulation, so our evaluation prioritizes the quality of generated behaviors over computational efficiency. Moreover, the runtime of our LLM/VLM-based framework is difficult to estimate consistently, as scene complexity and behavioral requirements affect input lengths and the number of model calls. A comprehensive and systematic efficiency evaluation across diverse scenes and tasks would therefore incur substantial token costs. Considering the scope of our study and the associated evaluation costs, we have not treated computational efficiency as a primary evaluation metric.

Implementation details: We have revised Appendix H to provide further implementation details on keyframe constraint generation, coordinate transformations, motion stitching, and inference configurations. The revision also includes condensed prompt examples for planning and motion generation, along with an illustrative input to the pretrained motion generator, to improve reproducibility.

Specifically, given a behaviour description, the avatar's current state, and scene information, the VLM first identifies the spatial targets and interaction constraints required for the action, including target positions, body pose requirements, and spatial relationships with interacting objects. For example, for a grasping action, the VLM determines the target contact position and how the hand should interact with the object. The system then uses these constraints to solve for joint poses through inverse kinematics (IK), followed by pose refinement based on skeletal constraints and local geometric checks. This process produces 3D skeletal keyframes that satisfy the required kinematic and spatial constraints. Finally, the keyframes are arranged according to the planned action timing and provided as conditioning inputs to the pretrained motion generator to produce continuous motion.

W6 & W7 "lack of modeling of real individuals" and "correspondence to actual human control strategies."

We thank the Reviewer for the insightful comments regarding the "lack of modeling of real individuals" and the "correspondence to actual human control strategies." However, we believe that these concerns about modeling real individuals are more closely aligned with the area of Applications to Neuroscience & Cognitive Science.

We would like to emphasize that our motivation is to build believable human-like agents. As discussed in Appendix D.4, believability relies on sustaining an "illusion of life," in which an agent exhibits stable, natural, and individually distinctive behavioral patterns.

However, we have made every effort to address the Reviewer's concerns:

W6: As explained in Appendix E.2, our work investigates the role of habitual behavior as a driving factor in human-like behavior generation. Accordingly, we focus on fine-grained human behavior simulation over relatively short time horizons, typically spanning a single day. Therefore, it is reasonable to assume that habit memory remains stable throughout the simulation period. Furthermore, to ensure the authenticity of the habitual behaviors themselves, we ground them in data collected from real humans.

W7: First, as illustrated in Eq. (4), our design is inspired by the computational model of Miller et al. (2019), aiming to integrate habit strength and goal-directed action values, which arise from distinct psychological processes, into a behavioral decision-making framework. The effectiveness of this design has been validated by Tab. 1 and our human evaluation.

Furthermore, as discussed in Appendix F, the psychological state intervention function is designed to introduce state dependence and behavioral dynamics into human-like agent behaviors, thereby enhancing their believability and human-likeness. The results in Tab. 2b further support its effectiveness.

We understand the Reviewer's concerns regarding our designed arbitration mechanism. To further validate its effectiveness, we conducted an additional human evaluation involving 17 participants, comparing three arbitration mechanisms: LLM-based (Tab. 2a), Random, and Ours. Behavioral trajectories were randomly sampled from 15 runs per method.

RandomLLMOurs
Random–47.1%35.3%
LLM52.9%–29.4%
Ours64.7%70.6%–

The table reports pairwise human preference rates (row over column). Our method achieves higher human preference rates than both alternatives.

[1] Towards Human-like Virtual Beings: Simulating Human Behavior in 3D Scenes, Liang et al., ICCV 2025

[2] Synthesizing Physical Character-Scene Interactions, Hassan et al., SIGGRAPH 2023

Reviewer_NgUzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GEMS, a framework for generating believable human-like behavior in 3D environments. Its cognitive module combines a Habitual Controller, which retrieves cue-associated behaviors from personalized habit memory; a Goal-directed Controller, which evaluates candidate actions through LLM-based imagined outcomes; and an Arbiter, which combines action values, habit strengths, individual preferences, and psychological states. The selected behavior is realized through a motion pipeline that decomposes actions and supplies scene-dependent keyframe constraints to a pretrained text-to-motion model. The authors construct a dataset of 571 cue–habit pairs and evaluate GEMS on 30 shared daily-life goals, with five runs per goal. Evaluation includes automated behavioral ratings, motion execution metrics, ablations, and two human studies totaling 94 participants. GEMS achieves the highest reported behavioral Mean score and improves motion success relative to the evaluated baselines.

Critical strengths

Human evaluations support the practical benefit of habitual control. GEMS is preferred over its Goal-only ablation by 35/53 participants in the behavioral-sequence study and 28/41 in the motion study, with reported two-sided binomial pp-values of 0.027 and 0.028. These results provide evidence beyond automated LLM ratings that the additional behavioral system improves perceived human-likeness.

The paper integrates behavior selection with executable scene-grounded motion and evaluates both components. The motion pipeline addresses a practical limitation of scene-agnostic text-to-motion generation. The keyframe ablation increases SSR from 52.22% to 86.24%, while the cognitive ablations support contributions from habit memory, prospective evaluation, and arbitration. Using the same LLM backbone for cognitive baselines and introducing ACTOR† with shared habit memory and motion-generation resources are useful comparison controls.

Critical weaknesses

The specific arbitration mechanism is less convincingly validated than the general benefit of adding habits. The human study compares the full system with Goal-only, which establishes the value of the additional behavior source but does not isolate the proposed Arbiter. ACTOR† and the automated ablations provide partial evidence, yet a closely matched alternative using the same habit candidates, goal-directed proposals, and computation budget is missing from the human evaluation. Similarly, the increased habit ratio under adverse psychological states follows directly from the designed weighting mechanism. Appendix F appropriately describes its parameterization as heuristic, but this experiment demonstrates controllability more clearly than psychological validity. The contribution would be better established by showing that context-sensitive arbitration improves believability over simpler alternatives with comparable overall habit frequency.

I really do not think this work is innovative enough, it is not workable in real life.

Questions for authors

Arbitration: Does GEMS remain preferable when compared with simpler selectors using the same habitual and goal-directed candidates, matched computation, and a comparable habit ratio? A human comparison would help determine whether the gain comes from the proposed coordination mechanism.

Motion evaluation: For Table 2c, what scene information and spatial constraints are supplied to each baseline? Can results be broken down by action category and native support? How often does successful execution of the generated plan also satisfy the original goal?

Reproducibility: Please specify how the VLM produces valid 3D skeletal keyframes, including input representation, coordinate conventions, pose constraints, timing, and any optimization or correction. How are collisions and attachment/release events handled during execution?

Human-study generalization: How are the ten samples per behavior source assigned across questionnaires, and are comparisons matched for goal and context? Do preference gains persist when uncertainty is estimated across behavioral samples as well as participants?

5 comments

Thank you for your thoughtful and detailed review of our work. We appreciate your recognition of the originality and significance of the GEMS framework. Below, we address the weaknesses and questions you raised:

W1 & Q1: Effectiveness of the Arbiter under matched habit ratios is not clear

Thank you for your attention to the effectiveness of our arbitration mechanism, Tab. 2a already demonstrates the superiority of our Arbiter over LLM-based arbitration.

However, we agree that comparisons under matched habit ratios would further strengthen our claims. As your suggested, we conduct additional quantitative and human evaluations against a simpler Random Arbiter, which selects the habitual candidate with probability λ\lambda when available, and otherwise selects the highest-valued goal-directed candidate. We calibrate λ\lambda to match the habit ratios of Ours and report results averaged over 15 independent runs.

MethodMetricSetting 1Setting 2Setting 3Avg.
OursHabit Ratio (%)52.9441.8341.65–
Mean ↑85.8787.3184.7485.97
RandomHabit Ratio (%)51.2343.9041.17–
Mean ↑69.8974.2275.8273.31

As shown above, Ours consistently outperforms Random across all three settings despite comparable habit ratios, demonstrating the benefit of context-sensitive arbitration. To further assess perceived believability, we conduct a human evaluation with 17 participants comparing three arbitration mechanisms: LLM-based (Tab. 2a), Random, and Ours. Behavioral trajectories are randomly sampled from 15 runs per method.

RandomLLMOurs
Random–47.1%35.3%
LLM52.9%–29.4%
Ours64.7%70.6%–

Pairwise win rates are reported in the table above (row over column). Ours achieves higher human preference rates than both alternatives, further confirming that the proposed Arbiter improves behavioral believability beyond merely controlling habit frequency. The detials are revised in Appendix. A1.

W2: Regarding the Conflation of Execution Capability and Differences in Executed Behaviors in Tab. 1

We thank the reviewer for this concern and clarify the evaluation objective and interpretation of Tab. 1.

First, regarding the evaluation protocol, as stated in Section 4.1, Tab. 1 assesses whether each agent's motion generation module can faithfully execute behaviors produced by its own cognitive module. Differences in action difficulty, sequence length, and granularity naturally arise across frameworks, reflecting their distinct behavioral and execution capabilities. Thus, our evaluation focuses on each framework's overall ability to translate cognitive decisions into 3D behaviors.

Second, regarding the evaluation metrics, SSR and GSR are widely adopted in prior works [1, 2]. While these metrics may be influenced by action sequence differences, Tab. 1 additionally reports Action Score (Act.) and Quality Score (Qual.), measuring alignment with target behavior descriptions and motion quality, respectively. These metrics assess performance across action difficulty, and granularity, as well as motion quality without depending on overall sequence length.

Moreover, Tab. 2d directly demonstrates the contribution of our keyframe mechanism to motion generation performance.

We also clarify two related points:

  1. Regarding Tab. 2c: This experiment examines whether existing human–scene interaction methods can handle the diverse actions produced by our Cognitive Module, motivating the need for our Motion Generation Module. We therefore use their official open-source implementations without additional adaptations.
  2. Regarding physical plausibility: Our work focuses on believable human-like behaviors, following distance-based evaluation criteria widely adopted in prior works [1, 2, 3]. Physics-based simulation and collision handling are beyond our primary scope. Nevertheless, we acknowledge their potential to further improve believability and will consider them in future work.

Q2: Purpose and Evaluation Details of Tab. 2c

As clarified in W1 & Q1, we would like to reiterate that Tab. 2c aims to evaluate whether existing motion generation methods can realize the diverse behavior descriptions produced by our cognitive module. To ensure a reasonable comparison, we randomly sample 45 individual actions from our cognitive module and allow each baseline to process scene information and spatial constraints following its original pipeline.

Furthermore, as suggested, following [4, 5], we categorize the results in Tab. 2c into object-interaction (OI) and non-object-interaction (Non-OI) actions.

MethodNon-OI SSR ↑Non-OI Act. ↑OI SSR ↑OI Act. ↑
HUMANISE451.28241.34
HOI-from-HLI352.26401.27
InfBaGel602.00241.44

The results show that existing methods still exhibit limited execution success and action-description alignment across both categories, further highlighting the need to support diverse motion generation.

Q3:Reproducibility

We thank the reviewer for highlighting the need for a clearer description of the motion-generation pipeline. Our motion module uses a VLM-centered framework to generate 3D skeletal keyframes through spatial constraint generation, skeletal pose estimation, and pose refinement, which is described as below.

3D Skeletal Keyframe Generation. Given a behaviour description, the avatar's current state, and scene information, the VLM first identifies the spatial targets and interaction constraints required for the action, including target positions, body pose requirements, and spatial relationships with interacting objects. For example, for a grasping action, the VLM determines the target contact position and how the hand should interact with the object. The system then uses these constraints to solve for joint poses through inverse kinematics (IK), followed by pose refinement based on skeletal constraints and local geometric checks. This process produces 3D skeletal keyframes that satisfy the required kinematic and spatial constraints. Finally, the keyframes are arranged according to the planned action timing and provided as conditioning inputs to the pretrained motion generator to produce continuous motion.

Collision Handling. Our system aims to reduce collisions through advance planning and geometric checks, rather than guaranteeing collision-free execution. Specifically, it plans navigation paths based on scene geometry and uses keyframe planning and local geometric checks during motion generation to reduce collisions between the agent and surrounding objects.

Object Interaction. The system aligns grasping and release events with the corresponding motion keyframes. During grasping, the object is attached to the avatar's hand and moves with it. Upon release, the attachment is removed, and the object's position and scene state are updated to support subsequent actions.

We have also revised Appendix H to provide further implementation details, including condensed prompt examples for planning and motion generation, along with an illustrative input to the pretrained motion generator, to improve reproducibility.

Q4: Sample Selection, Goal and Context matched, Uncertainty

Sample selection. Following [4], we randomly selected 10 of the 30 goals and sampled one behavior per goal from each of six sources, forming a fixed pool of 60 samples per setting. Each questionnaire compared all 15 method pairs, with a randomly selected goal for each pair. Non-human methods shared the same initialization, while human samples were matched by goal only.

Uncertainty across participants and behavioral samples.

Preference Rate ↑95% CILeave-One-Goal-Out
Video73.2%[59.9%, 85.4%][71.8%, 74.8%]
Behavior Sequence67.5%[54.3%, 79.8%][66.0%, 68.9%]

Following [5], we perform bootstrap resampling over both participants and goals. As shown above, GEMS consistently outperforms four non-human baselines, with confidence intervals entirely above 50% and stable preference rates under leave-one-goal-out analysis. These results demonstrate that the preference advantage remains robust when accounting for variability across participants and sampled goals.

[1] Towards Human-like Virtual Beings: Simulating Human Behavior in 3D Scenes, Liang et al., ICCV 2025

[2] Synthesizing Physical Character-Scene Interactions, Hassan et al., SIGGRAPH 2023

[3] WANDR: Intention-guided Human Motion Generation, Diomataris et al., CVPR 2024

[4] Generating Human Interaction Motions in Scenes with Text Control, Yi et al., ECCV 2024

[5] Vision-Guided Action: Enhancing 3D Human Motion Prediction with Gaze-informed Affordance in 3D Scenes, Yu et al., CVPR 2025

[6] VBench: Comprehensive Benchmark Suite for Video Generative Models, Huang et al., CVPR 2024

[7] Treating Stimuli as a Random Factor in Social Psychology, Judd et al., JPSP 2012

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.