acceptodds
Under review as a conference paper at ICLR 2027

PersonaAct: A Persona-Centric Dataset for Conversational Motion Generation

Abstract

For 3D digital characters to perform expressively, their motion should reflect not only what they say and do, but also their personalities. However, datasets that link persona descriptions to conversational motion remain limited, making it difficult to learn and evaluate how characters express their personalities through motion. To address this issue, we first present PersonaAct, a multimodal 3D motion dataset with 7,542 clips covering 302 personas. It pairs persona profiles and motion descriptions with synchronized speech and motion across different dialogue scenarios. Second, we evaluate motion quality and alignment with persona profiles, speech, and motion descriptions. To assess whether the motion fits the intended character, we introduce a persona–motion matching metric. Third, experiments show that current methods struggle to balance multiple conditions, particularly persona profiles and motion descriptions. These findings highlight key challenge in persona-conditioned motion generation. PersonaAct provides data and evaluation tools for developing and testing methods that address these challenges. Dataset visualizations and examples are available at https://anonymous.4open.science/r/PersonaAct-ICLR.

Primary area:
datasets and benchmarks
Keywords:
human motion generation, motion dataset and benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DE8dOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents PersonaAct, a dataset and benchmark for generating conversational body motion from speech audio, motion descriptions, and natural-language persona profiles.

The dataset contains 15.3 hours of performances across 302 personas, with synchronized speech, body, hand, and facial motion. The current benchmark focuses on body motion. The authors adapt existing generation models and introduce a learned persona–motion evaluator to measure how well motions match persona profiles.

Critical strengths
  • The dataset addresses a useful problem: generating conversational motion that reflects a natural-language persona. Capturing hands and faces alongside body motion also makes the data useful beyond the current benchmark.
  • The supplementary examples show a useful range of performances, from restrained gestures to large, expressive movements.
  • The conditioning ablations across different baselines are informative.
Critical weaknesses
  • Could the authors clarify whether any motion smoothing or artifact correction was applied beyond the temporal processing described in Appendix A? If videos of the original performances were recorded, will they also be released?
  • Do method rankings under P-M agree with human judgments? Could the authors also report the proportion of uncertain responses and inter-rater agreement?
Questions for authors

Please refer to the weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to reviewer DE8d

We thank the reviewer for the positive assessment and helpful feedback, and for recognizing the value of PersonaAct and our conditioning ablations. We address your questions below.

W1: Motion processing and availability of original recordings

Thank you for requesting clarification. Beyond the temporal processing described in Appendix A, we applied motion smoothing to improve temporal continuity. We recorded synchronized videos of the original performances, but these will not be publicly released to protect performers’ privacy. Researchers with a specific need may contact the authors to request access under strict restrictions. We will clarify these details in the revised paper.

W2: Agreement between the P-M evaluator and human judgments

Thank you for this suggestion. We evaluated 6 methods, with 30 clips per method and 10 ratings per clip (1,800 responses in total). Human scores are the percentage of all ratings selecting the target persona. P-M scores were computed on the same motion–persona pairs.The results are summarized below.

MeasureResult
Method-ranking agreement (Spearman’s ρ\rho)0.647
Uncertain responses12.44%
Pairwise rater agreement (uniform-random baseline: 33.33%)48.78%
Mean vote share of the more-selected persona64.67%
P-M scores show a positive correlation with human method scores (Spearman’s ρ = 0.647), providing evidence of correspondence at the method-ranking level.

Thank you again for recognizing the value of PersonaAct and for your helpful suggestions. We will incorporate the additional processing details and human-evaluation results into the revision to make our documentation and evaluation more complete.

Reviewer_GvwnOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PersonaAct, a multimodal dataset that combines human motion, character persona profiles, and synchronized text and speech. The dataset comprises 7,542 clips, with an average duration of 6.85 seconds. Its construction follows a three-stage pipeline: persona design, dialogue design, and performance capture.

To demonstrate the potential of the dataset, the authors introduce a new task: generating human motion conditioned on speech audio, textual descriptions, and persona profiles. They also propose a persona–motion alignment score to evaluate the consistency between a character's persona and the generated motion. The authors benchmark state-of-the-art human motion generation models fine-tuned on PersonaAct under different conditioning settings: audio-only, text-only, and audio-plus-text.

Finally, they conduct a user study to assess the quality of the dataset and evaluate the effectiveness of the proposed alignment metric.

Critical strengths
  • The paper is really clear and easy to follow. The proposed protocols, including both dataset construction and benchmark evaluation, are rigorously designed and well documented.

  • The dataset and the proposed evaluation metric represent valuable contributions to the field, opening up new research directions with practical applications in areas such as 3D animation and video games.

Critical weaknesses
  • The paper and supplementary materials would benefit from including qualitative video examples of generated samples to better illustrate the benchmark results.

  • The event synchronization distance provides a useful measure of speech–motion coordination, but it does not appear to explicitly evaluate lip synchronization, which is a key cue for assessing audio–visual speech alignment. Incorporating a measure of lip-sync quality would provide a more comprehensive evaluation of temporal synchronization between speech and generated motion.

Questions for authors
  • Persona generalization: How well does the model generalize to entirely unseen persona categories? Within a given category, to what extent do the training and test sets contain similar personas?
3 comments
AuthorsOpenReview ↗

Response to reviewer Gvwn

We thank the reviewer for the positive assessment and constructive suggestions, and for recognizing the value of our dataset and evaluation framework. We address your comments below.

W1: Qualitative examples of generated motion

Thank you for this suggestion. We have added full-body renderings of motions generated by all six benchmarked methods, alongside the corresponding reference motion. These examples are available at https://anonymous.4open.science/r/PersonaAct-visualization/. We appreciate this suggestion, which helps us present the benchmark more clearly.

W2: Lip synchronization and evaluation scope

Thank you for this suggestion. Our current benchmark focuses on full-body motion. To support tasks requiring facial motion evaluation, we additionally trained an audio–mouth synchronization evaluator inspired by Yang et al. [1], using our recorded ARKit mouth/jaw coefficients. We have added the implementation details and validation results to Appendix B and will release the evaluation code and weights.

Reference: [1] Yang et al. Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks, Methods, and Applications. CVPR, 2024.

Q1: Clarification of the unseen-persona setting.

Thank you for requesting clarification. Our unseen subsets are defined by holding out persona IDs from training. The split operates at the persona-ID level, rather than by holding out predefined persona categories. Using frozen BERT embeddings [1], we compared persona descriptions within each role/background category. Mean cosine similarity was 0.9158 between unseen and training profiles and 0.9143 among distinct training profiles, indicating comparable average textual similarity. This split is designed to assess within-distribution generalization to held-out persona profiles. We have added detailed split statistics in Appendix A.4 and separate seen/unseen benchmark results in Appendix E, allowing a direct comparison of performance on seen and held-out persona profiles under the same evaluation protocol. Thank you for this valuable suggestion. It has inspired us to consider extending the benchmark to evaluate generalization to entirely unseen persona categories in future work.

[1] Devlin et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL-HLT, 2019.

Thank you again for your positive assessment and constructive suggestions. We have added further visualizations, a supplementary lip-sync evaluation, and separate results for seen and unseen personas to the revised manuscript to better demonstrate the dataset’s capabilities and support future research.

I thank the authors for their responses and efforts made to address my concerns, particularly by providing qualitative visualizations and introducing an additional lip-synchronization evaluation metric.

However, I believe that the evaluation of persona generalization is an important aspect of this work and should be more explicitly incorporated into the final version of the manuscript to further strengthen its evaluation framework.

Overall, I appreciate the authors' efforts to improve the manuscript and lean toward acceptance.

Thank you for your positive assessment and support. We greatly appreciate your constructive suggestions for strengthening our evaluation framework. We will make the persona-generalization evaluation more explicit in the final manuscript. Thank you again for your time and careful consideration.

Reviewer_rcy8OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PersonaAct, a conversational motion dataset containing around 7.5K clips across more than 300 personas and ~15 hours of recordings. It pairs persona profiles and motion descriptions with speech, body & hand motion, facial expressions. The authors benchmark existing generation methods, introduce a persona–motion matching metric, and conduct conditioning ablations and human evaluations. The results highlight the difficulty of balancing multiple conditions simultaneously.

Critical strengths
  • The dataset addresses a relevant gap by combining natural-language persona profiles with conversational speech and motion. Having more than 300 personas broaden character coverage beyond single-character resources such as SuSuInterActs.
  • The recordings include body motion, hand gestures, facial expressions, and speech. This gives the dataset potential beyond the current body-only benchmark, including more complete character animation.
  • The collection process includes professional actors, reviewed scripts, manual inspection, and automatic integrity checks. These steps provide a useful foundation for building a reusable motion dataset.
  • The benchmark evaluates motion quality, description alignment, persona matching & speech coordination separately which makes competing strengths and weaknesses visible.
Critical weaknesses
  • This work provides dataset + benchmark contribution, but doesn’t build a novel method to solve the tasks on this dataset, limiting the overall contribution of this work.
  • The new persona metric needs stronger validation. Agreement with human judgments is around 70% on real motion and around 65% on generated motion, based on small studies with five participants. Random negative profiles may also make the task easier than distinguishing similar personas.
  • Important dataset collection information is missing such as performer details, including the number of actors and how personas were distributed among them. How were participants recruited to satisfy the constraint? Without this information, it is difficult to assess performance diversity or whether models could associate personas with individual actors’ movement styles.
  • Generalization to unseen personas is unclear. The main text describes a dialogue-level split, while Appendix A mentions unseen-persona subsets without reporting their sizes or separate results. The reported scores therefore do not establish how well the benchmark supports evaluation on new personas.
  • The dataset-consistency study does not isolate personality conveyed through motion. Participants receive speech alongside motion, so dialogue content or vocal delivery could reveal the intended character. The reported 94.1% also excludes uncertain responses without stating their frequency.
  • The speech-coordination metric produces results that need further explanation. Text-only MoMask achieves a better ESD than real motion despite receiving no audio and having a jerk ratio of 7.76. This raises the possibility that the metric rewards motion activity or coincidental timing rather than meaningful speech coordination.
  • Most comparable datasets are collected at 120 fps or higher, but this work collects motions at 30 fps.
  • The supplementary video lacks the complete mesh renderings and the facial expressions demonstrated look limited. Also, in some examples, the persona doesn’t match the tone of the speech. Also, the video lacks the results from the motion generation benchmarks.
Questions for authors
  • How many actors participated, how were personas assigned to them, and were any personas performed by multiple actors? Are actor identities available for evaluating generalization across performers? Was an IRB obtained and the actors were compensated?
  • What are the exact clip, dialogue, and persona counts in each split? Please clarify the unseen-persona subsets mentioned in Appendix A and provide separate results for seen and unseen personas.
  • Can you evaluate persona matching using harder negatives that share the same occupation or action but differ in personality? Are there examples of the same action or script performed under different personas that could help separate personality from action content?
  • For the human studies, what proportion of responses was uncertain, and how consistently did participants agree? Does the dataset-consistency result remain strong when speech is removed, and does the P-M evaluator preserve human rankings of the generation methods?
  • How does ESD respond to shuffled or time-shifted audio? These controls would help explain why a text-only model scores better than real motion and establish whether the metric measures audio-specific coordination.
  • Can the authors provide full-body renderings for some of the dataset samples and also some examples from generated results from methods used in benchmarking?
4 comments
AuthorsOpenReview ↗

Response to reviewer rcy8 —Part1

We thank the reviewer for the careful review and constructive feedback, and for recognizing the value of PersonaAct’s multimodal data and benchmark. We address your concerns point by point below.

W1: Dataset and benchmark contributions

Thank you for raising this point. PersonaAct’s primary contribution is a reusable dataset and benchmark for persona-conditioned conversational motion generation. Generating conversational motions that reflect a character’s persona is important for digital human interactions. Data supporting this task remain scarce and challenging to collect. To address this gap, we constructed PersonaAct. We also reveal how existing methods differ in satisfying multiple conditions, offering insights that we hope will help future work. By releasing the dataset and evaluation tools, we hope to support further research on persona-conditioned motion generation and provide the community with a shared basis for fair, transparent, and reproducible evaluation. This focus on reusable resources and systematic evaluation motivated our submission to the Datasets & Benchmarks area.

W2, W5 & Q3, Q4: Human validation of persona matching

We conducted additional user studies to address your questions. The evaluation protocols and results are detailed below.

P-M evaluator validation with harder negatives. To further enhance the validation of P-M, we conducted additional user studies to assess its agreement with human judgments under random and harder negatives. Following your suggestion in Q3, we selected harder negatives that share similar occupational or habitual-action contexts with the target persona but have different personality descriptions. The results are summarized in the table below.

EvaluationClips × ratings per clipTarget personaAlternative personaUncertainPairwise rater agreement (uniform-random: 33.33%)Mean vote share of the more-selected personaP-M agreement with human majority
Random negatives50 × 1065.40%24.60%10.00%56.49%70.60%79.49% (31/39)
Hard negatives30 × 1061.00%32.33%6.67%54.52%71.00%74.07% (20/27)

P-M agreement uses clips where one persona receives at least 6 of 10 votes. Pairwise agreement measures how often two raters give the same response, including “uncertain.” Mean vote share averages the more-selected persona’s share of all 10 ratings across clips.On clips with a strict human majority, P-M achieves 79.49% agreement with random negatives and 74.07% with harder negatives, providing additional evidence of correspondence with human judgments under both settings.

Persona matching from motion alone.

Participants in the above studies evaluated silent skeleton animations without speech. They selected the target persona more often than the alternative under both random negatives (65.40% vs. 24.60%) and hard negatives (61.00% vs. 32.33%). These results support the quality of PersonaAct by providing evidence of persona–motion consistency without speech cues.

Agreement with human method rankings.

We evaluated 6 methods, with 30 clips per method and 10 ratings per clip (1,800 responses in total). The human score is the percentage of all responses selecting the target persona. P-M scores were computed on the same pairs.

MeasureResult
Method-ranking agreement (Spearman’s ρ)0.647
Uncertain responses12.44%
Pairwise rater agreement (uniform-random: 33.33%)48.78%
Mean vote share of the more-selected persona64.67%

These results show positive correspondence between P-M and human method rankings. We will include the evaluation protocol and results in the revised paper.

Uncertain responses.

The tables above report uncertain-response rates for the additional user studies. In the original multimodal study with speech, uncertain responses accounted for 13.60% of all ratings.

Same-action or same-script performances.

Our current studies assess persona matching by comparing candidate persona descriptions for a given motion clip. Due to collection-budget constraints, we did not additionally record the same action or script under different personas. We will consider including such controlled performances in future extensions of PersonaAct.

AuthorsOpenReview ↗

Response to reviewer rcy8 —Part2

W3 & Q1: Performer details and persona assignment

Thank you for requesting these collection details. We recruited 6 actors with similar body proportions to reduce differences caused by body shape and make it easier to represent their motions using a common skeleton. We selected 6 actors from 20 candidates through screening and live auditions. They all had 1–2 years of acting experience. Each persona was assigned exclusively to one actor, no persona was performed by multiple actors, maintaining consistent portrayal across recordings. A motion-capture director with an academic background in arts and screenwriting oversaw actor selection, acting guidance, and quality control throughout recording.

These measures aim to reduce the influence of individual acting styles and maintain consistent, high-quality persona portrayals. We will release an anonymized actor ID for each clip to help researchers evaluate generalization across performers. We have clarified these details in Appendix A.1 of the revised paper.

W4 & Q2: Data splits and generalization to unseen personas

All clips from the same dialogue remain in the same split, and unseen persona IDs do not appear in training. The tables below report the numbers of personas, dialogues, and clips in each subset, followed by separate benchmark results for seen and unseen personas.

Dataset split statistics

SubsetPersonasDialoguesClips
Train2621,0315,328
Val-seen120120600
Val-unseen20102525
Test-seen120121625
Test-unseen2093464
Total (unique)3021,4677,542

Results on Test-seen

MethodNR@1 ↑R@2 ↑R@3 ↑FID ↓ESD ↓DiversityJerk RatioP-M Acc. ↑
Real Motion62549.1863.3071.740.0000.446222.5291.00090.97
EMAGE6253.397.2611.33486.1620.58532.2801.05746.33
DiffSHEG6259.0216.2922.1457.5950.2998‾\underline{0.2998}21.1813.38977.95
HY-Motion6256.4010.9415.71217.6140.362515.9111.193‾\underline{1.193}48.56
T2M-GPT62529.86‾\underline{29.86}41.60‾\underline{41.60}49.63‾\underline{49.63}19.3930.471022.3105.93293.84‾\underline{93.84}
MoMask62526.5937.9545.4125.032‾\underline{25.032}0.245022.071‾\underline{22.071}7.18195.13
SentiAvatar62438.5353.4062.2828.6020.438121.9782.01287.88

Results on Test-unseen

MethodNR@1 ↑R@2 ↑R@3 ↑FID ↓ESD ↓DiversityJerk RatioP-M Acc. ↑
Real Motion46445.0959.2266.770.0000.469522.4171.00074.27
EMAGE4644.749.4412.89484.8340.60582.3921.13857.93
DiffSHEG4648.5315.6022.3362.9420.3113‾\underline{0.3113}21.4233.61569.09
HY-Motion4645.569.8713.79234.0830.374015.7831.205‾\underline{1.205}60.57
T2M-GPT45919.2231.0739.3946.733‾\underline{46.733}0.595222.074‾\underline{22.074}8.59672.00‾\underline{72.00}
MoMask46422.24‾\underline{22.24}33.02‾\underline{33.02}40.09‾\underline{40.09}47.2150.236522.3478.58581.93
SentiAvatar46438.6252.2461.0836.8180.467221.9752.09871.61

We have added the split statistics to Appendix A.4 and the separate seen/unseen results to Appendix E, clarifying both the evaluation protocol and performance on each subset.

W6 & Q5: Validity of the speech–motion coordination metric

Thank you for highlighting this observation. We adopt ESD from SentiAvatar to measure temporal proximity between audio beats and motion events. Such event-based metrics may be affected by motion-event density and coincidental alignment. This is why we also report Jerk Ratio alongside ESD: MoMask’s lower ESD is accompanied by a Jerk Ratio of 7.76, indicating substantially greater jerk than real motion. Following your suggestion, we conducted additional controls.

ControlOriginal ESDESD after shuffling/shifting
Audio shuffling0.43140.4848
Time shift: ±0.5 s0.57580.5760
Time shift: ±1 s0.57580.5838
Time shift: ±2 s0.57580.6207

These results indicate sensitivity to audio pairing and larger offsets. We will clarify in the revised paper that ESD measures temporal proximity between detected audio beats and motion events, with GT serving as a reference. Each metric evaluates a different aspect of generated motion, so overall performance should be assessed by considering multiple metrics together.

AuthorsOpenReview ↗

Response to reviewer rcy8 —Part3

W7: Motion capture frame rate

Thank you for pointing out this ambiguity. Our raw motion data were captured at 110 fps, and we will also release the original motion data at 110 fps. Following the 20-fps setting used by T2M-GPT and MoMask on HumanML3D, we resampled the captured motion to 20 fps for our benchmark. We have clarified the 110 fps capture and 20 fps processing rates in Appendix A.2. We will also release the original 110 fps motion to support different temporal resolutions.

W8 & Q6: Qualitative visualizations and benchmark results

Thank you for this suggestion. We have added full-body renderings of dataset samples and qualitative examples from the benchmarked motion generation methods, available at https://anonymous.4open.science/r/PersonaAct-visualization/. We directly animate the avatar’s face using the recorded ARKit coefficients, making facial expressions easier to observe.

Regarding speech tone, actors followed the persona profiles and dialogue scripts under the motion-capture director’s guidance. So vocal delivery reflects both the persona and the immediate dialogue context, including the character’s emotional state.

Ethical considerations

Ethics approval.

Our project was reviewed and approved by our institutional ethics committee.

Actor consent and compensation.

The actors were informed about the study and agreed to the research use and public release of their processed motion data. They were compensated for their participation. Original performance videos will not be publicly released to protect their privacy.

Human evaluation.

Annotators were informed about the evaluation task before participation. They received compensation for the time and effort spent completing the evaluations.The evaluation results do not disclose annotators’ private information.

Thank you again for your detailed feedback. We hope these responses and additional results help clarify the points. We will incorporate these updates into the revised paper and supplementary material.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

As the rebuttal discussion period is approaching its deadline, we would like to kindly follow up on our response to your comments.

We sincerely hope that our clarifications and additional results have adequately addressed your concerns. We would greatly appreciate it if you could take another look at our response when time permits.

If our responses have resolved your concerns, we would also be grateful if you would consider updating your score accordingly.

Thank you very much for your time and thoughtful feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.