acceptodds
Under review as a conference paper at ICLR 2027

Bringing Scripts to Acting! Content-Variant Control for performance Emotion-Alignment of Acting generation

Abstract

We introduce Script-to-Acting, which generates controllable acting from a raw script. Existing text-to-motion and motion-stylization methods typically keep the motion content fixed and only rescale style or intensity, a paradigm we term content-consistent control. Acting is different: when the requested emotional state changes, the content of the performance can change with it. As the intensity along an emotion axis increases, the acting may shift from a submissive plea to more confrontational actions, such as a waving reprimand, a shove-away, or a shoulder shake. We therefore construct TheaterAct, a multimodal dataset for acting generation containing 7.2 hours of professional performance from 59 characters, with aligned scripts, dialogue, motion, and fine-grained emotion annotations. Based on this dataset, we design Acting Mind, Acting Morph, and Acting Creation. Acting Mind uses a dramaturgical context projector to interpret script, character, and dialogue through structured relationship, motivation, and emotion cues. Acting Morph represents dramatic emotion as a token-wise acting-mode matrix whose coordinates are grounded in professional performances indexed by script semantics, emotion axis, and continuous intensity. Rather than broadcasting a global style code, the matrix allocates dramatic control to individual script tokens and produces an emotion-aware motion feature for synthesis. Acting Creation then realizes this content-variant control on top of any pretrained motion generator, upgrading it to support emotion-aligned acting without architectural changes. Experiments show that our method outperforms existing motion-generation models in script alignment, emotion-intensity alignment, physical plausibility, and content-variant acting. Notably, the proposed modules can be integrated with multiple motion-generation backbones as a generic enhancement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Acting、Motion Generation、Diffusion Model 、Emotion Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Hm2YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a framework for generating human motion corresponding to theatrical lines by leveraging theatrical priors, such as play scripts.

The framework consists of three components: Acting Mind, which extracts structured information from the script; Acting Morph, which retrieves relevant examples from a theatrical motion database based on this information; and Acting Creation, which generates the corresponding human motion. Specifically, Acting Mind uses an LLM to extract emotional cues and contextual information about the characters from the script. Acting Morph then performs semantic and emotional similarity-based retrieval over a database of theatrical examples, combining the retrieved information into an acting mode matrix that characterizes the intended performance. Finally, Acting Creation leverages the outputs of the two preceding components to condition a human motion diffusion model.

To support this framework, the authors introduce TheaterAct, a dataset comprising human motions extracted from theatrical videos using off-the-shelf 3D human pose estimation methods, along with annotations describing the plays, characters, and corresponding lines.

Finally, the authors evaluate their approach against baseline methods using human preference studies, standard quantitative metrics, and newly introduced metrics designed to assess the intensity and emotional expressiveness of the generated motions.

Critical strengths
  • The paper addresses a highly relevant and new problem: theatrical human performance generation. This setting offers an interesting setup to investigate the relationship between emotion and human motion in a nuanced and naturalistic context, beyond conventional synthetic scenarios.

  • The proposed dataset is a valuable contribution to the field and provides a useful resource for future research on theatrical performance and expressive human motion generation.

Critical weaknesses
  • The paper is difficult to follow, and several aspects of the proposed approach would benefit from clearer explanations and a more structured presentation.

  • The evaluation protocol appears relatively limited and provides insufficient insight into the effectiveness of the proposed method. In particular, comparing against pretrained models trained on HumanML3D and prompted with theatrical lines may not provide a sufficiently strong baseline, as these inputs are likely to be out of distribution. Stronger baselines should be considered, such as straightforward fine-tuning of existing pretrained checkpoints on the proposed TheaterAct dataset, to better isolate the benefits of the proposed framework.

  • Similarly, the user study is biased and does not capture the subjective nature of theatrical performance and emotional expression. A more fine-grained evaluation would be valuable, for example, by asking participants to identify the emotions conveyed by the generated motions. Given the inherently subjective nature of the task, a carefully designed user study should be a central component of the evaluation.

  • The qualitative results are not entirely convincing. For instance, in the example "Frailty, thy name is woman" with increased anger intensity, the generated motion exhibits artifacts, including discontinuous movements. Moreover, the increased anger intensity appears to be expressed primarily through more vigorous arm and leg movements, rather than through a more nuanced and realistic portrayal of anger.

  • Further ablation studies would strengthen the evaluation. In particular, the contribution of each loss component should be investigated individually, rather than focusing primarily on the emotion loss, to better understand the role of each component in the overall framework.

Questions for authors
  • Is the selection of emotions considered in this work grounded in an established psychological framework or prior study? Clarifying the rationale behind the choice of emotions would strengthen the methodological foundation of the work.

  • Have the authors considered evaluating their approach against larger-scale pretrained motion generation models, such as HY-Motion (Tencent, 2025) or KIMODO (Rempe et al., 2026), which are trained on substantially larger datasets and may therefore support a broader range of prompts?

Reviewer_6gPNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces script-conditioned acting generation in which emotion intensity can change the action content. It combines script analysis, token-wise retrieval of professional motion motifs, and a conditioned motion generator. The authors also construct TheaterAct, a dataset aligning theatrical motion with scripts and emotional conditions under limited scope. Experiments on several motion backbones, component ablations, and a human study evaluate the approach.

Critical strengths

The motivation is clear and interesting. A more intense performance may require a different action than a larger version of the same gesture. This makes script-to-acting a useful extension of existing motion-control tasks. Testing four backbones and including a 52-participant human study provide useful support.

Critical weaknesses
  1. The emotion metrics need more independent validation. Equation 7 minimizes Lemo=−φe⋆(x^0)\mathcal L_{\mathrm{emo}}=-\varphi_{e^\star}(\hat x_0), and line 270 says the same φ\varphi is used for evaluation. Equation 9's RemoR_{\mathrm{emo}} measures an encoder-score percentile, not accuracy at the requested intensity. Higher scores therefore do not establish better control, especially for low targets. This raises a proxy-optimization concern (Gao et al., 2023); the user studies help, but its sampling and low-intensity coverage are unclear. Moreover, the user studies and dataset construction scope are too small.

  2. The contribution of token-wise allocation is not well separated from the retrieved prefix. SingleMorph is a useful control, but Table 3 does not report a fair comparison under each module (acting morph / acting creation) with the full model. Without prefix lengths or suffix-only results, it is also hard to tell how much of the content change comes from the new allocation and how much comes from retrieval and inpainting.

Reference

Gao, Leo, John Schulman, and Jacob Hilton. "Scaling laws for reward model overoptimization." International conference on machine learning. PMLR, 2023.

Questions for authors
  1. Can independent raters evaluate agreement with the requested intensity, including low targets? Please report the human study's scene counts, sampling procedure, and coverage across intensity levels.
  2. With matched fine-tuning and prefix lengths, does token-wise allocation improve motif switching and script fit over SingleMorph? Does the improvement remain when evaluating only the generated suffix?
  3. Can you scale up the dataset or provide further discussion on why the experimental discoveries are solid despite the limited scope?
Reviewer_WspFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Script-to-Acting, a framework that turns a raw script, a character, a dialogue line, and a continuous emotion axis/intensity into expressive full-body acting motion. It introduces three modules: Acting Mind (structured script interpretation and emotion vector), Acting Morph (token-wise retrieval of professional acting motifs instead of a global style code), and Acting Creation (conditioned motion generation with reconstruction, retrieved-prior anchoring, and frozen emotion reward). It also releases TheaterAct, a 7.2-hour, 730-clip multimodal theatrical dataset with script, character, dialogue, motion, emotion-axis, and intensity annotations. The paper argues that acting requires content-variant control: increasing intensity should change the acting motif, not merely rescale a fixed gesture.

Critical strengths
  1. Interesting problem formulation. Script-to-Acting is a meaningful step beyond text-to-motion and motion stylization; the distinction between content-consistent control and content-variant acting is clearly motivated.
  2. Dataset contribution is valuable. TheaterAct provides script-level, character-level, and line-level annotations with six emotion axes, continuous intensity, and motivation tags, which are rare in existing motion-language datasets.
  3. Token-wise acting-mode matrix is a reasonable design. Replacing a global style vector with token-wise motif binding is a plausible way to allow different parts of a line to realize different acting modes.
  4. Modular integration. Acting Morph and the frozen emotion reward are designed to plug into multiple backbones, and the paper tests MDM, MLD, MoMask, and MMM.
Critical weaknesses
  1. Main quantitative table is missing or incomplete in the provided PDF. The paper references Table 2 and “Table 2 layout” but only Table 3 (ablation) and Table 4 (HumanML3D) are visible. The main comparison against MDM, MLD, MoMask, and MMM on TheaterAct is not shown, so the headline claim cannot be verified.
  2. Duplicate contribution bullet. The TheaterAct dataset contribution is listed twice almost verbatim. This suggests careless writing and should be fixed.
  3. Retrieval-based anchoring may leak or copy test-time motion. The paper states the test play is excluded from the retrieval pool, which is good, but it does not quantify how much of the generated motion is copied from retrieved clips. A nearest-neighbor / copy-ratio analysis is needed.
Questions for authors
  1. How do you handle copyright, actor consent, and license for the theatrical source videos in TheaterAct?
  2. Why is the TheaterAct contribution bullet duplicated, and can you confirm the final contribution list?
  3. What are the exact physical-plausibility thresholds and contact-detection rules for sliding, floating, and penetration?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.