acceptodds
Under review as a conference paper at ICLR 2027

MoTween: In-Betweening for Part-wise Text-to-Motion Generation

Abstract

Text-to-motion generation aims to synthesize semantically consistent 3D human motions from natural language descriptions. Although recent approaches have achieved remarkable progress, existing methods either overlook the inherent hierarchical structure of the human body or struggle to capture the temporal dynamics described in the text. To address these limitations, we propose MoTween, a novel two-stage framework consisting of Hierarchical Vector Quantized Variational Autoencoder(Hierarchical VQ-VAE) and In-Betweening Motion Generation. In the first stage, we introduce a Hierarchical VQ-VAE that decomposes each pose into fine-grained body parts and coarse body groups, propagating group-level context into part representations to reflect the inherent body structure. In the second stage, we generate motion representations at semantically critical keyframes and subsequently complete the full sequence by in-betweening the remaining frames, enabling temporally coherent generation. Experiments on the HumanML3D and KIT-ML datasets demonstrate that MoTween outperforms existing state-of-the-art methods on both benchmarks, while also enabling flexible part-level motion editing. Code and models will be released soon.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-Motion Generation, Keyframe Prediction, Hierarchical VQ-VAE, Part-level Motion Editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AxZjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MoTween is a two-stage VQ text-to-motion model. Stage 1 encodes six parts and three groups (whole/upper/lower body) with separate encoders, refines part latents by cross-attention to the group latents, and quantizes them with part-specific codebooks. Stage 2 predicts from text a per-frame event class for each part (idle or ±x/±y/±z dominant velocity), and changes in this class define part-wise keyframes. A Keyframe Transformer generates latent anchors at those frames autoregressively, the gaps are linearly interpolated, and an In-Betweening Transformer (temporal, then part-wise attention) predicts the part tokens. Training event labels come from a rule-based procedure on GT motion. Results on HumanML3D and KIT-ML are slightly above MoGenTS; the appendix adds predictor accuracy, oracle keyframes, tokenizer comparisons, length-stratified results and slot-swap editing.

Critical strengths

Keyframe-first decoding is a sensible alternative to uniform AR or masked decoding in token-based T2M, and the paper tests it from several sides. With the same tokenizer, full-sequence AR gives FID 0.069 and random-mask completion 0.042, vs. 0.031 for MoTween (Tables 9 and 2). The advantage holds on long sequences with all methods re-run in one pipeline (Table 13), and the oracle-keyframe and weaker-predictor experiments (Tables 7-8) show how upstream errors propagate. The paper is clearly written and inference is fast (0.24 s).

Critical weaknesses
  • One-to-many generation is not addressed. As described, the pipeline is deterministic given the text: the event predictor outputs one class per frame, the Keyframe Transformer is a deterministic recursion (Eq. 3), and the token head takes "the index with the highest probability." The same prompt would then always yield the same motion, and MModality, a standard column on both benchmarks, is missing from Table 1. Relatedly, 95.8% frame-wise event accuracy and 82.8% keyframe F1 (±5 frames) from text alone are surprisingly high, because text does not determine when a hand or foot changes direction (the step phase during walking, for example). These numbers need a majority-class baseline and the class distribution, and the paper should state whether they are on the test split and whether GT length is given.

  • The gain hinges on hand-tuned pseudo-labels, and the SOTA claim is overstated. Event thresholds are set per "action category" and body part (App. B.2), but neither dataset has action categories and the paper does not say how they are assigned. The dependence is large: with one global setting, FID goes from 0.031 to 0.072 and Top-1 from 0.537 to 0.520 (Table 4), worse than the paper's own random-masking variant (0.042 / 0.523, Table 2). So keyframe guidance beats masked completion only with this tuning. On the benchmarks, gains over MoGenTS are within the reported intervals (HumanML3D FID 0.031±.002 vs. 0.033±.001; KIT Top-1 0.448±.005 vs. 0.445±.006), and published SALAD has clearly higher R-Precision (Top-1 0.581, Top-3 0.857; App. Table 12). The abstract and the "best results across all evaluation metrics" claim in Sec. 4.2.1 should be revised.

Questions for authors
  • Is anything sampled at inference (events, anchors, tokens)? Please report MModality and show several samples for one prompt. This is the point most likely to change my score.
  • How are action categories obtained for HumanML3D and KIT-ML? What does a majority-class predictor score on event accuracy and keyframe F1? What is the average number of keyframes per part relative to sequence length?
  • In Hierarchical Cross-Attention, does attention run over time? If each part attends to one group latent per frame, the softmax has a single key and the module reduces to adding a projected group latent. Likewise, does Slot Attention take CLIP token features or the pooled 512-d sentence embedding? With a single input vector, every slot receives the same update.
  • How is motion length set at inference, how do frame-level keyframes map to tokens if the encoder downsamples in time, and does Stage-2 training use GT or predicted keyframes?
  • In slot-swap editing, both the cross-slot self-attention in the event predictor and the part-wise attention in the In-Betweening Transformer let a swapped slot affect other parts. Do unedited parts actually stay fixed? Joint error on unedited parts relative to the base motion would show this.
Reviewer_HpJxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies single-person text-to-motion generation and proposes a two-stage framework combining hierarchical body representations with keyframe-guided in-betweening. In the first stage, the body is divided into six parts and three groups corresponding to the whole, upper, and lower body. Hierarchical cross-attention incorporates group-level context into the part features, which are then quantized using separate part-specific codebooks and reconstructed into full-body motion by a shared decoder. In the second stage, the model predicts directional motion events for each body part from text and uses event transitions to determine keyframe positions. It first generates sparse keyframe features and then produces the complete motion through interpolation and an in-betweening network with temporal and part-wise attention. Both keyframe positions and representations are predicted by the model, without requiring user-provided key poses. The paper evaluates generation on HumanML3D and KIT-ML and includes tokenizer comparisons, module ablations, keyframe prediction analyses, and demonstrations of part-level editing.

Critical strengths
  1. The hierarchical tokenizer incorporates whole-body and upper- and lower-body context before part-level quantization, helping balance local motion details with coordination across parts.
  2. It achieves FID values of 0.031 and 0.137 on HumanML3D and KIT-ML, respectively, performing strongly among the comparisons.
  3. The ablations and diagnostic experiments are fairly thorough and support the effectiveness of the main components.
Critical weaknesses
  1. Although the main text and appendix include some recent methods, the comparisons still rely substantially on earlier work and omit competitive methods in 2026, like MotionHiFlow.
  2. The supplementary video provides dynamic motion comparisons, but all results are shown as skeleton animations, which offer limited visibility into body surfaces, inter-limb contact, and penetration.
  3. The event labels rely on empirically selected parameters that vary by action category and body part, but the definition and source of the action categories, the handling of composite actions, and the complete parameter configuration remain unclear.
Questions for authors
  1. How are action categories defined and obtained when constructing event pseudo-labels? For sequences containing multiple actions, how are the applicable categories and parameters determined?
  2. The method figures and visualization figures in the paper employ three different forms of motion presentation, which can be unified.
  3. I recommend including competitive recent methods such as MotionHiFlow to strengthen the assessment of performance relative to current methods.
Reviewer_ocNAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MoTween combines a body-part/group VQ-VAE with text-predicted part-wise motion events. It generates sparse keyframe representations, interpolates intermediate features, and completes the sequence using temporal and part-wise attention. It reports strong text-to-motion results and qualitative part-level editing examples.

Critical strengths
  1. The body-structured representation and keyframe-first pipeline are coherent and achieve competitive results.
  2. The anatomical hierarchy differs from residual/temporal-scale quantization and provides a plausible way to exchange group-to-part context.
Critical weaknesses
  1. The incremental novelty needs clearer attribution. MoMask already supports temporal inpainting, and MoMask++ uses temporal multi-scale residual quantization. MoTween's anatomical hierarchy differs, but the gains attributable to that hierarchy remain insufficiently isolated.
  2. The benefit of event-based keyframe placement is unclear. Existing ablations change interpolation or the generation strategy together with placement, so they do not isolate why event boundaries are preferable to simpler sparse schedules.

I am open to raising my rating if the rebuttal clarifies the novelty beyond existing methods and demonstrates the benefits of the hierarchy and event-based keyframe placement through convincing ablations.

Questions for authors
  1. Could the hierarchy in Fig. 3(a) be compared with parameter-matched flat part attention or simple group-feature fusion, using the same codebook settings and Stage-2 training budget?
  2. Could you compare predicted event keyframes with uniform/random keyframes at matched per-part counts, keeping the generation architecture, interpolation, and training budget fixed?
  3. Does Table 2's "w/o Interpolation" retain the predicted keyframe anchors? Could you compare the full Stage 2 with standard masked generation using the same frozen tokenizer, text conditioning, and training budget?
  4. Have you identified why SALAD's Top-1 is 0.525 in your reproduction versus the reported 0.581 (Appendix Table 12)?

Comments on the paper

2 comments

According to the comparison with MotionHiFlow, the proposed is slightly inferior to the SOTA methods. Meanwhile, the authors should consider whether the shape and surface is necessary for this paper. I believe Figure 1-4 should be further polished.

We thank the reviewer for the follow-up. The revised PDF with updated Figs. 1–4 is uploaded.

1. Comparison with MotionHiFlow

We agree: MotionHiFlow has higher R-Precision (Top-1 0.563 vs. 0.537), and the paper no longer claims state-of-the-art. Our claim is narrower: with FID within MotionHiFlow's confidence interval on both benchmarks, MoTween is 3.4× faster (0.240 vs. 0.812 s), has the lowest FID among discrete-token models (MoGenTS 0.033, MoMask 0.045, MoMask++ 0.072), supports part-level editing through its part-wise tokens (Sec. 4.2.3), and is lower on every physical-plausibility metric of our previous response (penetration 0.057 vs. 0.079 cm, float 2.295 vs. 2.771 cm, skating 5.62 vs. 6.73%, capsule collision 3.04 vs. 6.91%). We state this trade-off explicitly in Sec. 4.2.1.

2. Shape and surface

Since both benchmarks and all compared methods are joint-based, we believe body shape and surface fall outside the scope of this work, and we have kept skeleton renderings throughout. To still address the underlying concern about contact and penetration, we add the shape-independent metrics from our previous response to App. G, and we now state this scope in Sec. 4.1. We would be happy to add surface renderings if the reviewer finds them helpful.

3. Figures 1–4

We have revised Figure. 1–4 for clarity and conciseness in the updated PDF, and we thank the reviewer for the suggestion; the figures are cleaner as a result. If the reviewer had specific aspects in mind beyond these changes, we would be glad to incorporate them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.