acceptodds
Under review as a conference paper at ICLR 2027

ForeMotion: Learning Causal Co-Speech Gestures through Future-Informed Supervision

Abstract

Generating co-speech gestures as speech unfolds requires predicting motion from incomplete acoustic and linguistic context. Complete training recordings, however, contain later speech that is unavailable to a causal generator at deployment. The challenge is to use this additional context for learning while preserving the generator's causal speech access at inference. We present ForeMotion, a framework for learning causal co-speech gestures through future-informed gesture supervision. ForeMotion represents motion as discrete gesture tokens and trains a shared autoregressive generator using paired-view gesture distillation. The causal gesture view receives a rolling window of past and current speech, whereas the future-conditioned gesture view accesses additional past and future audio and text from the training clip. Both views predict the same gesture targets from the same permitted motion history. The future-conditioned view produces probability distributions over possible gesture tokens. A gesture-token distillation loss trains the causal view to match these distributions, with gradients stopped through the supervisory predictions. At inference, only the causal view is executed, generating motion autoregressively from available speech and previously generated motion without requiring future speech. Experiments on BEATX demonstrate gains in measured gesture quality over the evaluated causal baselines, with benefits varying across future-context scopes, speaker settings, and evaluation metrics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Co-speech gesture generation, Multimodal learning, Autoregressive generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EHq5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ForeMotion, a gesture generation framework that leverages future speech context during training via stop-gradient distillation to improve a causal generator without increasing inference latency. Evaluated on the BEATX dataset across single- and multi-speaker setups, the method achieves notable reductions in FGD on single-speaker generation.

Critical strengths

1.The framework offers a clever mechanism for incorporating privileged future context during training without altering the causal inference pipeline. By sharing parameters between the causal and future-informed passes, the approach enhances target quality without introducing runtime lookahead or extra model overhead.

2.The empirical evaluation goes beyond comparing against external checkpoints by including proper causal baseline controls and perceptual studies. The proposed distillation strategy yields measurable FGD improvements over a regularized control, and the human evaluation demonstrates clear user preferences for the full system.

Critical weaknesses

1.The core mechanism remains unverified because the experiments do not isolate the specific benefit of matched future speech from general consistency regularization. Since the future pass is not independently trained and no controls disrupt future-speech alignment, it is unclear whether improvements arise from genuine future context or simply from the additional forward pass and training objective.

2.The most significant gains are restricted to the single-speaker setting, raising concerns about multi-speaker generalization and metric trade-offs. Multi-speaker FGD improvements are marginal, and gains in single-speaker generation come with minor regressions in beat alignment and diversity, while the user study pools ratings without controlling for participant-level variance.

Questions for authors

1.Does the expanded-context pass demonstrate higher accuracy on held-out gesture tokens, and can you provide a control experiment that disrupts future speech correspondence to verify that future context is the active driver of performance?

2.Are regularized control results available for the 23-speaker setup, and could you provide confidence intervals alongside a participant-aware analysis of the human preference data?

2 comments

Thank you for your careful review and recognition of the shared-parameter design and causal controls. We added a training-time context control, multi-speaker component ablations, gesture-composition analysis and pooled preference confidence intervals, detailed below.

Weakness 1: Future speech versus consistency regularization.

We trained a past-as-future control: future audio/text in the supervisory view are jointly replaced by a copy of the rolling past window, preserving the original past/current inputs. Both retain distillation, with data, initialization, budget, losses/schedules and causal-only evaluation settings matched.

Supervisory training contextFGD ↓BeatAlign →L1-Div →Facial-MSE ↓
Copied rolling past audio/text0.9860.75613.898.80
Corresponding future: paper reference0.5830.73113.777.82

Both trained models are evaluated causally. Corresponding future speech yields lower FGD and facial error with the objective unchanged, supporting a contribution from informative future context beyond adding another loss.

The views share one generator; the supervisory distribution is detached before guiding the causal prediction. Future context is used during training and dropped at deployment. Here and below, → means closer to the ground-truth reference in paper Table 1.

Weakness 2: Speaker settings, metric trade-offs and pooled preferences.

Paper Tables 1 and 2 show the larger single-speaker FGD gain and smaller multi-speaker body gain, alongside a larger facial improvement. Question 2 below adds the multi-speaker component study.

We also analyzed gesture composition. Scott, the single-speaker subject, has more deictic, iconic and metaphoric annotation time than the other speakers pooled:

SelectionScottOther 22 speakers
Test25.51%11.85%
Training selection26.18%10.86%

The larger single-speaker gain coincides with more annotation time for gestures conveying semantic information. This motivates our hypothesis that future speech is particularly useful when it clarifies an earlier gesture; composition is a possible contributor to the setting-dependent gains.

The single-speaker component study improves FGD and facial error while BeatAlign and L1-Div move farther from their ground-truth references. We report these complementary properties together. Repeated generation with fixed checkpoints assesses sampling variability; objective-metric confidence intervals and comparison statistics are being finalized. Perceptual intervals appear below.

Question 1: Held-out token accuracy and disrupted correspondence.

A held-out exact-token accuracy comparison between views has not been measured. The training control under Weakness 1 changes supervisory speech while retaining distillation, then evaluates the causal generator using decoded-motion FGD.

Exact-token accuracy measures agreement with a canonical RVQ encoding. Our soft targets and coherent residual-path sampling account for codebook geometry and dependence on earlier codes. We therefore prioritize decoded gesture quality, using the training controls and component studies to assess supervisory utility.

Question 2: Multi-speaker controls and participant-aware uncertainty.

The requested multi-speaker regularized-control results are now available:

ConfigurationToken reg.Distill.FGD ↓BeatAlign →L1-Div →Facial-MSE ↓
Regularized causal control✓✗0.4780.46712.515.83
Distillation only✗✓0.4720.45811.135.614
Full ForeMotion: paper reference✓✓0.4700.45310.425.21

Adding distillation gives a modest FGD improvement and lower facial error over the regularized control. The full configuration has the lowest reported values for both measures. We will include the study in the revision.

We calculated pooled perceptual preference rates and Wilson intervals:

ComparatorForeMotion selectionsPreference95% Wilson interval
CaMN26/4065.0%49.5%-77.9%
EMAGE38/4290.5%77.9%-96.2%
PyraMotion25/3375.8%59.0%-87.2%
MIBURI27/3871.1%55.2%-83.0%
Ground truth14/3935.9%22.7%-51.6%

Thanks for the detailed responses. My concerns have been mostly addressed, and I have decided to maintain my score.

Reviewer_ctcvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a gesture generation model that takes into account that some gestures (such as representational gestures) can precede speech. They address this by training a representation model that distills gesture info from the past and future during training. Then, they use causal inference that mainly relies on current speech and preceding motion. The study is relevant and is interesting. However, the contributions are minor, and the dataset that the authors use contains mainly beat gestures. This framework is more useful for representing semantic gestures.

Critical strengths
  • The authors target a relevant and timely issue: gestures can actually precede utterances, and current models predict them after speech
  • The study is well written and easy to follow, except for the core contribution, “PAIRED-VIEW GESTURE DISTILLATION”
Critical weaknesses
  • The empirical results based on the objective metrics seem weak. The study does not outperfom clearly outperform the baselines
  • The model evaluation should distinguish between representational gestures and beat gestures. This model might be more beneficial for representational gestures, as the authors cited in the introduction
  • The technical details of PAIRED-VIEW GESTURE DISTILLATION should be improved significantly. This section is the main controbiution and therefore should be better explained.
Questions for authors
  • Figures 1 and 2 both need significant improvement. The introduction explain the problem well, but figure 1 is not helpful in improving the understanding of the problem. The bodies in Figure 2 looks like AI generated, therefore, authors should improve the figure with proper editing and information.
  • Is it possible to isolate the effect of this model on the semantic gestures vs. beat gestures? I ask this because this framing (ForeMotion) should be better for semantic gestures.
1 comment

Thank you for your suggestions on the evaluation, gesture categories and presentation. We analyzed gesture composition and added a training-time context control, and will include these results and clearer explanations in the revision.\

Weakness 1: Objective results.

Paper Tables 1 and 2 report single-speaker FGD improvements over MIBURI and the regularized causal control. The multi-speaker body-FGD gain is smaller, while facial improvement is larger. To examine your concern about gesture categories, we analyzed the composition of the evaluated recordings:

Deictic/iconic/metaphoric annotation shareScottOther 22 speakers pooled
Test25.51%11.85%
Training selection26.18%10.86%

The larger single-speaker gain occurs for Scott, whose recordings contain more annotation time for gestures conveying semantic information in both selections. This motivates our hypothesis that future speech provides particularly useful supervision when it clarifies what an earlier gesture expresses. Gesture composition is therefore relevant to interpreting the difference between settings, and an aggregate score alone does not separate the contribution from each gesture category.

We also trained a past-as-future control, jointly replacing future audio/text in the supervisory view with a copy of the rolling past window while preserving the original past/current inputs. Data, initialization, budget, losses/schedules and causal-only evaluation settings are matched. Both retain the distillation objective.

Supervisory training contextFGD ↓Facial-MSE ↓
Copied rolling past audio/text0.9868.80
Corresponding future: paper reference0.5837.82

Both models are evaluated causally. The advantage of corresponding future speech supports the value of supervisory content beyond simply adding another training loss.

The revision will include this training control, the composition findings and repeated-generation uncertainty summaries.

Weakness 2 and Question 2: Semantic versus beat gestures.

The composition findings above address the category distribution in the single- and multi-speaker settings. They motivate examining whether the benefit varies with gesture type.

Category-specific FGD requires standardized windows, adequate feature samples and evaluation of both methods on the same intervals. Annotation durations and category frequencies vary considerably, so direct cropping would change the lengths and numbers of feature windows across categories. Our model comparisons use complete utterances; the annotation analysis characterizes their composition. We will present the composition results alongside this evaluation-protocol discussion to distinguish the observed setting-level gains from the hypothesis of category-dependent benefits.

Weakness 3: Explaining paired-view gesture distillation.

Training updates the gesture generator; regional motion codecs remain fixed. One shared generator performs two forward passes within a joint objective.

Both views receive identical permitted past/current audio/text, gesture history, within-step token prefixes and speaker identity. Only the supervisory view additionally receives future audio/text. Its expanded speech access changes the Temporal Transformer's state, which conditions the Kinematic Transformer. Neither view receives the target token or later gesture tokens.

The supervisory token distribution is detached and guides the causal distribution through forward KL, combined with token and auxiliary objectives. Deployment uses only the causal view.

We will rewrite Section 3.2 in input-to-distribution-to-loss order, define parameter sharing and stop-gradient explicitly, and expand implementation details. The contribution is training causal gesture generation with training-only future speech, supported by the added controls.

Question 1: Figures 1 and 2.

We will label Figure 1's current prediction time and visible speech context: bidirectional prediction accesses future audio/text, while causal prediction accesses only permitted past/current speech. The caption will identify this as a conceptual information-access comparison.

We will replace the human illustrations affected by super-resolution with clearer renderings. Figure 2 will show the shared generator, input paths, training-only future speech and direction of detached supervision.

Reviewer_eD9MOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper claims to show that future supervision is helpful for knowledge distillation for causally learning co-speech gesture generation. They use a setup with two alternativng objectives during training (an incremental one and one that can see the future) while at test time, only the incremental one is used. They show that gesture token regularisation along with distillation helps improve over MIBURI, which is a recently introduced online gesture generation model. The main claimed contribution is that future supervision helps when added along with gesture token regularisation.

Critical strengths

The main strength of this paper is that it shows that having future distillation helps in single-speaker online co speech gestures generation (as the margin over the multi-speaker setup is not very clear).

Critical weaknesses

A main limitation is that the paper doesn't conclusively demonstrate that the improvements that are found come from the future speech context. One could add a shuffled or mismatched future speech condition in order to demonstrate this more convincingly. The main improvement also only seems to be working for a single speaker from the current paper which is a big limitation and the difference with MIBURI is quite small when it comes to multi speaker settings (20% vs 2%) in motion quality and this in combination with no uncertainty estimates is a significant limitation for a system intended for interactive avatar applications. In addition, it's very important to have ablations in multi-speaker settings. Also, the paper uses quite a bit of jargon and is therefore somewhat difficult to read.

Questions for authors

is the future-conditioned view itself better in terms of token accuracy or FGD than the causal view? Do gesture onsets shift?

2 comments

Thank you for your constructive review. Following your suggestions, we added a training-time context control, multi-speaker component ablations and repeated generation evaluations. We will incorporate these results and clearer explanations into the revision.

Weakness 1: Contribution of future speech.

We trained a past-as-future control: future audio and text in the supervisory view are jointly replaced by a copy of the rolling past window, preserving the original past/current inputs. Data, initialization, budget, losses/schedules and causal-only evaluation settings are matched.

Training contextFGD ↓BeatAlign →L1-Div →Facial-MSE ↓
Copied rolling past0.9860.75613.898.80
Corresponding future: paper reference0.5830.73113.777.82

Both retain the distillation objective. Corresponding future speech yields lower FGD and facial error, supporting the value of supervisory content during training. Here and below, → means closer to the ground-truth reference in paper Table 1. With this new experiment, we can show that corresponding future speech reduces FGD and facial error relative to copied past speech under the same distillation objective, supporting a contribution from informative future context beyond simply adding another training loss.

Weakness 2: Smaller multi-speaker improvement.

We acknowledge the smaller body-FGD gain; the new multi-speaker component results appear under Weakness 4. We also analyzed gesture composition:

Deictic/iconic/metaphoric annotation shareScottOther 22 speakers
Test25.51%11.85%
Training selection26.18%10.86%

The larger single-speaker gain coincides with more annotation time for gestures conveying semantic information. This motivates our hypothesis that future speech is particularly useful when it clarifies an earlier gesture. Body and facial benefits also differ, indicating dependence on the measured property.

Weakness 3: Uncertainty estimates.

The revised ForeMotion is evaluated across five generation seeds on the same test set with fixed checkpoints. Values are mean ± SD; confidence intervals and p-values are being finalized.

SettingFGD ↓BeatAlign →L1-Div →Facial-MSE ↓
Single (n=5)0.5856 ± 0.01060.7384 ± 0.005913.440 ± 0.2577.563 ± 0.226
Multi (n=5)0.4747 ± 0.03660.443 ± 0.008010.760 ± 0.0275.430 ± 0.021

Weakness 4: Multi-speaker ablations.

We completed two additional component ablations and compare them with the full-model paper reference:

ConfigurationToken reg.Distill.FGD ↓BeatAlign →L1-Div →Facial-MSE ↓
Regularized causal control✓✗0.4780.46712.515.83
Distillation only✗✓0.4720.45811.135.614
Full ForeMotion: paper reference✓✓0.4700.45310.425.21

Adding distillation improves FGD and facial error over the regularized control. The full configuration also brings BeatAlign and L1-Div closer to ground truth.

Weakness 5: Terminology and clarity.

We will clarify Section 3.2 and the overview figure: one shared generator performs two forward passes within one joint objective, rather than alternating objectives. Both views share past/current speech, gesture prefixes and speaker identity; only the supervisory view receives future audio/text. Its detached distribution supervises the causal prediction through KL loss. Deployment runs only the causal view. We will define inputs before losses, illustrate visibility and stop-gradient direction, and simplify terminology.

Question 1: Future-conditioned prediction quality.

Weakness 1 evaluates future-informed training through decoded-motion FGD under causal-only evaluation. Exact-token accuracy measures agreement with a canonical RVQ encoding, whereas our soft targets and residual-path sampling account for codebook geometry and prefix dependence. We therefore prioritize decoded gesture quality alongside the training controls and component studies.

Question 2: Gesture onsets.

Supplementary comparison videos illustrate gesture-speech coordination; the perceptual study evaluates naturalness. We have not quantified a systematic onset shift. This requires distinguishing preparation from the meaning-bearing stroke and aligning it with its spoken expression (ter Bekke et al.). Revised captions and introduction will distinguish the timing motivation from the measured gesture-quality effects.

thank you for the careful response. In my view it clarifies the results for the single speaker setting, which shows a reliable improvement; however, the additional analyses also seem to show that indeed there is just a small and probably unreliable gain in the multi-speaker condition; also, with the uncertainty estimates, it seems that the 5-seed mean and SD highlight that the original values in the paper were much lower than would be expected from the multi-seed mean and SD. Was the original number a very lucky run or a different setting? I will increase my score to 2 due to the additional analysis, but I don't think it clears the acceptance threshold.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.