- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ForeMotion, a gesture generation framework that leverages future speech context during training via stop-gradient distillation to improve a causal generator without increasing inference latency. Evaluated on the BEATX dataset across single- and multi-speaker setups, the method achieves notable reductions in FGD on single-speaker generation.
Critical strengths
1.The framework offers a clever mechanism for incorporating privileged future context during training without altering the causal inference pipeline. By sharing parameters between the causal and future-informed passes, the approach enhances target quality without introducing runtime lookahead or extra model overhead.
2.The empirical evaluation goes beyond comparing against external checkpoints by including proper causal baseline controls and perceptual studies. The proposed distillation strategy yields measurable FGD improvements over a regularized control, and the human evaluation demonstrates clear user preferences for the full system.
Critical weaknesses
1.The core mechanism remains unverified because the experiments do not isolate the specific benefit of matched future speech from general consistency regularization. Since the future pass is not independently trained and no controls disrupt future-speech alignment, it is unclear whether improvements arise from genuine future context or simply from the additional forward pass and training objective.
2.The most significant gains are restricted to the single-speaker setting, raising concerns about multi-speaker generalization and metric trade-offs. Multi-speaker FGD improvements are marginal, and gains in single-speaker generation come with minor regressions in beat alignment and diversity, while the user study pools ratings without controlling for participant-level variance.
Questions for authors
1.Does the expanded-context pass demonstrate higher accuracy on held-out gesture tokens, and can you provide a control experiment that disrupts future speech correspondence to verify that future context is the active driver of performance?
2.Are regularized control results available for the 23-speaker setup, and could you provide confidence intervals alongside a participant-aware analysis of the human preference data?
2 comments
Thanks for the detailed responses. My concerns have been mostly addressed, and I have decided to maintain my score.
Thank you for your careful review and recognition of the shared-parameter design and causal controls. We added a training-time context control, multi-speaker component ablations, gesture-composition analysis and pooled preference confidence intervals, detailed below.
Weakness 1: Future speech versus consistency regularization.
We trained a past-as-future control: future audio/text in the supervisory view are jointly replaced by a copy of the rolling past window, preserving the original past/current inputs. Both retain distillation, with data, initialization, budget, losses/schedules and causal-only evaluation settings matched.
Both trained models are evaluated causally. Corresponding future speech yields lower FGD and facial error with the objective unchanged, supporting a contribution from informative future context beyond adding another loss.
The views share one generator; the supervisory distribution is detached before guiding the causal prediction. Future context is used during training and dropped at deployment. Here and below, → means closer to the ground-truth reference in paper Table 1.
Weakness 2: Speaker settings, metric trade-offs and pooled preferences.
Paper Tables 1 and 2 show the larger single-speaker FGD gain and smaller multi-speaker body gain, alongside a larger facial improvement. Question 2 below adds the multi-speaker component study.
We also analyzed gesture composition. Scott, the single-speaker subject, has more deictic, iconic and metaphoric annotation time than the other speakers pooled:
The larger single-speaker gain coincides with more annotation time for gestures conveying semantic information. This motivates our hypothesis that future speech is particularly useful when it clarifies an earlier gesture; composition is a possible contributor to the setting-dependent gains.
The single-speaker component study improves FGD and facial error while BeatAlign and L1-Div move farther from their ground-truth references. We report these complementary properties together. Repeated generation with fixed checkpoints assesses sampling variability; objective-metric confidence intervals and comparison statistics are being finalized. Perceptual intervals appear below.
Question 1: Held-out token accuracy and disrupted correspondence.
A held-out exact-token accuracy comparison between views has not been measured. The training control under Weakness 1 changes supervisory speech while retaining distillation, then evaluates the causal generator using decoded-motion FGD.
Exact-token accuracy measures agreement with a canonical RVQ encoding. Our soft targets and coherent residual-path sampling account for codebook geometry and dependence on earlier codes. We therefore prioritize decoded gesture quality, using the training controls and component studies to assess supervisory utility.
Question 2: Multi-speaker controls and participant-aware uncertainty.
The requested multi-speaker regularized-control results are now available:
Adding distillation gives a modest FGD improvement and lower facial error over the regularized control. The full configuration has the lowest reported values for both measures. We will include the study in the revision.
We calculated pooled perceptual preference rates and Wilson intervals: