acceptodds
Under review as a conference paper at ICLR 2027

ConGRets: Low-Resource, Low-Latency Co-Speech Gesture Synthesis via Contrastive Latent Prediction

Abstract

How small can a complete co-speech gesture system be while scoring on par with state-of-the-art generators? This question matters most for LLM-powered chatbots and dialogue systems, where a response exists as text before speech is synthesized: gestures could be generated without waiting for audio, yet most generators rely on audio the agent does not yet have. ConGRets separates motion and speaker representation learning from text-conditioned gesture prediction. We learn the motion representation contrastively rather than through a VAE or VQ-VAE, then freeze the encoder before decoder training. A separate contrastive encoder derives a global speaker descriptor from reference motion, enabling new-speaker conditioning without generator retraining. A fine-tuned text encoder and fusion head predict the next 10-dimensional motion latent from response text, speaker conditioning, and previous motion, trained by contrastive alignment with the motion space. Thus, one predictor supports memory-based synthesis by selecting and blending recorded units, hybrid synthesis by decoding the selected unit's latent, and direct synthesis by decoding the predicted latent. On 138 common takes from five BEAT speakers excluded from training, and without audio, memory-based synthesis achieves 58% lower position-space Fréchet Gesture Distance (FGD) than the lowest-FGD baseline among seven audio-conditioned systems, with pose diversity close to recorded motion. Hybrid synthesis achieves comparable FGD; direct synthesis achieves lower FGD than three baselines. All modes use at most 12 million generation-time parameters, 27× fewer than that baseline, and after enrollment, loading, and warm-up process the transcript of a 69-second utterance in 15–19 ms on GPU or 23–31 ms on CPU, over 30 times faster than the fastest evaluated baseline on the same input. These results demonstrate that compact text-conditioned prediction can deliver strong distributional gesture quality at low generation cost, providing a practical design for interactive conversational agents and motivating deployment on resource-constrained edge devices.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
co-speech gesture synthesis, text-driven gesture generation, contrastive representation learning, latent motion prediction, efficient inference, conversational agents, virtual humans, unseen-speaker conditioning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KfqLOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents ConGRets, a compact text-conditioned framework for 3D co-speech gesture generation without relying on speech audio. The method separates motion and speaker representation learning from text-conditioned gesture prediction. Specifically, it embeds two-second gesture units contrastively in a 10-dimensional latent space and uses a separate contrastive encoder to derive a global speaker descriptor from reference motion, enabling new-speaker conditioning without generator retraining. A fine-tuned text encoder and a fusion head predict the next motion latent from response text, speaker conditioning, and previous motion. The framework supports three synthesis modes: memory-based synthesis by selecting and blending recorded units, hybrid synthesis by decoding selected unit latents, and direct synthesis by decoding predicted latents.

Critical strengths

1.Detailed and transparent documentation of the experimental setup and results

Critical weaknesses
  1. Contrastive learning has been explored in prior co-speech gesture synthesis works, such as Semantic Motion Anchors(https://arxiv.org/abs/2605.30608 ) and SemConFlow(https://www.springerprofessional.de/en/semconflow-semantic-grounding-of-holistic-co-speech-gesture-gene/53692922), but the paper does not clearly distinguish its contributions.
  2. Lack of a user study to evaluate the perceptual quality and appropriateness of generated gestures.
  3. The paper does not provide ablation studies comparing text-only and text-audio conditioning within the same framework, leaving the impact of removing audio on gesture quality and speech-gesture synchronization unclear.
  4. Limited evidence of generalizability, as the model is trained only on BEAT and cross-dataset evaluations rely on target-speaker enrollment.
Questions for authors
  1. Could the authors clarify how the proposed method differs from these works(see above) and what its main contributions are?
  2. Could the authors provide a user study to evaluate the perceptual quality and appropriateness of the generated gestures?
  3. Could the authors provide ablation studies comparing text-only and text-audio conditioning within the same framework to clarify the impact of removing audio on gesture quality and speech-gesture synchronization?
  4. Given that the model is trained only on BEAT and cross-dataset evaluations rely on target-speaker enrollment, could the authors further demonstrate the generalizability of the proposed method by running experiments on other datasets, referencing SemTalk (https://openaccess.thecvf.com/content/ICCV2025/papers/Zhang_SemTalk_Holistic_Co-speech_Motion_Generation_with_Frame-level_Semantic_Emphasis_ICCV_2025_paper.pdf)?
Reviewer_kNGZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ConGRets, a text-first co-speech gesture system for dialogue agents, where the response exists as text before the speech audio is synthesized. It trains a contrastive motion encoder that embeds two-second motion units into a 10-D space and freezes it, trains a separate contrastive speaker encoder that pools ten minutes of reference motion into a speaker descriptor, and fine-tunes BERT-mini with a small MLP head to predict the next motion latent from the text, the descriptor, and the previous latent. One predictor supports three modes: memory-based (replay selected recorded units), hybrid (decode selected latents), and direct (decode predicted latents). On five held-out but enrolled BEAT speakers, memory-based synthesis reports 58% lower FGD than the best of seven adapted audio-conditioned baselines, with at most 12M generation-time parameters and 15-19 ms warm GPU time for a 69-second input.

Critical strengths
  1. For LLM-based dialogue agents the reply text exists before the audio, so removing the audio prerequisite is a real deployment need, and the one-predictor-three-modes design (replay, hybrid, direct) is a clean way to cover different cost-quality points.

  2. The efficiency numbers are measured on one canonical input with warm repeats and disclosed boundaries, all ablation runs are reported including invalid ones, the baseline adaptations and their budgets are itemized, and the reproduction procedure is released with detail. This level of auditability is valuable and is a good contribution to the community.

Critical weaknesses
  1. Random selection from the same library gets FGD 0.042, but the learned M only gets 0.157. Random also beats M on BEAT and on three of the four external packs. Even a static pose beats six of the seven baselines and ConGRets-D. So the 58% improvement comes from replaying recorded motion, not from the text-conditioned predictor. The authors should show at least one evaluation where learned selection beats random selection from the same library.

  2. The paper ran no human study. The paper also says the two main metrics do not measure text-gesture fit. GT-Sim is near zero for all learned systems, but 0.82-0.94 for the assembled reference. Also, permuting the transcript or changing the speaker descriptor barely changes the predictions. So we do not know whether the system works as a text-conditioned generator at all. The authors should run a human evaluation for comprehensive evaluation.

  3. M and H can select from the test speaker's own recorded motion, but the baselines get weaker reference access. The cluster centroids and the speaker descriptor may also include the scored take. DSG+ and EMAGE are cap-limited and not converged. And the 58% number comes from one seed, while the three main seeds span 0.072-0.184. So it is hard to know how much of the win is the method and how much is the privileged access. The authors should add a cross-speaker or capped-memory ablation and report all seeds in the main table.

Questions for authors

please respond to my raised weaknesses and I will consider raise my score to weak accept based on the response.

Reviewer_eSwgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents ConGRets, which predicts co-speech gestures directly from text instead of going from text to speech and then finally to gestures. Unlike previous methods, the main focus of the architecture is to be lightweight and low-latency so that the resulting system can be used for virtual avatars on top of conversational agents.

Method-wise, the paper used a contrastive latent prediction framework. This requires them to build speaker and motion representations first and then use them for direct text to gesture prediction.

Critical strengths
  • The demo video shows good semantic gestures which probably are a result from one of the main design choices of the paper i.e. direct text to gesture generation.
  • I appreciate authors' idea to directly predict gestures from text and removing speech as the bottleneck (which usually results in a cascaded system).
Critical weaknesses
  • The paper is very hard to read and understand. This is one of the major weaknesses which leads me to strongly reject the paper. Some examples:
  1. Ln 101-106: This paragraph is very weirdly written, the related work is just mentioned without properly grounding in the paper's motivation.
  2. The introduction follows no proper structure of introducing topic, related works and connecting them to the actual motivation.
  3. Ln. 050: instead of saying "early systems", the draft says "early virtual humans"
  • The absence of speech synchrony. I know that authors frame this as a future work but i believe for the task of co-speech gesture synthesis having speech synchrony is highly important. The demo video shows gestures that are weirdly misaligned with speech. Ln. 165 claims that the system provides no prosodic alignment which is a bad thing (not a strength!)

  • Importantly, authors should have also provided comparison with motion-matching based generation frameworks as well, as the final architecture is very similar to motion matching based generators.

  • one thing which is also weird is that direct synthesis shows lower FGD but the qualitative results show strong jitter and bad quality.

  • No BeatAlign metric is reported, which is standard in gesture generation literature

  • The appendix tables and results show so many results without properly explaining each row or their abbreviations. it looks like they were directly copied from code to the paper.

Questions for authors
  • Sec 3.1 and 3.2: Gesture generation is inherently a many-to-many generation problem and while contrastive latent prediction can have benefits in semantic retrieval and generation, how does it solve or take care of the problem of many to many generation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.