acceptodds
Under review as a conference paper at ICLR 2027

Bounding Latency Contribution of a Dyadic Avatar System

Abstract

Talking-head and dyadic models now achieve real-time inference, yet fast motion generation alone does not produce an interactive conversation. A live system must synchronize the model's output with speech from an external voice agent, schedule video frames for playback, and handle interruptions. Existing work reports throughput or aggregate latency but does not show how generation and serving interact to shape the latencies a user actually experiences. We introduce XXXX-1, a compact autoregressive flow-matching motion model together with an end-to-end system for live dyadic avatar interaction. We show how to turn chunk-based generation into a real-time conversation between a user and an avatar. We analytically derive the system's contribution to two user-facing latencies and validate it with two commercial voice agents. Further experiments demonstrate that XXXX-1 achieves leading performance among the compared dyadic models across visual quality, lip synchronization, and listening behavior, while its inference runtime operates in real time on data-center and consumer GPUs. We release the https://anonymous.4open.science/r/bounding-dydactic-latency/anonymized code.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
real-time conversational avatars, dyadic interaction, talking-head generation, streaming generation, ML systems, latency modeling, media scheduling, flow matching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tmh4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method to achieve interactive talking-head avatar generation for dyadic conversations, that is to say live conversations between two participants. The approach consists in a motion generator, an audio encoder, a renderer, and a streamer. The results give good continuity and stability, in particular because of the motion representation, I feel. The method is compared quantiatively with respect to several competing works, and with various metrics designed for video quality evaluation. The quantitative results are compared with these approaches, but only in still frames; the video results only show the proposed methods' output.

Critical strengths

1/ The problem of avatar generation is timely and useful. The goal is clearly stated and important. 2/ The results are of good quality, and are shown over long rollouts. It is clear that the choice of using the motion representation presented in section 3.1 has a positive influence on the stability of the method. Indeed, other approaches often may have subtly unrealistic motions, quite characteristic of Generative AI. The proposed approach does not seem to exhibit this, even if this does result in a slight wooden quality of the faces (the expressions remain quite neutral).

Critical weaknesses

1/ My main criticism is the heavily "system" orientation of the paper, and the explanations of the technical details, which are very hard to understand. It does not seem like a novel algorithm is proposed, rather a system of existing algorithmic bricks. The authors did not convey what was new very well, I find. The difficulty in understanding the technical details is due in large part to the lack of formalism in the paper, which I go into further detail below. I do not feel that the diagrams are clear enough to understand everything from them. Formulae and/or pseudo-algorithms would have been useful here.

2/ I will go here into more specifics concerning the lack of formalism. Apart from in the appendix, there are no notation or any equations, apart from at line 231. This necessarily makes the paper difficult to read, and hard to pin down what is going on exactly. This is particularly problematic for a research paper, which should have much more formalism. At the moment, it is a long textual description of the approach, and all technical details are pushed into the appendix, which is problematic. Some examples:

  • Line 107: "Each frame is represented by a 42-dimensional vector of head rotation and expression deformation derived from LivePortrait’s motion–appearance disentangled space (Guo et al., 2024). The renderer separately handles appearance. Further details are provided in Appendix". These details need to be in the main paper, especially the formalism and notation.
  • Line 121: "The conditional flow-matching Transformer (Lipman et al., 2023) autoregressively generates five-frame motion chunk ... ". Again, this needs to be formalised, and more details given. What is "other-audio" ?
  • What is a "Stack" exactly? I feel that this is a system terminology which is not super common in the present literature (autoregressive video models). This should be made clear.
  • Line 140: "To solve this, we build a chunk-based encoder through self-distillation of the original model without changing its architecture." You put the details in the appendix, but I feel that it should be in the main paper, because of its importance.
  • Line 150: "At initialization, it registers the source portrait and caches its LivePortrait appearance features and reference keypoints." Caches how? Where? With respect to what model?
  • Rendering pipeline: At no point do you say what model is used for rendering, or how it works from a technical viewpoint. The details are, again, pushed into the Appendix, but even looking, I could not find them. Is it a diffusion model? You say that a diffusion model is used for generating the motion, but what about the image/video renderer? How does it work?
  • Equation (no number, line 321): Why are TwaitT_{wait} and TpadT_{pad} random variables? It seems logical, but you do not explain why, or say it explicitly.

3/ There are no visual comparisons with other methods in the supplementary material. Why is this ? It seems important to do this, and would potentially be in your favour. As it is, the only visual comparisons are in Figures 7-8, and these do not really seem to favour your method. In particular in Figure 7, it seems that the proposed approach does not move much with respect to the ground truth (speaker). Video comparisons would help dispel this concern.

4/ Section 3.3 I found particularly difficult to read. It is a long sequence of textual description, with no notation at all for any of the objects introduced. This whole section would need to be re-written with clear, formal notation, otherwise I feel that it is impossible to understand or implement.

Questions for authors

1/ Can you summarise what is different in your approach, compared to previous ones, in algorithmic terms ? Apart from the system aspect, what is algorithmically different ? What is your added value ?

2/ The name of the method "XXXX-1" seems a bit odd. Is this a mistake on the part of the authors ? (maybe a macro that was not set ?). I would suggest creating a slightly more memorable name, in the interest of the method.

3/ Section 3.4. I had great trouble understanding this section, but most of all, I had trouble understanding in what sense the latency model was "analytical". When we say "analytical", we usually mean a problem has a closed-form solution, ie a mathematical formula to solve the problem. Where is this the case? In the equation? In general, I could not really understand what was going on in this Section, could you explain it simply, and indicating what you mean by "analytical"?

Small comments:

4/ Although I see that the term "dyadic" has been used in other publications, maybe you could define it here? It may not be obvious to all readers.

1 comment

We would like to thank reviewer tmh4 for the detailed critique of the formalism, which drove the revision.

W1. System orientation of the paper, novelty

  1. The system orientation is intended: the paper is submitted to ICLR's "infrastructure, software libraries, hardware, systems, etc." area. We agree the original text did not say clearly what is new, and we revised the contributions accordingly.
  2. The new element is the model-agnostic streamer together with its latency bounds: Algorithm 2 converts variable-size speech from an external voice agent into the fixed windows a chunk-based generator requires, and the paper derives closed-form bounds on what this conversion adds to response and interruption latency, in terms of the generator's timing parameters. The second contribution is the open end-to-end system as a whole. Inside it, the motion generation model is a fixed workload; its components are existing techniques, and we no longer present them as main contributions

W2. Difficult understanding of the paper

  • We have added formal definitions of the terms, two algorithms, and additional figures to improve presentation.
  • Model internals stay in the appendix by design, because the model is a fixed workload for the latency analysis
  • On the examples: "stack" now reads "system". Other-audio is the user's audio, as opposed to the avatar's own speech, and both terms are defined at first use. The renderer interface formalizes what the generator takes and returns; the motion vector and the encoder distillation are defined in the appendix. The frame synthesizer is not a diffusion model: it is LivePortrait's warping network and decoder, with the portrait features computed once at session start and cached. T_wait and T_pad are random because they depend on when and how the voice agent delivers its reply, which the system does not control. The revision says this explicitly.

W3. Lack of visual comparisons with other methods

  • We have added side-by-side comparison videos with the baselines in the supplementary material to assess the performance. On expressiveness: perceptually our expression variance and diversity are the closest to ground truth among the compared methods.

W4. Section 3.3. is difficult to read

We rewrote it with formal notation and a pseudo-algorithm (see above).

Q1: what is algorithmically different

Prior work reports model speed or generation-only latency. None of it covers the path from a voice agent's reply to playback, although it adds significant latency. Our contribution is this path: a window-formation algorithm whose rules give closed-form bounds on the added latency, validated live. Any chunk-based generator that meets the renderer interface can replace our model.

Q2: the name "XXXX-1"

It is the anonymized name for double-blind review, shared with the anonymized repository. A footnote now explains this.

Q3: in what sense the latency model is "analytical"

We meant closed form: each latency contribution is a sum of fixed and bounded random terms, so the bounds follow directly. We agree that "theoretically" overstated this, and the revision says "closed-form bounds".

Define "Dyadic"

Speech-driven talking head generation aims to generate facial and head movements conditioned on given audio. Dyadic systems extend this setting by conditioning motion generation on signals from both conversation participants.

Reviewer_XjCUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents XXXX, an end-to-end system for real-time dyadic avatar interaction that combines an autoregressive flow-matching motion model, a streaming audio encoder, and an audio-video scheduling backend. Its main contribution is an analysis of how audio buffering, lookahead, and rendering schedules contribute to response and interruption latency beyond model inference time. Under specified assumptions, the authors derive latency bounds and validate them using two commercial voice agents. Experiments also demonstrate competitive visual quality and listening-motion performance against existing dyadic models.

Critical strengths
  1. The paper connects audio buffering and rendering schedules to response and interruption latency through a conditional analytical model. This provides a useful framework for understanding why real-time generation alone does not guarantee responsive interaction and for identifying where an avatar system introduces additional delay.
  2. This work integrates motion generation, streaming audio encoding, rendering, synchronization, and interruption handling, with detailed engineering descriptions. Competitive results against evaluated dyadic models, runtime measurements across multiple GPUs, and integration with two commercial voice agents support its practical relevance and potential usefulness to researchers building interactive avatars.
Critical weaknesses
  1. The model combines LivePortrait representations, autoregressive flow matching, dual-stream audio conditioning, regional guidance, and correlated noise, but lacks controlled ablations to establish which components drive the improvements.
  2. The evaluation includes only ten turns per voice agent and does not systematically vary fragment sizes, arrival phases relative to the scheduling cycle, or consecutive-segment boundaries while maintaining the stated assumptions.
  3. Sec. 3 assumes that a newly arriving speech segment waits less than 200 ms iteration. However, rule R4 prevents admitting a new segment into a window where the preceding segment ends. It is unclear how the waiting-time bound holds when preceding speech remains queued.
  4. Lower pose variance and diversity than ground truth suggest potentially subdued head motion. Controlled ablations of the other participant’s audio and human evaluation would help establish whether listening reactions are natural and responsive.
Questions for authors
  1. Do you have existing controlled ablations using the same training data that isolate the contributions of the proposed components? In particular, does removing or shuffling the other participant’s audio meaningfully change listening behavior when self-audio is fixed?
  2. Are the total latencies in Table 3 measured independently from input/output event timestamps, or reconstructed by summing the four modeled terms?
  3. How is <200 ms guaranteed when preceding speech remains queued, given that R4 prevents a new segment from entering a window where another segment ends? Does the bound require an additional queue-state assumption?
  4. How did you verify that extending Avatar Forcing’s generation limit from 30 to 600 seconds did not degrade its performance? Do existing results on sequences within its original length limit support the same conclusions?

I would consider raising my rating if the authors satisfactorily clarify the issues above.

1 comment

We would like to thank reviewer XjCU for the concrete, answerable questions, which directly shaped the revision

Lack of controlled ablations

  1. We agree that the paper presented the model as if it were a main contribution, which invites this request. The revision reframes the motion generator as the system's fixed workload. The main contribution is the model-agnostic streamer with its latency bounds, and the second is the end-to-end system as a whole. The bounds are stated in the generator's timing parameters and hold for any chunk-based generator that satisfies the renderer interface (Sec. 3.1). New experiments check them across generator configurations (Sec. 6.3). Component ablations would not change any claim of the revised paper, so we leave them out of scope and have softened the quality claims to "competitive".
  2. However, two components had evidence in the previous version: self-distillation reduces the audio encoder's feature error from about 50% to 12% (Appendix D), and Figure 8 shows the effect of noise truncation (Appendix E).
  3. The new Appendix I.1 adds an ablation with the other-audio guidance scale set to 0. Without the other audio conditioning, PFD and rPCC get worse for both components (Table 8), so the listening metrics depend on the other participant's audio.

The evaluation includes only ten turns per voice agent and does not ... vary fragment sizes, arrival phases ... or consecutive-segment boundaries ...

  1. We have raised the number of turns in commercial voice agent runs to 50 per each (Table 2)
  2. We have added additional experiment Controlled sweep (Section 6.2) that systematically varies first fragment's arrival phase and fragment size. It demonstrates that latencies reach both ends of their bounds.
  3. The bound is not applicable to consecutive-segment boundaries by definition: a segment that follows earlier avatar speech is not a reply by the RLC definition, and the right-padding rule (Sec. 3.2) admits it on the step after the preceding segment ends.

It is unclear how the waiting-time bound holds when preceding speech remains queued

The bound concerns replies, and a reply by definition follows the user's speech while the avatar is silent, so no avatar speech is active or queued when its start marker arrives. The marker is admitted on the next step, so T_wait < 200 ms. A segment queued behind earlier avatar speech is not a reply: it plays after that speech by design, and R4 (the right-padding rule in the revised Sec. 3.2) only moves it to the step after the preceding segment ends. The revised Sec. 4.2 states this condition at the RLC definition.

Controlled ablations of the other participant's audio and human evaluation

  1. We add the other-audio ablation in Appendix I.1. Without the other audio, the reaction to the speaker gets worse (PFD and rPCC), while variance stays almost the same (Table 8). So the other audio changes how the avatar reacts, not how much it moves.
  2. We also add side-by-side comparison videos with the baselines in the supplementary material.
  3. The paper does not claim perceptual improvements in responsiveness; it bounds and measures the system's contribution to two user-facing latencies, which are defined as event-to-event intervals. A user study is out of scope, the revised Limitations paragraph states this.

Ablation on other audio conditioning

  1. We mentioned other audio conditioning ablation in the previous section.
  2. Other components ablations fall out of scope of the paper, since motion generator is not stated as a main contribution

"Are the total latencies in Table 3 measured independently from input/output event timestamps, or reconstructed by summing the four modeled terms?"

The total is measured independently, on the wall clock: from the moment the streamer receives the reply (start marker and first fragment) or the interruption, to the moment the first or the last speech frame is sent, which is its playback time on the media timeline. The streamer records the four terms separately when the reply or the interruption arrives in the scheduler, and their sum is the prediction (Appendix H.4). The revised Table 2 (Table 3 in the submitted version) reports both.

Avatar Forcing generation limit extension

  1. We changed only the range of Avatar Forcing's cached RoPE angles, from 30 s to 600 s, so the model is mathematically equivalent: positions within 30 s get the same angles as before, and later positions get the angles from the same RoPE formula. So on sequences within the original 30 s limit, the extended model is the original model, and its results there are unchanged.
  2. We have added side-by-side comparison videos with the baselines in the supplementary material to assess the performance
Reviewer_E3cYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an audio-driven talking-head stack consisting of a motion generator, a renderer, and a streamer; and analytically derives the system's contribution to user-facing response and interruption latencies based on scheduling rules.

Critical strengths

The system engineering effort to synchronize audio and video chunks from asynchronous update is acknowledged.

Critical weaknesses
  1. Lack of Machine Learning Novelty: This paper reads like a systems engineering or queuing theory technical report, not a machine learning conference paper. Deriving basic arithmetic bounds for latency ( T_lead +​T_present +T_wait+T_pad ) based on fixed chunk sizes requires no ML innovation. It is a trivial exercise in scheduling logic.

  2. Unprofessional Presentation and Placeholder Usage: The pervasive use of the placeholder "XXXX-1" throughout the text, tables, and figures is highly unprofessional and suggests this is an unfinished draft rather than a polished ICLR submission.

  3. Absence of Evaluation on Core Interactive Claims: The paper heavily promotes its ability to handle interruptions and support live dyadic interaction, yet provides NO quantitative evaluation on these core claims. There are no metrics for interruption handling success rates, no dedicated listening behavior metrics, and no user study to validate the claimed improvements in "user-facing" responsiveness. Relying solely on offline visual quality metrics (e.g., FID, FVD) computed on ground-truth audio completely fails to validate the paper's central premise of real-time interactive streaming.

Questions for authors

See Weaknesses.

1 comment

We would like to thank reviewer E3cY for the feedback.

Lack of machine learning novelty

The paper is submitted to ICLR's "infrastructure, software libraries, hardware, systems, etc." area, and it is a systems paper by intent. We have revised the contributions to state this directly. The first contribution is the model-agnostic streamer with the first, to our knowledge, closed-form bounds on its contribution to response and interruption latency. The second is the open system, in which the motion model is a fixed workload. A systems analysis of how head avatar generators behave when being deployed is in scope for that area.

Derived latency bounds are a trivial exercise in scheduling logic

The bounds are closed-form once the decomposition is known, but the decomposition itself is not obvious. On an RTX 4090, the renderer runs 3.2× faster than real time with a 39 ms time to first frame, yet the avatar adds 265–465 ms to the response. No prior dyadic or talking-head work explains that gap. For example, DyStream reports below 100 ms for its entire system, but with its windows and our reasonable choice of TleadT_{lead}, streaming alone adds 100 to 140 ms (Sec. 6.3). Additionally, revised upper bound derivation has non-obvious step that is valid under the stated assumptions only. The resulting bounds allow to predict the latency contribution of a chunk-based head motion generator before development.

Unprofessional presentation and placeholder usage

  1. XXXX-1 replaces the system's public name for double-blind review, since the name and the released code would identify the authors. We chose the token the code anonymization service uses, so that the paper and the anonymized repository refer to the system by the same name. The revised paper explains this in a footnote at the first mention in the main text.
  2. We have added formal definitions of the terms, two algorithms, and additional figures to improve presentation

Absence of quantitative evaluation of the paper's core claims

  1. Interruption handling is evaluated quantitatively. The previous version already reported it: the ILC columns of Table 3 (Table 2 in the revision) measure the system's contribution to interruption latency for both voice agents, decomposed into its terms and checked against the bound. The revision raises this to 50 turns per agent and checks it against the corrected 265–465 ms bound. A success or failure rate does not apply to our part: detecting an interruption is the voice agent's job, and once it sends one, the streamer's handling is deterministic (Algorithm 2, lines 21–23). The only quantity that varies is how long it takes, which the ILC measures. The paper defines the ILC in Sec. 4.2 as exactly this delay: from the voice agent's interruption signal to the end of avatar speech playback. The revised Sec. 3.1 also states that interruption detection belongs to the voice agent and that the streamer acts on its signal deterministically.
  2. The previous version already had a dedicated listening-behavior comparison: rPCC, P-FD, SID, Var in Table 2 (the listening-motion columns of Table 4 in the revision). Its metrics are computed only on frames where the generated participant is silent, so they measure listening rather than speaking. The Evaluation metrics paragraph states this (Sec. 4.1 in the previous version, Sec. 7 in the revision). Appendix I gives the same metrics with a second feature set (LivePortrait) and over full sequences.
  3. The new Appendix I.1 adds an ablation with self-audio fixed and the other-audio guidance scale set to 0. Without the other audio, PFD and rPCC get worse for both pose and expression (Table 8), so the listening metrics depend on the other participant's audio.

Relying solely on offline visual quality metrics to validate the paper's central premise

  1. The central premise is validated by the latency measurements, not by FID or FVD. The previous version already reported them from live sessions with two commercial voice agents (Table 3, now Table 2 with 50 turns per agent).
  2. The quality metrics answer a separate question: whether the fixed workload (motion generator) is competitive. For XXXX-1, they are computed through the same chunked streaming path. Ground-truth audio replaces live speech because the metrics require ground-truth motion that corresponds to the recorded audio. The baselines (DyStream, Avatar Forcing) run with their released offline inference, because they don't provide a streaming implementation. The revised Sec. 7 states all of this.

Absence of user study

The paper does not claim perceptual improvements in responsiveness; it bounds and measures the system's contribution to two user-facing latencies, which are defined as event-to-event intervals. A user study is out of scope, the revised Limitations paragraph states this.

Comments on the paper

1 comment
AuthorsOpenReview ↗

New significantly revised version of the paper

We thank the reviewers for detailed and constructive feedback. It inspired us to significantly update the content of the paper. In particular, the revision makes these changes:

  1. Reframed contributions. We have rearranged the content of the paper to highlight that the main contribution is the model-agnostic streamer with the analysis of its latency contribution and the closed-form bounds. The second contribution is the end-to-end system as a whole. The renderer's internal implementation is a fixed workload for the latency analysis, and we no longer present the dyadic talking head generator as a contribution.
  2. Formal system description (Sec. 3) We have added formal definitions of speech streams, a renderer's interface and added two algorithms. Algorithm 1 (rendering loop) and Algorithm 2 (window formation). We have also included new figures Fig. 2 and Fig. 3 to make the algorithms easier to understand.
  3. Streamer and the latency bounds are now reformulated as model-agnostic. Lower bound terms have been regrouped for improved clarity. Upper bound has been tightened. We lowered TleadT_{lead} from 100 to 60 ms, which covers the measured TTFF on an RTX 4090 GPU. The bounds are now 265–465 ms for both the RLC and the ILC, in place of 305–705 ms and 305–505 ms.
  4. Additional validation (Sec. 6). Latency validation with the commercial voice agents now has 50 turns instead of 10, newly added controlled sweep of arrival patterns, and emulated renderers with the timing configurations of other renderers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.