- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Routed Forcing, a training strategy for addressing the loss of motion and diversity in distilled streaming avatars. The authors observe that diversity loss is concentrated in the person region, while the mouth and background are less affected. The method applies Data-Forcing Distillation to the non-mouth person region at high teacher query noise levels and retains DMD elsewhere. Experiments on SpeakerVid and AVSpeech report improvements in motion, diversity, and visual quality while retaining the inference efficiency of the original student model.
Critical strengths
- Separating diversity loss by person, mouth, and background provides more insight than using a single score for the whole frame. This suggests a practical reason to handle these regions differently during distillation.
- The experiments feature controlled comparisons. Results show improvements across all four metric groups on both datasets. Component and noise threshold ablations further demonstrate the effectiveness of the proposed training rule in this setting.
Critical weaknesses
- The methodological contribution beyond DFD is limited. DFD already uses real video supervision and combines its updates with DMD. Routed Forcing retains these updates but replaces random mixing with a predefined rule based on semantic masks and a fixed noise threshold. While the regional observation is interesting, the method remains a targeted refinement of an existing training strategy rather than a significant advancement in distillation.
- The evaluation does not demonstrate that increased variation corresponds to meaningful motion diversity, and current metrics may increase due to artifacts rather than plausible gestures.
- FID and FVD measure quality, but direct evaluations of motion naturalness, temporal stability, and identity consistency are also needed.
- The reference point for diversity recovery is unclear. The method starts from LiveTalk, applies 1,000 additional DMD training steps, then adds 200 Routed Forcing steps. However, Figure 6 shows that extra DMD training reduces diversity. Table 2 omits diversity measurements for the original LiveTalk checkpoint and instead reports results for OmniAvatar-1.3B rather than the actual OmniAvatar-14B teacher model. These omissions make it difficult to determine whether the method truly improves upon the original balance between quality and diversity or merely recovers losses caused by intermediate training stages. Both reference models should be evaluated using the same diversity protocol.
- The causal explanation outweighs its supporting analysis. The introduction presents static collapse as a direct result of reverse KL mode seeking. However, selecting fewer modes does not necessarily mean choosing less dynamic ones. Appendix A provides a conditional explanation assuming that motion occupies directions of high conditional variance. But none of the above assumptions are verified in actual video representations.
Questions for authors
- What evidence demonstrates that semantic routing offers benefits beyond simply reducing overall DFD influence? Could the authors compare it to random masks of equal area and a global DFD baseline with matched update strength? These controls would clarify whether selecting specific regions provides advantages beyond just applying less DFD supervision.
- How does Routed Forcing compare to the original LiveTalk checkpoint and the OmniAvatar-14B teacher under the same quality and diversity evaluation protocol? Figure 6 shows that an additional 1,000 DMD steps reduce diversity.
- Does the proposed method enhance the balance between quality and diversity compared to the original checkpoint, or does it mainly recover losses caused by this intermediate training stage?
- How is spatial routing managed when a hand or body part in the real video aligns with background in the student rollout? Person masks are extracted from real videos, but the reported aggregate person IoU does not address correspondence at hands and moving boundaries. Could the authors provide alignment metrics for these regions and compare them to person masks extracted from student rollouts?
8 comments
Q3: Does the proposed method enhance the balance between quality and diversity compared to the original checkpoint, or does it mainly recover losses caused by this intermediate training stage?
A3: Routed Forcing improves the quality–diversity balance of the original checkpoint, rather than merely recovering losses caused by the intermediate training stage. As shown in Table 2, compared with the original LiveTalk, Routed Forcing achieves better visual quality, lip synchronization, and motion dynamics while maintaining comparable overall diversity. In addition, Table 1 of the main paper shows that Routed Forcing outperforms Self Forcing, which uses only DMD, on every evaluation metric. Together, these two comparisons support the improvement in the quality–diversity balance achieved by Routed Forcing.
Q4: How is spatial routing managed when a hand or body part in the real video aligns with background in the student rollout? The reported aggregate person IoU does not address correspondence at hands and moving boundaries. Could the authors provide alignment metrics for these regions and compare them to person masks extracted from student rollouts?
A4: We measured the coverage of hand regions in student-generated videos by the GT person mask. We approximated the generated hand regions using the convex hulls of DWPose hand keypoints, and computed the fraction of hand pixels covered by the GT person mask. Averaging across different input conditions and frames gives 80.34%. This indicates high coverage of the hands by the GT person mask.
In addition, we replaced the person mask with a version extracted from student rollouts and retrained RF while keeping all other settings unchanged. The results are shown below:
| Person Mask Source | FID↓ | FVD↓ | Sync-C↑ | Sync-D↓ | RAFT↑ | DINO-T↑ | CLIP↑ | DINO↑ | LPIPS↑ | L1↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| Real video (ours) | 25.43 | 309.92 | 4.07 | 10.94 | 7.80 | 1.25 | 3.55 | 1.58 | 6.09 | 2.29 |
| Student rollout | 25.72 | 309.88 | 4.08 | 10.95 | 7.69 | 1.27 | 3.50 | 1.60 | 6.02 | 2.26 |
Table 3. Ablation on the source of the person mask.
The results show that the two mask sources yield very similar results, with relative differences of no more than 1.6% on any metric.
We will include the hand coverage results and the person-mask source ablation in the final version.
W1: The methodological contribution beyond DFD is limited: Routed Forcing retains the DFD and DMD updates and replaces random mixing with a predefined rule based on semantic masks and a fixed noise threshold.
Response: We identify and validate allocation rules specifying at which noise stages and in which spatial regions real-data supervision should be applied. DMD and DFD provide two supervision objectives, but the objectives themselves do not determine how supervision should be allocated. Our analysis and controlled experiments show that, even when the amount of DFD and the training budget are held fixed, different temporal and spatial allocations can produce different, or even opposite, effects.
(1) When to apply it: With the same amount of DFD, high- and low-noise allocation produce different effects.
We fix the amount of DFD and change only the noise stage at which it is applied: random allocation (the original DFD recipe), the high-noise stage (), and the low-noise stage (). The three variants start from the same checkpoint and have the same training budget.
| Method | FID↓ | FVD↓ | Sync-C↑ | Sync-D↓ | RAFT↑ | DINO-T↑ | CLIP↑ | DINO↑ | LPIPS↑ | L1↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| Self Forcing (without DFD) | 27.17 | 320.91 | 4.03 | 10.97 | 5.62 | 1.07 | 3.07 | 1.37 | 4.96 | 2.14 |
| Random allocation (original DFD recipe) | 28.78 | 325.33 | 3.92 | 11.10 | 5.47 | 1.11 | 3.08 | 1.42 | 4.67 | 1.84 |
| Low-noise stage | 31.33 | 357.17 | 3.82 | 11.11 | 4.43 | 1.12 | 2.98 | 1.43 | 4.13 | 1.65 |
| High-noise stage | 26.61 | 311.85 | 4.01 | 11.02 | 7.50 | 1.22 | 3.56 | 1.60 | 5.92 | 2.22 |
Table 4. Noise-stage allocation comparison. All three DFD allocation variants apply supervision to the full frame, with approximately 50% of training updates allocated to DFD. Bold indicates the best result among the three DFD allocation variants.
High-noise allocation outperforms both random and low-noise allocation on all 10 metrics. Compared with random allocation, RAFT improves by 37.1% (5.47 → 7.50), LPIPS improves by 26.8% (4.67 → 5.92), and FID decreases from 28.78 to 26.61.
In contrast, low-noise allocation performs worse than Self Forcing without DFD on 8 of the 10 metrics, including visual quality, RAFT, and pixel-level diversity. This shows that introducing the same amount of real-data supervision does not necessarily yield improvements. The noise stage at which it is applied is critical.
(2) Where to apply it: With the same amount of DFD, region selection still provides additional benefits.
As shown in Table 1 in A1, with the same noise routing and total amount of DFD, the non-mouth human semantic mask achieves higher motion dynamics and diversity than uniform allocation and the random mask, as well as better visual quality and lip synchronization. Therefore, introducing the same amount of real-data supervision does not necessarily yield the same benefits. The spatial region to which it is applied is critical.
(3) The two allocation rules jointly improve quality, motion dynamics, and diversity.
Table 1 of the main paper shows the overall effects of combining the two rules. The original DFD recipe selects DMD and DFD with equal probability and improves over Self Forcing on only 8 of the 20 metrics across the two datasets. Routed Forcing, in contrast, improves all 20 metrics. On SpeakerVid / AVSpeech, RAFT improves by 38.8% / 45.6% and LPIPS-div improves by 22.8% / 24.5%, respectively, while visual quality improves and lip synchronization is maintained, with an absolute Sync-C increase of 0.04 on both datasets.
Therefore, our technical contribution consists of temporal and spatial supervision allocation rules supported by analysis and controlled experiments, with spatial masks and a timestep threshold serving as a simple implementation of these rules.
We will describe these two allocation rules and their contributions more explicitly in Section 1 of the final version and include the controlled experiments above.
Thank you for your valuable comments and for recognizing the value of our regional diversity analysis and the experimental support for our training rules from controlled comparisons and ablations on two datasets. We hope the following discussion can address your concerns!
A1: Following your suggestion, we added two controls. They use the same noise routing (Top-50%) and the same total amount of DFD as RF, changing only how DFD is applied spatially:
Table 1. Spatial allocation comparison.
The results show that the benefits of semantic routing do not come from using less DFD: with the same amount, location determines the effect. Compared with uniform allocation and the random mask, the semantic mask yields clearly higher motion dynamics and diversity (RAFT: 6.75 / 6.72 → 7.80, +15.6% / +16.1%, LPIPS: 5.63 / 5.61 → 6.09, +8.2% / +8.6%), as well as better visual quality and lip synchronization. Uniform allocation and the random mask produce almost identical results (RAFT: 6.75 vs. 6.72). With the same amount of DFD, uniform distribution and random placement perform similarly when locations are not selected based on semantics, whereas allocating DFD to human regions produces a clear improvement. This is consistent with the regional heterogeneity observed in Figure 1b of the main paper: diversity collapse is concentrated in non-mouth human regions, the mouth is constrained by audio, and the background is largely static. Applying DFD to human regions therefore substantially improves motion dynamics and diversity, while avoiding its application to the mouth prevents interference with lip synchronization.
We will include these spatial allocation controls in the final version.
A2: We compared Routed Forcing, the original LiveTalk checkpoint, and the OmniAvatar-14B teacher under the same evaluation protocol. The results are shown in Table 2.
Table 2. Comparison of visual quality, lip synchronization, motion dynamics, and diversity among Routed Forcing, the original LiveTalk, and the OmniAvatar-14B teacher.
Routed Forcing outperforms both the original LiveTalk and the OmniAvatar-14B teacher on all visual quality, lip synchronization, and motion dynamics metrics, while maintaining overall diversity comparable to the original LiveTalk. The OmniAvatar-14B teacher achieves the highest diversity scores. Its 14B parameter count is substantially larger than Routed Forcing's 1.3B, and its greater model capacity allows it to accommodate a richer set of generation modes.
We will include this comparison and the related analysis in the final version of the paper.