- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Uni-HaVi, a unified flow-matching model for egocentric video and 3D hand motion. DPG maps the wrist reprojection discrepancy to a 3D update through the camera Jacobian, and a learned adapter adjusts it before it is added to the predicted clean hand state.
Critical strengths
a. The paper turns video–hand consistency from an implicitly learned correspondence into an explicit geometric constraint inside the generative process by keeping hand motion in an explicit 3D articulated space.
b. Cross-modal consistency is measured by RGB evaluators that are independent of the model's own hand localization, at both the wrist and the whole-hand scale.
Critical weaknesses
a. The unified interface follows prior mixture-of-noise-level designs, as Sec. 2 states. Sec. 1 motivates joint generation by error propagation in two-stage pipelines, yet Table 2 reports reconstruction methods as reference points rather than matched baselines. Sec. 4.4 compares against a two-stage pipeline only on a single qualitative example. Please report this comparison quantitatively on the consistency metrics.
b. Eq. (4) adds the correction to the predicted clean hand state, which at a small noise level is already close to the output. Late guidance therefore approaches a direct edit of the final wrist, and Table 3 shows that Late DPG recovers nearly all of the gain. Please compare with a reprojection correction applied after sampling, and report the latency of the Off, Late, and Full settings against the methods in Table 1.
c. Sec. 1 argues that the video can specify the hand location early, while the 3D hand motion is still uncertain. Table 3 shows that guidance in this early stage improves wrist alignment far less than late guidance. The stated asymmetry therefore does not account for where the gain comes from.
Questions for authors
see above
5 comments
[Weak c] Concerns about the apparent conflict between DPG and our observation.
We agree that it is important to explain how our observation of early 2D stability relates to the guidance ablation results. Our observation that 2D video content stabilizes earlier motivated us to explore analytic updates and a learnable adapter for guiding hand-motion generation. However, our experiments show that applying guidance only at early sampling steps yields limited improvements.
The early availability of a stable visual target does not necessarily imply that early guidance will produce a comparably strong improvement in the final output.
To explain this behavior intuitively, we visualize hand-motion states in a two-dimensional plane using PCA (Fig. 3, right; lines 225–229). DPG modifies the flow-matching velocity at the current sample, thereby influencing subsequent generation. Specifically, each guided sampling step consists of three operations: (1) the Transformer blocks predict the clean hand state x̂₀; (2) DPG adjusts its 3D wrist position according to the 2D reprojection discrepancy, which changes the corresponding hand-motion velocity; and (3) the sampler advances to xₜ₋₁ using this modified velocity and the current sampling interval.
In the DPG design, the Transformer blocks at the following step receive the modified sampling state, but not the guidance signal as an explicit condition. They therefore infer the next clean prediction from the updated state without directly observing the correction that produced it. Our experiments show that the subsequent prediction does not necessarily preserve the preceding correction. This behavior is particularly relevant at early, high-noise sampling steps: although the video can already provide a relatively stable 2D target, the resulting correction may be attenuated by subsequent predictions. The same retention difficulty also affects purely analytic updates. Our ablation study was designed to examine this distinction between applying a correction at one step and retaining its benefit through the remaining sampling process.
To further investigate this, we conducted an exploratory experiment with token-conditioned DPG, termed DPG-T, on a relatively small dataset (400 for training, 40 for testing). Instead of applying the correction only after the Transformer prediction, DPG-T encodes the correction information into a token-space conditioning signal and injects it into the hand tokens before the conditioned Transformer forward pass, as illustrated in the right panel of Fig. 4 (lines 290–292, 468–474). At each guided sampling step, an initial prediction is used to construct the correction condition, after which a conditioned prediction determines the sampling update. The Transformer therefore receives explicit guidance information rather than only an implicitly modified sampling state.
Information flow within one guided sampling step (Fig. 4 in the revised manuscript, p. 6, lines 290–292):
Analytic DPG DPG-T (exploratory)
Current state x_t Current state x_t Current state x_t
↓ ↓ ↓
DiT prediction DiT prediction Initial DiT prediction
↓ ↓ ↓
Analytic correction Analytic correction Encode correction
α Δw_J Δw_J ↓
↓ ↓ Inject into hand tokens
│ Learned adapter of the same x_t
│ ↓ ↓
Corrected prediction Corrected prediction Conditioned prediction
↓ ↓ ↓
Sampling update Sampling update Sampling update
↓ ↓ ↓
x_{t−1} x_{t−1} x_{t−1}
Taken together, these results reconcile our observation with the ablation findings: a stable 2D target can provide guidance early in sampling, but the final benefit also depends on how effectively subsequent predictions retain the correction. The DPG-T results (Table 3, lines 445–453) suggest that explicitly conditioning the Transformer on correction information can help preserve its effect, allowing early-stage guidance to benefit the final output even after guidance is disabled.
| DPG-T setting | GT wrist reprojection error ↓ (px) | Absolute MPJPE ↓ (mm) |
|---|---|---|
| Off | 38.399 | 46.712 |
| Early: steps 10–19 | 25.783 | 37.234 |
| Sustained: steps 10–49 | 21.186 | 33.258 |
[Weak b] Comparison of DPG with post-hoc correction.
Thank you for your questions and suggestions regarding the DPG ablations. We agree that it is important to compare DPG with direct post-hoc correction. Post-hoc correction adjusts the final 3D wrist translations toward the 2D Locator targets without further generative sampling. The results below highlight a trade-off: post-hoc correction provides stronger 2D wrist-level alignment, whereas in-sampling DPG yields better finger and full-hand consistency with lower frame-to-frame acceleration magnitude. Lower acceleration magnitude alone does not establish better overall motion quality.
Table. In-sampling DPG versus post-hoc correction.
| Method | Wrist error ↓ (px) | Wrist PCK ↑ (%) | Soft IoU ↑ | Finger error ↓ (norm.) | Full Hand IoU ↑ | Accel. magnitude (mm/frame²) |
|---|---|---|---|---|---|---|
| DPG Off | 37.84 | 32.79 | 0.152 | 0.3781 | 0.2440 | 4.823 |
| DPG Off + post-hoc correction | 21.85 | 67.08 | 0.310 | 0.3785 | 0.2658 | 7.170 |
| DPG Full | 22.12 | 61.92 | 0.285 | 0.3215 | 0.2969 | 5.057 |
[Weak b] Inference latency.
Thank you for this suggestion. We agree that inference latency is also important for assessing the practical cost of incorporating guidance into flow-matching transformers. We measure inference latency on a single NVIDIA A100 80GB GPU with batch size 1, generating 61 frames at 832 × 480 resolution with 50 sampling steps. The results show that Full DPG adds 0.56 seconds to sampling and decoding compared with DPG Off; Late DPG adds 0.13 seconds. These results show that DPG introduces only a small computational overhead relative to the underlying generative model (Wan2.2 in our experiments).
We additionally report the Table 1 baseline implementations on the same GPU and input clips. All entries generate 61 frames at 832 × 480 resolution with batch size 1. We report mean ± sample standard deviation over five clips after excluding one warm-up clip. Timing covers sampling and RGB decoding, excluding model loading and output-file writing; it is not end-to-end application latency. Donk, HandWorld, and our model use 50 sequence-level sampling steps. The adapted DreamZero implementation uses 15 autoregressive chunks with 50 denoising steps per chunk and does not use an optimized KV cache. These measurements characterize the implementations evaluated in our experiments, rather than the fastest available implementations of each method.
Table. Inference latency of Table 1 baselines and different DPG configurations. (line 428; Table 4, lines 445–455)
| Configuration | Sampling ↓ (s) | Sampling + decoding ↓ (s) |
|---|---|---|
| Donk | 22.62 ± 0.03 | 28.21 ± 0.06 |
| DreamZero (adapted) | 476.29 ± 0.34 | 481.91 ± 0.36 |
| HandWorld | 77.00 ± 0.07 | 82.71 ± 0.11 |
| DPG Off | 23.26 ± 0.01 | 28.82 ± 0.03 |
| Late DPG | 23.38 ± 0.02 | 28.95 ± 0.04 |
| Full DPG | 23.82 ± 0.01 | 29.38 ± 0.02 |
We sincerely thank the reviewer for these valuable suggestions, particularly regarding the relationship between our observation of 2D stability and the DPG ablation results. We have conducted additional experiments and address each concern below.
[Weak a] Comparison with two-stage generation.
Thank you for suggesting further evaluation beyond the qualitative example. We compare our joint generation approach with a two-stage pipeline that first generates a video and then estimates hand poses from that video. The results below indicate that estimating hand poses from generated videos remains challenging.