acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Projective Guidance for Unified Video and 3D Hand Motion Generation

Abstract

Generating paired egocentric video and 3D hand motion can provide supervision for dexterous robot learning. Recent video–action models jointly generate both modalities, avoiding error propagation from sequential generation. However, they learn cross-modal correspondence implicitly, so the generated video and hand motion are often individually plausible yet spatio-temporally inconsistent. For example, given the same instruction, the 2D video hand trajectory may not align with the reprojected 3D hand motion trajectory on the image plane. To address this limitation, we introduce Uni-HaVi, a unified model for 3D hand motion and video generation. Compared with 3D hand motion generation, which gradually stabilizes in the later flow-matching steps, we observe that the hand trajectory in the generated video is largely determined within the first few steps. Exploiting this asymmetry, we propose Dynamic Projective Guidance (DPG), which uses a video hand locator and a learned adapter to progressively guide 3D hand motion generation toward the video hand trajectory throughout the flow-matching process. As a result, DPG enforces stronger spatio-temporal consistency between the video and 3D hand motion, while allowing their appearance and articulation to be independently refined during generative process. On EgoDex, DPG reduces wrist alignment error from 42.3 to 19.0 px and improves whole-hand IoU from 0.245 to 0.313, as measured by wrist and hand evaluators. By treating either modality as given or generated, Uni-HaVi supports video-to-hand-motion, hand-motion-to-video, and joint generation with a single model, and achieves strong video–hand spatio-temporal alignment and improved 3D hand-motion quality while maintaining competitive video quality.

Primary area:
generative models
Keywords:
Dynamic Projective Guidance, Unified Video-Hand Generation, 3D Hand Motion Generation, Egocentric Video Generation, Flow Matching, Cross-Modal Consistency, Geometric Guidance, Dexterous Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vv6tOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Uni-HaVi, a unified flow-matching model for egocentric video and 3D hand motion. DPG maps the wrist reprojection discrepancy to a 3D update through the camera Jacobian, and a learned adapter adjusts it before it is added to the predicted clean hand state.

Critical strengths

a. The paper turns video–hand consistency from an implicitly learned correspondence into an explicit geometric constraint inside the generative process by keeping hand motion in an explicit 3D articulated space.

b. Cross-modal consistency is measured by RGB evaluators that are independent of the model's own hand localization, at both the wrist and the whole-hand scale.

Critical weaknesses

a. The unified interface follows prior mixture-of-noise-level designs, as Sec. 2 states. Sec. 1 motivates joint generation by error propagation in two-stage pipelines, yet Table 2 reports reconstruction methods as reference points rather than matched baselines. Sec. 4.4 compares against a two-stage pipeline only on a single qualitative example. Please report this comparison quantitatively on the consistency metrics.

b. Eq. (4) adds the correction to the predicted clean hand state, which at a small noise level is already close to the output. Late guidance therefore approaches a direct edit of the final wrist, and Table 3 shows that Late DPG recovers nearly all of the gain. Please compare with a reprojection correction applied after sampling, and report the latency of the Off, Late, and Full settings against the methods in Table 1.

c. Sec. 1 argues that the video can specify the hand location early, while the 3D hand motion is still uncertain. Table 3 shows that guidance in this early stage improves wrist alignment far less than late guidance. The stated asymmetry therefore does not account for where the gain comes from.

Questions for authors

see above

5 comments

We sincerely thank the reviewer for these valuable suggestions, particularly regarding the relationship between our observation of 2D stability and the DPG ablation results. We have conducted additional experiments and address each concern below.

[Weak a] Comparison with two-stage generation.

Thank you for suggesting further evaluation beyond the qualitative example. We compare our joint generation approach with a two-stage pipeline that first generates a video and then estimates hand poses from that video. The results below indicate that estimating hand poses from generated videos remains challenging.

PipelineWrist error ↓ (px)Wrist PCK ↑ (%)Soft IoU ↑Motion-FID ↓Accel. magnitude (mm/frame²)
Joint generation + DPG Full22.1261.920.28512.7615.057
Generated video → HaMeR94.530.340.02324.01055.242
Generated video → HaWoR49.063.220.03614.85014.617

[Weak c] Concerns about the apparent conflict between DPG and our observation.

We agree that it is important to explain how our observation of early 2D stability relates to the guidance ablation results. Our observation that 2D video content stabilizes earlier motivated us to explore analytic updates and a learnable adapter for guiding hand-motion generation. However, our experiments show that applying guidance only at early sampling steps yields limited improvements.

The early availability of a stable visual target does not necessarily imply that early guidance will produce a comparably strong improvement in the final output.

To explain this behavior intuitively, we visualize hand-motion states in a two-dimensional plane using PCA (Fig. 3, right; lines 225–229). DPG modifies the flow-matching velocity at the current sample, thereby influencing subsequent generation. Specifically, each guided sampling step consists of three operations: (1) the Transformer blocks predict the clean hand state x̂₀; (2) DPG adjusts its 3D wrist position according to the 2D reprojection discrepancy, which changes the corresponding hand-motion velocity; and (3) the sampler advances to xₜ₋₁ using this modified velocity and the current sampling interval.

In the DPG design, the Transformer blocks at the following step receive the modified sampling state, but not the guidance signal as an explicit condition. They therefore infer the next clean prediction from the updated state without directly observing the correction that produced it. Our experiments show that the subsequent prediction does not necessarily preserve the preceding correction. This behavior is particularly relevant at early, high-noise sampling steps: although the video can already provide a relatively stable 2D target, the resulting correction may be attenuated by subsequent predictions. The same retention difficulty also affects purely analytic updates. Our ablation study was designed to examine this distinction between applying a correction at one step and retaining its benefit through the remaining sampling process.

To further investigate this, we conducted an exploratory experiment with token-conditioned DPG, termed DPG-T, on a relatively small dataset (400 for training, 40 for testing). Instead of applying the correction only after the Transformer prediction, DPG-T encodes the correction information into a token-space conditioning signal and injects it into the hand tokens before the conditioned Transformer forward pass, as illustrated in the right panel of Fig. 4 (lines 290–292, 468–474). At each guided sampling step, an initial prediction is used to construct the correction condition, after which a conditioned prediction determines the sampling update. The Transformer therefore receives explicit guidance information rather than only an implicitly modified sampling state.

Information flow within one guided sampling step (Fig. 4 in the revised manuscript, p. 6, lines 290–292):

Analytic                 DPG                      DPG-T (exploratory)

Current state x_t        Current state x_t        Current state x_t
        ↓                        ↓                        ↓
DiT prediction           DiT prediction           Initial DiT prediction
        ↓                        ↓                        ↓
Analytic correction      Analytic correction      Encode correction
     α Δw_J                    Δw_J                       ↓
        ↓                        ↓                Inject into hand tokens
        │                Learned adapter          of the same x_t
        │                        ↓                        ↓
Corrected prediction     Corrected prediction     Conditioned prediction
        ↓                        ↓                        ↓
Sampling update          Sampling update          Sampling update
        ↓                        ↓                        ↓
     x_{t−1}                  x_{t−1}                  x_{t−1}

Taken together, these results reconcile our observation with the ablation findings: a stable 2D target can provide guidance early in sampling, but the final benefit also depends on how effectively subsequent predictions retain the correction. The DPG-T results (Table 3, lines 445–453) suggest that explicitly conditioning the Transformer on correction information can help preserve its effect, allowing early-stage guidance to benefit the final output even after guidance is disabled.

DPG-T settingGT wrist reprojection error ↓ (px)Absolute MPJPE ↓ (mm)
Off38.39946.712
Early: steps 10–1925.78337.234
Sustained: steps 10–4921.18633.258

[Weak b] Comparison of DPG with post-hoc correction.

Thank you for your questions and suggestions regarding the DPG ablations. We agree that it is important to compare DPG with direct post-hoc correction. Post-hoc correction adjusts the final 3D wrist translations toward the 2D Locator targets without further generative sampling. The results below highlight a trade-off: post-hoc correction provides stronger 2D wrist-level alignment, whereas in-sampling DPG yields better finger and full-hand consistency with lower frame-to-frame acceleration magnitude. Lower acceleration magnitude alone does not establish better overall motion quality.

Table. In-sampling DPG versus post-hoc correction.

MethodWrist error ↓ (px)Wrist PCK ↑ (%)Soft IoU ↑Finger error ↓ (norm.)Full Hand IoU ↑Accel. magnitude (mm/frame²)
DPG Off37.8432.790.1520.37810.24404.823
DPG Off + post-hoc correction21.8567.080.3100.37850.26587.170
DPG Full22.1261.920.2850.32150.29695.057

[Weak b] Inference latency.

Thank you for this suggestion. We agree that inference latency is also important for assessing the practical cost of incorporating guidance into flow-matching transformers. We measure inference latency on a single NVIDIA A100 80GB GPU with batch size 1, generating 61 frames at 832 × 480 resolution with 50 sampling steps. The results show that Full DPG adds 0.56 seconds to sampling and decoding compared with DPG Off; Late DPG adds 0.13 seconds. These results show that DPG introduces only a small computational overhead relative to the underlying generative model (Wan2.2 in our experiments).

We additionally report the Table 1 baseline implementations on the same GPU and input clips. All entries generate 61 frames at 832 × 480 resolution with batch size 1. We report mean ± sample standard deviation over five clips after excluding one warm-up clip. Timing covers sampling and RGB decoding, excluding model loading and output-file writing; it is not end-to-end application latency. Donk, HandWorld, and our model use 50 sequence-level sampling steps. The adapted DreamZero implementation uses 15 autoregressive chunks with 50 denoising steps per chunk and does not use an optimized KV cache. These measurements characterize the implementations evaluated in our experiments, rather than the fastest available implementations of each method.

Table. Inference latency of Table 1 baselines and different DPG configurations. (line 428; Table 4, lines 445–455)

ConfigurationSampling ↓ (s)Sampling + decoding ↓ (s)
Donk22.62 ± 0.0328.21 ± 0.06
DreamZero (adapted)476.29 ± 0.34481.91 ± 0.36
HandWorld77.00 ± 0.0782.71 ± 0.11
DPG Off23.26 ± 0.0128.82 ± 0.03
Late DPG23.38 ± 0.0228.95 ± 0.04
Full DPG23.82 ± 0.0129.38 ± 0.02
Reviewer_46zhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Uni-HaVi, a flow-matching model for joint generation of egocentric video and 3D bimanual motion, as well as conditional generation in either direction. Its main contribution is Dynamic Projective Guidance (DPG), which locates 2D wrists in the evolving video and uses their reprojection discrepancy with the predicted 3D wrists, together with the camera projection Jacobian and a learned adapter, to adjust the 3D trajectories during sampling. On EgoDex, adding DPG reduces wrist alignment error from 42.32 to 19.02 pixels and increases hand-mask IoU from 0.245 to 0.313; the paper reports corresponding alignment gains on HOI4D.

Critical strengths

The most convincing result is the improvement in spatial consistency between the video and 3D hand motion. Table 1 shows substantial gains on both datasets, measured with an independent RGB-based wrist evaluator and whole-hand mask overlap. More importantly, the ablation in Table 3 holds the model weights, camera inputs, and initial noise fixed while varying the guidance. This makes the evidence for DPG more persuasive than a comparison across differently sized models or sampling conditions, and is the main reason I lean toward acceptance.

The unified formulation also has practical value. Section 3.1 and Table 2 cover joint generation as well as video-to-hand and hand-to-video generation within one model. This broader task coverage is useful for applications that require paired video and motion data.

Critical weaknesses

The evidence for image-plane alignment is stronger than the evidence for improved 3D motion quality or downstream robotic benefit. The paper does report 3D metrics, including Motion-FID and MPJPE, but these results do not clearly isolate DPG's effect on 3D accuracy. A 2D wrist location cannot resolve depth. On EgoDex, Motion-FID increases slightly from 3.40 to 3.54 after adding DPG (lower is better). The authors also note that the video-to-hand reconstruction comparisons in Table 2 do not use strictly matched training settings. The pouring experiment in Section 4.4 is a useful feasibility demonstration, but it is a single case. I therefore find the spatial-alignment claim well supported, while the broader claims about 3D accuracy and robotic learning would benefit from more careful qualification.

The contribution of the learned adapter relative to a well-tuned analytic geometric update remains insufficiently isolated. Table 3 shows that DPG outperforms the direct Jacobian update considered in the paper, but it is unclear whether the gap would persist with a tuned update scale, damping, or an analytic update applied only at late sampling steps.

There are also missing important related references, such as: SViMo: Synchronized Diffusion for Video and Motion Generation in Hand-object Interaction Scenarios, NeurIPS 2025

Questions for authors

With model weights and sampling seeds held fixed, what are the absolute MPJPE, PA-MPJPE and depth error for video-to-hand generation with DPG enabled versus disabled?

How does the analytic guidance in Table 3 perform after tuning the update scale or adding damping, or when applied only during late sampling steps? What happens when the adapter's reprojection-residual or noise-level input is removed?

How many independently generated trajectories were tested in the pouring experiment? Were other objects or tasks evaluated, and what were the simulation and real-robot success rates for each method?

3 comments

We sincerely thank the reviewer for recognizing our DPG ablation design. We also appreciate your constructive suggestions and address each concern below.

[Weak1, Q1] 3D accuracy with DPG.

We agree that additional experiments examining whether DPG improves 3D accuracy would be valuable. Motion-FID measures the distributional discrepancy between generated hand motions and motions in the test set; small differences in this metric do not directly establish differences in 3D accuracy. We therefore appreciate your suggestion to evaluate absolute MPJPE and PA-MPJPE for video-to-hand generation with and without DPG. We report this comparison in the table below. The results show that DPG also improves the 3D accuracy of the generated hand motion, providing more direct evidence.

DPGMPJPE ↓ (mm)PA-MPJPE ↓ (mm)Wrist depth MAE ↓ (mm)
Off38.7188.25828.646
Full27.8094.15715.600

[Weak2, Q2] Additional ablations of DPG and the analytic update.

We agree that additional ablations are important for understanding the contribution of each design choice. We have added the ablations below to examine the effects of separately removing the reprojection-residual or noise-level input to DPG, as well as different scales for the analytic update. All variants in the table below are evaluated using the same sampling seed and test set.

VariantWrist error ↓ (px)Wrist PCK ↑ (%)Soft IoU ↑Motion-FID ↓
DPG Full22.1261.920.28512.761
w/o reprojection-residual input22.8761.130.27212.552
w/o noise-level input27.7946.900.21612.442
DPG Off37.8432.790.15212.182
Analytic update, α = 0.2538.4033.780.14812.283
Analytic update, α = 0.538.4633.830.14912.313
Analytic update, α = 2.038.1534.290.15112.294
Analytic update, late-only34.7538.470.18812.336

The results suggest that noise-level conditioning is important for alignment. Removing the reprojection-residual input has a smaller effect. One possible explanation is that the analytic 3D update already conveys related correction information. Adjusting the analytic-update scale alone yields limited gains and, in some settings, slightly increases wrist reprojection error relative to DPG Off, supporting our use of a learned adapter rather than scale tuning alone.

[Q3] Real-world robotic evaluation.

Thank you for your suggestion. We agree that reporting success rates is also important. We evaluated 20 independently generated trajectories for each of two tasks—pouring and sweeping into a dustpan—in real-world robot experiments. A sweeping example in simulation and on a real robot is shown in Appendix G (Fig. 13). The success rates (SR) are reported below.

TaskSuccess rate (SR)
Pouring70% (14/20)
Sweeping into a dustpan55% (11/20)

[Weak3] Clarifications.

Thank you for your suggestions on improving the clarity of the manuscript. We have added a reference to SViMo in the revised manuscript (line 49). Among the public implementations used in our comparison, only HaMeR provided a fine-tuning interface; we therefore fine-tuned HaMeR and have clarified this setting in the revised manuscript (lines 418–420). The other methods were evaluated using their released pretrained checkpoints without additional fine-tuning.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end with less than one day remaining, we wanted to ensure that we have addressed all of your concerns satisfactorily. If there are any additional points or feedback you would like us to consider, please let us know. Your insights are invaluable to us, and we are eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Thank you for the responses, which have addressed most of my concerns. I will maintain my score.

Reviewer_RFoJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Uni-HaVi, a unified model for generating egocentric video and 3D hand motion. It addresses the problem that the generated video and hand motion can each look reasonable but may not align with each other. Based on the observation that the hand trajectory in the video becomes stable earlier during generation, the authors propose Dynamic Projective Guidance (DPG). It uses 2D wrist locations from the video, together with geometric correction and a learned adapter, to guide the 3D hand motion. The same model supports joint generation, video-to-hand-motion, and hand-motion-to-video generation. Experiments on EgoDex and HOI4D show clear improvements in video–hand alignment while maintaining competitive video quality.

Critical strengths
  • The paper addresses a concrete limitation of joint video and 3D hand motion generation: the generated video and hand motion can each look reasonable but do not align well with each other.
  • The observation that the 2D video hand trajectory becomes stable within the first few flow-matching steps is a nice finding. It also gives a clear motivation for the proposed guidance.
  • The improvement in wrist alignment is clear: 42.32 → 19.02 px on EgoDex and 33.92 → 21.04 px on HOI4D. These are good improvements on both datasets.
  • It is useful that a single model supports joint generation, video → hand motion, and hand motion → video. This makes the method flexible for different use cases.
Critical weaknesses
  • The finding of early stabilization of 2D flow-matching is interesting, but I would like to see more systematic analysis to support it. Table 3 provides some evidence, but it is unclear how consistent this observation is across different examples and datasets.
  • The guidance mainly aligns the hands in 2D images. How much does this improve the actual 3D alignment, especially in depth? The paper mentions the depth ambiguity, but needs more direct evidence of the 3D alignment.
  • The method section has too many equations and is a bit hard to follow. A more intuitive explanation of the main steps and why each component is needed would help readers understand the method.
  • For video → hand motion, the statement that “not all methods can be retrained under the same EgoDex setting” needs more explanation. Which methods were retrained, and which were not?
  • The robotics experiment is a useful example, but it only shows a pouring task. More tasks or quantitative success rates are needed.
Questions for authors

see weakness

3 comments

We sincerely thank the reviewer for recognizing our observations and motivation. We also appreciate your constructive suggestions and address each concern below.

[Weak1] Analysis of early stabilization during flow-matching sampling.

We agree that this observation warrants a more systematic analysis. The middle panel of Table 5 illustrates it across different examples. It shows that, during flow-matching sampling, the projected 3D wrist positions can remain far from the wrist locations in the video, as measured by pixel distance, whereas the 2D Locator identifies the wrist locations at an early sampling step. Following your suggestion for a more systematic analysis, we have added a sentence explaining this figure in the revised manuscript (lines 462–466). We also report the mean pixel distances across a mixed set of HOI4D and EgoDex samples in the table below.

Sampling stepProjected 3D wrist error (px)2D Locator wrist error (px)
0817.15 ± 502.2924.94 ± 24.59
5789.02 ± 470.7822.77 ± 32.72
10763.29 ± 441.9320.36 ± 29.14
20701.68 ± 374.0419.43 ± 27.39
30621.58 ± 295.0318.87 ± 29.43
40483.51 ± 182.5618.76 ± 26.99
45357.34 ± 108.4215.19 ± 12.61
46319.47 ± 92.0915.00 ± 11.84
47274.58 ± 77.1317.64 ± 18.51
48217.98 ± 65.4317.54 ± 16.02
49141.80 ± 64.2517.25 ± 13.18
Final50.07 ± 20.87—

[Weak2] 3D alignment with DPG.

We agree that evaluating 3D alignment beyond image-plane consistency is important. For joint generation, paired ground truth is unavailable for evaluating the 3D alignment of the generated outputs. We therefore evaluate 3D alignment in the video-to-hand setting, where paired ground truth is available, and report MPJPE, PA-MPJPE, and wrist depth MAE in the table below. The results show that DPG improves the 3D alignment of the generated hand motion, providing more direct evidence.

DPGMPJPE ↓ (mm)PA-MPJPE ↓ (mm)Wrist depth MAE ↓ (mm)
Off38.7188.25828.646
Full27.8094.15715.600

[Weak3 & Weak4] Clearer presentation and more intuitive figures.

Thank you for your suggestions on improving the clarity of the manuscript. To help readers follow the steps of our method, the right panel of Fig. 3 illustrates the sampling update (lines 225–229), and we have added an intuitive visualization (Fig. 4) and analysis in the ablation study, comparing the guidance information flow of analytic guidance, DPG, and DPG-T to help readers understand these designs and their ablation results (lines 290–292, 468–474, 486–487). Among the public implementations used in our comparison, only HaMeR provided a fine-tuning interface; we therefore fine-tuned HaMeR and have clarified this setting in the revised manuscript (lines 418–420).

[Weak5] Real-world robotic evaluation.

Thank you for your suggestion. We agree that reporting success rates is also important. We evaluated 20 independently generated trajectories for each of two tasks—pouring and sweeping into a dustpan—in real-world robot experiments. A sweeping example in simulation and on a real robot is shown in Appendix G (Fig. 13). The success rates (SR) are reported below.

TaskSuccess rate (SR)
Pouring70% (14/20)
Sweeping into a dustpan55% (11/20)

Thank you for the detailed responses. The authors have addressed most of my concerns. I will maintain my positive score.

Dear Reviewer,

Thank you for your kind response and for your continued support of our work. We sincerely appreciate your insightful comments, which have greatly helped us improve the paper. We will make sure to incorporate all the discussed revisions into the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.