acceptodds
Under review as a conference paper at ICLR 2027

Video-Conditioned Generative Joint 2D-3D Hand Motion Recovery

Abstract

Recovering faithful 3D hand motion from video remains challenging due to frequent occlusions and incomplete visual observations, which make frame-wise pose estimates unreliable and temporally inconsistent. To address this problem, we propose JoHan, a unified generative framework that recovers hand motion directly from video sequences without relying on intermediate per-frame pose predictions. Trained from scratch, our model jointly generates aligned 2D and 3D local hand pose sequences by learning their temporal dynamics and cross-representation correspondence. The generated 2D trajectories exploit direct spatial and temporal cues from the 2D images to guide the following generative 3D motion reconstruction, while the learned motion prior promotes temporal consistency. Their learned 2D–3D correspondence further enables recovery of the hand’s global position and orientation relative to the camera. Extensive experiments on challenging benchmarks demonstrate significantly improved accuracy and speed in local hand-pose and camera-space reconstruction. Notably, our method captures much better hand-motion dynamics, producing significantly smoother motion than previous methods while maintaining high per-frame pose accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Vision, Hand Motion Estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eySWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes JoHan, a video-conditioned generative framework for joint 2D and 3D hand motion recovery. Unlike existing approaches that depend on intermediate frame-wise pose estimates, JoHan directly generates aligned 2D and 3D local hand motion sequences from monocular videos using flow matching. The method introduces a 2D-guided fusion mechanism, where evolving 2D motion predictions guide 3D motion generation, together with reprojection supervision to encourage geometric consistency. Global hand orientation and translation are subsequently recovered using 2D–3D correspondences and a PnP solver.

Experiments on HO3Dv2, HO3Dv3, and HOT3D demonstrate competitive local hand reconstruction accuracy, improved temporal smoothness, and efficient inference. Ablation studies further support the benefits of joint 2D–3D generation and geometric supervision.

Critical strengths
  1. Meaningful joint modeling of 2D and 3D hand motion. The paper presents an interesting formulation that jointly generates temporally aligned 2D and 3D motion sequences without relying on externally estimated frame-wise hand poses as generation conditions. The 2D-guided fusion mechanism provides a reasonable way to incorporate image-space evidence into 3D motion reconstruction. This is a useful direction for addressing occlusion and temporal inconsistency in monocular hand reconstruction.

  2. Competitive reconstruction performance and computational efficiency. The experiments demonstrate strong local hand reconstruction accuracy and improved temporal smoothness. In particular, JoHan achieves a PA-MPJPE of 4.75 mm and an acceleration error of 2.69 m/s² on HOT3D. Table 3 also reports substantially faster hand-motion recovery than the compared baselines. The ablations provide useful evidence that joint generation and reprojection supervision contribute to reconstruction accuracy.

Critical weaknesses
  1. The methodological novelty of joint 2D–3D generation needs stronger justification. The central contribution is the joint generation of 2D and 3D hand motion through coupled flow-matching branches. However, the framework largely builds upon established components, including flow matching, spatiotemporal attention, 2D-guided conditioning, and reprojection supervision. Existing methods such as HandFlow and UniHand already investigate generative hand-motion reconstruction using visual and skeletal cues. Although JoHan differs by generating the 2D motion internally, the paper does not sufficiently establish why the proposed coupling mechanism represents a substantial methodological advance beyond existing conditional generation formulations. A more explicit analysis of the difference between joint generation and conventional conditional 3D generation would strengthen the contribution.

  2. Improvements in local pose accuracy do not consistently translate into better global motion reconstruction. This is an important concern because the paper aims to recover complete hand motion from video. In Table 2, JoHan achieves the best PA-MPJPE and acceleration error on HOT3D, but its world-space reconstruction remains behind several competing methods. Specifically, JoHan obtains a W-MPJPE of 55.05 mm, compared with 43.00 mm for HandFlow, and a WA-MPJPE of 20.36 mm, compared with 16.17 mm for HandFlow. These results indicate that the proposed global recovery stage may limit the overall performance. Since the framework relies on generated 2D–3D correspondences and a PnP solver, the authors should more carefully analyze the sources of global trajectory errors and clarify the practical advantages of the proposed approach beyond local reconstruction.

  3. The evaluation of temporal consistency and generative modeling remains incomplete. The paper claims that learning a joint motion distribution improves temporal coherence and resolves ambiguities under occlusion. However, temporal quality is mainly assessed through acceleration error, which does not necessarily distinguish physically plausible motion from excessively smoothed predictions. Furthermore, although the method is motivated by the one-to-many ambiguity of monocular observations, the experiments focus primarily on single-output reconstruction accuracy and do not sufficiently analyze the diversity or uncertainty of generated motion hypotheses. This makes it difficult to determine whether the generative formulation provides benefits beyond producing smoother point estimates.

Questions for authors
  1. Methodological novelty: Could the authors clarify the fundamental difference between the proposed joint 2D–3D flow and existing conditional generative approaches such as HandFlow and UniHand? Which aspect of the proposed coupling mechanism is essential beyond generating 2D guidance internally?

  2. Global motion accuracy: Table 2 shows that JoHan achieves better local pose accuracy but worse world-space errors than HandFlow. Could the authors clarify whether these errors primarily originate from PnP-based global recovery, generated 2D trajectories, or camera-motion estimation?

  3. Temporal consistency: Does the lower acceleration error necessarily correspond to more accurate motion dynamics? Could the authors provide additional evidence from existing evaluations that the improvement is not primarily caused by over-smoothing?

  4. Generative uncertainty: Since the method is motivated by ambiguity under occlusion, have the authors examined whether different samples produce meaningful and geometrically plausible motion hypotheses, rather than nearly identical predictions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer eySW [Part 1/2]

We thank the reviewer for recognizing the joint modeling formulation and the reconstruction and efficiency results. We address each concern below.

Q1. Methodological novelty:

The fundamental difference is that JoHan jointly generates 2D and 3D hand motion directly from video, rather than following HandFlow and UniHand's staged pipeline of deterministic regression followed by generation. As discussed in the paper's Related Work, this distinction has two main aspects:

1. Reliance on per-frame estimators. HandFlow uses HaMeR to obtain image features and 2D cues, while UniHand uses HaMeR for 2D motion conditions and first-frame pose initialization. Their 3D motion generation therefore depends on the quality and availability of these preceding deterministic estimates. JoHan instead conditions joint 2D–3D motion generation directly on full-frame video features, without requiring externally regressed 2D poses or an initial 3D pose. This removes the dependency on an upstream pose estimator, and the visual evidence remains available to the generators even when hand detection fails.

2. 2D–3D correspondences. HandFlow and UniHand use external 2D predictions to condition 3D generation, whereas JoHan jointly generates both sequences with 2D-Guided Fusion and reprojection supervision, promoting tighter alignment between 2D and local 3D motion. This alignment allows us to recover global placement directly from the reconstructed motion: given accurate 2D trajectories and local 3D poses, we restore the 2D joints to image coordinates and use their 2D–3D correspondences with the camera intrinsics in PnP to solve for global orientation and translation. JoHan therefore recovers camera-space placement geometrically, without requiring an additional global-pose prediction network.

Q2. Global motion accuracy:

JoHan's world-space errors appear to be primarily associated with limitations in hand detection, particularly when hands extend beyond the image boundaries.

To locate the sources of error, we separately replace the estimated camera trajectory with its GT rotation and translation. We also evaluate three substitutions along the hand-placement pipeline: GT bounding boxes, GT 2D trajectories supplied to PnP, and GT global hand orientation and translation, which bypass PnP placement.

HOT3D ground-truth substitutions. Errors are in mm; lower is better.

SettingW-MPJPE ↓WA-MPJPE ↓
JoHan (baseline)55.0520.36
With GT camera trajectory (rotation and translation)52.3319.47
With GT bounding boxes36.5212.20
With GT 2D trajectories at the PnP input27.6711.27
With GT global parameters25.067.95

Replacing only the camera trajectory reduces W-MPJPE from 55.05 to 52.33 mm and WA-MPJPE from 20.36 to 19.47 mm. This smaller improvement suggests that camera-motion estimation is not the dominant source of the observed world-space error.

The other three substitutions address coupled stages of hand placement: GT 2D and GT global hand parameters also bypass bbox-induced placement errors. The largest reduction between these ordered settings comes from GT boxes: 18.53 mm (55.05 to 36.52), followed by gaps of 8.85 mm to GT 2D and 2.61 mm to GT global hand parameters. These gaps are diagnostic comparisons, not an additive decomposition into independent error sources. They identify detection, which is not jointly optimized with the motion generators, as a major upstream bottleneck.

Failure inspection highlights hands extending outside the image as a challenging case for detection: boxes confined to the image boundaries may underestimate the full hand extent and misestimate its center and scale. These errors can affect 2D denormalization and subsequent PnP placement even when local 3D poses are accurate. As a diagnostic, excluding these cases (approximately 10% of the evaluated data) reduces W-MPJPE from 55.05 to 44.88 mm, illustrating the difficulty of these cases. Motivated by these detection limitations, we implemented a variant that predicts the hand center and square-box size directly from image features for 2D denormalization; its HOT3D results are reported below.

HOT3D hand-region prediction ablation.

Hand-region strategyPA-MPJPE (mm) ↓2D error (px) ↓W-MPJPE (mm) ↓WA-MPJPE (mm) ↓
JoHan (baseline)4.753.2755.0520.36
Directly predicted center and size4.754.2151.6420.52

The hand-region predictor has higher overall 2D localization error than the baseline, yet reduces W-MPJPE from 55.05 to 51.64 mm while leaving PA-MPJPE unchanged. This provides preliminary support for directly predicting hand bounds to address detector limitations.

We have added this localization analysis to Experiments in the revised manuscript and discuss the associated localization limitations and future work in the Conclusion.

AuthorsOpenReview ↗

Response to Reviewer eySW [Part 2/2]

Q3. Temporal consistency:

JoHan's lower acceleration error is accompanied by preserved wrist dynamics and improved reconstruction on small, fast-moving hands. These results support motion fidelity beyond smoothness.

Reconstruction of small, fast-moving hands. On the HOT3D small & fast subset (Reviewer 6SgP's W5), JoHan has the lowest reconstruction errors among the compared methods and a larger world-space advantage over HaWoR than on the full benchmark. These results indicate that JoHan's reconstruction advantage is more pronounced for small, fast-moving hands.

Preservation of fast motion. We analyze final predictions in a fixed window of one-second from the top decile of GT motion speed, separately for finger bending and wrist translation. The signal x(t)x(t) is the finger-bend angle with the largest range or wrist position projected onto the dominant motion direction.

A speed peak is a local maximum of ∣dx/dt∣|dx/dt|. GT and predicted peaks use the same GT-derived threshold and a minimum separation of 200 ms. Under this setting, We measure:

  • Amplitude ratio — motion range. Define A(x)=P95(x)−P5(x)A(x)=P_{95}(x)-P_{5}(x), where PqP_q is the temporal percentile, and average A(x^)/A(x)A(\hat{x})/A(x) equally across windows. 1 is ideal; lower values indicate attenuated motion and higher values indicate exaggerated motion.
  • Missed GT peaks — lost or mistimed events. The percentage of GT peaks without a matched prediction, using pooled peak counts across windows. 0% is ideal.
  • Extra predicted peaks — spurious or mistimed events. The percentage of predicted peaks without a GT match, using pooled peak counts across windows. 0% is ideal.
  • Matched-peak timing MAE — event timing. Mean ∣tpred−tGT∣|t_{\mathrm{pred}}-t_{\mathrm{GT}}| over all matched pairs, in milliseconds; 0 ms is ideal. Peak times lie on a 33.3 ms grid. Unmatched peaks are excluded, so this metric must be read alongside the missed- and extra-peak rates.

Motion amplitude and timing on fast events.

Motion / methodAmplitude ratio (ideal: 1)Missed GT peaks (%) ↓Extra predicted peaks (%) ↓Matched-peak timing MAE (ms) ↓
Finger bending / JoHan0.60436.726.458.6
Finger bending / HaWoR0.51734.029.869.0
Wrist translation / JoHan0.9424.814.348.3

JoHan retains 94.2% of GT wrist amplitude and misses 4.8% of peaks. For finger bending, it preserves more amplitude and has lower timing error than HaWoR, but a slightly higher missed-peak rate and amplitudes still below GT. These results show that JoHan preserves motion dynamics while improving reconstruction accuracy, suggesting that its gains cannot be attributed solely to over-smoothing.

Q4. Generative uncertainty:

JoHan maintains stable reconstruction accuracy across different sampling seeds or denoising steps while producing distinct, visually plausible hand poses under occlusion. We use the same HO3Dv3 inputs and 20 sampling steps for all five runs, and analyze these same predictions in two ways.

Accuracy against GT. We evaluate each of the five runs against GT, obtaining 7.013 ± 0.037 mm PA-MPJPE and 4.431 ± 0.014 px 2D error. The small variation indicates consistent reconstruction accuracy across seeds.

Diversity among the same five samples. For each frame, we compare the five predictions with one another, yielding 10 distinct pairs. These scores measure differences between predictions, rather than errors against GT. For pairwise PA-MPJPE, we average distances over the 20 non-wrist joints, both alignment directions, and all frame–pair combinations. Pairwise 2D distance averages Euclidean distances over the same joints, frames, and pairs without alignment. We report these differences by occlusion level; heavy occlusion means at least 10 of the 20 non-wrist joints are invisible.

Prediction diversity in inference.

Visibility subsetPairwise PA-MPJPE (mm)Pairwise 2D distance (px)
Light occlusion (≤5 invisible joints)2.0380.791
Heavy occlusion (≥10 invisible joints)2.8921.317

The larger pairwise differences under heavy occlusion show that the five samples retain greater variation when visual evidence is ambiguous. To illustrate their plausibility, the revised manuscript's Ablations section (“Pose Diversity under Occlusion”) visualizes these same five seeds for one frame from each of two videos. The examples show distinct finger poses consistent with the image evidence.

Thanks to the author for his careful and detailed rebuttal. The newly added global error attribution, fast motion preservation and diversity analysis under occlusion conditions have indeed responded to most of my previous doubts and made the experimental demonstration more complete. However, I still have questions about the core research motivation of the paper and the need for methodological innovation: the authors cite joint 2D–3D flow matching as the main contribution, but the current explanation puts more emphasis on its ability to avoid relying on external frame-by-frame pose estimation and guide 3D reconstruction through internally generated 2D trajectories. I don't yet fully understand why joint generation is necessary to solve these problems, rather than using a generative model that predicts 3D motion directly from video while being supervised by 2D geometry. In other words, beyond the improvement of experimental performance, what key limitations in existing conditional generation paradigms does joint modeling address, and what are its irreplaceable mechanistic advantages? I do not ask the author to add further experiments, but I hope that the author can help me establish a clearer logical chain based on the task motivation, method design and the actual benefits it brings. At present, I think this is a relatively complete work that has certain value and is on the edge of being accepted. However, its innovative narrative still makes me feel that it is closer to patching existing technical problems one by one, rather than a fully convincing unified research idea. If the author can clarify this core doubt, I am willing to increase the rating accordingly and will follow up the discussion in a timely manner

Our goal is to learn the joint distribution of local 2D and 3D hand motion conditioned on video, as illustrated in the Method section of the main paper. The distinctive advantage of this approach is that it uses complementary information from 2D and 3D motion to produce more accurate samples in both modalities while preserving both the diversity of the samples and the alignment between their 2D and 3D motions. Specifically, 2D trajectories provide explicit spatial and temporal cues about where the fingers appear and how they move across frames, while the 3D motion prior captures hand geometry and coordinated joint dynamics. Learning which image trajectories accompany which 3D motions should therefore improve reconstruction in both modalities. The ablation in the main paper supports this reasoning. JoHan reconstructs both 2D and 3D hand motion more accurately when the two are generated jointly than when each is generated independently.

Furthermore, we designed a reprojection loss to maintain consistency between the 2D and 3D motions sampled from the joint distribution. Our loss minimizes the discrepancy between the generated 2D motion and the projection of the corresponding generated 3D motion, encouraging the two trajectories in each sample to agree. This loss differs from those used in prior methods, which use reprojection to provide additional supervision by comparing the projection of the reconstructed 3D motion with ground-truth 2D trajectories. In the main paper, we explain the role of this loss in joint generation and demonstrate its effectiveness through the ablation in Table 4.

Also, our five-seed analysis and the examples in Figure 3 show that JoHan can generate diverse and plausible 2D and 3D motion hypotheses for the same video under heavy occlusion. At the same time, the stepwise 2D-to-3D guidance and the reprojection loss encourage geometric consistency between the two trajectories in each sample. By contrast, earlier approaches that condition 3D generation on a fixed 2D prediction face limitations in both diversity and consistency. First, for a given video, all 3D samples in these methods share the same 2D condition, even when the video supports several plausible image-plane trajectories. This restricts the range of 3D motions the model can explore, thus limiting the diversity of the generated motions. Second, conditioning alone does not explicitly enforce geometric agreement between the generated 3D motion and its 2D condition. In these methods, a reprojection loss computed against ground-truth 2D joints can even encourage the generated 3D motion to project onto the annotated trajectory, when the predicted 2D trajectory follows a different but plausible path. Such a mismatch between 2D and 3D motion can limit the accuracy of global recovery in these methods, whereas JoHan explicitly encourages agreement between its paired predictions, which benefits the use of PnP in our method.

Reviewer_6SgPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JoHan generates local 2D hand trajectories and local 3D MANO motion together: at each flow step, the predicted clean 2D coordinates guide the 3D branch, reprojection supervision links the branches during training, and box refinement with PnP recovers camera-space placement afterwards. The paper supports joint-model and reprojection ablations.

The open questions are why the 2D branch must be generative, why the baselines stay inside hand reconstruction, why global placement is solved after generation, and how the method behaves on small, fast hands.

Critical strengths
  • S1 — Joint generation and reprojection supervision are supported by controlled comparisons. PA-MPJPE is 7.8 mm against 8.9 mm for 3D-only generation, and 8.2 mm without reprojection supervision with Err2D 5.21→5.96 px (Table 4).
  • S2 — The reported camera-space and dynamic errors are useful. HO3Dv3 MPJPE is 32.8 mm against WiLoR's 53.5 mm (Table 1), and GT-box AccEr is 3.34 against 5.13 (Table 7). These measure dynamic accuracy on those benchmarks, not playback appearance.
Critical weaknesses
  • W1 — The method is an integration of established components, and the paper does not show what the integration adds (§3; Table 4). MANO local parameters, flow matching, a spatiotemporal transformer, clean-state prediction, coordinate-based guidance and PnP each have precedent, and I could not find what the combination adds beyond itself. Table 4 shows the combination works (7.8 against 8.9 mm; 8.2 mm without reprojection supervision), so my reservation is about attribution rather than performance; W2 is the comparison that would separate them.

  • W2 — A deterministic 2D estimator feeding the same 3D generator is not tested, so the value of generating 2D motion is unsupported (Eq.5; Table 4). Eq.5 passes encoded coordinates and visibility, not hidden 2D features (Lines 249–273), so OpenPose or a stronger detector fits the same interface once joints, normalization and confidence are aligned; the separate-training and noisy-teacher-forcing variants (12.89 and 9.49 mm, Lines 461–468) use ground-truth 2D in training and generated 2D at test time, so they test distribution matching instead. A run on the same 3D generator, video features, masks, split and Global Motion Recovery, with the 2D source identical between training and test, would settle it.

  • W3 — Baselines are chosen by application area, so the closest architectural relative is not compared (§4.2; Lines 274–279). The design is not hand-specific — the paper adopts D3DP's spatiotemporal architecture — and Kimodo (§4.1–4.2) already makes sequential clean predictions inside one network, with root and body branches conditioned on each other, whereas the 3D branch here reads the 2D branch only. One adapted general motion model under the same video inputs, MANO outputs and hand protocol would be enough, and full-body reconstruction tests are unnecessary.

  • W4 — Global translation and orientation are solved by post-hoc PnP, whereas Kimodo predicts them inside the network, and the paper gives neither a reason nor a comparison (Algorithm 2, steps 12–17; Eqs.7/8). Kimodo predicts root position and heading at every denoising step in the same network (§4.1–4.2), which is the end-to-end route and is generally expected to work better; here the denoised state holds no global motion, Eqs.7/8 use ground-truth global placement in training, and PnP runs after the flow loop. A matched comparison against a learned global branch interleaved with local generation would settle it; if the split is a deliberate trade-off, naming it (positioning versus temporal consistency, sensitivity to calibration) would be enough to judge it.

  • W5 — Small, fast hands outside hand-object interaction are untested (Appendix A.3–A.4). Evaluation uses hand-object datasets, and frames are resized to 256×256256\times256 (Lines 943–947) with no hand-size, speed or blur statistics, so I cannot tell whether the reported accuracy carries to dance-scale hand regions. Matched comparisons against strong hand baselines on such clips, with size, speed or blur-stratified errors where GT exists, would address it.

Minor Issues

  • No video demo.
  • Intrinsics. How KK is obtained for ordinary videos and updated after resizing, cropping or zooming (Algorithm 2; Appendix A.3).
  • Figure 2. Add �2��2� and �3��3� to the generator blocks (Line 241; Eq.3).
  • Visual conditioning. Whether tokens outside the hand box remain accessible at Lines 242–245 (Appendix A.1, Lines 858–862).
  • Sampling. State single sample, averaging or selection, and whether small differences persist across seeds (§4).
  • Visibility targets. Whether they mean physical visibility, annotation validity, or image-boundary validity (Eq.6; Lines 208–209).
Questions for authors

Please refer to previous sections.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6SgP [Part 1/3]

We thank the reviewer for the detailed suggestions and for recognizing the value of our joint-model and reprojection ablations. We address each concern in turn below.

W1 — The method is an integration of established components, and the paper does not show what the integration adds (§3; Table 4).

Our core contribution is the joint 2D–3D motion generation architecture, and we demonstrate its effectiveness for hand motion reconstruction. We thank the reviewer for citing Table 4 and acknowledging that the combined design is effective. For more detailed ablations of the individual components, please refer to our response to Reviewer UnnJ's W1. These controlled comparisons demonstrate the advantages of generative modeling over deterministic regression and the role of the 2D-Fusion block in improving 3D hand motion reconstruction. They also include the deterministic 2D regression variant with the same 3D generator requested in W2.

W2 — A deterministic 2D estimator feeding the same 3D generator is not tested, so the value of generating 2D motion is unsupported (Eq.5; Table 4).

We have implemented the requested ablation, which performs worse than joint generation on both metrics (see our response to Reviewer UnnJ's W1). For a fair comparison, the 2D regression branch uses the same spatiotemporal Transformer with AdaLN removed, while the rest of the network architecture, including the 3D flow generator, remains unchanged. On HO3Dv3, replacing 2D generation with regression increases PA-MPJPE from 7.8 to 8.5 mm and 2D error from 5.2 to 5.7 px.

In practice, we observed that 2D regression is more prone to low-quality predictions under heavy occlusion. Errors in the predicted 2D motion then propagate to the 3D generator through its conditioning, reducing 3D reconstruction accuracy. The generative variant handles these ambiguous cases more effectively, yielding better 2D and 3D motion reconstruction. This qualitative observation is consistent with our motivation for choosing generation over deterministic regression: to model plausible hand motion when visual evidence is incomplete or ambiguous.

W3 — Baselines are chosen by application area, so the closest architectural relative is not compared (§4.2; Lines 274–279).

We agree that Kimodo [1] and JoHan share a similar two-stage generation pipeline, in which one branch's clean prediction conditions the other at each denoising step. However, their conditioning signals differ substantially, and the success of root-motion conditioning for body motion does not necessarily carry over to hands. Kimodo generates global root motion to condition body motion, whereas JoHan generates 2D joint trajectories to guide local 3D hand motion.

In our early experiments, we tested wrist-motion conditioning on HO3D by supplying the ground-truth root (wrist) trajectory directly, including at test time. This evaluates the value of root-motion guidance without errors from a learned root predictor. We represented this GT global wrist trajectory as relative motion. Let Tt∈SE(3)T_t\in\mathrm{SE}(3) denote the wrist pose at frame tt, with rotation RtR_t and position ptp_t. For consecutive frames, we compute

\begin{bmatrix} R_{t-1}^{\top}R_t & R_{t-1}^{\top}(p_t-p_{t-1})\\\\ \mathbf{0}^{\top} & 1 \end{bmatrix}.$$ This transformation encodes both the relative rotation and the displacement expressed in the preceding wrist coordinate frame. Using this representation, we evaluated three conditioning settings on HO3D: the GT root trajectory alone, images alone, and the GT root trajectory together with images. **Hand motion generation with GT root-trajectory conditioning.** | Conditioning | PA-MPJPE (mm) ↓ | | :--- | ---: | | GT root trajectory only | 14.53 | | Image only | 10.86 | | GT root trajectory + image | 10.98 | Conditioning on the GT wrist trajectory alone yields a PA-MPJPE of 14.53 mm, compared with 10.86 mm for image-only conditioning. Even with the GT trajectory, adding wrist-motion guidance to the image-conditioned model gives 10.98 mm and does not improve reconstruction. Thus, in this setting, wrist-motion conditioning provides no additional accuracy benefit when visual evidence is already available.
AuthorsOpenReview ↗

Response to Reviewer 6SgP [Part 2/3]

W4 — Global translation and orientation are solved by post-hoc PnP, whereas Kimodo predicts them inside the network, and the paper gives neither a reason nor a comparison (Algorithm 2, steps 12–17; Eqs.7/8).

As explained in the Method section's Local Motion Representations paragraph, we learn local motion because it is constrained by hand anatomy and joint dynamics, whereas global placement varies with the task, body motion, and camera motion. HMP similarly notes that global hand orientation is less constrained than body orientation, for which gravity imposes stronger constraints. For example, a hand that translates with a fixed finger pose and one whose fingers continuously bend can share the same global translation and orientation. Global movement alone therefore does not determine local hand motion. JoHan reconstructs accurate 2D joint trajectories and local 3D hand motion with aligned 2D–3D correspondences. These correspondences, together with the camera intrinsics, allow PnP to recover global translation and orientation in camera space, so no additional prediction network is required.

Nevertheless, to address the reviewer's concern about learning global motion, we extended our ViT-L model with a network that predicts global translation and orientation. We embed the predicted global pose and supply it to the 3D flow-matching branch as an additional condition before local motion generation. During evaluation, we also use this predicted global pose to place the generated hand in camera space. The results on HO3Dv3 are reported below.

ViT-L with learned global-pose conditioning on HO3Dv3.

MethodPA-MPJPE (mm) ↓2D error (px) ↓MPJPE (mm) ↓
Baseline7.04.232.8
With global prediction7.24.660.9

As the table shows, the model with learned global motion yields an MPJPE closer to those of prediction-based methods such as WiLoR (53.5 mm) and HaMeR (54.7 mm), but remains substantially less accurate than our PnP-based recovery. Moreover, the additional global-pose conditioning does not improve hand-pose reconstruction. These comparisons support our choice to recover global pose with PnP rather than train an additional predictor.

W5 — Small, fast hands outside hand-object interaction are untested (Appendix A.3–A.4).

We examined existing dance-motion datasets [2–5], but they either lack corresponding RGB videos or hand-pose ground truth, or rely on HaMeR-generated hand annotations. These datasets therefore do not meet the requirements of our current evaluation protocol.

As an alternative, HOT3D contains small, rapidly moving hands in egocentric settings, allowing us to evaluate these challenging cases. To quantify hand-motion speed, we define the scale-normalized velocity as vrel=Lwrist/(αhandΔT)v_{\mathrm{rel}}=L_{\mathrm{wrist}}/(\alpha_{\mathrm{hand}}\Delta T), where LwristL_{\mathrm{wrist}} is the path length of the projected GT wrist over a one-second window, αhand\alpha_{\mathrm{hand}} is the median square root of the projected GT-joint bounding-box area over the window, and ΔT=1\Delta T=1 second. Speed is thus measured in hand scales per second.

Within the HOT3D test set, we define the small & fast subset as windows falling in both the lowest hand-size tertile (<30.91 pixels at a 256 × 256 image scale) and the highest speed tertile (≥2.689 hand scales/s), and the results are shown below.

Small & fast results. Lower is better. PA-MPJPE, camera-space MPJPE, W-MPJPE, and WA-MPJPE: mm; acceleration error: m/s².

MethodPA-MPJPE ↓Camera MPJPE ↓W-MPJPE ↓WA-MPJPE ↓AccEr ↓
HaMeR9.8353.36159.3938.7313.98
WiLoR6.8347.3291.1227.3812.56
Dyn-HaMR7.4463.49115.3934.056.85
HaWoR6.0155.4975.8022.9510.29
JoHan (ours)4.9937.0161.6522.403.03

JoHan achieves the lowest errors across all reported metrics on the small & fast subset. In particular, its W-MPJPE reduction relative to HaWoR increases from 3.7% on the full benchmark (55.05 vs. 57.18 mm) to 18.7% on small & fast hands (61.65 vs. 75.80 mm), showing a larger advantage in world-space motion reconstruction under these challenging conditions. JoHan also achieves the lowest PA-MPJPE (4.99 mm). Compared with the full set, JoHan therefore shows a more pronounced advantage on small, rapidly moving hands.

AuthorsOpenReview ↗

Response to Reviewer 6SgP [Part 3/3]

Minor Issues

  • No video demo. We provide video demonstrations in the supplementary material, covering laboratory settings and in-the-wild scenarios that illustrate JoHan's generalization beyond controlled environments.
  • Intrinsics. We use dataset-provided intrinsics for benchmarks and assumed intrinsics for uncalibrated videos. For resizing or cropping, we update them as K′=AKK'=AK, where AA is the image-coordinate transformation.
  • Visual conditioning. Yes. All full-frame visual tokens remain accessible to the motion generators; the bounding box provides an additional spatial cue without masking tokens outside it.
  • Sampling. Evaluation uses one generated sample per window. Accuracy remains stable across the tested seeds and flow-step settings; see our response to Reviewer UnnJ's Q2 for the ablation results. Different seeds also produce distinct, plausible hand poses under occlusion, as shown in the revised manuscript's Ablations section (“Pose Diversity under Occlusion”).
  • Visibility targets. The visibility label indicates whether a joint is visible in the current video frame. A joint is labeled invisible if it is occluded by an object or a hand, or lies outside the image boundaries; otherwise, it is labeled visible.
  • Figure 2. In the revised manuscript, we have added the labels fϕ2Df_{\phi_{\mathrm{2D}}} and fϕ3Df_{\phi_{\mathrm{3D}}} to the corresponding generator blocks in Figure 2.

References

[1] Rempe, Davis, et al. "Kimodo: Scaling Controllable Human Motion Generation." arXiv preprint arXiv:2603.15546, 2026.

[2] Li, Ronghui, et al. "FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation." ICCV, 2023.

[3] Li, Ruilong, et al. "AI Choreographer: Music Conditioned 3D Dance Generation with AIST++." ICCV, 2021.

[4] Grauman, Kristen, et al. "Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives." CVPR, 2024.

[5] Lin, Jing, et al. "Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset." NeurIPS, 2023.

Which 2D estimator exactly did you test?

Also, I intentionally mention using an off-the-shelf 2D estimator as a replacement because they are often trained on much more and more diverse data and thus can handle more challenging cases that lack 3D GT.

BTW, in the supp. video, the generated 3D meshes are not as temporally stable as the 2D keypoints. And all the demo cases are too easy compared to commonly tested cases by the 2D estimators.

Although dance videos often lack GT 3D annotation, humans can still easily tell which result is better. I would like to see some qualitative comparisons.

Reviewer_UnnJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper recovers single-hand MANO motion from monocular video with two flow-matching transformers trained from scratch on DINOv3 features: one generates normalized 2D joints, the other local 3D pose, and the 3D branch is conditioned on the 2D branch's current prediction at every flow step. Global orientation and translation are then solved by PnP from the generated 2D–3D correspondences, using refined and Kalman-smoothed detector boxes and known intrinsics. Experiments cover local accuracy on HO3Dv2/v3, world-space accuracy and smoothness on HOT3D, and runtime, with ablations on a smaller ViT-B backbone.

Critical strengths
  • Generating 2D and 3D local motion jointly, with step-wise 2D-to-3D guidance, is a clean idea that avoids conditioning on external per-frame pose estimates. The training ablation supports it: a separately trained factorization and noisy teacher forcing are both much worse than joint training with self-generated 2D guidance.
  • PnP from the generated 2D–3D correspondences gives a simple, geometrically grounded camera-space recovery. MPJPE on HO3Dv3 drops from about 53 to 32.8 mm. Recovered motion is consistently smoother on HOT3D.
  • The method is efficient and well documented, and code release is promised.
Critical weaknesses
  • Local accuracy gains are marginal and confounded. PA-MPJPE improves by only 0.1–0.2 mm over WiLoR on HO3Dv2/v3. Switching the backbone from ViT-B to ViT-L alone moves HO3Dv3 from 7.8 to 7.0 mm, more than any ablated component, and training data differs from the baselines'. There is no deterministic regressor with the same DINOv3-L, data and heads, and no variant that removes only the 2D-Guided Fusion. As a result, the gain cannot be attributed to the proposed design.
  • The smoothness and camera-space gains may come from pipeline components the baselines do not get. Box refinement plus Kalman filtering directly smooth translation and AccEr, and JoHan uses known intrinsics with PnP. Baselines receive no equivalent smoothing, it is unclear whether they used the same intrinsics, and there is no Kalman-off ablation. Low AccEr can also reflect over-smoothing: Fig. 5 trajectories are smooth but visibly deviate from GT. This reliance on a detector, refiner, Kalman filter, known intrinsics and HaWoR's camera module also makes "recovers hand motion directly from video" a potential overstatement.
  • Application scope is narrow. Only a single hand is handled, and all test sets are lab or egocentric capture, with no in-the-wild or interacting-hand evaluation.
Questions for authors
  • How are videos longer than 128 frames handled (windowing or stitching)? Are there boundary artifacts?
  • How sensitive are the results to the number of flow steps and to detector failures?
2 comments
AuthorsOpenReview ↗

Response to Reviewer UnnJ [Part 1/2]

We thank the reviewer for recognizing the joint 2D–3D formulation, global recovery, and efficiency of our method. We appreciate the thoughtful comments on attribution and pipeline fairness and address them point by point below.

W1. Local accuracy gains are marginal and confounded.

While we agree that adopting a larger backbone yields the biggest performance improvement, we should emphasize that this gain comes from the entire pipeline and does not diminish the importance of the other individual components, since removing any of them would significantly degrade performance. The reduction from 7.8 to 7.0 mm is obtained by upgrading the backbone within the complete JoHan framework, demonstrating the benefit of stronger visual features under our joint 2D–3D generation design. The contribution of this design should be assessed with the backbone held fixed, as in the controlled ablations below. Furthermore, even comparing to WiLoR with a larger ViT-H backbone, JoHan delivers consistent improvements with ViT-L, which is half the size of ViT-H. With ViT-H, JoHan further achieves a PA-MPJPE of 6.9 mm on HO3Dv2, compared with WiLoR's 7.5 mm using the same backbone size—a reduction of 0.6 mm. As a result, these improvements should not be considered marginal.

To better isolate the contributions to reconstruction accuracy, we implemented four ablations to evaluate the effectiveness of our proposed design:

  1. 2D regression only: a deterministic estimator predicts 2D local motion from video.
  2. 2D and 3D regression: a deterministic 3D local-motion estimator receives the predicted 2D motion from the regressor in (1) through cross-attention.
  3. 2D regression with 3D generation: JoHan's 3D motion generator is conditioned on the deterministic 2D predictions from (1), with the two branches trained jointly.
  4. 2D and 3D generation with cross-attention only: both generative branches are retained, while the dedicated 2D-Fusion module is replaced by the cross-attention connection used in (2).

The deterministic estimators in (1) and (2) retain JoHan's spatiotemporal Transformer architecture, but remove the AdaLN blocks used for denoising-timestep conditioning and replace the noisy motion tokens with learnable 2D/3D motion tokens. All four ablations use a DINOv3 ViT-B backbone and are evaluated on HO3Dv3. The comparison separates the choice of regression versus generation from the design of the connection between the two branches; variant (4) retains 2D-to-3D information exchange and therefore tests the dedicated fusion module, rather than removing all 2D guidance.

Deterministic prediction and fusion ablations on HO3Dv3.

MethodPA-MPJPE (mm) ↓2D error (px) ↓
JoHan (baseline)7.85.2
(1) 2D regression only--5.7
(2) 2D and 3D regression8.95.7
(3) 2D regression with 3D generation8.55.7
(4) 2D and 3D generation, cross-attention only8.55.4

These comparisons support two findings:

  1. Generation outperforms regression. Across the tested ablations, generative modeling yields better reconstruction accuracy than the regression alternatives. Joint 2D–3D generation achieves PA-MPJPE of 7.8 mm and 2D error of 5.2 px, compared with 8.9 mm / 5.7 px for 2D and 3D regression and 8.5 mm / 5.7 px for 2D regression with 3D generation. These comparisons support the benefit of generative modeling for hand motion reconstruction.
  2. The design of 2D-to-3D fusion matters. With both generative branches retained, the dedicated 2D-Fusion module achieves lower PA-MPJPE (7.8 vs. 8.5 mm) and 2D error (5.2 vs. 5.4 px) than cross-attention alone. Thus, the improvement depends on how the generated 2D guidance is incorporated, beyond simply providing a connection between the two branches.

We believe the ablations above demonstrate the contributions of our motion generation design and the 2D-Fusion blocks.

AuthorsOpenReview ↗

Response to Reviewer UnnJ [Part 2/2]

W2. The smoothness and camera-space gains may come from pipeline components the baselines do not get.

We follow the established HOT3D evaluation protocol, using dataset-provided camera intrinsics for all methods, including HaMeR and WiLoR. Our smoothing strategy is also part of JoHan's complete reconstruction pipeline. We respectfully believe that removing it from JoHan or adding it to another method does not, by itself, make the comparison between complete methods fairer. Nevertheless, to help the reviewer assess the contribution of these components, we provide additional experiments that remove bbox refinement and Kalman filtering from JoHan and apply both to HaWoR. The results are shown below:

Ablation study on bbox refinement and Kalman filtering.

MethodPA-MPJPE (mm) ↓W-MPJPE (mm) ↓WA-MPJPE (mm) ↓AccEr (m/s²) ↓
JoHan (baseline)4.7555.0520.362.69
JoHan w/o bbox refiner4.8966.6222.782.74
JoHan w/o bbox Kalman4.7557.7220.583.22
JoHan w/o both5.2078.3726.863.42
HaWoR (original setting)6.0557.1817.767.14
HaWoR + bbox refiner + bbox Kalman6.5050.2116.746.24

The results clearly indicate that JoHan achieves the lowest acceleration error among these variants, with or without the additional components (3.42 without both components, versus 7.14 for HaWoR). Although applying the same components to HaWoR improves world-space accuracy, its acceleration error remains higher than JoHan's, and its local accuracy deteriorates.

W3. Application scope is narrow.

We follow prior work in training and evaluating JoHan on established public datasets. Using these benchmarks enables comparison under standard protocols, but does not imply that JoHan is restricted to their capture environments. As shown in Appendix A.7, “Application in the Wild” (Figure 8), of the revised manuscript, JoHan also reconstructs detailed hand poses in internet videos beyond the benchmark scenarios, including object manipulation and hand gestures. For videos containing both hands, we apply the same model independently to each hand using hand-side conditioning and recover both hands in a shared camera coordinate system. This supports reconstruction of both hands in the same video, although the current model does not explicitly enforce inter-hand contact or prevent interpenetration. Following HaMeR and WiLoR, we use predefined camera intrinsics when calibration is unavailable.

Q1. How are videos longer than 128 frames handled (windowing or stitching)? Are there boundary artifacts?

JoHan processes long videos using windows of up to 128 frames with a 64-frame stride. Predictions for overlapping frames are blended, with greater weight assigned to predictions near each window center. On HOT3D, JoHan's local pose error (PA-MPJPE) is 4.63 mm at window joins versus 4.67 mm at nearby non-boundary frames, while the corresponding AccEr values are 2.50 and 2.53 m/s². These comparable errors indicate that window boundaries have little additional effect on reconstruction accuracy and temporal continuity in JoHan's evaluation.

Q2. How sensitive are the results to the number of flow steps and to detector failures?

JoHan maintains stable reconstruction accuracy across the tested random seeds and flow-step counts. We evaluated JoHan-L on HO3Dv3 using five random seeds and 10, 20, or 50 flow steps. The table reports mean PA-MPJPE ± sample standard deviation across seeds:

Flow stepsPA-MPJPE (mm) ↓
107.037 ± 0.036
207.013 ± 0.037
507.055 ± 0.043

The PA-MPJPE range across seeds remains below 0.09 mm at each step count, while mean errors differ by at most 0.042 mm across step counts. These results support JoHan's stability under the tested inference settings.

Table 4 in the main paper shows that local 3D pose reconstruction depends only weakly on detection-based conditioning: removing hand-mask conditions increases PA-MPJPE from 7.8 to 8.0 mm, as the generators retain access to full-frame visual evidence. Global motion recovery is rather affected as it requires the hand bounds for 2D denormalization. When detections are missing, these bounds are interpolated from neighboring valid frames.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank all reviewers for their thoughtful feedback. In addition to the point-by-point responses and additional experiments reported below, we have revised and expanded the manuscript to address the concerns raised during the rebuttal. Added or revised manuscript passages are highlighted in blue to distinguish them from the original text. The main updates are as follows:

  1. Generative versus deterministic 2D modeling (Table 4; “Formulation of 2D–3D Generation”). We added an ablation that replaces the 2D generator with a deterministic regressor while retaining the 3D generator, together with an analysis of its results and the benefit of generative modeling when visual evidence is incomplete.
  2. Pose diversity under occlusion (Figure 3; “Pose Diversity under Occlusion”). We added 3D mesh visualizations from five sampling seeds for frames from two HO3Dv3 videos, illustrating distinct, visually plausible hand poses under ambiguous observations.
  3. Global placement errors and limitations (Experiments and Conclusion). We added an analysis of how inaccurate hand bounds, particularly for hands extending outside the image, affect 2D denormalization and subsequent PnP placement. We also discuss this limitation and directions for improving global recovery.
  4. In-the-wild applications (Appendix A.7; Figure 8). We added full-frame outdoor examples of hand–plant interactions and signed gestures, showing the input images, generated 2D joints, and reconstructed 3D meshes.
  5. Method presentation and teaser (Figures 1 and 2). We explicitly labeled the 2D and 3D generator blocks in Figure 2 as fϕ2Df_{\phi_{\mathrm{2D}}} and fϕ3Df_{\phi_{\mathrm{3D}}} to clarify the correspondence between the architecture diagram and the method description. We also updated the teaser (Figure 1), replacing WiLoR with Dyn-HaMR in the qualitative comparison.
  6. Video demonstrations (Supplementary material). We provide video demonstrations covering laboratory settings and in-the-wild scenarios that illustrate JoHan's generalization beyond controlled environments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.