- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes JoHan, a video-conditioned generative framework for joint 2D and 3D hand motion recovery. Unlike existing approaches that depend on intermediate frame-wise pose estimates, JoHan directly generates aligned 2D and 3D local hand motion sequences from monocular videos using flow matching. The method introduces a 2D-guided fusion mechanism, where evolving 2D motion predictions guide 3D motion generation, together with reprojection supervision to encourage geometric consistency. Global hand orientation and translation are subsequently recovered using 2D–3D correspondences and a PnP solver.
Experiments on HO3Dv2, HO3Dv3, and HOT3D demonstrate competitive local hand reconstruction accuracy, improved temporal smoothness, and efficient inference. Ablation studies further support the benefits of joint 2D–3D generation and geometric supervision.
Critical strengths
-
Meaningful joint modeling of 2D and 3D hand motion. The paper presents an interesting formulation that jointly generates temporally aligned 2D and 3D motion sequences without relying on externally estimated frame-wise hand poses as generation conditions. The 2D-guided fusion mechanism provides a reasonable way to incorporate image-space evidence into 3D motion reconstruction. This is a useful direction for addressing occlusion and temporal inconsistency in monocular hand reconstruction.
-
Competitive reconstruction performance and computational efficiency. The experiments demonstrate strong local hand reconstruction accuracy and improved temporal smoothness. In particular, JoHan achieves a PA-MPJPE of 4.75 mm and an acceleration error of 2.69 m/s² on HOT3D. Table 3 also reports substantially faster hand-motion recovery than the compared baselines. The ablations provide useful evidence that joint generation and reprojection supervision contribute to reconstruction accuracy.
Critical weaknesses
-
The methodological novelty of joint 2D–3D generation needs stronger justification. The central contribution is the joint generation of 2D and 3D hand motion through coupled flow-matching branches. However, the framework largely builds upon established components, including flow matching, spatiotemporal attention, 2D-guided conditioning, and reprojection supervision. Existing methods such as HandFlow and UniHand already investigate generative hand-motion reconstruction using visual and skeletal cues. Although JoHan differs by generating the 2D motion internally, the paper does not sufficiently establish why the proposed coupling mechanism represents a substantial methodological advance beyond existing conditional generation formulations. A more explicit analysis of the difference between joint generation and conventional conditional 3D generation would strengthen the contribution.
-
Improvements in local pose accuracy do not consistently translate into better global motion reconstruction. This is an important concern because the paper aims to recover complete hand motion from video. In Table 2, JoHan achieves the best PA-MPJPE and acceleration error on HOT3D, but its world-space reconstruction remains behind several competing methods. Specifically, JoHan obtains a W-MPJPE of 55.05 mm, compared with 43.00 mm for HandFlow, and a WA-MPJPE of 20.36 mm, compared with 16.17 mm for HandFlow. These results indicate that the proposed global recovery stage may limit the overall performance. Since the framework relies on generated 2D–3D correspondences and a PnP solver, the authors should more carefully analyze the sources of global trajectory errors and clarify the practical advantages of the proposed approach beyond local reconstruction.
-
The evaluation of temporal consistency and generative modeling remains incomplete. The paper claims that learning a joint motion distribution improves temporal coherence and resolves ambiguities under occlusion. However, temporal quality is mainly assessed through acceleration error, which does not necessarily distinguish physically plausible motion from excessively smoothed predictions. Furthermore, although the method is motivated by the one-to-many ambiguity of monocular observations, the experiments focus primarily on single-output reconstruction accuracy and do not sufficiently analyze the diversity or uncertainty of generated motion hypotheses. This makes it difficult to determine whether the generative formulation provides benefits beyond producing smoother point estimates.
Questions for authors
-
Methodological novelty: Could the authors clarify the fundamental difference between the proposed joint 2D–3D flow and existing conditional generative approaches such as HandFlow and UniHand? Which aspect of the proposed coupling mechanism is essential beyond generating 2D guidance internally?
-
Global motion accuracy: Table 2 shows that JoHan achieves better local pose accuracy but worse world-space errors than HandFlow. Could the authors clarify whether these errors primarily originate from PnP-based global recovery, generated 2D trajectories, or camera-motion estimation?
-
Temporal consistency: Does the lower acceleration error necessarily correspond to more accurate motion dynamics? Could the authors provide additional evidence from existing evaluations that the improvement is not primarily caused by over-smoothing?
-
Generative uncertainty: Since the method is motivated by ambiguity under occlusion, have the authors examined whether different samples produce meaningful and geometrically plausible motion hypotheses, rather than nearly identical predictions?
4 comments
Response to Reviewer eySW [Part 2/2]
Q3. Temporal consistency:
JoHan's lower acceleration error is accompanied by preserved wrist dynamics and improved reconstruction on small, fast-moving hands. These results support motion fidelity beyond smoothness.
Reconstruction of small, fast-moving hands. On the HOT3D small & fast subset (Reviewer 6SgP's W5), JoHan has the lowest reconstruction errors among the compared methods and a larger world-space advantage over HaWoR than on the full benchmark. These results indicate that JoHan's reconstruction advantage is more pronounced for small, fast-moving hands.
Preservation of fast motion. We analyze final predictions in a fixed window of one-second from the top decile of GT motion speed, separately for finger bending and wrist translation. The signal is the finger-bend angle with the largest range or wrist position projected onto the dominant motion direction.
A speed peak is a local maximum of . GT and predicted peaks use the same GT-derived threshold and a minimum separation of 200 ms. Under this setting, We measure:
- Amplitude ratio — motion range. Define , where is the temporal percentile, and average equally across windows. 1 is ideal; lower values indicate attenuated motion and higher values indicate exaggerated motion.
- Missed GT peaks — lost or mistimed events. The percentage of GT peaks without a matched prediction, using pooled peak counts across windows. 0% is ideal.
- Extra predicted peaks — spurious or mistimed events. The percentage of predicted peaks without a GT match, using pooled peak counts across windows. 0% is ideal.
- Matched-peak timing MAE — event timing. Mean over all matched pairs, in milliseconds; 0 ms is ideal. Peak times lie on a 33.3 ms grid. Unmatched peaks are excluded, so this metric must be read alongside the missed- and extra-peak rates.
Motion amplitude and timing on fast events.
| Motion / method | Amplitude ratio (ideal: 1) | Missed GT peaks (%) ↓ | Extra predicted peaks (%) ↓ | Matched-peak timing MAE (ms) ↓ |
|---|---|---|---|---|
| Finger bending / JoHan | 0.604 | 36.7 | 26.4 | 58.6 |
| Finger bending / HaWoR | 0.517 | 34.0 | 29.8 | 69.0 |
| Wrist translation / JoHan | 0.942 | 4.8 | 14.3 | 48.3 |
JoHan retains 94.2% of GT wrist amplitude and misses 4.8% of peaks. For finger bending, it preserves more amplitude and has lower timing error than HaWoR, but a slightly higher missed-peak rate and amplitudes still below GT. These results show that JoHan preserves motion dynamics while improving reconstruction accuracy, suggesting that its gains cannot be attributed solely to over-smoothing.
Q4. Generative uncertainty:
JoHan maintains stable reconstruction accuracy across different sampling seeds or denoising steps while producing distinct, visually plausible hand poses under occlusion. We use the same HO3Dv3 inputs and 20 sampling steps for all five runs, and analyze these same predictions in two ways.
Accuracy against GT. We evaluate each of the five runs against GT, obtaining 7.013 ± 0.037 mm PA-MPJPE and 4.431 ± 0.014 px 2D error. The small variation indicates consistent reconstruction accuracy across seeds.
Diversity among the same five samples. For each frame, we compare the five predictions with one another, yielding 10 distinct pairs. These scores measure differences between predictions, rather than errors against GT. For pairwise PA-MPJPE, we average distances over the 20 non-wrist joints, both alignment directions, and all frame–pair combinations. Pairwise 2D distance averages Euclidean distances over the same joints, frames, and pairs without alignment. We report these differences by occlusion level; heavy occlusion means at least 10 of the 20 non-wrist joints are invisible.
Prediction diversity in inference.
| Visibility subset | Pairwise PA-MPJPE (mm) | Pairwise 2D distance (px) |
|---|---|---|
| Light occlusion (≤5 invisible joints) | 2.038 | 0.791 |
| Heavy occlusion (≥10 invisible joints) | 2.892 | 1.317 |
The larger pairwise differences under heavy occlusion show that the five samples retain greater variation when visual evidence is ambiguous. To illustrate their plausibility, the revised manuscript's Ablations section (“Pose Diversity under Occlusion”) visualizes these same five seeds for one frame from each of two videos. The examples show distinct finger poses consistent with the image evidence.
Thanks to the author for his careful and detailed rebuttal. The newly added global error attribution, fast motion preservation and diversity analysis under occlusion conditions have indeed responded to most of my previous doubts and made the experimental demonstration more complete. However, I still have questions about the core research motivation of the paper and the need for methodological innovation: the authors cite joint 2D–3D flow matching as the main contribution, but the current explanation puts more emphasis on its ability to avoid relying on external frame-by-frame pose estimation and guide 3D reconstruction through internally generated 2D trajectories. I don't yet fully understand why joint generation is necessary to solve these problems, rather than using a generative model that predicts 3D motion directly from video while being supervised by 2D geometry. In other words, beyond the improvement of experimental performance, what key limitations in existing conditional generation paradigms does joint modeling address, and what are its irreplaceable mechanistic advantages? I do not ask the author to add further experiments, but I hope that the author can help me establish a clearer logical chain based on the task motivation, method design and the actual benefits it brings. At present, I think this is a relatively complete work that has certain value and is on the edge of being accepted. However, its innovative narrative still makes me feel that it is closer to patching existing technical problems one by one, rather than a fully convincing unified research idea. If the author can clarify this core doubt, I am willing to increase the rating accordingly and will follow up the discussion in a timely manner
Our goal is to learn the joint distribution of local 2D and 3D hand motion conditioned on video, as illustrated in the Method section of the main paper. The distinctive advantage of this approach is that it uses complementary information from 2D and 3D motion to produce more accurate samples in both modalities while preserving both the diversity of the samples and the alignment between their 2D and 3D motions. Specifically, 2D trajectories provide explicit spatial and temporal cues about where the fingers appear and how they move across frames, while the 3D motion prior captures hand geometry and coordinated joint dynamics. Learning which image trajectories accompany which 3D motions should therefore improve reconstruction in both modalities. The ablation in the main paper supports this reasoning. JoHan reconstructs both 2D and 3D hand motion more accurately when the two are generated jointly than when each is generated independently.
Furthermore, we designed a reprojection loss to maintain consistency between the 2D and 3D motions sampled from the joint distribution. Our loss minimizes the discrepancy between the generated 2D motion and the projection of the corresponding generated 3D motion, encouraging the two trajectories in each sample to agree. This loss differs from those used in prior methods, which use reprojection to provide additional supervision by comparing the projection of the reconstructed 3D motion with ground-truth 2D trajectories. In the main paper, we explain the role of this loss in joint generation and demonstrate its effectiveness through the ablation in Table 4.
Also, our five-seed analysis and the examples in Figure 3 show that JoHan can generate diverse and plausible 2D and 3D motion hypotheses for the same video under heavy occlusion. At the same time, the stepwise 2D-to-3D guidance and the reprojection loss encourage geometric consistency between the two trajectories in each sample. By contrast, earlier approaches that condition 3D generation on a fixed 2D prediction face limitations in both diversity and consistency. First, for a given video, all 3D samples in these methods share the same 2D condition, even when the video supports several plausible image-plane trajectories. This restricts the range of 3D motions the model can explore, thus limiting the diversity of the generated motions. Second, conditioning alone does not explicitly enforce geometric agreement between the generated 3D motion and its 2D condition. In these methods, a reprojection loss computed against ground-truth 2D joints can even encourage the generated 3D motion to project onto the annotated trajectory, when the predicted 2D trajectory follows a different but plausible path. Such a mismatch between 2D and 3D motion can limit the accuracy of global recovery in these methods, whereas JoHan explicitly encourages agreement between its paired predictions, which benefits the use of PnP in our method.
Response to Reviewer eySW [Part 1/2]
We thank the reviewer for recognizing the joint modeling formulation and the reconstruction and efficiency results. We address each concern below.
Q1. Methodological novelty:
The fundamental difference is that JoHan jointly generates 2D and 3D hand motion directly from video, rather than following HandFlow and UniHand's staged pipeline of deterministic regression followed by generation. As discussed in the paper's Related Work, this distinction has two main aspects:
1. Reliance on per-frame estimators. HandFlow uses HaMeR to obtain image features and 2D cues, while UniHand uses HaMeR for 2D motion conditions and first-frame pose initialization. Their 3D motion generation therefore depends on the quality and availability of these preceding deterministic estimates. JoHan instead conditions joint 2D–3D motion generation directly on full-frame video features, without requiring externally regressed 2D poses or an initial 3D pose. This removes the dependency on an upstream pose estimator, and the visual evidence remains available to the generators even when hand detection fails.
2. 2D–3D correspondences. HandFlow and UniHand use external 2D predictions to condition 3D generation, whereas JoHan jointly generates both sequences with 2D-Guided Fusion and reprojection supervision, promoting tighter alignment between 2D and local 3D motion. This alignment allows us to recover global placement directly from the reconstructed motion: given accurate 2D trajectories and local 3D poses, we restore the 2D joints to image coordinates and use their 2D–3D correspondences with the camera intrinsics in PnP to solve for global orientation and translation. JoHan therefore recovers camera-space placement geometrically, without requiring an additional global-pose prediction network.
Q2. Global motion accuracy:
JoHan's world-space errors appear to be primarily associated with limitations in hand detection, particularly when hands extend beyond the image boundaries.
To locate the sources of error, we separately replace the estimated camera trajectory with its GT rotation and translation. We also evaluate three substitutions along the hand-placement pipeline: GT bounding boxes, GT 2D trajectories supplied to PnP, and GT global hand orientation and translation, which bypass PnP placement.
HOT3D ground-truth substitutions. Errors are in mm; lower is better.
Replacing only the camera trajectory reduces W-MPJPE from 55.05 to 52.33 mm and WA-MPJPE from 20.36 to 19.47 mm. This smaller improvement suggests that camera-motion estimation is not the dominant source of the observed world-space error.
The other three substitutions address coupled stages of hand placement: GT 2D and GT global hand parameters also bypass bbox-induced placement errors. The largest reduction between these ordered settings comes from GT boxes: 18.53 mm (55.05 to 36.52), followed by gaps of 8.85 mm to GT 2D and 2.61 mm to GT global hand parameters. These gaps are diagnostic comparisons, not an additive decomposition into independent error sources. They identify detection, which is not jointly optimized with the motion generators, as a major upstream bottleneck.
Failure inspection highlights hands extending outside the image as a challenging case for detection: boxes confined to the image boundaries may underestimate the full hand extent and misestimate its center and scale. These errors can affect 2D denormalization and subsequent PnP placement even when local 3D poses are accurate. As a diagnostic, excluding these cases (approximately 10% of the evaluated data) reduces W-MPJPE from 55.05 to 44.88 mm, illustrating the difficulty of these cases. Motivated by these detection limitations, we implemented a variant that predicts the hand center and square-box size directly from image features for 2D denormalization; its HOT3D results are reported below.
HOT3D hand-region prediction ablation.
The hand-region predictor has higher overall 2D localization error than the baseline, yet reduces W-MPJPE from 55.05 to 51.64 mm while leaving PA-MPJPE unchanged. This provides preliminary support for directly predicting hand bounds to address detector limitations.
We have added this localization analysis to Experiments in the revised manuscript and discuss the associated localization limitations and future work in the Conclusion.