acceptodds
Under review as a conference paper at ICLR 2027

Learning Pose-Pixel Synergy: Joint Egocentric Video and 3D Hand Pose Generation from a Single Image

Abstract

Predicting future actions based on the current state from an egocentric (Ego) perspective is vital for applications such as embodied AI. Previous works typically predict sparse hand poses or trajectories, or generate action videos relying on restrictive conditioning inputs. We observe that the two forms of Ego actions (i.e., hand pose and video) exhibit inherent synergy: poses provide abstract high-level structured dynamics, while videos preserve detailed pixel-level visual contexts. Inspired by this, we introduce an Egocentric Pose-Pixel Flow Network (EgoPFN), a unified flow-matching-based framework to jointly generate future Ego 3D hand poses and videos from a single image. To overcome the challenges of insufficient visual cues and the pose-pixel representation gap, we first develop a Geometry-aware Spatial-Temporal Propagation Module (GSTPM). It leverages geometric cues from the estimated first-frame depth map, which are integrated into visual features via spatial enhancement and temporal modulation, enabling explicit scene layout perception and persistent geometric guidance. Then, we propose a Pose-Pixel Synergy Module (PPSM), which learns pose flow from observation-anchored intermediate features and injects the learned structured pose information back into video latents via adaptive layer normalization, thereby enhancing the consistency between the generated poses and videos. Extensive experiments on our newly curated benchmark demonstrate that our EgoPFN outperforms many state-of-the-art video generation and hand pose prediction approaches by a large margin. Code will be released.

Primary area:
generative models
Keywords:
Pose-Pixel Synergy, Egocentric Video Generation, Egocentric 3D Hand Pose Forecasting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iAPTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EgoP2FN, a method that jointly generates future egocentric video frames and 3D hand poses from a single image and language instruction. The authors argue that hand poses provide structured action dynamics, while videos provide detailed visual context, and these two representations are complementary. The proposed method introduces a Geometry-aware Spatial-Temporal Propagation Module to incorporate geometric information and a Pose-Pixel Synergy Module to connect pose and video representations. Experiments are conducted on a new benchmark built from AssemblyHands, Ego-Exo4D, and EgoMe-pose, showing improvements over existing video generation and hand pose prediction methods.

Critical strengths
  1. The authors observe that hand poses and video frames provide complementary information for egocentric action prediction. Learning these two representations together is a meaningful research direction.

  2. GSTPM and PPSM address two important challenges: insufficient geometric information and the representation gap between pose and pixels. This makes the method more complete than only improving video generation quality. 20694_Learning_Pose_Pixel_Syne

  3. The paper evaluates both video generation metrics and 3D hand pose prediction metrics. It also provides ablation studies to analyze the effect of different components.

Critical weaknesses
  1. The difference from existing video generation frameworks needs more clarification. EgoP2FN is built on an existing flow-matching video generation framework and introduces GSTPM and PPSM. However, the paper does not fully analyze whether the improvement mainly comes from pose guidance, geometric guidance, or the strong backbone itself. More detailed contribution analysis is needed.

  2. The authors build a new benchmark from AssemblyHands, Ego-Exo4D, and EgoMe-pose. However, the test set only contains 400 samples, which is relatively small compared with the training set. It is unclear whether this benchmark is sufficient to measure the generalization ability in more complex real-world scenarios.

  3. The paper emphasizes that EgoP2FN does not require external pose conditions, while some controllable generation methods use ground-truth hand poses as inputs. Since these methods have different problem settings and input conditions, the comparison may not be fully fair. More details about model size, training data, and fine-tuning settings would be helpful.

  4. The causal relationship between generated videos and predicted poses is not deeply analyzed. The paper claims that pose-pixel synergy is the key idea, but the current experiments mainly show final performance improvements and visual examples. More analysis is needed to understand whether pose prediction directly improves video generation, or whether better video generation helps pose prediction.

Questions for authors
  1. EgoP2FN is built on the Cosmos-predict2.5-2B video generation framework. Would the proposed pose-pixel synergy modules still work effectively with other video foundation models?

  2. The ablation results show that both pose guidance and pose flow learning improve performance. Can the authors further analyze whether the improvement mainly comes from pose information or geometric information?

  3. Since the test set contains only 400 samples, can the authors provide larger-scale evaluation or cross-dataset evaluation to better demonstrate generalization ability?

  4. For baselines that use ground-truth pose conditions, can the authors provide a more controlled comparison with the same input conditions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer iAPT [Part 1/3]

Thanks to the reviewer’s insightful comments. As suggested by the reviewer, we have clarified the differences from existing frameworks, analyzed the contributions of different components, evaluated our modules on another video foundation model, conducted cross-dataset evaluations on a larger-scale dataset, conducted fair comparisons under the same input conditions, summarized more details of our method and baselines, and analyzed the causal relationship between generated videos and predicted poses. Our point-by-point responses are as follows:

1. W1: Clarification of differences from existing video generation frameworks

Different from existing general TI2V video generation frameworks that lack explicit action modeling, or controllable video generation models requiring restrictive conditioning inputs (e.g., future poses), our method aims to integrate the future Ego video generation and 3D hand pose prediction tasks in a unified flow-matching-based framework to learn pose-pixel synergy conditioned only on a single Ego image and the necessary language instruction. We adopt the universal DiT-based Cosmos-Predict2.5 as the backbone for jointly modeling the two tasks. Building on this framework, we develop a GSTPM to leverage geometric cues for explicit scene layout perception and persistent geometric guidance, and a PPSM to learn the pose flow and feed structured pose features back for enhancing pose-video consistency.

2. W1 & Q2: Contribution analysis of pose guidance, geometric guidance, and backbone

Thanks to the reviewer’s constructive comment. We have provided a more detailed ablation analysis to fully analyze the respective contributions of pose guidance, geometric guidance, and backbone. In the following table, SE and TM denote spatial geometric enhancement and temporal geometric modulation, respectively, while PG denotes pose guidance, and PF denotes pose flow learning. In the second row, removing pose guidance (w/o PG) causes a significant increase in FVD from 162.21 to 228.14. In the third row, we remove the geometric guidance (w/o SE&TM), which leads to MPJPE degrading from 139.74 to 196.19. In the fourth row, we fine-tune only the video generation backbone, resulting in substantially worse performance. These results indicate that pose guidance contributes more to the temporal coherence of generated Ego videos, whereas geometric guidance plays a more important role in accurate Ego 3D hand pose prediction.

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓MPJPE ↓PA-MPJPE ↓MPJVE ↓
Ours16.370.5750.397162.21139.7421.4426.48
SE+TM+PF (w/o PG)15.980.5610.414228.14155.5721.7327.77
PG+PF (w/o SE&TM)15.830.5590.422197.85196.1925.0227.52
Backbone Only15.290.5500.418246.18---

3. Q1: Would the proposed modules still work with other video foundation models?

Thanks to the reviewer’s constructive suggestion. We have integrated the proposed pose-pixel synergy modules into the Wan2.2-TI2V-5B video foundation model and compared three configurations: the original Wan2.2-TI2V, the fine-tuned Wan2.2-TI2V, and the Wan2.2-TI2V with our modules. As shown in the following table, Wan2.2-TI2V with our modules consistently outperforms the fine-tuned baseline on most image fidelity, video quality, and VBench-I2V metrics for Ego video generation. Moreover, it also accurately predicts Ego 3D hand poses, achieving an MPJPE of 175.03, a PA-MPJPE of 19.56, and an MPJVE of 34.71. Notably, Wan2.2-TI2V with our modules even surpasses our original EgoP$^2$FN on multiple metrics. These results demonstrate that the proposed pose-pixel synergy modules can work effectively with different video foundation models.

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓ISC ↑IBC ↑SC ↑MS ↑AQ ↑MPJPE ↓PA-MPJPE ↓MPJVE ↓
Wan2.2-TI2V12.550.4760.484524.230.9360.9460.8430.9800.400---
Wan2.2-TI2V (Fine-tuned)15.710.5580.421213.710.9630.9710.8850.9890.557---
Wan2.2-TI2V (+ Our modules)17.460.6000.357137.250.9780.9860.9320.9930.550175.0319.5634.71
AuthorsOpenReview ↗

Response to Reviewer iAPT [Part 2/3]

4. W2 & Q3: Larger-scale and cross-dataset evaluations

Thanks to the reviewer’s insightful suggestion. We have conducted additional cross-dataset evaluations on a larger-scale EgoDex [1] test set, which contains 3168 samples and 111 action categories. We have also evaluated two representative task-specific baselines, i.e., EgoControl for Ego video generation and EgoH4 for Ego 3D hand pose prediction. As shown in tables (a) and (b), our method consistently outperforms the video generation and 3D hand pose prediction baselines across diverse metrics, demonstrating the generalization ability of our method to larger-scale and more complex Ego scenarios.

(a) Results of Ego video generation on the EgoDex test set

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓ISC ↑IBC ↑SC ↑MS ↑AQ ↑
EgoControl19.920.7420.362318.590.9620.9680.9070.9920.650
Ours20.020.7500.328235.160.9760.9740.9110.9940.664

(b) Results of Ego 3D hand pose prediction on the EgoDex test set

MethodMPJPE ↓PA-MPJPE ↓MPJVE ↓
EgoH4245.0525.6122.41
Ours229.9423.6719.30

[1] Hoque R, Huang P, Yoon D, et al. Egodex: Learning dexterous manipulation from large-scale egocentric video[C]//International Conference on Learning Representations. 2026, 2026: 4218-4237.

5. Q4: Fair comparison under the same input conditions

Thanks to the reviewer’s constructive comment. We have conducted additional experiments to provide a more controlled comparison. Specifically, we have fine-tuned and evaluated the representative EgoControl baseline using the same input conditions (i.e., image and language instruction) as in our method. As shown in the following table, removing ground-truth hand pose conditions leads to performance degradation across all video generation metrics for EgoControl. In addition, despite not requiring ground-truth hand poses, our EgoP$^2$FN still outperforms the original fine-tuned EgoControl baseline with ground-truth hand pose conditions. These results further demonstrate the effectiveness of our method under the same input conditions without relying on external pose guidance.

MethodInput ConditionsPSNR ↑SSIM ↑LPIPS ↓FVD ↓ISC ↑IBC ↑SC ↑MS ↑AQ ↑
EgoControl (Fine-tuned)Image + Language Instruction + GT Hand Poses14.57‾\underline{14.57}0.479‾\underline{0.479}0.478‾\underline{0.478}174.60‾\underline{174.60}0.962‾\underline{0.962}0.9680.905‾\underline{0.905}0.990‾\underline{0.990}0.410‾\underline{0.410}
EgoControl (Fine-tuned)Image + Language Instruction12.580.3380.574614.320.9250.9400.8550.9830.392
OursImage + Language Instruction16.370.5750.397162.210.9800.965‾\underline{0.965}0.9310.9940.413

6. W3: More details of model size, training data, and fine-tuning settings

Thanks to the reviewer’s constructive comment. We have provided a detailed table comparing our EgoP$^2$FN with other video generation baselines in terms of model size, training data, input conditions, and fine-tuning settings, as follows:

MethodModel SizeTraining DataInput ConditionsFine-tuning Settings
Cosmos2BCosmos pretraining dataImage + Language Instruction-
Wan2.2-TI2V5BWan2.2 pretraining dataImage + Language Instruction-
CogVideoX-I2V5B35M video clips + 2B images (LAION-5B, COYO-700M)Image + Language Instruction-
AnimateAnyone$^{\dagger}$2B5K Internet-collected videosImage + Ground-truth Hand Poses-
Zhang et al.†^{\dagger}14BEgo4D datasetImage + Language Instruction + Ground-truth Hand Poses-
Cosmos$^{*}$2BCosmos-Predict2.5 pretrained weights + Our datasetImage + Language InstructionLoRA rank=512, Learning Rate=1e-5, 20000 iterations
EgoControl$^{*\dagger}$2BCosmos-Predict2.5 pretrained weights+ Nymeria dataset + Our datasetImage + Language Instruction + Ground-truth Hand PosesLoRA rank=512, Learning Rate=2e-5, 20000 iterations
EgoP$^2$FN (Ours)2BCosmos-Predict2.5 pretrained weights + Our datasetImage + Language InstructionLoRA rank=512 (Learning Rate=1e-5), GSTPM and PPSM (Learning Rate=4e-5), 20000 iterations

∗^{*} denotes methods that are further fine-tuned. †^{\dagger} denotes methods that use ground-truth hand poses as additional conditioning inputs.

AuthorsOpenReview ↗

Response to Reviewer iAPT [Part 3/3]

7. W4: Analysis of causal relationship between generated videos and predicted poses

Thanks to the reviewer’s insightful comment. We have conducted additional experiments to analyze the causal relationship between generated videos and predicted poses. In table (a), we train a baseline by removing the pose guidance that injects pose features into video generation, and report its video generation performance. The results show that removing pose guidance leads to consistent performance degradation, with FVD increasing from 162.21 to 228.14. In table (b), we remove the video generation branch and retain only the Ego 3D hand pose prediction branch, leading to performance degradations of 93.31, 3.42, and 2.14 in MPJPE, PA-MPJPE, and MPJVE, respectively. These results demonstrate the bidirectional benefits between video generation and pose prediction: pose guidance improves video generation and vice versa, further validating the effectiveness of the proposed method to learn pose-pixel synergy and jointly generate Ego videos and predict 3D hand poses.

(a) Pose → Video: Effect of Pose Guidance on Video Generation

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓
Ours16.370.5750.397162.21
Ours (w/o Pose Guidance)15.980.5610.414228.14

(b) Video → Pose: Effect of Video Generation on Pose Prediction

MethodMPJPE ↓PA-MPJPE ↓MPJVE ↓
Ours139.7421.4426.48
Ours (w/o Video Generation)233.0524.8628.62
Reviewer_nyHzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EgoP2FN, which jointly generates future egocentric videos and 3D hand poses from a single egocentric image and a language instruction. The method introduces spatial enhancement and temporal modulation to integrate geometric cues from the estimated first-frame depth map into visual features on the pre-trained Cosmos backbone, learns pose flow from observation-anchored intermediate features, and then feeds structured pose information back into video latents through AdaLN. The authors constructed a new benchmark from AssemblyHands, Ego-Exo4D and EgoMe-pose, and demonstrated the effectiveness of the complete system through video quality, pose error and component ablation experiments.

Critical strengths
  1. The research setup of the paper has practical significance. It only uses a single egocentric image and a language instruction to jointly generate future egocentric videos and 3D hand poses, which reduces the dependence on externally provided poses and provides a useful connection for visual future prediction and structured action modeling.

  2. The method is reasonably organized and supported by comprehensive basic experiments. The Geometry-aware Spatial-Temporal Propagation Module (GSTPM) integrates geometric cues through spatial enhancement and temporal modulation, while the Pose-Pixel Synergy Module (PPSM) learns pose flow from observation-anchored intermediate features and feeds structured pose information back into video latents through AdaLN. These modules address insufficient visual cues and the pose-pixel representation gap, forming a clear information path

Critical weaknesses
  1. In this paper, 18,916 training clips and 400 test samples are used, and the number of test samples from the three sources is only 75, 225 and 100 respectively, with corresponding sources involved in the training. The existing data are not necessarily insufficient for pre-trained model adaptation, but the current results mainly support the test performance within the existing data sources, and are not sufficient to establish extensive generalization ability.

  2. Table 3 supports the effectiveness of spatial geometric enhancement, temporal geometric modulation, pose guidance via AdaLN, and auxiliary pose regression supervision, but it does not fully distinguish gains from the conditioning information from those attributable to the specific architectural designs. For example, SE/TM lacks a comparison with a simpler conditioning mechanism that retains the same first-frame depth information, and pose flow learning lacks a direct pose regression baseline using the same observation-anchored intermediate features, supervision, and pose guidance interface.

  3. The auxiliary pose regression head provides observation-aware supervision in Eq. (11), and its prediction is integrated with the flow-based prediction in Eq. (12), with η = 0.5. The inference configuration when removing auxiliary supervision is not fully explained, so there remains an attribution gap between improving pose prediction and preventing shortcut denoising. It is recommended to clarify this configuration and present the latter as a mechanism explanation that requires further verification.

Questions for authors
  1. When removing auxiliary supervision in Table 3, how are the auxiliary pose regression head and η in Eq. (12) handled? Results using only the flow-based prediction, only the auxiliary pose regression prediction, and their integration should be reported separately to distinguish the effects of auxiliary supervision and prediction integration.

  2. With the same observation-anchored intermediate features, pose supervision, and pose guidance via AdaLN, how does direct pose regression perform relative to pose flow learning? Are there comparisons with simpler geometric cue injection or broadcasting the same first-frame geometric feature to all future temporal positions?

  3. Is there a direct quantitative evaluation of consistency between the generated egocentric videos and the predicted 3D hand poses? If not, can the evidence for improving video generation and hand pose prediction separately be clearly distinguished from the evidence for enhancing pose-video consistency?

  4. It is recommended to add larger and more diverse independent test sets to evaluate the generalization of EgoP2FN to unseen egocentric scenarios, objects, and action combinations.

4 comments
AuthorsOpenReview ↗

Response to Reviewer nyHz [Part 1/2]

Thanks to the reviewer’s insightful comments. As suggested by the reviewer, we have evaluated our method and baselines on a larger and more diverse test set, added comparisons with the simpler geometric conditioning mechanism and a direct pose regression baseline, clarified the configuration and potential role of auxiliary pose supervision, compared different pose prediction strategies, and conducted a quantitative evaluation of pose-video consistency. Our point-by-point responses are as follows:

1. W1 & Q4: Evaluate the generalization to unseen egocentric scenarios

Thanks to the reviewer’s constructive comment. We have evaluated our trained EgoP$^2$FN model on the larger and more diverse EgoDex [1] test set, which contains 3168 samples and 111 action categories. We have also evaluated two representative methods as task-specific comparison baselines, i.e., EgoControl for Ego video generation and EgoH4 for Ego 3D hand pose prediction. As shown in the tables (a) and (b), our EgoP$^2$FN consistently outperforms EgoControl across all video generation metrics and achieves more accurate Ego 3D hand pose predictions than EgoH4. These results demonstrate the generalization capability of our method to unseen egocentric scenarios with larger-scale data and more diverse action categories.

(a) Results of Ego video generation on the EgoDex test set

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓ISC ↑IBC ↑SC ↑MS ↑AQ ↑
EgoControl19.920.7420.362318.590.9620.9680.9070.9920.650
Ours20.020.7500.328235.160.9760.9740.9110.9940.664

(b) Results of Ego 3D hand pose prediction on the EgoDex test set

MethodMPJPE ↓PA-MPJPE ↓MPJVE ↓
EgoH4245.0525.6122.41
Ours229.9423.6719.30

[1] Hoque R, Huang P, Yoon D, et al. Egodex: Learning dexterous manipulation from large-scale egocentric video[C]//International Conference on Learning Representations. 2026, 2026: 4218-4237.

2. W2 & Q2: Comparison with the simpler geometric conditioning mechanism

Thanks to the reviewer’s constructive comment. As suggested by the reviewer, we have added a comparison with a simpler geometric cue injection baseline, which broadcasts the same first-frame geometric feature to all future temporal positions. As shown in the following table, our method consistently outperforms this baseline across all video generation and pose prediction metrics. Specifically, compared with our method, the simple geometric injection baseline causes a performance degradation of 52.94 in FVD and 31.16 in MPJPE. These results demonstrate that the proposed spatial geometric enhancement (SE) and temporal geometric modulation (TM) designs provide substantial benefits beyond simply injecting first-frame geometric cues.

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓MPJPE ↓PA-MPJPE ↓MPJVE ↓
Ours16.370.5750.397162.21139.7421.4426.48
Simple geometric injection16.310.5660.399215.15170.9021.9330.79

3. W2 & Q2: Comparison with a baseline trained only with direct pose regression

Thanks to the reviewer’s constructive suggestion. We have added an experiment with a baseline trained only with direct pose regression using the same observation-anchored intermediate features, supervision, and pose guidance interface. As shown in the following table, the regression-only baseline exhibits a performance drop of 35.51 in MPJPE and consistent degradation across all remaining video and pose metrics. These results demonstrate that the regression-only baseline is inferior to our observation-anchored pose flow learning under the same conditions, yielding less accurate Ego 3D hand pose prediction and resulting in less effective pose guidance for Ego video generation.

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓MPJPE ↓PA-MPJPE ↓MPJVE ↓
Ours16.370.5750.397162.21139.7421.4426.48
Regression-only baseline16.070.5680.409172.10175.2523.0629.11
AuthorsOpenReview ↗

Response to Reviewer nyHz [Part 2/2]

4. W3 & Q1: Clarification of configuration when removing auxiliary supervision and its potential role in preventing shortcut denoising

Thanks to the reviewer for pointing this out. When removing the auxiliary pose supervision in Table 3, we also remove the auxiliary regression head from the inference configuration (i.e., η=0\eta=0). The auxiliary regression directly predicts clean poses from intermediate features, thereby anchoring them to pose-aware observation information, which may encourage the flow matching head to learn from observations rather than relying on shortcut denoising. We have clarified that preventing shortcut denoising is a potential mechanism explanation rather than an established conclusion in the revised paper.

5. Q1: Comparison of different pose prediction strategies

Thanks to the reviewer’s constructive suggestion. We have conducted experiments to compare the pose prediction strategies using only the flow-based prediction (η=0\eta=0), only the auxiliary pose regression prediction (η=1\eta=1), and their integration (η=0.5\eta=0.5) during inference and reported their results separately. As shown in the following table, the flow-based (η=0\eta=0) and regression-based (η=1\eta=1) prediction branches exhibit different characteristics. In detail, Flow-based prediction achieves a lower PA-MPJPE (27.38 vs. 39.40), indicating more accurate local joint positions, whereas Regression-based prediction achieves a lower MPJVE (23.51 vs. 53.85), suggesting better temporal dynamics. By integrating the two predictions, our method achieves the best MPJPE (139.74) and PA-MPJPE (21.44) and a competitive MPJVE (26.48). These results demonstrate that our integration prediction strategy achieves high accuracy in both spatial positions and temporal dynamics.

Prediction StrategyMPJPE ↓PA-MPJPE ↓MPJVE ↓
Integrated Prediction (Ours, η=0.5\eta=0.5)139.7421.4426.48‾\underline{26.48}
Flow-based Prediction (η=0\eta=0)284.6227.38‾\underline{27.38}53.85
Regression-based Prediction (η=1\eta=1)266.40‾\underline{266.40}39.4023.51

6. Q3: Direct quantitative evaluation of pose-video consistency

Thanks to the reviewer’s insightful comment. We have conducted additional experiments to quantitatively evaluate the consistency between the generated Ego videos and the predicted 3D hand poses. Specifically, we project the predicted 3D hand poses onto the corresponding Ego video frames and compare their projections with the 2D hand keypoints estimated by an off-the-shelf hand pose estimation model [2]. We adopt Mean Reprojection Error (MRE) in pixels and Area Under the Curve (AUC) as consistency metrics.

We compare our full model with the SE+TM+PF variant as baseline, which removes pose guidance (PG), an important component for maintaining consistency between predicted poses and generated videos. As shown in the following table, our full model reduces MRE from 102.81 to 82.47 and improves AUC from 0.040 to 0.107 compared with the baseline. These results quantitatively demonstrate that the proposed pose guidance in our method facilitates achieving high consistency between generated Ego videos and predicted 3D hand poses.

MethodMRE↓AUC ↑
Ours82.470.107
Baseline102.810.040

[2] Vakunov A, Chang C L, Zhang F, et al. Mediapipe hands: On-device real-time hand tracking[C]//Workshop on Computer Vision for AR/VR. Seattle, WA, USA: arXiv, 2020, 2(4): 5.

Thanks for your response and the additional experiments. They address my concerns and support my initial positive assessment, so I will maintain my rating of 3.

Dear Reviewer nyHz,

Thank you for reviewing our rebuttal. We greatly appreciate your time and thoughtful consideration.

Best regards, The Authors

Reviewer_az3DOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Given the first frame of an egocentric video and a language instruction, EgoP2FN generates the future video together with the corresponding 3D hand poses. It fine-tunes the pretrained Cosmos-Predict2.5 video model and adds two modules. The first estimates a depth map from the first frame and uses it to enhance spatial features and to modulate features over time. The second predicts the hand poses (wrist positions and joints relative to the wrist) from intermediate video features, using flow matching together with an auxiliary regression head. The features of this pose branch are then injected into later video blocks through AdaLN, which the paper calls pose guidance. The authors build a benchmark of 18,916 training and 400 test clips from AssemblyHands, Ego-Exo4D and EgoMe-pose, and report better FVD and lower pose errors than fine-tuned video models and adapted pose-forecasting methods.

Critical strengths

1.Generating the video and the 3D hand poses together from a single frame and an instruction is a sensible setting, and unlike the pose-controlled video baselines, the model does not need future poses as input. Without the geometry module, adding pose guidance lowers FVD from 248.36 to 197.85 and MPJPE from 239.41 to 196.19 mm (Table 3). This is direct evidence that feeding the pose branch's features back into the video blocks improves both the generated video and the predicted poses. 2. Strong improvement over the baselines on both FVD and poses metrics.

Critical weaknesses
  1. The main claim, that the generated video and poses are consistent, is never measured. All quantitative metrics evaluate one output at a time: pose errors and PSNR/SSIM/LPIPS/FVD are computed against ground truth, and the VBench-I2V scores look only at the generated video. None of them checks whether a generated video and its generated poses show the same hand motion. I mean, such consistency is also not guaranteed by the design. Eq. 13 feeds the intermediate feature Z_p into the video branch, but the poses the model actually outputs, i.e. the flow samples blended with the regression head's prediction (Eq. 12), which has never reached the video branch. A direct test would project the generated 3D joints into the generated frames and compare them with 2D hand detections on those frames, without the need of post-hoc alignment used for the visualizations in Sec. 4.4.2. I think this should be reported for the full model and for a variant without pose guidance to show whether pose guidance improves agreement.

  2. Much of the MPJPE advantage over the pose baselines seems to come from the depth input, which the baselines do not get. With the same 2B backbone but without the geometry module and without pose guidance, MPJPE is 239.41 mm, worse than the adapted MADiff (179.85), EgoH4 (184.38), HaMeR (201.54) and Exo2EgoPose (203.30), so the pretrained backbone alone does not account for the gap. Adding pose guidance gives 196.19 mm, which is still behind MADiff and EgoH4, even though its PA-MPJPE of 25.02 mm already beats every baseline. Only when the geometry module is added does MPJPE fall to 139.74 mm. To show how much of the final margin is due to pose guidance, the paper needs the SE+TM+PF variant, or a baseline that is also given the depth map.

Questions for authors
  1. Requested experiment: Can you measure the agreement between each generated 3D pose sequence and its generated video, without post-hoc alignment, for both the full model and the SE+TM+PF variant? Please also say how you would handle camera intrinsics and hands that are occluded or out of view.
  2. Requested experiment: What MPJPE does the SE+TM+PF variant reach, and how does the strongest baseline (MADiff or EgoH4) perform when it is also given the depth map? Separately, how would a pose head trained only with the regression loss compare, using the same backbone and depth features? This would show whether flow matching is needed for the pose branch, or whether the auxiliary regression head does most of the work.
  3. Clarification: Was the fine-tuned Cosmos baseline trained with the same recipe as EgoP2FN?
3 comments
AuthorsOpenReview ↗

Response to Reviewer az3D [Part 1/2]

Thanks to the reviewer’s insightful comments. As suggested by the reviewer, we have conducted a quantitative pose-video consistency measurement, added an ablation study of the SE+TM+PF variant, evaluated pose baselines with depth inputs, and compared pose flow learning with direct pose regression. We have also clarified the training recipe of the fine-tuned Cosmos baseline. Our point-by-point responses are as follows:

1. W1 & Q1: Pose-video consistency measurement

Thanks to the reviewer’s insightful comment. Although the final blended poses are not directly injected into the video branch, the pose features ZpZ_p are jointly optimized by the flow matching and regression objectives during training, thereby enabling them to encode the structured future hand motion information. Such pose information is then conveyed to subsequent video blocks through feature-level pose guidance, guiding Ego video generation and improving pose-video consistency.

We have conducted additional experiments to quantitatively measure the consistency between the generated Ego 3D pose sequences and videos. Specifically, we project the generated 3D hand joints onto the generated frames without any post-hoc alignment and compare their projections with the 2D hand poses estimated by the widely adopted MediaPipe [1] toolbox. We use the official camera intrinsics provided by the original dataset for projection and perform joint-wise masking to mask out the invalid hand joints that are occluded or out of view. Moreover, we adopt Mean Reprojection Error (MRE) in pixels and Area Under the Curve (AUC) as consistency metrics. The results of our full model and the SE+TM+PF variant are reported in the following table. In detail, our full model reduces MRE from 102.81 to 82.47 and improves AUC from 0.040 to 0.107 compared with the SE+TM+PF variant. These results demonstrate that pose guidance effectively improves the agreement between the generated Ego 3D hand poses and video frames.

MethodMRE↓AUC ↑
Ours82.470.107
SE+TM+PF102.810.040

[1] Vakunov A, Chang C L, Zhang F, et al. Mediapipe hands: On-device real-time hand tracking[C]//Workshop on Computer Vision for AR/VR. Seattle, WA, USA: arXiv, 2020, 2(4): 5.

2. W2 & Q2: SE+TM+PF ablation to evaluate the contribution of pose guidance

Thanks to the reviewer’s constructive comment. We have added an ablation study of the SE+TM+PF variant, which retains SE, TM, and PF but removes PG (Pose Guidance), to show the final contribution of pose guidance. By injecting the pose features ZpZ_p into subsequent video blocks, PG allows video-generation gradients to propagate back to the shared representations, enhancing inter-task synergy through complementary pose and visual information. As shown in the following table, removing PG causes consistent degradation in both video generation and pose prediction performance. Specifically, FVD increases from 162.21 to 228.14, and MPJPE increases from 139.74 to 155.57. The full model also outperforms the SE+TM+PF variant on all remaining metrics. These results demonstrate the effectiveness of the proposed pose guidance in improving both Ego video generation and 3D hand pose prediction.

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓MPJPE ↓PA-MPJPE ↓MPJVE ↓
Ours16.370.5750.397162.21139.7421.4426.48
SE+TM+PF15.980.5610.414228.14155.5721.7327.77

3. W2 & Q2: Comparison to baselines with the depth map input

Thanks to the reviewer’s constructive suggestion. We have conducted additional experiments by providing depth maps as inputs to two strong pose baselines (i.e., MADiff and EgoH4), and reported the pose metrics of the SE+TM+PF variant. As shown in the following table, integrating depth maps leads to only marginal improvements for baselines, with decreases of 1.01 and 4.43 MPJPE for MADiff and EgoH4, respectively. Moreover, the SE+TM+PF variant achieves 155.57 MPJPE, 21.73 PA-MPJPE, and 27.77 MPJVE, consistently outperforming pose baselines with depth maps as input. In addition, it also significantly outperforms the PF-only variant by 83.84 in MPJPE. These results demonstrate that the improvements in pose prediction are primarily attributable to the proposed GSTPM, which effectively exploits depth cues through spatial geometric enhancement (SE) and temporal geometric modulation (TM), rather than merely adding a depth map as an additional input.

MethodMPJPE ↓PA-MPJPE ↓MPJVE ↓
MADiff179.8531.5470.51
MADiff w Depth178.8430.0869.57
EgoH4184.3832.8463.46
EgoH4 w Depth179.9532.1561.65
SE+TM+PF155.5721.7327.77
PF only239.4131.7932.92
AuthorsOpenReview ↗

Response to Reviewer az3D [Part 2/2]

4. Q2: Comparison with a pose head trained only with the regression loss

Thanks to the reviewer’s constructive suggestion. We have added an experiment with the pose head trained only with the regression loss using the same backbone and depth features. As shown in the following table, compared with our full model, the regression-only variant leads to a significant performance drop of 35.51 in MPJPE. It also exhibits consistent performance degradation across all remaining video generation and pose prediction metrics. These results indicate that a regression-only pose head is insufficient to achieve comparable pose prediction and video generation performance, whereas introducing the pose flow matching branch not only improves Ego 3D hand pose prediction accuracy but also provides more effective pose guidance for Ego video generation.

MethodPSNR ↑SSIM ↑LPIPS ↓FVD ↓MPJPE ↓PA-MPJPE ↓MPJVE ↓
Ours16.370.5750.397162.21139.7421.4426.48
Regression only16.070.5680.409172.10175.2523.0629.11

5. Q3: Was the fine-tuned Cosmos baseline trained with the same recipe as EgoP2FN?

The fine-tuned Cosmos baseline follows the same fine-tuning recipe as EgoP²FN, with identical configurations for all shared components. Both models use a learning rate of 1e-5 for LoRA parameters, while EgoP²FN additionally assigns a learning rate of 4e-5 to its newly initialized GSTPM and PPSM modules. We have further clarified the learning rate settings in the revised paper.

Dear Reviewer az3D,

Apologies for the follow-up. We would be grateful if you could review our rebuttal before the discussion period ends and let us know whether you have any remaining questions. If our responses address your concerns, would you consider updating the score accordingly?

We sincerely look forward to your feedback.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers and the AC for their thoughtful feedback

We sincerely thank all reviewers and the AC for their positive feedback: a meaningful research direction (Reviewer iAPT), a sensible problem setting (Reviewer az3D and nyHz), addressing important challenges (Reviewer iAPT), a more complete method (Reviewer iAPT), comprehensive experiments (Reviewer nyHz and iAPT), strong performance improvements (Reviewer az3D), and reasonable organization (Reviewer nyHz). In response to the reviewers’ concerns, we have added all the requested experiments, conducted more analyses, provided further clarifications, and so on. The detailed point-by-point responses are as follows:

AuthorsOpenReview ↗

A Gentle Follow-up on Our Rebuttal

Dear Reviewers,

Thank you for your thoughtful feedback. We have recently submitted our rebuttal with additional experiments, further analyses, and clarifications. As the discussion period is coming to a close, we would greatly appreciate it if you could take a look at our responses and let us know whether your concerns have been adequately addressed. Any further feedback would be highly appreciated.

Thank you again for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.