- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EgoP2FN, a method that jointly generates future egocentric video frames and 3D hand poses from a single image and language instruction. The authors argue that hand poses provide structured action dynamics, while videos provide detailed visual context, and these two representations are complementary. The proposed method introduces a Geometry-aware Spatial-Temporal Propagation Module to incorporate geometric information and a Pose-Pixel Synergy Module to connect pose and video representations. Experiments are conducted on a new benchmark built from AssemblyHands, Ego-Exo4D, and EgoMe-pose, showing improvements over existing video generation and hand pose prediction methods.
Critical strengths
-
The authors observe that hand poses and video frames provide complementary information for egocentric action prediction. Learning these two representations together is a meaningful research direction.
-
GSTPM and PPSM address two important challenges: insufficient geometric information and the representation gap between pose and pixels. This makes the method more complete than only improving video generation quality. 20694_Learning_Pose_Pixel_Syne
-
The paper evaluates both video generation metrics and 3D hand pose prediction metrics. It also provides ablation studies to analyze the effect of different components.
Critical weaknesses
-
The difference from existing video generation frameworks needs more clarification. EgoP2FN is built on an existing flow-matching video generation framework and introduces GSTPM and PPSM. However, the paper does not fully analyze whether the improvement mainly comes from pose guidance, geometric guidance, or the strong backbone itself. More detailed contribution analysis is needed.
-
The authors build a new benchmark from AssemblyHands, Ego-Exo4D, and EgoMe-pose. However, the test set only contains 400 samples, which is relatively small compared with the training set. It is unclear whether this benchmark is sufficient to measure the generalization ability in more complex real-world scenarios.
-
The paper emphasizes that EgoP2FN does not require external pose conditions, while some controllable generation methods use ground-truth hand poses as inputs. Since these methods have different problem settings and input conditions, the comparison may not be fully fair. More details about model size, training data, and fine-tuning settings would be helpful.
-
The causal relationship between generated videos and predicted poses is not deeply analyzed. The paper claims that pose-pixel synergy is the key idea, but the current experiments mainly show final performance improvements and visual examples. More analysis is needed to understand whether pose prediction directly improves video generation, or whether better video generation helps pose prediction.
Questions for authors
-
EgoP2FN is built on the Cosmos-predict2.5-2B video generation framework. Would the proposed pose-pixel synergy modules still work effectively with other video foundation models?
-
The ablation results show that both pose guidance and pose flow learning improve performance. Can the authors further analyze whether the improvement mainly comes from pose information or geometric information?
-
Since the test set contains only 400 samples, can the authors provide larger-scale evaluation or cross-dataset evaluation to better demonstrate generalization ability?
-
For baselines that use ground-truth pose conditions, can the authors provide a more controlled comparison with the same input conditions?
5 comments
Response to Reviewer iAPT [Part 2/3]
4. W2 & Q3: Larger-scale and cross-dataset evaluations
Thanks to the reviewer’s insightful suggestion. We have conducted additional cross-dataset evaluations on a larger-scale EgoDex [1] test set, which contains 3168 samples and 111 action categories. We have also evaluated two representative task-specific baselines, i.e., EgoControl for Ego video generation and EgoH4 for Ego 3D hand pose prediction. As shown in tables (a) and (b), our method consistently outperforms the video generation and 3D hand pose prediction baselines across diverse metrics, demonstrating the generalization ability of our method to larger-scale and more complex Ego scenarios.
(a) Results of Ego video generation on the EgoDex test set
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ | ISC ↑ | IBC ↑ | SC ↑ | MS ↑ | AQ ↑ |
|---|---|---|---|---|---|---|---|---|---|
| EgoControl | 19.92 | 0.742 | 0.362 | 318.59 | 0.962 | 0.968 | 0.907 | 0.992 | 0.650 |
| Ours | 20.02 | 0.750 | 0.328 | 235.16 | 0.976 | 0.974 | 0.911 | 0.994 | 0.664 |
(b) Results of Ego 3D hand pose prediction on the EgoDex test set
| Method | MPJPE ↓ | PA-MPJPE ↓ | MPJVE ↓ |
|---|---|---|---|
| EgoH4 | 245.05 | 25.61 | 22.41 |
| Ours | 229.94 | 23.67 | 19.30 |
[1] Hoque R, Huang P, Yoon D, et al. Egodex: Learning dexterous manipulation from large-scale egocentric video[C]//International Conference on Learning Representations. 2026, 2026: 4218-4237.
5. Q4: Fair comparison under the same input conditions
Thanks to the reviewer’s constructive comment. We have conducted additional experiments to provide a more controlled comparison. Specifically, we have fine-tuned and evaluated the representative EgoControl baseline using the same input conditions (i.e., image and language instruction) as in our method. As shown in the following table, removing ground-truth hand pose conditions leads to performance degradation across all video generation metrics for EgoControl. In addition, despite not requiring ground-truth hand poses, our EgoP$^2$FN still outperforms the original fine-tuned EgoControl baseline with ground-truth hand pose conditions. These results further demonstrate the effectiveness of our method under the same input conditions without relying on external pose guidance.
| Method | Input Conditions | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ | ISC ↑ | IBC ↑ | SC ↑ | MS ↑ | AQ ↑ |
|---|---|---|---|---|---|---|---|---|---|---|
| EgoControl (Fine-tuned) | Image + Language Instruction + GT Hand Poses | 0.968 | ||||||||
| EgoControl (Fine-tuned) | Image + Language Instruction | 12.58 | 0.338 | 0.574 | 614.32 | 0.925 | 0.940 | 0.855 | 0.983 | 0.392 |
| Ours | Image + Language Instruction | 16.37 | 0.575 | 0.397 | 162.21 | 0.980 | 0.931 | 0.994 | 0.413 |
6. W3: More details of model size, training data, and fine-tuning settings
Thanks to the reviewer’s constructive comment. We have provided a detailed table comparing our EgoP$^2$FN with other video generation baselines in terms of model size, training data, input conditions, and fine-tuning settings, as follows:
| Method | Model Size | Training Data | Input Conditions | Fine-tuning Settings |
|---|---|---|---|---|
| Cosmos | 2B | Cosmos pretraining data | Image + Language Instruction | - |
| Wan2.2-TI2V | 5B | Wan2.2 pretraining data | Image + Language Instruction | - |
| CogVideoX-I2V | 5B | 35M video clips + 2B images (LAION-5B, COYO-700M) | Image + Language Instruction | - |
| AnimateAnyone$^{\dagger}$ | 2B | 5K Internet-collected videos | Image + Ground-truth Hand Poses | - |
| Zhang et al. | 14B | Ego4D dataset | Image + Language Instruction + Ground-truth Hand Poses | - |
| Cosmos$^{*}$ | 2B | Cosmos-Predict2.5 pretrained weights + Our dataset | Image + Language Instruction | LoRA rank=512, Learning Rate=1e-5, 20000 iterations |
| EgoControl$^{*\dagger}$ | 2B | Cosmos-Predict2.5 pretrained weights+ Nymeria dataset + Our dataset | Image + Language Instruction + Ground-truth Hand Poses | LoRA rank=512, Learning Rate=2e-5, 20000 iterations |
| EgoP$^2$FN (Ours) | 2B | Cosmos-Predict2.5 pretrained weights + Our dataset | Image + Language Instruction | LoRA rank=512 (Learning Rate=1e-5), GSTPM and PPSM (Learning Rate=4e-5), 20000 iterations |
denotes methods that are further fine-tuned. denotes methods that use ground-truth hand poses as additional conditioning inputs.
Response to Reviewer iAPT [Part 3/3]
7. W4: Analysis of causal relationship between generated videos and predicted poses
Thanks to the reviewer’s insightful comment. We have conducted additional experiments to analyze the causal relationship between generated videos and predicted poses. In table (a), we train a baseline by removing the pose guidance that injects pose features into video generation, and report its video generation performance. The results show that removing pose guidance leads to consistent performance degradation, with FVD increasing from 162.21 to 228.14. In table (b), we remove the video generation branch and retain only the Ego 3D hand pose prediction branch, leading to performance degradations of 93.31, 3.42, and 2.14 in MPJPE, PA-MPJPE, and MPJVE, respectively. These results demonstrate the bidirectional benefits between video generation and pose prediction: pose guidance improves video generation and vice versa, further validating the effectiveness of the proposed method to learn pose-pixel synergy and jointly generate Ego videos and predict 3D hand poses.
(a) Pose → Video: Effect of Pose Guidance on Video Generation
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FVD ↓ |
|---|---|---|---|---|
| Ours | 16.37 | 0.575 | 0.397 | 162.21 |
| Ours (w/o Pose Guidance) | 15.98 | 0.561 | 0.414 | 228.14 |
(b) Video → Pose: Effect of Video Generation on Pose Prediction
| Method | MPJPE ↓ | PA-MPJPE ↓ | MPJVE ↓ |
|---|---|---|---|
| Ours | 139.74 | 21.44 | 26.48 |
| Ours (w/o Video Generation) | 233.05 | 24.86 | 28.62 |
Response to Reviewer iAPT [Part 1/3]
Thanks to the reviewer’s insightful comments. As suggested by the reviewer, we have clarified the differences from existing frameworks, analyzed the contributions of different components, evaluated our modules on another video foundation model, conducted cross-dataset evaluations on a larger-scale dataset, conducted fair comparisons under the same input conditions, summarized more details of our method and baselines, and analyzed the causal relationship between generated videos and predicted poses. Our point-by-point responses are as follows:
1. W1: Clarification of differences from existing video generation frameworks
Different from existing general TI2V video generation frameworks that lack explicit action modeling, or controllable video generation models requiring restrictive conditioning inputs (e.g., future poses), our method aims to integrate the future Ego video generation and 3D hand pose prediction tasks in a unified flow-matching-based framework to learn pose-pixel synergy conditioned only on a single Ego image and the necessary language instruction. We adopt the universal DiT-based Cosmos-Predict2.5 as the backbone for jointly modeling the two tasks. Building on this framework, we develop a GSTPM to leverage geometric cues for explicit scene layout perception and persistent geometric guidance, and a PPSM to learn the pose flow and feed structured pose features back for enhancing pose-video consistency.
2. W1 & Q2: Contribution analysis of pose guidance, geometric guidance, and backbone
Thanks to the reviewer’s constructive comment. We have provided a more detailed ablation analysis to fully analyze the respective contributions of pose guidance, geometric guidance, and backbone. In the following table, SE and TM denote spatial geometric enhancement and temporal geometric modulation, respectively, while PG denotes pose guidance, and PF denotes pose flow learning. In the second row, removing pose guidance (w/o PG) causes a significant increase in FVD from 162.21 to 228.14. In the third row, we remove the geometric guidance (w/o SE&TM), which leads to MPJPE degrading from 139.74 to 196.19. In the fourth row, we fine-tune only the video generation backbone, resulting in substantially worse performance. These results indicate that pose guidance contributes more to the temporal coherence of generated Ego videos, whereas geometric guidance plays a more important role in accurate Ego 3D hand pose prediction.
3. Q1: Would the proposed modules still work with other video foundation models?
Thanks to the reviewer’s constructive suggestion. We have integrated the proposed pose-pixel synergy modules into the Wan2.2-TI2V-5B video foundation model and compared three configurations: the original Wan2.2-TI2V, the fine-tuned Wan2.2-TI2V, and the Wan2.2-TI2V with our modules. As shown in the following table, Wan2.2-TI2V with our modules consistently outperforms the fine-tuned baseline on most image fidelity, video quality, and VBench-I2V metrics for Ego video generation. Moreover, it also accurately predicts Ego 3D hand poses, achieving an MPJPE of 175.03, a PA-MPJPE of 19.56, and an MPJVE of 34.71. Notably, Wan2.2-TI2V with our modules even surpasses our original EgoP$^2$FN on multiple metrics. These results demonstrate that the proposed pose-pixel synergy modules can work effectively with different video foundation models.