acceptodds
Under review as a conference paper at ICLR 2027

Perceive, Predict, and Plan: A Unified World Action Model for Egocentric Dexterous Manipulation

32 views

Abstract

We introduce , a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual–action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision–language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project website: https://p3-wam.pages.dev/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Action Models, Unified Model, Dexterous Manipulation, Egocentric Video, Hand Pose Tracking, World Model

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_At37OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes P3-WAM, a unified world action model for egocentric dexterous manipulation that integrates three complementary capabilities: 3D hand and camera motion reconstruction (PERCEIVE), action-conditioned video prediction (PREDICT), and goal-directed joint action and video generation (PLAN). Built upon a pretrained video diffusion model, the framework introduces a structured 3D action representation and a mixture-of-transformers architecture with bidirectional video–action attention. Through randomized modality visibility, the model learns all three tasks within a unified training framework.

Experiments across four egocentric manipulation datasets demonstrate improvements over single-mode training and strong performance against specialized baselines. The authors further evaluate planning quality through VLM and human assessments, along with qualitative real-robot demonstrations.

Critical strengths

1. A well-motivated and coherent unified framework.

The paper presents an interesting approach to connecting perception, prediction, and planning within a single world action model. The integration of structured 3D hand representations with a pretrained video diffusion backbone is well motivated, and the unified training strategy provides a natural way to share knowledge across these closely related tasks.

2. Strong and comprehensive experimental results.

The experimental evaluation is convincing, with consistent improvements over single-mode training across all three capabilities. The comparisons on multiple datasets demonstrate strong performance in both hand reconstruction and action-conditioned video prediction. The additional human evaluations, backbone comparisons, and qualitative robot demonstrations further strengthen the paper.

Critical weaknesses

1. Limited analysis of cross-task contributions in unified training.

While the paper demonstrates clear benefits of unified training over single-mode training, it does not fully investigate how the three tasks contribute to one another. The current experiments mainly compare individual task training against the full unified model, making it difficult to identify which task combinations are responsible for the observed improvements.

2. Lack of quantitative evaluation of camera trajectory prediction.

Camera motion is an important component of the proposed 3D action representation and is explicitly predicted by the model. However, the quantitative evaluations primarily focus on hand reconstruction accuracy and video generation quality, without separately reporting camera trajectory estimation errors. This makes it difficult to assess how accurately the model recovers camera motion, particularly over longer sequences.

Questions for authors

Q1. Cross-task contributions in unified training.

Could the authors provide additional ablations with pairwise task combinations, such as PERCEIVE + PREDICT, PERCEIVE + PLAN, and PREDICT + PLAN, alongside the full unified model? These comparisons would help clarify how the three capabilities benefit one another and whether certain task combinations contribute more significantly to the overall improvements.

Q2. Camera trajectory prediction accuracy.

Since camera motion is explicitly modeled and predicted alongside hand trajectories, could the authors provide quantitative evaluations of camera trajectory accuracy, such as rotation and translation errors against ground truth where available? An analysis of temporal drift would also help assess the reliability of camera motion estimation over longer sequences.

I will further raise my score if the questions above are discussed and solved clearly.

3 comments

Full response and visual evidence: Response to Reviewer At37.


We thank the reviewer for recognizing our coherent unified framework and the strong experimental results across perception, prediction, and planning. In response to your questions, we now report pairwise training ablations under a matched budget and quantitative camera evaluation covering rotation and translation accuracy, relative pose error, and temporal drift. These additions clarify cross-task contributions and demonstrate the accuracy and temporal stability of our camera predictions.


Pairwise task combinations

W1: Limited analysis of cross-task contributions in unified training.Q1: Cross-task contributions in unified training.

We compare Perceive + Plan, Predict + Plan, and Perceive + Predict under the same 5,000-update training budget as Unified and the single-mode models. The updated mode-ablation table reports Perceive and Predict results for all five settings.


Camera errors and temporal drift

W2: Lack of quantitative evaluation of camera trajectory prediction.Q2: Camera trajectory prediction accuracy.

We directly evaluate camera trajectories against ground truth using first-frame-relative rotation, scale-aligned camera-center error, adjacent-frame RPE, and temporal variation. Our model reduces camera-center error by 56.8%, rotation error by 26.4%, and translation RPE by 37.3% relative to VGGT. Examples A–C show raw trajectories across the full 5-second interval, with smaller final position and rotation errors.

We measure camera-position error at 2.5 and 5 seconds. Ours has lower error at both times and a smaller endpoint-minus-midpoint increase: 0.073 cm, compared with 0.194 cm for VGGT.

Position error (cm)OursVGGT
At 2.5 seconds0.2890.695
At 5.0 seconds0.3620.889
Endpoint − midpoint+0.073+0.194

Thank you for the rebuttal and additional experiments. All of my concerns have been addressed, and I will raise my score accordingly.

Dear Reviewer At37,

Thank you so much for your follow-up and for increasing your score! We truly appreciate your time, constructive comments, and support for our work. We will ensure that all discussed details are thoroughly updated in the revised manuscript.

Best regards,

The Authors

Reviewer_zZaPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a unified world action model that jointly learns three capabilities for egocentric dexterous manipulation, Perceive: 3D hand and camera motion reconstruction; Predict: action-conditioned video prediction; Plan: goal-conditioned joint action and video generation. The model integrates structured 3D action tokens with a pretrained video diffusion transformer through a mixture-of-transformers architecture and randomized modality visibility. Experiments on four datasets show improvements in hand reconstruction and action-conditioned video prediction over specialized baselines, while a vision-language model evaluation suggests better planning quality than video-generation baselines. Robot demonstrations illustrate the transfer of generated hand trajectories to physical manipulation.

Critical strengths
  1. Connecting perception, action-conditioned prediction, and goal-directed planning within one video-action model is well motivated. The three modes share a natural relationship through the underlying hand motions and their visual consequences.
  2. The paper evaluates perception and prediction on four datasets and includes comparisons with specialized baselines. The unified-versus-single-mode results provide evidence that joint training can improve multiple capabilities, rather than merely combining separate modules.
Critical weaknesses
  1. The main planning results rely on VLM scores, which may not reliably reflect physical feasibility, actual goal achievement, or executable action quality. The two robot demonstrations are illustrative rather than a systematic evaluation of task success, robustness, or generalization. This limits the strength of the paper's claims about dexterous manipulation.
  2. Although the paper compares unified training with single-mode training, it is not very clear how much of the improvement comes from cross-task knowledge transfer versus differences in effective training data, optimization, or exposure to additional supervision. More controlled ablations would strengthen the central claim that the three capabilities reinforce one another.
Questions for authors
  1. How well do VLM-based planning scores correlate with actual task completion and physical execution success? Can the authors provide quantitative robot experiments across multiple tasks and repeated trials?
  2. How accurately do the generated hand and camera trajectories correspond to the generated videos, and how sensitive is robot execution to discrepancies between the predicted trajectories and the visual dynamics?
  3. Are the single-mode and unified models matched in total training compute, data exposure, and optimization budget? Can the authors isolate the contribution of each additional training mode through controlled ablations?
3 comments

Full response and visual evidence: Response to Reviewer zZaP.


We thank the reviewer for recognizing the motivation for our unified framework and the evidence that joint training improves multiple capabilities. We now report closed-loop robot success rates over repeated Task26 trials, quantify agreement between generated hand trajectories and videos using an independent estimator, and isolate cross-task contributions through single-mode and pairwise comparisons with matched initialization, data mixture, batch size, and optimizer updates.


Task completion, repeated trials, and VLM scores

W1: The main planning results rely on VLM scores, which may not reliably reflect physical feasibility, actual goal achievement, or executable action quality. Q1: Can the authors provide quantitative robot experiments across multiple tasks and repeated trials?

We added repeated-trial closed-loop robot experiments with success measured by the benchmark’s automatic checker. After supervised fine-tuning, our model achieves 27/50 successes (54%) on Bench2Dex [1] Task26 under None and 22/50 (44%) under Full. The robot evaluation gives the comparison with Bench2Dex baselines.


Agreement between trajectories and visual dynamics

Q2: How accurately do the generated hand and camera trajectories correspond to the generated videos? How sensitive is robot execution to discrepancies between the predicted trajectories and the visual dynamics

We provide synchronized simulation–prediction videos from Ours with SFT, showing three successful Task26/Full rollouts at 1× speed. Please follow the video link to compare actual simulation on the left with model prediction on the right.

We further quantitatively evaluate the video-trajectory consistency by leveraging an independent estimator. We use HaMeR as a reference of hand motion in videos generated by Plan, then compare its estimates with Plan’s native hand trajectories. The mean PA-MPJPE is 11.69 mm on 320 generated frames conditioned on EgoDex inputs and 8.44 mm on 320 generated frames conditioned on GPT-image inputs. These results show close agreement between the predicted hand trajectories and the hand motion visible in the generated videos.


Matched training budget and cross-task contributions

Q3: Are the single-mode and unified models matched in total training compute, data exposure, and optimization budget?

Unified, single-mode, and the new mode-removal models use the same initializer, data mixture, global batch, and total number of optimizer updates. This means that for unified training, the total exposure to the same task setting is less than the corresponding single-mode setting, while still achieving better results on each of the subtasks.

W2: More controlled ablations would strengthen the central claim that the three capabilities reinforce one another.Q3: Can the authors isolate the contribution of each additional training mode through controlled ablations?

We added all three pairwise training controls: Perceive + Plan, Predict + Plan, and Perceive + Predict. The updated mode-ablation table compares all five settings on the same windows.


References

[1] Zhenjie Yang et al. Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands. arXiv:2609.15726, 2026.

I appreciate the authors for providing detailed responses and additional experiments. Most of my concerns are addressed. Therefore, I will increase my rating accordingly.

Dear Reviewer zZaP,

Thank you for your positive feedback and for taking the time to carefully consider our responses and additional experiments. We are glad that most of your concerns have been addressed and will make sure that all discussed details are thoroughly updated in the revised manuscript.

Best regards,

The Authors

Reviewer_hng7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces P³-WAM, a unified model for egocentric dexterous manipulation with three modes: PERCEIVE reconstructs hand and camera trajectories from video; PREDICT generates video conditioned on prescribed trajectories; and PLAN generates video together with hand and camera trajectories conditioned on an initial image, language, and an optional goal image. The model extends a pretrained Wan2.2 video backbone with compact hand/camera tokens, shared transformer weights, action-specific low-rank adaptations, and bidirectional video–action attention. Randomized modality visibility enables training across the three modes. Video generation uses flow matching, while action states are directly regressed, with PLAN retaining the action prediction from the final video-denoising step. Validation includes unified-versus-single-mode ablations, reconstruction and prediction benchmarks across four datasets, human and VLM assessments of generated videos, and two illustrative robot demonstrations.

Critical strengths

Controlled evidence supports the practical benefit of unified training. The single-mode comparisons match architecture, initialization, data mixture, and optimizer updates, making them informative about the benefit of the combined training recipe. Unified training reduces macro camera-space hand error and improves prediction PSNR. Planning improvements are also corroborated by human evaluation: Table 7 reports positive confidence intervals across all five dimensions, based on 21 complete pairs across 15 tasks. This provides meaningful evidence that sharing supervision across these tasks improves performance under a fixed training budget. The reconstruction and prediction results are supported by useful robustness checks.

Critical weaknesses

The explicit action outputs are not sufficiently validated independently of the generated videos. Most PLAN evaluation measures the visual quality and apparent task progression of generated videos. The quantitative trajectory-consistency test in Table 12 uses only eight image pairs and compares PLAN trajectories with reconstructions from the same model’s PERCEIVE branch. This establishes internal agreement but leaves geometric correctness and physical feasibility insufficiently tested. Furthermore, Appendix E.5 describes a pipeline beginning with generated video followed by scene/object reconstruction and retargeting, leaving unclear whether the robot executes PLAN’s native action predictions. The two qualitative robot examples lack attempted/successful trial counts. Predicted camera motion also receives no direct model-level quantitative evaluation; camera-space hand errors do not measure camera-trajectory accuracy. These gaps matter because the usefulness of the explicitly generated trajectories is central to the world–action contribution.

The methodological advance beyond existing unified formulations is insufficiently isolated. UWM already supports multiple video/action inference modes, and Aether unifies reconstruction, prediction, and goal-conditioned planning through randomized conditioning. P³-WAM’s extension to articulated hand–camera states is potentially valuable, but the experiments primarily compare the full architecture under unified and single-mode training. This establishes a benefit from the overall training recipe while leaving the contribution of bidirectional action-token coupling and action-specific adaptations unclear. In particular, the action-guidance experiment in Appendix F.4 does not separate the compact token pathway from the rendered skeleton and camera-ray controls. A focused comparison against a simpler shared-backbone formulation would clarify which proposed design contributes the additional capability and strengthen the case for the methodological contribution.

Questions for authors

Clarification of the existing robot experiments. Are the executed robot trajectories obtained directly by retargeting PLAN’s predicted hand and camera states, or is hand motion reconstructed again from the generated video, as Appendix E.5 appears to suggest? Please clarify the conversion pipeline and provide attempted and successful execution counts for each demonstrated task. This would determine how directly the robot results validate the proposed action outputs.

Validation of the claimed 3D outputs. Could you report relative camera rotation/translation errors and the resulting world-space hand-trajectory errors for PERCEIVE on the existing annotated evaluation data? For PLAN, is there an independent assessment of trajectory fidelity beyond reconstruction by the same model in Table 12? Such evidence would materially strengthen confidence in the geometric accuracy of the jointly generated trajectories.

Focused ablation of the proposed coupling. Under the same backbone, data, and training budget, how does the full model compare with a version that retains video-to-action prediction and skeleton/ray conditioning but removes action-to-video token attention? This control would help establish whether the proposed bidirectional coupling contributes to the planning improvement beyond shared training and auxiliary action supervision.

Regarding the Critical Weaknesses described above, I would appreciate further explanation.

3 comments

Full response and visual evidence: Response to Reviewer hng7.


Thank you for distinguishing agreement between our branches from independent validation. We now report camera rotation, scale-aligned translation, adjacent-frame RPE, and world-space hand errors against annotated data. The world-space hand measurement uses the model’s own camera prediction, rather than a ground-truth-camera conversion.


Native action execution

Q1: Are the executed robot trajectories obtained directly by retargeting PLAN’s predicted hand and camera states, or is hand motion reconstructed again from the generated video, as Appendix E.5 appears to suggest?

The robot demonstrations in the main paper use Plan’s predicted hand poses as input to optimization-based retargeting, which maps them into robot joint commands. The generated video is used to reconstruct the objects and scene and recover object motion over time. The hand poses supplied to retargeting come directly from Plan; there is no second hand-motion reconstruction step on the generated video. We have expanded Appendix E.5 to make these two paths explicit: video → scene and object reconstruction, and Plan hand poses → retargeting → robot joint commands → physical execution.

The explicit action outputs are not sufficiently validated independently of the generated videos. Most PLAN evaluation measures the visual quality and apparent task progression of generated videos.The two qualitative robot examples lack attempted/successful trial counts.Q2: For PLAN, is there an independent assessment of trajectory fidelity beyond reconstruction by the same model in Table 12?

As the robot demonstration in the main paper requires post-processing steps to convert MANO hands into robot native action space, the meaning of "success rate" would become obscure, as success / failure result can be affected by the post-processing steps. To this end, we supplement an additional bench2dex experiments [1], where we fine-tune our pre-trained model to output robot joint targets directly and evaluate closed-loop task success with the benchmark’s automatic checker. This gives a direct quantitative test of the model’s action output.

The Bench2Dex experiments execute Plan’s predicted robot joint targets directly and measure task success over repeated closed-loop trials. The execution setup and results describe the action decoding, control loop, and success counts. Success is determined by the simulator’s task checker, independently of the generated-video score.

The quantitative trajectory-consistency test in Table 12 uses only eight image pairs and compares PLAN trajectories with reconstructions from the same model’s PERCEIVE branch.

We provide synchronized simulation–prediction videos from Ours with SFT, showing three successful Task26/Full rollouts at 1× speed. Please follow the video link to compare actual simulation on the left with model prediction on the right.

We apply HaMeR to the videos generated by Plan and compare its estimates with Plan’s native hand trajectories, using 320 generated frames for each input group. HaMeR estimates agree with Plan’s native hand trajectories, with PA-MPJPE of 11.69 mm for EgoDex inputs and 8.44 mm for GPT-image inputs.


Camera motion evaluation

Predicted camera motion also receives no direct model-level quantitative evaluation; camera-space hand errors do not measure camera-trajectory accuracy. Q2: Could you report relative camera rotation/translation errors?

We now evaluate the predicted camera trajectories directly against ground truth. The camera evaluation reports rotation error, camera-center error, adjacent-frame RPE, and temporal variation, together with trajectory plots. Compared with VGGT, our model reduces camera-center error by 56.8%, rotation error by 26.4%, and translation RPE by 37.3%.

Contribution beyond existing unified formulations

The methodological advance beyond existing unified formulations is insufficiently isolated.

We admit that existing works have explored unified formulation in various fields. Aether models scene depth and camera trajectories for reconstruction, video prediction, and visual navigation. UWM couples robot-action and video diffusion to learn policies from robot demonstrations and action-free videos.

However, we argue that neither study isolates joint-training benefits across all constituent modes. We directly address this question with single-mode and pairwise controls that share the same initialization, data mixture, batch size, and update budget. These comparisons show that the unified mode mixture achieves the strongest overall results under a fixed training budget, supporting complementary benefits from training across different observation and prediction settings.


Decomposited Experiments

the experiments primarily compare the full architecture under unified and single-mode training.

We compare all three pairwise controls, Perceive + Plan, Predict + Plan, and Perceive + Predict, using the same initializer, data mixture, batch size, and 5,000-update budget as Unified. See the updated mode-ablation results.


World space hand metrics

Q2: Could you report the resulting world-space hand-trajectory errors for PERCEIVE on the existing annotated evaluation data?

We report hand MPJPE (mm) in camera and world coordinates. Alongside the full-benchmark results, we compare our method and Dyn-HaMR on the same matched hand-frames in the first-camera coordinate frame.

MPJPE (mm) / methodEgoDexARCTICHOT3DEgoLive
Camera-space — Ours32.9242.4787.0731.86
World-space — Ours32.8895.51149.3334.10
World-space — Dyn-HaMR104.25226.92210.55147.61

Training budget, decoupled experiments

Q3: Under the same backbone, data, and training budget, how does the full model compare with a version that retains video-to-action prediction and skeleton/ray conditioning but removes action-to-video token attention?

We clarify that all modes use the same backbone and training data. The training steps, FLOPs, are exactly the same. All variants use the same architecture, optimizer, and number of training updates. Action supervision is applied wherever valid annotations are available.

We compare the unidirectional and bidirectional attention models using the same inputs and sampling budget. Both allow action tokens to read video tokens; the bidirectional model also allows video tokens to read action tokens. In Plan, the bidirectional model achieves raw/PA-MPJPE of 150.16/9.67 mm, compared with 172.67/9.96 mm for the unidirectional model. For each of input windows, each model generates 32 complete trajectories from different noise initializations. For each metric, we take the lowest trajectory-level error among the 32 samples in each window, then average across windows. This best-of-32 score measures whether the sampled motions include an accurate trajectory, following the interpretation of best-of-k as favorable-tail coverage in Beyond Validation Loss, Section 4 [2].


References

[1] Zhenjie Yang et al. Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands. arXiv:2609.15726, 2026.

[2] Zhanyi Sun, Max Simchowitz, and Shuran Song. Beyond Validation Loss: Coverage and Reliability Dynamics in Robot Policy Pretraining. arXiv:2611.01565, 2026.

I thank the authors for their response and additional experiments. I believe some issues have been resolved as well, and I will raise my score accordingly.

Comments on the paper

2 comments

Full response and visual evidence: Common questions.


We sincerely thank the reviewers for the careful reviews. We appreciate the reviewers' recognition of our well-motivated unified framework, the benefits of joint training under a fixed training budget, and the strong hand-reconstruction and action-conditioned video-prediction results across four datasets. Reviewers also highlighted the human evaluation supporting improved planning quality and the robustness checks supporting our reconstruction and prediction results.

We invite the reviewers to our anonymous website responses to all reviewer comments at: common question, hng7, zZaP, and At37, which include comprehensive video and image results.


How accurate are the camera trajectories?

We evaluated the camera output of Perceive directly against annotated camera poses. Our model and the original VGGT-1B receive the same windows.

We express each trajectory relative to its first camera pose and score frames 1–80. Rotation error is the mean geodesic angle. Translation is the mean camera-center distance after fitting one nonnegative scale per clip; we apply that same scale when computing adjacent-frame relative pose error (RPE).

MethodRotation ↓Center ↓RPE rotation ↓RPE center ↓
Ours0.366°0.289 cm0.143°0.102 cm
Original VGGT0.497°0.669 cm0.141°0.162 cm

Our model reduces camera-center error by 56.8%, rotation error by 26.4%, and translation RPE by 37.3% relative to VGGT.

We also measured temporal variation. Translation acceleration is the mean norm of the second finite difference divided by the squared frame interval. VGGT has 43.0% larger translation acceleration. The acceleration residual against ground truth is also larger for VGGT (59.143 versus 40.496 cm/s²).

Raw trajectory measureOursVGGT
Translation acceleration40.486 cm/s²57.889 cm/s²
Angular acceleration65.288°/s²75.702°/s²

Examples A–C show camera centers in the first-camera coordinate frame, followed by position and rotation errors over time. Blue denotes our output, orange denotes VGGT, and gray denotes ground truth. All curves show raw outputs.

Please see the camera trajectory plots for Examples A–C and their per-frame position and rotation errors over five seconds.

In Examples A, B, and C, the final camera-center errors are 0.244, 0.605, and 0.085 cm, respectively, compared with 3.126, 2.163, and 0.239 cm for VGGT. These raw trajectories show smaller accumulated position drift, with lower endpoint rotation error in each example. In Example C, both methods remain close to the ground-truth trajectory, with mean camera-center errors of 0.088 cm for ours and 0.109 cm for VGGT.

Combining predicted MANO joints with predicted camera poses gives a world-space hand MPJPE of 32.879 mm and a joint-velocity error of 102.534 mm/s over the same evaluation windows. Predicted and annotated hand trajectories are expressed in the first-camera coordinate frame.


Do the action outputs lead to successful execution?

The main paper maps Plan’s predicted human hand poses to robot joint commands through optimization-based retargeting. To evaluate the action output directly in robot control space, we additionally fine-tune on Bench2Dex [1] demonstrations and execute the predicted robot joint targets. The policy takes the current three camera views, robot joint state, and task instruction, then predicts 32 joint targets at 20 Hz together with a future video. Each predicted joint delta is added to the joint state at the start of the chunk. The simulator executes these targets for 1.6 seconds before the policy receives the next observation. The generated video is saved for comparison; the executed actions come directly from the action output.

These experiments evaluate executable robot joint commands after post-training. They complement the human-hand results. Success is measured by the benchmark’s task checker, rather than by judging the generated video.

Simulation and prediction: Ours with SFT

Three successful Task26/Full examples from Ours with SFT. Left: actual simulation; right: model prediction. Each example shows the full rollout with all Plan queries aligned at 1× speed.

Please watch the synchronized simulation–prediction video to compare actual simulation on the left with model prediction on the right throughout the full rollouts.

Example 1 · Example 2 · Example 3


VLM scores and execution success on matched episodes

We analyze 100 Task26/Full episodes from Ours with SFT. Qwen3.8 Max 0902 scores five sampled Plan queries per episode on a 1–5 scale. We average valid query scores within each episode and pair them with simulator success and Bench2Dex’s normalized task-progress score (0–1), which measures the deepest stage reached.

ModelAction process ↑Instruction constraints ↑Physical interaction ↑Temporal consistency ↑Goal transition ↑Execution SR ↑
Ours with SFT2.8134.1884.9044.9722.99644.0%

Goal-transition scores track task progress (Spearman ρ = 0.636). Action-process scores correlate positively with both task progress (ρ = 0.553) and execution success (ρ = 0.299). The table reports correlations with both outcome labels.

VLM dimensionTask progress (Spearman ρ)Execution success (Spearman ρ)
Action process0.5530.299
Goal transition0.6360.207

Please see the per-episode VLM scatter plots. Each dot is one Task26/Full episode from Ours with SFT; dark bars and labels show mean VLM scores at each task-progress level.


What does each training mode contribute?

We compare all three pairwise training settings: Perceive + Plan (w/o Predict), Predict + Plan (w/o Perceive), and Perceive + Predict (w/o Plan), alongside Unified and Single Mode. The comparisons use the same original initializer, data mixture, global batch of 16, optimizer, and 5,000 total updates.

Perceive reports camera-space MANO joint error, root-relative error (RR), and Procrustes-aligned error (PA), all in millimeters. Predict reports PSNR, SSIM, LPIPS, and FVD. Single Mode uses separate Perceive-only and Predict-only checkpoints.

Training settingPerceive MPJPE ↓ (mm)Perceive RR ↓ (mm)Perceive PA ↓ (mm)Predict PSNR ↑Predict SSIM ↑Predict LPIPS ↓Predict FVD ↓
Unified94.3123.6911.5217.410.63950.41531223
w/o Predict107.626.8611.7716.930.62920.41691283
w/o Perceive119.426.6711.9017.280.63580.42201479
w/o Plan95.3524.6511.5317.330.63280.42171296
Single Mode127.828.8412.4716.850.62770.42411306

Please see the formatted mode-ablation table for the color-coded comparison. All settings share the same windows within each mode; green shading marks the top three results.

Unified's MPJPE is 94.31 mm, compared with 95.35 mm for w/o Plan and 127.8 mm for Single Mode; its PSNR is 17.41 dB, compared with 17.33 dB and 16.85 dB, respectively.


References

[1] Zhenjie Yang et al. Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands. arXiv:2609.15726, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.