acceptodds
Under review as a conference paper at ICLR 2027

VPTwin: Real-Sim-Real Video Prediction for Robotic Manipulation Planning

14 views

Abstract

While action-conditioned video prediction provides an intuitive world model for robotics, purely data-driven predictors often suffer from compounding errors and physically implausible hallucinations in long-horizon rollouts, severely undermining downstream action planning. We propose VPTwin, a Real-Sim-Real video prediction framework that anchors real-world future prediction using real-synchronized simulation twins. For a target manipulation task, a VLM reconstructs an executable digital twin from a real demonstration episode. To accommodate the ill-posed estimation of unobserved physical properties, Isaac Sim simulates multiple forward dynamic rollouts across randomized physical configurations under candidate action trajectories. Using these rollouts as in-context references, VPTwin harmonizes both domains, using simulation dynamics to enforce physical plausibility while capturing unmodeled contact interactions from real video. Furthermore, we establish a predictive planning loop using VPTwin to visually verify VLM-proposed actions and guide reliable real-world execution. Evaluations show substantial reductions in physical hallucinations during video prediction and marked improvements in manipulation planning performance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Real-Sim-Real video prediction, vision-based model predictive control, robotic manipulation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_QF38OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents VPTwin, a Real-Sim-Real video prediction framework that grounds future prediction in executable digital twins reconstructed by a VLM from real demonstrations. VPTwin-Core uses randomized, time-aligned simulation rollouts as physical context and incorporates synthetic failure co-training to model unsuccessful action outcomes. It then serves as a predictive visual verifier in staged planning, guiding the evaluation and refinement of VLM-proposed actions before real-world execution. Experiments report improved video prediction across rigid, deformable, and articulated manipulation tasks, alongside improved planning performance on two real-robot tasks.

Critical strengths

Simulation rollouts improve real-domain video prediction. VPTwin outperforms IWS in both settings, improving PSNR by about 3 dB and reducing LPIPS (Table 1). Training with ten rollouts also outperforms training with three when both models use the same three rollouts at test time (Table 4).

Critical weaknesses
  1. VPTwin-Core extends IWS with multi-rollout simulation conditioning, which I view as an incremental engineering contribution with insufficient methodological novelty. Ablations do not isolate the value of joint multi-rollout conditioning from increased training diversity. A single-rollout-conditioned baseline trained on the same rollout pool under a matched training budget would provide a more convincing comparison.

  2. The planning experiments lack an ablation that isolates the contribution of predicted-video verification. VPTwin also adds demonstration guidance and six planning substages relative to SIMPACT (Table 2; Appendix E), so it remains unclear how much of the improvement comes from video verification.

Questions for authors
  1. With the rollout pool and training budget fixed, does joint multi-rollout conditioning outperform conditioning on one randomly sampled rollout per training example? This would help separate the benefit of joint conditioning from training diversity.

  2. How does planning performance change when predicted-video verification is removed, while keeping demonstration guidance, the six substages, and the candidate-generation and refinement budget unchanged?

3 comments
AuthorsOpenReview ↗

Response to Reviewer QF38 (Part 1)

Thank you for the review. We trained the control you asked for and added the planning ablation. All numbers are in the revised PDF (changes in blue).

Q1. Novelty beyond adding rollout conditioning to IWS (Weakness 1).

We respectfully clarify that VPTwin introduces two distinct methodological contributions beyond IWS, rather than simply extending it with additional simulation inputs.

  1. First, physics-conditioned real-world video prediction: IWS predicts future observations from visual context and robot actions, without access to simulated future dynamics. VPTwin-Core introduces physics simulation as an additional source of predictive information, providing explicit hypotheses about object motion, contact, and articulation. The key motivation is to combine the physical structure provided by simulation with the visual realism learned from real-world data, rather than relying entirely on implicit dynamics learning from demonstrations.
  2. Second, multi-hypothesis conditioning under simulation uncertainty: A single simulation rollout can be misleading because physical properties such as friction, mass, and contact parameters cannot be reliably inferred from visual observations. Instead of committing to one potentially incorrect simulated future, VPTwin-Core jointly conditions on multiple rollouts generated under randomized physical parameters. This allows the predictor to exploit diverse physical hypotheses when forecasting real-world outcomes. As illustrated in Fig. 6B, VPTwin can predict real-world failures even when the simulated hypotheses disagree.

Q2. With the rollout pool and the training budget fixed, does joint multi-rollout conditioning outperform one randomly sampled rollout per training example? (Weakness 1, Question 1)

Yes (Table 4, App. K). We trained a single-rollout baseline that randomly samples one rollout from the same ten-rollout pool for each training example, with the same 10⁶ training steps. A single test rollout is also drawn at random (three draws, averaged). An outcome is correct if the predicted cup ends upright and on the correct slot, or if the predicted drawer is opened and then closed, as in all real videos of the test split.

Rollouts, training → testGrasp PSNRCorrect outcome (13 videos)Drawer PSNRCorrect outcome (10 videos)
None (IWS)23.6 dB622.5 dB7
1 → 1 (single-rollout baseline)26.7 dB1025.1 dB9
10 → 128.3 dB1126.4 dB9
10 → 10 (VPTwin-Core)28.3 dB1326.5 dB10

The results separate the contributions of simulation conditioning and of joint multi-rollout learning.

  • Simulation conditioning. Compared with IWS, single-rollout conditioning improves PSNR from 23.6 to 26.7 dB on Grasp Cup and from 22.5 to 25.1 dB on the drawer, and the correct outcomes from 6 to 10 and from 7 to 9.
  • Joint multi-rollout conditioning. With identical single-rollout test inputs, the jointly trained model further improves PSNR from 26.7 to 28.3 dB on Grasp Cup and from 25.1 to 26.4 dB on the drawer, and the correct grasp outcomes from 10 to 11.

The single-rollout and joint models train for the same number of steps using the same pool of ten rollouts per episode. For each training example, the single-rollout model samples one rollout from this pool at random, while the joint model uses all ten together. The gain from joint conditioning therefore cannot be explained by training-data diversity.

Fig. 6 (co-trained model of the planner) further shows why multi-rollout conditioning matters at test time. When different physical simulations produce conflicting outcomes for the same action, conditioning on a single rollout can lead to a misleading prediction. By jointly considering multiple plausible physical futures, VPTwin-Core still predicts the real outcome, including the real drawer failure of Fig. 6B. On the test videos of Table 4, the predicted outcome is correct for all 13 grasp and all 10 drawer videos with all ten rollouts, against 11 and 9 with one rollout. It does not simply take the majority either: in Fig. 6C only four of the ten rollouts move the cup to the center, yet the prediction from all ten moves it there, as in the real video. This is the practical motivation for multi-rollout conditioning: learning from uncertain and imperfect simulations rather than relying on a single physical hypothesis.

Together, these results support two complementary contributions: simulation conditioning provides physically grounded predictive information, and joint multi-rollout learning improves how the model exploits uncertain simulation hypotheses. We clarified these motivations and distinctions in the revised manuscript (Sec. 1, Sec. 4.4.2, App. K).

AuthorsOpenReview ↗

Response to Reviewer QF38 (Part 2)

Q3. Planning without predicted-video verification, with demonstration guidance, the six substages, and the budgets unchanged (Weakness 2, Question 2).

Added as SIMPACT+Substage (Table 3, Sec. 4.3, App. E and I). It runs our planner unchanged, and only the predicted-video feedback is removed: the judge and the refinement step see the nominal simulation only, and each later substage starts from the simulated instead of the predicted frame. Each planner planned and executed 24 real scenes, twice as many as before.

PlannerHardware successAcceptedAccepted failuresRejected successesPrecisionF1
SIMPACT5/24201500.250.40
SIMPACT+Substage9/24191110.420.57
VPTwin15/2417310.820.88

Without predicted-video verification, hardware success drops from 15/24 to 9/24 and the accepted failures rise from 3 to 11. The demonstration guidance, the six substages, and the candidate-generation and refinement budgets are identical in the two planners, so the difference is the contribution of VPTwin-Core's predicted-video feedback.

The ablation also shows why the predicted video is needed. With simulation-only verification, 11 of the 19 accepted plans fail on the robot, because the simulator does not capture the contact and friction variations of the real world. For this reason VPTwin commits an action only when both the simulation and the VPTwin-Core prediction pass, and its verdict before execution then matches the real outcome in 20 of 24 scenes instead of 12.

The verdicts are also more reliable on unseen placements. With the cup between the training slots (8 start images; Table 3, right), both plans that VPTwin accepts succeed on hardware (2/2), while SIMPACT+Substage accepts 2 plans that fail and SIMPACT accepts 6.

The predictor itself can be ablated in the same planner. With the predicted video as the only evidence of the judge on the 10 real drawer scenes, the failure co-trained predictor gives 8/10 hardware successes and the success-only predictor 0/10 (Table 3, bottom).

Summary. Both concerns are answered by matched experiments, and removing either contribution costs measurable performance: without joint rollout conditioning in training, PSNR drops from 28.3 to 26.7 dB on Grasp Cup and from 26.4 to 25.1 dB on the drawer under identical single-rollout test inputs (Table 4); without the predicted-video feedback in planning, hardware success drops from 15/24 to 9/24 and the accepted failures rise from 3 to 11 (Table 3). These are not incremental effects, and we would be grateful if the reviewer would reconsider the score in light of these results.

Thanks for your rebuttal! I have read it and will take it into consideration in the later discussion stage.

Reviewer_ZyjcOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

VPTwin is proposed to tackle the problem of unreliable action-conditioned video prediction for robotic manipulation, where learned predictors can generate physically implausible or overly optimistic outcomes and approximate simulators can misrepresent real interactions. Its contributions are threefold. First, a VLM authors executable digital twins from third-person demonstration keyframes and calibrated workspace information. Isaac Sim replays recorded robot actions under randomized physical parameters and object placements to generate digital cousins of the video demonstration. Second, VPTwin-Core conditions an IWS-derived latent video predictor on these time-aligned simulation rollouts. Real successful demonstrations supervise prediction, while additional synthetic failure co-training aims to reduce optimistic continuations. Third, a demonstration-guided staged planner proposes symbolic actions, converts them into joint trajectories through inverse kinematics, and verifies their outcomes using both simulation and predicted video. Rejected candidates are refined before the complete accepted trajectory is executed. The paper reports improved video prediction and encouraging preliminary hardware results.

Critical strengths
  1. VPTwin is a useful prediction mechanism with credible empirical support. Table 1 reports that VPTwin improves family-averaged PSNR by 2.99 dB and 3.32 dB in the measured-catalog and VLM-authored settings, respectively. Meanwhile, Appendix D specifies the matched optimizer, training budget, autoencoder, data splits, and action windows for the action-only comparator, and uses five training seeds. These results provide meaningful evidence for simulation-conditioned prediction as well as the comparison's reproducibility.

  2. VPTwin is a coherent system connecting prediction to real manipulation. The method integrates executable twin authoring, randomized physical contexts, synthetic failure supervision, and staged action refinement. VPTwin also examines context quantity, training diversity under matched inference contexts, failure supervision, and incorrect simulation outcomes. The hardware evaluation also executes rejected final plans separately to assess critic agreement, rather than assuming that every rejection corresponds to physical failure.

Critical weaknesses
  1. The incremental planning value of the predictor remains unclear. Appendix E gives VPTwin demonstration strips, cached motion guidance, six-stage decomposition, and predicted-video feedback, whereas SIMPACT searches and revises whole-skill plans without these additions. These are legitimate contributions of the complete framework, and Table 2 is a valid system-level comparison. However, it is unclear how much each system's component contributes to the better performance. This limits the evidence for the specific claim that learned real-domain forecasts improve physical action verification beyond an otherwise comparable simulator-based planner.

  2. The benefit of synthetic failure co-training lacks real-world grounding. Table 3, Figure 5, and Table 15 compare prediction of simulated failures, including simulated initial observations and targets. Appendix D.7 confirms that the co-trained predictor is used for hardware planning, but no matched success-only versus failure-co-trained hardware evaluation is reported. The complete system achieves 3/7 grasp successes and 5/5 drawer successes; its perfect critic precision reflects eight accepted plans. These are promising results, but they do not isolate whether synthetic failure supervision improves real failure anticipation or establish a general elimination of false acceptance. A controlled comparison between VPTwin with versus without failure co-training evaluated against real physical outcomes is missing.

Questions for authors
  1. To summarize in one sentence, could you provide a focused component ablation? With demonstration guidance, stage decomposition, action execution, and proposal/refinement budgets held comparable, how does the VPTwin planner using simulation-only feedback compare with the same planner augmented by VPTwin-Core? This comparison would clarify the predictor's additional value while preserving the other proposed planning contributions.

  2. To summarize in one sentence, Could you provide a further clarification of experimental results about the effectiveness of synthetic failure co-training’s real world grounding? Is there an evaluation of success-only versus failure-co-trained predictors on actual physical failures, starting from real observations? If initial images, joint-command logs, and videos were retained from the existing hardware evaluation, could both predictors be evaluated on the same executed trajectories with identical simulation contexts and judging criteria? Please report raw outcome counts, false acceptances, acceptance precision, and acceptance coverage. The separately executed rejected grasp plans and failed baseline executions may provide test cases if those records are available; I am not assuming that a reusable public trajectory dataset already exists.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Zyjc

Thank you for the careful review and for proposing concrete experimental designs. We ran the planning ablation you proposed and compared the two predictors on the real robot. All counts are in the revised PDF (changes in blue); coverage is the number of accepted plans divided by the number of scenes.

Q1. Component ablation: the same planner with simulation-only feedback versus with VPTwin-Core (Weakness 1, Question 1).

Added as SIMPACT+Substage (Table 3, Sec. 4.3, App. E and I). The demonstration strips, the cached motion notes, the six substages, action execution, and the proposal and refinement budgets are those of VPTwin. Only the predicted video is removed. Each planner planned and executed 24 real scenes, twice as many as before.

PlannerHardware successAccepted (coverage)False acceptancesFalse rejectionsPrecisionF1Verdict matches the real outcome
SIMPACT5/2420 (0.83)1500.250.409/24
SIMPACT+Substage9/2419 (0.79)1110.420.5712/24
VPTwin15/2417 (0.71)310.820.8820/24
  • Demonstrations and staging (SIMPACT → SIMPACT+Substage): 5/24 → 9/24.
  • Predicted video (SIMPACT+Substage → VPTwin): 9/24 → 15/24, with false acceptances falling from 11 to 3. Matched by scene, VPTwin succeeds where SIMPACT+Substage fails in 7 scenes, and the reverse happens once.
  • Precision of the judge at a larger scale. VPTwin now accepts 17 plans, more than twice as many as in the submission, and 14 of them succeed (precision 0.82), against 8 of 19 for the same planner without the predicted video (0.42) and 5 of 20 for SIMPACT (0.25). Its verdict matches the real outcome in 20 of 24 scenes, against 12 and 9. We report these raw counts in Table 3, Sec. 4.3, and App. E, and no longer state that false acceptances are eliminated.

Q2. Does synthetic failure co-training help against real physical outcomes? Could both predictors be evaluated under identical conditions, with raw counts, false acceptances, precision, and coverage? (Weakness 2, Question 2)

Yes. We compared the two predictors under identical conditions on the real robot, inside the planning loop: each predictor drives the same planner on the same 10 real drawer scenes, and every plan is executed (Table 3, bottom; Sec. 4.4.1; App. J). The scenes, the planner, the budgets, and the judging criteria are identical. The judge sees only the predicted video, which isolates the predictor, and every verdict is checked against a real outcome.

10 real drawer scenesSuccess-onlyFailure co-trained
Hardware success0/108/10
Accepted plans (coverage)8 (0.80)8 (0.80)
False acceptances82
False rejections02
Precision0.000.75
F10.000.75
  1. Failure detection. At the Pull Open substage, the judge accepts the first action in every scene from the success-only video, which can show a failed opening as an opened drawer (Sec. 4.4.1). From the co-trained video, which keeps the failure visible (Fig. 5), it rejects the incorrect action in every scene.
  2. Refine. Refinement starts only when a failure is detected. With the success-only video the action is never refined; with the co-trained video the judge accepts a refined action in every scene. A plan is accepted only if its other substages also pass, hence 8 accepted plans with either predictor.
  3. Real execution. Hardware success rises from 0/10 to 8/10. All 8 plans accepted with the success-only video fail on the robot, against 2 of 8 with co-training (precision 0.00 versus 0.75 at the same coverage).

With everything else fixed, synthetic failure co-training therefore changes the outcome on the real robot, and no real failure data is used for training. Beyond planning, VPTwin-Core also predicts real failures better than IWS: the real failure outcome of 15 of 21 recorded real failures, against 10 (App. M).

Summary. Both concerns are now answered on the real robot. In the matched planner, the predicted video raises hardware success from 9/24 to 15/24 and cuts the false acceptances from 11 to 3. With the predictor swapped on the 10 real drawer scenes, failure co-training raises hardware success from 0/10 to 8/10 and cuts the false acceptances from 8 to 2. We hope these results strengthen your assessment of the paper.

Reviewer_ZyjcOpenReview ↗

Reply to review

The reviewer thanks the authors for the clarifications. Most concernes have been addressed, and thus the rating remains positive and the confidence improves.

Reviewer_ZyjcOpenReview ↗

Reply to review

The reviewer thanks the authors for the clarifications. Most concernes have been addressed, and thus the rating remains positive and the confidence improves.

Reviewer_NxiYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes VPTwin, a Real-Sim-Real video prediction and robotic manipulation planning framework. The approach consists of three components:

  1. A VLM-assisted digital-twin construction pipeline that generates randomized physics rollouts in Isaac Sim.
  2. VPTwin-Core, an action-conditioned video prediction model incorporating time-aligned simulation rollouts as physical priors, together with synthetic failure co-training.
  3. A staged planning framework that uses predicted visual outcomes and simulation results to verify and refine VLM-proposed actions. Experiments show improvements in video prediction quality over an action-only baseline across six manipulation tasks. Real-world evaluations on two tasks demonstrate promising planning performance compared with SIMPACT and π0.5.
Critical strengths
  1. A promising integration of simulation-based physics priors and real-world video prediction. The idea of conditioning video prediction on multiple randomized simulation rollouts is meaningful, as it addresses the complementary limitations of purely learned world models and imperfect physical simulators. The video prediction experiments demonstrate consistent improvements over the action-only baseline, particularly in reducing temporal drift and preserving object and robot geometry.
  2. An end-to-end system with real-robot validation. The paper goes beyond evaluating video generation metrics and integrates the predictive model into robotic manipulation planning. The digital-twin construction pipeline, synthetic failure generation, and physical deployment demonstrate a reasonably complete system. The hardware experiments, although limited in scale, provide encouraging preliminary evidence of practical utility.
Critical weaknesses
  1. The real-world planning comparisons do not adequately isolate the contribution of VPTwin-Core. According to Appendix E, VPTwin uses demonstration-derived motion notes, teleoperation guidance, six-stage task decomposition, and per-stage action refinement. In contrast, the SIMPACT baseline performs whole-skill planning without the same demonstration guidance or staged decomposition. Consequently, the observed performance improvements cannot be confidently attributed to the proposed video prediction model rather than the additional planning structure and demonstration priors. A controlled comparison using identical planning architectures, candidate budgets, and demonstration information, differing only in the use of VPTwin-Core, is necessary to establish the central planning contribution. Furthermore, physical validation includes only 7 grasping trials and 5 drawer trials. This is insufficient to draw strong conclusions about general planning reliability. Although the method is described as closed-loop model predictive control, Appendix E indicates that the complete trajectory is planned before physical execution, without replanning from intermediate real-world observations.
  2. The evidence for physically grounded failure prediction and the necessity of multiple simulation hypotheses remains insufficient. The synthetic failure co-training results in Table 3 are evaluated against simulated failure trajectories rather than real-world failure videos. Thus, these experiments do not establish that the model can accurately predict unmodeled failures in real-world execution. Furthermore, Figure 6 suggests that much of the prediction improvement is already obtained from a single simulation rollout, while additional rollouts provide relatively limited inference-time gains. This raises questions about the specific contribution of multi-hypothesis physical reasoning beyond basic simulation conditioning. The paper primarily measures visual similarity using PSNR, LPIPS, and related metrics. More direct evaluation of physical consistency and failure-outcome accuracy would better support its central claims.
Questions for authors
  1. [Critical experiment] Can the authors provide a matched planning ablation in which both methods use identical demonstration guidance, substage decomposition, refinement budgets, and action generation, differing only in whether VPTwin-Core is used for verification? This would be essential for attributing planning improvements to the proposed predictor.
  2. [Critical clarification] Appendix B.3 states that digital-twin authoring uses demonstration frames up to 100% of the trajectory. For held-out test episodes, are the corresponding digital twins constructed using frames from those test episodes? If so, how is information from future ground-truth observations prevented from entering the prediction evaluation through the simulation context?
  3. [Critical experiment] How accurately does the failure-aware model predict actual real-world failures, rather than simulated failures? In addition, can the authors quantify the planning benefit of multiple randomized simulation rollouts compared with a single nominal simulation rollout under an otherwise identical setup?
3 comments
AuthorsOpenReview ↗

Response to Reviewer NxiY (Part 1)

Thank you for the detailed review. We address its three central concerns (the isolated contribution of VPTwin-Core, real failures, and multiple rollouts) with controlled experiments on the real robot and on real videos, and all three results favour VPTwin-Core:

  1. Isolated planning contribution. With matched demonstrations, substages, and budgets, adding VPTwin-Core raises hardware success from 9/24 to 15/24 and cuts the accepted failures from 11 to 3 (Q1).
  2. Real failures. On 21 recorded real failures, VPTwin-Core predicts the real failure outcome in 15, against 10 for IWS, and failure co-training raises hardware success on the 10 real drawer scenes from 0/10 to 8/10 (Q4).
  3. Multiple rollouts. With the predictor fixed, ten rollouts instead of one raise the correctly predicted outcomes from 11 to 13 of 13 real grasp test videos and from 9 to 10 of 10 real drawer test videos (Q5).

All changes are in the revised PDF (in blue).

Q1. A controlled comparison that differs only in the use of VPTwin-Core (Weakness 1, Question 1).

This comparison is now in the paper as SIMPACT+Substage (Table 3, Sec. 4.3, App. E and I). It runs our Algorithm 1 unchanged: the same demonstrations and motion notes, the same six substages, the same per-substage budget (one proposal plus three refinements), the same inverse kinematics, and the same success tests. Only the predicted-video feedback is removed: the judge and the refinement step see the nominal simulation only, without the predicted video, and each later substage starts from the simulated instead of the predicted frame. Each planner planned and executed 24 real scenes.

PlannerHardware successAcceptedAccepted failuresRejected successesPrecisionF1
SIMPACT5/24201500.250.40
SIMPACT+Substage9/24191110.420.57
VPTwin15/2417310.820.88

In the two staged planners the judge decides per substage, and a plan counts as accepted only when all six of its substages pass; SIMPACT judges the whole trajectory. There is therefore one verdict per scene and 24 per planner.

  • The gain is attributable to VPTwin-Core. With an identical planning architecture, candidate budget, and demonstrations, adding the predicted video raises hardware success from 9/24 to 15/24. Matched by scene, VPTwin succeeds where SIMPACT+Substage fails in 7 scenes, and the reverse happens once.
  • The simulation alone is not enough. Without the predicted video, the judge accepts 19 plans and 11 of them fail on the robot. The simulator does not capture the contact and friction variations of the real world. This observation is the basis of our design: an action is committed if and only if both the simulation and the VPTwin-Core prediction pass. With this rule the accepted failures drop from 11 to 3, and the verdict before execution matches the real outcome in 20 of 24 scenes instead of 12.

Q2. Physical validation with only 7 grasping and 5 drawer trials (Weakness 1).

The hardware study is now twice as large. Each of the 12 placements is set up twice, and every planner plans and executes all 24 scenes once; Table 20 (App. I) lists every execution with its verdict. The advantage of VPTwin holds at this scale: 15/24 against 9/24 and 5/24, and open-loop π0.5 fails in all 24 scenes.

Q3. "Closed-loop model predictive control" although the trajectory is planned before execution (Weakness 1).

By "closed-loop" we meant an iterative planning loop closed over predicted feedback, i.e., a prediction-feedback planning loop: at each substage the planner proposes, simulates, predicts, judges, and refines, and the loop closes over these predictions, not over live camera observations. We did not mean closed-loop control from real observations during execution; the complete trajectory is planned before the robot moves and executed once, as Algorithm 1 specifies.

Because the term invites this second reading, we replaced it. Sec. 3.3 is now titled "VPTwin-Based Staged Predictive Verification"; the Abstract, Sec. 1, and Fig. 1C describe a staged planner with predictive verification before execution, and "closed-loop" and "model predictive control" no longer describe the method anywhere in the paper. This changes the name, not the method or any result.

AuthorsOpenReview ↗

Response to Reviewer NxiY (Part 2)

Q4. Does the failure-aware model predict actual real-world failures, not only simulated ones? (Weakness 2, Question 3)

Yes. We recorded 21 real failures on the robot under teleoperation (16 grasp, 5 drawer) and predicted each one from its real first frame and its executed actions; both predictors are the failure co-trained models (Sec. 4.2, App. M).

Predicted failure outcome matches the real oneVPTwin-CoreIWS
Grasp Cup (16 recordings)11/167/16
Open and Close Drawer (5 recordings)4/53/5
Total (21 recordings)15/2110/21

The judge classifies the failure outcome of each video from the video alone: for the cup, left at its start, displaced, or fallen; for the drawer, never opened, left open, or opened and closed. On the real videos this classification agrees with the human labels for all 21 recordings.

Fig. 6B shows one of these recordings. The ten simulated rollouts of this real drawer failure conflict: four close the drawer and six leave it open. Conditioned on all ten, VPTwin-Core predicts the real failure: the drawer is left open.

The benefit of failure co-training is most pronounced during planning, when the simulated rollouts contain failure cues that a success-only predictor tends to ignore. We tested this on the real robot (Table 3, bottom; Sec. 4.4.1; App. J). The same planner plans the 10 real drawer scenes with the predicted video as the only evidence of the judge, with either predictor, and every plan is executed once.

10 real drawer scenes, video-only judgeSuccess-onlyFailure co-trained
Hardware success0/108/10
Accepted plans88
Accepted plans that fail82
Precision0.000.75

The difference arises at the Pull Open substage. With the success-only video, the judge accepts the first action in every scene, so the action is never refined. With the co-trained video, it rejects the incorrect action in every scene and accepts a refined one. A plan is accepted only if its other substages also pass, hence 8 accepted plans with either predictor. The success-only predictor can show a failed opening as an opened drawer (Sec. 4.4.1) or let the drawer disappear (Fig. 5); the co-trained predictor keeps the failure visible. No real failure data is used for training.

Q5. Do multiple randomized rollouts add anything beyond a single one? (Weakness 2, Question 3)

Yes. When the physical simulations of one action disagree, which happens because the reconstructed scene is uncertain and its physical parameters cannot be observed, a prediction from a single rollout depends on which rollout it is given. Given all ten rollouts, simulated under different physical parameters, the predicted outcome is correct on all 23 real grasp and drawer test videos. PSNR does not show this, because it is dominated by image quality (it saturates with one rollout, Fig. 9, formerly Fig. 6), so we measured the predicted outcome (Table 4, App. K).

We feed the recorded actions of the held-out real test videos to the predictors. A single test rollout is drawn at random, not fixed to the nominal one, and results are averaged over three draws. A grasp prediction is correct if the predicted cup ends upright and on the correct slot; a drawer prediction is correct if it shows the drawer opened and then closed, as in all real videos of the test split.

Rollouts, training → testGrasp PSNRCorrect outcome (13 videos)Drawer PSNRCorrect outcome (10 videos)
None (IWS)23.6 dB622.5 dB7
1 → 126.7 dB1025.1 dB9
10 → 128.3 dB1126.4 dB9
10 → 10 (VPTwin-Core)28.3 dB1326.5 dB10
  • At test time (rows 3 and 4, same model): with one random rollout the predicted outcome is correct in 11 of 13 grasp videos and 9 of 10 drawer videos; with all ten it is correct in all 13 and all 10.
  • In training (rows 2 and 3, same single-rollout test input): joint training on ten rollouts improves PSNR from 26.7 to 28.3 dB on Grasp Cup and from 25.1 to 26.4 dB on the drawer over training on one random rollout from the same pool with the same budget.

Fig. 6, made with the co-trained model of the planner, shows three real recordings whose rollouts disagree. Given one rollout, the predicted outcome depends on the rollout; given all ten, it matches the real outcome in all three. The prediction is not a majority vote: in Fig. 6C only four of the ten rollouts move the cup to the center, yet the prediction from all ten moves it to the center, as in the real video.

Planning. Inside the planner, the simulation check uses the nominal rollout, and VPTwin-Core adds a prediction conditioned on all ten randomized rollouts (Sec. 3.3). Adding this prediction cuts the accepted failures from 11 to 3 and raises hardware success from 9/24 to 15/24 (Table 3).

AuthorsOpenReview ↗

Response to Reviewer NxiY (Part 3)

Q6. Physical consistency and failure-outcome accuracy, beyond PSNR and LPIPS (Weakness 2).

Added as Table 2, on all six tasks. Every event is measured automatically by the same rule-based detector (color segmentation and fixed thresholds) in the predicted and in the real video of each test episode, and an error is the absolute difference between the two, averaged over all test videos of the task (App. L).

Task (real test videos)Physical eventVPTwin-CoreIWS
Grasp Cup (13)lift timing0.14 s1.32 s
Open and Close Drawer (10)opening profile0.53 cm1.37 cm
Wipe Table (10)contact onset0.08 s0.42 s
Place Cube into Drawer (6)opening profile0.57 cm1.30 cm
Pickup Cup and Cloth (10)cloth removal0.82 s1.06 s
Remove Cloth (5)cup reveal0.12 s0.23 s

VPTwin-Core predicts the physical event of every task more accurately than IWS. The outcome-level evaluations point the same way.

EvaluationMeasureVPTwin-CoreIWS
Real test videos, grasp / drawer (Table 4)predicted task outcome correct13/13; 10/106/13; 7/10
21 recorded real failures (App. M)predicted failure outcome correct15/2110/21
8 real off-grid grasp recordings (App. N)median final cup-position error2.0 cm12.0 cm

The prediction is therefore closer to the real physical events than IWS on event timing, articulation, task outcome, real failures, and unseen placements.

Q7. Are the twins of held-out test episodes built from frames of those episodes? Can future ground-truth frames reach the prediction through the simulation context? (Question 2)

No. All twins are authored from training-split episodes, so no frame of a test episode is used to build a twin. The keyframes up to 100% that App. B.3 mentions are those of the authoring episode, which belongs to the training split. The revision states it in Sec. 3.1 and App. B.3 ("Provenance").

A twin is authored once per initial-layout variant of a task (24 variants, App. B.3), not per episode. It holds static assets only (geometry, collision shapes, mass and friction ranges, articulation) and is shared by all episodes of that variant. To simulate a test episode, the twin places the objects as in the first frame of the episode and replays its recorded actions on ten seeds. The first frame and the actions are the inputs that every action-conditioned predictor receives, IWS included. The later frames of a test episode are not used and serve only as the prediction target, so no future ground-truth observation enters the simulation context.

Summary. The concerns behind the soundness assessment are now answered by direct measurements. With matched demonstrations, substages, and budgets, VPTwin-Core raises hardware success from 9/24 to 15/24 and cuts the accepted failures from 11 to 3. On the 10 real drawer scenes, failure co-training raises hardware success from 0/10 to 8/10. VPTwin-Core predicts the physical events of all six tasks more accurately than IWS, and the real failure outcome in 15 of 21 recorded real failures, against 10. We would be grateful if the reviewer would reconsider the score in light of these results.

Reviewer_3UAVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  1. This paper presents VPTwin, a system that combines a digital twin of a real scene with an action-conditioned video prediction model. Before the robot acts, the system predicts what may happen and uses the predictions to choose and improve its plan.

  2. The method uses a VLM to build a simulation of the task environment and objects. It simulates the same action under different physical settings and uses the resulting videos to predict what would happen in the real world. The model is also trained on simulated failure examples, so it can learn to predict the outcomes of unsuccessful actions.

  3. The experiments cover six video prediction task categories and two real-robot planning tasks. VPTwin performs better than the compared video prediction and planning methods in these settings.

Critical strengths
  1. Predicting failures before a robot acts is a useful goal. This approach could reduce the need for trial and error on real hardware.

  2. The method uses multiple simulations to account for uncertainty in physical properties. It also uses simulated failures to reduce the model’s tendency to predict success after learning from successful demonstrations. These are reasonable choices for the problem.

  3. The video predictor performs better than the IWS baseline trained under matched conditions. The full system also performs better than SIMPACT on two real-robot tasks. The authors execute rejected plans as well, which provides useful evidence for evaluating the system’s judgments.

Critical weaknesses
  1. The paper has discussed RealWonder in related work section and explained that VPTwin differs by using multiple simulations of the same action, with different physical settings, to predict a real-world video. However, some related work is not cited, discussed, or compared in the paper. WonderPlay [2] also combines physics simulation and video generation to create dynamic 3D scenes from a single image. VLMPC [1] predicts the outcomes of candidate actions in video and uses those predictions to choose robot actions. Please discuss these works and explain more clearly what VPTwin propose, how it uses simulation and real video data differently, and how it evaluates actions. [1] VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation [2] WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions

  2. Appendix shows that VPTwin differs from SIMPACT in several ways. Besides adding video prediction, VPTwin uses demonstrations and motion notes, and divides the task into six planning stages. These other changes may also improve the success rate. Table 2 therefore shows that the full VPTwin system works better, but it does not show how much of the improvement comes from video prediction. The authors could examine the same actions that have already been executed on the robot. For each action, they could compare a judgment based only on simulation with a judgment based on both simulation and predicted video, then check both against the real outcome. For example, if simulation predicts success but the robot fails, does the video help identify the failure in advance? This would show whether video prediction provides useful information beyond simulation.

  3. As I understand it, the paper trains the video prediction model using successful real demonstrations and simulated failures. The model then helps the planner choose and revise candidate actions. For each candidate, the planner runs multiple simulations, predicts a video, and asks a VLM to judge the result. If the action is revised, these steps are repeated. According to Appendix E, each candidate uses ten randomized simulations, and the initial plan at each stage can be revised up to three times. This may take considerable time, but the paper does not report how long it takes to evaluate one candidate or plan a complete task. Please report the time needed for data collection, simulation setup, simulated data generation, and model training. Please also report the time to evaluate one candidate and plan a full task. The hardware, use of parallel computation, and average numbers of candidates and revisions per task would also help readers understand the cost of the method.

  4. Section 3.3 describes the method as “closed-loop / model predictive control.” However, Algorithm 1 appears to plan all six stages before the robot executes the full trajectory. To plan each new stage, the system uses the previous stage’s predicted final frame and simulated state. It does not appear to use a new camera image from the robot after each stage. This distinction matters because “closed-loop” may suggest that the robot observes the real result of each action and adjusts its plan during execution. Please clarify what feedback the loop uses and whether the robot replans from new observations while executing the task.

Questions for authors
  1. What does VPTwin add beyond RealWonder, VLMPC, and WonderPlay? On the same set of actions, can predicted videos reveal failures that simulation misses?

  2. How long do data preparation, model training, candidate evaluation, and full-task planning take? What hardware and parallel computation are used?

  3. What does “closed-loop” mean here? Does the robot replan using new observations during execution?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3UAV (Part 1)

Thank you for the constructive review. Changes in the revised PDF are marked in blue.

Q1. Relation to RealWonder, WonderPlay, and VLMPC; what VPTwin proposes (Weakness 1, Question 1).

We added WonderPlay and VLMPC to Sec. 2.

  • RealWonder and WonderPlay use a physics solver to drive a video or scene generator. Their output is generated content, and no robot action is verified.
  • VLMPC scores candidate action sequences with a VLM-assisted cost on frames from a learned video predictor; no physics simulation checks the actions, which leaves it vulnerable to physical hallucinations.
  • VPTwin differs in three ways. How simulation is used: the same candidate action is rolled out in a VLM-authored twin under one nominal and nine randomized physical configurations, and the real-domain prediction is conditioned on all ten, so the simulation is a set of hypotheses, not a single answer. How real and simulated data are used: real videos supervise the successful episodes, and simulation provides the conditioning rollouts and the failure targets that real data lack. How actions are evaluated: in the full planner, an action is committed only if both the nominal simulation and the predicted real-domain video pass; otherwise the predicted video guides its refinement.

Q2. How much of the improvement comes from the video prediction? If the simulation predicts success but the robot fails, does the video identify the failure in advance? (Weakness 2, Question 1)

Most of the planning gain comes from the predicted video, and the video flagged a failure that the nominal simulation passed (at (−13.5, 0), below). SIMPACT+Substage is our planner with the same demonstrations, six substages, and budgets; only the predicted-video feedback is removed, so its judge and its refinement step see the simulation alone, and each later substage starts from the simulated frame instead of the predicted one (App. E). Each planner planned and executed 24 real scenes. Rejected plans were also executed, so every verdict is checked against the real outcome.

Planner (evidence shown to the judge)Hardware successAccepted plans that failVerdict matches the real outcomeReal failures rejected before execution
SIMPACT (simulation)5/2415 of 209/244 of 19
SIMPACT+Substage (simulation)9/2411 of 1912/244 of 15
VPTwin (simulation + predicted video)15/243 of 1720/246 of 9
  • How much comes from the video. Staging and demonstrations give 5/24 → 9/24, and the predicted video 9/24 → 15/24. On the same scenes, VPTwin succeeds in 7 scenes in which SIMPACT+Substage fails, and the reverse happens once.
  • Judgement before execution, checked against the real outcome. The verdict matches the real outcome in 20 of 24 scenes instead of 12, and the accepted plans that fail drop from 11 of 19 to 3 of 17 (precision 0.42 → 0.82).
  • Failures that the simulation passes. With the cup between the training slots, VPTwin accepts 2 plans and both succeed, while the simulation-only planners accept 5 (2 fail) and 8 (6 fail) (Table 3, right; App. N). At (−13.5, 0), VPTwin's candidate passed the simulation, but the predicted video showed the cup carried away or tilted in the last two substages; VPTwin rejected the plan, and it failed on hardware, as did the plans that both simulation-only planners accepted there.

What the predicted video adds to the simulation.

(1) Real-domain dynamics (Sec. 4.2, App. L). Against the real test videos, the lift-timing error is 0.14 s for the prediction and 0.44 s for the nominal simulation it is conditioned on; the drawer opening error is 0.53 cm against 0.95 cm.

(2) The correct outcome when the rollouts disagree (Table 4, Fig. 6). Physical parameters cannot be observed exactly, so the ten rollouts of one action can end differently. Conditioned on all ten, VPTwin-Core predicts the correct outcome for all 13 real grasp and all 10 real drawer test videos. In Fig. 6B (co-trained model of the planner), a recorded real drawer failure, four rollouts close the drawer, six leave it open, and the prediction shows the real failure.

(3) Feedback for refinement (App. E; Table 3). Both staged planners refine a rejected action with the same budget; the refinement step sees the failed simulation stills in SIMPACT+Substage and, in addition, the failed predicted stills in VPTwin. A refinement can only repair a failure that its evidence shows. The predictor swap on the 10 real drawer scenes (Table 3, bottom; the judge sees only the predicted video) shows this: with the success-only predictor, the judge accepts the first Pull Open action in every scene, so it is never refined (0/10 succeed); with the failure co-trained predictor, it rejects the incorrect action in every scene and accepts a refined one (8/10 succeed).

AuthorsOpenReview ↗

Response to Reviewer 3UAV (Part 2)

Q3. Time and cost (Weakness 3, Question 2).

We added a compute-and-time appendix (App. O, Table 23). It separates the costs paid once per task from those paid for every new scene.

Once per taskCost
Teleoperation data15 s × 91 episodes ≈ 23 min of recorded video per task on average (7–40 min); 2.3 h in total for the six tasks
Evaluation-only recordings21 real failures: 7.1 min; 8 off-grid grasps: ≈ 2.4 min
Twin authoring, VLM call 1 (scene parsing)≈ 2 min, one call for all tasks
Twin authoring, VLM call 2 (asset authoring)≈ 7 min, one call for all tasks
Training (10⁶ steps, one RTX 3080 Ti)VPTwin-Core 46 h; IWS 41 h
On-robot explorationnone (demonstrations only)
Once per sceneCost
Twin of a new scenethe object placement is read from the start image
Prediction of one video (one RTX 4090)VPTwin-Core 15.3 s; IWS 14.1 s
Planning one scene (one RTX 4090, median over the 24 scenes)VPTwinSIMPACT+SubstageSIMPACT
Time119 min113 min38 min
Evaluating one candidate16 min15 min2.6 min
Simulated rollouts per candidate10101

Where the planning time goes. The predicted video is not the bottleneck: VPTwin plans a scene in 119 min, and SIMPACT+Substage, which simulates the same ten rollouts per candidate without the video, in 113 min. Of the 16 min that VPTwin spends evaluating one candidate, about 14 min are spent on the ten simulated rollouts, while the video prediction takes about 15 s, a VLM proposal about 30 s, and a VLM judgement 13–25 s.

Simulation and parallel computation. One rollout alone takes 0.7 s per simulated frame, 2–3 min for a full trajectory; a substage candidate is simulated from the start of the episode with the committed substages replayed, so it costs about as much as a full trajectory. The ten rollouts of a candidate run in parallel on one GPU; they are independent and can be spread over several GPUs.

Candidate and revision budget. Each of the six substages has a budget of one proposal plus up to three refinements, identical for VPTwin and SIMPACT+Substage. SIMPACT samples ten whole-trajectory plans and allows up to three revisions (App. E).

When it is paid. Each scene is planned independently, before execution. The substage definitions of a task are computed once and reused for its later scenes. VPTwin needs no exploration on the robot. For reference, RAX (Huang et al., 2026b), which learns a task by online exploration on the real robot, reports robot sessions of 2.5–6 h per task (e.g., 3.5 h for wiping a stain).

Q4. What does "closed-loop" mean? Does the robot replan from new observations during execution? (Weakness 4, Question 3)

By "closed-loop" we meant an iterative planning loop closed over predicted feedback, i.e., a prediction-feedback planning loop: at each substage the planner proposes an action, simulates it, predicts the real-domain video, judges it, and refines it up to three times, and the loop closes over these predictions, not over live camera observations (Sec. 3.3). We did not mean closed-loop control from real observations during execution. The robot does not replan during execution: all six substages are planned before the robot moves, each later substage starts from the last predicted frame and the nominal simulator state of the committed candidate, and the complete trajectory is then executed once.

Because the term invites the second reading, we replaced it. Sec. 3.3 is now titled "VPTwin-Based Staged Predictive Verification"; the Abstract, Sec. 1, and Fig. 1C describe a staged planner with predictive verification before execution, and "closed-loop" and "model predictive control" no longer describe the method anywhere in the paper.

Summary. With matched demonstrations, substages, and budgets, the predicted video raises hardware success from 9/24 to 15/24 and catches a failure that the nominal simulation passes; planning time is dominated by simulation, which can be spread over several GPUs; and "closed-loop" is replaced by staged predictive verification. We hope these results strengthen your assessment of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the area chair and reviewers for their careful evaluation of VPTwin. In response to their feedback, we expanded the hardware study, added controls for video prediction and rollout conditioning, and evaluated physical outcomes on held-out real videos. The table summarizes the revisions and where to find them. Changes are marked in blue in the revised manuscript.

RevisionWhat we added or clarifiedLocation in revised manuscript
Expanded hardware study and matched planning ablationWe doubled the study to 24 scenes and added SIMPACT+Substage, which uses the same demonstrations, six substages, and refinement budget as VPTwin. Adding predicted-video feedback increased hardware successes from 9/24 to 15/24 and reduced accepted plans that failed on the robot from 11 to 3.Sec. 4.3, Tables 3 and 20, Apps. E and I
Failure co-training in the plannerOn the same 10 drawer scenes, we swapped the predictor while keeping the planner fixed. The judge used the start image and predicted frames. Hardware success increased from 0/10 with the success-only predictor to 8/10 with the predictor co-trained on synthetic failures.Sec. 4.4.1, Table 3 (bottom), App. J
One versus ten simulation rolloutsWe compared training on one randomly sampled rollout per example with training on all ten, using the same pool and number of steps. Joint training improved PSNR on both tasks with one test rollout. Giving the jointly trained model all ten at test time raised correct outcomes from 11/13 to 13/13 on Grasp Cup and from 9/10 to 10/10 on Open and Close Drawer.Sec. 4.4.2, Table 4, Fig. 6, App. K
Physical consistency across all six tasksWe added task-specific event measurements against held-out real videos. VPTwin-Core had lower error than IWS on all six tasks, including cup lift timing (0.14 vs. 1.32 s) and drawer opening profile (0.53 vs. 1.37 cm).Sec. 4.2, Table 2, App. L
Prediction of recorded real failuresWe recorded 21 real failures for evaluation. Given the first frame and recorded actions, VPTwin-Core predicted the observed failure outcome in 15/21 recordings, compared with 10/21 for IWS.Sec. 4.2, Fig. 6B, Table 21, App. M
Generalization to unseen cup placementsOn eight off-grid recordings, median final cup-position error was 2.0 cm for VPTwin-Core and 12.0 cm for IWS. In hardware planning from the eight start images, both plans accepted by VPTwin succeeded.Sec. 4.3, Table 3 (right), Fig. 10, Table 22, App. N
Twin construction and planning protocolWe clarified that the twin assets are authored from training episodes. At test time, the first frame initializes object placement. We also clarified that the planner verifies and refines actions before execution and that the robot executes the resulting trajectory without online replanning.Secs. 3.1 and 3.3, Fig. 1C, Apps. B.3 and E
Compute and planning timeWe added data-collection and training costs, video prediction time, and planning time for all three planners. Median planning time per scene was 119 min for VPTwin, 113 min for SIMPACT+Substage, and 38 min for SIMPACT.Table 23, App. O
Relation to prior workWe expanded the comparison with RealWonder, WonderPlay, and VLMPC to clarify how VPTwin uses several simulated outcomes of the same candidate action to predict and evaluate its real-world consequences.Sec. 2

We address each reviewer's questions in the accompanying responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.