- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents VPTwin, a Real-Sim-Real video prediction framework that grounds future prediction in executable digital twins reconstructed by a VLM from real demonstrations. VPTwin-Core uses randomized, time-aligned simulation rollouts as physical context and incorporates synthetic failure co-training to model unsuccessful action outcomes. It then serves as a predictive visual verifier in staged planning, guiding the evaluation and refinement of VLM-proposed actions before real-world execution. Experiments report improved video prediction across rigid, deformable, and articulated manipulation tasks, alongside improved planning performance on two real-robot tasks.
Critical strengths
Simulation rollouts improve real-domain video prediction. VPTwin outperforms IWS in both settings, improving PSNR by about 3 dB and reducing LPIPS (Table 1). Training with ten rollouts also outperforms training with three when both models use the same three rollouts at test time (Table 4).
Critical weaknesses
-
VPTwin-Core extends IWS with multi-rollout simulation conditioning, which I view as an incremental engineering contribution with insufficient methodological novelty. Ablations do not isolate the value of joint multi-rollout conditioning from increased training diversity. A single-rollout-conditioned baseline trained on the same rollout pool under a matched training budget would provide a more convincing comparison.
-
The planning experiments lack an ablation that isolates the contribution of predicted-video verification. VPTwin also adds demonstration guidance and six planning substages relative to SIMPACT (Table 2; Appendix E), so it remains unclear how much of the improvement comes from video verification.
Questions for authors
-
With the rollout pool and training budget fixed, does joint multi-rollout conditioning outperform conditioning on one randomly sampled rollout per training example? This would help separate the benefit of joint conditioning from training diversity.
-
How does planning performance change when predicted-video verification is removed, while keeping demonstration guidance, the six substages, and the candidate-generation and refinement budget unchanged?
3 comments
Response to Reviewer QF38 (Part 2)
Q3. Planning without predicted-video verification, with demonstration guidance, the six substages, and the budgets unchanged (Weakness 2, Question 2).
Added as SIMPACT+Substage (Table 3, Sec. 4.3, App. E and I). It runs our planner unchanged, and only the predicted-video feedback is removed: the judge and the refinement step see the nominal simulation only, and each later substage starts from the simulated instead of the predicted frame. Each planner planned and executed 24 real scenes, twice as many as before.
| Planner | Hardware success | Accepted | Accepted failures | Rejected successes | Precision | F1 |
|---|---|---|---|---|---|---|
| SIMPACT | 5/24 | 20 | 15 | 0 | 0.25 | 0.40 |
| SIMPACT+Substage | 9/24 | 19 | 11 | 1 | 0.42 | 0.57 |
| VPTwin | 15/24 | 17 | 3 | 1 | 0.82 | 0.88 |
Without predicted-video verification, hardware success drops from 15/24 to 9/24 and the accepted failures rise from 3 to 11. The demonstration guidance, the six substages, and the candidate-generation and refinement budgets are identical in the two planners, so the difference is the contribution of VPTwin-Core's predicted-video feedback.
The ablation also shows why the predicted video is needed. With simulation-only verification, 11 of the 19 accepted plans fail on the robot, because the simulator does not capture the contact and friction variations of the real world. For this reason VPTwin commits an action only when both the simulation and the VPTwin-Core prediction pass, and its verdict before execution then matches the real outcome in 20 of 24 scenes instead of 12.
The verdicts are also more reliable on unseen placements. With the cup between the training slots (8 start images; Table 3, right), both plans that VPTwin accepts succeed on hardware (2/2), while SIMPACT+Substage accepts 2 plans that fail and SIMPACT accepts 6.
The predictor itself can be ablated in the same planner. With the predicted video as the only evidence of the judge on the 10 real drawer scenes, the failure co-trained predictor gives 8/10 hardware successes and the success-only predictor 0/10 (Table 3, bottom).
Summary. Both concerns are answered by matched experiments, and removing either contribution costs measurable performance: without joint rollout conditioning in training, PSNR drops from 28.3 to 26.7 dB on Grasp Cup and from 26.4 to 25.1 dB on the drawer under identical single-rollout test inputs (Table 4); without the predicted-video feedback in planning, hardware success drops from 15/24 to 9/24 and the accepted failures rise from 3 to 11 (Table 3). These are not incremental effects, and we would be grateful if the reviewer would reconsider the score in light of these results.
Thanks for your rebuttal! I have read it and will take it into consideration in the later discussion stage.
Response to Reviewer QF38 (Part 1)
Thank you for the review. We trained the control you asked for and added the planning ablation. All numbers are in the revised PDF (changes in blue).
Q1. Novelty beyond adding rollout conditioning to IWS (Weakness 1).
We respectfully clarify that VPTwin introduces two distinct methodological contributions beyond IWS, rather than simply extending it with additional simulation inputs.
Q2. With the rollout pool and the training budget fixed, does joint multi-rollout conditioning outperform one randomly sampled rollout per training example? (Weakness 1, Question 1)
Yes (Table 4, App. K). We trained a single-rollout baseline that randomly samples one rollout from the same ten-rollout pool for each training example, with the same 10⁶ training steps. A single test rollout is also drawn at random (three draws, averaged). An outcome is correct if the predicted cup ends upright and on the correct slot, or if the predicted drawer is opened and then closed, as in all real videos of the test split.
The results separate the contributions of simulation conditioning and of joint multi-rollout learning.
The single-rollout and joint models train for the same number of steps using the same pool of ten rollouts per episode. For each training example, the single-rollout model samples one rollout from this pool at random, while the joint model uses all ten together. The gain from joint conditioning therefore cannot be explained by training-data diversity.
Fig. 6 (co-trained model of the planner) further shows why multi-rollout conditioning matters at test time. When different physical simulations produce conflicting outcomes for the same action, conditioning on a single rollout can lead to a misleading prediction. By jointly considering multiple plausible physical futures, VPTwin-Core still predicts the real outcome, including the real drawer failure of Fig. 6B. On the test videos of Table 4, the predicted outcome is correct for all 13 grasp and all 10 drawer videos with all ten rollouts, against 11 and 9 with one rollout. It does not simply take the majority either: in Fig. 6C only four of the ten rollouts move the cup to the center, yet the prediction from all ten moves it there, as in the real video. This is the practical motivation for multi-rollout conditioning: learning from uncertain and imperfect simulations rather than relying on a single physical hypothesis.
Together, these results support two complementary contributions: simulation conditioning provides physically grounded predictive information, and joint multi-rollout learning improves how the model exploits uncertain simulation hypotheses. We clarified these motivations and distinctions in the revised manuscript (Sec. 1, Sec. 4.4.2, App. K).