- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PhysCanvas is an action-conditioned visual–tactile world model for robot policy evaluation and learning. It represents multi-view RGB, tactile images, marker displacement, and task state in a unified canvas processed by a shared pretrained video backbone. Consistency-aware post-training uses scene, modality, and state reward models to encourage scene preservation, visual–tactile compatibility, and agreement with observed task outcomes. Across four manipulation tasks, adding state supervision to the model with scene-consistency supervision (M2 → M3) increases the reported mean task-wise correlation between simulated and real-world success rates from 0.602 to 0.885. Policy optimization in the full simulator improves π0.5’s mean real-world success rate from approximately 59.1% after SFT to 70.8%, a gain of 11.7 percentage points. The additional gain over policy optimization in the world model without consistency post-training is approximately 3.3 percentage points, from 67.5% to 70.8%.
Critical strengths
- An end-to-end pipeline with real-world validation. The paper connects multimodal world-model training, closed-loop imagined interaction, and policy optimization to evaluation on a physical robot. The reported improvements over SFT provide evidence that learning in the simulator can yield benefits beyond its internal reward scores.
- A unified representation that reuses video pretraining. The canvas places visual observations, tactile images, marker displacement, and task state within a shared visual representation. This provides a concrete way to reuse pretrained video models while generating the tactile displacement signals consumed by the evaluated policies.
- Consistency supervision with measurable benefits. The state objective is designed to match both successful and unsuccessful outcomes rather than uniformly favoring successful outcomes. Separately, adding modality-consistency supervision (M3 → M4) reduces all three marker-displacement errors, although contact-event F1 does not improve. The full consistency-post-trained simulator also improves average downstream performance, with gains varying across tasks.
Critical weaknesses
- The policy-evaluation protocol does not fully establish reliability. Figure 6 does not clearly specify how simulated success is determined. If it uses the State RM employed during training, independent validation is needed to distinguish improved outcome prediction from adaptation to that evaluator. Moreover, correlations pooled across three policy families do not establish reliable comparison of different versions of the same policy.
- The contributions of touch and the unified canvas are not isolated. The paper does not report RGB-only or separate tactile-branch comparisons under matched conditions, particularly for policy evaluation and real-world policy improvement. The experiments demonstrate the feasibility of the complete multimodal system, but do not establish the incremental benefit of tactile inputs or the advantage of the shared canvas over alternative architectures.
- The stability of the additional real-world gain is difficult to assess. Table 4 reports means and population standard deviations over three runs, but does not clarify whether these represent independent training runs or repeated evaluations of a fixed policy. The number of real-world trials for this experiment is also not explicitly stated. Consequently, the approximately 3.3-percentage-point average gain over the ordinary world-model simulator is difficult to assess relative to training and evaluation variability.
Questions for authors
- How is success determined for the simulated rollouts in Figure 6? If the State RM used during training also provides these judgments, is there any independent validation, such as human assessment, of the simulated outcomes?
- Do you have any existing RGB-only or separate tactile-branch comparison results, particularly for policy evaluation or real-world policy improvement, that could help isolate the contributions of tactile inputs and the unified canvas?
- What do the three runs in Table 4 represent: independent policy-training runs or repeated evaluations of a fixed trained policy? Please also clarify the number of real-world trials per task and method in each run, whether methods were evaluated under the same initial conditions, and how the evaluated checkpoints were selected.
5 comments
W2 & Q2 Ablation of the tactile branch
Thank you for the suggestion. We examine tactile input from three perspectives: visual generation quality (Table H3), world-model-based policy evaluation (Table H4), and policy improvement through RL (Table H5). Because the latter two require substantial real-robot testing and human assessment, we focus on the Test Tube and Bottle tasks.
Visual generation quality. Removing tactile input reduces performance on several visual metrics, including PSNR, SSIM, LPIPS, FID, and FVD. Nevertheless, RGB-only PhysCanvas remains competitive and outperforms all listed baselines on these five metrics (Table H3).
World-model-based policy evaluation. In Table H4, we average the three batches within each task–policy setting and compute the Pearson correlation between world-model and real-world success rates across the six resulting pairs. The correlation is 0.76 for RGB-only and 0.90 for RGB+Tactile, indicating closer agreement with real-world success rates when tactile input is included.
Policy improvement through RL. Table H5 reports real-world success rates for SFT and RL policies with and without tactile input. Averaged across the four task–policy settings, RGB-only RL improves success from 37.48% to 42.48%, a gain of 5.00 percentage points. RGB+Tactile RL improves success from 39.58% to 45.40%, a gain of 5.83 percentage points. RGB+Tactile RL achieves a higher mean success rate than RGB-only RL (45.40% vs. 42.48%), a difference of 2.93 percentage points. These results suggest that tactile input can improve downstream policy performance, while RL remains beneficial without tactile input.
Table H3. Visual generation metrics for RGB-only PhysCanvas, RGB+Tactile PhysCanvas, and baselines.
| Model | PSNR ↑ | SSIM ↑ | LPIPS ↓ | FID ↓ | FVD ↓ | Subject ↑ | Bg. ↑ | Smooth ↑ | EEF (px) ↓ | Object (px) ↓ |
|---|---|---|---|---|---|---|---|---|---|---|
| EVAC | 16.439 | 0.528 | 0.258 | 26.512 | 165.705 | 79.223 | 90.456 | 97.731 | 1.964 | 2.15 |
| Open-Sora | 17.916 | 0.599 | 0.228 | 34.762 | 183.86 | 78.519 | 90.234 | 98.157 | 1.425 | 1.426 |
| WoVR | 17.278 | 0.5972 | 0.203 | 26.086 | 138.243 | 80.247 | 91.082 | 98.059 | 2.792 | 1.164 |
| PhysCanvas RGB-only | 20.502 | 0.71 | 0.141 | 24.812 | 128.103 | 80.248 | 90.419 | 98.39 | 0.844 | 1.438 |
| PhysCanvas RGB+Tactile | 21.172 | 0.735 | 0.121 | 21.459 | 116.274 | 80.349 | 90.565 | 98.301 | 0.685 | 1.15 |
Table H4. Tactile-input ablation for world-model-based policy evaluation.
| Method | Test Tube, ACT | Test Tube, FM | Test Tube, Pi0.5 | Bottle, ACT | Bottle, FM | Bottle, Pi0.5 |
|---|---|---|---|---|---|---|
| Real World (RGB only) | 15%, 5%, 0% | 0%, 0%, 0% | 45%, 55%, 45% | 50%, 40%, 40% | 20%, 15%, 25% | 80%, 80%, 85% |
| Eval in WM (RGB only) | 40%, 10%, 10% | 5%, 30%, 20% | 55%, 70%, 70% | 30%, 25%, 30% | 45%, 30%, 45% | 55%, 50%, 60% |
| Real World (RGB+Tactile) | 15%, 10%, 0% | 0%, 0%, 0% | 45%, 60%, 45% | 55%, 30%, 35% | 20%, 15%, 25% | 85%, 85%, 95% |
| Eval in WM (RGB+Tactile) | 30%, 15%, 20% | 5%, 25%, 15% | 60%, 60%, 55% | 50%, 20%, 25% | 35%, 25%, 40% | 50%, 75%, 55% |
Table H5. Real-world success rates (%) of SFT and RL policies with and without tactile input.
| Method | Test Tube, FM | Test Tube, Pi0.5 | Bottle, FM | Bottle, Pi0.5 |
|---|---|---|---|---|
| RGB-only SFT | ||||
| RGB-only RL | ||||
| RGB+Tactile SFT | ||||
| RGB+Tactile RL |
W3 & Q3 Experimental protocol and policy improvement
Real-world evaluation protocol and checkpoint sensitivity. We apologize for not describing the experimental settings clearly. For each task and policy, we evaluate a fixed checkpoint over 60 real-world trials, divided into three batches of 20 trials conducted during separate, fixed time periods. We use default RL checkpoints obtained after 20 RL updates. Table H6 additionally examines checkpoints obtained after 20, 30, and 40 updates on the Test Tube and Bottle tasks. The average success rate increases slightly, which we attribute to normal variation across checkpoints.
Table H6. Real-world policy success rates with default and multiple RL checkpoints.
| Method | Test Tube, FM | Test Tube, Pi0.5 | Bottle, FM | Bottle, Pi0.5 |
|---|---|---|---|---|
| RL in WM (default) | ||||
| RL in WM+RM (default) | ||||
| RL in WM (multi ckpts) | ||||
| RL in WM+RM (multi ckpts) |
Modest additional gains for Pi0.5. In the aggregate results cited from the manuscript, Pi0.5 improves from 59.17% to 67.50% after RL, a gain of 8.33 percentage points. The additional reported gain from “RL in WM” to “RL in WM+RM” is 3.3 percentage points. Pi0.5 already achieves a relatively high success rate after RL in the standard world model. Further improvements in world-model generation quality may therefore yield diminishing returns, resulting in only modest gains in policy success rate.
Summary
Thank you again for your constructive feedback. Our responses address three main points:
- Independent evaluation: We clarify human assessment independent of reward-model predictions and report policy-specific correlations (Table H1).
- Tactile contribution: Tactile input improves several generation metrics, evaluation correlation, and mean RL success (Tables H3–H5).
- Protocol and robustness: We clarify the 60-trial protocol, examine checkpoints at 20/30/40 RL updates, and discuss checkpoint variation and diminishing returns for Pi0.5 (Table H6).
We hope these clarifications and additional results address your concerns, and we welcome further discussion.
W1 & Q1 Policy evaluation protocol
Evaluation independent of the reward models. As stated in lines 427–428 of the manuscript, we do not use the State Reward Model to determine evaluation outcomes. Instead, three human experts assess the evaluation samples, which are randomly assigned among them to reduce systematic assignment bias. Thus, the evaluation does not directly reuse the reward-model predictions employed during post-training.
Policy-specific correlations. Table H1 reports evaluation correlations separately for each policy. Pi0.5 has a lower correlation than ACT and FM. As shown in Table H2, Pi0.5 generally exhibits larger end-effector motion spans along the X and Y axes, suggesting a broader range of motion. Its actions may therefore be less well covered by the world model’s training distribution, potentially increasing rollout errors. In future versions, we will expand the action coverage of the training data to better accommodate diverse policy behaviors.
Table H1. Policy-specific evaluation correlations.
Table H2. Mean end-effector motion spans across ACT, FM and Pi0.5.