- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ForceWorld, an action-conditioned world model that jointly predicts multi-view RGB observations and external joint-torque proxies estimated from motor currents. Built on a pretrained video diffusion transformer, it introduces temporal torque encoding, direct action modulation, and load-weighted training. The generated visual and torque observations are returned to fixed force-aware policies for closed-loop imagined execution. Experiments evaluate prediction quality, apparent contact behavior, and the agreement between real and imagined policy performance across four manipulation tasks, three policy families, and several policy checkpoints.
Critical strengths
-
The paper addresses a useful evaluation problem and tests the predicted feedback in an actual interaction loop.Force-aware policies require an evaluation environment that reproduces their torque observations as well as their visual observations. ForceWorld supplies both modalities and allows the policy’s subsequent actions to depend on generated feedback. This gives the prediction model a concrete downstream purpose.
-
The real-versus-imagined comparisons provide encouraging initial evidence. The main evaluation includes 240 real and 240 imagined trials across three policy families and four tasks, with additional comparisons across checkpoints. The reported success-rate correlation and within-task ranking metric support the feasibility of the approach. Including failed demonstrations and excluding evaluation-policy rollouts from world-model training are also sensible choices that improve the credibility of this comparison.
Critical weaknesses
-
The incremental contribution needs a stronger demonstration of its value for policy evaluation. World-model-based policy evaluation has already been studied in WorldEval, and joint video–force generation is present in TACO. The temporal encoder, action modulation, and load-weighted objective are reasonable extensions, but the current experiments mainly establish prediction improvements. They do not show whether these additions produce substantially better policy evaluation than a visual world model combined with a simpler torque predictor. The backbone and conditioning configurations used for the retrained baselines also need to be clearer. Consequently, I am not yet convinced that the demonstrated evaluation benefit establishes a sufficiently strong advance over existing approaches.
-
The evidence for a reliable, reusable evaluator is limited in both coverage and statistical resolution. The main policy-evaluation results cover the same four tasks represented in training and a single robot platform. Excluding evaluation-policy trajectories provides useful separation, but there is no systematic evaluation on unseen tasks, objects, or contact conditions. Furthermore, the reported correlation of 0.956 is calculated over only twelve policy–task success rates, each estimated from twenty trials. The within-task MMRV is a useful complement, but uncertainty in the rankings is not quantified. These results leave open whether ForceWorld can reliably distinguish closely performing policies or support evaluation beyond the demonstrated task settings.
Questions for authors
-
What improvement in policy-evaluation accuracy comes specifically from joint visual–torque modeling? Could the authors compare against a matched visual world model with a simple independent torque predictor? Please also clarify the backbone, pretraining, and action-conditioning configurations used for the retrained TACO baseline.
-
With the world-model checkpoint held fixed, how well are policy rankings preserved under unseen objects or contact conditions? Could the authors quantify uncertainty in the within-task rankings and MMRV, given the twenty trials per policy–task pair?
-
How robust are the current-derived torque labels across recording sessions, payload changes, and temperature variation? Is there independent calibration or drift analysis beyond the free-motion and elastic-band checks, particularly for the high-load labels used in training and evaluation?
-
What is the wall-clock cost of an imagined rollout? Please clarify how execution-time force feedback is implemented within the approximately 4.8-second action chunks, including whether the evaluated FAWAM implementation retains its online residual correction.
3 comments
Q4 — Wall clock cost and feedback timing.
Including policy inference, an imagined rollout takes 1.7–12.1 minutes for ForceVLA and π0.5+τ and 2.4–19 minutes for FAWAM, depending on the task.
| Task | Robot time simulated | ForceVLA, π0.5+τ | FAWAM |
|---|---|---|---|
| Wipe board | 34 s | 1.9 min | 2.7 min |
| Unlock | 30 s | 1.7 min | 2.4 min |
| Screw bulb | 75 s | 4.1 min | 6.8 min |
| Peel cucumber | 220 s | 12.1 min | 19 min |
In the re-run trials, the policy receives new RGB and torque observations after every 10 executed actions (1 s), matching real-robot execution. The 4.8 s refers to the maximum world-model generation window, not the feedback interval.
FAWAM's online residual correction was disabled in both the real and the imagined evaluation.
Thank you for the additional experiments and clarifications. The comparison with a matched visual model plus an independent torque predictor directly addresses my main concern. The lower MMRV provides useful evidence that joint modeling benefits policy evaluation beyond improving prediction metrics alone.
The new-object experiment, ranking uncertainty analysis, and clarification of feedback timing also strengthen the evaluation. I appreciate the explicit reporting of rollout costs and the fact that FAWAM's residual correction was disabled in both real and imagined execution.
Some limitations remain. The object-generalization test is narrow, and the imagined success rates overestimate real performance despite preserving the observed ranking. The uncertainty analysis does not directly establish a significant advantage over the decoupled baseline, and high-load torque calibration on this platform remains limited. These points should qualify claims about generalization and evaluator reliability.
Overall, the response provides enough additional evidence to change my recommendation to 3 (weak acceptance). Please incorporate these results and qualifications into the manuscript.
We thank the reviewer for a detailed and expert review. We acknowledge the limitations in statistical resolution and coverage and restrict our claims to what the evidence supports. On the size of the increment, we respectfully differ, for the reasons below.
W1 — Contribution beyond prior work.
We argue that the core contribution of ForceWorld is to enable closed-loop evaluation of force-aware policies by jointly predicting video and torque conditioned on the policy's own actions and returning both as observations during rollout. TACO takes no action input and therefore cannot directly serve as a closed-loop policy evaluator. It generates joint video–force correction segments for post-training, without returning force observations to a policy during rollout. WorldEval returns generated images but cannot supply the torque observations required by force-aware policies; Table 6 shows the impact of withholding this feedback. The temporal encoder, action modulation, and load-weighted objective improve torque prediction for this closed-loop use (Table 2).
Q1 — Joint modeling benefits and baseline configurations.
We compare ForceWorld with a matched visual-only world model equipped with a separately trained torque prediction head. Using the same training data and closed-loop evaluation protocol, the decoupled baseline achieves a correlation of 0.926 and an MMRV of 0.058, compared with 0.956 and 0.0125 for ForceWorld. Without force information to constrain visual generation, a visual-only world model can produce penetration artifacts, as shown in our Push-T experiments. A separate torque prediction head then operates on these physically inconsistent frames, which can compromise torque accuracy during contact.
All retrained baselines use the same training trajectories as ForceWorld and are evaluated on the same held-out windows, with the same start frames and, where applicable, recorded actions; their backbones, pretraining and action conditioning are listed below. TACO and Ctrl-World are retrained from their released code and recipes.
W2 and Q2 — Coverage and ranking uncertainty.
Coverage. ForceWorld reproduces the observed real-world ranking of all three policies under this object change. We additionally evaluated the Wipe board task using a different blackboard eraser to test object generalization. The table below compares real and imagined success rates and policy rankings with the new object.
Uncertainty. We have quantified it from the per-cell success counts in Appendix D. The within-task rankings are significantly better than chance (exact permutation test, p = 0.005 for MMRV), and resampling the 20 trials of each cell gives an MMRV of 0.037 (95% interval [0.008, 0.092]), against 0.033 for a repeated real evaluation.
Q3 — Robustness of current derived labels.
Our weight tests indicate that the torque labels are consistent across recording sessions and operating time, and respond approximately proportionally to payload.