- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents BDWM, a generative world model for predicting the consequences of space-creating blade sweeps in robotic bin stowing. Conditioned on pre-contact RGB-D observations, item segmentation, and a planned sweep, a shared flow-matching DiT jointly transports a physics latent and a post-contact vision latent. The physics latent decodes to force/torque and end-effector traces, while the vision latent is supervised by post-stow DINOv3 features. The model additionally uses a force-weighted grounding loss to align physics-token attention with image regions near the blade. On approximately 248K logged real-robot cycles, BDWM reports lower trajectory and force errors than adapted policy/action-model baselines and deterministic regression. Frozen generated latents are also used for created-space prediction and stow-outcome classification, and an online chunking variant forecasts a short remaining horizon from an observed sweep prefix.
Critical strengths
-
Important task and unusually large real-robot dataset. Predicting force/torque and motion distributions before executing a contact-rich action is practically valuable, and the approximately 248K synchronized real-robot cycles provide a substantial empirical testbed.
-
Promising joint physical-visual modeling design. Co-generating force/torque, end-effector motion, and a post-contact visual latent is well motivated for partially observed contact dynamics. The component ablations indicate that image conditioning, post-stow supervision, spatio-temporal encoding, and grounding each contribute to trace prediction.
Critical weaknesses
-
The evaluation protocol does not establish robust generalization, and several headline metrics are insufficiently defined. The dataset is randomly split by cycle, with no evidence that bins, operation sessions, temporally adjacent cycles, or item configurations are separated across train and test. In sequential stowing data, neighboring cycles may share nearly identical scenes, so this split can substantially overestimate generalization. In addition, the reported force metric averages force and torque channels but labels the result in newtons, while the trajectory metric mixes positions, quaternions, linear/angular velocities, and actuator states but labels the result in meters. The normalization needed to make these aggregates meaningful is not specified. The main GP-NLL is also not fully defined in the appendix, and deterministic predictions could have been scored with CRPS as degenerate distributions rather than being marked undefined.
-
Several central claims go beyond what the experiments demonstrate. The no-F/T ablation shows that force prediction is useful auxiliary supervision, but it does not establish that the generated wrench causally conditions motion within the model; both outputs are jointly generated and may simply benefit from multitask regularization. The post-stow visual latent is never directly evaluated, despite claims of visual-future prediction. The downstream evidence is also weak: created-space prediction improves only modestly over regression and reads out a quantity closely related to the generated trajectory, while the stow-outcome table reports predicted class proportions against dataset prevalence rather than per-example accuracy, balanced accuracy, AUROC, AUPRC, or F1. Consequently, it does not demonstrate useful outcome classification.
Questions for authors
-
Is the train/test split disjoint by bin, physical stowing session, temporal sequence, and item configuration? Could the authors report results on a group-disjoint split that prevents adjacent cycles or closely related bin states from appearing in different partitions?
-
How are heterogeneous force, torque, position, quaternion, velocity, and actuator channels normalized before computing the aggregate RMSE, CRPS, and GP-NLL? Why are the resulting aggregate metrics labeled only in newtons and meters, and what is the exact GP-NLL formulation?
-
Could the authors report valid per-example metrics for stow-outcome prediction and directly evaluate the generated post-stow visual latent? These results would materially affect whether the claimed downstream and visual-prediction capabilities are supported.
4 comments
Adding requested tables and metrics
Per-channel and channel-composition results. As requested, we now report each F/T channel separately. Tables for blade torque (tx, ty, tz) and the rotation component of the trajectory are included, with RMSE, CRPS, best-of-50. We also evaluated a variant that removes the raw wrist channels and keeps only inertia-compensated signals (7-FT). It improves trajectory error (RMSE 0.0411 to 0.0290 m, CRPS 0.0123 to 0.0088) and gripper Fz error (15.71 to 9.28 N), and gives comparable blade-wrench accuracy (RMSE 7.15 vs. 7.31). The benefit of F/T prediction therefore does not depend on the raw wrist channels, which contain the arm’s own inertial and gravity loads.
Blade torque (EMA ep144, 4,000 cycles, K=50):
| Metric | tx | ty | tz |
|---|---|---|---|
| RMSE | 7.55 | 14.31 | 2.00 |
| CRPS | 3.79 | 7.61 | 0.60 |
| Best-of-50 | 2.49 | 3.85 | 0.81 |
Rotation component of the trajectory (best checkpoint, ep104)
| Metric | quat_x | quat_y | quat_z | quat_w |
|---|---|---|---|---|
| RMSE | 0.098 | 0.006 | 0.004 | 0.088 |
| CRPS | 0.019 | 0.002 | 0.002 | 0.020 |
| NLL | −0.46 | −5.34 | −5.71 | −1.27 |
| Best-of-50 | 0.008 | 0.002 | 0.001 | 0.008 |
Inertia-compensated signals only (7-FT) vs. all channels (13-FT), best checkpoint:
| Metric | 13-FT | 7-FT |
|---|---|---|
| Trajectory RMSE (m) | 0.0411 | 0.0290 |
| Trajectory CRPS | 0.0123 | 0.0088 |
| Trajectory NLL | −3.87 | −3.98 |
| Trajectory minADE | 0.0173 | 0.0169 |
| Trajectory minFDE | 0.0486 | 0.0445 |
| Blade wrench RMSE | 7.15 | 7.31 |
| Blade wrench CRPS | 3.32 | 3.42 |
| Blade wrench NLL | 2.22 | 2.12 |
| Gripper Fz RMSE (N) | 15.71 | 9.28 |
| Gripper Fz NLL | 5.62 | 3.79 |
Post-stow visual latent. We are evaluating the post-stow latent directly, comparing predicted and ground-truth DINO patch embeddings against pre-stow and per-position-mean baselines. We will report the prediction error in the appendix of the final manuscript.
Thank you for the detailed response. I am raising my rating to 3
Thank you
Thank you for the response and for raising our rating. We appreciate it.
Response to reviewer and summary of changes
We thank the reviewer for the careful and constructive review. We are glad the reviewer found the task important and the dataset unusually large (predicting force/torque and motion distributions before a contact-rich action is practically valuable, and ~248K synchronized real-robot cycles are a substantial testbed), and the joint physical-visual design promising (co-generating F/T, end-effector motion, and a post-contact visual latent is well motivated for partially observed contact dynamics, and the ablations show image conditioning, post-stow supervision, spatio-temporal encoding, and grounding each contribute).
W1/Q1/Q2: Evaluation protocol. Generalization: our split is randomized at the pod level, not the cycle level. A pod is a shelf with 4 faces, each containing a collection of bins. All cycles from a pod go to the same partition, so repeated stows into the same bin stay together and no bin or pre-stow scene appears in both training and test. We will rewrite l. 742-743 to state this. Test results are therefore already on unseen bins, so a bin-held-out split gives the same evaluation and we do not expect the results to change. The data also contains direct blade, wall, and corner inserts.
Metrics: we believe the concern stems from ambiguous wording, which we will fix. Metrics are computed per channel, and no units are mixed. Trajectory RMSE and best-of-50 (Tables 1, 2, 7-10) are translation error, in meters as labeled, and rotation errors (quaternion, angular velocity) are excluded. The F/T value is force error in N. Since all averaged channels share one unit, results do not depend on scaling, and the weighting concern does not arise. We will state which channels enter each score and add definitions and units. GP-NLL: we agree it is described too briefly and that the appendix gives only the per-element NLL. A 50-sample covariance has rank at most 49 and cannot be inverted directly, so we use Σ_reg = Σ̂ + εI, with ε fixed a priori based on validation data, the same for all methods and traces. Padded steps are masked, so only valid steps count. We will add the full definition to the main text and appendix.
W2/Q3: Claims beyond the experiments. Causal role of F/T: the reviewer notes the no-F/T ablation shows force prediction is useful auxiliary supervision but does not establish that the generated wrench causally conditions motion. By multi-task regularization we mean that training one shared latent to predict an extra output can improve the main output through additional supervision alone, with no physical or causal link between the outputs. We have evidence against it being the whole explanation: (1) adding other output channels gives no comparable gain in trajectory accuracy, which auxiliary supervision alone would predict; (2) removing or scrambling image conditioning drops performance to nearly the no-F/T level, which should not happen if the model only exploited kinematic coupling or generic auxiliary supervision.
Visual latent: it is not directly evaluated. Accurate force and trajectory prediction is the core objective, since cameras are occluded during manipulation and the loop is closed on force and position. The latent is a means to an end: it provides post-stow supervision and can be consumed by a downstream model. Consistent with our responses to the other reviewers, we are relabeling the contribution an action-conditioned bin dynamics model.
Downstream evidence: we agree the created-space improvement over regression is modest and related to the generated trajectory, and that the stow-outcome table reports predicted class proportions against dataset prevalence rather than per-example metrics. We have added these to Limitations and future work.
Revisions: specify pod-level randomization and that no bin or pre-stow scene spans partitions; state channels and units and add metric definitions to the appendix; add the full GP-NLL definition (estimator, ε, padding); relabel as an action-conditioned bin dynamics model; expand Limitations and future work on downstream evidence.