acceptodds
Under review as a conference paper at ICLR 2027

I Feel How You See: A Dynamics World Model for Predicting Forces and Kinematics for High-touch Robotic Manipulation

Abstract

In contact-rich manipulation, forces and torques are the fundamental drivers of contact dynamics, dictating how objects move and react to manipulation. Yet, most world models today forgo the grounding that comes from predicting plausible force/torque (F/T) traces alongside their predicted robot trajectories. Addressing this gap, we present a Bin Dynamics World Model, BDWM, for robotic bin stowing via space-creating sweep motions. Conditioned solely on pre-contact information (an RGB-D image, item segmentation, and the planned blade sweep), a single flow-matching velocity field jointly generates two latent streams: a physics latent that decodes to the full F/T and end-effector trajectory, and a vision latent that predicts the post-contact scene. Evaluated on real-robot cycles, BDWM reduces trajectory CRPS by  75% (0.009 vs. 0.036) over the strongest generative action-model baseline and trajectory RMSE by  26% (0.025 vs. 0.034 m) over the strongest baseline overall. Lightweight heads on the predicted latents further drive downstream tasks such as the prediction of space created after a sweep, reaching sub-centimeter accuracy compared to a task-specific regression baseline.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, contact-rich manipulation, robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_t4uKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents BDWM, a generative world model for predicting the consequences of space-creating blade sweeps in robotic bin stowing. Conditioned on pre-contact RGB-D observations, item segmentation, and a planned sweep, a shared flow-matching DiT jointly transports a physics latent and a post-contact vision latent. The physics latent decodes to force/torque and end-effector traces, while the vision latent is supervised by post-stow DINOv3 features. The model additionally uses a force-weighted grounding loss to align physics-token attention with image regions near the blade. On approximately 248K logged real-robot cycles, BDWM reports lower trajectory and force errors than adapted policy/action-model baselines and deterministic regression. Frozen generated latents are also used for created-space prediction and stow-outcome classification, and an online chunking variant forecasts a short remaining horizon from an observed sweep prefix.

Critical strengths
  1. Important task and unusually large real-robot dataset. Predicting force/torque and motion distributions before executing a contact-rich action is practically valuable, and the approximately 248K synchronized real-robot cycles provide a substantial empirical testbed.

  2. Promising joint physical-visual modeling design. Co-generating force/torque, end-effector motion, and a post-contact visual latent is well motivated for partially observed contact dynamics. The component ablations indicate that image conditioning, post-stow supervision, spatio-temporal encoding, and grounding each contribute to trace prediction.

Critical weaknesses
  1. The evaluation protocol does not establish robust generalization, and several headline metrics are insufficiently defined. The dataset is randomly split by cycle, with no evidence that bins, operation sessions, temporally adjacent cycles, or item configurations are separated across train and test. In sequential stowing data, neighboring cycles may share nearly identical scenes, so this split can substantially overestimate generalization. In addition, the reported force metric averages force and torque channels but labels the result in newtons, while the trajectory metric mixes positions, quaternions, linear/angular velocities, and actuator states but labels the result in meters. The normalization needed to make these aggregates meaningful is not specified. The main GP-NLL is also not fully defined in the appendix, and deterministic predictions could have been scored with CRPS as degenerate distributions rather than being marked undefined.

  2. Several central claims go beyond what the experiments demonstrate. The no-F/T ablation shows that force prediction is useful auxiliary supervision, but it does not establish that the generated wrench causally conditions motion within the model; both outputs are jointly generated and may simply benefit from multitask regularization. The post-stow visual latent is never directly evaluated, despite claims of visual-future prediction. The downstream evidence is also weak: created-space prediction improves only modestly over regression and reads out a quantity closely related to the generated trajectory, while the stow-outcome table reports predicted class proportions against dataset prevalence rather than per-example accuracy, balanced accuracy, AUROC, AUPRC, or F1. Consequently, it does not demonstrate useful outcome classification.

Questions for authors
  1. Is the train/test split disjoint by bin, physical stowing session, temporal sequence, and item configuration? Could the authors report results on a group-disjoint split that prevents adjacent cycles or closely related bin states from appearing in different partitions?

  2. How are heterogeneous force, torque, position, quaternion, velocity, and actuator channels normalized before computing the aggregate RMSE, CRPS, and GP-NLL? Why are the resulting aggregate metrics labeled only in newtons and meters, and what is the exact GP-NLL formulation?

  3. Could the authors report valid per-example metrics for stow-outcome prediction and directly evaluate the generated post-stow visual latent? These results would materially affect whether the claimed downstream and visual-prediction capabilities are supported.

4 comments
AuthorsOpenReview ↗

Response to reviewer and summary of changes

We thank the reviewer for the careful and constructive review. We are glad the reviewer found the task important and the dataset unusually large (predicting force/torque and motion distributions before a contact-rich action is practically valuable, and ~248K synchronized real-robot cycles are a substantial testbed), and the joint physical-visual design promising (co-generating F/T, end-effector motion, and a post-contact visual latent is well motivated for partially observed contact dynamics, and the ablations show image conditioning, post-stow supervision, spatio-temporal encoding, and grounding each contribute).

W1/Q1/Q2: Evaluation protocol. Generalization: our split is randomized at the pod level, not the cycle level. A pod is a shelf with 4 faces, each containing a collection of bins. All cycles from a pod go to the same partition, so repeated stows into the same bin stay together and no bin or pre-stow scene appears in both training and test. We will rewrite l. 742-743 to state this. Test results are therefore already on unseen bins, so a bin-held-out split gives the same evaluation and we do not expect the results to change. The data also contains direct blade, wall, and corner inserts.

Metrics: we believe the concern stems from ambiguous wording, which we will fix. Metrics are computed per channel, and no units are mixed. Trajectory RMSE and best-of-50 (Tables 1, 2, 7-10) are translation error, in meters as labeled, and rotation errors (quaternion, angular velocity) are excluded. The F/T value is force error in N. Since all averaged channels share one unit, results do not depend on scaling, and the weighting concern does not arise. We will state which channels enter each score and add definitions and units. GP-NLL: we agree it is described too briefly and that the appendix gives only the per-element NLL. A 50-sample covariance has rank at most 49 and cannot be inverted directly, so we use Σ_reg = Σ̂ + εI, with ε fixed a priori based on validation data, the same for all methods and traces. Padded steps are masked, so only valid steps count. We will add the full definition to the main text and appendix.

W2/Q3: Claims beyond the experiments. Causal role of F/T: the reviewer notes the no-F/T ablation shows force prediction is useful auxiliary supervision but does not establish that the generated wrench causally conditions motion. By multi-task regularization we mean that training one shared latent to predict an extra output can improve the main output through additional supervision alone, with no physical or causal link between the outputs. We have evidence against it being the whole explanation: (1) adding other output channels gives no comparable gain in trajectory accuracy, which auxiliary supervision alone would predict; (2) removing or scrambling image conditioning drops performance to nearly the no-F/T level, which should not happen if the model only exploited kinematic coupling or generic auxiliary supervision.

Visual latent: it is not directly evaluated. Accurate force and trajectory prediction is the core objective, since cameras are occluded during manipulation and the loop is closed on force and position. The latent is a means to an end: it provides post-stow supervision and can be consumed by a downstream model. Consistent with our responses to the other reviewers, we are relabeling the contribution an action-conditioned bin dynamics model.

Downstream evidence: we agree the created-space improvement over regression is modest and related to the generated trajectory, and that the stow-outcome table reports predicted class proportions against dataset prevalence rather than per-example metrics. We have added these to Limitations and future work.

Revisions: specify pod-level randomization and that no bin or pre-stow scene spans partitions; state channels and units and add metric definitions to the appendix; add the full GP-NLL definition (estimator, ε, padding); relabel as an action-conditioned bin dynamics model; expand Limitations and future work on downstream evidence.

AuthorsOpenReview ↗

Adding requested tables and metrics

Per-channel and channel-composition results. As requested, we now report each F/T channel separately. Tables for blade torque (tx, ty, tz) and the rotation component of the trajectory are included, with RMSE, CRPS, best-of-50. We also evaluated a variant that removes the raw wrist channels and keeps only inertia-compensated signals (7-FT). It improves trajectory error (RMSE 0.0411 to 0.0290 m, CRPS 0.0123 to 0.0088) and gripper Fz error (15.71 to 9.28 N), and gives comparable blade-wrench accuracy (RMSE 7.15 vs. 7.31). The benefit of F/T prediction therefore does not depend on the raw wrist channels, which contain the arm’s own inertial and gravity loads.

Blade torque (EMA ep144, 4,000 cycles, K=50):

Metrictxtytz
RMSE7.5514.312.00
CRPS3.797.610.60
Best-of-502.493.850.81

Rotation component of the trajectory (best checkpoint, ep104)

Metricquat_xquat_yquat_zquat_w
RMSE0.0980.0060.0040.088
CRPS0.0190.0020.0020.020
NLL−0.46−5.34−5.71−1.27
Best-of-500.0080.0020.0010.008

Inertia-compensated signals only (7-FT) vs. all channels (13-FT), best checkpoint:

Metric13-FT7-FT
Trajectory RMSE (m)0.04110.0290
Trajectory CRPS0.01230.0088
Trajectory NLL−3.87−3.98
Trajectory minADE0.01730.0169
Trajectory minFDE0.04860.0445
Blade wrench RMSE7.157.31
Blade wrench CRPS3.323.42
Blade wrench NLL2.222.12
Gripper Fz RMSE (N)15.719.28
Gripper Fz NLL5.623.79

Post-stow visual latent. We are evaluating the post-stow latent directly, comparing predicted and ground-truth DINO patch embeddings against pre-stow and per-position-mean baselines. We will report the prediction error in the appendix of the final manuscript.

Thank you for the detailed response. I am raising my rating to 3

Thank you for the response and for raising our rating. We appreciate it.

Reviewer_PTn9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies blade-sweep stowing in industrial fulfillment, where a blade is pushed into a cluttered bin to make room for a new item. Because the items it will touch are often occluded, the same scene and plan can lead to very different contact forces. The authors propose BDWM, a diffusion-based world model that, given the pre-contact RGB-D image and the planned sweep, samples the force/torque and end-effector trajectory of the sweep along with a latent of the post-sweep scene. On a large dataset of real robot cycles, BDWM outperforms a regression baseline and several adapted VLA policies, and its outputs are also used to predict the space created and the stow outcome. The paper claims three contributions: the BDWM model, evidence that jointly generating force and trajectory improves trajectory prediction, and the use of BDWM latents as features for downstream prediction.

Critical strengths
  1. Problem setting and dataset.

Because the blade contacts occluded items, the same scene and plan can lead to different forces, so the paper treats pre-contact prediction of the F/T and trajectory trace as conditional generation and evaluates it with CRPS as well as point error. The dataset has about 248K real robot cycles with synchronized F/T, RGB-D, segmentation, trajectories, and outcome labels, which is much larger than the hundreds to few thousand demonstrations used in most force-aware manipulation papers. If it is released as stated (l. 509), others can use it.

  1. Position encoding and attention loss.

The model uses a RoPE in metric bin coordinates that is shared by image patches and blade-position tokens, and a loss that guides the physics-to-image attention toward patches near the blade, weighted by contact force. Neither is specific to stowing.

Critical weaknesses
  1. Some issues in ablation

The No F/T prediction variant has the same inputs as the full model, including the planned sweep. In Table 2 its trajectory RMSE is 0.094 m and its best-of-50 error is 0.064 m. This is worse than the FC regression baseline (0.056 m) and worse than BDWM without the planned sweep (RMSE 0.084 m, best-of-KK 0.012 m). A model that receives the plan should not have a best-of-50 error of 6.4 cm when a model without the plan reaches 1.2 cm. This suggests that the variant is misconfigured or under-trained.

L351–353 also state that the regression baseline "trails even the weakest ablated BDWM variant on both" modalities, but three ablated variants in Table 2 have worse trajectory RMSE than regression.

  1. Definition of metrics:

All four scores are computed per channel and then averaged over channels (l. 290–292; A.5, l. 685–688). This causes the following problems.

The trajectory score averages 17 channels (Table 6): blade-tip position (m), a unit quaternion (no unit), linear velocity (m/s), angular velocity (rad/s), and blade, gripper, and pedestal positions (m). Only 7 of the 17 channels are in meters, but Tables 1, 2, and 7–10 label the trajectory RMSE and best-of-KK as "m". The F/T score averages 7 force channels (N) and 6 torque channels (Nm) (Table 5) and is labeled "N". The reported 0.025 m is therefore not a position error, and 5.75 N is not a force error. The average also depends on the choice of units. Expressing angular velocity in deg/s instead of rad/s multiplies the error of channels 10–12 by about 57, and expressing linear velocity in mm/s multiplies the error of channels 7–9 by 1000. Either change alters the average and can alter the ranking of methods. The weight of each channel in the average is set by its unit and typical magnitude, not by its relevance to the task.

Six of the 13 F/T channels are raw wrist force and torque. According to Table 5, they have no inertia compensation, contain the arm's own inertial and gravity loads, and are heavy-tailed. The F/T score therefore partly measures how well the model predicts the arm's own loads, not the contact forces, and these heavy-tailed channels may dominate the average. Table 5 mentions a 7-channel variant without the wrist channels, but no results are reported for it.

The NLL in Eq. 9 contains the term 12log⁡(2πσn2)\tfrac12\log(2\pi\sigma_n^2), so rescaling a channel by a factor ss shifts its NLL by log⁡s\log s. For example, expressing the seven meter-valued channels in millimeters would raise each of their NLL values by about 6.9. The average NLL over channels with different units therefore has no fixed meaning. Lines 343–345 take the negative trajectory NLL (−4.78) as evidence of "a sharp-predictive distribution", but the negative value mainly follows from σn\sigma_n being on the order of centimeters when expressed in meters. The same applies to GP-NLL. CRPS has the same unit as the variable, so its average over channels has the same problem as RMSE.

Several trajectory channels are probably close to constant during a sweep, for example the quaternion if the blade orientation is held fixed, and the pedestal positions. Any reasonable model has near-zero error on these channels, and they lower the average. In Fig. 3 the SWEEP phase covers only about 20–25 of roughly 70–130 valid steps, and outside this phase the observed trajectory largely follows the commanded plan, which the model receives as input. A large part of the 0.025 m may therefore come from steps and channels where copying the plan is enough. Per-channel results in native units, results restricted to the contact phase, and a baseline that outputs the commanded trajectory are needed to show how much of the score reflects contact dynamics.

GP-NLL is the main calibration score in Tables 1 and 2, but it is described in one sentence (l. 295–297). Line 293 refers to A.5 for the full definition, and A.5 defines only the per-element NLL (Eq. 9). A joint Gaussian over a 200-step trace needs a 200×200200\times200 covariance. The sample covariance of K=50K=50 samples has rank at most 49, so it is singular: its determinant is zero and it has no inverse. Restricting the trace to its 70–130 valid steps does not remove the problem. To obtain a finite value, the authors must have added a regularizer (for example Σ^+λI\hat\Sigma+\lambda I), fitted a parametric kernel, or used a shrinkage estimator. In the directions not spanned by the samples, the variance is then set by this choice, so the GP-NLL value depends on it and the ranking of methods may depend on it as well. The paper does not state which choice was made, how its hyperparameters were set, or how padded steps are handled.

Best-of-KK in Eq. 11 selects the best sample separately for each channel, and the result is then averaged over channels. Different channels can therefore use different samples, so the score does not measure "whether any sampled rollout matches the realized dynamics" (l. 299–300; A.5, l. 728–730). With 17 trajectory channels each taking the best of 50 samples, the score can be small even if no single sample is close on all channels. Eq. 11 also averages over all T=200T=200 steps, while the other scores use only valid steps (l. 291), so it is unclear whether padded steps are included.

  1. Leakage through the action input

The paper states that BDWM uses "only pre-stow inputs" (l. 49; see also l. 18–19), but it is not clear that the action input aa meets this condition. Line 196 calls aa "pre-contact planned action waypoints". Table 6 (l. 629–630) calls it "the commanded blade-tip motion", given as a 13-D pose and velocity vector at each timestep, and A.6 (l. 742) says the dataset contains "commanded and observed trajectories". The sweep is executed under an admittance controller that pushes with a target force (l. 137–139). This type of controller changes the position command at each control step according to the measured force. There are therefore two signals that could be called the command: the reference trajectory chosen before execution from the cost-map affordances (l. 139–140), and the force-adjusted command sent to the low-level controller during execution. Only the first is available before contact. If aa is the logged force-adjusted command, it already shows where the blade slowed down or stopped because of contact. Since the admittance law relates the deviation from the reference to the measured force, much of the F/T trace can also be recovered from it. The model would then not need to infer the occluded contact from the image.

Even if aa is the reference trajectory, its length can leak information. Traces have different numbers of valid steps (about 70–130 of 200), and the input includes a waypoint-validity signal (A.4, l. 679). If the valid length of aa is cut to match the executed trace, the input tells the model when the sweep ended. The "space created" target is defined as "the executed sweep length" (l. 411–412), so in either case it can be read almost directly from the input, and Table 3 would not show that BDWM features are useful. The large drop when the action input is removed (trajectory RMSE 0.025 → 0.084 m, Table 2) is consistent both with the plan being useful and with leaked information being removed. The paper should state which signal aa is and whether its length or validity mask depends on the executed trace, and it should report a baseline that uses the commanded trajectory as the prediction of the observed trajectory.

  1. Data split

The data are split 80/10/10 at random at the cycle level (l. 742–743). Repeated stows into the same bin can have nearly the same pre-stow scene, so a test cycle can have a near-duplicate in the training set, and the test scores may partly measure recall of bins seen during training. The paper states that generalization "is probed across bins and behaviors via held-out splits" (l. 466–467), but no experiment holds out bins or behaviors.

Questions for authors
  1. What is different in the "No F/T prediction" variant compared with the full model (autoencoder input channels, decoder input, loss weights, training length, early-stopping criterion)? Were hyperparameters re-tuned for the ablated variants? Why does this variant have a larger trajectory error than both the variant without the planned sweep and the FC regression baseline? Please also correct l. 351–353 to match Table 2.

  2. Are the metrics computed in raw or normalized units? Please report per-channel results in native units, at least for blade force and blade-tip position, results restricted to the SWEEP phase, and F/T results for the 7-channel variant without the wrist channels. Please define GP-NLL: which covariance estimator, kernel, or regularizer is used with K=50K=50 samples, and how are padded steps handled? In best-of-KK, is one sample selected for all channels or a different sample for each channel, and does Eq. 11 include padded steps?

  3. Is aa the reference trajectory planned before execution, or the force-adjusted command logged during execution? At test time, do the valid length or the validity mask of aa depend on the executed trace? What is the error of a baseline that uses the commanded trajectory as the prediction of the observed trajectory?

  4. How are repeated stows into the same bin assigned to the train and test sets? How do the results change on a split held out by bin? Which experiment supports the claim in l. 466–467?

3 comments
AuthorsOpenReview ↗

Response to reviewer and summary of changes

We thank the reviewer for the careful and detailed review. We are glad the reviewer found the problem setting and dataset well-motivated (the same scene and plan can yield different forces because items are occluded, so we treat pre-contact F/T and trajectory prediction as conditional generation; ~248K real robot cycles, much larger than most force-aware datasets, usable by others) and the position encoding and attention loss general (RoPE in metric bin coordinates shared by image patches and blade-position tokens; a force-weighted loss guiding attention toward patches near the blade; neither is specific to stowing).

W1/Q1: Ablation. No F/T variant: F/T channels are removed from the decoder output and the loss is changed accordingly. Everything else (planned-sweep conditioning, inputs, trajectory and visual channels, remaining loss weights, training length, early stopping) is identical to the full model. Hyperparameters were not re-tuned, to isolate the effect of F/T.

Higher trajectory error: F/T prediction links contact to the executed motion under our admittance controller. Without it, the latent has no contact-force supervision and must infer deviations from scene and plan alone. This is the effect the ablation measures, not a configuration error. Versus FC regression (0.056 m): regression is trained on squared error and outputs a point estimate, which RMSE favors, while BDWM RMSE is on sampled trajectories and includes sample variance. Versus the no-plan variant: the RMSE gap is small (0.094 vs. 0.084 m), and the best-of-50 gap reflects sample diversity. We believe presuming a training error is premature. The plan is just blade insertion poses and a few heuristic paths over the 3D mesh, and in production it comes from a feed-forward model on pre-stow RGB-D with very high agreement with the original optimization pipeline. It is only partially followed, since early force triggers change insertion depth, height, and sweep length. Models can therefore infer the plan from our inputs and do worse without force supervision.

L351–353 compare baselines with one another, not the ablated variants.

W2/Q2: Metrics. We believe this stems from ambiguous wording and will fix it. Metrics are computed per channel, and no units are mixed. Trajectory RMSE and best-of-50 (Tables 1, 2, 7–10) are translation error, in meters as labeled, and rotation errors are excluded. The F/T value is force error in N. Since all averaged channels share one unit, results do not depend on scaling (e.g., rad/s vs. deg/s) and the weighting concern does not arise. Force and position were prioritized because triggers and compliance act only on position axes, and most manipulations stay perpendicular to the bin. We will state which channels enter each score and add definitions, units, and missing metrics to the appendix.

GP-NLL: we agree it is too briefly described (A.5 gives only Eq. 9) and that a 50-sample covariance has rank ≤49. We use Σ_reg = Σ̂ + εI, with ε fixed a priori based on the validation data, the same for all methods and traces. Padded steps are masked, so only valid steps (70–130) count. We will add the full definition to the main text and A.5.

W3/Q3: Leakage. We believe this is a possible confusion. As described in Sections 3, 4, and 4.1, the planned sweep is a set of waypoints from 3D cost-map affordances, not the commanded action. We will clarify Table 6 and A.6, where the “commanded” wording may have contributed.

Valid length/mask: we compute the planned sweep length for the entire bin length. Execution follows the plan until a maximum force guard stops it or the fixed-force move no longer advances in Y. In a bin of infinitely compressible foam, we would sweep the full length. The planned length therefore does not reveal where execution ended. Sweep length prediction is used solely to match items to bins, so we plan the whole bin.

Commanded-trajectory baseline: the plan covers the full bin and execution deviates after force triggers, so its error would be quite large.

W4/Q4: Data split. Randomization is at the pod level, not the cycle level. A pod is a mobile shelf with 4 faces, each with 8–56 bins. All cycles from a pod go to the same partition, so repeated stows into a bin stay together and no bin or pre-stow scene appears in both training and test. Split cycle counts are close to the pod split. We will rewrite l. 742–743. Test results are therefore already on unseen bins, and a bin-held-out split gives the same evaluation. l. 466–467: the pod split supports generalization across bins, and the data contains direct blade, wall, and corner inserts. We did not hold out behaviors separately.

Revisions: clarify the conditioning input (method, Table 6, A.6); state channels and units and add the GP-NLL definition and missing metrics to the appendix; rewrite l. 742–743 to specify pod-level randomization.

Thank you for the detailed response. It resolves my main concern about the units and channel aggregation in the reported metrics, and your other answers, on the data split, the planned-sweep input, the GP-NLL definition, and the ablation, give reasonable explanations. Please make sure all of the revisions you described go into the final version of the paper. I am raising my rating to 3.

AuthorsOpenReview ↗

Thanking the reviewer for their prompt response

We want to thank you for your prompt response and for raising the rating. We also want to confirm that we are actively working on the promised ablations and they will be included in the final manuscript.

Reviewer_rNQBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents BDWM, a dynamics world model for contact-rich robotic manipulation. Given pre-contact RGB-D observations, object segmentation, and a planned action trajectory, the model jointly predicts force/torque (F/T) signals, end-effector trajectories, and post-contact visual representations. The method uses a flow-matching DiT to generate physics and visual latents, together with a peak-weighted loss and a spatio-temporal grounding loss to improve contact dynamics prediction.

The authors evaluate BDWM on approximately 250K real-robot manipulation trajectories. The results show improvements over several baselines, including π₀, π₀.₅, and ForceVLA, particularly in trajectory prediction accuracy and probabilistic forecasting metrics. The paper also explores downstream applications such as predicting created space and online dynamics forecasting.

Critical strengths
  1. The problem is meaningful and practically relevant. Most existing world models focus on predicting visual observations or motion trajectories. This work takes a step toward modeling physical interactions by incorporating F/T signals, which could be useful for contact-rich manipulation involving occlusion, uncertain contacts, and object deformation.

  2. The experiments are conducted on a relatively large real-robot dataset. The authors evaluate the method on approximately 250K manipulation trajectories and report clear improvements over the baselines across several prediction metrics. The ablation studies also provide useful evidence for the contributions of joint F/T prediction, visual representations, and different conditioning inputs.

  3. The joint modeling of physical and visual information is an interesting direction. The proposed framework generates F/T, motion trajectories, and post-contact visual features within a shared architecture. The spatio-temporal grounding loss is also a reasonable attempt to connect contact locations with physical responses.

Critical weaknesses
  1. The motivation for calling BDWM a world model is not fully convincing. The proposed method mainly generates F/T signals, end-effector trajectories, and post-contact visual features conditioned on the initial observation and planned action. While it does predict a future visual representation, it is unclear whether this representation can support subsequent state transitions or whether the model has learned meaningful environment dynamics beyond conditional sequence generation. The paper does not evaluate multi-step state prediction or physical consistency under different actions. As a result, the distinction between BDWM and a task-specific conditional dynamics predictor remains somewhat unclear.

  2. The mechanism behind the improvement from joint F/T and trajectory prediction is not sufficiently explained. This is one of the central claims of the paper. Table 2 shows that removing F/T prediction increases trajectory RMSE from 0.025 m to 0.094 m, but this result alone does not explain why predicting F/T leads to better motion prediction. In particular, the robot uses an admittance controller, where contact forces directly affect the executed motion. The improvement could therefore come from learning the controller's response rather than the underlying contact dynamics. It could also be a benefit of multi-task supervision through the shared latent representation. Moreover, the F/T channels include both inertia-compensated blade forces and raw wrist forces, the latter containing gravity and inertial effects. The paper does not clearly separate these factors or explain how different F/T signals contribute to trajectory prediction under different contact conditions.

  3. The task diversity is limited, and the generalizability of the method remains unclear. Although the dataset is relatively large, all experiments focus on a single blade-sweeping task with a fixed robot embodiment and control setup. Other manipulation tasks, such as grasping, insertion, rotation, or deformable object manipulation, can involve very different contact dynamics. In addition, the fixed F/T and trajectory representations, as well as the blade-position-based grounding mechanism, seem closely tied to the current setup. It is therefore difficult to judge whether the proposed method can generalize to other contact behaviors, robot embodiments, or control strategies.

4 . The practical value for robot decision-making has not been demonstrated. The paper reports promising offline prediction results and explores online forecasting, but the model has not been evaluated in closed-loop robot execution or actual action selection. It remains unclear whether the improved prediction accuracy would translate into better task success rates, failure detection, or more reliable manipulation. The method also requires iterative sampling and multiple generated trajectories, but the inference latency and computational cost are not sufficiently discussed.

Questions for authors
  1. World model formulation: Could the authors clarify what distinguishes BDWM from a conditional trajectory or dynamics generator? Can the predicted post-contact visual latent be used as the input for a subsequent state transition? It would also be helpful to understand whether the model can produce physically consistent predictions under different candidate actions.

  2. F/T and motion coupling: Could the authors explain more clearly why jointly predicting F/T leads to such a large improvement in trajectory prediction? How much of this improvement could be attributed to the admittance controller's force-response behavior? An additional ablation separating physical coupling from the benefits of multi-task supervision would be useful.

  3. F/T signals and contact conditions: Table 5 shows that the blade F/T signals are inertia-compensated, while the wrist signals include gravity and inertial loads. How are changes in end-effector payload handled? Could the authors evaluate the contributions of different F/T channels separately? It would also be interesting to see whether F/T prediction provides similar benefits during free-space motion, light contact, and strong contact.

  4. Generalization: Have the authors tested the method on different manipulation behaviors, object physical properties, or controller settings? Given that the current grounding mechanism is designed around the blade position, how well would it extend to tasks with more complex or multiple contact points?

  5. Practical deployment: What are the inference latency and computational costs of BDWM, particularly when generating multiple samples? Is the model fast enough for online robot operation? Also, could the authors provide more details on how the predicted F/T distributions could be used for actual decisions, such as detecting jams, identifying abnormal contacts, or predicting manipulation failures?

6 comments
AuthorsOpenReview ↗

Response to reviewer and summary of changes

We thank the reviewer for the careful and constructive review. We are glad the reviewer found the problem meaningful and practically relevant (incorporating F/T moves toward modeling physical interaction beyond visual or trajectory-only prediction), the evidence strong (~250K real-robot trajectories, clear gains over baselines, ablations supporting joint F/T prediction, visual representations, and conditioning inputs), and the joint physical-visual modeling with the spatio-temporal grounding loss an interesting direction.

W1/Q1: World model formulation. We agree. This is an active point of contention: visual-only models such as LeWorldModel face the same question relative to video generation (how can they be world models if they cannot forecast contact forces?). The term is used inconsistently, and we adopted a liberal definition. A stricter definition requires recursive rollout, which ours does not do, including in the sliding-window setting. We will relabel it a bin dynamics model (BDM). The visual latent can feed a later transition, though we expect drift. It is trained only to decode DINOv3 latents as regularization, since F/T prediction is the core objective and cameras are occluded during manipulation. Sharper RGB decoding is a next step. The added item also alters bin state but is not an input (out of scope). Physical consistency across candidate actions is not evaluated; will state this in Limitations.

W2/Q2/Q3: F/T-motion coupling. Our explanation:

  1. Multi-task: other output channels give no comparable gain; if auxiliary supervision explained it, they would help too.
  2. Controller: removing image conditioning drastically reduces performance (Table 2), and scrambling it drops performance almost to the no-F/T level. A model merely learning the controller from kinematics would not suffer.
  3. Channels: we will add an ablation separating inertia-compensated blade forces from raw wrist forces, reporting trajectory RMSE for each.

Hypotheses: predicting forces consistent with trajectories grounds them in physically plausible realizations, and likely eases predicting the admittance response. Our premise is that spatial/temporal conditioning on the pre-stow image and 3D affordances enables learning kinematics and dynamics; bringing all signals into the bin frame is deliberate, and predicting post-stow state adds value. F/T is the most impactful supervision among those we ablate. Ablating the controller is untenable (pure position control would disrupt production, trigger protective stops, and damage goods).

Payload: blade force is inertia-compensated. Wrist force cannot be, since item weight data and online estimators are unreliable, and weight is not a model input.

W3/Q4: Generalization. The method targets non-prehensile manipulation of confined objects at scale from production data, where manipulation is limited by robot design. Other manipulation (e.g., grasping) lacks data here, and validating it is outside our question: can a bin dynamics model reliably predict proprioceptive data from pre-contact information alone? We believe our experiments establish this well. On the fixed representations, we would appreciate clarification: F/T traces adapt to available sensors, trajectories can have arbitrary length, and the task-space frame (our robot plans in task space) could be swapped for others or joints.

Controller: fixed in production data; ablating it would impact customers.

Behaviors: direct blade insert (free-space insertion, then sweep), wall insert (sliding along the wall while pivoting and extending, multi-contact), and corner insert (seek wall, move down to displace items, multi-contact) are all in train/val/test. Sweeps are similar, but pre-sweep interactions differ, so multi-contact cases are present for grounding. We will provide a breakdown by behavior.

Objects: a large e-commerce fulfillment center, from dried-mealworm bags to heaters and milk cartons, limited only by the ≤14 in bin. We can confidently say this is one of the most diverse set of item properties ever documented.

W4/Q5: Practical value and cost. We make no real-time claims and will state this in Limitations. The sweep-length output drives asynchronous item-to-bin assignment, run in parallel with the previous cycle, so the budget is seconds, not ~20 ms. Sampling 50 trajectories takes ~45 s, the same order of magnitude as contemporary research works (e.g., VLAs). The model is far from production-ready for real-time control. Optimization paths: reflow (50→1-2 steps, ~1 s), one-shot K-sample flow-matching optimizations, and distillation.

Uses: (1) shown: latents refine sweep-length prediction, cutting error 30% and enabling feasible item-bin matching; (2) risk-aware MPPI with the chunked variant; (3) early failure signal if latents correlate with success; (4) offline RL with uncertainty quantification.

Summary of revisions: relabel as BDM; add the blade-vs-wrist ablation, update Limitations.

Reviewer_rNQBOpenReview ↗

Reply to the authors

Thank you to the authors for the detailed rebuttal information. I noticed that you have shortened "BDWM" as originally written in the paper to "BDM." However, two of the three contributions in your paper are related to BDWM. Based on your rebuttal, how will you reorganize and adjust your contributions?

AuthorsOpenReview ↗

Response on question regarding contributions

We thank the reviewer for this question. The rename changes the label but not the contributions to a great extent. Contributions (2) and (3) are claims about what the model does, and they rest on empirical results shown in the experiments and Appendix that the new name does not affect.

Contribution (2), joint F/T and trajectory generation. This is a claim about architecture and ablation: a single velocity field jointly transports the F/T and trajectory latents, and removing the F/T head raises trajectory RMSE from 0.025 m to 0.094 m (Table 2). Neither the result nor the architecture depends on whether the model is called a world model or a dynamics model. We are also adding the separate-latents ablation and the blade-versus-wrist analysis to narrow down the mechanism as suggested by other reviewers.

Contribution (3), downstream utility of the latents. This is a claim that lightweight heads on frozen latents can predict created space, giving pre-execution features for the upstream item-to-bin matching task (a 30% reduction in sweep-length prediction error). It is a statement about the learned representation, and the rename does not affect it. We will drop the closing sentence about a path for world models, since it relies on the label we are retiring. We will replace it with the narrower statement that the sampled traces can serve as feature extractors for downstream decision-making in this setting.

Please let us know if there are any other concerns and we will do our best to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.