acceptodds
Under review as a conference paper at ICLR 2027

Reliability-Aware World Model for Robot Policy Improvement

Abstract

Robot policies can improve by practicing inside a learned world model, but the model is wrong in places, and practicing there teaches the policy behaviors that fail on the real robot. Fixing the model needs new real data, and the usual way to get it is to run the policy on the robot and refit; most of those rollouts, however, land on interactions the model already predicts well, and few reach the places where it is wrong. We present a method in which the world model's own prediction says how far it can be trusted. A reliability score, computed from the self-consistency of the flow model's velocity field and from whether the predicted motion follows the commanded action, is used twice: it weights each imagined policy update, and it selects the interactions at which real data is collected. The world model is a video transformer with a dynamics head and an action- and contact-conditioned value head on a shared backbone, and its action-value gradients drive policy improvement. A flagged interaction is reached on the robot by restoring the logged start and replaying the action prefix; the real outcome refits the model before the next round. On three simulation and three real-robot tasks, two rounds of 20 real rollouts improve success over the initial policy by 15.8–17.1 and 25.0–28.1 points and are ahead of DSRL and a VLAW reproduction on every task at the same rollout budget. Controlled studies show that both uses of the score contribute, and that the collected data corrects the model where it was flagged.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world models, robot learning, robot manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_biNpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper aims to improve pre-trained robot policies by fine-tuning them inside a world model. The policy is fine-tuned on imagined rollouts, but these trajectories may drift from the real world and affect policy learning. To address this, the authors define a reliability score for state-action pairs. The score combines the model’s error in predicting the state (velocity-field self-consistency) with its action consistency. Imagined rollouts stop when this score exceeds a threshold, after which they continue in the real world or simulator to correct the model. Specifically, the authors reset the simulator or robot configuration to the beginning of the imagined trajectory and replay the actions taken in the world model until they reach the failure point. The rollout then continues in the real world or simulator. Experiments are conducted on three simulated tasks and three real-world tasks, using π0\pi_0 as the pre-trained policy and a video diffusion transformer as the backbone of the world model. The results show improvements over the base policy and two baselines. Additional experiments examine the role of the reliability score.

Critical strengths

The underlying problem is real and important: policy learning in imagination is prone to various errors, and this work aims to address them directly. The method builds on recent work and is evaluated in both real-world and simulated experiments.

Critical weaknesses

My first concern is the paper’s clarity and writing. The idea is simple, but I had to reread some passages to understand it. Parts of the method section describe task-specific details that could be moved to the experiments section. The text also presents the contact signal as a separate model input; why is it not included in the proprioceptive state? Many details about the method and experiments are missing: how the action chunk in rextr_\text{ext} is obtained, how wintrw_\text{intr} and wextw_\text{ext} were set, how the threshold τ\tau should be chosen, how the data in Sections 4.4 and 4.5 were collected, and what “disagreement” means in Section 4.4. A complete and clear description of how the models are fine-tuned is also missing. Finally, denoting the reliability score by r(s,a)r(s, a) is extremely confusing, since this notation is conventionally used for the reward.

The method seems tied to robots with end effectors, and this assumption is hard-coded into parts of the method and experiments, limiting its scope and applicability. Resetting to a state is also usually expensive and is not always feasible in practice.

The experiments have several weaknesses.

In Table 1, π0\pi_0 is not directly comparable to the other methods because it has not seen the 150 trajectories, and some of the improvement are attributable to just seeing those additional data. This is also apparent when comparing the performance gaps on real and simulated tasks: distribution shift hurts performance more on real tasks, where the gap between π0\pi_0 and Imagined is larger. Moreover, confidence intervals are mentioned in the text but omitted from the table. There appears to be substantial overlap between the confidence intervals for VLAW and the proposed method, but the table does not show this.

In Section 4.3, the world model with the contact signal as an additional input is labeled “ours” and compared against a vision-only value model to show that it is more responsive to actions. I do not see what makes adding this input specific to the proposed method. If the signal is available, any method could incorporate it and potentially benefit from the richer input.

Section 4.6 reports the error in resetting to a state. First, the measurements could be normalized relative to the initial scene and reported without units to make them more intuitive. Second, the experiment does not seem directly related to the method, and its purpose or importance is unclear.

An important missing experiment is an evaluation of sensitivity to the threshold used for switching to the real world/simulator.

Given the weaknesses in the experiments, the method’s limited scope and generality, and the rushed presentation, I lean toward rejecting the paper.

Questions for authors

Please see weaknesses.

6 comments

Thank you for your review. We address the presentation and methodological questions, clarify the shared initial data, and report additional hardware and threshold-sensitivity results in three parts. Changes in the revised manuscript are highlighted in blue.

Presentation and task-specific details

The revised method section focuses on the general formulation and the main steps of the loop: predicting and valuing action outcomes, assessing prediction reliability, and using the score to connect imagined updates with real feedback. We have moved the task-specific contact-signal sources and success-judgment procedures from Section 3.2 to the experimental setup in Section 4.1, placed the trigger-grid size and per-bin quota in Appendix A, and consolidated the FoldCloth restoration details in Appendix C. The method section retains the definitions, equations, and general execution procedure, with cross-references to these implementation details.

Contact inputs and their role in the value function

The world model receives observations oo, robot proprioceptive state ss, the commanded action chunk aˉ\bar a, contact signal cc, and the task instruction. Here, ss describes the arm configuration and gripper opening, while cc indicates whether an object is held. The first imagined chunk uses recorded real observations and states. Subsequent imagined chunks receive no new sensor readings, so their state inputs must be computed.

For ss, we use the current proprioceptive state and the arm-motion and gripper-opening changes specified by the action chunk to compute the next proprioceptive input. For cc, a command specifies that the gripper should close, but does not tell us whether it will hold the object or close on empty space. A contact head trained with real-trajectory contact labels therefore predicts the end-of-chunk contact state, which becomes the next chunk’s contact input. We write ss and cc separately to expose these two update rules. Concatenating them into one augmented state vector would leave the computations unchanged (Sections 3.2 and 3.4).

Contact sensing is an available input, and we do not claim its inclusion as a standalone contribution. Section 4.3 examines the value function used to improve the policy through action-value gradients: action conditioning enables it to distinguish candidate actions, and contact information helps it respond to grasp failures with little visual change. Table 2 isolates the contact contribution within the same action-conditioned critic: adding contact raises ranking accuracy from 78% to 96% and lowers return NRMSE from 0.15 to 0.08. This supports the choice of critic within our reliability-guided policy-improvement and data-collection loop. We have replaced “Ours” in Figure 5 with “Action- and contact-conditioned value” to identify the compared model directly.

Model fine-tuning and policy updates

The update order is described in Sections 3.2–3.4 and the algorithm in Appendix D, with optimization settings in Appendix A (Table 9). After collecting 20 trajectories in each round, we continue from the previous checkpoints, mixing newly collected data and replay data sampled uniformly from the original dataset in a 1:1 ratio. We first update the dynamics model with the flow-matching objective, then freeze the shared DiT and fit the Q-head using fitted Q evaluation; the contact head is trained against recorded contact labels in the same stage. Dynamics and Q fitting use batch sizes of 2 and 8, respectively, with 100k steps and a learning rate of 4×10−54\times10^{-5} for each.

The updated world model and critic are then frozen for the next imagined policy-improvement cycle. QAM updates only the policy’s rank-32 LoRA adapters using action-value gradients; the simulation policy-training setting is 5k steps at a learning rate of 10−610^{-6} (Table 9). Thus, collected real outcomes first update the prediction and value models, which then guide the next policy-improvement cycle. The revision consolidates the checkpoint-continuation and data-mixing details and explicitly states in Table 9 that its model-training settings are reused for each feedback round.

Reliability and reward notation

To distinguish prediction reliability from reward, the revision denotes the reliability score by ρ(s,aˉ)\rho(s,\bar a) and its two components by ρintr\rho_{\mathrm{intr}} and ρext\rho_{\mathrm{ext}}, while retaining rtr_t for reward. This notation is used consistently in the text, equations, figures, and algorithm.

Action consistency, score settings, and evaluation data

Section 3.3 describes how the action chunk in ρext\rho_{\mathrm{ext}} is obtained, and Appendix A (Table 9) specifies the settings of wintrw_{\mathrm{intr}}, wextw_{\mathrm{ext}}, and τ\tau. The action chunk used in ρext\rho_{\mathrm{ext}} is recovered from the predicted video: a visual probe extracts end-effector poses, and a three-layer inverse-dynamics MLP trained on demonstrations maps consecutive states to actions. The recovered actions form aˉimpl\bar a_{\mathrm{impl}}, whose L1L_1 discrepancy from the commanded chunk gives ρext\rho_{\mathrm{ext}}. To combine this signal with ρintr\rho_{\mathrm{intr}}, we set wintrw_{\mathrm{intr}} and wextw_{\mathrm{ext}} so that the weighted terms wintrρintrw_{\mathrm{intr}}\rho_{\mathrm{intr}} and wextρextw_{\mathrm{ext}}\rho_{\mathrm{ext}} each have an in-distribution mean of 0.5. The threshold τ\tau is then set to the 95th percentile of the combined score on held-out in-distribution predictions and shared across tasks, as specified in Table 9 and Section 3.4.

For the detection study in Section 4.4, Appendix B specifies the trajectory-level split and the comparison between imagined and executed outcomes of the same action. For each context from held-out PickPlace trajectories, split at the trajectory level, we generate a prediction and execute the commanded chunk in the simulator. “Disagreement” refers to a mismatch in grasp contact, object displacement, or gripper state—for example, a predicted grasp-and-lift when the object actually remains ungrasped and stationary. Simulator outcomes are read directly from state; preliminary labels from the probe are manually verified, with annotation tolerances set on in-distribution samples. The evaluation uses 600 predictions, including the 200-prediction annotated subset for detection validation. The policy ablations use the same two-round budget of 20 additional trajectories per round.

For the model-correction study in Section 4.5, Appendix B specifies a validation set fixed before collection. We use 300 executed transitions and use the initial score to divide them into 150 suspicious and 150 ordinary transitions. Their identities and labels remain unchanged, and the trajectories are excluded from collection, refitting, and setting the score weights and threshold, so correction is evaluated on the same transitions before and after model updates. The revision makes the cross-references to these existing definitions and protocols more explicit and clarifies the manual verification procedure and the 200/600 subset relationship.

End effectors and the manipulation setting

This work studies the improvement of π0\pi_0-based robot manipulation policies, with DSRL and VLAW evaluated in the same manipulation setting (Sections 3.1 and 4.1). End effectors are part of the platforms and tasks under study; reading their motion is how we implement the check that predicted manipulation follows the commanded actions.

The initial policy has seen the 150 trajectories

The π0\pi_0 policy in Table 1 has already undergone supervised fine-tuning on the initial 150 trajectories (Sections 3.1 and 4.1). Imagined starts from this same policy and improves it inside the world model without collecting additional real trajectories. Its improvement over π0\pi_0 is therefore obtained using the same initial real data.

Physical collection: purpose, cost, and normalized errors

As stated at the beginning of Section 4.6, the real-data collection stage requires the robot to reach the interaction selected by the reliability score so that its actual outcome can be collected. This is the direct purpose of the restoration-and-replay evaluation. Following the procedure in Figure 3, the robot restores the logged start and replays the action prefix. OpenJar, StackBox, and FoldCloth reach 19/20, 18/20, and 19/20 requested interactions, respectively (Table 8). The task-wise mean restoration time is 33–86 seconds and replay time is 12–22 seconds per completed rollout, with zero hands-on human time (Table 12).

Following the suggestion to provide a dimensionless reference, we express the measured errors relative to the relevant object sizes: the 14 cm lid diameter, 25 cm box side, and 38 cm spread garment width. Restoration centroid errors are 8.6% and 7.2% on OpenJar and StackBox; object-position errors after replay are 10.0% and 7.2%, respectively. The FoldCloth keypoint error is 6.3% of garment width. The corresponding tables and captions now include these reference sizes and percentages.

Section 4.6 also tests the effect of restoration error on policy improvement. Adding 1, 2, and 5 cm object-position offsets before replay in PickPlace gives Online-2 success rates of 41.7%, 39.3%, and 30.7%, compared with 42.4% under exact resets. Thus, the section measures both whether targeted collection can be carried out on hardware and how restoration accuracy affects the learning loop.

Confidence intervals and paired differences

Table 1 now includes the success counts and 95% confidence intervals for each entry. Since the methods are evaluated at the same initial placements, we also quantify uncertainty in their paired success-rate differences, accounting for the correspondence between the evaluations.

For the hardware policies trained with 40 additional trajectories, we repeated evaluation at the same 32 initial placements per task, giving two trials per method per placement. The table below reports the counts and success-rate differences, with 95% basic paired cluster-bootstrap intervals computed by resampling initial placements within each task and keeping both repetitions together. Across the three hardware tasks, the mean advantage is 6.8 percentage points, with a 95% interval of [1.6, 11.5] points and a two-sided paired permutation p-value of 0.0156.

TaskOursVLAW†Difference (pp)95% CI for the difference (pp)
OpenJar38/6434/64+6.3[-3.1, 12.5]
StackBox21/6414/64+10.9[0.0, 20.3]
FoldCloth22/6420/64+3.1[-3.1, 6.3]
Three-task aggregate81/19268/192+6.8[1.6, 11.5]

We also extended both methods to a third feedback round, with 60 additional training trajectories. Our success counts versus VLAW† are 23/32 versus 18/32 on OpenJar, 15/32 versus 7/32 on StackBox, and 15/32 versus 11/32 on FoldCloth. The corresponding margins are 15.6, 25.0, and 12.5 percentage points, providing further evidence of improvement with continued feedback. These results are reported separately by training stage.

Switching-threshold sensitivity

Section 4.4 reports a threshold-sensitivity test on the 200 annotated predictions: halving or doubling τ\tau changes at most four decisions (2%). The default threshold is the 95th percentile of the combined score on held-out in-distribution predictions, as specified in Appendix A, Table 9.

We additionally evaluated the effect on final policy performance by comparing the 90th-, 95th-, and 99th-percentile thresholds on PickPlace. All conditions use the same calibration set, initial checkpoints, and other training settings, with two feedback rounds of 20 additional trajectories each. Evaluated on the same 170 episodes, the final Online-2 policies achieve 71/170 (41.8%), 72/170 (42.4%), and 72/170 (42.4%), respectively; the 95th-percentile entry is the original Table 1 result. The maximum difference is one success, or 0.6 percentage points. Policy performance therefore remains stable across these threshold settings at the same data budget.

Thank you again for your comments. We look forward to your feedback on the revision.

Reviewer_UXPqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper uses a learned video world model to improve a robot policy through imagination. A score combining velocity-field self-consistency and action consistency down-weights unreliable policy updates and selects interactions for real feedback. A shared video backbone predicts future observations and supplies features to an action- and contact-conditioned critic. For a selected interaction, the robot restores the logged starting scene, replays the action prefix, and executes the flagged action chunk. The collected data update the model and critic. Experiments cover three simulation and three physical tasks, with two rounds of 20 additional trajectories per task.

Critical strengths
  1. The method addresses a practical problem: imagined policy updates can exploit model errors, while on-policy collection may spend much of its budget on interactions the model already predicts well. Using the score to control updates and direct collection is a sensible response.

  2. Table 6 separates update weighting from target selection and checks shuffled and constant weights. Table 7 tests model correction on a fixed held-out set. These controls help explain where the improvement comes from.

  3. The collection procedure is tested on three physical tasks. The paper reports restoration, replay, contact correspondence, and robot time, which helps assess whether the selected interactions can actually be reached.

Critical weaknesses
  1. Appendix B says the labels for the 600-chunk evaluation are obtained from predicted frames using the same probe and segmentation used elsewhere. The external score also uses the visual probe. Errors in this shared measurement pipeline could affect both the score and the outcome label, so the AUROC of 0.97 needs an independent check. The 200-prediction annotated set helps, but does not check the labels in this separate evaluation.

  2. Tables 8 and 11 report small position, yaw, and keypoint errors and high contact agreement. Appendix C explains the restoration routine, but not fully how each imagined-state measurement is compared with the reached state or how accurate the estimators are. Without calibration, I cannot tell how much of the reported discrepancy comes from replay and how much comes from measurement. This affects whether the real feedback can be treated as the outcome of the flagged interaction.

  3. The hardware results use 32 trials per entry. Relative to the stronger baseline, Online 2 adds two successes on OpenJar, four on StackBox, and one on FoldCloth. The paper acknowledges the wide confidence intervals, but does not quantify uncertainty in these differences. The three-seed PickPlace controls do not establish the stability of the hardware gains. A paired analysis of the existing evaluations, and repeated training runs where feasible, would strengthen this claim.

Questions for authors
  1. Table 5 retains 70% of 600 predictions, or 420 samples. The reported error rates of 13.5%, 6.5%, and 1.5% cannot be obtained by rounding integer error counts out of 420 to one decimal place. What are the error and retained-sample counts for each method? Were these values averaged across runs or computed using interpolation?

  2. Could a subset of the 600 predictions be checked manually or with an independently validated measurement procedure? Please report agreement with the current labels and whether the detection results change.

  3. For Tables 8 and 11, how are the imagined and reached poses, cloth keypoints, and contact modes obtained? Please specify the coordinate alignment and calibration errors, and whether failed triggers are included in the reported averages.

  4. Would a separate encoder initialized from the dynamics-trained checkpoint match the shared encoder in Table 2? The current comparison changes both parameter sharing and dynamics pretraining.

3 comments

Thank you for the detailed review. We clarify the labeling and measurement procedures, report additional hardware analysis, and address the Table 5 and encoder questions in two parts. Changes in the revised manuscript are highlighted in blue.

Manual verification of the 600 predictions (Weakness 1; Question 2)

Table 5 uses the final, manually verified labels for all 600 predictions. The visual probe and segmentation provide preliminary annotations to reduce annotation effort; these are then checked manually against the actual outcomes of the same actions, read directly from simulator state. The reported AUROC of 0.97 is computed using these manually verified labels.

The 200-prediction annotated set in Section 4.4 is a subset of these 600 predictions, organized by candidate failure type for detection validation and error-type analysis. Its annotations therefore cover samples in the Table 5 evaluation. We have clarified the manual verification procedure and this subset relationship in Appendix B.

Measurement procedures and calibration (Weakness 2; Question 3)

Table 8 evaluates restoration against the logged starting scene and replay against the imagined state at the same step; Table 11 further assesses geometry and contact correspondence at that interaction (Figure 3). All visual measurements are transformed into the same robot-base frame. The mean translation and rotation errors of this calibration are 1.2 mm and 0.35°.

For imagined states, segmentation and keypoint detection are combined with Depth Anything 3 to recover 3D geometry; reached states use measured camera depth. End-effector poses are obtained by gripper-template alignment on the imagined side and forward kinematics on the real side. Object position and yaw are estimated from segmented point clouds. Cloth error averages visible correspondences among ten predefined landmarks—two hem corners, four sleeve corners, two collar corners, and two buttons—with occluded points excluded.

We validate these estimators against reference poses and manually annotated multiview keypoints. On reference scenes, the visual end-effector estimator has errors of 2.6 mm and 0.43°. Object position/yaw errors for the DA3 and RGB-D pipelines are, respectively, 5.6 mm/0.82° and 3.2 mm/0.74° on OpenJar, and 8.4 mm/0.78° and 4.4 mm/0.92° on StackBox. Cloth-keypoint errors are 8.8 mm and 3.3 mm. We additionally check the generated images used for imagined-state measurements: mean 2D localization error is 3 pixels, and 3D errors are 3.6 mm for the end effector, 9.5 mm for objects, and 8.6 mm for cloth keypoints. These checks quantify measurement accuracy on both reference scenes and generated images.

For contact correspondence, the imagined mode is predicted by the contact head, while the reached mode is obtained from demonstration-calibrated gripper current. The current-based labels agree with the reference labels on all 20 holding and 20 empty-gripper samples, directly validating the physical-side contact labels used in Table 11.

Unsuccessful triggers are excluded from the mean errors in Tables 8 and 11 and included in the reported arrival rates: 19/20 for OpenJar, 18/20 for StackBox, and 19/20 for FoldCloth. For successful arrivals, the measured discrepancies are 6.5–10.4 mm for the end effector, 1.4–1.8 cm for object position, and 2.4 cm for cloth keypoints. These are state discrepancies measured through the above calibrated procedures. The feedback trajectories record the actual reached states, executed actions, and outcomes. We have added the measurement procedures and calibration results to Appendix C and made the averaging population explicit in the table captions.

Uncertainty in hardware improvements (Weakness 3)

We repeated the hardware evaluation of the policies trained with 40 additional trajectories. Each method was evaluated twice at each of the same 32 initial placements per task. On the original evaluations, our successful placements include all of VLAW†’s successful placements, with 2, 4, and 1 additional successes on OpenJar, StackBox, and FoldCloth. The repeated evaluations reproduce these outcomes except for one of our additional successes on StackBox.

We quantify uncertainty at the initial-placement level, keeping both repetitions together. The task-wise success-rate differences and 95% basic paired cluster-bootstrap intervals are reported below. Across the three hardware tasks, our mean advantage is 6.8 percentage points, with a 95% interval of [1.6, 11.5] points and a two-sided paired permutation p-value of 0.0156.

TaskOurs (both evaluations)VLAW† (both evaluations)Difference (pp)95% CI for the difference (pp)
OpenJar38/6434/64+6.3[-3.1, 12.5]
StackBox21/6414/64+10.9[0.0, 20.3]
FoldCloth22/6420/64+3.1[-3.1, 6.3]
Three-task aggregate81/19268/192+6.8[1.6, 11.5]

We also completed the third feedback round, reaching 60 additional training trajectories. Success rates are 23/32 versus 18/32 on OpenJar, 15/32 versus 7/32 on StackBox, and 15/32 versus 11/32 on FoldCloth, corresponding to margins of 15.6, 25.0, and 12.5 points. These results provide further hardware evidence that the advantage persists and increases with additional feedback. The revised results include the success counts and uncertainty estimates.

Table 5: counts and corrected denominator (Question 1)

Thank you for pointing out this inconsistency. Each method retains exactly 420 of the 600 predictions. The reported percentages were normalized by the full set of 600 predictions, whereas the table caption defines the error rate within the retained set of 420. The counts and corrected percentages under this definition are given below; they are computed directly, without averaging across runs or interpolation.

MethodRetained samplesErrors among retained samplesCorrected retained error
Distance to training data4208119.3%
Prediction-head ensemble420399.3%
Intrinsic signal420245.7%
External signal420337.9%
Combined score42092.1%

We have updated the retained-error column of Table 5 and the corresponding numbers in Section 4.4 to use this definition consistently. The error counts, AUROC values, and ordering of the methods are unchanged.

Dynamics-trained features and encoder sharing (Question 4)

Yes—under our frozen-encoder training protocol, a separate copy of the dynamics-trained encoder is an equivalent feature extractor. During Q fitting, the DiT parameters remain fixed (Section 3.2; Table 9). With the same checkpoint, inputs, and feature-extraction layers, the separate copy supplies the Q-head with exactly the same features as the shared encoder. Parameter sharing itself therefore introduces no additional feature-learning effect at this stage.

The “Separate” row in Table 2 uses a different checkpoint: the original pretrained Wan2.2 weights, without dynamics fine-tuning (Appendix B). Thus, the improvement from 88% to 96% ranking accuracy and from 0.11 to 0.08 return NRMSE supports the benefit of dynamics-trained features for value estimation. It should not be attributed to storing one shared encoder rather than two identical frozen copies.

We use sharing to let the Q-head reuse the dynamics encoder without maintaining a duplicate. Table 2 now explicitly labels whether the encoder has undergone dynamics fine-tuning and states that the encoders are frozen during Q fitting. Section 4.3 clarifies this interpretation.

Thank you again for your comments. We look forward to your feedback on the revision.

AuthorsOpenReview ↗

Follow-up to Reviewer UXPq

Dear Reviewer UXPq, thank you again for your detailed review. We know this is a busy period. The discussion closes on Nov 18, and if you have a moment, we would really appreciate hearing whether the label-verification and measurement-calibration details, the paired hardware analysis, the corrected Table 5, and our answer on the encoder comparison address your concerns, or whether something is still unclear. We are happy to answer follow-up questions until then.

Reviewer_kxjqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method to identify unreliable world model predictions and use them to guide real-world data collection. The reliability score is also used to down-weight unreliable predictions during policy optimization. Experiments in simulation and on real robots show consistent improvements over existing baselines.

Critical strengths
  1. Well-motivated problem. I like the idea of identifying where the world model is unreliable and collecting real-world data specifically to address those errors, rather than simply collecting more on-policy rollouts.

  2. Promising results. The method shows consistent improvements across simulation and real-robot tasks, and the ablations help support the main claims.

Critical weaknesses
  1. Missing comparison across iterations. The paper emphasizes iterative improvement, but only compares against the final performance of baselines like VLAW. Since VLAW is also iterative, I would like to see how the methods compare after each round of data collection.

  2. Method complexity and scalability. The overall approach seems quite complicated, with multiple components for reliability estimation and targeted data collection. In particular, I'm not sure how practical it is to restore specific states in more complex real-world tasks.

Questions for authors
  1. Could a pretrained VLM be used to detect unrealistic world model predictions? How would that compare to the proposed reliability score?

  2. How does velocity-field self-consistency work in highly multimodal settings, where multiple futures could all be valid?

  3. How do you reliably restore a specific state on a real robot? For example, in the middle of bimanual shirt folding, it seems difficult to reproduce the exact cloth configuration. How sensitive is the method to these restoration errors?

  4. Could you compare against VLAW after each iteration, rather than only at the end? This would help demonstrate the benefit of targeted data collection over standard on-policy collection.

4 comments

Thank you for your review. We have added a six-task comparison through three feedback rounds, VLM detector comparisons, and a study of multiple valid futures. We address your comments in three parts. Changes in the revised manuscript are highlighted in blue.

Round-by-round comparison (Weakness 1; Question 4)

We provide a round-by-round comparison with our VLAW reproduction (VLAW†) on all six tasks, extending both methods to three feedback rounds. Both methods start from the same policy fine-tuned on the initial 150 trajectories and receive 20 additional trajectories per task per round. Following Table 1, success rates are evaluated over 170 episodes per simulation task and 32 trials per real-robot task.

TaskVLAW† (20)Ours (20)VLAW† (40)Ours (40)VLAW† (60)Ours (60)
SinkFaucet32.934.739.443.541.849.4
OpenCabinet26.527.128.837.632.441.2
PickPlace30.632.430.642.434.146.5
OpenJar43.853.153.159.456.371.9
StackBox15.625.021.934.421.946.9
FoldCloth25.028.131.334.434.446.9

Success rates (%); column headers indicate cumulative additional trajectories.

Our method achieves higher success rates on all six tasks after every round. From 40 to 60 trajectories, our success rate increases by 3.5–5.9 percentage points on the simulation tasks and by 12.5 points on each real-robot task. The margin over VLAW† increases on five tasks and is maintained on OpenCabinet. Across all six tasks, our results with 40 additional trajectories already match or exceed those of VLAW† with 60. These results show continued improvement with additional feedback and greater data efficiency across rounds.

System complexity and restoration feasibility (Weakness 2)

System complexity. We address this concern through three aspects: how the system is organized, whether its components provide measurable benefits, and how much tuning is needed across tasks.

Overall design and component roles. The world model predicts action outcomes, and the value head evaluates actions to guide policy improvement. Two checks assess prediction reliability: velocity-field self-consistency evaluates geometric plausibility using the world model itself; a visual probe reads end-effector motion, and an inverse-dynamics network recovers the implied actions for comparison with the commands. The combined score weights imagined updates and selects interactions for real feedback. The collected outcomes then update the world model and value head.

Whether the components can be simplified. Controlled ablations quantify their contributions. Removing either reliability signal reduces PickPlace success by 10.0–11.8 percentage points at the original 40-trajectory budget (Table 4). Table 6 separately varies update weighting and score-selected collection, with shuffled and constant-weight controls. We also evaluate pretrained VLMs as alternative detectors: Qwen3-VL and Astra achieve AUROCs of 0.79 and 0.88, versus 0.97 for our combined score; the full comparison is provided in Part 2 (Question 1).

Tuning burden. All six tasks use the same calibration procedure for component weights and the reliability threshold. Halving or doubling the threshold, component weights, or noise-sample count changes at most four decisions on the existing 200-prediction detection-validation subset (Section 4.4).

Feasibility of restoration. Programmatic robot control composes perception, planning, and manipulation interfaces, then uses execution feedback to verify the outcome and revise or retry the action. This control pattern is demonstrated in Code as Policies, CaP-X, and the Astra-based GPT-GEAR. Applying the same feedback structure to scene restoration yields reusable reset procedures, as implemented on real robots in ENPIRE. Our RGB-D reset routine follows this structure with the logged starting scene as its target. After restoration, replaying the recorded action prefix reconstructs the selected intermediate interaction, including its contact and cloth configuration. Across three real-robot tasks and two platforms, 18–19 of 20 requests are reached per task, including 19/20 on bimanual FoldCloth (Table 8). Restoration and replay take 45–108 seconds per completed rollout, with zero hands-on human time (Table 12).

Pretrained VLM detectors (Question 1)

Pretrained VLMs can be used as prediction-error detectors, and we have added a direct comparison with Qwen3-VL and Astra without task-specific fine-tuning. Both receive the observation history and predicted frames, together with camera/coordinate information, the commanded end-effector trajectory, and gripper states, with physical units specified. We evaluate all three detectors on the same 600 held-out predictions used in Table 5, with the same manually verified labels.

DetectorAUROC ↑Error among the retained 70% ↓
Qwen3-VL0.7910.0% (42/420)
Astra0.886.4% (27/420)
Our combined score0.972.1% (9/420)

Our combined score achieves an AUROC of 0.97, compared with 0.79 and 0.88 for the VLMs. When retaining the 420 lowest-scoring predictions, it leaves nine errors, versus 42 for Qwen3-VL and 27 for Astra.

We further examine the existing 200-prediction annotated subset drawn from these 600 predictions to assess which error types the VLMs miss.

Prediction typeQwen3-VLAstra
Geometric failures: detected/total ↑35/45 (77.8%)41/45 (91.1%)
Action-unfaithful predictions: detected/total ↑24/46 (52.2%)32/46 (69.6%)
Correct predictions: false alarms/total ↓33/109 (30.3%)18/109 (16.5%)

Both VLMs detect geometric failures more readily than action-unfaithful predictions: Astra detects 41 of 45 geometric failures but only 32 of 46 action-unfaithful cases, while Qwen3-VL detects 35 and 24, respectively. Thus, even with the action commands provided, action-unfaithful predictions remain a particular difficulty for the VLM detectors. The overall comparison shows that our combined score screens the imagined predictions more accurately.

Multiple valid futures (Question 2)

Multiple futures can be valid under the same observation and action, for example grasping or slipping and different cloth configurations in FoldCloth. Our intrinsic score is evaluated separately for each predicted sample; it does not use disagreement between generated samples as the reliability criterion.

More specifically, Eq. (1) can be written as a weighted reconstruction error. For a predicted sample x^\hat x, let xσ=(1−σ)x^+σzx_\sigma=(1-\sigma)\hat x+\sigma z and x~=xσ−σvθ(xσ,σ)\tilde x=x_\sigma-\sigma v_\theta(x_\sigma,\sigma). The squared velocity residual is exactly ∥x^−x~∥2/σ2\|\hat x-\tilde x\|^2/\sigma^2. For an optimal velocity field, x~\tilde x is the conditional mean of the clean sample given the noised sample and the model inputs. Averaged over in-distribution samples, this reconstruction error reflects posterior uncertainty, which decomposes into within-mode and between-mode terms. When the noised sample still identifies its mode, the between-mode contribution is small, allowing distinct valid futures to receive low scores.

We examine this behavior on FoldCloth grasping interactions. We compare two groups of 25 starting conditions, generating four futures per condition while keeping the observation and commanded action fixed. One group has consistent valid outcomes across samples; the other contains multiple valid outcomes. The groups contain 92 and 88 valid predictions, respectively.

GroupStarting conditionsValid predictionsRelative change in mean ρintr\rho_{\mathrm{intr}}False alarms by the full method
Consistent valid outcomes2592Reference0/92 (0%)
Multiple valid outcomes2588+3.5%1/88 (1.1%)

Mean scores are computed on valid predictions; false alarms use the original combined score and threshold.

In FoldCloth grasping, multiple valid outcomes increase the mean intrinsic score by only 3.5%, with a false-alarm rate of 1.1%, close to the 0% rate in the consistent-outcome group. These results show that the method accommodates valid variations in grasp outcomes and cloth configurations while maintaining reliable judgments.

Restoration and sensitivity to its errors (Question 3)

As Figure 3 shows, the reset targets the logged real starting state s0s_0. The intermediate folding state is reached by replaying the recorded action prefix, rather than by directly arranging the cloth into an imagined configuration. The sequence is: restore the same start, replay the same prefix to reach the trigger, then execute the same flagged action chunk and record its real outcome. The intermediate cloth shape and grasp are therefore reconstructed through robot execution.

On FoldCloth, 19 of 20 triggers are reached (Table 8). At successfully reached interactions, the mean end-effector error is 8.6 mm and the mean error over visible corresponding cloth landmarks is 2.4 cm, equivalent to 6.3% of the garment’s 38 cm spread width. Contact-mode agreement across repeated replays is 95% (Table 11). These measurements assess both cloth shape and grasp reproduction.

In the PickPlace sensitivity test (Section 4.6), we first restore the logged simulator state and then add a 1, 2, or 5 cm offset to the object position before replaying the recorded action prefix. The prefix and flagged action chunk are kept unchanged, and data collection records the actual outcomes from the displaced scene. For each offset, we rerun the full two-round collection-and-update loop and evaluate the resulting Online-2 policy; the 0 cm condition uses exact resets:

Added reset offset0 cm1 cm2 cm5 cm
Online-2 success (%)42.441.739.330.7

With 1–2 cm offsets, success decreases by 0.7–3.1 percentage points and remains above VLAW† at the same 40-trajectory budget (30.6%). On FoldCloth, the same restoration-and-replay procedure supports improvement from 9.4% initially to 34.4% with 40 additional trajectories and 46.9% with 60.

References

Thank you again for your comments. We look forward to your feedback on the revision.

AuthorsOpenReview ↗

Follow-up to Reviewer kxjq

Dear Reviewer kxjq, thank you again for your review. We know this is a busy period. The discussion closes on Nov 18, and if you have a moment, we would really appreciate hearing whether the round-by-round comparison with VLAW†, the VLM detector comparison, and our answers on multiple valid futures and state restoration address your concerns, or whether something is still unclear. We are happy to answer follow-up questions until then.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You to All Reviewers and Summary of Our Responses

We thank Reviewers kxjq, UXPq, and biNp for their careful reviews. We appreciate their recognition that the problem is real and important, that directing real data collection to where the world model is unreliable is a sensible alternative to collecting more on-policy rollouts, and that the controlled experiments help explain where the improvement comes from.

All changes in the revised PDF are highlighted in blue. The added evaluations are in Appendix E (Tables 14–18) and the measurement details in Appendix C (Table 13). We summarize the main points below.

Comparison with VLAW† across feedback rounds (kxjq, biNp). Both methods are extended to three feedback rounds on all six tasks. Our method has the higher success rate on every task after every round, and with 40 additional trajectories it matches or exceeds VLAW† with 60 on every task (Table 14). With 60 trajectories, the margins are 7.6–12.4 points in simulation and 12.5–25.0 points on hardware.

Uncertainty of the improvement (UXPq, biNp). Table 1 now reports success counts and 95% confidence intervals. Since all methods are evaluated on the same episodes and initial placements, we test the paired difference from VLAW† at the 40-trajectory stage. On hardware, with every placement evaluated twice, the three-task difference is +6.8 points (95% interval [1.6, 11.5], p = 0.0156; Table 15). In simulation, the differences are +4.1 [1.2, 7.1], +8.8 [4.1, 12.9], and +11.8 [6.5, 16.5] points on SinkFaucet, OpenCabinet, and PickPlace, and every interval excludes zero (our Nov 19 reply to Reviewer biNp). Three training seeds of our method on PickPlace give 72, 72, and 68 successes out of 170, against 52 for VLAW† (Table 10).

Reliability score (UXPq, kxjq). The labels of all 600 predictions in Table 5 are manually verified against simulator outcomes, and the AUROC of 0.97 is computed with these labels. The retained-error column now uses the 420 retained predictions as its denominator, as the caption defines; the error counts, AUROC values, and ordering of methods are unchanged. Used as detectors on the same 600 predictions, Qwen3-VL and Astra reach AUROCs of 0.79 and 0.88 (Tables 5 and 16). On FoldCloth, when the same observation and action admit multiple valid futures, the mean intrinsic score changes by 3.5% and one of 88 valid predictions is flagged (Table 17).

Restoration and collection on real robots (kxjq, UXPq, biNp). The robot restores the logged starting scene and replays the recorded action prefix, so intermediate states, including cloth configurations, are reached through execution. On the three hardware tasks, 18–19 of 20 requested interactions are reached, including 19/20 on bimanual FoldCloth (Table 8), in 45–108 seconds per completed rollout with zero hands-on human time (Table 12). Appendix C now specifies how imagined and reached states are measured and reports the accuracy of each estimator (Table 13). With 1–2 cm reset offsets on PickPlace, success decreases by 0.7–3.1 points and stays above VLAW† at the same budget (Section 4.6).

Presentation, setup, and scope (biNp, UXPq). The method section now presents the general formulation, with task-specific details moved to Section 4.1 and the appendices, and the reliability score is denoted by ρ. Reviewer biNp confirmed that the presentation is clearer. All methods in Table 1 start from the same π0 policy, which is already fine-tuned on the initial 150 trajectories. Table 2 now labels whether each encoder has undergone dynamics fine-tuning. Switching thresholds at the 90th, 95th, and 99th percentiles give 71, 72, and 72 successes out of 170 on PickPlace (Table 18). On scope, the paper addresses robot manipulation; the end effector is shared by π0 and by the manipulation systems on which DSRL and VLAW are evaluated, and our experiments cover three robots and articulated, rigid, and deformable objects.

Thank you again for your time and helpful suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.