- Rating
- 2
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
This paper aims to improve pre-trained robot policies by fine-tuning them inside a world model. The policy is fine-tuned on imagined rollouts, but these trajectories may drift from the real world and affect policy learning. To address this, the authors define a reliability score for state-action pairs. The score combines the model’s error in predicting the state (velocity-field self-consistency) with its action consistency. Imagined rollouts stop when this score exceeds a threshold, after which they continue in the real world or simulator to correct the model. Specifically, the authors reset the simulator or robot configuration to the beginning of the imagined trajectory and replay the actions taken in the world model until they reach the failure point. The rollout then continues in the real world or simulator. Experiments are conducted on three simulated tasks and three real-world tasks, using as the pre-trained policy and a video diffusion transformer as the backbone of the world model. The results show improvements over the base policy and two baselines. Additional experiments examine the role of the reliability score.
Critical strengths
The underlying problem is real and important: policy learning in imagination is prone to various errors, and this work aims to address them directly. The method builds on recent work and is evaluated in both real-world and simulated experiments.
Critical weaknesses
My first concern is the paper’s clarity and writing. The idea is simple, but I had to reread some passages to understand it. Parts of the method section describe task-specific details that could be moved to the experiments section. The text also presents the contact signal as a separate model input; why is it not included in the proprioceptive state? Many details about the method and experiments are missing: how the action chunk in is obtained, how and were set, how the threshold should be chosen, how the data in Sections 4.4 and 4.5 were collected, and what “disagreement” means in Section 4.4. A complete and clear description of how the models are fine-tuned is also missing. Finally, denoting the reliability score by is extremely confusing, since this notation is conventionally used for the reward.
The method seems tied to robots with end effectors, and this assumption is hard-coded into parts of the method and experiments, limiting its scope and applicability. Resetting to a state is also usually expensive and is not always feasible in practice.
The experiments have several weaknesses.
In Table 1, is not directly comparable to the other methods because it has not seen the 150 trajectories, and some of the improvement are attributable to just seeing those additional data. This is also apparent when comparing the performance gaps on real and simulated tasks: distribution shift hurts performance more on real tasks, where the gap between and Imagined is larger. Moreover, confidence intervals are mentioned in the text but omitted from the table. There appears to be substantial overlap between the confidence intervals for VLAW and the proposed method, but the table does not show this.
In Section 4.3, the world model with the contact signal as an additional input is labeled “ours” and compared against a vision-only value model to show that it is more responsive to actions. I do not see what makes adding this input specific to the proposed method. If the signal is available, any method could incorporate it and potentially benefit from the richer input.
Section 4.6 reports the error in resetting to a state. First, the measurements could be normalized relative to the initial scene and reported without units to make them more intuitive. Second, the experiment does not seem directly related to the method, and its purpose or importance is unclear.
An important missing experiment is an evaluation of sensitivity to the threshold used for switching to the real world/simulator.
Given the weaknesses in the experiments, the method’s limited scope and generality, and the rushed presentation, I lean toward rejecting the paper.
Questions for authors
Please see weaknesses.
6 comments
Action consistency, score settings, and evaluation data
Section 3.3 describes how the action chunk in is obtained, and Appendix A (Table 9) specifies the settings of , , and . The action chunk used in is recovered from the predicted video: a visual probe extracts end-effector poses, and a three-layer inverse-dynamics MLP trained on demonstrations maps consecutive states to actions. The recovered actions form , whose discrepancy from the commanded chunk gives . To combine this signal with , we set and so that the weighted terms and each have an in-distribution mean of 0.5. The threshold is then set to the 95th percentile of the combined score on held-out in-distribution predictions and shared across tasks, as specified in Table 9 and Section 3.4.
For the detection study in Section 4.4, Appendix B specifies the trajectory-level split and the comparison between imagined and executed outcomes of the same action. For each context from held-out PickPlace trajectories, split at the trajectory level, we generate a prediction and execute the commanded chunk in the simulator. “Disagreement” refers to a mismatch in grasp contact, object displacement, or gripper state—for example, a predicted grasp-and-lift when the object actually remains ungrasped and stationary. Simulator outcomes are read directly from state; preliminary labels from the probe are manually verified, with annotation tolerances set on in-distribution samples. The evaluation uses 600 predictions, including the 200-prediction annotated subset for detection validation. The policy ablations use the same two-round budget of 20 additional trajectories per round.
For the model-correction study in Section 4.5, Appendix B specifies a validation set fixed before collection. We use 300 executed transitions and use the initial score to divide them into 150 suspicious and 150 ordinary transitions. Their identities and labels remain unchanged, and the trajectories are excluded from collection, refitting, and setting the score weights and threshold, so correction is evaluated on the same transitions before and after model updates. The revision makes the cross-references to these existing definitions and protocols more explicit and clarifies the manual verification procedure and the 200/600 subset relationship.
End effectors and the manipulation setting
This work studies the improvement of -based robot manipulation policies, with DSRL and VLAW evaluated in the same manipulation setting (Sections 3.1 and 4.1). End effectors are part of the platforms and tasks under study; reading their motion is how we implement the check that predicted manipulation follows the commanded actions.
The initial policy has seen the 150 trajectories
The policy in Table 1 has already undergone supervised fine-tuning on the initial 150 trajectories (Sections 3.1 and 4.1). Imagined starts from this same policy and improves it inside the world model without collecting additional real trajectories. Its improvement over is therefore obtained using the same initial real data.
Physical collection: purpose, cost, and normalized errors
As stated at the beginning of Section 4.6, the real-data collection stage requires the robot to reach the interaction selected by the reliability score so that its actual outcome can be collected. This is the direct purpose of the restoration-and-replay evaluation. Following the procedure in Figure 3, the robot restores the logged start and replays the action prefix. OpenJar, StackBox, and FoldCloth reach 19/20, 18/20, and 19/20 requested interactions, respectively (Table 8). The task-wise mean restoration time is 33–86 seconds and replay time is 12–22 seconds per completed rollout, with zero hands-on human time (Table 12).
Following the suggestion to provide a dimensionless reference, we express the measured errors relative to the relevant object sizes: the 14 cm lid diameter, 25 cm box side, and 38 cm spread garment width. Restoration centroid errors are 8.6% and 7.2% on OpenJar and StackBox; object-position errors after replay are 10.0% and 7.2%, respectively. The FoldCloth keypoint error is 6.3% of garment width. The corresponding tables and captions now include these reference sizes and percentages.
Section 4.6 also tests the effect of restoration error on policy improvement. Adding 1, 2, and 5 cm object-position offsets before replay in PickPlace gives Online-2 success rates of 41.7%, 39.3%, and 30.7%, compared with 42.4% under exact resets. Thus, the section measures both whether targeted collection can be carried out on hardware and how restoration accuracy affects the learning loop.
Confidence intervals and paired differences
Table 1 now includes the success counts and 95% confidence intervals for each entry. Since the methods are evaluated at the same initial placements, we also quantify uncertainty in their paired success-rate differences, accounting for the correspondence between the evaluations.
For the hardware policies trained with 40 additional trajectories, we repeated evaluation at the same 32 initial placements per task, giving two trials per method per placement. The table below reports the counts and success-rate differences, with 95% basic paired cluster-bootstrap intervals computed by resampling initial placements within each task and keeping both repetitions together. Across the three hardware tasks, the mean advantage is 6.8 percentage points, with a 95% interval of [1.6, 11.5] points and a two-sided paired permutation p-value of 0.0156.
| Task | Ours | VLAW† | Difference (pp) | 95% CI for the difference (pp) |
|---|---|---|---|---|
| OpenJar | 38/64 | 34/64 | +6.3 | [-3.1, 12.5] |
| StackBox | 21/64 | 14/64 | +10.9 | [0.0, 20.3] |
| FoldCloth | 22/64 | 20/64 | +3.1 | [-3.1, 6.3] |
| Three-task aggregate | 81/192 | 68/192 | +6.8 | [1.6, 11.5] |
We also extended both methods to a third feedback round, with 60 additional training trajectories. Our success counts versus VLAW† are 23/32 versus 18/32 on OpenJar, 15/32 versus 7/32 on StackBox, and 15/32 versus 11/32 on FoldCloth. The corresponding margins are 15.6, 25.0, and 12.5 percentage points, providing further evidence of improvement with continued feedback. These results are reported separately by training stage.
Switching-threshold sensitivity
Section 4.4 reports a threshold-sensitivity test on the 200 annotated predictions: halving or doubling changes at most four decisions (2%). The default threshold is the 95th percentile of the combined score on held-out in-distribution predictions, as specified in Appendix A, Table 9.
We additionally evaluated the effect on final policy performance by comparing the 90th-, 95th-, and 99th-percentile thresholds on PickPlace. All conditions use the same calibration set, initial checkpoints, and other training settings, with two feedback rounds of 20 additional trajectories each. Evaluated on the same 170 episodes, the final Online-2 policies achieve 71/170 (41.8%), 72/170 (42.4%), and 72/170 (42.4%), respectively; the 95th-percentile entry is the original Table 1 result. The maximum difference is one success, or 0.6 percentage points. Policy performance therefore remains stable across these threshold settings at the same data budget.
Thank you again for your comments. We look forward to your feedback on the revision.
Thank you for your review. We address the presentation and methodological questions, clarify the shared initial data, and report additional hardware and threshold-sensitivity results in three parts. Changes in the revised manuscript are highlighted in blue.
Presentation and task-specific details
The revised method section focuses on the general formulation and the main steps of the loop: predicting and valuing action outcomes, assessing prediction reliability, and using the score to connect imagined updates with real feedback. We have moved the task-specific contact-signal sources and success-judgment procedures from Section 3.2 to the experimental setup in Section 4.1, placed the trigger-grid size and per-bin quota in Appendix A, and consolidated the FoldCloth restoration details in Appendix C. The method section retains the definitions, equations, and general execution procedure, with cross-references to these implementation details.
Contact inputs and their role in the value function
The world model receives observations o, robot proprioceptive state s, the commanded action chunk aˉ, contact signal c, and the task instruction. Here, s describes the arm configuration and gripper opening, while c indicates whether an object is held. The first imagined chunk uses recorded real observations and states. Subsequent imagined chunks receive no new sensor readings, so their state inputs must be computed.
For s, we use the current proprioceptive state and the arm-motion and gripper-opening changes specified by the action chunk to compute the next proprioceptive input. For c, a command specifies that the gripper should close, but does not tell us whether it will hold the object or close on empty space. A contact head trained with real-trajectory contact labels therefore predicts the end-of-chunk contact state, which becomes the next chunk’s contact input. We write s and c separately to expose these two update rules. Concatenating them into one augmented state vector would leave the computations unchanged (Sections 3.2 and 3.4).
Contact sensing is an available input, and we do not claim its inclusion as a standalone contribution. Section 4.3 examines the value function used to improve the policy through action-value gradients: action conditioning enables it to distinguish candidate actions, and contact information helps it respond to grasp failures with little visual change. Table 2 isolates the contact contribution within the same action-conditioned critic: adding contact raises ranking accuracy from 78% to 96% and lowers return NRMSE from 0.15 to 0.08. This supports the choice of critic within our reliability-guided policy-improvement and data-collection loop. We have replaced “Ours” in Figure 5 with “Action- and contact-conditioned value” to identify the compared model directly.
Model fine-tuning and policy updates
The update order is described in Sections 3.2–3.4 and the algorithm in Appendix D, with optimization settings in Appendix A (Table 9). After collecting 20 trajectories in each round, we continue from the previous checkpoints, mixing newly collected data and replay data sampled uniformly from the original dataset in a 1:1 ratio. We first update the dynamics model with the flow-matching objective, then freeze the shared DiT and fit the Q-head using fitted Q evaluation; the contact head is trained against recorded contact labels in the same stage. Dynamics and Q fitting use batch sizes of 2 and 8, respectively, with 100k steps and a learning rate of 4×10−5 for each.
The updated world model and critic are then frozen for the next imagined policy-improvement cycle. QAM updates only the policy’s rank-32 LoRA adapters using action-value gradients; the simulation policy-training setting is 5k steps at a learning rate of 10−6 (Table 9). Thus, collected real outcomes first update the prediction and value models, which then guide the next policy-improvement cycle. The revision consolidates the checkpoint-continuation and data-mixing details and explicitly states in Table 9 that its model-training settings are reused for each feedback round.
Reliability and reward notation
To distinguish prediction reliability from reward, the revision denotes the reliability score by ρ(s,aˉ) and its two components by ρintr and ρext, while retaining rt for reward. This notation is used consistently in the text, equations, figures, and algorithm.