acceptodds
Withdrawn from ICLR 2027

Human-in-the-World-Model for Scalable Robot Post-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

24 views

Abstract

Post-training robot policies through human correction requires repeated physical execution, scene resets, and operator supervision, limiting the scale of corrective data collection. We propose Human-in-the-World-Model (Hi-WM), a robot policy post-training paradigm that collects human corrective supervision inside an action-conditioned world model, without requiring execution on the target robot during collection. A pretrained policy runs in closed loop on model-generated observations, and a human intervenes when the rollout becomes failure-prone. State caching, rollback, and branching allow operators to revisit the same failure state and collect multiple corrective continuations without repeating physical execution. The generated observations and corrective actions are combined with real-world demonstrations to post-train the policy. Across three real-world manipulation tasks involving rigid and deformable objects and two policy backbones, Hi-WM improves success rates by an average of 37.9 percentage points over the base policies and 19.0 percentage points over a baseline that augments training with successful autonomous world-model rollouts. Increasing the amount of virtual corrective data further improves real-world success in the tested range. These results demonstrate that human corrections collected in a learned world model can improve physical robot policies, supporting world models as reusable environments for corrective post-training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Post-train

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Ve7bOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Hi-WM, which collects human corrections inside a learned action-conditioned world model instead of on the physical robot. A pretrained policy (Diffusion Policy or π0) runs in closed loop in the world model. An operator takes over by keyboard, leader arm or VR when the rollout looks likely to fail. Cached states allow rollback and branching into multiple corrective continuations. The world model uses the encoder-dynamics-decoder architecture of Wang et al. (2026), trained with added failure and edge-case data. The method is tested on three bimanual YAM tasks: Fold Towel, Push-T and Route Rope. Hi-WM improves real-world success over the base policy and over WM-CL, a baseline that fine-tunes on successful autonomous world-model rollouts with retained data matched. About half of the gain over the base policy remains after this data-matched control (Table 2).

Critical strengths
  1. The real-robot evaluation covers two policy classes and three tasks, two of them with deformable objects. All methods within a task use the same initial configurations (Table 2, App. B.1).
  2. WM-CL is matched on retained trajectory count and action steps (B.3). This controls for added data volume, so it isolates human-corrected world-model data from autonomously generated world-model data.
  3. Edge-case augmentation reduces positioning error between world-model targets and robot execution by more than an order of magnitude (Fig. 6). This is the property that lets corrections made in the world model transfer to the robot.
  4. The cost projection is explicitly labeled illustrative, not measured (Fig. 7b caption). The convention for counting shared prefixes is disclosed (B.3).
  5. Gains hold on held-out Push-T scenes with appearance, background and distractor variation (Fig. 8). This suggests the corrections do not overfit to the training scene.
Critical weaknesses
  1. The new claim is not examined on its own. The stated observation, that the most valuable supervision lies near the learner's own failure states (Sec. 1), is the premise of DAgger and HG-DAgger (Ross et al., 2011; Kelly et al., 2019). The paper borders on presenting a system rather than a method. The underlying idea is HG-DAgger with the physical robot replaced by a world model, plus state caching. The new claim, that a learned world model can stand in for the robot in the correction loop, is never tested directly. The question that decides whether this works is whether the states the policy reaches in the world model, and the outcomes of human corrections there, match those on the real robot. No measurement addresses it. Two that would:

    • comparing failure modes or state distributions of the same policy in the world model and on the robot;
    • replaying collected corrections on the robot from reproducible failure states, which is feasible at least for Push-T.
  2. Unestablished world-model coverage where the policy lacks it. The approach offloads the policy's out-of-distribution issues onto the world model. It never establishes under what circumstances the world model can be expected to have data and accuracy where the policy does not.

    • Failure data is a plausible source of this asymmetry, since it informs dynamics but cannot be imitated. Its source and amount are not reported, however. If it comes from real rollouts of the base policy, robot execution and resets are still required. The saving would then be decoupling operator time from robot time, which is narrower than the claim that collection "requires no execution on the target robot" (Sec. 1).
    • For the edge-case data, it is unclear why the policy is not also trained on it.
    • World-model fidelity is evaluated only near demonstration states (Fig. 6, Table 1, Sec. 3.2.2). It is never evaluated at the policy's failure states, or for corrective actions that do not appear in the training data.
    • Prior-information symmetry: the world model sees real data the base policy does not. This affects the comparison against Base and the cost argument. It does not affect the comparison against WM-CL, which shares the same world model.
  3. The additional burden on the human is not characterized.

    • Not reported: operator-hours, number of operators, interventions, branches per branch point, collection wall-clock time, world-model inference rate, and the input device used for the reported results.
    • Relative to physical correction, the operator works from a generated single low-resolution view with no force feedback. The operator also audits the plausibility of world-model outcomes, since human judgment takes precedence in screening (B.4). Operators may unknowingly exploit optimistic world-model dynamics, and branching increases the number of decisions per failure.
    • Without these measurements, it is not possible to characterize whether this burden is worth the gains. One option is success versus cumulative operator-hours for Hi-WM (with world-model data collection counted as a fixed cost), WM-CL, physical HG-DAgger, and additional targeted demonstrations.
    • Fig. 7b, an illustrative projection with undisclosed assumptions, appears in the teaser Fig. 1 under "Cost effective scaling." The collection-time analysis in App. D is an accounting identity with none of its terms measured. The scaling result (Fig. 7c) has two increments, for DP only, and no WM-CL curve at matched budgets. That is too few points to show a trend or a scaling advantage over the alternative.
Questions for authors

Questions

  1. What are the source and amount of the failure and edge-case data used to train the world model? Does collecting it require running the policy on the robot?
  2. How do Hi-WM corrections compare with physical HG-DAgger corrections at matched operator time?
  3. What are the measured operator-hours, interventions, branches per branch point, input device and world-model frame rate?
  4. What fraction of collected corrections succeed when replayed on the real robot from matching initial states?
  5. Was WM-CL collected over two rounds, like Hi-WM? Is the world model retrained between rounds?
  6. Deformable dynamics from a single top view are where world-model accuracy for corrective actions seems least likely. What explains gains over WM-CL on Fold Towel and Route Rope that are comparable to Push-T?
  7. What is the automated rollout assessment? Does post-training use only intervention segments or full trajectories, and how are shared prefixes weighted?
  8. Is there one world model per task? What checkpoint initializes it, and does π0 start from the public pretrained checkpoint?
  9. Which backbone is used in Fig. 8, and how does WM-CL perform on the held-out scenes?

Minor and presentation

  • Sec. 3.2.1 cites an edge-case proportion ("approximately 30% of the regular-scene data volume") that does not appear in Table 1, which also uses a different denominator.
  • WM-CL is used in Sec. 3.1 before it is defined under Baselines.
  • Neither Fig. 7a nor Fig. 8 states its number of points or which backbone it uses.
  • Trial counts are repeated in the Table 2 caption, Sec. 3.1 and B.1.
  • Reference formatting: the 1X World Model entries.
Reviewer_2XhHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents a system for post-training robot control policies. The system allows to alter the robot policies with human demonstrations in a simulated environment.

Critical strengths
  1. The very idea of demonstrating robot behavior by a human in a simulation is interesting and inspiring. It is indeed beneficial in robot control problems where dexterous robot behavior can be presented by the human.
  2. The paper provides in proper, economical, terms a cost-benefit analysis of the presented approach.
Critical weaknesses
  1. The proposed approach is limited to robot control problems where (1) human demonstration is possible, (2) a world model is available and precise and (3) we prefer to not use the real robot in the (post-)training. This class of problems seems very narrow because (1) the robot need to be similar to a human hand or a pair of hands, which it not typical, and (2,3) if we prefer not to use the real robot, then we are not able to collect rollouts to identify the precise world model.
  2. Perhaps I missed something, but I never found the form of the robot control policy and how this policy is adjusted with the human demonstration. Without this information, it is difficult to discuss possible alternatives.
  3. The paper proposes to do something in simulation what is usually done in the real world. It is an economic novelty rather than technical.
Questions for authors
  1. What is the form of the robot policy and how is it adjusted with the human demonstrations?
  2. Please provide examples of a human demonstration that fixed an erratic robot behavior and could not be fixed with any form of automatic optimization of this behavior.
Reviewer_2b77OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a simple but effective method to collect human intervention in the interactive world simulator to build more robust and reliable robot policy. The paper shows impressive improvements on the downstream robotic manipulation tasks

Critical strengths
  • The idea is simple, intuitive and effective. While the idea itself is not super complicated and novel, I do appreciate such research that combines the right model with right technique to enable right downstream capabilities.
  • Real robot experiments show the effectiveness of the proposed method.
  • The paper is easy to follow.
Critical weaknesses
  • Figure 7 (b) might not be necessary. As a robotics research paper, it should only present robotics results authentically instead of projecting cost, which involves much more complicated market and economic factors and is totally out of this work's scope. This figure is also not very informative for the research insights and takes.
  • While the paper is very simple and effective, I think there can be more thorough analysis of the method. For example, the paper can be strengthened from a figure of x axis of data size (not data round) vs y axis of success rate. It could give readers better insights into how effective and valuable the human intervention data is compared to densely generated data. Although the policy improves, we need to disentangle factors of the data size and the data quality.
  • Does generalization of the policy shown in Section 3.3 come from the contribution of human intervention or the base policy?
Questions for authors
  • How is augmentation data in Table 1 exactly collected? For example, do authors replay the corrected data from WM in the real world to get the data? If yes, how do authors guarantee the consistency between the data in WM and real-world data?
  • Some very important but missing related works, including EXPO-FT and pi0.6
Reviewer_FtmLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel post-training paradigm that leverages a Human-in-the-World-Model supervision philosophy. By leveraging advances in world models, the authors demonstrate a system that simulates model performance and lets a human step in and correct uncertain or failed situations, which is significantly cheaper than doing so on a robot. By caching and rolling back to problematic states, the human can produce multiple supervised teleoperations instead of waiting for the robot to fail multiple times on a real system. Experiments cover three real robot manipulation tasks and two policies that, because of their supervision, improved substantially. The fundamental question answered in this work is whether world models can offer a cheaper substrate for collecting interventions and corrective demonstrations.

Critical strengths
  • Compelling practical motivation: It is no secret in the robotics community how painstaking data collection is for real hardware, and therefore any system that enables more efficient and impactful data collection is well motivated.
  • Substantial downstream improvement: The post-trained policies on Hi-WM demonstrate unquestionable improvements over the pretrained policies and, surprisingly, also generalize to novel environments.
  • Evaluation on real robots. The hallmark evidence is grounded in real-robot performance, not simulation, which further strengthens the impact of this work. Addresses the sim2real transfer question that often arises for demonstrations collected in simulated/world model environments.
Critical weaknesses
  1. Baselines do not establish the advantage of collecting corrections in a learned world model. While WM-CL is a useful alternative for filtering these policies, I would argue that this is not the primary baseline to compare against. The core contribution/claim of this paper is that collecting demonstrations in a world model is better than other existing paradigms, so with a fixed collection budget, other paradigms should also be used to collect post-training datasets. The most informative missing comparison is real-world human correction under a matched operator-time budget, ideally accompanied by an additional-real-demonstrations baseline. Even on one representative task, this would establish the practical tradeoff between correction quality and collection cost.

  2. Rollback and branching are not evaluated in isolation. I think the system needs ablation studies to motivate key architectural decisions. The current experiments compare the complete procedure with autonomous rollout augmentation. A focused ablation that reports both collection time and downstream success, along with the number of distinct branch points and continuations per point, would substantially strengthen the paper's claims.

  3. Analysis of the world model and discussion of the limitations feel insufficient. While world models can offer visually comparable/stable rollouts, it is also important to consider whether the consequences of interventions and actions align between the world model and the real robot. I'd be curious to see analysis of whether a corrective action/intervention in the world model reflects an intervention you would see in the real robot. Basically, the analysis should extend to similarity in action interventions seen in both.

Questions for authors
  1. At equal operator time, how does Hi-WM compare with physical human correction?
  2. What are the measured collection times, rejection rates, inference speed, and additional real-data requirements?
  3. How much does rollback improve collection efficiency or downstream performance over virtual correction without rollback?
  4. Do candidate corrective actions produce matching outcomes in the model and reality at failure-prone states?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.