acceptodds
Withdrawn from ICLR 2027

What Data Makes World Models Better Manipulation Planners? Beyond Successful Demonstrations with Diverse Action Experience

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models equip robots with the ability to predict the physical consequences of candidate actions prior to execution. However, the interaction data used to train these models is typically dominated by successful expert demonstrations. While such data ensures broad task-relevant state coverage, it offers limited experience of how alternative actions from similar states yield distinct future outcomes. In this work, we systematically investigate how the composition of robot interaction data influences world-model planning. We introduce Diverse Action Experience (DAE), a data-collection framework that branches alternative action trajectories from intermediate states of expert demonstrations, systematically allocating the interaction budget between broad state coverage and dense local action-outcome coverage. Evaluating multiple world-model architectures on the LIBERO benchmark, we measure performance through candidate-action ranking and closed-loop manipulation. Our experiments reveal that data composition substantially impacts planning behavior: varying state coverage, perturbation regimes, and interaction durations produces substantial differences in ranking fidelity and downstream task success. Furthermore, the preferred data composition depends strongly on the underlying world model. We also show that high aggregate ranking accuracy can obscure degenerate candidate selection, and that stronger candidate ranking does not automatically guarantee robust closed-loop control. These findings establish interaction-data composition as an important design dimension in world-model planning, alongside model architecture and planner design.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Joint-Embedding Predictive Architecture (JEPA), World Models, Robot Manipulation, Model-Based Planning, Trajectory Data Collection, Interaction Budget Allocation.

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_W2jJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Diverse Action Experience (DAE), a framework that collects diverse action branches from intermediate demonstration states. It studies how interaction-data composition affects world-model planning. The effects of state coverage, local action diversity, and trajectory duration vary across models. The finding, better candidate-action ranking does not always lead to better closed-loop task performance, is rather interesting.

Critical strengths
  1. Important research question. What data helps world models plan better is a question that many researchers care about. The paper studies an important problem in world-model learning.

  2. Good experimental design. The experiments compare different data compositions and test both candidate-action ranking and closed-loop task performance. These experiment provides insightful analysis.

  3. Detailed analysis. The paper analyzes how different models respond to changes in training data. It also shows why a high ranking score does not always mean better planning.

Critical weaknesses
  1. Limited practical guidance. The paper analyzes the experimental results in detail, but does not turn these observations into clear advice for training world models. The main conclusion is that data composition matters and that different models benefit from different data. I find this conclusion somewhat unsurprising. It is still unclear how these findings can help researchers decide what data to collect or how to build their training datasets.

  2. Limited evaluation scope. All experiments are conducted on LIBERO. The tasks may be too simple to test whether the findings hold in more complex settings. Models may perform well by fitting patterns specific to this benchmark, without learning more general planning capabilities. Testing on harder and more diverse tasks, such as those in BEHAVIOR-1K and RoboCasa, would make the conclusions more convincing.

  3. No real-robot validation. I consider this a critical weakness for a robotics paper because of the large gap between simulation and the real world. Real robots face perception errors, different contact dynamics, and imperfect action execution. These factors may change the conclusion on which training data is most impactful. Without real-robot experiments, it is hard to tell whether the findings can transfer to real-world manipulation.

Questions for authors
  1. Can the authors conclude their empirical observations into actionable guidance for constructing world-model training datasets? Without such guidance, it is difficult to clearly position the paper’s contribution beyond documenting model-dependent sensitivity to data composition.

  2. Can the authors extend the evaluation to more challenging benchmarks, such as BEHAVIOR-1K or some other LIBERO variants with harder and more diverse tasks, to assess whether the findings generalize beyond the current setting?

  3. Can the authors provide real-robot evaluation to determine whether the findings remain valid under real-world perception noise, contact dynamics, and execution uncertainty?

Reviewer_UKReOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how the composition of interaction data affects world-model planning for robotic manipulation. Its central question is: under a fixed interaction budget, should data collection cover more task-relevant states or gather more diverse action outcomes from fewer states? The authors propose the Diverse Action Experience (DAE) framework, with three main components:

  1. Interaction data that supplements successful demonstrations: In addition to a shared dataset of successful demonstrations D0D_0, the framework collects task-relevant interaction trajectories without requiring them to succeed, enriching the state–action–outcome relationships represented in the training data. The world model learns from these data through predictive learning, without task-success, reward, or progress labels.

  2. DAE construction through root states and action branches: An expert demonstration prefix is replayed to reach an intermediate task state, which serves as the root for multiple action branches. These branches vary the approach, grasp location, transport path, and release location to collect different action outcomes from similar starting states. Only the newly collected branch suffixes are added to the training data, while the execution cost of the expert prefix is included in the acquisition budget.

  3. A framework for controlled comparisons of data composition: By varying the number of root states and the number of branches per root, the authors construct D-BS configurations emphasizing broad state coverage and D-LS configurations emphasizing local action-outcome coverage. They further examine action perturbations and interaction-segment duration. Within the corresponding comparisons, the model architecture and training procedure are held fixed to study the effects of data composition.

Critical strengths
  • A well-motivated focus on interaction-data composition.

    The paper examines an important distinction between learning to imitate expert behavior and learning dynamics for planning: planners must distinguish the consequences of alternative actions. Framing data collection as a trade-off between broad state coverage and local action-outcome coverage provides a useful perspective beyond model architecture and planner design.

  • A structured framework for collecting task-relevant alternative-action experience.

    DAE branches multiple action trajectories from intermediate demonstration states, preserving task relevance while exposing the model to outcomes beyond expert execution. Retaining branches without filtering for final success or requiring reward and progress labels is a practical design choice. The root–branch formulation provides a concrete way to investigate different allocations of interaction experience.

Critical weaknesses
  • The central budget-allocation question remains insufficiently answered.

    The paper investigates the trade-off between broad state coverage and local action-outcome coverage under a fixed interaction budget, but the direct evidence is largely limited to candidate-action ranking in Table 3. For JEPA-WM at H=10H=10, both D-BS and D-LS underperform D0D_0, while their advantages at other horizons are inconsistent. The subsequent perturbation and duration experiments demonstrate configuration sensitivity without directly establishing how root–branch allocation affects closed-loop success. Consequently, the results do not sufficiently identify when each coverage strategy is beneficial or provide reliable guidance for budget allocation.

  • Key experimental configurations, training details, and concepts are insufficiently specified.

    Although the paper outlines the training objectives and evaluation framework, it does not adequately report the root and branch counts, interaction steps, perturbation magnitudes, segment lengths, or training sampling ratios for each data condition, making budget matching and comparison fairness difficult to verify. The relationship between Small/Short and Large/Long under Perturbation and Short and Long under Duration in Table 2 is also unclear. Furthermore, the construction and retrieval of the shared candidate pool, and the use of a shared goal scorer across different models, lack necessary detail. These omissions hinder both reproducibility and attribution of the observed performance changes.

  • The experimental objectives, evaluation metrics, and central conclusions are not connected by a clear chain of evidence.

    The state/local-coverage experiments primarily report ranking accuracy, whereas the perturbation and duration experiments primarily report closed-loop success, without directly connecting the same budget-allocation changes to both metrics. Metric interpretation is also ambiguous: Section 4.2 defines Top-1 as agreement with the true best candidate, yet Section 4.5 and Appendix A report an oracle result of only 40/50 without clarifying whether this instead measures task success. Moreover, regret is defined and stated to be reported, but corresponding results are absent. The experiments therefore more strongly support configuration sensitivity than further conclusions about how data allocation improves planning.

Questions for authors
  • Section 4.2 states that both Top-1 agreement and ranking regret are reported, but I could not locate any regret results in the main text or appendices. Could you provide these results or revise the statement accordingly?

  • The introduction asks whether a fixed interaction budget is better spent on broader task-relevant state coverage or more diverse action outcomes around fewer states. What answer does Q1 provide to this question, and under which conditions does either allocation perform better? Could you clarify how Q2–Q4 and the recovery-oriented data-collection experiment in Appendix C contribute to answering this central question?

  • In Tables 1 and 2, what do “Short” and “Long” mean under Perturbation, and how do they relate to “Short” and “Long” under Duration? Do they refer to the duration of the perturbation, the total collected trajectory length, or another quantity? Could you specify the perturbation magnitudes, durations in environment steps, and factors held fixed in each comparison, including how the interaction budget is matched?

  • What additional insight does Any Success provide beyond Terminal Success for studying data composition? Could you explain why transient task completion is relevant to the manipulation tasks considered here, and whether the gap between these metrics reveals different effects of the DAE configurations?

  • How is the shared candidate pool constructed, and how are the eight candidate action sequences retrieved for a given state? Could you specify their source, sequence lengths, retrieval criteria, and ordering, and clarify whether the pool is independent of the DAE training-data conditions? Given the strong fixed-slot-0 baseline, what determines which candidate occupies slot 0?

Reviewer_XkEJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how training-data composition affects WM planning. Starting from a dataset of successful demonstrations, it varies the allocation of additional experience between demonstration-derived root states and action branches from those states, evaluating two allocations for training and candidate action selection for two world models. It also studies how action perturbation and interaction duration affect closed-loop manipulation success for two other world models.

Critical strengths
  1. The paper puts forward an important and timely question for world models: what training data enables better closed-loop manipulation. - It aims to evaluate different ways of data collection to answer this question across several different world models, which is a useful study.
Critical weaknesses
  1. Evaluation, which is the primary aim of this paper, is very lacking and consistently very vague:
  • Only LIBERO in simulation. World Models are trained on much broader and more diverse data, and the resulting data-performance dynamics will likely be substantially different. Drawing conclusions based on just LIBERO training data is not very substantive.
  • not using the planner (always choosing candidate 0) in Table 4 is 54% success while planning with the 2 WMs are 8% and 18%, which suggests something is wrong with the setup
  • only 2 WM's are explored per data factor, and for some reason they are different for state / local coverage vs perturbation / duration
  • No explicit trial counts for perturbation / duration success rates in Tables 1 and 2
  • the actual variations and configurations aren't explained: just "D-BS for broader state coverage", "D-LS for denser local action-outcome coverage", ... and "short", "small", "mixed".
  • the framing in Section 3 discuss state vs local branching, but many results are on perturbation and duration, which are not explained. Moreover the state / local coverage metrics are only Top-1, whereas the other factors are closed-loop success.
  • Top-1 ranking agreement is used throughout the evaluation section but it is not a good metric for world-model planning. Actions can be not top-1 but have very little or equal regret with respect to the oracle action.
  • action proposal policy / mechanism is never explained
  1. It's unclear what the data collection "framework" being contributed is. The paper studies how different factors affect WM training and performance for closed-loop planning, but not an actual framework for systematically deciding what data to collect
  2. Sections 3.4, 3.5, 4.2 and Figure 2 are simply explaining standard world model training and planning, using considerable space while actually central details about data-selection and generation are not present.
Questions for authors

See weaknesses. A lot of details about the collection and evaluation are missing. Further, what framework for collecting data is recommended given the experimental findings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.