acceptodds
Under review as a conference paper at ICLR 2027

DexSimWAM: Scaling Dexterous World Action Models with Verified Simulation Data

Abstract

Learning transferable dexterous manipulation policies is constrained by the scarcity of large-scale robot data. Simulation offers scale, but synthesizing dexterous trajectories is substantially harder than for grippers: the data must cover diverse multi-finger contacts and hand–object configurations while satisfying language-specified grasp and motion semantics. We introduce DexSimWAM, which combines an agent-guided, retrieval-based synthetic-data pipeline with large-scale World Action Model (WAM) pre-training and task-specific post-training. Given a language-specified task and verifier feedback, the agent guides scene sampling, grasp retrieval, optimization-based path planning, and bounded expansion of a physically validated object-frame grasp bank. Deterministic filters and fixed whole-task verification yield a 16k-hour corpus spanning diverse objects, scenes, grasp styles, and unilateral and bimanual behaviors. On this corpus, DexSimWAM jointly pre-trains visual-dynamics prediction and synchronized arm–hand action generation. Task-specific post-training then adapts the shared WAM with scalable simulation data and targeted real-robot trajectories, supporting 6D object reorientation, functional pouring, articulated-object interaction, long-horizon semantic rearrangement, direct sim-to-real deployment, and physical-domain refinement. Project website: https://dexsimwam-anonymous.github.io/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, World Action Models, Agentic Robotics, Recursive Self-Improvement, Sim-to-Real

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gygZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an agent-guided synthetic-data pipeline for dexterous manipulation in which an LLM agent configures scene sampling, grasp retrieval, and planning, while deterministic filters and a whole-task physics verifier gate every trajectory; the result is a 16k-hour physically verified corpus used to jointly pre-train a dual-DiT world-action model (video dynamics + arm–hand action generation); the shared checkpoint is adapted per task via post-training on ~100 simulation or real-robot trajectories, and evaluated on six unilateral/bimanual tasks in both domains under base, position-, and visual-generalization settings, with ablations on data scale, adaptation, and sim–real composition.

Critical strengths
  1. The paper shows considerable engineering effort in constructing the data synthesis pipeline.
  2. The paper presents a comprehensive evaluation across the 6-task benchmark, covering aspects such as spatial and visual generalizability.
Critical weaknesses
  1. The conclusions of this work are questionable due to the narrow task distribution in the pretraining data and inadequate task evaluation experiments. a. The data synthesis method lacks novelty. Prior work such as HumanoidGen employs a similar method to synthesize dexterous manipulation demonstrations, sharing almost the same task distribution as this paper and also supporting a post-rendering step. b. The task distribution presented in the appendix reveals a limited range of tasks within the pretraining dataset, which suggests low task diversity and raises concerns about the effectiveness of pretraining on such data. c. The number of distinct assets used in data synthesis is not reported in the paper, which may further exacerbate the lack of diversity in the pretraining data. d. The simulation evaluation appears to be an in-domain assessment, as the model is evaluated on simulation tasks from which its post-training data was synthesized. This experimental design lacks sufficient evidence to demonstrate the broad effectiveness of the proposed data synthesis method. e. The paper does not provide results on any open-sourced simulation benchmarks, making it difficult to systematically assess the model's generalization capability to unseen tasks or environments compared to other baselines.
  2. Flaws in experimental design and a lack of focus in the experiments make the results unreliable. a. The comparison is unfair: the baselines (PI05, LDA-1B, Fast-WAM) were pretrained on parallel-gripper data but only post-trained on dexterous manipulation, whereas DexSimWAM is pretrained and post-trained entirely on dexterous manipulation ("dexterous-hand-native"). This experimental design fails to provide adequate support for the authors' claim, casting doubt on its validity. b. Nearly half of the experiment section is devoted to reporting aspects unrelated to the claimed main contributions. For the relevant experiments, the authors spend most of the text describing the experimental setup but fail to provide sufficient analysis of the reported results. c. Only one task is used to evaluate pretraining scaling performance, which makes the conclusion prone to bias. The paper is not ready for publication due to the writing issues mentioned in the question section below and the concerns mentioned above.
Questions for authors
  1. Could you provide evaluation results and comparisons with other models on established simulation benchmarks (e.g., DexVerse, HumanoidGen, GR-1 Tabletop Robocasa)?
  2. Can the proposed data synthesis method support sim-to-real policy transfer using only synthesized demonstrations?
  3. Line 204: Could you please clarify what the "cuRobo constraints and budgets" generated by the LLM refer to? Providing concrete examples would be helpful.
  4. Line 206: Given that the framework provides preference context to allow agents to apply different sampling strategies, is there any quality control mechanism in place? Could this flexibility introduce bias into the synthesized demonstrations, potentially affecting pretraining effectiveness?
  5. Line 211: The "Drawer state coverage" does not appear to be shown in the figure. Could you clarify this? Additionally, is there a distinction between the state representation of an articulated object like a drawer and that of a rigid object like a bottle?
  6. In the data generation process, the feasibility of dexterous grasping is guaranteed, but its semantics are ignored. Could this neglect affect the semantic alignment of the collected demonstrations? (For example, "pour the water from a pot" implies grasping the handle rather than the rim.)
  7. The task categories in the pretraining dataset appear to be limited, primarily focusing on drawer opening/closing, transportation of specific objects, and some pick-and-place tasks. Could you report the number of assets used in synthesizing the pretraining dataset, as object diversity is crucial for diversifying instructions in general-purpose training? Furthermore, could you please explain what the "paired failure" task refers to in the pretraining dataset?
  8. Could you please explain your experiment design in response to the points mentioned in the "weakness" section above? Questions in this part can be asked and discussed only if we ensure the experiment design is complete and fair.

The following are some additional writing and formatting issues in the paper:

  1. Figure 1: The font size in the pipeline part is too small to read clearly.
  2. Line 215: There is excessive white space.
  3. Line 205: The phrase "fixed scheduler" is confusing.
  4. Lines 359 and 369: The bolded sentences are unrelated to their corresponding main text.
Reviewer_7AaZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a pipeline that generates 16,000 hours of physically verified simulation data for dexterous manipulation, then uses it to pretrain a WAM. Its main contribution is the data pipeline rather than a new model architecture.

Experiments on six simulated and real-robot tasks show that this pretraining helps, with ablations supporting the benefits of more data. However, each task still needs substantial separate fine-tuning, so the results mainly demonstrate a useful starting point for task-specific policies.

Critical strengths

The core contributions are a scalable pipeline for generating physically verified dexterous manipulation data, and evidence that more pretraining data improves downstream performance. The scaling results are promising, but cover only three data sizes on one task. Since performance is still rising at the largest scale tested, larger-scale and broader experiments would help establish how far the gains extend.

Critical weaknesses
  • Limited scaling evidence. The data-scaling study covers only three settings on one task. Performance is still improving at the largest scale, so the results establish a positive trend but leave the shape and limits of that trend unclear. More data scales and tasks would strengthen this central contribution.

  • Unclear adaptation efficiency. Each task requires a separate 60k-step fine-tuning run. Without downstream learning curves, it is difficult to tell whether pretraining reduces the adaptation cost or mainly improves performance after extensive optimization.

  • The pipeline’s gains are not fully isolated. The strongest configuration combines agent guidance, adaptive sampling, and grasp-bank expansion. More controlled ablations would clarify which components account for the throughput improvement, particularly how much the agent adds over simpler sampling strategies.

Questions for authors

See weaknesses

Reviewer_frmbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this work, the authors propose DexSimWAM, a method to pre-train a World Action Model (WAM) using a synthetic data pipeline that generates physically verified simulation trajectories. The pipeline is used to generate 16k hours of dexterous simulation trajectories for large-scale pre-training and task-specific post-training of a WAM. The results are subsequently evaluated on six distinct dexterous manipulation tasks in simulation and on a real robot.

Critical strengths

The core strength of the paper is that most of the setup around architecture and training (especially pre-training) is clear and the methodology is generally sound. There is a detailed discussion of the tasks, assets and simulation parameters. Similarly, the WAM architecture description is comprehensive and well-motivated. The description of the pre-training flow overall is complete and makes the resulting pre-training easy to reproduce. Lastly, the qualitative results demonstrated in the paper and supplementary material cover all tasks and add a lot of clarity.

Critical weaknesses

My main overall criticism is that for the experimental results, the paper's readability suffers from covering a lot of breadth, which affects the depth of analysis and leaves more questions than answers, with concrete examples below:

  1. Table 2 only shows the effect of pre-training and post-training on a single task each, which are two of the most important components in the pipeline. The effect of pre-training and post-training should be more clear.
  2. The work relies on a proprietary agent guide (GPT-6 Astra) and explores the use of a Jev guide as a comparison, but does not check whether other (open-source or closed-source) agents affect the data generation.
  3. If the models are currently post-trained per task (see Question 1 below), it would be very interesting to see a comparison of their generalization capabilities without task-specific post-training or a discussion of why such a comparison would not make sense. pi_0.5 and Fast-WAM are designed to be generalist models, so post-training on every single task seems to be costly without any additional discussion.
  4. Section H in the appendix in particular seems very confusing. It devotes nearly six pages to code agents and WAM orchestration which are not part of the main pipeline and it is not clear to me why it is included in the work, as it seems like preliminary exploratory analysis for future work that does not meaningfully help with the methodology described in the main paper. To a lesser extent, this is also true of introducing training-time RTC in Section F.

Overall, I think the readability of the paper could be substantially improved by focusing on the core contribution and rewriting or simplifying key sections. While my initial recommendation is a weak reject, I am open to raising my score if the weaknesses mentioned above and open questions are addressed and look forward to the discussion with the authors.

Questions for authors
  1. Do I understand this correctly that the baseline comparisons are for post-trained (i.e., fine-tuned) models for each specific task? For instance, on the "6D-Pose Transfer" task, are you comparing DexSimWAM post-trained on the "6D-Pose Transfer" task with Fast-WAM and pi_0.5 also post-trained on the "6D-Pose Transfer" task each and the comparison on a different task would then be each model post-trained on that specific task?
  2. What is the exact difference between the basic, position generalization and visual generalization settings? Most of the results are evaluated on all three settings, but the paper and appendix do not seem to include the exact difference between these evaluation settings.
  3. Is each post-trained model specific to its evaluation setting? (i.e., is there only a single post-trained DexSimWAM model for the "6D-Pose Transfer" task or does each evaluation setting have its own post-trained model)
  4. While the WAM architecture and inputs/outputs are clear to me thanks to the detailed description in the appendix, I cannot say the same for the LLM agent. Could you please provide an example input and output for the LLM agent or describe it in more detail?
  5. Table 15 shows that the PnP success rate on the real robot increases as the proportion of real post-training data increases. What is the advantage of using a 50:50 split or any simulation data in post-training in this case? Did you explore the case where all real post-training data is retained and augmented by simulation data? Does the real robot success rate increase or decrease in that case?
  6. Are the models in Table 11 all post-trained on the same split of simulation and real data?
  7. Based on Table 6, the majority of the pretraining data is from PnP. Why is that the case? Also, what is the difference between "General-object upright transfer" and the specific objects detailed below it?
  8. Is there a failure mode analysis for the different tasks?
  9. Why are the physical verification and rendering done by MuJoCo and Isaac Sim respectively? Why not combine the two and use either MuJoCo or Isaac Sim?
4 comments

Thank you for the detailed rebuttal addressing the weaknesses and questions I had previously asked. After reviewing your responses, I believe my W2, W4, Q1, Q2, Q3, Q4, Q6 and Q7 have been addressed (though for W4 specifically, I would advise you to remove that additional section from the revised manuscript and explore the other work as a separate project). I have additional comments and clarifying questions for the other points based on the new findings are as follows:

  1. With regard to your response on W1, it appears that the success rate on the additional simulated tasks is significantly lower than the reported results in Table 2 (in both the cases with and without pre-training). For instance, the success rate of 27.1% with both pre-training and post-training on Tabletop Waste Clearing in simulation is significantly lower than the results reported on the real robot without any pre-training in Table 2(a). Why is that the case? Also, the fact that the post-training is only studied on a single example task remains a significant limitation, especially given the seemingly high variance between the success rate of different tasks and its importance to the pipeline.
  2. I appreciate your including the new comparison with open-source models in addition to GPT-6 Astra. The examples included on the project website help clarify the role of the Agent-as-Guide and the inputs and outputs. The illustrative examples are clear, though I would like to see a raw example of the full prompt, input context and output configuration in a revised version of the manuscript for reproducibility later on (it is not needed for the rebuttal).
  3. For the response to W3 (as well as reviewer gygZ's comments), how were the five tasks selected from the 24 GR-1 Tabletop tasks?
  4. I understand the purpose of the experiment discussed in Q5 and your clarifications provide some insight as to how it was conducted. However, the experimental setup here is still somewhat unclear to me. I assume that for each setup, 100 samples were used based on a different split depending on the setup. Is the sampling here done once prior to the post-training for each setup? Are these samples fixed at the beginning of the post-training (e.g., for a 50-50 split, 50 samples are randomly selected from the simulation and 50 samples are selected from real-world demonstrations and then subsequently used for post-training)?
  5. For the failure analysis, could you shed some light on some examples of the different failure modes? I do not expect a full analysis of the different failure modes for the rebuttal, but some limited examples would help demonstrate where e.g. post-training via simulation and post-training via real-world demonstrations would differ or lead to some additional insight. This is essential especially given the importance of the synthetic data in the pipeline.
  6. I am not entirely convinced by your response to Q9. While I understand separating physics verification and data generation, I do not understand the philosophy behind using MuJoCo for one and Isaac Sim for another. Was there any practical benefit to doing so or was it just an engineering decision made by you?

Given the time constraints, please note that I do not expect additional experiments or results here for any questions. However, additional clarifications by you would be helpful in better understanding the contributions for a final rating.

Thank you for the responses to the follow-up questions and additional clarifications. The failure modes and examples provided are certainly helpful and the selection of the RoboCasa tasks makes sense. For 4, does this mean that the dataset available throughout the post-training when there is a mix of simulation and real-world trajectories is larger than the dataset available when only simulation or only real-world trajectories are used? If so, how can that not be seen as a dataset augmentation given that you have all the samples from both datasets available during post-training and the samples are drawn from each simulation?

Thank you for the fruitful discussions and detailed responses. All my questions have been answered and I do not have any additional comments as of right now. I have a more positive view of the paper overall based on the rebuttal. In the initial review, my rating was between a 1 and a 2 leaning slightly towards a 2. After this discussion, my current score is now between a 2 and a 3 leaning slightly towards a 2 (though that can change based on the other reviewers' comments).

The strengths of the rebuttal and the reason for the improvements are the additional results on open-source agents and established open-source benchmarks as requested by reviewer gygZ as well as the failure mode examples. The conceptual clarifications also help with better understanding the work and should be included in future revisions of the paper.

I still think the current state of the paper is borderline, especially because the primary weakness is a limited study of the effects of pre-training and post-training which is fundamental to the results being ready for publication. The issues with the lack of focus in the experiments (i.e., breadth instead of depth) persist and lead to open questions for the work's conclusion. These are concerns that reviewers 7AaZ and gygZ also expressed and I share those concerns. As such, I maintain my score for now given all the information available to me at the time, though I might revise my score to a 3 based on further discussion with other reviewers and the AC. Thank you again for the clarifying comments!

Thank you for the constructive discussion and for recognizing the additional results and clarifications. We acknowledge your remaining concerns and will further study the effects of pre-training and post-training across more settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.