acceptodds
Under review as a conference paper at ICLR 2027

From World Models to World Action Models: Rethinking Next-State Prediction

Abstract

Predicting the next state is a core paradigm of World Models for modeling physical dynamics, emphasizing prediction fidelity. As World Models evolve into World-Action Models (WAMs), existing methods still fix the next state before training as RGB, a single latent feature, or a static combination of predefined targets, thereby constraining action learning to the inductive biases preserved by a particular representation. To address this limitation, we propose CF-WAM, a dynamic next-state prediction framework that samples visual, semantic, geometric, and interaction projections of the same future, standardizes them into a common video form, and supervises a unified WAM across these projections. The action-relevant constraints exposed by these projections accumulate across training steps, forcing WAM to capture the underlying state-transition structure that supports multiple projections of the same action-conditioned future. This dynamic mechanism also provides a natural cross-embodiment dynamics reference frame for Human and Robot learning. By jointly learning across different next-state parameterizations, heterogeneous Human and Robot experience can bypass appearance differences and directly contribute to shared state-transition learning, improving cross-embodiment generalization. Experiments show that CF-WAM improves both training efficiency and final control performance, while translating Human experience effectively into policy gains. CF-WAM achieves state-of-the-art performance on RoboCasa-GR1 with an average success rate of 82.50%, while reaching 82.65% on LIBERO-Plus and up to 84.00% in real-world evaluations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-Action Models, Dynamic Future-State Prediction, Egocentric Manipulation, Ego–Robot Joint Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EyHUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CF-WAM studies how the choice of future prediction target affects action learning. During training, it samples one of four targets: RGB, semantic features, depth, or interaction segmentation. These targets share a video representation and are specified through the language condition. The model also learns from human and robot demonstrations using a partly shared action space. Experiments on RoboCasa-GR1, LIBERO-Plus, and six real-world tasks show improvements over the reported baselines, with ablations examining the target representations and the use of human data.

Critical strengths
  1. The main idea is straightforward and useful. Sampling one future representation at a time gives the model access to several types of supervision without requiring it to predict all of them in every update. Table 2 provides a useful check: success rises from 76.25% with RGB alone to 79.00% as the other projections are added. The individual gains are modest, but the trend supports the design choice.

  2. The experimental results are encouraging across several settings. On RoboCasa-GR1, the full model reaches 82.50%, compared with 75.17% for WALA. I found Table 10 particularly helpful: keeping the Human-Robot action-space design fixed, replacing visual supervision with dynamic supervision raises success from 78.25% to 82.50%. This makes the benefit of the proposed training scheme easier to assess. The LIBERO-Plus and real-world results add support, and the 84.00% real-world average is achieved with a single projection, without oracle selection.

Critical weaknesses
  1. My main reservation concerns the explanation for the static baseline's poor performance. Appendix A.3.1 matches total training compute, which is a sensible comparison for practical use. However, the static model processes four future streams per update, so the number of updates it receives also matters. The 68.75% versus 79.00% result supports the efficiency of dynamic supervision, but it does not establish that attention competition is the main cause of the gap. I would be more convinced by a comparison of training curves against optimizer updates. The attention plots alone do not settle this point.

  2. I would also like to see a small check of what the frozen video VAE retains from the non-RGB targets. Depth maps and segmentation masks contain sharp boundaries that may not survive the shared encoding equally well. The downstream results suggest that these targets are useful, but they do not tell us which details the model can actually learn from them. A few before-and-after reconstructions, with a simple depth or boundary error measure, would help.

Questions for authors
  1. Under the matched compute budget in Appendix A.3.1, how many optimizer updates does each variant receive? Do the training curves suggest that Static Multi-State is still improving when training stops?

  2. How well does the frozen Wan2.2 VAE reconstruct the depth and interaction-segmentation targets? Could you show a few examples before and after encoding and decoding, particularly around object boundaries?

  3. Table 4 shows that the best inference projection varies across tasks. For a new task, which projection would you use by default, and what evidence would guide a different choice without relying on test outcomes?

3 comments
AuthorsOpenReview ↗

To Question 1 (Static Multi-State), Question 2 (VAE reconstruction) and Question 3 (Inference projection selection)

We sincerely thank you for the time and care you have devoted to reviewing our manuscript, and for your encouraging assessment of CF-WAM’s design and experimental results. We particularly appreciate your insightful perspective on both the methodological and practical implications of our approach. Your questions about training efficiency, target reconstruction, and inference-time projection selection have helped us strengthen our experimental analysis and explain our design choices more clearly. We are grateful for these thoughtful and constructive suggestions. We have carefully considered each point, added relevant experiments and clarifications, and addressed your questions point by point below.

To Question 1 (Static Multi-State):

We thank the reviewer for highlighting the importance of optimizer update counts when interpreting these results. Training efficiency is one advantage of dynamic supervision. Dynamic-State predicts one future projection per sample, whereas Static Multi-State processes all four within each update. Under the original matched-compute budget, Static Multi-State received approximately 26k optimizer updates, while all other variants received 100k. The training curves plotted against optimizer updates show that Static Multi-State was still improving at 26k updates. With approximately twice the original compute budget, its success rate reached 75.92%. When trained for the same 100k optimizer updates as Dynamic-State, its success rate further increased to 76.58%, remaining 2.42 percentage points below Dynamic-State’s 79.00%, with approximately four times the total training compute. Additional updates narrowed the gap observed under the original matched-compute budget. With optimizer update counts matched, Dynamic-State still achieved a higher success rate using approximately one-quarter of the training compute. Figure 7 shows more diffuse attention from the static variant’s action branch to video tokens, suggesting competition among future projection streams for the action branch’s attention. With the batch size and video specifications held constant, CF-WAM can accommodate additional supervision modalities with almost no increase in per-update training compute. Static Multi-State must process an additional video stream at every update for each new modality, making multimodal scaling more difficult under a fixed compute budget.

To Question 2 (VAE reconstruction):

We encode and decode depth maps and interaction segmentation masks using the frozen Wan2.2 VAE. The evaluation covers four subsets and 1,000 unique frames. We compare each reconstruction with its corresponding target before encoding and average the metrics across frames. Depth reconstruction achieves an MAE of 2.42 mm. For interaction segmentation, the foreground mIoU is 99.85%, and the Boundary F1 score is 99.996% with a 3-pixel tolerance, indicating high agreement between the reconstructed masks and the original targets in both foreground regions and boundaries. If the paper is accepted, we will include side-by-side comparisons of the original depth maps and interaction segmentation masks, their VAE reconstructions, and enlarged views around object boundaries in the appendix of the final version.

To Question 3 (Inference projection selection):

CF-WAM’s dynamic projection sampling is primarily used during training. For each training sample, one of the four future projections is sampled. At inference, the same checkpoint predicts only the selected projection.

For new real-world tasks, we use Interaction by default. It achieves the highest average success rate among the individual projections on both the six tasks in Table 4 and the five OOD settings in Table 11, at 84.00% and 79.20%, respectively. Based on the average results in Table 4, Interaction, Visual, and Geometric perform similarly, and any of them can serve as a fixed inference projection. Semantic has a slightly lower average success rate.

Beyond the default setting, task instructions and characteristics can provide heuristic guidance if a projection is to be selected for a particular task. Geometric may be considered for tasks that depend more heavily on 3D spatial relationships. Semantic may be considered for tasks emphasizing object categories or semantic distinctions. Visual may be considered when task progress is primarily assessed through appearance or deformation, such as towel folding.

In future work, we will further explore agent-driven adaptive projection selection, allowing an agent to directly adjust the language prompt based on the task instruction, current observation, and execution stage to select the future projection to predict.

Thank you for the detailed responses and the additional experiments, which largely address my concerns. In particular, on why Static Multi-State falls below Dynamic-State, the added comparison under matched updates shows that most of the compute-matched gap disappears. I think this means that training efficiency is not just one advantage of dynamic sampling over static multi-state supervision but the main one.

For the remaining 2.42 points, I agree with the response that competition among the future projection streams may play a role. However, the attention plots in Fig. 7 appear to use the compute-matched Static model, so its diffuse pattern may still reflect fewer updates. Section 4.3.3 could therefore present competition as a hypothesis instead of the cause, and report the comparison under matched updates as well.

The new results are consistent with my original assessment, so I remain positive about the paper and keep my score of 3.

We sincerely thank you for taking the time to review our responses and for your thoughtful suggestions. We appreciate your recognition of training efficiency as one of the advantages of dynamic sampling over static multi-state supervision. We will clarify this benefit in Section 4.3.3.

Please let us know if there are any specific questions or anything holding back raising the score. We would be happy to provide any additional information needed.

Reviewer_e985OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper attempts to imbibe physical priors necessary for robot manipulation into video backbones used for training world action models (WAMs). It achieves this by training the same video backbone that produces future RGB images in WAMs to also produce geometric, semantic and interaction projections of future in the form of depth, segmentation maps and contact representations respectively, by specifying the target via prompts. Results show that method can additionally benefit from human videos better to achieve state-of-the-results on Robocasa-GR1, improvements on OOD settings of Libero-Plus and real evaluation set needing specialized priors (eg. depth understanding, semantic understanding) for succeeding.

Critical strengths
  1. Simple and new WAM approach. The method introduces no architectural changes from the video backbone for handling different modalities, directly supervising it with additional modalities. Repurposing video backbone without architectural changes has been studied for handling multiple viewpoints before [1] but not for multiple modalities.
  2. Comprehensive sim and real evaluation suite demonstrates effectiveness of method. The real suite in particular includes tasks requiring geometric and semantic understanding for improved performance.
    • Benefits with human data. Paper shows the method can benefit better from human videos (pick-place subset of EgoDex) by learning structural, geometric and interaction priors from them.

[1] Video Generators are Robot Policies. Liang et al. arxiv 2025.

Critical weaknesses
  1. Concerns regarding evaluations and results. Paper doesn't compare to state-of-the-art VLA architectures (MolmoAct2 in Table 3) that incorporate depth supervision. The real evaluation (Table 4) does not include a complete methodthat supervises with all modalities at once. Without validating that this still works and beats prior works like other variants, the method needs additional determination of what modalities to train for each task. Further, it is not clear if the architectures in prior works (eg. CoRE-VLA) would do better when trained with the same modalities.
  2. Modeling choices not completely justified. The paper does not compare to prior strategies to incoporate multiple modalities, example using different specialized heads [1], controlnet guidance [2], interleaving different modalities in VAE inputs [3], or supplying the model with first frame of different modalities [1,2].

[1] TesserAct: Learning 4D Embodied World Models. Zhen et al. ICCV 2025.

[2] Cosmos Transfer 2.5: World Simulation with Video Foundation Models for Physical AI. NVIDIA. arxiv 2025.

[3] Spatially Aware World Action Model via Geometric Latent Diffusion. Gonzalez et al. arxiv 2026.

Questions for authors

See weaknesses for missing comparisons.

Missing analysis:

  1. The quality of generations across modalities is not studied. It would be interesting to study if the different modalities benefit each other (or benefit RGB prediction) and the prediction on OOD data improves with human data.
  2. Figure 1. tsne visualization: Can authors report quantitative measurements that show features are hard to classify across modalities in intermediate representations.

Other minor concerns:

  1. Can the authors measure and report statistical significance of the gains over ablations in simulation and real suites?
  2. L220: Why does the method input RGB in first frame before encoding the target? The frozen VAE used to encode was trained on video sequences that do not have substantial variation across frames (like in between first timestep RGB and second timestep depth which are completely different). The authors mention separately encoding first frame and replacing first latent anyways (in L792-3), so it is not clear why it is also necessary to input RGB for encoding target video latent.
  3. L201: "without RGB correspondence": Can the authors clarify this statement given that RGB prediction branch still exists.
  4. L037: Hafner et al. 2019a may not be the best citation for world action models that predict future states and actions at once [1].

[1] World Action Models: The Next Frontier in Embodied AI. Wang et al. 2026.

4 comments
AuthorsOpenReview ↗

To Weakness 1 and Minor Concern 4

We sincerely thank you for your careful and thoughtful review, and for the time and expertise you have devoted to assessing our work. We greatly appreciate your encouraging assessment and your recognition of CF-WAM’s simple design, comprehensive evaluation, and effective use of human demonstrations. Your detailed and constructive suggestions have helped us strengthen the experimental evidence and clarify the presentation of our method. We have carefully considered each point and addressed your comments and questions below.

To Weakness 1 (Table 4):

Thank you for pointing out that our description of the training and inference settings in Table 4 was insufficiently clear. Table 4 already evaluates the full CF-WAM model. The same model is trained using all four future projections as supervision, without requiring task-specific selection of training modalities or retraining. Visual, Semantic, Geometric, and Interaction in Table 4 denote four inference settings. All four rows evaluate the same checkpoint trained with all four projections. Each row’s label indicates only which future projection the model generates during evaluation.

We have also added a real-world evaluation of the Static Multi-State variant from Table 2, which simultaneously supervises all four future projections for each sample. Its average success rate is 41.00%. We will explicitly distinguish training supervision from inference settings in the caption of Table 4 and the main text of the final manuscript.

To Minor Concern 4 (L37):

Thank you for recommending this survey. We will cite Wang et al. (2026) in the introduction of the revised manuscript to more accurately describe WAMs as models that combine future-state prediction with action generation.

To Weakness 1 (MolmoAct2 baseline):

Thank you for suggesting the inclusion of a VLA baseline that uses depth supervision. We conducted an additional evaluation on LIBERO-Plus using the official MolmoAct2 checkpoint, which achieved an average success rate of 78.96%. CF-WAM achieves an average success rate of 82.65%, exceeding this result by 3.69 percentage points. We will include this result in Table 3 of the revised manuscript.

To Weakness 1 (Comparison with CoRE-VLA):

We compare the Vis.+Geo. variant in Table 2 with CoRE-VLA because both methods use RGB and depth as visual modalities during training. On RoboCasa-GR1, Vis.+Geo. achieves an average success rate of 77.50%, exceeding CoRE-VLA’s reported 56.50% by 21.00 percentage points. This shows that CF-WAM still outperforms CoRE-VLA on this benchmark even when the visual modalities used during training are restricted to RGB and depth.

AuthorsOpenReview ↗

To Weaknesses 1 and 2 (Alternative multimodal designs)

To Weaknesses 1 and 2 (Alternative multimodal designs):

In embodied AI, a key challenge in scaling training is how to use diverse supervision signals from different data sources. The Bitter Lesson highlights the long-term value of general methods that can continue to benefit from increasing computational resources. Following this perspective, we argue that for world-action models to continually benefit from larger datasets from multiple sources, the prediction tasks for different future-state representations need a unified formulation, allowing heterogeneous supervision to accumulate within a single model. CF-WAM unifies different modalities within a language-conditioned video prediction task, providing a path toward multimodal scaling for embodied models. RGB, semantic, depth, and interaction targets use a common video format and share the same video VAE and prediction head, with text specifying the prediction target. This supports learning state transitions shared across embodiments from unpaired human demonstrations and robot trajectories, while facilitating the incorporation of compatible supervision modalities into the existing architecture. Even as the number of supervision modalities increases, each training sample still requires predicting only one future video stream.

Unifying multimodal future supervision within a single video prediction task is one of CF-WAM’s core contributions. New modalities that can be converted into compatible video representations can be incorporated into training with almost no architectural changes. The alternative designs require additional model adaptations or guidance signals to support the same unified training. CoRE-VLA would require extensions to its original action prediction architecture to incorporate the same future supervision, making it relatively less extensible. Modality-specific heads require output modules for new targets, while latent-frame interleaving requires allocating sequence positions to new modalities. ControlNet and multimodal first-frame conditioning require additional control signals and observations, respectively. RGB and depth can be acquired directly using an RGB-D camera, but semantic features and interaction segmentation must be generated at deployment if used as first-frame conditions. CF-WAM constructs these representations offline as training targets and requires only the current RGB observation as visual input at deployment.

Table 2 evaluates the per-sample sampling design with the training data, four supervision targets, and total training compute held fixed. On RoboCasa-GR1, Dynamic-State, which samples one projection per training example, achieves 79.00%, whereas Static Multi-State, which predicts all four simultaneously, achieves 68.75%. Controlled comparisons with the other designs would require implementing each corresponding variant and retraining under comparable settings.

AuthorsOpenReview ↗

To Question 1 (Future prediction quality) and Question 2 (Figure 1)

To Question 1 (Future prediction quality):

We have added two offline evaluations to examine whether multimodal supervision improves future RGB prediction and whether incorporating human data improves OOD prediction. Each comparison uses the same trajectories, current observations, task instructions, and prediction horizon. We compare predictions with the corresponding future targets from the same trajectories, using PSNR for RGB, MAE over valid pixels for depth, foreground mIoU for interaction segmentation, and DINOv3-PCA representation MAE for semantics.

We conducted an additional evaluation on the PnPBottleToCabinetClose task in RoboCasa-GR1, comparing Dynamic-State with the corresponding single-modality models across four future prediction targets. All models are evaluated on the same 50 trajectories. In each comparison, both models receive identical current observations and task instructions, predict the same target over the same horizon, and are evaluated against the same reference targets.

Future prediction targetMetricSingle-modality trainingDynamic-State (four modalities)
RGBPSNR ↑ (dB)32.1532.45
DepthDepth MAE over valid pixels ↓ (mm)1.7441.559
Interaction segmentationForeground mIoU ↑0.7840.820
SemanticsDINOv3-PCA representation MAE ↓3.3853.205

Compared with single-modality training, Dynamic-State improves all four future prediction metrics. RGB PSNR increases from 32.15 dB to 32.45 dB, depth MAE decreases by 0.185 mm, interaction segmentation mIoU increases by 3.6 percentage points, and semantic representation MAE decreases by 0.180. This trend is consistent with the RoboCasa-GR1 task success rates reported in Table 2, where replacing Visual-only with Dynamic-State increases success from 76.25% to 79.00%. Together, these results support our design motivation that learning multiple future projections within a shared state space provides complementary supervision and helps the model learn state representations useful for action prediction.

Effect of human data on OOD prediction. We use the w/o Human and CF-WAM checkpoints from Table 11 to evaluate future RGB, depth, interaction segmentation, and semantic prediction on the same set of real-world OOD trajectories in the Human-like setting. For each modality, both models use identical current observations, task instructions, prediction horizons, and reference targets.

Future prediction targetMetricw/o HumanCF-WAM
RGBPSNR ↑ (dB)28.5031.47
DepthDepth MAE over valid pixels ↓ (mm)2.3361.812
Interaction segmentationForeground mIoU ↑0.7440.799
SemanticsDINOv3-PCA representation MAE ↓3.7623.304

With human data, CF-WAM improves all four future prediction metrics on the same real-world OOD trajectories. Compared with w/o Human, RGB PSNR increases by 2.97 dB, depth MAE over valid pixels decreases by 0.524 mm, foreground mIoU for interaction segmentation increases by 5.5 percentage points, and DINOv3-PCA semantic representation MAE decreases by 0.458. This trend is consistent with the closed-loop evaluation in Table 11. With Visual fixed as the inference projection, incorporating human data increases the average task success rate across real-world OOD settings from 45.60% to 75.60%. These results indicate that the OOD generalization benefits of human data are reflected in both multimodal future prediction and closed-loop task execution.

To Question 2 (Figure 1):

We quantify the degree of modality mixing in the intermediate representations using 15-nearest-neighbor purity. The purity at L10 is 0.265, close to the shuffled-label baseline of 0.250, and lower than the values at the input layer (1.000) and output layer (0.998).

AuthorsOpenReview ↗

To Minor Concern 1 (Statistical significance), Minor Concern 2 (L220, RGB first-frame conditioning) and Minor Concern 3 (L201)

To Minor Concern 1 (Statistical significance):

Across repeated evaluations, the maximum observed fluctuations in success rate were ±0.4 percentage points in simulation and ±6.0 percentage points in real-world experiments. Within these ranges, even if CF-WAM’s success rate shifts downward and that of the corresponding ablation variant shifts upward, Dynamic-State retains an advantage of at least 1.95 percentage points over Visual-only in simulation, while incorporating human data retains an advantage of at least 18.00 percentage points in real-world OOD evaluations. Therefore, across the evaluated simulation and real-world ablation comparisons, our method achieves performance gains that exceed the maximum observed fluctuations. These advantages persist even when both results simultaneously shift in the least favorable directions.

To Minor Concern 2 (L220, RGB first-frame conditioning):

During training, we require two types of VAE latents: the latent of the current RGB frame serves as the observation condition, while the latents of future target frames provide training supervision. At inference, the model generates future latents from noise, so the VAE only needs to encode the current RGB frame alone. The single-frame encoding described in lines 792–793 refers to the inference stage.

During training, we prepend the current RGB frame to the four future target frames and obtain both the conditioning latent and the future target latents in a single encoding pass. Wan2.2’s causal VAE ensures that the first latent depends only on the current RGB frame, while subsequent latents use temporal features cached from the first frame, so that both encoding and decoding of the future targets use the current RGB observation as context.

If the sequence were first encoded with a depth frame at the beginning and the first latent were then replaced with the RGB latent, the future latents would still have been encoded using temporal features from the initial depth frame. The decoder, however, would first process the substituted RGB latent and then use the resulting temporal feature cache to reconstruct subsequent depth frames. Encoding and decoding would therefore rely on different first-frame information, potentially increasing reconstruction error. In our pipeline, the RGB frame is already used as the first frame during encoding. After adding noise, we restore the clean RGB latent obtained from the same encoding pass to the first position, preserving the noise-free observation condition, and compute the loss only over future latents. We will explicitly distinguish the training and inference encoding procedures in the revised manuscript.

To Minor Concern 3 (L201):

Here, “without RGB- or trajectory-level correspondence” means that paired RGB observations or aligned trajectories are not required between human and robot data. The RGB prediction branch remains in the model. When an RGB target is sampled, the model predicts future RGB frames conditioned on the current RGB frame from the same demonstration. When predicting depth, semantic, or interaction targets, it likewise uses the current RGB frame from that demonstration as the condition. Thus, the required temporal correspondence is maintained within each demonstration, while human and robot demonstrations do not need to be paired. We will clarify in the revised manuscript that this statement refers to learning without paired human–robot RGB videos, rather than learning without RGB videos.

Reviewer_9bhZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents CF-WAM, a world action model that predicts multiple visual representations and annotations beyond RGB videos. For visual prediction, the paper introduces a dynamic training scheme that supports RGB, DINO feature, depth, and segmentation prediction. For action prediction, the paper introduces a masked loss scheme that excludes certain elements from the loss calculation when a specific embodiment lacks those elements. CF-WAM achieves state-of-the-art performance on RoboCasa-GR1, with an average success rate of 82.50%, while achieving 82.65% on LIBERO-Plus and up to 84.00% in real-world evaluations.

Critical strengths
  1. The method figure is clear and easy to follow, providing a good overview of the proposed approach. For visual prediction, the paper introduces a dynamic training scheme that supports RGB, DINO feature, depth, and segmentation prediction. For action prediction, it proposes a masked loss scheme that excludes missing action elements from the loss calculation, allowing the model to handle different robot embodiments.

  2. The paper demonstrates strong performance across multiple benchmarks. CF-WAM achieves state-of-the-art results on RoboCasa-GR1, with an average success rate of 82.50%. It also achieves 82.65% on LIBERO-Plus and up to 84.00% in real-world evaluations, demonstrating its effectiveness in both simulation and real-world settings.

Critical weaknesses
  1. Although predicting multiple future representations can improve policy performance, this idea is not entirely new. Flex-π[1] has already explored joint prediction of RGB, DINO features, and 3D pointmaps, achieving strong performance. The paper would benefit from a clearer discussion of this closely related work in the method section and a direct comparison with Flex-π as a baseline.

  2. In Flex-π, 3D pointmaps can be directly encoded using the Wan-VAE with high reconstruction accuracy and low MSE, without additional training. In contrast, CF-WAM converts continuous metric depth values into discrete RGB values. This quantization may introduce some loss of depth precision. The paper would benefit from an ablation study examining how this encoding choice affects depth reconstruction quality and downstream policy performance.

  3. Given the limited methodological novelty, the paper would benefit from more extensive evaluations, especially on challenging real-world manipulation tasks. For example, Flex-π demonstrates performance on a long-horizon self-repair task. Evaluating CF-WAM on similarly challenging tasks would provide stronger evidence of its robustness and generalization capabilities.

[1] Flex-π: A Multi-Stream World-Action Model with Compute Flexibility, arXiv:2608.10860

Questions for authors
  1. Could the authors clarify the main methodological differences between CF-WAM and Flex-π, particularly regarding multi-representation future prediction? Would it be possible to include Flex-π as a baseline for a more direct comparison?

  2. Does converting continuous metric depth values into discrete RGB values introduce quantization errors that affect geometric accuracy? Could the authors provide an ablation comparing the proposed depth encoding with directly encoding 3D pointmaps using Wan-VAE, as done in Flex-π?

  3. Can CF-WAM handle more challenging real-world tasks, such as the long-horizon self-repair task demonstrated in Flex-π? It would be interesting to see whether the proposed method remains effective on tasks requiring precise manipulation, long-horizon planning, and recovery from errors.

  4. I'm also curious about the training setup. The paper states that the model was trained on 4 nodes, each equipped with 8 NVIDIA B200 GPUs, using a global batch size of 32. Does this mean that each GPU processes one sample per iteration? Given that each B200 has 180 GB of memory, would allocating an entire GPU to a single sample be wasteful? Could you also report the GPU memory usage during training?

8 comments
AuthorsOpenReview ↗

To Question 1 (Comparison with Flex-π)

Thank you for your comments. Flex-π is concurrent work. We address your comments and questions below.

To Question 1 (Comparison with Flex-π)

To Question 1(a) (Methodological differences):

Thank you for recognizing CF-WAM’s performance across multiple benchmarks. First, the two works address different research questions. Flex-π focuses on enabling multi-stream world-action models to flexibly select input and output streams at deployment according to the available observations and compute budget. CF-WAM focuses on unifying the prediction task formulation across different future-state representations and treating the choice of future representation as a dynamic training variable. Within this unified framework, CF-WAM learns state changes shared across embodiments from unpaired human demonstrations and robot trajectories.

Second, the two methods use different prediction architectures and supervision schemes for multi-representation future prediction. CF-WAM unifies appearance, semantics, geometry, and interaction within a single language-conditioned video generation task. The model takes the current RGB observation as its visual condition, with language specifying the future projection to predict. During training, CF-WAM samples one of the four future projections for each example and computes the future-state prediction loss only for the selected projection. All four projections share the same video VAE and world prediction head. These representations correspond to different projections of the same underlying future-state manifold, and their complementary constraints accumulate within the same world model through parameter updates across training steps. At inference, the Wan video branch generates the specified future projection stream in the shared latent space, while the action branch simultaneously predicts an action sequence. Flex-π uses separate prediction heads for RGB, DINO features, and 3D pointmaps on a shared backbone. During training, it computes prediction losses for all three future representation streams and actions for each sample and combines these losses in a single parameter update. At inference, Full joint mode jointly generates actions and all three future representation streams. The model also supports generating only a subset of future streams. We will clarify this distinction in the method section of the revised manuscript.

Finally, compared with Flex-π’s design of organizing prediction streams by modality, CF-WAM’s unified prediction task supports joint learning from unpaired human and robot data and facilitates the expansion of supervision modalities and training data sources. First, CF-WAM replaces the staged transfer paradigm of pretraining on human demonstrations followed by post-training on robot trajectories with direct co-training at the state-transition level. Within the same training stage, we sample from unpaired human demonstrations and robot trajectories and dynamically select one future projection as the supervision target for each example, enabling a single world-action model to learn state changes shared across embodiments from both data sources. Second, CF-WAM can incorporate new modalities using the existing architecture. Flex-π predicts DINO features through a dedicated head, whereas CF-WAM converts DINOv3 features into three-channel semantic image sequences using PCA. These sequences and interaction segmentation videos share the same VAE and world prediction head. With the batch size and video specifications held constant, each sample still predicts only one stream when compatible new modalities are introduced, resulting in almost no increase in per-update training compute. Third, CF-WAM expands the data sources available for action learning. Human wrist motion and robot end-effector motion provide action supervision in a shared end-effector space, while robot-specific joint dimensions are supervised exclusively by robot trajectories. Together, these advantages provide a path toward exploring additional supervision modalities and scaling training with larger datasets from multiple sources within the same architecture.

AuthorsOpenReview ↗

To Question 1 (Comparison with Flex-π) and Question 2 (Depth encoding and geometric accuracy)

To Question 1(b) (Flex-π baseline):

Flex-π is pretrained on robot data from AGIBOT World and subsequently fine-tuned for each experimental setting. Without aligning the training data and evaluation protocols, directly comparing the two systems’ scores would reflect differences in both pretraining settings and model design, making it difficult to isolate the effect of the future supervision scheme itself.

To examine simultaneous supervision of multiple future streams, we implemented the four-stream Static Multi-State variant within the CF-WAM framework, keeping the backbone, training data, and four future targets unchanged. Under the same total training compute budget, Dynamic-State achieves a success rate of 79.00%, compared with 68.75% for the four-stream variant. When optimizer update counts are matched, the four-stream variant reaches 76.58% using approximately four times the training compute, remaining 2.42 percentage points below Dynamic-State. This controlled comparison shows that, within the same framework, per-sample projection sampling uses a fixed training budget more effectively.

To Question 2 (Depth encoding and geometric accuracy):

CF-WAM focuses on unifying the prediction task formulation across different future-state representations and exploring how to learn shared state changes from unpaired human demonstrations and robot trajectories. Depth encoding is an implementation component of this framework, rather than the focus of our study. It incorporates geometric information from different sources into unified future-state supervision.

1. Encoding design. CF-WAM’s depth encoding is designed to provide unified geometric supervision across simulation, real-world robot data, and human demonstrations, motivating an encoding scheme with broader applicability. Compared with Flex-π, once metric depth is available, we map it directly into three-channel videos, eliminating the need for XYZ back-projection based on camera intrinsics. Thus, human videos with metric depth but without camera intrinsics can also be incorporated into the same video training pipeline.

2. Encoding accuracy and depth coverage. We added a controlled depth encoding comparison using the same 40 simulated depth frames and 10 real-world depth frames to evaluate CF-WAM’s pseudocolor RGB encoding and Flex-π’s XYZ pointmap encoding. Each representation is inverted to recover metric depth, and MAE is computed against the original depth. Under the default configuration, Flex-π’s floating-point pointmaps yield MAEs of 16.55 mm and 637.05 mm, respectively. The corresponding MAEs for CF-WAM’s 8-bit encoding are 0.352 mm and 0.599 mm. Retaining floating-point color values reduces both MAEs to approximately 10−410^{-4} mm.

Floating-point representation avoids 8-bit quantization, but Flex-π’s default preprocessing clips depths in the range 1.5<z<2.01.5 < z < 2.0 m to 1.5 m and sets points with z≥2.0z \ge 2.0 m to zero. These originally valid pixels remain included in the error calculation, so the larger MAEs reflect the depth coverage limits of this configuration. CF-WAM uses a nonlinear mapping with 3 m as a scale parameter, preserving finer quantization resolution at close range while also encoding depths beyond 3 m. Across both evaluated sets of frames, CF-WAM’s 8-bit encoding achieves an MAE below 1 mm, providing broader depth coverage with submillimeter mean quantization error.

3. Downstream performance. In Table 2, adding this geometric supervision to RGB supervision increases the RoboCasa-GR1 success rate from 76.25% to 77.50%.

AuthorsOpenReview ↗

To Question 3 (Real-world evaluation) and Question 4 (Batch size and GPU memory usage)

To Question 3 (Real-world evaluation)

As shown in Table 4, we have conducted systematic evaluations on six real-world tasks, with 50 closed-loop trials per task, covering challenges such as precise end-effector alignment for test-tube insertion, deformable-object manipulation in towel folding, and sustained contact during wiping. We also evaluated CF-WAM under five real-world OOD settings (Table 11) to assess its generalization to scenarios outside the training distribution.

Flex-π’s gripper self-repair results were obtained by training on robot teleoperation demonstrations supplemented with task-specific DAgger correction trajectories. These trajectories were recorded when operators took over the robot following ManiFlow’s execution errors and corrected its actions. CF-WAM can likewise incorporate such data into training, but this use of additional data is independent of the future prediction method itself. We further explore a complementary approach by directly co-training with collected human manipulation demonstrations. We find that, with the robot training data held fixed, adding human manipulation demonstrations alone enables CF-WAM to complete multi-stage tasks involving precise manipulation more frequently and to recover autonomously from naturally occurring errors more often. The additional supervision comes directly from human manipulation, avoiding the need for additional DAgger-style robot trial-and-error and corrective data collection through teleoperation takeovers. To quantify these two benefits, we added a 10-step real-world chemistry experiment.

The robot sequentially pours the solution and vitamin C powder from test tubes into a beaker, stirs the mixture with a glass rod, and then returns the test tubes to their designated positions and organizes the equipment. We record naturally occurring errors during execution. An autonomous recovery is counted when the robot corrects an error and completes the corresponding step without human intervention. Both groups use the same robot training data, inference projection, and real-world evaluation conditions, with 50 rollouts per group. With robot demonstrations alone, the model completes all steps in 13 rollouts (26%) and autonomously recovers from 6 of 43 naturally occurring errors (13.95%). After adding independently collected human manipulation demonstrations, the model completes all steps in 27 rollouts (54%) and autonomously recovers from 28 of 51 naturally occurring errors (54.90%).

Our results show that direct co-training with demonstrations of humans performing the task can also improve multi-stage task completion and autonomous recovery from naturally occurring errors. These demonstrations can be collected independently of robot trial-and-error, providing a complementary path for expanding the training data sources available to robot policies. In future work, we will explore combining these two approaches.

To Question 4 (Batch size and GPU memory usage):

Thank you for pointing out the issue in our description of the batch size. The original manuscript incorrectly reported the per-GPU microbatch size of 32 as the global batch size. Training uses four nodes, each equipped with eight NVIDIA B200 GPUs, with gradient accumulation set to 1. Each GPU therefore processes 32 samples per iteration, and the actual global batch size is 4×8×32=10244 \times 8 \times 32 = 1024. During training, the average GPU memory usage is approximately 70 GB per GPU. We will consistently correct the batch size in Section 4.1, Appendix A.1.5, and Table 7, and include the memory usage statistics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.