acceptodds
Withdrawn from ICLR 2027

P-WAM: Perception-World-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World Action Models (WAMs) have recently shown strong potential for robot manipulation by jointly modeling future visual states and actions, thereby capturing environment dynamics and physical regularities for closed-loop control. However, existing WAMs typically rely on frozen reconstruction-oriented visual VAEs for initial environment perception, which may limit their ability to capture object-level semantics and fine-grained spatial information. In this work, we propose Perception-World-Action Models (P-WAM), a simple and general framework that enhances WAMs with pretrained visual perception encoders while preserving their world dynamics modeling. P-WAM introduces complementary semantic and spatial perceptual features from discriminative visual encoders into the video-action DiT through lightweight connector modules. We instantiate P-WAM with SigLIP and DINOv3 to provide language-aligned semantics and dense spatial representations. Through comprehensive representation probing, experiments on the challenging RoboCasa simulation benchmark, and diverse real-world robot evaluations, we show that P-WAM improves the object-level semantic and fine-grained spatial information encoded in video-action DiT representations and leads to stronger closed-loop policy performance. These results suggest that integrating visual perception with generative world-action modeling is a promising direction for improving WAMs.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Robot Learning, Visual Perception, Generative World Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_i3tUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the lack of object-level semantics and fine-grained spatial structure in VAEbased perception for World Action Models (WAMs), and proposes P-WAM, which preserves the original VAE-based generative video-action backbone and prediction objectives while adding a discriminative perception branch based on SigLIP and DINOv3 and alternately injecting fused perception tokens into later DiT cross-attention layers. Experiments show that P-WAM improves policy success rates in RoboCasa simulation and real-world robot tasks while maintaining comparable SSIM/FID for future-frame prediction. It demonstrates the complementary effectiveness of discriminative visual perception for generative world-action modeling.

Critical strengths
  1. Well-motivated and practical architectural integration. P-WAM adds pretrained discriminative perception to a shared video–action backbone while retaining the original VAE pathway, denoising targets, and prediction objectives. The reported overhead is modest—approximately 12% additional FLOPs and 4% higher inference latency— while future-frame prediction quality remains comparable on the reported SSIM and FID metrics. 【pp. 4–6, Table 1】
  2. Consistent gains over a strong baseline. P-WAM outperforms Cosmos Policy across all five reported RoboCasa evaluation seeds, raising average success from 65.7% to 67.7% (t=8.16, p<0.01). Real-world success improves from 64.0% to 73.7%, with gains on five of seven tasks. 【Table 5; pp. 7–8, Table 3】
  3. Useful controls addressing alternative explanations. Additional training of Cosmos Policy reaches only 65.6%, and random-initialized encoders reach 66.3%. These controls provide evidence that pretrained perception contributes to the gains beyond additional training steps or architectural capacity alone. 【p. 8】
  4. Complementary evaluation of representations and behavior. The paper combines frozen video-action DiT representation probing, simulation, real-world deployment, and architectural ablations. The probe shows improved foreground mIoU, supporting more decodable perceptual information in the shared representation. 【Fig. 3, Tables 2–4】
Critical weaknesses
  1. Limited conceptual novelty for ICLR. P-WAM primarily integrates established components—pretrained visual encoders, feature pooling and fusion, and cross-attention conditioning—into an existing world–action backbone. The paper acknowledges precedents for separate perception and generation interfaces and alternating condition injection. Although practically useful, the integration provides limited new modeling principles or mechanistic insight, leaving its conceptual contribution somewhat modest for ICLR. 【pp. 1, 3–5】
  2. Mechanistic validation and preservation of world modeling remain insufficient. Concurrent improvements in segmentation probing and policy success do not establish that enhanced semantic and spatial perception causes the policy gains. The pretraining control does not isolate this specific mechanism. Likewise, unchanged SSIM and slightly improved FID do not adequately establish preserved action-conditioned dynamics, contact behavior, or physical consistency. Since later backbone layers and output projections are updated, retaining the generative pathway does not guarantee preservation of its functionality. 【pp. 5, 7–8, Fig. 3, Table 1; p. 18, Table 9】
  3. Statistical robustness and ablation conclusions remain limited. The five reported random seeds demonstrate evaluation stability, but whether each seed involves independent retraining is not clearly documented. Most ablations lack multiple seeds or confidence intervals, making small differences—such as 0.6 percentage points over SigLIP alone and 1.0 point over concatenation—insufficient to establish robust designsuperiority. Although App. A provides per-task results, the main paper lacks separate ID/OOD breakdowns and variance estimates for the key ablations, limiting assessment of the distribution and reliability of the gains. 【p. 7, Table 4; p. 16, Tables 5–6; p. 21】
  4. Generality is not yet established beyond the evaluated setting. The framework is tested only on the Cosmos Policy backbone, while real-world evaluation covers one robot platform and seven tasks, with 25 trials per task. Benefits are uneven: simulated articulated-object tasks show a slight decrease, and real-world pouring and clothfolding tasks remain unchanged. These results support effectiveness in the tested settings, but broader benefits across WAM architectures and manipulation skills remain unproven. 【p. 7, Table 3; p. 16, Table 6; pp. 19–21】
Questions for authors

Q1: What is the key technical novelty beyond integrating existing components, particularly relative to Cosmos3 and prior perception–generation or hybrid VLA/WAM approaches? Q2: What evidence links improved semantic/spatial perception to policy gains? Could targeted interventions distinguish this mechanism from general pretraining benefits? Q3: How is preserved world-modeling capability established beyond SSIM/FID—for example, through action-conditioned prediction, contact behavior, physical consistency, or long-horizon consistency?

Reviewer_XPgEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes P-WAM, a perception-enhanced World Action Model that incorporates pretrained SigLIP and DINOv3 visual representations into Cosmos Policy through lightweight connectors and alternating cross-attention. The method aims to improve object-level semantics and spatial understanding while preserving the original world dynamics modeling capabilities. Experiments on RoboCasa and real-world manipulation tasks show improvements over Cosmos Policy, supported by representation probing and architectural ablations.

Critical strengths
  1. Meaningful motivation. The paper investigates whether reconstruction-oriented VAE representations provide sufficient perceptual information for WAMs. Exploring the complementary roles of discriminative perception and generative world modeling is a relevant research direction.

  2. Reasonably comprehensive experimental validation. The authors evaluate both simulation and real-world manipulation, including multiple random seeds and useful controls for additional training and pretrained visual encoders. The real-world results are encouraging, particularly for object-centric manipulation tasks.

Critical weaknesses
  1. The method mainly combines existing visual encoders with a WAM through standard connectors and cross-attention. The technical contribution appears relatively incremental, which similiar to Cambrian-1, Mini-Gemini.

  2. Insufficient comparisons. The paper lacks comparisons with several recent methods, including π0.5, StarVLA, VLAct, LingBot-VLA, LingBot-VA, Fast-WAM, and DreamZero.

  3. Limited benchmark coverage. The evaluation primarily focuses on RoboCasa and seven real-world tasks. More diverse benchmarks would help establish the generality of the approach.

  4. Unclear perception bottleneck. Cosmos Policy already learns visual representations through action supervision. The limited improvement in segmentation probing does not convincingly demonstrate that perception is a major bottleneck in existing WAMs.

  5. Unclear contribution of world modeling. It remains unclear whether the improvements arise from better visual perception alone or from its interaction with world dynamics modeling. An ablation without future-video prediction supervision would be informative.

Questions for authors
  1. Necessity of additional perception. Given that Cosmos Policy has already been trained with action supervision, what evidence demonstrates that its learned representations suffer from a significant perception bottleneck? Could the authors provide more targeted evaluations, such as object grounding, semantic discrimination, or spatial localization, rather than relying primarily on foreground segmentation?

  2. Interaction between perception and world modeling. How much does joint world-action modeling contribute to the improvements? For example, could the authors ablate the future visual prediction objective during P-WAM fine-tuning while retaining action supervision? This would help clarify whether continued world-modeling supervision contributes to the gains beyond adding stronger perceptual features.

  3. Generalization and robustness. Could the authors provide separate ID/OOD results, particularly for unseen object instances and visual appearances? Given the pretrained perception encoders, stronger generalization would be an important justification for the proposed approach.

Reviewer_b534OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Perception-World-Action Models (P-WAM) enhance World Action Models (WAMs) by integrating pretrained visual perception encoders to improve object-level semantics and fine-grained spatial information. Using lightweight connector modules, P-WAM combines semantic and spatial features from SigLIP and DINOv3 into video-action DiT representations, leading to better closed-loop policy performance in robot manipulation tasks.

Critical strengths
  • The main design point of this paper, Discriminative Perception Pathway, is well-motivated and indeed effective in experimental validation.
  • The ablation study is relatively thorough, the writing is smooth, and the real-world experiments are relatively sufficient.
Critical weaknesses
  • From a design perspective, the Discriminative Perception Pathway is simple yet effective. However, the core technological innovation points need to be clearly explained. Many works have added auxiliary Perception Pathways to vla or wam architectures, such as dream-vla, vla-jepa, and jepa-wam. They all enhance feature discrimination starting from image features, thereby improving policy success rates. From my understanding, the module proposed in this paper is not novel.

  • From an experimental perspective, first of all, the experiments on RoboCasa do not demonstrate the effectiveness of the method; many new works are not compared against it—for example pi0.5, fast-wam, gr00t-n1.7, being h0.7 etc. The study only includes experiments on RoboCasa without incorporating results from many mainstream experimental benchmarks such as libero, libero-plus, robotwin2.0 etc. In terms of ablation studies (Table 4), while key results are provided there is no horizontal comparison with other methods that introduce discriminative image features as mentioned in my first point above.

Questions for authors
  • There are relatively few methods for real-world experiment comparisons, and more experimental results need to be supplemented.
  • No real-world experiment videos.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.