acceptodds
Under review as a conference paper at ICLR 2027

Decoupled-WAM: Decoupling Imagination and Action for Efficient and Flexible World Action Models

19 views

Abstract

World Action Models (WAMs) transfer temporal and physical priors from video generative models to robot control. As future-video generation becomes less central at inference time, the representation transferred from the video model to the action policy becomes a key architectural question. Existing WAMs typically pass video-side attention states through layer-wise interfaces, binding the Action Expert to the internals of a particular video model. We introduce Decoupled-WAM, which instead places the video-action boundary at a shared dense planning feature. The Video Expert combines a design-time subset of intermediate layer outputs into a fixed-shape representation, while every Action Expert block constructs its own visual keys and values to read that representation through gated cross-attention. This interface supports Action Expert capacity variation, early termination after the deepest selected video tap, and replacement of the Video Expert while retaining a trained Action Expert. On LIBERO, ten-layer and one-layer Action Experts achieve 98.5% and 98.2% success, respectively; the one-layer model operates at 40.5 ms per action query and 24.7 queries/s. It further reaches 75.3% on LIBERO-Plus, while the default model obtains 91.9% on RoboTwin 2.0. With the one-layer Action Expert frozen, a five-layer student Video Expert retains 94.7% LIBERO success compared with 98.2% for its paired teacher. Retention is substantially weaker under distribution shifts, showing that interface regression does not preserve all aspects of the teacher’s closed-loop behavior. These results characterize the capabilities and current limits of the shared video-action interface.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZEuEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Decoupled-WAM, which replaces the layer-by-layer aligned MoT/KV-cache exchange between the Video and Action Expert in WAMs with a shared, fixed-shape dense planning feature. This idea supports varying action-side depth, early exit after the deepest tap, and replacing the Video Expert via interface distillation with the Action Expert frozen. Experiments show that the action side can be compressed aggressively at little accuracy cost, while Video Expert replacement through the interface preserves most in-distribution behavior (94.7%) but degrades sharply out of distribution (74.9% to 38.1% on LIBERO-Plus).

Critical strengths

S1. The motivation is good: current WAMs wire the Action Expert layer by layer into a specific video model, which makes changing the video side hard; asking for a shared feature interface instead is a useful starting point.

S2. A one-layer Action Expert still reaches 98.2% on LIBERO at 40.5 ms and 24.7 queries/s, which shows that the action side can be made much smaller without losing much accuracy.

Critical weaknesses

W1. The 5-layer student retains 94.7% on LIBERO but drops from about 74.9% to 38.1% on LIBERO-Plus. This shows that this method does not equal behavior preservation. The paper notes this, but the “practical route” phrasing in the conclusion still reads somewhat optimistic; more cautious wording is advisable.

W2. Several comparisons have very small gaps, e.g., 98.5% vs 98.2% and 91.9% vs 91.8%, but no random seeds, confidence intervals, or multi-run variance are reported.

W3. Table 4 packs two experiments into one structure with a duplicated row, which hurts readability.

Questions for authors

For the very small performance gaps (e.g., 98.5% vs. 98.2% and 91.9% vs. 91.8%), could the authors report results over multiple random seeds with mean±std?

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZEuE

Thank you for the careful review. Our video-replacement experiment shows that interface-only distillation retains 94.7% LIBERO success with the entire trained Action Expert frozen.

Student robustness (W1)

A five-layer Video Expert learns to match the teacher’s planning interface using no action loss or video-generation objective. It achieves 94.7% LIBERO success at 30.2 ms per query. This demonstrates that the shared interface offers a practical target for replacing the video provider while retaining the trained consumer.

Appendix D reports the paired results.

EvaluationPaired teacherS5 + frozen Action Expert
Standard LIBERO98.5%94.7%
Core-100074.9%38.1%

Core-1000 is a diagnostic LIBERO-Plus subset. Its larger decline shows that robustness is harder to retain through interface matching alone, as already acknowledged in the abstract and conclusion. The experiment establishes provider replacement on standard LIBERO and identifies robustness retention as the main challenge for extending this capability. We will make the subset label explicit in the main text.

Random seeds and small gaps (W2, Q1)

Appendix E explicitly reports the single-seed setting. Repeating full training runs exceeds our rebuttal compute and time budget. We treat the small success-rate gaps as point estimates. Our main conclusions draw on the measured efficiency gains and experiments that hold one module fixed while adapting the other.

Table 4 (W3)

We will separate action-depth variation from video replacement and label each reference run. Appendix D explains that the A1 depth result is 98.2%, while its paired distillation result is 98.5%, using the same checkpoint with different evaluation backends and rollout batches. We will use the paired teacher in the replacement panel and align the main-text references.

Reviewer_acTUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

World Action Models pair a video expert with an action expert layer by layer, so the depth of one fixes the depth of the other. The paper keeps joint training and replaces that pairing with one dense feature, merged from a few chosen video layers, that each action block reads through its own attention. Three capabilities follow. Action heads can be far shallower, though each depth is trained separately. A five-layer video model can replace the video expert with the action side frozen, reaching 94.7 in distribution. Video computation can stop after the last chosen layer. The method reaches 98.5 on LIBERO, 74.9 on LIBERO-Plus and 91.9 on RoboTwin 2.0. Its default configuration runs at 147.3 ms per action chunk, against 278.9 ms for Fast-WAM.

Critical strengths
  1. This work reduces the video–action boundary to one dense tensor, mixed from a few selected layers, which every action block reads through its own projections. The depth of the action head is then no longer tied to the depth of the video model. One, ten and thirty layer heads, each a separately trained pair, reach 98.2, 98.5 and 98.6.
  2. The work shows that the video expert is replaceable once the action expert has been trained, provided the substitute produces the P the consumer expects. A five-layer student trained only on the frozen consumer's LayerNormed P reaches 94.7, with no action loss and no layer correspondence to recover. A separately trained layer-wise KV student reaches 15.75, and the student itself falls to 38.1 under shift.
Critical weaknesses
  1. The paper shows no identifiable advantage on any of the three benchmarks. LIBERO is 98.5, level with LingBot-VA and below 99.0 for Faster-WAM. LIBERO-Plus is 74.9, below 79.0 for DoT-Faster and 82.3 for Faster-WAM. These gaps fall between 0.1 and 1.3 points, and no variance or seed count is reported anywhere, so the numbers support neither superiority over baselines nor superiority over the paper's own ablation variants. The only measurable gain is latency, and it is measured in simulation alone, with no validation in a physical control loop.
  2. None of the three capabilities is attributed to the interface itself. Depth comes from three separately trained pairs and replacement from distilling one student, so the claimed modularity is a set of retraining recipes rather than a property shown with the video features held fixed. The fixed-shape property the paper emphasises holds for any pooled feature, and Section 4.2 itself notes that it means depth decoupling rather than compression. Nothing analyses what P carries or why layers 10, 15 and 20 are the right taps, so the design offers no mechanism, and what separates it from DoT-Faster and Faster-WAM is a softmax mixture over a few layers.
Questions for authors

The contribution is placed on three capabilities, variable action depth, a replaceable video expert and early exit from video computation. Each is obtained by retraining, three pairs for depth, one distillation for replacement, a changed configuration for early exit. The results on three benchmarks are parity or below, with a 7.4 point deficit under shift. What is the claimed contribution, and which of the three capabilities improves any metric the paper reports.

1 comment
AuthorsOpenReview ↗

Response to Reviewer acTU

Thank you for asking us to connect the interface to concrete benefits. Our study asks how much computation an existing WAM needs, and how its experts can be changed while retaining useful control. These are policy-research objectives alongside improving maximum benchmark success.

What improves? (W1, Q1)

OperationMeasured benefitControl result
Action depth, A10 → A1147.3 → 40.5 ms/queryLIBERO 98.5% → 98.2%
Stop video execution at L20160.477 → 145.014 ms/queryIdentical action chunks on tested inputs
Remove unused video blocks5.601B → 3.964B loaded parametersBlocks after the last tap are removed
Replace video provider with S5S5 takes 30.225 ms/queryPaired LIBERO 98.5% → 94.7%

Appendix B.3 measures model computation on one H100, excluding VAE encoding and preprocessing. The early-stop comparison uses the same A10 checkpoint and inputs. Depth variants are separately trained. The additional RoboTwin results show a 0.44-point Combined drop from A10 to A1 under the same constrained recipe.

Appendices A.4 and E explicitly report one training seed and one evaluation seed. We treat small success-rate gaps as single-run observations. Section 6 states that control evaluation is limited to simulation.

The baseline discrepancy also needs clarification. Our DoT-Faster comparison uses the August v1 available before our September submission. Its October v2 reports the updated result. For Faster-WAM by Zhao et al., our 73.6% comes directly from Table 3 of its official arXiv paper. We could not trace the review’s 82.3% figure to that table.

Evidence for module reuse (W2)

The depth experiments establish that the interface supports different Action Expert sizes. The video-replacement experiment holds the complete trained Action Expert fixed. We have also completed A-F1, which trains a new one-layer Action Expert with the existing Video Expert and feature mixer frozen.

ExperimentHeld fixedLIBERO success
Video replacementTrained Action Expert94.7%
A-F1 adaptationExisting Video Expert and mixer98.45%

A-F1 uses the frozen Video Expert and mixer from A-U1’s final checkpoint. This provider originally served a one-layer Action Expert. The new consumer starts from the last block of the original pretrained 30-layer ActionDiT, with fresh visual-cross Q/O and proprioceptive encoder parameters. It receives no trained A-U1 consumer weights.

We train for 21,700 steps on 8 H100s, with global batch 128, bf16, and seed 42, costing approximately 135.4 H100 GPU-hours. We report the final checkpoint.

SuiteSuccessesSuccess (%)
Spatial495/50099.00
Object498/50099.60
Goal491/50098.20
Long485/50097.00
Overall1969/200098.45

This shows that a frozen provider can support a separately trained consumer. Together with frozen-Action video replacement, it provides reuse evidence on each side separately.

Interface analysis (W2)

Appendix C includes fixed-checkpoint weight interventions, layer deletion, gradients, and interaction analysis. On 120 selected rollouts, learned fusion scores 70.0%, equal fusion 65.83%, and one-hot inputs 0–7.5%. These interventions show that control depends on the fused input distribution.

The separate 11-layer analysis finds weak correlation between learned weights and deletion effects (0.262), with substantial interactions arising in visual cross-attention. These results examine how the interface is used. The exact taps remain an empirical choice.

Reviewer_QmEjOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Decoupled WAM, a method that fuses video-backbone intermediate representations efficiently through a learned interface to enable fast inference while steal reaching acceptable performance. This interface also opens up different design choices, like arbitrarily deep action experts, distillation of the video backbone to a smaller more efficient student and early termination of the video backbone layers.

Critical strengths

The main strengths of the paper are:

  1. While some notations are a bit confusing, the overall writing of the paper is clear and underlines the contributions of the work very clearly.
  2. RoPE misalignment is a main issue that your reference work FasterWAM (DoT-Faster), and the proposed method resolves this in a principled way.
  3. Early stopping is an interesting lever for efficiency specific to WAMs, and could probably be pushed much further, this work is a good initial test.
Critical weaknesses

The work in its current state has several critical weaknesses :

  1. The baseline choice seems arbitrary to serve the method, and is not a fair comparison. For example in the LIBERO-Plus table, the π0.5\pi_{0.5}, Qwen-Robomanip, ImageWAM and many other much better baselines are missing.

  2. Section 4.3 states that the method enables action-capacity variation, video replacement and early stopping. However it seems like the work you improve (Faster WAM, Ma et. al.), already provides the action capacity variation, and that the video replacement increases failure rate drastically (98.2% to 94.7% is almost a x3 degradation, collapse on robustness testing). The fact that two out of the three stated advantages of the method are either not novel or are underperforming is a major issue.

  3. You choose the three layer tap for your main model, but why and how you chose this subset isn't clear, especially given that the scores for three layer and all layer fusion are equivalent.

  4. Overall novelty is limited given DoT-Faster which the authors reference as prior work and is highly similar, and the paper doesn't seem to highlight a significant improvement over the cited method.

Questions for authors

Q1: What is the performance of V20A1 on Robotwin2.0?


Q2: Can you explain the choice of baselines for your tables? You seem to be quoting ImageWAM on LIBERO but not in the robustness study, which is important since the method reaches 82.2% SR on LIBERO-Plus with the same latency as DecoupledWAM. Same issue with DoT-Faster, which is quoted as 75% but seems to claim 78% in their paper, with latency close to your V20A1 model. Are those numbers taken for the respective papers, or reruns on your end?


Q3: How did you choose the three layer subset? 90.14% and 89.95% are completely equivalent results, and it poses the question of how you got to that? Is there a way to choose S that limits max S for early stopping without comprimising performance that gets highlighted by your method?


Q3.1: You use a fixed interface throughout the paper. Have you at any moment in your studies tried a dynamic PP? Based on flow matching time, current action expert layer or anything else?


Q4: While it is considered concurrent work given the date of publication and you will not be penalized by this, could you try to summarize the advantages of your method over FasterWAM (Ma et al, 2026) in its current version? Given that both methods are doing similar things and DoT-Faster gets better results with their 1 layer action expert model, it would be interesting to explain what you consider the stronger point of your method compared to the one you cite in your introduction.

6 comments
AuthorsOpenReview ↗

Response to Reviewer QmEj

Thank you for the detailed questions. Our goal is to simplify communication within an existing WAM so that its experts can be resized or replaced at lower computational cost. We evaluate control performance together with these architectural and efficiency benefits.

Novelty relative to DoT-Faster and Faster-WAM (W2, W4, Q4)

We acknowledge DoT-Faster’s shallow action modules and Faster-WAM’s efficient KV-based communication. Our method exports video-layer outputs as one shared feature PP. Each action block constructs its own K/V to read PP. This gives a smaller Video Expert a single matching target while retaining the trained Action Expert, and allows video execution to stop once PP is ready.

Our contribution is this interface design and the changes it supports. A simple weighted mixture already achieves competitive control performance. The experiments examine its flexibility, computational benefits, and current limitations.

Baseline selection and result sources (W1, Q2)

We select baselines for their relevance to our research question. FastWAM and efficient WAMs are the closest references for simplifying video–action communication. Broader policies provide context, with results that also reflect different representations, training data, and pretraining. We retain this comparison scope. Improvements in flexibility and efficiency provide value alongside improvements in maximum success rate.

Appendix B.3 separates published success rates from local timings of released checkpoints. ImageWAM’s 147.8 ms is close to our default model’s 147.3 ms. Our A1 model takes 40.5 ms.

The DoT-Faster discrepancy comes from the paper version. Our September submission compares with its August 3 arXiv v1, which reports 75.0% on LIBERO-Plus. The 78.3% result appears in v2, released on October 6, after our submission. We will make the cited version explicit. Version history

For Faster-WAM by Zhao et al., Table 3 of the official arXiv paper reports 73.6%, which is the value we cite. We could not trace the review’s 82.3% figure to that table.

Why choose L10/L15/L20? (W3, Q3)

We chose three taps to increase sparsity, taking the beginning, middle, and end of L10–L20. We did not search all layer combinations.

The following ablations share the same constrained training recipe. They use RoboTwin three-camera data at 384 resolution, non-idle filtering, and stride-5 rotating-hashed sampling. Training uses 8 H100s, global batch 192, 30,000 updates, and learning rate 10−410^{-4}. Different micro-batch and accumulation settings preserve the same global batch.

Video feature sourcesAction layersClean (%)Random (%)Combined (%)
L10/L15/L201090.9289.3690.14
L10–L201090.2088.5289.36
L1–L301089.9689.9489.95
L10/L15/L20/L301090.7689.8990.33
L10/L15/L20190.4788.9389.70

We agree that 90.14% and 89.95% are close. Three taps complete PP at L20, allowing the remaining video layers to be skipped. Adding L30 improves Random success to 89.89% and Combined success to 90.33%, but requires the full video depth. We retain three taps as a lower-compute choice.

These results belong to the ablation training scale. The main experiment uses dense stride-1 windows without non-idle filtering, global batch 1,024, and 29,355 updates. It has approximately 30.06 million sample presentations, compared with 5.76 million for the ablations. These counts include repeated presentations.

V20A1 on RoboTwin 2.0 (Q1)

V20A1 achieves 90.47% Clean, 88.93% Random, and 89.70% Combined under the constrained recipe above. Its Combined result is 0.44 percentage points below the corresponding A10 model.

Evaluation follows the standard protocol in Appendix B.2, with 50 tasks and 100 accepted episodes per task and condition. We have no full-recipe A1 RoboTwin result, and training that model exceeds our rebuttal budget.

Dynamic interfaces (Q3.1)

We have not tried dynamic interfaces. We kept the design simple because the fixed interface already met our control-performance target. Each observation produces a new PP, reused across action layers and flow steps. Each layer has its own reader. Appendix C.2 examines sensitivity through static changes to the fusion weights.

W2: You did not address the video replacement part of the weakness. Video Expert replacement is presented as one of the three capabilities enabled by the interface, but it substantially degrades performance: LIBERO drops from 98.5% (paired teacher) to 94.7%, and LIBERO-Plus collapses from 74.9% to 38.1%, for a 40.5 to 30.2 ms gain. Is this something we simply have to accept from the method?


Novelty relative to DoT-Faster and Faster-WAM (W4, Q4)

You are highlighting the changes that you made, which I understand, but my question was about improvements. It would be good if the authors could simply explain what their method brings in terms of actual improvements over previous works and baselines.

Currently, the model doesn't really outperform baselines: on LIBERO-Plus, it reaches the same score as DoT-Faster in its first version, and it is outperformed by ImageWAM on both LIBERO-Plus and RoboTwin (see next section).


Baseline selection (W1, Q2)

I understand the focus on efficient WAMs, but ImageWAM fits that scope: no embodied pretraining, and your own harness measures it at the same latency as your default model (147.8 vs 147.3 ms). It is included in Table 1 (LIBERO), where it is within noise of Decoupled-WAM, but omitted from Tables 2 and 3, where it outperforms it significantly (82.2% vs 74.9%; 93.38% vs 91.92%). For Q2: the 82.2% is ImageWAM's episode-weighted score computed from its Table 3 categories (83.1% as printed), the same aggregation as your Table 2.

Also, claims should be made against a defined class of comparable methods (e.g. WAMs without embodied pretraining), not "among methods in the table" (Sec. 5.2): the latter depends only on which baselines the authors chose to include, cannot be verified or challenged and gives an incomplete idea of the competitiveness, as the omission of ImageWAM clearly shows.


Latency measurements:

I have concerns about how comparable the latency numbers in Table 1 / App. Table 5 are, as other papers report quite different timings, often on weaker hardware:

  • Fast-WAM: 278.9 ms on your H100 (model compute only) vs 170.5 ms on a consumer grade GPU in DoT-Faster (App. E, Table 7, same boundary; 196.0 ms with text and VAE). That is 1.6× slower on hardware with 2–3× the memory bandwidth.
  • Faster-WAM (Zhao et al.): 263.4 ms on your H100 vs 252.95 ms on an NVIDIA L20 in the original paper, a much weaker GPU.
  • ImageWAM (Table 11): about the same speed as Fast-WAM on an A6000 (eager 198 vs 194 ms; compiled 69 vs 80 ms), but 0.53× Fast-WAM in your harness (147.8 vs 278.9 ms).

Could the authors please:

  1. State the execution mode for all methods (eager vs. compiled, precision, batch size). Compilation alone changes Fast-WAM latency by ~2.4× (ImageWAM, Table 11), more than most gaps in Table 1, and compiled timings are the relevant basis for deployment. Please also specify what is excluded for ImageWAM (FLUX.2 VAE, Qwen3 text encoder) and which checkpoint was timed.
  2. Time DoT-Faster in your harness (code public since 14 Oct), as it is the closest architectural baseline.
  3. Report the latency of the V20A10 checkpoint behind the 91.9% RoboTwin result. The only configuration competitive in latency with DoT-Faster (V20A1) has no full-recipe RoboTwin result.

Why choose L10/L15/L20? (W3, Q3)

At 10,000 combined episodes, the tap configurations span only ~1 pt (89.36–90.33), and your three-tap choice (90.14) is pretty much indistinguishable from single-layer cross-attention (89.38, p≈0.08), all-layer fusion (89.95) and the matched Fast-WAM baseline (89.87). Only the move from pooled AdaLN to dense cross-attention clearly exceeds noise, which suggests the fusion itself contributes little in-distribution. An ablation on LIBERO-Plus, where methods do separate, would be more informative.

AuthorsOpenReview ↗

Response to Reviewer 1 (1/2): Video replacement, novelty, and baselines

Thank you for the follow-up. We clarify the practical benefits of video replacement, report new same-hardware timings for DoT-Faster, and explain what the tap ablation establishes.

1. Video replacement lowers the memory barrier while retaining the trained Action Expert (W2).

Replacing the video provider while keeping the trained Action Expert frozen reduces the policy from 5.07B to 0.99B parameters and peak allocated inference memory from 9.57 to 1.97 GiB. Given an existing teacher, the distillation run peaked at 18.2 GiB allocated and 18.9 GiB reserved per GPU, within the memory capacity of a 24 GiB consumer GPU. These reductions lower the hardware barrier for follow-up research and make inference on memory-constrained devices more practical.

The performance cost is real. In the paired evaluation, success falls from 98.5% to 94.7% on standard LIBERO and from 74.9% to 38.1% on Core-1000, a diagnostic subset of LIBERO-Plus. We would retain the teacher where robustness to these shifts is important. The experiment establishes that a much smaller video provider can be trained through the shared interface without retraining the Action Expert. The observed robustness loss is a limitation of this student, and our results do not establish that it is inherent to the interface. Retaining robustness during replacement is an important direction for future work.

2. Compared with DoT-Faster v1, we obtain comparable success at lower measured inference cost (W4, Q4).

Across the three benchmarks in our submission, our reported success rates match or exceed those reported in the first version of DoT-Faster. The small differences support comparable performance. The concrete gain is that a shared feature built from a few video layers supports this level of performance while allowing us to adjust the Action Expert’s depth and replace the video provider separately.

On LIBERO-Plus, V20A1 scores 75.4%, compared with 75.0% reported by DoT-Faster v1. In our same-hardware eager benchmark, where both generate 32 actions per chunk, V20A1 takes 42.76 ms and uses 9.57 GiB of peak allocated memory, compared with 47.36 ms and 11.66 GiB for DoT-Faster. The frozen-consumer experiment demonstrates a separate benefit: we can train a smaller provider by matching the shared feature alone and reuse the trained Action Expert.

Table 1. LIBERO-Plus success rates (%). The DoT-Faster row is from the first arXiv version.

MethodBackgroundCameraLanguageLightingObjectRobotSensorAvg.
DoT-Faster (v1, reported)57.067.992.194.382.749.082.375.0
Ours (V20A1)56.755.093.694.481.871.774.375.4

3. ImageWAM is a relevant system baseline, and Fast-WAM is the reference for the interface change (W1, Q2).

We agree that ImageWAM is a relevant efficiency baseline and that its higher scores on LIBERO-Plus and RoboTwin belong in the discussion of overall competitiveness. However, our main question is what changes when we redesign the interface while keeping the Fast-WAM video pretraining source fixed. The absence of embodied pretraining alone does not control for that choice. ImageWAM uses a different pretrained image representation and shows that it can support strong manipulation performance. Its results inform the comparison between full systems. Comparisons within the Fast-WAM family address what the interface design enables. This architectural contribution can be evaluated independently of whether we exceed ImageWAM’s task success.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.