acceptodds
Under review as a conference paper at ICLR 2027

Separating Action from Prediction in World Action Models

Abstract

World action models (WAMs) learn latent variables from observations and use them both to predict how the scene evolves and to generate actions. Prediction has to account for the whole scene, whereas the actions depend on only part of it. However, WAMs fitted to demonstrations provide no guarantee of generating actions only from the latent variables on which the demonstrated actions depend, termed action-relevant latent variables, because a model may imitate equally well while entangling them with nuisance variables. We therefore separate what the actions may depend on from what prediction needs, and impose a sparsity constraint only on the estimated structure between latent states and actions. We prove that, under this constraint, the latent variables used to generate actions are an invertible function of the action-relevant ones alone, and the true structure is identified up to a permutation of the latent variables. These guarantees require neither sufficient temporal dependence nor assumptions on how unobserved regimes evolve, and hold even when the action-relevant latent variables are statistically dependent on the rest. We approximate the constraint in a pretrained WAM by adding a sparse routing path to its action head, while the world-modeling objective stays unchanged. Synthetic experiments support the theory, and on simulated and real robots, the routing path improves aggregate closed-loop success.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Robotics, World Action Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cSY9OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the different representation needs of environment prediction and action generation in world-action models. It proposes to separate action-relevant factors from other scene information by restricting the information used for action generation. The authors establish theoretical results under several identifiability assumptions and implement this idea through a sparse routing module built on a pretrained world-action model. The experiments cover synthetic data, simulation, and real-world robots. The method is evaluated through latent factor recovery, closed-loop control, and visual context interventions. The paper reports improvements in control performance and attempts to explain these gains through routing visualizations and token ablations.

Critical strengths
  1. The problem formulation is well motivated.
    The paper allows world prediction to retain full scene information while limiting the information used for action generation. It also allows action-relevant and irrelevant factors to be statistically correlated. This provides a useful perspective for studying task-relevant representations.

  2. The closed-loop experiments support the practical effectiveness of the module.
    The authors report improved robot performance under matched base models and training settings. The intervention comparing selected tokens with random tokens also provides some functional evidence that the routing mechanism selects useful information.

Critical weaknesses
  1. Focused attention is not sufficiently verified as the mechanism behind action-relevant learning and performance improvements.
    The sparsity loss itself encourages top-k attention to concentrate, so focused attention maps cannot serve as independent evidence of the proposed mechanism. Attention sinks may assign high weights to tokens with little semantic importance. Background patch tokens may also act as registers that store global information or support internal computation, meaning that a token's spatial location does not necessarily reflect the source of its information. These phenomena have been documented in prior work, but the paper does not provide targeted analyses to rule them out. Therefore, visual overlap between attention hotspots and manipulated objects is insufficient to show that the model has identified factors that actually drive actions.

The authors provide a C-only intervention, but this analysis only covers the four tasks where the proposed method performs best. It is also unclear whether the upstream dense pathway has already provided full visual information to the action hidden states. In addition, the robot baseline removes both the router and sparse regularization, making it difficult to separate the contribution of the new architecture from the sparsity constraint. The current results support the effectiveness of the combined method, but do not sufficiently establish the connection between focused attention, action-relevant information selection, and control improvements.

  1. The synthetic experiments do not match the key theoretical conditions.
    Proposition 1 requires the Jacobian row vectors for the same action dimension across different states to have sufficient linear independence. However, Appendix C uses a linear action mechanism, where hidden regimes only apply scalar scaling to the same matrix. As a result, the corresponding gradients remain collinear and cannot satisfy the stated span condition. In addition, the version without sparsity already recovers the relevant factors reasonably well, so the current results do not clearly demonstrate the unique contribution of sparse regularization to factor separation. The paper should provide validation that satisfies the theoretical conditions or narrow the conclusions supported by these experiments.
Questions for authors

Key Questions That May Affect My Evaluation

  1. Have attention sink and register effects been examined?
    Please clarify whether the backbone or router contains explicit register tokens or special tokens. Have the authors analyzed persistent attention hotspots, high-norm tokens, or background tokens that may aggregate global information? What evidence rules out these effects when interpreting the spatial attention maps?

  2. What evidence shows that focused attention improves actions rather than just changing the visualization?
    Could the authors add a baseline that keeps the same router but removes sparse regularization? It would also be useful to examine action responses when attention concentration is kept similar but the locations receiving attention are changed. This could help separate the effect of attention concentration from actual information selection.

  3. Do the selected tokens actually reduce dependence on irrelevant information?
    Please clarify whether C-only completely blocks the upstream dense information pathway and report the intervention results on the full task set. Could the authors also test paired observations where task-relevant states remain fixed but backgrounds change, together with comparisons where target poses change? These tests would help verify whether actions are stable under irrelevant changes and sensitive to relevant ones.

  4. How does the synthetic generator satisfy the theoretical conditions?
    Please explain how a linear action mechanism with only scalar scaling can satisfy the Jacobian span condition. If it does not, could the authors provide a nonlinear synthetic experiment that satisfies the condition? Please also compare against the non-sparse version and the VAE with sparse regularization baseline mentioned in the appendix.

Separate Reporting Check: Not Used in My Current or Final Score, but Requires Clarification

Table 2 and Appendix F.1 refer to a 50k-step checkpoint, while Appendix H states that C-only and random-matched interventions use a fixed 60k policy. Please confirm whether all three conditions in Table 2 use exactly the same model. Also, please clarify whether this difference is a documentation error, a different way of counting training steps, or the use of different checkpoints. Providing the corresponding checkpoint identifiers and evaluation configurations would help resolve this issue.

At present, I can only confirm an inconsistency in the descriptions, not that different checkpoints were actually mixed in the experiments. I do not use this issue to lower my score.

8 comments
AuthorsOpenReview ↗

We are very grateful for your insightful and constructive feedback

Thank you for your constructive feedback and for recognizing the value of our inductive bias, theory, and framework. In response, we added new experiments and expanded the discussion.Please find the detailed responses as follows:


Q1. Have attention sink and register effects been examined?

A1. We inspected the backbone-to-router pathway and the tokens eligible for router attention. We found no dedicated learned register bank in the inspected pathway. The two types of non-spatial boundary tokens in the router’s effective context are listed below.

TokenFunctionCorresponds to an image region
EOSMarks the end of a text sequenceNo
<|vision_start|>Marks the start of a visual segmentNo

Prior work shows that boundary tokens can receive substantial attention while contributing little to the attention readout [1], and that tokens with limited semantic relevance can act as attention sinks [2]. To test the role of these tokens in CWAM, we ran two interventions on a fixed checkpoint. Both interventions are applied within the router.

  • Special-V-zero. We set the boundary tokens’ projected value vectors to zero. For a fixed router input, their keys and attention weights stay the same, while their direct contribution to the router output is removed.
  • Special-masked. We mask the boundary tokens before softmax, which renormalizes the attention weights over the remaining eligible context tokens.

We selected three RoboCasa365 tasks that cover distinct manipulation demands. CoffeeSetupMug requires transporting a mug and aligning it with a placement target. OpenDrawer requires locating the handle and moving an articulated object along its permitted direction. TurnOnSinkFaucet requires locating and actuating a small component on a fixed fixture. We evaluate the original router and both interventions with 100 rollouts per task per condition, for 900 rollouts in total.

TaskConditionSpecial Attn.Router ΔAction ΔSuccess
CoffeeSetupMugOriginal6.37%0037%
Special-V-zero6.37%3.42%0.02336%
Special-masked0%8.76%0.04634%
OpenDrawerOriginal5.74%0070%
Special-V-zero5.74%2.61%0.01769%
Special-masked0%7.23%0.03868%
TurnOnSinkFaucetOriginal6.18%0037%
Special-V-zero6.18%3.17%0.02638%
Special-masked0%9.14%0.05135%
3-task meanOriginal6.10%0048.0%
Special-V-zero6.10%3.07%0.02247.7%
Special-masked0%8.38%0.04545.7%

Router Δ measures the relative change in router output, and Action Δ measures the normalized change in the predicted action chunk. Results are averaged across the three tasks. Zeroing the projected values of EOS and <|vision_start|> while preserving their attention weights yielded a normalized action change of 0.022. Mean success changed from 48.0% to 47.7%. This suggests that these two boundary-token types make a limited direct value contribution to action prediction under the tested protocol.

Visual-token norms and spatial attention. Even in the router call with the largest norm disparity, the maximum pre-LayerNorm visual-token norm was 1.454× the median. The mean Spearman correlation between projected key norms and attention weights across visual tokens, computed within each head and router call, was 0.045. Together, these measurements provide little evidence that unusually large token norms explain the routing patterns we observed. In the illustrated CoffeeSetupMug rollout, some heads shifted their attention with the mug as it crossed patch boundaries, showing that their hotspots tracked the task object across spatial locations.

As reported in A3, we also changed wall and floor textures while keeping the task-relevant initial state fixed. Full CWAM showed smaller success-rate drops than the matched router without sparsity under both texture changes, providing complementary behavioral evidence of reduced sensitivity to the tested background changes.

To test the effect on control, we permuted the assignment of attention weights across visual tokens while preserving the weight distribution, entropy, and top-kk mass. On CoffeeSetupMug, using the same checkpoint and evaluation episodes, success fell from 37% to 14%. This 23-percentage-point drop shows that control performance depends on how the router assigns its learned attention weights, even at fixed attention concentration.

[1] Kobayashi et al. Attention is Not Only a Weight: Analyzing Transformers with Vector Norms. EMNLP, 2020.

[2] Xiao et al. Efficient Streaming Language Models with Attention Sinks. ICLR, 2024.

AuthorsOpenReview ↗

We are very grateful for your insightful and constructive feedback

Q2. Does sparsity improve control beyond adding an attention branch?

A2. We added a matched control retaining the same router architecture, learned gate, and 5× router learning-rate multiplier, while removing only the sparsity loss.

ModelRoboCasa365Real robot
Cosmos 3 baseline24%23/50 (46%)
Router Without Sparsity30%26/50 (52%)
Full CWAM43%41/50 (82%)

RoboCasa365 results are averaged across 17 tasks, with 100 rollouts per task. The Franka evaluation uses 50 trials per model. Adding the router without sparsity improves performance, and the sparsity loss provides a further improvement under the matched architecture and optimization settings. The additional attention branch and its learning-rate multiplier therefore explain only part of the observed gains.


Q3. Do the selected tokens reduce dependence on irrelevant information? Does C-only block the upstream dense pathway?

A3. C-only restricts which visual tokens are available at the evaluated action interface. Upstream visual information can still enter through the shared backbone or be carried by contextualized tokens. We therefore interpret C-only as an intervention on the visual context available at that interface.

We extended the evaluation to all 17 RoboCasa365 tasks using the same token budget for C-only and Random matched.

Evaluation setFull CWAMC-onlyRandom matched
Original four tasks71.3%60.3%24.5%
Remaining 13 tasks34.2%26.0%13.0%
All 17 tasks42.9%34.1%15.7%

C-only exceeds Random matched by 18.4 percentage points across all 17 tasks and by 13.0 points on the 13 tasks outside the original subset. The selected tokens therefore provide more useful conditioning than an equal number of random tokens under this intervention.

Paired visual perturbations. We evaluated three tasks covering object transport and alignment (CoffeeSetupMug), articulated-object manipulation (OpenDrawer), and local component manipulation (TurnOnSinkFaucet). Each checkpoint was evaluated without retraining for 100 rollouts per task per condition, totaling 2,100 rollouts per model. We used the same initial states and seeds across models and conditions. Task-relevant initial object configurations, physics, camera parameters, and success criteria were held fixed. We report the unweighted mean across the three tasks.

  • Original scene. No modification.
  • Checkerboard background. Replace wall and floor textures with a two-color checkerboard.
  • Speckled background. Replace the same textures with a fixed random block-pattern texture.
  • Reduced lighting. Scale ambient, diffuse, and specular lighting components by 0.7.
  • Added bottle and box. Introduce rendering-only distractor geometries.
  • Gaussian image noise. Add Gaussian noise with a standard deviation of 5/2555/255.
  • Gaussian image blur. Apply Gaussian blur with a 1-pixel radius to the 256 × 256 input images.
ConditionCosmos 3Router Without SparsityFull CWAM
Original scene20%34%48%
Checkerboard background7%12%30%
Speckled background11%19%36%
Reduced lighting9%8%24%
Added bottle and box2%6%22%
Gaussian image noise4%8%19%
Gaussian image blur2%1%12%

Full CWAM achieved higher absolute success than both controls under every tested perturbation. The paired rollouts show that this advantage persists under the tested background and distractor changes.

AuthorsOpenReview ↗

We are very grateful for your insightful and constructive feedback

Q4. Does the synthetic generator satisfy the Jacobian span condition?

A4. Thank you for pointing this out. In the original generator, the regime-specific action matrices have the form Mc=αcMM_c=\alpha_c M, so the gradient for a fixed action dimension ii is ∇sai=αcMi,⋅\nabla_s a_i=\alpha_c M_{i,\cdot}. Changing the regime only rescales this gradient, yielding a span of dimension at most one. The original experiment therefore illustrates predictive block recovery on that generator.

Revised generator and span verification. We draw s∼N(0,I20)s\sim\mathcal{N}(0,I_{20}) with four action-relevant and sixteen nuisance coordinates. We generate observations using the invertible nonlinear map o=Q2g(Q1s)o=Q_2g(Q_1s), where Q1Q_1 and Q2Q_2 are orthogonal and g(u)=u+0.2tanh⁡(u)g(u)=u+0.2\tanh(u) acts elementwise. The eight action dimensions follow

\qquad B_{ij}\ne 0.$$ Holding the action noise fixed, we stack the gradients with respect to the four relevant coordinates at $s_{1:4}=e_1,\ldots,e_4$. For each action dimension, this gives $$G_i=(I_4+\mathbf{1}\mathbf{1}^{\top})\operatorname{diag}(B_{i1},\ldots,B_{i4}), \qquad \det G_i=5\prod_{j=1}^{4}B_{ij}\ne 0.$$ Thus, for every action dimension, the gradients span the four-dimensional action-relevant subspace, satisfying the Jacobian span condition. **Matched sparsity comparison.** We evaluate CWAM and VAE with and without action-head sparsity. Full CWAM and VAE + column sparsity use the same column-wise group-$\ell_2$ penalty and the same coefficient $\lambda$. For the action-head weight matrix $W\in\mathbb{R}^{d_a\times d_z}$, the penalty is $$\mathcal{L}_{\mathrm{sp}} = \frac{1}{d_z}\sum_{j=1}^{d_z}\lVert W_{:,j}\rVert_2.$$ Each column contains the weights connecting one latent coordinate to all action dimensions. Both variants without sparsity use $\lambda=0$. Within each seed, all four methods share the generator and data and use the same four-dimensional selection budget. | Method | Relevant $R^2$ ↑ | Nuisance sensitivity $D_N$ ↓ | Selected/full action MSE ratio (closer to 1) | |---|---:|---:|---:| | CWAM without sparsity | 0.9780 ± 0.0043 | 0.0620 ± 0.0183 | 2.044 ± 0.366 | | VAE without sparsity | 0.9490 ± 0.0135 | 0.1366 ± 0.0368 | 2.914 ± 0.576 | | VAE + column sparsity | 0.9820 ± 0.0041 | 0.0352 ± 0.0100 | 1.686 ± 0.191 | | **Full CWAM** | **0.9928 ± 0.0017** | **0.0092 ± 0.0031** | **1.154 ± 0.088** | Relevant $R^2$ measures how well a nonlinear probe recovers the true action-relevant coordinates from the selected block on held-out data. $D_N$ is the fraction of the standardized block’s squared Jacobian norm attributable to nuisance coordinates. The selected/full ratio divides the action MSE using the selected coordinates by the action MSE using the full representation. Adding column sparsity improves all three mean metrics within both model families. With the same penalty and coefficient, Full CWAM achieves higher relevant $R^2$, lower nuisance sensitivity, and a selected/full MSE ratio closer to one than VAE + column sparsity. These results support improved block recovery on a generator whose Jacobian span condition we verified analytically. --- **Reporting Clarification.** Did all three conditions in Table 2 use the same checkpoint, and why do the reported step counts differ? **Response.** Thank you for flagging this inconsistency. We confirm that all three conditions in Table 2 used the **same EMA weights** from the export package `CWAM-E35-S1-I50000-EMA-BF16-20260826`, saved at `iter_000050000` (global step 50,000). The associated continuation-training configuration was `action_policy_robocasa365_s1_cont5k_nano`. The reference to a “60k policy” in Appendix H is a documentation error. We will correct it to “50k checkpoint.” All three conditions used the same task split, episode IDs, initial states, and evaluation seeds. The intervention changed only the visual context available at the action interface. Full CWAM used the full visual context, C-only retained the router-selected tokens, and Random matched retained an equal number of randomly sampled tokens. All three conditions were evaluated without retraining.
Reviewer_MjW5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to separate action-relevant information from the representations used for video prediction in world action models. The authors derive identifiability results under sparsity assumptions and add a sparsity-regularized attention router to the action head of Cosmos 3. Synthetic experiments examine recovery of action-relevant variables, while simulated and real-robot experiments show improved closed-loop success.

Critical strengths

The distinction between information needed for scene prediction and action generation is meaningful. Restricting the action pathway while preserving the full representation for prediction offers a promising way to reduce reliance on irrelevant visual information. The closed-loop improvements across simulation and a real-robot task support its practical potential.

Critical weaknesses

The contribution of sparse routing is not isolated on the robot benchmarks. The main comparison adds both an attention router and a sparsity loss. Since the router implements cross-attention, the gains could come from the additional attention branch rather than sparse selection. The synthetic comparison and inference-only token interventions do not resolve this ambiguity, weakening the evidence for the proposed mechanism.

Questions for authors
  1. Can the authors compare the same attention router with and without the sparsity loss on the robot benchmarks?
  2. How does the token-level attention constraint enforce the dependency sparsity required by the theory when the dense visual-to-action pathway remains active? What changes when that pathway is disabled?
  3. Can the authors directly test robustness to nuisance changes, such as altered backgrounds or distractor objects with the task-relevant state fixed?
5 comments
AuthorsOpenReview ↗

We truly appreciate your thoughtful and constructive comments

Q1. Can the authors compare the same attention router with and without the sparsity loss on the robot benchmarks?

A1. We agree that the original main comparison did not isolate the contribution of sparsity from that of the additional attention branch. We therefore introduce a matched training ablation: Router without sparsity retains the same router architecture, parameters, learned gate, and 5× router learning-rate multiplier, but removes the sparsity loss during training.

For RoboCasa365, each of the 17 tasks is evaluated over 100 rollouts, totaling 1,700 rollouts per model. We report the arithmetic mean of the 17 per-task success rates. For Franka stacking, each model is evaluated over 50 real-robot trials.

ModelRoboCasa365: 17-task averageReal robot
Cosmos 3 baseline24%23/50 (46%)
Router without sparsity30%26/50 (52%)
Full CWAM43%41/50 (82%)

The improvement without sparsity is plausible because the router still performs action-conditioned cross-attention: queries derived from action hidden states adaptively aggregate clean visual and task context, and the action-training objective can favor useful information even without an explicit sparsity penalty. Sparsity adds an inductive bias toward concentrated routing beyond this predictive weighting.

Full CWAM further improves success over the same router without sparsity by 13 percentage points on RoboCasa365 and 30 percentage points on Franka stacking. Thus, the gains cannot be explained solely by the additional attention branch, parameters, or learning-rate multiplier. This training-time comparison directly addresses the contribution of sparsity on the robot benchmarks, independently of the synthetic experiments and inference-only token interventions. We will include these results in the revision.


Q2. How does the token-level constraint enforce theoretical dependency sparsity when the dense pathway remains active? What changes when that pathway is disabled?

A2. Our theory constrains the functional dependencies of the complete action mechanism on latent scene variables, with CtC_t representing the recovered action-relevant state block.

Directly enforcing this constraint in Cosmos 3 presents two practical challenges:

  1. Latent scene variables and their action dependencies are not explicitly observed in robot demonstrations, while contextualized tokens may mix multiple variables; sparse attention therefore does not directly imply sparse latent dependencies.

  2. Visual information can enter action hidden states through the shared backbone before routing occurs. Enforcing the theoretical constraint would require separating information needed for video prediction from that available to action generation throughout the architecture, rather than only at the final readout.

We therefore adopt a sparsity-regularized router as a practical, theory-guided approximation that encourages action-relevant conditioning while retaining the pretrained joint video–action architecture. The action hidden state hh carries the ongoing action-denoising computation, whereas the router output rr supplies action-conditioned contextual information. Gated residual fusion,

h′=h+gr,h' = h + gr,

integrates these complementary representations. Diffusion Policy similarly distinguishes noisy action tokens from observation conditioning [1], while Flamingo’s gated cross-attention [2] and DiT’s gated residual modulation [3] provide related architectural precedents.

These precedents motivate gated residual fusion as a practical, theory-guided approximation tailored to the joint video–action architecture. The design encourages selective use of action-relevant information while preserving the shared scene representations and action-denoising computation. Specifically, the sparsity loss regularizes the router branch, promoting concentrated contextual readout, while the backbone retains broader scene information for joint modeling. Extending this branch-level regularization to dependency sparsity across the complete action mechanism, while preserving the benefits of joint generative modeling, is an important and challenging direction for future research. We will clarify this architectural motivation and the connection between our theoretical principle and its practical realization in the revision.

[1] Chi et al. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. RSS, 2023.

[2] Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS, 2022.

[3] Peebles and Xie. Scalable Diffusion Models with Transformers. ICCV, 2023.

AuthorsOpenReview ↗

We truly appreciate your thoughtful and constructive comments

Q3. Can the authors directly test robustness to nuisance changes while fixing the task-relevant state?

A3. We additionally conduct a controlled robustness evaluation on three RoboCasa365 tasks: CoffeeSetupMug (object transport and alignment), OpenDrawer (articulated-object manipulation), and TurnOnSinkFaucet (local component manipulation). Each fixed checkpoint is evaluated without retraining for 100 rollouts per task under each of seven conditions, totaling 2,100 rollouts per model. We report the arithmetic mean of the three per-task success rates.

We reuse the same initial states and seeds across models and conditions, keeping task-relevant object configurations, physics, camera parameters, and success criteria unchanged. The conditions are:

  • Original scene: no modification.
  • Checkerboard background: replace textures used exclusively by walls and floors with a two-color checkerboard.
  • Speckled background: replace the same background textures with a fixed random block-pattern texture.
  • Reduced lighting: scale ambient, diffuse, and specular lighting components by 0.7.
  • Added bottle and box: introduce rendering-only distractor geometries.
  • Gaussian image noise: add Gaussian noise with a standard deviation of 5/2555/255.
  • Gaussian image blur: apply Gaussian blur with a 1-pixel radius to the 256×256256 \times 256 input images.
ConditionCosmos 3Router Without SparsityFull CWAM
Original scene20%34%48%
Checkerboard background7%12%30%
Speckled background11%19%36%
Reduced lighting9%8%24%
Added bottle and box2%6%22%
Gaussian image noise4%8%19%
Gaussian image blur2%1%12%

CWAM’s sparsity-regularized router encourages conditioning to concentrate on action-relevant information, rather than incidental visual cues. This design favors more stable conditioning when visual appearance changes but the underlying task remains unchanged. Consistent with this motivation, Full CWAM outperforms both controls across all tested perturbations, supporting the benefit of emphasizing action-relevant context for robustness to changes in backgrounds, lighting, distractors, and sensor quality. We will include these results and the evaluation protocol in the revision.

Thanks for your response! All my concerns have been solved!

Reviewer_Ba9QOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies world action models, which predict future video and also output robot actions. The authors argue that actions should depend only on the part of the scene that matters for the task, not on everything the model uses to predict video. They prove that adding a sparsity constraint on how actions depend on the latent variables makes those action-relevant variables identifiable. In practice, they add a small attention module (a “router”) to Cosmos 3 that is pushed to look at only a few image tokens. With this module, success rate improves significantly on RoboCasa365 and on a real cube-stacking task, with small gains on LIBERO.

Critical strengths
  • The method is supported by theory that highlights the importance of sparsity to for the action to be robust to noisy features.
  • The method is simple and plugs into an existing mode such as Cosmos.
  • The gains are significant on various tasks, and the real-robot result .
Critical weaknesses
  • The baseline has no router at all. CWAM adds a new attention layer, new parameters, and a 5× learning-rate multiplier on those parameters Those could explain the gain, without sparsity playing a role. It would be nice to ablate the effect of router without the sparsity loss.
  • While the router and sparsity allows to improve results, it unclear which token are dropped from the action head. It would be nice to illustrate what token are selected/discarded to validate the initial intuition that irrelevant scene part are detrimental.
  • The router’s output is added back to the action token through a learned gate (line 287), so the action head could in theory leverage all token beyond the selected one, so it's unclear how important is this selection.
Questions for authors
  • What are the results with the router but without the sparsity loss?
  • Why is it necessary to add back the router output to the action token instead of using the router output directly?
  • How does the method do when backgrounds, textures or distractors change? Could show qualitatively that the router allows to discard distractor from the scene?
4 comments
AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback

Dear Reviewer,

Thank you for raising these important questions about the contributions of sparsity and residual fusion, and the interpretation of token selection. We conducted two additional ablations and clarify below what our architecture and experiments establish.


Q1. Could the gains come from the additional attention layer, parameters, and 5× learning-rate multiplier, rather than sparsity? What happens without the sparsity loss?

A1. We introduce two additional ablations to examine the contributions of sparsity regularization and residual fusion:

  • Router without sparsity: retains the router architecture, learned gate, and 5× router learning-rate multiplier, but removes the sparsity loss.
  • Direct router readout without the skip connection: replaces hi′=hi+grih'_i = h_i + g r_i with hi′=grih'_i = g r_i, removing the direct contribution of the original action hidden state at this fusion point. We discuss the motivation for residual fusion and its relationship to our theoretical formulation in our response to Q3.

For RoboCasa365, each of the 17 tasks is evaluated over 100 rollouts, totaling 1,700 rollouts per model. We report the arithmetic mean of the 17 per-task success rates.

For Franka stacking, each model is evaluated over 50 real-robot trials. The results are shown below.

ModelRoboCasa365: 17-task averageReal robot
Cosmos 3 baseline24%23/50 (46%)
Router without sparsity30%26/50 (52%)
Direct router readout without the skip connection31%28/50 (56%)
Full CWAM43%41/50 (82%)

Full CWAM substantially outperforms the same router without sparsity regularization, supporting the contribution of sparsity regularization beyond the additional attention branch and its optimization settings.

The improvement without sparsity is expected: removing the sparsity loss does not remove the router's action-conditioned attention. Queries derived from action hidden states still adaptively aggregate clean visual and task context, and the action-training objective can favor useful information even without an explicit sparsity penalty. This can improve performance even without sparsity; the sparsity loss further regularizes how this additional context is selected.

Useful predictive weighting does not guarantee a minimal action-relevant dependency structure: a dense router may still rely on redundant or spuriously correlated context. Sparsity introduces an additional inductive bias toward concentrated routing. Our matched comparison therefore distinguishes the benefit of adding the router from the further improvement obtained by regularizing its routing distribution.


Q2. Which tokens are selected or discarded? Since the original pathway remains available, how important is the selection?

A2. We agree that “discarded” requires clarification. The sparsity loss concentrates routing mass on a small subset of clean context tokens.

The original Table 2 reported fixed-checkpoint context interventions on four tasks. To provide more comprehensive evidence, we extend these interventions to all 17 tasks in our RoboCasa365 evaluation suite. C-only restricts the action branch’s visual context to router-selected tokens, while Random Matched provides an equal number of randomly selected tokens. These interventions use the same trained checkpoint rather than separately trained models. We report the arithmetic mean of per-task success rates:

Task groupFull CWAMC-onlyRandom Matched
Original four tasks in Table 271.3%60.3%24.5%
Remaining 13 tasks34.2%26.0%13.0%
All 17 tasks42.9%34.1%15.7%

Across the complete 17-task evaluation, router-selected tokens outperform an equal number of randomly selected tokens. With the token budget matched, this provides behavioral evidence that the router preferentially selects action-relevant information, rather than merely reducing the number of context tokens. The remaining gap to Full CWAM suggests that complementary context remains useful.

AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback

Q3. Why is it necessary to add back the router output to the action token instead of using the router output directly?

A3. Thank you for this question. Our theory constrains which latent scene variables the action mechanism depends on, with CtC_t representing the recovered action-relevant state block. In Cosmos 3, we approximate this principle through a sparsity-regularized readout over clean context tokens, while retaining the joint video–action backbone and its rectified-flow objective.

The action hidden state hh incorporates the noisy action, denoising timestep, and multimodal interactions from the shared backbone. The router output rr, by contrast, aggregates contextual features conditioned on this action state. Residual fusion integrates selectively routed conditioning into the ongoing denoising computation, rather than replacing that computation with a context aggregate. With our linear action head,

v^a=Wd(h+gr)+bd=(Wdh+bd)+gWdr,\hat{v}_a = W_d(h + gr) + b_d = (W_dh + b_d) + gW_dr,

so the routed branch contributes directly to the predicted action velocity. Gated cross-attention in Flamingo [1], residual adapters [2], and the conditional branch in ControlNet [3] provide related architectural precedents, although they do not establish the necessity of residual fusion for CWAM.

Our ablation supports this choice: direct router readout without the action-state skip connection achieves 31% on RoboCasa365 and 56% on Franka stacking, compared with 43% and 82% for full CWAM. RoboCasa365 results average success rates over 17 tasks, each evaluated for 100 episodes; Franka results use 50 trials per variant. These results demonstrate a benefit from residual fusion in our setting, rather than its universal necessity.

The gated residual design is a practical architectural approximation of our theoretical principle for joint video–action training and deployment. While the theory characterizes an action mechanism that depends only on the action-relevant state block, our implementation encourages action-relevant information use through a sparsity-regularized context readout, integrated into the existing action-denoising representation. The dense backbone pathway remains available, making this a selective routing design rather than global hard token removal. This preserves the theoretical motivation while accommodating the shared generative architecture, without imposing the exact mechanism-level support constraint analyzed in the theory. We will clarify this connection between the theoretical formulation and its practical realization in the revision.

[1] Alayrac et al. Flamingo: a Visual Language Model for Few-Shot Learning. NeurIPS, 2022.

[2] Houlsby et al. Parameter-Efficient Transfer Learning for NLP. ICML, 2019.

[3] Zhang et al. Adding Conditional Control to Text-to-Image Diffusion Models. ICCV, 2023.

AuthorsOpenReview ↗

We sincerely appreciate your time and thoughtful feedback

Q4. Does the router discard distractors when backgrounds, textures, or distractors change?

A4. We additionally conduct a controlled robustness evaluation on three RoboCasa365 tasks: CoffeeSetupMug (object transport and alignment), OpenDrawer (articulated-object manipulation), and TurnOnSinkFaucet (local component manipulation). Each fixed checkpoint is evaluated without retraining for 100 rollouts per task under each of seven conditions, totaling 2,100 rollouts per model. We report the arithmetic mean of the three per-task success rates.

We reuse the same initial states and seeds across models and conditions, keeping task-relevant object configurations, physics, camera parameters, and success criteria unchanged. The conditions are:

  • Original scene: no modification.
  • Checkerboard background: replace textures used exclusively by walls and floors with a two-color checkerboard.
  • Speckled background: replace the same background textures with a fixed random block-pattern texture.
  • Reduced lighting: scale ambient, diffuse, and specular lighting components by 0.7.
  • Added bottle and box: introduce rendering-only distractor geometries.
  • Gaussian image noise: add Gaussian noise with a standard deviation of 5/2555/255.
  • Gaussian image blur: apply Gaussian blur with a 1-pixel radius to the 256×256256 \times 256 input images.
ConditionCosmos 3Router Without SparsityFull CWAM
Original scene20%34%48%
Checkerboard background7%12%30%
Speckled background11%19%36%
Reduced lighting9%8%24%
Added bottle and box2%6%22%
Gaussian image noise4%8%19%
Gaussian image blur2%1%12%

CWAM’s sparsity-regularized router encourages conditioning to concentrate on action-relevant information, rather than incidental visual cues. This design favors more stable conditioning when visual appearance changes but the underlying task remains unchanged. Consistent with this motivation, Full CWAM outperforms both controls across all tested perturbations, supporting the benefit of emphasizing action-relevant context for robustness to changes in backgrounds, lighting, distractors, and sensor quality. We will include these results and the evaluation protocol in the revision.

AuthorsOpenReview ↗

Further follow-up before the discussion concludes

Dear Reviewer,

Thank you for your time and constructive feedback. As the discussion period draws to a close, we would like to briefly summarize the additional evidence and clarifications provided in response to your concerns:

  • Contribution of sparsity beyond the added router. Our matched ablation retains the same router architecture, parameters, gate, and learning-rate multiplier while removing only the sparsity loss. Full CWAM achieves 43% vs. 30% on RoboCasa365 and 82% vs. 52% on real-robot stacking, supporting the additional contribution of sparsity regularization.

  • Behavioral relevance of the selected tokens. We extended the fixed-checkpoint token interventions to all 17 RoboCasa365 tasks. Router-selected context achieves 34.1% success, compared with 15.7% for an equal number of randomly selected tokens. We also clarified that selection operates within the router branch, while the original backbone pathway remains available.

  • Theoretical motivation and support for gated residual fusion. The gated residual design is a practical architectural approximation of our theoretical principle for joint video–action training and deployment, integrating selectively routed context into the ongoing action-denoising computation. Successful uses of related mechanisms in Flamingo, residual adapters, and ControlNet support this architectural choice, while our ablation results further demonstrate its effectiveness in CWAM.

  • Robustness to nuisance changes. We added controlled evaluations on three tasks under seven conditions, using matched initial states and seeds. Full CWAM outperforms both the baseline and the router without sparsity across all tested perturbations, including background changes, lighting changes, distractors, noise, and blur.

We hope these additions help address your questions.We would also be more than glad to provide any further clarification. If our responses resolve your concerns, we would also really grateful if you could reflect this in your final assessment.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.