- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the different representation needs of environment prediction and action generation in world-action models. It proposes to separate action-relevant factors from other scene information by restricting the information used for action generation. The authors establish theoretical results under several identifiability assumptions and implement this idea through a sparse routing module built on a pretrained world-action model. The experiments cover synthetic data, simulation, and real-world robots. The method is evaluated through latent factor recovery, closed-loop control, and visual context interventions. The paper reports improvements in control performance and attempts to explain these gains through routing visualizations and token ablations.
Critical strengths
-
The problem formulation is well motivated.
The paper allows world prediction to retain full scene information while limiting the information used for action generation. It also allows action-relevant and irrelevant factors to be statistically correlated. This provides a useful perspective for studying task-relevant representations. -
The closed-loop experiments support the practical effectiveness of the module.
The authors report improved robot performance under matched base models and training settings. The intervention comparing selected tokens with random tokens also provides some functional evidence that the routing mechanism selects useful information.
Critical weaknesses
- Focused attention is not sufficiently verified as the mechanism behind action-relevant learning and performance improvements.
The sparsity loss itself encourages top-k attention to concentrate, so focused attention maps cannot serve as independent evidence of the proposed mechanism. Attention sinks may assign high weights to tokens with little semantic importance. Background patch tokens may also act as registers that store global information or support internal computation, meaning that a token's spatial location does not necessarily reflect the source of its information. These phenomena have been documented in prior work, but the paper does not provide targeted analyses to rule them out. Therefore, visual overlap between attention hotspots and manipulated objects is insufficient to show that the model has identified factors that actually drive actions.
The authors provide a C-only intervention, but this analysis only covers the four tasks where the proposed method performs best. It is also unclear whether the upstream dense pathway has already provided full visual information to the action hidden states. In addition, the robot baseline removes both the router and sparse regularization, making it difficult to separate the contribution of the new architecture from the sparsity constraint. The current results support the effectiveness of the combined method, but do not sufficiently establish the connection between focused attention, action-relevant information selection, and control improvements.
- The synthetic experiments do not match the key theoretical conditions.
Proposition 1 requires the Jacobian row vectors for the same action dimension across different states to have sufficient linear independence. However, Appendix C uses a linear action mechanism, where hidden regimes only apply scalar scaling to the same matrix. As a result, the corresponding gradients remain collinear and cannot satisfy the stated span condition. In addition, the version without sparsity already recovers the relevant factors reasonably well, so the current results do not clearly demonstrate the unique contribution of sparse regularization to factor separation. The paper should provide validation that satisfies the theoretical conditions or narrow the conclusions supported by these experiments.
Questions for authors
Key Questions That May Affect My Evaluation
-
Have attention sink and register effects been examined?
Please clarify whether the backbone or router contains explicit register tokens or special tokens. Have the authors analyzed persistent attention hotspots, high-norm tokens, or background tokens that may aggregate global information? What evidence rules out these effects when interpreting the spatial attention maps? -
What evidence shows that focused attention improves actions rather than just changing the visualization?
Could the authors add a baseline that keeps the same router but removes sparse regularization? It would also be useful to examine action responses when attention concentration is kept similar but the locations receiving attention are changed. This could help separate the effect of attention concentration from actual information selection. -
Do the selected tokens actually reduce dependence on irrelevant information?
Please clarify whether C-only completely blocks the upstream dense information pathway and report the intervention results on the full task set. Could the authors also test paired observations where task-relevant states remain fixed but backgrounds change, together with comparisons where target poses change? These tests would help verify whether actions are stable under irrelevant changes and sensitive to relevant ones. -
How does the synthetic generator satisfy the theoretical conditions?
Please explain how a linear action mechanism with only scalar scaling can satisfy the Jacobian span condition. If it does not, could the authors provide a nonlinear synthetic experiment that satisfies the condition? Please also compare against the non-sparse version and the VAE with sparse regularization baseline mentioned in the appendix.
Separate Reporting Check: Not Used in My Current or Final Score, but Requires Clarification
Table 2 and Appendix F.1 refer to a 50k-step checkpoint, while Appendix H states that C-only and random-matched interventions use a fixed 60k policy. Please confirm whether all three conditions in Table 2 use exactly the same model. Also, please clarify whether this difference is a documentation error, a different way of counting training steps, or the use of different checkpoints. Providing the corresponding checkpoint identifiers and evaluation configurations would help resolve this issue.
At present, I can only confirm an inconsistency in the descriptions, not that different checkpoints were actually mixed in the experiments. I do not use this issue to lower my score.
8 comments
We are very grateful for your insightful and constructive feedback
Q2. Does sparsity improve control beyond adding an attention branch?
A2. We added a matched control retaining the same router architecture, learned gate, and 5× router learning-rate multiplier, while removing only the sparsity loss.
| Model | RoboCasa365 | Real robot |
|---|---|---|
| Cosmos 3 baseline | 24% | 23/50 (46%) |
| Router Without Sparsity | 30% | 26/50 (52%) |
| Full CWAM | 43% | 41/50 (82%) |
RoboCasa365 results are averaged across 17 tasks, with 100 rollouts per task. The Franka evaluation uses 50 trials per model. Adding the router without sparsity improves performance, and the sparsity loss provides a further improvement under the matched architecture and optimization settings. The additional attention branch and its learning-rate multiplier therefore explain only part of the observed gains.
Q3. Do the selected tokens reduce dependence on irrelevant information? Does C-only block the upstream dense pathway?
A3. C-only restricts which visual tokens are available at the evaluated action interface. Upstream visual information can still enter through the shared backbone or be carried by contextualized tokens. We therefore interpret C-only as an intervention on the visual context available at that interface.
We extended the evaluation to all 17 RoboCasa365 tasks using the same token budget for C-only and Random matched.
| Evaluation set | Full CWAM | C-only | Random matched |
|---|---|---|---|
| Original four tasks | 71.3% | 60.3% | 24.5% |
| Remaining 13 tasks | 34.2% | 26.0% | 13.0% |
| All 17 tasks | 42.9% | 34.1% | 15.7% |
C-only exceeds Random matched by 18.4 percentage points across all 17 tasks and by 13.0 points on the 13 tasks outside the original subset. The selected tokens therefore provide more useful conditioning than an equal number of random tokens under this intervention.
Paired visual perturbations. We evaluated three tasks covering object transport and alignment (CoffeeSetupMug), articulated-object manipulation (OpenDrawer), and local component manipulation (TurnOnSinkFaucet). Each checkpoint was evaluated without retraining for 100 rollouts per task per condition, totaling 2,100 rollouts per model. We used the same initial states and seeds across models and conditions. Task-relevant initial object configurations, physics, camera parameters, and success criteria were held fixed. We report the unweighted mean across the three tasks.
- Original scene. No modification.
- Checkerboard background. Replace wall and floor textures with a two-color checkerboard.
- Speckled background. Replace the same textures with a fixed random block-pattern texture.
- Reduced lighting. Scale ambient, diffuse, and specular lighting components by 0.7.
- Added bottle and box. Introduce rendering-only distractor geometries.
- Gaussian image noise. Add Gaussian noise with a standard deviation of .
- Gaussian image blur. Apply Gaussian blur with a 1-pixel radius to the 256 × 256 input images.
| Condition | Cosmos 3 | Router Without Sparsity | Full CWAM |
|---|---|---|---|
| Original scene | 20% | 34% | 48% |
| Checkerboard background | 7% | 12% | 30% |
| Speckled background | 11% | 19% | 36% |
| Reduced lighting | 9% | 8% | 24% |
| Added bottle and box | 2% | 6% | 22% |
| Gaussian image noise | 4% | 8% | 19% |
| Gaussian image blur | 2% | 1% | 12% |
Full CWAM achieved higher absolute success than both controls under every tested perturbation. The paired rollouts show that this advantage persists under the tested background and distractor changes.
We are very grateful for your insightful and constructive feedback
Q4. Does the synthetic generator satisfy the Jacobian span condition?
A4. Thank you for pointing this out. In the original generator, the regime-specific action matrices have the form , so the gradient for a fixed action dimension is . Changing the regime only rescales this gradient, yielding a span of dimension at most one. The original experiment therefore illustrates predictive block recovery on that generator.
Revised generator and span verification. We draw with four action-relevant and sixteen nuisance coordinates. We generate observations using the invertible nonlinear map , where and are orthogonal and acts elementwise. The eight action dimensions follow
\qquad B_{ij}\ne 0.$$ Holding the action noise fixed, we stack the gradients with respect to the four relevant coordinates at $s_{1:4}=e_1,\ldots,e_4$. For each action dimension, this gives $$G_i=(I_4+\mathbf{1}\mathbf{1}^{\top})\operatorname{diag}(B_{i1},\ldots,B_{i4}), \qquad \det G_i=5\prod_{j=1}^{4}B_{ij}\ne 0.$$ Thus, for every action dimension, the gradients span the four-dimensional action-relevant subspace, satisfying the Jacobian span condition. **Matched sparsity comparison.** We evaluate CWAM and VAE with and without action-head sparsity. Full CWAM and VAE + column sparsity use the same column-wise group-$\ell_2$ penalty and the same coefficient $\lambda$. For the action-head weight matrix $W\in\mathbb{R}^{d_a\times d_z}$, the penalty is $$\mathcal{L}_{\mathrm{sp}} = \frac{1}{d_z}\sum_{j=1}^{d_z}\lVert W_{:,j}\rVert_2.$$ Each column contains the weights connecting one latent coordinate to all action dimensions. Both variants without sparsity use $\lambda=0$. Within each seed, all four methods share the generator and data and use the same four-dimensional selection budget. | Method | Relevant $R^2$ ↑ | Nuisance sensitivity $D_N$ ↓ | Selected/full action MSE ratio (closer to 1) | |---|---:|---:|---:| | CWAM without sparsity | 0.9780 ± 0.0043 | 0.0620 ± 0.0183 | 2.044 ± 0.366 | | VAE without sparsity | 0.9490 ± 0.0135 | 0.1366 ± 0.0368 | 2.914 ± 0.576 | | VAE + column sparsity | 0.9820 ± 0.0041 | 0.0352 ± 0.0100 | 1.686 ± 0.191 | | **Full CWAM** | **0.9928 ± 0.0017** | **0.0092 ± 0.0031** | **1.154 ± 0.088** | Relevant $R^2$ measures how well a nonlinear probe recovers the true action-relevant coordinates from the selected block on held-out data. $D_N$ is the fraction of the standardized block’s squared Jacobian norm attributable to nuisance coordinates. The selected/full ratio divides the action MSE using the selected coordinates by the action MSE using the full representation. Adding column sparsity improves all three mean metrics within both model families. With the same penalty and coefficient, Full CWAM achieves higher relevant $R^2$, lower nuisance sensitivity, and a selected/full MSE ratio closer to one than VAE + column sparsity. These results support improved block recovery on a generator whose Jacobian span condition we verified analytically. --- **Reporting Clarification.** Did all three conditions in Table 2 use the same checkpoint, and why do the reported step counts differ? **Response.** Thank you for flagging this inconsistency. We confirm that all three conditions in Table 2 used the **same EMA weights** from the export package `CWAM-E35-S1-I50000-EMA-BF16-20260826`, saved at `iter_000050000` (global step 50,000). The associated continuation-training configuration was `action_policy_robocasa365_s1_cont5k_nano`. The reference to a “60k policy” in Appendix H is a documentation error. We will correct it to “50k checkpoint.” All three conditions used the same task split, episode IDs, initial states, and evaluation seeds. The intervention changed only the visual context available at the action interface. Full CWAM used the full visual context, C-only retained the router-selected tokens, and Random matched retained an equal number of randomly sampled tokens. All three conditions were evaluated without retraining.
We are very grateful for your insightful and constructive feedback
Thank you for your constructive feedback and for recognizing the value of our inductive bias, theory, and framework. In response, we added new experiments and expanded the discussion.Please find the detailed responses as follows:
Q1. Have attention sink and register effects been examined?
A1. We inspected the backbone-to-router pathway and the tokens eligible for router attention. We found no dedicated learned register bank in the inspected pathway. The two types of non-spatial boundary tokens in the router’s effective context are listed below.
EOS<|vision_start|>Prior work shows that boundary tokens can receive substantial attention while contributing little to the attention readout [1], and that tokens with limited semantic relevance can act as attention sinks [2]. To test the role of these tokens in CWAM, we ran two interventions on a fixed checkpoint. Both interventions are applied within the router.
We selected three RoboCasa365 tasks that cover distinct manipulation demands. CoffeeSetupMug requires transporting a mug and aligning it with a placement target. OpenDrawer requires locating the handle and moving an articulated object along its permitted direction. TurnOnSinkFaucet requires locating and actuating a small component on a fixed fixture. We evaluate the original router and both interventions with 100 rollouts per task per condition, for 900 rollouts in total.
Router Δ measures the relative change in router output, and Action Δ measures the normalized change in the predicted action chunk. Results are averaged across the three tasks. Zeroing the projected values of
EOSand<|vision_start|>while preserving their attention weights yielded a normalized action change of 0.022. Mean success changed from 48.0% to 47.7%. This suggests that these two boundary-token types make a limited direct value contribution to action prediction under the tested protocol.Visual-token norms and spatial attention. Even in the router call with the largest norm disparity, the maximum pre-LayerNorm visual-token norm was 1.454× the median. The mean Spearman correlation between projected key norms and attention weights across visual tokens, computed within each head and router call, was 0.045. Together, these measurements provide little evidence that unusually large token norms explain the routing patterns we observed. In the illustrated CoffeeSetupMug rollout, some heads shifted their attention with the mug as it crossed patch boundaries, showing that their hotspots tracked the task object across spatial locations.
As reported in A3, we also changed wall and floor textures while keeping the task-relevant initial state fixed. Full CWAM showed smaller success-rate drops than the matched router without sparsity under both texture changes, providing complementary behavioral evidence of reduced sensitivity to the tested background changes.
To test the effect on control, we permuted the assignment of attention weights across visual tokens while preserving the weight distribution, entropy, and top-k mass. On CoffeeSetupMug, using the same checkpoint and evaluation episodes, success fell from 37% to 14%. This 23-percentage-point drop shows that control performance depends on how the router assigns its learned attention weights, even at fixed attention concentration.
[1] Kobayashi et al. Attention is Not Only a Weight: Analyzing Transformers with Vector Norms. EMNLP, 2020.
[2] Xiao et al. Efficient Streaming Language Models with Attention Sinks. ICLR, 2024.