acceptodds
Under review as a conference paper at ICLR 2027

STaC-WAM: Asymmetric World-Action Models via Spatio-Temporal Action-Conditioning

Abstract

World Action Models (WAMs) improve robot policies by coupling action gener- ation with predictive visual modeling. However, efficient inference remains chal- lenging. Implicit WAMs avoid test-time future-video generation by using a pre- dictive video backbone, but they still lack an explicit mechanism for transforming predictive visual representations into compact action-conditioning features. Video features encode high-dimensional visual dynamics, including object interactions, motion patterns, and local details, whereas action generation only requires a subset relevant for control. Without such an interface, the action expert must implicitly learn this transformation through extra capacity, making action generation com- putationally expensive. We introduce STaC-WAM, an asymmetric implicit WAM that couples a large predictive video backbone with a lightweight action expert through a Spatio-Temporal Action-Conditioning (STaC) bridge. Instead of di- rectly exposing the action expert to deep video representations, STaC compresses multi-layer video features into compact action-conditioning representations while preserving object-level structure and fine-grained visual details. Our experiments suggest a broader design principle for WAMs with predictive visual backbones: visual modeling and action generation need not scale symmetrically. With the STaC bridge, a 6-layer Action DiT outperforms the 30-layer Fast-WAM baseline on LIBERO, improving success rate from 97.6% to 98.85% while increasing infer- ence speed by 2.10x, with pronounced gains on LIBERO-Long. Evaluations on simulated and real-world tasks further demonstrate the robustness of this asym- metric design. Ablations show that Temporal Similarity Alignment provides the largest gain, while Object Assignment Alignment and the Object-Guided Dual- Path Decoder provide complementary benefits under aggressive compression.

Primary area:
applications to robotics, autonomy, planning
Keywords:
WAM, efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1FYQOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes STaC-WAM to address the inference overhead caused by depth coupling between the video backbone and action expert in implicit world-action models. Its core component is a spatio-temporal action-conditioning bridge that groups multi-layer video KV features, preserves structural information through object-slot compression, Object Assignment Alignment (OAA), and Temporal Similarity Alignment (TSA), and reconstructs action-conditioning features through object and detail pathways. This design reduces the action expert from 30 to 6 layers while keeping the video backbone unchanged. The paper reports an improvement in average LIBERO success rate from 97.60% to 98.85% and a reduction in end-to-end inference latency from 192.83 ms to 91.68 ms, with additional evaluations on RoboTwin, five real-robot tasks, and a subset of VLABench tasks.

Critical strengths
  1. STaC-WAM reduces inference latency from 192.83 ms to 91.68 ms while improving LIBERO success from 97.60% to 98.85%, demonstrating a promising performance–latency trade-off.
  2. Combining object-centric compression with a complementary detail pathway is a reasonable approach to preserving control-relevant information, with component ablations supporting their complementarity.
  3. Experiments cover LIBERO, the full RoboTwin benchmark, and five real-robot tasks, providing evidence across simulated and physical manipulation settings.
Critical weaknesses
  1. Table 3 compares compression strategies and alignment objectives, but lacks an MLP compression baseline trained with the same TSA supervision and does not report parameter counts or latency for each variant. It therefore remains unclear how much of STaC’s improvement comes from structured compression rather than auxiliary supervision or differences in model capacity.
  2. Faster-WAM already decouples video-backbone and action-head depths through cross-layer KV fusion, while Light-WAM also explores compact action interfaces built from multi-layer video representations. The paper should clarify the additional value of its object structure, temporal alignment, and dual-path reconstruction relative to these designs and provide corresponding comparisons.
  3. The manuscript does not sufficiently specify the TSA prediction head, how the spatial prior is incorporated into the detail decoder, or key training hyperparameters and real-robot evaluation protocols.
Questions for authors
  1. How does MLP compression with the same TSA supervision compare with full STaC in success rate, parameter count, and latency?
  2. What advantages does STaC provide over the interfaces in Faster-WAM and Light-WAM under comparable settings?
  3. Could you specify the TSA head, spatial-prior integration, training hyperparameters, and real-robot evaluation protocol, and provide the teacher checkpoint or pseudo-labels needed to reproduce OAA?
5 comments
AuthorsOpenReview ↗

Response to Weaknesses (1)

Thank you for taking the time to review our work so carefully. We greatly appreciate your thoughtful and constructive feedback, which helped us clarify what STaC contributes and strengthen the experimental evaluation.

W1. Improvement from structured compression

Table A. Interface and supervision ablations

IDInterfaceTSAOAALIBERO Avg. SR (%)RoboTwin Clean SR (%)RoboTwin Randomized SR (%)Bridge Params. (B)Total Params. (B)End-to-End Latency (ms)
A1Fast-WAM✗✗97.6091.8891.7806.02192.83
A2MLP✗✗97.6084.2085.260.115.3374.42
A3MLP(Large)✗✗97.6084.2485.301.386.59113.84
A4MLP✓✗97.7584.3285.400.115.3374.42
A5MLP(Large)✓✗97.8086.2887.441.386.59113.84
A6Simple Fusion✓✗95.5082.9483.4805.2172.30
A7Slot/STaC✗✗97.6584.7085.241.386.5991.68
A8Slot/STaC✓✗98.2088.6289.341.386.5991.68
A9Full STaC✓✓98.8592.2592.111.386.5991.68

Table A compares the original MLP + TSA (A4), the parameter-matched large MLP + TSA (A5), and Slot/STaC + TSA (A8). A5 and A8 use the same six-layer Action DiT and TSA objective, with OAA disabled in both. Each has 1.38B bridge parameters and 6.59B total inference parameters.

Under these matched conditions, A8 improves success over A5 by 0.40 percentage points on LIBERO and 2.34/1.90 points on RoboTwin Clean/Randomized, while reducing measured latency from 113.84 to 91.68 ms. Adding OAA gives the full STaC model (A9), which reaches 98.85% on LIBERO and 92.25%/92.11% on RoboTwin Clean/Randomized at 91.68 ms.

W2. Relation to Faster-WAM and Light-WAM

Faster-WAM reduces the cost of explicit future conditioning by restricting visual K/V reads to 8 of its 30 action layers. Each denoising step still runs all 30 layers. STaC-WAM compresses the K/V from the current observation into six groups, enabling a six-layer action expert. In our matched RTX 5090D benchmark, STaC-WAM is 2.07× faster, and the two papers report similar success rates on LIBERO and RoboTwin 2.0. Zhao et al.

Light-WAM reduces computation with a compact video backbone, learned-query pooling over three selected video layers, and single-pass action regression. Its training efficiency study reports 1.99B loaded parameters, 0.44B trainable parameters, and 4.25× the training throughput of Fast-WAM. STaC-WAM draws on all 30 video layers and combines object slots with a spatial detail pathway to condition a six-layer action expert. Across the two papers, STaC-WAM reports RoboTwin 2.0 success rates about 16 percentage points higher in both settings. The two systems differ in backbone, model size, and action decoder. Li et al., 2026

MethodLIBERO Avg. SR (%)RoboTwin 2.0 Clean SR (%)RoboTwin 2.0 Randomized SR (%)Inference latency (ms)Timing setup
Faster-WAM (Zhao et al.)99.092.7892.26189.63Our matched RTX 5090D benchmark
Light-WAM (Li et al.)97.276.476.367.57Our matched RTX 5090D benchmark
STaC-WAM98.8592.2592.1191.68Our matched RTX 5090D benchmark

On the RTX 5090D, STaC-WAM and Zhao et al.’s Faster-WAM were timed with five action-denoising steps, while Light-WAM used its native single-pass action head. Reported success rates follow each paper’s evaluation settings.

AuthorsOpenReview ↗

Response to Weaknesses (2)

W3. Implementation and evaluation details

TSA prediction head. The head takes the reconstructed KV features from the current latent frame, which are also supplied to the Action DiT. For the K branch in LIBERO, the input consists of 98 tokens, each represented by a 3,072-dimensional vector. The head applies LayerNorm followed by a linear projection from 3,072 to 98 dimensions to each token. Each output score corresponds to one position in the next latent frame. This yields a 98×9898\times98 matrix, where entry (i,j)(i,j) scores the predicted similarity between current position ii and future position jj.

For group gg and branch ξ∈K,V\xi\in{K,V}, the logits are Zg,tξ=Linearξ(LayerNormξ(F~g,tξ))Z_{g,t}^{\xi}=\mathrm{Linear}^{\xi}(\mathrm{LayerNorm}^{\xi}(\widetilde F_{g,t}^{\xi})). We apply softmax along the future-position axis so that each row is a distribution over positions in the next latent frame. During training, we use cross-entropy to match this prediction to a target distribution computed from the cross-frame cosine similarities of the normalized, group-averaged source features. The next latent frame is used only to construct the target; it is not an input to the prediction head. K and V have separate heads, and neither head is run at inference.

Spatial prior in the detail decoder. Row ii of the slot-mediated prior Πgξ\Pi_g^\xi indicates which source positions are favored when reconstructing output position ii. The detail decoder also computes content scores for these source positions. It first updates learned positional queries by cross-attending to the pooled source features Fˉgξ\bar F_g^\xi. It then projects the updated queries to Qgdet,ξQ_g^{\mathrm{det},\xi} and the pooled features to Kgdet,ξK_g^{\mathrm{det},\xi}, yielding content logits Δgξ=Qgdet,ξ(Kgdet,ξ)⊤/d\Delta_g^\xi=Q_g^{\mathrm{det},\xi}(K_g^{\mathrm{det},\xi})^\top/\sqrt d.

We smooth the spatial prior by mixing it with a uniform distribution, Π~gξ=(1−ρ)Πgξ+ρ/L\widetilde\Pi_g^\xi=(1-\rho)\Pi_g^\xi+\rho/L, and add it as a log bias before the final retrieval softmax:

Agdet,ξ=softmaxj(αξlog⁡Π~gξ+βξΔgξ)A_g^{\mathrm{det},\xi}=\mathrm{softmax}_j(\alpha^\xi\log\widetilde\Pi_g^\xi+\beta^\xi\Delta_g^\xi).

Here, αξ\alpha^\xi and βξ\beta^\xi are learned positive scalars, and jj indexes source positions in the current frame. Before normalization, the retrieval weight is proportional to (Π~gξ)αξexp⁡(βξΔgξ)(\widetilde\Pi_g^\xi)^{\alpha^\xi}\exp(\beta^\xi\Delta_g^\xi). This gives preference to source positions consistent with the slot assignments while allowing the content scores to adjust the retrieval. The decoder reads Agdet,ξFˉgξA_g^{\mathrm{det},\xi}\bar F_g^\xi, then applies LayerNorm and a linear projection before the token-wise router fuses the resulting detail features with the object-path features.

Training. We use eight H200 GPUs for each benchmark. LIBERO uses a per-GPU batch size of 16 for 10 epochs, and RoboTwin uses 24 for five epochs. Both use AdamW with β1=0.9\beta_1=0.9, β2=0.95\beta_2=0.95, weight decay 0.01, gradient accumulation of one, and an initial learning rate of 10−410^{-4} with cosine decay. The main STaC configuration uses three slots and one Slot Attention iteration. We set λTSA=0.5\lambda_{\mathrm{TSA}}=0.5, λOAA=0.1\lambda_{\mathrm{OAA}}=0.1, and the TSA temperature τ=0.07\tau=0.07.

Real-robot evaluation. We use a GALAXEA R1 Lite mobile manipulator with two 6-DoF arms and two grippers. The policy receives wrist and third-person camera observations at 224×224224\times224 resolution. We evaluate five tasks spanning basic manipulation and real-time decision-making. We collect 80 expert demonstrations per task and evaluate 50 rollouts per task, with small random perturbations to initial object positions and poses.

We will include these details in the revision.

AuthorsOpenReview ↗

Response to Questions

Thank you for your thoughtful questions and for engaging so closely with our work. We appreciate the opportunity to explain our design choices more clearly and share additional results.

Q1. Success rate, parameter count, and latency of MLP + TSA and full STaC

IDInterfaceLIBERO Avg. SR (%)RoboTwin Clean/Randomized SR (%)Bridge/Total Params. (B)End-to-End Latency (ms)
A4MLP + TSA97.7584.32 / 85.400.11 / 5.3374.42
A5Large MLP + TSA97.8086.28 / 87.441.38 / 6.59113.84
A9Full STaC (TSA + OAA)98.8592.25 / 92.111.38 / 6.5991.68

Compared with MLP + TSA (A4), full STaC improves success by 1.10 percentage points on LIBERO and 7.93/6.71 points on RoboTwin Clean/Randomized at an additional 17.26 ms. Compared with the parameter-matched Large MLP + TSA (A5), full STaC improves success by 1.05 points on LIBERO and 5.97/4.67 points on RoboTwin Clean/Randomized while reducing latency by 22.16 ms.

Q2. Advantages under comparable settings

Faster-WAM (Zhao et al.) fuses neighboring video layers by taking a weighted sum of K/V at each token position. STaC uses object slots to guide the retrieval of local details from the source K/V when reconstructing each output patch. This reconstruction runs once per observation and produces six K/V groups for a six-layer Action DiT. Each denoising step therefore executes six action blocks, compared with 30 in Faster-WAM, eight of which read video K/V.

Light-WAM pools features from three selected video layers into one vector shared by all action steps. STaC retains spatially indexed K/V, so different action queries can attend to different image regions as the action chunk is denoised.

Table A tests the bridge with the backbone, layer grouping, six-layer Action DiT, and TSA objective fixed, with OAA disabled. Replacing our equal-weight grouped K/V fusion control (A6) with the STaC bridge (A8) improves success by 2.70 percentage points on LIBERO and 5.68/5.86 points on RoboTwin Clean/Randomized. Against the parameter-matched Large MLP + TSA (A5), STaC improves success by 0.40 points on LIBERO and 2.34/1.90 points on RoboTwin.

In the matched RTX 5090D benchmark reported in our response to Weakness 2, STaC-WAM runs 2.07× faster than Faster-WAM (91.68 vs. 189.63 ms), with similar success rates reported in the respective papers. Light-WAM runs faster in the same timing benchmark (67.57 ms), while STaC-WAM’s reported RoboTwin success rates are 15.85/15.81 percentage points higher in the Clean/Randomized settings.

Q3. TSA head, spatial prior, training, real-robot protocol, and OAA resources

See our response to Weakness 3 for the implementation and evaluation details. The VideoSAUR teacher checkpoints used to generate the OAA masks for LIBERO, RoboTwin 2.0, and the real-robot experiments are all available at Hugging Face.

If the new results and clarifications address your main concerns, we would appreciate it if you could reconsider your overall assessment.

Reviewer_1Sq6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes STaC-WAM, which introduces a STaC bridge between a predictive video backbone and a shallower action expert. The bridge combines Slot Attention compression, object and temporal alignment objectives, and a dual-path decoder to transform deep world-model features into action-conditioning representations. The method retains a 30-layer world model while reducing the action expert to six layers. The authors evaluate the approach on LIBERO, RoboTwin, and real-robot tasks, reporting an increase in average LIBERO success rate from 97.60% to 98.85% and a reduction in inference latency from 192.83 ms to 91.68 ms.

Critical strengths
  1. Useful component analyses and substantial implementation details. The paper separately examines OAA, TSA, and the dual-path decoder. The combined alignment objectives and dual-path architecture achieve better results, providing some evidence of their complementary roles. The supplementary material includes the main model implementation, training objectives, training configurations, and LIBERO evaluation code, facilitating understanding and inspection of the method.
Critical weaknesses
  1. The methodological novelty appears incremental, and the added value of the structured bridge has not been sufficiently isolated. The paper largely follows Fast-WAM’s world-model and action-generation framework, with its main additions concentrated in the compression and reconstruction interface between the two experts, incorporating established components such as Slot Attention. Such a combination can constitute a contribution, but the current experiments do not sufficiently establish the advantage of the specific object-centric design over a simpler interface. With the same six-layer action expert, Table 3 reports 97.60% for MLP compression and 97.65% for slot compression without alignment losses; slot compression with TSA achieves 98.20%, increasing to 98.85% when OAA is added. However, the absence of a comparison between MLP+TSA and slot+TSA under matched supervision makes it difficult to separate the contributions of the architecture and the auxiliary training objectives. The existing joint-loss and dual-path ablations support the complementary roles of these components, but do not sufficiently isolate the additional value of the more complex structure over a simpler interface. This is the main reason for my reservations about the methodological contribution.

  2. The depth ablation does not disentangle the effects of conditioning compression and action-model capacity. In Section 4.4.1, the authors attribute the performance degradation at very small depths to the loss of predictive information caused by excessive compression. However, the experiments in Figure 5(b) simultaneously vary the number of action-conditioning groups and the depth of the action expert, coupling stronger conditioning compression with reduced action-model capacity. Consequently, these results demonstrate a performance–efficiency trade-off across depths, but do not establish whether the degradation primarily stems from information loss in the conditioning representations or insufficient action-model capacity. This limits the interpretation of the compression bottleneck and the rationale for choosing the action-expert depth.

Questions for authors
  1. Added value of the bridge architecture: With the same six-layer action expert, comparable training budgets, and the same TSA objective, how do MLP+TSA and slot+TSA compare in success rate, parameter count, and inference latency? Please clarify shared settings, such as the decoder configuration, and include the full model with OAA as a reference to distinguish the effects of architectural design from those of auxiliary supervision.

  2. Source of the gains from TSA: The existing ablations show that TSA improves task performance, but it remains unclear whether its benefits depend on correct temporal correspondence. Could the authors provide a control using same-frame similarity or shuffled temporal correspondences, while keeping other training settings unchanged, to examine whether the gains are specifically related to temporal structure rather than merely to additional auxiliary supervision?

  3. Interpretation of the depth ablation: As the action expert becomes shallower, how can the performance degradation be attributed to information loss from conditioning compression versus insufficient action-model capacity? Are there controls that hold action-expert capacity fixed, or other diagnostic evidence supporting the explanation in Section 4.4.1?

3 comments
AuthorsOpenReview ↗

Response to Weaknesses

Thank you for the insightful review. Your feedback helped us design more informative controls and refine our interpretation of the results.

W1. Added value of the structured interface

STaC uses object slots both to compress grouped video K/V and to guide patch-level reconstruction. For each output patch, the decoder uses the patch’s and source tokens’ assignments to the same slots to build a spatial prior, then combines the retrieved source details with features decoded from the slots.

Table A. Interface and supervision ablations

IDInterfaceTSAOAALIBERO Avg. SR (%)RoboTwin Clean SR (%)RoboTwin Randomized SR (%)Bridge Params. (B)Total Params. (B)End-to-End Latency (ms)
A1Fast-WAM✗✗97.6091.8891.7806.02192.83
A2MLP✗✗97.6084.2085.260.115.3374.42
A3MLP(Large)✗✗97.6084.2485.301.386.59113.84
A4MLP✓✗97.7584.3285.400.115.3374.42
A5MLP(Large)✓✗97.8086.2887.441.386.59113.84
A6Simple Fusion✓✗95.5082.9483.4805.2172.30
A7Slot/STaC✗✗97.6584.7085.241.386.5991.68
A8Slot/STaC✓✗98.2088.6289.341.386.5991.68
A9Full STaC✓✓98.8592.2592.111.386.5991.68

To directly test the added value of the structured bridge over a simpler interface, we compare the parameter-matched large MLP + TSA (A5) with Slot/STaC + TSA (A8) in Table A. Both variants use the same six-layer Action DiT and TSA objective, with OAA disabled in both. They also have the same inference parameter counts (1.38B for the bridge and 6.59B in total).

With the same TSA objective, matched inference parameter counts, and OAA disabled in both variants, replacing the large MLP (A5) with STaC’s slot compression and dual-path reconstruction (A8) improves success by 0.40 percentage points on LIBERO and 2.34/1.90 points on RoboTwin Clean/Randomized. Adding OAA to the same STaC bridge (A8 → A9) yields a further 0.65 points on LIBERO and 3.63/2.77 points on RoboTwin Clean/Randomized.

W2. Conditioning compression and action-expert capacity

We keep the video backbone at 30 layers and vary the number of conditioning groups and the action-expert depth independently. Each action layer receives one group's K/V. With fewer groups than action layers, group outputs are reused across consecutive layers. In C2, one group conditions all six layers. In C4, each of the three groups conditions two consecutive layers.

Table C. Conditioning groups and action-expert depth on LIBERO

IDConditioning GroupsAction Expert DepthLIBERO Avg. SR (%)
C11196.90
C21696.90
C33398.10
C43698.25
C56698.85

With one conditioning group, the one-layer and six-layer action experts both achieve 96.90% on LIBERO (C1 and C2). With three groups, increasing the action depth from three to six layers improves success by 0.15 percentage points (C3 to C4). At a fixed depth of six layers, increasing the number of conditioning groups from one to three to six raises success from 96.90% to 98.25% to 98.85% (C2, C4, and C5).

These comparisons indicate that conditioning granularity has a larger effect on success than action-expert depth in this sweep. We will revise Section 4.4.1 to reflect this result and present the loss of predictive information as an explanation for the effect of coarser conditioning.

AuthorsOpenReview ↗

Response to Questions

Thank you for your thoughtful questions, especially your suggestion to include same-frame and shuffled-frame controls.

Q1. MLP + TSA versus Slot/STaC + TSA under matched settings

All four variants use the same six-layer Action DiT, source K/V grouping, policy-training schedule, and TSA target construction. For A4 and A5, the TSA head reads the dense K/V produced directly by the MLP. A8 and A9 use the same Object-Guided Dual-Path Decoder architecture, and the TSA head reads its fused features. The table below reproduces the relevant results from Table A.

IDInterfaceOAALIBERO SR (%)RoboTwin Clean / Randomized SR (%)Bridge / Total Params. (B)End-to-End Latency (ms)
A4MLP + TSA✗97.7584.32 / 85.400.11 / 5.3374.42
A5Large MLP + TSA✗97.8086.28 / 87.441.38 / 6.59113.84
A8Slot/STaC + TSA✗98.2088.62 / 89.341.38 / 6.5991.68
A9Full STaC✓98.8592.25 / 92.111.38 / 6.5991.68

At the same inference parameter count, A8 reduces latency by 22.16 ms relative to A5. Relative to the smaller MLP (A4), A8 gains 0.45 points on LIBERO and 4.30/3.94 points on RoboTwin Clean/Randomized at an additional 17.26 ms.

Q2. Same-frame and shuffled-frame TSA targets

Across D1–D3, we fix the STaC architecture, OAA supervision, TSA head, loss weight, temperature, and training settings. We vary only the frame pairs used to construct the TSA targets. D1 pairs each latent frame with the next one, D2 pairs each frame with itself, and D3 keeps the source frames in order and randomly permutes the target frames within each clip before pairing them.

Table D. TSA target ablations on LIBERO

IDTSA TargetLIBERO Avg. SR (%)
D0No TSA (OAA only, submission Table 3(a))97.50
D1Adjacent-frame similarity (original)98.85
D2Same-frame similarity97.70
D3Shuffled-frame similarity95.45

Relative to OAA alone (97.50%), adjacent-frame TSA improves LIBERO success by 1.35 percentage points, and same-frame TSA improves it by 0.20 points. Shuffled-frame TSA reduces success by 2.05 points. These results support the role of correctly paired adjacent frames in TSA.

Q3. Interpretation of the depth ablation

Table C disentangles the two changes in the original depth ablation. With the Action DiT fixed at six layers, reducing the number of conditioning groups from six to one (C5 → C2) reproduces the full 1.95-point drop observed when both the group count and action depth are reduced (C5 → C1). With one conditioning group, increasing the action depth from one to six layers (C1 → C2) yields the same 96.90% success rate. These controls identify coarser conditioning as the main source of the observed performance drop, consistent with the loss of action-relevant predictive information under stronger compression.

If the new results and clarifications address your main concerns, we would appreciate it if you could reconsider your overall assessment.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 1Sq6,

We hope this message finds you well. As the discussion period draws to a close, we wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please do not hesitate to reach out. Your insights are invaluable to us, and were eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Best regards, Authors.

Reviewer_jU8nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes STaC-WAM, which inserts a feature transformation module, STaC, between the video model and the action network of Fast-WAM. This module aggregates every 5 layers of video features via Slot Attention, and then combines them with the original features to reconstruct the input required by the action network, reducing the action network from 30 layers to 6 layers. During training, it introduces object grouping supervision OAA (with pseudo-labels from VideoSAUR) and temporal feature supervision TSA. The paper reports that the average SR on LIBERO increases from 97.60% to 98.85%, and inference latency decreases from 192.83 ms to 91.68 ms, and it also evaluates manipulation performance through real-robot experiments.

Critical strengths
  1. The problem is practically meaningful. STaC-WAM optimizes the redundant computation in action denoising and reports about a 2.1× speedup while maintaining a high success rate.
  2. The experimental coverage is good. It includes multiple simulation benchmarks and real-robot tasks, and provides component ablations and visualization analysis of internal representations.
Critical weaknesses
  1. The approach is complex, and the source of the gains is unclear. In Table 3, MLP compression already reaches 97.60%, and Slot Attention without additional supervision reaches only 97.65%. The full STaC-WAM reaches 98.85%, but it adds auxiliary supervision and a complex decoding structure, which increases training (and data preparation) cost. Key comparisons such as "MLP + TSA" are missing, so it is hard to know whether the gains come from the structural design or from additional training.
  2. There is a lack of comparison with important recent work. Faster-WAM also improves efficiency by simplifying the action network and is a highly relevant comparison.
  3. Some details are unclear. Why 3 slots were chosen and how many iterations were used are neither explained nor ablated. For RoboTwin 2.0 task Place Burger Fries, why there is a 98.5% SR?
Questions for authors
  1. Can STaC be compared with simpler structures under the same auxiliary supervision? In particular, MLP + TSA and simple cross-layer feature fusion + TSA.
  2. Please compare STaC-WAM with Faster WAM.
  3. Is VideoSAUR used with off-the-shelf weights, or is it trained separately on this paper's data? What is the cost of generating pseudo label?
  4. Can you provide ablations on the number of slots and the number of slot attention iterations?
6 comments
AuthorsOpenReview ↗

Response to Weaknesses (1)

Thank you for your careful reading and constructive feedback. We particularly appreciate your suggestions for isolating the contribution of the STaC bridge and your close attention to the RoboTwin evaluation protocol. Both helped us make the experimental evidence clearer.

W1. Source of the gains from the structured bridge

To isolate the contribution of the structured bridge, we added three controls trained with TSA: a standard MLP, a large MLP with the same inference parameter count as STaC, and simple cross-layer K/V fusion. We also report these ablations on RoboTwin Clean and Randomized, where the differences between variants are more pronounced than on LIBERO, whose success rates are near saturation.

A2–A8 share the same 30-layer video backbone, grouping of source layers into six groups of five, and six-layer Action DiT, with matched training and evaluation settings. Within each group of five video layers, the MLP baselines (A2–A5) concatenate the features from the same spatial position across all five layers along the channel dimension, then project the concatenated features with an MLP. Simple Fusion (A6) uses an equal-weight average across the five layers at each spatial position, averaging keys and values separately. Both interfaces produce one dense K/V pair per group. OAA requires token-to-slot assignments, which only STaC produces in this comparison. We measure the effect of OAA by comparing A8 (TSA) with A9 (TSA + OAA). A1 uses the original Fast-WAM architecture with a 30-layer action expert. Parameter counts refer to the inference model, and end-to-end latency is measured under the same settings on an RTX 5090D at batch size one.

Table A. Interface and supervision ablations

IDInterfaceTSAOAALIBERO Avg. SR (%)RoboTwin Clean SR (%)RoboTwin Randomized SR (%)Bridge Params. (B)Total Params. (B)End-to-End Latency (ms)
A1Fast-WAM✗✗97.6091.8891.7806.02192.83
A2MLP✗✗97.6084.2085.260.115.3374.42
A3MLP(Large)✗✗97.6084.2485.301.386.59113.84
A4MLP✓✗97.7584.3285.400.115.3374.42
A5MLP(Large)✓✗97.8086.2887.441.386.59113.84
A6Simple Fusion✓✗95.5082.9483.4805.2172.30
A7Slot/STaC✗✗97.6584.7085.241.386.5991.68
A8Slot/STaC✓✗98.2088.6289.341.386.5991.68
A9Full STaC✓✓98.8592.2592.111.386.5991.68

The large MLP and STaC have the same inference parameter counts (1.38B for the bridge and 6.59B in total). On RoboTwin Clean and Randomized, adding TSA to the large MLP (A3 → A5) raises success rates from 84.24% and 85.30% to 86.28% and 87.44%, respectively. With the same TSA supervision and OAA disabled in both variants, replacing the large MLP with the STaC bridge (A5 → A8) improves success rates by 2.34 and 1.90 percentage points. Adding OAA (A8 → A9) yields further gains of 3.63 and 2.77 points, bringing the full model to 92.25% and 92.11%.

AuthorsOpenReview ↗

Response to Weaknesses (2)

W2. Comparison with Faster-WAM

Thank you for pointing out this closely related work. Of the two works named Faster-WAM, only Zhao et al.’s had a publicly available implementation at the time of benchmarking. We measured latency for their implementation and STaC-WAM on the same RTX 5090D GPU under matched inference settings. STaC-WAM achieves a 2.07× speedup, and the two papers report similar success rates on standard LIBERO and RoboTwin 2.0.

MethodMeasured latency (ms) ↓LIBERO SR (%) ↑RoboTwin 2.0 Clean SR (%) ↑RoboTwin 2.0 Randomized SR (%) ↑
Faster-WAM189.6399.092.7892.26
STaC-WAM91.6898.8592.2592.11

Success rates are from the respective papers. Zhao et al. report a latency of 252.95 ms with 10 denoising steps, and we measured 189.63 ms with five denoising steps using their public implementation on the same RTX 5090D used for STaC-WAM.

Faster-WAM makes explicit future conditioning cheaper by restricting visual K/V reads to 8 of its 30 action layers. Every denoising step still incurs the cost of running all 30 layers. However, STaC-WAM directly reduces this recurring cost by compressing K/V from the current observation into six conditioning groups, enabling a six-layer action expert.

W3. Slot configuration and the RoboTwin result

Our default configuration uses three slots and one Slot Attention iteration to keep the representation compact and inference latency low.

Table B. Slot Attention ablations on LIBERO

IDSlotsIterationsLIBERO Avg. SR (%)End-to-End Latency (ms)
B11196.2591.36
B22196.7591.94
B33198.8591.68
B44198.8592.80
B55198.8091.39
B63299.0598.49
B73399.15104.00

We evaluated STaC-WAM on all 50 RoboTwin tasks using two distinct evaluation seeds, with 100 rollouts per task and condition for each seed. Because we observed noticeable seed-dependent variability in RoboTwin evaluation, we use two independent evaluation seeds to reduce dependence on any single evaluation run. For each condition (clean and randomized), we report the mean success rate across the two runs. For Place Burger Fries, the reported 98.5% in each condition corresponds to 197 successes out of 200 rollouts. We will clarify this protocol in Section 4.2.2 and in the captions of Tables 2 and 4.

AuthorsOpenReview ↗

Response to Questions

Thank you for your thoughtful questions. We appreciate the opportunity to clarify our approach and share additional results.

Q1. MLP + TSA and simple cross-layer fusion + TSA

Table A reports MLP + TSA (A4), Large MLP + TSA (A5), and simple grouped K/V fusion + TSA (A6). A8 is the STaC + TSA reference. The TSA head reads the MLP output in A4 and A5, the fused K/V in A6, and the reconstructed K/V in A8. All four use the same TSA target construction, temperature, and loss weight.

Q2. Faster-WAM benchmark results and latency

We report our comparison with Zhao et al.’s Faster-WAM in the response to Weakness 2. We measured inference latency for both methods on the same RTX 5090D under matched settings. The success rates are reported from the respective papers.

Q3. VideoSAUR training and pseudo-label cost

We train a separate VideoSAUR teacher for LIBERO and RoboTwin 2.0 using each benchmark's main-camera training observations. We sample videos at 10 Hz and resize frames using bicubic interpolation while preserving aspect ratio. Each teacher uses three 128-dimensional slots and is trained for 100,000 steps on eight A800 GPUs with a batch size of 64 per GPU. Training takes approximately 10–12 hours per benchmark. Generating the slot masks on eight A800s takes approximately 30 minutes for LIBERO and 6–8 hours for RoboTwin. Mask generation is an offline preprocessing step for OAA.

We have also released the corresponding VideoSAUR teacher checkpoints for LIBERO, RoboTwin 2.0, and the real-robot experiments on Hugging Face. These checkpoints enable the regeneration of OAA pseudo-labels and facilitate the reproducibility of the overall STaC-WAM pipeline.

Q4. Ablation over slot count and iteration count

Table B. Slot Attention ablations on LIBERO

IDSlotsIterationsLIBERO Avg. SR (%)End-to-End Latency (ms)
B11196.2591.36
B22196.7591.94
B33198.8591.68
B44198.8592.80
B55198.8091.39
B63299.0598.49
B73399.15104.00

For each slot count, we use OAA pseudo-labels from a VideoSAUR teacher with the corresponding number of slots, trained with the same recipe and hyperparameters across slot counts. With one iteration, increasing the slot count from two to three raises LIBERO success from 96.75% to 98.85%. Adding a fourth or fifth slot yields at most another 0.05 percentage points. With three slots, increasing from one iteration to two or three improves success by 0.20 and 0.30 percentage points while adding 6.81 and 12.32 ms of inference latency, respectively.

As described in the paper, the three-slot assignments organize visual tokens into spatially coherent regions corresponding to the robot arms, task-relevant objects, and surrounding workspace. This object-centric organization is transferred from compression to reconstruction through the object-path assignments and slot-mediated spatial prior, preserving task-relevant structural information. Individual slots do not have fixed semantic identities across episodes. Representative visualizations are available in our Hugging Face.

If the additional results and clarifications address your main concerns, we would appreciate your reconsideration of the overall rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the AC and All Reviewers

Dear AC and Reviewers,

We sincerely appreciate the time, effort, and constructive feedback of the AC and all reviewers. We summarize the review outcome and our additional experimental findings below for your convenience.

Overall Outcome

Reviewer 1FYQ confirmed that our additional experiments and clarifications fully addressed their concerns and maintained their positive rating. Reviewer 1Sq6 acknowledged the strengthened empirical evidence and practical value of our approach, with their remaining reservation focusing on methodological novelty. Reviewer jU8n acknowledged the effectiveness of our approach and its clear latency advantage, with their remaining concerns primarily focusing on architectural complexity and additional training costs. The discussion has largely shifted from empirical effectiveness to methodological novelty and architectural complexity, which we further clarify.

Strengths Highlighted by Reviewers

  1. Practically meaningful research problem and favorable performance–efficiency trade-off. STaC-WAM achieves a 2.10× inference speedup over Fast-WAM while improving LIBERO success from 97.60% to 98.85% (Reviewers jU8n, 1FYQ).
  2. Effective object-centric interface design. Combining object-centric compression with a complementary detail pathway preserves control-relevant information, with component ablations supporting their complementary roles (Reviewers 1FYQ, 1Sq6).
  3. Comprehensive experimental evaluation. Experiments cover LIBERO, the full RoboTwin benchmark, and five real-robot tasks, together with component ablations and representation visualizations (Reviewers jU8n, 1FYQ).
  4. Substantial implementation and reproducibility details. The supplementary material provides the main model implementation, training objectives, configurations, and evaluation code (Reviewer 1Sq6).

Weaknesses Raised and Our Responses

  • Added value of the structured interface (All reviewers). With matched parameters and TSA supervision, STaC + TSA outperforms Large MLP + TSA by 2.34/1.90 points on RoboTwin Clean/Randomized, reducing latency from 113.84 to 91.68 ms. OAA yields further gains of 3.63/2.77 points (Table A).
  • Comparison with recent WAMs (Reviewers jU8n, 1FYQ). Under matched benchmarking settings, STaC-WAM achieves a 2.07× speedup over Faster-WAM with comparable reported success rates, and approximately 16-point higher RoboTwin success rates than Light-WAM in both settings.
  • Conditioning granularity versus action-expert capacity (Reviewer 1Sq6). Increasing conditioning groups from one to six at fixed depth improves LIBERO success by 1.95 points, whereas increasing action-expert depth from three to six at fixed conditioning granularity yields only 0.15 points (Table C).
  • Temporal correspondence in TSA (Reviewer 1Sq6). Adjacent-frame, same-frame, and shuffled-frame TSA achieve 98.85%, 97.70%, and 95.45%, respectively, demonstrating the importance of temporal correspondence (Table D).
  • Slot configuration (Reviewer jU8n). Additional ablations validate three slots and one iteration as an effective performance–latency trade-off (Table B).
  • Implementation and reproducibility (Reviewers jU8n, 1FYQ). We clarify the TSA head, spatial prior, training configurations, VideoSAUR costs, and evaluation protocols, and release the VideoSAUR teacher checkpoints.
  • Conceptual and methodological contribution (Reviewer 1Sq6). We address a fundamental architectural question: must predictive visual modeling and action generation scale symmetrically? Our findings reveal that conditioning granularity can matter more than action-expert depth, providing broader insights into asymmetric WAM design.
  • Architectural complexity and training cost (Reviewer jU8n). Despite its more sophisticated interface, STaC-WAM substantially accelerates inference over the contemporaneous Faster-WAM. VideoSAUR is training-only and introduces no inference overhead.

Closing Remarks

Our additional controlled experiments demonstrate that the structured interface provides benefits beyond auxiliary supervision alone, while enabling substantial inference acceleration. We appreciate that the reviewers have acknowledged the strengthened empirical evidence and practical merits of our approach. We believe that the additional findings further substantiate both the methodological contribution and practical significance of STaC-WAM. We kindly ask the AC to take these findings into consideration when evaluating our work.

We would be happy to address any further questions and sincerely thank the AC and all reviewers again for their time and constructive feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.