- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper places action generation at the bottleneck of a hierarchical visual backbone and uses a subsequent synthesis pathway for future prediction. Joint training provides predictive supervision, while action-only inference skips the synthesis stages. Experiments cover LIBERO, LIBERO-Plus, and RoboTwin.
Critical strengths
-
The architecture offers a clear separation between training capacity and action-time computation, with a useful performance–memory trade-off.
-
Evaluation covers standard manipulation, distribution shifts, and bimanual tasks. The matched action-only training comparison provides evidence that predictive supervision benefits control.
Critical weaknesses
-
The distinctive R/W design is weakly justified empirically. In Table 3(b), convolution with retained detail scores 65.4 versus 65.1 for the reference, while removing detail reduces performance by only 0.3 points. These results do not establish the necessity of orthogonal analysis–synthesis or retained detail for control.
-
Prediction quality and deployment advantages need stronger qualification. Selected prediction comparisons follow each policy’s own actions, confounding prediction accuracy with trajectory difficulty; Table 2 retains heterogeneous baseline protocols, and Appendix Table 5 excludes visual encoding and decoding. These measurements support narrower conclusions than uniformly matched control superiority and complete-system efficiency.
-
Evaluation is limited to simulation, with no real-robot experiment. Given the emphasis on deployment efficiency and scene robustness, the reported benchmark results and computational profiles leave open whether these benefits translate to physical manipulation under sensor noise, calibration errors, and contact uncertainty.
-
Fig.1 and 2 do not clearly communicate the architectural intuition or information flow. The visual symbols and module labels are difficult to interpret without repeatedly consulting the text, particularly regarding what is compressed, what bypasses action generation, and how the retained features support future prediction. Explicit tensor labels and clearer distinctions between the action and prediction pathways would substantially improve readability.
Questions for authors
-
What benefit does R/W provide over convolutional compression, particularly for future prediction?
-
Which RoboTwin baselines share Lute’s training data, checkpoint-selection procedure, and first-attempt evaluation?
-
Do the action-only latency and memory measurements include all required encoders and resident model weights?
Other suggestions (not decisive for my recommendation): Distinguish percentage-point improvements from relative percentages; state how prediction examples were selected; clarify the architectural advance over Fast-WAM’s training-time prediction and action-only inference.
5 comments
Response to Reviewer KZNf (2/3)
Q3. Not all of them in every row—as noted in the paper (Sec. 3.4, Lines 206–222; Table 1 caption, Line 240; Sec. 4.3, Line 464; and Appendix Table 5 caption, Line 621), it depends on the specific measurement configuration. Specifically, Table 3(c) includes the visual encoder, while the two Action rows distinguish full resident model weights (3.87 GiB) from active action weights only (1.66 / 5.75 GiB). Detailed configurations are listed below.
All tests use the same GPU with Batch Size = 1, BF16, 10-step Euler integration, and cached text features (encoded once per episode):
| Metric / Model | Table 3(c): Closed-Loop Policy (, w/ Visual Encoding) | Table 5: Backbone-Only (, w/o Visual Enc./Dec.) |
|---|---|---|
| Lute 1.4B (A0.87B) | Joint: 192.6 ms / 3.89 GiB (full weights); Action: 134.3 ms / 3.87 GiB (full weights; 124.6 ms / 3.85 GiB w/ feature reuse); Action: 124.6 ms / 1.66 GiB (0.87B action weights only) | Joint: 137.79 ms / 2.97 GiB (full backbone weights) |
| Lute 5.0B (A3.05B) | Joint: 361.0 ms / 10.55 GiB (full weights); Action: 250.6 ms / 5.75 GiB (3.05B action weights only) | Joint: 254.83 ms / 9.50 GiB (full backbone weights) |
| Protocol | Peak allocated memory; 5 warmup / 20 timed runs | Peak reserved memory; 3 warmup / 10 timed runs |
- Table 3(c) vs. Table 5 Latency: Table 3(c) encodes the current frame and processes a 3-frame window (); Table 5 takes a single pre-encoded latent frame () without visual encoding/decoding.
- Joint vs. Action Latency (~30% saving): With full weights resident, skipping the return prediction path (Blocks 19–30) reduces latency from
192.6 msto134.3 ms(124.6 msw/ feature reuse) for 1.4B, and from361.0 msto250.6 msfor 5.0B. - Joint vs. Action Memory (~50% saving): With full weights resident, 1.4B action-only memory is
3.87 GiB. Since the action stream is independent of the prediction branch (Eq. 8), loading only active action weights (0.87B / 3.05B) lowers memory to1.66 GiB / 5.75 GiB. We will clarify both residency settings in the revised captions.
Response to Reviewer KZNf (3/3)
W2 and Q2. We understand your concern, but we respectfully disagree on several points. First, WAM action decisions emerge autonomously from the model, making it mechanistically impossible to manually force different WAMs onto identical trajectories; our rollout comparisons were not cherry-picked by score (details will be provided in the following response). Second, Table 2(b) (Line 273) already explicitly marks co-trained policies with , and Lute holds a substantial advantage over baselines under the exact same protocol. Finally, the computational core of our architectural innovation lies in the backbone (as the text encoder are shared across works); Table 3(c) and Figure 5 already provide full policy-level timing including encoding, whereas the backbone-only protocol in Appendix Table 5 was specifically designed to accommodate Light-WAM's structural particularity.
Specifically:
(1) Fairness of Visual Rollout Comparisons. A WAM is fundamentally a VLA with future rollout capability (or a world model with decision-making). A reliable WAM must simultaneously grasp the task goal, action execution, and the resulting visual changes; if it rapidly loses structural coherence during multi-step recursive rollouts (without fresh ground-truth observations), its world-modeling capability is questionable. Because WAMs do not accept custom external actions during inference, starting from identical initial scenes and 2 warmup rounds to evaluate the consistency between each model's predicted frames and its own execution is the only physically self-consistent way to evaluate WAM world modeling.
(2) Sample Selection. Lute does not rank first on every single-sample visual metric and exhibits corrective steering along its trajectories—which is precisely the entry point of our closed-loop behavior analysis (e.g., Fig. 6). Among strong models of comparable scale (Lute 5B, Fast-WAM, DreamWAM in Fig. 3), single-sample metrics trade wins, confirming we did not cherry-pick by score. Due to space limits, Fig. 3 selects 1 of 4 camera-perturbation candidates, and Fig. 7 selects 1 case per suite (Spatial, Goal, Object) with clear dual-view visibility after 2 warmup rounds to contrast MAE and Edge-F1. For unbiased statistics, we pre-specified the evaluation set before viewing any outputs—the original 7 cases (+32/+64 steps) plus 16 new stratified cases at +32 steps (4 Spatial, 6 Goal, 6 Object)—and computed the unweighted mean over all rollouts including failures (full lists and per-case metrics will be added to the Appendix):
| Method (16-Episode Mean, +32 Steps) | External Edge-F1 | Wrist Edge-F1 | External MAE | Wrist MAE |
|---|---|---|---|---|
| Lute 1.4B | 0.6753 | 0.2169 | 0.0869 | 0.1926 |
| Light-WAM | 0.0225 | 0.1270 | 0.0938 | 0.1305 |
Lute 1.4B achieves higher dual-view Edge-F1 and better external MAE, while Light-WAM yields lower wrist MAE due to smooth, blurry predictions.
(3) Table 2 and RoboTwin Protocol (Response to Question 2). All WAM baselines in Table 2(a) and all dagger-marked () multi-task models in Table 2(b) (Line 273: "Daggers mark co-trained policies", including Fast-WAM, $\pi_{0.5}$, and X-VLA) follow the exact same Fast-WAM protocol as Lute: jointly training a single shared model on 27,500 demonstrations across all 50 RoboTwin 2.0 tasks (50 Clean + 500 Randomized per task) and evaluating the same final checkpoint on both Easy and Hard under first-attempt evaluation. Under this strictly aligned protocol, Lute 5B holds a clear advantage over matched baselines.
(4) Efficiency and the Rationale of Table 5. Table 3(c) and Figure 5 already report policy-level costs including visual encoding; Appendix Table 5 measures backbone cost specifically to accommodate Light-WAM: Light-WAM applies spatial pooling to future visual targets, outputting predictions in a downsampled latent space that relies on a separate generation/decoding bypass (App. C.4). While Light-WAM speeds up action-only inference, neither its speed nor its prediction quality is optimal in joint action+visual inference; Table 5 places both models on equal footing to show that Lute, as a complete WAM, has a lighter and faster joint-inference backbone than Light-WAM (see Q3).
Response to Reviewer KZNf (1/3)
Thank you for your thorough review and constructive feedback. Please allow us to address your comments and questions below.
W1 and Q1. R/W is faster and achieves comparable control performance to the parameter-heavy convolutional compression method. The design goal of R/W and retained detail is not to boost control alone, but to let action generation operate on only 1/16 of the visual tokens while preserving spatial information for prediction; the comparable control scores across the variants in Table 3(b) show that this partitioning incurs no control loss.
Summary coefficients enter the compact action stage while detail coefficients bypass it for prediction; strict orthogonality guarantees exact input reconstruction when coefficients are unaltered (W∘R=I, adding only 12D learnable scalars). Since their benefit should be evaluated on prediction (working as a world model), we pre-specified 32 held-out episodes across Spatial, Goal, Object, and Long before inspecting any outputs, and compared +32-step prediction quality (merged into Table 3(b) and App. C):
Conv +)0.087/0.1950.689 / 0.232R/W -)0.133 / 0.2080.482 / 0.150R/W +)0.087 / 0.1910.686 / 0.225Fixed Haar +)64.60.093 / 0.2050.671 / 0.209Because convolutional compression introduces many more parameters across multi-step flow-matching inference, R/W provides two key benefits:
+32steps,Full R/Wachieves the lowest wrist MAE (0.191) and matchesConv +in boundary fidelity (Edge-F1), while clearly outperformingFixed Haar +andR/W -.W3 and W4. Thank you for this helpful suggestion. We are currently collecting real-robot data and will supplement sim-to-real results in the revised manuscript. We will redraw Figures 1 and 2 with explicit tensor dimensions and clearer pathway distinctions in the revised manuscript.
Meanwhile, reading out actions at the 1/16 compact stage and omitting the return prediction branch during inference are intrinsic computational properties of the network architecture—the resulting ~30% latency reduction and ~50% memory savings are independent of specific robot hardware. We have already demonstrated the robustness of the system to various types of noise in the initial conditions of the scenario on the simulator. Investigating how physical sensor noise, calibration errors, and contact uncertainty affect policy control requires substantial data collection and systematic benchmarking across baselines to chart the direction for Sim2Real, which merits a dedicated standalone study in its own right.
Others.
(1) We will standardize these expressions across the manuscript as follows:
(2). Example Selection. See W2 and Q2.
(3). Architectural Advance Over Fast-WAM.While both perform predictive training and action-only inference, Lute introduces two structural differences: