acceptodds
Under review as a conference paper at ICLR 2027

Lute: Folding Visual Hierarchies Around Action for World Action Models

Abstract

World–action models generate actions and predict the resulting future world states. Existing methods reduce future generation or reuse predictive features to make control more efficient. These strategies can still leave dense visual processing on the action path, while downsampling prediction targets removes spatial detail needed for world estimation. We introduce Lute (distinct strings, one shared resonant body), where folding visual hierarchies around action formation keeps action readout compact while preserving spatial detail for future prediction. Computation proceeds from visual analysis to action generation and then future prediction. Hierarchical processing progressively compresses visual features, with action tokens introduced only at the most compact stage. A paired analysis and synthesis transform retains complementary spatial information that bypasses action formation and supports subsequent prediction conditioned on action representations. All stages use the same block design, with roles defined by spatial scale and information access. Joint flow learning lets both objectives shape the shared hierarchy. When only actions are required, inference omits the subsequent prediction stages. Experiments show competitive control and improved tradeoffs between scene robustness and memory cost over efficient WAM baselines, while retaining structured future predictions.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Scene Robustness, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KZNfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper places action generation at the bottleneck of a hierarchical visual backbone and uses a subsequent synthesis pathway for future prediction. Joint training provides predictive supervision, while action-only inference skips the synthesis stages. Experiments cover LIBERO, LIBERO-Plus, and RoboTwin.

Critical strengths
  1. The architecture offers a clear separation between training capacity and action-time computation, with a useful performance–memory trade-off.

  2. Evaluation covers standard manipulation, distribution shifts, and bimanual tasks. The matched action-only training comparison provides evidence that predictive supervision benefits control.

Critical weaknesses
  1. The distinctive R/W design is weakly justified empirically. In Table 3(b), convolution with retained detail scores 65.4 versus 65.1 for the reference, while removing detail reduces performance by only 0.3 points. These results do not establish the necessity of orthogonal analysis–synthesis or retained detail for control.

  2. Prediction quality and deployment advantages need stronger qualification. Selected prediction comparisons follow each policy’s own actions, confounding prediction accuracy with trajectory difficulty; Table 2 retains heterogeneous baseline protocols, and Appendix Table 5 excludes visual encoding and decoding. These measurements support narrower conclusions than uniformly matched control superiority and complete-system efficiency.

  3. Evaluation is limited to simulation, with no real-robot experiment. Given the emphasis on deployment efficiency and scene robustness, the reported benchmark results and computational profiles leave open whether these benefits translate to physical manipulation under sensor noise, calibration errors, and contact uncertainty.

  4. Fig.1 and 2 do not clearly communicate the architectural intuition or information flow. The visual symbols and module labels are difficult to interpret without repeatedly consulting the text, particularly regarding what is compressed, what bypasses action generation, and how the retained features support future prediction. Explicit tensor labels and clearer distinctions between the action and prediction pathways would substantially improve readability.

Questions for authors
  1. What benefit does R/W provide over convolutional compression, particularly for future prediction?

  2. Which RoboTwin baselines share Lute’s training data, checkpoint-selection procedure, and first-attempt evaluation?

  3. Do the action-only latency and memory measurements include all required encoders and resident model weights?

Other suggestions (not decisive for my recommendation): Distinguish percentage-point improvements from relative percentages; state how prediction examples were selected; clarify the architectural advance over Fast-WAM’s training-time prediction and action-only inference.

5 comments
AuthorsOpenReview ↗

Response to Reviewer KZNf (1/3)

Thank you for your thorough review and constructive feedback. Please allow us to address your comments and questions below.

W1 and Q1. R/W is faster and achieves comparable control performance to the parameter-heavy convolutional compression method. The design goal of R/W and retained detail is not to boost control alone, but to let action generation operate on only 1/16 of the visual tokens while preserving spatial information for prediction; the comparable control scores across the variants in Table 3(b) show that this partitioning incurs no control loss.

Summary coefficients enter the compact action stage while detail coefficients bypass it for prediction; strict orthogonality guarantees exact input reconstruction when coefficients are unaltered (W∘R=IW\circ R = I, adding only 12D12D learnable scalars). Since their benefit should be evaluated on prediction (working as a world model), we pre-specified 32 held-out episodes across Spatial, Goal, Object, and Long before inspecting any outputs, and compared +32+32-step prediction quality (merged into Table 3(b) and App. C):

VariantControl Success (LIBERO-Plus)MAE Ext. / Wrist ↓\downarrow (+32)Edge-F1 Ext. / Wrist ↑\uparrow (+32)
Convolution, w/ detail (Conv +)65.40.087 / 0.1950.689 / 0.232
R/W, w/o detail (R/W -)64.80.133 / 0.2080.482 / 0.150
Full R/W (learned rotation, R/W +)65.10.087 / 0.1910.686 / 0.225
Fixed Haar, w/ detail (Fixed Haar +)64.60.093 / 0.2050.671 / 0.209

Because convolutional compression introduces many more parameters across multi-step flow-matching inference, R/W provides two key benefits:

  • Structure & Efficiency: It provides a strictly orthogonal summary–detail decomposition where synthesis reuses the transposed analysis matrix (∥x∥22=∣S∣2+∥Δ∥22\|x\|_2^2 = |S|^2 + \|\Delta\|_2^2), guaranteeing exact reconstruction and local energy preservation with only 12D12D scalars across two levels while avoiding the ~8%–20% joint-inference slowdown from transposed convolutions on the return path.
  • Future Prediction: At +32 steps, Full R/W achieves the lowest wrist MAE (0.191) and matches Conv + in boundary fidelity (Edge-F1), while clearly outperforming Fixed Haar + and R/W -.

W3 and W4. Thank you for this helpful suggestion. We are currently collecting real-robot data and will supplement sim-to-real results in the revised manuscript. We will redraw Figures 1 and 2 with explicit tensor dimensions and clearer pathway distinctions in the revised manuscript.

Meanwhile, reading out actions at the 1/16 compact stage and omitting the return prediction branch during inference are intrinsic computational properties of the network architecture—the resulting ~30% latency reduction and ~50% memory savings are independent of specific robot hardware. We have already demonstrated the robustness of the system to various types of noise in the initial conditions of the scenario on the simulator. Investigating how physical sensor noise, calibration errors, and contact uncertainty affect policy control requires substantial data collection and systematic benchmarking across baselines to chart the direction for Sim2Real, which merits a dedicated standalone study in its own right.

Others.

(1) We will standardize these expressions across the manuscript as follows:

ChangeppRelative Gain
62.1% →\rightarrow 65.5%+3.45.48%
30.2% →\rightarrow 38.6%+8.427.81%
75.47% →\rightarrow 76.83%+1.361.80%

(2). Example Selection. See W2 and Q2.

(3). Architectural Advance Over Fast-WAM.While both perform predictive training and action-only inference, Lute introduces two structural differences:

    1. Actions are read out solely at the compact stage where visual tokens are reduced to 1/16 of the original resolution; (See Q3)
    1. Complementary spatial details bypass the action stage and enter the prediction synthesis pathway directly. (See W1)
AuthorsOpenReview ↗

Response to Reviewer KZNf (2/3)

Q3. Not all of them in every row—as noted in the paper (Sec. 3.4, Lines 206–222; Table 1 caption, Line 240; Sec. 4.3, Line 464; and Appendix Table 5 caption, Line 621), it depends on the specific measurement configuration. Specifically, Table 3(c) includes the visual encoder, while the two Action rows distinguish full resident model weights (3.87 GiB) from active action weights only (1.66 / 5.75 GiB). Detailed configurations are listed below.

All tests use the same GPU with Batch Size = 1, BF16, 10-step Euler integration, and cached text features (encoded once per episode):

Metric / ModelTable 3(c): Closed-Loop Policy (L=3L=3, w/ Visual Encoding)Table 5: Backbone-Only (L=1L=1, w/o Visual Enc./Dec.)
Lute 1.4B (A0.87B)Joint: 192.6 ms / 3.89 GiB (full weights); Action: 134.3 ms / 3.87 GiB (full weights; 124.6 ms / 3.85 GiB w/ feature reuse); Action: 124.6 ms / 1.66 GiB (0.87B action weights only)Joint: 137.79 ms / 2.97 GiB (full backbone weights)
Lute 5.0B (A3.05B)Joint: 361.0 ms / 10.55 GiB (full weights); Action: 250.6 ms / 5.75 GiB (3.05B action weights only)Joint: 254.83 ms / 9.50 GiB (full backbone weights)
ProtocolPeak allocated memory; 5 warmup / 20 timed runsPeak reserved memory; 3 warmup / 10 timed runs
  • Table 3(c) vs. Table 5 Latency: Table 3(c) encodes the current frame and processes a 3-frame window (L=3L=3); Table 5 takes a single pre-encoded latent frame (L=1L=1) without visual encoding/decoding.
  • Joint vs. Action Latency (~30% saving): With full weights resident, skipping the return prediction path (Blocks 19–30) reduces latency from 192.6 ms to 134.3 ms (124.6 ms w/ feature reuse) for 1.4B, and from 361.0 ms to 250.6 ms for 5.0B.
  • Joint vs. Action Memory (~50% saving): With full weights resident, 1.4B action-only memory is 3.87 GiB. Since the action stream is independent of the prediction branch (Eq. 8), loading only active action weights (0.87B / 3.05B) lowers memory to 1.66 GiB / 5.75 GiB. We will clarify both residency settings in the revised captions.
AuthorsOpenReview ↗

Response to Reviewer KZNf (3/3)

W2 and Q2. We understand your concern, but we respectfully disagree on several points. First, WAM action decisions emerge autonomously from the model, making it mechanistically impossible to manually force different WAMs onto identical trajectories; our rollout comparisons were not cherry-picked by score (details will be provided in the following response). Second, Table 2(b) (Line 273) already explicitly marks co-trained policies with †\dagger, and Lute holds a substantial advantage over baselines under the exact same protocol. Finally, the computational core of our architectural innovation lies in the backbone (as the text encoder are shared across works); Table 3(c) and Figure 5 already provide full policy-level timing including encoding, whereas the backbone-only protocol in Appendix Table 5 was specifically designed to accommodate Light-WAM's structural particularity.

Specifically:

(1) Fairness of Visual Rollout Comparisons. A WAM is fundamentally a VLA with future rollout capability (or a world model with decision-making). A reliable WAM must simultaneously grasp the task goal, action execution, and the resulting visual changes; if it rapidly loses structural coherence during multi-step recursive rollouts (without fresh ground-truth observations), its world-modeling capability is questionable. Because WAMs do not accept custom external actions during inference, starting from identical initial scenes and 2 warmup rounds to evaluate the consistency between each model's predicted frames and its own execution is the only physically self-consistent way to evaluate WAM world modeling.

(2) Sample Selection. Lute does not rank first on every single-sample visual metric and exhibits corrective steering along its trajectories—which is precisely the entry point of our closed-loop behavior analysis (e.g., Fig. 6). Among strong models of comparable scale (Lute 5B, Fast-WAM, DreamWAM in Fig. 3), single-sample metrics trade wins, confirming we did not cherry-pick by score. Due to space limits, Fig. 3 selects 1 of 4 camera-perturbation candidates, and Fig. 7 selects 1 case per suite (Spatial, Goal, Object) with clear dual-view visibility after 2 warmup rounds to contrast MAE and Edge-F1. For unbiased statistics, we pre-specified the evaluation set before viewing any outputs—the original 7 cases (+32/+64 steps) plus 16 new stratified cases at +32 steps (4 Spatial, 6 Goal, 6 Object)—and computed the unweighted mean over all rollouts including failures (full lists and per-case metrics will be added to the Appendix):

Method (16-Episode Mean, +32 Steps)External Edge-F1 ↑\uparrowWrist Edge-F1 ↑\uparrowExternal MAE ↓\downarrowWrist MAE ↓\downarrow
Lute 1.4B0.67530.21690.08690.1926
Light-WAM0.02250.12700.09380.1305

Lute 1.4B achieves higher dual-view Edge-F1 and better external MAE, while Light-WAM yields lower wrist MAE due to smooth, blurry predictions.

(3) Table 2 and RoboTwin Protocol (Response to Question 2). All WAM baselines in Table 2(a) and all dagger-marked (†\dagger) multi-task models in Table 2(b) (Line 273: "Daggers mark co-trained policies", including Fast-WAM†\dagger, $\pi_{0.5}$†\dagger, and X-VLA†\dagger) follow the exact same Fast-WAM protocol as Lute: jointly training a single shared model on 27,500 demonstrations across all 50 RoboTwin 2.0 tasks (50 Clean + 500 Randomized per task) and evaluating the same final checkpoint on both Easy and Hard under first-attempt evaluation. Under this strictly aligned protocol, Lute 5B holds a clear advantage over matched baselines.

(4) Efficiency and the Rationale of Table 5. Table 3(c) and Figure 5 already report policy-level costs including visual encoding; Appendix Table 5 measures backbone cost specifically to accommodate Light-WAM: Light-WAM applies spatial pooling to future visual targets, outputting predictions in a downsampled latent space that relies on a separate generation/decoding bypass (App. C.4). While Light-WAM speeds up action-only inference, neither its speed nor its prediction quality is optimal in joint action+visual inference; Table 5 places both models on equal footing to show that Lute, as a complete WAM, has a lighter and faster joint-inference backbone than Light-WAM (see Q3).

Reviewer_8ZAFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Lute is a hierarchical world-action model designed for language-conditioned robotic manipulation tasks. The model compresses visual tokens through three distinct stages: "Slow," "Mid," and "Fast", with action tokens integrated exclusively at the "Fast" stage. By employing paired orthogonal transformations, the model preserves spatial details for subsequent visual prediction, while action representations are incorporated into the prediction branch. The model utilizes a joint flow matching framework (Lipman et al., 2023) to simultaneously train for action generation and the prediction of future visual features. Because the action flow does not depend on potentially noisy future visual inputs, the prediction branch can be bypassed during inference and only action execution is required. The paper evaluates two model variants of different scales on benchmarks, including LIBERO, LIBERO-Plus, and RoboTwin 2.0, reporting on control success rates, robustness, inference costs, example predictions, and ablation studies for individual components. The core contribution of this work lies in achieving an improved trade-off between efficient action computation and informative future prediction.

Ref

  • Lipman, Y., Chen, R. T. Q., Ben-Hamu, H., Nickel, M., and Le, M. (2023). Flow Matching for Generative Modeling. International Conference on Learning Representations (ICLR). Paper.
Critical strengths
  1. The architecture offers a clear way to reduce action computation. The joint execution of "action generation" and "image prediction" has a precedent in GR-1 (Wu et al., 2024). Lute organizes these tasks through spatial compression, detail preservation, and an explicit action-output mechanism. Equation (8) defines a one-way dependency from "action generation" to "prediction," allowing the control flow to bypass the predictive branch. Table 3(c) highlights the specific advantages of Lute 1.4B: compared to joint inference, performing only action inference reduces the median latency from 192.6 ms to 124.6 ms and lowers peak memory usage from 3.89 GiB to 1.66 GiB. These results demonstrate that this architecture effectively reduces the model's computational overhead.

  2. The comparison at fixed action capacity supports predictive training. Table 2 compares Lute 1.4B with a variant trained exclusively on actions; both models feature an identical parameter count of 0.87B dedicated to action-related components. The results show that, with the incorporation of predictive training, the success rate on LIBERO-Plus tasks increases from 62.1% to 65.5%, while the success rate on RoboTwin 2.0 Hard tasks rises from 30.2% to 38.6%. These findings demonstrate that, given a fixed capacity for the action branch, introducing predictive supervision provides tangible benefits. This comparison offers the strongest empirical support for the "joint learning" design approach.

Ref

  • Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, J., Li, X., Liu, M., Li, H., and Kong, T. (2024). Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation. International Conference on Learning Representations (ICLR). Paper.
Critical weaknesses
  1. The claims about practical deployment lack real-robot validation. All reported evaluations use simulation benchmarks. No real-world experiments are reported. The paper emphasizes scene robustness and practical deployment costs. Yet the effects of physical sensing, contact, and actuation remain untested. The efficiency evidence also has a limited scope. Table 5 reports backbone costs with cached text features. It excludes visual encoding and decoding and no end-to-end performance measurements on a physical robot are reported. Related ICLR papers provide relevant examples of physical validation. GR-1 evaluates real-robot manipulation and generalization (Wu et al., 2024). RDT-1B also evaluates these capabilities on a physical bimanual robot (Liu et al., 2025). For Lute, the lack of physical validation limits my confidence in the practical robustness-efficiency trade-off. This is a major reason for my weak-rejection assessment.

  2. The specific benefit of the detail-preserving hierarchy is not sufficiently validated. Figures 3 and 7 present only a few selected prediction examples, supplemented by perceptual or structural metrics; however, the paper fails to report overall prediction accuracy across specific task sets and experimental episodes. Furthermore, the results in Table 3(b) also fail to clearly demonstrate the advantages of "retaining details": in the LIBERO-Plus benchmark, the R/W configuration with details achieved a score of 65.1%, the convolutional configuration with details scored 65.4%, and the R/W configuration without details scored 64.8%. While the table reports success rates for control tasks, it provides no corresponding predictive metrics. These results are insufficient to substantiate a significant advantage for the proposed "orthogonal detail-retention mechanism." The existing evidence lends more support to the general concept of "predictive training" rather than the specific "analysis/synthesis" design presented in the paper. This gap in the argument further weakens the case for the paper's acceptance.

Ref

  • Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, J., Li, X., Liu, M., Li, H., and Kong, T. (2024). Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation. International Conference on Learning Representations (ICLR). Paper.
  • Liu, S., Wu, L., Li, B., Tan, H., Chen, H., Wang, Z., Xu, K., Su, H., and Zhu, J. (2025). RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation. International Conference on Learning Representations (ICLR). Paper.
Questions for authors
  1. Clarification of deployment scope and available evidence. Were any real-robot experiments performed? If so, please report any available tasks, trial counts, matched baselines, and success rates. Besides, are end-to-end latency measurements available? Please specify the timing boundaries and replanning settings for Table 3(c) and Figure 5. Table 5 explicitly measures backbone costs only. Moreover, which deployment conclusions are intended to remain limited to simulation bench?

  2. Clarification and targeted validation of prediction quality. How were the examples in Figures 3 and 7 selected? How many tasks and episodes were evaluated? Can you report aggregate prediction metrics, including unsuccessful rollouts? This would help separate predictor quality from differences in policy trajectories.

  3. Targeted validation of the proposed mechanism. Can you report future-prediction accuracy for the Table 3(b) variants under the same evaluation protocol? Does R/W with detail improve prediction over both convolution with detail and R/W without detail? As GR-1 already motivates joint image prediction and action learning (Wu et al., 2024). What additional benefit does Lute's particular hierarchy provide? This evidence would directly affect the assessment of the paper contribution.

Ref

  • Wu, H., Jing, Y., Cheang, C., Chen, G., Xu, J., Li, X., Liu, M., Li, H., and Kong, T. (2024). Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation. International Conference on Learning Representations (ICLR). Paper.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8ZAF (1/3)

Thank you for your constructive suggestions. Please allow us to answer your questions and clarify several facts below.

W1. We are currently collecting real-robot data and will supplement real-robot inference results in the revised manuscript, which requires some time.

Admittedly, this submission does not include physical robot experiments; however, as stated in the Abstract ("In complex simulations") and Section 4.1 (Line 234, "Action latency and peak memory quantify deployment cost"), our paper focuses on the practical robustness–efficiency trade-off on simulation benchmarks rather than physical hardware integration. In fact, model inference always runs on the GPU—regardless of whether the downstream actuator is a simulator or a physical robot, the computational efficiency of the model remains unchanged.

  • (detail also in Q3.) Regarding GR-1 (Wu et al., 2024), RDT-1B (Liu et al., 2025), and Evaluation Scope (See Q3):
    1. RDT-1B is already evaluated as a core bimanual baseline in Table 2(b) on RoboTwin 2.0, where Lute 1.4B / 5B achieves 38.60% / 52.50% on Hard, substantially outperforming RDT-1B (13.72%).
    2. Consistent with recent WAM baselines (Light-WAM, DreamWAM, Faster-WAM, WorldVLA), we use controlled, reproducible benchmarks (LIBERO, LIBERO-Plus with 7 perturbation axes, and bimanual RoboTwin 2.0) to isolate the algorithmic gains of multi-scale hierarchy and predictive supervision.
  • (detail also in Q1.) Full-Model Efficiency Including Encoders (Table 3(c) includes current-frame visual encoding with a 3-frame history window L=3L=3, whereas Table 5 uses a single pre-encoded latent frame L=1L=1 without visual encoding/decoding; see Question 1 for the full table):
    • Latency Reduction (~30%): Under the same setting with current-frame visual encoding, L=3L=3 history, and full resident weights, skipping the return prediction path reduces 1.4B latency from 192.6 ms to 134.3 ms (and 5.0B from 361.0 ms to 250.6 ms). We use the same T5 text encoder as Fast-WAM and related works, so relative efficiency rankings remain unchanged (running live text encoding adds ~`60 ms` across all models).
    • Memory Reduction (~50%): With full weights resident, 1.4B action-only memory is 3.87 GiB. Because the action stream is independent of the prediction branch (Eq. 8), loading only active action weights (0.87B / 3.05B) reduces peak memory from 3.89 / 10.55 GiB to 1.66 / 5.75 GiB.

Q3. We have provided the future-prediction accuracy and analyzed its impact on prediction quality above (please refer to Weakness 2). While GR-1 interleaves dense visual and action tokens within a single uniform-resolution causal Transformer, Lute's multi-scale folding introduces three structural decouplings:

  1. Compact Action Interaction: Actions are read out at the Fast stage after two spatial reductions, compressing visual tokens to 1/16 of the original resolution;
  2. Orthogonal Detail Bypass: Complementary spatial details Δs,Δm\Delta_s, \Delta_m bypass the action stage directly to the return pathway, recovering the 15/16 high-frequency spatial information omitted from the action stage (see Weakness 2);
  3. One-Way Triangular Dependency: Future visual latents enter the synthesis path only after action readout (∂vA/∂Zu+=0\partial v_A / \partial Z_u^+ = 0), allowing action execution to completely skip the prediction branch—reducing latency by 30% (192.6 ms → 134.3 ms) and action-only memory to 1.66 GiB (detail also in Q1.).
AuthorsOpenReview ↗

Response to Reviewer 8ZAF (2/3)

W2 and Q2. The prediction examples were not cherry-picked by score, and the samples are drawn from the inference rollouts of the models in Tables 1 and 2. Lute's architecture further improves inference speed and reduces computational cost, as supported by our experiments. Detailed answers follow:

Contribution and Claim. Building upon prior work (Fast-WAM) showing that visual prediction aids action inference, Lute's core contribution is demonstrating that compressing visual tokens yields both faster speed and stronger control, while our R/W module resolves the information mismatch between action inference and visual prediction—enabling action execution without decoding future visuals while preserving superior visual predictions.

(1) Benefit of Predictive Supervision. In Line 274, "VLA indicates only action training" means training without predictive supervision under fixed action-branch capacity. Adding visual prediction boosts LIBERO-Plus from 62.1% to 65.5% and RoboTwin Hard from 30.2% to 38.6%.

(2) The R/W Structure Itself. While the structures in Table 3(b) achieve similar control success rates (within 0.3–0.8 percentage points), R/W's orthogonal decomposition and retained detail are specifically designed to restore spatial structure for future prediction (W∘R=IW\circ R = I, adding only 12D12D scalar parameters). We evaluated +32-step closed-loop future predictions across 32 pre-specified held-out episodes spanning Spatial, Goal, Object, and Long before inspecting any outputs (added to Table 3(b)):

VariantControl Success ↑\uparrow (LIBERO-Plus)MAE Ext. / Wrist ↓\downarrow (+32)Edge-F1 Ext. / Wrist ↑\uparrow (+32)
Convolution, w/ detail (Conv +)65.40.087 / 0.1950.689 / 0.232
R/W, w/o detail (R/W -)64.80.133 / 0.2080.482 / 0.150
Full R/W (learned rotation, R/W +)65.10.087 / 0.1910.686 / 0.225
Fixed Haar, w/ detail (Fixed Haar +)64.60.093 / 0.2050.671 / 0.209

Discarding retained detail (R/W -) causes a sharp degradation in both pixel error and boundary agreement, confirming that visual synthesis relies on spatial detail that bypasses the action stage. Meanwhile, learned orthogonal rotations (R/W +) consistently improve over fixed wavelets (Fixed Haar +) in both views and match parameterized convolutions (Conv +) in control success, pixel error (achieving the lowest wrist and dual-view MAE), and boundary retention—while adding only 12D12D parameters and avoiding the extra parameters and ~8%–20% joint-inference slowdown of transposed convolutions on the return path.

(3) Aggregate Prediction Quality Across the Full Episode Set. Figs. 3 and 7 were selected for suite coverage and interaction visibility, whereas our aggregate evaluation covers all pre-specified episodes (including failed rollouts).

Due to page limits, Fig. 3 selects 1 of 4 camera-perturbation candidates, and Fig. 7 selects 1 case per suite (Spatial, Goal, Object) with clear dual-view visibility after 2 warmup rounds, without selecting for favorable scores (e.g., in Fig. 3, Fast-WAM's external metrics and DreamWAM's wrist metrics exceed Lute 5B, as discussed in Limitations). For unbiased statistics, we pre-specified the evaluation set before viewing any outputs—the original 7 cases (+32/+64 steps) plus 16 new stratified cases at +32 steps (4 Spatial, 6 Goal, 6 Object)—and computed unweighted mean closed-loop metrics across all rollouts including failures:

Method (16-Episode Mean, +32 Steps)External Edge-F1 ↑\uparrowWrist Edge-F1 ↑\uparrowExternal MAE ↓\downarrowWrist MAE ↓\downarrow
Lute 1.4B0.67530.21690.08690.1926
Light-WAM0.02250.12700.09380.1305

Lute 1.4B achieves higher dual-view Edge-F1 and better external MAE, while Light-WAM has lower wrist MAE due to smooth, blurry predictions.

AuthorsOpenReview ↗

Response to Reviewer 8ZAF (3/3)

Q1.

Real-Robot Experiments & Simulation Scope: No real-robot experiments were performed in the initial submission (see W1); all deployment conclusions are currently scoped to our simulation benchmarks.

Timing Boundaries and Replanning. As noted in its caption, Table 5 uses cached text features, excludes visual encoding/decoding, and generates a 32-step action chunk and t+4t+4 visual prediction. End-to-end action-only latency including current-frame visual encoding is 134.3 ms (1.4B); Table 3(c) and Table 5 differ only in timing boundaries and history length.

All tests use the same GPU with Batch Size = 1, BF16, 10-step Euler integration, and cached text features (encoded once per episode):

Metric / ModelTable 3(c): Closed-Loop Policy (L=3L=3, w/ Visual Encoding)Table 5: Backbone-Only (L=1L=1, w/o Visual Enc./Dec.)
Lute 1.4B (A0.87B)Joint: 192.6 ms / 3.89 GiB (full weights); Action: 134.3 ms / 3.87 GiB (full weights; 124.6 ms / 3.85 GiB w/ feature reuse); Action: 124.6 ms / 1.66 GiB (0.87B action weights only)Joint: 137.79 ms / 2.97 GiB (full backbone weights)
Lute 5.0B (A3.05B)Joint: 361.0 ms / 10.55 GiB (full weights); Action: 250.6 ms / 5.75 GiB (3.05B action weights only)Joint: 254.83 ms / 9.50 GiB (full backbone weights)
ProtocolPeak allocated memory; 5 warmup / 20 timed runsPeak reserved memory; 3 warmup / 10 timed runs
  • Table 3(c) vs. Table 5 Latency: Table 3(c) encodes the current frame and processes a 3-frame window (L=3L=3); Table 5 takes a single pre-encoded latent frame (L=1L=1) without visual encoding/decoding.
  • Joint vs. Action Latency (~30% saving): With full weights resident, skipping the return prediction path (Blocks 19–30) reduces latency from 192.6 ms to 134.3 ms (124.6 ms w/ feature reuse) for 1.4B, and from 361.0 ms to 250.6 ms for 5.0B.
  • Joint vs. Action Memory (~50% saving): With full weights resident, 1.4B action-only memory is 3.87 GiB. Since the action stream is independent of the prediction branch (Eq. 8), loading only active action weights (0.87B / 3.05B) lowers memory to 1.66 GiB / 5.75 GiB. We will clarify both residency settings in the revised captions.

Figure 5 (left) uses the exact same policy-level timing boundary as Table 3(c) action-only mode; each forward pass predicts a 32-step action chunk and triggers replanning after executing the first QQ (replan) steps.

End-to-End Latency. Including all encoders on every step adds the live text encoder forward pass (~`60 ms`); any remaining latency in physical deployment comes from hardware communication and robot actuation delay.

Reviewer_NnewOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Lute places action generation at the bottleneck of a shared multiscale visual backbone. Learned orthogonal R/W transforms pass compact summaries toward the action stage and retain spatial details for a predictive return pathway. Actions and future visual features are trained jointly with flow matching, while action-only inference skips the return pathway. Experiments cover LIBERO, LIBERO-Plus, and RoboTwin 2.0. The larger model reports 76.8% success on LIBERO-Plus and 52.5% on RoboTwin Hard.

Critical strengths
  1. The placement of action generation is the most interesting part of the architecture. Action tokens interact with visual features only at the compact stage, while the return pathway retains access to spatial detail. Since future latents enter after action readout, prediction can be omitted without changing the action field. Table 3(c) demonstrates the resulting savings with the same trained parameters: the smaller model’s latency drops from 192.6 to 124.6 ms.

  2. The matched-capacity comparison supports the value of predictive training. LIBERO-Plus success increases from 62.1% to 65.5%, and RoboTwin Hard success increases from 30.2% to 38.6%. I find the Hard result particularly convincing: the additional prediction task produces a substantial control improvement without requiring visual generation during execution.

  3. The fixed-depth stage-allocation study is informative. Moving depth from Mid to Slow reduces both Plus and Long performance, providing evidence for the chosen distribution of computation within the hierarchy.

Critical weaknesses
  1. In Table 3(b), convolution achieves 65.4%, the R/W reference achieves 65.1%, and removing retained detail gives 64.8%. These results leave the specific benefit of R/W unresolved. The authors correctly note that control is more sensitive to the compression operator than to the detail branch. However, prediction is the main reason for retaining detail, and prediction metrics are not reported for these variants. I cannot tell from the current ablation whether orthogonality and detail retention improve the control–prediction trade-off over a learned convolution. This is the main piece of evidence I would like to see added.

  2. The prediction results are mostly selected examples. Figures 3 and 7 show recognizable scene structure, but do not establish how consistently this holds across tasks and horizons. Figure 3 also uses the future observations resulting from each policy’s own actions. This evaluates prediction along each model’s rollout, though the resulting differences in motion and future states complicate comparisons of prediction error. The evaluation needs a clearly specified episode set and aggregate metrics to support a broader conclusion about predictive quality.

  3. The runtime tables need more configuration details. The smaller model’s joint-inference latency is 192.6 ms in Table 3(c) and 137.79 ms in Table 5, but the settings accounting for this difference are not clear. Table 5 explicitly measures backbone cost and excludes visual encoding and decoding. The integration schedule, history settings, and components counted in peak memory should be stated alongside these measurements. In particular, whether prediction weights remain resident affects the interpretation of action-only memory usage.

  4. Variation across runs is not clearly reported. Section 4.3 specifies Seed = 0 for the ablations, which makes differences of a few tenths of a percentage point difficult to interpret. On LIBERO-Plus, Appendix Table 4 reports 75.47% for DreamWAM and 76.83% for Lute 5B; this comparison belongs in Table 2(a) as well. The number of training runs and evaluation seeds should be given. For RoboTwin, the paper should also distinguish Lute’s training configuration from the source protocols used for the published baseline results.

Questions for authors
  1. What are the prediction errors for convolution, R/W without retained detail, and full R/W? Have you tested fixed Haar instead of the learned rotation?

  2. How many episodes are used for the prediction metrics, and how were they selected? Are aggregate results across tasks and horizons available?

  3. Which settings explain the latency difference between Tables 3(c) and 5? Please specify the integration schedule, history length, resident components, and end-to-end action latency.

  4. How many training runs and evaluation seeds were used? Please also clarify the RoboTwin demonstration counts, joint versus per-task training, and Easy/Hard training configurations.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Nnew (1/2)

Thank you for your constructive feedback. Please allow us to address your questions below.

W1 and Q1. The design philosophy of R/W is to improve the control–prediction trade-off by partitioning the past visual information needed for action generation and future prediction with minimal parameter overhead while maintaining inference speed. We have added the prediction metrics and analyzed them below.

Q1. We did not test fixed Haar initially. Because the specific information along the analysis pathway and the predictive return pathway differs, a spatial projection is required; therefore, we did not consider it.

The four variants in Table 3(b) differ by only 0.3–0.5 percentage points from the reference on LIBERO-Plus, confirming that control primarily depends on the low-frequency summary, whereas retained detail Δ\Delta is designed to restore spatial structure for future prediction. Mechanistically, R/W orthogonally decomposes each 2×22\times2 patch into 1 summary component SS (entering the action stage) and 3 detail components Δ\Delta (bypassing it). Strict orthogonality (W∘R=IW\circ R = I, ∥x∥22=∣S∣2+∥Δ∥22\|x\|_2^2 = |S|^2 + \|\Delta\|_2^2) ensures complementary, lossless decomposition using only 12D12D scalars across two levels. By contrast, standard convolution lacks this constraint and requires learnable transposed convolutions on the return path (adding parameters and optimization overhead), slowing multi-step diffusion/flow joint inference by ~8% (~20% under multi-view).

We evaluated +32+32-step prediction quality on 32 held-out episodes (spanning Spatial, Goal, Object, and Long) pre-specified before inspecting any model outputs:

VariantControl Success (LIBERO-Plus)MAE Ext. / Wrist ↓\downarrow (+32)Edge-F1 Ext. / Wrist ↑\uparrow (+32)
Conv, w/ detail (Conv +)65.40.087 / 0.1950.689 / 0.232
R/W, w/o detail (R/W -)64.80.133 / 0.2080.482 / 0.150
Full R/W (learned rotation, R/W +)65.10.087 / 0.1910.686 / 0.225
Fixed Haar, w/ detail (Fixed Haar +)64.60.093 / 0.2050.671 / 0.209

Replacing learned rotation with fixed Haar does not cause action control to collapse, but noticeably degrades visual prediction quality, while Full R/W matches Conv + in predictive fidelity without extra convolutional overhead.

W4 and Q4. Run-to-run variation will be added in the revision. While most performance gains are clearly visible, please allow us to further explain the smaller differences below. We will update the table to include the new comparison. Protocol differences are already noted in the main text.

We used 1 training run and 1 set of evaluation seeds. Our RoboTwin protocol is consistent with Fast-WAM and related works, as detailed below:

  • Evaluation Variance. Using 5 evaluation seeds (42–46) on a fixed 100-episode stratified LIBERO-Plus subset (20 episodes per difficulty level, 14–15 per perturbation category), Lute achieves 81%, 75%, 81%, 79%, and 84%, with a mean of 80.00% and a sample SD of 3.32%.
  • Ablation Seeds. Table 3(b) uses evaluation seed 0; although structural variants show close success rates, each is evaluated over a very large sample size under the full main-experiment protocol, reducing estimator variance. We did not select random seeds; instead, we fixed the seed to 42 consistently with prior works.
  • Training & Evaluation Runs. Each model is trained once and evaluated in a single pass, matching the standard LIBERO-Plus single-attempt protocol with seed 42 (identical to Fast-WAM and Faster-WAM). We will add these details.
  • DreamWAM. We will add DreamWAM to Table 2(a) to match Appendix Table 4: DreamWAM achieves 75.47% vs. 76.83% for Lute 5B.
  • RoboTwin Protocol. Lute jointly trains a single shared model on 27,500 demonstrations, while published baselines retain their source protocols. Consistent with Fast-WAM, we jointly train on mixed Clean and Randomized demonstrations and evaluate the same checkpoint on both Easy and Hard. All WAM baselines in Table 2(a) and all dagger-marked (†\dagger) co-trained policies in Table 2(b) (Line 273: Fast-WAM$\dagger$, π0.5†\pi_{0.5}\dagger, X-VLA$\dagger$) follow the exact same Fast-WAM protocol as Lute, under which Lute 5B holds a clear advantage.
AuthorsOpenReview ↗

Response to Reviewer Nnew (2/2)

Q2 and W2. Due to submission file size limits, the displayed samples were selected solely for task-suite coverage and initial-frame visibility, whereas our quantitative conclusions rely entirely on the unweighted mean over the full pre-specified candidate set (including failed rollouts) locked before inspecting any outputs. Because WAMs make autonomous action decisions, we cannot manually prescribe their trajectories. We provide the exact sample sources to ensure reproducibility, followed by quantitative metrics below.

Episode Count and Selection: To fit page limits, Fig. 3 selects 1 of 4 camera-perturbation candidates, and Fig. 7 selects 1 case per suite (Spatial, Goal, Object) with clear dual-view object/gripper visibility after 2 warmup rounds. They were not cherry-picked by score (e.g., in Fig. 3, Fast-WAM's external metrics and DreamWAM's wrist metrics exceed Lute 5B). For unbiased statistics, we pre-specified the evaluation set before viewing any outputs—the original 7 cases (+32/+64 steps) plus 16 new stratified cases at +32 steps (4 Spatial, 6 Goal, 6 Object)—and computed the unweighted mean across all closed-loop rollouts without dropping failures (full per-case tables will be added to the Appendix):

Method (16-Episode Mean, +32 Steps)External Edge-F1 ↑\uparrowWrist Edge-F1 ↑\uparrowExternal MAE ↓\downarrowWrist MAE ↓\downarrow
Lute 1.4B0.67530.21690.08690.1926
Light-WAM0.02250.12700.09380.1305

Lute 1.4B achieves higher dual-view Edge-F1 and better external MAE, whereas Light-WAM yields lower wrist MAE. At the macro level, Lute accurately restores scene states; locally, consistent with our visual figures, Light-WAM produces blurry, smoothed predictions that numerically favor pixel-averaging metrics like wrist MAE.

W3 and Q3. The differences between Table 3(c) and Table 5 stem from the observation history length (3 frames vs. 1 frame), whether the visual encoder is included, and whether prediction-branch weights remain resident in GPU memory.**

All tests use the same GPU with Batch Size = 1, BF16, 10-step Euler integration, and cached text features (encoded once per episode):

Metric / ModelTable 3(c): Closed-Loop Policy (L=3L=3, w/ Visual Encoding)Table 5: Backbone-Only (L=1L=1, w/o Visual Enc./Dec.)
Lute 1.4B (A0.87B)Joint: 192.6 ms / 3.89 GiB (full weights); Action: 134.3 ms / 3.87 GiB (full weights; 124.6 ms / 3.85 GiB w/ feature reuse); Action: 124.6 ms / 1.66 GiB (0.87B action weights only)Joint: 137.79 ms / 2.97 GiB (full backbone weights)
Lute 5.0B (A3.05B)Joint: 361.0 ms / 10.55 GiB (full weights); Action: 250.6 ms / 5.75 GiB (3.05B action weights only)Joint: 254.83 ms / 9.50 GiB (full backbone weights)
ProtocolPeak allocated memory; 5 warmup / 20 timed runsPeak reserved memory; 3 warmup / 10 timed runs
  • Table 3(c) vs. Table 5 Latency: Table 3(c) encodes the current frame and processes a 3-frame window (L=3L=3); Table 5 takes a single pre-encoded latent frame (L=1L=1) without visual encoding/decoding.
  • Joint vs. Action Latency (~30% saving): With full weights resident, skipping the return prediction path (Blocks 19–30) reduces latency from 192.6 ms to 134.3 ms (124.6 ms w/ feature reuse) for 1.4B, and from 361.0 ms to 250.6 ms for 5.0B.
  • Joint vs. Action Memory (~50% saving): With full weights resident, 1.4B action-only memory is 3.87 GiB. Since the action stream is independent of the prediction branch (Eq. 8), loading only active action weights (0.87B / 3.05B) lowers memory to 1.66 GiB / 5.75 GiB. We will clarify both residency settings in the revised captions.

Thanks authors for the response. After considering your clarifications in light of my own assessment and the key points raised by the other reviewers, I have decided to keep my original score.

We appreciate your feedback and wish you a happy day every day.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Manuscript Revisions and New Real-World Experiments

We sincerely thank the AC(s) and all reviewers for their constructive feedback. We have uploaded the revised manuscript, with all updates highlighted in red.

  • R/W Ablations & Contribution (Sec. 4.3 Table 3(b), App. C Table 8): We added fixed-Haar comparisons as well as +32+32-step pixel discrepancy (MAE) and boundary agreement (Edge-F1) metrics across held-out episodes. These results confirm that while compact summaries suffice for control, R/W's learned orthogonal decomposition and retained details Δ\Delta are essential for restoring spatial structure in future prediction without the parameter or latency overhead of transposed convolutions.

  • Sample Selection & Benchmark Protocols (Sec. 4.1 Table 2, Sec. 4.2, App. B.1 Table 7, App. E.1): We clarified the RoboTwin co-trained setup and added DreamWAM in Table 2, documented the pre-locked visualization selection rules in Appendix E.1, and reported aggregate closed-loop prediction metrics across 16 uncurated episodes (including failed rollouts) in Section 4.2 and Appendix B.1 (Table 7).

  • Redrawn Figures (Sec. 1 Fig. 1, Sec. 3 Fig. 2): We redrew Figures 1 and 2 to explicitly label stage-wise tensor dimensions, illustrate the Fast stage and orthogonal R/W decomposition, and clearly separate the compact action path from the optional prediction synthesis path.

  • Runtime Profiling Boundaries (Sec. 4.3 Table 3(c), App. B Table 6): We clarified the exact timing boundaries, history lengths, encoder inclusions, and memory residency definitions across both runtime tables.

  • Real-World Robot Evaluation (Sec. 4.5 Table 4 & Fig. 7, App. D.4, App. E Fig. 9): We added physical manipulation experiments on a dual-camera AgileX PiPER platform across three sequential tasks and three distinct initial arm poses (50 trials per pose), benchmarking Lute against RDT-1B (w/o visual pred.), Fast-WAM, and Faster-WAM. We evaluated joint multi-task success, reverse-transfer generalization, closed-loop inference speed, multi-step visual prediction quality, and 3D trajectory tracking. Overall, Lute achieves the highest manipulation and transfer success, faster joint inference, sharper visual forecasts, and smoother physical trajectories.

Regarding limitations, manual resets cannot guarantee 100% identical scene restoration across trials (so we deliberately shuffled object placements between runs while ensuring all target objects remained visible in both cameras), and because real-world teleoperated data collection and closed-loop rollouts are highly time-consuming, evaluation is currently scoped to a moderate number of tasks and trials.

AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank the Area Chair and reviewers for their time and effort. Below is a brief summary of our contribution, the reviewer consensus, and how we addressed the raised concerns.

Core Contribution: Lute folds a shared multiscale DiT hierarchy around compact action readout via lightweight orthogonal R/W transforms. This lets predictive supervision improve control while skipping visual synthesis during action-only inference, addressing what visual information decision-making needs and how to organize it without slowing down execution.

Reviewer Consensus: All three reviewers rated Presentation as 3 ("Clear and easy to follow") and recognized both the separation of training capacity from inference cost and the control gains from matched-capacity predictive training.

Concerns and Resolutions:

  • Real-world validation (8ZAF, KZNf): We added physical AgileX PiPER experiments across three tasks, zero-shot transfer, and three camera poses (Sec. 4.5, Table 4, Fig. 7, App. Fig. 9), where Lute outperforms all baselines. Reviewer 8ZAF noted that these experiments address their / his / her original gap.
  • R/W necessity and efficiency (Nnew, 8ZAF, KZNf): We added prediction ablations (Tables 3(b), 8) and matched timing benchmarks. With only 12D12D parameters, R/W matches convolutional quality at much lower cost, giving a better speed-accuracy trade-off over multi-step diffusion inference.
  • Minor clarifications and presentation (all): We clarified reporting details, benchmark/profiling boundaries, and aggregate prediction metrics (Tables 2, 3(c), 6, 7), added multi-seed checks, and redrew vector Figures 1 and 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.