acceptodds
Under review as a conference paper at ICLR 2027

SLIM-0.5B: Learning Action-Grounded Predictive Latents for Robot Manipulation

Abstract

Compact robot manipulation policies need representations that capture how actions change observations. Large multimodal backbones and pixel-level world models provide semantic and predictive structure, but can devote substantial computation to details beyond those needed for control. We propose **SLIM** (**S**elf-supervised **L**atent **I**nteraction **M**odel), a compact policy that learns action-grounded predictive representations within the same backbone used for action generation. A Mixture-of-Transformers (MoT) jointly models observation latents and continuous action tokens. In the first stage, SLIM learns these representations through self-supervised masked trajectory prediction. It reconstructs actions from current and future observations and predicts future visual representations from current observations and actions, using an exponential moving average (EMA) target encoder to provide the prediction targets. The second stage builds on the visual representations learned in the first stage and trains the shared backbone for action generation with flow matching, requiring neither future observations nor explicit future generation at deployment. With 0.47B trainable policy parameters, SLIM achieves competitive performance on LIBERO and CALVIN and reaches 77.45% success on zero-shot LIBERO-Plus. It also outperforms and Fast-WAM in average task progress across five real-world tasks and four evaluation settings. In our H100 inference benchmark, SLIM achieves and speedups per inference call over and Fast-WAM, respectively, while reducing per-call FLOPs by over 76% relative to both baselines. These results support bidirectional latent prediction as an effective training approach for compact manipulation policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot manipulation, self-supervised learning, latent dynamics, robust robot control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fdibOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SLIM, which establishes a link between visual representations and action representations through the joint training of an FDM (Forward Dynamics Model) and an IDM (Inverse Dynamics Model). However, its prediction of future features relies on a given action, lacking a training objective for predicting future observations when the action is unknown. Consequently, it remains to be verified whether these representations can provide future information for action generation, and whether they can generalize to unseen tasks and environments.

  1. Architecture: Employs a compact dual-stream MoT framework to integrate visual and action data, utilizing a single backbone network for both representation learning and action generation. During deployment, the model generates actions directly, eliminating the need to predict future visual frames.
  2. Training: The process begins with joint learning of inverse and forward dynamics, inferring actions from current and future observations, and predicting future features based on current observations and actions, with feature supervision provided by an EMA encoder. This is followed by fine-tuning via flow matching, enabling the model to generate actions conditioned on current observations and instructions.
Critical strengths
  1. Representation learning integrating forward and inverse dynamics: By simultaneously learning to infer actions from observed changes and to predict future features based on actions, the resulting representations account for both actions and their consequences.
  2. Comprehensive experimental validation: The approach is evaluated across both simulated and real-world robotic platforms, under environmental perturbations, and in long-horizon tasks. Furthermore, the effectiveness of combining forward and inverse dynamics is confirmed through ablation studies.
Critical weaknesses
  1. Verification of representation stability remains limited: In FDM, the supervisory target evolves throughout the training process, posing a risk of co-degradation alongside the predicted representations. While EMA can mitigate this, it does not guarantee the prevention of representation collapse. Further validation is required to assess stability at larger scales and over longer training durations.
  2. The loss-weighting ratio between FDM and IDM significantly impacts model performance: In the paper’s experiments, the optimal overall performance was achieved with an IDM:FDM ratio of 0.125:1. However, it remains uncertain whether this ratio will remain effective across more datasets or larger model scales, and achieving strong results may necessitate manual hyperparameter tuning.
Questions for authors
  1. It is recommended to extend Stage-1 training and report effective rank (Eff. rank), top eigenvalue energy concentrations (Top-1, Top-5), and token cosine similarity at different training epochs to assess representation stability.
  2. It is recommended to train and evaluate SLIM on RoboTwin with different IDM:FDM loss-weight ratios, comparing the default ratio of 0.125:1 against alternatives to assess whether strong performance requires careful manual tuning.
2 comments

W1 / Q1: Can EMA prevent representation collapse, and is training stable?

Thank you for raising this question. To complement the endpoint statistics in Appendix A.3, we further compare the complete Stage-1 training trajectories of EMA and No-EMA under matched configurations: both models are trained for 3 epochs and 17,736 optimizer updates, with representation diagnostics recorded every 100 steps.

The training curves in Figure 7 (Appendix A.3) show that, without EMA, effective rank decreases while feature-energy concentration and token similarity gradually increase. EMA maintains greater representation diversity. The following values, also reported in Table 5, are averaged over the final 20 diagnostic records of each run:

MetricEMA: 3 epochsNo-EMA: 3 epochsEMA: 6 epochs
Effective rank60.5014.3571.53
Top-1 energy fraction0.0980.3850.102
Top-5 energy fraction0.3600.6630.335
Token cosine similarity0.0710.4240.117
Normalized L10.2500.1590.236

No-EMA achieves lower late-stage prediction error, but this is accompanied by a continuing decline in representation diversity. Therefore, lower prediction loss alone cannot establish representation quality or training stability. The target representations also evolve separately in the two runs, so prediction errors are not computed against the same fixed target.

Furthermore, starting from the same initialization, we double the EMA training budget to 6 epochs / 35,472 updates and extend the cosine learning-rate decay schedule accordingly. The extended run maintains relatively high effective rank and low energy concentration, with no pronounced representation collapse observed. However, effective rank decreases during the sixth epoch and token similarity rises late in training, so the results should not be described as continuous improvement across all metrics.

The original downstream ablations provide complementary evidence: EMA, No-EMA, and fixed DINOv2 targets achieve LIBERO-Plus success rates of 77.45%, 66.82%, and 71.81%, respectively.

Together, the results support EMA mitigating representation degradation within the tested budget and maintaining representation diversity at twice that budget.


Summary: How do representations help the policy when actions are unknown, and can they generalize to unseen tasks and environments?

Thank you for highlighting the need to clarify the connection between action-conditioned prediction and policy generation. SLIM transfers the visual encoder, control backbone, and future slots learned through dynamics pretraining into an action flow-matching policy. At deployment, it generates actions directly from current observations, state, and task conditions, without ground-truth future actions or observations. We examine this design at four levels.

1. Does dynamics pretraining improve action generation?

The A/C comparison (Table 9) matches stage-wise data, architecture, token capacity, update budgets, and stage-boundary optimization, changing only the Stage-1 objective:

SettingStage-1 dataStage-1 objectiveStage-2 data and objectiveLIBEROLIBERO-Plus
A: Full SLIM (original)Four suites + LIBERO-90IDM+FDMFour suites, Policy-only97.50%77.45%
C: Objective control (new)Four suites + LIBERO-90Policy-onlyFour suites, Policy-only96.00%66.79%

Policy-only denotes optimizing only the action flow-matching loss. Full SLIM improves LIBERO/LIBERO-Plus success by 1.50/10.66 percentage points, supporting the contribution of dynamics pretraining to downstream policy performance beyond additional data or token capacity.

2. Is future information still present without ground-truth actions?

We freeze the Stage-1 and Stage-2 models and train separate linear probes to predict future-feature changes through the same policy input path, providing only current observations, state, and language. Using a fixed target encoder and identical trajectory splits, Stage-1/Stage-2 slots reduce test MSE by 14.55%/7.13% relative to the static baseline that predicts the future equals the present (Appendix A.10, Table 14).

This supports the presence of limited decodable information about future changes in Stage-2 slots without ground-truth action inputs. Decodability is weaker than in Stage 1, consistent with Stage 2 no longer applying future-prediction supervision.

3. Do pretrained slots provide transfer value for the final policy?

Starting from the same Stage-1 checkpoint, we reinitialize only the slots before Stage 2, retaining all other weights and the full training setup (Appendix A.10, Table 13). LIBERO/LIBERO-Plus success decreases from 97.05%/78.02% for the matched reproduction baseline to 96.00%/75.22%.

The 1.05/2.79 percentage-point decreases at unchanged capacity support the transfer value of pretrained slot initialization, but this benefit cannot be attributed solely to the decodable future information they contain.

4. What generalization conclusions do these results support?

CALVIN ABC→D evaluates performance in an unseen environment, while LIBERO-Plus evaluates zero-shot robustness under perturbations. They support environmental generalization and robustness within their respective evaluation settings.


W2 / Q2: Does the 0.125:1 ratio depend on tuning, and can it generalize to RoboTwin?

Thank you for raising the question of hyperparameter transferability. Figure 6(b) compares multiple IDM:FDM loss ratios on LIBERO-Plus and CALVIN. 0.125:1 achieves the best overall performance among the tested settings, providing the empirical basis for selecting it as the default ratio.

We plan to conduct additional comparisons on RoboTwin using 0.125:1, 0.5:1, and 1:1. However, training and evaluation on RoboTwin require substantial time. If verified results that support further assessment become available before the rebuttal deadline, we will provide an update. Our current conclusions remain limited to the validated LIBERO-Plus and CALVIN settings.

Reviewer_mzRbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SLIM learns a compact manipulation policy through inverse-action reconstruction and action-conditioned future-latent prediction within a shared MoT backbone, using an EMA visual encoder for prediction targets. A second stage trains flow-matching control without future observations at deployment. Experiments cover LIBERO, LIBERO-Plus, CALVIN, and real-world manipulation.

Critical strengths
  1. The training and deployment design is coherent. SLIM combines inverse-action reconstruction and future-latent prediction within a compact shared backbone, then transfers it to direct control without requiring explicit future generation at inference.
  2. The experiments include useful controls. Figure 6 examines pretraining and IDM/FDM weighting, while Appendices A.2–A.3 compare EMA, fixed-DINOv2, and online stop-gradient targets. The reported 77.45% LIBERO-Plus success and 4.556 CALVIN average length support the practical value of the training recipe.
Critical weaknesses
  1. The action-grounded mechanism is not sufficiently isolated. Joint IDM/FDM training improves over either objective alone, but this does not distinguish action-dependent dynamics learning from additional predictive supervision or multitask regularization. Whether future prediction meaningfully depends on actions rather than visual continuity remains untested, limiting attribution of the bidirectional-training gain.
  2. The contribution of transferred future slots remains unclear. Given precedents for predictive policies without test-time future rollout, SLIM’s contribution lies in its specific training and transfer design. However, Stage 2 uses only the action loss, and no control isolates whether pretrained slots retain useful predictive structure or mainly provide additional latent capacity.
  3. The LIBERO comparison leaves data exposure ambiguous. Stage 1 includes LIBERO-90 and the target suites, whereas Stage 2 uses only the target suites. The Stage-2-only baseline matches optimization steps, but its access to LIBERO-90 is unspecified. The LIBERO-Plus gain may therefore partly reflect additional data coverage. The same-data CALVIN comparison provides complementary evidence.
  4. The real-world advantage needs stronger uncertainty and reproducibility reporting. All policies use the same 750 demonstrations, but each task-condition pair has only 10 trials. Progress averages lack uncertainty estimates, and baseline fine-tuning budgets and hyperparameters are insufficiently detailed. These omissions make the robustness of the reported advantage difficult to assess.
Questions for authors
  1. How does future prediction change with correct, shuffled, or removed actions while evaluation examples and target representations remain fixed? Does action-conditioned pretraining outperform an otherwise matched action-free predictor in downstream control?
  2. Does transferring pretrained future slots improve over reinitializing the same slots before Stage 2? Is predictive information retained in their representations after policy training?
  3. Does the step-matched Stage-2-only LIBERO baseline also use LIBERO-90, and does the pretraining advantage persist with matched data access?
  4. Can you provide uncertainty estimates for real-world progress scores, results across training seeds, and the baseline fine-tuning configurations?
4 comments

W1 / Q1: Does FDM use action information, rather than merely benefiting from visual continuity or additional supervision?

We examine this through a frozen-model diagnostic and a retrained downstream control.

Frozen-model diagnostic (Appendix A.9, Table 12). We fix the reproduced Stage-1 checkpoint, current observations, and future targets, changing only the FDM action input. We evaluate 5,376 samples from 168 validation trajectories across 84 tasks under correct, shuffled, and zeroed actions. Shuffled actions come from another trajectory of the same task, averaged over three shuffle seeds; zeroing is applied after normalization.

Action inputNormalized L1 ↓MSE ↓Cosine distance ↓
Correct0.254643.037490.07128
Shuffled0.349833.142780.13866
Zeroed0.333533.135260.12663

Normalized L1 measures mean absolute error after independently applying non-affine LayerNorm to predictions and targets; MSE uses the original features.

Relative to correct actions, shuffling and zeroing increase normalized L1 by 0.09520 (37.39%) and 0.07889 (30.98%), respectively. Task-cluster bootstrap 95% confidence intervals for these paired increases are [0.09131, 0.09913] and [0.07560, 0.08229]. For shuffling, we average the three seeds per sample before bootstrapping. Both interventions increase mean normalized L1 in all 84 tasks, supporting FDM's reliance on actions matched to the observations.

Downstream control (Appendix A.9, Table 11). We train an action-free FDM variant with identical Stage-1 data, architecture, update budget, and EMA supervision. Only normalized action values in the FDM branch are zeroed; action tokens, positional encodings, and attention paths remain unchanged. IDM and Stage-2 policy training still use real actions.

SettingSourceLIBEROLIBERO-Plus
Full SLIM: action-conditioned FDMOriginal97.50%77.45%
Action-free FDMNew96.05%71.00%
Decrease after removing action values—1.45 pp6.45 pp

Future-prediction supervision remains present, yet downstream performance decreases. Thus, the gains cannot be explained solely by additional future supervision. Together, these experiments support that matched actions improve future-feature prediction, and action-conditioned pretraining benefits downstream policies. They neither exclude contributions from visual continuity nor establish complete causal dynamics. The downstream comparison uses one training seed and does not establish significance across training seeds.


W2 / Q2: Do pretrained slots aid transfer, and retain predictive information after Stage 2?

We separately examine the transfer value of pretrained slots and the decodability of future information after policy training.

Transfer value (Appendix A.10, Table 13). Starting from the same Stage-1 checkpoint, we reinitialize only the future slots before Stage 2, retaining all other pretrained weights, token capacity, and the full Stage-2 training setup.

SettingLIBEROLIBERO-Plus
Retain pretrained slots: matched reproduction97.05%78.02%
Reset slots before Stage 296.00%75.22%
Decrease after resetting1.05 pp2.79 pp

With capacity unchanged, resetting reduces performance, supporting the transfer value of pretrained slot initialization beyond merely adding tokens.

Predictive information after policy training (Appendix A.10, Table 14). We freeze the corresponding Stage-1 and Stage-2 models and train separate linear probes to predict future-feature changes, Δz=zt+h−zt\Delta z = z_{t+h}-z_t, using the same input path, fixed target encoder, and trajectory splits. Inputs contain only current observations, state, and language—no ground-truth actions or future observations.

Representation / baselineTest MSE ↓Reduction from static baseline
Static baseline: future equals current0.068552—
Stage-1 slots0.05857814.55%
Stage-2 slots0.0636637.13%

Future changes remain linearly decodable from Stage-2 slots, although less accurately than from Stage-1 slots. This is consistent with Stage 2 optimizing action generation without future-prediction supervision. Decodability alone does not establish that the policy causally uses this information.

W3 / Q3: Does pretraining help under matched data?

The original CALVIN comparison (Figure 6(a)) uses identical ABC training data: two-stage training improves average sequence length from 4.391 to 4.556. For LIBERO, we separate objective and data contributions:

Stage-1 dataStage-1 objectiveStage-2 objectiveLIBEROLIBERO-PlusSource
Four suitesPolicy-onlyPolicy-only96.20%68.30%Original
Four suitesIDM+FDMPolicy-only96.40%74.93%New (B)
Four suites + LIBERO-90Policy-onlyPolicy-only96.00%66.79%New (C)
Four suites + LIBERO-90IDM+FDMPolicy-only97.50%77.45%Original (A)

Policy-only denotes action flow matching. Stage 2 always uses four suites. The original Policy-only baseline trains continuously without a stage transition. Two-stage variants use 17,736 + 81,200 updates and global batch 128.

Matched data (Appendix A.9, Table 9). A/C match architecture, stage-wise data, update budgets, and stage-boundary optimization, changing only the Stage-1 objective. A improves LIBERO/LIBERO-Plus by 1.50/10.66 percentage points, supporting dynamics pretraining under matched data. Without LIBERO-90, B still improves LIBERO-Plus by 6.63 points over the continuous Policy-only baseline, although this comparison also includes the optimization restart.

Additional data (Appendix A.9, Table 10). A/B retain the dynamics objectives and update budgets. Adding LIBERO-90 improves LIBERO/LIBERO-Plus by 1.10/2.52 points. Thus, broader task coverage provides additional gains but cannot alone explain SLIM's improvements.


W4 / Q4: Real-world uncertainty, training seeds, and configurations

All methods use 750 demonstrations and 5 tasks × 4 conditions × 10 trials = 200 trials per method. Mean progress and 95% confidence intervals are:

ConditionSLIMπ0.5\pi_{0.5}Fast-WAM
Overall67.75 [63.25, 72.25]56.75 [51.75, 61.75]40.00 [35.25, 44.50]
Standard86.00 [79.00, 93.00]72.00 [63.00, 80.00]71.00 [60.00, 81.00]
Distractors63.00 [54.00, 72.00]54.00 [43.00, 65.00]39.00 [28.00, 50.00]
Lighting73.00 [64.00, 82.00]47.00 [36.00, 58.00]48.00 [37.00, 59.00]
Background49.00 [38.00, 59.00]54.00 [44.00, 64.00]2.00 [0.00, 5.00]

Progress uses a 0–100 scale. We use 100,000 stratified bootstrap resamples, resampling trials within each task–condition pair and weighting tasks and conditions equally. Intervals reflect fixed-checkpoint evaluation uncertainty on the tested tasks and conditions, not training randomness or generalization to new settings. Rounded overall means match the original 67.8, 56.8, and 40.0; SLIM leads overall, but not in every condition.

ConfigurationSLIMπ0.5\pi_{0.5}Fast-WAM
InitializationDINOv2; Stage 2 loads Stage-1 weightsπ0.5\pi_{0.5}-DROID, JAXWan video weights; no robot-policy checkpoint
EpochsStage 1: 3; Stage 2: 202323
Seed424242
OptimizerAdamWAdamWAdamW
Learning rateStage 2: policy 10−410^{-4}; vision 10−510^{-5}2.5×10−52.5\times10^{-5}10−410^{-4}
ScheduleStage 2: 5,000-update warmup + cosine1,000-update warmup + cosine5% warmup + cosine
Global batch12832128
Trainable modulesVision and policy; cached language featuresFull fine-tuningDiT and proprioceptive encoder; others frozen
Prediction / execution horizon16 / 1016 / 1032 / 10
CheckpointFinalFinalFinal

Budgets match total epochs, including SLIM Stage 1, not optimizer updates or compute.

Each method currently reports one training seed. At five minutes per trial, evaluation requires approximately 16.7 hours per checkpoint, excluding resets and maintenance. ACT also reports single-seed comparisons on two real-world tasks [1]. Nevertheless, cost and prior practice do not resolve training variance: trial-level intervals cannot replace multiple training seeds, and we make no claim of cross-seed statistical superiority.

Additional training-seed experiments are underway, with robot evaluation planned. We will supplement results if completed before the rebuttal deadline.

[1] Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS, 2023. doi:10.15607/RSS.2023.XIX.016.

Reviewer_mzRbOpenReview ↗

Official Comment by Reviewer mzRb

Thank you for the additional experiments, which have largely addressed my previous concerns.

My remaining question concerns the motivation for jointly learning FDM and IDM in Stage 1. While the ablations suggest that combining them improves downstream performance, could the authors explain what complementary information the two objectives encourage the model to learn, and how this benefits subsequent policy training? In particular, what does IDM contribute beyond action-conditioned forward prediction, and why should their combination produce representations that are more useful for action generation in Stage 2?

Thank you for acknowledging the additional experiments. We jointly use IDM and FDM to encourage representations to retain both the visual consequences of actions and the action information that explains those changes. The objectives use related training data but require the model to solve different conditional prediction problems.

FDM learns “what visual changes will occur given an action.” It predicts future latent representations from the current observation and action, encouraging the model to capture relationships between actions and environmental changes. However, actions are known inputs to FDM; reducing future-feature prediction error does not directly require the model to generate actions from visual information.

IDM complements this by learning “what actions explain the observed changes.” It reconstructs actions from current and future observations, directly constraining visual representations through action-reconstruction error. This encourages them to retain control-relevant changes, beyond information useful for predicting future appearance. Moreover, IDM uses action flow matching to directly pretrain the action-generation branch subsequently used for policy learning. Its role is therefore not merely to add a visual prediction constraint, but also to train the mapping from visual information to action outputs.

Together, the objectives provide complementary initialization for Stage 2. FDM constrains representations to capture the visual consequences of actions, while IDM requires these representations to support action reconstruction. The shared visual encoder and control backbone receive both forms of supervision. Stage 2 then transfers these components to a policy that generates actions using only current observations, state, and task conditions, without requiring ground-truth future observations. We expect this initialization to provide useful visual information and action-generation capabilities for subsequent policy learning.

The existing experiments support the following conclusions:

Evidence and locationComparisonSupported conclusion
Figure 6(b)Joint IDM/FDM versus IDM-only and FDM-onlyAcross the tested LIBERO-Plus and CALVIN settings, the joint objective achieves better overall performance, directly supporting the downstream value of combining the objectives
Appendix A.9: Frozen-model action interventionsFix observations and targets, changing only FDM inputs to shuffled or zeroed actionsMean prediction error increases in all 84 tasks, showing that FDM uses matched action information
Appendix A.9: Action-free FDMRetain IDM and future supervision, removing only FDM’s action-value conditioningLIBERO-Plus decreases from 77.45% to 71.00%, showing that explicit action conditioning in FDM provides downstream value even with IDM retained
Appendix A.9: A/C controlMatch stage-wise data, architecture, update budgets, and optimization procedures, replacing Stage-1 IDM/FDM with Policy-onlyLIBERO/LIBERO-Plus improve from 96.00%/66.79% to 97.50%/77.45%, supporting the overall transfer benefit of dynamics pretraining for subsequent action generation

In particular, the joint-objective versus FDM-only comparison in Figure 6(b) most directly addresses “what IDM contributes beyond forward prediction”: adding action reconstruction further improves downstream performance. The other controls show that the forward branch uses action conditioning and that the overall gains cannot be explained solely by additional data or training budgets.

Our interpretation is therefore that FDM encourages shared representations to capture the visual consequences of actions, while IDM requires them to support action reconstruction and pretrains the action-generation branch; Stage 2 transfers these jointly trained components to a policy using only current information.

Reviewer_W2UaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SLIM, a compact vision-language-action (VLA) model with 0.47B trainable parameters, utilizing a DINOv2 encoder and a Mixture-of-Transformers (MoT) trunk, where observation and action streams interact through joint attention and language is incorporated via cross-attention. The training process consists of two stages. In stage 1, the model learns to reconstruct actions from current and future latents (inverse dynamics), and predict future latents from current latents and actions supervised with an EMA target (forward dynamics). In stage 2, a flow matching policy is trained to estimate actions conditioned on learned future slot embeddings, without an explicit future generation. SLIM achieves competitive performance against larger baselines while offering faster inference.

Critical strengths

Competitive performance with a compact model size

  • With 0.47B trainable parameters and no pre-training on external robot datasets, SLIM achieves competitive performance on LIBERO and CALVIN ABC->D, compared to much larger VLA/WAM baselines. It reaches 77.45% on zero-shot LIBERO-Plus, which evaluates robustness under various perturbation types. The results suggest that robustness under distribution shift can be achieved by a compact policy, which is usually attributed to large VLM backbones or video world models.

An inference-efficient architecture from training-only dynamics prediction

  • The inverse- and forward-dynamics objectives are applied only in stage 1 to shape the representation of the backbone and the learned future slots enable the policy to predict actions without generating future rollouts or latent decoding. This yields efficient inference, addressing the inference cost which is a practical bottleneck that both large VLAs and WAMs share. The FLOPs and memory figures are more convincing than the wall-clock latencies, which were measured in PyTorch eager mode.
Critical weaknesses

Limited novelty and missing prior work

  • The core ingredients closely follow prior work, several of which are neither cited nor compared as well. The two-stream trunk is essentially the MMDiT[1] architecture and nearly identical to DUST[2] which already applied joint attention between an action stream and a visual stream. FLARE[3] places learnable future tokens into a diffusion policy and supervises them with EMA target of future observation. Masked Trajectory Models already learn forward and inverse dynamics through masking in a single transformer. The overall contribution therefore appears to be a recombination of existing components of prior work.

Insufficient evidence that the proposed objectives drive the gains

  • On LIBERO and LIBERO-Plus, stage 2 is trained on the four target suites, but the stage 1 data includes LIBERO-90, regarded as pretraining, and the SLIM is marked as having no extra pre-training. Baselines including OpenVLA-OFT are usually trained only on the four suites. Although LIBERO-90 is in-domain and far smaller than other VLA's large-scale pre-training datasets, the stage 1 ablation (Fig. 6a) changes both the training objectives and the data at the same time, making it difficult to disentangle their effects. In addition, the contribution of learned future slots should be distinguished from that of generic extra tokens. Since the paper's central claim is that action-grounded latent prediction causes the improved performance, the current experiments cannot rule out the extra data or the additional tokens.

[1] Esser, Patrick, et al. "Scaling rectified flow transformers for high-resolution image synthesis." ICML (2024).

[2] Won, John, et al. "Dual-stream diffusion for world-model augmented vision-language-action model." arXiv preprint arXiv:2510.27607 (2025).

[3] Zheng, Ruijie, et al. "Flare: Robot learning with implicit world modeling." arXiv preprint arXiv:2505.15659 (2025).

Questions for authors

Q1. [Clarification] What is the essential difference between SLIM and FLARE/DUST/MTM? Which components of SLIM are novel relative to these works and which experiments isolate their contributions?

Q2. [Clarification] Is the full 8-step action chunk executed before replanning, i.e., is the action horizon equal to the execution horizon? At what control frequency does the policy run? Since compute is reported per call while chunk lengths differ across models, does SLIM's efficiency advantage hold per executed action, and under what execution settings were the baselines run?

Q3. [Clarification + Additional Experiment] To separate the effect of the stage 1 objectives from that of the additional LIBERO-90 data, could you report i) SLIM with stage 1 trained only on the four target suites and ii) a stage 2-only baseline trained on both LIBERO-90 and four target suites, under the same total number of training steps?

Q4. [Clarification + Additional Experiment] The learned future slots are kept in stage 2, but no experiment separates their effect from that of generic extra tokens. How would you show that the gains come from the predictive information carried by these slots rather than from adding extra tokens, through a controlled comparison under the same data and training steps?

9 comments

W1 / Q1: Differences from Related Work and Novelty

Thank you for highlighting the need for clearer comparisons. SLIM jointly learns visual representations through complementary dynamics tasks, then transfers them to an action-generation policy. Stage 1 combines action reconstruction from current/future observations (IDM) with action-conditioned future-latent prediction (FDM), jointly training the visual encoder and control backbone against adaptive EMA targets. Stage 2 transfers the encoder, backbone, and future slots to action-only policy training. The dual-stream backbone follows MMDiT; our contribution concerns this learning and transfer mechanism, rather than attention, future tokens, or EMA individually.

Key differences (Section 2 of the revised manuscript):

  • vs. DUST: SLIM learns an adaptive visual latent space through dynamics pretraining, whereas DUST jointly generates actions and future features in a frozen visual representation space. DUST uses dual-modality flow matching; SLIM combines flow-matching action reconstruction with direct EMA-target future-feature regression, then transfers the learned encoder, backbone, and slots to a policy with action-only sampling.

  • vs. FLARE: SLIM uses explicit inverse and forward dynamics to pretrain transferable policy representations, whereas FLARE combines action generation with future-feature alignment during policy training. FLARE first learns action-aware embeddings and uses EMA targets for downstream alignment. SLIM jointly trains inverse dynamics (Zt,Zt+H→At)(Z_t, Z_{t+H} \to A_t) and clean-action-conditioned forward dynamics (Zt,At→Zˉt+H)(Z_t, A_t \to \bar{Z}_{t+H}), then transfers the pretrained components to policy training without future supervision.

  • vs. MTM: SLIM combines adaptive visual latent prediction with flow-matching action reconstruction in a two-stage visual-control framework. MTM unifies dynamics prediction and control through masked trajectory reconstruction. SLIM jointly learns the visual encoder and control backbone using complementary IDM/FDM objectives, with EMA future-feature targets, then transfers the backbone and slots to action-generation training.

Comparison with DUST/FLARE. Their official project pages still indicate that code is forthcoming, preventing evaluation of their official implementations on LIBERO-Plus. We therefore use the LIBERO/CALVIN results reported in DUST (DUST Appendix A.1, Tables 9–10):

MethodTrainable policy parametersPretrained initializationLIBERO ↑CALVIN ABC→D ↑
SLIM0.472BDINOv2-B/14; frozen T5-small97.50%4.556
DUSTNot reportedFrozen Eagle VLM; full-backbone reference size ≈2.1B96.20%3.91
FLARE†Not reportedFrozen Eagle VLM; full-backbone reference size ≈2.1B96.20%3.75

† FLARE is the DUST authors’ adapted reproduction (DUST Appendix A.3). SLIM achieves higher reported scores without a pretrained VLM, exceeding DUST/FLARE† on CALVIN by 0.646/0.806.

Comparison with MTM. Since official MTM provides no LIBERO visual–language recipe, we adapted its code using two RGB views, robot state, frozen T5-small features, and masked observation/action reconstruction. We compare against SLIM B with the same target-task coverage:

MethodTraining-task coverageLIBERO ↑Source
MTM, visual–language adaptationFour target suites60.55%New
SLIM BFour target suites96.40%New
Full SLIM AStage 1: four suites + LIBERO-90; Stage 2: four suites97.50%Original

MTM is evaluated over 40 tasks × 50 official initial states = 2,000 rollouts. SLIM B performs better under the same task coverage; architecture and optimization settings are not identical.

Controlled ablations further support SLIM’s design: replacing Stage-1 IDM/FDM with action loss, removing FDM action values, and fixing DINOv2 targets reduce LIBERO-Plus success by 10.66, 6.45, and 5.64 percentage points, respectively (Appendix A.9, Tables 9 and 11).

W2 / Q3: P.T., LIBERO-90, and Same-Data Controls

1. P.T. labeling and four-suite comparisons

Stage 1 uses LIBERO-90, whereas baselines such as OpenVLA-OFT typically train downstream on only four target suites.

P.T. denotes policy/world-model pretraining on robot data outside the target benchmark, excluding generic vision, language, or video initialization. LIBERO-90 is benchmark-internal, hence the no-external-robot-pretraining label, but adds data beyond the four target suites.

B uses only four suites in both stages (Appendix A.9). Original baselines appear in Table 1:

MethodLIBEROLIBERO-PlusSource
OpenVLA-OFT97.10%69.60%Original
π₀94.10%69.30%Original
Fast-WAM97.60%50.00%Original
SLIM B: four suites only96.40%74.93%New
SLIM A: with LIBERO-90 in Stage 197.50%77.45%Original

Without LIBERO-90 or external robot-data pretraining, SLIM still exceeds OpenVLA-OFT/Fast-WAM on LIBERO-Plus by 5.33/24.93 points. Removing LIBERO-90 reduces SLIM’s LIBERO/LIBERO-Plus scores by 1.10/2.52 points, but its robustness advantage persists. Thus, the gains cannot be explained solely by additional LIBERO-90 data.

2. Separating objectives from data coverage in Figure 6a

The original LIBERO ablation varies objectives and data coverage. We separate these effects (Appendix A.9, Tables 9–10):

Stage-1 dataStage-1 objectiveStage-2 objectiveLIBEROLIBERO-PlusSource
Four suitesPolicy-onlyPolicy-only96.20%68.30%Original
Four suitesIDM+FDMPolicy-only96.40%74.93%New, B
Four suites + LIBERO-90Policy-onlyPolicy-only96.00%66.79%New, C
Four suites + LIBERO-90IDM+FDMPolicy-only97.50%77.45%Original, A
  • A/C: objective contribution. With matched architecture, stage-wise data, and optimization procedure, IDM+FDM improves LIBERO/LIBERO-Plus by 1.50/10.66 points.
  • B versus four-suite Policy-only: gains without LIBERO-90. LIBERO-Plus improves by 6.63 points. This comparison includes the stage transition and optimizer reset.
  • A/B: additional-data contribution. With matched dynamics objectives and update budgets, LIBERO-90 adds 1.10/2.52 points.

Q2: Execution Horizon, Control Frequency, and Efficiency

SLIM predicts and executes eight actions before replanning in LIBERO. At 20 Hz, each chunk spans 0.4 simulated seconds, giving 2.5 replanning calls per simulated second. Wall-clock frequency also includes inference and execution overhead.

Table 8 (Appendix A.7) reports inference cost per policy call. For latency TT, compute FF, and execution horizon HexecH_{\mathrm{exec}}, amortized costs are:

\qquad F_{\mathrm{action}}=\frac{F}{H_{\mathrm{exec}}}.$$ | Model | Prediction / execution horizon | Sampling steps | ms/call | GFLOPs/chunk | ms/executed action | GFLOPs/executed action | |---|---:|---:|---:|---:|---:|---:| | SLIM | 8 / 8 | 4 | **60.61** | **490.73** | **7.58** | **61.34** | | π₀.₅ | 10 / 10 | 10 | 193.14 | 4714.59 | 19.31 | 471.46 | | Fast-WAM | 32 / 10 | 10 | 360.64 | 2090.07 | 36.06 | 209.01 | Measured on **one H100 80GB, batch 1, BF16, PyTorch eager**. SLIM/Fast-WAM use cached language features. Environment stepping, communication, and robot execution are excluded. π₀.₅ predicts 10 steps in this timing benchmark and 16 in real-world experiments. SLIM has lower amortized costs in this setup. **Amortized cost is not individual-action response latency:** SLIM’s full chunk still takes 60.61 ms. ## W2 / Q4: Predictively Trained Future Slots or Additional Token Capacity? Future slots carry action-conditioned future-latent predictions in Stage 1 and are retained in Stage 2. **Action-grounded predictive training and learned representations improve performance beyond additional tokens alone.** **A/C** retains identical stage-wise data, update budgets, backbone, and slot capacity, replacing only Stage-1 IDM/FDM with Policy-only training (Appendix A.9, Table 9). Full SLIM achieves **97.50%/77.45%** on LIBERO/LIBERO-Plus, versus **96.00%/66.79%** for C. This supports dynamics training beyond data or capacity alone, but does not isolate the slots’ contribution. Starting from **the same Stage-1 checkpoint**, we reinitialize only the future slots, retain all other pretrained weights, and apply identical Stage-2 training and evaluation (Appendix A.10, Table 13). Reset slots remain trainable: | Setting | LIBERO | LIBERO-Plus | |---|---:|---:| | Retain pretrained slots, matched reproduction | **97.05%** | **78.02%** | | Reset slots only | 96.00% | 75.22% | | Decrease | 1.05 pp | 2.79 pp | **With unchanged token capacity, retaining pretrained slots improves final performance, supporting the transfer value of their predictively learned initialization.**

We thank the reviewer again for the review. Since the discussion period is short, we would be grateful to hear whether our responses address the reviewer’s concerns, and we would be glad to discuss any remaining question.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for recognizing SLIM’s competitive performance with a compact model, its inference efficiency, and the additional evidence supporting the robustness benefits of dynamics pretraining. Your feedback helped us clarify the contribution and strengthen its experimental support.

The main additions during rebuttal are:

  • Contribution and controls: Clarified comparisons with prior work, matched-data objective comparisons, training without LIBERO-90, and slot reinitialization at unchanged token capacity.
  • Predictive learning and robustness: Action-input interventions, future-information probes, perturbation-specific results, and paired action-prediction diagnostics.
  • Training and evaluation: Extended EMA training, a joint Policy+IDM/FDM comparison, per-executed-action efficiency estimates, and trial-level confidence intervals for real-world evaluation.

Together, these controlled experiments support our core contribution: dynamics pretraining and pretrained-slot transfer improve policy performance beyond additional data or token capacity alone. We will clearly state the remaining limitations in the final manuscript.

We appreciate your careful consideration and the opportunity to improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.