- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SLIM, which establishes a link between visual representations and action representations through the joint training of an FDM (Forward Dynamics Model) and an IDM (Inverse Dynamics Model). However, its prediction of future features relies on a given action, lacking a training objective for predicting future observations when the action is unknown. Consequently, it remains to be verified whether these representations can provide future information for action generation, and whether they can generalize to unseen tasks and environments.
- Architecture: Employs a compact dual-stream MoT framework to integrate visual and action data, utilizing a single backbone network for both representation learning and action generation. During deployment, the model generates actions directly, eliminating the need to predict future visual frames.
- Training: The process begins with joint learning of inverse and forward dynamics, inferring actions from current and future observations, and predicting future features based on current observations and actions, with feature supervision provided by an EMA encoder. This is followed by fine-tuning via flow matching, enabling the model to generate actions conditioned on current observations and instructions.
Critical strengths
- Representation learning integrating forward and inverse dynamics: By simultaneously learning to infer actions from observed changes and to predict future features based on actions, the resulting representations account for both actions and their consequences.
- Comprehensive experimental validation: The approach is evaluated across both simulated and real-world robotic platforms, under environmental perturbations, and in long-horizon tasks. Furthermore, the effectiveness of combining forward and inverse dynamics is confirmed through ablation studies.
Critical weaknesses
- Verification of representation stability remains limited: In FDM, the supervisory target evolves throughout the training process, posing a risk of co-degradation alongside the predicted representations. While EMA can mitigate this, it does not guarantee the prevention of representation collapse. Further validation is required to assess stability at larger scales and over longer training durations.
- The loss-weighting ratio between FDM and IDM significantly impacts model performance: In the paper’s experiments, the optimal overall performance was achieved with an IDM:FDM ratio of 0.125:1. However, it remains uncertain whether this ratio will remain effective across more datasets or larger model scales, and achieving strong results may necessitate manual hyperparameter tuning.
Questions for authors
- It is recommended to extend Stage-1 training and report effective rank (Eff. rank), top eigenvalue energy concentrations (Top-1, Top-5), and token cosine similarity at different training epochs to assess representation stability.
- It is recommended to train and evaluate SLIM on RoboTwin with different IDM:FDM loss-weight ratios, comparing the default ratio of 0.125:1 against alternatives to assess whether strong performance requires careful manual tuning.
2 comments
Summary: How do representations help the policy when actions are unknown, and can they generalize to unseen tasks and environments?
Thank you for highlighting the need to clarify the connection between action-conditioned prediction and policy generation. SLIM transfers the visual encoder, control backbone, and future slots learned through dynamics pretraining into an action flow-matching policy. At deployment, it generates actions directly from current observations, state, and task conditions, without ground-truth future actions or observations. We examine this design at four levels.
1. Does dynamics pretraining improve action generation?
The A/C comparison (Table 9) matches stage-wise data, architecture, token capacity, update budgets, and stage-boundary optimization, changing only the Stage-1 objective:
| Setting | Stage-1 data | Stage-1 objective | Stage-2 data and objective | LIBERO | LIBERO-Plus |
|---|---|---|---|---|---|
| A: Full SLIM (original) | Four suites + LIBERO-90 | IDM+FDM | Four suites, Policy-only | 97.50% | 77.45% |
| C: Objective control (new) | Four suites + LIBERO-90 | Policy-only | Four suites, Policy-only | 96.00% | 66.79% |
Policy-only denotes optimizing only the action flow-matching loss. Full SLIM improves LIBERO/LIBERO-Plus success by 1.50/10.66 percentage points, supporting the contribution of dynamics pretraining to downstream policy performance beyond additional data or token capacity.
2. Is future information still present without ground-truth actions?
We freeze the Stage-1 and Stage-2 models and train separate linear probes to predict future-feature changes through the same policy input path, providing only current observations, state, and language. Using a fixed target encoder and identical trajectory splits, Stage-1/Stage-2 slots reduce test MSE by 14.55%/7.13% relative to the static baseline that predicts the future equals the present (Appendix A.10, Table 14).
This supports the presence of limited decodable information about future changes in Stage-2 slots without ground-truth action inputs. Decodability is weaker than in Stage 1, consistent with Stage 2 no longer applying future-prediction supervision.
3. Do pretrained slots provide transfer value for the final policy?
Starting from the same Stage-1 checkpoint, we reinitialize only the slots before Stage 2, retaining all other weights and the full training setup (Appendix A.10, Table 13). LIBERO/LIBERO-Plus success decreases from 97.05%/78.02% for the matched reproduction baseline to 96.00%/75.22%.
The 1.05/2.79 percentage-point decreases at unchanged capacity support the transfer value of pretrained slot initialization, but this benefit cannot be attributed solely to the decodable future information they contain.
4. What generalization conclusions do these results support?
CALVIN ABC→D evaluates performance in an unseen environment, while LIBERO-Plus evaluates zero-shot robustness under perturbations. They support environmental generalization and robustness within their respective evaluation settings.
W2 / Q2: Does the 0.125:1 ratio depend on tuning, and can it generalize to RoboTwin?
Thank you for raising the question of hyperparameter transferability. Figure 6(b) compares multiple IDM:FDM loss ratios on LIBERO-Plus and CALVIN. 0.125:1 achieves the best overall performance among the tested settings, providing the empirical basis for selecting it as the default ratio.
We plan to conduct additional comparisons on RoboTwin using 0.125:1, 0.5:1, and 1:1. However, training and evaluation on RoboTwin require substantial time. If verified results that support further assessment become available before the rebuttal deadline, we will provide an update. Our current conclusions remain limited to the validated LIBERO-Plus and CALVIN settings.
W1 / Q1: Can EMA prevent representation collapse, and is training stable?
Thank you for raising this question. To complement the endpoint statistics in Appendix A.3, we further compare the complete Stage-1 training trajectories of EMA and No-EMA under matched configurations: both models are trained for 3 epochs and 17,736 optimizer updates, with representation diagnostics recorded every 100 steps.
The training curves in Figure 7 (Appendix A.3) show that, without EMA, effective rank decreases while feature-energy concentration and token similarity gradually increase. EMA maintains greater representation diversity. The following values, also reported in Table 5, are averaged over the final 20 diagnostic records of each run:
No-EMA achieves lower late-stage prediction error, but this is accompanied by a continuing decline in representation diversity. Therefore, lower prediction loss alone cannot establish representation quality or training stability. The target representations also evolve separately in the two runs, so prediction errors are not computed against the same fixed target.
Furthermore, starting from the same initialization, we double the EMA training budget to 6 epochs / 35,472 updates and extend the cosine learning-rate decay schedule accordingly. The extended run maintains relatively high effective rank and low energy concentration, with no pronounced representation collapse observed. However, effective rank decreases during the sixth epoch and token similarity rises late in training, so the results should not be described as continuous improvement across all metrics.
The original downstream ablations provide complementary evidence: EMA, No-EMA, and fixed DINOv2 targets achieve LIBERO-Plus success rates of 77.45%, 66.82%, and 71.81%, respectively.
Together, the results support EMA mitigating representation degradation within the tested budget and maintaining representation diversity at twice that budget.