acceptodds
Under review as a conference paper at ICLR 2027

Reconstructing is Not Acting: Action-Centric Latent Dynamics Modeling

Abstract

Latent action models (LAMs) learn action representations from unlabeled videos by inferring latent actions from visual transitions and reconstructing future states. However, we identify a fundamental : lower reconstruction error does not necessarily yield better latent dynamics or downstream performance. We attribute this mismatch to two underconstrained aspects of reconstruction-based latent dynamics modeling: (i) the inverse dynamics model (IDM) is not explicitly encouraged to distinguish action-related transitions from nuisance appearance, and (ii) the forward dynamics model (FDM) can underutilize the inferred latent action by exploiting predictive shortcuts from the current state. To address both limitations, we propose , a lightweight action-centric framework that strengthens both action extraction and action utilization. Specifically, its Action Query IDM (AQ-IDM) employs learnable action queries and gated aggregation to selectively extract rich action-related transition cues without strong information bottlenecks. And its Action Token FDM (AT-FDM) projects latent actions into action tokens that progressively interact with evolving state representations, enabling continuous state-aware action conditioning. ACT-LAM further streamlines feature processing to concentrate model capacity on latent dynamics modeling. Extensive experiments on several robotic datasets and the VP benchmark demonstrate stronger latent action consistency, forward dynamics, and downstream visual planning performance with fewer trainable parameters and lower computational overhead. In particular, ACT-LAM surpasses the previous state of the art by **7.6%** on the aggregated VP success rate.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Latent Action Model, Action Learning, Visual Planning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4tZ6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the mismatch between reconstruction quality and action learning in latent action models. The authors propose ACT-LAM, which combines an Action Query IDM for extracting action-related information and an Action Token FDM for better action conditioning. Experiments show improvements in linear probing, action utilization, downstream visual planning, and computational efficiency.

Critical strengths

The main strength of the paper is that it addresses an important issue in latent action learning. The observation that better reconstruction does not necessarily lead to better action representations is meaningful and interesting.

Moreover, the proposed framework is simple and efficient. It reduces trainable parameters while achieving better downstream planning performance.

Finally, the experiments cover multiple datasets and evaluation settings, including linear probing, action utilization, visual prediction, and robotic planning. The component-level comparisons also provide useful information for understanding the proposed design.

Critical weaknesses

The main weakness of this paper is that higher action utilization does not necessarily indicate better forward dynamics. Proposed utilization metric measures the increase in prediction error when latent actions are replaced with constant actions. This measures sensitivity to actions, but not prediction quality. For example, in Table 3, ACT-LAM performs substantially worse than DiLA at longer rollout horizons. This raises questions about whether AT-FDM actually improves forward dynamics.

Also, the action-centric property of AQ-IDM is not fully demonstrated. The linear probing results are positive, but the improvements over DiLA in Table 2 are relatively small. The attention visualizations and singular-value analysis also do not directly show that AQ-IDM separates action-related information from nuisance appearance.

Questions for authors

Q1. What would be the possible reason for the higher action utilization when the long-horizon prediction error is substantially worse?

Q2. Can you clarify how AQ-IDM distinguishes actual action information from changes in background or appearance?

Q3. Are the improvements from combining AQ-IDM and AT-FDM consistent across multiple random seeds in the downstream planning experiments?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 4tZ6 (1/1)

Q1: Utilization and Rollout Error

Thank you for this question. Action utilization and absolute prediction error measure different aspects of the FDM. The former measures the relative benefit of using the original latent actions instead of a constant action, whereas the latter measures prediction accuracy.

As shown in Table A1, AT-FDM’s high utilization is accompanied by a larger error under the constant-action control. A possible explanation is that AT-FDM relies more on transition-specific latent actions. Thus, replacing them with a constant action causes greater degradation that accumulates during long-horizon rollout. Importantly, it does not mean that AT-FDM always has a higher error with the original actions. In our matched FDM comparison on SSV2, AT-FDM achieves the highest utilization while its original-action MSE at horizon 15 is lower than AdaLN-FDM.

Q2: Action and Appearance

Thanks for pointing this out. AQ-IDM is designed to extract action information from changes between consecutive states. It first computes temporal differences between compact states, aiming to emphasize transition-related changes while reducing the influence of persistent appearance. Learnable queries then attend to these spatial transition features, and a learned gate weights the resulting per-query latent vectors to produce the final latent action.

To examine its sensitivity to background changes, we replaced the white background in Push-T transitions with a blue-color or textured background, while preserving the non-background pixels. We kept AQ-IDM and the existing action probe frozen. As shown in Table C1, action prediction MSE remains comparable to the original background result under both background changes. It increases from 0.199 to 0.218 with the textured background. This suggests that the latent actions retain action predictive information under the background interventions.

Table C1. Normalized action-probing MSE under background interventions.

\hline \text{Original background}\downarrow & \text{Blue background}\downarrow & \text{Textured background}\downarrow \\\\ \hline 0.199 & 0.196 & 0.218 \\\\ \hline \end{array}$$ > **Q3: Multiple-seed Results** Please refer to **Table A2**.
Reviewer_4tZ6OpenReview ↗

Thanks for the response.

Thank you for the additional explanations and experimental results. However, I believe my main concerns about the relationship between action utilization and forward dynamics performance remain. Therefore, I have decided to maintain my original score.

AuthorsOpenReview ↗

Thanks for your response.

Thank you for your continued consideration. Our central observation is that lower rollout error does not necessarily imply better latent action representations or stronger action utilization. ACT-LAM addresses action extraction and action utilization through a lightweight architecture, achieving better downstream planning with fewer parameters and lower computational cost.

We evaluate AT-FDM through two complementary forms of evidence. Action utilization measures the relative benefit of the original latent actions over a constant-action baseline, supporting stronger action utilization. The planning ablations assess its practical contribution.

We acknowledge that these improvements coexist with mixed prediction-error results and a remaining limitation in long-horizon rollouts. The supported conclusion is that AT-FDM strengthens action conditioning and contributes to planning performance, while ACT-LAM improves the aggregate VP² score from 41.44 to 49.04 with substantially fewer parameters and less computation.

Reviewer_mZUeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses two potential limitations of modern latent action models (LAMs): the inverse dynamics model (IDM) may encode background distractors or noise rather than action-relevant information, and the forward dynamics model (FDM) may underutilize the latent action inferred by the IDM when reconstructing the next state, relying instead on predictive information in the current state. The authors argue that these effects may explain why lower reconstruction loss does not necessarily translate into better action representations for control and task completion. They propose architectural changes to address these problems, assess their contributions through ablations, and report action-probing comparisons and VP² benchmark evaluations against AdaWorld and DiLA.

Critical strengths
  1. The architectural description in the method section is well written and supported by a clear graphical overview. The design changes align with the paper’s motivation.
  2. The proposed method achieves higher aggregate success rates than DiLA and AdaWorld on the evaluated VP² tasks.
  3. The evaluation combines action probing, latent representation analyses, and downstream visual planning, verifying method’s effectiveness beyond reconstruction quality alone.
Critical weaknesses
  1. The authors claim identifying the reconstruction–action mismatch in existing LAMs as a key contribution. However, closely related limitations have already been studied extensively by https://arxiv.org/abs/2502.00379v5, https://arxiv.org/abs/2506.15691v3, https://arxiv.org/abs/2608.19613v1#S5.SS5, and https://arxiv.org/abs/2605.20223v1. The paper should more clearly distinguish its specific empirical findings from these established concerns.
  2. Some method details are insufficiently explained in the main text. Although the appendix specifies the dynamics adapter’s architecture, the fusion operation lacks a clear computational description. The notation and computation of (LsymL_{\mathrm{sym}}) and (LvarL_{\mathrm{var}}) also require clarification, and the reconstruction-loss indexing (ftf_t in one place and ft+1f_{t+1} in another) should be made consistent or explicitly explained.
  3. Slower singular-value decay is a limited proxy for action quality: it indicates less concentration of latent variation into a few dimensions, but those dimensions could still encode backgrounds, camera motion, or noise. The accompanying action-probing results do not establish that the additional dimensions capture action-relevant variation.
  4. Although the paper includes component ablations, an (LvarL_{\mathrm{var}}) ablation, and comparisons of FDM conditioning mechanisms, the contributions of individual AQ-IDM design choices remain insufficiently isolated. In particular, the downstream benefits of learnable queries, transition differencing, and gated aggregation are unclear. Attention visualizations provide limited evidence of its contribution to task success. Moreover, the VP2VP^2 component ablation uses only one random seed.
  5. Despite promising results on VP2VP^2, the evaluation offers limited breadth. General claims about action quality and nuisance appearance would benefit from broader manipulation tasks, such as those in RoboTwin 2.0, and controlled appearance perturbations, such as those in LIBERO-Plus.
Questions for authors
  1. Could you provide additional ablations of AQ-IDM design choices and, beyond the existing attention visualizations, analyze what the learnable queries capture? Do they consistently specialize into distinct action-related slots across examples?
  2. Could you provide evaluation results for ACT-LAM, DiLA, and AdaWorld using the same planner and matched adaptation protocols on additional tasks or controlled appearance perturbations, either within VP2VP^2 or using selected tasks from RoboTwin 2.0 or LIBERO-Plus?
  3. How does ACT-LAM compare to CoMo under your pretraining and VP2VP^2 settings?
5 comments
AuthorsOpenReview ↗

Response to Reviewer mZUe (1/3)

Q1: Distinction from Prior Works

Thank you for pointing this out. Three of the four papers are already cited in our submission, and we appreciate these studies as complementary analyses of latent-action learning.

Specifically, we organize them into three lines. (i) LAOM [B1] and Zhang et al. [B2] primarily study the abstraction of latent actions. They analyze how distractors, observation noise, or action supervision affect the action-relevant information encoded in the latent. (ii) Bu et al. [B3] conduct a broad empirical study over 41 LAM design choices and examine the relationship between proxy metrics and downstream manipulation performance. (iii) Lee et al. [B4] focus on exogenous leakage, showing how exogenous information can enter the latent and induce a shortcut in the FDM.

In contrast, ACT-LAM explicitly formulates the reconstruction–action mismatch as two coupled but separable problems: (i) extracting action-relevant transition information in the IDM, and (ii) utilizing the resulting latent action in the FDM. We develop AQ-IDM and AT-FDM to address these two stages, respectively. We evaluate them through action probing, latent representation analyses, and downstream visual planning. Thus, ACT-LAM provides an explicit IDM–FDM diagnosis and an architectural treatment of both action extraction and action utilization in a practical LAM.

  • [B1] Nikulin et al. Latent Action Learning Requires Supervision in the Presence of Distractors. ICML 2025.
  • [B2] Zhang et al. What Do Latent Action Models Actually Learn? NeurIPS 2025.
  • [B3] Bu et al. What Matters for Latent Actions in Robot Learning. arXiv:2608.19613, 2026.
  • [B4] Lee et al. Why Latent Actions Fail, and How to Prevent It. arXiv:2605.20223, 2026.

Q2: Clarification of Method Details

We apologize for the confusion. We clarify the information as follows.

Fusion operation: Given the predicted dynamics state s^t+1\hat{s}_{t+1}, a lightweight structure decoder first maps it to spatial patch tokens. We then combine these tokens with the initial-frame DINO feature f0f_0 using spatial self-attention. After that, an output projection produces the reconstructed feature f^t+1\hat{f}_{t+1}. We use the initial-frame feature as a visual anchor that provides appearance details complementary to the predicted compact state. Unlike DiLA [B5], our fusion operator only utilizes this initial-frame condition and does not include a content encoder or content memory.

Lsym\mathcal{L}_{\text{sym}} & Lvar\mathcal{L}_{\text{var}}: Following DiLA [B5], we use a cosine-similarity objective for Lsym\mathcal{L}_{\mathrm{sym}} (Eq.(10)) to enforce that inverse temporal transitions produce opposite latent action vectors. In addition, we introduce Lvar\mathcal{L}_{\text{var}} to preserve transition-specific variation in latent actions by penalizing dimensions whose empirical standard deviation falls below σ=0.05\sigma=0.05.

Reconstruction-loss indexing: We apologize for this notation inconsistency. It is a typographical error. In our submission, ftf_t denotes the current input feature, while ft+1f_{t+1} denotes the future feature to be reconstructed. We will correct Eq.(10) in the revision.

  • [B5] Zhang et al. DiLA: Disentangled Latent Action World Models. ICML 2026.
AuthorsOpenReview ↗

Response to Reviewer mZUe (2/3)

Q3: Singular-Value Analysis

Thank you for the insightful question. Following [B6], we use the singular-value spectrum as a diagnostic of latent-action diversity and effective dimensionality. We agree that the spectrum alone cannot establish action relevance, since a slower decay could also reflect nuisance variation. Therefore, Figure 3 should not be interpreted in isolation. Combined with Table 2, where ACT-LAM achieves the lowest linear-probing MSE on both benchmarks. The results support the conclusion that ACT-LAM learns a more diverse latent-action representation that is also better aligned with robot actions.

To better understand whether the additional spectral directions are action-relevant, we conducted a linear probing experiment on the Block Pushing transitions (see Table B1). We applied PCA to the latent actions from training trajectories and projected them onto the top 16 or 32 principal components. We then trained linear probes on these projected representations and evaluated them on held-out trajectories.

As shown in Table B1, ACT-LAM achieves lower action-prediction MSE than the other models using either the top 16 or top 32 principal components. Its MSE further decreases when components 17–32 are included, indicating that these additional directions contain action-predictive information. This finding is consistent with the slower spectral decay in Figure 3 and the better linear-probing results in Table 2.

Table B1. Action-probing MSE using different numbers of principal components. Values are reported as mean ± population std over three trajectory splits.

\hline \text{Model} & \text{Top-16 MSE}\downarrow & \text{Top-32 MSE}\downarrow \\\\ \hline \text{MFormer} & 4.697 \pm 0.113 & 5.202 \pm 0.205 \\\\ \text{DiLA} & 3.878 \pm 0.242 & 5.044 \pm 0.261 \\\\ \text{ACT-LAM} & \mathbf{2.870 \pm 0.186} & \mathbf{2.657 \pm 0.106} \\\\ \hline \end{array}$$ - [B6] Garrido et al. RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rank. ICML 2023. > **Q4: Ablations and Random Seeds** Thanks for your comments. We agree that our current ablation evaluates AQ-IDM as a complete module and does not separately quantify the contributions of learnable queries, transition differencing, and gated aggregation. These ablations cannot be reliably obtained by modifying existing checkpoints. Changing any of these choices alters the resulting latent-action distribution. Thus, each variant requires retraining the LAM, repeating action-to-latent adaptation, and rerunning downstream evaluation. Given this computational cost and the limited rebuttal period, we were unable to complete the fine-grained ablation. To make the computational constraint concrete, we measured the time cost of our actual pipeline. LAM pretraining takes approximately 1–2 days on four RTX A6000 GPUs, followed by 0.5–1 h for action-to-latent adaptation and 4–5 h for joint fine-tuning. $VP^2$ evaluation is also expensive. The runs require 22–26 h for a RoboSuite evaluation, 5–7 h for a RoboDesk upright-block evaluation, and 22–28 h for a RoboDesk open-slide evaluation. Thus, even a 4-seed RoboSuite comparison requires approximately **89–103 GPU hours** of evaluation. To address the concern about statistical reliability, we additionally expanded the $VP^2$ component ablation from one seed to four seeds and report the corresponding results in **Table A2**.
AuthorsOpenReview ↗

Response to Reviewer mZUe (3/3)

Q5: Broader Manipulation Tasks

Thank you for this suggestion. We conducted additional experiments on LIBERO-Plus, extending the MPC evaluation pipeline of VP2VP^2 to broader manipulation tasks and controlled appearance perturbations. We evaluated ACT-LAM, DiLA, and AdaWorld under identical conditions, but none achieved a successful rollout in the evaluated trajectories. These results suggest that the evaluated tasks remain challenging for the current MPC pipeline. We therefore regard stronger policy learning with latent-action supervision as a promising direction for future work.

Q6: Learnable-Query Analysis

To better understand the learnable queries, we analyzed their attention maps and per-query latent vectors on RT-1. The queries share a substantial portion of their spatial attention, with an average 80% overlap among their top-10% attention patches. This suggests that they consistently aggregate information from a common set of salient transition regions. Moreover, across the 8 queries, the association scores with robot translation and rotation are on average 6.7× and 2.3× higher than those obtained after shuffling the action labels, respectively . These observations are consistent with AQ-IDM’s design of aggregating action-relevant transition information.

Q7: ACT-LAM vs CoMo

Thank you for the question. CoMo[B7] is highly relevant to our work. CoMo is a continuous latent motion model that uses a Motion Q-Former with learnable queries. We believe a controlled comparison with CoMo's Motion Q-Former is the most informative way to answer this question, and ACT-LAM outperforms it under our pretraining and VP² settings.

The official CoMo implementation uses a different visual backbone (MAE-ViT versus DINOv2), prediction space (RGB frames versus compact states), and decoder. To isolate the IDM contribution, MFormer preserves CoMo’s core design while sharing our experimental protocols. MFormer and AQ-IDM only therefore differ only in the IDM.

Linear Probing (Table 2): the full ACT-LAM model achieves lower MSE than MFormer on Push-T (0.007 versus 0.016) and Block Pushing (0.031 versus 0.048).

Planning (Table A2, four seeds): replacing MFormer with AQ-IDM improves mean success rates from 64.00 to 70.50 on RoboSuite, 41.67 to 48.33 on Blue Button, and 57.50 to 60.83 on Green Button.

Efficiency (Table 15): AQ-IDM requires 41% fewer training FLOPs (64.3 versus 109.3 GFLOPs) and achieves 1.7× throughput (118.1 versus 68.9 frames/s).

These results demonstrate AQ-IDM’s effectiveness and efficiency relative to the controlled CoMo-style IDM baseline. An end-to-end evaluation of the official CoMo model remains to be completed. We will train the official CoMo implementation and report the results in the final version.

  • [B7] Yang et al. CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning. CVPR 2026.
Reviewer_PLbXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a reconstruction–action mismatch in latent action models: better future-frame reconstruction does not necessarily imply more useful action representations or better downstream control. It proposes ACT-LAM, combining an Action Query IDM that aggregates spatial transition features through learnable queries and gating, and an Action Token FDM that progressively updates action tokens alongside state representations. The architecture also reduces feature-processing overhead.

Validation includes pretraining on four video datasets, linear action probing on Push-T and Block Pushing, action-conditioning comparisons, autoregressive prediction, and VP² visual planning. ACT-LAM improves probing results and raises the simulator-normalized aggregate planning score from DiLA’s 41.44 to 49.04, a 7.60 percentage-point gain, while reducing trainable parameters from 123M to approximately 55M. Prediction results are mixed: short-horizon errors improve, but long-horizon reconstruction remains weaker.

Critical strengths

Evaluation connects representation learning to actual control. Linear probing and planning provide evidence beyond reconstruction metrics. The substantial gains on the blue- and green-button tasks make the reconstruction–control distinction practically consequential. The method improves performance while reducing training cost. The parameter reduction is accompanied by measured improvements in FLOPs, throughput, and memory under a specified hardware protocol, strengthening the practical contribution.

Critical weaknesses

The utilization metric does not independently establish better dynamics. It measures relative improvement over replacing actions with their training-set mean. A higher score can reflect worse constant-action predictions rather than lower error with correct actions. Absolute errors are needed to interpret this result, especially because Table 9 shows that adding AT-FDM worsens longer-horizon prediction relative to the baseline. Attribution of downstream gains remains incomplete. ACT-LAM changes feature processing, both dynamics modules, and the loss through the variance term. Although several ablations are provided, the planning component ablation covers only RoboSuite with one seed (Appendix D.3), and the variance-loss ablation is limited to probing. This leaves uncertainty about which changes explain the aggregate planning gain.

Questions for authors

Clarification: Can you report both absolute correct-action and constant-action prediction errors for the matched FDM comparisons in Table 14? How should the utilization gains be reconciled with the longer-horizon degradation in Table 9? Assessment-relevant validation: Under identical losses and training schedules, do AQ-IDM and AT-FDM each improve aggregate VP² performance across multiple seeds? How much of the gain depends on the variance loss?

4 comments
AuthorsOpenReview ↗

Response to Reviewer PLbX (1/2)

Q1: Action Utilization

Thank you for your constructive comments! Action utilization measures the relative benefit of conditioning an FDM on the original latent actions compared with a constant latent action. It complements absolute prediction error by quantifying the contribution of action conditioning. To clarify their relationship, Table A1 reports rollout state MSEs under both conditions for the matched FDM comparisons.

At horizon 15, AT-FDM achieves the highest utilization on both SSV2 and RT-1, indicating that its predictions benefit more from the latent actions. On SSV2, AT-FDM also achieves the lowest original-action MSE. On RT-1, AdaLN-FDM achieves a lower MSE, while AT-FDM remains competitive. The higher constant-action error of AT-FDM arises from its greater reliance on latent actions, whose replacement with a constant action can cause prediction errors to accumulate during long-horizon rollout. These results support the effectiveness of AT-FDM’s action conditioning.

Table A1: Absolute errors under original and constant actions at horizon 15

\hline \text{Dataset} & \text{FDM} & \text{Original-action MSE}\downarrow & \text{Constant-action MSE}\downarrow & \text{Utilization}\uparrow \\\\ \hline \text{SSV2} & \text{Add-FDM} & 5.299 \pm 0.168 & 5.939 \pm 0.104 & 10.8 \pm 1.3\\% \\\\ \text{SSV2} & \text{Concat-FDM} & 5.707 \pm 0.533 & 5.861 \pm 0.062 & 2.7 \pm 8.2\\% \\\\ \text{SSV2} & \text{XAttn-FDM} & 5.391 \pm 0.092 & 5.937 \pm 0.093 & 9.2 \pm 2.7\\% \\\\ \text{SSV2} & \text{AdaLN-FDM} & 4.988 \pm 0.077 & \mathbf{5.816 \pm 0.007} & 14.2 \pm 1.3\\% \\\\ \text{SSV2} & \text{AT-FDM} & \mathbf{4.975 \pm 0.091} & 8.537 \pm 0.127 & \mathbf{41.7 \pm 0.8\\%} \\\\ \hline \text{RT-1} & \text{Add-FDM} & 4.617 \pm 0.111 & 5.820 \pm 0.120 & 20.7 \pm 0.5\\% \\\\ \text{RT-1} & \text{Concat-FDM} & 4.879 \pm 0.262 & 6.162 \pm 0.252 & 20.9 \pm 1.1\\% \\\\ \text{RT-1} & \text{XAttn-FDM} & 4.701 \pm 0.147 & 5.968 \pm 0.082 & 21.2 \pm 2.0\\% \\\\ \text{RT-1} & \text{AdaLN-FDM} & \mathbf{4.477 \pm 0.038} & \mathbf{5.750 \pm 0.072} & 22.1 \pm 0.4\\% \\\\ \text{RT-1} & \text{AT-FDM} & 4.546 \pm 0.164 & 6.219 \pm 0.168 & \mathbf{26.9 \pm 0.8\\%} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Response to Reviewer PLbX (2/2)

Q2: More ablations

Thank you for pointing this out. As shown in Table A2, we conducted additional ablation studies to examine whether the improvements are consistent across random seeds. Specifically, we extended the VP² component ablation to assess the individual contributions of AQ-IDM and AT-FDM, as well as the effect of the variance loss Lvar\mathcal{L}_{\text{var}}.

All the designs contribute to downstream performance. Adding AQ-IDM to the AT-FDM-only variant improves mean success rates by 1.00%, 64.16%, and 13.33%, respectively. Adding AT-FDM to the AQ-IDM-only variant yields improvements of 0.75% 42.50%, and 4.17%.

The variance loss also contributes substantially. Removing it reduces success rates from 71.25 to 67.25, 90.83 to 74.17, and 65.00 to 37.50, respectively. These results quantify the contributions of our designs.

Table A2: More ablation results (4 seeds)

\hline \text{Method} & \text{RoboSuite} & \text{Blue Button} & \text{Green Button} \\\\ \hline \text{PlainLAM} & 49.00 \pm 2.92 & 4.17 \pm 1.44 & 12.50 \pm 5.46 \\\\ \text{PlainLAM w/o STT} & 50.00 \pm 4.08 & 21.67 \pm 3.73 & 31.67 \pm 3.73 \\\\ \text{MFormer} & 64.00 \pm 2.45 & 41.67 \pm 7.64 & 57.50 \pm 6.40 \\\\ \text{AT-FDM only} & 70.25 \pm 1.79 & 26.67 \pm 4.08 & 51.67 \pm 1.67 \\\\ \text{AQ-IDM only} & 70.50 \pm 2.29 & 48.33 \pm 8.98 & 60.83 \pm 10.9 \\\\ \text{ACT-LAM w/o }L_{var} & 67.25 \pm 3.27 & 74.17 \pm 3.63 & 37.50 \pm 6.82 \\\\ \text{ACT-LAM} & \mathbf{71.25 \pm 3.63} & \mathbf{90.83 \pm 2.76} & \mathbf{65.00 \pm 7.64} \\\\ \hline \end{array}$$
Reviewer_PLbXOpenReview ↗

Post-rebuttal assessment and rating

Thank you for the absolute-error comparisons and the additional four-seed planning ablations. The new experiments strengthen the evidence for the contributions of AQ-IDM, AT-FDM, and the variance loss, and substantially address my concern about the limited component-level planning evaluation.

My concern about interpreting action utilization remains. On SSV2, AT-FDM and AdaLN-FDM have nearly identical original-action errors, while AT-FDM has a substantially higher constant-action error. On RT-1, AT-FDM does not achieve the lowest original-action error. These results support stronger dependence on actions, but do not independently establish more accurate forward dynamics. Please distinguish these conclusions explicitly and explain their relationship to the longer-horizon degradation in Table 9. The scope of the new planning ablations and their matched training settings should also be stated clearly.

I am maintaining my rating of 3 (Weak acceptance). The practical planning gains and additional ablations support acceptance, while the remaining uncertainty about dynamics quality and the breadth of component-level validation limits the strength of my recommendation.

AuthorsOpenReview ↗

Thanks for the response.

Thanks for pointing out this distinction. We recognize that our claim about “improved forward dynamics” was too broad and will revise it to distinguish action dependence, predictive accuracy, and planning performance.

Table A1 supports stronger action dependence, while the longer-horizon degradation in Table 9 (Table 11 in the revised manuscript) identifies a remaining limitation in long-horizon prediction. These findings are compatible because stronger dependence on latent actions does not guarantee lower rollout error. We clarify that utilization is an action-conditioning diagnostic rather than the evidence of predictive accuracy.

The planning ablations provide complementary evidence of AT-FDM’s practical contribution. AT-FDM improves mean success across all three evaluated tasks. These results support its usefulness for planning despite mixed prediction-error results.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their insightful and constructive reviews of our manuscript. Reviewers PLbX and 4tZ6 recognized that our paper is well-written and easy to follow, and that our method is simple and efficient. All the reviewers (PLbX, mZUe,4tZ6) also acknowledged that our evaluation covers multiple datasets and settings.

Based on all the reviews, here we provide a general response to the concerns raised by multiple reviewers. The individual responses are commented on below each review.

  • Regarding questions about experiments, we have addressed the concerns as follows:
    • We add multi-seed ablations (For Reviewer PLbX, mZUe,4tZ6)
    • We run experiments on a new benchmark (LIBERO-Plus). (For Reviewer mZUe)
  • Regarding questions about clarification for details, we have addressed the concern as follows:
    • We clarify our action utilization metric (For Reviewer PLbX, 4tZ6)
    • We discuss related works with our manuscript (For Reviewer mZUe)
    • We clarify the singular-value analysis (For Reviewer mZUe)
    • We clarify the action-centric property of AQ-IDM (For Reviewer 4tZ6)

We have also uploaded a revised version of our submission according to the suggestions provided by reviewers, with major changes highlighted in purple. We look forward to further feedback and discussion. Please feel free to let us know if further details/explanations are helpful.

Best, Authors.

Dear Reviewers,

Thank you very much for your efforts and constructive comments. Please note that the rolling discussion will end soon. If you have any further concerns or suggestions, kindly let us know. We are more than happy to address them and revise our paper accordingly.

Warm regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.