- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the mismatch between reconstruction quality and action learning in latent action models. The authors propose ACT-LAM, which combines an Action Query IDM for extracting action-related information and an Action Token FDM for better action conditioning. Experiments show improvements in linear probing, action utilization, downstream visual planning, and computational efficiency.
Critical strengths
The main strength of the paper is that it addresses an important issue in latent action learning. The observation that better reconstruction does not necessarily lead to better action representations is meaningful and interesting.
Moreover, the proposed framework is simple and efficient. It reduces trainable parameters while achieving better downstream planning performance.
Finally, the experiments cover multiple datasets and evaluation settings, including linear probing, action utilization, visual prediction, and robotic planning. The component-level comparisons also provide useful information for understanding the proposed design.
Critical weaknesses
The main weakness of this paper is that higher action utilization does not necessarily indicate better forward dynamics. Proposed utilization metric measures the increase in prediction error when latent actions are replaced with constant actions. This measures sensitivity to actions, but not prediction quality. For example, in Table 3, ACT-LAM performs substantially worse than DiLA at longer rollout horizons. This raises questions about whether AT-FDM actually improves forward dynamics.
Also, the action-centric property of AQ-IDM is not fully demonstrated. The linear probing results are positive, but the improvements over DiLA in Table 2 are relatively small. The attention visualizations and singular-value analysis also do not directly show that AQ-IDM separates action-related information from nuisance appearance.
Questions for authors
Q1. What would be the possible reason for the higher action utilization when the long-horizon prediction error is substantially worse?
Q2. Can you clarify how AQ-IDM distinguishes actual action information from changes in background or appearance?
Q3. Are the improvements from combining AQ-IDM and AT-FDM consistent across multiple random seeds in the downstream planning experiments?
3 comments
Thanks for the response.
Thank you for the additional explanations and experimental results. However, I believe my main concerns about the relationship between action utilization and forward dynamics performance remain. Therefore, I have decided to maintain my original score.
Thanks for your response.
Thank you for your continued consideration. Our central observation is that lower rollout error does not necessarily imply better latent action representations or stronger action utilization. ACT-LAM addresses action extraction and action utilization through a lightweight architecture, achieving better downstream planning with fewer parameters and lower computational cost.
We evaluate AT-FDM through two complementary forms of evidence. Action utilization measures the relative benefit of the original latent actions over a constant-action baseline, supporting stronger action utilization. The planning ablations assess its practical contribution.
We acknowledge that these improvements coexist with mixed prediction-error results and a remaining limitation in long-horizon rollouts. The supported conclusion is that AT-FDM strengthens action conditioning and contributes to planning performance, while ACT-LAM improves the aggregate VP² score from 41.44 to 49.04 with substantially fewer parameters and less computation.
Response to Reviewer 4tZ6 (1/1)
Thank you for this question. Action utilization and absolute prediction error measure different aspects of the FDM. The former measures the relative benefit of using the original latent actions instead of a constant action, whereas the latter measures prediction accuracy.
As shown in Table A1, AT-FDM’s high utilization is accompanied by a larger error under the constant-action control. A possible explanation is that AT-FDM relies more on transition-specific latent actions. Thus, replacing them with a constant action causes greater degradation that accumulates during long-horizon rollout. Importantly, it does not mean that AT-FDM always has a higher error with the original actions. In our matched FDM comparison on SSV2, AT-FDM achieves the highest utilization while its original-action MSE at horizon 15 is lower than AdaLN-FDM.
Thanks for pointing this out. AQ-IDM is designed to extract action information from changes between consecutive states. It first computes temporal differences between compact states, aiming to emphasize transition-related changes while reducing the influence of persistent appearance. Learnable queries then attend to these spatial transition features, and a learned gate weights the resulting per-query latent vectors to produce the final latent action.
To examine its sensitivity to background changes, we replaced the white background in Push-T transitions with a blue-color or textured background, while preserving the non-background pixels. We kept AQ-IDM and the existing action probe frozen. As shown in Table C1, action prediction MSE remains comparable to the original background result under both background changes. It increases from 0.199 to 0.218 with the textured background. This suggests that the latent actions retain action predictive information under the background interventions.
Table C1. Normalized action-probing MSE under background interventions.
\hline \text{Original background}\downarrow & \text{Blue background}\downarrow & \text{Textured background}\downarrow \\\\ \hline 0.199 & 0.196 & 0.218 \\\\ \hline \end{array}$$ > **Q3: Multiple-seed Results** Please refer to **Table A2**.