- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TCLA, a framework for learning temporally contextualized latent action representations for robot manipulation. It combines action-chunk reconstruction, a task-progress representation derived from the DCT of executed action history, and a latent flow-matching objective with an EMA target encoder. After pretraining, downstream policies predict latent actions, which are decoded using task and historical context. The method is evaluated with several policy architectures on RoboTwin, multi-task learning, and five real-world manipulation tasks. The reported results show improvements in task success, downstream training convergence, and motion smoothness.
Critical strengths
- Studying how action representations and temporal context affect robot policy learning is a worthwhile direction with potential benefits for manipulation policies.
- Experiments span different policy families and both simulated and physical environments, providing encouraging evidence of the framework’s applicability.
Critical weaknesses
- TCLA changes both the action representation and the decoding process by introducing historical action context. The current experiments do not clearly distinguish the benefits of temporal representation learning from those of additional history-conditioned decoding. Although Table 5 provides objective ablations, it does not fully isolate these factors, limiting confidence in the claimed contribution of the proposed latent action space.
- The reported convergence improvements mainly consider downstream policy training while excluding the additional TCLA pretraining stage. The pretraining cost, data requirements, and computational overhead are not sufficiently quantified. Therefore, faster downstream convergence does not necessarily demonstrate better end-to-end training efficiency.
- Table 3 compares separately trained single-task ACT policies with a jointly trained multi-task ACT+TCLA policy. This changes both the training setting and the action representation, making it difficult to attribute the improvement specifically to TCLA. A multi-task ACT baseline using raw actions under the same training conditions is necessary.
- The paper reports reduced jerk, but does not sufficiently control for execution duration, trajectory characteristics, or differences in metric computation. Moreover, the main results lack clear uncertainty estimates across independent training runs. These limitations weaken the evidence for the claimed improvements in motion quality and robustness.
Questions for authors
- Which inputs, modules, and objectives are retained in each Table 5 variant?
- When the progress objective is removed, does the history encoder and phase conditioning remain active and trainable?
- Do the raw-action baselines receive equivalent action-history information?
- Is there a control that separates the benefit of historical conditioning from that of the latent action representation?
We thank the reviewer for the constructive feedback and address the concerns below.
(1) TCLA is a unified framework that learns temporally structured latent action representations through two complementary objectives: global task-phase modeling and local temporal forward prediction. History-conditioned decoding is an integral part of this design, allowing latent actions to adapt to different execution stages and improving action modality consistency and motion smoothness. The two objectives are the meaning of "Temporally Contextualized" that we have presented in the title.
In addition, Table 5 demonstrates that temporal representation learning is beneficial even without task-progress modeling, while incorporating phase information further improves performance. Beyond improved success rates, phase modeling also enables task-progress estimation and supports memory-dependent manipulation, such as sequential button pressing. These results highlight the complementary benefits of temporal representation learning and global phase conditioning. We will further clarify the retained modules and objectives in each ablation.
(2) Pretraining Efficiency. TCLA pretraining is lightweight due to the low-dimensional action inputs and compact representation model. As detailed in Appendix B.6, the proposed action representation model contains only 9.3M trainable parameters. Moreover, the 3.5M-parameter Latent Dynamics Module is not used during inference. This additional overhead is relatively small compared with the billion-parameter-scale architectures of X-VLA, GO-1, and Fast-WAM. More importantly, the pretrained representation is reusable across BAKU, X-VLA, GO-1, and Fast-WAM, amortizing its one-time cost across different policies. We will clarify the computational overhead and distinguish downstream convergence from total training cost.
(3) Multi-task Learning. Table 3 aims to demonstrate a qualitative capability leap: extending a single-task policy architecture to multi-task learning. TCLA enables one unified ACT policy to handle ten tasks, achieving better performance with only 1/10 of the aggregate downstream training steps.
This experiment highlights the multi-task capability enabled by TCLA, rather than isolating the latent representation contribution under identical multi-task settings. We will clarify this distinction.
(4) Jerk metrics are averaged over recorded evaluation trajectories, with consistent evaluation settings within each policy family. We will provide more details on metric computation, control intervals, and evaluation protocols.
We thank the reviewer again for the valuable suggestions.