acceptodds
Withdrawn from ICLR 2027

Learning Temporally Contextualized Latent Action Space for Robot Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robot policies are commonly trained to predict low-level action chunks from observations. However, raw action spaces are weakly aligned with task progress and lack inherent temporal semantics. In this work, we propose an action representation learning framework that learns a Temporally Contextualized Latent Action (TCLA) space. Beyond action reconstruction, we introduce two pretraining objectives: a temporal forward prediction objective in latent space and a task-progress estimation objective using episode-level spectral action context. Through experiments, we show that our method consistently improves training efficiency, success rates, and motion smoothness across pretrained Vision-Language-Action (VLA) models and World Action Models (WAMs), as well as policies trained from scratch. For example, on a RoboTwin task set, TCLA improves X-VLA from 55.6% to 89.0% in just 10k steps, and improves FastWAM from 20.6% to 40.8% at epoch 5. It also provides a new perspective on extending single-task policies to multi-task settings, highlighting the broader potential of temporal action pretraining.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Manipulation, Action Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_D6rFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TCLA, a framework for learning temporally contextualized latent action representations for robot manipulation. It combines action-chunk reconstruction, a task-progress representation derived from the DCT of executed action history, and a latent flow-matching objective with an EMA target encoder. After pretraining, downstream policies predict latent actions, which are decoded using task and historical context. The method is evaluated with several policy architectures on RoboTwin, multi-task learning, and five real-world manipulation tasks. The reported results show improvements in task success, downstream training convergence, and motion smoothness.

Critical strengths
  1. Studying how action representations and temporal context affect robot policy learning is a worthwhile direction with potential benefits for manipulation policies.
  2. Experiments span different policy families and both simulated and physical environments, providing encouraging evidence of the framework’s applicability.
Critical weaknesses
  1. TCLA changes both the action representation and the decoding process by introducing historical action context. The current experiments do not clearly distinguish the benefits of temporal representation learning from those of additional history-conditioned decoding. Although Table 5 provides objective ablations, it does not fully isolate these factors, limiting confidence in the claimed contribution of the proposed latent action space.
  2. The reported convergence improvements mainly consider downstream policy training while excluding the additional TCLA pretraining stage. The pretraining cost, data requirements, and computational overhead are not sufficiently quantified. Therefore, faster downstream convergence does not necessarily demonstrate better end-to-end training efficiency.
  3. Table 3 compares separately trained single-task ACT policies with a jointly trained multi-task ACT+TCLA policy. This changes both the training setting and the action representation, making it difficult to attribute the improvement specifically to TCLA. A multi-task ACT baseline using raw actions under the same training conditions is necessary.
  4. The paper reports reduced jerk, but does not sufficiently control for execution duration, trajectory characteristics, or differences in metric computation. Moreover, the main results lack clear uncertainty estimates across independent training runs. These limitations weaken the evidence for the claimed improvements in motion quality and robustness.
Questions for authors
  1. Which inputs, modules, and objectives are retained in each Table 5 variant?
  2. When the progress objective is removed, does the history encoder and phase conditioning remain active and trainable?
  3. Do the raw-action baselines receive equivalent action-history information?
  4. Is there a control that separates the benefit of historical conditioning from that of the latent action representation?
1 comment

We thank the reviewer for the constructive feedback and address the concerns below.

(1) TCLA is a unified framework that learns temporally structured latent action representations through two complementary objectives: global task-phase modeling and local temporal forward prediction. History-conditioned decoding is an integral part of this design, allowing latent actions to adapt to different execution stages and improving action modality consistency and motion smoothness. The two objectives are the meaning of "Temporally Contextualized" that we have presented in the title.

In addition, Table 5 demonstrates that temporal representation learning is beneficial even without task-progress modeling, while incorporating phase information further improves performance. Beyond improved success rates, phase modeling also enables task-progress estimation and supports memory-dependent manipulation, such as sequential button pressing. These results highlight the complementary benefits of temporal representation learning and global phase conditioning. We will further clarify the retained modules and objectives in each ablation.

(2) Pretraining Efficiency. TCLA pretraining is lightweight due to the low-dimensional action inputs and compact representation model. As detailed in Appendix B.6, the proposed action representation model contains only 9.3M trainable parameters. Moreover, the 3.5M-parameter Latent Dynamics Module is not used during inference. This additional overhead is relatively small compared with the billion-parameter-scale architectures of X-VLA, GO-1, and Fast-WAM. More importantly, the pretrained representation is reusable across BAKU, X-VLA, GO-1, and Fast-WAM, amortizing its one-time cost across different policies. We will clarify the computational overhead and distinguish downstream convergence from total training cost.

(3) Multi-task Learning. Table 3 aims to demonstrate a qualitative capability leap: extending a single-task policy architecture to multi-task learning. TCLA enables one unified ACT policy to handle ten tasks, achieving better performance with only 1/10 of the aggregate downstream training steps.

This experiment highlights the multi-task capability enabled by TCLA, rather than isolating the latent representation contribution under identical multi-task settings. We will clarify this distinction.

(4) Jerk metrics are averaged over recorded evaluation trajectories, with consistent evaluation settings within each policy family. We will provide more details on metric computation, control intervals, and evaluation protocols.

We thank the reviewer again for the valuable suggestions.

Reviewer_5DHsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TCLA pretrains a continuous action representation using reconstruction, flow matching between successive action-chunk latents, and episode-progress prediction from a spectral representation of past actions. A downstream policy predicts latent actions, which a task- and history-conditioned decoder converts into executable chunks. The experiments cover several policy architectures, ten RoboTwin tasks and their Hard variants, and five physical manipulation tasks.

Critical strengths

The improvements are substantial and span different policy architectures. Table 1 reports best observed success increases from 46.4% to 71.4% for BAKU, 82.8% to 91.8% for X-VLA, 75.8% to 81.4% for GO-1, and 42.2% to 54.8% for Fast-WAM. The advantage appears early in downstream training as well: X-VLA reaches 89.0% at 10k steps versus 55.6% for the baseline. These are useful results for both policies trained from scratch and pretrained policies.

The physical evaluation strengthens this empirical case. Aggregate success increases from 43/100 to 78/100, with the clearest change on sequential button pressing, from 3/20 to 20/20 (Table 4). The representation is also described concretely enough to understand how temporal context enters deployment. Table 5 includes a reconstruction-only variant and objective/EMA ablations, so the evidence goes beyond comparing an entirely new system against raw-action prediction.

Critical weaknesses

The experimental protocol leaves important comparison details unspecified. Section 3.1 and Appendix D do not give demonstration counts, the data and duration used for representation pretraining, or the downstream optimization and action-execution settings for each baseline/TCLA pair. Appendix D supplies the representation optimizer, but does not resolve these questions. This prevents a clear assessment of data parity and reproducibility; it is not evidence that the settings actually differ. The convergence curves establish an advantage versus downstream training steps, while total training cost remains unknown once representation pretraining and per-step costs are included.

The interpretation of the improvement is broader than the controls justify. Equation 13 adds a direct history-conditioned decoding path, so the full-system gain cannot be attributed solely to a better latent prediction target. Table 5 helps, but does not say whether removing task-progress supervision also removes phase conditioning. Likewise, Table 3 demonstrates a useful shared ACT+TCLA policy, but its comparison with separately trained single-task ACT policies does not establish an advantage over a shared raw-action ACT with comparable task/history inputs. These limitations narrow the mechanistic and multi-task conclusions; they do not erase the value of the complete interface.

Questions for authors
  1. Please specify the demonstration counts, representation-pretraining data and steps, and downstream optimization/chunk-execution settings. Are data and execution settings matched within every pair? What cost is included in the efficiency claim?
  2. What exactly changes in the task-progress ablation? Can existing results separate supervision from phase conditioning, and clarify the multi-task benefit relative to a shared raw-action ACT? Otherwise, please restrict the attribution to the full TCLA interface.
1 comment

We thank the reviewer for recognizing TCLA's consistent improvements across policy architectures and real-world tasks. We appreciate the constructive questions and clarify the following points.

Q1. Experimental Protocol and Training Efficiency.

We follow the standard RoboTwin Clean2Clean and Clean2Random evaluation settings, using 50 demonstrations per task and executing each predicted action chunk in full during evaluation.

Regarding training cost, TCLA pretraining is highly efficient due to the low dimensionality of action data and the lightweight representation modules. More importantly, the pretrained action representation is reusable across different policy architectures (BAKU, X-VLA, GO-1, and Fast-WAM), without requiring separate pretraining for each model. Thus, the one-time pretraining overhead is small and can be amortized across downstream policies. We will clarify the experimental configurations and distinguish downstream convergence efficiency from total training cost.

Q2. Phase Latent Ablation and Multi-Task Learning.

(1) Phase latent ablation. Task-progress prediction serves as an auxiliary training objective for the phase latent, encouraging it to capture temporal task progression. Importantly, the corresponding ablation in Table 5 removes the entire phase latent component, including its auxiliary progress supervision and participation in downstream conditioning. Therefore, this experiment evaluates the contribution of the complete phase latent rather than isolating the progress prediction loss. We will clarify this distinction in the paper.

(2) Multi-task learning. We would like to emphasize that Table 3 demonstrates a qualitative capability leap enabled by TCLA: extending a single-task policy architecture to multi-task learning. The original raw-action ACT requires independently trained policies for different tasks and lacks an explicit multi-task instruction interface. By introducing TCLA, we enable a single unified policy to perform all ten tasks, achieving better overall performance with approximately 1/10 of the aggregate training cost. This represents a fundamental shift from task-specific policy learning to shared multi-task learning, rather than merely an incremental improvement in success rate. Therefore, Table 3 is intended to highlight this new capability enabled by TCLA, rather than to isolate the contribution of latent representations against a shared raw-action baseline.

We appreciate the reviewer's suggestions and will revise the manuscript to clarify these experimental details and the scope of our conclusions.

Reviewer_EDTNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Temporally Contextualized Latent Actions (TCLA), a framework for learning continuous action representations through action reconstruction, latent temporal forward prediction, and task-progress supervision. The authors use flow matching between consecutive action latents and spectral features of historical actions to introduce temporal context into the learned action space. The resulting representations serve as prediction targets for downstream robot policies. Experiments on RoboTwin and five real-world manipulation tasks demonstrate improvements in success rate, training convergence, and motion smoothness across several policy architectures.

Critical strengths
  • Meaningful research direction. The paper studies the design of action representations rather than focusing exclusively on policy architectures. This is an important direction for improving the efficiency of robot policy learning.
  • Substantial empirical gains across different policies. TCLA consistently improves performance for BAKU, X-VLA, GO-1, and Fast-WAM. And the paper includes simulation, robustness evaluation, multi-task learning, and real-world experiments. The improvements in motion smoothness provide useful additional evidence beyond task success rates.
Critical weaknesses
  • Insufficient justification for the latent forward prediction objective. The paper does not clearly explain why flow matching is necessary for learning temporally structured action representations. In particular, why not first consider a simpler objective, such as direct next-latent regression, instead of predicting the velocity between consecutive action latents?
  • Unclear contribution of task-progress supervision. Is task-progress supervision really beneficial? According to Table 5, adding task-progress supervision to the "w/o latent prediction and task progress" baseline actually decreases the success rate from 41.6% to 11.6%. This counterintuitive result raises concerns about the effectiveness of the proposed task-progress objective.
  • Potentially unfair comparisons with baselines. TCLA introduces several additional modules and a separate action representation pretraining stage, making the comparisons with raw-action baselines potentially unfair. It is unclear whether the improvements come from the proposed temporal objectives or simply from the additional model capacity and training. The authors should clarify the computational and data costs and provide better-controlled comparisons.
  • Insufficient discussion of related work and limited presentation quality. The paper lacks a sufficiently thorough comparison with existing action representation learning methods, particularly in the experiments. Moreover, several figures are not very informative and provide limited insights beyond the numerical results. More meaningful comparisons and analyses would strengthen the paper.
  • Limited benchmark coverage and potential task selection bias. Evaluating only a manually selected 10-task subset of RoboTwin is insufficient to demonstrate general effectiveness and raises concerns about potential cherry-picking.
Questions for authors

As shown in the weaknesses.

1 comment

We sincerely thank the reviewer for recognizing the strengths of our work and providing constructive suggestions. We address the concerns point by point below.

1. Justification for Latent Forward Prediction with Flow Matching

Our current design was selected after exploring multiple architectural and objective formulations during the development of this framework. We appreciate the reviewer's suggestion and will carefully consider it. We will supplement the paper with additional analyses to better demonstrate the advantages of the chosen formulation.

2. Effectiveness of Task-Progress Supervision

The performance degradation observed when using task-progress supervision alone arises from the accumulation of errors between progress estimation and action prediction. Introducing latent forward prediction helps mitigate this issue by providing additional temporal constraints on the learned action representations, as discussed in our analysis. This also explains why the two objectives exhibit complementary effects and why their combination achieves substantially better performance.

3. Fairness of Comparisons and Computational Overhead

As detailed in Appendix B.6, the proposed action representation model contains only 9.3M trainable parameters. Moreover, the 3.5M-parameter Latent Dynamics Module is not used during inference. This additional overhead is relatively small compared with the billion-parameter-scale architectures of X-VLA, GO-1, and Fast-WAM.

Furthermore, the pretrained action representation is frozen during downstream policy training. In essence, TCLA provides a new set of structured action targets for policy learning, rather than requiring substantial modifications to or additional training of the policy backbone.

4. Comparison with Related Action Representation Methods

To the best of our knowledge, our work is among the first to explicitly incorporate full-trajectory temporal context into action representation pretraining, whereas most existing approaches primarily focus on modeling action representations within individual action chunks.

Therefore, we believe that evaluating TCLA across diverse policy architectures provides meaningful evidence of its general applicability, particularly given the lack of directly comparable methods targeting the same problem formulation.

Nevertheless, we appreciate the reviewer's suggestion and will consider including additional comparisons in the future to further strengthen our evaluation.

5. RoboTwin Benchmark Coverage

We selected a ten-task subset of RoboTwin primarily because our evaluation involves testing a large number of intermediate training checkpoints to systematically examine training convergence and learning efficiency. Expanding this evaluation to the entire benchmark would introduce substantial computational costs. Furthermore, as we have described in the experiment setup, the selection of this task subset is informed by prior work on action-space design for robotic manipulation, rather than being arbitrarily chosen.

Importantly, beyond RoboTwin simulation, we also provide success-rate evaluations and video demonstrations on five real-world manipulation tasks. These additional experiments offer further evidence of the effectiveness and applicability of TCLA beyond the selected simulation tasks.

We thank the reviewer again for the valuable feedback, which will help us improve the clarity and experimental rigor of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.