acceptodds
Withdrawn from ICLR 2027

TERRA: Learning Transportable Latent Actions through Temporal Effect Representation and Relational Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent actions supervise robot policies with action-like codes inferred from visual transitions, and their usefulness hinges on two questions: what a code keeps from a transition, and whether it still means the same thing when reused in a different initial state. The first is a tension in time: an endpoint difference discards how mo- tion unfolds, while the full sequence admits nuisance variation. The second is left open by reconstruction, which only ever observes a latent together with the state it came from. We argue that both questions can be answered in the same place. TERRA (Temporal Effect Representation and Relational Alignment) describes a transition by a compact temporal effect, its net feature change together with a low- order within-window dynamics component, and learns a continuous latent from this effect. The same effect space then serves as the reference for reuse: Effect- Anchored Transport (EAT) decodes a latent in other initial states and anchors the resulting effect to the one observed at its source, so that the latent is shaped by what it does across contexts rather than only by the transition it came from. With frozen linear readers, TERRA predicts actions more accurately than UniVLA and a LAPA-style baseline, degrades more slowly under visual distractors, and keeps transported transitions faithful to the donor action as the recipient context moves farther away; a same-budget control shows that these gains come largely from EAT. At matched pretraining scale, the complete system reaches 93.4% average success on LIBERO, compared with 91.8% for UniVLA

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Latent Actions, Robot Learning, Self-Supervised Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vmiGOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TERRA, a framework for learning continuous latent actions from video transitions. Its Temporal Effect Representation combines net visual feature changes with a low-order temporal dynamics component. Effect-Anchored Transport then decodes latent actions in other initial states and regularizes the resulting feature changes to preserve the source transition’s effect direction. Trained on 29 Open X-Embodiment sources, TERRA is evaluated through frozen linear action readers, visual distractor tests, and cross-context transport diagnostics. The authors report improved action readout and transport stability, alongside 93.4% average success on LIBERO compared with 91.8% for UniVLA at matched pretraining scale, using different policy interfaces.

Critical strengths
  1. The paper tackles a useful question: what latent actions should capture and whether they retain their meaning across different states.
  2. The same-budget reconstruction baseline and temporal-component ablations provide useful evidence for the proposed approach.
  3. The authors clearly acknowledge the limitations and provide per-source results and bootstrap intervals.
Critical weaknesses
  1. There are no real-robot experiments, and transport is evaluated through a linear reader rather than physical execution. Recovering the donor action from generated features does not show that it is feasible or produces the intended effect in the recipient state.
  2. The LIBERO comparison changes both the latent representation and the policy interface. Without closed-loop ablations using the same policy head, it is unclear what drives the improvement. Results also use only one fine-tuning seed.
  3. The gains from EAT are not fully isolated. In particular, the paper lacks a reconstruction-plus-invariance baseline, making it hard to separate the proposed transport mechanism from standard invariance regularization.
  4. Robustness is mainly supported by offline readout metrics. Perturbed closed-loop rollouts are shown only qualitatively, so it remains unclear whether the representation gains translate into more reliable control.
Questions for authors
  1. Can you test whether transported actions are physically feasible and produce the intended effects in recipient states, rather than relying only on a linear reader?
  2. How do endpoint-only, TER, and TER+EAT compare in closed-loop performance with the same policy head and multiple training seeds?
  3. How does reconstruction plus invariance regularization compare with full EAT under the same training budget?
  4. Can you report quantitative success rates for the perturbed closed-loop evaluations described in Appendix B.2?
Reviewer_LHvgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TERRA, a framework for learning latent actions that can be reused across different initial states. It combines a temporal effect representation, which captures both net feature change and coarse within-window dynamics, with EAT, which encourages decoded effects to remain consistent across contexts. The authors evaluate action decodability, robustness to visual distractors, cross-context transport, and downstream performance on LIBERO.

Critical strengths

The paper addresses an important limitation of reconstruction-based latent action learning: reconstructing an observed transition does not necessarily make its latent action reusable in other states. I also appreciate the controlled experiments, particularly the matched-budget reconstruction baseline and the analysis of how transport performance changes with recipient-state distance.

Critical weaknesses

While the results are promising, the writing is a bit hard to follow and I do not think the current evidence fully supports the central claims.

  1. The transport objective does not clearly establish action consistency across states. EAT assumes that preserving the direction of a donor's visual effect is a useful way to preserve its action semantics in a different state. However, the same action can produce different visual effects depending on the recipient state and its dynamics. The transport evaluation in Figure 4 measures whether a linear reader can recover the donor action from generated feature changes, rather than whether the resulting transition is physically valid or appropriate for the recipient state. I think this leaves a gap between the proposed notion of transport and actual action reuse.

  2. The gains are not fully isolated from other training components. While the matched-budget reconstruction control in Table 2 provides useful evidence for the benefit of EAT, the individual contributions of its multiple objectives are not fully isolated. It seems like LIBERO comparison is between complete systems rather than a controlled comparison of latent representations. TERRA and UniVLA differ in both the latent-action model and the policy interface. Despite this additional methodological complexity, the improvement in average success rate is relatively small, and Appendix B.2 states that the closed-loop results use only a single fine-tuning seed with no reported variability across training runs. These limitations make it difficult to determine how much of the downstream improvement comes from the proposed representation and how robust the reported gain is.

Minor thing, the paper could better distinguish its contribution from prior work that also uses observed effects to construct latent actions. The specific use of a shared effect descriptor for latent construction and transport supervision is interesting, but the broader motivation is not entirely new. I believe Latent Actions from Factorized Transition Effects under Agent Ambiguity shows similar motivation of latent action learning with cross context reuse.

Questions for authors
  1. The authors acknowledge that EAT is a visual effect regularizer rather than a guarantee of physically correct counterfactual actions. However, when the same action produces different effects depending on contact or object configuration, enforcing donor–recipient effect alignment may introduce incorrect supervision. What assumptions make this alignment appropriate, and under what conditions might it become counterproductive?

  2. Could the authors add a controlled experiment that evaluates transport using actual recipient-state rollouts rather than only generated feature changes? For example, applying the same donor action in multiple recipient states with known dynamics would allow the predicted effects to be compared with physically realized transitions. This would directly test whether EAT learns reusable actions rather than merely producing feature changes that remain decodable as the donor action.

  3. Could the authors provide an invariance-only refinement baseline and a downstream comparison using the same policy interface? These controls would help separate the benefit of effect-based transport from nuisance regularization and policy design differences.

Reviewer_NF7GOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

TERRA learns continuous latent actions by compressing visual transitions into their net feature change and a low-order temporal component. Effect-Anchored Transport (EAT) then decodes donor latents in other initial states and aligns the generated effects with the donor’s observed effects. Experiments across 29 OXE sources show improved action decoding and robustness, while the complete system achieves 93.4% LIBERO success versus 91.8% for retrained UniVLA. The results are promising, but the specific benefit of recipient-conditioned effect supervision and the validity of the transport evaluation remain insufficiently established.

Critical strengths
  • A clear and motivated construction. The temporal component preserves information unavailable from endpoints alone, while EAT explicitly constrains latent reuse across initial states. Neither component requires action labels during representation learning.

  • Broad representation evaluation with useful controls. The experiments cover 29 real robot-data sources, use shared frozen visual features and episode-level reader splits, and train transport readers only on real transitions. The far-context transport proxy improves on 22 of 29 sources.

  • Evidence of improvement beyond additional training. Equal-update refinement reduces Action NMSE from .784 with reconstruction to .723 with full EAT. The complete system also improves over retrained UniVLA on all four LIBERO suites, without executing the transport machinery during policy inference.

Critical weaknesses
  • The benefit of the central design choice is not isolated. Olaf-World already aligns latent-action representations with frozen temporal-effect targets, while ViVLA decodes donor latents in other initial states and enforces latent-cycle consistency. TERRA makes a distinct choice: supervising the recipient-decoded effect against the donor’s observed effect. However, Table 2 and Figure 4 compare the EAT primarily with additional reconstruction. They do not establish whether placing effect supervision after recipient decoding provides an advantage over directly aligning latents with the same effect targets. Authors also do not separately isolate effect anchoring, relational consistency, and distribution matching.

  • Robustness and downstream evidence remain limited. The moving-video-patch experiment is useful but covers a restricted perturbation: a (64 ×\times 64) patch in (224 ×\times 224) images. Perturbed LIBERO rollouts are qualitative rather than quantitative. Moreover, the 93.4% versus 91.8% comparison uses different policy interfaces and a single fine-tuning seed, leaving attribution of the improvement and its variability uncertain (Appendices B.2–B.3).

  • The gains are not separated from augmentation and policy-interface changes. Action NMSE is .784 for additional reconstruction, .764 for EAT without invariance, and .723 for full EAT. A reconstruction-plus-invariance control is therefore important but absent. Endpoint-only representation learning with full EAT is also untested, leaving the temporal component’s contribution to the final system unclear. Downstream, both the tokenizer and policy interface change, and LIBERO uses one fine-tuning seed. Consequently, the 1.6-point success improvement cannot be attributed specifically to TER/EAT.

Questions for authors
  • Does decoding into another state add value beyond directly supervising the latent with the observed visual effect? Starting from the same checkpoint, compare full EAT with two controls: remove Eq. 7; or replace it with a small prediction head that takes only the donor latent (z) and predicts the fixed effect descriptor (\Psi(\Delta F)) computed from that same observed donor clip. This replacement directly supervises what the latent encodes, without generating a transition in another state for this loss (inspired by Olaf-World). Keep all remaining losses and training budgets fixed, and report action-recovery error and LIBERO success.

  • Does effect anchoring remain beneficial when grasp outcomes change? In a LIBERO manipulation task, execute the same close-and-lift action chunk with the object centered between the fingers, offset, and outside the grasp. Compare the generated recipient effects with actual simulator outcomes. Does EAT correctly accommodate whether the object is lifted, displaced, or left behind, and does it improve control over an otherwise identical model without effect anchoring?

  • Do the gains persist under a controlled downstream robustness evaluation? Compare full EAT against reconstruction plus the same nuisance-invariance loss, using identical policy heads and at least three fine-tuning seeds. What are the mean success rates and variability on clean LIBERO and the LIBERO-Plus perturbations already used for qualitative examples? Report results separately by perturbation type to show where EAT helps or hurts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.