acceptodds
Withdrawn from ICLR 2027

Are Latent Actions Transferable Across Visual Contexts?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reusing behavioral knowledge from unlabeled videos in new scenes requires transferable action representations. Latent action models (LAMs) learn representations from visual transitions to predict future states, but may entangle action information with source visual context. Existing benchmarks lack a common protocol for evaluating cross-context latent-action transfer. We introduce the **L**atent **A**ction **T**ransfer Benchmark (**LATBench**), a unified protocol for evaluating action transfer across visual contexts. LATBench extracts an action representation from a source video and applies it to a different target video: the source provides the action, while the target provides the visual context. It evaluates (1) action-effect fidelity (whether the source action effect is reproduced), (2) target-context preservation (whether the target object and scene are preserved), and (3) source-content leakage (whether source appearance is copied into the generated video). We first assess how well existing LAMs transfer across visual contexts and examine how representation design affects transferability. We then investigate the balance between action information and source content in learned representations and test the limits of transfer under longer rollouts, camera motion, and viewpoint changes. Our experiments reveal a key insight: effective transfer requires suppressing source content while retaining action information that supports state changes in the target context; lower leakage alone is insufficient.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Latent Action Model, Action Transferability, Cross-Context Transfer, Benchmarking and Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CknjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LATBench proposes to standardize an evaluation methodology that is increasingly common in LAMs: action transfer across visual contexts. The paper uses the SSv2 and HRDexDB datasets to construct their benchmark and propose three different metrics to evaluate existing LAMs and explain their performance: action-effect fidelity (AEF), target-context preservation (TCP), and source-content leakage (SCL).

Critical strengths

The paper's goal is directionally sound and compelling. This direction is valuable for the LAM/embodied AI/world modeling community. The paper methodology is clear and the overall presentation quality is good.

Critical weaknesses

First, the lead question "can latent actions learned from video reproduce action effects across visual contexts while preserving the target object and scene?" is one that is already extensively explored in prior literature [cf., IGOR, ViPRA, Olaf-World, DiLA, Learning Latent Action World Models In The Wild, as cited by the authors already; plus others such as CoLA-World, Wang et al., ICML 2026; DisMo, Ressler-Antal et al., NeurIPS 2025]. That is not necessarily a weakness. However, it is unclear to me from reading the paper what sets this benchmark apart from previous efforts -- an explicit discussion of how and why settings & metrics differ, and explicit experiments establishing the advantage and value of the proposed settings and metrics.

Second, there are multiple methodological weaknesses that could undermine the relevance of the insights derived from the explorations around the benchmark. I believe that, when fixed or otherwise addressed, the benchmark could be of value to the community:

  • Poor interpretability of the proposed metrics: without a reference of how evaluators perform on ground truth videos (with assigned ground truth action or random described action), random actions fed to the decoder, no-op actions, it's difficult to understand how meaningful the results are. Human agreement is positive but the sample size is small. Additionally, how reliable judges are is not well established (Table 19).
  • The trained decoder is a major confounder: L1 reconstruction produces blurry renders (that judges then penalize) for latent representations that discard information (as they should), have high uncertainty regions, or leave the future frame underspecified. Diffusion/FM decoders are a valid alternative. Moreover, decoders are not matched (App D.2 contradicts Sec 4.2) in spatial resolution (output res and patch size) which disadvantages some LAMs.
  • The experiment often compare high-dimensional latent representations with very compressed ones. Very compressed ones can encode less informations and therefore would leak less. This is a confounder of compression vs what is learnt. An additional experiment comparing low-rank projection of the high-res feature is needed (e.g., via PCA)
Questions for authors

The asks for the authors should be quite clearly laid out in the critical weaknesses section already -- if the relation to previous explorations (and thus also the contribution of this work) is made clear, and the potential methodological issues are sufficiently addressed, I think this could be an acceptable paper.

Reviewer_3BY2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LATBench, a benchmark for transferring latent actions across visual contexts. A latent action extracted from a source video is applied tothe initial frame of a target video via a trained pixel decoder. Three VLM evaluators assess action-effect fidelity (AEF), target-context preservation (TCP), and source-content leakage (SCL), these scores are combined into a Transfer Score.

The authors compare four pretrained LAMs (LAPA, UniVLA, villa-X, and LaWAM) on subsets of SSv2 and HRDexDB, and conduct controlled experiments examining the effects of the visual encoder, IDM inputs, and latent bottleneck.

Main findings:

  1. Learned latent actions reduce SCL compared with DINO feature differences.
  2. V-JEPA 2.1 with CoMo-style IDM inputs achieves the highest average Transfer Score across the two datasets among the evaluated combinations.
  3. Discretizing latent actions reduces SCL but lowers the Transfer Score.
Critical strengths
  1. Direct and interpretable evaluation of action transfer. LATBench tests whether an action’s effect can be reproduced in a new visual context. Decomposing evaluation into AEF, TCP, and SCL is a useful approach to distinguishing successful action transfer from copying source appearance. The results show that low leakage alone does not guarantee successful transfer.

  2. Controlled investigation of representation design choices. In 5.2, the authors vary the visual encoder, IDM inputs, and latent bottleneck while keeping the training data and decoding protocol fixed. This helps to clarify how representation design affects transfer performance and complements the comparison of off-the-shelf LAMs.

Critical weaknesses
  1. Insufficient validation of the reliability of VLM scores and the robustness of the conclusions. The reported correlation with human judgments supports the general usefulness of the evaluation, but does not establish its reliability for individual components or small differences between configurations. Validation covers only the overall Transfer Score on 80 generated outputs from 20 source–target pairs (four models per pair), although many conclusions rely on changes in AEF/TCP/SCL, particularly SCL. Agreement among the three human raters is also not reported. Moreover, the ± values in the tables reflect variability across three VLM evaluators, rather than confidence intervals over evaluation samples or variability across training runs. Without these uncertainty estimates, the robustness of the comparative conclusions in 5.2, including the recommendation of V-JEPA 2.1 + CoMo as the default choice, remains unclear.

  2. Missing pretraining details limit the interpretation of Table 1. The off-the-shelf LAMs were trained on different datasets, but the exact checkpoints and pretraining data are not specified. Table 1 therefore compares models as a whole, without separating the contributions of architecture and training data.

  3. Limited significance of the findings. The practical insights provided by the benchmark are limited. The most robust finding is that learned latent actions exhibit less leakage than DINO feature differences. This is expected by design, since the latent bottleneck limits the amount of information transmitted. The remaining recommendations, including the choice of V-JEPA 2.1 + CoMo, are not supported by uncertainty estimates (W1), and have not been validated on downstream tasks. It therefore remains unclear how they can guide the choice of a LAM in practice.

Questions for authors
  1. Robustness of the conclusions. Could you provide confidence intervals for the differences in Transfer Score between configurations in 5.2, computed over evaluation samples separately for each dataset? In particular, is the advantage of V-JEPA 2.1 + CoMo over the second-best configuration on average (DINOv3 + LAPA-style) robust enough to recommend it as the default? Which reported advantages remain supported?
  2. Connection to downstream performance. You note that downstream results also depend on the policy and control components. Still, could you check, even for a few configurations under the same protocol, whether the Transfer Score ranking agrees with downstream performance?
  3. Source of degradation. The degradation in 5.4 may come from the action representations, the decoder, or both. Could you show self-reconstruction (actions from a video applied to its own initial frame) on the same challenging examples? Comparing it with cross-context transfer would show whether the degradation already appears without transfer.

Clarifications

  1. Checkpoints and pretraining. Which checkpoints and pretraining data were used for the four off-the-shelf LAMs and for LAPA-DINOv2/v3 (5.3)? Was overlap with the SSv2 and HRDexDB evaluation videos checked?
  2. Evaluation pairs and transfer types. How are SSv2 pairs constructed (number of pairs, within or across action classes, repeated videos)? Is the applicability of the source action to the target initial state verified? How are the pairs of both datasets distributed across cross-scene, cross-object and cross-embodiment transfer?
  3. Data filtering. How was the manual filtering of the SSv2 validation set performed?
Reviewer_4UhJOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed a new benchmark, LATBench, to test whether action movements learned from one video can transfer to a completely different scene or object.

To measure success, the authors look at whether the action actually happens, whether the new scene stays intact, and whether visual details from the original video accidentally leak into the new one.

They found that models must actively remove the original visual background while preserving the core motion details, showing that simply removing source content is not enough.

Through experiments comparing different models, they found that combining a V-JEPA visual encoder with motion-difference inputs gave the best overall transfer performance.

Finally, they confirmed that these models still struggle with longer video sequences, camera movements, and changing viewing angles.

Critical strengths

Most prior latent-action benchmarks rely on same video reconstruction, feature probing, or full robot control, while prior cross-video transfer demos used incompatible setups and mostly qualitative examples. LATBench freezes each representation and trains a shared pixel decoder to score three clear failure modes: whether the right motion happens (AEF), whether the target object/embodiment stays intact (TCP), and whether source appearance leaks in (SCL), to have a clean way to compare models side by side.

Beyond benchmarking existing models, the paper runs controlled ablations over visual encoders, IDM inputs, and bottlenecks. Comparing raw feature differences, continuous VAEs, and discrete VQ-VAEs shows an important lesson: blocking source leakage is necessary, but overly tight discrete bottlenecks throw away motion details needed to cleanly update the target image, hurting both action fidelity and target preservation.

Critical weaknesses
  • Every metric in LATBench depends on three VLM judges (Gemini 3 Flash Preview, Gemini 3.5 Flash Lite, and DeepSeek V4 Flash Vision Exp) with long, multi-step prompts. Human validation only covers 80 outputs from 20 video pairs and only checks the combined Transfer Score, not AEF, TCP, or SCL individually. Meanwhile, the appendix tables show large disagreements across the three VLM judges. For instance, on HRDexDB, AEF for LAPA jumps from 0.040 under one judge to 0.387 under another (Table 4), and SCL rates often differ by more than 10x (Tables 15, 19). Relying on Preview and Exp API models also makes future leaderboard comparisons hard to reproduce as endpoints change or retire.

  • The paper never tests these representations on real robot tasks. As a result, there is no evidence that scoring higher on these visual image metrics (like continuous models beating discrete ones) actually leads to better robot control.

Questions for authors
  • Given that proprietary VLM endpoints (Gemini 3 Flash Preview, Gemini 3.5 Flash Lite, DeepSeek V4 Flash Vision Exp) evolve over time, how will benchmark comparisons remain reproducible for future external submissions without drift in automated scoring?

  • The SSv2 evaluation suite is limited to six planar directional movements, excluding classes like opening and closing because of high variance. Given that manipulation often involves non-rigid, multi-stage. How sensitive is the finding regarding V-JEPA + CoMo-style inputs when applied to non-planar manipulation tasks beyond basic human-robot grasping?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.