- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LATBench proposes to standardize an evaluation methodology that is increasingly common in LAMs: action transfer across visual contexts. The paper uses the SSv2 and HRDexDB datasets to construct their benchmark and propose three different metrics to evaluate existing LAMs and explain their performance: action-effect fidelity (AEF), target-context preservation (TCP), and source-content leakage (SCL).
Critical strengths
The paper's goal is directionally sound and compelling. This direction is valuable for the LAM/embodied AI/world modeling community. The paper methodology is clear and the overall presentation quality is good.
Critical weaknesses
First, the lead question "can latent actions learned from video reproduce action effects across visual contexts while preserving the target object and scene?" is one that is already extensively explored in prior literature [cf., IGOR, ViPRA, Olaf-World, DiLA, Learning Latent Action World Models In The Wild, as cited by the authors already; plus others such as CoLA-World, Wang et al., ICML 2026; DisMo, Ressler-Antal et al., NeurIPS 2025]. That is not necessarily a weakness. However, it is unclear to me from reading the paper what sets this benchmark apart from previous efforts -- an explicit discussion of how and why settings & metrics differ, and explicit experiments establishing the advantage and value of the proposed settings and metrics.
Second, there are multiple methodological weaknesses that could undermine the relevance of the insights derived from the explorations around the benchmark. I believe that, when fixed or otherwise addressed, the benchmark could be of value to the community:
- Poor interpretability of the proposed metrics: without a reference of how evaluators perform on ground truth videos (with assigned ground truth action or random described action), random actions fed to the decoder, no-op actions, it's difficult to understand how meaningful the results are. Human agreement is positive but the sample size is small. Additionally, how reliable judges are is not well established (Table 19).
- The trained decoder is a major confounder: L1 reconstruction produces blurry renders (that judges then penalize) for latent representations that discard information (as they should), have high uncertainty regions, or leave the future frame underspecified. Diffusion/FM decoders are a valid alternative. Moreover, decoders are not matched (App D.2 contradicts Sec 4.2) in spatial resolution (output res and patch size) which disadvantages some LAMs.
- The experiment often compare high-dimensional latent representations with very compressed ones. Very compressed ones can encode less informations and therefore would leak less. This is a confounder of compression vs what is learnt. An additional experiment comparing low-rank projection of the high-res feature is needed (e.g., via PCA)
Questions for authors
The asks for the authors should be quite clearly laid out in the critical weaknesses section already -- if the relation to previous explorations (and thus also the contribution of this work) is made clear, and the potential methodological issues are sufficiently addressed, I think this could be an acceptable paper.