How Do Latent Actions Generalize? Probing Ordered Transfer Structure in Action-Free Representations

Latent action models learn transition representations from visual data without robot-action annotations, providing a potential interface between large-scale visual experience and downstream policy learning. If latent actions are intended to support generalization, however, a key question is how transition information should be organized in the latent space. We study whether action-free latent representations develop an ordered refinement structure that is systematically related to cross-distribution generalization. To make this process observable, we introduce a weak nested-access bias that exposes progressively expanded prefixes of the same transition representation, without action supervision, predefined token semantics, or assumed high-level/low-level skill hierarchies. We then analyze how action-related, state-transition, and environment-specific information evolve across refinements and distribution shifts. Across LIBERO, LIBERO-Plus, and MimicGen, we observe a consistent pattern: refinement increases in-distribution action recoverability, while environment-specific information also becomes more decodable and OOD performance does not improve monotonically. Under stronger distribution shifts, intermediate refinements can outperform the complete latent representation, with similar non-monotonic behavior in closed-loop control. These findings suggest that generalizable latent-action learning depends not only on how much transition information is learned, but also on how that information is organized during learning, offering a useful direction for designing more transferable latent-action objectives.