Representation Selection in End-to-End Feature-Predictive World Models

Latent world models support planning by evaluating candidate action sequences in compact visual representations. While fixed-target models predict exhaustive visual details, joint-embedding architectures avoid explicit reconstruction through end-to-end latent prediction; By dynamically updating the target encoder, they inherently turn the prediction objective into an implicit feature selector for predictable information. Under finite representation capacity and structural assumptions, this preference inherently marginalizes less predictable, task-relevant factors. We formalize this phenomenon as predictability bias. We then analytically derive how explicit target supervision redirects this capacity allocation, giving a sufficient condition under which retaining target variables improves alignment between the native planning score and task cost, even when pure prediction error rises. Controlled exact-whitening and action-conditioned experiments empirically verify this predictability-driven factor selection. In a separate four-task continuous control evaluation, countering this bias via explicit target supervision improves native closed-loop success under identical deployment conditions, suggesting that beyond merely fitting temporal dynamics, the joint prediction objective can act as an implicit filter that shapes the composition of the latent space.