Consolidating Specialist Representations into a Shared World Model

Independently trained world-model specialists provide task-specific representations whose latent coordinates need not be compatible with a shared predictor. We study how to consolidate these representations into one task-conditioned world model. Our approach freezes specialist encoders, aligns their latent spaces, and trains a shared model with cached supervision and a gradual transition to student representations. For quantized supervision, applying the same similarity map to a latent space and its codebook preserves assignments within that codebook. We compare continuous, discrete, and hybrid supervision on two-, three-, and six-task collections of pixel-based control problems. In our current single-training-seed evaluation, continuous and hybrid consolidation achieve 80.3% macro-average success on six tasks, compared with 50.0% for our native joint-training baseline; aligning hybrid supervision improves its point estimate from 75.3% to 80.3%. These results motivate further study of representation alignment for specialist consolidation, while leaving its robustness across training seeds and the causes of supervision-type differences to be established.