acceptodds
Under review as a conference paper at ICLR 2027

Consolidating Specialist Representations into a Shared World Model

Abstract

Independently trained world-model specialists provide task-specific representations whose latent coordinates need not be compatible with a shared predictor. We study how to consolidate these representations into one task-conditioned world model. Our approach freezes specialist encoders, aligns their latent spaces, and trains a shared model with cached supervision and a gradual transition to student representations. For quantized supervision, applying the same similarity map to a latent space and its codebook preserves assignments within that codebook. We compare continuous, discrete, and hybrid supervision on two-, three-, and six-task collections of pixel-based control problems. In our current single-training-seed evaluation, continuous and hybrid consolidation achieve 80.3% macro-average success on six tasks, compared with 50.0% for our native joint-training baseline; aligning hybrid supervision improves its point estimate from 75.3% to 80.3%. These results motivate further study of representation alignment for specialist consolidation, while leaving its robustness across training seeds and the causes of supervision-type differences to be established.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, multi-task learning, knowledge distillation, representation alignment, discrete representations

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4UcsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for consolidating independently trained specialist world models into one shared, task-conditioned world model. The method first freezes each specialist encoder, constructs a task-specific codebook, and caches its latent features and code assignments. It then maps all specialist features and codebooks into the latent space of a reference specialist using a similarity transformation. Finally, a shared student encoder and predictor are trained using the aligned specialist supervision. The paper studies continuous, hybrid, and discrete supervision. During inference, the shared model receives the observation, action, goal image, and task ID, and uses latent predictions with MPC for closed-loop control. Experiments are conducted on two-, three-, and six-task collections and compare the consolidated models with separately trained specialists, single-task students, and a native joint-training baseline.

Critical strengths

The paper studies an important problem: jointly training one world model on multiple tasks can cause substantial task interference. Consolidating existing specialists is a reasonable alternative to training a multi-task model completely from scratch. The extract-align-consolidate pipeline is conceptually straightforward.

Critical weaknesses

Figure 1 shows that the aligned representation has greater task separability than both unaligned concatenation and native joint training. This suggests that alignment preserves or increases task-specific clustering rather than producing a common task-invariant representation. This raises two possibilities:

  1. If latent trajectories remain within separate task clusters, the extent of cross-task representation sharing is unclear. In this case, the relative rotation and placement of the clusters may have limited effect, particularly because the predictor already receives the task ID.
  2. If latent trajectories or code assignments enter another task’s cluster, the model may treat states from different tasks as neighbors. Such cross-task matching is meaningful only if those states have compatible physical meaning and dynamics.

Could the authors visualize the ground-truth and predicted latent trajectories before and after alignment, report the task origin of selected codebook entries, and clarify what latent-space mechanism is responsible for the observed performance improvement?

Questions for authors
  1. Since the model already knows the task ID during inference, why is consolidation necessary instead of selecting the corresponding specialist? Why not train a task-ID-free model?

  2. Why does feature alignment improve performance when the aligned latent space still clearly separates the tasks?

  3. Please provide latent trajectories before and after alignment. Do trajectories remain within their own task clusters or interact with other task clusters?

  4. Since the loss does not explicitly align cross-task transitions, what mechanism makes the aligned representations useful to the shared predictor?

  5. How often does a latent select a code originating from another task? If this occurs, how do the authors verify that it is not selecting an incorrect task representation?

  6. Why is a codebook needed instead of directly aligning the continuous features, given that M4 performs as well as M2?

  7. Why is the codebook-size ablation only performed on single-task models? Please evaluate codebook size in the multi-task consolidation setting.

  8. Is there a controlled ablation that independently measures the contribution of specialist-encoder supervision?

  9. Please clarify the definitions of M0–M5, why M1 was not run, and why several variants change multiple factors simultaneously.

Reviewer_PS3EOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a method to distill the knowledge of task-specific expert world models in a single, task-conditioned world model. The method works in four steps:

  1. Train (or reuse) task-specific world models.
  2. Encode observations from each task using the associated world model, and fit a codebook with 8192 codes.
  3. Map expert features into the feature space of the PushT expert. The authors claim that such mapping preserves assignments and their ordering. Codebooks are also concatenated, producing a unified codebook with N×8192N \times 8192 codes, where NN is the number of tasks.
  4. Distill the experts into a single task-conditioned model. At each step, the student regresses the teacher representations, alongside predicting either the teacher targets or student latents over a sequence, decided by a Bernoulli sample.
    1. The hybrid and discrete variants include a KL divergence term matching the distribution of student code assignments to the top-32 codes from the teacher.

Models are trained over 16 epochs, and the student gradually moves from predicting the teacher targets to its own latents. They compare their approach to a LeWorldModel baseline with joint-training. The baseline performs reasonably well in 4/6 tasks, but collapses on PushT and Reacher. The distilled student instead performs well on all tasks.

The authors further show that continuous-only supervision performs comparably to discrete, i.e., codebook-based. Their combination also performs well, with a regression on the "Scene" environment. Finally, the authors ablate the number of codes on PushT and show that while using a smaller number of codes does not affect hybrid supervision, performance collapses for the discrete-only variant, quantify the performance improvement due to feature alignment, and show that low validation error does not guarantee control ability.

Critical strengths
  1. The paper tackles an interesting problem.
  2. The proposed method substantially improves over a joint-training baseline, and does not collapse on any individual environment.
Critical weaknesses
  1. The writing is dense, hard to parse and claims are significantly hedged. It's difficult to understand what the paper claims exactly, and numbers are often presented before the reader can grasp their significance. In addition, figures are unclear, and are very hard to decipher without having read a significant portion of the text. Furthermore, the experiment labels, i.e., MX, force the reader to do several round trips between their definitions and the results discussion. Finally, the appendix includes a "planned validation protocol" section, which cannot be assessed.
  2. The performance improvement due to alignment is almost exclusively on PushT, which also happens to be the target domain. Hence, the performance improvement might be simply due to reduced interference for that specific task. It would be interesting to see if the performance improvements persist as the target task for alignment is changed.
  3. To my understanding, LeWorldModel is not designed to be multi-task, and it's unclear if the baseline uses the same task embeddings and action space as the distilled student. Furthermore, the task-specific collapse of the baseline is not investigated in depth, making it hard to tell whether collapse on some tasks is inevitable in joint training or the baseline is under-tuned. Comparing the proposed method to a baseline that is designed to be multi-task would improve the paper.
  4. Method-wise, the motivation of using a codebook is unclear to me. Discrete supervision performs marginally worse than continuous, and the hybrid variant matches continuous on average, but is 8 points worse on the least-performing task.
  5. I am unsure whether feature alignment qualifies as a contribution by itself, as it seems to just be an application of a similarity transform?
  6. The proposed method requires training NN task-specific experts for NN tasks, which would be hard to scale as the encoder size, number of tasks, and their complexity grows.

Assessing the paper purely on the technical contribution I would lean towards a weak reject, but in my opinion the manuscript is not yet ready for publication, hence the reject.

Questions for authors
  1. What happens if another task is picked rather than PushT for target alignment?
  2. Would it be possible to compare the proposed method to a baseline designed to be multi-task?
  3. Why were models trained for only 16 epochs?
Reviewer_3h7xOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors train one task-conditioned world model by distilling from several frozen single-task world models. Each teacher's latent space is mapped onto a reference teacher's by a scaled orthogonal Procrustes fit, optionally quantized against a per-task codebook, and cached; the student regresses these targets while its predictor inputs are switched from teacher to student latents over training. On six pixel-based control tasks planned with MPC, the student reaches 80.3% macro success, against 50.0% for the platform's native joint-training model and 81.0% for the six teachers. The method is multi-teacher feature distillation with Procrustes alignment, and the large gain is measured against a joint-training baseline that fails on two of the six tasks; against its own teachers the student is on par.

Critical strengths

Consolidating independently trained world models into one deployable model is a practically relevant problem, and the deployed student needs none of the training-time components.

Critical weaknesses
  • The method is established multi-teacher feature distillation with Procrustes alignment, and the components added on top show no clear benefit. Continuous supervision, which uses no codebook, matches the hybrid variant, so the codebook machinery and the assignment property of Equation 3 are not needed for the best result; the paper also notes that this property no longer holds once the codebooks are pooled. The gain from alignment is driven by a single task on a single seed.

  • The large improvement over joint training comes from a collapsed baseline. Nearly all of it is on two tasks where the native model almost always fails, and the paper does not present this baseline as a fair test of joint training. Against its own teachers the student is only on par.

  • Figure 1 overstates the result. Its headline chart claims the student beats its teacher on PushT, but the plotted values combine the best tasks of two different variants and match no single trained model. In the six-task setting the student is below its teacher on PushT.

  • Presentation quality is very low. Figure 1 is overcrowded, Figure 2 has overlapping and overflowing text, the paper relies on opaque internal model labels, and an appendix lists planned experiments without results. Together with the scope of the AI-use statement, the paper reads as largely AI-produced, without sufficient revision by the authors.

Questions for authors

please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.