acceptodds
Under review as a conference paper at ICLR 2027

Compositional Context Encoding for Out-of-Distribution Generalization in Modular World Models

Abstract

World models used in model-based reinforcement learning must generalize to physical configurations unseen during training. A key challenge arises in compositional out-of-distribution (OOD) settings: the parameters of each mechanism span a wide range, but training varies each mechanism in isolation, whereas test time requires simultaneous extrapolation across all mechanisms. We show that a naive global context encoder fails in this setting because the joint OOD input it encounters at test time was never seen in any form during training, even though the parameters of each individual mechanism remain within the training distribution. We propose Compositional Context Encoding (CCE), which assigns one independent GRU encoder to each causal mechanism. Grounded in the Independence of Causal Mechanisms (ICM) principle, each encoder reads only the transition history of its own state subspace, so the joint OOD test input is automatically decomposed into per-encoder single-sided OOD inputs that the training distribution does cover. A lightweight auxiliary loss aligns inferred mechanism parameters with oracle values during training, enabling stable long-horizon rollouts. On a decoupled double-pendulum benchmark with 64 compositional OOD test configurations, CCE achieves – lower 20-step rollout MSE than a parameter-free monolithic baseline across three random seeds. By contrast, a global-encoder variant performs worse than the parameter-free baseline, confirming that the compositional structure is essential.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model-based reinforcement learning, compositional generalization, out-of-distribution generalization, world models, independence of causal mechanisms, context encoding, modular architectures, meta-reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DFT5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies context encoding for world models when training varies each subsystem’s parameters in isolation but testing varies all of them. It proposes Compositional Context Encoding (CCE), which uses one GRU encoder per subsystem, each reading only its own substate, plus an auxiliary loss that regresses inferred parameters onto ground-truth values. The approach is tested on a decoupled double-pendulum benchmark.

Critical strengths

The question of how context encoders generalize to unseen parameter combinations is relevant to model-based RL.

Critical weaknesses
  1. All experiments use a single toy environment where the proposed factorization holds by construction. That is clearly substandard for ICLR.

  2. The results consist of one environment and one table, shown again as one bar plot, over three seeds. All comparisons are internal variants, oracle references, or a deliberately weakened lower bound. No existing baseline (e.g., PEARL, CaDM) is evaluated.

  3. The central claim that a global encoder fails rests on a single-seed appendix experiment, and it is unclear whether that model used the auxiliary loss the paper shows is decisive. That auxiliary loss requires ground-truth physical parameters for every training trajectory, and the method fails without it. Such labels are only available in simulation and make the paper not really applicable in practice.

Questions for authors

Q1: Can you add meaningful baselines and more experiments?

Q2: How does the approach work without ground truth?

Reviewer_jx4rOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary
  • The paper provides a method that avoids the difficulty of encoding long contexts by leveraging a way to break up the global context into multiple local ones.
Critical strengths

The problem targeted by the paper is important, as long-context problems are often encountered in decision-making.

Critical weaknesses

The paper is very poorly written. For example, section 3.5 is missing. I think the background section is supposed to be related work?

Questions for authors

How will the method work if the subsestems are not known a-priori ?

Reviewer_FoNrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at when a world model has to generalize to unseen combinations of subsystem parameters. Training varies each subsystem one at a time, but test time has everything off simultaneously. Their argument is that a global context encoder breaks here because the concatenated input is off-manifold even if each subsystem individually looks in-distribution. The fix is straightforward: give each mechanism its own GRU encoder that only reads transitions from its own state slice, grounded in the ICM factorization. They also add an auxiliary loss matching inferred parameters to oracle values, which turns out to matter a lot for rollout stability. Experiments are on a decoupled double pendulum.

Critical strengths

The single-sided vs joint OOD mismatch is a failure mode that I haven't seen spelled out clearly in the MBRL literature. The observation that the encoder input itself shifts, even when marginal subsystem distributions look fine, is correct and worth pointing out. The auxiliary-loss ablation is genuinely informative: CCE without it has nearly the same single-step MSE but falls apart in rollout. This is a useful warning that single-step metrics don't tell you anything about long-horizon behavior.

Critical weaknesses

The experimental scale is well below ICLR main-track level. Two independent pendulums, three parameters each, four discrete levels. The entire pitch is "decompose joint OOD into k single-sided OOD problems" — but k=2 is the only thing tested. There's no evidence this survives k=3, continuous parameters, or any system where subsystems actually interact. The main baseline is a strawman. MonoNoPhi has no parameter conditioning at all. The comparison I actually care about — a global encoder feeding a grouped/modular world model — is pushed to Appendix C and only run on one seed. If that baseline also fails, it should be a full table entry with three seeds, not a footnote. The auxiliary loss needs ground-truth physical parameters during training. In any real MBRL setting you don't have this. The paper presents it as a lightweight regularizer but it's actually a fairly strong supervision signal. There are no policy results. It's a world-model paper and the only metric is rollout MSE on a toy pendulum. No Dreamer, no PPO, no downstream reward. The seed variance is large. The oracle grouped model itself swings from 0.34 to 1.15 ×10⁻² across seeds. That makes the "3.5–4.9× improvement" number feel a bit fragile.

Questions for authors

Why is ContextAwareGroupedWM (Appendix C) only on seed 42? Can you add it to Table 1 with all three seeds? What happens at k=3 or k=4? The decomposability claim is the whole paper. How sensitive is performance to λ? Only λ=0 and λ=0.1 are reported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.