In-Context Adaptation of Dynamics Models for Control

Learned dynamics models typically fail to generalize to environments with unseen physical parameters. Standard approaches add adaptation machinery, such as test-time gradient updates or an inferred latent variable trained with an auxiliary objective. We show that this machinery is not what decides the matter, and that whether a model adapts in context is settled by how it is trained and by where it is tested. A plain sequence model, with no module for inferring the environment, no loss beyond next-state prediction, and no positional encoding, adapts to unseen physics in a single forward pass with its weights frozen. Its context is simply the recent transitions, each kept whole so that a state, the action taken, and the resulting next state stay together. On the six continuous-control environments of lee2020cadm, under their protocol and interaction budget, it outperforms the vanilla, stacked, and meta-learned model-based baselines everywhere, and an ensemble variant clears both variants of the purpose-built context-aware method outright on three of the six while staying inside seed noise on two more. Interventions that leave the weights untouched show what the model uses. Permuting which next state follows which transition costs almost as much as removing the context outright, and replacing the transitions by their average costs two-thirds of the benefit, so the model addresses individual transitions rather than a summary of them. What the context is worth also depends on where the model is tested, moving by a factor of five within one environment between held-out ranges that sit equally far from training. We report where the approach does not work. On Ant, the context is worth nothing, and no intervention we tried changes that, including a fivefold widening of the training distribution.