In-context Imitation Learning under Task Drift with Tabular Foundation Models

Behavior cloning is typically formulated as an offline problem: a policy is trained on a fixed demonstration dataset and deployed without further adaptation. In practice, changes in object configurations, initial states, goals, or other task conditions can substantially alter the input distribution and degrade performance. We refer to this setting as task drift and study how a policy can adapt incrementally from small sets of new demonstrations without retraining from scratch. In-context learning provides a natural mechanism for such adaptation without updating model weights. Existing robotic in-context learners, however, acquire this capability through large-scale, robot-specific training, limiting their applicability when demonstrations are scarce and reducing transfer across environments and embodiments. We instead formulate behavioral cloning as tabular in-context regression. A tabular foundation model, pretrained to infer predictive relationships from labeled rows, receives compact feature–action pairs constructed from visual and proprioceptive demonstrations. Accumulating the resulting table provides a memory-efficient policy without training a robot-specific action predictor. We also introduce a fixed-memory alternative to table accumulation, using a learned prompt that is updated through prompt distillation, incorporating the current distribution while preserving the predictive content of the preceding prompt. Our method achieves 88.1% final success, compared with 47.5% for the strongest incremental baseline. It also obtains -3.8 forgetting, indicating backward transfer, while all baselines exhibit substantial forgetting.