acceptodds
Under review as a conference paper at ICLR 2027

In-context Imitation Learning under Task Drift with Tabular Foundation Models

Abstract

Behavior cloning is typically formulated as an offline problem: a policy is trained on a fixed demonstration dataset and deployed without further adaptation. In practice, changes in object configurations, initial states, goals, or other task conditions can substantially alter the input distribution and degrade performance. We refer to this setting as task drift and study how a policy can adapt incrementally from small sets of new demonstrations without retraining from scratch. In-context learning provides a natural mechanism for such adaptation without updating model weights. Existing robotic in-context learners, however, acquire this capability through large-scale, robot-specific training, limiting their applicability when demonstrations are scarce and reducing transfer across environments and embodiments. We instead formulate behavioral cloning as tabular in-context regression. A tabular foundation model, pretrained to infer predictive relationships from labeled rows, receives compact feature–action pairs constructed from visual and proprioceptive demonstrations. Accumulating the resulting table provides a memory-efficient policy without training a robot-specific action predictor. We also introduce a fixed-memory alternative to table accumulation, using a learned prompt that is updated through prompt distillation, incorporating the current distribution while preserving the predictive content of the preceding prompt. Our method achieves 88.1% final success, compared with 47.5% for the strongest incremental baseline. It also obtains -3.8 forgetting, indicating backward transfer, while all baselines exhibit substantial forgetting.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Imitation Learning, Behavioral Cloning, Domain Adaptation, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_i1UNOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies incremental imitation learning under task drift, where a robotic policy must adapt to changing task conditions while retaining performance on previously encountered distributions. The authors formulate behavior cloning as tabular in-context regression using a pretrained TabPFN model. Visual observations are encoded using a frozen pretrained visual encoder, compressed through partial least squares (PLS) using expert action supervision, and combined with proprioceptive states to form feature-action tables.

The paper proposes two adaptation strategies: accumulating demonstrations in the model's context and maintaining a fixed-size learnable prompt updated through incremental prompt distillation. The latter uses the preceding prompt to preserve knowledge while incorporating demonstrations from the current distribution.

Experiments on six MetaWorld tasks with three sequential distribution shifts show strong performance. The incremental prompt-tuning method achieves 88.1% final success, compared with 47.5% for the strongest incremental baseline, while reporting a forgetting score of -3.8.

Critical strengths
  1. The authors use a pretrained tabular foundation model as an action predictor, which is a conceptually simple and potentially useful alternative to conventional robot-specific policy training. The method separates visual representation learning from the action prediction mechanism and exploits the ability of TabPFN to infer input-output relationships from labeled examples. This formulation is particularly attractive in settings where repeatedly training a policy after every distribution shift is undesirable.
  2. The fixed-size prompt-distillation approach provides a reasonable mechanism for retaining previously acquired information without storing historical raw demonstrations. The proposed objective combines prediction on new demonstrations with constraints derived from the preceding prompt, keeping the TabPFN predictor frozen.
Critical weaknesses
  1. The incremental experiments compare the proposed prompt-distillation method against R3M, BeT, ACT, and Diffusion Policy, which are sequentially fine-tuned using only the current distribution. However, the proposed method explicitly preserves information from previous distributions through a 512-row feature-action prompt, whereas the baselines do not receive comparable replay or persistent-memory mechanisms. Therefore, the large performance gap may reflect the availability of persistent memory rather than a specific advantage of tabular foundation models or the proposed distillation objective. The paper also lacks comparisons with simple nonparametric or tabular regression alternatives, such as nearest-neighbor regression, kernel ridge regression, or similarly compact exemplar-memory methods. These are particularly important because the central formulation reduces imitation learning to regression over low-dimensional feature-action pairs. Without such comparisons, it is difficult to identify the contribution of TabPFN itself. Additionally, there is no direct ablation removing the previous-prompt distillation term in Equation (11), which makes it difficult to quantify the importance of the proposed retention mechanism.
  2. The experiments are restricted to six MetaWorld tasks, each containing only three stages of geometric task drift. Although the results are encouraging, they do not establish that the method remains effective over substantially longer sequences, larger distribution shifts, or different robot embodiments. Moreover, the complete system is not independent of robot-specific learning resources: it relies on a visual encoder pretrained on robot-manipulation data and an action-supervised PLS projection fitted to the initial demonstrations. The latter remains frozen across subsequent task shifts, potentially limiting adaptation when newly relevant visual features are absent from the original representation. Finally, the reported negative forgetting of -3.8 +- 3.6 is relatively modest compared with its variability. The results clearly support reduced forgetting, but provide weaker evidence for robust backward transfer than the paper's wording suggests.
Questions for authors
  1. Can the authors compare against conventional policies equipped with a comparable persistent-memory budget, such as replay from stored feature-action examples, and against simple regression approaches using the same frozen representation? Additionally, what is the effect of removing the previous-prompt distillation term in Equation (11)? These comparisons would clarify how much of the improvement comes from TabPFN, persistent memory, and the proposed distillation mechanism, respectively.
  2. How does performance change when the number of drift stages increases beyond three? Since the PLS projection is fitted only on the initial distribution and then frozen, can the authors evaluate whether substantial later shifts introduce action-relevant visual information that the initial projection fails to preserve?
1 comment

We thank the reviewer for the careful assessment and constructive suggestions. We have added several experiments and revised the manuscript to address the concerns.

Critical Weaknesses

  1. We thank the reviewer for these suggestions and have added experiments accordingly. We clarify that the fixed-size prompt is not a replay memory: it contains learnable rows analogous to the optimized parameters of a conventional policy. Thus, both our incremental method and the baselines are replay-free. To strengthen the comparison, we additionally equipped every baseline with EWC, which explicitly regularizes parameter updates to reduce forgetting. We also ablated the prompt-distillation term in our method. The table reports success rates averaged across all tasks.
MethodMean(%)Final(%)Forgetting(%)
R3M66.947.559.9
BeT54.037.262.4
ACT45.034.346.7
DP35.530.935.7
R3M-EWC73.569.226.0
BeT-EWC53.445.036.1
ACT-EWC42.134.639.4
DP-EWC38.433.131.4
Ours (without prompt distillation)78.676.18.9
Ours (prompt distillation)84.088.1−3.8

EWC reduces forgetting and improves performance; for example, R3M's final success increases from 47.5% to 69.2%, while forgetting decreases from 59.9 to 26.0 points. However, substantial forgetting remains across all EWC baselines, and their performance remains considerably weaker than ours. Removing prompt distillation from our method increases forgetting and reduces final success from 88.1% to 76.1%. This confirms that prompt distillation contributes substantially to performance.

We also added nonparametric regression methods that retain and reuse demonstrations. All methods use exactly the same PLS features and proprioceptive state, isolating the effect of the action-prediction mechanism.

MethodMean(%)Final(%)Forgetting(%)
Kernel regression78.084.4−4.4
VINN75.981.6−7.3
Gaussian process82.385.4−4.5
Ours (full context)84.793.4−9.6

The nonparametric methods do not exhibit forgetting because they retain all demonstrations. Our method nevertheless achieves the strongest performance using the same representation and demonstrations. This supports a benefit from the pretrained tabular model beyond reusing prior examples.

  1. We appreciate the reviewer's comment regarding broader evaluation. As explained in the appendix, each drift stage must occupy a distinct region and remain unsolvable by policies trained on preceding stages. Because the robot workspace is finite, additional stages eventually overlap with previously covered regions and no longer constitute meaningful new drift, limiting the sequence to three stages. We also require a simulator with a scripted expert that remains reliable under drift; Meta-World was the only simulator that met this requirement. Nevertheless, all compared methods encounter the same validated distributions, and our method retains a substantial advantage. To examine whether the frozen projection restricts adaptation, we added a PLS-dimension ablation and a variant without PLS.
PLS dimensionMean(%)Final(%)Forgetting(%)Trainable parametersInference time(ms)
1676.980.9−2.214,336176
3284.088.1−3.822,528182
6483.388.4−4.938,912243
12874.686.0−8.171,680251
25673.075.6−2.1137,216279
51260.970.4−1.6268,288563
1,02435.154.7−7.4530,4321,029
No PLS38.127.835.9792,5761,536
No PLS (full context)92.596.6−2.0—1,666

Very small projection dimensions, such as 16, discard useful information, whereas 32–64 dimensions perform well. Performance can decline at larger PLS dimensions, suggesting overfitting due to the increased number of fitted parameters. The joint full-context variant avoids this overfitting issue when PLS is removed because the pretrained TabPFN predictor is used directly without optimization. However, using the full visual representation increases storage and computation: inference takes 1,666 ms, which is 9.15 times slower than the 182 ms required with the 32-dimensional PLS representation. We therefore use PLS because it maintains strong performance while being considerably more efficient.

Regarding backward transfer, achieving positive backward transfer remains challenging in continual learning, where most methods primarily aim to reduce forgetting. We agree that the observed negative forgetting of −3.8±3.6-3.8 \pm 3.6 percentage points provides limited evidence of backward transfer. We have therefore tempered the corresponding claim in the revised manuscript to avoid overstating this result.

Questions

Question 1 is addressed by the EWC, prompt-distillation, and nonparametric regression experiments above. Question 2 is addressed by the drift-construction explanation and PLS ablation above.

We hope these additional experiments and clarifications address the reviewer’s concerns. If so, we respectfully ask the reviewer to consider updating their score.

Reviewer_E75COpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual imitation learning under geometric distribution shifts within a fixed task, reformulating behavioral cloning as tabular in-context regression to separate general-purpose prediction using a frozen tabular foundation model (TabPFN) from task-specific adaptation through demonstration-derived feature–action tables supplied as context. As a memory-efficient variant, it maintains a fixed-size table of learnable feature–action rows (a prompt), optimized on new demonstrations with a distillation objective to preserve prior behavior. Across six MetaWorld tasks, the method achieves 88.1% final success without retaining past demonstrations, compared with 47.5% for the strongest sequential fine-tuning baseline, while exhibiting positive backward transfer. Under joint training with all accumulated demonstrations, the method remains competitive with the baselines.

Critical strengths

I think the core insight of this paper is a practical bridge between general-purpose in-context prediction and robot control, using an off-the-shelf tabular foundation model with demonstration-derived feature–action tables. This decoupling offers a promising direction for compact adaptive behavior models, where task-specific knowledge can be updated through an editable data context rather than retraining the predictor, with the potential to benefit from future advances in tabular foundation models.

Critical weaknesses

The evaluation is limited to MetaWorld, and the analysis provides limited insight into when the pretrained tabular predictor offers advantages and what constrains its generalization, making broader applicability difficult to assess.

Questions for authors

In the joint setting of Table 1, how would the proposed method compare with approaches that store and reuse demonstrations, such as VINN or DINOBot? I would be interested in understanding what the pretrained tabular model contributes beyond retrieval or trajectory replay when inferring actions from demonstrations.

I would be interested in how scaling each model component affects performance. Have authors explored larger pretrained visual encoders or tabular foundation models, and whether any generalization limits in robotic manipulation remain as these components scale?

It would be helpful to compare training compute with the baselines under both the joint and incremental settings in Table 1. Could these comparisons highlight potential efficiency gains from the proposed framework?

I wonder how learning prompt rows compares with selecting representative demonstration rows at each phase. Separately, how does the row-based formulation account for temporal dependencies within demonstrations? Have authors also considered learning and appending 512 new rows per phase, allowing the memory budget to grow?

I am curious how much of the difficulty comes from learning through visual representations. Would using privileged state observations from MetaWorld instead of image features help clarify this?

As a future direction, how might the table formulation connect to multi-task learning, reinforcement learning, or partially observable control through additional metadata, such as task semantics, rewards, or temporal information? It would be interesting to discuss the opportunities and limitations of these connections.

Addressing these questions could help clarify the broader applicability of the proposed approach and strengthen its impact.

4 comments

We thank the reviewer for the thoughtful questions. We also appreciate the reviewer’s recognition of our work and overall positive assessment. We address the concerns below:

Critical Weaknesses

We thank the reviewer for raising this limitation. As explained in the appendix, constructing valid drift sequences requires configurable task geometry, a scripted expert that remains reliable, and distinct distributions not covered by earlier stages. Meta-World was the only simulator we examined that met these requirements. Within this controlled setting, our method substantially outperforms the baselines under both incremental and joint adaptation. We have also added the experiments below to clarify the benefits of our proposed method.

Questions

  1. We appreciate this suggestion and added kernel regression, VINN, and Gaussian-process baselines. All methods use the same PLS visual features and proprioceptive state, isolating the effect of the prediction mechanism. The table reports success rates averaged across all tasks.
MethodMean(%)Final(%)Forgetting(%)
Kernel regression78.084.4−4.4
VINN75.981.6−7.3
Gaussian process82.385.4−4.5
Ours (full context)84.793.4−9.6

As expected, these methods do not exhibit forgetting because they retain all demonstrations. Nevertheless, our method achieves the highest mean and final success using the same demonstrations and state representation. This result indicates that the pretrained tabular foundation model provides benefits beyond retaining or retrieving previous examples.

  1. We considered testing different tabular foundation models but found no directly applicable alternatives, as most are limited to classification. Support for tabular foundation models capable of regression is comparatively recent: the original TabPFN supported only classification, while later versions extended the framework to regression. We use TabPFN-3 in the paper because it provides the regression interface required for continuous action prediction. For visual encoding, we evaluated alternatives including DINOv2 and observed comparable performance. We selected R3M because it is pretrained on robot-manipulation data and is substantially smaller, providing a better efficiency-performance tradeoff for our setting. Exploring larger models remains an interesting direction for future work.

  2. We thank the reviewer for this suggestion. We have included training times. The values report the total incremental training time across all three stages of the basketball task.

MethodTotal training time
R3M52 min
BeT11 min
ACT100 min
DP40 min
Ours (prompt tuning)53 min
Ours (full context)2 min

The optimization time of our prompt-based method is comparable to that of the other baselines, largely reflecting differences in network size and trainable parameter count. The full-context variant requires no optimization and is therefore substantially faster. Its reported time consists entirely of extracting visual embeddings, fitting the PLS projection, constructing the feature-action rows, and placing them in the TabPFN context.

  1. Our formulation assumes a controlled Markov process in which the current observation and proprioceptive state contain the information required to select an action. The policy therefore learns a Markovian mapping from state to action and does not explicitly model temporal dependencies. For the prompt, we maintain 512 rows across all phases because our objective follows the standard bounded-memory continual learning setting, where the memory should not grow with the number of adaptation steps. Appending 512 rows per phase is possible, but it corresponds to an expanding-memory setting similar to the full-context variant.

  2. We appreciate this suggestion and added an experiment in which we remove the visual encoder and provide Meta-World's privileged state representation directly to TabPFN. The table reports success rates averaged across all tasks.

Input representationMean(%)Final(%)Forgetting(%)
Image features + proprioception84.793.4−9.6
Privileged state96.398.1−1.8

With privileged state information, the method achieves near-ceiling performance. This indicates that a substantial portion of the difficulty comes from recovering state information from visual observations, rather than from learning the state-to-action mapping.

  1. To extend the work to multi-task learning or partially observable control, the context table could be augmented with metadata such as task identifiers, language semantics, or observation histories. Task metadata could support shared multi-task contexts, while history features could extend the approach to partially observable control. We will discuss these opportunities and limitations for future work.

We hope these additional experiments and clarifications address the reviewer’s concerns and more clearly establish the contribution of the proposed method.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns and questions.

I would first like to clarify the basis of my initial positive assessment. I view this work within the broader scope of continual imitation learning, particularly memory-based, non-parametric approaches to adaptation. Relative to existing in-context imitation learning methods [1], I find value in bridging an off-the-shelf tabular foundation model to robotic imitation learning with little or no environment-specific training of the action predictor, and in evaluating this approach under continual adaptation.

From this perspective, I see the potential of maintaining and updating demonstration memory with little or no policy optimization. This motivated my suggestion to compare the full-context variant against VINN and other non-parametric regression methods. The additional comparisons in Response 1 and the training-time results in Response 3 are valuable and help clarify this advantage.

However, several concerns remain insufficiently addressed.

My primary concern is not the choice of Meta-World itself but the criteria that select the task drifts. The formal definition requires only that the trajectory distribution changes, yet the benchmark accepts a candidate drift only when the previous full-context policy achieves at most 10% success and the scripted expert achieves at least 80%. The rebuttal does not explain why these thresholds define a valid drift. Because the first criterion filters candidates by the failure of the proposed method's own full-context policy, the benchmark retains only the shifts that this policy cannot solve without new demonstrations, and I cannot assess how well the resulting evaluation represents task drift more broadly.

Existing continual imitation learning benchmarks such as LIBERO [2] already provide controlled spatial variations with human-collected demonstrations. LIBERO-Spatial keeps a common manipulation objective while varying the spatial configuration, which appears consistent with the paper's formal definition of task drift. The proposed method needs only demonstrations to adapt, so the requirement for a reliable scripted expert follows from the authors' drift construction procedure and not from the method. I would therefore find the evaluation more convincing if the authors justified the selection criteria and evaluated the method on a compatible subset of an existing benchmark such as LIBERO-Spatial, or explained what prevents such an evaluation.

I appreciate the clarification regarding proprioceptive state and the controlled Markov process formulation. This assumption is reasonable at the level of the underlying state. However, extending it to visual observations requires more careful discussion. Even Meta-World tasks may be partially observable from the policy's visual input. It is unclear whether the accumulated context can implicitly compensate for missing state information, or whether the proposed method simply learns a reactive state-action mapping. Since the table does not explicitly preserve temporal dependencies within trajectories, I would appreciate a clearer account of what information is available at inference time and how partial observability affects the formulation.

I am also not convinced by the response concerning model selection. The claim that no directly applicable alternatives exist because most tabular foundation models are limited to classification is difficult to justify, given publicly available regression-capable tabular foundation models [3, 4]. I would suggest acknowledging the absence of these comparisons instead of claiming that no applicable alternatives exist.

Furthermore, I find the response to the final question disappointing, as it largely restates the suggested extensions without explaining how the table would carry temporal information, task conditioning, or memory adaptation. Related studies outside imitation learning already apply tabular foundation models to control by constructing state-action value or policy-return prediction tables [5, 6], and I believe contrasting their formulations with the proposed feature-action context would clarify what the robotic imitation setting specifically requires.

Despite these remaining concerns, I still believe this work can make a useful contribution by connecting memory-based in-context adaptation with the continual imitation learning community. The additional experiments in the rebuttal strengthened the empirical case, and the efficiency of the full-context approach remains a clear strength. I believe the authors can address most of the gaps above through revised discussion without changing the proposed approach, and I hope these comments help them do so.

My assessment remains marginally positive and I therefore maintain my weak accept recommendation, although I would understand a rejection decision given the concerns above. I thank the authors for their careful engagement throughout the discussion.

References

[1] Fu et al. In-Context Imitation Learning via Next-Token Prediction (ICRT). ICRA 2025.

[2] Liu et al. LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning. NeurIPS 2023.

[3] Ma et al. TabDPT: Scaling Tabular Foundation Models on Real Data. NeurIPS 2025.

[4] Zhang et al. Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models. NeurIPS 2025.

[5] Schiff et al. Gradient Free Deep Reinforcement Learning With TabPFN. 2025.

[6] Ou and Dümbgen. Can Tabular Foundation Models Guide Exploration in Robot Policy Learning? 2026.

We thank the reviewer for the careful follow-up and for identifying several points that require clearer qualification.

The thresholds are operational benchmark-construction criteria rather than part of the formal definition of task drift. The first criterion ensures that each new geometric distribution presents a meaningful adaptation problem. If a policy with access to all preceding demonstrations already performs well on a candidate drift, that stage provides limited evidence about learning from newly supplied demonstrations. We therefore require near-zero prior success as a conservative indication that the preceding data do not adequately cover the drift distribution. The 80% scripted-expert threshold serves a separate purpose: all evaluated methods require demonstrations from the new distribution, so the expert must solve the configuration reliably enough to provide valid training data. We also manually inspected the accepted distributions to confirm that they represent distinct geometric configurations. We will clarify these motivations and the scope of the resulting benchmark.

We appreciate the suggestion to consider LIBERO. Although it shares a common manipulation objective, its standard benchmark represents each spatial variation as a distinct language-conditioned task with its own instruction and task identity. It therefore evaluates continual learning across a sequence of related tasks, whereas our setting keeps the task fixed and changes only its configuration. Using LIBERO directly would consequently require task or language conditioning and would extend our formulation toward multi-task continual learning. Nevertheless, LIBERO is a valuable complementary setting, and we will clarify this distinction and consider extending the method to language-conditioned continual imitation learning.

We thank the reviewer for pointing out the regression-capable models in [3,4]. We were not aware that these models provided applicable regression interfaces, and we will acknowledge that we have not yet compared against them. We will examine these models to determine how they compare with the TabPFN model used in our experiments.

A partially observable extension could concatenate the current features with a fixed-size encoding of the observation-action history. In preliminary experiments, we explored an echo-state network, whose recurrent reservoir summarizes trajectory history without task-specific parameter training. Its reservoir state was concatenated with the PLS and proprioceptive features before constructing the table. We did not observe a meaningful improvement, likely because the evaluated Meta-World tasks are well approximated by reactive control given the available multi-view observations and proprioception; the scripted expert itself also selects actions without trajectory history. Temporal windows or action chunking offer other possible extensions. These mechanisms may become important in settings with stronger perceptual aliasing or genuinely hidden state.

For multi-task learning, a task identifier can be included as an additional table column. The tabular foundation model could then condition its prediction on this attribute and identify demonstrations associated with the queried task. However, a shared context would grow as tasks are added; a more efficient alternative is to retrieve only the corresponding demonstration rows and place them in context. Because the predictor remains frozen, switching tasks would require context retrieval rather than retraining.

For reinforcement learning, the table would require a different prediction target, such as an action-value estimate or return, and rows could additionally contain rewards or next states. This differs from our imitation learning formulation, in which expert actions provide direct supervised targets; reinforcement learning introduces exploration, credit assignment, and potentially bootstrapped targets. We thank the reviewer for highlighting [5,6] and will contrast their action-value and policy-return prediction tables with our feature-action context to clarify what is specific to robotic imitation learning.

We thank the reviewer again for engaging thoughtfully with our rebuttal and for raising these important points, which have helped us clarify the scope, assumptions, and limitations of our work.

Reviewer_dT5xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies incremental imitation learning under task drift, evaluated through geometric changes to the conditions of a fixed task. It converts demonstrations into feature–action tables using frozen R3M visual features, an action-supervised PLS projection fitted on the initial demonstrations, and proprioception, then uses a frozen TabPFN model to predict actions. It proposes both full-context accumulation and a fixed-size learned prompt. During incremental adaptation, the prompt is optimized using new demonstrations and a distillation objective over the previous prompt. Experiments on six MetaWorld tasks, each with three successive distributions, show strong retention and higher aggregate final success than the evaluated behavioral-cloning baselines under both incremental and joint training.

Critical strengths
  1. The tabular formulation enables practical reuse of a pretrained predictor for robot control. Converting demonstrations into feature–action tables allows an off-the-shelf TabPFN model to predict actions without robot-specific training of the action predictor’s weights. The full-context variant incorporates new demonstrations simply by appending rows, avoiding repeated predictor optimization. Its 93.4% final success in the joint setting demonstrates that this unconventional formulation can produce effective policies on the evaluated tasks.

  2. The results demonstrate impressive resistance to catastrophic forgetting. Under sequential adaptation, the proposed method achieves 88.1% final success compared with 47.5% for the strongest incremental baseline. Its average success-based forgetting is −3.8 ± 3.6 percentage points, supporting strong retention with a small average improvement on earlier distributions, while the evaluated baselines exhibit substantial degradation. Although these experiments concern changing conditions within individual tasks, the retention demonstrated in that setting is a substantive empirical strength.

  3. The learned prompt combines adaptation with compact, fixed-size context storage. Prompt distillation incorporates new demonstrations while carrying forward historical knowledge through a 512-row learned context, without retaining historical trajectories. The incremental method performs comparably to the strongest baseline trained on all accumulated demonstrations, making the storage–performance tradeoff practically appealing. The prompt-size analysis further shows how context capacity affects performance.

Critical weaknesses
  1. The continual-learning comparison needs a baseline explicitly trained to reduce forgetting. The retention results are impressive, but the incremental baselines simply fine-tune on new demonstrations. A useful control would apply prediction distillation to the R3M–MLP baseline: retain a frozen previous policy, then fit the new expert actions while penalizing changes from the previous policy’s predictions on those same observations. This adapts the Learning without Forgetting principle without requiring historical demonstrations, although additional model memory should be reported. An ablation removing prompt distillation would also help distinguish the benefit of the tabular formulation from that of its retention objective.

  2. The broader imitation-learning claims require substantially more evidence. The joint-training results are promising, but evaluation covers only six simulated MetaWorld tasks, with separate task-specific policies and three geometric distributions per task. Although the appendix explains the constraints behind this benchmark construction, robustness under longer sequences and more varied distribution shifts remains unresolved. These experiments support the stated same-task adaptation setting, but broader claims about the advantages of tabular imitation learning should be qualified accordingly.

  3. The frozen visual representation may restrict future adaptation. The method compresses visual features from 1,536 to 32 dimensions using only the initial demonstrations, then freezes that projection. If later conditions require visual distinctions discarded by this projection and unavailable through the remaining inputs, prompt updates alone cannot recover them. Without representation-dimension ablations or targeted evaluation of newly relevant visual information, the experiments do not establish how well this design supports broader drift. The storage reduction should therefore be presented alongside its potential information-loss tradeoff.

Questions for authors
  1. Experimental request: How much does the previous-prompt distillation term contribute to retention? Please compare prompt adaptation with and without this term. A complementary control would add prediction distillation to the R3M–MLP baseline, matching the previous policy’s outputs on current observations while learning the new expert actions, with its additional memory cost reported.

  2. Clarification and experimental request: Do the evaluated shifts require visual information that was not relevant to the initial demonstrations? Please also evaluate sensitivity to the dimensionality of the frozen PLS projection. This would help clarify whether the representation can support newly required distinctions.

  3. Experimental request: Can the authors evaluate longer adaptation sequences or more varied shifts within the same task? Such evidence would strengthen the continual-adaptation claim beyond the three-stage geometric sequences. A shared multi-task evaluation would additionally support broader imitation-learning claims; otherwise, those claims should remain limited to the demonstrated setting.

3 comments

We thank the reviewer for the thoughtful and constructive review and for recognizing the strengths of our work. We address the raised concerns below:

Critical Weaknesses

  1. We appreciate the request for a baseline explicitly designed to reduce forgetting and have included the results. While prediction distillation can be applied to a regression policy such as R3M, it is not directly applicable to all architectures, particularly Diffusion Policy, which defines a generative process. We therefore used EWC, which can be applied consistently to all four baselines and penalizes changes to parameters identified as important to the previous policy. We also ablated the prompt-distillation term in our method. The table reports success rates averaged across all tasks.
MethodMean (%)Final (%)Forgetting (%)
R3M66.947.559.9
BeT54.037.262.4
ACT45.034.346.7
DP35.530.935.7
R3M-EWC73.569.226.0
BeT-EWC53.445.036.1
ACT-EWC42.134.639.4
DP-EWC38.433.131.4
Ours (without prompt distillation)78.676.18.9
Ours (prompt distillation)84.088.1−3.8

EWC generally reduces forgetting and improves performance; for example, R3M's final success increases from 47.5% to 69.2%, while forgetting decreases from 59.9 to 26.0 percentage points. However, substantial forgetting remains across all EWC baselines, and their performance remains considerably weaker than ours. Removing prompt distillation from our method increases forgetting from −3.8 to 8.9 percentage points and reduces final success from 88.1% to 76.1%. This confirms that prompt distillation contributes substantially to performance.

  1. We appreciate the reviewer's comment regarding broader evaluation. As explained in the appendix, constructing valid drift sequences requires a simulator with configurable geometric distributions, a scripted expert that remains reliable, and shifts that are not already covered by earlier distributions, which limited the benchmark to six tasks and three stages. Nevertheless, our method provides a substantial improvement over the baselines in both the incremental and joint settings. We have also added comparisons with nonparametric methods that store and reuse demonstrations, reported in our response to Reviewer E75C. These results further demonstrate the benefit of our proposed method.

  2. We appreciate this concern and added an ablation over the PLS (partial least squares) dimension, together with a variant that removes PLS and uses the complete 1,536-dimensional visual representation.

PLS dimensionMean (%)Final (%)Forgetting (points)Trainable parametersInference time (ms)
1676.980.9−2.214,336176
3284.088.1−3.822,528182
6483.388.4−4.938,912243
12874.686.0−8.171,680251
25673.075.6−2.1137,216279
51260.970.4−1.6268,288563
1,02435.154.7−7.4530,4321,029
No PLS38.127.835.9792,5761,536
No PLS (full context)92.596.6−2.0—1,666

Very small projection dimensions, such as 16, discard useful information, whereas 32–64 dimensions perform well. Performance can decline at larger PLS dimensions, suggesting overfitting due to the increased number of fitted parameters and the limited number of trajectories. The joint full-context variant avoids this overfitting issue when PLS is removed because the pretrained TabPFN predictor is used directly without any optimization. However, using the full visual representation substantially increases storage and computation: inference takes 1,666 ms, which is 9.15 times slower than the 182 ms required with the 32-dimensional PLS representation. We therefore use PLS because it maintains strong performance while being considerably more efficient.

Questions

Questions 1 and 2 are addressed by the EWC, prompt-distillation, and PLS ablations above. Regarding Question 3 on longer sequences or more varied shifts within the same task, we deliberately retain only shifts for which policies trained on previous stages achieve near-zero success, ensuring that each stage introduces a meaningful new distribution. Extending the sequence with overlapping distributions would increase its length without creating a genuine adaptation challenge. We have clarified this construction constraint and limited our claims to continual adaptation across distinct distributions of the same task.

We hope these additional experiments and clarifications address the reviewer’s concerns. If so, we respectfully ask the reviewer to consider updating their score.

I want to thank the authors for the detailed response to my concerns.

  1. Regarding needing a baseline that is explicitly designed to reduce forgetting, I find the EWC comparison to be very informative as it still falls short of the proposed methodology. Solid point proven.
  2. Regarding the need for broader evaluation on other benchmarks, I understand the difficulties, but still find it underwhelming that no further benchmarking (even short horizon tasks of other levels of difficult/dexterity e.g.) were shown to truly quantify and scope out the meaningfulness of the contribution.
  3. Regarding PLS dimensions, I appreciate the attempt to quantify the sensitivity of PLS dimensions given current task, but I think not being able to answer the impact of higher PLS dimensions beyond performance on existing benchmarks inference time is also slightly underwhelming, as one would expect the performance to truly shine given longer, harder task scenarios.

Overall, I still find value in this work's current contribution as a first-stab at using tabular architectures to improve continuous learning, so I would understand if the decision is to accept this paper. However, with the points above, I would like to maintain my current assessments.

We thank the reviewer for the thoughtful follow-up and for recognizing that the additional EWC comparison addresses the continual learning baseline concern.

We appreciate the reviewer’s suggestion to conduct more extensive experiments and agree that a broader experimental evaluation would further strengthen the paper. As discussed in our rebuttal, however, adaptation to task drift remains relatively underexplored, and existing benchmarks are not designed specifically for constructing controlled sequences of task drift. We therefore developed a benchmark that allowed us to isolate this setting and evaluate all methods under the same validated distributions. Within this benchmark, our method consistently outperforms all baselines. Moreover, our contribution is not limited to preventing forgetting: our approach also outperforms established imitation-learning baselines when all demonstrations are available, demonstrating the benefit of the tabular formulation beyond continual adaptation. We hope the reviewer will take these practical limitations into consideration when assessing the contributions of our work and that the necessarily limited experimental scope will not overshadow the novelty and potential significance of the proposed approach.

We thank the reviewer again for their careful consideration of our work.

Comments on the paper

2 comments

Dear Reviewers,

As the discussion period is coming to an end, I would greatly appreciate it if you could take a moment to review our rebuttal. Please feel free to ask any further questions or raise any remaining concerns. If you find our responses satisfactory, I would also be grateful if you could consider updating your scores accordingly.

Thank you very much for your time and thoughtful feedback!

Authors

Hi everyone, please acknowledge the authors’ rebuttal with a brief reply, even if your assessment is unchanged (if you already did or are about to do so, please ignore this message).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.