acceptodds
Under review as a conference paper at ICLR 2027

KEEP: Knowledge-Preserving Expert Distillation for Embodied Policies

10 views

Abstract

We seek to adapt a pre-trained VLA policy for novel manipulation tasks while preserving previously acquired knowledge. This is a non-trivial task, as naive fine-tuning causes catastrophic forgetting, and existing remedies, such as co-training with replayed pre-training data or merging the task experts with the generalist in the weight space, usually trade retention for acquisition. To address these challenges, we introduce KEEP, a method for novel skill acquisition in generalist VLA policies via the distillation of task experts in the flow-field space. The method proceeds by (i) fine-tuning a lightweight LoRA-based task expert, (ii) computing for every training sample a competence gate that compares the expert's and the frozen generalist's own flow-matching errors, and (iii) training a student, initialized from the generalist, to match the expert's velocity field where the expert is competent and the generalist's field everywhere else. We thoroughly evaluate KEEP on a real robot with 5,150 rollouts, covering generalist evaluations, three single-task settings (bottle pouring, plate stacking, breaker box flipping), and a two-task setting with joint and sequential curricula. Every model is scored on its task suites and on a 100-episode generalist suite against the pre-trained policy, LoRA task experts, full fine-tuning with weight merging, and co-training with replayed pre-training data followed by weight merging. KEEP matches the task experts on the novel tasks while keeping generalist performance within the noise of the pre-trained policy, whereas the strongest baseline loses close to fifteen points. In real-to-sim validated simulation, we further ablate every component of KEEP over more than 250,000 episodes, including the gate, the anchor, and the data mixture. Taken together, our results indicate that the flow field, rather than the weight space, is the more favorable place to compose a task expert with a generalist policy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, vision-action-language models, novel task adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kDMwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KEEP, a dual-teacher distillation framework for adapting flow-based Vision-Language-Action policies while preserving generalist capabilities. A competence gate compares the flow-matching errors of a LoRA expert and a frozen generalist, guiding skill injection and knowledge anchoring. Extensive real-robot, REALM simulation, and LIBERO experiments demonstrate improved acquisition–retention trade-offs.

Critical strengths

KEEP introduces a sample-dependent dual-teacher distillation mechanism that dynamically balances novel-skill acquisition and generalist knowledge preservation, offering a meaningful alternative to global weight-space merging. The paper provides extensive empirical validation, including 5,150 real-robot rollouts and over 250,000 simulated episodes. KEEP achieves 81% novel-task success and 59% generalist success, outperforming the strongest merging baseline (57% and 46%). The additional LIBERO evaluations and systematic ablations further strengthen the experimental evidence.

Critical weaknesses

W1. KEEP requires a separately trained LoRA expert and access to replayed pre-training data for each new skill, limiting applicability when historical data are unavailable and increasing adaptation costs.

W2. Limited validation of long-term continual learning. The evaluation covers at most two new skills, and Figure 3 shows substantial forgetting without earlier-task replay, leaving long-term knowledge preservation and scalability to longer task sequences unestablished.

W3. Although flow-matching error is mathematically linked to demonstration-action prediction error, the paper does not directly validate whether it reliably predicts teachers' closed-loop task competence, particularly on student-generated trajectories.

W4. Confounded ablation of anchoring and replay. In Figure 4, removing the anchor term also removes replay data, confounding their contributions and preventing an isolated assessment of the anchoring objective under identical replay settings.

W5. Although Appendix A.7 reports substantially higher training costs for KEEP, the paper lacks acquisition–retention comparisons under matched end-to-end GPU-hour budgets, including expert training and student rollout costs.

Questions for authors

Please refer to Weaknesses W1–W5 for my questions and concerns.

4 comments

Thank you for the detailed review. Three of the weaknesses led to new experiments (W2, W4 and W5), and the others to clarifications in the paper. We answer them in order.

W1: KEEP needs a LoRA expert and replayed pre-training data

Both are indeed properties of the method, and the paper states them as limitations. However, we would like to note two points. First, the replay data is the same data that the co-training baselines use, i.e., a uniform subset of about 5% of DROID. Second, training the LoRA expert accounts for 1.9 of the 5.6 hours that adding a skill takes. The ablation of the co-training ratio (Figure 5, Section 4.3) also shows that the replay data is needed for the anchor term to act on.

Retaining prior skills without any of the original training data is, to our knowledge, still an open problem. The methods we evaluate that do not use replay data, FFT + RETAIN and the FFT models trained with a weight penalty, do not reach KEEP’s combination of novel-task and generalist success (Figure 2, Appendix A.6). In its current form, KEEP uses the replayed actions to compute the competence gate and to form the noised action at which the anchor term is evaluated. Both uses could in principle be replaced: assigning every replayed sample to the base policy performs similarly to the gate (Table 1), and the anchor term can be evaluated on the student’s own actions (Figure 5). Investigating the method without access to the original data is therefore an interesting direction for future work, and we expect world models, as a source of such data, to play a useful role there.

W2: At most two new skills, and long-term preservation is not established

We extended the evaluation to five skills. On LIBERO, we add them one at a time, in three task orders, and re-evaluate the generalist suite and every learned skill after each stage (new Section 4.2, Figure 4, Appendix A.2). KEEP retains generalist success and outperforms the baseline on the novel tasks also in this five-skill setting:

  • Generalist success stays at the level of the base policy through all five skills: 90.9% for KEEP and 91.8% for Co-Train + RETAIN, compared to 92.0% for the base policy.
  • On the novel tasks KEEP outperforms Co-Train + RETAIN at every stage, by 6 to 13 points, and reaches 79.3% after five skills, compared to 71.7% for Co-Train + RETAIN. The novel tasks are evaluated on out-of-distribution splits of increasing difficulty (easy, medium and hard). For the first three skills these are the splits of RETAIN, and for the two additional skills we defined splits in the same style (Appendix A.2). The two methods are within about two points of each other on the easy split, while KEEP outperforms Co-Train + RETAIN by at least 7 points on both the medium and the hard split at every stage, and by up to 23 points on the hard split.

Earlier skills are retained by replaying their demonstrations at every stage. This is the mechanism shown in Figure 3, and [1] found the same for pre-trained VLAs on LIBERO. Since both methods replay the earlier skills, the advantage of KEEP lies in how well each new skill is acquired, which is most visible on the harder splits. However, retention is harder on the real robot than on LIBERO. There, Co-Train + RETAIN, which uses the same replay data as KEEP, loses about twenty points of generalist success compared to the base policy (61%), averaged over the three tasks. KEEP stays within three points of the base policy (Figure 2). In other words, the advantage of KEEP in retaining the original tasks is larger on the real robot than on LIBERO.

W3: Does the flow-matching error predict the teachers’ task competence, particularly on student-generated trajectories?

The competence gate addresses a narrower question than task competence. On each training sample it compares the two teachers and selects the one whose predicted action is closer to the demonstrated action, i.e., the one with the lower flow-matching error. It does not predict how well a teacher performs the task, and it does not use any rollouts. Section 3 now explicitly describes the assumption behind the competence gate.

Whether this comparison selects the right teacher is tested in Table 1. There we replace the competence gate by an oracle that knows the source of each sample and assigns every new-skill sample to the LoRA expert and every replayed sample to the base policy. The competence gate is as good as or better than this oracle (row "Task label" in Table 1) on all three measures: 71.1% vs. 68.2% on pouring, 91.6% vs. 87.4% on stacking, and 59.8% vs. 59.1% on the generalist suite. This shows that comparing flow-matching errors selects the teacher at least as well as knowing the source of each sample.

Regarding student-generated trajectories, the gate is always computed on demonstrations, where the target is known, and not on the student’s own actions. The student’s own actions serve only as the input at which the expert is queried for the inject term, which raises novel-task success from 71.1% to 81.3% without a measurable cost to generalist success (Figure 5).

W4: The ablation of the anchor also removed the replay data

You are right, thank you for pointing this out. We have rerun the ablation. The runs without the anchor term now use the same 80/20 mixture as KEEP. Generalist success is 20.1–21.0% without the anchor term, compared to 59.8% with it on identical data, and novel-task success is 61.1–61.9% without it, compared to 81.3% with it. This shows that the retention comes from the anchor term, and is not provided by the replay data alone. These numbers are now reported in Figure 5 and Section 4.3 of the revised paper (Figure 4 in the original submission) and replace the previously reported numbers.

W5: No comparison under matched end-to-end GPU-hours

The cost is now in the main text (Section 4): about 45 GPU-hours for KEEP, including the training of its LoRA expert and the student’s own ODE integration, compared to about 17 for co-training followed by merging. We also trained R1 baselines, FFT + RETAIN and Co-Train + RETAIN, at a matched budget of 45 GPU-hours (13,000 steps under our recipe R1; denoted RETAIN-L in Appendix A.6 and Figures 16 and 17). The results in simulation after merging are given in the table below. Neither matched-budget run closes the gap on the novel tasks. FFT + RETAIN also retains much less generalist success than KEEP. Co-Train + RETAIN retains 60.7%, compared to 59.8% for KEEP. Please note that simulation overestimates the generalist success of the co-trained baselines by 14 to 25 points relative to the real robot, but not that of KEEP (Appendix A.5). Independently of these runs, continuing the stronger RETAIN recipe (R2) beyond its stopping point lowers both success rates (Appendix A.6, Figure 17), so the gap between the baselines and KEEP on the novel tasks does not come from a smaller training budget.

Method (45 GPU-hours)PouringStackingGeneralist
FFT + RETAIN-L37.880.632.3
Co-Train + RETAIN-L24.773.860.7
KEEP71.191.659.8

Success rate [%] in simulation. The generalist column is the mean over the two single-task models, as in Table 1.

[1] Liu et al., Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning, ICML 2026

Thank you for the additional experiments and detailed cost analysis. I acknowledge that the matched-budget comparisons strengthen the empirical evidence for KEEP. However, my primary concern regarding its practical adaptation cost and scalability remains unresolved. Each new skill still requires training a separate LoRA expert followed by costly student distillation, while preserving previously acquired skills continues to depend on historical data replay.

Recent VLA continual learning methods, such as CLARE (IEEE ROBOTICS AND AUTOMATION LETTERS, 2026), have explored replay-free adaptation. They demonstrate that these dependencies are not universal requirements of continual VLA learning.

Given the importance on efficiency and scalability, I decide to maintain my original rating.

Thank you for the reply, and for acknowledging that the matched-budget comparison strengthens the evidence. We would like to clarify how CLARE relates to our setting, which we also discuss in Section 2 and Appendix A.6.

CLARE shows that sequentially added tasks can be retained without replay data. It does not evaluate the question our paper addresses: retention of the pre-trained policy's own abilities. In both its simulation and real-robot experiments, the policy is evaluated only on the tasks added during continual learning, and no result is reported on the original tasks of the pre-trained policy. In our paper this is the central measure: the base policy reaches 61% on our generalist suite, and KEEP stays within noise of it. CLARE's router always activates an adapter in each adapted layer, so the base policy is never run unmodified, and there is no route back to it. Whether a routing approach can retain the base policy's abilities without access to the pre-training distribution is therefore untested.

CLARE is also demonstrated at a different scale. In simulation, its base policies are compact models of about 200M parameters pre-trained by the authors on LIBERO-90, not a large pre-trained VLA such as π0.5\pi_{0.5}. Our results are on π0.5\pi_{0.5} with 5,150 real-robot rollouts, and in this setting RETAIN is the retention method previously demonstrated, which we compare against in every experiment, including at a matched budget. KEEP also produces a single network with the size and inference cost of the base policy, whereas CLARE adds parameters with every task.

We agree that replay-free retention is an important goal, and the paper states the need for replay data and a LoRA expert as limitations, as well as the decline in novel-task success over longer skill sequences. For a single fixed-size network that keeps the abilities of a large pre-trained VLA, we believe replay-free retention remains an open problem, and we will expand the discussion of CLARE in the paper accordingly.

Reviewer_YTabOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes KEEP, a distillation method that adds a skill to a flow-based VLA to enable the policy to master new skills without forgetting old ones. KEEP works by weighted distillation from two teachers (the frozen base VLA, and a LoRA task-finetuned expert) according to their flow-matching errors. The authors run very extensive evaluation of their proposed method in the real world and in simulation, and find that KEEP significantly beats baselines in skill acquisition and retention, and ablate the important design decisions in KEEP.

Critical strengths
  1. The empirical evaluation in the paper is extremely strong. The authors evaluate their method and baseline on real world robots totalling 5150 rollouts, with 50 evaluation seeds per task. Such rigor is rare in current robotic papers. To compare to the best baseline RETAIN, the authors also experiment with two training settings (R1 & R2), and evaluate both in the real world. The ablations are done in simulation, and they extensively ablate the important design decisions.
  2. The question of “skill acquisition without forgetting” is an important question, and the method the authors present is intuitive and easy to understand, and has strong performance.
Critical weaknesses
  1. The method depends on a weighting of the distillation from two different teacher policies depending on the flow loss. The assumption of the method is that flow matching loss correlates with how “good” the teacher policy is. However, flow matching loss does not necessarily correlate with the performance of a policy, and this is not shown in the paper
Questions for authors
  1. [suggestion] I would suggest keeping the figure captions much shorter and to the point, and leave details in the actual paper text.
  2. In Figure 4, it is shown that smaller temperature (e.g. 0.002) is important for good performance, indicating that usually one of the teachers dominate the distillation loss. However, the first subfigure also shows that you need both distillation losses. Do you know why this is?
  3. For future work, is there another objective you found to be meaningful that can also serve as the gate/weighting for the two distillation losses? That would make this paper applicable to non flow-based VLA as well.
  4. Given that the skill is mostly distilled into the LLM backbone (Figure 5), is there a metric based on the LLM backbone we can use, instead of flow loss? (I’m not asking for new experiments, but rather just curious if the authors have tried anything)
3 comments

Thank you for the review and for the questions, which helped us state more precisely what the competence gate does and does not assume.

Weakness: flow-matching loss need not correlate with how good a policy is

We agree with the general statement, and the competence gate does not rely on it. The gate never judges a teacher in absolute terms. On each training sample it compares the two teachers against the same demonstration and selects the one whose predicted action is closer to the demonstrated action, i.e., the one with the lower flow-matching error. Details are given in Appendix A.1: a teacher's flow-matching error on a sample is the squared distance between the action chunk it predicts and the demonstrated action chunk, divided by t², where t is the flow time of the noised sample. Since both teachers are evaluated on the same noised sample, this factor is the same for both and does not change which of them is closer.

We do not show a direct correlation between flow-matching error and task success, and we do not claim this is the case. What we show is that this per-sample comparison selects the teacher at least as well as knowing whether each sample comes from the new-skill demonstrations or from the replay data. An oracle given the true source of every sample reaches 59.1% generalist success, compared to 59.8% for the competence gate, and 3.5 points less on the novel tasks (Table 1). The competence gate is close to one on over 99% of new-skill samples and close to zero on about four fifths of replayed samples. Section 3 now states the assumption explicitly and points to this test. The extension we describe below, under your question on other objectives, would replace this comparison with a direct estimate of how useful each teacher's action is for the task. This is closer to the notion of a "good" teacher in your comment.

Suggestion: shorter captions

Thank you for this suggestion. The captions of Figures 2, 3 and 6 are roughly half their previous length, the caption of Figure 5 is about a third shorter, and the details are in the text (Figures 5 and 6 were Figures 4 and 5 in the original submission). Figure 1 is redrawn. We have also re-written its caption to focus on what the figure shows.

Question: small temperatures work best, yet both losses are needed. Why?

The competence gate is nearly binary per sample, but different samples are assigned to different teachers. New-skill samples are assigned to the LoRA expert, so the inject term (the first term of Equation 3 in the main paper) acts on them. Replayed samples are mostly assigned to the base policy, so the anchor term acts on them (the second term of Equation 3 in the main paper). Each sample uses essentially one term, but the batch as a whole uses both terms. Removing one of the terms would leave one kind of sample without supervision. Please note that this assignment is not the same as splitting the samples by their source: the gate selects the expert on about a fifth of the replayed samples, and it performs at least as well as an oracle that knows the source of each sample (Table 1).

We have rerun the ablation to demonstrate this directly. Without the anchor term, on the same 80/20 data mixture as KEEP, generalist success falls from 59.8% to 20.1–21.0%. Without the expert's target, fitting the demonstrations instead, novel-task success falls from 81.3% to 47.6–50.6% (Figure 5, Section 4.3). We added a sentence to Section 3 that describes the two cases.

Question: another objective that could serve as the gate, for VLAs that are not flow-based

The competence gate needs only a per-sample score for each teacher against the demonstration. Any training loss could provide such a score. Examples include the token likelihood of the demonstrated action for autoregressive policies, or the regression error for policies with a regression head. In a similar manner, the comparison of the two teachers' scores would then play the same role.

The extension that we find most natural goes one step further and borrows from offline reinforcement learning. Instead of asking which teacher is closer to the demonstration, one would estimate, from the same offline data, how much better one teacher's action chunk is than the other's for completing the task, and weight the two teachers by that estimate. This keeps the structure of the method, does not depend on the action head, and measures the usefulness of an action for the task rather than its distance to a demonstration. The key challenge, however, remains and is similar to the key challenge of offline reinforcement learning. Such estimates must be reliable on actions that are absent from the data.

Question: a metric based on the LLM backbone

We have not tried a metric based on the LLM backbone. However, our finding that the skill is written mostly into the LLM backbone (Figure 6) suggests that it is a natural place to look. Our method relies on the LoRA expert being reliable on the inputs it was trained on, and the backbone could be used to test this directly. For example, one could embed a sample with the backbone and measure how close it lies to the expert's training data. Such a gate would check whether a sample resembles the inputs the expert was trained on, instead of comparing flow-matching errors, and it would not depend on the action head. We do not know whether it would identify the better teacher as well as the comparison of flow-matching errors does. We find this an interesting direction, and we thank you for the suggestion.

I thank the authors for their answers, and maintain my positive review of the paper

Reviewer_XoquOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper is on robotics foundation models, addressing the problem of how to adapt to new individual tasks while preserving generalist competence.

Critical strengths

Experimental results that improve on the state of the art for the settings considered in the paper.

Critical weaknesses

I found the paper difficult to follow, and I cannot recommend for acceptance a paper that I do not fully understand. Acronyms are used without first defining them (e.g. VLA). Terms are used without references or definitions (e.g. π0.5\pi_{0.5} on line 64, x1x_1 and x2x_2 one line 77, "base policy" on line 309 -- I can guess, but I should not have to; clarity is of utmost importance in a scientific paper). There are many examples of phrasing that are not fully clear at the point of use. As one example, on line 55, I am unclear on what "ground truth regression" refers to, and on line 200, what the "fixed budget" refers to. In general, I found myself reading many sentences and paragraphs multiple times just to understand what was being conveyed at a high level. Overall, the writing does not flow well; and information that should be easy to find in a single location is distributed throughout the paper and not easy to find. The paper lacks a background section that explains the basic terms, concepts, and methods (e.g. flow matching) that form the background for this work and would facilitate fast and clear understanding of the scientific contributions. The abstract is unnecessarily long. Figure 1 is a complicated illustration of a relatively simple concept. In this figure, I counted 8 different colors and 6 different text styles. Are all these useful distinctions? Is there really a benefit from using italicized text in this figure except for the mathematical symbols? The photographs are too small to meaningfully add to the figure. In the paper, I do not see a section that clearly and succinctly describes the problem setting, the assumptions made, and the learning objective. Unnecessary abbreviations, such as SR for success rate, degrade the reading experience. Evaluation settings are not always explained or justified (e.g. why the specific 30 second limit on line 278).

I am unclear on the setting explored in the paper and its motivation. My understanding is that the authors are aiming to add capacity to perform a new skill to an existing policy, while retaining the capabilities of this earlier policy. If this is the case, I find it curious that evaluation is done with the addition of only one and two skills, as it would then be important to evaluate the approach as the number of skills continues to grow.

In particular, I am unclear on the role of the new policy learned. Does it replace the generalist base policy as the new generalist base policy? If the setting is the introduction of a few additional skills, then what are the benefits compared to having a generalist policy plus a few specialized policies for each desired new skill? This is especially important given the additional computational complexity of the proposed approach, which should be explicitly discussed in the main paper, given that the proposed algorithm performs additional work on top of LORA fine-tuning for the new task.

Questions for authors
  1. Please explain the problem setting in a self-sufficient single paragraph, clearly and concisely, including the state of the robot prior to training, the objective of the training, and the training data.

  2. Please clearly and concisely motivate the need to scientifically explore this setting.

3 comments

Thank you for the careful review. It made us reread the paper with fresh eyes, and it showed us several places where we had assumed too much of the reader. We have revised the paper along the lines you suggested, and we believe it is now clearer. We answer your two questions first, then the points on the setting, and then list the changes that follow from your comments.

Question 1: the problem setting

We start from a pre-trained generalist robot policy, which we call the base policy. It is a vision-language-action (VLA) model that maps camera images, a language instruction and the robot’s joint state to a short sequence of future actions. In our real-robot experiments this is the π0.5 policy trained on DROID, a large dataset of teleoperated manipulation with a Franka Emika Panda arm. Before adaptation, this policy performs a broad range of everyday manipulation tasks (61% success on our 50-instruction generalist suite) but largely fails on the new skill (2–16% success). We are given three things: the weights of this base policy, about 55 minutes of teleoperated demonstrations of the new skill, and a random subset (about 5%) of the base policy’s original training data. There is no interaction with the environment during training, and no reward. Fine-tuning the base policy on the demonstrations alone yields an expert that performs the new skill well but loses much of its original competence. The goal is a single policy, with the same architecture and inference cost as the base policy, that performs the new skill as well as such an expert, while its success on the original tasks stays at the level of the base policy. We report both quantities for every method. A shorter version of this paragraph now opens Section 3.

Question 2: the motivation

Generalist policies are deployed in settings their pre-training data does not cover, so adding skills after pre-training is an important capability. However, the standard ways of doing it can significantly damage the policy. In our real-robot experiments, LoRA fine-tuning lowers generalist success from 61% to 7–33%, and the strongest existing remedy, weight merging (interpolating the weights of the fine-tuned policy with those of the base policy), still loses about fifteen points while acquiring less of the skill. The scientific question is whether a skill can be added to the base policy at no (or limited) cost to what it already knows. Our results indicate that this is possible when the base policy and an expert fine-tuned on the new skill are composed in the space of their outputs rather than their weights. We believe this is because weight merging applies one coefficient to all inputs, so improving the new skill lowers success on the original tasks, whereas our method chooses for each training sample which of the two models to follow (more details in Appendix A.1).

More than two skills

Thank you for pointing that out. We agree that this was missing. We now add five skills one at a time on the LIBERO benchmark, in three task orders, and re-evaluate the generalist suite and every learned skill after each stage, i.e., after each newly added skill (new Section 4.2 and Figure 4). Generalist success stays at the level of the base policy through all five skills (90.9% for KEEP, compared to 92.0% for the base policy), and our approach (KEEP) outperforms the weight-merging baseline on the novel tasks at every stage, by 6 to 13 points. Novel-task success declines for both methods as skills accumulate, which we state as a limitation.

The role of the new policy

Yes, it replaces the base policy. In the skill chains described above, the policy produced at each stage is the starting point of the next stage and the teacher that preserves the earlier skills. Section 3 now says this explicitly.

Why not the base policy plus one expert per skill

Keeping the base policy and adding one expert per skill avoids forgetting, but it needs a mechanism that decides which model to run for each instruction, and each expert performs only its own task. A single policy does not need such a mechanism, and in our experiments it matches or exceeds the expert on the expert’s own task (72%, 94% and 78% for KEEP, compared to 62–86% for the LoRA experts on the robot). We believe this is because KEEP trains the single policy on the expert's predicted velocities, which are a lower-variance target than the raw demonstrations, and supervises it on its own generated actions (Sections 4.1 and 4.3). The introduction now states this alternative and why we do not pursue it.

Computational cost

The main text now has a paragraph on computational cost (Section 4). Adding one skill takes about 45 GPU-hours with KEEP, including the training of the LoRA expert it builds on. This compares to about 17 GPU-hours for co-training followed by weight merging. This cost is paid once, at training time, and the resulting policy has the inference cost of the base policy. Training the weight-merging baselines for the same 45 GPU-hours does not close the gap on the novel tasks (Appendix A.6).

Clarity of the presentation

Thank you again for the detailed suggestions on the presentation. Below is how we addressed each of them in the revised manuscript:

  • VLA is defined in the abstract and at first use in the introduction.
  • π0.5 (your line 64) is introduced as our base policy, with its reference. The symbols on line 77 are removed.
  • “Base policy” (line 309) is defined in the introduction and again in Section 3, and we now use this name consistently for that model.
  • “Ground-truth regression” (line 55) is now “regression onto the demonstrated actions”. “Fixed budget” (line 200) is replaced by a plain statement of what the two weights do.
  • Section 3 opens with a problem-setting paragraph and a preliminaries paragraph on flow-matching policies. The assumption behind the competence gate (our per-sample choice between the expert and the base policy) is stated in Section 3.
  • The abstract is shorter. Figure 1 is redrawn with fewer colors and text styles, no italics outside mathematical symbols, and larger photographs.
  • “Success rate” is spelled out in the text. The abbreviation remains only in figures and tables, and it is defined once in Section 4.
  • The 30-second limit is justified: it lies above the average demonstration length of the slowest task (22 seconds).
Reviewer_XoquOpenReview ↗

Thank you for the clarifications.

I thank the authors for their answers, the problem setting and the motivation are now clear. I am taking a careful look at the revised paper. I have no further questions to the authors.

Comments on the paper

1 comment

We thank the three reviewers for their time and for the detailed feedback. It led to new experiments and to a substantial revision of the paper. A revised PDF is uploaded, with all changes marked in blue.

Summary of the reviews

All three reviewers noted that the results improve on existing methods in the settings we study. Reviewers YTab and kDMw also pointed to the extent of the evaluation: 5,150 real-robot rollouts, the comparison with RETAIN under two training settings, the LIBERO study, and the ablations of the design choices. Reviewer YTab considered the question important and the method intuitive and easy to understand. Reviewer kDMw described the sample-dependent weighting of two teachers as a meaningful alternative to global weight-space merging.

The weaknesses raised fall into five groups:

  1. the evaluation adds at most two skills (Xoqu, kDMw);
  2. the ablation of the anchor term also removed the replay data (kDMw);
  3. the training cost is higher and was not discussed in the main text (Xoqu, kDMw);
  4. it is not shown that the flow-matching error identifies the better teacher (YTab, kDMw);
  5. the paper was hard to follow (Xoqu, with a suggestion on captions from YTab).

We address each below. Replies to the individual reviews follow under each review.

1. Longer skill sequences

We now add five skills one at a time on LIBERO, in three task orders, and re-evaluate the generalist suite and every learned skill after each stage, i.e., after each newly added skill (new Section 4.2, Figure 4, Appendix A.2). Generalist success stays at the level of the base policy through all five skills (90.9% for KEEP and 91.8% for Co-Train + RETAIN, compared to 92.0% for the LIBERO base policy). On the novel tasks KEEP outperforms Co-Train + RETAIN at every stage, by 6 to 13 points, and reaches 79.3% after five skills, compared to 71.7% for Co-Train + RETAIN. We believe the better performance can be attributed to the fact that weight merging pulls every newly fine-tuned model back toward the previous policy with a single coefficient shared by all inputs, which gives up part of each new skill, whereas KEEP selects the teacher for each sample (Appendix A.1). Novel-task success declines for both methods as skills accumulate, which we discuss as a limitation and which remains an open problem.

2. Anchor and replay in the ablation

We reran the anchor-free runs on the same 80/20 mixture as KEEP. Without the anchor, generalist success is 20.1–21.0%, compared to 59.8% with the anchor on identical data. This shows that the retention comes from the anchor term and is not provided by the replay data alone. In the revised PDF, these runs replace the earlier ones in Figure 5 and Section 4.3 (Figure 4 in the original submission), which used novel-task data only.

3. Training cost

The cost is now discussed in Section 4. Adding one skill takes about 45 GPU-hours with KEEP, including its LoRA expert, compared to about 17 GPU-hours for co-training followed by merging. This cost is paid only once, at training time, and the resulting policy has the same inference cost as the base policy. FFT + RETAIN and Co-Train + RETAIN runs given the same 45 GPU-hours reach 59.2% and 49.2% novel-task success in simulation, compared to 81.3% for KEEP, at 32.3% and 60.7% generalist success, compared to 59.8% for KEEP (Appendix A.6, Figure 17).

4. Flow-matching error in the competence gate

The competence gate does not need the flow-matching error to predict task success in general. It compares the two teachers on the same demonstration and selects the one that predicts the demonstrated action better. We now state this assumption in Section 3 and validate it against an oracle that knows the true source of every sample, which performs about the same (59.1% generalist success, compared to 59.8% for the competence gate, and 3.5 points lower novel-task success, Table 1). The gate is always computed on demonstrations, not on student-generated actions.

5. Clarity

We revised the text to improve its clarity and accessibility. Section 3 now opens with a self-contained problem setting and a preliminaries paragraph. Terms are defined at first use, and the pre-trained model is called the base policy throughout. The abbreviation SR is spelled out in the text, the 30-second limit is justified, and the assumption behind the competence gate is stated in Section 3. The abstract and the captions of Figures 2, 3, 5 and 6 are shorter, and Figure 1 is redrawn.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.