acceptodds
Under review as a conference paper at ICLR 2027

Co-Evolving Robot Orchestrators and Policies through Deployment

Abstract

Vision-language-action (VLA) policies trained on large datasets are capable within their training domains, yet they still fail to generalize to the variety of situations a robot meets in real-world deployment. Agentic robot systems complement the policy with a vision-language model (VLM) orchestrator that learns when to call the policy, how to instruct it, and when to use scripted skills instead. However, because the harness is built around a frozen policy that has limited language steerability, the orchestrator can avoid the policy's failures but never overcome them. The policy becomes the bottleneck of the whole system. Fine-tuning the policy can remove this bottleneck, but updating it alone decouples it from an orchestrator tuned to its old behavior. We propose Robo-COP, in which the orchestrator and policy co-evolve during deployment. Robo-COP curates skill demonstrations from its own executions, fine-tunes the policy when this data can address recurring failures, and adopts each new policy only after it improves the skills it was trained for. Across ten simulated RoboLab tasks, Robo-COP raises mean held-out success from 64.8% to 73.8% over the same harness with a frozen policy, while fine-tuning on a fixed schedule without verification reaches only 65.8%. On three real-world tasks, Robo-COP raises held-out success from 38.3% to 50.0%. Robo-COP turns deployment into a self-improving flywheel in which robots learn by doing, with each improvement in execution producing better data for the next round of learning. Videos and code are available at https://robo-cop.pages.dev/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Embodied Agents, Self-improving Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GXNtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is about agentic robot systems, where an LLM writes code that mixes scripted motions with calls to a VLA policy. Systems like this can learn over time which steps to give to the VLA, but the VLA itself never gets better, so it ends up capping the whole system. ROBO-COP tries to fix that during deployment. It records everything the robot does in the VLA's format, has LLM judges cut episodes into skills, and keeps the skills that worked, even from episodes that failed overall. A controller decides when there's enough data to fine-tune. Each new checkpoint is tried out before it's kept, and old memory notes like "don't use the VLA for grasping" are flagged so the system gives the new policy another chance.

Critical strengths

I liked this paper. The problem is real and well put: a harness that only routes around a weak policy can never do better than that policy. The point about stale memory was new to me. Once the policy changes, the advice the orchestrator wrote about the old one can quietly keep it from ever trying the new one. The fix is simple and sensible: mark that advice as unverified instead of deleting it.

Several design choices feel like they come from people who have actually run robots. Recording scripted and VLA actions as one stream means the scripted side can teach the policy. Keeping a good grasp from an episode that later failed uses data that would normally be thrown away. Treating each new checkpoint as a candidate, rather than just deploying it, is the right instinct when the robot is real.

The writing is clear, the appendix is thorough (prompts, rules, per-task tables), and the limitations section is honest, including the tasks where ASPIRE wins. The fixed-schedule baseline is a good choice, because it shows that simply fine-tuning on deployment data isn't enough.

Critical weaknesses

My main concern is how much of the gain actually comes from fine-tuning. Per task, ROBO-COP beats the frozen control on five tasks and loses on five, so the average rests on a few large swings. Two of the biggest have nothing to do with the policy update. On Restack top on loose, ROBO-COP never kept a new policy, yet it scores 30 points above the frozen control, with the same VLA. On Recall stack sort, the +54 comes mostly from the frozen control collapsing from 61% to 6% on held-out. Without Recall, the average gain drops to about 4 points. So I'm not yet convinced the improvement comes from updating the policy rather than from how memory happened to evolve in each run.

I'm also not fully convinced the verifier is doing its job. Three of the six tasks where a candidate was adopted end up below the frozen control on held-out (Bananas out of bin, Biggest fruit, and Recover misplaced by a little). On the tasks the VLA can already solve alone, ROBO-COP is a bit below plain π0.5.

The "self-improving flywheel" framing is stronger than what the curves show. After the 30 shared trials, ROBO-COP stays around 73%, while the frozen control slides down to 66%. That looks more like "doesn't get worse" than "keeps getting better". That's still useful, but the abstract should say so.

Smaller points. The idea of keeping skills from failed episodes is never ablated on its own, and the fixed-schedule baseline removes the controller and the verifier at the same time, so I can't tell which one helps. The real-robot gain comes from a single task (7/20 vs 14/20); the other two are identical between methods. And since each task gets its own fine-tuned policy, it would be good to say clearly that the result is a set of per-task specialists rather than a better general policy.

To be clear, I think the system is well designed and the direction is worth pursuing. My score is about the evidence, not the idea.

Questions for authors
  1. On Restack the policy was never changed, yet ROBO-COP beats the frozen control by 30 points. Where does that come from? And how would you show that the remaining gains come from fine-tuning and not from memory differences? This is the main thing that would change my score.
  2. Could you compare against curating only from successful episodes, with everything else the same? This is the paper's core data idea, and right now nothing tests it directly. A clear gap here would make the contribution much more concrete.
  3. On Bananas out of bin and Biggest fruit, the verifier kept candidates that do worse on held-out. What did it see during those windows? I'm trying to understand whether skill-level finish rates are a good enough signal, or whether the margin is just too loose.
  4. Why does the frozen harness get worse over deployment? If it's memory piling up bad advice, part of ROBO-COP's advantage might come from the memory revision step rather than from fine-tuning. It would be good to know which.
2 comments

Thank you for the detailed review! We ran the checkpoint-swap and success-only curation experiments and examined the deployment logs to address your questions.

Q1: Where do the gains come from?

To provide a fair comparison, we reran the held-out evaluations three times with the same GPU model. On RestackTopOnLoose, ROBO-COP trained and evaluated two candidate checkpoints but rejected both, so its final VLA is identical to Frozen's. With the new results, ROBO-COP scores 57.3% versus 46.0% for Frozen, averaged over three evaluation seeds.

Although neither candidate checkpoint was retained, the two systems developed different memories during deployment. ROBO-COP tried candidate checkpoints and marked avoidance advice UNVERIFIED after rejection. Over the last 35 deployment trials, its orchestrator called the VLA in 31 trials, versus 2 for Frozen. The verification stage also provided additional observations of VLA performance that could inform subsequent orchestration.

To examine the contribution of fine-tuning, we held ROBO-COP's final memory fixed and swapped the checkpoint on all six tasks with an adopted update:

TaskBase π0.5Adopted checkpoint
RecallStackSort19/5030/50
RecoverMisplaced24/5043/50
XrBiggestFruit31/5038/50
RecycleCupClutter16/5017/50
RecycleSort227/5025/50
BananasOutOfBin28/5026/49

The adopted checkpoint scores higher on three tasks (+11, +19 and +7 successes out of 50). The remaining three show smaller observed differences, including two small decreases.

On RecallStackSort, the base policy with Frozen's memory scores 18/50, compared with 19/50 using ROBO-COP's memory. Changing memory therefore adds 2 percentage points, while changing the checkpoint with ROBO-COP's memory fixed adds 22 points in this evaluation.

Q2: Success-only curation. We retrained the adopted round's policy on segments from successful episodes only, keeping the recipe and memory fixed. Compute limited us to three tasks:

TaskSuccess-onlyROBO-COP
RecallStackSort30.0%56.0%
RecoverMisplaced76.0%88.0%
RecycleCupClutter50.0%49.3%

ROBO-COP results are averaged over three evaluation seeds; success-only results use one evaluation seed. These results support retaining successful segments from failed episodes on RecallStackSort and RecoverMisplaced, with similar performance on RecycleCupClutter.

(Continued in part 2.)

Q3: Verifier decisions on BananasOutOfBin and XrBiggestFruit. We examined the evidence available during the verification windows:

  • BananasOutOfBin: During verification, the system using the candidate completed 17/25 tasks (68%), compared with a 43% reference from the previous policy's last 30 trials. At the skill level, the candidate completed 55/73 executions; each task trial can contain multiple skill executions. However, the previous policy had only 3 skill executions available for comparison because the orchestrator rarely called it, making the skill-level comparison uncertain.
  • XrBiggestFruit: The round-2 candidate was adopted at the end of the 100-trial deployment budget, after 7 verified trials (5/7 = 71% versus a 77% reference).

Verification consumes the same limited deployment budget as task execution, restricting how much evidence the verifier can collect before each decision. Consequently, promising deployment results do not guarantee held-out improvement. We will state this limitation explicitly.

Q4: Why does Frozen degrade, and does the advantage come from memory revision or fine-tuning? Our memory mechanism is imperfect. As context accumulates, useful advice may become harder for the orchestrator to retrieve or apply consistently, while past failures may encourage continued avoidance of the VLA. The deployment logs show declining VLA use in Frozen: from trials 1–30 to trials 66–100, the share of trials calling the VLA falls from 33% to 11% on BananasOutOfBin, from 27% to 0% on RecycleCupClutter, and from 93% to 66% on XrBiggestFruit.

ROBO-COP addresses this through both memory revision and policy adaptation. Memory revision lets the orchestrator reconsider avoidance advice after a checkpoint decision. Fine-tuning distills successful execution segments into the VLA's weights, potentially reducing dependence on the orchestrator recalling and applying textual advice. This does not make the policy immune to forgetting, but provides another way to retain useful behavior.

The checkpoint-swap results support a contribution from fine-tuning: with final memory held fixed, the adopted checkpoint scores higher on RecallStackSort, RecoverMisplaced and XrBiggestFruit. RestackTopOnLoose also improves without retaining a new checkpoint, so we agree that orchestration also contributes to the observed gains.

Reviewer_KB4XOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents ROBO-COP, a deployment-time loop that updates a VLM coding orchestrator, including its episodic memory, together with the VLA policy (pi0.5) it calls. Executions that mix scripted primitives and VLA calls are logged as a single action stream in the policy's action space. Text and video judges split this stream into labeled skill segments, and successful segments are kept even when they come from failed episodes. An LLM controller decides when there is enough data to LoRA-fine-tune. Each candidate is then tested over a window of deployment trials. It is adopted only if task success and per-skill finish rates stay within a margin of the incumbent and an LLM reviewer agrees. After every adoption or reversion, memory entries that steer the orchestrator away from the VLA are marked as unverified.

Critical strengths

Prior harnesses assume a fixed policy. The paper points out that memory written for an old checkpoint (for example, "grasp with scripted motion, the VLA fails") will stop the system from ever finding out that an updated policy works. It also notes that a reverted candidate leaves the same kind of misleading memory behind. Anyone building agentic robot systems with an updatable policy will run into this, and marking stale advice rather than deleting it is a simple, reasonable fix.

Critical weaknesses
  1. The headline gain can't be separated from run-to-run variance. The paper's core claim is that co-evolution, and the policy update in particular, produces the improvement. A single seed, where memory alone can swing a task by 30 points, does not support that claim.
  2. The individual mechanisms are not isolated, and the case for verification is thin. There are no ablations for memory revision, for keeping segments from failed episodes, or for the video judge, even though these are listed as contributions. And There is no direct evidence that the 5 rejected candidates were actually worse. The support is the indirect fixed-schedule comparison plus one forced-training diagnostic on a real task.
Questions for authors
  1. For each task with an adopted update, what is held-out success when the final memory is held fixed and only the checkpoint is swapped (base vs. adopted)?
  2. How does each reverted candidate score on held-out trials under the same memory?
  3. Do you have an ablation without UNVERIFIED marking, and one that curates only from successful episodes? After a memory revision, how often did the orchestrator actually retry the VLA on a skill it had been avoiding?
2 comments

Thank you for the detailed review! We ran the checkpoint swap (Q1), evaluated rejected candidates (Q2), and ran the success-only part of Q3.

W1: Run-to-run variance. We now report three held-out evaluation repeats per cell. Mean success across the 10 tasks is 70.3 ± 1.5% for ROBO-COP versus 67.7 ± 1.6% for Frozen; ± is the standard deviation of the 10-task mean across repeats. These are evaluation repeats of one adaptation run per task, not independent adaptation runs. We could not complete another set of adaptation runs, each requiring 100 deployment trials plus training per task, during rebuttal. We will state this limitation explicitly.

Q1: Checkpoint swap with final memory held fixed.

For all six tasks with an adopted update, holding ROBO-COP's final memory fixed:

TaskBase π0.5Adopted checkpoint
RecallStackSort19/5030/50
RecoverMisplaced24/5043/50
XrBiggestFruit31/5038/50
RecycleCupClutter16/5017/50
RecycleSort227/5025/50
BananasOutOfBin28/5026/49

The adopted checkpoint scores higher on three tasks (+11, +19 and +7 successes out of 50). The remaining three show smaller observed differences, including two slightly decreases. On RecallStackSort, the base policy with Frozen’s memory scores 18/50, compared with 19/50 using ROBO-COP’s memory. Thus, changing memory adds 2 percentage points, while changing the checkpoint with ROBO-COP’s memory fixed adds 22 points in this evaluation.

Q2: Rejected candidates under the same memory. We report three selected comparisons below with ROBO-COP's final memory and compared them with the policies retained at the corresponding decisions.

Task and roundRejected candidateRetained policy
RestackTopOnLoose r121/5031/50 (base)
RecycleSort2 r128/5027/50 (base)
XrBiggestFruit r133/5031/50 (base)

On RestackTopOnLoose, the rejected candidates scored below the retained policy (21/50 versus 31/50). On RecycleSort2 and XrBiggestFruit, the rejected candidates scored slightly higher (28/50 versus 27/50 and 33/50 versus 31/50). From the additional experiments, we could observe that the verifier generally makes reasonable decision on the checkpoint adoption.

W2 / Q3: Mechanism analysis.

Success-only curation. We retrained the adopted round's policy on segments from successful episodes only, keeping the recipe and memory fixed. Compute limited us to three tasks:

TaskSuccess-onlyROBO-COP
RecallStackSort30.0%56.0%
RecoverMisplaced76.0%88.0%
RecycleCupClutter50.0%49.3%

ROBO-COP results are averaged over three evaluation seeds; success-only results use one evaluation seed.

Memory revision. Its purpose is to let the orchestrator retry the VLA after a checkpoint decision. On RecoverMisplaced, RecycleCupClutter, RestackTopOnLoose and RecycleSort2, the orchestrator called the VLA in 0, 0, 0 and 1 of the 10 trials before the first candidate was tried, versus 8, 9, 9 and 4 of the 10 trials after the decision and revision.

Thank you for the detailed rebuttal and additional experiments. However, I still have major concerns regarding the statistical significance and component-level justification of the method. With three evaluation repeats, the mean gain over the frozen baseline shrinks from +9.0 pp to +2.6 pp on a single adaptation run, leaving the main co-evolution claim largely within noise. Moreover, half of the adopted checkpoints fail to beat the base policy under fixed memory, two of the three evaluated rejected candidates slightly outperform the retained policy, and end-to-end ablations for UNVERIFIED marking and the video judge remain missing. So my decision to reject stands, though I am raising my rating from 1 to 2 in recognition of the added evidence.

Reviewer_Z9x5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a framework for updating a VLA policy and its orchestrator’s memory during deployment. It reports an increase in mean held-out success from 64.8% to 73.8% in simulation and from 38.3% to 50.0% on three physical tasks.

I find the motivation convincing, and the results are promising. My main concern is that it is difficult to tell which components drive the improvement and how consistent the gains would be across runs.

Critical strengths
  • The paper identifies a practical issue: an orchestrator may continue avoiding certain skills based on past failures, even after an updated policy becomes capable of performing them.

  • The treatment of partial success is sensible. Successful subskills can be retained from failed episodes, and the candidate policy is not credited for work completed by fallback actions.

  • The implementation and experimental results are described clearly and in useful detail.

Critical weaknesses
  • The contribution of the individual components remains unclear. For example, removing policy-dependent memory revision or training only on successful episodes would help establish how much these design choices contribute.

  • Each method has only one adaptation run per task. Since the collected data and memory evolve throughout deployment, the results may be sensitive to the particular learning history. Repeated runs would make the conclusions more convincing.

Questions for authors
  • Do the improvements hold across independent adaptation runs?

  • How does the system behave over a longer sequence of policy updates? In particular, how well does it keep memory consistent with the deployed checkpoint and retain skills acquired by earlier policy versions?

1 comment

Thank you for the careful review and for recognizing the stale-memory problem!

W1: Contribution of individual components. We ran the success-only curation ablation and examined deployment logs for policy-dependent memory revision.

  • Success-only curation. We retrained the adopted round's policy on segments from successful episodes only, with the same recipe and memory. Given the time limit, we evaluated three tasks and will report results for the remaining adopted tasks in the final version.
TaskSuccess-onlyROBO-COP
RecallStackSort30.0%56.0%
RecoverMisplaced76.0%88.0%
RecycleCupClutter50.0%49.3%

ROBO-COP results are averaged over three evaluation seeds; success-only results use one evaluation seed.

  • Policy-dependent memory revision. Its purpose is to let the orchestrator retry the deployed VLA after a checkpoint decision rather than continue relying on avoidance advice written for an earlier policy. On RecoverMisplaced, RecycleCupClutter, RestackTopOnLoose and RecycleSort2, the orchestrator called the VLA in 0, 0, 0 and 1 of the 10 trials before the first candidate was tried, versus 8, 9, 9 and 4 of the 10 trials after the decision and memory revision.

W2 / Q1: Consistency across runs. We now report three held-out evaluation repeats per cell. ROBO-COP leads in every repeat: 70.3 ± 1.5% versus 67.7 ± 1.6% mean success. Here, ± is the standard deviation of the 10-task mean across repeats. Each adaptation run requires 100 deployment trials plus training per task, and we could not complete a second full set during rebuttal. We will state this limitation explicitly in the revision.

Q2: Longer sequences of policy updates. Several tasks underwent multiple candidate rounds. RecycleCupClutter adopted a round-1 checkpoint and then replaced it with a round-2 checkpoint. RecoverMisplaced, RecycleSort2 and XrBiggestFruit rejected a round-1 candidate and adopted a round-2 candidate. After each adoption or reversion, memory entries that steer the orchestrator away from the VLA are marked UNVERIFIED so the deployed checkpoint can be tried again. Before adopting a new checkpoint, the verifier also re-tests skills adopted in earlier rounds against its retention criterion. On RecycleCupClutter, the round-2 checkpoint finished 29 of 49 executions of the round-1 skill and passed this check. Longer horizons for more than 3 updates remain future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful feedback and for recognizing the practical importance of stale memory when policies change, the value of retaining successful skills from failed episodes, and the thoughtful system design. Below, we summarize the new results; detailed comparisons and deployment-log analyses appear in our individual responses.

1. Repeated held-out evaluation. We repeated the held-out evaluations using three evaluation seeds and the same GPU model. The updated per-task results are:

TaskFrozenROBO-COPΔ
BananasInBinTwoMore69.3%92.7%+23.3
BananasOutOfBin82.7%53.7%−29.0
KitThreeBinsBlocks96.7%84.0%−12.7
RecallStackSort33.3%56.0%+22.7
RecoverMisplaced90.0%88.0%−2.0
RecycleCupClutter41.3%49.3%+8.0
RecycleSort236.0%51.3%+15.3
RestackTopOnLoose46.0%57.3%+11.3
XrBiggestFruit85.3%72.7%−12.7
XrExceptOrange96.0%98.0%+2.0
Mean ± SD67.7 ± 1.6%70.3 ± 1.5%+2.6

Results are averaged over three evaluation seeds. Δ is in percentage points, calculated before rounding; ± is the standard deviation of the 10-task mean across repeats. ROBO-COP scores higher on 6 of 10 tasks, with uneven gains across tasks. These are evaluation repeats of one adaptation run per task, not independent adaptation runs. We could not complete another full set of adaptation runs during rebuttal and will state this limitation explicitly.

2. Separating checkpoint and memory effects. We held ROBO-COP's final memory fixed and swapped the checkpoint on all six tasks with an adopted update:

TaskBase π0.5Adopted checkpoint
RecallStackSort19/5030/50
RecoverMisplaced24/5043/50
XrBiggestFruit31/5038/50
RecycleCupClutter16/5017/50
RecycleSort227/5025/50
BananasOutOfBin28/5026/49

These are single-evaluation results. The adopted checkpoint scores higher on three tasks, with smaller observed differences on the remaining three, including two small decreases.

On RestackTopOnLoose, ROBO-COP trained and evaluated two candidate checkpoints but retained neither, so its final VLA is identical to Frozen's. Nevertheless, the two systems developed different memories during deployment: ROBO-COP's verification trials provided additional observations of VLA performance, and avoidance advice was marked UNVERIFIED after rejection. Over the last 35 deployment trials, ROBO-COP's orchestrator called the VLA in 31 trials, versus 2 for Frozen. These differences help explain how ROBO-COP scores 57.3% versus 46.0% despite having the same final checkpoint, with both results averaged over three evaluation seeds.

3. Success-only curation. We retrained the adopted round's policy using only segments from successful episodes, keeping the recipe and memory fixed. Compute limited us to three tasks:

TaskSuccess-onlyROBO-COP
RecallStackSort30.0%56.0%
RecoverMisplaced76.0%88.0%
RecycleCupClutter50.0%49.3%

ROBO-COP results are averaged over three evaluation seeds; success-only results use one evaluation seed. These results support retaining successful segments from failed episodes on RecallStackSort and RecoverMisplaced, with similar performance on RecycleCupClutter.

4. Verification under a limited deployment budget. We added held-out comparisons of rejected candidates and examined the evidence available during adoption decisions. The reported RestackTopOnLoose comparison supports the rejection decision (21/50 for the candidate versus 31/50 for the retained policy), while rejected candidates on RecycleSort2 and XrBiggestFruit score slightly higher than the retained policies. These comparisons provide a retrospective check rather than a guarantee of reliable selection.

Verification consumes the same limited deployment budget as task execution, restricting how much evidence can be collected before each decision. Our response to GXNt details the verification windows on BananasOutOfBin and XrBiggestFruit and explains why promising deployment evidence does not guarantee held-out improvement.

We will incorporate the updated results and analyses into the revision, clarify the contributions of policy adaptation and orchestration, and explicitly state the limitations of short verification windows and evaluation repeats without independent adaptation runs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.