- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper is about agentic robot systems, where an LLM writes code that mixes scripted motions with calls to a VLA policy. Systems like this can learn over time which steps to give to the VLA, but the VLA itself never gets better, so it ends up capping the whole system. ROBO-COP tries to fix that during deployment. It records everything the robot does in the VLA's format, has LLM judges cut episodes into skills, and keeps the skills that worked, even from episodes that failed overall. A controller decides when there's enough data to fine-tune. Each new checkpoint is tried out before it's kept, and old memory notes like "don't use the VLA for grasping" are flagged so the system gives the new policy another chance.
Critical strengths
I liked this paper. The problem is real and well put: a harness that only routes around a weak policy can never do better than that policy. The point about stale memory was new to me. Once the policy changes, the advice the orchestrator wrote about the old one can quietly keep it from ever trying the new one. The fix is simple and sensible: mark that advice as unverified instead of deleting it.
Several design choices feel like they come from people who have actually run robots. Recording scripted and VLA actions as one stream means the scripted side can teach the policy. Keeping a good grasp from an episode that later failed uses data that would normally be thrown away. Treating each new checkpoint as a candidate, rather than just deploying it, is the right instinct when the robot is real.
The writing is clear, the appendix is thorough (prompts, rules, per-task tables), and the limitations section is honest, including the tasks where ASPIRE wins. The fixed-schedule baseline is a good choice, because it shows that simply fine-tuning on deployment data isn't enough.
Critical weaknesses
My main concern is how much of the gain actually comes from fine-tuning. Per task, ROBO-COP beats the frozen control on five tasks and loses on five, so the average rests on a few large swings. Two of the biggest have nothing to do with the policy update. On Restack top on loose, ROBO-COP never kept a new policy, yet it scores 30 points above the frozen control, with the same VLA. On Recall stack sort, the +54 comes mostly from the frozen control collapsing from 61% to 6% on held-out. Without Recall, the average gain drops to about 4 points. So I'm not yet convinced the improvement comes from updating the policy rather than from how memory happened to evolve in each run.
I'm also not fully convinced the verifier is doing its job. Three of the six tasks where a candidate was adopted end up below the frozen control on held-out (Bananas out of bin, Biggest fruit, and Recover misplaced by a little). On the tasks the VLA can already solve alone, ROBO-COP is a bit below plain π0.5.
The "self-improving flywheel" framing is stronger than what the curves show. After the 30 shared trials, ROBO-COP stays around 73%, while the frozen control slides down to 66%. That looks more like "doesn't get worse" than "keeps getting better". That's still useful, but the abstract should say so.
Smaller points. The idea of keeping skills from failed episodes is never ablated on its own, and the fixed-schedule baseline removes the controller and the verifier at the same time, so I can't tell which one helps. The real-robot gain comes from a single task (7/20 vs 14/20); the other two are identical between methods. And since each task gets its own fine-tuned policy, it would be good to say clearly that the result is a set of per-task specialists rather than a better general policy.
To be clear, I think the system is well designed and the direction is worth pursuing. My score is about the evidence, not the idea.
Questions for authors
- On Restack the policy was never changed, yet ROBO-COP beats the frozen control by 30 points. Where does that come from? And how would you show that the remaining gains come from fine-tuning and not from memory differences? This is the main thing that would change my score.
- Could you compare against curating only from successful episodes, with everything else the same? This is the paper's core data idea, and right now nothing tests it directly. A clear gap here would make the contribution much more concrete.
- On Bananas out of bin and Biggest fruit, the verifier kept candidates that do worse on held-out. What did it see during those windows? I'm trying to understand whether skill-level finish rates are a good enough signal, or whether the margin is just too loose.
- Why does the frozen harness get worse over deployment? If it's memory piling up bad advice, part of ROBO-COP's advantage might come from the memory revision step rather than from fine-tuning. It would be good to know which.
2 comments
Q3: Verifier decisions on BananasOutOfBin and XrBiggestFruit. We examined the evidence available during the verification windows:
- BananasOutOfBin: During verification, the system using the candidate completed 17/25 tasks (68%), compared with a 43% reference from the previous policy's last 30 trials. At the skill level, the candidate completed 55/73 executions; each task trial can contain multiple skill executions. However, the previous policy had only 3 skill executions available for comparison because the orchestrator rarely called it, making the skill-level comparison uncertain.
- XrBiggestFruit: The round-2 candidate was adopted at the end of the 100-trial deployment budget, after 7 verified trials (5/7 = 71% versus a 77% reference).
Verification consumes the same limited deployment budget as task execution, restricting how much evidence the verifier can collect before each decision. Consequently, promising deployment results do not guarantee held-out improvement. We will state this limitation explicitly.
Q4: Why does Frozen degrade, and does the advantage come from memory revision or fine-tuning? Our memory mechanism is imperfect. As context accumulates, useful advice may become harder for the orchestrator to retrieve or apply consistently, while past failures may encourage continued avoidance of the VLA. The deployment logs show declining VLA use in Frozen: from trials 1–30 to trials 66–100, the share of trials calling the VLA falls from 33% to 11% on BananasOutOfBin, from 27% to 0% on RecycleCupClutter, and from 93% to 66% on XrBiggestFruit.
ROBO-COP addresses this through both memory revision and policy adaptation. Memory revision lets the orchestrator reconsider avoidance advice after a checkpoint decision. Fine-tuning distills successful execution segments into the VLA's weights, potentially reducing dependence on the orchestrator recalling and applying textual advice. This does not make the policy immune to forgetting, but provides another way to retain useful behavior.
The checkpoint-swap results support a contribution from fine-tuning: with final memory held fixed, the adopted checkpoint scores higher on RecallStackSort, RecoverMisplaced and XrBiggestFruit. RestackTopOnLoose also improves without retaining a new checkpoint, so we agree that orchestration also contributes to the observed gains.
Thank you for the detailed review! We ran the checkpoint-swap and success-only curation experiments and examined the deployment logs to address your questions.
Q1: Where do the gains come from?
To provide a fair comparison, we reran the held-out evaluations three times with the same GPU model. On RestackTopOnLoose, ROBO-COP trained and evaluated two candidate checkpoints but rejected both, so its final VLA is identical to Frozen's. With the new results, ROBO-COP scores 57.3% versus 46.0% for Frozen, averaged over three evaluation seeds.
Although neither candidate checkpoint was retained, the two systems developed different memories during deployment. ROBO-COP tried candidate checkpoints and marked avoidance advice UNVERIFIED after rejection. Over the last 35 deployment trials, its orchestrator called the VLA in 31 trials, versus 2 for Frozen. The verification stage also provided additional observations of VLA performance that could inform subsequent orchestration.
To examine the contribution of fine-tuning, we held ROBO-COP's final memory fixed and swapped the checkpoint on all six tasks with an adopted update:
The adopted checkpoint scores higher on three tasks (+11, +19 and +7 successes out of 50). The remaining three show smaller observed differences, including two small decreases.
On RecallStackSort, the base policy with Frozen's memory scores 18/50, compared with 19/50 using ROBO-COP's memory. Changing memory therefore adds 2 percentage points, while changing the checkpoint with ROBO-COP's memory fixed adds 22 points in this evaluation.
Q2: Success-only curation. We retrained the adopted round's policy on segments from successful episodes only, keeping the recipe and memory fixed. Compute limited us to three tasks:
ROBO-COP results are averaged over three evaluation seeds; success-only results use one evaluation seed. These results support retaining successful segments from failed episodes on RecallStackSort and RecoverMisplaced, with similar performance on RecycleCupClutter.
(Continued in part 2.)