acceptodds
Under review as a conference paper at ICLR 2027

CAPEX: Efficiently Distilling Foundation Model Behavior into Deployable Robot Policies through Experience-Adaptive Reasoning

Abstract

Robot learning has largely relied on human-teleoperated demonstrations to acquire effective learnable behaviors. However, human-operated data collection processes can be unintuitive, difficult to scale, and inherently asynchronous. We explore an alternative: distilling physical behavior from general-purpose multimodal foundation models into deployable robot policies by using the foundation model itself as an autonomous demonstrator. While sufficiently capable models can generate successful zero-shot manipulation trajectories, repeatedly invoking them during physical execution is slow and expensive, limiting their utility as scalable data generators. As a solution, we introduce CAPEX, an experience-conditioned demonstration collection framework that uses execution experience from previous attempts to adapt how frequently the foundation model must observe, reason, and replan. We evaluate across RoboCasa tasks and on physical Franka and bimanual YAM-arm platforms, measuring task success, model calls, token usage, collection time, and cost. We further train Diffusion Policy and ACT on matched sets of human-teleoperated and foundation-model-generated demonstrations to evaluate the downstream learning value of autonomously collected data. We find that CAPEX increases the number of successful demonstrations by 4.3x while reducing the cost per successful demonstration by 80%. Policies trained on CAPEX-generated data approach the performance of those trained on matched human demonstrations; with longer training, this gap largely closes for policies trained from scratch. These results suggest that foundation models can serve as scalable sources of reusable robot experience. Project page: https://capex-paper.github.io/

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLM, robot learning, autonomous data collection, imitation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yLtbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Collecting robot demonstrations through human teleoperation is difficult to scale, while using foundation models as autonomous demonstrators requires slow and costly model calls. The paper proposes a framework that uses a frozen multimodal foundation model to generate waypoint plans and collect demonstrations for training robot policies. CAPEX uses past execution outcomes to calibrate waypoint confidence and decide when to replan, while summaries and successful examples from previous attempts guide future plans. On RoboCasa, CAPEX increases collection success from 12.5% to 53.6% and reduces the cost per successful demonstration by 80%. Experiments in simulation and on real robots show that policies trained from scratch on CAPEX demonstrations achieve success rates close to those trained on human demonstrations, while fine-tuned Diffusion Policy performs worse with CAPEX data.

Critical strengths

Using foundation models for demonstration collection is an important and timely problem. Adapting planning and execution based on accumulated experience is also a promising approach to improving collection efficiency.

Critical weaknesses
  • Demonstration quality warrants further evaluation beyond task success and aggregate collection efficiency. The project videos appear to show unnatural pauses and non-smooth transitions between waypoints, which may affect downstream policy learning. Quantifying motion smoothness and systematically comparing demonstration completion times against human demonstrations would strengthen the evaluation.
  • Despite the improvement over the baseline, the reported 6.6 minutes per successful demonstration still indicates low collection throughput.
  • Although the learned calibrator includes distance to a VLM-selected 3D point, that point's semantic role may vary across calls, and the calibrator lacks direct visual features or explicit object geometry. This raises concerns about its generality to more challenging tasks. For example, in contact-rich peg insertion, reachability depends strongly on the peg and hole geometry and their relative alignment.
  • The related-work discussion omits CaP-X [1], (accepted at ICML 2026) a relevant framework for foundation-model-driven robot manipulation that could also support demonstration collection.

[1] Fu L, Yu J, El-Refai K, et al. CaP-X: A framework for benchmarking and improving coding agents for robot manipulation[J]. arXiv preprint arXiv:2603.22435, 2026.

Questions for authors
  • What advantages does CAPEX offer for demonstration collection compared to CaP-X?
  • Could Table 1 additionally report tokens per successful demonstration? This would complement monetary cost with a measure independent of changing API prices and applicable to locally deployed models such as Qwen.
5 comments
AuthorsOpenReview ↗

Response to Reviewer yLtb (Part 1/2)

Dear Reviewer yLtb,

Thank you for the concrete suggestions and for recognizing using foundation models for demonstration collection as an important and timely problem. Your review led us to compare CAPEX demonstrations with human demonstrations, evaluate the calibrator on unseen tasks, discuss CaP-X, and report token usage per successful demonstration.

W1: Demonstration quality

"Quantifying motion smoothness and systematically comparing demonstration completion times against human demonstrations would strengthen the evaluation."

Thank you for suggesting this comparison. We find that CAPEX demonstrations take longer and are slightly less smooth than human demonstrations, but have shorter end-effector paths. More importantly, policies trained from scratch on CAPEX demonstrations achieve comparable success to those trained on human demonstrations.

We compared 193 successful CAPEX demonstrations with human demonstrations from the same initial robot and scene states (App. G.5):

Median over 193 pairsCAPEXHuman
End-effector path length1.35 m1.58 m
Completion time15.0 s9.5 s
Action jerk0.170.13

Action jerk is the mean L1 norm of the second difference of translational arm commands. We hypothesize that the longer completion times are largely due to the executor moving more slowly between waypoints and pausing to check whether each waypoint was reached. These pauses also provide the execution outcomes used to train our calibrator. Although they make the trajectories slower, the shorter path lengths suggest that CAPEX still follows relatively direct spatial routes.

To test whether these differences affect downstream learning, we trained policies on matched quantities of CAPEX and human demonstrations from identical initial states and evaluated them in our submission. After 100k training steps, ACT trained from scratch achieves 50% success with either dataset, while Diffusion Policy trained from scratch achieves 45% with CAPEX demonstrations versus 47% with human demonstrations (Sec. 4.5, Fig. 4). These results suggest that, despite being slower and slightly less smooth, CAPEX demonstrations can provide comparable training supervision without requiring human teleoperation.

W2: Collection throughput

"Despite the improvement over the baseline, the reported 6.6 minutes per successful demonstration still indicates low collection throughput."

We agree that CAPEX's current collection throughput is slower than human teleoperation. However, the main advantage is that these demonstrations can be collected without a human operator controlling the robot, at an API cost of USD 1.83 per successful demonstration including failed attempts. We investigated the inference bottleneck and found that 4.1 of the 6.6 minutes (62%) are spent waiting for model responses. At the provider's current fastest API tier of approximately 300 output tokens/s, we project that collection time would decrease to 3.9 minutes per demonstration (App. G.4).

CAPEX's capabilities are still limited by the foundation model. On the same benchmark, the open Qwen model achieves only 1.4% success with CAPEX compared with 53.6% for GPT-6 Astra. We expect further improvements in foundation models to significantly improve the demonstration yield and throughput.

AuthorsOpenReview ↗

Response to Reviewer yLtb (Part 2/2)

W3: Generality of the calibrator

"that point's semantic role may vary across calls, and the calibrator lacks direct visual features or explicit object geometry"

You are right that the grounded point may have different semantic roles across calls. Our calibrator only uses the distance from a waypoint to the last located 3D point, without assigning it a semantic label. We tested whether this feature is necessary by removing it, and found that AUROC only decreases from 0.754 to 0.740 (App. G.3). This suggests that the calibrator does not depend strongly on the semantic consistency of the grounded point.

More importantly, the calibrator learns from each task's own execution outcomes rather than relying on a geometric model transferred from another task. To test whether this generalizes to new tasks, we evaluated the same calibrator design on six tasks never used during development, fitting its weights separately for each task from prior execution outcomes. We find that calibration reduces expected calibration error from 0.15 to 0.03 and improves AUROC from 0.61 to 0.73. It also improves both metrics across every benchmark task family, including doors, drawers, buttons, and knobs (App. G.3). We provide the full unseen-task evaluation in our response to dGg6 (W3 & Q3).

For your peg-insertion example, we hypothesize that repeated stalls near contact would teach the calibrator to shorten its execution horizon and request more observations where precise alignment matters. However, we have not tested peg insertion, and adding direct visual or contact features could further improve these predictions and is a promising direction for future work.

W4 & Q1: Advantages for demonstration collection compared to CaP-X

"What advantages does CAPEX offer for demonstration collection compared to CaP-X?"

Thank you for pointing us to CaP-X [1], which we now discuss in Sec. 2. CaP-X studies how coding agents can generate robot programs using a library of perception and control primitives, evaluating across 187 tasks. CAPEX instead focuses on using a frozen general-purpose foundation model to autonomously collect demonstrations for downstream policy training. There are some key differences between the methods, specifically for demonstration collection.

CAPEX uses a simpler action interface, requiring only a pixel-to-3D grounding tool and waypoint executor rather than a larger library of human-crafted perception and control primitives. This reduces the task-specific engineering needed to begin collecting demonstrations. CAPEX also learns from its own collection experience: prior execution outcomes calibrate how far to execute a plan, while successful demonstrations provide references for future attempts, all without updating the foundation model. In addition, CAPEX records successful executions as training data at 20 Hz and directly evaluates their downstream learning value. We find that policies trained from scratch on CAPEX demonstrations achieve comparable success to those trained on matched human demonstrations (Sec. 4.5; W1 above). We also measure the cost, time, requests, and tokens required per successful demonstration, including failed attempts (Table 1), to evaluate the practical cost of collecting a training dataset.

We see the approaches as complementary. CAPEX's experience-based calibration and reuse of successful executions could also benefit a coding agent such as CaP-X, since the underlying goal is to use previous collection experience to improve future collection.

Q2: Tokens per successful demonstration

"Could Table 1 additionally report tokens per successful demonstration?"

Thank you for this suggestion. We added tokens per successful demonstration to Table 1, calculated by summing input and output tokens across all attempts, including failures, and dividing by the number of successful demonstrations.

AgentTokens / demonstration
Astra (Base)903k
Astra + CAPEX309k
Qwen (Base)∞ (no successes)
Qwen + CAPEX38.2M

For Astra, CAPEX reduces token usage by 2.9× and API cost by 5× per successful demonstration. The larger cost reduction comes partly from prompt caching: 64% of CAPEX's input tokens are cached, compared with 34% for the base agent (App. G.4). Thus, the efficiency improvement is not solely due to API pricing or caching, as CAPEX also reduces the total tokens required to obtain usable demonstrations.

[1] Fu et al. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. ICML 2026.

Thank you again for your thoughtful suggestions. They helped us strengthen the evaluation of demonstration quality, generalization, and collection efficiency, and we would be happy to continue the discussion about our findings.

Thank you for your response. I think the measured 6.6 minutes per successful demonstration remains slow and limits practical collection throughput. Using agents to collect training data is a good idea, but existing approaches such as CaP-X could also serve this purpose, and the conceptual arguments provided are not backed by empirical comparisons demonstrating CAPEX’s advantages. Therefore, I still find the paper slightly below the standard for ICLR and encourage the authors to further improve the work.

Reviewer_yQ7tOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

CAPEX is a method for leveraging foundation VLM models as demonstrators, for the purpose of distilling behaviors into faster, cheaper policies. CAPEX evaluates GPT-6 Astra's capacity for serving as such a demonstrator, and proposes a three-part method for extracting demonstrations efficiently and cheaply. The CAPEX method consists of three main ideas (1) prompting for a multi-waypoint plan, where the VLM is asked to return its own confidence that the robot can reach each waypoint, (2) self-supervised confidence calibration, using the outcome of past demonstrations to serve as labels for the logistic regression calibrator, which is re-run before the start of each new demonstration collection episode, and (3), a memory which summarizes the outcome of previous demonstration attempts, as well as containing some examples of prior successes. Together, this allows CAPEX to adaptively determine when re-planning with the foundation model is necessary, versus just naively re-querying the foundation model at every step, during the process of collecting demonstrations. The authors demonstrate that CAPEX yields a higher fraction of successful demonstrations compared with a naive approach, and that the CAPEX demos result in similar downstream policy success rates as human teleoperated demonstrations, across several simulated and real-world manipulation tasks.

Critical strengths
  1. Since the release of GPT-6 Astra, for the first time, there have been demonstrations of frontier models directly controlling robots successfully. This paper makes practical compromises (predicting waypoints rather than high-frequency action targets, re-planning only when the forecasted success of the previous plan falls below a threshold) in order to leverage that capability beyond creating a single demo video. Papers like CAPEX that probe into what foundation models can already express, and how to extract that information efficiently, are timely and relevant to the field.
  2. The contribution does not stop at demo yield, it also analyzes the impact on downstream policy learning. The authors verify that the CAPEX demonstrations are similarly good as human demonstrations for policy learning. Also, the self-supervised calibration procedure is simple and effective; it relies on no human labeling, and results in confidences that are well-calibrated with the actual execution outcomes, whereas the raw VLM outputs are overconfident.
Critical weaknesses
  1. It is not clear where CAPEX's benefits actually come from. CAPEX is motivated as an efficiency strategy, so my expectation was that it would be cheaper (in money and time) than a naive approach which re-queries the foundation model after executing the previous prediction (base), and that it's success rate would be similar to base. However, the results in Figure 2 show that CAPEX is succeeding much more often than base. It is plausible that the multi-waypoint approach is reducing the effective horizon of the task, and CAPEX is therefore succeeding more often, but after looking into the details, it seems that CAPEX and base also differ in several ways that could impact the result (how much information about the task the prompt encodes, whether it has access to locate_pixel tool, etc.) - this makes the result hard to interpret. More isolated ablations (prompt vs. query frequency vs. memory) are necessary in order to understand the result. This is especially true since Figure 3a compares CAPEX with and without memory, and the success rates are separated by ~7 percentage points (still meaningful, but much less dramatic than CAPEX vs. base) even though, to my reading, deleting the memory gets rid of most of the stated contributions of CAPEX (presumably no confidence calibration, no memory).
  2. I am not convinced that this approach is actually practical; 6.6 minutes per successful demo in sim and even ~2 minutes for simple tasks in real is very slow (what is the time comparison with a human demonstration?), and the tasks are much simpler than ones that genuinely rely on expensive human teleoperation (vs. scripting, MimicGen style augmentation, etc.). It would be more interesting to understand the capabilities/limits of current foundation models for synthesizing demonstrations for tasks that are out of range for other synthetic data generation approaches. Since CAPEX's primary stated benefit is practicality, and it relies on the assumption that foundation models are not fast and cheap enough to run directly in closed-loop which may not be true except in the near-term, I think the burden for demonstrating practicality is high.
Questions for authors
  1. Have you tried seeding CAPEX with a few successful human demonstrations?
  2. What happens on tasks where human demonstrations are genuinely slow/costly?
  3. What are the failure modes of CAPEX / base demonstrators? Do they differ at all / are there ones that emerge consistently across methods / tasks?
  4. My feeling is that re-planning is most necessary when the predicted waypoint is wrong, not when the robot failed to reach the predicted waypoint. Do you have any analysis on the utility of waypointing reaching as a calibration label (i.e. does calibrated reach probability relate to episode success)?
7 comments
AuthorsOpenReview ↗

Response to Reviewer yQ7t (Part 1/4)

Dear Reviewer yQ7t,

Thank you for your thoughtful review! We are glad you find the work timely and relevant. Here, we respond to your questions and concerns.

W1: Component ablation

"More isolated ablations (prompt vs. query frequency vs. memory) are necessary in order to understand the result."

Thank you for your suggestion to individually test the impact of each innovation. We have run an ablation study where we add each component one-by-one to the base model. We randomly sampled six tasks across families from the 15 benchmark tasks with a CAPEX success and show results in Sec. 4.4, Table 2 in the revised paper (also below for your convenience).

AgentSuccess (120 episodes)Cost / demonstration (USD)Requests / demonstration
Base (one waypoint per request)14%7.39200
+ CAPEX prompt and locate_pixel41%2.6690
+ multi-waypoint plans53%1.1831
+ calibration and memory (full CAPEX)62%1.3726

One key distinction is that we measure efficiency per successful demonstration (Sec. 3, Eq. 2). Failed attempts still consume model calls and robot time without producing training data, so increasing the success rate also improves collection efficiency.

"It is plausible that the multi-waypoint approach is reducing the effective horizon of the task, and CAPEX is therefore succeeding more often"

Your hypothesis is supported by our new ablation. With the prompt and grounding tool held fixed, multi-waypoint planning increases success from 41% to 53% while reducing requests per demonstration from 90 to 31. We hypothesize that fewer decisions reduce opportunities for errors to compound, while planning several waypoints together helps preserve a coherent strategy rather than reconstructing it after every motion. Planning ahead may also help when the robot's arm moves into the scene and obscures relevant objects.

However, executing longer plans is not always better. Once a waypoint is missed, subsequent waypoints may no longer be valid, so the useful execution horizon depends on the motion and task. We find that a fixed execution horizon achieves 56% success compared with 62% for CAPEX's adaptive horizon, which learns when to replan from prior execution outcomes. We discuss this comparison further in our response to dGg6 (W2; App. G.2).

Your reading of Fig. 3a also highlights an important distinction. Clearing memory removes both the successful references and reliability summaries given to the model, as well as the execution outcomes used to fit the calibrator. Without these outcomes, calibration falls back to the model's stated confidence. However, the agent still retains the improved prompt, grounding tool, and multi-waypoint planning, explaining why it remains relatively strong without memory. Cross-episode experience provides the additional improvement, particularly once the robot has collected a successful demonstration of the task (Sec. 4.3).

W2.1: Human demonstration comparison

"6.6 minutes per successful demo in sim and even ~2 minutes for simple tasks in real is very slow (what is the time comparison with a human demonstration?)"

We agree that human teleoperation is currently faster per demonstration. CAPEX takes 6.6 minutes per successful demonstration, including failed attempts, while Zhao et al. [1] report 30–60 minutes to collect 50 real-robot demonstrations including resets, roughly 0.6–1.2 minutes each. However, CAPEX's advantage is that it does not require human demonstrations or teleoperation. A human operator must be trained and present to collect every demonstration, whereas CAPEX can generate trajectories directly from camera observations and a task instruction. This is particularly useful when collecting demonstrations for a new robot or task, where setting up teleoperation and training an operator can require substantial effort.

Your question also led us to investigate where the collection time is spent. We find that robot motion itself takes a median of 15.0 seconds for CAPEX versus 9.5 seconds for humans on matched initial states. Of CAPEX's 6.6 minutes per demonstration, 4.1 minutes (62%) is spent waiting for model responses, averaging 6.7 seconds per request. At the provider's current fastest API tier of approximately 300 output tokens/s, we project that collection time would decrease to 3.9 minutes per demonstration (App. G.4). We further compare robot motion time in our response to yLtb (W1; App. G.5).

[1] Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS 2023.

AuthorsOpenReview ↗

Response to Reviewer yQ7t (Part 2/4)

Title: Response to Reviewer yQ7t (Part 2/4)


W2.2: Benefits of CAPEX with improved models

"it relies on the assumption that foundation models are not fast and cheap enough to run directly in closed-loop which may not be true except in the near-term"

Thank you for bringing this up. We ran a simulated analysis to find that although faster and cheaper models reduce the inference bottleneck, CAPEX still improves how reliably the model generates demonstrations. To determine whether CAPEX's improvements depend on current model inference latency, and to quantify the impact of faster models, we replayed the recorded executions at 300 output tokens/s (the current fastest API tier) and with instantaneous token generation (infinite token generation speed), while retaining our observed per-request overhead of approximately 2.1 seconds (App. G.4). On the six ablation tasks, where CAPEX's measured 4.8 minutes corresponds to 6.6 on the full benchmark, we find:

Agent (six tasks)SuccessRequests / demonstrationMeasured (min)At 300 tokens/s (min)Instantaneous generation (min)
Base14%20031.419.417.9
+ CAPEX prompt and locate_pixel41%9010.97.26.7
Full CAPEX62%264.82.92.6

Times are minutes per successful demonstration. Even with instantaneous generation, the agent in the second row, which has CAPEX's prompt and tool but still asks for one waypoint per request, takes 6.7 minutes per demonstration compared with CAPEX's 2.6 minutes, because it still requires more model requests and has more failed attempts. Thus the benefits to demonstration generation offered by CAPEX remain, even with faster and cheaper models. Moreover, the resulting student policies run locally at 20 Hz without requiring any foundation-model inference, preserving the value of distilling the collected experience into deployable policies.

W2 & Q2: More complex tasks

"the tasks are much simpler than ones that genuinely rely on expensive human teleoperation (vs. scripting, MimicGen style augmentation, etc.)"

We agree that our tasks do not cover the full range of difficult manipulation skills. Our closest example is the in-air bowl handover on the physical bimanual robot, which requires coordinating two arms to transfer a bowl without putting it down. CAPEX collects this skill without any human demonstrations or task-specific scripts, achieving 95% success at 2.3 minutes and USD 0.88 per successful demonstration, compared with 30% success, 7.6 minutes, and USD 2.73 for the base agent (Sec. 4.6).

Unlike scripting or MimicGen-style augmentation, which require predefined behaviors or source demonstrations, CAPEX generates the behavior directly from camera observations and a language instruction. This distinction becomes particularly useful for new robots and tasks where no demonstrations or scripts are available. However, CAPEX's capabilities are still limited by the foundation model. On the same benchmark, the open Qwen model achieves only 1.4% success with CAPEX compared with 53.6% for GPT-6 Astra (Table 1). We expect improvements in foundation models to expand the range of tasks CAPEX can autonomously collect, while our new seeding experiment (Q1) and failure analysis (Q3) help identify the remaining limitations.

AuthorsOpenReview ↗

Response to Reviewer yQ7t (Part 3/4)

Q1: Seeding with human demonstrations

"Have you tried seeding CAPEX with a few successful human demonstrations?"

Thank you for this suggestion. We tested whether providing a human demonstration can help CAPEX on tasks where it has not yet collected a success of its own. We find that a single human demonstration can substantially improve success when the model does not understand the intended manipulation, but does not necessarily resolve difficulties with precise contact. For example, seeding TurnOnSinkFaucet increases success from 0/20 to 18/20, while CloseBlenderLid remains at 0/20.

We selected the three benchmark tasks where CAPEX never collected a successful demonstration. For each, we provided one human demonstration from the same kitchen, outside the 20 evaluation initial states. The model received an image strip at the first request of each episode and the demonstration's waypoints at every request, keeping the prompt, tools, and other settings unchanged (App. G.7).

TaskWithout human demoWith one human demo
TurnOnSinkFaucet0/2018/20
OpenDrawer0/203/20
CloseBlenderLid0/200/20

To better understand what is going on, we looked at the model's proposed actions. For the faucet, the human demonstration teaches the model the correct motion: it proposes a sideways lever swing in all 20 seeded episodes, compared with only 1/20 without the demonstration. It achieves USD 1.25 and 3.1 minutes per successful demonstration. However, in the drawer task, the drawer remains unmoved in 13 of the 17 failures, while the blender lid is moved but not properly seated in 19/20 episodes. This suggests that a demonstration can resolve uncertainty about what motion to perform. Despite this, precise physical execution remains a challenge, as shown in the table.

Our goal is to eliminate the need for human demonstrations, but this experiment shows that CAPEX can also benefit from them when available. Human demonstrations and the robot's own successful trajectories can both serve as in-context references. The remaining contact-related failures further motivate the execution limitations discussed in Q3.

Q3: CAPEX and base demonstrator failure modes

"What are the failure modes of CAPEX / base demonstrators? Do they differ at all / are there ones that emerge consistently across methods / tasks?"

Thank you for suggesting this analysis. We classified every failed episode by its final simulator state across the 360 matched initial states (App. G.6). We find that the largest difference is whether the agent manages to move the task object or target part at all. Failures leaving them unmoved decrease from 37% with the base agent to 18% with CAPEX. However, both agents continue to struggle with precise contact and fixture manipulation. The table summarizes the failure categories as percentages of all 360 episodes per agent:

Final state of failed episodeBaseCAPEX
Object or target part did not move37%18%
Object or part moved, but not to target20%18%
Target reached, success condition not met20%2%
Wrong fixture part or direction6%5%
Navigation target not reached5%3%
Total failed episodes87.5%46.4%

The shared limitations are clearest on three tasks where neither agent exceeds 5% success. In OpenDrawer, the drawer remains unmoved in 18/20 base and 17/20 CAPEX episodes. In TurnOnSinkFaucet, the wrong part or axis moves in 15/20 and 14/20 episodes, respectively. In CloseBlenderLid, the lid moves but is not properly seated in 13/20 and 18/20 episodes. We hypothesize that sparse waypoint control without force or tactile feedback struggles with sustained, well-aligned contact. This is consistent with our human-seeding experiment (Q1), where showing the model the intended motion does not necessarily give it the precision to execute that motion successfully.

Reviewer_dGg6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CAPEX, a framework for using a multimodal foundation model to autonomously collect robot demonstrations. Instead of querying the model after every waypoint, CAPEX asks it to propose longer plans, uses outcomes from previous attempts to decide how much of each plan to execute, and provides prior successful attempts as context. Across RoboCasa and two physical robot platforms, CAPEX improves collection success and reduces model calls, time, and API cost. The paper also shows that the collected demonstrations can be used to train Diffusion Policy and ACT policies.

Critical strengths
  • The paper studies an important practical question: whether expensive foundation-model behavior can be converted into reusable robot-policy data.
  • The evaluation is broad and practical, reporting success, model calls, collection time, API cost, and downstream policy performance in both simulation and the real world.
  • Calibrating the model's confidence using actual execution outcomes is intuitive and useful. The paper clearly shows that the model's raw confidence is poorly calibrated.
  • The comparison with matched human demonstrations is valuable because it tests whether the collected trajectories are actually useful for policy learning.
Critical weaknesses
  1. My primary concern is that the main comparison does not isolate what CAPEX contributes. The base agent predicts only one waypoint at a time and does not have multi-waypoint planning, locate_pixel, confidence calibration, or cross-episode memory. CAPEX receives all of these at once. Therefore, the improvement from 12.5% to 53.6% could come from the better action interface, perception tool, prompt, memory, or calibration. The same issue appears in the YAM experiment. A cleaner comparison would add these components one at a time while keeping the model, prompt, tools, and action space fixed.

  2. I am not fully convinced about the novelty of the method. Asking the model for longer action chunks, replanning when confidence is low, and showing it previous successes are all natural ideas, with close connections to the adaptive execution and memory methods discussed in related work. Combining them into a working data-collection system is useful, but currently feels more like a well-engineered recipe than a strong new technical contribution. Stronger ablations and comparisons with fixed action chunks or existing adaptive-replanning methods would help establish what is genuinely new.

  3. It is unclear whether the reported efficiency reflects performance on a held-out setting or the final system after tuning. CAPEX learns from earlier attempts on the same task, and the 20 evaluation starts are always processed in one fixed order. The reported cost includes failed collection attempts, which is good, but the paper does not explain how the prompt, calibration features, memory rules, plan length, and the tunable threshold tau=0.5 were chosen, and what was the cost of these design experiments. Results that measure a tuned CAPEX model's ability from zero to one across different start orders and on held-out tasks would make the efficiency claim more convincing.

  4. Minor: The opening paragraph of Section 3 is difficult to follow because several symbols are introduced without explanation, including s_i, T_e, and Succ_l. Equation 3 also refers to recent-action memory as m_t in the text but uses h_t in the equation.

Questions for authors
  • How does CAPEX compare with a base agent given the same multi-waypoint interface and locate_pixel tool, but without calibration or memory?
  • Were any of the evaluation tasks used to tune the prompt, memory rules, plan length, calibration features, or tau?
  • Do the gains persist on held-out tasks and when the order of collection attempts is changed?
4 comments
AuthorsOpenReview ↗

Response to Reviewer dGg6 (Part 1/2)

Dear Reviewer dGg6,

Thank you for your careful review and for recognizing the value of testing foundation models as sources of robot demonstrations. We ran all four experiments you suggested: a component comparison, tasks never used in development, reversed collection order, and fixed-length execution. They help us explain where CAPEX's gains come from and test whether they extend beyond the tasks used to develop it.

W1 & Q1: Component ablation

"A cleaner comparison would add these components one at a time while keeping the model, prompt, tools, and action space fixed."

Thank you for your suggestion to individually test the impact of each innovation. We have run an ablation study where we add each component one-by-one to the base model. We find that adding the CAPEX prompt and locate_pixel tool increases success rate from 14% to 41%, adding multi-waypoint plans then increases it to 53%, and adding calibration and memory further increases it to 62% while reducing model requests per demonstration from 200 to 26. We randomly sampled six tasks across families from the 15 benchmark tasks with a CAPEX success and show results in Sec. 4.4, Table 2 in the revised paper (also below for your convenience).

AgentSuccess (120 episodes)Cost / demonstration (USD)Requests / demonstration
Base (one waypoint per request)14%7.39200
+ CAPEX prompt and locate_pixel41%2.6690
+ multi-waypoint plans53%1.1831
+ calibration and memory (full CAPEX)62%1.3726

W2: Novelty and comparison with fixed action chunks

"Combining them into a working data-collection system is useful, but currently feels more like a well-engineered recipe than a strong new technical contribution."

"Stronger ablations and comparisons with fixed action chunks or existing adaptive-replanning methods would help establish what is genuinely new."

Thank you for asking us to clarify the technical contribution and compare against fixed-length execution. We ran an experiment with fixed 3-waypoint chunks and found that they achieve 56% success compared with 62% for CAPEX. Averaged over two runs, fixed chunks interrupt plans more often (54% versus 23%) while still executing more unreliable waypoints (35% unreached versus 28%). This shows that the challenge is not simply reducing the number of model calls, but learning when the robot should execute a plan and when it needs to observe and replan (see Sec. 4.4 for additional experiment details).

CAPEX introduces three innovations to address the challenges in using a general-purpose multimodal foundation model as an autonomous robot demonstrator. The revised introduction clarifies these innovations (multi-waypoint interface, calibrated plan execution, and learning from experience) and Sec. 4.4, Table 2 shows how these innovations improve the cost and efficiency of demonstration collection over the base model.

A key challenge is that the frozen model's confidence does not reliably predict physical execution. Across all 970 plans of the multi-waypoint agent, its stated confidence would select every waypoint for execution. On CAPEX's waypoints from the same six tasks, average confidence is 0.93, although only 72% are reached. We hypothesize that the model's confidence reflects whether a waypoint makes sense for the task rather than whether the robot can execute it. CAPEX instead calibrates reach probabilities from the robot's own execution outcomes, improving AUROC from 0.57 to 0.75 and reducing expected calibration error from 0.24 to 0.05 on the full benchmark (Sec. 4.3), without human labels, model fine-tuning, or access to model internals.

Regarding existing adaptive-replanning methods, we could not do a direct comparison because the methods cited in Sec. 2 typically rely on learned policies with dense action chunks, internal uncertainty signals, or supervision from task demonstrations, which are not directly available when a frozen API model generates sparse robot waypoints. CAPEX instead obtains physical execution labels directly from the robot's controller and uses them to learn a lightweight, per-task reach predictor. Together with successful trajectories reused as references for future planning, this allows the same frozen model to improve its demonstration collection through experience, without requiring human demonstrations or additional model training.

AuthorsOpenReview ↗

Response to Reviewer dGg6 (Part 2/2)

W3 & Q3: Unseen task evaluation and reversed collection order experiment

"Do the gains persist on held-out tasks and when the order of collection attempts is changed?"

We ran new experiments on held-out tasks as well as with reordering, and find that CAPEX's gains persist both on tasks never used in development and when the order of collection attempts is reversed. To test generalization beyond the held-out initial states in our original submission, we randomly selected six tasks, stratified by family, from the 47 RoboCasa365 atomic tasks outside our benchmark. None of these tasks was used during development, and all 120 episodes use previously unused kitchens. CAPEX achieves 71% success compared with 38% for the base agent, outperforming it on every task. Per successful demonstration, CAPEX reduces API cost from USD 1.99 to USD 0.85, model requests from 55 to 17, and collection time from 8.8 to 2.9 minutes (Sec. 4.2, App. G.1).

To test sensitivity to collection order, we reversed the 20 initial states on each of the six component-comparison tasks. CAPEX achieves 63% success, nearly identical to 62% in the original order (App. G.2). This suggests that the collection gains are not sensitive to the original ordering of attempts. We also find that on 14 of the 15 benchmark tasks where CAPEX succeeds, the first success occurs within two attempts. Thus, the robot typically acquires its own successful task reference early in collection and can reuse it to guide subsequent attempts from different initial states (App. G.2).

W3 & Q2: Design choices and their cost

"Were any of the evaluation tasks used to tune the prompt, memory rules, plan length, calibration features, or tau?"

The initial robot and scene states used for evaluation were held out, and all settings were fixed before evaluation (App. C). The prompt, calibration features, and memory rules were developed on a set of scenes not used in evaluation. We fix τ = 0.5 across tasks, requiring later waypoints to be at least as likely to reach their targets as not (App. G.2). The model itself chooses the proposed plan length, averaging 4.4 waypoints and never exceeding 12, so our cap of 24 does not constrain the observed plans.

Development runs and debugging consumed approximately USD 1.1k in API calls, alongside prototyping under a USD 200/month subscription. These costs are separate from the reported demonstration collection costs.

W4: Notations Thank you for catching these. We have updated them in the revised manuscript (Sec. 3).

Thank you again for the detailed feedback. Your suggested experiments helped us better isolate the contributions and evaluate generalization to unseen tasks and different collection orders. We hope these additional results can help address your concerns, and we would be happy to continue the discussion about our findings.

W1: I appreciate the ablation; it adds meaningful technical understanding to the paper and helps isolate the contributions of different components.

W2: I appreciate the experiment, and it shows that the fixed 3-waypoint chunks perform on par with CAPEX. My concerns about novelty remain unaddressed. CAPEX key innovations: multi-waypoint execution and the locate_pixel tool are not fundamental changes that prior works have not explored. And the motivation and story are unclear: there is no single takeaway from the paper on what works.

W3: Held-out task results address my key concern here. I'd also urge the authors to extend their evaluation to measure how one would evaluate a CAPEX-like system, including the costs of tuning, on a completely new task set. However, for the purpose of this work, held-out improvement over the base model suffices (though it would be nice if ablations are also evaluated on this setup).

Overall, due to the novelty concerns, I'm inclined to keep my score.

Dear Reviewer dGg6,

Thank you for the follow-up. We are glad that the component ablation and the held-out tasks addressed your concerns. Beyond locate_pixel and the multi-waypoint interface, our contributions also include calibrated plan execution and learning from experience, each designed for a frozen general-purpose foundation model that we use without fine-tuning it or accessing its internals. These innovations, taken together, make CAPEX, to our knowledge, the first general-purpose autonomous robot demonstrator in which a frozen foundation model directly generates the robot's actions. Three comparisons from our new experiments demonstrate its value for the research community.

Comparison with CaP-X [1], the current state-of-the-art coding agent for robot manipulation. CAPEX reaches comparable success to CaP-X on its two single-arm pick-and-place tasks and completes a trial about three times faster. See our follow-up response to Reviewer yLtb (point 2) for the experiment details.

Comparison with hand-scripting. CAPEX collected an in-air bowl handover on our physical bimanual robot at 95% success from one instruction sentence, whereas hand-writing a program for a two-arm handover is reported to take experienced robot programmers 2–3 weeks [1] (vs. a few minutes to write one instruction sentence in CAPEX).

Comparison with human teleoperation. The advantage of CAPEX over human teleoperation is in how easily CAPEX can scale. A teleoperated demonstration takes an operator's time, whereas a CAPEX demonstration takes only compute, so collection can run unattended and in parallel. We ran 18 collection processes in parallel and collected 275 successful demonstrations per hour (about 13 seconds per demonstration), compared with 0.6–1.2 minutes for a typical human demonstration [2].

These comparisons showcase the potential of CAPEX to become a new standard tool for the robotics research community that can accelerate the pace of generating robot demonstrations.

Evaluation on a completely new task set

"measure how one would evaluate a CAPEX-like system, including the costs of tuning, on a completely new task set"

Thank you for the suggestion! We ran a new experiment on the remaining 47 atomic tasks of RoboCasa365 [3] outside our benchmark. We ran this task set with the prompt and all settings unchanged, so there was no tuning cost. We observed that the success rate of demonstration generation is statistically indistinguishable from the success rate on our 18-task benchmark, which shows that CAPEX is indeed a general-purpose autonomous demonstrator.

We will add these experiments to the revised paper (App. G).

[1] Fu et al. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. ICML 2026 (Fig. 8 and App. K).

[2] Zhao et al. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS 2023.

[3] Nasiriany et al. RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots. ICLR 2026.

Thank you again for your suggestions, and let us know what you think of our findings!

Comments on the paper

1 comment

Dear AC and reviewers,

Thank you for the thoughtful feedback and appreciation of CAPEX's technical contribution, generalization, collection efficiency, and the learning value of its demonstrations.

We start by summarizing key contributions recognized by reviewers in the original submission:

  1. CAPEX makes general-purpose foundation models suitable demonstrators for autonomous robot policy learning without modifying the frozen model [dGg6, yQ7t, yLtb].

  2. Extensive evaluations show that CAPEX substantially reduces the cost and time to create successful robot demonstrations with downstream learning value comparable to human demonstrations [dGg6, yQ7t].

Results from the new experiments requested by reviewers showcase the potential of CAPEX as a scalable alternative to teleoperation for generating robot demonstrations. Key highlights include:

1. Comparison with CaP-X [yLtb]. We ran CaP-Agent0, the state-of-the-art Code-as-Policy agent of CaP-X, from its public code release on all five single-arm tasks of the CaP-X benchmark. CAPEX matches or exceeds its success on every task, with 82% against 75% on average. On the four tasks that both systems complete, CAPEX is 3.4× faster and 5.4× cheaper per successful demonstration. The full comparison is in the revised manuscript (Sec. 4, App. G.8; response to yLtb).

2. Task-agnostic [dGg6, yQ7t, yLtb]. To test generalizability, we add six new tasks in previously unused kitchens, and CAPEX achieves 71% collection success versus 38% for the base agent, reducing collection time from 8.8 to 2.9 minutes per successful demonstration. We used the same prompt and settings, with only the task instruction changing. We also ran CAPEX on all 65 atomic tasks of RoboCasa365, changing only the task instruction. On the 47 tasks outside our benchmark, its success rate is statistically indistinguishable from that on our 18 benchmark tasks (50% vs. 48%). Reversing the collection order on the six component-comparison tasks preserves success at 63% versus 62% in the original order. These experiments suggest that the framework can adapt to new tasks without task-specific development (App. G.1–G.3; responses to dGg6 and yQ7t).

3. Demonstration throughput [yQ7t, yLtb]. By eliminating the need for a human teleoperator, CAPEX can be run in parallel, which scales demonstration throughput linearly. With 56 processes on the six tasks of Sec. 4.5, CAPEX collected 200 successful demonstrations in 8.8 minutes of wall-clock time (1457 demonstrations per hour), at USD 0.56 each and with no operator time. The 3,075 human demonstration trajectories of these six tasks released with RoboCasa365 last a median of 12.3 s, so even a teleoperator who never paused for resets or failed attempts would collect at most about 290 per hour, 5× fewer than 56 CAPEX processes (Sec. 4.2, App. G.4; responses to yQ7t and yLtb).

4. Ablation Study [dGg6, yQ7t]. We ran a new ablation study and find that adding the CAPEX prompt and locate_pixel to the base model raises success from 14% to 41%. Multi-waypoint planning further raises it to 53%, and calibration and memory to 62%, while requests per successful demonstration decrease from 200 to 26 (full CAPEX). Together, multi-waypoint planning, calibration and memory raise success from 41% to 62% over the agent with the same prompt and locate_pixel (p<0.001p<0.001), while reducing requests per demonstration by 71% and cost by 48%. The full comparison, including cost and explanations of each component, is in our revised manuscript (Sec. 4.4, Table 2; App. G.2).

To address additional concerns, we also conducted new experiments on fixed-length execution and seeding with human demonstrations, and added analyses of calibration, failure modes, motion quality, inference time, and token usage (App. G.2–G.7).

Together, the results from the new experiments show that CAPEX can change the trajectory of how quickly robot demonstrations are generated. Changes in the revised paper are highlighted in blue.

Best,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.