- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Collecting robot demonstrations through human teleoperation is difficult to scale, while using foundation models as autonomous demonstrators requires slow and costly model calls. The paper proposes a framework that uses a frozen multimodal foundation model to generate waypoint plans and collect demonstrations for training robot policies. CAPEX uses past execution outcomes to calibrate waypoint confidence and decide when to replan, while summaries and successful examples from previous attempts guide future plans. On RoboCasa, CAPEX increases collection success from 12.5% to 53.6% and reduces the cost per successful demonstration by 80%. Experiments in simulation and on real robots show that policies trained from scratch on CAPEX demonstrations achieve success rates close to those trained on human demonstrations, while fine-tuned Diffusion Policy performs worse with CAPEX data.
Critical strengths
Using foundation models for demonstration collection is an important and timely problem. Adapting planning and execution based on accumulated experience is also a promising approach to improving collection efficiency.
Critical weaknesses
- Demonstration quality warrants further evaluation beyond task success and aggregate collection efficiency. The project videos appear to show unnatural pauses and non-smooth transitions between waypoints, which may affect downstream policy learning. Quantifying motion smoothness and systematically comparing demonstration completion times against human demonstrations would strengthen the evaluation.
- Despite the improvement over the baseline, the reported 6.6 minutes per successful demonstration still indicates low collection throughput.
- Although the learned calibrator includes distance to a VLM-selected 3D point, that point's semantic role may vary across calls, and the calibrator lacks direct visual features or explicit object geometry. This raises concerns about its generality to more challenging tasks. For example, in contact-rich peg insertion, reachability depends strongly on the peg and hole geometry and their relative alignment.
- The related-work discussion omits CaP-X [1], (accepted at ICML 2026) a relevant framework for foundation-model-driven robot manipulation that could also support demonstration collection.
[1] Fu L, Yu J, El-Refai K, et al. CaP-X: A framework for benchmarking and improving coding agents for robot manipulation[J]. arXiv preprint arXiv:2603.22435, 2026.
Questions for authors
- What advantages does CAPEX offer for demonstration collection compared to CaP-X?
- Could Table 1 additionally report tokens per successful demonstration? This would complement monetary cost with a measure independent of changing API prices and applicable to locally deployed models such as Qwen.
5 comments
Response to Reviewer yLtb (Part 2/2)
W3: Generality of the calibrator
"that point's semantic role may vary across calls, and the calibrator lacks direct visual features or explicit object geometry"
You are right that the grounded point may have different semantic roles across calls. Our calibrator only uses the distance from a waypoint to the last located 3D point, without assigning it a semantic label. We tested whether this feature is necessary by removing it, and found that AUROC only decreases from 0.754 to 0.740 (App. G.3). This suggests that the calibrator does not depend strongly on the semantic consistency of the grounded point.
More importantly, the calibrator learns from each task's own execution outcomes rather than relying on a geometric model transferred from another task. To test whether this generalizes to new tasks, we evaluated the same calibrator design on six tasks never used during development, fitting its weights separately for each task from prior execution outcomes. We find that calibration reduces expected calibration error from 0.15 to 0.03 and improves AUROC from 0.61 to 0.73. It also improves both metrics across every benchmark task family, including doors, drawers, buttons, and knobs (App. G.3). We provide the full unseen-task evaluation in our response to dGg6 (W3 & Q3).
For your peg-insertion example, we hypothesize that repeated stalls near contact would teach the calibrator to shorten its execution horizon and request more observations where precise alignment matters. However, we have not tested peg insertion, and adding direct visual or contact features could further improve these predictions and is a promising direction for future work.
W4 & Q1: Advantages for demonstration collection compared to CaP-X
"What advantages does CAPEX offer for demonstration collection compared to CaP-X?"
Thank you for pointing us to CaP-X [1], which we now discuss in Sec. 2. CaP-X studies how coding agents can generate robot programs using a library of perception and control primitives, evaluating across 187 tasks. CAPEX instead focuses on using a frozen general-purpose foundation model to autonomously collect demonstrations for downstream policy training. There are some key differences between the methods, specifically for demonstration collection.
CAPEX uses a simpler action interface, requiring only a pixel-to-3D grounding tool and waypoint executor rather than a larger library of human-crafted perception and control primitives. This reduces the task-specific engineering needed to begin collecting demonstrations. CAPEX also learns from its own collection experience: prior execution outcomes calibrate how far to execute a plan, while successful demonstrations provide references for future attempts, all without updating the foundation model. In addition, CAPEX records successful executions as training data at 20 Hz and directly evaluates their downstream learning value. We find that policies trained from scratch on CAPEX demonstrations achieve comparable success to those trained on matched human demonstrations (Sec. 4.5; W1 above). We also measure the cost, time, requests, and tokens required per successful demonstration, including failed attempts (Table 1), to evaluate the practical cost of collecting a training dataset.
We see the approaches as complementary. CAPEX's experience-based calibration and reuse of successful executions could also benefit a coding agent such as CaP-X, since the underlying goal is to use previous collection experience to improve future collection.
Q2: Tokens per successful demonstration
"Could Table 1 additionally report tokens per successful demonstration?"
Thank you for this suggestion. We added tokens per successful demonstration to Table 1, calculated by summing input and output tokens across all attempts, including failures, and dividing by the number of successful demonstrations.
| Agent | Tokens / demonstration |
|---|---|
| Astra (Base) | 903k |
| Astra + CAPEX | 309k |
| Qwen (Base) | ∞ (no successes) |
| Qwen + CAPEX | 38.2M |
For Astra, CAPEX reduces token usage by 2.9× and API cost by 5× per successful demonstration. The larger cost reduction comes partly from prompt caching: 64% of CAPEX's input tokens are cached, compared with 34% for the base agent (App. G.4). Thus, the efficiency improvement is not solely due to API pricing or caching, as CAPEX also reduces the total tokens required to obtain usable demonstrations.
[1] Fu et al. CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation. ICML 2026.
Thank you again for your thoughtful suggestions. They helped us strengthen the evaluation of demonstration quality, generalization, and collection efficiency, and we would be happy to continue the discussion about our findings.
Thank you for your response. I think the measured 6.6 minutes per successful demonstration remains slow and limits practical collection throughput. Using agents to collect training data is a good idea, but existing approaches such as CaP-X could also serve this purpose, and the conceptual arguments provided are not backed by empirical comparisons demonstrating CAPEX’s advantages. Therefore, I still find the paper slightly below the standard for ICLR and encourage the authors to further improve the work.
Response to Reviewer yLtb (Part 1/2)
Dear Reviewer yLtb,
Thank you for the concrete suggestions and for recognizing using foundation models for demonstration collection as an important and timely problem. Your review led us to compare CAPEX demonstrations with human demonstrations, evaluate the calibrator on unseen tasks, discuss CaP-X, and report token usage per successful demonstration.
W1: Demonstration quality
Thank you for suggesting this comparison. We find that CAPEX demonstrations take longer and are slightly less smooth than human demonstrations, but have shorter end-effector paths. More importantly, policies trained from scratch on CAPEX demonstrations achieve comparable success to those trained on human demonstrations.
We compared 193 successful CAPEX demonstrations with human demonstrations from the same initial robot and scene states (App. G.5):
Action jerk is the mean L1 norm of the second difference of translational arm commands. We hypothesize that the longer completion times are largely due to the executor moving more slowly between waypoints and pausing to check whether each waypoint was reached. These pauses also provide the execution outcomes used to train our calibrator. Although they make the trajectories slower, the shorter path lengths suggest that CAPEX still follows relatively direct spatial routes.
To test whether these differences affect downstream learning, we trained policies on matched quantities of CAPEX and human demonstrations from identical initial states and evaluated them in our submission. After 100k training steps, ACT trained from scratch achieves 50% success with either dataset, while Diffusion Policy trained from scratch achieves 45% with CAPEX demonstrations versus 47% with human demonstrations (Sec. 4.5, Fig. 4). These results suggest that, despite being slower and slightly less smooth, CAPEX demonstrations can provide comparable training supervision without requiring human teleoperation.
W2: Collection throughput
We agree that CAPEX's current collection throughput is slower than human teleoperation. However, the main advantage is that these demonstrations can be collected without a human operator controlling the robot, at an API cost of USD 1.83 per successful demonstration including failed attempts. We investigated the inference bottleneck and found that 4.1 of the 6.6 minutes (62%) are spent waiting for model responses. At the provider's current fastest API tier of approximately 300 output tokens/s, we project that collection time would decrease to 3.9 minutes per demonstration (App. G.4).
CAPEX's capabilities are still limited by the foundation model. On the same benchmark, the open Qwen model achieves only 1.4% success with CAPEX compared with 53.6% for GPT-6 Astra. We expect further improvements in foundation models to significantly improve the demonstration yield and throughput.