- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PrimitiveSuite, a robot agent interface exposing manipulation through direct control (C0), geometric targets (C1), and semantic skills (C2). The interface affects vision-language model agents across models, tasks, and manipulation phases, and proposes Chain-of-Primitives traces as in-context demonstrations for composing these interfaces. Results on LIBERO-Pro and RoboMME show that no fixed interface works best in every setting, while COP-based ICL improves primitive composition, especially for weaker orchestration models.
Critical strengths
- The paper aims at a practical question that how the robot api partitions reasoning and control between an agent and its executors. The finding that preferred abstraction varies by model and manipulation phase is useful.
- The COP ablation is informative: call traces provide much more benefit than visual demonstrations alone, and the failure analysis suggests that the traces reduce redundant calls rather than improving low-level execution directly.
Critical weaknesses
- The comparison changes both the agent-facing abstraction and the executor behind it. For example, C2 grounding uses SAM3 plus a VLM and scripted motion, C1 uses depth back-projection and scripted motion, while C0 exposes short direct action chunks. Thus, the results establish the value of these interface-and-executor bundles, but do not isolate the effect of abstraction level or the agent's interface selection alone. This distinction matters for the paper's central interpretation.
- Some baseline numbers are taken from prior work and do not cover the same evaluation cells. The paper acknowledges this for RATS, but the overall ranking can still be hard to interpret as a fully matched comparison.
Questions for authors
- Can the authors clarify which conclusions are about the exposed interface and which depend on the different grounding/control backends? Is there a matched executor comparison for at least one task?
- How sensitive are COP-ICL results to the source model, retrieval strategy, and number of demonstrations? Can this be evaluated on more than Qwen3-VL-8B/LIBERO-Pro?
- What are the per-episode API costs and for the main agent configurations, and how stable are the results across models?
3 comments
Q2. CoP-ICL Generalization and Demonstration Sensitivity
We thank the reviewer for this helpful question. Our main CoP-ICL evaluation covers five models across LIBERO-Pro and RoboMME, while the controlled demonstration-format ablation focuses on Qwen3-VL-8B. Each evaluation episode retrieves up to two successful CoP demonstrations from other task identities, containing primitive calls, execution results, and rationales.
| Model | LIBERO-Pro: No ICL → CoP ICL | RoboMME: No ICL → CoP ICL |
|---|---|---|
| Qwen3-VL-8B | 8.75% → 53.75% | — |
| Qwen3.8-27B | 64.50% → 77.38% | 58.88% → 62.13% |
| Claude Opus 5 | 70.75% → 75.50% | 92.13% → 92.25% |
| GPT-6 Astra | 78.75% → 82.38% | — |
| GPT-5.4 | — | 50.25% → 55.00% |
We further construct a traceable pool of 50 replay-validated demonstrations generated by Qwen3-VL-32B and evaluate two model sizes under the same retrieval protocol:
| Model | No CoP | CoP | Gain |
|---|---|---|---|
| Qwen3-VL-8B | 8.75% | 57.50% | +48.75 pp |
| Qwen3-VL-32B | 62.50% | 63.75% | +1.25 pp |
These findings support model-dependent benefits from procedural ICL, especially for smaller models. We acknowledge that the new pool is predominantly C2, 36.25% of evaluation cells retrieve at least one identical normalized instruction, and the original pool could not be recovered. Thus, this is not a controlled source-model sensitivity study; retrieval and shot-count sensitivity remain open.
Q3. Per-Episode API Costs and Stability
We appreciate the request for computational costs and statistical uncertainty. We provide per-episode calls, tokens, control steps, wall time, and available API-cost estimates across representative models and CoP conditions.
Per-Episode Computational Cost
| Model | CoP | Agent Turns | Total Calls | Tokens (k) | Steps | Wall (s) | API Cost |
|---|---|---|---|---|---|---|---|
| Claude Opus 5 | × | 3.8 | — | — | 253.7 | — | $0.19–0.22* |
| GPT-6 Astra | × | 3.9 | — | — | 251.6 | — | $0.25–0.31* |
| Qwen3.8-27B | × | 3.7 | — | — | 264.4 | — | $0.007–0.010* |
| Qwen3-VL-8B | × | 4.6 | 8.4 | 34.9 | 323.6 | 27.0 | Local |
| Qwen3-VL-8B | ✓ | 4.3 | 6.6 | 35.7 | 276.6 | 24.9 | Local |
| Qwen3-VL-32B | × | 4.9 | 6.9 | 46.2 | 269.0 | 51.4 | Local |
| Qwen3-VL-32B | ✓ | 4.0 | 6.0 | 32.3 | 267.5 | 44.4 | Local |
USD values are historical uncached ranges across interface configurations, not exact CoP-condition costs; Qwen3.8-27B uses a hosted-price proxy. Local GPU costs are unavailable. Historical and supplemental evaluations use different protocols.
With CoP, 8B total calls decrease by 21.5% and wall time by 7.7%, while 32B token usage decreases by 30.3% and wall time by 13.7% in the matched supplemental conditions.
Thanks for the detailed feedback, Really appreciate the effort for the dense experiment results. I keep my score positive.
We sincerely thank the reviewer for recognizing the importance of agent–executor interfaces, the model-dependent abstraction findings, and the value of our CoP ablation. We greatly appreciate the constructive feedback and address each concern below.
W1. Interface Abstraction and Executor Dependence
We appreciate this important observation and agree that interface effects should be distinguished from executor capabilities. Our original comparisons evaluate interface–executor configurations, not abstraction levels in isolation. We clarify this distinction and provide two diagnostics:
E1. Executor-Side Attainable Success
C0 is a theoretical bound; C1/C2 use best-of-eight oracle search with rollback.
E2. Model-Side Object-Reference Accuracy
These results reveal a model–executor trade-off: C1 offers higher attainable success (82.0% vs. 71.2%) but requires more precise geometric reasoning, whereas C2 achieves higher model-side reference accuracy. This motivates our study of model–interface compatibility. We acknowledge that these diagnostics use different protocols and do not establish a universal abstraction ranking.
W2. Unmatched Baseline Comparisons
We thank the reviewer for identifying this concern. We agree that published baseline results should not be interpreted as fully matched comparisons. RoboMME baselines were not rerun using our agent harness, and differences in models, training data, controllers, and inference budgets remain. Although our RATs table covers all eight splits, exact task IDs, seeds, and denominators have not been fully reconciled. We clarify these limitations in the revised manuscript.
Q1. Fair Interface Comparison and Model-Dependent Effects
We thank the reviewer for this important suggestion. To better distinguish interface usability from executor capability, we introduce E3, a model-scale ablation comparing end-to-end success against executor-side attainable success (E1) on the same 80 evaluation cells. The aligned E1 references are 80.00% for C1 and 73.75% for C2.
E3. End-to-End Success Relative to Executor-Side Attainable Success
The results highlight the importance of model-side interface usability. Despite C1's higher executor-side attainable success, both models achieve higher end-to-end success through C2. Moreover, CoP substantially improves 8B on C2 (+37.50 pp), while its effect on 32B is modest (+2.50 pp) and can even be negative on C1 (−22.50 pp). These findings suggest that interface effectiveness and CoP benefits depend on model capabilities, not executor capability alone. We acknowledge that E1-relative rates are descriptive and do not fully isolate backend effects.