acceptodds
Under review as a conference paper at ICLR 2027

Chain-of-Primitives Shapes Robot-Agent Adaptation

Abstract

Vision-language models (VLMs) increasingly control robots through manipulation primitives, ranging from low-level commands to geometric targets and semantic skills. How these capabilities are exposed to the agent determines which parts of spatial reasoning and control the model must solve itself. Existing systems, however, typically fix this interface by design. We introduce PrimitiveSuite, which exposes a common set of manipulation capabilities through three agent-facing interfaces: direct control, geometric specification, and semantic invocation. We find that no single interface is consistently best: the preferred abstraction varies across models, tasks, and even manipulation phases. Rather than prescribing this choice, we allow agents to compose heterogeneous primitives during execution and represent their decisions as a Chain-of-Primitives (CoP). Successful CoPs provide a natural form of in-context experience, demonstrating how available primitives should be selected, parameterized, and sequenced. CoP-based ICL substantially improves composition, while visual demonstrations alone provide little benefit. GPT-6 Astra reaches 82.4% success on LIBERO-Pro, and Claude Opus 5 reaches 92.3% on RoboMME, outperforming representative VLA, coding-agent, and VLM-harness baselines. Our results shift the design question from choosing a fixed robot interface to enabling agents to adaptively compose interfaces from experience.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic manipulation, agentic robotics, in-context learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5Q3HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PrimitiveSuite, a robot agent interface exposing manipulation through direct control (C0), geometric targets (C1), and semantic skills (C2). The interface affects vision-language model agents across models, tasks, and manipulation phases, and proposes Chain-of-Primitives traces as in-context demonstrations for composing these interfaces. Results on LIBERO-Pro and RoboMME show that no fixed interface works best in every setting, while COP-based ICL improves primitive composition, especially for weaker orchestration models.

Critical strengths
  • The paper aims at a practical question that how the robot api partitions reasoning and control between an agent and its executors. The finding that preferred abstraction varies by model and manipulation phase is useful.
  • The COP ablation is informative: call traces provide much more benefit than visual demonstrations alone, and the failure analysis suggests that the traces reduce redundant calls rather than improving low-level execution directly.
Critical weaknesses
  • The comparison changes both the agent-facing abstraction and the executor behind it. For example, C2 grounding uses SAM3 plus a VLM and scripted motion, C1 uses depth back-projection and scripted motion, while C0 exposes short direct action chunks. Thus, the results establish the value of these interface-and-executor bundles, but do not isolate the effect of abstraction level or the agent's interface selection alone. This distinction matters for the paper's central interpretation.
  • Some baseline numbers are taken from prior work and do not cover the same evaluation cells. The paper acknowledges this for RATS, but the overall ranking can still be hard to interpret as a fully matched comparison.
Questions for authors
  1. Can the authors clarify which conclusions are about the exposed interface and which depend on the different grounding/control backends? Is there a matched executor comparison for at least one task?
  2. How sensitive are COP-ICL results to the source model, retrieval strategy, and number of demonstrations? Can this be evaluated on more than Qwen3-VL-8B/LIBERO-Pro?
  3. What are the per-episode API costs and for the main agent configurations, and how stable are the results across models?
3 comments

We sincerely thank the reviewer for recognizing the importance of agent–executor interfaces, the model-dependent abstraction findings, and the value of our CoP ablation. We greatly appreciate the constructive feedback and address each concern below.

W1. Interface Abstraction and Executor Dependence

We appreciate this important observation and agree that interface effects should be distinguished from executor capabilities. Our original comparisons evaluate interface–executor configurations, not abstraction levels in isolation. We clarify this distinction and provide two diagnostics:

E1. Executor-Side Attainable Success

C0C1C2
100%*82.0%71.2%

C0 is a theoretical bound; C1/C2 use best-of-eight oracle search with rollback.

E2. Model-Side Object-Reference Accuracy

Qwen3-VLC1 GeometricC2 Semantic
2B0.5%*95.6%
4B37.5%100.0%
8B49.3%95.6%
32B74.1%94.7%

These results reveal a model–executor trade-off: C1 offers higher attainable success (82.0% vs. 71.2%) but requires more precise geometric reasoning, whereas C2 achieves higher model-side reference accuracy. This motivates our study of model–interface compatibility. We acknowledge that these diagnostics use different protocols and do not establish a universal abstraction ranking.

W2. Unmatched Baseline Comparisons

We thank the reviewer for identifying this concern. We agree that published baseline results should not be interpreted as fully matched comparisons. RoboMME baselines were not rerun using our agent harness, and differences in models, training data, controllers, and inference budgets remain. Although our RATs table covers all eight splits, exact task IDs, seeds, and denominators have not been fully reconciled. We clarify these limitations in the revised manuscript.

Q1. Fair Interface Comparison and Model-Dependent Effects

We thank the reviewer for this important suggestion. To better distinguish interface usability from executor capability, we introduce E3, a model-scale ablation comparing end-to-end success against executor-side attainable success (E1) on the same 80 evaluation cells. The aligned E1 references are 80.00% for C1 and 73.75% for C2.

E3. End-to-End Success Relative to Executor-Side Attainable Success

ModelCoPC1 SuccessC1 / E1C2 SuccessC2 / E1
8B×2.50%3.12%21.25%28.81%
8B✓6.25%7.81%58.75%79.66%
32B×47.50%59.38%61.25%83.05%
32B✓25.00%31.25%63.75%86.44%

The results highlight the importance of model-side interface usability. Despite C1's higher executor-side attainable success, both models achieve higher end-to-end success through C2. Moreover, CoP substantially improves 8B on C2 (+37.50 pp), while its effect on 32B is modest (+2.50 pp) and can even be negative on C1 (−22.50 pp). These findings suggest that interface effectiveness and CoP benefits depend on model capabilities, not executor capability alone. We acknowledge that E1-relative rates are descriptive and do not fully isolate backend effects.

Q2. CoP-ICL Generalization and Demonstration Sensitivity

We thank the reviewer for this helpful question. Our main CoP-ICL evaluation covers five models across LIBERO-Pro and RoboMME, while the controlled demonstration-format ablation focuses on Qwen3-VL-8B. Each evaluation episode retrieves up to two successful CoP demonstrations from other task identities, containing primitive calls, execution results, and rationales.

ModelLIBERO-Pro: No ICL → CoP ICLRoboMME: No ICL → CoP ICL
Qwen3-VL-8B8.75% → 53.75%—
Qwen3.8-27B64.50% → 77.38%58.88% → 62.13%
Claude Opus 570.75% → 75.50%92.13% → 92.25%
GPT-6 Astra78.75% → 82.38%—
GPT-5.4—50.25% → 55.00%

We further construct a traceable pool of 50 replay-validated demonstrations generated by Qwen3-VL-32B and evaluate two model sizes under the same retrieval protocol:

ModelNo CoPCoPGain
Qwen3-VL-8B8.75%57.50%+48.75 pp
Qwen3-VL-32B62.50%63.75%+1.25 pp

These findings support model-dependent benefits from procedural ICL, especially for smaller models. We acknowledge that the new pool is predominantly C2, 36.25% of evaluation cells retrieve at least one identical normalized instruction, and the original pool could not be recovered. Thus, this is not a controlled source-model sensitivity study; retrieval and shot-count sensitivity remain open.

Q3. Per-Episode API Costs and Stability

We appreciate the request for computational costs and statistical uncertainty. We provide per-episode calls, tokens, control steps, wall time, and available API-cost estimates across representative models and CoP conditions.

Per-Episode Computational Cost

ModelCoPAgent TurnsTotal CallsTokens (k)StepsWall (s)API Cost
Claude Opus 5×3.8——253.7—$0.19–0.22*
GPT-6 Astra×3.9——251.6—$0.25–0.31*
Qwen3.8-27B×3.7——264.4—$0.007–0.010*
Qwen3-VL-8B×4.68.434.9323.627.0Local
Qwen3-VL-8B✓4.36.635.7276.624.9Local
Qwen3-VL-32B×4.96.946.2269.051.4Local
Qwen3-VL-32B✓4.06.032.3267.544.4Local

USD values are historical uncached ranges across interface configurations, not exact CoP-condition costs; Qwen3.8-27B uses a hosted-price proxy. Local GPU costs are unavailable. Historical and supplemental evaluations use different protocols.

With CoP, 8B total calls decrease by 21.5% and wall time by 7.7%, while 32B token usage decreases by 30.3% and wall time by 13.7% in the matched supplemental conditions.

Thanks for the detailed feedback, Really appreciate the effort for the dense experiment results. I keep my score positive.

Reviewer_myBdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Strengh:

  • The question of the correct interface between FM and robotics is important.
  • The evaluation contains useful controls.
  • The procedural-context ablation produces a clear and useful result.

Weakness:

  • The proposed method, Chain-of-Primitives, is an ordered sequence of primitive calls, augmented with execution feedback. And is quite limited in novelty given existing works like EMPIRE, ASPIRE, etc.
  • The experiments do not isolate the benefit of learning to compose abstraction levels.
  • The abstraction-level conclusions remain dependent on the particular executors.
Critical strengths

See summary

Critical weaknesses

See summary

Questions for authors

See summary

2 comments

We sincerely thank the reviewer for recognizing the importance of foundation model–robot interfaces, the value of our experimental controls, and the insights from our procedural-context ablation. We appreciate the thoughtful concerns and address each below.

W1. Limited Novelty of Chain-of-Primitives

We appreciate the reviewer's observation and agree that primitive sequencing itself is not novel. Our contribution is not a new sequencing algorithm, but a systematic investigation of how foundation models interact with robotic primitives at different abstraction levels, and how procedural in-context learning (ICL) improves their utilization.

Inspired by Chain-of-Thought, CoP makes intermediate embodied decisions explicit through primitive calls, execution results, and rationales. PrimitiveSuite further enables systematic study of three agent-facing abstraction levels (C0–C2).

We clarify our distinction from related work:

MethodTask-demo ICLICL ContextAction InterfaceVisual ICLForm Ablation
Code as Policies✓Instruction–codeControl APIs××
CaP-X×API examplesHigh/low APIs××
ASPIRE×Repair skillsPerception/control××
ENPIRE×Research summariesCode + tools/VLAs××
Harness VLA✓Primitive traces + rulesAnalytic + VLA××
VoLoAgent×Online historyVLA/WAM + tools××
Ours✓CoP: calls, results, rationalesC0/C1/C2 per call✓✓

✓ = used or evaluated; × = not reported under these criteria.

Unlike approaches primarily focused on improving policies or skills, we investigate the interaction between model capabilities, interface abstraction, and procedural demonstrations. Our ICL-form ablation (8.8% → 53.8%) further highlights the importance of how primitive experience is represented.

W2. Insufficient Isolation of Cross-Level Composition

We thank the reviewer for highlighting the need to disentangle primitive utilization from cross-level composition. Our additional comparison provides the following results:

ModelCoP ICLFixed C1Fixed C2Composition
Claude Opus 5×53.8%66.2%70.0%
Claude Opus 5✓63.8%66.2%75.0%
Qwen3.8-27B×50.0%59.4%65.0%
Qwen3.8-27B✓60.6%61.9%71.3%

Two observations emerge. First, composition achieves the highest success rates for both models, suggesting value in flexible interface selection. Second, CoP ICL improves fixed C1 by +10.0 pp (Opus) and +10.6 pp (Qwen), indicating benefits even without cross-level switching.

We acknowledge that these configurations were evaluated under different protocols. Thus, while the results support the utility of primitive composition and procedural ICL, they do not establish the independent causal benefit of learned cross-level switching. We will clarify this distinction in the paper.

W3. Executor Dependence of Abstraction-Level Conclusions

We appreciate this important observation and agree that abstraction-level performance depends on both model-side reasoning and executor-side capabilities. Our diagnostics illustrate this interaction.

E1. Executor-Side Attainable Success

OracleC0C1C2
Success100%*82.0%71.2%

C0 is a conditional theoretical bound; C1/C2 use best-of-eight oracle search with rollback.

E2. Model Accuracy in Specifying Interface Parameters

Qwen3-VLC1 GeometricC2 Semantic
2B0%*95.6%
4B37.5%100.0%
8B49.3%95.6%
32B74.1%94.7%

2B C1 uses incorrect coordinate units. E2 measures parameter accuracy, not task success; denominators differ, and 8B/32B evaluations are incomplete.

These results highlight a key trade-off: C1 offers higher executor-side attainable success (82.0% vs. 71.2%) but requires more precise geometric reasoning. In contrast, C2 provides a lower attainable success ceiling but substantially higher model-side parameter accuracy, especially for smaller models.

Thus, a more capable executor does not necessarily yield higher task success unless the model can effectively use its interface. This model–executor compatibility motivates our study of abstraction levels and CoP. We acknowledge that E1 and E2 use different evaluation protocols and that E2 is preliminary; therefore, these findings do not establish a universal abstraction-level ranking or the causal benefit of adaptive switching.

Dear Reviewer myBd,

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would greatly appreciate it if you could review our responses and let us know whether they address your concerns.

If any points remain unclear or unresolved, please let us know, and we would be happy to provide further clarification. We would also appreciate it if you could reconsider your assessment after reviewing our responses.

Thank you for your time and consideration.

Sincerely, The Authors

Reviewer_e17POpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PRIMITIVESUITE, a multi-level primitive library exposing three agent-robot interfaces (C0: direct control, C1: geometric, C2: semantic). The authors find no single interface is universally optimal; capable agents can compose heterogeneous primitives into a Chain-of-Primitives (CoP)​ during execution. Providing successful CoPs as in-context demonstrations significantly boosts performance outperforming VLA and coding-agent baselines.

Critical strengths

1.The experiments are valuable. The paper investigates how the agent–executor prompt interface affects the apparent capabilities of robot-use agents. 2.The evaluation covers multiple model families, analyzes the performance–cost trade-off of different agents, and uses two simulation benchmarks.

Critical weaknesses
  1. The primitive intervention is not sufficiently isolated From Appendix B, Composition Prompt, it appears that C0, C1, and C2 are implemented through three different prompts that provide the agent with different types of information. However, the distinction between C1 and C2 is not entirely clean. Semantic information in C2, such as object segmentation or object detection, implicitly contains the target’s spatial location and therefore includes information that is also required by C1. In this sense, C1 and C2 seem to form a progressive hierarchy of information: both ultimately provide a target location to the executor, and both use motion planning. In contrast, C0 uses a different low-level policy or control mechanism. Therefore, the comparison is not conducted within a single unified system.
  2. Limited novelty The main idea chain-of-primitives appears to summarize a prompt pattern that is already common in agentic robotics. Direct-control, geometric, and semantic primitives are all relatively standard approaches. The authors should more explicitly distinguish their contribution from existing work on Code-as-Policy, tool-use agents, hierarchical robot skills, semantic manipulation APIs, and systems.
  3. Lack of real-robot experiments Given that the proposed method is relatively simple the absence of real-robot experiments is a significant limitation.
Questions for authors
  1. What exact low-level controller is used to execute the robot actions in C0, C1, C2 and the VLA baselines? Are all methods using delta control (relative action representation) across both LIBERO-Pro and RoboMME? Please provide the relevant implementation details and parameter settings.

  2. Please clarify the differences between this work and other code-as-policy or Astra-driven robot approaches.

  3. Please further explain why the combination of C0, C1, and C2 achieves better results, and under what scenarios should C0, C1, C2, and the combination be used respectively?

4 comments

We sincerely thank the reviewer for recognizing the value of our experiments, the breadth of our evaluation across models and benchmarks, and the soundness of our study. We appreciate the constructive feedback and address each concern below.

W1. Spatial-Reasoning Responsibility and Execution Backends

We thank the reviewer for identifying this important ambiguity. We agree that C1 and C2 both involve spatial information and motion planning. Their distinction lies in how spatial reasoning is allocated between the agent (System 2) and executor (System 1). For example, placing a mug on a shelf requires end-effector increments (C0), explicit geometric targets (C1), or semantic arguments such as place(mug, shelf) (C2).

All three ultimately use benchmark-native low-level controllers, but differ in their agent-facing interfaces and execution routines. We acknowledge that the original comparisons evaluate interface–executor configurations, not abstraction level alone. We have clarified this distinction in the Introduction and Sec. 3.

Our additional diagnostics illustrate the trade-off between executor capability and model-side interface usability:

E1. Maximum Achievable Task Success

C0C1C2
100%*82.0%71.2%

C0 is a theoretical upper bound, not a measured result. C1/C2 use best-of-eight oracle search with rollback.

E2. Model Accuracy in Specifying Interface Parameters

Qwen3-VLC1 GeometricC2 Semantic
2B0%*95.6%
4B37.5%100.0%
8B49.3%95.6%
32B74.1%94.7%

Q1. Exact Controllers and Delta Representations

We appreciate the request for implementation details. Not all methods use delta control, and the low-level controllers differ across benchmarks:

MethodLIBERO-ProRoboMME
C0/C1/C2OSC_POSE (relative EEF control)pd_joint_pos (absolute joint targets)
VLA/WAM baselinesExact run configurations not uniformly verifiedRoboMME specifies absolute targets; SimpleMemVLA documents pd_joint_pos

In LIBERO-Pro, C0 accepts relative end-effector increments, while C1/C2 generate targets executed through the same OSC_POSE backend. In RoboMME, C0 converts relative displacements into waypoints for joint-position control. Only Composition was evaluated on RoboMME. We have expanded Appendix A.7 (p. 18) with action limits, controller settings, and baseline details.

W2 / Q2. Novelty Beyond Existing Tool-Use Systems

We thank the reviewer for raising this important question. We agree that primitive sequencing and individual abstraction levels are not novel. Our contribution is a unified C0–C2 primitive suite and a systematic study of how procedural in-context learning improves primitive utilization by frozen foundation models.

We clarify our distinctions from related work:

MethodTask-demo ICLICL ContextAction InterfaceVisual ICLForm Ablation
Code as Policies✓Instruction–codeControl APIs××
CaP-X×API examplesHigh/low APIs××
ASPIRE×Repair skillsPerception/control××
ENPIRE×Research summariesCode + tools/VLAs××
Harness VLA✓Primitive traces + rulesAnalytic + VLA××
VoLoAgent×Online historyVLA/WAM + tools××
Ours✓CoP: calls, results, rationalesC0/C1/C2 per call✓✓

✓ = used or evaluated; × = not reported under these criteria.

Our central distinction is the joint study of multi-level interface composition and procedural ICL, including controlled comparisons of different demonstration formats.

With the model and primitive library fixed, success improves from 8.8% (no ICL) and 11.3% (visual keyframes) to 47.5% (primitive calls) and 53.8% (calls + results + rationales).

Even with an explicit API guide, CoP improves Qwen3-VL-8B from 8.75% to 57.50% (+48.75 pp), while Qwen3-VL-32B improves from 62.50% to 63.75% (+1.25 pp), highlighting model-dependent benefits.

These findings demonstrate the value of procedural demonstrations beyond API descriptions or visual examples, without model fine-tuning or executor modifications. We acknowledge that these experiments do not independently establish the benefit of learned cross-level routing.

W3. Lack of Real-Robot Experiments

We appreciate the reviewer's suggestion and agree that real-robot validation would strengthen the practical significance of our findings. Unfortunately, we currently lack access to the necessary robotic hardware, limiting our evaluation to simulation. We acknowledge this limitation in the revised manuscript and identify validation across physical robot platforms as an important direction for future work.

Q3. Why Combine Abstraction Levels, and When Should Each Be Used?

We thank the reviewer for this insightful question. Composition allows the model to select an interface based on task requirements and its own reasoning capabilities: C2 for semantic operations, C1 for explicit geometric decisions and recovery, and C0 for fine-grained control.

Our comparison demonstrates the potential benefits of this flexibility:

ModelCoP ICLFixed C1Fixed C2Composition
Claude Opus 5×53.8%66.2%70.0%
Claude Opus 5✓63.8%66.2%75.0%
Qwen3.8-27B×50.0%59.4%65.0%
Qwen3.8-27B✓60.6%61.9%71.3%

Composition achieves the highest success rates for both models, with CoP ICL further improving Composition by +5.0 pp (Opus) and +6.3 pp (Qwen). These comparisons suggest benefits from flexible interface selection, although the evaluation protocols do not isolate its independent causal contribution.

We further provide qualitative case studies with simulator frames in Appendix A.10, Fig. 7, illustrating when composition helps and when a single interface suffices:

  • Astra (plate placement): After a C2 grasp fails, switching to C1 enables recovery through explicit geometric targets.
  • Opus (bowl placement): Selecting C1 succeeds where fixed C2 fails, without requiring an interface switch.
  • Qwen (two-mug task): Fixed C2 succeeds in 385 steps, while Composition takes 520 steps, showing that additional composition is not always beneficial.

Thank you for your answers. I now have the information I needed.

We sincerely thank the reviewer for their time and thoughtful engagement. We are glad that our responses have addressed your questions. We hope that the additional experiments and clarifications have helped strengthen the paper. Please let us know if you have any remaining questions or concerns. We would be happy to address them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.