acceptodds
Under review as a conference paper at ICLR 2027

Agent Priors-guided Policy Learning

Abstract

Robots that learn from a few demonstrations often require two forms of generalization. Compositional generalization recombines skills to solve new tasks, and skill generalization lets the learned policy behind each skill work in new situations. The two depend on each other, yet information is lost between composition and the skills it calls. Where a skill works is determined by the structure its policy is trained with, while composition sees the skill only through a separate description, such as a name, an instruction, or a symbolic operator, that omits this structure. Our key idea is to use each policy's structural prior as part of the interface between composition and the skill. A structural prior states what a behavior depends on, for example that a grasp depends only on the gripper's pose relative to the object. Built into training, it shapes where the policy generalizes; stated in language, it tells composition where the policy applies. We instantiate this idea in Agent Priors-guided Policy Learning (APPL). A construction agent segments complete demonstrations into reusable skills, proposes several structural priors for each skill, and trains and verifies one policy per prior. A runtime agent then selects among these prior-specific policies and composes them toward new task goals using their interfaces. Across MetaWorld and long-horizon ManiSkill tasks, APPL improves out-of-distribution skill generalization and enables previously unseen skill compositions; ablating the interface information substantially reduces performance. These results support the use of training-time structural assumptions as a bridge between skill learning and skill composition.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Agent, Compositional Generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Q3SUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an agent-guided framework for policy learning and skill composition to improve generalization to out-of-distribution settings and long-horizon tasks. Given demonstrations, the agent segment (i.e., skills) and proposes multiple policy variations to improve generalization. Each policy is trained and evaluated, with its design, description, and evaluation results stored for future use. During inference, the agent selects and combines appropriate policies based on the task and current state. Experiments on simulated manipulation tasks demonstrate improved out-of-distribution generalization and performance on long-horizon tasks.

Critical strengths
  • The paper demonstrates that agents can automatically design policies that substantially improve performance. This is an impactful finding, as manually designing such policies can be inefficient and challenging. Furthermore, the proposed skill composition framework demonstrates that these policies can be combined to solve long-horizon tasks rather than only individual skills.
  • The paper clearly explains each component and its role step by step, providing a useful foundation for future research. The ablation studies help clarify the contribution of each component, while the detailed prompts support reproducibility and further research on agent-driven policy learning.
Critical weaknesses

Scalability under limited resources: The proposed method requires training multiple policies, which can be computationally expensive, especially when using large models such as VLAs. Furthermore, policy verification requires additional rollouts, which can be costly in real-world settings. How could the proposed framework scale under limited computational and physical resources?

Questions for authors
  • How does the framework determine whether each policy has converged? Different policy designs and representations may require different training budgets, making convergence difficult to assess and compare fairly.
  • Could the authors provide concrete examples of the "variations it is expected to tolerate" the prior description did_i? Does it have prior knowledge of the expected evaluation environments or distribution shifts? Also, understanding what 'known limitation' was hard for me. Concrete examples of the generated interface would help readers better understand how this information is determined and used.
Reviewer_VLLFOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Agent Priors-guided Policy Learning (APPL). The main contribution is to use a skill structural prior to connect task level reasoning and skill level policy execution. The core insight is the use of such a structural prior both as an inductive bias in training and as a description of where the policy applies at runtime. The proposed method uses two agents, practically implemented as LLMs. The first one, called construction agent, obtains a set of task demonstrations and a "task knowledge". The construction agent segments the demonstrations into skills, proposing different priors for each skill and training a policy for each prior. Such an agent also defines an interface that can be used to define when a skill can be applied. A second runtime agent (composer) inspects policy interfaces to decide the sequence of policies to be used in order to accomplish the goal. The method is validated on two experiments. The first experiment investigates the role of the priors for a different number of demonstrations on MetaWorld comparing against a diffusion policy and a policy with a fixed relational prior. The paper reports that APPL outperforms both baselines, with the largest gains when the construction agent can receive feedback from an execution. The second experiment studies whether the runtime agent can compose the skills on long-horizon tasks and whether this helps retain the performance on out of distribution cases. Finally, an ablation shows that hiding the interface information reduces performance.

Critical strengths
  • The idea of having multiple variations of the same skill is interesting. Other works have already proposed this idea (such as the mentioned Agentic Skill Discovery) but this paper also provides a concrete method on how to choose them at runtime.
  • The results reported in Table 1 are strong. Even the first design proposed by the agent (q1) without any feedback, outperforms the considered baselines.
  • The authors report all the details of the experiments (including prompts, hyperparameters and analysis of failure modes). The experiments are transparent.
Critical weaknesses
  • Most of the components of the proposed method are built around the use of LLMs. The agent segments the trajectories, decides and implements the priors, produces the interface and so on. The appendix mentions that the experiments use GPT-6 Astra and GPT-5.6 Sol with reasoning effort max. These are frontier models that perform extraordinarily well on benchmarks, so it is hard to decouple the real contribution of the method from the capability of the underlying model to solve the task in isolation. The experiments partially support this concern. The baseline Agent+VLA keeps the runtime agent but drops the prior-based library and yet it remains competitive on compositions (6/16 vs 8/16). A comparison with the same model solving the task without the framework's components, and results with weaker models, would make the claims more convincing.
  • The paper is hard to follow. Section 3 introduce a heavy formalism, most of which is decorative and remains unused in the rest of the paper. The concept of structural prior is used since the abstract, but it gets properly introduced only in 4.1. The use of task knowledge K remains vague and is never properly defined. Most of the introduced components translate into text used to prompt the LLM, for instance the applicability region of Eq.3.
  • The prompts used contain some domain-specific suggestions, for instance in Exp.2 prior design and implementation, the prompt specifies a requirement that the relevant motion is invariant to object displacement. This hint likely helps the agent to design the appropriate priors because object displacement is what is being test in OOD. As a result, it is unclear whether the agent can discover useful priors unguided. Additionally, this depends on model capabilities (read weakness 1). Removing all the domain-knowledge from the prompt would help assess how much the agent contributes to the prior design.
  • The paper is motivated by generalizing from few demonstrations to new tasks because human demonstrations are costly. However, the paper shows that skills identified by the agent only apply to a specific task. Therefore, the paper does not evaluate skill reuse across tasks, which limits the compositional generalization claim.
Questions for authors
  • How do results change if you use a weaker/open LLM?
  • How does APPL compare with the same LLM being asked to solve the problem without all of frameworks component?
  • KK is introduced but never precisely defined. What does it contain? How can the contribution of the framework be separated the knowledge injected through K?
  • Is the applicability region (AiA_i in Eq.3) ever computed explicitly? One of the paper's claims is that using the same prior in training and in the description improves interface faithfulness. Have you observed whether policies actually succeed from states inside AiA_i and fail outside it?
  • How do the results of exp. 2 change if the instruction to target object displacement is removed from the prior-design prompt?
  • Can a library built for one task generalize to another?
Reviewer_BS47OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

To address robots’ limited ability to generalize to object position changes from few demonstrations and failures in skill composition caused by high-level planners’ limited knowledge of skill applicability, this paper proposes APPL. In APPL, a construction agent segments complete demonstrations into reusable skills and designs multiple structural priors for each skill to guide policy learning. These priors also inform a runtime agent’s selection and composition of the learned policies. Experiments on MetaWorld and ManiSkill demonstrate improved skill generalization and task composition with few demonstrations.

Critical strengths
  • APPL presents a clear pipeline that guides diffusion policy training with structural priors and records skill handoff conditions and verification results, improving task success with few demonstrations.
  • The paper provides extensive baseline comparisons and ablation studies to support the effectiveness of APPL.
Critical weaknesses
  • The paper should further discuss the benefits of structural priors and APPL when demonstrations are limited but diverse. For example, when the blue block appears at varied locations rather than only on the left side of the table.
  • The evaluation is relatively small: the long-horizon experiments include only 40 cases each for object position shifts and task start/end variations, and 16 novel composition cases. This limits the evidence for the method’s reliability.
Questions for authors
  • The paper should further discuss the benefits of structural priors and APPL when demonstrations are limited but diverse. For example, when the blue block appears at varied locations rather than only on the left side of the table.
  • The evaluation is relatively small: the long-horizon experiments include only 40 cases each for object position shifts and task start/end variations, and 16 novel composition cases. This limits the evidence for the method’s reliability.
  • The paper should explain why success rates in Tab 1 decrease as the number of demonstrations increases from 5 to 20, and whether data quality contributes to this trend.
  • Training multiple policies for each skill increases training and development costs. For example, the long-horizon experiments train four policies per skill.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.