- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a semantic skill transfer framework that reuses learned high-level policies across tasks with different state and action spaces. The method transfers which skill to execute, while the target uses its own low-level controller to execute the skill. It consists of (i) semantic state mapping, which maps source and target states into a shared representation; (ii) skill correspondence, which matches equivalent skills across tasks; and (iii) target adaptation, which refines the transferred policy through reinforcement learning.
The paper evaluates transfer across three settings: related tasks, discrete-to-continuous control, and different computational domains. Results show improved initial performance and reduced target learning.
Critical strengths
The paper shows that learned high-level policies can transfer not only between related tasks but also across discrete and continuous control and even different computational domains. This supports the generality of semantic skill transfer beyond conventional task-specific skill reuse.
Critical weaknesses
- Reliance on manually defined semantic correspondence. The method assumes that semantic state mappings and skill correspondences between source and target tasks are known. However, identifying these correspondences can be challenging in more complex environments (such as high-dimensional robot manipulator control). This assumption limits the practical applicability of the framework. Could authors give clarification on this?
Questions for authors
-
Semantic correspondence: What information is required to define the semantic state mapping and skill correspondence between source and target tasks? How much task-specific knowledge must be provided?
-
Benefit of learned skill transfer: Some transfer scenarios involve relatively simple high-level sequences, such as picking up and delivering an object or recursively solving subproblems. How much benefit comes from transferring a learned policy rather than using a manually defined high-level policy?
Response to Reviewer vuTT
Thank you for asking what prior knowledge is needed for transfer. The skill–embedding decomposition separates reusable knowledge from task-specific information. For each task pair, we specify a compact description of task progress and a correspondence between source and target skills. The source policy learns the higher-level rule for choosing skills from experience. The target then uses that rule while learning its own values or additional decisions. Automatically learning the descriptions and correspondences across complex tasks and multiple levels is a challenging direction for future work.
Thank you also for asking about hand-written policies. A hand-written high-level policy assumes that someone already knows the correct sequence of decisions. For complex tasks, discovering this rule can be the difficult part. Our approach learns this higher-level rule in the source task; at transfer time, the task-specific information identifies the relevant progress and matches the skills. In BabyAI, the learned and hand-written selectors succeed on the same 63 of 64 missions, with no BabyAI high-level training (Appendix D.1). Thus the learned rule achieves the same mission-level coverage as a manually constructed one using compact semantic information.
The recursive experiment illustrates the same point. A hand-written executor requires the rule for choosing and combining subproblems to be known in advance. We learn this rule in Merge Sort, and the transferred policy executes all twelve Matrix Chain evaluation problems correctly before target learning (Section 5; Figure 6). The same hierarchy trained in Matrix Chain reaches at least 95% success on every problem after 80–320 target decisions, with a median of 240. The target can also introduce a further level to learn where to split each matrix chain, while reusing the learned rule to execute the resulting decomposition.
In multi-request Taxi, the learned source rule chooses when to navigate, pick up, and drop off a passenger; the Taxi navigator carries out the movements. Together, they implement Serve(p), a skill for completing one request. A new scheduler learns which request to serve next. After 10,000 target interactions, the transferred hierarchy reaches a mean return of 34.04–34.81 across five runs, compared with 25.42 for a fixed request order (Section 3; Appendix C.2.2).