acceptodds
Under review as a conference paper at ICLR 2027

Multi-Level Semantic Skills for Transfer: Exploration, Control, and Adaptation across Discrete and Continuous Domains

Abstract

Hierarchical methods in reinforcement learning (RL) offer a natural substrate for transfer: a skill can keep the same semantic role even when the state representation, action space, or surrounding task changes. We exploit this principle by adapting a recently introduced multi-level RL framework, where learned policies become higher-level actions and can be reused through task-specific implementations. We develop three progressively broader forms of transfer. First, semantically meaningful skills learned in one task are reused to guide exploration and accelerate learning in related tasks. Second, hierarchical knowledge is carried from discrete state and action spaces to continuous states and continuous control by replacing lower-level implementations while preserving higher-level semantic structure. Third, learned hierarchical relationships are adapted to apparently different domains by separating reusable semantic structure from target-specific choices, including recursive decompositions. Experiments across navigation, control, and structured computation demonstrate the resulting gains in learning and adaptation when semantic structure is shared. Because these skills admit natural-language descriptions, they also provide an interpretable interface for communication, coordination, and cooperation in agentic systems.

Primary area:
reinforcement learning
Keywords:
Hierarchical reinforcement learning, transfer learning, semantic skills, continuous control, exploration, skill composition, agentic systems.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vuTTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a semantic skill transfer framework that reuses learned high-level policies across tasks with different state and action spaces. The method transfers which skill to execute, while the target uses its own low-level controller to execute the skill. It consists of (i) semantic state mapping, which maps source and target states into a shared representation; (ii) skill correspondence, which matches equivalent skills across tasks; and (iii) target adaptation, which refines the transferred policy through reinforcement learning.

The paper evaluates transfer across three settings: related tasks, discrete-to-continuous control, and different computational domains. Results show improved initial performance and reduced target learning.

Critical strengths

The paper shows that learned high-level policies can transfer not only between related tasks but also across discrete and continuous control and even different computational domains. This supports the generality of semantic skill transfer beyond conventional task-specific skill reuse.

Critical weaknesses
  • Reliance on manually defined semantic correspondence. The method assumes that semantic state mappings and skill correspondences between source and target tasks are known. However, identifying these correspondences can be challenging in more complex environments (such as high-dimensional robot manipulator control). This assumption limits the practical applicability of the framework. Could authors give clarification on this?
Questions for authors
  • Semantic correspondence: What information is required to define the semantic state mapping and skill correspondence between source and target tasks? How much task-specific knowledge must be provided?

  • Benefit of learned skill transfer: Some transfer scenarios involve relatively simple high-level sequences, such as picking up and delivering an object or recursively solving subproblems. How much benefit comes from transferring a learned policy rather than using a manually defined high-level policy?

1 comment
AuthorsOpenReview ↗

Response to Reviewer vuTT

Thank you for asking what prior knowledge is needed for transfer. The skill–embedding decomposition separates reusable knowledge from task-specific information. For each task pair, we specify a compact description of task progress and a correspondence between source and target skills. The source policy learns the higher-level rule for choosing skills from experience. The target then uses that rule while learning its own values or additional decisions. Automatically learning the descriptions and correspondences across complex tasks and multiple levels is a challenging direction for future work.

Thank you also for asking about hand-written policies. A hand-written high-level policy assumes that someone already knows the correct sequence of decisions. For complex tasks, discovering this rule can be the difficult part. Our approach learns this higher-level rule in the source task; at transfer time, the task-specific information identifies the relevant progress and matches the skills. In BabyAI, the learned and hand-written selectors succeed on the same 63 of 64 missions, with no BabyAI high-level training (Appendix D.1). Thus the learned rule achieves the same mission-level coverage as a manually constructed one using compact semantic information.

The recursive experiment illustrates the same point. A hand-written executor requires the rule for choosing and combining subproblems to be known in advance. We learn this rule in Merge Sort, and the transferred policy executes all twelve Matrix Chain evaluation problems correctly before target learning (Section 5; Figure 6). The same hierarchy trained in Matrix Chain reaches at least 95% success on every problem after 80–320 target decisions, with a median of 240. The target can also introduce a further level to learn where to split each matrix chain, while reusing the learned rule to execute the resulting decomposition.

In multi-request Taxi, the learned source rule chooses when to navigate, pick up, and drop off a passenger; the Taxi navigator carries out the movements. Together, they implement Serve(p), a skill for completing one request. A new scheduler learns which request to serve next. After 10,000 target interactions, the transferred hierarchy reaches a mean return of 34.04–34.81 across five runs, compared with 25.42 for a fixed request order (Section 3; Appendix C.2.2).

Reviewer_6e1AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors analyze the ability for transfer in hierarchical reinforcement learning. They focus on "semantic skills" and show that these transfer across in several different contexts.

Critical strengths
  • The authors provide a quite comprehensive set of both empirical and theoretical results. This exceeds the normal expectations here. I cannot find anything obviously wrong with them.
  • The problem of how to transfer tools between domains remains something that has quite a high impact potential given how cross-domain AI systems have become.
Critical weaknesses
  • Transfer in hierarchical reinforcement learning is, I believe, a well-studied topic, dating back much more than a year. The authors hint at a complete comparison being in the appendix, but when they are proposing something new in a long line of past works, this really is not sufficient. A work that represents the latest in a long line of inquiry must be positioned directly within the context of existing works early on, as otherwise it's quite impossible to assess novelty. Is the novelty here just that the skills happen to use natural language to define them (an option induces a policy as well)? Even that, I believe, has been studied before. Either way, the research question they pose is, linguistically speaking, I believe, already answered. This would certainly need to change.
  • Given the above, I struggle to actually see where these "semantic skills" would make sense, but one of the older, "orthogonal" approaches would not make sense. As these other approaches are orthogonal, I suppose a comparison might not be possible, but then it needs to be very clear WHY it would not be applicable in a certain problem setting and WHY that problem setting is necessary to be framed in that way.
Questions for authors
  • What is the exact novelty of this work? Transfer using hierarchical reinforcement learning has been heavily studied before, so how is this different? Is it just that skills are bits of natural language? Has that not been studied before? Natural language goals in RL have most certainly been a topic for well over five years.
  • From what I can understand here, you are defining a skill to be some latent value that induces a policy? Is that correct? Can you provide a formal definition of this? I think that would help me understand the novelty more.
  • Are the results presented here statistically significant?

Please also respond to the points raised in the weaknesses section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6e1A

Thank you for asking us to clarify the contribution and its relationship to earlier hierarchical reinforcement learning.

The MMDP framework of Yang and Maggioni (2026) builds higher-level actions from parameterized families of partial policies. Its skill–embedding decomposition separates reusable knowledge from the information needed to apply it to a particular task. High-level decisions in our examples often correspond to meaningful stages of task progress. The English skill names help readers understand the decisions; the learned policy uses a compact description of task progress.

This construction reflects a distinction between learning a rule and executing it. A twenty-line proof may take a year to discover but only minutes to write, while a routine thousand-line derivation may be easy to discover but take much longer to write. In the MMDP framework, difficulty is defined by the first level at which an optimal policy belongs to a prescribed policy family, rather than by execution time.

Earlier methods study different parts of transfer. Steccanella et al. (2020), for example, transfer options between regions of a state partition shared by tasks, then learn a high-level policy in the target. We transfer the learned rule for choosing matched skills through compact semantic information. Our three scenarios progress from related tasks, to direct discrete-to-continuous transfer, to abstract recursive computation. They measure how much target learning is saved by reusing the source rule. Each paired comparison holds the target hierarchy and lower-level implementation fixed, isolating the contribution of the transferred decisions. Section 6 also bounds the resulting policy-value loss under its stated assumptions. We have made this relationship to earlier work explicit in the Introduction and Appendix A.

Thank you also for asking about statistical evidence. On 40 new PointMaze layouts, transfer improves the area under the early return curve in all 40 cases (two-sided sign test: p = 1.8 × 10⁻¹²). It reduces the interactions needed to meet the success criterion on 35 layouts. Each layout contributes one paired target-training run with a fixed source policy and continuous controller. The statistical test was performed after the experiments. The PointMaze results appear in Section 4 and Figure 5. Appendix C.1 gives the statistical analysis, and Appendix C.3.3 gives the complete results for all 40 layouts.

Reviewer_6e1AOpenReview ↗

Response to authors

I appreciate the changes made by the authors and I believe they add strength to the paper. However, I still feel that the positioning relative to the HRL literature is quite poor and the one statistical test added covers only a single experiment, not the full range of results the paper relies on (also ALL statistical tests are performed after the experiments; that is normal). So I will maintain my score.

AuthorsOpenReview ↗

Clarification on HRL positioning and evidence across experiments

Thank you for taking another look and for recognizing the improvements. We appreciate your further comments and would like to make the relationship to earlier HRL and the evidence across settings more explicit.

Relationship to earlier HRL

Options provide the temporal-abstraction foundation. Andreas et al. (2017) learn reusable modules from policy sketches; transfer uses a new sketch or learns a new high-level policy in the target. Steccanella et al. (2020) transfer options between regions of a shared invariant state space and learn the target manager.

Our study uses compact semantic state descriptions and specified skill correspondences to reuse a rule learned in the source task for choosing and composing skills. Using the MMDP framework, we formulate and analyze this reuse across related tasks, discrete-to-continuous control, and recursive computation, measuring the new target learning it saves (Sections 2–5; Appendix A). Section 6 bounds policy-value loss when skill effects or implementations change, under the stated assumptions.

Evidence across experiments

Thank you for clarifying the point about statistical testing. The other scenarios also include repeated target-training comparisons:

  • In Taxi, the normalized area under the success-rate curve over the first 20,000 target interactions increases by 0.247–0.267 across five target-training runs, compared with learning the same hierarchy without transfer (Section 3; Appendix C.2.1).
  • In Matrix Chain, both methods use the same hierarchy for recursive computation. One transfers the execution rule learned in Merge Sort; the other learns an execution rule directly in Matrix Chain. The transferred rule solves all twelve evaluation problems correctly before target training. Across five target-training runs, the method trained directly in Matrix Chain reaches at least 95% success on every problem after 80–320 target decisions, with a median of 240 (Section 5; Figure 6; Appendix C.4.2).

These results describe the observed learning gains and their variation across five target-training runs in each setting.

For the PointMaze numerical results and sign test, we refer to our previous response and to Section 4, Figure 5, and Appendices C.1 and C.3.3. The two hierarchical methods use the same trained continuous controller, which remains unchanged during target learning. The transfer method reuses the same learned source policy across the new mazes. Each maze contributes one paired target-training run.

Thank you again for your time and careful assessment.

Reviewer_y9k7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyses transfer learning in the Hierarchical Reinforcement Learning settings across three different settings. Transfer learning of the skills themself, to improve exploration and accelerate learning in a related task. Transfer learning of the high level decision policy assuming a usefull function mapping the high-level skill/abstract action to the new skill in the new environment. And third transfering just the hierarchical relationship.

Critical strengths
  • I think the main strength of the paper is that it tackles a setting that is usually overlooked in the Hierarchical Reinforcement Learning literature. To the best of my knowledge, most of the transfer learning work in Hierarchical Reinforcement Learning focuses on transferring the underlying learned options, but rarely concretely asks when and how the high-level policy and structure can transfer across similar domains.
Critical weaknesses
  • The paper is hard to read and moves directly to transfer learning without concretely explaining the necessary background and formal structure of the hierarchical setting considered, such as the MDP and SMDP formulations, options as abstract actions.

  • The paper has a somewhat unusual structure, without dedicated background or conclusion sections, making it difficult to follow.

  • The limitations of the approach and its assumptions are not clearly explained in the main paper but are instead deferred to the appendix.

  • The main weakness of the paper is that it assumes access to prior knowledge to perform the transfer, without explaining how an agent could learn this prior knowledge from experience. Similarly, it does not explain how the high-level state space is learned in the first place, which is one of the main challenges and open research directions in the HRL literature.

Questions for authors
  • See weaknesses.

  • One related work that comes to mind is Hierarchical reinforcement learning for efficient exploration and transfer, I suggest the authors to look at it as a possible related work reference

2 comments
AuthorsOpenReview ↗

Response to Reviewer y9k7

Thank you for asking us to clarify the prior knowledge used for transfer. For each task pair, the task designer specifies a compact state description sufficient for the reusable decision rule and a correspondence between source skills and their target implementations (Section 2).

The source policy learns how to choose skills from this description. In the target, the agent learns values from new experience to refine transferred choices or guide additional decisions. In single-request Taxi, four task-progress flags query the source policy, while both hierarchical methods learn target values on the full 500-state space. In the three-request task, the learned passenger-service rule becomes a Serve skill, and a new policy learns the request order.

Section 6 now states the common Markov decision-state assumptions used by Theorem 1. Appendix B gives the full details. We also restored the Conclusion.

Thank you for suggesting Steccanella et al. (2020). Their method partitions a common invariant state space into regions, learns options for moving between these regions, and trains a new high-level policy in each target task.

The MMDP framework of Yang and Maggioni (2026) instead constructs higher-level actions from parameterized families of partial policies. Its skill–embedding decomposition separates reusable knowledge from task-specific information. In our examples, higher-level decisions often correspond to meaningful stages of task progress. This structure allows learned rules to be reused across tasks and levels using compact semantic information.

Our three experiments progressively broaden the transfer setting. MazeBase+ and Taxi share related task sequences; PointMaze directly transfers a discrete routing rule to continuous control; and Merge Sort and Matrix Chain are two abstract computational tasks that share a recursive-execution rule despite their different purposes. These examples show how reusable reasoning structure can connect tasks beyond a common physical environment. The Introduction and Appendix A now compare Steccanella et al.'s region-based option transfer with our reuse of learned higher-level decision rules through compact semantic information.

Automatically discovering such descriptions and skill correspondences in complex environments is a very challenging problem, particularly across multiple levels. We regard this as an important direction for future work.

Thank you for your response. I share the same concerns as reviewer 6e1A that the positioning relative to the HRL literature is poor and make it hard to asses this work.

That said, I believe the paper offers some value, as I explained in my review, by investigating when and how high-level policies and structures can transfer across similar domains. To the best of my knowledge, this aspect has not been studied in the HRL literature.

I'm inclined to maintain my score at the moment and I will look forward to discuss about this with the other reviewers in the next phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.