acceptodds
Under review as a conference paper at ICLR 2027

Multi-Task Reinforcement Learning with Language-Encoded Gated Policy Networks

Abstract

Multi-task reinforcement learning often relies on task metadata—such as brief natural-language descriptions—to guide behavior across diverse objectives. We present Lexical Policy Networks (LEXPOL), a language-conditioned mixture-of-policies architecture for multi-task RL. LEXPOL encodes task metadata with a text encoder and uses a learned gating module to select or blend among multiple sub-policies, enabling end-to-end training across tasks. On MetaWorld benchmarks, LEXPOL matches or exceeds strong multi-task baselines in success rate and sample efficiency, without task-specific retraining. To analyze the mechanism, we further study settings with fixed expert policies obtained independently of the gate and show that the learned language gate composes these experts to produce behaviors appropriate to novel task descriptions and unseen task combinations. These results indicate that natural-language metadata can effectively index and recombine reusable skills within a single policy.

Primary area:
reinforcement learning
Keywords:
Multi-task Reinforcement Learning, Language-Conditioned Policy Learning, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C4ZSOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes the Lexical Policy Network, a language-conditional architecture for multi-task reinforcement learning. A pretrained text encoder maps task descriptions to contextual embeddings, and a gated MLP generates softmax weights over multiple subpolicies, performing operations that are a weighted sum of their outputs. Subpolicies and gates are trained end-to-end using SAC. Separate settings freeze pretrained subpolicies and train gates on composite tasks. Experimental results show that LEXPOL achieves significant improvements over baselines on MT10 and MT50.

Critical strengths
  1. At both reported training budgets, LEXPOL achieves higher average success rates than the multi-task baselines evaluated on MetaWorld MT10 and MT50. Combining this with CARE further improves the performance of 2M steps. The standard error results for ten seeds provide useful evidence of their empirical competitiveness.

  2. Ablation changes the number of expert policies and compares six pretrained language encoders. Other experiments examined freezing versus joint training policies, integration with CARE, and MiniGrid combined courses. This coverage helps evaluate the architecture for different configurations and training setups.

Critical weaknesses
  1. The method is insufficiently specified, and its stated architecture is inconsistent with the interpretation of the main composition experiment. Section 3.1 and Figure 1 define the gate as a function only of the language context, while Section 2 states that this context is fixed for an entire episode. Thus, for a fixed task and fixed model parameters, the weights do not depend on the current state. Nevertheless, in my understanding, Figure 2 tests how the gating network selects between two already pre-trained policies when retraining the gating network on a new composite task, which does not follow from the stated architecture that the dominant gate weight should be the same throughout the state space for the same task.

  2. The experiments do not isolate the claimed contributions of language semantics, dense policy composition, or generalization, and the literature analysis is inadequate. Language only conditions the gate, so task IDs or learned task embeddings could provide the same task-disambiguation signal for the fixed training tasks. The composition experiments also involve additional training on the new tasks, rather than demonstrating generalization to unseen instructions or compositions without further learning. Missing top-1 routing and inference-cost comparisons, together with limited discussion of recent related work, further weaken the justification for the proposed design.

  3. Algorithm 1 is inadequate as a training specification: it omits the training losses and does not explain how the blended action distribution, log-probabilities, or gradients are computed under SAC, and it uses the notation aia_i and aima_i^m inconsistently. Figure 2's caption refers to top/bottom images even though the panels are left/right, and Table 2 misspells MOORE as MOOORE. The manuscript also lacks an AI-use disclosure section.

Questions for authors
  1. Does the implemented gate take only zcontextz_{context}, or does it also receive state, history, or task-progress information? Is the task description fixed throughout a composite-task episode? Please give the exact quantity visualized in Figure 2 and explain how its spatially varying colors arise under the architecture in Section 3.1. How is progress through "red goal, then blue goal" represented?

  2. How does LEXPOL perform when the language input is replaced by one-hot task IDs, learned/random task embeddings, or shuffled descriptions under otherwise identical settings? Do semantic similarities between task-description embeddings correspond to similarities in the learned gate weights or reusable skills?

  3. What success rate and inference cost are obtained when the gate selects its top-1 expert before any actor evaluation, compared with the current dense weighted-action method? Could you compare this approach with executing only the top-1 policy to assess the performance-cost trade-off?

  4. What motivated the selected policy counts in Table 8 and Table 9? It seems that the increase of policy count does not contribute to a increase performance on MT10, how does a k = 10 model, especially with task-based routing, compare with ten separately trained SAC agents under explicitly matched interaction budgets?

  5. Since LEXPOL uses language instructions to condition policy selection, have you evaluated its generalization to OOD tasks or unseen task compositions without additional training?

  6. Can you compare full-policy and internal-module routing with matched parameter counts and budgets?

Reviewer_QZWZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Lexical Policy Networks (LEXPOL), a language-conditioned architecture for multi-task reinforcement learning that uses a pretrained text encoder and a learned softmax gate to combine the action outputs of multiple sub-policies receiving a common state representation. LEXPOL is evaluated on MetaWorld MT10 and MT50 at several training budgets and performs competitive or higher relative to the evaluated multi-task baselines.

Critical strengths
  1. The method is simple, and the combination with CARE looks promising.
  2. Evaluation across task scales and training budgets is solid.
Critical weaknesses

I'm not familiar with this area. I'm happy to increase my score according to the other reviewers' comments and the authors' rebuttal.

  1. As the authors acknowledged, the proposed LEXPOL only matches or exceeds the baselines, making its effectiveness questionable.
  2. The task description is fixed within an episode, and Algorithm 1 computes the gating weights solely from its embedding, without state, history, or task-progress inputs. Therefore, for a fixed trained model and instruction, the gate should remain constant throughout the episode. However, Figure 2 is described as showing spatially varying policy dominance and is used to support sequential skill composition, which is confusing.
Questions for authors
  1. Typo: "LEXPOL matches or matches or exceeds".
  2. Whether the reported uncertainties are standard deviations or standard errors, which per-run statistic is tested, and how the significance markers are obtained.
Reviewer_Z8yYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Lexical Policy Networks (LEXPOL), a multi-task reinforcement learning method that encodes natural-language task descriptions with BERT. The resulting embedding is passed to a gating MLP, which produces softmax weights for combining the action outputs of k SAC sub-policies. The authors position this approach in contrast to CARE (Sodhani et al., 2021): CARE gates over state encoders, whereas LEXPOL gates over policy outputs. The empirical evaluation covers MetaWorld MT10 and MT50, a T-maze experiment with frozen expert policies, a hybrid LEXPOL + CARE variant, and additional experiments on Minigrid.

Critical strengths
  1. The method is simple and easy to implement.
  2. The experiments use 10 seeds and report Welch t-tests with Bonferroni correction.
  3. The appendix includes ablations on the number of sub-policies and on the choice of language encoder.
Critical weaknesses
  1. Limited technical novelty :
  • The idea of mixing multiple experts or policies through task-conditioned attention has been explored in several prior works, including AMESAC (Cheng et al., 2023), MOORE (Hendawy et al., 2024), Soft Modularization (Yang et al., 2020), and CARE (Sodhani et al., 2021). The main distinction of LEXPOL appears to be the use of pre-trained BERT embeddings in place of learned task embeddings. However, since each MetaWorld task is paired with a single fixed description, the BERT embedding may effectively serve as a task identifier rather than a source of semantic information.
  1. Overstated optimality claim of single-task SAC/PPO :
  • The claim that SAC and PPO “learn optimal performance” in single-task settings (Sec. 2–3) is not accurate. Neither algorithm guarantees optimality under deep function approximation, and SAC optimizes a maximum-entropy objective. The paper’s own results (0.93 on MT10, 0.72 on MT50) also contradict this claim. I would suggest revising the statement and describing single-task SAC as a reference rather than a strict upper bound.
  1. Readability :
  • The manuscript would benefit from careful proofreading. The phrase “matches or matches or exceeds” appears repeatedly throughout the paper, likely as an artifact of a global text replacement. There are also several typos, such as “MOOORE” (Table 2), “Attenion” (Sec. 4.1.1), “disengtanglement” (Sec. 5.2), and “solve solve” (Sec. 2). In addition, Table 12 lists the policy and Q-function learning rates twice. These issues are minor individually, but together they give the impression that the manuscript is not yet ready for publication. I would encourage the authors to revise the text thoroughly.
Questions for authors

Q1) According to Algorithm 1 (lines 6–8), the gating weights depend only on the context embedding, not on the state. The mixture weights should therefore be constant within a task. Could the authors explain how the state-dependent policy dominance in Figure 2, and the policy switching required for the sequential “red then blue” task, arise under this formulation? In particular, does the gate in fact take the state as input, and how is “policy dominance” computed?

Q2) Could the authors report results where the BERT embedding is replaced with a one-hot task ID or a learnable task embedding? Since each MetaWorld task uses a single fixed description, this comparison seems essential for isolating the contribution of the semantic content of language.

Q3) The abstract states that LEXPOL generalizes to novel task descriptions and unseen task combinations. Have the authors evaluated zero-shot performance on MetaWorld, for example with paraphrased descriptions or held-out tasks? Such results would provide more direct support for this claim than the T-maze experiment, in which the gate is retrained on the new task.

Q4) Were the baseline results reproduced by the authors under the same protocol, or taken from prior work? For instance, MOORE was originally evaluated on MT10-rand and MT50-rand with random goal positions, which differs from the protocol used here. Could the authors clarify the source and setup of each baseline, the MetaWorld version used, and whether “2 million timesteps for each environment” refers to per-task or total environment steps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.