acceptodds
Under review as a conference paper at ICLR 2027

Language-Guided Safe Policy Optimization for Multi-Agent Locomotion control

Abstract

Can high-level language guidance improve safe coordination in multi-agent continuous control? We study this question in safety-constrained multi-agent reinforcement learning (MARL) for cooperative robotic locomotion, where multiple agents jointly control a shared body under an explicit safety budget. We introduce a framework that augments existing multi-agent policy optimisers with an episodic language-based meta-controller and reward penalisation. During training, the meta-controller periodically queries a large language model (LLM) using a compact textual summary of the system state and recent safety statistics to generate a cooperative strategy. The strategy is embedded and incorporated into each agent's observation through a learned gating module, while the underlying policy is trained under centralised training with decentralised execution (CTDE). Unlike Lagrangian constrained-MARL methods, our approach uses fixed reward penalisation and does not require additional cost critics or dual-variable updates. Experiments across six Safe Multi-Agent MuJoCo locomotion tasks show that language-guided policy optimisation improves the reward–safety trade-off across multiple robotic locomotion settings. In particular, LLM-HAPPO achieves higher rewards than standard constrained MARL baselines and a matched HAPPO variant using the same reward shaping, while generally maintaining low safety costs. The framework is also compatible with different MARL backbones. Overall, our results demonstrate the potential of episodic language guidance as a lightweight mechanism for improving safety-aware multi-agent policy optimisation without replacing the learned decentralised policies.

Primary area:
reinforcement learning
Keywords:
Safe Multi-Agent Reinforcement Learning, Large Language Models, Language-Guided Policy Optimization, Multi-Agent Coordination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ti7KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a plug-in framework for safe cooperative MARL in which an LLM acts as a slow-timescale meta-controller. Every few episodes it receives a text summary of the robot's state and recent safety costs and returns a high-level cooperative strategy. The strategy is embedded and fed into each agent's observation through a learned gate, while safety is handled by a fixed reward penalty rather than Lagrangian or cost-critic methods. The framework is applied to MAPPO and HAPPO on six Safe Multi-Agent MuJoCo locomotion tasks, where LLM-HAPPO is reported to achieve higher reward than constrained MARL baselines while keeping costs generally below the limit, supported by several ablations.

Critical strengths

The paper includes a relatively broad range of ablations, including comparisons with reward-shaped HAPPO, a non-language conditioning baseline, and a rule-based strategy generator, as well as studies on the gating mechanism, sentence encoders, LLM backbones, query frequency, and the penalty scheme. These experiments cover many of the key design choices and represent a reasonable attempt to isolate which components contribute to the reported performance, although some of the results also raise additional questions discussed below.

Critical weaknesses
  1. Heavy reliance on manual, task-specific design. The LLM does not directly observe the environment. Instead, its input is produced by a handcrafted, rule-based state interpreter that converts selected simulator variables into semantic descriptions such as whether the robot is upright, leaning, unstable, or fallen. Each task additionally requires manually specified agent roles, task descriptions, safety descriptions, few-shot examples, and a task-dependent penalty coefficient. Though the paper acknowledges that these abstractions are benchmark-specific and manually engineered, this substantially weakens the claim that the framework is a plug-in method (section 4 and Appendix G) as applying the approach to a new environment appears to require considerable domain knowledge and prompt engineering, rather than simply attaching an LLM module to an existing MARL algorithm.

  2. No evaluation of scalability to larger multi-agent systems. All experiments use 2 to 6 agents controlling a single robot body. Scalability is a central concern in MARL, and it is unclear how the method behaves as the number of agents grows. A single shared strategy embedding is broadcast to all agents, a short LLM output must coordinate every agent, and the prompt grows with the number of agent roles. There are hints of degradation even at this small scale: the 6-agent HalfCheetah task is where both LLM-HAPPO and LLM-MAPPO show their highest and most variable costs. Without experiments on settings with many more agents, the paper's claims about improving multi-agent coordination are not well supported.

Questions for authors
  1. How much of the gain depends on the handcrafted state interpreter? In Table 9, the rule-based baseline uses the same handcrafted summaries without an LLM and performs close to LLM-HAPPO on several tasks. On 4×2 Ant and 6×1 HalfCheetah, it achieves better safety with comparable rewards. What does the LLM contribute beyond these human-designed rules?
  2. What does language add beyond an auxiliary conditioning variable? The non-language baseline in Table 4 performs poorly on several tasks, far below LLM-HAPPO. Could the authors explain how this encoder was trained and matched in capacity? As it stands, it is unclear whether the comparison isolates the effect of language or whether the baseline is under-optimized.
  3. How does the method scale to larger numbers of agents? All experiments use at most 6 agents. Appendix D.4 mentions Humanoid results that do not appear in the paper; were larger configurations attempted, and how did they perform?

There are a few presentation issues, particularly duplicated sentences and short paragraphs in the Related Work section and Section 4.1. These issues should be cleaned up, but they do not affect my overall evaluation of the paper.

4 comments

W1: We thank the reviewer for this observation. Our framework investigates whether high-level language guidance can improve safe coordination in multi-agent continuous control. The structured state interpreter, agent roles, and task-specific descriptions serve an important purpose: they provide the LLM with relevant environmental information, agent responsibilities, and safety requirements, enabling it to generate more informed, context-aware, and task-appropriate cooperative strategies. Rather than directly prescribing low-level actions, these components provide the context from which the LLM generates high-level guidance for decentralized agents.

Importantly, the plug-in nature of our framework refers to its compatibility with existing MARL algorithms, rather than fully automated adaptation to arbitrary environments. The language-conditioning and gating mechanism can be integrated into both HAPPO and MAPPO and other backbones without modifying their underlying policy optimization procedures. We acknowledge that constructing these task-specific descriptions requires domain knowledge. However, this design allows the LLM to receive meaningful environmental and safety context instead of relying on raw numerical observations alone. Our contribution is therefore a modular framework for incorporating LLM-generated cooperative guidance into existing MARL policies, rather than eliminating the need for environment-specific information. While the handcrafted components provide structured task and safety context, they do not directly determine the strategy generated by the LLM. The LLM uses this information to produce high-level cooperative guidance, which is then incorporated into the agents' policies through the learned gating mechanism.

To directly address the reviewer's concern about handcrafted priors, we conducted an additional ablation removing the state-to-text interpreter while retaining the LLM strategy generator, sentence encoder, gating mechanism, and HAPPO backbone. Both variants use matched training and evaluation settings, including five random seeds. The results show that LLM-HAPPO without the interpreter remains competitive, while the full model achieves higher mean rewards on five of six tasks. These findings indicate that handcrafted state abstractions can improve performance but are not the sole source of the observed gains.

TaskFull LLM-HAPPO Reward ↑Full Cost ↓No-Interpreter Reward ↑No-Interpreter Cost ↓
2×4 Ant2643.87 ± 68.130.20 ± 0.272509.46 ± 65.620.16 ± 0.32
4×2 Ant2521.42 ± 69.410.98 ± 0.902515.28 ± 76.380.00 ± 0.00
2×3 HalfCheetah2624.41 ± 80.611.10 ± 2.802121.98 ± 345.6214.12 ± 15.47
2×3 Walker2d2064.41 ± 148.742.28 ± 2.662139.85 ± 77.700.48 ± 0.57
3×1 Hopper1522.08 ± 190.800.80 ± 1.661348.53 ± 502.384.14 ± 6.31
6×1 HalfCheetah2099.01 ± 290.2415.24 ± 19.101878.48 ± 348.514.02 ± 2.37

Both methods used the same hyperparameter tuning and optimization settings, including the HAPPO optimizer, learning rate, training budget, penalty coefficient, cost limit, and update interval. Both were evaluated using five matched random seeds and ten episodes per seed, ensuring a fair comparison.

W2,Q3: Scalability to Larger Multi-Agent Systems

Table: Comparison of LLM-HAPPO with different safety penalty coeff and constrained MARL baselines on 9|8 Humanoid.Results are mean ± standard deviation over 5 seeds, with 10 deterministic evaluation episodes per seed and cost limit 25.

MethodR ↑C ↓Success Rate ↑
LLM-HAPPO(λ=0\lambda=0)2011.00 ± 1196.41344.24 ± 210.860.00 ± 0.00
LLM-HAPPO(λ=2\lambda=2)682.22 ± 49.4141.26 ± 12.680.26 ± 0.25
LLM-HAPPO(λ=3\lambda=3)711.09 ± 171.9235.38 ± 19.010.52 ± 0.40
LLM-HAPPO(λ=5\lambda=5)610.78 ± 71.1121.38 ± 3.290.78 ± 0.34
MAPPO-Lag607.51 ± 90.8130.82 ± 3.570.30 ± 0.17
MACPO465.53 ± 52.7923.06 ± 7.530.46 ± 0.31
MACUP563.52 ± 21.9422.56 ± 7.400.50 ± 0.32

To investigate scalability, we extended our evaluation from 2–6 agents to the 9|8 Humanoid configuration (9 agents, each controlling 8 action dimensions). We also conducted a penalty coefficient sweep (λ=0,2,3,5\lambda=0,2,3,5) to examine the reward–safety trade-off. LLM-HAPPO (λ=5\lambda=5) achieves a mean reward of 610.78±71.11610.78\pm71.11, mean safety cost of 21.38±3.2921.38\pm3.29, and constraint-satisfaction rate of 0.78±0.340.78\pm0.34. Compared with MAPPO-Lag, MACPO, and MACUP, our method achieves higher mean reward, lower mean safety cost, and a higher constraint-satisfaction rate in this configuration. Although the prompt may grow with the number of agent roles, the additional nine-agent Humanoid results reported above demonstrate that the framework can operate beyond the original six-agent settings.

Q1:

As discussed in W1, our additional no-interpreter ablation demonstrates that LLM-guided coordination remains effective without handcrafted state interpretation. Regarding Table 9, we acknowledge that rule-based guidance achieves competitive rewards and lower safety costs on some tasks, particularly 4×2 Ant and 6×1 HalfCheetah. However, LLM-HAPPO achieves higher mean rewards across all six tasks. Unlike predefined rule mappings, the LLM generates context-dependent cooperative strategies, providing additional empirical benefits beyond handcrafted rules.

Q2: We thank the reviewer for this question. The non-language baseline uses a trainable MLP (16→64→3216 \rightarrow 64 \rightarrow 32) to map the same structured state and safety features into a 32-dimensional conditioning embedding. It is trained end-to-end with the HAPPO policy using the same gating mechanism, optimizer, training budget, penalty coefficient, cost limit, and evaluation protocol.

To ensure a fair comparison, we reevaluated the learned numerical control (Table 4) using five matched random seeds. Although the updated baseline improves over the originally reported results, LLM-HAPPO achieves higher mean rewards and lower mean safety costs across all six tasks. These findings suggest that pretrained language-based strategy guidance provides benefits beyond an auxiliary learned conditioning variable, although the comparison does not isolate semantic reasoning from differences in pretrained representations and encoder capacity.

TaskLLM-HAPPO Reward ↑LLM-HAPPO Cost ↓Learned Numerical Control Reward ↑Learned Numerical Control Cost ↓
2×4 Ant2643.87 ± 68.130.20 ± 0.272459.72 ± 445.9821.63 ± 38.14
4×2 Ant2521.42 ± 69.410.98 ± 0.902498.90 ± 134.173.20 ± 5.15
2×3 HalfCheetah2624.41 ± 80.611.10 ± 2.80596.88 ± 792.2926.80 ± 34.26
3×1 Hopper1522.08 ± 190.800.80 ± 1.66626.02 ± 555.6425.54 ± 15.21
2×3 Walker2d2064.41 ± 148.742.28 ± 2.66881.23 ± 667.8934.59 ± 36.94
6×1 HalfCheetah2099.01 ± 290.2415.24 ± 19.10966.03 ± 1091.12105.44 ± 158.18

Our additional experiments clarify the contribution of LLM-generated guidance beyond handcrafted state interpretation, rule-based strategies, and learned numerical conditioning. The results demonstrate consistent reward improvements across most evaluated tasks, while also highlighting task-dependent safety trade-offs. Furthermore, the nine-agent Humanoid experiments provide additional evidence of the framework's applicability to larger multi-agent configurations. We acknowledge that task-specific design remains necessary and that broader scalability evaluations are important future directions. We hope these additional results and clarifications address the reviewer's concerns and strengthen the empirical support for our proposed framework.

Thank you for the detailed response and the additional experiments. However, my concerns are not fully addressed.

  1. W1. Regarding the additional no-interpreter ablation, more clarification on its implementation details is needed, particularly regarding how the LLM inputs and prompts are constructed in this setting.
  2. Q2. Although the authors clarify the implementation of the non-language baseline, I remain concerned about the comparability of the two methods. The proposed approach uses a powerful pretrained LLM, whereas the non-language baseline relies on a relatively simple MLP trained from scratch. Matching the output dimension does not match the representational capacity or pretrained knowledge of the two systems.

I therefore maintain my score.

W1: We thank the reviewer for the thoughtful follow-up questions and constructive feedback. In the no-interpreter ablation, we remove only the handcrafted physical-state description from the LLM input. All other components of the LLM-HAPPO pipeline remain unchanged.

At every strategy-generation step, the environment state is still summarized for bookkeeping, but the task-specific human-written interpretation of physical quantities such as: torso height, pitch, roll, uprightness, falling status or recovery urgency is not passed to the LLM. The LLM instead receives the information that is common to the task also provided in the paper appendix as a prompt structure:

  • the task objective, such as maintaining the target velocity;
  • the number of agents and their roles;
  • the safety objective and cost limit;
  • the cooperative nature of the action;
  • the instruction to produce a conservative, short-horizon coordination strategy;
  • the recent strategy text or strategy history, when history is enabled by the original configuration.

The system prompt still instructs the model to act as a cooperative multi-agent control strategist, respect safety constraints, coordinate agent roles, and return a concise actionable strategy. The user prompt still contains the task description, role assignments, and the safety information. We do not give the model an alternative manually designed description of the physical state.

The generated strategy is processed exactly as in the main method. The text is encoded with the same frozen sentence encoder, projected to the same 32-dimensional conditioning representation, and passed through the same learned gating module. The gated representation is concatenated with the original policy observation before the actor network produces actions. The actor, critic, gate, and all HAPPO parameters are trained with the same optimizer, learning rate, reward shaping, penalty coefficient, rollout length, update schedule, LLM frequency, number of environment steps, and five random seeds as the full LLM-HAPPO condition. The LLM is queried at the same fixed frequency as in the main experiment rather than at every environment step. Low-level actions continue to depend on the instantaneous environment observation, so the ablation does not replace feedback control with open-loop language.

The purpose of this design is to isolate the contribution of the handcrafted state-to-text interpreter. The LLM remains present and retains access to task-level semantics, agent roles, and safety context, but it does not receive the human-designed interpretation of the current physical state. Therefore, any performance difference between full LLM-HAPPO and this ablation reflects the value of providing detailed state summaries to the language-conditioning pipeline, subject to the usual variance of the five-seed experiments.

Q2: Capacity and pretrained knowledge

We agree that matching only the output dimension does not make a randomly initialized MLP equivalent to a pretrained LLM. Our claim is therefore narrower than a full comparison of computational capacity or prior knowledge. The numerical baseline isolates whether an auxiliary conditioning vector, with the same interface to the policy, is sufficient to reproduce the gains. It uses the same structured state and safety features, a trainable 16→64→32 MLP, the same 32-dimensional conditioning output, projection/gating pathway, HAPPO optimizer, reward shaping, seeds, and training budget. The encoder is trained end-to-end from scratch, it is not presented as a parameter-count or knowledge-equivalent replacement form model. We will revise the paper to state this limitation explicitly. The comparison should be interpreted as testing semantic/task-conditioned guidance versus learned numerical conditioning, not as claiming equal representational capacity between a pretrained LLM and a small MLP. The no-interpreter controls further separate the effects of merely adding a conditioning pathway and of using the handcrafted interpreter. We agree that a stronger control would require a substantially larger pretrained or pretrained-free sequence model with matched compute and data, which is beyond the present study.

Reviewer_QSuLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an LLM-guided safe multi-agent reinforcement learning framework for cooperative robotic locomotion. The method generates high-level coordination strategies using an LLM based on handcrafted state summaries and recent safety information. The framework is integrated with existing MARL algorithms, including MAPPO and HAPPO, which are tested in Safe MuJoCo locomotion tasks.

Critical strengths
  1. The proposed framework is straightforward, well-structured, and easy to follow. The integration of LLM-generated strategies with existing MARL algorithms through a gating mechanism is clearly explained, making the overall approach intuitive and relatively easy to understand.

  2. The experimental results are promising across multiple Safe Multi-Agent MuJoCo environments.

  3. The supplementary videos provide an intuitive visualization of the learned locomotion behaviors. The demonstrated coordination and movement patterns appear impressive.

Critical weaknesses

1. Limited Novelty

The overall novelty of the proposed method appears limited. The main idea is to use an LLM to generate high-level strategies, encode them into embeddings, and concatenate these embeddings with the agents' observations through a gating mechanism. The underlying MARL algorithms remain unchanged. Overall, the approach seems to be a relatively straightforward combination of existing techniques, without substantial methodological contributions to either MARL or safe RL.

2. Unclear Motivation and Effectiveness of the Gating Mechanism

The motivation for introducing the gating mechanism is not sufficiently convincing. It is unclear why directly incorporating LLM embeddings into the observations would be insufficient. More importantly, Figure 5 shows that the gate values gradually decrease as training progresses, suggesting that the policy increasingly ignores the LLM guidance.

3. Fixed Multiplier λ\lambda

The proposed reward shaping is essentially equivalent to a Lagrangian objective with a fixed multiplier. This is a well-established approach in safe RL and does not introduce a new safety optimization mechanism. It is also unclear why the authors prefer manually selected penalty coefficients over standard primal-dual optimization, where the Lagrangian multiplier is adaptively updated according to constraint violations. Although Appendix D.8 compares fixed and adaptive penalties, the motivation for relying on task-specific fixed coefficients remains insufficiently justified. Moreover, the proposed method does not provide any guarantees of constraint satisfaction.

4. Limited Theoretical Contributions

The theoretical results in Section 4.1 and the Appendix appear rather straightforward. These results seem closer to theoretical sanity checks than substantial theoretical contributions. For example, extending the state space to include a fixed language embedding and applying standard policy gradient identities are direct consequences of existing RL theory.

5. Insufficient Ablation Studies on the Gating Mechanism

Although the authors provide an ablation study comparing gated and non-gated variants in Table 7, the results are not entirely convincing. The non-gated variant achieves slightly higher rewards on Ant and lower safety costs on Hopper, suggesting that the gating mechanism does not consistently improve performance.

Questions for authors

1. Effectiveness of Language Guidance

In Table 4, the non-language baseline performs substantially worse than LLM-HAPPO on several tasks. For example, the reward on HalfCheetah drops from approximately 2624 to 220. Could the authors explain why such a large performance gap occurs and whether the non-language baseline has been sufficiently optimized? Additionally, the rule-based baseline in Table 9 achieves competitive performance on some tasks. Could the authors further clarify what specific benefits the LLM-generated guidance provides over simpler rule-based strategies?

2. Language Guidance During Evaluation

The paper mentions that LLM strategies are periodically generated during training, while no online LLM inference is required during evaluation. Could the authors clarify how the language embeddings are obtained and used during evaluation? Are the embeddings fixed after training, or can they adapt to changing states and safety conditions? If fixed embeddings are used, how does the policy maintain the benefits of adaptive language guidance during execution?

6 comments

W1: Limited Novelty

We thank the reviewer for the constructive feedback. Our work investigates a central research question: Can high-level language guidance improve safe coordination in multi-agent continuous control?

Our contribution lies in integrating LLM-generated cooperative strategies into decentralized continuous-control policies through a hierarchical, two-timescale framework. The LLM operates at a slower timescale to generate team-level strategies from structured environmental and safety information, while decentralized actors perform fine-grained control. An observation-dependent gating mechanism enables each agent to adaptively incorporate the shared strategy embedding according to its local observations. This allows agents to use common high-level guidance while maintaining decentralized execution.

Importantly, our contribution is not a new underlying MARL optimization algorithm, but a language-guided coordination mechanism designed to complement existing CTDE-based algorithms. To examine whether the improvements arise merely from adding an auxiliary conditioning channel, we introduce a stronger learned numerical control (non-language baseline) , As shown in the second table of our response to Reviewer 1. This baseline receives the same structured state and safety features and produces a 32-dimensional conditioning vector through a trainable MLP. Both methods share the HAPPO backbone, gating mechanism, safety penalty, training budget, and evaluation protocol.

Across six Safe MaMuJoCo tasks, LLM-HAPPO achieves higher mean rewards and lower mean safety costs than the learned numerical control. The improvements are particularly pronounced on HalfCheetah, Hopper, and Walker2d, while the numerical control remains competitive on Ant. These results demonstrate that the language-guided conditioning pipeline provides empirical benefits beyond simply adding a learned numerical conditioning channel.

We emphasize that our contribution is the integration and empirical evaluation of LLM-generated strategic guidance for safe multi-agent continuous control, rather than a new MARL optimizer or formal safety algorithm. We will clarify this distinction and incorporate the additional experimental comparison in the revised manuscript.

W2: Unclear Motivation and Effectiveness of the Gating Mechanism

We thank the reviewer for this observation. The gating mechanism is designed to allow each decentralized agent to adaptively regulate the influence of high-level language guidance based on its current observation. Unlike direct concatenation, which provides the full strategy embedding to the policy at every timestep, the learned gate modulates its contribution according to the agent's local state:

\tilde{o}_t^i = [o_t^i;\alpha_t^i z_k].$$ This allows the policy to attenuate the strategy signal when it is less relevant, rather than relying on its full magnitude throughout execution. Regarding Figure 5, the decreasing gate values indicate reduced average reliance on the language-conditioning channel as training progresses. This behavior is consistent with the interpretation that high-level guidance may be more useful during early learning, while the learned policies increasingly handle routine control. However, we do not claim that the figure alone establishes this explanation or proves that the policy has internalized the language guidance. We further examine the effectiveness of gating through the gated versus no-gate ablation in Table 7. The gated variant achieves higher mean rewards on HalfCheetah (2579.03 vs. 2485.55) and Hopper (1517.89 vs. 1128.87), whereas the no-gate variant achieves a slightly higher mean reward on Ant. These findings suggest that gating can be beneficial, although its effectiveness is task-dependent rather than universal. We will clarify the motivation for gating, the interpretation of Figure 5, and the scope of the ablation results in the revised manuscript.

W3: Fixed Multiplier

Our framework incorporates safety considerations directly into the learning objective through fixed-coefficient reward shaping: rtsafe=rt−λct,r_t^{\mathrm{safe}} = r_t - \lambda c_t, where rtr_t denotes the environment reward, ctc_t is the instantaneous safety cost, and λ\lambda is a fixed penalty coefficient. We select λ\lambda through a small parameter sweep and keep it fixed across all seeds for each task. Although this objective is mathematically related to a Lagrangian formulation with a constant multiplier, the distinction lies in the optimisation procedure. Our approach directly optimises the shaped reward without maintaining a separate cost critic or performing dual-variable updates. In contrast, primal-dual methods adapt the multiplier based on estimated constraint violations.

Furthermore, our fixed-versus-adaptive penalty comparison in Appendix D.8 shows that adaptive penalties can achieve modest reward improvements in some environments but exhibit larger cost fluctuations and more frequent constraint violations. This empirical observation motivates our choice of fixed penalties in the evaluated settings.

We recognize that fixed coefficients require task-specific selection and do not provide theoretical guarantees of constraint satisfaction. Our claims therefore concern empirical safety performance rather than formal safety guarantees. We will clarify this distinction and the rationale for selecting fixed penalties in the revised manuscript.

W4: Limited Theoretical Contributions

Our theoretical analysis aims to establish a mathematical foundation for integrating episodic language guidance into multi-agent policy optimization, rather than introducing new convergence or formal safety guarantees.

Specifically, the LLM-generated strategy embedding zkz_k is held fixed during the corresponding rollout block. Under the assumption that this embedding depends only on information available before the block, we define the augmented state xt=(st,zk)x_t=(s_t,z_k). Conditional on zkz_k, the resulting process admits a standard Markov formulation, allowing existing policy-gradient and performance-difference identities to remain applicable.

The significance of this result is that language-generated strategies can be incorporated as slow-timescale contextual information within CTDE, while HAPPO/MAPPO performs low-level policy optimization without requiring a new policy-gradient derivation. The learned gate forms part of the policy parameterization and regulates the influence of this context on decentralized actors.

We emphasize that these results establish structural compatibility with standard policy optimization, rather than policy optimality, convergence, or constraint-satisfaction guarantees. The effectiveness of language-guided coordination is instead evaluated empirically. We will clarify this distinction and the assumptions underlying the analysis in the revised manuscript.

W5: Insufficient Ablation Studies on the Gating Mechanism

We thank the reviewer for this observation. The gating mechanism is designed to adaptively regulate the influence of language guidance based on each agent's local observation, rather than guarantee improvements across all tasks.

As shown in Table 7, the gated variant achieves higher rewards on HalfCheetah (2579.03 vs. 2485.55) and Hopper (1517.89 vs. 1128.87), along with lower safety costs on HalfCheetah (15.52 vs. 23.16). We acknowledge that the non-gated variant achieves slightly higher rewards on Ant and lower safety costs on Hopper.

These findings indicate that the benefits of gating are task-dependent. Nevertheless, the results demonstrate that observation-dependent gating can improve performance in certain environments. We will clarify this interpretation and the limitations of the ablation study in the revised manuscript.

Q1: To strengthen the comparison, we reevaluated the learned numerical control (non-language, Table 4) using five matched random seeds. Although the updated baseline performs better, a substantial performance gap remains. The learned numerical control is trained end-to-end and is neither frozen nor randomly initialized without optimization. It receives the same structured state and safety features as the language pipeline and maps them into a 32-dimensional conditioning representation through a trainable MLP 16→64→3216 \rightarrow 64 \rightarrow 32. The resulting representation is incorporated through the same gating mechanism and actor interface. Both methods use the same HAPPO optimizer, training budget, penalty coefficient, cost limit, conditioning dimension, five random seeds, and evaluation protocol. Therefore, the observed performance gap cannot be attributed to fewer evaluation seeds or a shorter training budget for the numerical baseline.

The numerical MLP must learn a useful mapping from structured state and safety features to coordination signals through policy-gradient optimization. This may be challenging in long-horizon continuous-control tasks, particularly HalfCheetah, Walker2d, and Hopper. In contrast, the LLM provides pretrained knowledge for generating high-level coordination strategies, a plausible explanation is that the LLM supplies a pretrained high-level strategy prior, whereas the numerical encoder must learn its representation solely through the RL objective. Regarding the rule-based baseline, Table 9 shows that predefined strategies can be competitive on some tasks; however, LLM-HAPPO achieves higher mean rewards across all six tasks, suggesting that context-dependent language guidance provides additional benefits beyond fixed strategy mappings.

As shown in the table below, LLM-HAPPO achieves higher mean rewards and lower mean safety costs across all six tasks, with lower reward variability. While the learned numerical control remains competitive on Ant, the improvements are particularly pronounced on HalfCheetah, Hopper, and Walker2d. These findings provide additional evidence that the language-mediated strategy-generation pipeline offers benefits beyond simply introducing an auxiliary learned conditioning channel.

TaskLLM-HAPPO Reward ↑LLM-HAPPO Cost ↓Learned Numerical Control Reward ↑Learned Numerical Control Cost ↓
2×4 Ant2643.87 ± 68.130.20 ± 0.272459.72 ± 445.9821.63 ± 38.14
4×2 Ant2521.42 ± 69.410.98 ± 0.902498.90 ± 134.173.20 ± 5.15
2×3 HalfCheetah2624.41 ± 80.611.10 ± 2.80596.88 ± 792.2926.80 ± 34.26
3×1 Hopper1522.08 ± 190.800.80 ± 1.66626.02 ± 555.6425.54 ± 15.21
2×3 Walker2d2064.41 ± 148.742.28 ± 2.66881.23 ± 667.8934.59 ± 36.94
6×1 HalfCheetah2099.01 ± 290.2415.24 ± 19.10966.03 ± 1091.12105.44 ± 158.18

Q2: Clarification on Language Guidance During Evaluation

Training-time language guidance: During training, the LLM generates a high-level cooperative strategy every 10 training episodes using structured summaries of the environment state and recent safety statistics. Each strategy is encoded into a 32-dimensional embedding and supplied to the agents through learned gating mechanisms. The embedding is retained between successive updates. Consequently, agents are trained under periodically changing language guidance, allowing the policy and gating networks to learn how to use strategy information across different training conditions.

Evaluation-time embedding: At evaluation, no additional LLM queries are made. The strategy text associated with the selected training checkpoint is saved and re-encoded when the checkpoint is loaded to reconstruct the 32-dimensional embedding zevalz_{\mathrm{eval}}. This embedding remains fixed throughout evaluation and is not regenerated in response to changes in the environment or safety conditions.

Adaptation without online LLM inference: Although zevalz_{\mathrm{eval}} is fixed, its contribution to the policy is observation-dependent. At each timestep tt, agent ii computes

\tilde{o}_t^i = [o_t^i;\,\alpha_t^i z_{\mathrm{eval}}],$$ where $g_{\phi_i}$ is the learned gating network and $\alpha_t^i \in [0,1]$ controls the contribution of the language embedding. The action is then selected according to $$a_t^i \sim \pi_{\theta_i}(\cdot \mid \tilde{o}_t^i).$$ Thus, the policy responds to changing observations through both its observation-dependent action selection and the dynamically computed gate. In particular, the gate can increase or attenuate the contribution of the stored strategy depending on the current observation. However, it does not modify the semantic content of the embedding or generate a new strategy.
Reviewer_6TbbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an LLM-guided framework for safe multi-agent reinforcement learning (MARL) in continuous robotic locomotion. A frozen LLM periodically generates high-level cooperative strategies from handcrafted natural-language summaries of the robot state, agent roles, and recent safety statistics. These strategies are encoded into fixed-dimensional embeddings and injected into decentralized policies through a learnable gating mechanism. Safety is addressed through fixed reward penalization, without additional cost critics or dual-variable updates.

The framework is evaluated on six Safe Multi-Agent MuJoCo locomotion tasks using MAPPO and HAPPO, with comparisons against constrained MARL baselines and ablations involving reward shaping, non-language conditioning, rule-based guidance, and alternative LLMs. LLM-HAPPO achieves higher mean rewards than a reward-shaped HAPPO baseline across four matched tasks, while generally maintaining low safety costs. However, safety improvements are not uniform, and the rule-based baseline performs comparably on some tasks. The results suggest that LLM-generated guidance can improve empirical performance, but the extent to which these gains arise from semantic reasoning rather than handcrafted state abstractions and strategy priors remains unclear. The method also provides no formal safety guarantee.

Critical strengths
  1. A modular approach to integrating high-level guidance into MARL. The framework separates episodic strategy generation from low-level continuous control, incorporating LLM-generated guidance through a learnable gating mechanism without modifying the underlying policy optimizer. This design is relatively lightweight and compatible with multiple MARL backbones, making it a potentially useful direction for exploring high-level guidance in safe control.

  2. Broad empirical evaluation with relevant ablations. The paper evaluates six Safe Multi-Agent MuJoCo tasks and includes comparisons against constrained MARL baselines, reward-shaped HAPPO, non-language conditioning, and rule-based guidance. LLM-HAPPO consistently improves mean reward over reward-shaped HAPPO on four matched tasks, while generally maintaining low safety costs. The ablations also expose important limitations, including the competitive performance of rule-based guidance on some tasks, providing a useful basis for assessing the method's actual contribution.

Critical weaknesses
  1. The contribution of LLM-based reasoning is not sufficiently isolated. The state-to-text interpreter relies on handcrafted, task-specific heuristics, while the system prompts explicitly encode high-level control rules, such as prioritizing balance recovery when unstable. The rule-based baseline achieves comparable performance on some tasks, yet the experiments do not establish whether the gains arise from the semantic content of LLM-generated strategies or from the handcrafted abstractions and strategy priors. The non-language comparison is also inconclusive, given its limited number of seeds and substantial variance. This makes the central contribution of using an LLM, rather than a simpler rule-based or learned conditioning module, unclear.

  2. The safety claims are not sufficiently supported by the proposed optimization method. Safety is handled through fixed reward penalization rather than explicit constraint enforcement, and the authors acknowledge that the method provides no formal safety guarantee. Moreover, safety-cost improvements over reward-shaped HAPPO are not consistent across tasks. Given the paper's emphasis on safe policy optimization, stronger evidence of reliable constraint satisfaction, including variability and violation rates, is needed to substantiate its safety-related claims.

Questions for authors
  1. Contribution of LLM reasoning: Given that the state-to-text interpreter and system prompts already encode task-specific abstractions and high-level control rules, what evidence supports the claim that LLM-generated strategies provide benefits beyond these handcrafted priors? In particular, can the authors clarify why the rule-based baseline performs comparably on some tasks and whether the observed gains are attributable to the semantic content of the generated strategies?

  2. Fairness and robustness of the non-language comparison: Table 4 reports substantially worse and more variable results for the non-language baseline, evaluated with three seeds versus five for LLM-HAPPO. Were both methods given comparable hyperparameter tuning and optimization opportunities? A matched-seed comparison would materially affect confidence in the claimed advantage of language-based conditioning.

4 comments

Q1,W1: Contribution of LLM-Generated Guidance Beyond Handcrafted Priors

We thank the reviewer for the constructive feedback. Our framework investigates whether LLM-generated high-level strategies can improve safe multi-agent coordination. The handcrafted interpreter and prompts provide environmental and safety context, while the LLM generates cooperative strategies that guide decentralized agents through the learned gating mechanism.

Ablation of the handcrafted state-to-text interpreter.

To directly address the reviewer's concern about handcrafted priors, we conducted an additional ablation removing the state-to-text interpreter while retaining the LLM strategy generator, sentence encoder, gating mechanism, and HAPPO backbone. Both variants use matched training and evaluation settings, including five random seeds.

TaskFull LLM-HAPPO Reward ↑Full Cost ↓No-Interpreter Reward ↑No-Interpreter Cost ↓
2×4 Ant2643.87 ± 68.130.20 ± 0.272509.46 ± 65.620.16 ± 0.32
4×2 Ant2521.42 ± 69.410.98 ± 0.902515.28 ± 76.380.00 ± 0.00
2×3 HalfCheetah2624.41 ± 80.611.10 ± 2.802121.98 ± 345.6214.12 ± 15.47
2×3 Walker2d2064.41 ± 148.742.28 ± 2.662139.85 ± 77.700.48 ± 0.57
3×1 Hopper1522.08 ± 190.800.80 ± 1.661348.53 ± 502.384.14 ± 6.31
6×1 HalfCheetah2099.01 ± 290.2415.24 ± 19.101878.48 ± 348.514.02 ± 2.37

The full LLM-HAPPO achieves higher mean rewards on five of six tasks, while the no-interpreter variant achieves lower mean costs on four tasks. Importantly, the no-interpreter variant remains competitive, particularly on Ant and Walker2d. These results demonstrate that handcrafted state abstractions can improve performance in certain environments, but the language-guided framework does not depend exclusively on the handcrafted interpreter. The no-interpreter control demonstrates that LLM conditioning can remain effective from task-level and safety context alone, while the full model retains the advantage of explicit physical-state summaries when fine-grained state information is important. Thus, the interpreter is useful but not the sole source of performance.

Comparison with rule-based guidance. Table 9 compares LLM-HAPPO against a deterministic rule-based strategy generator using the same state summaries, sentence encoder, gating mechanism, and HAPPO backbone. LLM-HAPPO achieves higher mean rewards across all six tasks while maintaining mean safety costs below the prescribed budget of 25. Although the rule-based baseline achieves lower costs on some tasks, both methods remain within the mean cost budget. These results demonstrate the empirical benefit of LLM-generated strategies over predefined strategy mappings.

W2: Empirical Safety Performance and Constraint Satisfaction

We agree that fixed reward penalization should not be described as explicit constraint enforcement. Our method optimizes the shaped objective r′=r−λcr' = r - \lambda c with a task-specific fixed penalty; it does not maintain a learned Lagrange multiplier, an auxiliary cost critic, or a formal feasibility guarantee. The role of the penalty is therefore to bias policy learning toward lower-cost behavior, not to guarantee that every trajectory satisfies the cost limit.

Comparison with reward-shaped HAPPO: We acknowledge that safety-cost improvements over HAPPO+RS are not uniform across tasks. As reported in our original experiments, LLM-HAPPO achieves lower mean safety costs on 2×4 Ant (0.200.20 vs. 0.380.38), 2×3 HalfCheetah (1.101.10 vs. 2.722.72), and 2×3 Walker2d (2.282.28 vs. 3.313.31). On 3×1 Hopper, HAPPO+RS achieves a lower mean cost (0.120.12 vs. 0.800.80). However, LLM-HAPPO achieves higher mean rewards across all four tasks, while maintaining mean costs below the prescribed budget of 25.

Safety variability and constraint satisfaction. Our experiments report mean safety costs and standard deviations across five random seeds, allowing the variability of safety performance to be assessed. We emphasize that mean costs below the budget do not imply zero constraint violations, and our method does not guarantee per-episode feasibility. Accordingly, we interpret the results as empirical evidence of safety-aware learning rather than strict constraint satisfaction. We will clarify that the safety claims to distinguish empirical improvements from formal guarantees. Our contribution is that LLM-generated guidance can improve coordination and reward performance while maintaining low mean safety costs across several tasks.

Q2, W1: Comparison with learned numerical conditioning. To determine whether the gains from LLM-HAPPO arise from language-based guidance rather than merely from adding an auxiliary conditioning channel, we further reevaluate a learned numerical control(non language, Table 4) under a matched training protocol. The control replaces only the LLM generation and sentence-encoding pipeline. It receives the same structured state and safety features used to construct the language prompt, and maps the resulting 16-dimensional feature vector through a trainable 16→64→3216\rightarrow64\rightarrow32 multilayer perceptron. Its 32-dimensional output is passed through the same learned gate and concatenated with each decentralized actor's observation. The numerical encoder is optimized jointly with the HAPPO actor objective and is checkpointed and restored for evaluation; it is therefore a learned control, rather than a frozen or randomly initialized feature map. We evaluated both methods using the same five random seeds, with ten evaluation episodes per seed.

As shown in the table below, LLM-HAPPO achieves higher mean rewards and lower mean safety costs across all six tasks, with lower reward variability. While the learned numerical control remains competitive on Ant, the improvements are particularly pronounced on HalfCheetah, Hopper, and Walker2d. These findings provide additional evidence that the language-mediated strategy-generation pipeline offers benefits beyond simply introducing an auxiliary learned conditioning channel.

TaskLLM-HAPPO Reward ↑LLM-HAPPO Cost ↓Learned Numerical Control Reward ↑Learned Numerical Control Cost ↓
2×4 Ant2643.87 ± 68.130.20 ± 0.272459.72 ± 445.9821.63 ± 38.14
4×2 Ant2521.42 ± 69.410.98 ± 0.902498.90 ± 134.173.20 ± 5.15
2×3 HalfCheetah2624.41 ± 80.611.10 ± 2.80596.88 ± 792.2926.80 ± 34.26
3×1 Hopper1522.08 ± 190.800.80 ± 1.66626.02 ± 555.6425.54 ± 15.21
2×3 Walker2d2064.41 ± 148.742.28 ± 2.66881.23 ± 667.8934.59 ± 36.94
6×1 HalfCheetah2099.01 ± 290.2415.24 ± 19.10966.03 ± 1091.12105.44 ± 158.18

Both methods used the same hyperparameter tuning and optimization settings, including the HAPPO optimizer, learning rate, training budget, penalty coefficient, cost limit, and update interval. Both were evaluated using five matched random seeds and ten episodes per seed, ensuring a fair comparison. Overall, these additional experiments strengthen the evidence for the contribution of LLM-generated guidance. Compared with rule-based strategies and learned numerical conditioning, LLM-HAPPO achieves higher mean rewards across all six tasks while maintaining mean safety costs below the prescribed budget. Furthermore, LLM-HAPPO exhibits lower reward variability than the learned numerical control across all six tasks. The no-interpreter ablation further shows that LLM-HAPPO achieves higher mean rewards on five of six tasks, while the no-interpreter variant remains competitive and achieves lower mean safety costs on four tasks. These findings suggest that handcrafted state abstractions can be beneficial but are not essential for effective language-guided coordination, supporting the empirical effectiveness of the proposed framework.

Thank you for the detailed clarification and additional experiments. I appreciate the effort the authors have made to address the concerns.

Regarding the no-interpreter ablation, I now understand that the handcrafted physical-state descriptions are removed from the LLM input, rather than being replaced with raw observations or another numerical representation. The LLM still receives task-level information, agent roles, safety objectives, and strategy history, while the underlying MARL policy continues to receive real-time environment observations for low-level control. This resolves my concern about the implementation of this ablation.

However, I think this distinction also has important implications for interpreting the results. The ablation demonstrates that task-level LLM-generated strategic conditioning can remain useful without explicit physical-state descriptions. However, it does not demonstrate that the LLM can independently understand or reason about real-time physical states. In fact, without access to such information, the LLM's role appears closer to providing a task-conditioned strategic prior than performing state-dependent strategic reasoning.

This also leaves an important question insufficiently answered: What makes LLM-generated strategies more useful than human-designed strategies in this setting? The rule-based baseline shows that LLM-HAPPO achieves higher mean rewards across all six tasks, which is encouraging. However, the rule-based baseline achieves lower mean safety costs on several tasks, and the current results do not sufficiently explain what unique capability of LLM-based strategy generation accounts for the observed improvements.

Similarly, the updated numerical baseline provides a more consistent comparison in terms of seeds and training protocols. However, as the authors acknowledge, a small MLP trained from scratch is not a capacity- or knowledge-matched comparison to a pretrained LLM. Therefore, these results provide evidence for the effectiveness of the proposed language-mediated pipeline over this particular numerical conditioning baseline, but do not establish that language-based reasoning itself is the essential source of the improvement.

I appreciate that the authors have also clarified the limitations of fixed reward penalization and narrowed their safety claims accordingly. The additional experiments and clarifications have certainly improved the empirical evidence and the presentation of the work.

Nevertheless, my main concern remains that the paper does not sufficiently establish the necessity or unique contribution of LLM-based reasoning for safe multi-agent coordination, beyond providing an additional task-conditioned strategy signal. This is a central aspect of the paper's motivation and claimed contribution, rather than merely a missing implementation detail.

Overall, I appreciate the authors' efforts and consider several of my original concerns addressed. However, the remaining issue affects my assessment of the paper's core contribution. I therefore maintain my original rating.

We thank the reviewer for the thoughtful follow-up and for acknowledging our additional experiments and clarifications. We would like to further clarify our contribution.

In no-interpreter setting, the LLM receives task-level information, agent roles, safety objectives, and strategy history, but not handcrafted physical-state descriptions. The decentralised policies continue to receive real-time observations for low-level control. Therefore,this ablation demonstrates that LLM-generated task-conditioned strategies can remain useful without explicit physical state interpretation, rather than establishing independent LLM reasoning about real-time physical states.

Contribution of LLM-generated strategies beyond handcrafted rules: Rule-based comparison is designed to examine the source of high-level strategy guidance. Both variants use the same handcrafted state summaries, sentence encoder, conditioning interface, learned gating mechanism, HAPPO backbone, reward penalty, and training protocol. The key difference is the strategy generator: a deterministic human-designed rule mapping versus a pretrained LLM that periodically generates cooperative strategies. Under this comparison, LLM-HAPPO achieves higher mean rewards across all six tasks. This provides empirical evidence that LLM-generated guidance offers benefits beyond the specific handcrafted strategy mappings evaluated in our experiments. We acknowledge that the rule-based baseline achieves lower mean safety costs on some tasks. In safe RL, agents are allowed to optimize rewards subject to a predefined safety-cost budget. Our results show higher mean rewards while maintaining mean safety costs below this budget across all evaluated tasks.

We also agree that a numerical encoder trained from scratch is not equivalent to a pretrained LLM in representational capacity or prior knowledge. This comparison is intended to examine whether an additional learned conditioning pathway alone can reproduce the observed improvements. The results show that the full LLM-guided pipeline outperforms this numerical baseline under matched training and evaluation settings, although they do not establish that semantic reasoning is the sole cause of the improvement.

Our evidence supports the practical value of pretrained LLM-generated strategic conditioning over the specific human-designed rule-based and learned numerical alternatives tested here. It does not establish that LLM reasoning is uniquely necessary for safe MARL, and we will remove wording that could be read as making that stronger claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.