- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a plug-in framework for safe cooperative MARL in which an LLM acts as a slow-timescale meta-controller. Every few episodes it receives a text summary of the robot's state and recent safety costs and returns a high-level cooperative strategy. The strategy is embedded and fed into each agent's observation through a learned gate, while safety is handled by a fixed reward penalty rather than Lagrangian or cost-critic methods. The framework is applied to MAPPO and HAPPO on six Safe Multi-Agent MuJoCo locomotion tasks, where LLM-HAPPO is reported to achieve higher reward than constrained MARL baselines while keeping costs generally below the limit, supported by several ablations.
Critical strengths
The paper includes a relatively broad range of ablations, including comparisons with reward-shaped HAPPO, a non-language conditioning baseline, and a rule-based strategy generator, as well as studies on the gating mechanism, sentence encoders, LLM backbones, query frequency, and the penalty scheme. These experiments cover many of the key design choices and represent a reasonable attempt to isolate which components contribute to the reported performance, although some of the results also raise additional questions discussed below.
Critical weaknesses
-
Heavy reliance on manual, task-specific design. The LLM does not directly observe the environment. Instead, its input is produced by a handcrafted, rule-based state interpreter that converts selected simulator variables into semantic descriptions such as whether the robot is upright, leaning, unstable, or fallen. Each task additionally requires manually specified agent roles, task descriptions, safety descriptions, few-shot examples, and a task-dependent penalty coefficient. Though the paper acknowledges that these abstractions are benchmark-specific and manually engineered, this substantially weakens the claim that the framework is a plug-in method (section 4 and Appendix G) as applying the approach to a new environment appears to require considerable domain knowledge and prompt engineering, rather than simply attaching an LLM module to an existing MARL algorithm.
-
No evaluation of scalability to larger multi-agent systems. All experiments use 2 to 6 agents controlling a single robot body. Scalability is a central concern in MARL, and it is unclear how the method behaves as the number of agents grows. A single shared strategy embedding is broadcast to all agents, a short LLM output must coordinate every agent, and the prompt grows with the number of agent roles. There are hints of degradation even at this small scale: the 6-agent HalfCheetah task is where both LLM-HAPPO and LLM-MAPPO show their highest and most variable costs. Without experiments on settings with many more agents, the paper's claims about improving multi-agent coordination are not well supported.
Questions for authors
- How much of the gain depends on the handcrafted state interpreter? In Table 9, the rule-based baseline uses the same handcrafted summaries without an LLM and performs close to LLM-HAPPO on several tasks. On 4×2 Ant and 6×1 HalfCheetah, it achieves better safety with comparable rewards. What does the LLM contribute beyond these human-designed rules?
- What does language add beyond an auxiliary conditioning variable? The non-language baseline in Table 4 performs poorly on several tasks, far below LLM-HAPPO. Could the authors explain how this encoder was trained and matched in capacity? As it stands, it is unclear whether the comparison isolates the effect of language or whether the baseline is under-optimized.
- How does the method scale to larger numbers of agents? All experiments use at most 6 agents. Appendix D.4 mentions Humanoid results that do not appear in the paper; were larger configurations attempted, and how did they perform?
There are a few presentation issues, particularly duplicated sentences and short paragraphs in the Related Work section and Section 4.1. These issues should be cleaned up, but they do not affect my overall evaluation of the paper.
4 comments
W2,Q3: Scalability to Larger Multi-Agent Systems
Table: Comparison of LLM-HAPPO with different safety penalty coeff and constrained MARL baselines on 9|8 Humanoid.Results are mean ± standard deviation over 5 seeds, with 10 deterministic evaluation episodes per seed and cost limit 25.
| Method | R ↑ | C ↓ | Success Rate ↑ |
|---|---|---|---|
| LLM-HAPPO() | 2011.00 ± 1196.41 | 344.24 ± 210.86 | 0.00 ± 0.00 |
| LLM-HAPPO() | 682.22 ± 49.41 | 41.26 ± 12.68 | 0.26 ± 0.25 |
| LLM-HAPPO() | 711.09 ± 171.92 | 35.38 ± 19.01 | 0.52 ± 0.40 |
| LLM-HAPPO() | 610.78 ± 71.11 | 21.38 ± 3.29 | 0.78 ± 0.34 |
| MAPPO-Lag | 607.51 ± 90.81 | 30.82 ± 3.57 | 0.30 ± 0.17 |
| MACPO | 465.53 ± 52.79 | 23.06 ± 7.53 | 0.46 ± 0.31 |
| MACUP | 563.52 ± 21.94 | 22.56 ± 7.40 | 0.50 ± 0.32 |
To investigate scalability, we extended our evaluation from 2–6 agents to the 9|8 Humanoid configuration (9 agents, each controlling 8 action dimensions). We also conducted a penalty coefficient sweep () to examine the reward–safety trade-off. LLM-HAPPO () achieves a mean reward of , mean safety cost of , and constraint-satisfaction rate of . Compared with MAPPO-Lag, MACPO, and MACUP, our method achieves higher mean reward, lower mean safety cost, and a higher constraint-satisfaction rate in this configuration. Although the prompt may grow with the number of agent roles, the additional nine-agent Humanoid results reported above demonstrate that the framework can operate beyond the original six-agent settings.
Q1:
As discussed in W1, our additional no-interpreter ablation demonstrates that LLM-guided coordination remains effective without handcrafted state interpretation. Regarding Table 9, we acknowledge that rule-based guidance achieves competitive rewards and lower safety costs on some tasks, particularly 4×2 Ant and 6×1 HalfCheetah. However, LLM-HAPPO achieves higher mean rewards across all six tasks. Unlike predefined rule mappings, the LLM generates context-dependent cooperative strategies, providing additional empirical benefits beyond handcrafted rules.
Q2: We thank the reviewer for this question. The non-language baseline uses a trainable MLP () to map the same structured state and safety features into a 32-dimensional conditioning embedding. It is trained end-to-end with the HAPPO policy using the same gating mechanism, optimizer, training budget, penalty coefficient, cost limit, and evaluation protocol.
To ensure a fair comparison, we reevaluated the learned numerical control (Table 4) using five matched random seeds. Although the updated baseline improves over the originally reported results, LLM-HAPPO achieves higher mean rewards and lower mean safety costs across all six tasks. These findings suggest that pretrained language-based strategy guidance provides benefits beyond an auxiliary learned conditioning variable, although the comparison does not isolate semantic reasoning from differences in pretrained representations and encoder capacity.
| Task | LLM-HAPPO Reward ↑ | LLM-HAPPO Cost ↓ | Learned Numerical Control Reward ↑ | Learned Numerical Control Cost ↓ |
|---|---|---|---|---|
| 2×4 Ant | 2643.87 ± 68.13 | 0.20 ± 0.27 | 2459.72 ± 445.98 | 21.63 ± 38.14 |
| 4×2 Ant | 2521.42 ± 69.41 | 0.98 ± 0.90 | 2498.90 ± 134.17 | 3.20 ± 5.15 |
| 2×3 HalfCheetah | 2624.41 ± 80.61 | 1.10 ± 2.80 | 596.88 ± 792.29 | 26.80 ± 34.26 |
| 3×1 Hopper | 1522.08 ± 190.80 | 0.80 ± 1.66 | 626.02 ± 555.64 | 25.54 ± 15.21 |
| 2×3 Walker2d | 2064.41 ± 148.74 | 2.28 ± 2.66 | 881.23 ± 667.89 | 34.59 ± 36.94 |
| 6×1 HalfCheetah | 2099.01 ± 290.24 | 15.24 ± 19.10 | 966.03 ± 1091.12 | 105.44 ± 158.18 |
Our additional experiments clarify the contribution of LLM-generated guidance beyond handcrafted state interpretation, rule-based strategies, and learned numerical conditioning. The results demonstrate consistent reward improvements across most evaluated tasks, while also highlighting task-dependent safety trade-offs. Furthermore, the nine-agent Humanoid experiments provide additional evidence of the framework's applicability to larger multi-agent configurations. We acknowledge that task-specific design remains necessary and that broader scalability evaluations are important future directions. We hope these additional results and clarifications address the reviewer's concerns and strengthen the empirical support for our proposed framework.
Thank you for the detailed response and the additional experiments. However, my concerns are not fully addressed.
- W1. Regarding the additional no-interpreter ablation, more clarification on its implementation details is needed, particularly regarding how the LLM inputs and prompts are constructed in this setting.
- Q2. Although the authors clarify the implementation of the non-language baseline, I remain concerned about the comparability of the two methods. The proposed approach uses a powerful pretrained LLM, whereas the non-language baseline relies on a relatively simple MLP trained from scratch. Matching the output dimension does not match the representational capacity or pretrained knowledge of the two systems.
I therefore maintain my score.
W1: We thank the reviewer for the thoughtful follow-up questions and constructive feedback. In the no-interpreter ablation, we remove only the handcrafted physical-state description from the LLM input. All other components of the LLM-HAPPO pipeline remain unchanged.
At every strategy-generation step, the environment state is still summarized for bookkeeping, but the task-specific human-written interpretation of physical quantities such as: torso height, pitch, roll, uprightness, falling status or recovery urgency is not passed to the LLM. The LLM instead receives the information that is common to the task also provided in the paper appendix as a prompt structure:
- the task objective, such as maintaining the target velocity;
- the number of agents and their roles;
- the safety objective and cost limit;
- the cooperative nature of the action;
- the instruction to produce a conservative, short-horizon coordination strategy;
- the recent strategy text or strategy history, when history is enabled by the original configuration.
The system prompt still instructs the model to act as a cooperative multi-agent control strategist, respect safety constraints, coordinate agent roles, and return a concise actionable strategy. The user prompt still contains the task description, role assignments, and the safety information. We do not give the model an alternative manually designed description of the physical state.
The generated strategy is processed exactly as in the main method. The text is encoded with the same frozen sentence encoder, projected to the same 32-dimensional conditioning representation, and passed through the same learned gating module. The gated representation is concatenated with the original policy observation before the actor network produces actions. The actor, critic, gate, and all HAPPO parameters are trained with the same optimizer, learning rate, reward shaping, penalty coefficient, rollout length, update schedule, LLM frequency, number of environment steps, and five random seeds as the full LLM-HAPPO condition. The LLM is queried at the same fixed frequency as in the main experiment rather than at every environment step. Low-level actions continue to depend on the instantaneous environment observation, so the ablation does not replace feedback control with open-loop language.
The purpose of this design is to isolate the contribution of the handcrafted state-to-text interpreter. The LLM remains present and retains access to task-level semantics, agent roles, and safety context, but it does not receive the human-designed interpretation of the current physical state. Therefore, any performance difference between full LLM-HAPPO and this ablation reflects the value of providing detailed state summaries to the language-conditioning pipeline, subject to the usual variance of the five-seed experiments.
Q2: Capacity and pretrained knowledge
We agree that matching only the output dimension does not make a randomly initialized MLP equivalent to a pretrained LLM. Our claim is therefore narrower than a full comparison of computational capacity or prior knowledge. The numerical baseline isolates whether an auxiliary conditioning vector, with the same interface to the policy, is sufficient to reproduce the gains. It uses the same structured state and safety features, a trainable 16→64→32 MLP, the same 32-dimensional conditioning output, projection/gating pathway, HAPPO optimizer, reward shaping, seeds, and training budget. The encoder is trained end-to-end from scratch, it is not presented as a parameter-count or knowledge-equivalent replacement form model. We will revise the paper to state this limitation explicitly. The comparison should be interpreted as testing semantic/task-conditioned guidance versus learned numerical conditioning, not as claiming equal representational capacity between a pretrained LLM and a small MLP. The no-interpreter controls further separate the effects of merely adding a conditioning pathway and of using the handcrafted interpreter. We agree that a stronger control would require a substantially larger pretrained or pretrained-free sequence model with matched compute and data, which is beyond the present study.
W1: We thank the reviewer for this observation. Our framework investigates whether high-level language guidance can improve safe coordination in multi-agent continuous control. The structured state interpreter, agent roles, and task-specific descriptions serve an important purpose: they provide the LLM with relevant environmental information, agent responsibilities, and safety requirements, enabling it to generate more informed, context-aware, and task-appropriate cooperative strategies. Rather than directly prescribing low-level actions, these components provide the context from which the LLM generates high-level guidance for decentralized agents.
Importantly, the plug-in nature of our framework refers to its compatibility with existing MARL algorithms, rather than fully automated adaptation to arbitrary environments. The language-conditioning and gating mechanism can be integrated into both HAPPO and MAPPO and other backbones without modifying their underlying policy optimization procedures. We acknowledge that constructing these task-specific descriptions requires domain knowledge. However, this design allows the LLM to receive meaningful environmental and safety context instead of relying on raw numerical observations alone. Our contribution is therefore a modular framework for incorporating LLM-generated cooperative guidance into existing MARL policies, rather than eliminating the need for environment-specific information. While the handcrafted components provide structured task and safety context, they do not directly determine the strategy generated by the LLM. The LLM uses this information to produce high-level cooperative guidance, which is then incorporated into the agents' policies through the learned gating mechanism.
To directly address the reviewer's concern about handcrafted priors, we conducted an additional ablation removing the state-to-text interpreter while retaining the LLM strategy generator, sentence encoder, gating mechanism, and HAPPO backbone. Both variants use matched training and evaluation settings, including five random seeds. The results show that LLM-HAPPO without the interpreter remains competitive, while the full model achieves higher mean rewards on five of six tasks. These findings indicate that handcrafted state abstractions can improve performance but are not the sole source of the observed gains.
Both methods used the same hyperparameter tuning and optimization settings, including the HAPPO optimizer, learning rate, training budget, penalty coefficient, cost limit, and update interval. Both were evaluated using five matched random seeds and ten episodes per seed, ensuring a fair comparison.