- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The research background of this paper is cooperative multi-agent reinforcement learning (MARL), with experimental scenarios set in the StarCraft micromanagement benchmarks SMAC and SMACv2, for example, 5 allied units versus 6 enemy units. The trained reinforcement learning policies are able to decide how each unit should move and attack based on observations. In this scenario, an LLM advisor is added as a global consultant that provides big-picture decisions. Against this background, the authors argue that this form of cooperation has several problems: the advisor's instructions may not account for the actual situation of the units, the units may fully accept a major instruction from the advisor even when doing so is problematic, and the advisor receives no feedback after issuing instructions, making it difficult to target its execution in the next round. They therefore design three dimensions: D1 decision authority: who decides whether to adopt the suggestion? D2 semantic granularity: at what granularity is the suggestion adopted? D3 feedback mechanism: what feedback can the advisor see? On this basis, the authors propose CLoGA, whose main contribution is a closed-loop mechanism that can selectively adopt LLM suggestions and feed execution conditions back to the LLM. Specifically, the authors do not modify the parameters of the original reinforcement learning policy; instead, they insert a learnable gating module into the policy's action generation process and divide the LLM's suggestions into four semantic dimensions, namely formation, maneuver, target selection, and context. The policy can decide, separately for each dimension, the degree to which the suggestion is adopted based on the current actual state of the units and the content of the suggestion, rather than directly accepting or rejecting the entire suggestion. In addition, the authors feed feedback information for each dimension back to the LLM, enabling it to adjust the next round of suggestions according to the execution of the previous round's suggestions. In terms of experiments, the authors mainly validate the method on 8 maps of SMAC and further test it in some scenarios of HAPPO and SMACv2. The experiments compare CLoGA with the original reinforcement learning policy, fixed gating, no gating, shuffled suggestions, and removal of the feedback mechanism, and analyze the role of different designs through ablation experiments. The experimental results show that CLoGA achieves an average win rate of 0.970 on the selected SMAC maps, higher than the original policy's 0.888; it also achieves certain improvements in HAPPO and SMACv2 scenarios. The experiments also find that directly injecting LLM suggestions without effective gating may lead to performance degradation on some maps, indicating that selective adoption of suggestions has certain value.
Critical strengths
The authors propose a fine-grained, learnable gating mechanism that enables downstream agents to selectively incorporate the LLM advisor's suggestions across different semantic dimensions, which is quite human-like. Experiments on SMAC show a substantial improvement over the original MAPPO baseline, with the average win rate increasing from 0.888 to 0.970 across the eight evaluated maps, providing empirical evidence for the effectiveness of the proposed approach.
Critical weaknesses
The novelty is limited. The core components mentioned in the paper, such as learnable gating and LLM-RL feedback, already have related ideas in the literature. The current contribution is more like a targeted integration of existing techniques, and the core idea of assigning learnable weights to different types of inputs has long existed. In addition, the paper does not conduct sensitivity experiments on the suggestion refresh interval, which the authors explicitly list as an experiment not yet completed. This means that although the current performance gains are supported by experiments, it remains unclear how sensitive the results are to key design choices, and it is also difficult to judge whether the advantages of the method can be stably generalized to a broader range of tasks.
Questions for authors
Section 4.1 of the paper mentions the use of κ=8 as the advisory refresh interval but does not report a sensitivity analysis. Considering the trade-off among instruction freshness, feedback, and cost, could the authors clarify whether the gains are robust under different refresh intervals?
2 comments
Dear Reviewer xrhX,
We would like to kindly follow up on our previous responses. We would appreciate it if you could let us know whether any further questions, comments, or clarifications are required.
Thank you.
Best regards,
Authors
Response to the Review
Thank you for the positive assessment. We now report the refresh-interval sensitivity you asked for: κ = 4 matches κ = 8 (difference ≤ 0.001), κ = 16 loses 0.007, and every setting stays above the frozen base on all three maps. On novelty, we agree that learned gating and LLM feedback each exist in the literature; our claim concerns how they are aligned, and new controls test that claim.
Q1: sensitivity to the refresh interval κ. We ran κ ∈ {4, 8, 16} with the same 3 seeds and 200k-step protocol on three maps chosen to differ in tactics: 3m (short engagements), 5m_vs_6m (mid-battle maneuver switching) and MMM2 (heterogeneous roles), as reported in Sec. 4.1 and Appendix Table S6a:
A refresh also fires whenever a directive expires, so κ bounds the delay between advice–feedback exchanges rather than a directive's lifetime. Shortening it to 4 nearly doubles the query cost without a gain, consistent with an 8-step window already giving a stable feedback signal since the advisor itself assigns durations of 8.0–10.2 steps on average (Appendix D.7). Lengthening it to 16 delays feedback and costs 0.007. The effect is small next to the gains over the base: even at κ = 16 the win rates (0.986 / 0.941 / 0.951) exceed both the frozen base (0.740 / 0.890 / 0.930) and the unfrozen-training control (0.862 / 0.914 / 0.938, Table 2b). We therefore read κ = 8 as the cheapest setting on a plateau and do not claim it is optimal. Sec. 4.1 and Appendix D.8 are revised accordingly.
W1: novelty. Learned gates and LLM feedback each exist in prior work, and Sec. 2 now says so, including residual policy learning. The contribution we claim is the alignment between the gate partition and the advisor's vocabulary: it makes the executor's per-dimension adoption both a filter on the advice and a readable signal returned to the advisor. A generic gate only filters. Here the same partition also shows the advisor which part of its plan the executor followed, so the loop can correct one field instead of the whole directive. Three results test this claim rather than assume it.
Together, these results show that aligning per-head gates with the advisor's vocabulary is what lets the executor filter imperfect advice per dimension and return a useful feedback signal while keeping the base policy intact.