acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Listen: Closed-Loop LLM Advisory for Cooperative Multi-Agent Reinforcement Learning

Abstract

Large language models (LLMs) can serve as tactical advisors in cooperative multi-agent reinforcement learning (MARL), yet existing advisory channels fix the adoption decision outside the executing agents and provide no feedback on whether advice was followed or effective. We organize the interface around three dimensions: who decides adoption (D1), at what semantic granularity (D2), and what feedback the advisor receives (D3). We propose Closed-Loop Gated Advisory (CLOGA), a lightweight adapter for frozen recurrent executors that resolves all three: (i) semantic gating decomposes each directive into heads and lets each agent modulate adoption per dimension through learned, state-conditioned gates; (ii) a feedback channel returns per-head gate coefficients, relative reward, and situation drift to the frozen advisor for in-context revision without weight updates; and (iii) corrections are bounded residuals at the action readout rather than writes into recurrent memory. The system assumes a centralized advisory service at execution while preserving local recurrent backbones; this is an explicit channel assumption, not strictly communication-free decentralized execution. We establish conditional exact readout recovery and a readout-state deviation bound; both are pointwise results at a fixed history, not guarantees about joint-policy return. On SMAC and SMACv2 with MAPPO and HAPPO executors, CLOGA raises the weakest base by +0.25 and the imperfect-tier average by +0.09, while ungated injection of the same advice degrades three of four controlled maps; in a gate-type×feedback comparison, per-head gating converts feedback into a +0.024 gain against +0.006 for an otherwise identical learned scalar gate, suggesting that granularity and observability interact.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Reinforcement Learning, Large Language Model Advisor, Closed-Loop Feedback, Semantic Gating, Cooperative Coordination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xrhXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The research background of this paper is cooperative multi-agent reinforcement learning (MARL), with experimental scenarios set in the StarCraft micromanagement benchmarks SMAC and SMACv2, for example, 5 allied units versus 6 enemy units. The trained reinforcement learning policies are able to decide how each unit should move and attack based on observations. In this scenario, an LLM advisor is added as a global consultant that provides big-picture decisions. Against this background, the authors argue that this form of cooperation has several problems: the advisor's instructions may not account for the actual situation of the units, the units may fully accept a major instruction from the advisor even when doing so is problematic, and the advisor receives no feedback after issuing instructions, making it difficult to target its execution in the next round. They therefore design three dimensions: D1 decision authority: who decides whether to adopt the suggestion? D2 semantic granularity: at what granularity is the suggestion adopted? D3 feedback mechanism: what feedback can the advisor see? On this basis, the authors propose CLoGA, whose main contribution is a closed-loop mechanism that can selectively adopt LLM suggestions and feed execution conditions back to the LLM. Specifically, the authors do not modify the parameters of the original reinforcement learning policy; instead, they insert a learnable gating module into the policy's action generation process and divide the LLM's suggestions into four semantic dimensions, namely formation, maneuver, target selection, and context. The policy can decide, separately for each dimension, the degree to which the suggestion is adopted based on the current actual state of the units and the content of the suggestion, rather than directly accepting or rejecting the entire suggestion. In addition, the authors feed feedback information for each dimension back to the LLM, enabling it to adjust the next round of suggestions according to the execution of the previous round's suggestions. In terms of experiments, the authors mainly validate the method on 8 maps of SMAC and further test it in some scenarios of HAPPO and SMACv2. The experiments compare CLoGA with the original reinforcement learning policy, fixed gating, no gating, shuffled suggestions, and removal of the feedback mechanism, and analyze the role of different designs through ablation experiments. The experimental results show that CLoGA achieves an average win rate of 0.970 on the selected SMAC maps, higher than the original policy's 0.888; it also achieves certain improvements in HAPPO and SMACv2 scenarios. The experiments also find that directly injecting LLM suggestions without effective gating may lead to performance degradation on some maps, indicating that selective adoption of suggestions has certain value.

Critical strengths

The authors propose a fine-grained, learnable gating mechanism that enables downstream agents to selectively incorporate the LLM advisor's suggestions across different semantic dimensions, which is quite human-like. Experiments on SMAC show a substantial improvement over the original MAPPO baseline, with the average win rate increasing from 0.888 to 0.970 across the eight evaluated maps, providing empirical evidence for the effectiveness of the proposed approach.

Critical weaknesses

The novelty is limited. The core components mentioned in the paper, such as learnable gating and LLM-RL feedback, already have related ideas in the literature. The current contribution is more like a targeted integration of existing techniques, and the core idea of assigning learnable weights to different types of inputs has long existed. In addition, the paper does not conduct sensitivity experiments on the suggestion refresh interval, which the authors explicitly list as an experiment not yet completed. This means that although the current performance gains are supported by experiments, it remains unclear how sensitive the results are to key design choices, and it is also difficult to judge whether the advantages of the method can be stably generalized to a broader range of tasks.

Questions for authors

Section 4.1 of the paper mentions the use of κ=8 as the advisory refresh interval but does not report a sensitivity analysis. Considering the trade-off among instruction freshness, feedback, and cost, could the authors clarify whether the gains are robust under different refresh intervals?

2 comments
AuthorsOpenReview ↗

Response to the Review

Thank you for the positive assessment. We now report the refresh-interval sensitivity you asked for: κ = 4 matches κ = 8 (difference ≤ 0.001), κ = 16 loses 0.007, and every setting stays above the frozen base on all three maps. On novelty, we agree that learned gating and LLM feedback each exist in the literature; our claim concerns how they are aligned, and new controls test that claim.

Q1: sensitivity to the refresh interval κ. We ran κ ∈ {4, 8, 16} with the same 3 seeds and 200k-step protocol on three maps chosen to differ in tactics: 3m (short engagements), 5m_vs_6m (mid-battle maneuver switching) and MMM2 (heterogeneous roles), as reported in Sec. 4.1 and Appendix Table S6a:

κ3m5m_vs_6mMMM2Refreshes per episode (range over maps)
40.993 ± .0070.948 ± .0190.957 ± .0155.8–19.4
8 (default)0.993 ± .0080.947 ± .0210.958 ± .0143.0–11.3
160.986 ± .0110.941 ± .0230.951 ± .0181.8–6.4

A refresh also fires whenever a directive expires, so κ bounds the delay between advice–feedback exchanges rather than a directive's lifetime. Shortening it to 4 nearly doubles the query cost without a gain, consistent with an 8-step window already giving a stable feedback signal since the advisor itself assigns durations of 8.0–10.2 steps on average (Appendix D.7). Lengthening it to 16 delays feedback and costs 0.007. The effect is small next to the gains over the base: even at κ = 16 the win rates (0.986 / 0.941 / 0.951) exceed both the frozen base (0.740 / 0.890 / 0.930) and the unfrozen-training control (0.862 / 0.914 / 0.938, Table 2b). We therefore read κ = 8 as the cheapest setting on a plateau and do not claim it is optimal. Sec. 4.1 and Appendix D.8 are revised accordingly.

W1: novelty. Learned gates and LLM feedback each exist in prior work, and Sec. 2 now says so, including residual policy learning. The contribution we claim is the alignment between the gate partition and the advisor's vocabulary: it makes the executor's per-dimension adoption both a filter on the advice and a readable signal returned to the advisor. A generic gate only filters. Here the same partition also shows the advisor which part of its plan the executor followed, so the loop can correct one field instead of the whole directive. Three results test this claim rather than assume it.

  1. A scalar gate with identical adapters, gate networks and inputs gains +0.007 from feedback, against +0.024 for per-head gates (contrast +0.018, positive in 3/3 seeds and in all 12 map × seed cells, Table 3 and Appendix Table S6b). The two differ only in whether the gate is shared across heads.
  2. Single-head interventions (Sec. 5.7, Table 5 and Appendix Table S8) show that closing the head named for a behavior changes that behavior: closing the targeting head on 3m lowers kill concentration from 0.81 to 0.34, and closing the maneuver head on 5m_vs_6m moves engagement distance from 6.1 to 7.9. Forcing the formation gate on 3m from 0.22 to 1.0 also lowers win rate by 0.055 (Table S8), showing that a low gate withholds a harmful part of the directive.
  3. Gated residuals at the readout recover the base policy exactly when the gates close or the advisor is absent (Proposition 1). Alternatives without this structure are less safe: ungated injection falls below the base on three of four maps (Table 3), and unfrozen continued training lowers a converged policy (8m_vs_9m: 0.980 → 0.961, Table 2b).

Together, these results show that aligning per-head gates with the advisor's vocabulary is what lets the executor filter imperfect advice per dimension and return a useful feedback signal while keeping the base policy intact.

Dear Reviewer xrhX,

We would like to kindly follow up on our previous responses. We would appreciate it if you could let us know whether any further questions, comments, or clarifications are required.

Thank you.

Best regards,

Authors

Reviewer_cfeuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CLoGA, an LLM-guided decision-making framework for cooperative multi-agent reinforcement learning. The method enables RL agents to selectively adopt LLM-generated tactical advice through semantic gating and uses execution feedback to refine subsequent advice. Experiments on SMAC and SMACv2 demonstrate performance improvements over pretrained policies.

Critical strengths

S1. The paper introduces an intuitive and interpretable advisory framework that combines semantic-level gating with closed-loop feedback. S2. The paper evaluates the method across different MARL algorithms and task settings, with component ablations and mechanism analyses demonstrating improvements over pretrained policies.

Critical weaknesses

W1. Insufficient comparisons with related methods. The paper mainly compares CLoGA with internal baselines and ablations. Although several related LLM-guided MARL methods are discussed, the paper does not provide direct comparisons with them. This limits the evidence for CLoGA's advantages over existing approaches. W2. Unclear necessity of the LLM advisor. The state summarizer, tactical vocabulary, and prompts incorporate substantial handcrafted StarCraft knowledge. Without a rule-based advisor baseline using the same information, it is difficult to isolate the contribution of the LLM. W3. Insufficient validation of semantic-level adoption. Semantic gating operates on hidden-state residuals, but gate coefficients do not necessarily reflect actual tactical behavior. The paper lacks head-specific intervention experiments to establish this correspondence. W4. Limited generalizability. Experiments are restricted to StarCraft environments and a single LLM backbone. The reliance on domain-specific tactical vocabularies and state summaries further limits evidence of general applicability.

Questions for authors

Q1. Can the authors compare CLoGA with closely related LLM-guided MARL methods, or explain why such comparisons are infeasible? Q2. How would CLoGA perform with a rule-based advisor using state information and tactical vocabulary instead of an LLM? Q3. Can the authors provide head-specific intervention experiments to verify that semantic gating produces the intended tactical behaviors?

4 comments
AuthorsOpenReview ↗

Response to the Review

Thank you for the clear review. Your concerns reduce to two questions: is the semantic-level claim substantiated, and is the LLM needed? We now provide head-specific interventions (W3), a rule-advisor control (W2) and a second LLM family (W4). For W1, the paper already compares against prior adoption paradigms, as reimplementations under a frozen-executor protocol, and we have made that scope explicit.

W1/Q1: comparison with prior LLM-guided MARL methods. Table 2(a) is not limited to internal ablations: InstructPPO, Exact concat and the fixed and ungated injection arms reimplement the adoption mechanisms of prior work under matched checkpoints, summaries and the same 200k-step budget (Sec. 5.2, Appendix C, Table S5). Each isolates a different design choice. InstructPPO follows the KL-to-prior regularization of Hu & Sadigh (2023), where a fixed weight decides how much advice the policy absorbs. Exact concat is the closest frozen-executor analogue of observation augmentation (Jain & Grossmann, 2025), injecting the directive without gating. Fixed or ungated injection (Table 3) is the closest analogue of scheduled override (Siedler & Gemp, 2025). CLoGA reaches 0.970 on the controlled suite against 0.920, 0.926, and 0.928 / 0.890 for these arms. InstructPPO updates the whole actor and still falls below the base on 3s5z_vs_3s6z (0.874 vs. 0.900) and MMM2 (0.921 vs. 0.930), consistent with a uniform weight forcing the policy to absorb mismatched parts of a directive together with useful ones. Because the original systems differ in training setup, environment and LLM interface, reimplementing their adoption mechanisms under a single frozen-executor protocol lets us compare the interfaces on equal terms (Sec. 5.2, Appendix C).

W2/Q2: rule-based advisor. Your concern is that handcrafted StarCraft knowledge, not the LLM, may carry the result. We built an advisor from the summarizer's own tactical-hint and target lines, with the same JSON vocabulary, adapter, gates and budget (controlled-suite means; base 0.888; Sec. 5.6, Appendix Table S7):

AdvisorUngatedGated, open loopGated, closed loop
Rule0.8860.9240.948
LLM (Qwen3.5-9B)0.8900.9460.970

With our interface the rule advisor gains +0.060 over the base, so the interface does not require an LLM, while the LLM adds a further +0.022. Without our interface, those same handcrafted rules fail to improve on the base (0.886 vs. 0.888 in Table S7). It is the per-head gating (+0.038) and closed-loop feedback (+0.024) that make the rule advisor useful, while the LLM adds another +0.022 on top (+0.035 on 3m and 5m_vs_6m). In short, the interface itself produces most of the gain, and the LLM adds a smaller, separate gain when directives need to change during an episode.

W3/Q3: head-specific interventions. Gate coefficients measure openness, not behavioral effect, so we test the link directly. At evaluation time we clamp one gate and keep the others at their learned values, then measure the behavior the head is named for: kill concentration (share of team attacks on the lowest-HP enemy) for targeting, and engagement distance (allied-to-enemy center distance) for maneuver (new Sec. 5.7, Table 5 and Appendix Table S8).

InterventionWin rateBehavior metric
3m, none0.993kill concentration 0.81
3m, targeting → 00.8510.34 (frozen base: 0.33)
3m, formation / maneuver → 00.988 / 0.9720.79 / 0.77
5m_vs_6m, none0.947engagement distance 6.1
5m_vs_6m, maneuver → 00.8987.9 (frozen base: 7.6)
5m_vs_6m, targeting → 00.9216.3

Closing the head named for a behavior changes that behavior, and closing the other heads does not: targeting governs focus fire on 3m, and maneuver governs engagement distance on 5m_vs_6m. In a dose–response sweep on 3m (Appendix Table S8), win rate is flat for g_targeting ≥ 0.5, whereas forcing g_formation to 1.0 costs 0.055 (0.993 → 0.938). A low gate is therefore not passive indifference: the learned formation gate of 0.22 withholds a directive component that would break focus fire. Together, the single-head removals in Table 5 and the dose–response sweep in Table S8 show that each gate controls the behavior its head is named for and filters out harmful advice components.

W4: generality. We separate two layers. The adapter and feedback protocol assume only that the directive splits into K semantic slices, and they are unchanged across HAPPO executors (+0.09 on 5m_vs_6m, +0.05 on MMM2, Table 4), SMACv2 (+0.04 to +0.07, Table 4), rule and LLM advisors (Table S7) and a second LLM family (Llama-3.1-8B-Instruct, complete method: 0.966 vs. 0.970 for Qwen3.5-9B, Sec. 5.5, Table S6c). The summarizer and vocabulary are the domain-specific interface and would be redesigned for a new domain (Sec. 3, Appendix G), while the gated adapter and feedback protocol stay the same.

Dear Reviewer cfeu,

We would like to kindly follow up on our previous responses. We would appreciate it if you could let us know whether any further questions, comments, or clarifications are required.

Thank you.

Best regards,

Authors

Thank you for the additional experiments, which address most of my previous concerns and strengthen the empirical evidence. However, the methodological novelty remains limited, and the applicability beyond StarCraft environments is still unvalidated. These additions do not substantially change my assessment of the paper's overall research contribution. Therefore, I maintain my original rating.

AuthorsOpenReview ↗

Follow-up Response to Reviewer cfeu

Thank you for your reply and for confirming that our previous experiments addressed your concerns on baselines (W1), the rule-based advisor (W2), and head-specific interventions (W3).

To address your remaining concerns on applicability beyond StarCraft (W4) and methodological novelty, we added two non-StarCraft benchmarks—MPE Cooperative Navigation and Google Research Football—and clarify below why the cross-domain evidence establishes a general interface principle rather than a StarCraft-specific design.

W4: Validation beyond StarCraft (MPE and Google Research Football). Following Section 4.2, we kept the 4-head GatedReadout adapter (H=64, K=4, H/K=16), the 21-dimensional encoding layout (Table S2), the feedback equations (Eq. 3–5), and all hyperparameters (Table S4) unchanged. We replaced only the text summarizer and the three tactical field names. On MPE simple_spread (3 agents, 3 landmarks), Head 1 is spacing (disperse / compact / maintain), Head 2 is motion (advance / damped_approach / yield / hold), and Head 3 is landmark assignment (conflict-free target priority). On GRF academy_3_vs_1_with_keeper (3 attackers vs. 1 defender and keeper), Head 1 is spacing (spread_wide / compact_box / maintain), Head 2 is progression (dribble_advance / pass_open / shoot / hold), and Head 3 is target player (open_winger / far_post / ball_carrier). Full results and keyframes are in Appendix D.8 (Table S6c) and Appendix F (Figures S12–S13), summarized in Section 5.5:

ArmMPE simple_spread (Coverage)GRF 3_vs_1_with_keeper (Scoring)
Frozen MAPPO (base)0.7620.643
CLoGA (per-head, full FB)0.882 (+0.120)0.778 (+0.135)
Learned tactical gates (g1 / g2 / g3)0.19 / 0.48 / 0.580.13 / 0.52 / 0.47

Without modifying the adapter or hyperparameters, CLoGA improves MPE coverage by +0.120 (0.762 to 0.882) and GRF scoring by +0.135 (0.643 to 0.778). The learned gates reproduce the same selective filtering seen on StarCraft. On MPE (Figure S12), frozen MAPPO sends two agents to L3 and leaves L1 uncovered (2/3 coverage). Under CLoGA, agents open assignment (g3 = 0.58) and motion (g2 = 0.48) to cover all three landmarks while keeping dispersion low (g1 = 0.19) once on target. On GRF (Figure S13), frozen MAPPO runs centrally into the defender and gets blocked. Under CLoGA, agents open progression (g2 = 0.52, pass_open then shoot) and target selection (g3 = 0.47, open_winger then far_post) to score into the open corner while keeping spacing low (g1 = 0.13) so compact_box advice does not pull wingers into traffic. Thus, the adapter and feedback protocol transfer beyond StarCraft with only a vocabulary swap.

Methodological contribution. We respectfully disagree that CLoGA is merely a combination of standard gating and an LLM prompt. Prior LLM-MARL interfaces treat a directive as a single block (uniform KL regularization in InstructPPO, observation concatenation, or a scalar gate), forcing the policy to accept or reject the whole instruction at once. Our results across SMAC, MPE, and GRF identify a structural failure mode of that paradigm and resolve it through three coupled findings:

  1. Non-additive coupling between semantic gating and feedback. If gating and feedback were independent add-ons, feedback would help a scalar gate as much as a per-head gate. Instead, our 2x2 ablation (Table 3, Table S6b) shows that closed-loop feedback adds only +0.007 under an identical learned scalar gate, versus +0.024 under per-head semantic gates (interaction contrast +0.018, positive in all 12 map-seed cells). A scalar gate cannot tell the advisor which field failed, whereas aligning each readout residual block with a vocabulary field (Eq. 1–5) turns the gate vector into credit-assigned feedback.

  2. Interface bottleneck over advisor capacity. Our advisor-matched control (Sec. 5.6, Appendix Table S7) shows that without semantic gating, neither handcrafted rules (0.886) nor a 9B LLM (0.890) improves on the frozen base (0.888). Adding our interface raises the rule advisor by +0.060 and the LLM by +0.082. The primary bottleneck in advisory MARL is therefore executor-side semantic filtering and feedback, not prompt engineering or model scale alone.

  3. Bounded readout intervention without corrupting converged policies. Unlike full-actor updates (InstructPPO drops below base on 3s5z_vs_3s6z and MMM2; +200k unfrozen training drops on 8m_vs_9m), keeping bounded residuals at the readout preserves the base policy (Propositions 1–2), gives verifiable head-to-behavior control (Sec. 5.7, Table 5, Table S8), and transfers unchanged to HAPPO, SMACv2, MPE, and GRF.

With all four weaknesses (W1–W4) now addressed with empirical controls across three domains, we hope these additions resolve your remaining reservations and support a higher score.

Reviewer_J4NKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how frozen recurrent MARL executors should use tactical advice from a frozen LLM, framing the interface along three axes: who decides adoption, at what semantic granularity, and whether the advisor sees outcomes. The proposed CLoGA encodes a JSON directive into four semantic heads, each adding a bounded residual to a disjoint block of the actor's readout state through a learned state-conditioned gate. Every eight steps, per-head gate values, a reward signal and state drift are returned to the LLM (Qwen3.5-9B) as text for in-context revision. Experiments use frozen MAPPO and HAPPO checkpoints on SMAC and SMACv2 with 3 seeds. The paper reports +0.25 win rate on the weakest base (3m), +0.09 on the imperfect-tier average, and a 2x2 comparison in which feedback helps per-head gating (+0.024) more than a scalar gate (+0.006).

Critical strengths
  1. The interface is framed cleanly. Splitting the advisory channel into who decides adoption, at what granularity, and with what feedback is a useful way to organize prior work, and the zero-initialized, masked design guarantees a fallback to the base policy.

  2. The controls within the design space are careful. The scalar-gate control keeps all adapters and gate networks and only shares the coefficient, so granularity is not confounded with parameter count. The finding that ungated injection falls below the base on three of four maps is useful for anyone adding LLM advice to a trained policy.

  3. The scoping is candid. The authors state that the system is not strict CTDE, that gate openness is not behavioral influence, and that the refresh interval was not swept, and they provide advisor-output statistics and full transcripts.

Critical weaknesses
  1. The frozen bases are weak (MAPPO reaches only 0.74 on 3m, where near 100% is typical), and this map supplies the headline gain. There is no baseline that simply continues training or fine-tunes the base policy for the same 200k steps.

  2. The contribution of the LLM is not separated from the extra global information it receives. The state summary already contains hand-written tactical hints and the advice is highly stereotyped per map, yet there is no rule-based advisor or direct-feature baseline.

  3. The interaction effect is a 0.018 difference over 3 seeds with 32-episode evaluations and no confidence interval, which is within noise, yet it is stated prominently in the abstract.

  4. Related work on residual policy learning (Silver et al., 2018; Johannink et al., 2019) is missing.

Questions for authors

1.. How many steps were the frozen MAPPO checkpoints trained, and why is 3m at 0.74? What win rate does continued training of the unfrozen base actor reach in the same 200k steps?

  1. Could you add a non-LLM heuristic advisor that maps the summarizer's own tactical hint and target line to the same JSON vocabulary, trained with the identical adapter? This would show whether the LLM adds anything over the centralized summary.

  2. For the 2x2 comparison, what is the seed-level confidence interval of the interaction contrast (PH_full minus PH_no) minus (SC_full minus SC_no)? Does it exclude zero?

  3. What is the wall-clock fraction spent in uncached LLM calls at evaluation time, and how does performance change if the advisor is removed at test time after training?

  4. A kappa sweep over {4, 8, 16}, and one non-SMAC domain, would help establish that the interface is not tied to StarCraft vocabulary.

3 comments
AuthorsOpenReview ↗

Response to the Review[1/2]

Thank you for a review that examines the controls closely. We read your central concern as whether CLoGA's gains reflect the interface itself or confounds: a weak base, extra global information, and training budget. Two new controls address it: continuing to train the unfrozen base for the same 200k steps reaches 0.924 on the controlled suite against 0.970 for CLoGA, and a rule-based advisor on the identical summary already recovers 0.060 of CLoGA's 0.082 gain.

W1/Q1: base competence and a training-matched baseline. The frozen MAPPO checkpoints were trained for 2M steps on 2s_vs_1sc (1.000), 5M steps on 2c_vs_64zg (0.990), 5m_vs_6m (0.890) and 8m_vs_9m (0.980), 10M steps on 3s5z_vs_3s6z (0.900), MMM2 (0.930) and corridor (0.980) as well as all HAPPO and SMACv2 bases, and 1M steps for the intermediate snapshot on 3m (0.740) (Sec. 5.1, Appendix A.7). This competence span (0.74–1.00, Figure S5) is part of the design: it tests whether an advisory channel lifts suboptimal policies while leaving near-optimal ones intact. On hard and super-hard maps, full 5M–10M runs plateau below 1.00 (giving the 0.89–0.93 imperfect bases and 0.98–0.99 saturated bases). On easy maps where full runs reach 1.00 (as in 2s_vs_1sc), we took the 1M-step snapshot on 3m (0.740) so the imperfect tier also includes a homogeneous focus-fire setting alongside the heterogeneous and asymmetric ones. Every arm loads the exact same checkpoint file on each map and trains for 200k steps.

Because 3m starts at 0.740, its +0.25 gain reflects larger headroom. On the three 5M–10M imperfect bases (5m_vs_6m, 3s5z_vs_3s6z, MMM2) the gains are +0.03 to +0.06 (mean 0.907 → 0.946), alongside +0.05 to +0.09 on HAPPO and +0.04 to +0.07 on SMACv2 (Table 4). Even across the three 10M-step controlled maps alone (3s5z_vs_3s6z, MMM2, corridor; base mean 0.937), CLoGA reaches 0.962 whereas InstructPPO (0.923) and Exact concat (0.921) fall below the base.

You asked what continued training of the base would reach. We unfroze actor and critic and continued standard MAPPO for the same 200k steps (3 seeds, same protocol, Table 2b and Table S6c):

Map (base steps)Frozen+200k unfrozenCLoGA
3m (1M)0.7400.8620.993
5m_vs_6m (5M)0.8900.9140.947
3s5z_vs_3s6z (10M)0.9000.9210.933
MMM2 (10M)0.9300.9380.958
corridor (10M)0.9800.9760.996
8m_vs_9m (5M)0.9800.9610.985
Controlled-suite mean (3m, 3s5z, MMM2, corridor)0.8880.9240.970
Imperfect-tier mean0.8650.9090.958

CLoGA is higher on all six maps (+0.131 on 3m, +0.012 to +0.033 elsewhere), and unfrozen training lowers the converged 8m_vs_9m policy (0.980 → 0.961) while CLoGA leaves it intact (0.985). Continued training updates every weight, which helps an under-trained policy but can move a converged one away from a good solution, as on 8m_vs_9m. CLoGA keeps the backbone frozen and bounds the residual, returning the base exactly when gates close. While a longer or tuned schedule would narrow the gap on weak bases, under the same 200k budget CLoGA reaches higher win rates on all six maps and leaves saturated policies intact (Sec. 5.2, Table 2b).

W2/Q2: separating the LLM from the centralized summary. Your concern is that the summary already carries hand-written tactical hints, so the LLM may add little beyond them. We built the advisor you describe: a deterministic rule set that maps the summarizer's tactical-hint and target lines to the same 7-field JSON vocabulary, with the same adapter, gates and 200k budget. In closed loop it applies the feedback logic of our prompt (switch the ignored head, halve the duration under high drift). Controlled-suite means (base 0.888):

AdvisorUngatedGated, open loopGated, closed loop
Rule0.8860.9240.948
LLM (Qwen3.5-9B)0.8900.9460.970

Most of the gain belongs to the interface: with gating and feedback the rule advisor already gains +0.060 over the base, and the LLM adds a further +0.022 in both open and closed loop. Without gating neither advisor improves on the base, so the failure sits in the injection pathway, not the advice source. Gating helps both advisors (+0.038 rule, +0.056 LLM), and feedback adds +0.024 to each. The LLM margin is smallest on corridor (+0.008), where one directive suffices throughout, and largest on 3m and 5m_vs_6m (+0.035), where directives change within an episode. Even against rules taken straight from the summarizer's hints, the gating and feedback interface accounts for +0.060 of the gain, and the LLM adds a consistent +0.022 when directives need to be revised during a fight (Sec. 5.6, Appendix D.9).

AuthorsOpenReview ↗

Response to the Review[2/2]

W3/Q3: the interaction contrast. We agree the original statement was stronger than three seeds support and have softened it: the abstract and Sec. 5.3 now present the result as a directional observation. We also report uncertainty in Appendix D.8 and Table S6b. With Δ_int = (PH_full − PH_no) − (SC_full − SC_no), the feedback gains are +0.0245 for per-head and +0.0065 for scalar gating, so Δ_int = +0.018, computed from the per-map cell values of Table 3 (the rounded suite means shown there read as 0.017). Per seed, Δ_int is +0.016, +0.019 and +0.019 (95% t-interval [0.013, 0.023], excluding zero), and it is positive in each of the 12 map × seed cells. The per-map means are +0.021 on 3m, +0.030 on 3s5z_vs_3s6z, +0.016 on MMM2 and +0.005 on corridor. The cells share seeds and are not independent, so we attach no p-value to the 12/12 count, and with three seeds and cells that do not hold the advice stream fixed we keep these caveats in the paper.

Why we expect an interaction: with a shared gate the feedback text can report only one adoption number for three heads, whereas per-head gates let it say that formation was adopted and maneuver ignored, which tells the advisor which field to revise. Together with the positive contrast in all 12 map × seed cells (Table S6b), this shows that returning per-head adoption makes the feedback loop more effective than a single shared gate.

Q4: wall-clock cost and test-time advisor removal. Uncached LLM calls account for under 5% of evaluation wall-clock time on a co-located A100 (Appendix A.8, Table S6a). Removing the advisor at test time returns the base policy exactly by construction (Proposition 1): on 3m the win rate is 0.740 and the logits match the checkpoint to floating-point precision. Because this outcome is guaranteed, it mainly checks the implementation rather than whether the adapter depends on the advice content. The informative controls are State-only (adapter without directive, Table 2a), which reaches 0.886 against a 0.888 base, and shuffled advice (Table 3), which reaches 0.867. Neither extra capacity nor unrelated advice produces the gain.

Q5: refresh interval and a second domain.

κ3m5m_vs_6mMMM2Refreshes per episode (range over maps)
40.9930.9480.9575.8–19.4
8 (default)0.9930.9470.9583.0–11.3
160.9860.9410.9511.8–6.4

κ = 4 matches κ = 8 (difference ≤ 0.001) at nearly twice the query rate, and κ = 16 loses 0.007 (Sec. 4.1, Appendix D.8, Table S6a). A refresh also fires whenever a directive expires, so κ caps the delay between advice–feedback exchanges. Even at κ = 16, every setting stays well above both the frozen base and the unfrozen-training control on all three maps, so we treat κ = 8 simply as a cheap setting on a flat plateau.

We have not tested a non-StarCraft domain. The summarizer and vocabulary are domain-designed by construction (Sec. 3, Appendix G), and the adapter assumes only some semantic partition of the directive, so a new domain requires designing the interface rather than changing the adapter. Within StarCraft, the same adapter and protocol work without change across training algorithms (HAPPO: +0.09 on 5m_vs_6m, Table 4), randomized compositions (SMACv2: +0.04 to +0.07, Table 4), rule and LLM advisors (Table S7), and a second LLM family (Llama-3.1-8B-Instruct: 0.966 vs. 0.970 for Qwen3.5-9B, Sec. 5.5, Table S6c).

W4: residual policy learning. Thank you. Sec. 2 now cites Silver et al. (2018) and Johannink et al. (2019) as the closest classical precedent. RPL adds a state-conditioned correction to a fixed controller. CLoGA also corrects a frozen policy, but conditions the residual on an external language directive, applies it at the readout state, and scales it per semantic head with gates that the executor learns and the advisor observes. With the gates closed, the base policy is recovered exactly.

Dear Reviewer J4NK,

We would like to kindly follow up on our previous responses. We would appreciate it if you could let us know whether any further questions, comments, or clarifications are required.

Thank you.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.