acceptodds
Withdrawn from ICLR 2027

Evidence-Gated Adoption of LLM-Generated Graph Priors for Multi-Agent Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-guided multi-agent reinforcement learning (MARL) increasingly uses coordination graphs as structural priors, yet lacks an evidence boundary between graph proposal and policy optimization. Across 36 conditions, 15 LLM graphs underperform a graph-free reference, while proposer confidence is weakly informative. We treat each proposed graph as a hypothesis whose adoption requires stage-specific evidence. FSGAC makes a pre-policy graph-admission decision from relative probe improvement, typed complexity, and propagation sensitivity; PCEVC audits retained edges using paired deletion and clipped DR-style estimation; ITSC regularizes cross-domain variance in audited type effects. FSGAC correctly classifies 33/36 conditions under grouped leave-one-scenario-out evaluation. PCEVC reports 3/176 harmful decisive accepted edges in a post-hoc MPE/SMACv2 subtotal. The full pipeline has 8.2% maximum reported benchmark-shift degradation. Across MPE, SMACv2, and MDO-Sim, the pipeline ranks first against 14 baselines, and the graph gate transfers to a matched Llama proposer. These results support an evidence-based adoption protocol for graph-prior MARL, with separate graph, edge, and transfer checks.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Learning, Large Language Model, LLM-guided multi-agent reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_zBksOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether LLM-generated coordination graphs should be adopted in multi-agent reinforcement learning (MARL). The authors propose a three-stage framework: FSGAC evaluates graph proposals before policy optimization using empirical diagnostics; PCEVC audits individual graph edges through paired deletion and doubly robust-style estimation; and ITSC introduces variance-based regularization to improve the stability of semantic coordination types under distribution shifts.

Experiments on MPE, SMACv2, and MDO-Sim report improved graph-admission decisions, edge-level risk estimates, and downstream MARL performance compared with several baselines.

Critical strengths
  1. The paper raises a meaningful question: LLM-generated coordination structures are not necessarily beneficial for MARL, and graph proposals should not automatically be incorporated into policy optimization. Distinguishing graph proposal from evidence-based graph adoption is a potentially useful research direction.

  2. The experiments consider graph-level admission, edge-level auditing, and robustness under distribution shifts. This is a reasonable evaluation perspective because improved average return alone does not necessarily establish the reliability of an externally proposed coordination structure.

Critical weaknesses

1. The organization and presentation are substantially below the standard expected at ICLR, making the methodological contribution difficult to evaluate.

The manuscript introduces numerous specialized abbreviations, including FSGAC, PCEVC, ITSC, LCB, DR, and several related quantities, without sufficiently explaining their roles and relationships at a conceptual level. The Abstract is particularly difficult to follow because it concentrates heavily on technical terminology and experimental numbers rather than clearly communicating the underlying methodological idea.

The Introduction and Method sections also lack a coherent progression from the identified problem to the proposed solution. Section 3 contains several definitions, lemmas, diagnostic proxies, and optimization objectives, but the motivation for introducing each mathematical component and its relationship to the overall framework are not sufficiently explained.

For example, Lemma 3.2 introduces an algebraic residual decomposition, Definition 3.3 introduces a propagation-sensitivity proxy, and Definition 3.4 combines these quantities into a graph-admission score. However, the manuscript does not convincingly explain why this particular combination should identify graph structures that improve downstream MARL performance.

Similarly, the transition from graph-level admission (FSGAC) to edge-level auditing (PCEVC) and typed regularization (ITSC) is presented primarily as a sequence of mathematical operations rather than a well-justified algorithmic framework.

Although Figure 1 provides a high-level illustration, it does not sufficiently explain the interaction between these components or how the complete framework is trained and executed.

Overall, the presentation makes it difficult to determine which parts represent genuine methodological innovations and which are adaptations of established statistical techniques. This substantially weakens the paper's accessibility, reproducibility, and overall contribution.

2. The theoretical justification and experimental evidence do not sufficiently establish the soundness and necessity of the proposed framework.

The theoretical analysis contains several components whose mathematical validity is narrower than the claims the framework seeks to support.

First, the FSGAC graph-admission score in Eq. (6) combines empirical residual improvement, a complexity-based penalty, and a propagation-sensitivity proxy. However, these quantities are largely heuristic, and the manuscript explicitly acknowledges that the resulting diagnostic is neither an unbiased estimator nor a lower bound on downstream return improvement. Consequently, it remains unclear why the sign of this score should reliably determine whether an LLM-generated graph benefits subsequent policy optimization. The theoretical motivation does not establish this connection.

Second, PCEVC estimates policy-conditioned one-step edge effects under a fixed policy snapshot. Although the authors acknowledge this limitation, the proposed framework uses these estimates to determine which edges should be retained for subsequent learning. A positive one-step effect does not necessarily imply that retaining an edge improves long-horizon coordination or the performance of the subsequently optimized policy. This distinction is particularly important in MARL, where agent interactions and coordination requirements evolve during training.

Third, ITSC regularizes cross-domain variance using typed contributions derived from previously audited edges. However, the audit masks and reliability weights remain fixed while the underlying policy and type representations continue changing. The manuscript does not adequately justify why the original edge-level evidence remains reliable throughout this optimization process.

The empirical results are encouraging but do not fully resolve these issues. In particular, the graph-admission experiment includes 36 conditions derived from only 12 underlying scenarios, limiting the independent evidence for generalization. Moreover, the component ablations mainly demonstrate changes in performance when individual stages are removed, rather than directly validating the theoretical assumptions or establishing the necessity of the proposed diagnostic formulations.

I would also encourage stronger comparisons with simpler alternatives, such as direct validation-based graph selection, standard edge pruning, and conventional variance regularization under comparable computational budgets. Without these comparisons, it is difficult to determine whether the proposed three-stage framework provides benefits beyond combining established techniques.

Overall, the framework contains several potentially useful ideas, but the methodological connections, theoretical justification, and experimental evidence are not sufficiently convincing to establish a sound and distinctive contribution at ICLR level.

Questions for authors
  1. What theoretical or empirical evidence explains why the FSGAC diagnostic in Eq. (6), computed using frozen probes and a reference policy, should predict whether a graph improves downstream MARL optimization? Why is the proposed combination of residual improvement, complexity, and propagation sensitivity preferable to simpler validation-based selection?

  2. PCEVC estimates one-step effects under a fixed policy. What evidence supports using these estimates to select edges for subsequent long-horizon policy optimization? Do the authors have results demonstrating that the audited edge rankings remain useful after the policy changes?

  3. Since ITSC updates the policy and type assignments while retaining fixed audit masks and weights, how is the validity of the original edge-level evidence maintained? Could the authors clarify whether stale audit information negatively affects optimization?

  4. Can the authors clarify how the benefits of the proposed diagnostic and regularization mechanisms are distinguished from simpler graph-selection, edge-pruning, or variance-regularization alternatives? This distinction would materially affect my assessment of the algorithmic contribution.

Reviewer_ivj2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an evidence-gated adoption protocol for LLM-generated coordination graphs in multi-agent reinforcement learning (MARL). Rather than assuming that an LLM-proposed graph should be used because it appears plausible or is generated with high confidence, the paper treats graph adoption as a hypothesis-testing problem. The framework contains three stages: (1) FSGAC; (2) PCEVC; and (3) ITSC. Experiments are conducted over 36 conditions from 12 scenarios across MPE, SMACv2, and MDO-Sim, against 14 baselines.

Critical strengths
  1. The paper’s central premise is valuable: an LLM-generated coordination graph should not be adopted merely because it is plausible-looking or associated with high proposer confidence.

  2. The paper distinguishes graph-level admission, edge-level diagnosis, and end-to-end transfer, rather than conflating these distinct objectives.

  3. The paper explicitly acknowledges important limitations, including that edge-level evidence is restricted to one-step policy-mediated effects and does not establish long-horizon or post-retraining benefit.

Critical weaknesses
  1. The central graph-admission result is based on only 36 conditions from 12 underlying scenarios, with three prompt variants per scenario. Thus, the apparently strong result of 33/36 correct decisions may be statistically fragile and may not reflect robust generalization to genuinely new tasks, environments, graph types, or LLM proposers. In particular, prompt variants are not equivalent to independently sampled task scenarios.

  2. The paper uses ten evaluation seeds and bootstrap procedures for certain admission decisions, and the main end-to-end return comparisons and regularizer ablations are presented largely as point estimates or descriptive gaps.

  3. PCEVC is presented as an edge-level auditing mechanism, but its validation is limited to policy-conditioned one-step effects.

  4. The harmful-edge count is reported only for the post-hoc MPE/SMACv2 subtotal; MDO-Sim contributes correlations but no corresponding harmful-edge risk count.

  5. The presentation is occasionally overly dense. The paper introduces many acronyms and technical quantities in a short space, which makes the method difficult to parse on a first reading. A simple end-to-end running example—from an LLM-proposed graph, through FSGAC admission, PCEVC edge decisions, and ITSC training—would significantly improve accessibility.

Questions for authors

see weakness

If the authors can address my concerns well, I will be willing to increase the score.

Reviewer_2kLbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Paper Summary

  • Overview: This paper proposes a three-stage pipeline for screening, pruning, and regularizing LLM-generated coordination graphs in Multi-Agent Reinforcement Learning (MARL).
  • Contributions:
    1. FSGAC (Pre-training Gate): Evaluates candidate LLM graphs before policy training by fitting frozen value probes on initial rollouts and computing a heuristic score based on residual error reduction, graph complexity, and value variation.
    2. PCEVC (Mid-training Edge Pruning): At a training checkpoint, iterates through retained edges and uses clipped doubly-robust (DR) counterfactual estimation to drop edges that do not improve one-step policy values.
    3. ITSC (Training Regularizer): Applies a cross-domain variance penalty to the values of retained semantic edge types to improve out-of-distribution (OOD) transfer.
  • Validation: Evaluated across 12 scenarios (with 3 prompt variants each, framed as 36 conditions) on MPE, SMACv2, and a custom MDO-Sim environment, comparing against 14 baselines and evaluating zero-shot OOD shifts.
Critical strengths

Critical Strengths

  • Practical Problem Formulation: Addressing the unreliability of LLM-generated priors in MARL is a well-motivated problem. Treating external LLM graphs as testable hypotheses rather than ground-truth priors—and attempting to filter out harmful graphs or edges before and during training—is a sensible and practical direction.
  • Comprehensive Ablation Effort: The authors attempt to isolate the impact of each stage of their pipeline (graph gating, edge pruning, and variance regularization) across multiple environments, proposers (Qwen and Llama), and corruption levels. Additionally, the results seem to give improvement over baselines. I appreciate the full pipeline nature of the approach. I think this has really interesting ideas and results, with a bit more work it can be improved further
Critical weaknesses
  • Poor Presentation, Clarity, and Over-Formalization: The paper is difficult to read. The abstract and introduction heavily rely on undefined acronyms (FSGAC, PCEVC, ITSC, DR, etc.) that are only defined pages later, alongside inconsistent terminology. Furthermore, the methodology is buried beneath dense definitions and lemmas that restate standard algebraic identities (e.g., Lemma 3.2 is simply the law of cosines/variance decomposition) rather than novel theoretical results. Much of the text reads like a disconnected list of caveats and metrics rather than a cohesive narrative.
  • Crucial Omissions in Experimental and Methodological Details: Key mechanics required to understand and evaluate the method are missing:
    • OOD Graph Expansion: In the 2×2\times agent and novel-role OOD settings, the paper never explains how a static, pruned graph G=(V,E,T)G = (\mathcal{V}, \mathcal{E}, \mathcal{T}) is expanded to accommodate new agent nodes or how edges/embeddings cijc_{ij} for unseen agent pairs are determined without re-querying the LLM.
    • Vague OOD Specifications: The environment modifications (e.g., "jammer" replacements in SMACv2, "rewired" environment topologies) are under-specified, making the OOD results hard to interpret.
    • Missing Audit Timing & Overhead: PCEVC audits edges at a "common checkpoint tat_a", but tat_a is never specified (e.g., 10% vs. 80% of training). Total wall-clock/compute overhead for rollout collection, probe fitting, LLM refinement, and O(∣E∣)O(|\mathcal{E}|) paired rollouts is also omitted.
  • Limited Component Novelty and Lack of Qualitative/Structural Analysis: As acknowledged in Section 2, the individual components rely on standard tools (residual variance, clipped DR, V-REx-style variance penalties), and ITSC feels somewhat orthogonal to the core premise of evaluating graph priors. Because the novelty rests entirely on the empirical pipeline, the lack of qualitative analysis is a major gap: the paper never visualizes or interrogates the final learned graphs, nor does it test retraining from scratch with the final pruned graphs to isolate whether gains come from the graph topology itself, the curriculum effect of mid-training pruning, or the auxiliary losses.
Questions for authors

Clarification Requests (Material to Assessment):

  • OOD Graph Expansion: When evaluating zero-shot on 2×2\times agents (e.g., 3 to 6 agents), how is the static FSGAC-admitted and PCEVC-pruned graph topology adapted to the new agents? How are edges and frozen LLM relation embeddings (cijc_{ij}) assigned to the new agent pairs?
  • PCEVC Checkpoint (tat_a): At what exact training step tat_a (out of the 5×1065 \times 10^6 budget) is PCEVC executed, and how sensitive is the pipeline to this choice?
  • Computational Overhead: What is the exact sample and compute cost of the pre-training rollouts (π0\pi_0), probe fitting, and the 50-rollout-per-edge audit relative to standard MARL training?
  • OOD Setup Details: Can you precisely define how the "jammer" role operates in SMACv2 and what "environment graph structure rewired" means for environments that do not natively use fixed communication graphs?

Suggestions for Future Work / Revisions:

  • Retraining from Scratch: Train a policy from scratch using the final PCEVC-pruned graph (without mid-training changes or ITSC) to isolate the intrinsic value of the discovered topology from training-time regularization dynamics.
  • Graph Interrogation: Include visualizations and qualitative analyses of the accepted vs. rejected graphs and pruned edges to demonstrate what semantic structures the pipeline actually discovers and why they are useful.
  • Headline Framing: Frame the experimental scope around the 12 base scenarios rather than "36 conditions" in the abstract, as the 36 conditions are simply 3 prompt variations per scenario.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.