acceptodds
Under review as a conference paper at ICLR 2027

Discovering Multiagent Learning Algorithms with Large Language Models

11 views

Abstract

Much of the advancement in Multi-Agent Reinforcement Learning (MARL) for imperfect-information games has historically depended on the manual, iterative refinement of algorithmic baselines. Recently, evolutionary coding agents powered by Large Language Models (LLMs) have emerged as powerful tools to automate this discovery process. In this work, we deploy one of such agentic frameworks, AlphaEvolve, to navigate the design spaces of two distinct game-theoretic paradigms: counterfactual regret minimization (CFR) and policy-space response oracles (PSRO). This automated search yielded two algorithms: Volatility-Adaptive Discounted (VAD-) CFR and Smoothed Hybrid Optimistic Regret (SHOR-) PSRO, which are consistently competitive with state-of-the-art human-designed baselines across an 18-game evaluation suite spanning Poker, Goofspiel, Liar's Dice, Blotto, and Battleship variants. However, because the LLM optimizes for fitness on a specific training set, it often constructs highly synergistic, complex mechanisms tailored to those environments. Through systematic train-only ablation criteria, we distill these discoveries into two minimal closed-form solvers: Warm-started Optimistic Predictive (WOP-)CFR and Projection Matching (PM-)PSRO. Ten independent search runs confirm that the distilled cores recur while the overfitted machinery differs across runs. Distillation isolates this stable algorithmic core, achieving superior held-out performance at drastically reduced structural complexity: WOP-CFR improves on scaled variants within family and maintains parity out of family, while PM-PSRO improves in both regimes and most strongly out of family.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
multiagent reinforcement learning, meta learning, counterfactual regret minimization, policy space response oracle, LLM-based algorithm discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5DZQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper leverages the AlphaEvolve agent to discover new MARL algorithms, focusing on the CFR and PSRO algorithms. For CFR, the authors consider how the instantaneous counterfactual regrets are processed, how to derive the current iteration’s strategy from the accumulated regrets, and how to integrate the current iteration’s policy into the final average strategy. For PSRO, the authors consider how to derive the meta strategies during the oracle training and evaluation phases. The authors also distill the discovered algorithms to two minimal closed-form solvers, WOP-CFR and PM-PSRO. Experiments demonstrate the superior performance in a set of different games.

Critical strengths
  • The idea of automatically discovering new algorithms by using LLMs/agents in the MARL field is interesting and relevant to the community.

  • The discovered algorithms deliver good performance in most of the training/testing games compared to baselines.

Critical weaknesses
  • The main concern is that this paper is a simple application of AlphaEvolve to the two distinct game-theoretic algorithms CFR and PSRO, leaving the technical novelty unconvincing. From my perspective, adapting the original CFR and PSRO to AlphaEvolve, i.e., constructing the search spaces for them, is not so novel but just a prerequisite for applying AlphaEvolve. Even with that, the designed spaces are also simple, since these two algorithms still have many other elements that can impact their performance, e.g., the hyperparameters and the training of the best response oracle (though this paper only considers the exact responses, which also further narrow down and simplify the search spaces).

  • Instead, I feel that distillation is also an important part of the method that should be presented in the technical part. The authors put it in the experiments, making it hard to grasp the details.

  • As mentioned in the first point, this paper uses exact oracles in best response computation and exact payoff values in meta-game construction, which further simplifies the search spaces. As far as I know, even exact oracles or value payoffs could still be hard to compute for some large games such as 3-player leduc.

Questions for authors

See weaknesses.

Reviewer_DS6JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper applies AlphaEvolve to discover update rules for two multi-agent learning frameworks, counterfactual regret minimization (CFR) and policy-space response oracles (PSRO). The search produces VAD-CFR and SHOR-PSRO. The authors then use ablation analysis to simplify these discoveries into WOP-CFR and PM-PSRO, aiming to retain useful algorithmic mechanisms while removing specialized heuristics. Evaluation uses four training games and fourteen held-out OpenSpiel games with exact payoff and exploitability computation, supplemented by random normal-form games, additional search trajectories, and extended-horizon tests. The main contribution is the empirical algorithmic discoveries and their subsequent analysis and simplification, rather than a new evolutionary discovery framework.

Critical strengths
  1. Useful algorithmic discoveries. Applying AlphaEvolve to CFR and PSRO yields interpretable algorithm variants with competitive performance on held-out games, demonstrating the potential of LLM-driven automated research in multi-agent learning.

  2. Interesting distillation analysis. Simplifying the evolved programs produces substantially simpler solvers with improved aggregate held-out performance. This provides a useful direction for understanding and refining automated discoveries, supported by clear presentation and detailed code and prompts.

Critical weaknesses
  1. Limited methodological novelty. The discovery framework largely follows existing AlphaEvolve. Contributions mainly lie in domain adaptation and the resulting algorithms, with limited comparison against simpler search or tuning alternatives.

  2. Incomplete support for the distilled core. Individual component ablations and the final simplified solver do not systematically resolve component interactions or redundancy. The stated retention threshold also appears inconsistent with some PSRO distillation choices, weakening the claim of a stable core selected by a reproducible rule.

Questions for authors
  1. Distillation threshold. Section 4.2 retains PSRO components when their ablation gives strain<−0.05s_{\mathrm{train}} < -0.05. Table 2 reports −0.069-0.069 for no_blending_orm and −0.104-0.104 for no_annealing, yet both mechanisms are removed in PM-PSRO. How do these choices follow the stated rule, and how were the thresholds chosen?

  2. Component interactions. Were paired or sequential ablations used to examine interactions and redundancy? What evidence distinguishes a stable algorithmic core from one of several effective simplified configurations?

Reviewer_9KUPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This manuscript makes three main contributions, as outlined in the introduction:

  • First, the manuscript introduces a modular framework for CFR and PSRO algorithms, which decomposes them into independently modifiable components, enabling LLM-driven evolutionary search over algorithmic update rules and structures.
  • Second, the manuscript proposes a general propose-and-distill methodology, which identifies and retains the essential mathematical mechanisms discovered by an LLM-driven evolutionary search through systematic ablation experiments to produce simpler, generalizable algorithms.
  • And the last one, two distilled solvers, closed loop equilibrium solvers, WOP-CFR and PM-PSRO, which achieve overall lower aggregate exploitability over their original LLM-discovered counterparts on held-out games.

The manuscript evaluates baselines across 18 different games, of which four are training games, and the rest are held-out games spanning Poker, Goofspiel, Liar's Dice, Colonel Blotto, and Battleship variants. Here, the exploitation is computed via game-tree traversal. WOP-CFR achieves the lowest exploitability on 14 out of 18 games, while PM-PSRO achieves the lowest exploitability on 12 out of 18 games.

Critical strengths

1. Algorithmic simplicity and generalizability:

WOP-CFR and PM-PSRO retain only the essential mathematical mechanisms identified through the ablation studies of their original LLM-discovered counterparts. Both algorithms achieve an overall improvement in performance despite their simple and closed-form formulations. For PM-PSRO, this improves performance on unfamiliar game families, as seen in the Out-of-family IQM log-improvement (Table 2, page 26). While WOP-CFR achieves comparable aggregate performance on unfamiliar game families, as seen in the Out-of-family IQM log-improvement (Table 1, page 6).


2. Strong empirical performance against baselines:

Across 18 games, PM-PSRO achieves the lowest reported exploitability on 12 games, and WOP-CFR achieves the lowest reported exploitability on 14 games. This demonstrates competitive performance against strong and relevant baselines.


3. Systematic ablations support the proposed distillation methodology:

The manuscript investigates the individual components through systematic ablations. The distilled WOP-CFR and PM-PSRO achieve better reported results than their respective raw discoveries, VAD-CFR and SHOR-PSRO. This provides a direct experimental support for the manuscript's central methodological claim.


4. Reproducibility across independent evolutionary searches:

The manuscript investigates the recurrence of similar mathematical mechanisms across ten independently evolving search trajectories. This provides evidence that the essential components are reproducible within the adopted search framework despite the variation in the additional heuristics.

Critical weaknesses

Dependence on a single LLM backbone:

The manuscript already addresses this limitation in lines 472-475.

The manuscript relies exclusively on Gemini 2.5 Pro and uses the same prompt design throughout the experiment. Modern frontier LLMs like Gemini 2.5 Pro are robust to prompt design, but drastic variations in the prompt may still produce different results. [1] demonstrate that minor variations in the prompt phrasing and structure can lead to accuracy deviations. Similarly, different frontier models may produce different quality of the discovered algorithms or the recurrence of their essential mechanisms. [2] demonstrate that the choice of LLM can materially influence code-generation performance. Consequently, the extent to which the proposed discovery-and-distillation methodology generalizes across different LLM backbones and prompt designs remains ambiguous.


References.

[1] Mohamed Insaf Ismithdeen, Muhammad Uzair Khattak, and Salman Khan. 2025. Promptception: How Sensitive Are Large Multimodal Models to Prompts?. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 23950–23985, Suzhou, China. Association for Computational Linguistics.

[2] D. Palla and A. Slaby, "Evaluation of Generative AI Models in Python Code Generation: A Comparative Study," in IEEE Access, vol. 13, pp. 65334-65347, 2025, doi: 10.1109/ACCESS.2025.3560244

Questions for authors

I seek clarification on the generalization across LLM backbones and the sensitivity to prompt design.

  1. How would the proposed discovery-and-distillation methodology perform on alternative LLM backbones, such as frontier models like ChatGPT and Claude, or even open source models like Llama? An ablation on an alternative LLM backbone is enough for my clarification.

  2. I want to understand how the proposed methodology would perform in substantial changes to the instruction. An ablation on exploring the influence of different prompt structures on the quality of the discovered algorithms or the recurrence of their essential mathematical mechanisms is enough for my clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.