- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper leverages the AlphaEvolve agent to discover new MARL algorithms, focusing on the CFR and PSRO algorithms. For CFR, the authors consider how the instantaneous counterfactual regrets are processed, how to derive the current iteration’s strategy from the accumulated regrets, and how to integrate the current iteration’s policy into the final average strategy. For PSRO, the authors consider how to derive the meta strategies during the oracle training and evaluation phases. The authors also distill the discovered algorithms to two minimal closed-form solvers, WOP-CFR and PM-PSRO. Experiments demonstrate the superior performance in a set of different games.
Critical strengths
-
The idea of automatically discovering new algorithms by using LLMs/agents in the MARL field is interesting and relevant to the community.
-
The discovered algorithms deliver good performance in most of the training/testing games compared to baselines.
Critical weaknesses
-
The main concern is that this paper is a simple application of AlphaEvolve to the two distinct game-theoretic algorithms CFR and PSRO, leaving the technical novelty unconvincing. From my perspective, adapting the original CFR and PSRO to AlphaEvolve, i.e., constructing the search spaces for them, is not so novel but just a prerequisite for applying AlphaEvolve. Even with that, the designed spaces are also simple, since these two algorithms still have many other elements that can impact their performance, e.g., the hyperparameters and the training of the best response oracle (though this paper only considers the exact responses, which also further narrow down and simplify the search spaces).
-
Instead, I feel that distillation is also an important part of the method that should be presented in the technical part. The authors put it in the experiments, making it hard to grasp the details.
-
As mentioned in the first point, this paper uses exact oracles in best response computation and exact payoff values in meta-game construction, which further simplifies the search spaces. As far as I know, even exact oracles or value payoffs could still be hard to compute for some large games such as 3-player leduc.
Questions for authors
See weaknesses.