acceptodds
Under review as a conference paper at ICLR 2027

Modular Discovery of General Game-Playing Algorithms with Large Language Models

Abstract

General Game Playing across arbitrary games from rules alone remains challenging due to differing algorithmic requirements across game classes and strict decision-time constraints. Rather than hand-designing search heuristics for specific domains, can we leverage Large Language Models (LLMs) to discover general game-playing algorithms? Because language models can propose and refactor structured code, they provide an expressive proposal engine for exploring the space of algorithmic designs. We introduce a multi-agent LLM meta-learning system to co-evolve game-agnostic procedural search mechanisms in C++ alongside domain heuristics synthesized directly from game rules. Controlling the compute budget, we benchmark the discovered mechanisms across more than 400 diverse environments, including OpenSpiel training and held-out games, procedural simulation engines, and games with deep neural policy-value representations trained via PPO. Evaluated via AlphaRank stationary distributions and Soft Condorcet Optimization (SCO) against 15 established MCTS baselines, the discovered search mechanisms consistently achieve top-tier ratings and pairwise ballot majorities over most baselines across independent evolutionary runs, generalizing to unseen human-designed and procedurally synthesized games and remaining competitive with baselines on frozen neural network representations.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Monte-Carlo tree search, General game playing, LLM-based algorithmic discovery, Meta learning, AlphaEvolve

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ze1MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper applies AlphaEvolve to discover game-playing search algorithms by co-evolving shared search mechanisms and game-specific heuristics across multiple games. The search mechanisms modify components such as tree selection, value backups, and root action selection. Experiments across training, held-out, procedurally synthesized, and neural-policy/value settings show improvements over established MCTS baselines in several settings, while performance with frozen PPO networks is statistically comparable to tuned PUCT.

Critical strengths

The paper provides extensive empirical evaluation of the discovered search mechanisms. It covers held-out human-designed games, synthesized games, and frozen neural policy/value networks, with comparisons against MCTS baselines under both time and simulation budgets. Evaluating five independent evolutionary runs also helps assess the consistency of the results. The transfer across games provides useful evidence for automated search-code discovery beyond a single domain.

Critical weaknesses
  • I find the methodological novelty limited. The paper uses AlphaEvolve to evolve MCTS-style search code. Separating shared search mechanisms from game-specific knowledge is a useful design, but the contribution appears primarily to be an application of an existing code-evolution framework to general game search.

  • The additional benefit with learned policy/value networks is unclear. In the PPO evaluation, the evolved mechanisms do not significantly improve over tuned PUCT. This limits the demonstrated practical advantage in an important setting for strong game-playing systems.

Questions for authors

Could the authors clarify what they consider the main methodological contribution beyond applying AlphaEvolve to MCTS-style search? In particular, what does the proposed factorized co-evolution add beyond evolving a shared search mechanism across multiple games?

1 comment
AuthorsOpenReview ↗

Thank you for your review

We thank Reviewer Ze1M for the assessment, for noting the breadth of the empirical evaluation across 5 independent runs, and for two questions that go to the core of the paper.

W1 and Q1: what factorized co-evolution adds

We agree that the code-mutation engine is AlphaEvolve. Standard AlphaEvolve evolves a single program against a fixed evaluator; our contribution is structuring multi-game search discovery as a multi-agent system—one MechanismCoordinator agent evolving a shared mechanism archive A_mech across 50 games, and 50 per-game KnowledgeWorker agents evolving heuristic archives A_know(g) (Algorithm 2).

The coordinator and workers communicate through shared archives with decoupled credit: when a worker generates a verified heuristic κ' (Appendix B.1), it samples a mechanism m ~ A_mech, evaluates <m, κ'> against fixed reference PUCT <m_ref, κ'> and the standalone prior π_κ', and reports headroom-normalized Amp and Imp back to the coordinator's running cross-game score of m, paced by a completion quorum (Section 3.3, Appendix B.2).

On Q1 (what co-evolving knowledge adds over evolving a shared mechanism alone under fixed default heuristics):

  1. Non-trivial inputs for prior- and value-dependent operators. Under uniform default priors, terms such as sqrt(1 − π(a)) in first-play urgency are constant across actions and receive no selection signal. Co-evolved priors and value estimators give the coordinator informative inputs during evolution; four of the five runs discovered prior-dependent FPU schedules (revised Table 20; supplementary code).
  2. Preventing opponent saturation (Section 3.4, Appendix B.3). Against static opponents, reference PUCT saturates the payoff ceiling in tactical games (Hpuct ≤ 0.05 ΔU), dropping those games from Amp. Promoting the co-evolved champion (m*_e, κ*_{g,e}) to opponent re-expands headroom and restores valid selection coverage in connect_four, gomoku and hex from 0% to 85–91%.
  3. Credit separation (Section 3.3). Ranking heuristics under fixed m_ref while ranking mechanisms by marginal headroom gain (Amp, Imp) over m_ref on the same heuristic prevents a strong heuristic from masking a weak mechanism.

Generalization beyond MCTS and games. Because the coordinator–worker protocol only requires factoring an algorithm into a domain-agnostic procedural core m and task-specific heuristics κ_g, the same generalist–specialist multi-agent meta-learning architecture transfers directly to other algorithmic domains—such as combinatorial optimization and SAT/MILP solvers (co-evolving a general branching/restart policy m alongside problem-family scoring heuristics κ_g), automated theorem proving (co-evolving proof-search control m with domain tactic priors κ_g), and classical or robotic planning (co-evolving graph/tree search m with task-specific cost-to-go heuristics κ_g)—using the same invariant-reference Amp/Imp credit assignment and quorum-paced shared archives.

W2: benefit with learned policy and value networks

We agree with your reading. On the 17-game neural tier the evolved mechanisms are at statistical parity with tuned PUCT, and we have updated Section 4.3, Section 6, and Appendix K.4 in the revised PDF to state this directly.

  • 50 ms budget. SCO 508.91 ± 0.67 versus 508.05 ± 1.12, and a pairwise win rate of 52.9% ± 8.0% (Table 1). Across the five runs the evolved mechanism ranks first in three and second in two (original Table 13; revised Table 17).
  • Fixed-simulation budget. PUCT is slightly ahead: 507.42 ± 0.57 versus 506.73 ± 1.37, with a win rate of 45.9% ± 6.2% (original Table 15; revised Table 19).

What the tier does show:

  1. The mechanisms transfer without ever seeing a network. They were evolved from a bare seed (fixed c_puct = 2.0, no first-play urgency, no root noise, no schedule) using only symbolic C++ heuristics. On frozen PPO networks they match a PUCT baseline whose exploration constant was tuned on the training suite. PUCT was also tuned without neural networks, so neither side is adapted to this tier.
  2. They are ahead of the methods designed for learned priors and values, under both budgets: Gumbel AlphaZero (win rate 70.4% ± 4.1% at 50 ms; 68.1% fixed-simulation), RPO (69.2% ± 8.0%; 67.3%) and MENTS (79.7% ± 3.6%; 76.7%).
  3. The gain over PUCT is concentrated in particular games. These head-to-head episode win rates are from the same evaluation logs and are not yet in the PDF. They are the games with the largest gains, not a representative sample; on the remaining games the two are close or PUCT is ahead, as the 52.9% aggregate implies.
    • Hex (50 ms): 86.0% five-seed mean (93%, 97%, 96%, 46%, 98% for Runs 1–5; Run 4 loses).
    • Dark Hex, abrupt (50 ms): 75.6% five-seed mean, above 50% in all five runs.
    • Phantom Tic-Tac-Toe (50 ms): 58.0% five-seed mean, above 50% in all five runs.
Reviewer_SojUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a modular LLM-driven evolutionary framework for discovering general-purpose MCTS algorithms. The central idea is to factorize game-playing agents into game-agnostic procedural search mechanisms and game-specific knowledge modules, and to co-evolve them through separate but interacting evolutionary processes. The framework employs an invariant PUCT reference, headroom-normalized objectives, and adaptive opponent progression to guide cross-game algorithm discovery.

The authors evaluate the resulting search mechanisms against 15 established MCTS baselines across 50 training games, 19 held-out games, 360 procedurally generated games, and 17 games with frozen PPO policy-value networks. Performance is assessed primarily through AlphaRank and Soft Condorcet Optimization, supplemented by exploitability analysis and comparisons with specialized game engines. The discovered mechanisms outperform several baselines in cross-game rankings, although improvements under neural representations are more limited.

Critical strengths
  1. The separation between game-agnostic procedural mechanisms and game-specific knowledge is a sensible and practically useful formulation. The invariant reference search, normalized cross-game objectives, and asynchronous co-evolutionary curriculum provide a coherent approach to a difficult credit-assignment problem. This is the strongest aspect of the paper.

  2. The evaluation covers heterogeneous game families, including held-out environments, procedurally generated games, and frozen neural representations. The comparisons against 15 MCTS baselines under matched computational constraints provide evidence that the framework can discover competitive search mechanisms beyond individual training games.

Critical weaknesses

The representative BIFV-MCTS algorithm primarily combines established ideas, including PUCT scheduling, first-play urgency, root Dirichlet exploration, and value-based tie-breaking, with relatively simple game-property-dependent scaling factors. The broader evolutionary framework is interesting, but the discovered mathematical operators appear incremental.

More importantly, Appendix Table 16 shows that removing any one of the four proposed operators improves performance on all 360 synthesized games. Removing FPU annealing also improves performance on the training and held-out tiers by 9.2 and 13.9 percentage points, respectively. These results weaken the argument that the full BIFV combination represents a consistently superior or synergistic search design. Stronger evidence is needed to establish which discovered components are genuinely consequential.

The exploitability evaluation in Appendix D.2 tabularizes stochastic root visit-count distributions, whereas the deployed C++ agent selects a single action through the final action selector. Therefore, the reported exploitability does not necessarily characterize the policy actually used for gameplay.

In addition, the pairwise preference definition in Eq. (18) uses the number of evaluation games as its denominator, while other sections report decisive-only rates after excluding ties and timeouts, such as 285/401 for Run 5. The relationship between these quantities and the main-table win rates is insufficiently specified.

Finally, the Run 4 implementation in Listing 10 does not appear to provide an atomic update of the coupled visit-count and value statistics required for its claimed nonlinear Robbins–Monro backup under parallel execution. These issues do not invalidate every experimental result, but they limit confidence in several central interpretations.

The reference at line 543 should be corrected to:

Chaslot, G., Bakkes, S., Szita, I., & Spronck, P. (2008). Monte-Carlo Tree Search: A New Framework for Game AI. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 4(1), 216–217. https://doi.org/10.1609/aiide.v4i1.18700

Questions for authors
  1. Table 16 shows that removing each proposed BIFV operator improves performance on the largest benchmark tier, consisting of 360 synthesized games. What evidence supports the claim that the full operator combination is synergistic or preferable to the strongest ablated variants? This is important for assessing the actual algorithmic contribution.

  2. Does the exploitability analysis measure the policy actually executed by the agent, or a distinct stochastic policy derived from root visit counts? If they differ, how do the conclusions about robustness and exploitability transfer to the deployed action-selection policy?

  3. lease reconcile Eq. (18), the exclusion of ties and timeouts, and the reported decisive-only statistics such as 285/401 and 279/391. Are the pairwise percentages in Tables 1 and 15 conditional on decisive games, or calculated over all evaluated environments?

  4. How does the Run 4 implementation guarantee a correct Robbins–Monro value update under four-thread parallel search? Listing 10 updates the value sum and visit count separately, without an apparent lock or compare-and-swap operation protecting their joint consistency.

2 comments
AuthorsOpenReview ↗

Thank you for your review

We thank Reviewer SojU for a careful technical review and for highlighting the strengths of our factorized formulation, invariant reference search, headroom-normalized objectives, and cross-game evaluation. Each of the technical issues raised pointed to a concrete improvement we have made in the revised manuscript.

W1 and Q1: BIFV operators and Table 16

Thank you for flagging Appendix Table 16. Checking the standalone ablation pipeline revealed evaluation configuration issues in how the ablated variants were launched, which caused the anomalous numbers in that table. We have removed Table 16 (and the accompanying operator-synergy phrasing in Appendix J.4) from the revised PDF and are currently rerunning the ablation with the corrected evaluation setup; we will include the updated results in the final version and post them here if the run finishes during the discussion period.

Importantly, this issue was isolated to the standalone operator-ablation script in the appendix and does not affect any of the main benchmark evaluations or the central claims of the paper. On the first part of W1, we agree that BIFV-MCTS recombines established MCTS ideas (PUCT schedules, first-play urgency, root Dirichlet noise, value tie-breaking) with game-property scaling factors. We present BIFV-MCTS as an interpretable case study of one discovered mechanism (with its full C++ code in Appendix J.2, Listing 1), while the primary claim of the paper is at the framework level: across five independent evolutionary runs, our factorized co-evolutionary system consistently discovers mechanisms that rank first or second among 16 algorithms on every tier under the 50 ms budget (Tables 1 and 17) and generalize zero-shot across held-out and procedurally synthesized games. That result does not rely on Table 16.

W2a and Q2: visit-count policy versus executed policy

Thank you for pointing this out. The submitted Appendix D.2 measured only the averaged root visit-count policy (π_visit), whereas gameplay executes the discrete action returned by FinalActionSelector (π_exec). In the revised paper (Section 4.2 and Appendix D), we re-ran the tabularization with M = 200 searches per information state across all 5 evolution seeds and report both π_visit and π_exec.

The core conclusion of Section 4.2—that the procedural mechanism (m*) transfers reliably across seeds whereas co-evolved heuristics (κ*) can overfit—holds across both policies:

Game (B = 1000, Table 4)Visit policy (π_visit): PUCT → <m*, κ_base> (Best m*)Executed policy (π_exec): PUCT → <m*, κ_base> (Best m*)
Kuhn poker0.1681 → 0.1199 (0.1199)0.1622 → 0.1478 (0.1399)
Leduc poker1.4998 → 0.9455 (0.9108)1.2791 → 1.0209 (1.0209)
Liar's dice0.5662 → 0.5655 (0.4667)0.5259 → 0.5696 (0.5011)
  • Pure mechanism transfer (<m*, κ_base>, Tables 7–12): Across all 15 (game, budget) cells, <m*, κ_base> lowers exploitability over PUCT in 15/15 cells on π_visit and 11/15 cells on π_exec. On Leduc, the >10× simulation efficiency gain holds on π_exec as well: <m*, κ_base> at B = 100 (1.0977) already beats PUCT at B = 1000 (1.2791) and reaches 1.0209 (−20.2%) at B = 1000 (with higher π_exec only at B ≤ 50 on Leduc and B ≥ 500 on Liar's dice).
  • Modular attribution and FinalActionSelector (Appendix D.6): In all five mechanisms FinalActionSelector picks a most-visited root action (with BIFV-MCTS using 0.1 × Q_norm only to break integer visit ties). Under κ_base, root Dirichlet noise in m* diversifies discrete root selections across determinized trials, whereas sharp co-evolved priors (π*) concentrate π_exec onto exploitable near-pure actions—reinforcing that m* is the module that transfers.

W2b and Q3: Eq. (18), ties and timeouts, 285/401, Tables 1 and 15

  1. Denominator. The percentages in Tables 1 and 15 (revised Table 19) are computed per seed over co-evaluated games: games whose tournament had produced payoff estimates for both candidates. They are not conditional on decisive games and not computed over all 446 games. Ties stay in the denominator, with the tied pair ordered by their position in the AlphaRank ballot. The sentence in Appendix H.3 that said ties and timeouts were excluded was wrong and is corrected (revised Eq. 20).

  2. Where 285/401 and 279/391 came from. Those fractions were from an earlier checkpoint (282/402, 279/391, 262/401, 270/402, 285/401). Tables 1 and 15 were computed from a later one (280/408, 284/406, 279/412, 275/408, 292/409; mean 69.0% ± 0.7%). Both means round to 69.0%, so the mismatch went unnoticed. The revised paper uses the later fractions throughout.

  3. Tournament snapshot vs. final logs (documented in revised Appendix H.3). Each per-game tournament runs under a two-hour limit, and at the snapshot when Table 1 was computed 406–412 of the 446 games per run had estimates for both candidates. In the final logs that number is 428–434. On the final logs (revised Table 16), m*_evo versus PUCT under the 50 ms budget (fixed-simulation in parentheses) is:

    ConventionWin rate
    Ties counted as losses72.2% ± 0.8% (73.8% ± 1.5%)
    Half credit for ties73.5% ± 0.8% (75.1% ± 1.5%)
    Ties excluded74.1% ± 0.8% (75.8% ± 1.5%)
    Ties and unfinished games as losses, over all 44669.6% ± 0.8% (71.3% ± 1.5%)

    Under the convention of Table 1 the final-log figure is 72.2%, against 69.0% in Table 1, so Table 1 is conservative. For the final version we will regenerate Table 1 and the per-seed tables directly from the final logs and report the exact co-evaluated game count per tier.

W2c and Q4: Run 4 backup under four threads

Your reading of Listing 10 is correct: because total_value and visits are stored as separate atomics without a joint lock or compare-and-swap, Run 4's nonlinear (n+1)^−0.82 update acts as a lock-free relaxed approximation under 4-thread parallel search rather than an exact sequential Robbins–Monro backup (whereas the other four runs use linear additive accumulation). In the revised paper, we removed the secondary-seed mechanistic writeups and the exact Robbins–Monro convergence claim from Section 4.3 and Appendix J.4 so the manuscript focuses strictly on the primary BIFV-MCTS case study (while retaining Run 4's C++ code in the supplementary material). Because Run 4 was evolved and evaluated in the same 4-thread engine, its empirical benchmark results are unaffected.

W3: Chaslot et al. (2008)

Thank you. Our entry combined the author list of a different 2008 Chaslot et al. paper with the wrong venue. The revised paper uses the AIIDE reference you gave.

Reviewer_EzyxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper addresses the challenge of general game playing for AI, where traditional handcrafted search algorithms (like MCTS variants) often struggle to transfer across game classes. Instead of manually designing search heuristics, the authors propose using LLMs to automatically discover game-agnostic search algorithms. They introduce a multi-agent LLM meta-learning framework built on top of AlphaEvolve that co-evolves C++ simulation-based search mechanisms.

Critical strengths
  • The core insight to decouple the agent into a game-agnostic C++ procedural search mechanism and game-specific heuristics is an interesting topic.
Critical weaknesses
  • The presentation of the paper is poor.
  • A 66-page appendix with raw C++ source code pasted is redundant
  • This paper applies an existing meta-learning framework (AlphaEvolve) to an existing modular MCTS template over an existing game framework (OpenSpiel). The novelty is limited.
Questions for authors
  • The 15 baselines in Table 1 are described as "tuned on the training suite." What was the tuning budget and search space for each baseline?
  • What is the total GPU/CPU-hours consumed across all five evolutionary runs?
  • Is the discovery process reproducible without proprietary access to Gemini 3.5 Flash?
2 comments

Q2: compute across the five evolutionary runs

Appendix B.6 reported tokens and API cost. The fuller breakdown:

  • GPUs. None during evolution. All search evaluation ran on CPU and the LLM was called through its API. Separately, training the PPO networks for the 17-game neural tier used about 4,346 accelerator chip-hours.
  • Up to the reported champions (about 6,400 candidates per run, reached after 7.4–23.2 wall-clock hours): about 9,500–12,000 CPU-core-hours over the five runs, and about 300M input and 550M output tokens per run.
  • Full runs. The jobs were left running for 4.5–6.2 days (694.8 hours in total, 94k–100k candidates per run) to check long-horizon stability. That used about 158,000–207,000 CPU-core-hours.
  • Caveat (noted in revised Appendix B.6). The five reported runs did not log token counts directly; the token figures are estimates calibrated on instrumented runs under the identical configuration.

Q3: reproducibility without Gemini 3.5 Flash

We have not re-run the discovery process with another model, so we cannot claim it reproduces without Gemini 3.5 Flash.

What does not depend on any LLM is the result itself. BIFV-MCTS is given as complete C++ in Appendix J.2 (Listing 1) with its mathematical operators in Section 4.3, Appendix J.4, and revised Table 20, while the seed and all five discovered mechanisms are provided as complete C++ in the supplementary material. They can be compiled against OpenSpiel and evaluated on CPU. The LLM is used only as a mutation operator that receives a text prompt (printed in Appendix J.3, Listing 2) and returns a code edit, so the pipeline does not rely on fine-tuning or model internals.

I appreciate the authors for their detailed response. However, I still have concerns about this paper, particularly regarding the novelty of the method. Therefore, I have decided to maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.