acceptodds
Under review as a conference paper at ICLR 2027

Regularized policy gradient with learned mixtures of Gaussians for games with continuous actions

Abstract

Most successes of superhuman game-playing algorithms are in games with discrete actions, yet in auctions, robotics, sports, or trading, actions are nearly continuous. Prior techniques either rely on expert-designed discretizations or are sample inefficient. We present a scalable policy-gradient algorithm for large sequential games with continuous or mixed discrete and continuous actions. It combines magnetic mirror descent with a mixture of Gaussians reparametrization, trained via self-play. We show that it approximates equilibrium in games where gradient descent fails. In sequential games, it outperforms neural fictitious self-play and matches or outperforms the final strategies of policy space response oracles with 3.5–5.5 fewer samples. In heads-up no-limit Texas hold'em, it performs on par with Slumbot.

Primary area:
reinforcement learning
Keywords:
two-player zero-sum games, imperfect information games, mixed equilibria, mixture of Gaussians, policy-gradient, Gaussian reparametrization, Poker

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tHRrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to approximate Nash equilibria in large, sequential, two-player zero-sum games with continuous action space. As action discretization may disregards the relationships among discrete actions and ordinary policy gradients may not converge, the paper proposes Magnetic Mixture Policy Optimization that combines Gaussian mixture policies, Magnet KL regularization, self-play. It gives some guarantee under quite restrictive conditions, and evaluate the proposed method on some continuous-action games.

Critical strengths

The potential contribution is showing that this combination can produce effective mixed strategies in continuous action games with relatively few environment interactions.

Critical weaknesses

I found this draft difficult for human to read, as it uses many concepts/notions without properly defining them, and suddenly presents a pile of sentences without a coherent reasoning. This gives me impression that the writing may have relied heavily on LLMs. I personally have no objection to using LLMs to assist with writing, but I think at least one of the authors should read once the generated content from beginning to end to check the readability.

For example, the second and third paragraphs of the Introduction section.

From the summary given by AI, the draft seems to focus on games with continuous action spaces. The proposed method combines several well-established techniques, i.e., magnetic mirror descent, Gaussian mixture models, and KL regularization. Honestly, I struggle to find any genuinely novel insights or ideas beyond this combination of existing tools.

Questions for authors

See above

3 comments

We would like to defend ourselves against the AI claims. This paper was not written by AI. We did use LLMs to polish the writing, as we acknowledged in the paper. However, all of the presentation decisions were ours, including the decision to explain why mixing between several distinct strategies is done this way. The concepts discussed in Paragraphs 2 and 3 of the introduction are standard in game theory, including the fact that optimal strategies often require mixing among several actions. Below we expand on the topics of those paragraphs. If the reviewer thinks it is appropriate to expand in the paper as well, we can also include such expansions in the camera-ready version of the paper.

Paragraph 2

Paragraph 2 aims to show that in adversarial problems, if one player always follows the same deterministic strategy, it can be exploited. Consider Rock-paper-scissors, if one player always plays Rock, then a rational opponent will always pick Paper and win every time. It is known that the optimal strategy in Rock-Paper-Scissors is to mix all 3 actions with equal probability. This paragraph argues that the same thing happens even when the actions are continuous rather than discrete.

One may argue that the Gaussian reparametrization already mixes between several actions. However, if we consider the penalty kick in football (soccer), and the action as the angle at which the ball is kicked. Let us assume that the optimal strategy for such a scenario mixes equally often between kicking the leftmost and rightmost parts of the net, without shooting in the middle. Such a strategy is impossible to model with a single Gaussian. This motivates our use of a Gaussian mixture model rather than a single Gaussian, as in PPO or SAC.

Paragraph 3

Paragraph 3 then explains that if you naively use policy-gradient algorithms, such as PPO, in discrete action games, they will not converge to the optimal strategy. For example, in Rock-paper-scissors, PPO does not converge to a uniform strategy (this is similar to the behavior we show in Figure 7 in the Appendix for continuous matching pennies). This is a problem that has been well studied in prior work. To the best of our knowledge, the only truly scalable approach that converges to these optimal strategies in discrete action games is to use KL regularization (as in algorithms such as RNaD, MMD, or NashPG). This paragraph motivates why MMPO uses the KL-divergence regularization. We further expand on this in Section 3.2, where we show that the KL divergence is indeed beneficial in the Gaussian reparametrization.

Whole method

MMPO parametrizes the strategy as a Gaussian mixture model because it has greater expressive power than a single Gaussian and allows for mixing among several, possibly vastly different, actions. MMPO has two heads: one Gaussian head predicts KK means and standard deviations, while the second categorical head predicts a categorical distribution over those components. Section 3.1 explains that MMPO needs to use KL divergence in the policy-gradient loss for the categorical head, because otherwise the categorical head may end up in a Rock-Paper-Scissors scenario, where it fails to converge to an optimal mixture of the individual components. Section 3.2 shows that using the KL-divergence for the Gaussian head is beneficial, because it can guarantee its convergence in games where plain PPO would not converge, which we empirically verify in Section 4.1. Section 3.3 then only formalizes the whole policy-gradient loss of MMPO.

Known components

While we agree that our work can be viewed as a combination of known components, that does not mean it lacks valuable insights. We show that naively applying PPO to continuous action games can yield suboptimal strategies (Sections 4.1-4.3). We prove that the theoretical guarantees of KL-regularization translate from discrete action games to some classes of continuous action games when using Gaussian reparametrization. We also explain that naively extending the PPO to a mixture of Gaussians without the KL-divergence for the categorical head may result in the cyclical behavior seen in discrete action games. This means that the KL divergence for the categorical head is not merely an arbitrary choice but a principled solution. We would argue that the fact that our work uses known concepts but warns about the potential dangers of their combination and shows how to combine them well is a strength of the approach, not its weakness.

I appreciate the revisions to the introduction. But I still found the results in Sections 2 and 3 difficult to verify. My main concern is that the convergence analysis covers a single Gaussian, while MMPO jointly learns a Gaussian mixture. In my view, a convergence claim for the full algorithm would require substantial additional work.

All references below are to the revised manuscript.


My main concern is the gap between the motivation for MMPO and the setting covered by its theory. The paper motivates learned Gaussian mixtures through games in which equilibrium play requires randomization over several distinct actions. Theorem 1 establishes convergence for a single Gaussian with a fixed magnet. When multiple Gaussian components are learned jointly, both their weights and their parameters change, and the theorem does not establish convergence for those coupled updates.

Section 2 assumes compact action sets (line 74), whereas the Gaussian policies in Theorem 2 have unbounded support. How are samples outside the action set handled? If they are clipped to the boundary, please justify why the stated exploitability bound still holds.

Section 3.3 uses an upper bound on the mixture KL divergence as a surrogate regularizer (lines 206–209). This bound can be loose. Could the authors clarify how this surrogate gap affects the policy updates?

I found the theoretical results in Sections 2 and 3 difficult to verify because several definitions are implicit or appear only after the notation is used. For example, Theorem 1 states that the iterates converge to the unique solution of VI(VR,Π)\mathrm{VI}(V^R,\Pi) (line 170). Even after chasing down the proof, I couldn't find the precise inequality. It would also help to state both explicitly before the theorem, specify the coordinates in which V(π)V(\pi) and R(π)R(\pi) are defined, and explain the roles of LVL_V, δR\delta_R, and σmin⁡\sigma_{\min} when they first appear. This is a minor presentation issue, but it significantly affects readability.

Thank you for the ongoing discussion and for pointing out problems with the presentation of the paper!

Mixture of Gaussians theory

Theorems 1-3 use a parametrization of strategy as a single Gaussian, and we are explicit about this in the paper. The goal of those theorems is not to say how the whole MMPO converges. They only serve as supporting arguments for using KL regularization for the Gaussian heads (similarly, Section 3.1 argues the same for the categorical head).

We would like to be clear that we are not claiming that MMPO is guaranteed to converge to a Nash equilibrium. To the best of our knowledge, there is no scalable policy-gradient algorithm for sequential continuous action games that models mixed actions and provably converges to Nash equilibrium. MMPO serves as a heuristic algorithm for these types of problems. While it may be possible to define a restricted class of games in which MMPO would provably converge, our approach was to empirically demonstrate that MMPO finds strong strategies across domains where gradient-based algorithms are unlikely to provide guarantees, such as sequential games or games with continuous support.

Compact action sets

Thank you for pointing this out. This is an oversight on our side, because in Theorems 2 and 3, we implicitly assumed that even if the action sets are defined over compact sets, the utility function can be extended beyond this legal action set with the same bounded second derivative. We have added a note that the Gaussian mean is within the action set (line 155), Assumption 2, which explicitly states that it uses the extended utility over the whole real number set, and an explanation for this assumption (lines 189-195). We also modified the bodies of the Theorems and propositions to directly use this new assumption.

In our implementation, when an action outside of this set is sampled, we clip it to the set. With the clipping, the same proof no longer holds, because it breaks the monotonicity assumption, so it may require additional steps to find a bound for that setup. Again, the point of Theorems 1-3 is only to show the benefit of the Gaussian KL regularization, that is why we did not try to construct a proof for the clipping setting, since the Theorems already assume an idealized setting.

Theorem 1

The fact that the VI(VR,Π)\mathrm{VI}(V^R,\Pi) converges to the unique solution directly follows from the last line of the proof. By Proposition 1, there is a unique solution. By Eq. (2), k→∞  ⟹  πk=π∗k \to \infty \implies \pi^k = \pi^*. We have explicitly added this to the proof for completeness.

KL divergence estimate

We have run an experiment in which we trained the MMPO using either the true KL divergence estimator or our weighted-sum estimator. The entire experiment is in Appendix D.5. With both KL estimators, the MMPO converges to roughly the same value. We also show in that experiment how the two estimators differ during one run.

Reviewer_LwxaOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper extends MMD style regularized policy optimization to games with continuous action space. It uses a mixture of Gaussian policy with discrete categorical head and continuous Gaussian head. The authors evaluate MMPO on a suite of continuous games and variants of pokers, beating or on par with previous best results.

Critical strengths
  • The paper is well motivated, well positioned and easy to follow. It explains prior works and connections to prior works well.
  • The authors have provided both theoretical and empirical justifications to the method. The empirical evaluations are strong, covering multiple games including larger sequential ones. It either achieves the best performance, or achieves comparable performance with previous methods with much less samples.
  • The Texas hold'em results against Slumbot demonstrates its scalability and strong performance.
Critical weaknesses

Not a "critical" weakness to me, but MMPO could be viewed as an incremental change over the MMD method, or natural extension of MMD into continuous action domain, and the choice of mixture of Gaussian makes a lot of sense.

A missing baseline is PPO with the same mixture of Gaussian architecture.

Questions for authors
  • What will happen if the PPO also use the same mixture of Gaussian architecture?
  • I would like to see a coefficient sweep over the regularization strength on one or two environments.
2 comments

We would like to thank the reviewer for their review and suggestion for the ablation experiment.

Regularization effect

We have run an ablation experiment using the small games in Section 4.2 with different regularization magnitudes. We have included this experiment in Appendix D.4. This ablation answers both questions because PPO with a mixture of Gaussians is an ablation in which both regularizations are set to 0. Surprisingly, it turns out that in the Circle game, the Gaussian regularization is necessary for convergence, whereas for the Two-point game, the categorical regularization is necessary. We think this is because of the structure of the optimal strategies, where in the Circle game, the Nash equilibrium mixes over infinitely many actions. Hence, the position of the Gaussians is more important than the distribution over them. In contrast, the Two-point game requires a specific distribution between a fixed number of points. In the Glicksberg-Gross game, even the PPO with a mixture of Gaussians performs relatively well.

Thanks for you response. I confirm that I have read it and I keep my score.

Reviewer_kaqbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a mixture-of-gaussians approach to handling continuinous games, using MMD-style regularization to help with convergence. It validates these on some continuous equivalents of standard benchmarks and against Slumbot for a larger scale validation.

Critical strengths

I like this paper. It contributes a useful tool to continuous game-solving, validates that it works on a variety of benchmarks, and seems to be sound. It mostly does not overstate its claims and has appropriate baselines.

Critical weaknesses
  • The theory section of the paper seems to present some results that don't seem extremely relevant to the full method of the paper, and I am not sure what additional insights it provides. I have questions about this, so I may remove this weakness after discussion.

Nits:

  • The Leduc and Kuhn games in the main text should be more clearly labeled as continuous. I had to check the appendix to confirm that something odd wasn't happening.
Questions for authors
  • One thing I was confused about is the separate clipping terms for the categoricals and the Gaussians. Is there a particular reason to prefer that instead of some sort of joint clipping?
  • In equation 6, for the KL between the two normal distributions, is it between the sampled normal or the full weighted sum? Why one versus the other?
  • I don't quite understand what point section 3.2 is making in the context of the paper. As in, I understand the contexts of the section, but why is this relevant to the papers contributions?
  • Why does the right hand side of figure 5 seemingly have a different win rate than the stated win rate against slumbot? The stated win rate has much wider errors.
  • Is there a comparison with just running PPO on slumbot? You don't need to run it, I am just curious.
1 comment

We would like to thank the reviewer for their review, improvement suggestions, and questions.

Leduc and Kuhn labeled as continuous

Thank you for pointing this out. We have updated it in the revision.

Clipping term

We opted for this because the method can be easily explained as a combination of the two loss terms, both of which are versions of MMD, one (categorical) loss corresponds to the discrete MMD as Sokota et al. defined it. The second (Gaussian) loss corresponds to the Gaussian MMD we introduced in Section 3.2. We think that joint clipping is also possible. However, using the same ratio, the clipping would be slightly more aggressive.

KL divergence of the Mixture of Gaussians

We use a weighted sum of the individual KL divergences because it has lower variance than the sampled one.

Structure of the paper

Our rationale behind this structure is:

  • Section 3.1 aims to show that the KL regularization is necessary for the categorical head.
  • Section 3.2 shows that the KL regularization for Gaussian reparametrization is beneficial in some games.

Without Section 3.2, it may not be obvious why we used the KL regularization for the Gaussian head of the MMPO. This Section shows that it also provides theoretical benefits in some games. In the ablation we have added to Appendix D.4, we empirically show that both of the regularizations are beneficial.

Mismatch between the plot and the reported value

We apologize for the confusion. We have used two separate comparisons against Slumbot. In the figure, we used 150k games against Slumbot for each checkpoint. We then ran a second comparison, which played 900k games to improve the statistical significance of the results; as a result, the two values differ. We agree with the reviewer that this is confusing, and we have updated the figure to include this second estimate at the end. We have also included details about the games played for the evaluation in the Appendix.

PPO with Slumbot

We have not run this experiment ourselves. However, one of the works we reference in the paper, AlphaHoldem, has used PPO. That work uses a discrete, predefined abstraction for the bets and uses self-play against K-best checkpoints. Based on their results, they outperform Slumbot by ~100 mbb/hand after 3 days of training on 8 GPUs. However, they do not report Local best response, which is a better estimate of the bots strength. Our experiment was not done to build the strongest bot, but to show that MMPO can be scaled to large games and still perform well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.