acceptodds
Under review as a conference paper at ICLR 2027

DiffNash: Finding Nash Equilibrium in High-Dimensional Continuous Action Spaces with Diffusion Policies

Abstract

Computing equilibria in high-dimensional continuous games requires learning and representing rich, multi-modal strategy distributions, something conventional neural policies struggle to achieve. Diffusion models offer a natural solution, but standard diffusion training requires direct samples from a fixed target distribution, which are generally unavailable under iterative game-solving updates. We propose a diffusion-based framework for equilibrium computation that enables diffusion-based variants of Online Mirror Descent (OMD) and Fictitious Play (FP). Our approach derives a reweighted score-matching objective that trains diffusion policies without requiring direct samples from each updated policy. Our main method, , combines the fast convergence of OMD with the sample efficiency of FP by reusing past actions through a replay buffer. To correct the resulting distribution shift from sample reuse, we derive an unbiased estimator of the difference between current and past policies’ Evidence Lower Bounds, whose exponential provides a tractable proxy for the corresponding importance weight without evaluating diffusion likelihoods. We evaluate our approach on forest protection games and continuous Colonel Blotto games, reducing the Nash gap over the strongest baseline by and , respectively, with minimal hyperparameter tuning.

Primary area:
generative models
Keywords:
Diffusion Model, Equilibrium Computation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6T6HOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DiffNash, a framework for computing approximate equilibria in continuousaction games using diffusion policies to represent mixed strategies. It derives reweighted denoising objectives for online mirror descent (OMD) and a fictitious-play variant, and introduces replay of opponent and self-actions. Replay samples are weighted using an exponentiated stochastic estimate of the difference between current and behavior-policy evidence lower bounds (ELBOs). Experiments on Cournot, forest protection, and continuous Colonel Blotto games provide evidence that the framework can improve equilibrium-learning performance.

Critical strengths
  1. A useful representation for mixed strategies. Diffusion policies provide an expressive representation of multimodal strategies in continuous-action games. The experiments, particularly the Blotto examples, illustrate why policy expressiveness can matter for equilibrium computation.

  2. A principled connection to equilibrium updates. Proposition 1 connects the idealized OMD update to a reweighted denoising objective. This provides a meaningful foundation for the framework, independently of whether the practical replay correction is exact.

  3. Encouraging empirical evidence and informative ablations. The paper evaluates several game configurations, reports uncertainty across seeds, and includes replay-component ablations and hyperparameter studies. Section 5.3 and Appendix F.4 distinguish the optimization benefit of self-action replay from the computational benefit of opponent replay. Improvements are not uniform: DiffFP has lower mean Nash gaps in the symmetric Blotto settings in Table 2. Nevertheless, the overall results support the practical potential of the approach.

Critical weaknesses
  1. The replay-weight proxy is insufficiently validated. Proposition 2 does not establish the accuracy of the exponentiated, clipped, self-normalized weights. A tractable density-ratio diagnostic would help identify when the correction is reliable.

  2. The benefit of weighting is not fully isolated from the benefit of replay. The existing component ablations are useful, but a matched comparison of corrected and unweighted replay would better distinguish distribution-shift correction from the effect of historical sample diversity.

  3. Equilibrium-quality estimates depend on approximate best responses. Appendix D.1 describes the evaluation procedures, but the resulting Nash-gap estimates are not certified upper bounds on exploitability. Search-budget sensitivity checks would strengthen confidence in the reported rankings.

Questions for authors
  1. Can you quantify/qualify the unbiasedness of the final estimator?
  2. On a tractable diagnostic problem, how closely do the proposed weights match exact or accurately estimated density ratios?
  3. Are the main performance rankings stable under increased best-response evaluation budgets?
4 comments
AuthorsOpenReview ↗

W2: IS correction is Necessary / W1&Q2: Ablation on Clipping and Normalization / W3: Result is Robust to Different Search Budgets

We thank the reviewer for the qusestions.

W2 : IS weight correction is Necessary

To isolate the contribution of importance weighting from the benefit of replay itself, we conduct an additional ablation on the two forest-security settings shown in Figure 4. We compare DiffNash against variants without replay and with unweighted replay. The result is added to Line 475-488 of the updated paper.

SettingMethodNash Gap ↓ (mean ± 95% CI)
F2, m=n=3No Replay, No IS631.6 ± 150.8
With Replay, No IS625.6 ± 137.1
DiffNash (Ours)352.9 ± 132.8
F3, m=n=5No Replay, No IS866.8 ± 224.4
With Replay, No IS702.3 ± 355.6
DiffNash (Ours)556.6 ± 163.9

Replay alone increases sample diversity and stabilizes training, particularly in F3, where it reduces the mean Nash gap by 19.0% compared with the baseline Without Replay or IS. However, unweighted replay does not account for distribution shifts between historical and current policies. By using our proposed ELBO-based IS correction, DiffNash further reduces the mean Nash gap by 43.6% on F2 and 20.7% on F3 compared with unweighted replay. This result highlights the effectiveness of our ELBO-based importance sampling estimator in mitigating the distribution shift induced by sample reuse and demonstrating the necessity of such correction.

W1/Q2: Accuracy of the clipped, self-normalized weights. Tractable density-ratio diagnostic.

One possible alternative for approximating the density ratio is using probability flow ODE [1], which maps an action to the noise prior and integrates the flow’s divergence to estimate its log density. However, computing the log-density difference for a batch of 128 opponent samples takes 10.8 ms with our ELBO-based estimator, compared with 6.33s using the probability flow ODE. This 586x slowdown makes using the probability flow ODE throughout the training too costly. In practice, we use our ELBO-based density estimator to perform diagnostics and confirm the effects of clipping and self-normalization. We discuss the effect of exponentiation later with our response to Q1.

Without clipping, exponentiating our ELBO-based estimator to approximate importance weights can occasionally cause numerical overflow and destabilize training. Hence, we clip importance weights to limit extreme values and stabilize training, a common practice also applied in PPO objectives [2]. On Cournot (d=30), 6.42% of exponentiated ELBO estimates reach the clipping threshold over 600 iterations, dropping to only 1.9% during the more stable iterations 101–500.

Self Normalization is also common in policy training. For example, PPO performs advantage normalization [2]. We run our method without self normalization on Cournot (d=30), and show self normalization significantly reduces the distance to Nash.

SettingDistance to Nash ↓
With self-normalization2.827 ± 0.051
Without self-normalization8.066 ± 0.338

Ablation on Clipping/Self-normalization is added to Appendix F.6.

W3/Q3: Search Budget

To evaluate the robustness of our Nash-gap estimates, we conduct an additional sensitivity analysis by varying the evolutionary best-response search budget from 25 to 100 candidates, including the original budget of 50, on the F2 and F3 forest-security games. This robustness check is added to Appendix F.9.

Performance under varying search budget

TypeSearch BudgetDeepFPNeural OMDDiffFPOurs
F2 (m=n=3)25656.1 ± 90.4462.4 ± 234.3795.6 ± 106.1236.0 ± 72.9
50 (Original Budget)687.4 ± 78.5500.1 ± 177.2821.6 ± 101.3300.7 ± 126.4
75648.15 ± 73.86539.90 ± 225.91871.72 ± 24.10369.63 ± 37.42
100693.9 ± 74.0565.9 ± 199.3883.7 ± 12.5316.3 ± 81.1
F3 (m=n=5)25636.0 ± 129.2733.8 ± 296.9689.4 ± 341.7467.2 ± 130.0
50565.7 ± 119.7761.1 ± 271.1800.9 ± 121.3528.2 ± 142.7
75760.03 ± 60.74756.12 ± 235.31789.38 ± 83.77638.23 ± 124.21
100742.8 ± 110.2884.7 ± 93.9870.6 ± 148.0590.6 ± 154.4

DiffNash consistently achieves the lowest mean estimated Nash gap across all search budgets in both settings, showing our result is robust to variations in search budget size.

Reference:

[1] Song, Yang, et al. Score-based generative modeling through stochastic differential equations. ICLR.

[2] Schulman, John, et al. Proximal policy optimization algorithms.

AuthorsOpenReview ↗

Q1: Robustness of the final estimator

W2/Q1: quantify/qualify the unbiasedness of the final estimator

Let k be the current policy iteration, k' the iteration where the action is sampled. Recall Et,ϵ[Δ^]=ELBO⁡k(a0)−ELBO⁡k′(a0)\mathbb{E}_{t,\epsilon}[\widehat{\Delta}] = \operatorname{ELBO}^{k}(\mathbf{a}_0) - \operatorname{ELBO}^{k'}(\mathbf{a}_0) is our unbiased estimator of the ELBO difference. After taking exponentiation, an upward bias is introduced. Formally, by Jensen's inequality, E[eΔ^]≥eE[Δ^]\mathbb{E}[e^{\widehat{\Delta}}] \geq e^{\mathbb{E}[\widehat{\Delta}]}, where the expectation is over t,ϵ\epsilon. We make this point explicit in Line 302-304, 516-522, and we further justify the robustness of our ELBO-based estimator in Appendix F.8.

Despite its upward bias, we quantify the robustness of our exponentiated ELBO estimator by showing: 1. the upward bias can be bounded under mild assumption, 2. clipping prevents excessive upward bias, 3. our estimator greatly outperforms an unbiased estimator of the exponentiated ELBO difference empirically, 4. its empirical effectiveness.

Point 1: Bounded Upward Bias

This bias can be characterized and bounded. Fix an action a. If the estimation error ϵ=Δ^(a)−E[Δ^(a)]\epsilon=\widehat{\Delta}(a)-\mathbb{E}[\widehat{\Delta}(a)] is σ2\sigma^2-sub-Gaussian, then the upward bias introduced by exponentiation satisfies

0≤E[eΔ^(a)]−eE[Δ^(a)]≤eE[Δ^(a)](eσ2/2−1).0 \leq \mathbb{E}[e^{\widehat{\Delta}(a)}]-e^{\mathbb{E}[\widehat{\Delta}(a)]} \leq e^{\mathbb{E}[\widehat{\Delta}(a)]}(e^{\sigma^2/2}-1).

See Proposition 5 in Appendix F.8 for the proof. This result shows although exponentiation introduces upward bias, its magnitude can be bounded under mild assumptions on the estimation error.

Point 2: Clipping Mechanism

As discussed in our response to W2/Q1, clipping prevents the upward bias from hurting training stability. Moreover, only 1.9% the exponentiated ELBO difference reach the clipping threshold during after the first 100 iterations on Cournot (d=30), suggesting that the upward bias does not cause frequent extreme importance weights.

Point 3: Unbiased Estimator is Possible, but Performs Much Worse

An unbiased estimator of the exponentiated ELBO difference can be constructed as follows. Let M∼Poisson(λ)M\sim\mathrm{Poisson}(\lambda) with λ>0\lambda>0, independently of i.i.d. unbiased estimates Δ^k,k′(a;j)\widehat{\Delta}^{k,k'}(a;j). We define w^k,k′(a)\widehat{w}^{k,k'}(a) by

w^k,k′(a)=∏j=1M(1+Δ^k,k′(a;j)λ)\widehat{w}^{k,k'}(a)=\overset{M}{\underset{j=1}{\prod}}\left(1+\frac{\widehat{\Delta}^{k,k'}(a;j)}{\lambda}\right),

with the empty product equal to 1. We provide the unbiasedness proof in Appendix F.8. The estimator requires E[M]= λ\lambda score-network calls on average. This unbiased estimator suffers high variance and requires a random number of samples. We fix other components and replace our ELBO-based estimator in DiffNash to the unbiased one, and run it on Cournot (d=30) for 600 iterations.

MethodDistance to Nash ↓
DiffNash (Ours)2.827 ± 0.051
Unbiased (λ=1)11.147 ± 0.935
Unbiased (λ=5)9.410 ± 0.483
Unbiased (λ=10)7.898 ± 0.608
Unbiased (λ=20)6.936 ± 0.363

DiffNash requires only one score network call per sample reuse, whereas the unbiased estimator requires λ\lambda calls on average. Our ELBO-based estimator achieves a substantially closer approximation to Nash equilibrium, even when the unbiased estimator uses 20 times as many network calls on average.

Point 4: Empirical Effectiveness

Our prior response to W2 shows that using our exponentiated ELBO-based estimator as the IS weight reduces the mean Nash gap by 43.6% on F2 and 20.7% on F3 compared with unweighted replay, demonstrating its empirical effectiveness.

Thank you for the clarifications and additional experiments. The weighted versus unweighted replay ablation and the best-response budget sensitivity checks substantially address two of my concerns. The main remaining limitation is that the clipping and normalization ablations do not directly validate the weights against actual density ratios. Likewise, controlling bias relative to the exponentiated ELBO difference does not establish accuracy relative to the true density ratio. Please make this distinction explicit and qualify the claim that importance correction is “necessary.” The rebuttal strengthens the empirical case, and I maintain my weak acceptance recommendation.

AuthorsOpenReview ↗

Comparison Between Our ELBO-based Estimator and True Density Ratio

We thank the reviewer for the constructive feedback. We add further comparison between Our ELBO-based Estimator and True Density Ratio here. The new experiment is added to Appendix F.11. We hope the reviewer could take the new result into consideration.

Probability Flow ODE [1] enables computing true probability density for a diffusion policy. We therefore compare our ELBO-based density-ratio estimates with this ODE-based reference to assess the robustness of our estimator.

Evaluating density ratio under probability Flow ODE is 586x times slower than our lightweight ELBO-based method. Since using Probability Flow ODE to compute true density ratio throughout the training run is too expensive, we evaluate selected checkpoints from the beginning, middle, and end of training in Cournot (d=1d=1) and Blotto Symmetric (d=6d=6). During training, our replay buffer retains samples from the preceding 32 iterations. Thus, at each checkpoint, we evaluate reusing actions generated 1, 5, and 20 iterations earlier to assess agreement between the ELBO-based estimates and the ODE-based reference across different replay ages. For each action, we estimate the ratio of its density under the current policy to that under the behavior policy that generated it, using both our estimator and probability flow ODE.

The discrepancy between the ELBO-based estimates and the probability flow ODE reference remains moderate throughout training. For Cournot, it becomes negligible at iteration 600, with a mean absolute difference of 0.0052. For symmetric Blotto, the mean absolute difference also decreases significantly, from 0.3527 at iteration 1000 to 0.1022 at iteration 2000. These observations suggest improved agreement with the ODE-based density-ratio reference as training approaches convergence.

Since our ELBO-based method is empirically over 500×500\times faster than Probability flow ODE. These results together with the empirical effectiveness of our ELBO-based IS correction support its use in practical training.

DomainBehavior iterationCurrent iterationELBO based resultPF-ODE based result
Cournot991001.02941.0671
951001.16290.9589
801001.12610.8971
2993000.99660.9961
2953000.99521.0012
2803000.96711.0144
5996001.00000.9951
5956000.99980.9932
5806000.99890.9947
Blotto Symmetric991001.24170.9821
951001.51081.5197
801002.07592.1437
99910001.21931.5209
99510001.04150.9200
98010001.21621.8512
199920001.02320.9176
199520001.09860.9386
198020001.08961.0485

Reference: [1] Song, Yang, et al. Score-based generative modeling through stochastic differential equations. ICLR.

Reviewer_k9CeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiffNash, a method for computing equilibria in continuous-action games using diffusion policies. It implements Online Mirror Descent (OMD) updates through reweighted score matching and reuses past actions from a replay buffer. To correct the distribution shift between past and current policies, it approximates importance weights by exponentiating an estimate of the ELBO difference. The method is evaluated on the forest security game and continuous Colonel Blotto game.

Critical strengths
  1. Reusing past actions with importance weighting is a reasonable approach. I find the low-cost estimation of ELBO differences for approximating importance weights particularly interesting.

  2. The paper derives a loss for implementing the policy update and proves that the ELBO-difference estimator is unbiased.

Critical weaknesses
  1. The theoretical results do not establish guarantees for equilibrium computation. The results concern the policy-update loss and the ELBO-difference estimator. They do not establish whether DiffNash converges to a Nash equilibrium or how accurately it approximates one. I do not require a convergence proof for the entire method, but the scope of the theoretical results is narrower than the problem the paper aims to solve.

  2. The ELBO-based importance weights need further validation. An unbiased estimator of the ELBO difference does not necessarily give an accurate importance weight after exponentiation. The paper does not sufficiently examine how these affect the correction of distribution shift. The importance-sampling ablations are useful, but it is unclear whether they keep replay fixed and change only the weighting. A comparison using the same replay samples with and without importance weighting would help distinguish the benefit of replay from that of the proposed correction.

Questions for authors
  1. Is there theoretical analysis or experimental evidence that exponentiating the ELBO-difference estimate accurately approximates the true importance weight?

  2. How does ELBO-based importance weighting compare with reusing the same past samples without weighting?

4 comments
AuthorsOpenReview ↗

W1: Convergence Clarification/ W2/Q2: Ablation shows the necessity of Importance Sampling Correction

We thank the reviewer for the questions.

W1: Convergence Proof of DiffNash

The reviewer’s question about convergence has two aspects: (1) whether the solver Online Mirror Descent (OMD) converges to Nash, and (2) whether DiffNash’s diffusion policy accurately represents the policy updates prescribed by OMD.

OMD is a no-regret algorithm [1]. When both players play OMD in a continuous action two player zero-sum game, their empirical strategy converges to the set of Nash equilibrium asymptotically [2]. Therefore, if the player can parameterize the target update in OMD, convergence is guaranteed theoretically. This is introduced in Section 2.3 (Line 154-157). Empirically, OMD is also commonly used to compute Nash equilibrium, for example in mean-field game [3] and game-theoretical alignment [4].

DiffNash's policy loss is globally minimized by the score function of the noise-corrupted OMD target update at each time t, which is all we need to parameterize and sample from the target update in OMD using diffusion. Hence, when trained perfectly, DiffNash inherits the theoretical convergence guarantee from OMD. In practice, with finite network capacity and a finite number of gradient descent iterations, optimization may approach the global minimum without attaining it exactly. We acknowledge that this assumption of recovering global minimizer is idealized in the updated paper (Line 801-807). However, this limitation is shared by all neural networks based methods (including all our baselines) and is therefore not an unique disadvantage to our method.

W2/Q2: Reuse Past Sample without Importance Weighting

We perform the ablation for the two settings presented in Figure 4 (forest security game). We show the necessity of adding importance sampling (IS) correction. The result is added to Line 475-488 of the updated paper.

SettingMethodNash Gap ↓ (mean ± 95% CI)
F2, m=n=3No Replay, No IS631.6 ± 150.8
With Replay, No IS625.6 ± 137.1
DiffNash (Ours)352.9 ± 132.8
F3, m=n=5No Replay, No IS866.8 ± 224.4
With Replay, No IS702.3 ± 355.6
DiffNash (Ours)556.6 ± 163.9

Replay alone increases sample diversity and stabilizes training, particularly in F3, where it reduces the mean Nash gap by 19.0% compared with the baseline Without Replay or IS. However, naively reusing past samples still adds a significant distribution shift to the data samples. By using our proposed ELBO-based IS correction, DiffNash further reduces the Nash gap relative to just using replay by 44.1% on F2 and 35.8% on F3. This result highlights the effectiveness of our ELBO-based importance sampling estimator in mitigating the distribution shift induced by sample reuse and demonstrating the necessity of such correction.

Reference:

[1] Krichene, Walid, et al. The hedge algorithm on a continuum. ICML.

[2] Balandat, Maximilian, et al. Minimizing regret on reflexive Banach spaces and Nash equilibria in continuous zero-sum games. NeurIPS.

[3] Perolat, Julien, et al. Scaling up mean field games with online mirror descent. AAMAS.

[4] Munos, Rémi, et al. Nash learning from human feedback. ICML.

AuthorsOpenReview ↗

W2/Q1: Our ELBO-based Estimator Remains Robust After Taking Exponentiation

W2/Q1: How does taking exponentiation affect our ELBO-based estimator?

Let k be the current policy iteration, k' the iteration where the action is sampled. Recall that Et,ϵ[Δ^]=ELBO⁡k(a0)−ELBO⁡k′(a0)\mathbb{E}_{t,\epsilon}[\widehat{\Delta}] = \operatorname{ELBO}^{k}(\mathbf{a}_0) - \operatorname{ELBO}^{k'}(\mathbf{a}_0). By Jensen's inequality, E[eΔ^]≥eE[Δ^]\mathbb{E}[e^{\widehat{\Delta}}] \geq e^{\mathbb{E}[\widehat{\Delta}]}, where the expectation is over t,ϵ\epsilon. Thus, exponentiating our estimator introduces an upward bias relative to the exponentiated ELBO difference. We make this point explicit in Line 302-304, 516-522, and we further justify the robustness of our ELBO-based estimator in Appendix F.8.

Beyond the empirical effectiveness of our ELBO-based estimator demonstrated in our response to W2/Q2, our estimator remains robust because: 1. the upward bias can be bounded under mild assumption, 2. clipping prevents excessive upward bias, 3. our estimator greatly outperforms an unbiased estimator of the exponentiated ELBO difference empirically.

Point 1: Bounded Upward Bias

This bias can be characterized and bounded. Fix an action a. If the estimation error ϵ=Δ^(a)−E[Δ^(a)]\epsilon=\widehat{\Delta}(a)-\mathbb{E}[\widehat{\Delta}(a)] is σ2\sigma^2-sub-Gaussian, then the upward bias introduced by exponentiation satisfies

0≤E[eΔ^(a)]−eE[Δ^(a)]≤eE[Δ^(a)](eσ2/2−1).0 \leq \mathbb{E}[e^{\widehat{\Delta}(a)}]-e^{\mathbb{E}[\widehat{\Delta}(a)]} \leq e^{\mathbb{E}[\widehat{\Delta}(a)]}(e^{\sigma^2/2}-1).

See Proposition 5 in Appendix F.8 for the proof. This result shows although exponentiation introduces upward bias, its magnitude can be bounded under mild assumptions on the estimation error.

Point 2: Clipping Mechanism

We clip importance weights to limit extreme values and stabilize training, a common practice also applied in PPO objectives [5]. On Cournot (d=30), only 6.42% of exponentiated ELBO estimates reach the clipping threshold over 600 iterations, dropping to 1.9% during the more stable iterations 101–500, suggesting that the upward bias does not cause frequent extreme importance weights.

Point 3: Unbiased Estimator is Possible, but Performs Much Worse

An unbiased estimator of the exponentiated ELBO difference can be constructed as follows. Let M∼Poisson(λ)M\sim\mathrm{Poisson}(\lambda) with λ>0\lambda>0, independently of i.i.d. unbiased estimates Δ^k,k′(a;j)\widehat{\Delta}^{k,k'}(a;j). We define w^k,k′(a)\widehat{w}^{k,k'}(a) by

w^k,k′(a)=∏j=1M(1+Δ^k,k′(a;j)λ)\widehat{w}^{k,k'}(a)=\overset{M}{\underset{j=1}{\prod}}\left(1+\frac{\widehat{\Delta}^{k,k'}(a;j)}{\lambda}\right),

with the empty product equal to 1. We provide the unbiasedness proof in Appendix F.8. The estimator requires E[M]= λ\lambda score-network calls on average. This unbiased estimator suffers high variance and requires a random number of samples. We fix other components and replace our ELBO-based estimator in DiffNash to the unbiased one, and run it on Cournot (d=30) for 600 iterations.

MethodDistance to Nash ↓
DiffNash (Ours)2.827 ± 0.051
Unbiased (λ=1)11.147 ± 0.935
Unbiased (λ=5)9.410 ± 0.483
Unbiased (λ=10)7.898 ± 0.608
Unbiased (λ=20)6.936 ± 0.363

DiffNash requires only one score network call per sample reuse, whereas the unbiased estimator requires λ\lambda calls on average. Our ELBO-based estimator achieves a substantially closer approximation to Nash equilibrium, even when the unbiased estimator uses 20 times as many network calls on average.

Reference: [5] Schulman, John, et al. Proximal policy optimization algorithms.

I thank the authors for the additional experiments and analyses. In particular, the comparison with reusing past samples without correction addresses my request and alleviates my concern about the effectiveness of the proposed importance weighting. I also appreciate the additional analysis of the bias introduced by exponentiation.

However, I believe that the theoretical support remains limited. The added bias analysis addresses the error introduced by exponentiating an estimate of the ELBO difference, but it does not address the discrepancy between the ELBO difference and the true log-density ratio. Therefore, the accuracy of the approximation to the true importance weight remains unclear. Moreover, the explanation that DiffNash inherits OMD’s convergence guarantees when trained perfectly does not, by itself, provide theoretical support for how diffusion policies overcome the difficulties conventional neural policies face in representing and learning the required strategy distributions.

In light of these points, I suggest softening statements in the introduction such as “provably recovering the correct target update” to clarify the conditions and scope of the theoretical guarantees.

Nevertheless, I believe it is appropriate to evaluate this paper primarily on its empirical merits. These theoretical limitations alone should not outweigh its contribution. I therefore raise my rating in recognition of the additional experimental evidence addressing my initial concerns.

I should note that I have limited expertise in judging whether the experimental settings and evaluation methods meet the standards of this field. Subject to confirmation from reviewers with relevant expertise that there are no major issues with these aspects, I would be willing to support this paper.

AuthorsOpenReview ↗

Thank you for Acknowledging our Empirical Strength / Theoretical Limitation of Projecting Gaussian Noise

We thank the reviewer for recognizing our method’s empirical strengths and further clarifying their concern. We appreciate their suggestion and will make the assumptions and conditions explicit to soften our theoretical claim. We include a brief theoretical discussion about why projecting Gaussian noise is worse at modeling multimodal distribution compared to diffusion.

Convergence of DiffNash

We answer both aspects: (1) why prior methods (projecting latent Gaussian noise) fail at capturing multimodal distributions, (2) why the diffusion model can capture multimodal distribution.

Why Projecting Latent Gaussian Noise Fails at Modeling Multimodal Distribution

[1] theoretically answers why simple pushforward network (projecting Gaussian noise) fails to model multimodal distributions theoretically and empirically. We include a concise theoretical discussion here.

Let ν\nu be the target distribution on Rd\mathbb{R}^d. We aim to learn a generator g:Rp→Rdg:\mathbb{R}^p\to\mathbb{R}^d that transforms Gaussian noise Z∼N(0,Ip)Z\sim\mathcal N(0,I_p) into samples from ν\nu. Assume gg is LL-Lipschitz, and let φ\varphi and Φ\Phi denote the standard Gaussian density and CDF. For a Borel set A⊆RdA\subseteq\mathbb{R}^d, define

Aε:={x∈Rd:dist(x,A)≤ε}A_\varepsilon := \lbrace x \in \mathbb{R}^d : \mathrm{dist}(x,A) \leq \varepsilon \rbrace

ν+(∂A):=lim inf⁡ε→0+ν(Aε)−ν(A)ε\nu^+(\partial A) := \liminf_{\varepsilon \to 0^+} \frac{\nu(A_\varepsilon)-\nu(A)}{\varepsilon} Here, ν+(∂A)\nu^+(\partial A) measures how quickly probability mass increases when AA expands outward. [1]'s Theorem 1 proves the following statement:

Theorem 1 (boundary inequality). If g(Z)∼νg(Z)\sim\nu, then for every Borel set AA with 0<ν(A)<10<\nu(A)<1 and 0<ν+(∂A)<∞0<\nu^+(\partial A)<\infty,

L\geq \frac{\varphi\left(\Phi^{-1}(\nu(A))\right)} {\nu^+(\partial A)}. }$$ When $\nu$ has well-separated modes, we can choose $A$ to contain one or more modes, with its boundary passing through the low-density region between them. This makes $\nu^+(\partial A)$ small. Provided $\nu(A)$ stays away from zero and one, the numerator remains bounded away from zero, requiring a large $L$. Thus, modeling multimodal distribution A requires a generator with a large Lipschitz constant. However, Lipschitz constraints are commonly used to stabilize generative-model training, while large Lipschitz constants can lead to training instability. Instead of learning a one-step push-forward map, a diffusion model learns a time-dependent score function that gradually transforms to data. Therefore, the above limitation does not apply to diffusion model learning. Empirically, Diffusion is known as better at modeling multimodal distribution compared to pushforward generative models, in settings like image modeling [1] and RL policy modeling [2]. ### **Why Diffusion Can Model The Target Distribution** We present the SDE view of diffusion model [3]. In diffusion model, the forward diffusion $$\mathrm dx=f(x,t)\mathrm dt+g(t)\mathrm dw$$ progressively adds noise to the data distribution $p_0$. With an appropriate noise schedule and sufficiently large $T$, $p_T$ approaches a Gaussian distribution. The reverse-time SDE $$\mathrm dx= \left[f(x,t)-g(t)^2\nabla_x\log p_t(x)\right]\mathrm dt +g(t)\mathrm d\bar w$$ is integrated from $T$ to $0$. With the exact score and initialization from $p_T$, it recovers $p_0$. For DDPM, the corresponding continuous-time coefficients are $$f(x,t)=-\tfrac12\beta(t)x, \qquad g(t)=\sqrt{\beta(t)}.$$ DDPM's forward process discretizes this SDE, while its denoising sampler approximates the reverse SDE using a learned score. In DiffNash, $p_0$ is the OMD target, whose noise-corrupted scores $\nabla_x\log p_t(x)$ minimize our reweighted score-matching objective. With exact score learning and initialization from $p_T$, exact reverse-time sampling recovers this target; finite-step DDPM sampling approximates it. **Finding the global minimum is an ideal assumption, and we have soften our theoretical claim in Line 65-67.** ## **Comparison Between Our Estimator and True Density Ratio** We compare our ELBO-based density-ratio estimates with a probability flow ODE reference [3], finding moderate discrepancies that further decrease in later training stages. This experiment is added to Appendix F.11; see our latest response to Reviewer 6T6H for details. Reference: [1] Salmona, Antoine, et al. Can push-forward generative models fit multimodal distributions?. NeurIPS 2022. [2] Wang, Zhendong, et al. Diffusion policies as an expressive policy class for offline reinforcement learning. ICLR 2023. [3] Song, Yang, et al. Score-based generative modeling through stochastic differential equations. ICLR.
Reviewer_bSDNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the problem of computing equilibria in high-dimensional continuous games and proposes DiffNash, a diffusion-based framework for equilibrium computation. The authors derive a reweighted score-matching objective that trains the diffusion model to realize Online Mirror Descent (OMD) updates. To improve sample efficiency and training stability, DiffNash reuses past actions through a replay buffer and approximately corrects the resulting distribution shift with ELBO-based importance-sampling strategy. Experimental results show that DiffNash achieves lower mean estimated Nash gaps than the compared baselines in the forest security and Colonel Blotto games.

Critical strengths
  1. The problem addressed in the paper is of practical significance.
Critical weaknesses
  1. The limitations of existing methods are not sufficiently discussed. The core components substantially overlap with prior work, and the technical novelty of the proposed method is not clearly articulated.

  2. The paper emphasizes equilibrium computation in high-dimensional continuous action spaces. Although the forest security and continuous Colonel Blotto experiments demonstrate the ability to learn multimodal mixed strategies, the action dimensionality per player in these tasks is at most 10. The highest-dimensional experiment, the 30-dimensional Cournot game, involves independent markets and has a unique pure-strategy equilibrium. Therefore, the evidence for learning multimodal mixed strategies in higher dimensions remains limited.

Questions for authors
  1. In Figure 4, some DiffNash guard placements appear distant from tree clusters, and their role in intercepting attackers is unclear. Could the authors clarify what constitutes an effective guard deployment and provide further interpretation of the results in Figure 4?

  2. For the continuous Colonel Blotto experiments in Table 2, all methods are trained for 40K gradient updates per player, but action-generation costs differ across methods. Could the authors additionally report Nash gap versus wall-clock time, including the costs of action generation and replay correction?

  3. Could the authors provide additional experiments at higher dimensions to further evaluate DiffNash in games requiring multimodal mixed strategies?

3 comments
AuthorsOpenReview ↗

Response to W1, W2, Q1, Q3

We thank the reviewers for the insightful questions.

W1: Limitation of Existing Methods / Our Novelty

With the additional page, we update the related work (section 4) to better emphasize the difference between our method and prior work. The details of all baselines are discussed between Line 360-370.

In prior equilibrium computation literature, neural network policies (DeepFP/Neural OMD) project latent Gaussian noise to player action. Such a design is known to have mode collapse in high dimension space [1]. DiffNash instead uses diffusion policies to model richer strategy distributions. While it shares the goal of neural policy representation with existing methods, its architecture and training formulation differ substantially. To our knowledge, the only prior diffusion-based method in our setting is DiffFP, which we include as a baseline. Since sampling from the target Fictitious Play (FP) update is intractable, DiffFP uses gradients to guide samples toward high utility regions and trains diffusion on these samples. This heuristics struggles when gradient is uninformative, which explains its inferior average performance in forest security and Blotto games. In contrast, DiffNash uses the reweighted score matching loss that principally recovers the target update in online mirror descent (OMD).

The reweighting technique we use to parameterize OMD update exists in diffusion RL, but it’s non-trivial to adapt it to our setting. In equilibrium computation, the player’s environment is non-stationary, i.e., each player’s policy will affect the transition dynamics of the opponent player. We need to consider such non-stationarity when designing the loss function (Proposition 1).

W2&Q3: Higher Dimension

We extend our evaluation to 20-dimensional action space for both forest-security game and Colonel Blotto game. DiffNash reduces the mean Nash gap by 35.3% on radial Blotto and 50.9% on forest F2 relative to the strongest baseline in each setting.

TypeAction DimensionDeepFPNeural OMDDiffFPOurs
Forest F2m=n=10 (dim: 20)636 ± 51.1697.9 ± 244.3701.9 ± 86.7312.5 ± 118.2
Blotto Radial200.700 ± 0.0011.070 ± 0.6722.877 ± 0.0290.453 ± 0.068

We emphasize that 20 dimensions represents a meaningful scale for continuous-action policy learning, particularly in the context of existing benchmarks.

First, we evaluate much higher-dimensional action spaces than prior work in equilibrium computation. DeepFP, a key baseline, considers at most action dimension 6.

Second, diffusion policies are widely used in RL with high-dimensional action spaces and commonly evaluated on MuJoCo [3], a standard RL benchmark. [2] evaluates its diffusion policy on Mujoco’s Humanoid task, whose action dimensionality is 17, comparable to our 20-dimensional setting.

Q1: what constitutes effective guard deployment / Further interpret Figure 4?

Ideally, a guard should be positioned within a tree cluster; the second best option is between the tree cluster and the nearest point on the forest’s periphery, where the guard can intercept attackers who approaches from the periphery. DiffNash achieves much better coverage compared to all baselines across settings. In F3, we cover all tree clusters. In F2, we miss the upper tree cluster. However, if we increase the gradient budget from 40k to 60k, we observe DiffNash having higher coverage in F2’s upper tree cluster. We show the updated plot in Appendix F.10, showing longer training improves convergence.

AuthorsOpenReview ↗

Response to Q2: DiffNash's performance remains robust when evaluated against wall-clock training time.

Q2: Performance v.s. Wall clock time

TLDR: Given an hour of training time, DiffNash outperforms the strongest baseline in two Blotto settings by 17.3%. Compared to generating fresh samples, our replay mechanism cuts training time by 45.7% while also significantly reducing the Nash gap by increasing sample diversity.

Performance Against Wall clock time

We compare Nash gap versus wall-clock training time, reporting results for radial-band Blotto (n=6) here and the symmetric setting in Appendix F.7.1 (Table 6). Across both settings, DiffNash outperforms the strongest baseline by 17.3%. On radial-band Blotto, other baselines perform better initially, but DiffNash achieves the lowest mean Nash gap at 30 and 60 minutes, improving over the strongest baseline by 23.4% and 28.2%, respectively. With sufficient training time (>= 30mins), these results are consistent with those under a fixed budget of 40K gradient updates.

Nash gap v.s. Wall-Clock time

EquilibriumMethod1 min5 min30 min60 min40k gradient steps
Radial (n=6)DeepFP0.800 ± 0.1990.805 ± 0.2010.806 ± 0.2010.806 ± 0.2010.688 ± 0.001
Neural OMD0.944 ± 0.0170.565 ± 0.1970.599 ± 0.0350.645 ± 0.0190.836 ± 0.236
DiffFP1.513 ± 0.1701.073 ± 0.0241.051 ± 0.0461.043 ± 0.0331.038 ± 0.056
DiffNash2.027 ± 0.1140.941 ± 0.1020.459 ± 0.0860.463 ± 0.1360.501 ± 0.139

Wall-clock time For action generation

DiffNash generates actions more slowly but achieves a substantially lower Nash gap with sufficient training, justifying the added cost. Similar efficiency–performance tradeoffs are common in diffusion-based RL, where diffusion policies have demonstrated strong performance despite their higher computational costs [1]. This result is in Appendix F.7.2. Action generation may be further sped up by one step generation method such as MeanFlow [4], which we leave as future work.

MethodMean time per action
DiffNash0.187 ms
Neural OMD0.0317 ms

Replay Mechanism Reduces Run Time

We compare DiffNash against a variant that always generates fresh samples (with no replay or IS), holding the number of gradient updates fixed at 40k. Replay reduces total training time by approximately 45.7% across the two Blotto settings, while also reducing the mean Nash gap by 49.6% on radial-band Blotto and 81.6% on symmetric Blotto. This result is in Appendix F.7.3.

GameMethodTraining wall-clock time (min)Final Nash gap ↓
Radial band, n = 6DiffNash (ours)63.9 ± 0.20.480 ± 0.152
Radial band, n = 6DiffNash (No replay, No IS)117.9 ± 0.20.953 ± 0.023
Symmetric, n = 6DiffNash (ours)64.1 ± 0.10.990 ± 0.145
Symmetric, n = 6DiffNash (No replay, No IS)117.7 ± 0.35.390 ± 1.014

Wall-clock time for IS correction

We quantify the impact of IS correction using the 30-dimensional Cournot game over 600 outer iterations. Replay reduces training time by approximately 43.8% compared with generating fresh samples, while adding IS correction increases wall-clock time by only 3.8% relative to unweighted replay. Hence, IS correction adds minimal overhead. This result is in Appendix F.7.4.

MethodTotal wall-clock time
No replay, no IS41.24 ± 0.21 min
Replay, no IS23.17 ± 0.20 min
DiffNash (Ours)24.04 ± 0.16 min

References:

[1] Wang et al.(2023). Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. ICLR.

[2] Ma et al. (2025). Efficient Online Reinforcement Learning for Diffusion Policy. ICML.

[3] Gymnasium. MuJoCo Humanoid Environment.

[4] Geng, Zhengyang, et al. NeurIPS 2026.

AuthorsOpenReview ↗

Please let us know if our rebuttal has addressed your questions. Thank you very much for your time!

Dear reviewer bSDN,

Please let us know if our rebuttal has addressed your questions. Thank you very much for your time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Follow-up

Dear reviewers,

Thank you for taking the time to review our paper and for recognizing that it addresses "a problem with practical significance", that our loss provides a "principled connection to equilibrium updates", that our low-cost ELBO difference estimation is "particularly interesting", and that our empirical results are "encouraging" and "informative".

We have made substantial efforts during rebuttal, including running the following major experiments to address your questions:

(1) Experiments on 20-dimensional action space (Reviewer bSDN)

(2) Performance Comparison w.r.t. wall clock time (Reviewer bSDN)

(3) Ablation on using replay mechanism without IS correction (Reviewer k9Ce and 6T6H)

(4) Analyzing how taking exponentiation, clipping, and self normalization may impact our ELBO-difference estimator. (Reviewer k9Ce and 6T6H)

(5) Robustness against Search Budget in Nash Gap computation (Reviewer 6T6H)

We have also clarified the connections to existing work, the interpretation of Figure 4 (Reviewer bSDN), and the convergence results (Reviewer k9Ce). We already updated our rebuttal responses and incorporated your feedback into the revised draft. Please let us know if you have any remaining concerns so we can address them before the Wednesday deadline. Thank you again for your time and insightful comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.