- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DiffNash, a framework for computing approximate equilibria in continuousaction games using diffusion policies to represent mixed strategies. It derives reweighted denoising objectives for online mirror descent (OMD) and a fictitious-play variant, and introduces replay of opponent and self-actions. Replay samples are weighted using an exponentiated stochastic estimate of the difference between current and behavior-policy evidence lower bounds (ELBOs). Experiments on Cournot, forest protection, and continuous Colonel Blotto games provide evidence that the framework can improve equilibrium-learning performance.
Critical strengths
-
A useful representation for mixed strategies. Diffusion policies provide an expressive representation of multimodal strategies in continuous-action games. The experiments, particularly the Blotto examples, illustrate why policy expressiveness can matter for equilibrium computation.
-
A principled connection to equilibrium updates. Proposition 1 connects the idealized OMD update to a reweighted denoising objective. This provides a meaningful foundation for the framework, independently of whether the practical replay correction is exact.
-
Encouraging empirical evidence and informative ablations. The paper evaluates several game configurations, reports uncertainty across seeds, and includes replay-component ablations and hyperparameter studies. Section 5.3 and Appendix F.4 distinguish the optimization benefit of self-action replay from the computational benefit of opponent replay. Improvements are not uniform: DiffFP has lower mean Nash gaps in the symmetric Blotto settings in Table 2. Nevertheless, the overall results support the practical potential of the approach.
Critical weaknesses
-
The replay-weight proxy is insufficiently validated. Proposition 2 does not establish the accuracy of the exponentiated, clipped, self-normalized weights. A tractable density-ratio diagnostic would help identify when the correction is reliable.
-
The benefit of weighting is not fully isolated from the benefit of replay. The existing component ablations are useful, but a matched comparison of corrected and unweighted replay would better distinguish distribution-shift correction from the effect of historical sample diversity.
-
Equilibrium-quality estimates depend on approximate best responses. Appendix D.1 describes the evaluation procedures, but the resulting Nash-gap estimates are not certified upper bounds on exploitability. Search-budget sensitivity checks would strengthen confidence in the reported rankings.
Questions for authors
- Can you quantify/qualify the unbiasedness of the final estimator?
- On a tractable diagnostic problem, how closely do the proposed weights match exact or accurately estimated density ratios?
- Are the main performance rankings stable under increased best-response evaluation budgets?
4 comments
Q1: Robustness of the final estimator
W2/Q1: quantify/qualify the unbiasedness of the final estimator
Let k be the current policy iteration, k' the iteration where the action is sampled. Recall is our unbiased estimator of the ELBO difference. After taking exponentiation, an upward bias is introduced. Formally, by Jensen's inequality, , where the expectation is over t,. We make this point explicit in Line 302-304, 516-522, and we further justify the robustness of our ELBO-based estimator in Appendix F.8.
Despite its upward bias, we quantify the robustness of our exponentiated ELBO estimator by showing: 1. the upward bias can be bounded under mild assumption, 2. clipping prevents excessive upward bias, 3. our estimator greatly outperforms an unbiased estimator of the exponentiated ELBO difference empirically, 4. its empirical effectiveness.
Point 1: Bounded Upward Bias
This bias can be characterized and bounded. Fix an action a. If the estimation error is -sub-Gaussian, then the upward bias introduced by exponentiation satisfies
See Proposition 5 in Appendix F.8 for the proof. This result shows although exponentiation introduces upward bias, its magnitude can be bounded under mild assumptions on the estimation error.
Point 2: Clipping Mechanism
As discussed in our response to W2/Q1, clipping prevents the upward bias from hurting training stability. Moreover, only 1.9% the exponentiated ELBO difference reach the clipping threshold during after the first 100 iterations on Cournot (d=30), suggesting that the upward bias does not cause frequent extreme importance weights.
Point 3: Unbiased Estimator is Possible, but Performs Much Worse
An unbiased estimator of the exponentiated ELBO difference can be constructed as follows. Let with , independently of i.i.d. unbiased estimates . We define by
,
with the empty product equal to 1. We provide the unbiasedness proof in Appendix F.8. The estimator requires E[M]= score-network calls on average. This unbiased estimator suffers high variance and requires a random number of samples. We fix other components and replace our ELBO-based estimator in DiffNash to the unbiased one, and run it on Cournot (d=30) for 600 iterations.
| Method | Distance to Nash ↓ |
|---|---|
| DiffNash (Ours) | 2.827 ± 0.051 |
| Unbiased (λ=1) | 11.147 ± 0.935 |
| Unbiased (λ=5) | 9.410 ± 0.483 |
| Unbiased (λ=10) | 7.898 ± 0.608 |
| Unbiased (λ=20) | 6.936 ± 0.363 |
DiffNash requires only one score network call per sample reuse, whereas the unbiased estimator requires calls on average. Our ELBO-based estimator achieves a substantially closer approximation to Nash equilibrium, even when the unbiased estimator uses 20 times as many network calls on average.
Point 4: Empirical Effectiveness
Our prior response to W2 shows that using our exponentiated ELBO-based estimator as the IS weight reduces the mean Nash gap by 43.6% on F2 and 20.7% on F3 compared with unweighted replay, demonstrating its empirical effectiveness.
Thank you for the clarifications and additional experiments. The weighted versus unweighted replay ablation and the best-response budget sensitivity checks substantially address two of my concerns. The main remaining limitation is that the clipping and normalization ablations do not directly validate the weights against actual density ratios. Likewise, controlling bias relative to the exponentiated ELBO difference does not establish accuracy relative to the true density ratio. Please make this distinction explicit and qualify the claim that importance correction is “necessary.” The rebuttal strengthens the empirical case, and I maintain my weak acceptance recommendation.
Comparison Between Our ELBO-based Estimator and True Density Ratio
We thank the reviewer for the constructive feedback. We add further comparison between Our ELBO-based Estimator and True Density Ratio here. The new experiment is added to Appendix F.11. We hope the reviewer could take the new result into consideration.
Probability Flow ODE [1] enables computing true probability density for a diffusion policy. We therefore compare our ELBO-based density-ratio estimates with this ODE-based reference to assess the robustness of our estimator.
Evaluating density ratio under probability Flow ODE is 586x times slower than our lightweight ELBO-based method. Since using Probability Flow ODE to compute true density ratio throughout the training run is too expensive, we evaluate selected checkpoints from the beginning, middle, and end of training in Cournot () and Blotto Symmetric (). During training, our replay buffer retains samples from the preceding 32 iterations. Thus, at each checkpoint, we evaluate reusing actions generated 1, 5, and 20 iterations earlier to assess agreement between the ELBO-based estimates and the ODE-based reference across different replay ages. For each action, we estimate the ratio of its density under the current policy to that under the behavior policy that generated it, using both our estimator and probability flow ODE.
The discrepancy between the ELBO-based estimates and the probability flow ODE reference remains moderate throughout training. For Cournot, it becomes negligible at iteration 600, with a mean absolute difference of 0.0052. For symmetric Blotto, the mean absolute difference also decreases significantly, from 0.3527 at iteration 1000 to 0.1022 at iteration 2000. These observations suggest improved agreement with the ODE-based density-ratio reference as training approaches convergence.
Since our ELBO-based method is empirically over faster than Probability flow ODE. These results together with the empirical effectiveness of our ELBO-based IS correction support its use in practical training.
| Domain | Behavior iteration | Current iteration | ELBO based result | PF-ODE based result |
|---|---|---|---|---|
| Cournot | 99 | 100 | 1.0294 | 1.0671 |
| 95 | 100 | 1.1629 | 0.9589 | |
| 80 | 100 | 1.1261 | 0.8971 | |
| 299 | 300 | 0.9966 | 0.9961 | |
| 295 | 300 | 0.9952 | 1.0012 | |
| 280 | 300 | 0.9671 | 1.0144 | |
| 599 | 600 | 1.0000 | 0.9951 | |
| 595 | 600 | 0.9998 | 0.9932 | |
| 580 | 600 | 0.9989 | 0.9947 | |
| Blotto Symmetric | 99 | 100 | 1.2417 | 0.9821 |
| 95 | 100 | 1.5108 | 1.5197 | |
| 80 | 100 | 2.0759 | 2.1437 | |
| 999 | 1000 | 1.2193 | 1.5209 | |
| 995 | 1000 | 1.0415 | 0.9200 | |
| 980 | 1000 | 1.2162 | 1.8512 | |
| 1999 | 2000 | 1.0232 | 0.9176 | |
| 1995 | 2000 | 1.0986 | 0.9386 | |
| 1980 | 2000 | 1.0896 | 1.0485 |
Reference: [1] Song, Yang, et al. Score-based generative modeling through stochastic differential equations. ICLR.
W2: IS correction is Necessary / W1&Q2: Ablation on Clipping and Normalization / W3: Result is Robust to Different Search Budgets
We thank the reviewer for the qusestions.
W2 : IS weight correction is Necessary
To isolate the contribution of importance weighting from the benefit of replay itself, we conduct an additional ablation on the two forest-security settings shown in Figure 4. We compare DiffNash against variants without replay and with unweighted replay. The result is added to Line 475-488 of the updated paper.
Replay alone increases sample diversity and stabilizes training, particularly in F3, where it reduces the mean Nash gap by 19.0% compared with the baseline Without Replay or IS. However, unweighted replay does not account for distribution shifts between historical and current policies. By using our proposed ELBO-based IS correction, DiffNash further reduces the mean Nash gap by 43.6% on F2 and 20.7% on F3 compared with unweighted replay. This result highlights the effectiveness of our ELBO-based importance sampling estimator in mitigating the distribution shift induced by sample reuse and demonstrating the necessity of such correction.
W1/Q2: Accuracy of the clipped, self-normalized weights. Tractable density-ratio diagnostic.
One possible alternative for approximating the density ratio is using probability flow ODE [1], which maps an action to the noise prior and integrates the flow’s divergence to estimate its log density. However, computing the log-density difference for a batch of 128 opponent samples takes 10.8 ms with our ELBO-based estimator, compared with 6.33s using the probability flow ODE. This 586x slowdown makes using the probability flow ODE throughout the training too costly. In practice, we use our ELBO-based density estimator to perform diagnostics and confirm the effects of clipping and self-normalization. We discuss the effect of exponentiation later with our response to Q1.
Without clipping, exponentiating our ELBO-based estimator to approximate importance weights can occasionally cause numerical overflow and destabilize training. Hence, we clip importance weights to limit extreme values and stabilize training, a common practice also applied in PPO objectives [2]. On Cournot (d=30), 6.42% of exponentiated ELBO estimates reach the clipping threshold over 600 iterations, dropping to only 1.9% during the more stable iterations 101–500.
Self Normalization is also common in policy training. For example, PPO performs advantage normalization [2]. We run our method without self normalization on Cournot (d=30), and show self normalization significantly reduces the distance to Nash.
Ablation on Clipping/Self-normalization is added to Appendix F.6.
W3/Q3: Search Budget
To evaluate the robustness of our Nash-gap estimates, we conduct an additional sensitivity analysis by varying the evolutionary best-response search budget from 25 to 100 candidates, including the original budget of 50, on the F2 and F3 forest-security games. This robustness check is added to Appendix F.9.
Performance under varying search budget
DiffNash consistently achieves the lowest mean estimated Nash gap across all search budgets in both settings, showing our result is robust to variations in search budget size.
Reference:
[1] Song, Yang, et al. Score-based generative modeling through stochastic differential equations. ICLR.
[2] Schulman, John, et al. Proximal policy optimization algorithms.