- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the combined effects of environmental payoff noise and mutation on evolutionary learning dynamics in multi-agent systems. This paper is directly built on previous references, Wang et al (2026) and Bauer et al. (2026), and the authors formulate a noisy replicator–mutator dynamics (NRMD) model. They analyze equilibrium existence and stability in two-strategy games and examine the effects of noise and mutation on cooperation and oscillatory dynamics. The paper also proposes a population-based learning algorithm, Population-Replicator Policy Gradient (PRPG).
Critical strengths
The paper attempts to connect theoretical analysis with numerical and learning experiments across several settings.
Critical weaknesses
My main concern is about theoretical novelty of this work. Replicator-mutator dynamics have been extensively studied in evolutionary game theory, and the effects of stochastic perturbations on evolutionary dynamics have also received a lot attention. This paper reads like just a simple combination of noise and mutation, which have already been studied in the two papers explicitly cited by the authors. The results derived from the considered two-strategy case seems also not scalable to multiple strategy learning.
- The proposed NRMD model essentially combines these existing mechanisms in a single dynamical equation. The motivation for this particular combination is not clearly justified.
- The paper emphasizes the effects of noise and mutation on equilibria, limit cycles, and transitions between different dynamical regimes. However, the existence of stable limit cycles, interior equilibria, and multistability in evolutionary game dynamics is not new by itself.
- The paper is framed as a contribution to multi-agent learning, but the relationship between the proposed NRMD model and general multi-agent learning algorithms remains unclear.
- Some technical results also require clarification or correction. For example, in Section 3.2, the stochastic logit dynamics in Eq. (8) appear inconsistent with the Itô correction in Eq. (9). If both the drift and diffusion coefficients are multiplied by the learning-rate parameter (\alpha), the noise-induced Itô drift should scale quadratically with (\alpha).
Questions for authors
-
What is the fundamental theoretical novelty of NRMD? Except for combining the noise term and the mutation term, which results are really new and cannot be explained by existing replicator-mutator or stochastic evolutionary game theory?
-
What new dynamical phenomenon is established in Section 4.1? How do the authors distinguish the reported coexistence of limit cycles and equilibria from known behavior in evolutionary dynamical systems? Can they provide a more rigorous bifurcation analysis supporting the claimed subcritical transition?
-
How is the proposed NRMD model formally connected to multi-agent reinforcement learning? Can the authors derive the continuous-time dynamics from the actual update rules of PRPG?
-
In PRPG, What is the specific policy-gradient component, and what advantage does it provide over traditional evolutionary learning approaches?
2 comments
I thank the authors for these responses, which help me better to understand this work.
Response to Reviewer gJ25
We thank Reviewer gJ25 for the technical points, in particular the one on the Itô coefficient, which led us to find a notational ambiguity and an error in §4.1.
W1 — is this just a combination? The equation is a sum of two known terms; we will say so in one sentence. What is not in either parent:
W2 — §4.1. Your request made us redo the analysis, and it changes our conclusion.
At the symmetric point πi is equal for all i, so πi−πˉ=0 and the noise term 21[(σi−σˉ)2−Var(σ)] is quadratic in the perturbation: its gradient vanishes. The Jacobian there is the classical replicator one and is independent of k — numerically Reλ=0.166667 for k=0,0.5,1,1.5,2,3. Mutation shifts the spectrum by exactly −M, so Reλ(k,M)=6α−β−M, and the cycle terminates at MH=(α−β)/6, independent of the noise scale (MH=1/6 at α,β=2,1). The amplitude vanishes continuously: $0.418,,0.256,, 0.168,,0.093,,0.063$ at MH−M=0.0667,…,0.0017, i.e. 1.54(MH−M)1/2 with log–log slope 0.50 over 1.5 decades — a supercritical Hopf bifurcation.
This corrects our earlier reading: the coexistence window came from integrating the 3-D field without projecting onto the simplex; in simplex coordinates the 60-IC scan gives a single attractor at every M. We will replace Fig. 2 (right) and the §4.1 paragraph. The corrected statement is also the better answer to your question: the interaction puts a phase boundary at a location independent of k, which neither parent can express.
W3 — connection to MARL. softmax policy → score-function update → diffusion (8) → Itô (9) → drift (3) → mutation from the entropy bonus (Proposition 3) → discrete update (17), whose mean-field limit is (3) → Algorithm 1, where reproduction replaces the per-agent gradient. Proposition 6 closes the loop. Caveats we will state: this is a mean-field limit (small step, large P), not an identity at finite P, and Algorithm 1 is not a gradient method.
W4 — the policy-gradient component. In the continuous form (17), Δza=η(1[a=ai]A^i−M(za−zˉ)): a score-function term with baseline, plus the entropy contraction. The advantage over a purely evolutionary scheme is that A^i keeps per-agent credit assignment, which a scalar population fitness cannot. In Algorithm 1 the gradient is replaced by fitness-proportional reproduction on the same parameter; we will say so in the caption.
W5 — Eq. (8) versus (9), and the α scaling. You are right that the correction must be quadratic in α, and it is: Eq. (9) contains αx(1−x)⋅α(21−x)(Δσ)2 — two factors of α; symbolically the correction has absolute degree 2 in α, against degree 1 for the drift. Two things obscured this:
k*xi/sqrt(s)). A reported k is therefore already the dimensionless k of Eq. (3). In §3.1 we will state this and give the raw-scale threshold kraw>2/s; no reported number changes.