acceptodds
Under review as a conference paper at ICLR 2027

Noisy Replicator-Mutator Dynamics in Multi-Agent Learning

Abstract

Evolutionary game theory predicts where a population of learning agents ends up, but the classical replicator equation is derived for a payoff matrix that does not change. Two recent lines of work relax it in different ways. Wang et al. (2026) show that global payoff noise adds a multiplicative term , with the intensity of the payoff fluctuation, to the replicator equation, stabilising coexistence and generating stable limit cycles; Bauer et al. (2026) show that an additive mutation term turns it into a replicator-mutator system whose interior equilibria are last-iterate attracting, which yields convergence guarantees for multi-agent reinforcement learning. The two perturbations have been analysed on their own; their interaction has not been studied. We derive the system that carries both, which we call Noisy Replicator-Mutator Dynamics (NRMD), from a stochastic policy-gradient update; the two terms act on different quantities, so they can be set independently. Our contributions are threefold: (i) we give a complete two-strategy theory of NRMD, consistent with its -strategy form at ; (ii) we validate the effect of noise in multi-agent experiments; and (iii) we propose a novel algorithm, Population-Replicator Policy Gradient (PRPG), that exploits NRMD. Code and data will be released.

Primary area:
reinforcement learning
Keywords:
multi-agent learning, evolutionary game theory, large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gJ25OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the combined effects of environmental payoff noise and mutation on evolutionary learning dynamics in multi-agent systems. This paper is directly built on previous references, Wang et al (2026) and Bauer et al. (2026), and the authors formulate a noisy replicator–mutator dynamics (NRMD) model. They analyze equilibrium existence and stability in two-strategy games and examine the effects of noise and mutation on cooperation and oscillatory dynamics. The paper also proposes a population-based learning algorithm, Population-Replicator Policy Gradient (PRPG).

Critical strengths

The paper attempts to connect theoretical analysis with numerical and learning experiments across several settings.

Critical weaknesses

My main concern is about theoretical novelty of this work. Replicator-mutator dynamics have been extensively studied in evolutionary game theory, and the effects of stochastic perturbations on evolutionary dynamics have also received a lot attention. This paper reads like just a simple combination of noise and mutation, which have already been studied in the two papers explicitly cited by the authors. The results derived from the considered two-strategy case seems also not scalable to multiple strategy learning.

  1. The proposed NRMD model essentially combines these existing mechanisms in a single dynamical equation. The motivation for this particular combination is not clearly justified.
  2. The paper emphasizes the effects of noise and mutation on equilibria, limit cycles, and transitions between different dynamical regimes. However, the existence of stable limit cycles, interior equilibria, and multistability in evolutionary game dynamics is not new by itself.
  3. The paper is framed as a contribution to multi-agent learning, but the relationship between the proposed NRMD model and general multi-agent learning algorithms remains unclear.
  4. Some technical results also require clarification or correction. For example, in Section 3.2, the stochastic logit dynamics in Eq. (8) appear inconsistent with the Itô correction in Eq. (9). If both the drift and diffusion coefficients are multiplied by the learning-rate parameter (\alpha), the noise-induced Itô drift should scale quadratically with (\alpha).
Questions for authors
  1. What is the fundamental theoretical novelty of NRMD? Except for combining the noise term and the mutation term, which results are really new and cannot be explained by existing replicator-mutator or stochastic evolutionary game theory?

  2. What new dynamical phenomenon is established in Section 4.1? How do the authors distinguish the reported coexistence of limit cycles and equilibria from known behavior in evolutionary dynamical systems? Can they provide a more rigorous bifurcation analysis supporting the claimed subcritical transition?

  3. How is the proposed NRMD model formally connected to multi-agent reinforcement learning? Can the authors derive the continuous-time dynamics from the actual update rules of PRPG?

  4. In PRPG, What is the specific policy-gradient component, and what advantage does it provide over traditional evolutionary learning approaches?

2 comments
AuthorsOpenReview ↗

Response to Reviewer gJ25

We thank Reviewer gJ25 for the technical points, in particular the one on the Itô coefficient, which led us to find a notational ambiguity and an error in §4.1.

W1 — is this just a combination? The equation is a sum of two known terms; we will say so in one sentence. What is not in either parent:

  • Proposition 2 (separation). Shared shocks give an O(1)O(1) term; independent shocks give O(1/N)O(1/N), i.e. nothing in the mean-field limit. This decides whether the mechanism is observable in MARL at all, and Wang et al. assume a common shock rather than deriving the condition.
  • Proposition 1 (first-order immunity). Updates affine in the sampled payoff cancel the shock exactly; only updates with the payoff in an exponential inherit it. §4.3's PPO control (clipping, GAE, critic; 16 cells, no effect) is therefore a prediction.
  • Proposition 3. An entropy bonus is a mutation bias with M(x)=ηβψ(x)M(x)=\eta\beta\psi(x) and 0<ψ≤140<\psi\le\tfrac14; Bauer et al. treat MM as free.
  • Proposition 5. Mcrit(k)=max⁡{0,s(k2/2−1)}M_{\mathrm{crit}}(k)=\max\{0,s(k^{2}/2-1)\} is the only quantity that mixes ss, kk and MM.
  • Corollary 1. The window x∗(k)>M/(2s)x^{*}(k)>M/(2s) explains contradictory reports: the attributable regime is k∈(2,∼2)k\in(\sqrt2,\sim2).
  • §4.4. The class boundary with matched policy sets: +0.410+0.410 versus +0.020+0.020.

W2 — §4.1. Your request made us redo the analysis, and it changes our conclusion.

At the symmetric point πi\pi_i is equal for all ii, so πi−πˉ=0\pi_i-\bar\pi=0 and the noise term 12[(σi−σˉ)2−Var(σ)]\tfrac12[(\sigma_i-\bar\sigma)^{2}-\mathrm{Var}(\sigma)] is quadratic in the perturbation: its gradient vanishes. The Jacobian there is the classical replicator one and is independent of kk — numerically Re λ=0.166667\mathrm{Re}\,\lambda=0.166667 for k=0,0.5,1,1.5,2,3k=0,0.5,1,1.5,2,3. Mutation shifts the spectrum by exactly −M-M, so Re λ(k,M)=α−β6−M,\mathrm{Re}\,\lambda(k,M)=\frac{\alpha-\beta}{6}-M , and the cycle terminates at MH=(α−β)/6M_H=(\alpha-\beta)/6, independent of the noise scale (MH=1/6M_H=1/6 at α,β=2,1\alpha,\beta=2,1). The amplitude vanishes continuously: $0.418,,0.256,, 0.168,,0.093,,0.063$ at MH−M=0.0667,…,0.0017M_H-M=0.0667,\dots,0.0017, i.e. 1.54 (MH−M)1/21.54\,(M_H-M)^{1/2} with log–log slope 0.500.50 over 1.5 decades — a supercritical Hopf bifurcation.

This corrects our earlier reading: the coexistence window came from integrating the 3-D field without projecting onto the simplex; in simplex coordinates the 60-IC scan gives a single attractor at every MM. We will replace Fig. 2 (right) and the §4.1 paragraph. The corrected statement is also the better answer to your question: the interaction puts a phase boundary at a location independent of kk, which neither parent can express.

W3 — connection to MARL. softmax policy → score-function update → diffusion (8) → Itô (9) → drift (3) → mutation from the entropy bonus (Proposition 3) → discrete update (17), whose mean-field limit is (3) → Algorithm 1, where reproduction replaces the per-agent gradient. Proposition 6 closes the loop. Caveats we will state: this is a mean-field limit (small step, large PP), not an identity at finite PP, and Algorithm 1 is not a gradient method.

W4 — the policy-gradient component. In the continuous form (17), Δza=η(1[a=ai]A^i−M(za−zˉ))\Delta z_a=\eta(\mathbf{1}[a=a_i]\hat A_i-M(z_a-\bar z)): a score-function term with baseline, plus the entropy contraction. The advantage over a purely evolutionary scheme is that A^i\hat A_i keeps per-agent credit assignment, which a scalar population fitness cannot. In Algorithm 1 the gradient is replaced by fitness-proportional reproduction on the same parameter; we will say so in the caption.

W5 — Eq. (8) versus (9), and the α\alpha scaling. You are right that the correction must be quadratic in α\alpha, and it is: Eq. (9) contains α x(1−x)⋅α(12−x)(Δσ)2\alpha\,x(1-x)\cdot\alpha(\tfrac12-x)(\Delta\sigma)^{2} — two factors of α\alpha; symbolically the correction has absolute degree 2 in α\alpha, against degree 1 for the drift. Two things obscured this:

  1. We call α2x2(1−x)2(Δσ)2\alpha^{2}x^{2}(1-x)^{2}(\Delta\sigma)^{2} the "diffusion coefficient"; it is its square. The coefficient is αx(1−x)Δσ=O(α)\alpha x(1-x)\Delta\sigma=O(\alpha).
  2. Δσ\Delta\sigma in Eq. (3) is the s\sqrt{s}-rescaled amplitude (Wang's σa=s a~\sigma_a=\sqrt{s}\,\tilde a), not the raw increment, which is Δσ/s\Delta\sigma/\sqrt{s} — exactly what Algorithm 1 line 6 injects (k*xi/sqrt(s)). A reported kk is therefore already the dimensionless kk of Eq. (3). In §3.1 we will state this and give the raw-scale threshold kraw>2/sk_{\mathrm{raw}}>\sqrt{2/s}; no reported number changes.

I thank the authors for these responses, which help me better to understand this work.

Reviewer_BL2MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce Noisy Replicator-Mutator Dynamics (NRMD) as replicator dynamics with two corrections: a multiplicative noise term and an additive bias coming from and inspired by two different previous works. They study the resulting dynamics, present some experiments, and propose/test a variation of PPO based on the dynamics.

Critical strengths

The topic is well-known and the paper follows a natural research direction in its purpose of studying the dynamics resulting from two different approaches to the same issue: namely, going beyond a fixed payoff matrix assumption.

Critical weaknesses

The main weakness of the paper, in my opinion, is the presentation. It is densely written, and often new notation and terms are used without explaining them. The results stated in theorems/propositions are not really explained: it seems that the statements contain parts of explanations or justifications that would deserve more space. Section 3 is a long list of subsections and paragraphs that are not well connected together in a clear sequential flow. Section 3, in particular, is missing a real discussion or comparison of the NRMD with the dynamics from which it is derived. A reader must really put some effort to do this by themselves after having understood the convergence properties described by the various propositions. I could list many minor things but it would be outside the scope of this review.

My intuition is that the text has been produced mostly by LLMs, but without a proper revision by the authors themselves. Let me be clear: I do not see anything wrong in utilizing LLMs for this purpose (the authors have clearly stated it on page 10 as well), but the final result is still difficult for me to follow, and I am familiar with the topics of the paper. This is enough for me to reject the paper.

Questions for authors

What is the benefit of considering NRMD w.r.t. the dynamics that inspire it?

1 comment
AuthorsOpenReview ↗

Response to Reviewer BL2M

We thank Reviewer BL2M for the frank assessment. We do not dispute it: the paper is dense, it uses notation before defining it, the propositions carry their own justifications, and §3 reads as a list of topics rather than an argument. We will rewrite §3 and §4.1 rather than patch them. Concretely:

  • a four-sentence roadmap at the head of §3 giving the logic chain;
  • a four-row table comparing NRMD with the two dynamics it is built from (term, what each ingredient gives alone, what it cannot give, what the combination adds);
  • every symbol defined at first use, plus a notation table in the appendix;
  • explanatory clauses moved out of the proposition statements, and one sentence after each proposition stating what it asserts and why it is there;
  • §3 and §4.1 rewritten sentence by sentence by the authors.

One correction to the summary, since it affects how the paper reads: PRPG is not a variation of PPO. PPO appears in §4.3 only as a negative control — a standard learner on which the mechanism is predicted to have no effect, and does not. PRPG (Algorithm 1) is a population learner in which reproduction is fitness-proportional; it contains no clipping, no GAE and no critic.

Q — what is the benefit of NRMD relative to the dynamics that inspire it? The equation is a sum of two known terms, and we say so; the contribution is the analysis of the sum, which is not the union of the two analyses. Four things exist only for the pair:

  1. A joint prediction that neither parent can state. Mcrit(k)=max⁡{0,s(k2/2−1)}M_{\mathrm{crit}}(k)=\max\{0,s(k^{2}/2-1)\} (Proposition 5): the mutation needed to regularise the all-defection boundary grows as k2k^{2}. Wang et al. have no MM and cannot price regularity; Bauer et al. have no kk and cannot make the price move.
  2. A boundary between learner classes. Proposition 1 (payoff-linear updates cancel the shock exactly at first order) and Proposition 2 (the term survives only if the shock is common) together say which learners inherit the effect. This is why our negative control is predicted rather than incidental, and why the paper is about multi-agent learning rather than about an ODE.
  3. A dictionary between an algorithm knob and a dynamical parameter. Proposition 3 gives M(x)=ηβψ(x)M(x)=\eta\beta\psi(x) for an entropy bonus, so an implementer can set the mutation strength instead of hoping for it. Bauer et al. take MM as given.
  4. An interaction result in the cyclic game. The noise-induced cycle is destroyed at a Hopf bifurcation whose location does not depend on the noise scale (see our reply to gJ25). Neither ingredient alone has a transition with this property.

The empirical counterpart is §4.4: one environment, one shock process, matched policy sets, twenty seeds, +0.410+0.410 for the replicator-class learner against +0.020+0.020 for the gradient-class learner on the same parameter.

Reviewer_RcMgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies replicator-mutator dynamics under global payoff noise shared by a population of agents. The authors introduce Noisy Replicator-Mutator Dynamics (NRMD), which combines a noise-induced correction with an additive mutation term. They formulate the dynamics for multiple strategies and analyze equilibria and convergence primarily in two-strategy games. They also propose Population-Replicator Policy Gradient (PRPG) and provide experiments in tabular and deep multi-agent learning settings, as well as with LLM agents.

Critical strengths

I appreciate that the empirical evaluation goes beyond tabular settings and considers learning with neural network function approximation. Section 4.3 implements a learner with a network-parameterized softmax policy and examines how mutation and payoff noise affect its behavior. This is a useful step toward understanding whether the insights from population dynamics extend to more practical multi-agent learning settings.

Critical weaknesses

My main concern is that an important prior result on replicator-mutator dynamics with noisy feedback is missing, making the novelty of the paper unclear. Section 2 states that prior work has studied mutation without noise or noise without mutation. However, Abe et al. (2023) [1] propose Mutation-Driven Multiplicative Weights Update (M2WU), a discrete-time algorithm corresponding to replicator-mutator dynamics, and establish almost-sure last-iterate convergence to its stationary point under noisy feedback. The paper cites Abe et al. (2022), but does not discuss this subsequent result. Although the present paper studies shared payoff shocks and noise-induced changes in population dynamics, which differ from the setting of Abe et al. (2023), the claim that noise and mutation have only been studied separately is too broad. I believe that a direct comparison of the noise models, update rules, and convergence guarantees is necessary to clarify what is new in NRMD and Proposition 6.

Additionally, I found it difficult to follow the main argument throughout the paper. Paragraph headings are frequently used to introduce new topics, but the connections between the paragraphs are often unclear, making it difficult to identify the main message of each section. For example, Section 3.2 abruptly introduces a diffusion process for the logit without sufficiently explaining why this process is considered or how it relates to the learning updates motivating the paper. The discussion then moves to “First-order immunity of the payoff-linear class,” but the connection to the preceding derivation and the purpose of considering this class are not sufficiently explained. I believe that explaining why each topic is introduced and how it contributes to the section’s main argument, rather than relying on paragraph headings to organize the discussion, would make the paper substantially easier to follow.

[1] Kenshi Abe, Kaito Ariu, Mitsuki Sakamoto, Kentaro Toyoshima, and Atsushi Iwasaki. Last-Iterate Convergence with Full and Noisy Feedback in Two-Player Zero-Sum Games. AISTATS, 2023.

Questions for authors

My main concerns and questions are outlined in the Critical Weaknesses section. Additionally, I have the following questions:

In Section 3.1, what does “symmetric” mean for the population of NN agents? Does it mean that all agents receive the same payoff, or that they share the same payoff function while their realized payoffs can differ depending on their actions and opponents?

Proposition 3 expresses the entropy term as a mutation bias with state-dependent strength M(x)M(x), whereas Proposition 5 and Corollary 1 assume a constant mutation strength MM. Appendix C notes that these constant-MM results are only approximations in the interior, but the main text uses the correspondence to interpret the subsequent conditions in terms of the entropy coefficient. Under what conditions can M(x)M(x) be approximated by a constant, and which conclusions of Proposition 5 and Corollary 1 remain valid for the entropy-regularised update? In particular, how does M(x)M(x) vanishing at the boundaries affect this interpretation?

1 comment
AuthorsOpenReview ↗

Response to Reviewer RcMg

We thank Reviewer RcMg for the careful reading and for pointing us to Abe et al. (2023), which we had missed.

W1 — Abe et al. (2023). The omission is real. We will cite it in §2, replace "studied apart" with the narrower shared payoff shocks and mutation have not been analysed jointly, and add the following comparison.

Abe et al. analyse M2WU, a discrete-time replicator–mutator algorithm, in two-player zero-sum games under noisy feedback, proving almost-sure last-iterate convergence. Their noise is feedback noise — zero-mean, independent across players — not a shock to the game. Proposition 2 is exactly the statement that such noise cannot survive the population average: with independent shocks Var[Δx∣x]=α2x2(1−x)2(Δσ)2/N=O(1/N)\mathrm{Var}[\Delta x\mid x]=\alpha^{2}x^{2}(1-x)^{2}(\Delta\sigma)^{2}/N=O(1/N), so the term vanishes in the mean-field limit at any noise magnitude. Our shock is common and multiplies every payoff, so it enters the deterministic field at second order. Different objects: theirs is robustness of a converging algorithm, ours changes the equilibria. Neither implies the other. The update rules differ for the same reason: M2WU's continuous limit is the replicator–mutator ODE, whereas our shock scales every payoff by the same 1+kξ1+k\xi, so the update cannot average it away.

On Proposition 6: it is the standard ODE method (Benaïm, Borkar) applied to the combined field, and we will say so rather than presenting it as new. What is specific is the O(1/t)O(1/t) rate under λc>12\lambda c>\tfrac12 and the explicit n≥3n\ge3 caveat that the limit set need not be a fixed point.

W2 — readability, and §3.2. Accepted. The diffusion is introduced there because the sign of the Itô correction depends on which coordinate the shock enters, and that same fact decides which learners inherit the mechanism; the two halves of §3.2 are one argument, which the paper never says. §3.2 will open with that sentence, and the immunity paragraph with "the other side of the same computation is…". Paragraph headings will no longer be the only organising device.

Q1 — what "symmetric" means. The game, not the realised payoffs. All NN agents share one A\mathbf{A} and one action set; the realised payoff of agent ii is π(ai,aj)\pi(a_i,a_j), so it differs across agents and rounds. Common are the matrix and the shock: one ξ∼N(0,1)\xi\sim\mathcal N(0,1) per generation, applied to every interaction. Proposition 2 is why that commonality matters. We will define this in §3.1 in one sentence.

Q2 — M(x)M(x) versus constant MM. This is the sharpest question in the review, and the paper buries the answer in Remark 2.

  • M(x)=ηβ ψ(x)M(x)=\eta\beta\,\psi(x) with ψ(x)=x2(1−x)2log⁡1−xx/(12−x)\psi(x)=x^{2}(1-x)^{2}\log\frac{1-x}{x}\big/(\tfrac12-x), 0<ψ≤140<\psi\le\tfrac14, ψ(12)=14\psi(\tfrac12)=\tfrac14, and ψ(x)=2x2log⁡1x(1+o(1))\psi(x)=2x^{2}\log\frac1x(1+o(1)) as x→0x\to0.
  • Hence constant MM is valid on any [δ,1−δ][\delta,1-\delta], with Meff=ηβ ψ(x^)M_{\mathrm{eff}}=\eta\beta\,\psi(\hat x) at the operating point x^\hat x; the relative error is O(δ2log⁡1δ)O(\delta^{2}\log\frac1\delta).
  • Inherited. The interior content of Proposition 5 — odd number of roots, smallest one attracting, k2k^{2} scaling — carries over with M→MeffM\to M_{\mathrm{eff}}.
  • Not inherited. The boundary-slope clause M>Mcrit(k)M>M_{\mathrm{crit}}(k). Because M(x)(12−x)=ηβ x2(1−x)2log⁡1−xx=O(x2log⁡1x)=o(x)M(x)(\tfrac12-x)=\eta\beta\,x^{2}(1-x)^{2}\log\frac{1-x}{x}=O(x^{2}\log\frac1x)=o(x), F′(0)=s(k2/2−1)F'(0)=s(k^{2}/2-1) for every ηβ\eta\beta: an entropy bonus cannot remove the boundary degeneracy, so McritM_{\mathrm{crit}} is a property of the controlled constant-MM model.
  • Corollary 1 survives only vacuously: its premise is a positive floor at k=0k=0, which the entropy bias does not produce, so the inequality holds for every k>2k>\sqrt2 and is not what limits that learner — the boundary staying attracting is.
  • Reading. The budget is a budget on ηβ\eta\beta at the operating point, not a guarantee that the boundary is regularised. Consistent with §4.3, where the entropy-only run collapses to H=0.005H=0.005.

We will promote this from the appendix to a paragraph in §3.3, and mark which clauses of Proposition 5 and Corollary 1 are inherited.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to the reviewers — summary of changes

We thank all three reviewers for the time they spent on the paper. The reviews agree on two things — the writing is hard to follow, and the paper does not say clearly enough what it adds to Wang et al. (2026) and Bauer et al. (2026) — and we take both seriously. In the revision we will replace §3 and §4.1. Four changes are common to all replies:

  1. Structure. §3 opens with a four-sentence roadmap stating the logic chain (where the shock enters → which learners are immune → what mutation changes → whether a learner reaches it), each paragraph ends with a forward pointer, and every proposition is followed by one sentence saying what it asserts and why it matters. Explanatory clauses move out of the proposition statements into the surrounding prose.
  2. A direct comparison with the two parent dynamics (a four-row table: term, what each ingredient gives alone, what it cannot give, what the combination adds). This is the single change that answers the "what is the benefit" question in all three reviews.
  3. Abe et al. (2023) is added to §2 with an explicit comparison of noise models.
  4. §3.1 states the noise normalisation explicitly (Δσ\Delta\sigma is the s\sqrt{s}-rescaled amplitude) and the Itô coefficient is written out so that its α2\alpha^{2} scaling is unambiguous.

One correction we volunteer, because it changes a claim: prompted by Reviewer gJ25's request for a rigorous bifurcation analysis we recomputed the rock–paper–scissors transition. It is a supercritical Hopf bifurcation at MH=(α−β)/6M_H=(\alpha-\beta)/6, independent of kk, not a subcritical transition with a coexistence window; the earlier reading came from integrating the three-dimensional field without projecting onto the simplex. Details in the reply to gJ25; Fig. 2 (right) and the §4.1 paragraph will be replaced.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.