acceptodds
Under review as a conference paper at ICLR 2027

Data, Objective, and Architecture: When Do Independent Policy-Gradient Learners Collude?

Abstract

Generative AI agents increasingly make sequential decisions in multi-agent environments, where independently reward-maximizing agents can nevertheless learn cooperative or collusive behavior. We study how two defining components of modern generative agents— and —shape collusion under independent policy-gradient learning. Using a repeated pricing game as a controlled social-dilemma environment, we develop an analytical framework that separates what an agent learns its pre-training data, what it learns under its post-training objective, and what it can through its context architecture. We first show that next-token pre-training acts as a behavioral-transmission mechanism: under realizability and standard generalization conditions, the learned policy recovers the teacher's history-conditional action distribution, so pre-training determines the behavioral initialization from which subsequent multi-agent learning begins. We then characterize independent own-reward population policy-gradient dynamics. With one-step memory, we derive an exact action-value gap showing that the gradient favors high-price actions only when the opponent's learned reciprocity exceeds an explicit threshold. This yields a patience condition and an invariant non-collusive basin: sufficiently non-collusive initial policies converge toward competitive behavior, while crossing the threshold is not by itself sufficient for collusion. Memory changes these dynamics by creating additional continuation-value channels. Without learned memory, agents cannot retaliate, but collusion can still arise through coordinated discovery; we derive a separate threshold below which policy-gradient learning eliminates high-price behavior. For learned context length , we analyze a soft-attention policy and derive a sufficient non-collusion threshold that decreases with while remaining strictly positive for every fixed discount factor below one. Thus, longer memory enlarges the set of future states through which current actions can affect an opponent's behavior and correspondingly shrinks the guaranteed competitive basin. Together, our results show that pre-training shapes multi-agent learning through , while memory reshapes its , providing a tractable learning-theoretic account of emergent collusion under independent policy-gradient learning.

Primary area:
learning theory
Keywords:
Generative AI, AI pricing agents, collusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_juzuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates collusion among independent policy-gradient learners in repeated pricing games. It analyzes three factors: pre-training data, reward-based learning objectives, and memory architecture. The authors establish a pre-training imitation bound, derive an exact reciprocity threshold for one-step memory, and characterize sufficient non-collusion conditions for memoryless and longer-context policies.

Critical strengths
  1. The paper addresses an important question about how collusive behavior may emerge from independent multi-agent learning.

  2. The exact action-value gap and reciprocity threshold provide useful insights into how an opponent's conditional responses affect policy-gradient learning.

  3. The separation of pre-training, learning objectives, and memory effects offers a useful theoretical perspective. The assumptions are generally explicit, with detailed proofs in the appendix.

Critical weaknesses
  1. My main concern is the distinction between high-price behavior and strategic collusion. Under the payoff definition in Eq. (1), the collusion index in Eq. (5) reduces to the discounted average frequency of high-price actions. Even two agents independently choosing high prices with probability 0.5 would yield an expected index of 0.5, without any coordination. The metric therefore captures high-price outcomes rather than strategic collusion itself. Although the authors acknowledge this distinction, I am not convinced that the main theoretical results fully address when collusion actually emerges. Most results establish conditions for avoiding high-price behavior, rather than sufficient conditions for stable collusion.

  2. Proposition 1 provides an average imitation-error bound, but this does not directly guarantee the initialization conditions required by Propositions 3 and 5. In particular, the connection between pre-training error and membership in the non-collusive basin remains unclear.

  3. In the memoryless setting, a persistent coordination regime is activated after consecutive joint high-price actions. However, the paper does not clearly specify its post-discovery dynamics. It is also unclear whether this coordination mechanism emerges from learning or is externally imposed.

  4. The analysis relies on restricted policy classes and exact population-gradient dynamics. I would also like to see stronger evidence that the decreasing sufficient threshold for longer contexts reflects an actual change in the non-collusive basin, rather than simply a looser bound. Some numerical validation would help clarify these issues.

Questions for authors

Q1. How does the collusion index distinguish genuine strategic coordination from independently occurring high-price actions? Can the authors clarify which theoretical results establish collusive behavior rather than merely high-price outcomes?

Q2. Can the authors establish a formal connection between the pre-training imitation bound and the non-collusive initialization conditions?

Q3. What happens after the persistent coordination regime is triggered in the memoryless model? Is this mechanism learned or externally specified?

Q4. Does longer memory actually reduce the non-collusive basin, or does it only weaken the sufficient theoretical guarantee? Could the authors provide numerical evidence to clarify this distinction?

6 comments

We thank the reviewer for the detailed feedback and for recognizing the importance of the research question, the exact action-value characterization, and the separation of training mechanisms.

We have addressed Q1–Q4 through revised model definitions, new theoretical propositions, complete appendix proofs, and additional numerical illustrations. The revised paper provides a clearer distinction between high-price outcomes and strategic reciprocity, establishes a formal connection between pretraining and post-training, fully specifies the memoryless benchmark, and strengthens the analysis of evolving attention and context length.

Q1. High-price outcomes versus strategic collusion

We have clarified the economic meaning of the payoff-based collusion index and introduced separate theoretical measures of synchronization and strategic reciprocity.

Under our payoff specification, the discounted index reduces exactly to discounted high-price incidence: κδ=∑tωtX1,t+X2,t2.\kappa^\delta=\sum_t\omega_t\frac{X_{1,t}+X_{2,t}}{2}.

We therefore distinguish high-price outcomes from the strategic mechanism generating them.

We additionally introduce a new proposition linking contemporaneous synchronization and lagged reciprocity under one-step reactive policies:

St=ρ1ρ2St−1,Cov⁡(Xi,t,X−i,t−1)=ρiVar⁡(X−i,t−1).\mathcal S_t=\rho_1\rho_2\mathcal S_{t-1}, \qquad \operatorname{Cov}(X_{i,t},X_{-i,t-1}) =\rho_i\operatorname{Var}(X_{-i,t-1}).

These identities establish that contemporaneous synchronization can vanish even when strategic reciprocity is strong, while lagged dependence identifies the reciprocity coefficient within the specified policy class.

The revised manuscript consequently distinguishes high-price incidence, synchronization, and the history-dependent incentives characterized by the action-value gap. This provides a more precise interpretation of what our convergence propositions establish.

Revised-paper evidence for Q1: Section 3, p. 4, lines 202–215 (Equations (9)–(11)) derives the exact high-price-incidence identity and defines separate synchronization and lagged-response statistics. Proposition 3, p. 6, lines 293–310 (Equations (19)–(21)), establishes the covariance identities. Lemma 1, p. 12, lines 620–632, proves the index identity; Appendix E, p. 15, lines 784–803, proves Proposition 3. Proposition 4’s interpretation at p. 7, lines 324–326, specifies that the convergence claim concerns high-price initiation/incidence rather than strategic punishment.

Q2. Formal connection between pretraining and post-training

We have introduced a new finite-sample theorem that explicitly connects the pretraining generalization bound to post-training safe-basin membership.

Let ζ>0\zeta>0 denote the teacher's margin below the relevant safe threshold, μ>0\mu>0 the minimum probability of the required teacher contexts, and ξ\xi the policy-extraction error.

We establish that the pretrained student enters the invariant non-collusive basin whenever εn(η)2μ+ξ<ζ,\sqrt{\frac{\varepsilon_n(\eta)}{2\mu}}+\xi<\zeta, where εn(η)\varepsilon_n(\eta) is the finite-sample pretraining KL bound.

The existing post-training convergence propositions then apply under their respective assumptions.

This result directly connects training-data quality, history coverage, student approximation accuracy, and subsequent independent policy-gradient dynamics. It also identifies when average predictive accuracy is insufficient to certify downstream behavior. The complete proof is provided in the appendix.

Revised-paper evidence for Q2: Proposition 1, p. 5, lines 241–259, supplies the average KL bound but explicitly notes the coverage gap (lines 256–259). Section 7 begins with that gap at p. 9, lines 464–466. Proposition 9, pp. 9–10, lines 484–520, specifies assumptions (C0)–(C3) (lines 486–491), proves the probability bound (Equation (39), lines 496–500), and states the basin-entry condition (Equation (40), lines 501–508). Appendix B, pp. 13–14, lines 687–727, derives the result. The coverage illustration is in p. 10, lines 521–525 and Appendix M, p. 26, lines 1398–1403.

Q3. Specification of the memoryless coordination mechanism

We have fully specified the K=0K=0 coordination-discovery benchmark, including its state transitions, post-discovery policy, and learning parameters.

The environment maintains a joint-high-price streak counter DtD_t. Consecutive HHHH outcomes increment the counter, any other outcome resets it, and reaching mm activates an absorbing coordination regime.

Before discovery, agents choose HH with probability σ(αi)\sigma(\alpha_i). After discovery, their high-price probability is a fixed externally specified parameter qCq^C. Only the pre-discovery policy parameters are updated.

We have also added an appendix lemma formalizing the counterfactual coupling and recoupling argument.

The resulting proposition establishes a sufficient condition for avoiding entry into this explicitly specified regime. We now distinguish this benchmark from endogenous learning of retaliatory strategies, which is analyzed separately through the learned-memory policies.

Revised-paper evidence for Q3: Section 3, p. 4, lines 170–183 fully defines the state counter and transition (Equation (6)), the pre-/post-trigger policy (Equation (7)), and which parameters are trained. Section 6, p. 7, lines 335–357 presents Proposition 5 as a sufficient non-entry statement and distinguishes the external mode from endogenous punishment. Appendix H, pp. 17–19, lines 884–983, specifies the fixed post-trigger action law and proves the result; Lemma 4 (lines 889–906) gives the counterfactual recoupling argument.

Q4. Context length, evolving attention, and the non-collusive basin

We have substantially strengthened the memory analysis beyond the original context-length bound.

First, we derive an exact lag-specific sensitivity characterization and the sharper inequality χij≤tanh⁡(∣λi∣aij4)≤∣λi∣aij4.\chi_{ij}\leq\tanh\left(\frac{|\lambda_i|a_{ij}}4\right)\leq\frac{|\lambda_i|a_{ij}}4.

This allows us to establish a refined continuation-value bound based on the policy's actual attention distribution rather than context length alone.

Second, we explicitly incorporate the evolution of attention weights and response slopes during policy-gradient learning. The revised analysis establishes:

  • A finite-time guarantee for the original unconstrained gradient flow, accounting for parameter drift.
  • A global convergence guarantee for a bounded but still trainable attention architecture.

Revised-paper evidence for evolving parameters: Proposition 8, p. 8, lines 404–418, distinguishes the original unconstrained finite-learning-time certificate from the bounded-but-trainable global guarantee (Equation (34)). Appendix J, pp. 21–23, lines 1080–1201, derives parameter-speed bounds (Lemma 7, lines 1091–1119), moving-policy influence bounds (Lemma 8, lines 1122–1157), and the two gradient-flow conclusions (lines 1161–1201). This explicitly treats changes in response slopes and attention logits during learning.

For the latter, with bounded response slope Λ\Lambda, attention concentration RR, and uK=ΛR4K∑j=1Kδju_K=\frac{\Lambda R}{4K}\sum_{j=1}^{K}\delta^j,

we establish the sufficient condition: ΓuK+2ΔuK2<Δ\Gamma u_K+2\Delta u_K^2<\Delta.

This condition guarantees negative high-price action-value gaps throughout learning and convergence of high-price initiation from the specified competitive history.

Crucially, for fixed Λ,R,δ\Lambda,R,\delta, uKu_K decreases with KK. Thus, our refined analysis shows that longer available context can strengthen, rather than weaken, a non-collusion guarantee when learned attention remains sufficiently diffuse.

This complements the original worst-case bound and demonstrates why nominal context length alone does not determine the strength of strategic feedback. We explicitly distinguish these sufficient guarantees from claims about the exact attraction basin, whose monotonicity is not established.

Revised-paper evidence for context-length interpretation and numerical checks: Corollary 2 (pp. 8–9, lines 419–428; Equation (35)) states monotonicity of the original sufficient threshold, whereas Proposition 8 and the explanation at p. 8, lines 411–420, show that the bounded-attention certificate can improve with K. The qualifications appear at pp. 8–9, lines 426–449. Appendix M, p. 27, lines 1441–1451 and Figures 2–3, compares the sharper and original certificates, explicitly without claiming measured monotonicity of true attraction basins.

Summary of contributions

The revisions establish three connections that are central to the paper:

  1. Pretraining → post-training: A finite-sample theorem linking teacher behavior and generalization accuracy to invariant policy-gradient basins.
  2. Learning dynamics → strategic behavior: Exact reciprocity and synchronization identities that distinguish high-price outcomes from history-dependent incentives.
  3. Memory architecture → learning guarantees: Refined sensitivity bounds and trajectory-aware results for evolving attention parameters, including conditions under which longer context improves the non-collusion guarantee.

Together, these results provide a more complete theoretical account of how behavioral initialization and learned memory interact with independent policy-gradient learning. We believe the revised manuscript directly resolves the specification and interpretation concerns raised in the review and substantially strengthens the original analysis.

We appreciate the reviewer's detailed questions, which provided an opportunity to make these contributions more explicit.

Reviewer_EHXyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper analyzes independent learners in a repeated pricing game. One-step memory yields an exact reciprocity threshold for high-price incentives. Restricted policies admit conditional non-collusion guarantees under exact gradient flow.

Critical strengths

The paper studies a question of interest to the multi-agent learning community: when independently trained agents may collude.

Critical weaknesses
  1. I find the overall contribution insufficiently novel.

    1. Section 4 is a straightforward application of standard finite-class generalization arguments for autoregressive maximum-likelihood learning.
    2. Equation (1) is strategically equivalent to the donation-game form of the repeated Prisoner's Dilemma. Making this connection explicit is essential for positioning the results against the relevant literature. The patience condition in Section 5 follows directly from the action-value identity and expresses the familiar trade-off between future reciprocity and immediate gains from defection. Prior work already analyzes defection stability and initialization-dependent basins under independent reactive-strategy learning ("Fujimoto and Kaneko, Emergence of exploitation as symmetry breaking in iterated prisoner's dilemma, Physical Review Research, 2019"). Further work analyzes related payoff-gradient dynamics and describes the established reciprocity threshold for reactive strategies ("LaPorte, Hilbe, and Nowak, Adaptive dynamics of memory-one strategies in the repeated donation game, PLOS Computational Biology, 2023").
  2. The presentation obscures the main contribution. The abstract is lengthy and overloaded with technical details and case-specific conclusions, making its central message difficult to identify. Sections 4–6 would benefit from brief opening paragraphs explaining the question addressed, its connection to the overall argument, and the main takeaway before introducing notation and propositions. As written, the transitions into the technical statements are abrupt.

Questions for authors

Could the authors clarify the substantive contribution of Sections 5–6 to the analysis of policy-gradient learning, relative to the prior work cited above?

2 comments

We thank the reviewer for feedback on novelty, presentation, and the contributions of Sections 4–6. We have revised the manuscript with a new cross-stage theoretical result, stronger analysis of learned attention, and clearer positioning relative to prior work.

1. Substantive theoretical contributions

We expanded Related Work (lines 105–130) to explicitly position our analysis against Fujimoto and Kaneko (2019) and LaPorte et al. (2023), and clarify the donation-game equivalence (lines 139–143).

Our central contribution characterizes how predictive pretraining determines the initialization of independent policy-gradient learning, and how learned memory affects subsequent learning dynamics and non-collusive guarantees.

The revised manuscript makes three contributions:

(i) A formal pretraining-to-policy-gradient guarantee (Section 7, Proposition 9, lines 484-520; proof in Appendix B). We introduce a new proposition connecting finite-sample pretraining accuracy to the invariant non-collusive basins derived for post-training. We show that teacher-policy margin, history coverage, and policy-extraction accuracy jointly determine whether a pretrained student enters a guaranteed non-collusive region. This establishes an explicit connection between the data used for pretraining and the subsequent dynamics of independently reward-optimizing agents.

(ii) Parameter-level policy-gradient dynamics (Section 5, Proposition 4, lines 317-327; proof in Appendix G). Beyond the exact reciprocity identity, we derive the gradient dynamics of the trainable logit parameters and establish forward invariance and convergence properties for the resulting conditional action probabilities. The contribution is not the classical patience condition itself, but its role in characterizing the learning trajectories of independently updated parametric policies.

(iii) Learning with evolving attention and context (Section 6, Propositions 7-8, lines 388-418; proofs in Appendices I-J). We extend the analysis beyond fixed memory-one strategies to trainable positional attention. The revision derives exact lag-specific sensitivities and sharper continuation-value bounds. Importantly, we further account for changes in response slopes and attention weights during learning, establishing a finite-time guarantee for unconstrained gradient flow and a global guarantee for a bounded but trainable attention architecture. These results distinguish effective learned memory from nominal context length.

2. Relationship to prior work

We have rewritten the abstract (lines 014–024) and revised the section introductions:

  • Section 4, lines 218–220: behavioral transmission and its role in policy initialization.
  • Section 5, lines 264–271: reciprocal incentives and their relationship to parameter-level policy gradients.
  • Section 6, lines 332–333: the distinction between exogenous coordination discovery and learned memory, including conservative and sharper attention-dependent bounds and their limitations.
  • Section 7, lines 464–466: the missing connection between averaged imitation accuracy and uniform safe-basin initialization, addressed through teacher-history coverage, a safe teacher margin, and controlled policy-extraction error.

These studies establish important results on adaptation, reciprocity, and initialization-dependent dynamics in repeated social dilemmas. Our contribution is the connection between predictive pretraining, certified policy initialization, and independent gradient learning with evolving attention. We now clearly distinguish these results from the classical donation-game reciprocity condition and the standard finite-class generalization bound.

3. Presentation improvements

We have rewritten the abstract to emphasize this central contribution and added introductory explanations to Sections 4–7 that identify each section's research question, its role in the framework, and its main takeaway before presenting technical results.

The revised theoretical progression is:

Pretraining and behavioral transmission → policy-gradient incentives → learned-memory dynamics → finite-sample initialization guarantees.

We have also refined the interpretation of the propositions to distinguish exact results, sufficient bounds, and convergence guarantees.

Where the presentation clarification is visible: The introductory synthesis at pp. 1–2, lines 050–089, distinguishes the data, objective, and architecture channels. The transition at p. 9, lines 464–466, explicitly explains why the Section 7 result is needed after Sections 4–6. The conclusion (p. 10, lines 529–539) states the guarantees and their limits in a compact form.

We believe these revisions materially strengthen both the technical contribution and its presentation, and directly address the reviewer's concerns about novelty and the connection between Sections 4–6. We appreciate the opportunity to clarify these contributions.

In addition to the revisions described in our previous response, we would like to share two new empirical findings relevant to the original pretraining-to-post-training framework and the role of policy architecture. These additional changes are marked in orange in Section 9 and Appendix U.5.1.

1. Economic attraction versus conditional-policy stability. Experiments C1–C3 (Figures 18–28) examine independent sampled own-profit learning. While the seven representative C1 treatments reproduce the qualitative exact-flow outcomes, the untied joint-history experiments reveal a distinction between economic stability and stability of the entire conditional policy. All reported strongly contingent untied three-period policies satisfy the economic and visited-history stability checks, whereas none satisfies the all-history behavioral check (Figure 27). High-price probabilities at rarely visited histories can increase even while joint high pricing is maintained.

2. Behavioral transmission and subsequent neural learning. We additionally fit two independent two-layer causal Transformer actors to matched synthetic conditional policies and subsequently update all trainable parameters using independent sampled own-profit gradients, without an imitation loss (Figure 29). Despite close initial behavioral imitation, the strongly contingent neural policies attain terminal mean joint high-price incidences of approximately 0.20 and 0.60 for one- and three-period contexts, respectively, compared with near one for the matched tabular policies.

These findings reinforce the distinction between successful behavioral transmission and the subsequent preservation of transmitted behavior under independent reward adaptation. They demonstrate the empirical relevance of policy parameterization even when initialization is behaviorally matched, and motivate further investigation of cross-history parameter sharing. We do not claim to have causally isolated parameter-sharing interference or established a finite-sample extraction guarantee for unrestricted neural networks.

The new findings and limitations are summarized in Section 9, the Conclusion, and Appendix U.7. These are additional supporting analyses of the original framework. We appreciate your consideration of this update.

Reviewer_PfqdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies a stylized two players, two actions game, where the probability of each player to play one action depends of the past actions of the other player. It shows that a policy trained to minimize the log-likelihood of the actions of a given policy reproduces the actions of that given policy.That, when one player can influence the other, its optimal actions might not be to maximize the instantaneous reward. Finally, that last sections gives conditions, depending on how many past actions players observe, under which learning converges to always playing the same action.

Critical strengths

S1: The calculations are correct.

Critical weaknesses

The abstract and introduction are particularly ambitious and aim general results. However, all the paper presents a very stylized model, where everything can be derived easily, and no general results can be derived from. The results are textbooks. The paper has crucial lacking of flow in presentation. I strongly recommend rejection.

Questions for authors

I do not have any specific question for the authors.

4 comments

Our model is deliberately stylized to isolate mechanisms of strategic interaction that are difficult to disentangle in complex LLM systems. The contribution lies in developing a tractable abstraction that enables transparent, rigorous analysis of these mechanisms. Tractability and the use of established analytical tools do not, by themselves, imply that the findings lack novelty or relevance.

We agree that the pretraining Proposition 1 concerning imitation of a target policy reflects a standard result in machine learning. We include it for completeness in establishing the model, and do not claim it as a novel contribution. We can make this explicit and replace its proof with an appropriate reference.

This distinction matters when assessing our main collusion findings. The revised manuscript explicitly explains how our model differs from prior models. To substantiate the broader characterization of our findings as “textbook,” we ask the reviewer to identify prior results that subsume those findings under comparable assumptions in setup of LLM architecture and collusion in textbook.

Lastly, the abstract presents our contribution as a model as abstraction for the complex system, for explaining the system’s behavior, and the accompanying experiments support the relevance of the identified mechanisms beyond the analytical setting. We are happy to revise claims the reviewer considers too broad. But the review does not say which claims need revision or which parts are unclear, so it is difficult to know what changes would address these concerns.

I thank the authors for their answer.

The manuscript has changed substantially, a large part of the revised version appears in blue. I believe the paper would therefore need a new round of review rather than an update of the current one.

My main concern remains. The model is so stylized that it is hard to tell which results or methods would generalize to a more complex model, while the paper is motivated by LLM pricing (see the introduction). The revised abstract now says the results do not concern "general convergence of unrestricted language models". I agree, but the title, introduction and Table 1 still frame the paper that way. The title should state explicitly that the work concerns a stylized model. In addition, the stage game (eq. 1) is an additive donation game: there is no demand, no price grid, and no undercutting incentive that depends on the rival's price, which are the defining features of a pricing game.

By "textbook" I meant that the core results (Props. 1–4) are classical, and that the remaining ones (Props. 5–8) are elementary and loose.

Prior results, as requested.

  • Prop. 1: you agree it is standard.
  • Prop. 2 and Cor. 1, this is classical condition for reactive strategies in the repeated donation game. See Nowak & Sigmund (1990).
  • Prop. 4: this is the adaptive-dynamics picture for reactive strategies (Nowak & Sigmund 1990; LaPorte et al. 2023). It is also the analytical counterpart of Foerster et al. (2018, LOLA).
  • Prop. 3: a two-line application of the law of total covariance.
  • Props. 5–8: They are contraction bounds of a standard type. They are also loose by your own numbers.

The paper does not answer its own question.

The title asks when learners collude. Every learning result (Props. 4–9) is a sufficient condition for not colluding. No result shows that collusion emerges, or how data, objective or architecture would cause it.

Experiments. Your response says the experiments support the relevance of the mechanisms beyond the analytical setting. Appendix M uses only the restricted policy classes of Sections 5–6, so these are numerical checks of the propositions, not evidence beyond them.

References

Nowak, M. A., & Sigmund, K. (1990). The evolution of stochastic strategies in the Prisoner’s Dilemma. Acta Applicandae Mathematicae,

LaPorte, P., Hilbe, C., & Nowak, M. A. (2023). Adaptive dynamics of memory-one strategies in the repeated donation game. PLOS Computational Biology,

Foerster, J. N., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., & Mordatch, I. (2018). Learning with opponent-learning awareness. In Proceedings of AAMAS 2018.

Thank you for the follow-up and the specific references. In response to the combined reviewer feedback, we have strengthened the original analysis and added targeted robustness extensions. We distinguish below what clarifies the original results, what provides supporting evidence, and what extends the model. For ease of reference, we have marked the changes made in this latest revision in red to distinguish them from the changes made in the previous revision, which are marked in blue.

1. Original contribution (clarification and strengthened analysis). Our framework separates predictive initialization, individual reward adaptation, and memory-dependent feedback. We do not claim novelty for maximum likelihood, classical reciprocity, or total covariance individually. Nowak and Sigmund (1990) and LaPorte et al. (2023) study reactive and memory-dependent adaptive dynamics; Foerster et al. (2018) differentiate through anticipated rival updates, unlike our independent gradients. Our contribution connects conditional imitation to invariant learning regions and evolving-attention certificates (Propositions 7–9). (Section 7, Eq. 42; Appendix B.1) constructs exact audited-endpoint extraction and improves the conditional error rate from n−1/4n^{-1/4} to n−1/2n^{-1/2} under the existing finite-class realizability and exact maximum-likelihood assumptions. Appendix P.1 now reports three settings activating the sharper certificate, each with 350/350 safe fitted pairs. These strengthen the original data channel without changing the model. Table 1 is intended to compare separation of learning channels, not absence of memory; we recognize the need to align its wording and the title with the stylized scope.

2. Explicit pricing (new extension). Appendix Q (Q.1–Q.5) introduces a differentiated-product pricing game with a finite price grid, logit demand and marginal costs. Unlike Eq. (1), the immediate undercutting gain depends on the rival's price. In our binary calibration, it is approximately 0.1313 against a high-priced rival and 0.2599 against a low-priced rival. Lemmas 14–15 generalize the one-step action-value expression and sufficient low-price guarantee to nonadditive payoffs, under their stated assumptions. The original independent own-profit learning rule is unchanged.

3. When high prices are sustained (new complementary results). Proposition 10 (Section 8, lines 452–457; Appendix R) establishes an open, forward-invariant attraction region under independent own-profit logit gradients. With separately trainable initial, cooperative-history and punishment-history probabilities, the sufficient patience condition is δ>T−RT−P.\delta>\frac{T-R}{T-P}.

Proposition 11 and Appendix S extend positive attraction to longer joint-history policies. Corollaries 3–4 connect conditional teacher behavior to these positive regions through finite-sample imitation and extraction conditions. Unlike a fixed-policy incentive calculation, these results establish attraction under simultaneous parameter updates. Appendix T separately gives the complete interior basin classification for a memoryless mixed-reward objective (Lemmas 18–19). We distinguish this explicitly aligned objective from tacit own-profit cooperation. The positive results require contingency to be seeded; they do not claim spontaneous discovery from arbitrary initializations. Our original one-sided safety guarantees remain valuable for pre-deployment auditing even without a complete basin classification.

4. Experiments beyond the original assumptions (supporting analysis and robustness tests). Section 9 and Appendix U now report substantially broader controlled experiments. A1 follows completed predictive imitation with independent own-profit learning: contingent teachers yield stabilized joint high prices in 50/50 dataset pairs in each binary market, versus 0/50 for unconditional-high or discounted-frequency-matched teachers (Appendix U.1, Figure 1). A2–A3 introduce sampled policy gradients and learned attention. Experiment B uses two-, five- and nine-price demand grids and untied categorical policies beyond the binary theorem; strong contingent initializations sustain near-maximal profits, with negative net gains from forced undercutting (Appendix U.4, Figures 9–12). C examines positive attraction and untied joint-history policies; D explores objective-dependent selection and stochastic updates. These experiments extend beyond several assumptions of the original propositions, but remain controlled parametric learning experiments, not unrestricted LLM training.

These changes respond to the collective reviewer requests. The original stage game, independent-gradient framework, and data–objective–architecture decomposition remain intact. The clarifications and supporting analyses strengthen the original contribution; the explicitly labeled extensions complement it rather than constitute a replacement submission. We appreciate your consideration.

Following our previous response, we would like to share an additional update addressing your earlier concern about whether the mechanisms identified in our analytical framework extend to more general learning settings. We have added experiments with sampled policy gradients and causal Transformer policies. For ease of reference, these additional changes are marked in orange in Section 9 and Appendix U.5.1.

1. Learning beyond exact population gradients. Experiments C1–C3 (Figures 18–28) replace exact population gradients with finite-horizon sampled own-profit gradients and independent Adam updates. All seven representative initialization treatments in C1 reproduce the qualitative exact-flow outcomes. The broader experiments also identify departures from the exact-flow results: some initially unconditional-high policies sustain joint high prices from supplied cooperative histories in longer-context configurations, and untied policies can exhibit stable economic outcomes while remaining behaviorally unstable at rarely visited histories. These results clarify both the empirical relevance and the limitations of our analytical guarantees.

2. Learning beyond restricted policy classes. We additionally introduce two independent two-layer causal Transformer actors (Figure 29). We first fit the actors to matched synthetic history-conditional policies and subsequently update all trainable parameters using each firm's own sampled discounted returns, without an imitation loss during post-training. Under strong contingent initialization, the neural policies initially exhibit near-unit joint high-price incidence but attain terminal means of approximately 0.20 and 0.60 for one- and three-period contexts, respectively, compared with near one for the matched tabular policies. Unconditional-high neural initializations reach near-zero joint high-price incidence.

These results show that successful behavioral imitation need not preserve history-dependent responses under subsequent neural learning. They identify a boundary to the tabular attraction guarantees, without establishing the cause of neural deterioration or a causal effect of context length.

Together, these experiments provide additional evidence on the extent to which our identified mechanisms transfer beyond the original analytical policy classes. The findings and limitations are summarized in Section 9, the Conclusion, and Appendix U.7. They complement the results described in our previous response without changing the original model or theoretical analysis. We appreciate your earlier suggestion, which motivated these additional experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.