acceptodds
Under review as a conference paper at ICLR 2027

Persistent Partners Raise Prices Among Learning Agents

Abstract

When pricing agents meet repeatedly on a platform, the platform decides who faces whom. We ask whether that choice moves the prices the agents learn, and whether a rise comes with learned punishment. In a pre-registered randomised experiment in the Bertrand duopoly of Calvano et al., each agent's price is set by a tabular Q-learning module, not by the small language model attached to it, and we randomise whether each agent keeps its partner, sees its rival's prices and can send messages. Keeping the same partner raises the level of profits, averaged over training, by 0.27 of the gap between competitive and monopoly profit (95% CI 0.20 to 0.35, all twenty paired runs positive), our registered primary result, and the resting price by 0.17 of the Nash-to-monopoly range (post hoc). A plain tabular learner reproduces the effect in all 25 further blocks, and there one permanent partner raises the level more than about three do (+0.23 against +0.05, exploratory). Where rival prices are hidden, the price-setting module cannot see a cut, so cannot punish it, yet the resting price rises as much and the rise lasts to the end of training, while with visible rivals it shrinks with longer training (post hoc). Where the rival is visible, a static best responder accounts for a third to a half of what a forced-deviation probe reads as punishment, on the starts where the rival can see the cut, and net of it the registered test of learned punishment is inconclusive. A test that looks only for punishment would thus miss the rise where the rival is hidden, while a check for profitable deviations flags most of those prices (post hoc). In an exploratory extension, untrained Qwen2.5 7B and 14B models under one prompt show the effect when the rival's price is left out of the prompt and inconsistently when it is shown, the 7B result replicating on fresh blocks, while two other model families show none.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent reinforcement learning, algorithmic collusion, Q-learning, algorithmic pricing, multi-agent safety, emergent coordination, Bertrand competition, pre-registration, randomisation inference, language model agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YW6aOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper address a question of whether, when pricing agents are arranged by the platform to meet each other, the platform choice that decides which one faces other one matters in terms of deciding what agents may learn. The paper proposes a wide range of predictions and hypothesis, and through experiments it shows that nuanced results of punishment and price decision in pricing agents' behaviors.

Critical strengths

None

Critical weaknesses

The foremost issue is: this manuscript is very hard to read, and is a disservice to reader. The writing contains a lot of internal "lab process" that should not appear in a paper: To name a few, a huge number of labeled predictions and hypotheses, "registered", "post hoc". More extreme examples like "A simulated review of an earlier draft raised three objections", "An earlier version of this paragraph reported" are never something one should expect from a scientific paper. It's honest to show the AI usage (which suggest the writing heavily AI authored), but it is a quality problem to leave all such log level lab notes in a writing and expect a reader can extract what may be useful. Transparency is not a substitute for authoring a sounding writing.

The manuscript in its current form significantly makes it hard to assess the quality, and such an unpolished form merits a rejection on its own.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer YW6a

Thank you for the direct review. You are right. The main text read like a lab notebook, with prediction labels, the history of the analyses, and sentences like "an earlier version of this paragraph reported". None of this helps a reader judge the work.

We rewrote the main text around one line of argument: the question, the design, the main result, the case where punishment is impossible, and what follows from it. Sentences of that kind are gone from the paper. We kept one distinction: the four registered tests versus everything else. A reader needs it to judge the evidence, so we state the rule once, in Section 4. "Post hoc" and the prediction labels are gone from the main text. "Registered" appears only where it tells the reader what we fixed in advance. The pre-registration record and the dated list of deviations are in an appendix, for readers who want to audit the study.

We uploaded the revised PDF. We would be grateful if you could read it and tell us whether it is now possible to judge the work.

Reviewer_hDHeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether partner persistence changes the prices learned in a differentiated Bertrand duopoly. A preregistered factorial experiment varies matching, rival-price visibility and communication, using hybrid agents whose greedy prices are controlled by tabular Q-learning. Fixed partners increase checkpoint-averaged greedy-probe profit levels by 0.274 on the Nash-to-monopoly scale, and a separate replication with a plain tabular learner supports the effect across 25 additional blocks. The paper also proposes subtracting a static best-response benchmark from forced-deviation responses, although the registered test of the resulting enforcement contrast remains inconclusive. In hidden-rival conditions, the frozen greedy policy cannot respond to a rival's deviation, yet the persistence effect remains. Exploratory mechanism tests are partly consistent with spontaneous coupling without establishing it. An extension to prompted language models finds a replicated hidden-price effect for Qwen2.5-7B, while results with visible rival prices and other model families are mixed or null.

Critical strengths
  1. The main intervention result is consistent and supported by meaningful controls. All twenty registered fixed–shuffled contrasts are positive, and the plain tabular replication is positive in all 25 additional blocks (Table 1; Appendix F). The two-agent control and the matched ablation with a zeroed language-model head substantially reduce explanations based solely on the episode clock or policy-gradient update cadence. These experiments provide useful evidence that the structure of training interactions changes pricing outcomes in the studied system.

  2. The paper carefully distinguishes high prices from strategic enforcement. The hidden-rival frozen policy provides a particularly clear case: it cannot condition its actions on a rival's deviation, yet persistence increases its price level. The authors appropriately describe H6 as inconclusive, rather than interpreting non-rejection as evidence of no enforcement. They also report failed mechanism predictions, limitations of the swap test, and unsuccessful LLM replications. This careful separation of observations, mechanisms and generalisation makes the empirical contribution more informative.

Critical weaknesses
  1. The connection between the registered endpoint and sustained economic outcomes needs further validation. The primary level is measured after a 40-round greedy rollout from four grid initialisations, with learning frozen, rather than from prices and profits realised during training (Section 4). Appendix D reports cycling or large-drift sham trajectories on at least one start for 61% of fixed and 66% of shuffled probe points. Excluding starts with drift of at least eight bins leaves the contrast largely unchanged, which is reassuring, but does not establish robustness to cycle phase or to averaging over a sufficiently long evaluation window. The tabular replications use the same probe, so they do not independently resolve this measurement issue. This matters because the headline concerns the prices that the agents reach, and the abstract's “averaged over training” wording can be read as referring to realised outcomes.

  2. The finite-horizon conservativeness claim for the enforcement correction requires an additional assumption. Appendix H.1, lines 1398–1400, states that a non-negative unit-gain response kernel makes the truncated residual non-positive. This does not follow without a restriction on the sign of the reference-response path. For example, let mt=REmyopic(t)m_t=\mathrm{RE}_{\mathrm{myopic}}(t), use a one-round delay h1=1h_1=1, and set m1=−1m_1=-1, m25=1m_{25}=1, with all other terms zero and identical prehistory. At horizon 25, Emyopic=0E_{\mathrm{myopic}}=0 and E=1E=1, giving Enet=1>0E_{\mathrm{net}}=1>0 despite a matched, non-strategic, non-negative unit-gain response. The infinite-horizon identity remains correct. The distinction matters for the implemented 25-round probe, particularly when trajectories cycle. The authors should qualify the finite-horizon guarantee and separate it from the empirical calibration of the diagnostic.

Overall, I lean towards weak acceptance because the main persistence effect is consistently supported for the defined probe and the substantive conclusions are generally restrained. The outcome-validation gap and the finite-horizon error prevent a stronger recommendation.

Questions for authors
  1. Validation of the primary outcome: Does the fixed–shuffled contrast persist when prices and profits are averaged over complete cycles or a sufficiently long frozen-policy evaluation window, rather than read at the nominal rest point? Where existing logs permit, a comparison with realised training outcomes would also help. Please retain the preregistered outcome as primary and present these as sensitivity analyses. This would clarify whether the result reflects sustained pricing differences rather than sensitivity to the probe's readout.

  2. Scope of the enforcement guarantee: What additional path assumptions justify the finite-horizon sign claim in Appendix H.1, and do the recorded probe trajectories satisfy them? Please correct or qualify that statement and distinguish the valid infinite-horizon identity from the 25-round diagnostic. A sensitivity check using longer horizons on the existing frozen policies would help assess the practical size of truncation effects.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hDHe

Thank you for the careful review. Both of your technical points are correct. We checked them on the runs we already had.

1. Does the result depend on how the probe reads the level?

We checked this in two ways.

Prices played during training. The run logs record, at each of the eight checkpoints, the mean profit level over the previous 1,000 rounds, as actually played and with exploration. Averaged over the checkpoints, persistent agents reach 0.55 and rematched agents 0.31.

Full cycles of the frozen policy. In the trained agent, the value table sets the greedy price. So a frozen pair is a deterministic map on the last two prices. We replayed all 2,560 probe starts from the saved tables and got every registered value back exactly. After it settles, each pair enters a short cycle (median length 2, longest 6; a third are fixed points). The average over this cycle is the long-run average of the frozen policy. So the result no longer depends on the phase of the cycle or on the length of the window.

We used the same aggregation and the same test as the registered analysis:

outcomecontrast95% CIpositive
registered, rest point+0.274[+0.195, +0.350]20/20
played in training, 8 checkpoints+0.241[+0.190, +0.295]20/20
played in training, rival hidden+0.309[+0.265, +0.360]10/10
frozen policy, full cycle+0.263[+0.190, +0.330]20/20
frozen policy, full cycle, rival hidden+0.311[+0.270, +0.355]10/10

On the last 2,000 training rounds alone, the contrast is +0.33 [+0.20, +0.47], positive in 16 of 20. This estimate is noisier because it uses only one window.

You also noted that the tabular replication uses the same probe. The tabular learner is deterministic for a given seed. So we re-ran its training exactly (the Q tables match the saved ones at every checkpoint) and recorded the prices it played. On its 25 replication blocks, the contrast is +0.242 [+0.220, +0.260] on played prices and +0.293 [+0.260, +0.325] on full cycles. The registered readout gives +0.306 [+0.270, +0.340]. All three are positive in 25 of 25 blocks.

The registered outcome stays primary. The new results are in Appendix I as sensitivity analyses, and the scripts are in the supplementary archive. We also removed "averaged over training" from the abstract. It referred to the eight checkpoints of the frozen-policy probe, and it was easy to misread.

2. Appendix H.1 of the submission

Your counterexample is correct. A non-negative kernel is not enough. The reference path must also be non-negative at the end of the window. In our probe trajectories, it is negative somewhere in the window for 1,087 of 2,560 starts, so we cannot assume it. The revision states the finite-horizon result with both conditions and keeps the infinite-horizon identity separate (Appendix H, paragraph "Finite horizon"). It now presents the 25-round score as an empirical diagnostic, with no guaranteed sign.

We also ran the longer horizons that you suggested, on the same frozen policies. The punishment contrast (called the enforcement contrast in the submission) is +1.48 [-1.55, +4.85] at 25 rounds, +1.71 [-4.75, +8.45] at 50 and +3.37 [-8.8, +15.8] at 100. The smallest effect of interest is 4. The result is inconclusive at every horizon. The intervals get wider as the window grows, because cycling adds up in the window. So the longer windows cannot tightly bound the effect of the truncation, but they show no sign that it changes the verdict. The punishment hypothesis stays inconclusive.

Do these analyses answer your two concerns?

Thank you for the detailed response and the additional analyses.

The realised-training and full-cycle analyses substantially address my first concern. The reported persistence contrasts of +0.241 [0.190, 0.295] during training and +0.263 [0.190, 0.330] over complete frozen-policy cycles, together with the corresponding tabular replication, support the conclusion that the main effect is not an artefact of the original probe’s readout phase. I also appreciate that the preregistered outcome remains primary, that these checks are presented as sensitivity analyses, and that the potentially misleading wording in the abstract has been clarified.

On the second point, I appreciate your acknowledgement of the counterexample and your clarification that the finite-horizon sign guarantee requires an additional condition on the reference-response path. Presenting the 25-round score as an empirical diagnostic without a guaranteed sign, while keeping the infinite-horizon identity separate, addresses the issue I raised. I also agree with the cautious interpretation of the longer-horizon checks: the enforcement result remains inconclusive, and the widening intervals neither tightly bound truncation effects nor establish an absence of enforcement.

Based on the evidence reported in your response, my two technical concerns are substantially addressed. I maintain my weak-accept recommendation. The main persistence effect is now supported by more directly interpretable outcome measurements, while the limited scope of the experimental setting and the uncertainty about enforcement remain relevant to the overall assessment.

Thank you for taking the time to check the new analyses, and for your clear summary. We are glad that both technical points are resolved.

Reviewer_296bOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper looks at agents playing in the Bertrand pricing game following the setting of Calvanno et al. In particular, it focuses on the question of whether persistence with the partner changes the level of collusion between agents. They find that swapping out for random partners leads to significantly less collusion.

Critical strengths
  • I think that the goal of this paper is very timely and important for the community to grapple with
  • I believe the result that swapping out partners in the pricing game leads to less collusion
  • Experiments are conducted with multiple LLMs (although the selection is particularly old and small)
Critical weaknesses

While I am quite excited by work in this area, I find the overall conclusion of this paper to be extremely predictable and lacking relevance:

  • Collusion in the Bertrand game requires sustained coordination between partners, obviously if the partners are swapped out coordination needs to be re-established. So I think anyone familiar with this game would find this behavior predictable.
  • The authors are correct that sustained coordination with persistent partners is the dominant setting in the literature on this topic. However, what they don't even acknowledge is that this is because it is the most common setting in the real-world and what makes sense. The concept that the "platform" can control which agents are paired together is not motivated in the text by any examples or explanation. To me, as someone familiar with the literature, I cannot think of any examples where this is even an option. So I really need significant convincing about the real-world relevance.
  • The writing quality of this paper is extremely poor and resembles many of the worst characteristics of LLM writing. It randomly talks about "arms" just like Claude is prone to, it berates readers with low-level details without maintaining a clear narrative structure, and doesn't empathize with the information revealed to readers. A good example of the narrative structure point is the conclusion paragraph, which talks about low level details rather than summarizing the narrative. AI use aside, it is just very bad writing that makes this paper extremely difficult for humans to follow. A good example of the last point is the fact that the two hypothesis considered are called "H1" and "H6". I guess H2 through H5 were not interesting.
Questions for authors
  • Q1: Can you please provide some real examples that motivate where it would be possible to swap out partner agents in a real-world pricing game? What fraction of real-world pricing use cases do you believe this is relevant to?
  • Q2: With regard to the paragraph starting with "The two-part design follows a negative result" (line 209), why did you use these older models? It was established in [1] that CoT based model e.g. the Qwen model distillations of DeepSeek-R1 lead to out of the box collusion, which could have avoided some complicating factors in the experiments.
  • Q3: What made you choose the name H6 for the secondary hypothesis?

[1] Riemer, Matthew, et al. "Position: Collusion Risks Among AI Reasoning Agents Justify Certification Requirements for Making Market Decisions." Forty-third International Conference on Machine Learning Position Paper Track. 2026.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 296b (1/2)

(Part 1 of 2)

Thank you for the review. We answer your three weaknesses in order, and then Q2. We keep this reply short. The revised PDF gives the numbers behind each sentence.

1. "The result is predictable": coordination must be re-established

We tested this idea directly. It explains part of our results, but not all of them.

If coordination only has to be re-established, a change of partner should cost only the time to learn again. This is true. In the tabular learner, after one change to a new partner, prices recover within about 2,500 rounds. At the end, agents that changed are only 0.027 [-0.005, +0.060] below agents that stayed (new Table 2).

The same idea predicts that partners who never change cost nothing, because nothing must be re-established. This is false. We gave each agent a second, permanent partner in a quarter of its markets. The level fell by 0.169 [0.140, 0.195], about two thirds of the whole rise. One value table per partner recovers most of this loss, and agents with such tables keep the persistent level even under full rematching. So prices fall when one learner's experience is mixed across partners. A single change is cheap. These runs are exploratory and use the tabular learner. We replicated them on new blocks, with predictions written before the runs.

It is also not coordination by punishment. When the rival's price is hidden, the table that sets the price cannot see a price cut, so it cannot punish one. But persistence raises prices just as much there: +0.31 on prices played during training, in 10 of 10 paired runs, and +0.346 [+0.295, +0.395] on 25 new tabular blocks. We did not pre-register this split by rival visibility, and the paper says so.

We agree with part of your point. Learners that keep meeting may drift up together (spontaneous coupling, Banchio and Mantegazza, 2022). This is a form of coordination, and rematching breaks it. Our tests agree with this account, but they do not prove it. Also, the direction of the effect was not clear in advance. Weis et al. (2026) find that a diverse pool of co-players helps cooperation, and our registration named this as the competing prediction.

Our claim is narrow. To our knowledge, no earlier study of algorithmic pricing randomises whether agents keep or change partners during learning. And when the rival is hidden, an audit that looks only for punishment would miss the effect.

2. Relevance: where can partners change? (Q1)

We see three cases. In none of them did we test the effect in a real market.

Measurement. Simulation studies train the same learners together. In our experiment, this setup raises the measured level. So a reported level may partly come from the pairing.

Certification. Riemer et al., the paper you pointed us to, write that "agents must be evaluated against a diverse population of counterparts, including self-play and known collusive and competitive policies" (Section 7, arXiv v1). If agents keep learning during such a test, rotating counterparts may measure lower prices than one persistent rival would produce after deployment. This argument has limits. In an exploratory tabular test against that mix, the measured level is only 0.04 below that of a persistent pair. And for frozen agents, our result says nothing.

Platforms. Ad exchanges choose which bidders to call for each auction (Azari, Heavlin, Heidari, Lin and Todorova, 2017). On Amazon's marketplace, sellers that use algorithmic repricers compete for the same Buy Box, and the platform decides which offers are eligible (Chen, Mislove and Wilson, WWW 2016). These are only partial analogues. In both, an intermediary shapes which algorithms meet, but neither re-pairs rivals at random like our design.

We cannot give a fraction of real pricing cases, and we do not claim that it is most of them. When geography fixes the rival, this lever does not exist. The paper also states a limit that works against us: the effect needs one partner at a time, for several hundred rounds. A firm that prices several products with one learner, against a different rival for each product, already has several permanent partners, with no platform. In our tabular runs, this setup loses most of the effect, unless the learner can tell its rivals apart. Whether rivals can be identified is also a design choice. There is a precedent with human dealers: Simaan, Weaver and Whitcomb (Journal of Finance, 2003) argue that showing dealer identities in Nasdaq quotes mattered for implicit collusion. The revised introduction now opens with the platform examples, and Section 8 states these limits.

AuthorsOpenReview ↗

Response to Reviewer 296b (2/2)

(Part 2 of 2)

3. Writing, and "H6" (Q3)

You were right about the writing. We rewrote the main text around one line of argument, and a new Figure 1 shows the design and the main result. The word you objected to is gone. The conclusion now sums up the argument. The hypothesis numbers came from the pre-registration. There, we made some hypotheses exploratory or postponed them before any data came in. The paper now names the two hypotheses by what they test: the price hypothesis and the punishment hypothesis.

4. The models, and the role of the language model (Q2)

This is the weaker part of the paper, and the revision says so. In the trained agent, the value table sets the price. We tested one condition (rival visible, no messages). There, zeroing and freezing the language model's head leaves the contrast unchanged (+0.204 against +0.206). The contrast also stays positive when the language model can overrule the table in 49% and then 79% of settled states (+0.198, then +0.159). So the result is about learning agents in a market where someone designs the matching. The backbone is small (0.5B) because we trained 40 pre-registered runs with reinforcement learning.

We ran the models you suggested, in the same prompted design with the rival's price hidden. We wrote our predictions before the runs. DeepSeek-R1-Distill-Qwen-7B sets high prices from the start, as Riemer et al. report: about 0.70 of the way from competitive to monopoly profit. But it does not show the persistence effect (-0.011 [-0.090, +0.070]). We had predicted an effect for this model, and it did not appear. DeepSeek-R1-Distill-Llama-8B gives -0.275 [-0.520, -0.040], in the other direction. The token budget often cut their reasoning short. With a larger budget, the effect does not appear either (Appendix E). Among untrained models, only the Qwen2.5 instruction-tuned models show the effect. So the effect is not universal across agents, and the abstract now says this.

If the revision answers these points, we would be grateful if you would reconsider your score.

Reviewer_296bOpenReview ↗

Re: Response to Reviewer 296b (1/2)

Thank you for engaging with my review, but I found your reply confusing. I still am really struggling over any valid use cases. It would only be useful for certification if it resembled a real-world use case. The main one you bring up is for the controllers of platforms instituting this. However, this doesn't make sense to me because I can't see the logic either from the platform perspective or consumer perspective. From the platform's perspective driving down prices is not something they would chose to do unless forced to by the public / government. Moreover, restricting the set of competitors to a subset of the full set is a discriminatory practice that could be viewed as unethical even if prices are lower. For the public / government it is also unclear that they would prefer this alternative because the comparison isn't really between transient or persistent restricted competition, but rather between transient restricted competition and persistent unrestricted competition.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: what changed in the revision

Thank you all for your reviews.

Writing (296b, YW6a). Two of you found the paper hard to read. We agree. The main text had too much protocol detail: prediction labels, hypothesis numbers, and the history of the analyses. We moved this detail to the appendix and removed nothing. The main text now tells one story. When learning agents keep the same partner, they learn higher prices. This also happens when an agent cannot see its rival's price, so it cannot punish a price cut. A new Figure 1 shows the design and the main result. A new Table 2 shows what a change of partner costs. The revised PDF is online.

New analyses (hDHe, 296b). Each reply gives the details. In short:

  • The main result also holds on the prices that the agents played during training, not only on the probe of frozen agents.
  • The finite-horizon claim in Appendix H.1 of the submission was wrong. We corrected it.
  • New tabular runs show how much persistence the effect needs: one partner at a time, for several hundred rounds. The paper states this as a limit.
  • In exploratory tabular runs, replicated on new blocks, one change of partner costs little. A second permanent partner in a quarter of an agent's markets removes about two thirds of the rise. Agents that keep one value table per partner keep the high prices even under full rematching.
  • We ran two DeepSeek-R1 distilled models, with our predictions written before the runs. They do not show the effect, also with a larger reasoning budget. The paper reports this (Section 7, Appendix E).

We post now so that we have time to discuss with you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.