acceptodds
Withdrawn from ICLR 2027

Restoring Co-Adaptation: An Endpoint Audit of Asymmetric Training in Multi-Agent Market Making

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When an endpoint produced by asymmetric multi-agent training is evaluated with all policies frozen, the evaluation disables the learning responses whose suppression helped shape that endpoint. We distinguish frozen persistence from algorithm- and horizon-relative co-adaptive response stability and introduce a restoration audit: resume all suppressed update channels from the learned endpoint and compare the resulting system-statistic change with a same-budget continuation from a co-adaptive reference. In a controlled three-agent market-making testbed, staged train-and-freeze training increases mean minimum quoted spread (MQS) relative to simultaneous co-adaptation by (95% CI ; 20/20 seeds positive). After 600 restored episodes, the mean recovery fraction is (95% CI ), whereas the matched simultaneous continuation drifts by only (95% CI ). A factorial audit shows that the tested spread shift is associated with the interaction of long active-agent blocks and elevated placeholders (primary interaction , 95% CI ; 20/20 seeds positive). The disadvantage follows the first-trained role across identities, and Independent DQN learners reproduce the prespecified block, placeholder, and restoration contrasts in the same testbed. Restoration is not an equilibrium or exploitability test; within this controlled setting, it shows that frozen persistence can overstate stability under the specified resumed-learning dynamics.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, evaluation methodology, co-adaptation, training-path dependence, non-stationarity, response stability, market making

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 2 reviews, above 0% of the others.
Reviewer_jFyJOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work considers in multi-agent setting for market, asymmetric training evaluation may have issue with agents are unfrozen. This work prosed to model this by introducing a restoration audit with all update channels are resumed. In a controlled testbed the author shows, with careful statistics, that the audit does show some traits for this particular cases, including testbed shift and the overstated stability for frozen persistence.

Critical strengths

This work introduces an clean separation conceptually: frozen persistence and co-adaptive response stability. Such an idea is neat and should be encouraged.

The experiment design is clean, rigorous, and comes with with strong statistics results. This is really good.

Critical weaknesses

That said, this work also have issues that prevents acceptance in its current form.

For presentation, this work seems to be hard to follow given the high density of information (which are sometimes not necessary). The writing requires significant effort in parsing.

The proposed method is for a widely acknowledged issues, but without proper comparison with prior works addressing such issues. As the authors admitted, issues with frozen evaluation is a consensus in multi-agent RL. There are a range of prior works dealing with that. This paper mentioned some of them (Shoham et al., 2007, Foerster et al. 2018(the manuscript mistakes it for year 2017), Lanctot et al. 2017, and Timbers et al. 2022), but only briefly in discussion without actual comparison the proposed method with them. This missing strongly limits the contribution of this work.

The proposed method's generalizability is questionable. No cross domain analysis beyond the toy testbed environment is conducted. To justify the proposed method as general, at least some cross domain experiments are expected. Furthermore, while the manuscript is good at establish statistically strong estimation, it does not produce an analysis of mechanism. Prior works like HAPPO/A2PO (the manuscript cites them) already show some insights into the mechanism, thus we expect this work should also provide insights to some degree.

Domain (market-making) specific contribution is also limited. Many market mechanisms important for studying market making, such as those related to limited order book, are missing from the testbed. This does not really handles market making well from first principle. Furthermore, although this work identifies widening spreads, it does not show from a market making point of view who benefit from the spread, i.e. the welfare, an import topic for market making.

Questions for authors

None

Reviewer_P85AOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at a problem with asymmetric multi-agent training. When some agents are frozen during training, the final policies may look stable when evaluated together, but this may change once all agents start learning again.

The authors propose a restoration audit, basically resuming joint learning from the trained policies and checking whether the observed outcome still holds. They test this in a three-agent market-making environment. Staged training leads to wider quoted spreads compared to simultaneous training, but the difference almost disappears after restoring joint learning. There are also experiments on block lengths, placeholder policies, agent order, and a DQN version of the setup.

Critical strengths
  1. I think the distinction between a frozen outcome and what happens when agents start adapting again is a useful point. In MARL, we often evaluate a collection of trained policies as if the other agents will just stay fixed. That is not always a good assumption, especially when the training itself involved freezing some of them. The paper makes this issue quite concrete.

  2. The experiments are fairly careful for what the authors want to show. In particular, they compare restoration with continued simultaneous training, rather than just showing that the staged outcome changes after more training. The results also seem quite consistent across seeds, and the DQN experiments give some additional support. So I don't have major doubts about the main observation in this particular environment.

Critical weaknesses
  1. Limited insight beyond the specific setup.

The results are convincing within this particular setup. But I am less sure what we should learn from them beyond this example. Is this something we should generally expect from asymmetric training? Or does it depend on the particular game, the learning algorithm, or the way the training stages are arranged?

The paper does not really explore these questions. The DQN experiment is useful, but it still uses the same environment and essentially the same learning problem. I would have liked to see more discussion of when this phenomenon should happen and when it should not. Right now, it feels more like an interesting observation about this training setup than a general insight into multi-agent learning.

  1. The main conceptual contribution feels somewhat limited.

I also have some doubts about how much is really new in the proposed restoration audit. That agents may behave differently when other agents start learning again is already a fairly basic concern in multi-agent learning. The paper gives this a clearer evaluation framing, which is useful, but the actual procedure is essentially to resume training and observe what changes.

The authors distinguish this from exploitability and best-response evaluation, and I agree that these are different questions. But I am not sure this alone establishes a substantial new evaluation methodology. For example, does this audit tell us something that we would otherwise miss when evaluating existing MARL methods? Would it change which algorithm we consider better? The paper does not really show that. So while I find the message reasonable, I am less convinced by the significance of the contribution.

Questions for authors
  1. Do the authors have any intuition about what properties of a game or learning process make this restoration effect more likely? I understand that the paper is not claiming generality, but some understanding of why this happens would make the result more interesting beyond the market-making example.

  2. What would be a concrete case where the proposed audit changes our assessment of a MARL method? I think this would help clarify its value as an evaluation tool, rather than simply a way of showing that policies can change under further joint training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.