acceptodds
Under review as a conference paper at ICLR 2027

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets

Abstract

Power markets are a natural testbed for multi-agent reinforcement learning (MARL), where multiple self-interested participants repeatedly submit bids. A market-clearing mechanism then determines dispatch and prices subject to power grid constraints and market settlement rules. However, existing MARL environments typically focus on a single market setting, implement simplified clearing mechanisms, or rely on CPU-based optimization solvers that slow large-scale training and limit the systematic study of bidding strategies and market behavior. We introduce PowerMarketJax, a benchmark suite for MARL across five power markets: day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer double auctions, and local flexibility. Each environment implements its own clearing, pricing, and settlement rules while providing a common framework for learning and evaluation. We find that learned bidding behavior depends strongly on the market design: independent learners can miss better strategies when gains require many agents to change together, when more profitable strategies lie beyond a region of lower profit, or when profits disappear as more agents adopt the same strategy. PowerMarketJax implements both market simulation and policy training in JAX, allowing the entire pipeline to run on the GPU with 1 024 1 200 parallelisms across both environments and market participants, achieving up to 33 speedup over CPU-based baselines. Our open-source benchmark is available at: https://github.com/powermarketjax/PowerMarketJax.

Primary area:
datasets and benchmarks
Keywords:
Power Markets, Multi-Agent Reinforcement Learning, JAX, Benchmark Environment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fkEaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PowerMarketJax implements five electricity-market environments and independent PPO/SAC learners in JAX. The suite combines bidding, market clearing, settlement, and policy updates on the GPU. The authors validate clearing against reference implementations and an independent solver, measure runtime, and examine coordinated markups, crowded battery arbitrage, and manipulation of flexibility demand.

Critical strengths
  1. Moving constrained market clearing into the same execution pipeline as learning addresses a real bottleneck. The released suite covers markets with different settlement rules and includes detailed configurations. Its engineering contribution is supported by reference checks across all five markets, price comparisons with HiGHS, and a measured end-to-end training speedup in the balancing market. Together, these make a credible case that the suite can lower the cost of studying strategic behavior in electricity markets.

  2. The analysis goes beyond reporting agent returns. In the flexibility market, disabling the contribution of strategic charging to the procurement baseline largely removes the excess profit. That intervention helps explain what the agents have learned and illustrates how the suite could reveal unintended incentives in a market design.

Critical weaknesses
  1. [Major] Strategic fidelity remains uncertain for the day-ahead commitment approximation. This market rounds a relaxed commitment solution before solving dispatch. Appendix D.4 reports modest production-cost gaps against exact commitment across three networks under truthful bids, which is useful evidence. However, strategic bids can change both commitment and the marginal generator, and a small cost gap does not establish that profitable deviations are preserved. I would like a targeted check of learned bids against exact commitment, followed by dispatch and settlement under the stated pricing rule, comparing feasibility and individual profits. This limits confidence in the day-ahead strategic conclusions, although it does not invalidate the other market environments or the demonstrated computational benefit.

  2. [Minor] The interpretation of coordinated profit gains needs a clearer connection to the learning objective. In the day-ahead experiment, the learned outcome resists the tested unilateral deviations, yet a more profitable joint markup is presented as something independent learners miss. These agents optimize their own profits; a coordinated increase in producer profit is not automatically a better solution to that game. The observed difference between joint and unilateral incentives is informative, but should not itself be treated as a failure of learning. Separating these criteria would correct the interpretation without changing the experimental result.

Questions for authors

See weaknesses.

4 comments

W1 Response:

Thank you for this comment. We agree that the relaxed commitment may change the results. To test this, we add new experiments in Appendix E.2. We take the strategic bids learned from IPPO-PS and IPPO-NoPS on case29gb (all 10 training seeds × 36 evaluation days), and clear them with our relaxed method and the mixed-integer linear programming (MILP) solved by HiGHS.

For each bid profile, both methods use the same strategic bids and market clearing rules. Commitment, dispatch, prices, and profits are recomputed separately for each method. Detailed results are provided in Appendix E.2, with the key findings summarized below.

  1. Feasibility: all evaluated instances are feasible without load shedding under both methods.
  2. Cost and prices: across the 10 seeds, the median daily production-cost gap is 1.3–2.8%; and the median relative hourly difference in load-weighted LMP is 0–1.1%.
  3. Profits: Profit rankings among the dispatched generators are strongly correlated between the two methods (Spearman ρ = 0.90–0.98). For IPPO-NoPS, our relaxed method yields revenue, cost, and profit of 2,142, 1,588, and 553 (£ million over the 36 evaluation days), compared with 2,236, 1,561, and 674 under MILP. The corresponding absolute relative errors are 4.20%, 1.73%, and 17.95%, respectively.

In summary, the gaps between the relaxed method and the MILP are acceptable, because it does not affect feasibility, changes prices only slightly, and leaves the profit ranking almost unchanged. This means the qualitative results and our conclusions are kept the same.

Following your comment we have:

  1. Added the comparison between our relaxed method and exact MILP in Appendix E.2 with results of feasibility, prices and profits.
  2. Added and updated two summary tables (Tables 6, 7) of all fidelity results in Appendix E.
  3. Updated the Limitations in Section 6 to discuss the approximation's effect on clearing accuracy and future JAX-based solvers that directly handle binary variables.
  4. Added references showing that relaxing commitment binary variables is common practice in Appendix C.1.1, both in unit commitment studies (Morales-España et al., 2013; Gómez et al., 2026) and in practical market clearing, such as the integer relaxation method used by PJM for fast-start pricing (FERC, 2019).

Reference: Morales-España et al. "Tight and compact MILP formulation for the thermal unit commitment problem." IEEE Trans. Power Syst., 2013. Gómez et al. "Relax-and-round strategies for solving the unit commitment problem with AC power flow constraints." Optim. Eng., 2026. FERC. PJM Interconnection, L.L.C.: Order on Paper Hearing. Docket No. EL18-34-000, 2019.

W2 Response:

We agree, and we thank the reviewer for pointing this out. This is exactly what we intended to say. Each agent (participant) is trained to maximize its own profit, and the learned policies resist the tested unilateral deviations. The joint markup is a case that we construct with human knowledge, in which all generators raise their bids together (collusion). It is not an outcome that self-interested learners are expected to reach.

In the original manuscript, our wording suggested that the independent learners miss a better solution because they do not reach the higher profit of the joint markup. This was a problem of expression in our writing. We did not intend to describe it as a failure of learning.

We appreciate this suggestion, which helps us state the point clearly. Following your comments, we have:

  1. Revised Section 5.1 to clarify the individual learning objective and distinguish the tested unilateral deviations from the joint markup, which is only shown as a reference.
  2. Revised the conclusion in Section 6 to remove the interpretation that not reaching the higher-profit joint outcome represents a failure of independent learning.

Dear Reviewer fkEa,

Thank you again for your time in reviewing our work. We’d like to check if our rebuttal has clarified your concerns. If there are still any unclear parts in our work, please let us know and we would be happy to engage further before the discussion period closes.

Thanks for your reply. I will maintain my positive score.

Dear Reviewer fkEa,

Thank you for your positive feedback and support. We sincerely appreciate your time and constructive comments.

Best regards, All authors

Reviewer_pYSsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper attempts to provide a complete model of the power market as partially observable stochastic games (POSGs) to be used as a multi-agent reinforcement learning (MARL) benchmark. The model is complete in the sense that all levels of the market are taken into account (M1-M5: day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer double auctions, and local flexibility) rather than just a sub-set. The benchmark simulation and learning the five distribution levels is all developed in a single execution pipeline/graph in Jax for 33 fold speedup compared to cpu.

In the experiment section, a mix of limited datasets are used at different levels where various gains and effects are stated and market shaping effects are sought explained.

Critical strengths

It may be a good start and direction for an efficient market simulator with a lot of potential, but if it is a significant simplification and to that uses arbitrary collection of datasets the stated utility as is does not convince.

Critical weaknesses

Real data is used and bids on one level affect prices in the next, however the data used at different levels is a combination of British, Belgian and Swiss datasets of different magnitude and scope.

It is very doubtful the outcomes of these bidding simulations have anything to do with a real power grid and this reviewer was really looking forward to a measure of how well the simulation reproduces actual market prices and dynamics.

Authors state the following under limitations: "the five environments simplify aspects of real market operation, network modeling, participant constraints, and market rules". But the goal of modeling all five levels was to have a realistic market, so with the stated limitations this was not achieved.

In some experiments 3 random seeds are used, others 3-10. This seems quite arbitrary and definitely not enough.

The authors demonstrate that if all bidders join forces they would be able to sell at alpha=2 times their marginal cost, but that is highly unrealistic and of limited interest for a strategy.

Questions for authors

Why does the curve of IPPO-PS in Figure 3 look so differently and why wouldn't the agents learn to find a more similar final bidding strategy?

4 comments

W1 Response:

We thank the reviewer. PowerMarketJax spans generation, transmission, distribution, and end users. Ideally, all five markets would use data from one country, including bids, prices, networks, and constraints. However, countries may not operate all five markets or release all relevant data. For example, Great Britain (GB) lacks a large-scale P2P market, while household smart-meter data face privacy restrictions.

We therefore build the benchmark mainly on European systems with similar market structures. The three closely coupled transmission-level markets (M1–M3) use GB data, M4 uses smart-meter data from 1,200 Belgian households, and M5 uses the 129-bus SwissDN feeder. Researchers can also study a consistent subset within one country, such as M1–M3 in GB.

In the revised manuscript, we have:

  1. Explained market coverage and data limitations at the end of Section 3, and clarified that PowerMarketJax supports studying subsets of the five markets.
  2. Clarified the European test systems and each market’s data source in Section 5.

W2 Response:

Thank you for the suggestion. We have added comparisons between simulated and historical prices in Appendix E.3 (Fig 13):

  1. Australian NEM: Using the historical offers from 151 generators, the Spearman correlations with historical prices are 0.60 for daily means and 0.75 across 8,760 hours (Fig 13a).
  2. GB: The simulated prices track the GB Market Index Price, with Spearman correlations of 0.81 for daily means and 0.69 across 8,640 hours (Fig 13b).

These comparisons support the benchmark's ability to capture broad seasonal and daily price patterns, although the magnitudes of individual peaks and troughs can differ.

W3 Response:

Our focus is short-term electricity markets, including participant bidding, clearing, and settlement under network and time-coupling constraints. This allows us to study how outcomes change as more participants adopt learning-based bidding. Other market and operation aspects, such as long-term contracts and transient power-system dynamics, are outside this scope.

The key question is whether these simplifications preserve the mechanisms being studied. We have clarified two assumptions:

  1. Network. We use steady-state network approximations, including DC power flow for transmission markets, which is the standard model in market clearing.
  2. Settlement. We omit uplift payments and taxes, which are decided by regulators.

Our findings are conditional on these assumptions. We have revised the Limitations in Section 6 and added comparisons with historical prices in Appendix E.3, as detailed in our response to W2.

W4 Response:

Thank you for the suggestion. We have rerun all M1–M5 experiments with 10 random seeds per learner and updated the corresponding results (Section 5, Table 9). We show 95% confidence interval across seeds as shaded bands (e.g., Fig 3). With 10 seeds, the main findings remain unchanged, while the reported numbers are updated.

W5 Response:

We agree. The joint markup is a manually specified collusive reference in which all generators raise their bids together. We use it to distinguish joint from individual incentives, rather than as a learned strategy or learning target.

In the revised manuscript, we have:

  1. Clarified in Section 5.1 the role of the joint-markup reference.
  2. Revised Section 6 to distinguish higher joint profit from the agents' individual learning objective and remove its interpretation as a learning failure.

As Reviewer fkEa also noted, each learner maximizes its own profit, so the relevant question is whether a generator can gain by changing only its own bid (details can be found in our response to Reviewer fkEa's W2). Our main result addresses this question. For example, in M1, when one generator raises its bid above the learned level (α ≈ 1.5) while the others keep their learned bids, its profit falls (Fig 3(b)). The learned policies are therefore close to the best each generator can do alone, and the M1 design does not reward raising a bid alone (This also shows that the current M1 market design is effective in this respect.)

Q1 Response:

The IPPO-PS curve differs because all 66 generators share one policy network. Their gradients are pooled, so early updates move all markups up together and profit rises, as under a joint markup. Each generator's own-profit signal then favours a lower bid (Fig 3(b)), and the shared markup returns to about 1.5. With agent-specific networks (NoPS), each generator follows its own gradient and this rise-and-fall curve does not appear.

Regarding the final strategy, uniformly high markups would require collusion among generators. However, this is unlikely under independent learning. If all generators bid high, each has an incentive to bid slightly lower to increase its chances of being dispatched for securing sales, which pull prices down. This is consistent with Section 5.1 and our responses to W5 above and W2 of Reviewer fkEa.

Dear Reviewer pYSs,

Thank you again for your time in reviewing our work. We’d like to check if our rebuttal has clarified your concerns. If there are still any unclear parts in our work, please let us know and we would be happy to engage further before the discussion period closes.

Thank you for your clarifications and taking the comments into account.

Stating that the proposed work is a research benchmark does not excuse that fact that model dynamics do not represent real interactions between different layers and that agents can unite and drive the benchmark to unrealistic states, etc.

As these fundamental weaknesses are still inherent in the model and mixed reduced geographic dataset still remain is not enough for this reviewer to change the final grade.

We thank the reviewer for further comments, but we respectfully disagree with the concerns about interaction between different layers. The sequencing and integration of electricity markets depend on the market design. Separating all markets is one extreme, and coupling all markets tightly is another. The reality sits between, which is what we model in this paper. M1–M3 are like the New York Stock Exchange, and M4–M5 like local supermarkets and farmers' markets. They are related, but a change on the exchange does not quickly reach our lunch price. Our model dynamics represent the real interactions between layers: M2 and M3 use the M1 commitment and schedules, and M4 and M5 use the M1–M2 prices as reference. In practice, layers are linked but cleared separately:

  • M1 vs. M2. At the ERCOT (Texas, USA) morning peak on 15 January 2025, the day-ahead price was about USD60/MWh, but the real-time price was about USD2,050/MWh [1].
  • M4–M5 vs. M1–M2. Refs. [3]-[4] clearly state that M4 P2P trades are "within the same balance group, keeping the balance position of the BRP unchanged", and M5 local flexibility market "could also remain separate markets".

[1] GridStatus, "Locational Marginal Price – ERCOT," https://www.gridstatus.io/live/ercot?date=2024-01-13to2024-01-17 (free login required; screenshot provided in [2]). Accessed 18 November 2026.

[2] https://raw.githubusercontent.com/powermarketjax/PowerMarketJax/9b85fe43ba30668ff2c83ee763ba14ab2bf1edbe/Material/ercot_hb_north_lmp_2025-01-13_to_16.png

[3] Pressmair, G., et al. (2021). "Overcoming barriers for the adoption of Local Energy and Flexibility Markets: A user-centric and hybrid model." Journal of Cleaner Production, 317, 128323.

[4] Leonardo Meeus (2020). "The Future of Electricity Markets with Distribution Constraints." Oxford Energy Forum, Issue 124, pp. 21–24.

If you still have concerns, please feel free to let us know. Thank you.

Reviewer_JzcuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PowerMarketJax, a multi-agent reinforcement learning benchmark for electricity markets based on JAX, covering five types of electricity markets. This framework integrates market clearing and strategy training into GPU, and analyzes bidding behavior under different market mechanisms through IPPO and SAC. The experiment demonstrated the improvement of computational efficiency and the learning behavior of agents under different market mechanisms.

Critical strengths
  • Efficient and unified MARL experimental framework for the electricity market.
  • The paper integrates five types of electricity markets into a unified JAX framework, while retaining their respective market clearing and settlement mechanisms.
  • The overall workload is relatively large, covering various market modeling, GPU acceleration implementation, and comprehensive experimental analysis. This framework significantly improves the efficiency of environmental interaction and provides a practical tool for large-scale MARL research.
Critical weaknesses
  1. Insufficient novelty and necessity of the benchmark. There have been works such as SustainGym, ASSUME, OpenGridGym, and MARLEM that support reinforcement learning, physical constraints, or various market mechanisms in the electricity market. The main new contributions of this article are focused on the integration and computing acceleration of JAX, but it is not yet clear which important research needs cannot be met by existing platforms.

  2. Lack of strong validation of the unique research value of benchmarks. The paper has not yet clearly demonstrated which important research that the benchmark can support is difficult to accomplish on existing platforms. The experiments mainly use independent PPO and SAC, lacking stronger MARL baselines and targeted evaluation, making it difficult to determine their actual driving effect on MARL research.

Questions for authors

1.Compared to existing MARL platforms such as ASSUME, OpenGridGym, and MARLEM, what important MARL research can PowerMarketJax support that is difficult to conduct on existing platforms?

2.In addition to computational acceleration, can you further validate the evaluation value of this benchmark in issues such as coordinated learning through a broader comparison of MARL algorithms?

2 comments

Q1 Response:

Thank you for this comment. We agree that existing platforms support RL and physical constraints, but generally focus on selected market settings. SustainGym's market task uses one learning battery; ASSUME supports adaptive bidding in day-ahead markets; OpenGridGym studies distribution markets; and MARLEM focuses on local-market cooperation.

PowerMarketJax provides broader coverage through five representative markets: day-ahead wholesale, real-time balancing, ancillary services, P2P trading, and local flexibility. These span transmission and distribution levels, retain their own clearing and settlement rules, and share interfaces, PPO/SAC baselines, and evaluation procedures.

First, these tasks are needed because market rules change which strategies are profitable. In our experiments, higher bids can increase day-ahead prices and profits. In balancing, however, higher offers can cause generators to lose dispatch to competitors and earn less. Performance in one market therefore does not establish performance in another.

Second, these tasks are needed to examine how a strategy’s profitability changes as more participants adopt it. In our 1,200-household P2P case, a charging/discharging schedule becomes unprofitable at around 210-240 adopters. Simultaneous charging raises purchase prices, while simultaneous discharging lowers sale prices. This feedback cannot be established through fixed-price or single-participant evaluation.

These examples show why the benchmark needs both different market rules and interacting participants. Existing platforms might be extended to reproduce individual tasks, but do not directly provide this five-task suite with common baselines and evaluation code. PowerMarketJax makes these comparisons available without requiring researchers to rebuild each market.

Finally, PowerMarketJax combines five market tasks with JAX-based market clearing and learning on the GPU, enabling rapid, large-scale MARL experiments. Researchers can train and compare bidding policies under different market rules and agent populations, identify which MARL methods work well in each market, and develop new MARL policies suited to different market mechanisms. This supports faster development and systematic evaluation of new MARL methods and bidding strategies for electricity markets.

PlatformMarketsFeatures
SustainGymReal-time energyOne learning battery; network-constrained clearing.
ASSUMEDay-aheadAdaptive bidding.
OpenGridGymDistributionCompeting agents; interchangeable market and grid models.
MARLEMLocal energyStorage participation; support for multi-agent cooperation.
PowerMarketJaxDay-ahead, balancing, ancillary services, P2P, and local flexibilityCommon MARL benchmark; GPU-based clearing and learning.

Following your comment we have:

  1. Clarified the benchmark's novelty and necessity in Section 2
  2. Added a comparison table in Section 2 summarizing market coverage and main features
  3. Clarified in Section 2 how GPU execution supports rapid, large-scale experiments
  4. Explained how market rules and strategy adoption affect profits in the Introduction section

Q2 Response:

We thank the reviewer for this suggestion. We have added a coordinated learning benchmark and the corresponding evaluation.

First, we broaden the comparison of MARL algorithms with MATD3. MATD3 learns in a coordinated way through centralized critics that use information from all generators during training, while each agent still maximizes its own profit. In M1, MATD3 reaches a profit similar to that of the independent PPO and SAC learners, with more variance across seeds. The result shows that coordinated learning does not lead to higher profit.

Second, we test the learned bids against two types of change. 1) When one generator changes its individual bid by any learning method, it cannot exceed the best learned policy. 2) When all generators raise their bids together, total profit reaches £697M (at α = 2). As Reviewer fkEa mentions, the individual learning strategy cannot learn this, because each agent optimizes its own profit. A coordinated increase in producer profit is not a better solution for these agents, and not reaching it is not a failure of learning.

Third, following Reviewer pYSs's comment, all experiments now use 10 random seeds, so the results are more reliable for both the independent learners and the new MATD3 baseline. With these additions, the results show more clearly what the benchmark can evaluate beyond a learning curve: whether the learned bids are the best each agent can do on its own, and whether a market design discourages collusion.

Following your comment we have:

  1. Added MATD3 as a coordinated MARL baseline in Section 5.1
  2. Replotted Fig. 3(a) by adding its learning curve
  3. Updated all experiments with 10 random seeds
  4. Revised Section 5.1 to explain the algorithm comparison and bid-deviation results

Dear Reviewer Jzcu,

Thank you again for your time in reviewing our work. We’d like to check if our rebuttal has clarified your concerns. If there are still any unclear parts in our work, please let us know and we would be happy to engage further before the discussion period closes.

Comments on the paper

1 comment

General Response

We thank all reviewers for their helpful comments. All the review comments have been address in seperate response and all the revised texts in manuscript have been marked in blue.

The main revisions are summarized below:

  1. Novelty and research value (Reviewer Jzcu): Added a comparison with existing platforms (Section 2), and clarified why both different market rules and interacting participants are needed (Section 1).
  2. MARL baselines (Reviewer Jzcu): Added MATD3 as a coordinated learning benchmark, together with an evaluation for bid deviation (Section 5.1).
  3. Interpretation of the joint markup (Reviewers fkEa, pYSs, Jzcu): Clarified that the joint markup is a collusive reference, not a learning target, and such that not reaching it is not a failure of learning (Sections 5.1 and 6).
  4. Fidelity (Reviewers fkEa, pYSs): Compared market outcomes of our relaxed commitment method with the exact MILP under learned strategic bids (Appendix E.2), and compared simulated prices with historical NEM and GB market data (Appendix E.3).
  5. Data and scope (Reviewer pYSs): Clarified the data source for each market and the modeling assumptions (Sections 3, 5, and 6).
  6. Random seeds (Reviewer pYSs): Re-ran all experiments with 10 random seeds. The main findings are unchanged (Section 5).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.