acceptodds
Under review as a conference paper at ICLR 2027

Learning Stock Trading Policies via Barycenter-Based Adversarial Inverse Reinforcement Learning

Abstract

Designing effective trading strategies using reinforcement learning remains challenging due to delayed and noisy rewards, poor exploration, and the difficulty of enforcing explicit risk constraints. In this work, we propose BRaG, a barycenter-based adversarial inverse reinforcement learning framework for stock trading that learns trading behavior from multiple heterogeneous expert strategies. BRaG aggregates expert demonstrations using a performance-weighted Wasserstein barycenter, yielding a stable pseudo-expert representation that captures shared structure across diverse trading styles. This representation is used to pretrain a trading policy via adversarial imitation learning, which alleviates unstable exploration during reinforcement learning. The pretrained policy is subsequently refined using reinforcement learning with true market rewards. To ensure risk-aware decision-making, BRaG incorporates control barrier functions that constrain action execution and regularize policy learning to satisfy drawdown limits. We evaluate the proposed approach on four major global equity markets, including the US, UK, Indian, and Taiwanese indices. Across all the markets, the proposed approach achieves stronger performance than both classical trading rules and recent deep reinforcement learning methods, while exhibiting more stable risk characteristics.

Primary area:
reinforcement learning
Keywords:
Inverse Reinforcement Learning, Stock Trading, Risk-Aware Reinforcement Learning, Multi-Expert Learning, Quantitative Finance

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VbWBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

BRaG combines performance-weighted expert aggregation, GAIL pretraining, PPO fine-tuning, and a control-barrier action filter. The expert target is constructed from projected state-action samples. Experiments on four equity markets report higher returns and Sharpe ratios than the evaluated baselines, with additional component and aggregation ablations on DJI.

Critical strengths
  1. The evaluation covers four markets, includes transaction costs, and reports five-run variability in the appendix. The return and Sharpe improvements are consistent across the four reported test settings.
  2. The DJI ablations include useful controls: Sharpe-weighted sampling without a barycenter, uniform expert sampling, and removal of pretraining, fine-tuning, or the barrier filter. These comparisons address the main components more directly than an aggregate comparison with trading baselines alone.
Critical weaknesses
  1. The central barycenter construction is not sufficiently specified or justified. Equation (5) gives separate one-dimensional quantile averages, but Section 4.2 only describes averaging and back-projecting them to obtain joint samples. It does not establish that this reconstruction solves either Equation (4) or a defined sliced objective. Clipping actions against cash and holdings does not ensure that reconstructed market states and state-action distributions are reachable under the trading dynamics. This matters because the reconstructed distribution determines the GAIL reward.
  2. The stated drawdown enforcement is stronger than the method supports. The rolling estimate in Equation (12), with κ = 1 in Table 2, is not a guaranteed lower bound on future returns. Satisfying Equation (15) for predicted wealth therefore does not ensure compliance for realized wealth. The paper reports drawdown improvements, but does not report violation rates against the chosen limit; the main experiment's drawdown limit is also missing from Table 2.
Questions for authors
  1. What is the exact reconstruction algorithm, including feature scaling and the treatment of inconsistent projections? How is the resulting target checked for compatibility with reachable states and policy occupancy?
  2. What drawdown limit is used in each main experiment, how often is it exceeded, and what happens when the action projection has no feasible solution? Is the intended claim empirical risk reduction or a formal constraint guarantee?
Reviewer_yXCMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a barycenter‑based inverse reinforcement learning approach to learn a stable trading policy from multiple diverse expert trading strategies. The method first generates policy representations via a Sharpe‑ratio‑driven performance‑weighted Wasserstein barycenter. It then learns a pre‑trained policy using inverse reinforcement learning and adversarial imitation learning. Finally, the policy is fine‑tuned with PPO. The approach also incorporates risk‑aware action execution and policy constraints. Backtesting across multiple markets demonstrates that it can achieve superior returns.

Critical strengths

As to quantitative strategy, learning an unknown reward function via inverse reinforcement learning to construct effective policies is a promising idea and a worthwhile research direction. And, the model construction is clearly presented.

Critical weaknesses

While constructing quantitative trading strategies via inverse reinforcement learning is a direction worthy of exploration, I have substantial doubts regarding the feasibility of the proposed scheme and the reproducibility of experimental results.

  1. The core premise of this method relies on the availability of "experts" for learning. However, the paper adopts various classical quantitative strategies as the "experts", such as trend-following and regression-based strategies. These strategies carry inherent strong limitations and exhibit drastically different performance across varied market regimes. Therefore, I have serious doubts about the feasibility of using such strategies for inverse reinforcement learning to learn effective trading policies. This work resembles building a sophisticated "ensemble strategy" using machine learning, yet the experiments are only validated under bull-market conditions. Moreover, comparisons against existing ensemble-strategy baselines are absent.
  2. Although the paper includes a REPRODUCIBILITY STATEMENT, a large amount of experimental details are missing. The authors state that the state representation is built with "eight technical indicators", yet they provide no introduction of what these eight indicators are. And, more examples include how training data is preprocessed, which normalization scheme is adopted, and the timestamp used for ranking stocks by market capitalization. These preprocessing steps directly determine whether look-ahead bias is introduced into experiments. The authors neither release source code nor elaborate on these critical details in the manuscript.
Questions for authors
  1. The authors claim to learn superior policies from diverse expert strategies. From a reinforcement‑learning perspective, expert policies are conventionally defined (or intuitively presumed) as optimal or near‑optimal policies. From the perspective of financial trading, however, the four expert strategies adopted by the authors (excluding the PPO agent) are not experts in the reinforcement‑learning sense. They are classical trading strategies whose limited performance is also demonstrated in the authors’ own experiments and are generally far from optimal. On another note, if these strategies qualify as expert policies, could other baselines evaluated in the paper such as the simple Buy‑Hold strategy also be treated as experts for training BRaG? After all, Buy‑Hold is not inferior to these expert strategies under certain market scenarios (e.g., on the DJI dataset).
  2. The authors propose to generate policy representations via the Wasserstein barycenter. Nevertheless, variables within a state are inherently coupled. How is such coupling preserved for the newly generated states? If coupling is broken in the generated states, the authors need to further elaborate why reinforcement‑learning can still learn effectively from such data.
  3. The authors evaluate a wide range of baselines and show that the proposed BRaG achieves better cumulative returns. Nevertheless, the paper does not illustrate what concrete actions BRaG takes to realize such returns. For instance: why do these expert strategies fail under certain conditions? How exactly does BRaG avoid such losses? Does BRaG mitigate losses by falling back to one of the existing expert strategies, or does it learn an uninterpretable mixed strategy? These questions are critical in financial trading, as they determine whether practitioners can trust the deployed method. Moreover, this directly bears on the core claim of this paper: that BRaG learns better policies from these expert demonstrations.
  4. Validation solely on bull‑market periods is insufficient. Supplementing experimental results covering diverse market regimes are necessary, including bearish and sideways‑oscillating market conditions. And, comparisons against ensemble‑strategy baselines are also important.
  5. Reproducibility concerns. While the authors provide detailed descriptions of compared baselines, many critical details concerning financial data processing are omitted, which are essential for reproducibility: 5.1) The authors use daily closing prices as part of the state, yet they do not clarify whether returns are computed using the closing price of the current day or the subsequent day. If returns are computed using the same‑day closing price, this corresponds to practically infeasible trades. Though this may be acceptable for pure research purposes, it reduces the real‑world practical value of the work. 5.2) Multiple datasets are utilized, yet data‑processing pipelines are not elaborated. This includes missing‑value handling and normalization schemes. Most importantly, it remains unclear whether look‑ahead bias is introduced. 5.3) Stocks are selected based on market‑capitalization ranking, but the timestamp at which market capitalization is measured for ranking is unspecified. 5.4) Eight technical indicators are used to construct the state space; however, the exact indicators and their computation hyperparameters are not specified. The authors are advised to provide the full list of indicators together with their calculation formulas and parameters.
Reviewer_UK6wOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a Deep Reinforcement Learning based approach for financial trading that is called BRaG, a barycenterbased adversarial inverse reinforcement learning framework. It combined expert strategies with learned behaviours. Ad hoc regularizers related to market indices are also incorporated.

Critical strengths

none

Critical weaknesses

No methodological contribution in the field of DRL. Very limited and unconvincing validation to selected stock markets, for specific periods and daily candles. Definately overfiting based on my expeience.

Questions for authors

Does the approach add any significant impovement that can be verified in other domain bayond fiancial trading? Why such limited validation? How the specific markets, duration, candles, train/validatio/test sets have been selected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.