acceptodds
Under review as a conference paper at ICLR 2027

Correlation Structure Governs Scalarization Bias in a Group-Relative Reinforcement Learning System for Portfolio Optimization

Abstract

This paper introduces Pareto-GRPO, a group-relative reinforcement learning system with verifiable rewards for portfolio rebalancing under four objectives: Sharpe ratio, negative maximum drawdown, diversification ratio, and negative turnover. At each rebalancing step the policy samples a group of candidate weight vectors, scores them with a closed-form auditable objective, and reduces the resulting objective matrix to a per-candidate advantage through one of two interchangeable schemes: a Minkowski-weighted (MW) linear combination, or a one-time computation of an in-group rank-and-crowding-distance statistic adapted from NSGA-II. Working under an equicorrelation asset model, this paper applies a differential-geometric treatment to the mean-variance frontier and a spectral decomposition to the asset covariance matrix. Using this framework, this paper derives a closed-form relationship between average pairwise asset correlation and the curvature of the Sharpe-drawdown trade-off surface. This paper demonstrates that as asset correlation falls, the region of this surface that cannot be reached via linear reduction grows monotonically. This analysis is extended to a finite-sample statement, built from Chernoff-type concentration bounds on a two-component sampling mixture, of what an in-group rank statistic can actually recover of that region. Further results establish an unbiasedness and variance-reduction property of the group-relative baseline, a trust-region bound on the clipped surrogate, and a consistency result for the Bradley-Terry pairwise variant as a maximum-likelihood estimator of the group's latent dominance ranking. Together these results yield a testable law: linear reduction should track rank-based reduction in concentrated, high-correlation markets, and increasingly lag it as the universe diversifies. The law is tested with a walk-forward protocol across six equity universes spanning several continents, using Wilcoxon signed-rank tests with Holm-Bonferroni correction and a Friedman rank test across markets. The empirical pattern matches the theoretical prediction on all six markets (S&P 500 tech subset, NASDAQ 100 subset, NIFTY 50, Russell 1000 Value, Dow 30, STOXX Europe 50): linear reduction leads on the four higher-correlation panels and rank-based reduction leads on the two lowest-correlation panels, though the cross-market Friedman statistic, while directionally consistent, remains underpowered at this sample size. A secondary job-scheduling instance, analyzed via a submodularity argument for the hypervolume indicator, indicates the reduction machinery is not specific to finance.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, portfolio optimization, group-relative policy optimization, Pareto frontier, scalarization, NSGA-II, quantitative finance

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VXdAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

Problem. Portfolio rebalancing over N=10N=10 assets as a four-objective problem (Sharpe, −MDD, diversification ratio, −turnover).

Method (Pareto-GRPO). A GRPO-style loop without a critic. At each 5-day rebalancing step the policy samples G=16G=16 candidate weight vectors from a two-component Dirichlet, scores each with the closed-form objective vector on the realized return window, and reduces the 16×416\times4 objective matrix to a per-candidate advantage by one of two schemes. MW uses fixed linear weights (0.4,0.3,0.2,0.1)(0.4,0.3,0.2,0.1). The NSGA-II variant applies one non-dominated sort with crowding distance to the single group. Composite and Bradley-Terry variants are also defined.

Theory. Under an equicorrelation model the minimum-variance point and frontier curvature are monotone in ρ\rho, so the region unreachable by linear scalarization shrinks as correlation rises. A detection-probability result states that a single group of 16 samples sees an indentation only if its width exceeds about 17% of the local objective range.

Claim. Linear reduction should win on high-correlation universes and rank-based reduction on low-correlation ones. Six ten-ticker panels are ranked by realized average correlation. The realized winner matches the stated prediction on all six. A Friedman test across panels gives p=0.311p=0.311.

Critical strengths

1. The question is well posed. Whether the choice between linear and rank-based reduction can be predicted from a property of the asset universe is a sensible question, and holding the entire training loop fixed while swapping only the reduction step is the right controlled design.

2. The paper is open about its weakest results. It states that the headline Friedman test is not significant, that the rank-based variant suffers a training pathology on the training panel, and that the single-group detector is much weaker than a full evolutionary search.

Critical weaknesses

1. The central empirical claim is circular, and the headline results are in-sample.

The theory delivers a monotone relationship between correlation and the size of the unreachable region. It does not deliver a threshold ρˉ\bar\rho at which the winner should switch. The "Predicted leader" column of Table 3 therefore cannot have been written before the results were seen. The four-versus-two split is read off the outcomes, and the sign-test p=0.016p=0.016 built on it is not a test of anything. One of the six panels (A) is the training set. The pre-registered Friedman test gives p=0.311p=0.311. The per-market Wilcoxon tests announced in Section 6 are never reported. Table 2 and Figure 1, presented as the full evaluation, are training-set metrics by the paper's own statement in Section 5.

2. The modelling ignores the gap between historical and realized statistics, and the panel ranking uses test-period information.

The theory treats Σ\Sigma as a fixed equicorrelation matrix. Nothing in the framework addresses distribution shift, which is the central difficulty in portfolio construction. The empirical test ranks panels by realized average correlation computed over the full 2014–2026 sample, which includes every test fold. A prediction that depends on test-period statistics is not an ex ante prediction. Fixing this requires either an ex ante correlation estimate per fold or a theory that models the drift, both of which change the paper's structure.

3. The experiments are too thin and too opaque to support the claims.

IssueEvidence
BaselinesOnly Markowitz MVO, Risk Parity, Equal Weight, and a scalar PPO. No RL portfolio baseline, no textbook NSGA-II, no modern allocation method
MarginsMarkowitz beats NSGA-II on panels E and C. The losing reduction trails Equal Weight on four of five test panels. The rank-based variant has Sharpe 0.507 and MDD 56.9% on the training panel vs. 1.035 and 30.5% for Equal Weight
TrainingNumber of steps and stopping rule are not stated. Gradient norm for the rank-based run goes to zero after step 200 (Section 7). Objective traces in Figure 1 do not appear converged
ProtocolNot stated whether the policy trained on A is transferred to B–F or retrained. Fold counts and horizons hkh_k not given. Ticker lists and selection rule not given. Numerical ρˉ\bar\rho not reported, only high/medium/low
CodeNot released. Proprietary infrastructure under NDA

The best-performing in-sample variant (Composite) is never evaluated on the test panels. Whether any parameter is set with test data in view cannot be determined from the paper.

Questions for authors

Q1. Pre-specification of the prediction. State the rule, fixed before any result was seen, that maps a panel's ρˉ\bar\rho to a predicted leader. If no such rule existed, remove the sign test and the "all six match" claim.

Q2. Ex ante correlation. Recompute the panel ranking using only training-window correlation for each fold, and report whether the ordering and the claimed pattern survive.

Q3. Out-of-sample headline. Replace Table 2 and Figure 1 with out-of-sample walk-forward results, and report the per-market Wilcoxon tests with Holm correction that Section 6 describes.

Q4. Training protocol. State the number of training steps, the stopping criterion, whether the policy is trained once on A and transferred or retrained per panel, the number of folds, and the horizons hkh_k. Provide the ticker lists and the selection rule for each panel.

Q5. Baselines. Add at least one RL portfolio baseline (e.g. Jiang et al., 2017), a textbook generational NSGA-II with the same objectives, and one modern allocation method such as hierarchical risk parity.

Q6. Convergence and collapse. Report training curves to convergence for all variants, and explain how the rank-based variant's gradient collapse is separated from the correlation-curvature explanation. Evaluate the Composite variant on panels B–F.

Q7. Objective evaluation window. Over what horizon are Sharpe and MDD computed for each candidate at a 5-day rebalancing interval? These statistics are extremely noisy on short windows, and the paper does not say.

Q8. Figures. Figure 1 is unreadable at print size. Split it into separate figures with legible fonts and distinguishable colours.

Reviewer_naHZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper propose Pareto-GRPO, a group-relative RL algorithms with verifiable rewards for portfolio management. The authors derive a closed-form relationship between average pairwise asset correlation and the curvature of Sharpe-drawdown trade-off surface. The findings are further tested on 6 different stock markets.

Critical strengths
  • RL has been widely used for portfolio optimization. However, RL algorithms with verifiable optimization bounds related to reward functions are new direction.
  • The closed-form relationship between average pairwise asset correlation and the curvature of the Sharpe-drawdown trade-off surface derived in this work is meaningful for the design of RL-based portfolio management algorithms.
  • Experiments on 6 different stock markets make the results convincing.
Critical weaknesses
  • The writting of this work need to be significantly improved. The current version is hard to follow with lots of typos and formating issues. Many terminlogies have no explanation which is impossible to understand for readers with general AI/RL background.
  • The motivation is not clear. I suggest adding some introduction on task background and limitation of existing work to motivation this paper. At present, I do not get why correlation, RL and portfolio management need to be studied together.
  • The formulas and notations should be improved. For most readers, it should be very hard to follow as the authors missed lots of preliminary definition with many unfamiliar abbreviation for ML communities.
  • Table 2 is confusing. As the authors claim experiments on 6 datasets, why there is only performance on one dataset here?
  • The Share ratio is quite poor with very high MDD. It is better to check the detailed behavior of RL algorithms otherwise this experiments may indicate an RL algorithms with very few buy/sell actions and hold the assets for a long time.
  • The size of all experiments figures is too small and hard to distinguish.
  • Detailed description on the formulation of MDP in this setting is required. The deisgn of state, action and reward can have a high impact on experimental results in this domain.
Questions for authors

See weakness above.

Reviewer_sbQtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Pareto-GRPO, a group-relative reinforcement learning framework for multi-objective portfolio optimization. It compares linear scalarization with rank-based reduction using non-dominated sorting and crowding distance. The paper theoretically investigates the relationship between asset correlation, Pareto-frontier curvature, and scalarization bias, and evaluates the proposed approach on six equity markets.

Critical strengths
  1. The paper addresses a relevant problem in multi-objective optimization and reinforcement learning, with practical applications to portfolio management.
  2. It attempts to connect theoretical analysis with empirical validation through financial market experiments.
  3. The experimental protocol includes walk-forward evaluation, multiple random seeds, and statistical testing.
Critical weaknesses
  1. The analysis of the mean–variance frontier does not adequately establish the claimed monotonic relationship between correlation and the unreachable region of the actual four-objective Pareto frontier.
  2. The claimed unbiasedness of the group-relative policy gradient requires further justification, since the group mean includes each candidate's own reward.
  3. Six market panels are insufficient to establish a general relationship between asset correlation and the relative performance of the two reduction strategies. The cross-market Friedman test is not statistically significant.
  4. The rank-based method is not compared sufficiently with full-fledged multi-objective optimization algorithms under comparable computational budgets.
  5. The treatment of transaction costs, potential survivorship bias, and the temporal evaluation protocol requires greater clarity.
Questions for authors
  1. How can the claimed monotonicity of the curvature functional be rigorously established, given that it depends on multiple quantities that vary with asset correlation?
  2. How can the mean–variance analysis justify conclusions about the unreachable region of a four-objective Pareto frontier?
  3. Is the group-relative policy gradient unbiased when the baseline includes the candidate's own reward? If so, under what assumptions?
  4. Does the proposed method outperform full-fledged multi-objective optimization algorithms under comparable computational budgets?
  5. The main empirical findings need to be validated across more independent markets or controlled synthetic environments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.