acceptodds
Under review as a conference paper at ICLR 2027

Distributional Epistemic Robustification for Sequential Portfolio Optimization

Abstract

Portfolio reinforcement-learning systems must operate with policies frozen before future market regimes are observed, because return dynamics shift between historical training and deployment. Overcoming this requires separating ordinary market-outcome risk from uncertainty about which portfolio action remains valuable, rather than treating all predictive dispersion as one risk signal. Distributional value learning is well suited to representing stochastic portfolio returns, yet its within-model return law leaves epistemic ambiguity unresolved once historical data support multiple plausible value functions, especially in offline frozen-policy extraction. Within that subcase, distributional critic ensembles address model uncertainty through predictive disagreement, but aggregate dispersion can mix within-critic outcome variation with between-critic disagreement about expected action values and thereby penalize the wrong uncertainty. We therefore propose Distributional Epistemic Robustification (DER), which separates return variation from disagreement across learned value functions and yields exact structural guarantees for the extractor together with a conditional shift-aware bound. Experiments on a strictly time-ordered, cost-aware A-share benchmark show that DER attains the highest aggregate and worst-calendar Active ARR among the external baselines in the primary comparison. A matched uncertainty-target control shows that penalizing cross-critic expected-value disagreement preserves substantially more return than penalizing within-critic or total dispersion while remaining competitive with ensemble-mean extraction.

Primary area:
reinforcement learning
Keywords:
Portfolio Optimization, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kjXQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an action-selection rule that penalizes disagreement among the mean predictions of frozen distributional critics. It establishes deterministic properties and a conditional regret bound, and evaluates the method through portfolio backtests and matched-critic ablations.

Critical strengths

I appreciate the frozen deployment protocol and matched-critic ablation. Keeping the critics fixed is a meaningful control: it makes it easier to see whether a gain comes from the action-selection rule or from better value estimation.

Critical weaknesses
  1. The methodological contribution and theoretical justification are limited. The method combines familiar ideas from distributional RL, ensembles, and pessimistic action selection, while several theoretical properties follow directly from the proposed envelope. My main concern is Assumption 1: it supplies the central link between ensemble disagreement and prediction error, without a nonvacuous residual bound for the stated training procedure. Critics can agree while sharing substantial bias. I do not see what in the analysis rules out this possibility or makes the uncertainty penalty reliable in the intended setting.

  2. The empirical evidence is too weak. In Section 5.4, the matched comparison against ensemble-mean selection shows an average advantage of 0.36 percentage points and wins in only three of five repeats. These repeats share the same market history and primarily assess training randomness, so they do not give me much confidence in a dependable advantage across market conditions. The benefit of distributional critics over matched scalar critics is also unclear. In my view, the empirical evidence is too weak for this paper to be seriously considered for acceptance.

Questions for authors
  1. What justifies Assumption 1 for the actual training procedure? Can the residual be independently bounded or calibrated when ensemble members share systematic errors?
  2. Which controlled result establishes a benefit over matched scalar critics using the same extractor? How do the three wins in five repeats support the claimed reliability?
Reviewer_tE9COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DER chooses portfolio actions by scoring each one as μ − β·σ, where μ is the average and σ the disagreement of the expected returns from three C51 critics. It deliberately ignores within-critic return variance. The method is tested as a frozen policy on A-share data from 2021 to 2025.

Critical strengths

The time-ordered protocol is careful, trading costs are realistic, and results are reported for every year.

Critical weaknesses
  1. Limited novelty. Without the framing, the deployed rule is a standard ensemble lower-confidence-bound score, the same mean-minus-std penalty used in Q-ensemble methods such as UCB over Q-ensembles (Chen et al., 2017), SUNRISE (Lee et al., 2021), PBRL (Bai et al., 2022), and An et al. (2021). Separating aleatoric from epistemic uncertainty in distributional ensembles has also been done before, for example by Nikolov et al. (ICLR 2019) with C51 plus bootstrapped heads, and by Clements et al. (2019). That a risk-neutral objective should not penalize return variance is expected rather than a new insight. Theorems 1–3 (ball support function, linear scalarization, Lipschitz margin) are elementary. The regret bounds hold equally at β = 0, so they don't show the penalty helps.

  2. The advantage is not statistically significant. No significance tests are reported, and both headline gains are within repeat-to-repeat noise:

    • Against the best baseline: 6.97 ± 1.13 vs 6.39 ± 1.42 for Decision Mamba-Finite, with n = 5 each. A Welch t-test on the reported numbers gives t ≈ 0.7 and p ≈ 0.5.
    • Against its own β = 0 ablation: DER gains 0.36 pp with 3/5 wins. With a paired SD of about 2.3 read from Figure 1b, this gives t ≈ 0.35 and p ≈ 0.7.
    • Risk-adjusted metrics: the Sharpe differences (0.890 vs 0.877 vs 0.874) are negligible relative to their SDs.

    The claims of "highest" performance are therefore not supported. The paper cites work on backtest overfitting (White 2000; Bailey et al. 2017; Harvey et al. 2016) but applies none of the corresponding tests. More repeats, paired tests, and multiple markets are needed.

  3. Missing key baselines. These are needed to isolate the contribution:

    • Ensemble-mean C51 (β = 0) on the same repeats as the main table. It currently appears only in a separate retrained cohort.
    • A scalar-critic ensemble with the same μ − β·σ extraction. The extractor uses only expectations, so it is unclear whether the "distributional" part matters at all.
Questions for authors
  1. Can you report paired significance tests for DER against the best baseline and against β = 0? With n = 5, the 0.58 and 0.36 pp gains seem within noise.
  2. What does ensemble-mean C51 (β = 0) score on the same repeats as Table 1?
  3. How does a scalar-critic ensemble perform with the same μ − β·σ rule? Does the distributional part matter?
  4. How does the best fixed rule selected on 2020 perform on the test years?
  5. Why is DER the worst method in 2022, the bear-market year?
Reviewer_pE1SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Distributional Epistemic Robustification (DER), a two-step method for stock portfolio trading. When models trade in future markets without retraining, they must separate market outcome risk from the uncertainty about valuable portfolio action. DER trains multiple AI critics to predict return probability charts, converts each chart into a single average return, and penalizes only the disagreement among these averages. The paper shows mathematical guarantees for action stability and error bounds under future market shifts using the proposed method. The proposed method is tested on A-share trading.

Critical strengths
  • the paper deals with an important problem in stock trading, how to adapt the trading strategy to the test regime where new data may imply scenarios not seen during model training.
  • the proposed method provides some theoretical justification. Experiments is organized to address specific research questions.
Critical weaknesses
  • While the paper distinguishes between within-critic return variation and cross-critic disagreement, there is a lack of intuition and justification why cross-critic disagreement is bad.
Questions for authors
  • why are these particular baselines methods picked to compare to the proosed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.