acceptodds
Under review as a conference paper at ICLR 2027

Mean-Preserving Exploration for Continuous-Time Portfolio Learning

Abstract

We study a fundamental question in continuous-time reinforcement learning: how can we introduce exploration while ensuring that the mean exploratory action exactly recovers the target optimal policy? Using continuous-time portfolio selection as a canonical setting, we investigate two exploration mechanisms. First, we formulate entropy-regularized exploratory control and solve the associated exploratory Hamilton–Jacobi–Bellman (HJB) equations under both state-independent and state-dependent temperatures. We show that, in both cases, the mean of the optimal exploratory policy coincides exactly with the target optimal policy, while the temperature controls its dispersion. Second, we develop a direct Gaussian exploration formulation and establish the same mean-preserving property for both state-independent and state-dependent variances. Our results reveal a common principle behind these formulations: exploration can be designed through policy dispersion while preserving the optimal exploitation decision through its mean. This provides a tractable framework for separating exploration from exploitation in continuous-time reinforcement learning.

Primary area:
reinforcement learning
Keywords:
Continuous-time reinforcement learning, exploratory control, mean-preserving exploration, entropy regularization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_QVvnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when Gaussian exploration preserves the optimal mean policy in a continuous-time portfolio problem. It identifies conditions on entropy temperatures and exploration covariances, including particular state-dependent choices, under which the optimal mean remains equal to the deterministic optimal control. These results motivate an actor–critic-based algorithm that learns the policy mean using a prescribed state-dependent covariance. A example illustrates how exploration changes the value and incurred risk while preserving the optimal mean. The proposed learning algorithm is not evaluated empirically.

Critical strengths

(S1: Clear explanation of mean preservation) The derivative-ratio argument explains why adding zero-mean noise does not automatically preserve the optimal policy after optimisation. The main derivations are sound.

(S2: Useful distinction between exploration mechanisms) The paper shows why a state-dependent covariance and an entropy temperature cannot be chosen interchangeably. The example of a temperature that fails to preserve the mean makes this distinction concrete.

Critical weaknesses

(W1. Unclear implications for exploration design) The paper characterises mean-preserving exploration and quantifies its cost, but does not establish a criterion for choosing among the admissible designs. In particular, constant covariance already preserves the optimal mean, and it remains unclear when adding the state-dependent component is worthwhile or how its strength should be selected.

(W2. Modest advance over existing results) Gaussian exploration and recovery of the classical optimal policy through its mean are already established in Wang and Zhou (2020) and Dai et al. (2026). The present paper adds a compatibility characterization and state-dependent constructions, including conditions under which mean preservation fails. I find this to be a small extension and remain unconvinced that their significance beyond the earlier results is sufficiently established, and I'm not sure which previously unresolved cases the characterization addresses and why those cases matter.

(W3. No evidence of a learning benefit) The authors explicitly acknowledge this, but they provide neither convergence analysis nor empirical evaluation of Algorithm 1. It does not show whether the proposed exploration helps the algorithm learn.

Questions for authors

(Q1: Addresses W1) What exploration choice should a reader make differently after reading this paper? In particular, when should state-dependent covariance be preferred to constant covariance?

(Q2: Addresses W2) What is the main insight beyond the earlier mean-recovery results and the preservation of quadratic value functions? Please explain its significance for exploration design.

(Q3: Addresses W3) Does the proposed algorithm benefit from state-dependent exploration under the same sampling budget? An analysis or a controlled comparison with constant covariance would help establish why these alternatives matter.

(Q4: Connection between theory and algorithm) What practical guidance does the state-dependent temperature construction in Section 4.2 provide beyond establishing existence? Its role in the overall contribution is unclear to me, since the algorithm in Sections 5–6 uses a separate prescribed covariance design. Can the authors explain more clearly?

(Q5. Comparison with TD3) How does the proposed algorithm compare with TD3 using the same state-dependent Gaussian noise under matched environment interaction budgets? Including TD3 with constant Gaussian noise would help distinguish the benefit of the covariance design from that of the actor–critic updates.

(C1) Maybe a personal choice, but there are no dedicated related works section but included in the introduction. There also seem to be quite few citations. I think the introduction and the thoery section would benefit from some more citations to help position this work. Also the authors should include a dedicated related works section to help position the contribution of this work.

Reviewer_qZw3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies mean-preserving Gaussian exploration in continuous-time stochastic control, using an unconstrained fixed-target portfolio problem with a quadratic terminal objective. It considers two exploration mechanisms: entropy-regularized randomized control and direct Gaussian exploration with prescribed covariance. The authors derive conditions under which the optimal Gaussian policy retains the mean of the original deterministic optimal control, including certain state-dependent exploration configurations. They also propose a trajectory-based actor–critic algorithm and provide a one-asset analytical example evaluating closed-form terminal losses.

Critical strengths

The paper provides explicit analytical conditions for mean preservation under selected state-dependent entropy temperatures and Gaussian covariances. These constructions offer properties that could serve as reference cases for continuous-time exploration methods, although their scope and practical significance remain quite limited.

Critical weaknesses

1. Limited theoretical novelty and significance.

The main results largely reduce to standard linear–quadratic stochastic control calculations: mainly through a quadratic value-function form and HJB coefficient matching. The optimal mean is preserved because Vx/Vxx=x−wV_x/V_{xx}=x-w for a value function of the form V(t,x)=P(t)(x−w)2+B(t)V(t,x)=P(t)(x-w)^2+B(t). Gaussian exploration and the separation between the optimal mean and exploration variance have already been studied in Wang et al. (2020) [1] and Wang and Zhou (2020) [2], while Dai et al. (2023) [3] examines Gaussian policy randomization and recovery of the original optimal control through its mean. Although the specific state-dependent compatibility conditions may be new, the paper does not establish their significance beyond a modest extension of established LQ techniques.

2. No meaningful experimental validation or learning benchmarks.

The proposed trajectory-based actor–critic algorithm is never evaluated. The only numerical example substitutes parameters into closed-form formulas for a one-asset model: there is no data-driven learning experiment, no learning or convergence results, and no comparison with established continuous-time RL algorithms such as Jia and Zhou (2022) [4] or Huang et al. (2024) [5] as well as traditional numerical methods. Thus, there is no evidence that the algorithm learns successfully or offers any practical advantage.

3. Severe restriction to quadratic value functions, with unclear generality.

The mean-preservation results depend critically on the special quadratic structure produced by linear wealth dynamics and a squared terminal objective. The exploration rules are constructed specifically to retain this quadratic HJB structure. With a general nonquadratic value function, the identity Vx/Vxx=x−wV_x/V_{xx}=x-w fails and mean preservation no longer follows. The paper provides no corresponding result for nonlinear dynamics, nonquadratic objectives, or constrained controls, leaving the broader relevance of the theory unclear.

References

[1] H. Wang, T. Zariphopoulou, and X. Y. Zhou. "Reinforcement Learning in Continuous Time and Space: A Stochastic Control Approach." Journal of Machine Learning Research, 21(198):1–34, 2020.

[2] H. Wang and X. Y. Zhou. "Continuous-Time Mean–Variance Portfolio Selection: A Reinforcement Learning Framework." Mathematical Finance, 30(4):1273–1308, 2020. doi:10.1111/mafi.12281.

[3] M. Dai, Y. Dong, Y. Jia, and X. Y. Zhou. "Data-Driven Merton's Strategies via Policy Randomization." arXiv:2312.11797, 2023.

[4] Y. Jia and X. Y. Zhou. "Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms." Journal of Machine Learning Research, 23(275):1–50, 2022.

[5] Y. Huang, Y. Jia, and X. Y. Zhou. "Mean–Variance Portfolio Selection by Continuous-Time Reinforcement Learning: Algorithms, Regret Analysis, and Empirical Study." arXiv:2412.16175, 2024.

Questions for authors
  1. What is the substantive new theoretical result beyond existing Gaussian exploration and mean-recovery literature [1–3], and why is it significant rather than an application of quadratic HJB matching?

  2. Can the authors evaluate Algorithm 1 in a genuinely data-driven setting with unknown market coefficients, and compare policy accuracy, convergence, and computational cost against established numerical methods, discrete RL methods, as well as continuous-time RL baselines?

  3. To what extent can the mean-preservation results survive beyond quadratic value functions, such as under nonquadratic terminal costs or constrained controls? If they cannot, what is the intended broader applicability?

Reviewer_L5gGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when exploration preserves the optimal deterministic policy through the mean of an optimized stochastic policy. In a continuous-time portfolio problem with deterministic coefficients and quadratic terminal loss, it characterizes mean preservation through the quadratic structure of the value function. It establishes conditions for entropy-regularized exploration and prescribed Gaussian covariances, including state-dependent designs.

The paper also proposes a model-free actor–critic procedure and provides a closed-form example distinguishing exploration cost from the performance of the extracted mean policy. The contribution is primarily theoretical; the learning procedure is not empirically evaluated or accompanied by a convergence guarantee.

Critical strengths
  1. The paper addresses the meaningful problem of preserving the optimal deterministic policy under exploration. It explains why zero-mean action noise alone is insufficient, since exploration changes the state distribution and potentially the policy being optimized.

  2. The derivative-ratio condition provides a coherent explanation of mean preservation across both exploration formulations. The compatibility condition for state-dependent entropy temperatures and the result for additive and multiplicative Gaussian exploration turn this principle into concrete characterizations.

  3. The paper distinguishes exploration cost from deployed-policy performance, entropy regularization from prescribed covariance, and continuous-time guarantees from finite-step implementation. It also clearly acknowledges that the control results do not establish convergence of the proposed learning algorithm.

Critical weaknesses
  1. The theoretical results rely on a narrowly structured setting with deterministic market coefficients, unconstrained actions, and quadratic terminal loss. Although the state-dependent exploration extensions are useful, their reliance on quadratic closure limits the breadth of the contribution beyond this specific control problem.

  2. The proposed actor–critic procedure has neither convergence analysis nor empirical evaluation. While this limitation is acknowledged, the optimal-control results do not establish reliable recovery of the target policy from data, leaving the practical benefit for policy learning unresolved.

Questions for authors

Could the authors clarify which aspects of the mean-preservation characterization are specific to the portfolio model and which extend to a broader class of linear-quadratic control problems? This would help assess the scope of the theoretical contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.