- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when Gaussian exploration preserves the optimal mean policy in a continuous-time portfolio problem. It identifies conditions on entropy temperatures and exploration covariances, including particular state-dependent choices, under which the optimal mean remains equal to the deterministic optimal control. These results motivate an actor–critic-based algorithm that learns the policy mean using a prescribed state-dependent covariance. A example illustrates how exploration changes the value and incurred risk while preserving the optimal mean. The proposed learning algorithm is not evaluated empirically.
Critical strengths
(S1: Clear explanation of mean preservation) The derivative-ratio argument explains why adding zero-mean noise does not automatically preserve the optimal policy after optimisation. The main derivations are sound.
(S2: Useful distinction between exploration mechanisms) The paper shows why a state-dependent covariance and an entropy temperature cannot be chosen interchangeably. The example of a temperature that fails to preserve the mean makes this distinction concrete.
Critical weaknesses
(W1. Unclear implications for exploration design) The paper characterises mean-preserving exploration and quantifies its cost, but does not establish a criterion for choosing among the admissible designs. In particular, constant covariance already preserves the optimal mean, and it remains unclear when adding the state-dependent component is worthwhile or how its strength should be selected.
(W2. Modest advance over existing results) Gaussian exploration and recovery of the classical optimal policy through its mean are already established in Wang and Zhou (2020) and Dai et al. (2026). The present paper adds a compatibility characterization and state-dependent constructions, including conditions under which mean preservation fails. I find this to be a small extension and remain unconvinced that their significance beyond the earlier results is sufficiently established, and I'm not sure which previously unresolved cases the characterization addresses and why those cases matter.
(W3. No evidence of a learning benefit) The authors explicitly acknowledge this, but they provide neither convergence analysis nor empirical evaluation of Algorithm 1. It does not show whether the proposed exploration helps the algorithm learn.
Questions for authors
(Q1: Addresses W1) What exploration choice should a reader make differently after reading this paper? In particular, when should state-dependent covariance be preferred to constant covariance?
(Q2: Addresses W2) What is the main insight beyond the earlier mean-recovery results and the preservation of quadratic value functions? Please explain its significance for exploration design.
(Q3: Addresses W3) Does the proposed algorithm benefit from state-dependent exploration under the same sampling budget? An analysis or a controlled comparison with constant covariance would help establish why these alternatives matter.
(Q4: Connection between theory and algorithm) What practical guidance does the state-dependent temperature construction in Section 4.2 provide beyond establishing existence? Its role in the overall contribution is unclear to me, since the algorithm in Sections 5–6 uses a separate prescribed covariance design. Can the authors explain more clearly?
(Q5. Comparison with TD3) How does the proposed algorithm compare with TD3 using the same state-dependent Gaussian noise under matched environment interaction budgets? Including TD3 with constant Gaussian noise would help distinguish the benefit of the covariance design from that of the actor–critic updates.
(C1) Maybe a personal choice, but there are no dedicated related works section but included in the introduction. There also seem to be quite few citations. I think the introduction and the thoery section would benefit from some more citations to help position this work. Also the authors should include a dedicated related works section to help position the contribution of this work.