- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies deterministic policy learning through independent Gaussian parameter perturbations at each decision step. It introduces an auxiliary objective , analyzes its discrepancy from the deterministic deployment objective , and presents a two-time-scale actor–critic method with a linear TD critic. The theoretical results address objective smoothness, finite-sample convergence under Markovian sampling, and the transfer of stationarity guarantees to deterministic deployment. Experiments evaluate PSBE and a nonlinear-critic extension on three MuJoCo tasks.
Critical strengths
- The paper addresses a meaningful question: how optimizing a policy with parameter-space exploration relates to the performance of its deterministic counterpart at deployment.
- The analysis considers coupled actor–critic updates, dependent samples, and critic approximation error, extending beyond an idealized exact-gradient setting.
Critical weaknesses
- The abstract introduces notation and asymptotic rates without adequately establishing the motivation or explaining the conceptual significance of the results.
- Redundant definitions, inconsistent notation, unresolved cross-references, and conflicting statements about dimension dependence make the theoretical development unnecessarily difficult to follow and verify.
- The sampling protocol is unclear because Algorithm 1 uses the reset kernel , whereas the TD fixed-point definition in Eq. (20) uses the original transition kernel .
- Corollary 10 states an -stationarity guarantee without retaining the critic approximation error from Theorem 9 or specifying an assumption that makes this error sufficiently small.
Questions for authors
- Theorem 5 gives a smoothness bound proportional to , whereas the following discussion states . Which dependence is intended, and how does the proof establish it?
- Are transitions used in the TD target sampled from or , and how is this reconciled with both Eq. (20) and the claim of operating without resets?
- Should Corollary 10 include an residual, or does it require an additional condition relating critic approximation error to ?