acceptodds
Withdrawn from ICLR 2027

Parameter Step-Based Exploration For Learning Deterministic Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In this work, we investigate deterministic policy learning through an auxiliary objective, \(J_\sigma\), induced by step-based Gaussian perturbations of the policy parameters, where \(\sigma>0\) denotes the exploration scale. First, we establish discrepancy bounds quantifying the gap between the auxiliary and deterministic objectives under both Lipschitz and smooth Markov Decision Processes (MDPs), of order \(O(d\sigma)\) and \(O(d\sigma^2)\), respectively. Furthermore, analyzing a simultaneous actor-critic architecture under realistic Markovian sampling rather than idealized i.i.d. sampling, we prove that our approach reaches an \(\varepsilon\)-stationary point of \(J_\sigma\), up to critic approximation error, with a sample complexity of \(O(d^1.5\sigma^-3\varepsilon^-2)\) in the Lipschitz case and \(O(d\sigma^-2\varepsilon^-2)\) in the smooth case. Under additional smoothness, we also bound the gradient discrepancy between the auxiliary and deterministic objectives by \(O(d^1.5\sigma)\), thereby relating stationarity of the explored objective to stationarity of the deterministic deployment objective as the exploration scale decreases. Finally, we evaluate PSBE and a deep variant, Deep-PSBE, on standard MuJoCo continuous-control benchmarks against parameter- and action-space exploration baselines.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Deterministic Policy Gradient, Actor-Critic, Parameter Space Exploration, Zeroth-Order Optimization, Gaussian Smoothing, Markovian Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f9a1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies deterministic policy learning through independent Gaussian parameter perturbations at each decision step. It introduces an auxiliary objective JσJ_\sigma, analyzes its discrepancy from the deterministic deployment objective JDJ_D, and presents a two-time-scale actor–critic method with a linear TD critic. The theoretical results address objective smoothness, finite-sample convergence under Markovian sampling, and the transfer of stationarity guarantees to deterministic deployment. Experiments evaluate PSBE and a nonlinear-critic extension on three MuJoCo tasks.

Critical strengths
  • The paper addresses a meaningful question: how optimizing a policy with parameter-space exploration relates to the performance of its deterministic counterpart at deployment.
  • The analysis considers coupled actor–critic updates, dependent samples, and critic approximation error, extending beyond an idealized exact-gradient setting.
Critical weaknesses
  • The abstract introduces notation and asymptotic rates without adequately establishing the motivation or explaining the conceptual significance of the results.
  • Redundant definitions, inconsistent notation, unresolved cross-references, and conflicting statements about dimension dependence make the theoretical development unnecessarily difficult to follow and verify.
  • The sampling protocol is unclear because Algorithm 1 uses the reset kernel P~\widetilde P, whereas the TD fixed-point definition in Eq. (20) uses the original transition kernel PP.
  • Corollary 10 states an ϵ\epsilon-stationarity guarantee without retaining the critic approximation error from Theorem 9 or specifying an assumption that makes this error sufficiently small.
Questions for authors
  • Theorem 5 gives a smoothness bound proportional to d/σ\sqrt d/\sigma, whereas the following discussion states d/σd/\sigma. Which dependence is intended, and how does the proof establish it?
  • Are transitions used in the TD target sampled from PP or P~\widetilde P, and how is this reconciled with both Eq. (20) and the claim of operating without resets?
  • Should Corollary 10 include an O(Ξcritic)O(\Xi_{\mathrm{critic}}) residual, or does it require an additional condition relating critic approximation error to ϵ\epsilon?
Reviewer_p3GeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies Parameter Step-Based Exploration (PSBE), a zero-order actor-critic algorithm. The main results provide sample complexity guarantees (for convergence to a stationary point) for the Lipschitz case and the smooth case. Empirical evaluations were also provided.

Critical strengths
  1. The paper aims to provide finite sample analysis of a zero-order actor critic method which was absent in literature.
Critical weaknesses
  1. The algorithm isn't new or novel—it seems to be almost identical to ZOAC. However, the ZOAC paper didn't have a finite-time analysis. I also don't see much novelty on the analysis side, as it seems to use known techniques.

  2. The assumptions on the MDP, such as Lipschitz continuity and smoothness, seem very restrictive to me. There are actor-critic analyses that don't use these (Xu et al., Wu et al., and many more), so why are they needed here?

  3. The d4d^4 dependence in the final result is quite suboptimal—this can be extremely large for large neural networks found in practice.

  4. There seem to be numerous issues in the analysis:

    i. The paper says no "resets" are used, but their algorithm says it samples from P~\widetilde P, which, by definition, requires resets to the initial distribution.

    ii. Also, the use of P~\widetilde P is not correct in the algorithm, since the Bellman equation (for a fixed policy) uses PP instead of P~\widetilde P. Thus, you need samples from PP for the critic.

    iii. The discount factor is missing from the performance difference lemma.

These issues may be fixed (except for the "no resets" part, since resets seem unavoidable), but I am unsure about the correctness of the entire proof.

Questions for authors

Based on the weaknesses listed:

  1. Please clarify the novelty of the paper, since I am currently not very convinced by its algorithmic or technical novelty.

  2. Please describe when the assumptions hold, especially the Lipschitz and smoothness assumptions on the MDP.

  3. Can you explain how the upper bounds obtained compare to the optimal/lower bounds and, if there is a gap, why it arises?

Reviewer_qeTcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies deterministic policy learning through Gaussian parameter perturbations resampled at each decision step. It analyzes the discrepancy between exploratory training and deterministic deployment and presents a finite-sample analysis of simultaneous actor–critic updates under Markovian sampling.

Critical strengths

The explicit distinction between the exploratory training objective and the deterministic deployment objective provides a clear formulation.

Considering dependent Markovian samples and critic approximation error makes the analysis relevant to practical actor–critic learning.

Critical weaknesses

The motivation for this particular exploration mechanism is insufficient. Deterministic-policy methods already support stochastic exploration during training, and stochastic-policy methods can also be evaluated deterministically. Therefore, the desire for deterministic deployment alone does not establish why step-wise parameter perturbations are needed. The paper should identify a concrete limitation of established approaches that its formulation resolves.

The incremental contribution relative to existing work needs clearer justification. ZOAC already combines step-wise parameter-space exploration with a zeroth-order actor–critic framework. Although the present paper adds convergence analysis, it does not sufficiently explain what new understanding or actionable guidance this analysis provides about exploration, or when the proposed approach should be preferred.

The experiments do not convincingly establish practical advantages. DDPG and TD3 are evaluated with modified linear critics, while Deep-PSBE uses a nonlinear critic. Direct comparisons with ZOAC, standard TD3, and SAC with deterministic evaluation are missing. Consequently, the reported results provide limited evidence that the proposed mechanism offers benefits over established alternatives.

Questions for authors

Beyond establishing convergence for the existing exploration mechanism, what distinct theoretical insight does the analysis provide relative to ZOAC and prior deterministic actor–critic analyses?

Can the authors demonstrate an advantage over ZOAC, standard TD3, and SAC under matched interaction budgets and comparable model capacity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.