acceptodds
Under review as a conference paper at ICLR 2027

Geometry-Adapted Stepping for Stochastic Control via Walk-On-Spheres

Abstract

Reinforcement learning (RL) algorithms with a continuous action space rely on discrete (Euler-Maruyama) time-stepping to simulate continuous-time stochastic dynamics, introducing temporal discretization error and requiring ad hoc collision handling to respect geometric boundaries. These issues may be improved with smaller integration steps, but this increases overall simulation cost, creating a tradeoff between performance and speed. We introduce Walk-on-Spheres (WoS) Monte Carlo rollouts that perform geometry-aware stepping to traverse the state space more efficiently, are collision-free by construction, and do not require time discretization. In the setting of quadratic control costs, relevant to KL-regularized reinforcement learning, we utilize the Doob -transform to learn an optimal control policy for fixed and goal-conditioned navigation problems with only terminal rewards. We evaluate our method on 2D point-maze and 3D HM3D indoor navigation scenes, demonstrating significant speedups in rollout time while exceeding baseline performance under equal compute budget.

Primary area:
reinforcement learning
Keywords:
Stochastic Control, Walk on Spheres, Reinforcement Learning, Maximum Entropy Reinforce Learning, Hamilton Jacobi Bellman Equation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UMeEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a Walk-on-Spheres (WoS) approach to stochastic control with terminal rewards and quadratic control costs. Through the Hopf–Cole and Doob transforms, the optimal control is related to a harmonic function and its induced sphere-exit distributions. The method learns these distributions using von Mises–Fisher mixtures, replacing small time-discretized steps with geometry-adapted sphere jumps. Experiments compare against reinforcement-learning and path-integral-control baselines on 2D mazes and 3D HM3D scenes.

Overall, I am slightly positive. The paper presents a compelling connection between geometry-aware Monte Carlo sampling and a structured stochastic-control problem, with promising computational results in both 2D and 3D. Its scope is narrower than general reinforcement learning, and the continuous-control interpretation and experimental reliability need clarification. Nevertheless, I believe the methodological contribution is sufficient to support weak acceptance.

Critical strengths
  1. The connection between WoS and stochastic control is interesting and well motivated. Under the stated assumptions, the transformation from the HJB equation to a harmonic problem provides a principled basis for learning sphere-exit distributions. This offers a concrete alternative to time-discretized rollouts.

  2. The computational benefit is supported by experiments on complex geometry. Geometry-adapted jumps reduce rollout steps and network queries. The 2D and 3D experiments, including goal-conditioned navigation, show promising performance under a shared compute cap.

Critical weaknesses
  1. The relationship between the learned policy and executable continuous control remains incomplete. The exact Doob kernel corresponds to the optimal drift, but an arbitrary learned exit distribution need not correspond to a position-only Markov control. Appendix F defines a path measure by filling sphere interiors with conditioned Brownian motion; how this relates to the control-policy class used by the baselines needs clarification. The method also remains restricted to a particular first-exit problem with terminal rewards and quadratic control costs.

  2. Some experimental comparisons require more careful interpretation. WoS skips the interior Brownian paths, so its sphere-jump polyline length is not directly comparable to the finely sampled path length of Euler–Maruyama rollouts. Runtime statistics are conditioned on successful walks despite substantial success-rate differences. The reported learning curves also use one run per method, leaving training variability insufficiently characterized.

Questions for authors
  1. Does a general learned sphere-exit policy correspond to a position-only continuous drift? If not, how should its policy class and objective be compared with those of the Euler–Maruyama baselines?

  2. Could you report repeated-seed results and rollout costs across all attempts, alongside the successful-walk statistics?

  3. Could you clarify that Lemma 3.1 establishes fixed-policy asymptotic consistency rather than finite-sample unbiasedness or convergence of online training? How sensitive is training to importance-weight concentration?

4 comments

We thank the reviewer for their careful review of the paper. We paraphrase and address the raised concerns below.

The relationship between the learned policy and executable continuous control remains incomplete. The exact Doob kernel corresponds to the optimal drift, but an arbitrary learned exit distribution need not correspond to a position-only Markov control. Appendix F defines a path measure by filling sphere interiors with conditioned Brownian motion; how this relates to the control-policy class used by the baselines needs clarification.

Does a general learned sphere-exit policy correspond to a position-only continuous drift? If not, how should its policy class and objective be compared with those of the Euler–Maruyama baselines?

A learned exit distribution corresponds to a position-only Markov diffusion at convergence. When qθq_\theta matches the Doob kernel pBhp_B^h, it gives the optimal drift a∗=σ2∇log⁡ha^* = \sigma^2 \nabla \log h (Theorems 2.1 and 3.1). Before convergence, qθq_\theta need not correspond to a position-only Markov diffusion, but it is still an admissible control, as its control cost (Lemma F.1) is finite for a vMF mixture with finite κ\kappa and finitely many steps. The path measure in Appendix F is the law of the walk under this policy, so its control cost (Lemma F.1) is the same Equation 4 objective that we report for the baselines.

We now state Equation 4 over the wider class of admissible controls, and show in Lemma C.1 that the optimal admissible control is the Markov state-feedback drift a∗(Xt)=σ2∇log⁡h(Xt)a^*(X_t) = \sigma^2 \nabla \log h(X_t). Thus, there is no advantage to operating within this wider class, and the comparison with the Markov Euler--Maruyama baselines is fair.

Regarding executable continuous control, our Walk-on-Spheres based policy can potentially be interpreted via the well-established framework of options in hierarchical RL (Sutton, Precup, Singh, 1999), where each sphere represents a time-abstracted sub-problem in planning defined in terms of the sphere exit distribution qθq_\theta (see Section 2.1, lines 122--125). Such formalism allows an executable continuous controller to be naturally hierarchical, where a high-level policy chooses the sphere exit, and a low-level controller carries the walk to it. Design of such low-level controller from a trained qθq_\theta is left as future work.

Could you clarify that Lemma 3.1 establishes fixed-policy asymptotic consistency rather than finite-sample unbiasedness or convergence of online training?

We agree with the reviewer. Lemma 3.1 establishes fixed-policy asymptotic consistency, and not finite-sample unbiasedness nor convergence of the online training. We have updated the text in the document to clarify this, and replaced “unbiased”' with “consistent” in Section 3.1.

How sensitive is training to importance-weight concentration?

Regarding importance weight concentration, we ran an ablation that dilutes the importance weights by mixing the policy with a defensive sampling distribution (Appendix J). The average reward is highest with no dilution, and drops as the dilution increases. We suspect the numerical instabilities (or outliers) that come with sharp importance weight regions get amortized away by our neural network, which is fitted over many walks.

The method also remains restricted to a particular first-exit problem with terminal rewards and quadratic control costs.

Our work provides a direct method to solve linearly solvable continuous stochastic control problems via a consistent walk on spheres integrator. This integrator takes large steps in complex geometric domains without suffering from any approximation error associated with time discretization. Consequently, our method does not require any hyperparameter tuning or ablation on the Δt\Delta t parameter used by other continuous control algorithms. We also demonstrate our method’s efficiency over standard RL baselines in point navigation problems.

To the best of our knowledge, our approach of using a discretization-free integrator has not been applied before in continuous control or RL, potentially opening the door to a broad range of follow-on work. Given the maturity of the field using Euler-Maruyama methods, we look forward to developing further algorithmic extensions, e.g., actor-critic and value-based methods that leverage a WoS integrator, and extending the scope of problems it can tackle, e.g. to include path-dependent rewards, more general (angular, configuration space, non-positional, etc) state spaces and problems with no closed form doob tilted transition kernel.

Some experimental comparisons require more careful interpretation. WoS skips the interior Brownian paths, so its sphere-jump polyline length is not directly comparable to the finely sampled path length of Euler–Maruyama rollouts. Runtime statistics are conditioned on successful walks despite substantial success-rate differences. The reported learning curves also use one run per method, leaving training variability insufficiently characterized.

Could you report repeated-seed results and rollout costs across all attempts, alongside the successful-walk statistics?

We run 3 training seeds per scene for HM3D scenes and 2D maze scenes, per algorithm (baselines + our method). We have updated the training curves for Figure (2, 3, 9, 10) to reflect the changes. We have also updated all tables reporting metrics with mean ±\pm std across 3 different training seeds, instead of evaluation points. Our method still demonstrates strong performance in average reward, success rate, rollout speed compared in 11 / 12 HM3D scenes (See Tables 14, 17, 20, 22, 25, 28). And unlike some baselines (like SAC, PPO, PICE), our method also demonstrates lower variance in training across different seeds (see training curves). The 2D maze scene runs are yet to finish. Hence the related tables contain N/A for now. We’ll report them as soon as they are available.

Additionally, we made the baselines more competitive. We equipped all policy networks with the same hash-grid encoder, and trained each baseline with its best Δt\Delta t from an ablation (Tables 3 and 4, Appendix H). Even against these baselines, our method wins in 11 / 12 HM3D scenes in average reward. Note for our method, we do not require any such hyper parameter tuning, as it completely avoids time discretization.

Regarding rollout costs, we originally chose time per successful walk because, in a first exit problem, a poorly trained policy looks fast, as its walks fail early on nearby walls. Following the reviewer's recommendation, we now report the unconditional rollout time across all attempts in Table 1, and move time per successful walk to the appendix (Tables 13, 21, 29). Our method is still faster under the new metric.

Regarding walk length, we agree that the polyline length is not comparable between the two integrators. It is not intrinsic for either of them, as a continuous Brownian path has infinite length, so the Euler--Maruyama length grows as Δt→0\Delta t \to 0. We keep it in the appendix with this caveat, and add the control cost 12E∫0τ∥at∥2dt=σ2DKL(π ∥ PW)\frac{1}{2}\mathbb{E}\int_0^\tau \|a_t\|^2 dt = \sigma^2 D_{KL}(\pi \,\|\, \mathbb{P}_W) (Lemma E.1) to Table 1, which is computed the same way for both integrators.

We thank the reviewer for their valuable comments. In response, we have revised the manuscript to include new figures, an additional lemma, and further experiments addressing the points raised. As the discussion period closes by end of day today (AoE), we would be grateful if the reviewer could take a look at the revised manuscript and our posted responses, and let us know whether they resolve the concerns.

I appreciate the author's effort in the rebuttal, and I have no further questions and remain positive.

Reviewer_kcj1OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a Walk-on-Spheres (WoS) based approach for a restricted class of continuous-time stochastic control problems with terminal rewards and quadratic control costs. The key idea is to replace Euler-Maruyama time stepping with geometry-adaptive WoS transitions that directly sample exits from maximally inscribed spheres. Using a Hopf-Cole transformation and the Doob hh-transform, the optimal controlled diffusion can be characterized through a harmonic function hh, and the corresponding optimal sphere-exit distribution is approximated by a neural von Mises-Fisher mixture. The model is trained online using importance-weighted Monte Carlo estimates of the Doob-transformed exit distribution.

The method is evaluated on 2D mazes and 12 HM3D indoor scenes under fixed-goal and goal-conditioned settings, and compared against REINFORCE, SAC, PPO, PICE, MPPI, and an FEM reference in 2D. The reported results show substantially higher success/reward and much lower rollout cost than the Euler-Maruyama-based baselines under the chosen compute budget.

Critical strengths
  1. The use of Walk-on-Spheres for geometry-aware stochastic control is a technically interesting idea. Replacing small Euler-Maruyama steps with maximal sphere exits can dramatically reduce the number of simulated transitions and geometric/neural-network queries in cluttered environments. This is a potentially useful computational abstraction for first-exit stochastic control problems.
  2. The paper makes a coherent connection between WoS, the Doob hh-transform, and linearly-solvable stochastic control. For the restricted terminal-reward/quadratic-control setting, the derivation of the optimal Doob-transformed exit kernel is conceptually clean, and the 2D FEM reference provides a useful sanity check for the idealized setting.
Critical weaknesses
  1. The connection between the learned WoS policy and the original stochastic control problem is not fully established. The paper does not show that an arbitrary learned exit distribution qθ(ν∣x)q_\theta(\nu\mid x) corresponds to a valid Markov state-feedback diffusion of the form assumed in the original objective. This leaves a central gap between the method actually optimized and the control problem the paper claims to solve.
  2. The empirical validation and presentation are insufficiently rigorous. The main results rely on one training run per method, use different representation architectures across methods, and report a “time per successful walk” metric that conflates computational speed with success rate. The paper is also not sufficiently polished: for example, Figures 2 and 3 contain very small plots that are difficult to read, which further reduces the clarity of the experimental evidence.
Questions for authors
  1. The paper optimizes a learned sphere-exit distribution qθ(ν∣x)q_\theta(\nu\mid x), but it is unclear whether every such policy corresponds to a valid Markov state-feedback diffusion dXt=aθ(Xt)dt+σdWtdX_t=a_\theta(X_t)dt+\sigma dW_t. Please clarify the precise relationship between the learned WoS policy and the control class defined in Equation 4.
  2. The main experimental results appear to use one training run per method, while the reported uncertainty is over evaluation start states rather than independent training seeds. Could you provide results over multiple training seeds to demonstrate that the reported gains are statistically robust?
  3. WoS/PICE use a multi-resolution hash-grid encoder, whereas SAC/PPO/REINFORCE use standard MLPs. How much of the reported performance gap remains when the baselines are given comparable representation capacity?
  4. The reported “time per successful walk” depends on both computational throughput and success probability. Could you additionally report an unconditional runtime measure, such as time per rollout or steps/queries per second, to isolate the actual speedup of the WoS integrator?
4 comments

We thank the reviewer for their careful review of the paper. We paraphrase and address the raised concerns below.

The connection between the learned WoS policy and the original stochastic control problem is not fully established. The paper does not show that an arbitrary learned exit distribution qθ(ν∣x)q_\theta(\nu | x) corresponds to a valid Markov state-feedback diffusion of the form assumed in the original objective. This leaves a central gap between the method actually optimized and the control problem the paper claims to solve.

The paper optimizes a learned sphere-exit distribution qθ(ν∣x)q_\theta(\nu | x), but it is unclear whether every such policy corresponds to a valid Markov state-feedback diffusion dXt=aθ(Xt)dt+σdWtdX_t = a_\theta(X_t) dt + \sigma dW_t. Please clarify the precise relationship between the learned WoS policy and the control class defined in Equation 4.

We agree that an arbitrary learned exit distribution qθq_\theta does not, in general, correspond to a Markov state-feedback diffusion, and we do not claim it does. The exact sphere kernel pBh(y∣x)=PB(y∣x) h(y)/h(x)p_B^h(y \mid x) = P_B(y \mid x)\, h(y)/h(x) corresponds to the optimal Markov drift a∗=σ2∇log⁡h=∇Va^* = \sigma^2 \nabla \log h = \nabla V, where VV solves the HJB equation (Theorems 2.1 and 3.1). In practice, we fit pBhp_B^h with a vMF mixture, using a finite number of walks and training iterations, so qθq_\theta recovers dXt=a∗dt+σdWtdX_t = a^* dt + \sigma dW_t only in the limit qθ→pBhq_\theta \to p_B^h. Away from this limit, qθq_\theta is still an admissible control, as its control cost (Lemma F.1) is finite for a vMF mixture with finite κ\kappa and finitely many steps.

We now state Equation 4 over the wider class of admissible controls, and show in Lemma C.1 that the optimal admissible control is the Markov state-feedback drift a∗(Xt)=σ2∇log⁡h(Xt)a^*(X_t) = \sigma^2 \nabla \log h(X_t). Thus, there is no advantage to operating within this wider class, and the comparison with the Markov Euler--Maruyama baselines is fair.

The empirical validation and presentation are insufficiently rigorous. The main results rely on one training run per method, use different representation architectures across methods

The main experimental results appear to use one training run per method, while the reported uncertainty is over evaluation start states rather than independent training seeds. Could you provide results over multiple training seeds to demonstrate that the reported gains are statistically robust?

WoS/PICE use a multi-resolution hash-grid encoder, whereas SAC/PPO/REINFORCE use standard MLPs. How much of the reported performance gap remains when the baselines are given comparable representation capacity?

Post deadline, we equipped all policy networks with the same hash-grid encoder, so that all methods have the same spatial representation capacity, and retrained all baselines (one run per method). We additionally performed a Δt\Delta t ablation (Tables 3 and 4, Appendix H), and trained each baseline with its best Δt\Delta t (see Choice of Δt\Delta t, Appendix H) based on this hyperparameter search. Please note our method does not require any such hyperparameter tuning unlike any of the baselines, as it completely avoids time-discretization.

The revised document contains updated results which demonstrate that even with the new competitive baselines, our method still wins in 11 / 12 HM3D scenes in average reward.

We ran 3 training seeds per scene for HM3D scenes and 2D maze scenes, per algorithm (baselines + our method). We have updated the training curves for Figure (2, 3, 9, 10) to reflect the changes. We have also updated all tables reporting metrics with mean ±\pm std across 3 different training seeds, instead of evaluation points. Our method still demonstrates strong performance in average reward, success rate, rollout speed compared in 11 / 12 HM3D scenes (See Tables 14, 17, 20, 22, 25, 28). And unlike some baselines (like SAC, PPO, PICE), our method also demonstrates lower variance in training across different seeds (see training curves). The 2D maze scene runs are yet to finish. Hence the related tables contain N/A for now. We’ll report them as soon as they are available.

and report a “time per successful walk” metric that conflates computational speed with success rate.

The reported “time per successful walk” depends on both computational throughput and success probability. Could you additionally report an unconditional runtime measure, such as time per rollout or steps/queries per second, to isolate the actual speedup of the WoS integrator?

In a first exit problem, a poorly trained policy looks fast, as its walks fail early on nearby walls leading us to report time per successful walk metric. But, following the reviewer's recommendation, we now report the unconditional rollout time in Table 1, and move time per successful walk to the appendix (Tables 13, 21, 29). Our method is still faster under the new metric.

The paper is also not sufficiently polished: for example, Figures 2 and 3 contain very small plots that are difficult to read, which further reduces the clarity of the experimental evidence.

We apologize for the small font in the figures. We have updated all figures to have the same visible font size.

We thank the reviewer for their valuable comments. In response, we have revised the manuscript to include new figures, an additional lemma, and further experiments addressing the points raised. As the discussion period closes by end of day today (AoE), we would be grateful if the reviewer could take a look at the revised manuscript and our posted responses, and let us know whether they resolve the concerns.

The additional experiments and clarifications address some of my concerns, but they do not change my overall assessment. I still find the contribution limited in scope, and the manuscript’s writing and visual presentation, particularly the readability and consistency of the figures, fall below the standard I expect for an ICLR paper. Despite the improvements made during rebuttal, I am not convinced that the contribution and presentation are sufficiently strong to warrant acceptance. I therefore maintain my current rating.

Reviewer_17c1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

While prior RL work relies on discrete time approximation to continuous-time systems, this propose designing an RL method for continuous action spaces and continuous time (stochastic optimal control), around Walk-on-Spheres (WoS), and can take large jumps through state space by relying on the (known) geometry on the environment. This is instanced problems with terminal rewards, and quadratic control costs, which relates KL regularization of the trajectory distribution to a prior of Brownian motion. In the RL setting, the optimal policy tilts the transitions along paths with function h, which is an exponentiated value function. Practically, is trained to try to approximate this tilted distribution by parameterizing the exit direction. The policy is then optimized using rollouts which construct an MC estimate of the tilted distribution, which is then used as a target for the policy, by minimizing forward KL. Empirically, this algorithm (WoS), is compared to various baselines utilizing standard discretized time, such as model-free RL, and MPPI planning, evaluating on 2D/3D point navigation. Compared to baselines, the method is able to perform better (e.g. success) and has better wall-clock efficiency.

Critical strengths

The paper proposes a unique perspective on approaching continuous time RL problems using walk on spheres, and strong efficiency on 2D/3D point navigation problems.

Critical weaknesses
  • The compared baselines use the discrete time, and the proposed method WoS method (Alg 1) is the only method that uses the WoS environment. It may be useful to assess is the WoS enviroment integrator could be used by a normal algorithm, like REINFORCE, which would help assess the contribution of the environment versus the algorithm.
  • It is also unclear if the RL baselines are properly configured/tuned on the HM3D environment, which achieve very low success rate. It may be useful to train longer, both to see if the proposed method can solve HM3D, and if baselines can eventually catch up. There are also design decisions (sparse reward, fixed standard deviation), and the baselines do not get to leverage the environment structure (nearest wall distance), which is utilized for WoS. For example, an asymmetric critic, in PPO could use privileged wall structure information.
  • Following the description n Figure 2 for “one run per method”, it seems that single runs/seeds are used thorough the paper? it would be important to report multiple seeds (e.g. 3-5), and reporting/displaying both mean and std to be able to determine the reliability of results.
  • The success of the method may also rely on broad coverage (or uniform sampling) for initial states. Following algorithm 1, only the first action nu_n,0 and later actions in the trajectory are not supervised. I could understand that random initial states overcome this choice, as the policy starts different distances/positions away from goals.
  • Several other weaknesses are candidly discussed in the limitation section; however that the method is currently specific to one problem structure, not general RL problems: point navigation, with instantaneous transitions, available distance to boundary, and a specific reward structure. This may limit the current applicability of this work.
Questions for authors

Why is the Euler step size 0.1 for HM3D compared to 0.01 for the 2D maze? Could this be too large?

6 comments

We thank the reviewer for their careful review of the paper. We paraphrase and address the raised concerns below.

It is also unclear if the RL baselines are properly configured/tuned on the HM3D environment, which achieve very low success rate. It may be useful to train longer, both to see if the proposed method can solve HM3D, and if baselines can eventually catch up.

Why is the Euler step size 0.1 for HM3D compared to 0.01 for the 2D maze? Could this be too large?

The reported success rate is low as we did not run hyperparameter tuning for the baseline algorithms on the Δt\Delta t parameter. Post deadline, we ran an ablation over different Δt\Delta t values (See Table 3 and Table 4 in the revised document we uploaded), and also equipped all the policies with the same hashgrid encoder (so they get the same spatial representation capacity). These changes led to the baselines becoming more competitive, but our method still outperforms in 11 out of 12 HM3D scenes. For the 2D maze, we are running the training - we will report back when results are available.

Please note, in addition to outperforming the baseline methods, our method does not require Δt\Delta t tuning as it completely avoids time discretization.

There are also design decisions (sparse reward, fixed standard deviation), and the baselines do not get to leverage the environment structure (nearest wall distance), which is utilized for WoS. For example, an asymmetric critic, in PPO could use privileged wall structure information.

The baselines also use geometric queries to obtain structure information, namely ray-intersection queries to detect wall collisions, while WoS uses distance queries instead. Figure 13 demonstrates how Euler-Maruyama based methods fail under the absence of ray-intersection queries. While it is true that distance information is more informative than ray-intersection queries, building variants of baseline methods that use such information would constitute construction of wholly new methods to compare against. Our goal was to compare our algorithm with the most standard baselines with minor restrictions in place to make sure we are solving the same problem.

Following the description n Figure 2 for “one run per method”, it seems that single runs/seeds are used thorough the paper? it would be important to report multiple seeds (e.g. 3-5), and reporting/displaying both mean and std to be able to determine the reliability of results.

We ran 3 training seeds per scene for HM3D scenes and 2D maze scenes, per algorithm (baselines + our method). We have updated the training curves for Figure (2, 3, 9, 10) to reflect the changes. We have also updated all tables reporting metrics with mean ±\pm std across 3 different training seeds, instead of evaluation points. Our method still demonstrates strong performance in average reward, success rate, rollout speed compared in 11 / 12 HM3D scenes (See Tables 14, 17, 20, 22, 25, 28). And unlike some baselines (like SAC, PPO, PICE), our method also demonstrates lower variance in training across different seeds (see training curves). The 2D maze scene runs are yet to finish. Hence the related tables contain N/A for now. We’ll report them as soon as they are available.

The success of the method may also rely on broad coverage (or uniform sampling) for initial states. Following algorithm 1, only the first action nu_n,0 and later actions in the trajectory are not supervised. I could understand that random initial states overcome this choice, as the policy starts different distances/positions away from goals.

Yes, the success of our method relies on the ability to uniformly sample starting positions from the domain. However, for this specific sparse reward problem, this is also true for other baselines, as demonstrated in a small experiment that we ran. In particular, we trained the SAC / REINFORCE / PPO baselines while sampling the start locations from a restricted region. All baselines and our method fail to learn the correct policy. In this sparse reward setting with restricted starting location far from the goal, all methods begin with a roughly Brownian policy whose random walker is highly unlikely to reach the goal and provide relevant training signals.

Unlike our method, the baseline techniques train on intermediate walk steps, but still struggle due to the reason above. While modifying our method to train on intermediate spheres is possible and would likely improve sample efficiency with both terminal and path dependent rewards, we leave such extensions as future work. (See Appendix L, Rebuttal Figures, Figure 15).

The compared baselines use the discrete time, and the proposed method WoS method (Alg 1) is the only method that uses the WoS environment. It may be useful to assess is the WoS enviroment integrator could be used by a normal algorithm, like REINFORCE, which would help assess the contribution of the environment versus the algorithm.

Yes, we agree that REINFORCE can be used for training a policy with the WoS-based environment integrator. However, the goal of our paper was to establish the WoS integrator in the context of linearly solvable stochastic control and recognize the optimal sphere exit kernel as the Doob h-transform. Establishing such a theoretical relationship within the standard REINFORCE framework is not as straightforward, though we see this as an interesting future direction of exploration that potentially generalizes our method in the stochastic control problems it can handle.

however that the method is currently specific to one problem structure, not general RL problems: point navigation, with instantaneous transitions, available distance to boundary, and a specific reward structure. This may limit the current applicability of this work.

Our work provides a direct method to solve linearly solvable continuous stochastic control problems via a consistent walk on spheres integrator. This integrator takes large steps in complex geometric domains without suffering from any approximation error associated with time discretization. Consequently, our method does not require any hyperparameter tuning or ablation on the Δt\Delta t parameter used by other continuous control algorithms. We also demonstrate our method’s efficiency over standard RL baselines in point navigation problems.

To the best of our knowledge, our approach of using a discretization-free integrator has not been applied before in continuous control or RL, potentially opening the door to a broad range of follow-on work. Given the maturity of the field using Euler-Maruyama methods, we look forward to developing further algorithmic extensions, e.g., actor-critic and value-based methods that leverage a WoS integrator, and extending the scope of problems it can tackle, e.g. to include path-dependent rewards and more general (angular, configuration space, non-positional, etc) state spaces.

Finally, we note that our Walk-on-Spheres based policy can also be interpreted via the well-established framework of options in hierarchical RL (Sutton, Precup, Singh, 1999). Each sphere may represent a time-abstracted sub-problem in planning defined in terms of sphere exit distribution qθq_\theta (See section 2.1, 122-125). This presents an interesting formalism that can be used to extend our sphere exit policies to more classical step-based controllers.

First, I appreciate the Figures and Tables for the increased training seeds., this amends the concern on “one run per method”

Thank you for also sharing comparison in the Δt\Delta t ablation. A further question I have is the effect of the discount factor for the RL algorithm,γ\gamma, as changing Δt\Delta t effects the problem’s horizon. Was this originally swept over? is this varied with Δt\Delta t?

I also appreciate the experiments covering the fixed start location. However, I am still apprehensive about its implications on needing uniform sampling. Additionally, I am still concerned if 2D/3D point navigation is sufficient to demonstrate “the goal … to establish the WoS integrator in the context of linearly solvable stochastic control”. Could the authors highlight the relevant applications or prior literature that meet the assumptions of the method?

Finally, In regards to your comment replying to “ WoS enviroment integrator could be used by a normal algorithm, like REINFORCE, which would help assess the contribution of the environment versus the algorithm.”:

I appreciate the response to the comment, and also the interesting observation that “Walk-on-Spheres based policy can also be interpreted via the well-established framework of options in hierarchical RL “. However, it still seems like studying the integration with RL would be useful to determine importance of Alg 1 , and the the empirical integration seems feasible as the REINFORCE policy draw on the same sphere-exit action space, mixture policy, and use per-step KL and terminal rewards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.