- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a Walk-on-Spheres (WoS) approach to stochastic control with terminal rewards and quadratic control costs. Through the Hopf–Cole and Doob transforms, the optimal control is related to a harmonic function and its induced sphere-exit distributions. The method learns these distributions using von Mises–Fisher mixtures, replacing small time-discretized steps with geometry-adapted sphere jumps. Experiments compare against reinforcement-learning and path-integral-control baselines on 2D mazes and 3D HM3D scenes.
Overall, I am slightly positive. The paper presents a compelling connection between geometry-aware Monte Carlo sampling and a structured stochastic-control problem, with promising computational results in both 2D and 3D. Its scope is narrower than general reinforcement learning, and the continuous-control interpretation and experimental reliability need clarification. Nevertheless, I believe the methodological contribution is sufficient to support weak acceptance.
Critical strengths
-
The connection between WoS and stochastic control is interesting and well motivated. Under the stated assumptions, the transformation from the HJB equation to a harmonic problem provides a principled basis for learning sphere-exit distributions. This offers a concrete alternative to time-discretized rollouts.
-
The computational benefit is supported by experiments on complex geometry. Geometry-adapted jumps reduce rollout steps and network queries. The 2D and 3D experiments, including goal-conditioned navigation, show promising performance under a shared compute cap.
Critical weaknesses
-
The relationship between the learned policy and executable continuous control remains incomplete. The exact Doob kernel corresponds to the optimal drift, but an arbitrary learned exit distribution need not correspond to a position-only Markov control. Appendix F defines a path measure by filling sphere interiors with conditioned Brownian motion; how this relates to the control-policy class used by the baselines needs clarification. The method also remains restricted to a particular first-exit problem with terminal rewards and quadratic control costs.
-
Some experimental comparisons require more careful interpretation. WoS skips the interior Brownian paths, so its sphere-jump polyline length is not directly comparable to the finely sampled path length of Euler–Maruyama rollouts. Runtime statistics are conditioned on successful walks despite substantial success-rate differences. The reported learning curves also use one run per method, leaving training variability insufficiently characterized.
Questions for authors
-
Does a general learned sphere-exit policy correspond to a position-only continuous drift? If not, how should its policy class and objective be compared with those of the Euler–Maruyama baselines?
-
Could you report repeated-seed results and rollout costs across all attempts, alongside the successful-walk statistics?
-
Could you clarify that Lemma 3.1 establishes fixed-policy asymptotic consistency rather than finite-sample unbiasedness or convergence of online training? How sensitive is training to importance-weight concentration?
4 comments
Part 2
Some experimental comparisons require more careful interpretation. WoS skips the interior Brownian paths, so its sphere-jump polyline length is not directly comparable to the finely sampled path length of Euler–Maruyama rollouts. Runtime statistics are conditioned on successful walks despite substantial success-rate differences. The reported learning curves also use one run per method, leaving training variability insufficiently characterized.
Could you report repeated-seed results and rollout costs across all attempts, alongside the successful-walk statistics?
We run 3 training seeds per scene for HM3D scenes and 2D maze scenes, per algorithm (baselines + our method). We have updated the training curves for Figure (2, 3, 9, 10) to reflect the changes. We have also updated all tables reporting metrics with mean std across 3 different training seeds, instead of evaluation points. Our method still demonstrates strong performance in average reward, success rate, rollout speed compared in 11 / 12 HM3D scenes (See Tables 14, 17, 20, 22, 25, 28). And unlike some baselines (like SAC, PPO, PICE), our method also demonstrates lower variance in training across different seeds (see training curves). The 2D maze scene runs are yet to finish. Hence the related tables contain N/A for now. We’ll report them as soon as they are available.
Additionally, we made the baselines more competitive. We equipped all policy networks with the same hash-grid encoder, and trained each baseline with its best from an ablation (Tables 3 and 4, Appendix H). Even against these baselines, our method wins in 11 / 12 HM3D scenes in average reward. Note for our method, we do not require any such hyper parameter tuning, as it completely avoids time discretization.
Regarding rollout costs, we originally chose time per successful walk because, in a first exit problem, a poorly trained policy looks fast, as its walks fail early on nearby walls. Following the reviewer's recommendation, we now report the unconditional rollout time across all attempts in Table 1, and move time per successful walk to the appendix (Tables 13, 21, 29). Our method is still faster under the new metric.
Regarding walk length, we agree that the polyline length is not comparable between the two integrators. It is not intrinsic for either of them, as a continuous Brownian path has infinite length, so the Euler--Maruyama length grows as . We keep it in the appendix with this caveat, and add the control cost (Lemma E.1) to Table 1, which is computed the same way for both integrators.
Followup
We thank the reviewer for their valuable comments. In response, we have revised the manuscript to include new figures, an additional lemma, and further experiments addressing the points raised. As the discussion period closes by end of day today (AoE), we would be grateful if the reviewer could take a look at the revised manuscript and our posted responses, and let us know whether they resolve the concerns.
I appreciate the author's effort in the rebuttal, and I have no further questions and remain positive.
Part 1
We thank the reviewer for their careful review of the paper. We paraphrase and address the raised concerns below.
A learned exit distribution corresponds to a position-only Markov diffusion at convergence. When qθ matches the Doob kernel pBh, it gives the optimal drift a∗=σ2∇logh (Theorems 2.1 and 3.1). Before convergence, qθ need not correspond to a position-only Markov diffusion, but it is still an admissible control, as its control cost (Lemma F.1) is finite for a vMF mixture with finite κ and finitely many steps. The path measure in Appendix F is the law of the walk under this policy, so its control cost (Lemma F.1) is the same Equation 4 objective that we report for the baselines.
We now state Equation 4 over the wider class of admissible controls, and show in Lemma C.1 that the optimal admissible control is the Markov state-feedback drift a∗(Xt)=σ2∇logh(Xt). Thus, there is no advantage to operating within this wider class, and the comparison with the Markov Euler--Maruyama baselines is fair.
Regarding executable continuous control, our Walk-on-Spheres based policy can potentially be interpreted via the well-established framework of options in hierarchical RL (Sutton, Precup, Singh, 1999), where each sphere represents a time-abstracted sub-problem in planning defined in terms of the sphere exit distribution qθ (see Section 2.1, lines 122--125). Such formalism allows an executable continuous controller to be naturally hierarchical, where a high-level policy chooses the sphere exit, and a low-level controller carries the walk to it. Design of such low-level controller from a trained qθ is left as future work.
We agree with the reviewer. Lemma 3.1 establishes fixed-policy asymptotic consistency, and not finite-sample unbiasedness nor convergence of the online training. We have updated the text in the document to clarify this, and replaced “unbiased”' with “consistent” in Section 3.1.
Regarding importance weight concentration, we ran an ablation that dilutes the importance weights by mixing the policy with a defensive sampling distribution (Appendix J). The average reward is highest with no dilution, and drops as the dilution increases. We suspect the numerical instabilities (or outliers) that come with sharp importance weight regions get amortized away by our neural network, which is fitted over many walks.
Our work provides a direct method to solve linearly solvable continuous stochastic control problems via a consistent walk on spheres integrator. This integrator takes large steps in complex geometric domains without suffering from any approximation error associated with time discretization. Consequently, our method does not require any hyperparameter tuning or ablation on the Δt parameter used by other continuous control algorithms. We also demonstrate our method’s efficiency over standard RL baselines in point navigation problems.
To the best of our knowledge, our approach of using a discretization-free integrator has not been applied before in continuous control or RL, potentially opening the door to a broad range of follow-on work. Given the maturity of the field using Euler-Maruyama methods, we look forward to developing further algorithmic extensions, e.g., actor-critic and value-based methods that leverage a WoS integrator, and extending the scope of problems it can tackle, e.g. to include path-dependent rewards, more general (angular, configuration space, non-positional, etc) state spaces and problems with no closed form doob tilted transition kernel.