- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces PINN-URL, which extends Umbrella Reinforcement Learning with PINNs. The proposed method replaces the original semi-gradient updates for the value function and discounted state occupancy density with direct minimization of their governing equation residuals. The method is evaluated on two low-dimensional benchmarks, Multi-Valley Mountain Car and StandUp, against the original URL and several PPO-based baselines.
Critical strengths
-
This paper is generally straightforward and easy to follow.
-
The experimental results demonstrate substantially reduced variability across independent runs compared with the original URL method. The authors also provide additional PDE residual analyses to support the observed improvements in training consistency.
Critical weaknesses
1. Limited Novelty
The methodological novelty of this work appears to be rather limited. Applying PINNs to reinforcement learning and PDE-based policy evaluation has already been widely explored in prior literature. The additional use of SIREN architectures, exact action summation, and modified training configurations does not appear to introduce a fundamentally new methodological contribution. The paper also provides no new theoretical analysis or convergence guarantees.
2. Lack of Related Work Section
The paper lacks a Related Work section. Although several relevant studies are briefly mentioned in the Introduction, the paper does not provide a sufficiently detailed comparison with existing physics-informed reinforcement learning and PDE-based policy evaluation methods. In particular, continuous-time reinforcement learning (CTRL) is a relatively less explored area compared to discrete-time RL, making a comprehensive discussion of prior work especially important.
3. Limited Applicability to Discrete Action Spaces
The proposed PINN-URL method appears to be limited to discrete action spaces, as the computation of the value and density residuals relies on exact summation over all possible actions.
4. Extremely Limited Experimental Evaluation
The proposed method is only conducted on two toy cases. The baseline methods are also rather limited. The experiments primarily compare PINN-URL against PPO and its variants, without including other competitive continuous-time reinforcement learning methods. In particular, existing PDE-based reinforcement learning and physics-informed policy iteration approaches are not considered. Furthermore, PPO is a trajectory-based method that learns from sampled environment interactions, whereas PINN-URL directly exploits known system dynamics and governing equations. These methods operate under substantially different information assumptions, making the comparisons less informative.
5. Insufficient Evaluation of Time Discretization
Although the proposed method is formulated within a continuous-time reinforcement learning framework, all experiments involving PINN-URL are conducted with a fixed numerical time step of (\Delta t = 0.1). The authors investigate different time-step sizes only for the PPO baselines, without conducting a corresponding sensitivity analysis for the proposed method. Experiments with different discretization resolutions, while preserving the same physical simulation horizon, would provide stronger evidence of the consistency between the continuous-time formulation and its numerical implementation.
6. Overstated Claims Regarding Sparse-Reward Problem
In most sparse-reward RL settings, the agent does not have prior knowledge of the rewarding regions and must discover them through exploration. However, in this work, the authors explicitly sample an additional 2,500 states from the known reward regions at each training iteration, substantially simplifying the learning problem. This setting raises concerns about the claim that the proposed method effectively addresses sparse-reward exploration. In particular, the authors do not provide an ablation study without reward-region-specific sampling, making it difficult to determine how much of the reported improvement comes from the proposed method itself rather than the additional prior knowledge.
Questions for authors
Please refer to the weaknesses discussed above for my main concerns. Also, could the authors clarify the underlying mathematical formulation of the proposed framework? Is it formulated as a continuous-time Markov decision process (MDP), a deterministic continuous-time control problem, or another formulation? In particular, how is the stochastic policy incorporated into the continuous-time dynamics, and under what assumptions does the policy-averaged vector field accurately characterize the resulting state occupancy distribution? And how can the value function converge in this problem formulation? A more rigorous explanation of the relationship between the underlying control process and the governing PDEs would be appreciated.
3 comments
Response to Reviewer fXmM (2/3): Experimental Evidence and Sampling
We address the reviewer's concerns regarding baseline selection, time discretization, and additional value-focused sampling.
4. Experimental comparisons and fairness
We agree that evaluations on higher-dimensional systems and comparisons with additional PDE-based methods would strengthen the evidence for broader applicability. Our primary objective was to determine whether the proposed field-learning procedure improves Original URL on its established benchmarks, rather than to establish superiority over all RL methods.
Original URL is the directly comparable baseline: both methods use the same governing equations, known state-rate field and its divergence, initial density, reward function, and policy-learning framework. They are evaluated over 20 independent runs of 1.2 million training iterations, with common benchmark and evaluation protocols.
On these benchmarks, PINN-URL improves normalized return AUC from 0.329 to 0.512 in MVMC and from 0.982 to 1.274 in StandUp, alongside substantially lower equation residuals and reduced run-to-run variability (Table 1; App. D).
We distinguish this comparison from PPO. PPO learns from environment transitions rather than the same analytical model information, so its results serve as a trajectory-based reference, not a model-information- or compute-matched comparison. Likewise, equal iteration counts for the two URL methods do not establish equal FLOPs or wall-clock costs, since their field architectures and updates differ.
Other PDE-based approaches address related but different control objectives. Adapting them to the same discounted sparse-reward tasks and density-dependent Umbrella objective would allow further controlled comparisons. We do not claim superiority over methods that were not evaluated. These distinctions are explicit in Sec. 4.1, Sec. 1.1, and App. G.
5. Time discretization
The time-step experiments in Appendix G.5 concern PPO, not PINN-URL. PINN-URL learns its value and density fields from continuous-time differential equations whose strong-form residual objectives do not explicitly depend on the simulator integration step.
However, the learned policy is evaluated in a numerically discretized environment, where changing the integration step or action-update frequency can affect observed returns. All reported URL and PINN-URL evaluations use the benchmark step .
Thus, the continuous-time formulation does not by itself establish time-step invariance of evaluated policy performance. We have clarified this distinction without presenting the PPO sensitivity experiments as PINN-URL experiments.
6. Additional reward-region samples
The motivation for additional value-focused collocation was already described in the original submission (Sec. 3, Eq. (12), and App. B).
In MVMC, the rewarded position interval occupies approximately 5.05% of the position domain. Uniform sampling therefore provides relatively limited coverage of this narrow region. The additional 2,500 states are used directly only in the value-equation residual, to address localized value-approximation error. They are not additional environment interactions and do not directly enter the policy or density losses.
We acknowledge that this refinement uses known reward-region geometry. It does not establish reward-blind exploration in an unknown environment; the study operates within URL's known-reward and known-dynamics setting.
We have added a five-seed comparison with and without additional value samples, evaluated using a shared bank of 250 deterministic episodes per checkpoint. The normalized AUCs are 0.51203 ± 0.00256 and 0.51227 ± 0.00402, respectively; final returns are 0.52946 ± 0.00299 and 0.52469 ± 0.00221.
The learning curves show closely similar integrated performance, while the value-field comparison reveals a localized difference near the rewarded region. These are descriptive observations, not evidence that additional sampling has exactly zero effect on return or a conclusive isolation of its causal contribution.
The results are presented in Appendix F.4, Figures 11–12 and Table 8. Adaptive value-residual sampling is a direction for future work.
Response to Reviewer fXmM (3/3): Mathematical Formulation and Policy-Averaged Dynamics
We appreciate the request to clarify the underlying continuous-time formulation, the role of the stochastic policy, and the interpretation of value-function convergence. We have expanded Sec. 2.1 and added a detailed derivation in Appendix H.
7. Mathematical formulation and policy-averaged dynamics
Original URL considers deterministic action-dependent state-rate fields,
together with a stationary stochastic state-feedback policy . Its continuous-ensemble formulation uses the policy-averaged drift
The stochastic policy therefore enters the continuous-time dynamics through the state-dependent averaging of the action-specific vector fields.
Appendix H provides a mathematical interpretation through a rapid-action-resampling limit. If actions are sampled conditionally on the current state at intervals , and the one-step state map satisfies , then for a sufficiently smooth test function ,
=\bar v_\pi\cdot\nabla_s f+O(h).$$ The action-sampling contribution to the covariance of the state increment is $O(h^2)$; its contribution per unit time vanishes as $h\to0$. Under appropriate regularity assumptions, the limiting interior dynamics therefore have generator $\bar v_\pi\cdot\nabla_s$. **Discounted occupancy density** For a fixed policy, let $\lambda=|\log\gamma|>0$. The normalized discounted occupancy measure is $$\mu_\pi=\lambda\int_0^\infty e^{-\lambda t}\mu_t\,dt.$$ Combining probability conservation with this discounted measure gives, where sufficiently regular densities exist, the interior transport equation $$\nabla_s\cdot(p_\pi\bar v_\pi) +\lambda(p_\pi-p_0)=0.$$ The divergence includes the state dependence of the policy probabilities; they are not treated as constants when differentiating the policy-averaged drift. **Fixed-policy value and convergence** For a fixed policy and its corresponding occupancy density, assume that the policy-averaged Umbrella reward is bounded in magnitude by $R$ and that the averaged state flow is well defined with compatible boundary behavior. The infinite-horizon discounted value integral is then finite, with \$ |V_\pi(s)|\leq \frac{R}{\lambda}, \$ and its finite-horizon truncation error is bounded by \$ |V_\pi(s)-V_{\pi,T}(s)| \leq\frac{R}{\lambda}e^{-\lambda T}. \$ Where differentiable, the value satisfies \$ \bar v_\pi\cdot\nabla_s V_\pi +\bar r_U^\pi-\lambda V_\pi=0. \$ For fixed policy and density, the associated discounted Bellman operator over a duration $h$ contracts bounded value functions by $e^{-\lambda h}$. **Scope of these statements** These results establish convergence of the discounted fixed-policy value integral under explicit assumptions. They do **not** establish convergence of neural residual minimization or coupled policy optimization. The continuous-ensemble equations describe an idealized limiting process. Finite-step stochastic action sampling, state clipping, and greedy finite-horizon evaluation need not produce identical occupancy distributions. Likewise, normalization of the theoretical occupancy measure does not imply exact normalization of the learned neural density. Appendix H now provides these derivations and qualifications, while Appendix A documents the boundary-representation limitations. We hope these explanations address the mathematical concerns and would be glad to clarify any remaining points.
Response to Reviewer fXmM (1/3): Contribution, Related Work, and Action Spaces
We appreciate the reviewer's detailed assessment and the recognition of improved consistency across independent runs. We agree that the paper should distinguish more clearly what PINN-URL contributes beyond established physics-informed techniques, how it relates to previous work, and which aspects of the formulation extend beyond the tested action spaces. We have addressed these points in the revised manuscript, with changes marked in blue.
1. Methodological contribution
Our contribution is not the introduction of PINNs, SIRENs, or automatic differentiation individually. Rather, it is a physics-informed learning procedure for the coupled value and occupancy-density equations of Umbrella Reinforcement Learning (URL).
Original URL establishes these equations but trains their neural approximations through stochastic semi-gradient updates. PINN-URL instead minimizes their squared strong-form residuals using full gradients and exact expectations over discrete actions, while retaining the original URL policy update.
This distinction matters because the density enters the effective Umbrella reward, which determines the value equation and contributes to the modified advantage used for policy learning. Improving the accuracy and consistency of these coupled fields is therefore central to URL, rather than an auxiliary prediction problem.
The practical evidence supports this modification. Across 20 independent runs, PINN-URL achieves approximately 63–318-fold lower final median equation-residual MSEs, reaches high policy returns earlier, and substantially reduces run-to-run variability (Sec. 5.2; App. D). The architecture–learning ablations, already included in the original submission (Sec. 5.4; App. F), provide complementary evidence about the contributions of the evaluated configurations.
We do not claim a new general PINN theory or a convergence guarantee for coupled neural optimization. Our contribution is the proposed field-learning procedure and its demonstrated empirical advantages on the evaluated benchmarks.
2. Relationship to previous work
We agree that the original Introduction did not sufficiently distinguish PINN-URL from related physics-informed and PDE-based RL methods. We have added a dedicated Related Work subsection (Sec. 1.1).
The revised discussion compares our formulation with physics-informed model-based RL, physics-informed optimal control, and PDE-based policy iteration, including approaches with convergence analysis. It also discusses related state-density methods.
The central distinction is that PINN-URL directly learns the stationary transport and value equations of URL, coupled through the density-dependent Umbrella reward, while retaining URL's policy-learning framework. We now explain this relationship explicitly rather than suggesting that applying PINNs to RL is itself new.
3. Discrete and continuous actions
The present implementation uses finite action spaces, allowing exact policy-weighted action sums. This is a restriction of the evaluated implementation, not an intrinsic requirement of the governing continuous-time equations.
For continuous actions, the sums formally become integrals under a policy density. Their practical evaluation would require numerical integration or sampling, careful differentiation of policy-dependent expectations, and a suitable treatment of continuous-action entropy. In particular, squaring a Monte Carlo estimate of a residual can introduce bias.
We have clarified this scope in Sec. 3. Continuous-action implementation and experimental validation remain future work.