acceptodds
Under review as a conference paper at ICLR 2027

PINNs Meet Umbrella Reinforcement Learning: Improved Convergence and Training Consistency

Abstract

Hard reinforcement learning problems combine sparse or delayed rewards, state traps, and the absence of a terminal state. Umbrella Reinforcement Learning (URL) addresses them with a continuous ensemble of agents described by a discounted state occupancy density and a steady-state value function, which enter the policy update through an entropy-augmented reward and a modified advantage. The original URL implementation trains these two quantities with stochastic semi-gradient updates that do not directly enforce their governing equations. We introduce PINN-URL, a physics-informed training scheme that learns both of them by minimizing their squared strong-form residuals, using sinusoidal representation networks (SIRENs) and exact sums over discrete actions, while retaining the URL policy update. On Multi-Valley Mountain Car and StandUp, PINN-URL converges faster and to a higher final return, with substantially lower run-to-run variability. Tuned PPO, including long-horizon, reward-shaped, and locally imitation-assisted variants, does not reach comparable return at the target discretization within its interaction budget. In representative runs, PINN-URL's learned value function and occupancy density agree more closely with policy-dependent simulation references. Ablations show that the joint state–action entropy is essential on Multi-Valley Mountain Car, and that neither the network architecture nor the learning configuration alone is advantageous on both benchmarks.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, physics-informed neural networks, umbrella reinforcement learning, continuous-time reinforcement learning, sparse rewards, state occupancy density, value function approximation, hard reinforcement learning problems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fXmMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces PINN-URL, which extends Umbrella Reinforcement Learning with PINNs. The proposed method replaces the original semi-gradient updates for the value function and discounted state occupancy density with direct minimization of their governing equation residuals. The method is evaluated on two low-dimensional benchmarks, Multi-Valley Mountain Car and StandUp, against the original URL and several PPO-based baselines.

Critical strengths
  1. This paper is generally straightforward and easy to follow.

  2. The experimental results demonstrate substantially reduced variability across independent runs compared with the original URL method. The authors also provide additional PDE residual analyses to support the observed improvements in training consistency.

Critical weaknesses

1. Limited Novelty

The methodological novelty of this work appears to be rather limited. Applying PINNs to reinforcement learning and PDE-based policy evaluation has already been widely explored in prior literature. The additional use of SIREN architectures, exact action summation, and modified training configurations does not appear to introduce a fundamentally new methodological contribution. The paper also provides no new theoretical analysis or convergence guarantees.

2. Lack of Related Work Section

The paper lacks a Related Work section. Although several relevant studies are briefly mentioned in the Introduction, the paper does not provide a sufficiently detailed comparison with existing physics-informed reinforcement learning and PDE-based policy evaluation methods. In particular, continuous-time reinforcement learning (CTRL) is a relatively less explored area compared to discrete-time RL, making a comprehensive discussion of prior work especially important.

3. Limited Applicability to Discrete Action Spaces

The proposed PINN-URL method appears to be limited to discrete action spaces, as the computation of the value and density residuals relies on exact summation over all possible actions.

4. Extremely Limited Experimental Evaluation

The proposed method is only conducted on two toy cases. The baseline methods are also rather limited. The experiments primarily compare PINN-URL against PPO and its variants, without including other competitive continuous-time reinforcement learning methods. In particular, existing PDE-based reinforcement learning and physics-informed policy iteration approaches are not considered. Furthermore, PPO is a trajectory-based method that learns from sampled environment interactions, whereas PINN-URL directly exploits known system dynamics and governing equations. These methods operate under substantially different information assumptions, making the comparisons less informative.

5. Insufficient Evaluation of Time Discretization

Although the proposed method is formulated within a continuous-time reinforcement learning framework, all experiments involving PINN-URL are conducted with a fixed numerical time step of (\Delta t = 0.1). The authors investigate different time-step sizes only for the PPO baselines, without conducting a corresponding sensitivity analysis for the proposed method. Experiments with different discretization resolutions, while preserving the same physical simulation horizon, would provide stronger evidence of the consistency between the continuous-time formulation and its numerical implementation.

6. Overstated Claims Regarding Sparse-Reward Problem

In most sparse-reward RL settings, the agent does not have prior knowledge of the rewarding regions and must discover them through exploration. However, in this work, the authors explicitly sample an additional 2,500 states from the known reward regions at each training iteration, substantially simplifying the learning problem. This setting raises concerns about the claim that the proposed method effectively addresses sparse-reward exploration. In particular, the authors do not provide an ablation study without reward-region-specific sampling, making it difficult to determine how much of the reported improvement comes from the proposed method itself rather than the additional prior knowledge.

Questions for authors

Please refer to the weaknesses discussed above for my main concerns. Also, could the authors clarify the underlying mathematical formulation of the proposed framework? Is it formulated as a continuous-time Markov decision process (MDP), a deterministic continuous-time control problem, or another formulation? In particular, how is the stochastic policy incorporated into the continuous-time dynamics, and under what assumptions does the policy-averaged vector field accurately characterize the resulting state occupancy distribution? And how can the value function converge in this problem formulation? A more rigorous explanation of the relationship between the underlying control process and the governing PDEs would be appreciated.

3 comments
AuthorsOpenReview ↗

Response to Reviewer fXmM (1/3): Contribution, Related Work, and Action Spaces

We appreciate the reviewer's detailed assessment and the recognition of improved consistency across independent runs. We agree that the paper should distinguish more clearly what PINN-URL contributes beyond established physics-informed techniques, how it relates to previous work, and which aspects of the formulation extend beyond the tested action spaces. We have addressed these points in the revised manuscript, with changes marked in blue.

1. Methodological contribution

Our contribution is not the introduction of PINNs, SIRENs, or automatic differentiation individually. Rather, it is a physics-informed learning procedure for the coupled value and occupancy-density equations of Umbrella Reinforcement Learning (URL).

Original URL establishes these equations but trains their neural approximations through stochastic semi-gradient updates. PINN-URL instead minimizes their squared strong-form residuals using full gradients and exact expectations over discrete actions, while retaining the original URL policy update.

This distinction matters because the density enters the effective Umbrella reward, which determines the value equation and contributes to the modified advantage used for policy learning. Improving the accuracy and consistency of these coupled fields is therefore central to URL, rather than an auxiliary prediction problem.

The practical evidence supports this modification. Across 20 independent runs, PINN-URL achieves approximately 63–318-fold lower final median equation-residual MSEs, reaches high policy returns earlier, and substantially reduces run-to-run variability (Sec. 5.2; App. D). The architecture–learning ablations, already included in the original submission (Sec. 5.4; App. F), provide complementary evidence about the contributions of the evaluated configurations.

We do not claim a new general PINN theory or a convergence guarantee for coupled neural optimization. Our contribution is the proposed field-learning procedure and its demonstrated empirical advantages on the evaluated benchmarks.

2. Relationship to previous work

We agree that the original Introduction did not sufficiently distinguish PINN-URL from related physics-informed and PDE-based RL methods. We have added a dedicated Related Work subsection (Sec. 1.1).

The revised discussion compares our formulation with physics-informed model-based RL, physics-informed optimal control, and PDE-based policy iteration, including approaches with convergence analysis. It also discusses related state-density methods.

The central distinction is that PINN-URL directly learns the stationary transport and value equations of URL, coupled through the density-dependent Umbrella reward, while retaining URL's policy-learning framework. We now explain this relationship explicitly rather than suggesting that applying PINNs to RL is itself new.

3. Discrete and continuous actions

The present implementation uses finite action spaces, allowing exact policy-weighted action sums. This is a restriction of the evaluated implementation, not an intrinsic requirement of the governing continuous-time equations.

For continuous actions, the sums formally become integrals under a policy density. Their practical evaluation would require numerical integration or sampling, careful differentiation of policy-dependent expectations, and a suitable treatment of continuous-action entropy. In particular, squaring a Monte Carlo estimate of a residual can introduce bias.

We have clarified this scope in Sec. 3. Continuous-action implementation and experimental validation remain future work.

AuthorsOpenReview ↗

Response to Reviewer fXmM (2/3): Experimental Evidence and Sampling

We address the reviewer's concerns regarding baseline selection, time discretization, and additional value-focused sampling.

4. Experimental comparisons and fairness

We agree that evaluations on higher-dimensional systems and comparisons with additional PDE-based methods would strengthen the evidence for broader applicability. Our primary objective was to determine whether the proposed field-learning procedure improves Original URL on its established benchmarks, rather than to establish superiority over all RL methods.

Original URL is the directly comparable baseline: both methods use the same governing equations, known state-rate field and its divergence, initial density, reward function, and policy-learning framework. They are evaluated over 20 independent runs of 1.2 million training iterations, with common benchmark and evaluation protocols.

On these benchmarks, PINN-URL improves normalized return AUC from 0.329 to 0.512 in MVMC and from 0.982 to 1.274 in StandUp, alongside substantially lower equation residuals and reduced run-to-run variability (Table 1; App. D).

We distinguish this comparison from PPO. PPO learns from environment transitions rather than the same analytical model information, so its results serve as a trajectory-based reference, not a model-information- or compute-matched comparison. Likewise, equal iteration counts for the two URL methods do not establish equal FLOPs or wall-clock costs, since their field architectures and updates differ.

Other PDE-based approaches address related but different control objectives. Adapting them to the same discounted sparse-reward tasks and density-dependent Umbrella objective would allow further controlled comparisons. We do not claim superiority over methods that were not evaluated. These distinctions are explicit in Sec. 4.1, Sec. 1.1, and App. G.

5. Time discretization

The time-step experiments in Appendix G.5 concern PPO, not PINN-URL. PINN-URL learns its value and density fields from continuous-time differential equations whose strong-form residual objectives do not explicitly depend on the simulator integration step.

However, the learned policy is evaluated in a numerically discretized environment, where changing the integration step or action-update frequency can affect observed returns. All reported URL and PINN-URL evaluations use the benchmark step Δt=0.1\Delta t=0.1.

Thus, the continuous-time formulation does not by itself establish time-step invariance of evaluated policy performance. We have clarified this distinction without presenting the PPO sensitivity experiments as PINN-URL experiments.

6. Additional reward-region samples

The motivation for additional value-focused collocation was already described in the original submission (Sec. 3, Eq. (12), and App. B).

In MVMC, the rewarded position interval occupies approximately 5.05% of the position domain. Uniform sampling therefore provides relatively limited coverage of this narrow region. The additional 2,500 states are used directly only in the value-equation residual, to address localized value-approximation error. They are not additional environment interactions and do not directly enter the policy or density losses.

We acknowledge that this refinement uses known reward-region geometry. It does not establish reward-blind exploration in an unknown environment; the study operates within URL's known-reward and known-dynamics setting.

We have added a five-seed comparison with and without additional value samples, evaluated using a shared bank of 250 deterministic episodes per checkpoint. The normalized AUCs are 0.51203 ± 0.00256 and 0.51227 ± 0.00402, respectively; final returns are 0.52946 ± 0.00299 and 0.52469 ± 0.00221.

The learning curves show closely similar integrated performance, while the value-field comparison reveals a localized difference near the rewarded region. These are descriptive observations, not evidence that additional sampling has exactly zero effect on return or a conclusive isolation of its causal contribution.

The results are presented in Appendix F.4, Figures 11–12 and Table 8. Adaptive value-residual sampling is a direction for future work.

AuthorsOpenReview ↗

Response to Reviewer fXmM (3/3): Mathematical Formulation and Policy-Averaged Dynamics

We appreciate the request to clarify the underlying continuous-time formulation, the role of the stochastic policy, and the interpretation of value-function convergence. We have expanded Sec. 2.1 and added a detailed derivation in Appendix H.

7. Mathematical formulation and policy-averaged dynamics

Original URL considers deterministic action-dependent state-rate fields,

s˙=v(s,a),\dot{s}=v(s,a),

together with a stationary stochastic state-feedback policy π(a∣s)\pi(a\mid s). Its continuous-ensemble formulation uses the policy-averaged drift

vˉπ(s)=∑a∈Aπ(a∣s)v(s,a).\bar v_\pi(s)=\sum_{a\in\mathcal A}\pi(a\mid s)v(s,a).

The stochastic policy therefore enters the continuous-time dynamics through the state-dependent averaging of the action-specific vector fields.

Appendix H provides a mathematical interpretation through a rapid-action-resampling limit. If actions are sampled conditionally on the current state at intervals hh, and the one-step state map satisfies Th(s,a)=s+h v(s,a)+O(h2)T_h(s,a)=s+h\,v(s,a)+O(h^2), then for a sufficiently smooth test function ff,

=\bar v_\pi\cdot\nabla_s f+O(h).$$ The action-sampling contribution to the covariance of the state increment is $O(h^2)$; its contribution per unit time vanishes as $h\to0$. Under appropriate regularity assumptions, the limiting interior dynamics therefore have generator $\bar v_\pi\cdot\nabla_s$. **Discounted occupancy density** For a fixed policy, let $\lambda=|\log\gamma|>0$. The normalized discounted occupancy measure is $$\mu_\pi=\lambda\int_0^\infty e^{-\lambda t}\mu_t\,dt.$$ Combining probability conservation with this discounted measure gives, where sufficiently regular densities exist, the interior transport equation $$\nabla_s\cdot(p_\pi\bar v_\pi) +\lambda(p_\pi-p_0)=0.$$ The divergence includes the state dependence of the policy probabilities; they are not treated as constants when differentiating the policy-averaged drift. **Fixed-policy value and convergence** For a fixed policy and its corresponding occupancy density, assume that the policy-averaged Umbrella reward is bounded in magnitude by $R$ and that the averaged state flow is well defined with compatible boundary behavior. The infinite-horizon discounted value integral is then finite, with \$ |V_\pi(s)|\leq \frac{R}{\lambda}, \$ and its finite-horizon truncation error is bounded by \$ |V_\pi(s)-V_{\pi,T}(s)| \leq\frac{R}{\lambda}e^{-\lambda T}. \$ Where differentiable, the value satisfies \$ \bar v_\pi\cdot\nabla_s V_\pi +\bar r_U^\pi-\lambda V_\pi=0. \$ For fixed policy and density, the associated discounted Bellman operator over a duration $h$ contracts bounded value functions by $e^{-\lambda h}$. **Scope of these statements** These results establish convergence of the discounted fixed-policy value integral under explicit assumptions. They do **not** establish convergence of neural residual minimization or coupled policy optimization. The continuous-ensemble equations describe an idealized limiting process. Finite-step stochastic action sampling, state clipping, and greedy finite-horizon evaluation need not produce identical occupancy distributions. Likewise, normalization of the theoretical occupancy measure does not imply exact normalization of the learned neural density. Appendix H now provides these derivations and qualifications, while Appendix A documents the boundary-representation limitations. We hope these explanations address the mathematical concerns and would be glad to clarify any remaining points.
Reviewer_7mXFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PINN-URL, a modified implementation of Umbrella Reinforcement Learning. URL represents a continuous ensemble of agents through a discounted occupancy density and a steady-state value function. These fields satisfy a stationary transport equation and a continuous-time value equation. The original URL updates their neural approximations with stochastic semi-gradients. The proposed PINN-URL instead minimizes both equation residuals using six-layer SIRENs and exact action expectations while retaining URL’s policy update. The authors then show that on MVMC and StandUp benchmarks, PINN-URL improves mean return and AUC, reduces variability and residual errors, and outperforms the tested PPO variants.

Critical strengths
  • organisation of the paper is clear
  • the paper is generally readable
Critical weaknesses
  • The benchmarks are way too simple. Both benchmarks are two-dimensional, deterministic, fully observed systems introduced in the prior URL work.
  • The comparison also seems unfair. PINN-URL assumes exact access to more information: the vector field, its divergence, the admissible domain, the initial density, the reward-region geometry, etc.
  • The novel appears to be quite limited. The URL equations and semi-gradient updates already exist. It appears that PINN-URL simply applies a standard squared strong-form residual objective, automatic differentiation, and SIREN representation to those equations. Maybe there are practical benefits, but the current benchmarks do not convincingly demonstrate this.
Questions for authors
  • How does PINN-URL perform on higher-dimensional, stochastic, or partially observed tasks?
  • How does PINN-URL compare with baselines given the same model information and computational budget?
  • What is the precise boundary condition for the density equation under each environment, and how does the input map impose the corresponding flux relation?
  • Which gains come specifically from residual minimization rather than SIRENs or other training changes?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 7mXF (1/2): Scope, Fairness, and Contribution

We appreciate the reviewer's careful assessment. The questions about benchmark scope, comparison fairness, and the contribution beyond established PINN techniques are important. Our central claim is that PINN-URL improves the empirical learning and consistency of URL's coupled fields on the two established benchmarks, not that we have demonstrated general scalability or a theoretical convergence guarantee. We address the points below; the overall changes are summarized in our General Response, and the corresponding manuscript revisions are highlighted in blue.

1. Benchmark scope

MVMC and StandUp are two-dimensional, deterministic, and fully observed. We selected them because they are the original URL benchmarks, allowing a direct comparison under common dynamics and evaluation conditions. Their difficulty is associated with sparse rewards, state traps, and nonterminal dynamics, rather than high-dimensional observations. They also provide distinct transport settings: MVMC has a rectangular domain and divergence-free state-rate field; StandUp has a constrained domain and nonzero divergence (Sec. 2.3; App. A). We agree that these results alone cannot establish performance on higher-dimensional, stochastic, or partially observed problems. The revised Sec. 6 states this scope explicitly.

2. Model information and comparison budget

PINN-URL does not receive additional analytical dynamics information relative to Original URL. Both use the known action-dependent state-rate field, its divergence, the initial density, and the reward, together with the same simulator dynamics and boundary representations. Both retain the same policy architecture. Thus, our primary comparison tests two URL field-learning configurations with common underlying model information.

We also matched the training-iteration and evaluation protocols: 20 independent runs per method, 1.2 million iterations per run, γ=0.95\gamma=0.95, effective entropy coefficient 0.010.01, and policy evaluation on the same 250 initial states, time step, and episode horizons (Sec. 4; Apps. B–C). Each iteration updates the policy, value, and density networks. However, the field architectures, residual computations, and value-focused collocation differ. Consequently, we do not equate the iteration budget with identical FLOPs or wall-clock cost.

Within this explicitly defined comparison, PINN-URL delivers a final return of 0.531 ± 0.002 versus 0.454 ± 0.053 in MVMC and 1.304 ± 0.007 versus 1.183 ± 0.411 in StandUp. Normalized return AUC increases from 0.329 to 0.512 and 0.982 to 1.274, respectively (Table 1). These results demonstrate substantially earlier and more consistent high-return learning over the same training-iteration horizon, not a measured wall-clock speedup. PPO serves as a trajectory-based reference; because it learns from interactions without the same analytical model information, we do not present it as a compute- or information-matched alternative (Sec. 4.1; App. G).

3. Methodological contribution

We agree that PINNs, SIRENs, and automatic differentiation are established tools. The contribution lies in how the coupled URL value and occupancy-density equations are learned. Original URL uses stochastic semi-gradient field updates; PINN-URL minimizes the corresponding squared strong-form residuals with full gradients and exact expectations over discrete actions, while preserving URL's policy update. These fields are not auxiliary predictions: the density enters the effective Umbrella reward, and the value gradient enters the modified advantage that drives the policy update.

Across 20 runs, the final median mean-squared equation residuals are reduced by approximately 63–318-fold (Sec. 5.2; App. D). Together with the return and variability results, this provides evidence of improved numerical field learning and training consistency. We do not claim that this establishes convergence of the coupled neural optimization process.

AuthorsOpenReview ↗

Response to Reviewer 7mXF (2/2): Boundary Conditions and Ablations

4. Density boundary conditions

Both methods inherit the same fixed, environment-specific input representations from Original URL, no separate boundary-loss term is used. The distinction between the constraints imposed by these representations and an exact physical zero-flux condition is now explicit in Sec. 3 and App. A.

For MVMC, the value and density fields use h(x,u)=(x,c(u)2,c(u)d(x))h(x,u)=(x,c(u)^2,c(u)d(x)), where c(u)=cos⁡(π(u+0.07)/0.14)c(u)=\cos(\pi(u+0.07)/0.14) and d(x)=min⁡{∣x+0.99∣,∣x−0.99∣}d(x)=\min\{|x+0.99|,|x-0.99|\}. For either field FF, this implies ∂uF(x,±0.07)=0\partial_u F(x,\pm0.07)=0 and F(xb,u)=F(xb,−u)F(x_b,u)=F(x_b,-u) at xb=±0.99x_b=\pm0.99. For density, the latter symmetry yields the integrated signed position-flux relation ∫−0.070.07u p(xb,u) du=0\int_{-0.07}^{0.07}u\,p(x_b,u)\,du=0, since the position rate is uu. It does not impose zero normal flux pointwise at each boundary state.

For StandUp, the input map h(ϕ1,ϕ2)=(sin⁡ϑ1,sin⁡ϑ2)h(\phi_1,\phi_2)=(\sin\vartheta_1,\sin\vartheta_2), with the coordinate transformation defined in App. A, imposes derivative constraints at the edges of the transformed domain. These are not, by themselves, exact pointwise physical flux conditions. Both simulators also clip states instead of implementing specular reflection, so we do not assert exact agreement between their finite-step boundary dynamics and the neural constraints. Appendix A now states these relations and limitations explicitly.

5. Attribution of improvements

This question was already investigated in the submitted paper through the architecture–learning swaps (Sec. 5.4; App. F). We have expanded the explanation and updated Fig. 10 to show both full methods and the swapped configurations.

The ten-seed results reveal a useful benchmark-dependent contrast (Table 2; App. F.2). On MVMC, the Original URL field architecture trained with the PINN-URL learning configuration obtains 0.524 ± 0.013, near complete PINN-URL at 0.529 ± 0.003. Conversely, SIREN fields with Original URL learning obtain 0.032 ± 0.034. This points to the importance of the learning configuration on MVMC.

On StandUp, SIRENs with Original URL learning reach 1.253 ± 0.094, whereas Original URL fields with PINN-URL learning finish at 0.854 ± 0.247, following an earlier improvement; complete PINN-URL reaches 1.313 ± 0.005. The relative contributions of architecture and learning configuration therefore depend on the benchmark, with the complete PINN-URL configuration producing the most consistent outcomes across the two tasks.

These swaps compare learning configurations: residual minimization, action expectations, sampling, and optimization settings change together. They do not isolate residual minimization as the sole cause. What the results establish is the practical benefit of the complete, evaluated PINN-URL method, supported by the separate field-residual analysis.

We would be glad to clarify any remaining questions.

Reviewer_Uar8OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces a new variant of Umbrella RL -- Phyisics informed Neural-network based RL. It shows improvement on both Multi Valley Mountain Car and Stand UP.

Critical strengths

The paper has some really strong results and shows demonstrable progress against standard RL problems. The experimental set up is sound, the baselines to compare against make sense. Overall this is a solid RL contribution that pushes the field forwards.

Critical weaknesses

The paper is undoubtedly difficult to follow without prior experience with URL. As an academic paper this is fine, but it did make it harder to review without Umbrella-specific RL expertise. I think this could be improved with a slightly expanded background section (or suitable appendix). This could also make it easier for a reader to engage with your work without needing to read multiple Umbrella RL papers.

Despite being "hard problems" both environments are relatively simple, low dimensional state-space, I think the paper would be improved by showing PINNRL can provide improvements to more complex problems.

Questions for authors

What drove the selection process for the environments chosen?

1 comment
AuthorsOpenReview ↗

Response to Reviewer Uar8

We appreciate the reviewer's positive assessment and the suggestions concerning accessibility and benchmark selection. We agree that the paper should explain the main quantities of Umbrella Reinforcement Learning (URL) without requiring readers to consult earlier work. We have expanded the background, clarified the purpose of the benchmarks, and stated the limits of the experimental conclusions more explicitly. The changes are marked in blue; our General Response summarizes the revisions across all reviews.

1. Background and accessibility

The submitted paper presented the governing URL equations, but the connections between them deserved a clearer explanation. Section 2.1 now describes how the initial state distribution and policy-averaged dynamics determine the discounted state occupancy density, how this density enters the effective Umbrella reward through joint state–action entropy, and how the resulting value function and its gradient enter the modified advantage used in policy learning.

Section 2.2 also distinguishes Original URL's semi-gradient updates for the value and density fields from PINN-URL's minimization of their governing equation residuals. The underlying URL formulation and policy update remain unchanged.

2. Motivation for benchmark selection

We selected MVMC and StandUp for three reasons.

First, both benchmarks were introduced with Original URL (Nuzhin and Brilliantov, 2025). Using them allows a direct comparison on the same tasks, with common underlying dynamics and evaluation conditions.

Second, both combine sparse rewards, state traps, and nonterminal dynamics. In MVMC, the agent must build oscillations to escape the outer valleys and sustain rewarding behavior near the central hill. In StandUp, reaching the upright region requires a coordinated sequence of folding and unfolding movements. In neither environment does entering the rewarded region terminate the dynamics.

Third, the benchmarks provide distinct mathematical settings. MVMC has a rectangular state domain and a divergence-free state-rate field, whereas StandUp has a geometrically constrained domain and nonzero divergence. This allows us to examine the field-learning procedure under different transport conditions.

These properties are described in Section 2.3 and Appendix A.

3. Benchmark complexity and generalization

We agree that both benchmarks are low-dimensional and that experiments on more complex systems would strengthen the evidence for broader applicability. In the URL framework, however, the term hard refers to sparse rewards, state traps, and the absence of terminal states, rather than high state-space dimensionality.

Our objective was to investigate whether physics-informed field learning improves the existing URL implementation on its established benchmarks. Across 20 independent runs, PINN-URL achieves earlier high returns, lower governing-equation residuals, and substantially reduced run-to-run variability.

These results support the proposed method on the evaluated problems, but do not establish scalability to higher-dimensional, stochastic, or partially observed systems. We have clarified this distinction in Section 6. Extensions to these settings remain directions for further mathematical and experimental investigation.

We would be glad to clarify any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions

We thank the reviewers for their feedback on the mathematical formulation, methodological contribution, experimental comparisons, and scope of our results. We have revised the manuscript to address these points more explicitly and to make the evidence supporting our conclusions easier to assess. The revisions are marked in blue.

The original submission already included architecture–learning ablations in Section 5.4 and Appendix F, as well as the motivation for additional value-focused collocation in Section 3 and Appendix B. We have clarified what these analyses establish, improved their presentation, and added further experimental evidence in response to the questions raised.

Summary of revisions

LocationRevision
Sec. 1.1Expanded related work and clarified the distinction from existing physics-informed and PDE-based RL methods.
Secs. 2.1–2.2Clarified the URL formulation, discounted occupancy density, entropy, and coupled field-learning procedure.
Sec. 3 and App. BClarified residual-based learning and the role of additional value-focused collocation.
App. ASpecified the boundary relations imposed by the neural representations and their limitations.
Sec. 4 and App. CClarified model-information assumptions, evaluation protocols, and differences between URL and PPO training budgets.
Sec. 5.4 and Apps. F.2–F.3Expanded the interpretation of the existing architecture–learning ablations and updated Figure 10.
App. F.4Added the five MVMC runs for comparison with and without additional value samples.
App. GExpanded PPO diagnostics and clarified time-discretization and training-budget comparisons.
Sec. 6 and App. HClarified the scope of our conclusions and added the policy-averaged transport and fixed-policy value derivations.

Additional experimental evidence

To examine the role of additional goal-region value samples, we evaluated five MVMC runs per configuration using a common bank of 250 deterministic evaluation episodes per checkpoint. The normalized AUCs with and without additional samples are 0.51203 ± 0.00256 and 0.51227 ± 0.00402, respectively. The corresponding final returns are 0.52946 ± 0.00299 and 0.52469 ± 0.00221 (mean ± sample standard deviation).

The learning curves show closely similar integrated performance, while the representative value fields reveal a localized difference near the rewarded region (Appendix F.4, Figures 11–12 and Table 8). The existing architecture–learning ablations provide complementary evidence: the relative importance of field representation and learning configuration differs between MVMC and StandUp. We interpret these results in terms of the complete PINN-URL configuration rather than attributing all improvements to a single component.

Remaining limitations

We agree that evaluating PINN-URL on higher-dimensional, stochastic, and partially observed tasks, extending the implementation to continuous actions, and comparing against additional physics-informed and PDE-based reinforcement learning methods would strengthen the study. These directions require further methodological development and experimental validation beyond the discussion period. Although PINN-URL learns from continuous-time equations, the sensitivity of its evaluated policy returns to the simulator time step has not been independently investigated; the time-step experiments in Appendix G concern PPO. We also do not claim a general convergence guarantee for the coupled neural optimization procedure.

The completed revisions will be retained in the camera-ready manuscript if accepted. We address each reviewer's specific concerns in the corresponding response thread and would be glad to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.