acceptodds
Under review as a conference paper at ICLR 2027

Controlling PINN Optimization Dynamics with Reinforcement Learning

Abstract

Training a physics-informed neural network (PINN) requires decisions about loss weights, optimizer settings, and the allocation of updates as the solution evolves. We study reinforcement learning (RL) as a controller for these decisions in a holographic inverse problem: reconstructing a scalar potential jointly with bulk fields constrained by gravitational equations and reference data. We formulate the control loop and analyze eight experience-collection schedules for a localized model-based controller, using completed replicates and a non-localized reference. The three strongest individual final-controller evaluations have potential root mean square errors of , , and . Schedule means and medians reveal substantial variation, while saved cycle evaluations show that additional training can reverse earlier gains. Examining the recovered wells and their rising branches further distinguishes a low scalar loss from accurate potential reconstruction. These results identify promising individual trajectories and motivate controller evaluation that accounts for physical geometry, replicate stability, and the distribution of experience across training stages.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Physics-Informed Neural Networks, Reinforcement Learning, Scientific Machine Learning, PINN Optimization, Adaptive Training, Holographic Inverse Problems, Potential Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_aMkYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using a model-based reinforcement learning controller to dynamically adjust PINN optimization settings, including learning rate, Adam parameters, and residual loss weights. The primary experiments investigate eight experience-collection schedules that vary the amount and timing of RL training data. Evaluation is conducted on a single holographic inverse problem involving scalar potential reconstruction. The authors report substantial variation across schedules and replicates, with some promising individual results but incomplete recovery of the target potential.

Critical strengths

The paper reports replicate variability, unsuccessful runs, and non-monotonic performance across controller-training cycles rather than focusing solely on the best outcomes. Its examination of reconstructed potential geometry, beyond scalar optimization loss, can also be valuable for assessing whether numerical improvements translate into meaningful solutions.

Critical weaknesses

1. Lack of benchmarks against existing optimization methods.

The paper compares eight experience-collection schedules for essentially the same RL controller, but provides no comparison against conventional or alternative adaptive PINN optimization methods. No baselines are included. Several important baselines should be considered such as tuned Adam/L-BFGS, adaptive loss weighting (Wang et al. (2021) [1] or ReLoBRaLo (Bischof & Kraus, 2025) [2] as an example), as well as established hyperparameter optimization approaches, including Bayesian optimization for PINNs (Escapil-Inchauspé & Ruz, 2023) [3] and dynamic methods such as Population Based Training (Jaderberg et al., 2017) [4] or PB2 (Parker-Holder et al., 2020) [5].

2. Limited novelty and inconclusive experimental contribution.

The general idea of using RL to dynamically adjust optimization parameters has already been studied in Dynamic Algorithm Configuration (Biedenkapp et al., 2020) [6], while Bihlo (2024) [7] investigates learned optimization specifically for PINNs. The paper does not clearly establish a substantial methodological advance over these directions. Its main distinguishing contribution appears to be the study of experience-collection schedules, but the eight configurations vary multiple factors simultaneously, including memory allocation, exploration, stopping criteria, and training-window lengths, without matched computational budgets. The evaluation further covers only one specialized inverse problem, with three replicates for most schedules and one for some. Consequently, the experiments neither isolate the effect of experience allocation nor establish generalizable insights into RL-based PINN optimization.

References

[1] S. Wang, Y. Teng, and P. Perdikaris. "Understanding and Mitigating Gradient Flow Pathologies in Physics-Informed Neural Networks." SIAM Journal on Scientific Computing, 43(5):A3055–A3081, 2021. doi:10.1137/20M1318043.

[2] R. Bischof and M. A. Kraus. "Multi-Objective Loss Balancing for Physics-Informed Deep Learning." Computer Methods in Applied Mechanics and Engineering, 439:117914, 2025. doi:10.1016/j.cma.2025.117914.

[3] P. Escapil-Inchauspé and G. A. Ruz. "Hyper-Parameter Tuning of Physics-Informed Neural Networks: Application to Helmholtz Problems." Neurocomputing, 561:126826, 2023. doi:10.1016/j.neucom.2023.126826.

[4] M. Jaderberg et al. "Population Based Training of Neural Networks." arXiv:1711.09846, 2017.

[5] J. Parker-Holder, V. Nguyen, and S. J. Roberts. "Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits." Advances in Neural Information Processing Systems 33, pp. 17200–17211, 2020.

[6] A. Biedenkapp, H. F. Bozkurt, T. Eimer, F. Hutter, and M. Lindauer. "Dynamic Algorithm Configuration: Foundation of a New Meta-Algorithmic Framework." Proceedings of ECAI 2020, pp. 427–434, 2020. doi:10.3233/FAIA200122.

[7] A. Bihlo. "Improving Physics-Informed Neural Networks with Meta-Learned Optimization." Journal of Machine Learning Research, 25(14):1–26, 2024.

Questions for authors
  1. Can the authors provide comparisons against conventional PINN optimization, adaptive loss weighting, and non-RL hyperparameter optimization approaches? How much benefit does RL provide relative to its additional computational cost?

  2. What is the specific methodological contribution beyond existing dynamic algorithm configuration and learned PINN optimization?

1 comment
AuthorsOpenReview ↗

Response to reviewer aMkY

We appreciate the reviewer’s concrete suggestions and the positive assessment of our geometry-based evaluation and reporting of unsuccessful runs. We agree that tuned conventional optimizers, adaptive loss weighting, and non-RL hyperparameter-optimization methods are relevant comparisons, and that the current single-problem study cannot establish generality.

The revised paper adds a fixed-Adam vanilla PINN sweep in Table 3 and measured elapsed times in Table 4. Vanilla PINN mean best potential RMSEs are 1.83 at 1M epochs (187.6 min), 1.85 at 3M (492.5 min), and 1.86 at 10M (1532.0 min). Selected RL results include Schedule 7 at mean RMSE 1.49 and 188.9 min, and Schedules 1–3 at means 1.38–1.56 and 354.8–398.4 min. RL timings include controller training and experience collection; collection uses eight parallel workers. Since runs are unpaired and the compute setups differ, these tables provide useful context but do not establish equal-compute superiority.

We recognize the connection to learned optimizer control and algorithm configuration. The focus here is a sequential controller acting on the evolving PINN training process, with experience-collection schedules evaluated for one holographic inverse problem; it is not a claim of a general-purpose advance over dynamic algorithm configuration or learned optimization. We have not added direct experiments against L-BFGS, adaptive weighting, Bayesian optimization, PBT, or PB2, and now provide the vanilla Adam reference while keeping those broader comparisons as future work. The paper’s conclusions remain limited to a proof of concept on this equation.

Reviewer_6P88OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses RL to control PINN training for a holographic inverse problem. It compares eight experience-collection schedules for a localized model-based controller. The results show some promising individual runs, but also substantial instability and a mismatch between low training loss and accurate physical reconstruction.

Critical strengths
  1. Controlling PINN training is a reasonable problem to study. Evaluating the recovered potential’s shape is also useful.
  2. The paper reports unsuccessful runs and deterioration across training cycles, rather than focusing only on the best results. This makes the limitations of the current approach easier to assess.
Critical weaknesses
  1. I think the contribution of this paper is limited. The work mainly applies existing RL control ideas and localization techniques to a particular inverse problem. It is unclear what new insight emerges beyond this application.
  2. The empirical evidence is insufficient in the current version. The main experiments compare different schedules for the same controller, while the non-localized reference also uses RL. Without more comparisons against fixed training schedules or adaptive loss weighting, it is unclear whether RL offers enough benefit to justify its added complexity. With only one to three runs per configuration and substantial variation, the results do not establish a reliable improvement. Overall, I find the contribution too limited and the evaluation too incomplete to support acceptance.
  3. Several settings change together, including memory size, exploration, training windows, stopping rules, and residual scales. Most schedules have only three runs, and some have one. These results make it difficult to identify which design choices matter.
Questions for authors
  1. Does RL outperform well-tuned non-RL alternatives under the same total computational budget, including controller training and experience collection?
  2. Can the schedule effects be isolated through controlled comparisons with matched initial conditions, budgets, and more independent runs?
2 comments
AuthorsOpenReview ↗

Response to reviewer 6P88

Thank you for recognizing the value of evaluating recovered geometry and for the positive assessment of our reporting of unsuccessful runs. We agree that three replicates for most schedules and substantial between-run variation limit the strength of statistical conclusions; because multiple schedule factors vary together, the current experiments also do not isolate the causal effect of each design choice.

The revised paper adds a fixed-Adam vanilla PINN sweep (Table 3) and measured wall-clock context (Table 4). Vanilla PINN means are 1.83 at 1M epochs (n=4, 187.6 min), 1.85 at 3M (n=4, 492.5 min), and 1.86 at 10M (n=3, 1532.0 min). For the RL schedules, Schedule 7 has mean RMSE 1.49 (n=3, 188.9 min), while Schedules 1–3 have means 1.38–1.56 (n=3 each, 354.8–398.4 min). Thus Schedule 7 is a useful similar-time reference to the 1M vanilla run, and Schedules 1–3 have lower observed means and elapsed times than the 3M and 10M references. The strongest individual RL endpoints (0.761, 0.770, 1.14) are individual outcomes, not typical schedule performance.

These are descriptive comparisons across unpaired runs and differing compute setups, not controlled equal-compute evidence. RL elapsed time includes experience collection and controller training, with collection parallelized across eight workers. We have therefore added comparative context while retaining the paper’s limited proof-of-concept scope; establishing schedule effects or statistical superiority will require matched, larger experiments.

Thank you for the response and new experiments. However, I think the contribution of this paper remains limited and the current version is not suitable for ICLR. I will therefore keep my score.

Reviewer_kYmAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is about training physics-informed neural network (PINN). Training PINN needs tuning some of the hyperparameters. The motivation is set in the intro: the impact of residuals, boundary information propagation, and the need for adaptation as the approximation improves.

The paper propose an RL framework for optimizing such parameters. The input is residuals and training context, and it is mapped to optimization settings. The application is limited to the holographic inverse problem.

The contributions are claimed as follows:

  • Proposing a sequential control formulation to map residual observations to optimization actions and experience collection.
  • Quoting: "analyze eight schedules for localized FLEXplore, varying the distribution of memories, exploration, and the length of training stages."
  • Quoting: "Third, we combine aggregate and replicate results with saved cycle potentials to examine when numerical progress corresponds to recovery of the target geometry."

The main method is presented in Section 3, with 3.1 on the policy model that controls PINN training trajectory and reward, and training objective in section 3.3.

Section 4 is dedicated to details of the experimental setup, and results are presented in Section 5.

Critical strengths

Given that the method seems quite familiar, it is difficult to highlight a critical strength.

Critical weaknesses

In general, the paper applied a version of well known appoarch (RL based hyperparameter optimization) to a single problem (holographic inverse problem) of specific method (PINN). Therefore, the scope is limited, and I have difficulty finding the novelty in method. It is not clear whether the paper tries to solve a problem in hyperparameter optimiziation, PINN optimization or holographic inverse problem. It is difficult to see from experiments if the method brings any gain (see more below).

More comments below.

  • The paper doesn't compare against vanilla hyperparameter search, fixed or standard learning-rate schedules, or any non-RL training. This is important to see the final gain.
  • Training challenges mentioned in this work is not, at a higher level, much different from training other neural networks. Namely, one can introduce a second optimization level that maps available information from the lower level optimization loop to the action space of optimization parameters. There are many works in the literature (on meta-learning and bi-level optimization) that address these challenges.
  • Meta-learning in PINN has also been discussed in the following work and needs to be discussed:
  • More specifically, the proposed controller is a special case of learned hyperparameter control (the papers I listed above). The observation is the training statistics, the action is learning rate, Adam moments, and other optimization parameters, and the reward is log-loss improvement.
  • The controller is trained only on a single problem instance, so the generalization of the approach is not shown.
  • I feel that the terms like memory targets and window are not completely standard RL terms. For example, memory target means, I think, the number of transitions used for training, or window is episode horizon. It would be good to map these terminology to standard RL formalism and terminology, making the paper easier to understand.
Questions for authors
  • One of the main challenges of such meta-learning methods is the computation costs. Can the authors comment on the cost compared to simple hyperparameter search and fixed scheduler based PINN training?
  • Can author report numbers for other methods like vanilla PINN training, hyperparameter tuned optimizers like Adam? This is important to see if the proposed method provide gains.
2 comments
AuthorsOpenReview ↗

Response to reviewer kYmA

We thank the reviewer for highlighting both the relationship to learned optimizer control and the need for a conventional reference. We agree that our evidence is scoped to one holographic inverse problem; the contribution is an initial PINN-specific study of a sequential control loop, not a claim of a general-purpose optimizer or an exhaustive hyperparameter-search method. The controller selects training actions from the evolving PINN state during an inner training trajectory, after which the PINN parameters are updated under those actions.

The revised paper now reports a fixed-Adam vanilla PINN sweep in Table 3: mean best potential RMSEs are 1.83, 1.85, and 1.86 at 1M, 3M, and 10M epochs (n=4, 4, and 3; observed wall times 187.6, 492.5, and 1532.0 minutes). Table 4 adds selected RL schedules with elapsed time including experience collection and controller training: Schedule 7 has mean RMSE 1.49 at 188.9 minutes; Schedules 1–3 have means 1.38–1.56 at 354.8–398.4 minutes. These give a conventional reference, but do not establish statistical significance or parity with every non-RL tuning scheme. RL experience collection uses eight parallel workers, so wall-clock time reflects a parallel workflow. We have added baseline context while keeping the paper’s scope limited to this proof of concept; L-BFGS, adaptive weighting, and black-box hyperparameter methods remain relevant future comparisons.

I would like to thank the authors for their response. The new experiment on Adam remains useful. However, my expectation of the research work is to have findings applicable to larger PINN training, unless holographic inverse problems pose a particular challenge and are impactful on their own (for example, like CryoEM). I am not convinced about this importance, and in general, the scope of experiments and baselines remains limited. Therefore, unfortunately, I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.