acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking Optimizers to Solve Inverse Problems with Differentiable Physics Simulators

Abstract

Solving inverse problems with differentiable physics simulators holds the potential to revolutionize scientific discovery and engineering design, as it enjoys both the strict physical correctness from rigorous numerical physics simulators, and the high efficiency and effectiveness from automatic differentiation and gradient-based optimization. However, currently, this paradigm faces performance issues in optimization. In this work, we target benchmarking the performance of different optimizers to solve various inverse problems. We construct 12 differentiable physics simulators spanning physics domains including discrete mechanics, continuous mechanics, atomistic simulations, rendering, and semi-empirical physics models. Based on these simulators, we design corresponding inverse problems that can be categorized into parameter identification, inverse design, and optimal control. Finally, we conduct extensive experiments to compare the performance of different optimizers, including regular first-order methods, approximate second-order methods, as well as global optimizers, on these inverse problems, and analyze the results to provide insights on how to choose and design optimizers for differentiable programming. We hope such benchmarks can inspire the development of more effective optimizers, and further promote the applications of differentiable programming in various scientific and engineering domains.

Primary area:
datasets and benchmarks
Keywords:
Differentiable Programming, Differentiable Physics Simulator, Inverse Problem, Optimizer, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A7PTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a benchmark comprising 12 differentiable physics models and corresponding inverse problems covering parameter identification, inverse design, and optimal control. It compares 12 optimizers from first-order, approximate second-order, and global-search families, reporting objective values, runtime, and memory usage.

The unified implementation is potentially useful, but the current benchmark resembles a collection of small examples more than a validated tool for practical optimizer selection. Several tasks have limited discriminative power, and the evaluation does not sufficiently establish robustness or transfer of the findings to larger problems. A benchmark need not introduce a new algorithm, but it should provide reliable comparisons and meaningful new insights. I do not think the current submission demonstrates these sufficiently.

Critical strengths
  1. The unified PyTorch implementation spans several physical domains and could reduce the effort required to test optimizers across different inverse problems.

  2. Including runtime and memory alongside objective values is useful, since optimizer iterations can have substantially different computational costs.

Critical weaknesses
  1. The benchmark’s practical significance is insufficiently established. Several tasks provide little separation between optimizers, including projectile motion, flight control, and rigid-body control. The problems are deliberately small, with at most 1,100 optimization variables, and their rankings are not validated across increasing problem scales or difficulty levels. Consequently, it is unclear whether the results offer reliable guidance for realistic applications. The conclusions largely reiterate familiar tradeoffs rather than establishing actionable relationships between problem characteristics and optimizer performance.

  2. The evaluation protocol limits conclusions about reliability and efficiency. Non-converged runs are excluded before hyperparameter selection and aggregation. Although invalid numerical results should not be treated as successful optimization, excluding failures without incorporating failure rates can obscure robustness. Moreover, objective values and per-step costs alone do not establish performance under comparable total computational budgets, particularly when population methods include additional local searches.

Questions for authors
  1. What actionable optimizer-selection guidance follows from this benchmark beyond the observation that different tasks favor different methods?

  2. Can the optimizer rankings be shown to persist as resolution, parameter count, or rollout length increases?

  3. How are non-converged runs defined, and what are the failure rates for each method? Would including failures change the conclusions?

  4. How do the methods compare under matched wall-clock budgets, including all local searches and objective evaluations?

5 comments
AuthorsOpenReview ↗

Official Comment to Reviewer A7PT (Part 1)

Thank you for your precise assessment and constructive comments. We appreciate your recognition that such a unified benchmark is useful for testing optimizers across different inverse problems. Below, we address the proposed weaknesses and questions one by one.

W1 (a) Several tasks provide little separation

On one hand, we agree with the reviewer on this point to a large extent. We had thought about it and realized this is rooted in a very tricky dilemma, that the famous problems in differentiable programming are the ones that have already been verified solvable by current optimizers, while for some other problems, none of the existing optimization methods can obtain outstanding results, which is the major motivation why we built this benchmark. In the future, we plan to introduce more problems where most or all existing optimizers fail.

On the other hand, we would like to highlight that, with the current set of simulators and problems, different types of optimizers already show performance differences. For the problems where optimizers show similar performances, we gain the knowledge that such tasks are not bottlenecked by the optimizers.

W1 (b) No validation across increasing problem scales & Q2 Optimizer performances across different resolutions, parameter counts, or rollout lengths

Thank you for this suggestion. In principle, the problem remains the same mathematical structure for different sizes, hence the suitable optimizer types should hold. However, this conjecture needs to be examined. We reran the topology optimization problem on different mesh resolutions, and the mass-spring problems on different rollout lengths, with 3 fresh seeds and 100 epochs.

Resolution of the Topology Optimization problem.

Optimizer30×2030\times 20 (600)42×2842\times 28 (1176)60×4060\times 40 (2400)
SGD1.536±0.0071.536 \pm 0.0071.540±0.0051.540 \pm 0.0051.649±0.0131.649 \pm 0.013
RMSprop1.563±0.0011.563 \pm 0.0011.529±0.0121.529 \pm 0.0121.526±0.0101.526 \pm 0.010
Adam1.495±0.007\mathbf{1.495 \pm 0.007}1.483±0.013\mathbf{1.483 \pm 0.013}1.476±0.0041.476 \pm 0.004
AdamW1.496±0.0041.496 \pm 0.0041.489±0.0021.489 \pm 0.0021.475±0.004\mathbf{1.475 \pm 0.004}
L-BFGS1.574±0.0141.574 \pm 0.0141.550±0.0011.550 \pm 0.0011.536±0.0101.536 \pm 0.010
K-FAC1.564±0.0081.564 \pm 0.0081.527±0.0071.527 \pm 0.0077.150±0.0497.150 \pm 0.049
Shampoo1.562±0.0061.562 \pm 0.0061.496±0.0041.496 \pm 0.0047.164±0.0217.164 \pm 0.021
SOAP2.022±0.0422.022 \pm 0.0422.090±0.0962.090 \pm 0.0961.489±0.0031.489 \pm 0.003
RS1.544±0.0081.544 \pm 0.0081.704±0.1901.704 \pm 0.1901.852±0.0041.852 \pm 0.004
SA2.664±0.0672.664 \pm 0.0673.043±0.0163.043 \pm 0.0163.706±0.0253.706 \pm 0.025
PS1.536±0.0081.536 \pm 0.0081.583±0.0141.583 \pm 0.0141.747±0.0101.747 \pm 0.010
DE2.564±0.0352.564 \pm 0.0352.902±0.0422.902 \pm 0.0423.411±0.0703.411 \pm 0.070

From the table, we can see that Adam and AdamW remain the best on every mesh resolution. Global optimizers simulated annealing and differential evolution are significantly worse, even with higher computational cost. The performance of Shampoo and K-FAC degrades significantly as the mesh size grows.

Rollout length of the mass–spring problem.

Optimizer2550100
SGD0.201±0.0020.201 \pm 0.0020.210±0.0160.210 \pm 0.0160.197±0.0160.197 \pm 0.016
RMSprop0.208±0.0040.208 \pm 0.0040.162±0.0200.162 \pm 0.0200.088±0.0480.088 \pm 0.048
Adam0.172±0.0340.172 \pm 0.0340.148±0.0200.148 \pm 0.0200.146±0.0100.146 \pm 0.010
AdamW0.218±0.0060.218 \pm 0.0060.173±0.0200.173 \pm 0.0200.162±0.0050.162 \pm 0.005
L-BFGS0.249±0.0000.249 \pm 0.0000.246±0.0020.246 \pm 0.0020.224±0.0340.224 \pm 0.034
K-FAC0.209±0.0050.209 \pm 0.0050.171±0.0180.171 \pm 0.0180.183±0.0090.183 \pm 0.009
Shampoo0.182±0.0080.182 \pm 0.0080.110±0.007\mathbf{0.110 \pm 0.007}0.048±0.010\mathbf{0.048 \pm 0.010}
SOAP0.189±0.0190.189 \pm 0.0190.156±0.0070.156 \pm 0.0070.110±0.0270.110 \pm 0.027
RS0.165±0.0170.165 \pm 0.0170.114±0.0240.114 \pm 0.0240.102±0.0210.102 \pm 0.021
SA0.185±0.0210.185 \pm 0.0210.161±0.0190.161 \pm 0.0190.110±0.0140.110 \pm 0.014
PS0.150±0.026\mathbf{0.150 \pm 0.026}0.137±0.0360.137 \pm 0.0360.135±0.0220.135 \pm 0.022
DE0.185±0.0060.185 \pm 0.0060.171±0.0110.171 \pm 0.0110.132±0.0200.132 \pm 0.020

From the results, we can see that Shampoo basically keeps its leading position. However, for the problem with shorter rollout length, particle swarm and random search show advantages, indicating that for shorter rollout length, the long-rollout problem does not dominate the mathematical characteristics of the problem.

AuthorsOpenReview ↗

Official Comment to Reviewer A7PT (Part 2)

W1 (c) Establishing actionable relationships between problems and optimizers & Q1 Actionable optimizer-selection guidance

Thank you for proposing this suggestion. We structured the optimizer selection guidelines in the flowchart below. Note that this is for reference only, as realistic problems can involve more complicated factors and concerns.

Start: new differentiable inverse problem
            |
            v
      Long rollout?
            |
            +-- yes --> approximate second-order
            |
            +-- no
                      |
                      v
              Ill-conditioned?
                      |
                      +-- yes --> approximate second-order
                      |
                      +-- no
                                |
                                v
                        High-dimensional?
                                |
                                +-- yes --> first-order
                                |
                                +-- no, low-dimensional and nonconvex,
                                |    initialization uncertain
                                |         --> global
                                |
                                +-- no, low-dimensional, well-conditioned,
                                     not strongly nonconvex
                                          --> first-order

On the other hand, we would also like to emphasize that we think a novel optimizer designed specifically for differentiable physics simulators is called for, which is our original motivation for building this benchmark.

W2 (a) Non-converged runs are excluded & Q3 Definition of non-converged runs and failure rates

To guarantee fair comparison, we use a fixed hyperparameter scan for all physics simulators and inverse problems. This leads to the result that very unreasonable hyperparameters can be applied to a specific problem, causing convergence failures. But such failure has little correlation with the performance or stability of a specific optimizer on a specific problem. That is the reason why we only report the best hyperparameter configuration with mean±std over 3 random initializations.

In the current procedure, non-converged runs are first identified by humans to observe unreasonable outcome results. More specifically speaking, in the current benchmark and experimental settings, only the topology optimization problem based on the elastic mechanics simulator produces non-converged runs, in which case non-physical negative objectives are generated. Hence in the experiment, we define a run as non-converged if any of the following holds: the optimizer step raises an exception; the best objective is non-finite (NaN or Inf); or, specifically for topology optimization, the compliance becomes a negative value, which is a failed equilibrium solve rather than a better design.

For the topology optimization problem, 20 of 621 runs (3.2%) have a negative best loss: 18/108 differential-evolution runs (16.7%) and 2/36 K-FAC runs (5.6%). No other optimizers on this task have non-converged runs. The other 11 inverse problem tasks have no failures under the same rule. For all problems and optimizers, none of the reported results (3 random initializations of the best hyperparameter configuration) contain non-converged runs.

AuthorsOpenReview ↗

Official Comment to Reviewer A7PT (Part 3)

W2 (b) Per-step costs are not sufficient & Q4 Comparison under matched wall-clock budgets

Thank you for proposing this suggestion. We reran all 12 tasks, 3 seeds, and 100 epochs, and recorded a timestamp after every epoch. The curves are plotted and added in the revised manuscript as Appendix C.3 Objective versus Wall-Clock Time. For each task, the matched budget is the mean wall-clock time at which the fastest first-order or approximate second-order method finishes 100 epochs. A run that has already finished keeps its final best value.

Where the method with the best objective is itself a fast local method, the matched budget selects the same method. Global methods are behind, because the budget buys them on the order of ten steps against a hundred for the local method. L-BFGS still attains the best objective on tight binding and the battery fit. Shampoo still attains the best objective on the fluid design and the mass–spring reach. Adam still attains the best objective on topology, with AdamW unresolved from it. L-BFGS still attains the best objective on atmospheric scattering and flight, ahead of a narrow band of other local methods. On the fluid design, the global methods do reach Shampoo’s plateau, but only after a run many times longer than this budget. On mass–spring, they stay behind even after a full run.

The best method changes when one local method is much more expensive per step. On tire fitting and Gaussian splatting, the matched budget is a fraction of a second and a first-order method attains the best objective, while L-BFGS is still inside its line search. L-BFGS is the best on those two tasks only when it is allowed to finish that longer run. DFT runs the other way: L-BFGS is the first method to finish, and Adam, SGD, and particle swarm are already at numerical zero. Simulated annealing and differential evolution are still far above that floor and reach it only later.

Projectile and the rigid-body setpoint do not separate under either clock. On the projectile problem, all twelve methods sit on the same float32 value within a tenth of a second. On the rigid-body problem, the twelve methods form a narrow band, and the global methods are only a few steps into their search when the local methods have already finished.

Reviewer_ghjnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a benchmark study that systematically compares the performance of optimization methods for inverse problems involving differentiable physics simulators. Rather than proposing a new optimization method, the study evaluates existing methods across diverse problems, motivated by the observation that commonly used optimizers such as Adam are not necessarily optimal. The benchmark covers parameter identification, inverse design, and optimal control tasks based on 12 physics simulators. A total of 12 optimization methods are compared, including first-order methods, approximate second-order methods, and global optimization methods. The numerical experiments evaluate not only optimization performance but also computational time and GPU memory usage. The results show that L-BFGS generally performs well in parameter identification tasks, whereas no single optimizer consistently outperforms the others across inverse design and optimal control tasks. The authors conclude that there is no universally optimal method and that the choice of optimizer should depend on the characteristics of the problem and computational costs.

Critical strengths

A major strength of this paper is that it addresses the highly important question of the effectiveness of optimization methods for inverse problems in physics-informed machine learning (PIML) through comprehensive numerical experiments. Since conducting experiments on this scale requires a substantial investment of time and computational resources, I believe this work will make a valuable contribution to the research of many scientists in the field.

Critical weaknesses

A weakness of this paper is that the numerical experiments did not identify any specific rules for selecting optimization methods. This means that the study does not provide a systematic procedure that other researchers can follow when choosing appropriate optimization methods for their own problems.

Questions for authors

Based on the findings from these comprehensive numerical experiments, would it be possible to develop a decision-making framework, such as a flowchart, to help researchers select appropriate optimization methods for solving inverse problems in physics-informed machine learning?

2 comments

Thank you for your inspiring comments and for your recognition of the importance of the topic and the comprehensive effort. In the following, we address the weaknesses and questions in turn.

W1: No specific rules for selecting optimization methods.

Thank you for raising this concern. From our experimental results, we summarize general optimizer selection rules in the table below regarding four important factors: dimension, ill-conditioning, nonconvexity, and whether the physics process is a long rollout. We will also see to make such practical guidelines clearer in a modified version of the manuscript in a suitable form (bulleted list or flowchart).

Problem attributeSuitable Optimizer
Ill-conditioned least-squares, spectral, or semi-empirical fitSecond-order
Same fit, but a line search is too expensiveFirst-order
High-dimensional design with an expensive linear solveFirst-order
Low-dimensional, nonconvex, uncertain initialization, no long rolloutGlobal
Long rollout, neither ill-conditioned nor strongly nonconvexFirst-order
Long rollout, ill-conditioned or nonconvexApproximate second-order, or adaptive first-order
Low-dimensional, well-conditioned, not strongly nonconvexFirst-order
Inner loop or tight latencyFirst-order

Q1: A flowchart of the optimization decision-making framework

Thank you for this great suggestion. A flowchart of this kind can be really handy. We structured the optimizer selection guidelines in the flowchart below. Note that this is for reference only, as realistic problems can involve more complicated factors and concerns.

Start: new differentiable inverse problem
            |
            v
      Long rollout?
            |
            +-- yes --> approximate second-order
            |
            +-- no
                      |
                      v
              Ill-conditioned?
                      |
                      +-- yes --> approximate second-order
                      |
                      +-- no
                                |
                                v
                        High-dimensional?
                                |
                                +-- yes --> first-order
                                |
                                +-- no, low-dimensional and nonconvex,
                                |    initialization uncertain
                                |         --> global
                                |
                                +-- no, low-dimensional, well-conditioned,
                                     not strongly nonconvex
                                          --> first-order

On the other hand, we would also like to emphasize that we think a novel optimizer designed specifically for differentiable physics simulators is called for, which is our original motivation for building this benchmark.

I sincerely apologize for my delayed reply. I was unwell for some time, and I also had to make a major life decision during this period, which prevented me from responding sooner.

Thank you for your detailed response. The table and the flowchart directly address my main concern, and I believe practical guidelines of this kind will considerably increase the value of the benchmark for researchers working on inverse problems with differentiable simulators. I also agree that your results call for optimizers designed specifically for differentiable physics simulators, and I see this as an important direction for future work.

I have a few suggestions for incorporating these guidelines into the revised manuscript.

First, the table and the flowchart are not yet fully consistent. For example, the table recommends first-order methods for long-rollout problems that are neither ill-conditioned nor strongly nonconvex, whereas the flowchart sends every long-rollout problem to approximate second-order methods. Unlike the flowchart, the table also restricts the high-dimensional rule to design problems with an expensive linear solve, and it includes cases (an expensive line search, tight latency) that the flowchart does not. In addition, the flowchart does not cover some cases, such as low-dimensional nonconvex problems with a reliable initial guess. It is also unclear whether “second-order” in the first row of the table refers to the approximate second-order methods in the paper. I would recommend reconciling the two before including them in the manuscript.

Second, it would be helpful to show which benchmark results support each rule. For example, you could characterize each task in terms of the four factors you identified and indicate which optimizers performed best on it. Naming the specific methods that performed best within each category, rather than only the category, would also make the guidelines more actionable.

Third, conditioning and the degree of nonconvexity are often unknown before a problem is solved. A brief note on how practitioners might assess these attributes in advance would make the flowchart easier to apply in practice. Possible approaches include a rough estimate of the Hessian spectrum using Hessian-vector products, or a few short multi-start runs.

Overall, I will maintain my current rating of weak acceptance.

Reviewer_86sXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a set of benchmarks that implements 12 differentiable physics simulators across multiple physics domains, defines inverse problems spanning parameter identification, inverse design, and optimal control, and systematically compares first-order, approximate second-order, and global optimizers.

Critical strengths

The benchmark brings diverse physical models and inverse problems into a common framework, providing a useful foundation for cross-domain optimizer evaluation. Its coverage of first-order, approximate second-order, and hybrid global–local methods enables comparisons beyond the standard baselines typically used in individual applications.

Critical weaknesses

What new scientific insight does the benchmark establish?”, rather than merely “Is this a useful benchmark?” The paper claims that the benchmark provides insights into optimizer selection and design for differentiable programming. However, the current paper reads like primarily an infrastructure + empirical benchmarking rather than a new ML/optimization method.

Basically, while the benchmark is useful, but the paper does not introduce a new optimization algorithm or a sufficiently novel theoretical result. Many of the conclusions are intuitive: cheap first-order methods are useful for expensive simulators, while global methods help with nonconvex problems. I have derived no new insight from this paper. More specific concerns include:

  • Limited benchmark discrimination. Several tasks provide little separation between methods. Tables 5, 6 show mostly identical scores on Projectile, Flight Dynamics, and Rigid Body.

  • Incomplete efficiency comparison. Final losses after 100 optimizer epochs and per-step runtimes do not show which optimizer reaches a given accuracy faster, since each epoch involves different amounts of computation across methods. Objective v.s. wall-clock-time curves would better support the practical recommendations.

  • Evaluation excludes unsuccessful runs. Removing numerically invalid results is reasonable, but the reported losses exclude non-converged runs and therefore do not capture overall reliability. The paper should define the exclusion criteria and report failure rates alongside the losses, particularly to support its conclusions about robustness.

  • Limited evaluation of initialization sensitivity. The benchmark uses a single initialization scheme per task (σ_init = 0.05 in Eq. 52), without systematically varying the perturbation scale or starting-point distribution. This limits the evidence for the claimed robustness benefits of global optimizers under random initialization.

Questions for authors

Could the authors report values at higher numerical precision in Tables 5,6 to clarify if the apparent ties reflect similar performance or rounding?

Could the authors provide objective-versus-wall-clock-time curves, define the criteria for excluding non-converged runs, report failure rates alongside losses, and test a wider range of initialization scales and starting-point distributions to support their recommendations on optimizer choice and claims about robustness?

6 comments
AuthorsOpenReview ↗

Official Comment to Reviewer 86sX (Part 1)

Thank you very much for the careful review and the concrete comments, as well as your recognition that the benchmark provides a useful foundation for evaluation on diverse physical domains and optimizers. In the following, we address the weaknesses and questions one by one.

W1: Scientific insight

As a benchmark paper, we claim our major contribution as to bring diverse physics simulators, inverse problems, and optimizers into a unified framework for easy comparisons. We don't intend to design a new optimizer ourself, but to help researchers in the optimization area to realize such requests and to provide a playground easy to test new methods that can effectively applies to diverse physics processes and different inverse problems.

We also would like to emphasize that the benchmark provides systematic and comparative evidence about when established optimizer choices succeed, reverse, or have little effect. Such evidence has scientific value independently of proposing a new method or theorem, difficult to obtain without such a unified framework. The resulting evidence goes beyond the general observation that different problems favor different methods, such as the conclusions that curvature, global optimizer, or long-rollout dynamics don't universally bring benefits or difficulties to the inverse problems.

W2: Limited benchmark discrimination

On one hand, we agree with the reviewer on this point to a large extent. We had thought about it and realized this is rooted in a very tricky dilemma, that the famous problems in differentiable programming are the ones that have already been verified solvable by current optimizers, while for some other problems, none of the existing optimization methods can obtain outstanding results, which is the major motivation why we built this benchmark. In the future, we plan to introduce more problems where most or all existing optimizers fail.

On the other hand, we would like to highlight that, with the current set of simulators and problems, different types of optimizers already show performance differences. For the problems where optimizers show similar performances, we gain the knowledge that such tasks are not bottlenecked by the optimizers.

W3: Incomplete efficiency comparison & Q2: Provide objective-versus-wall-clock-time curves

Thank you for proposing this suggestion. We reran all 12 tasks, 3 seeds, and 100 epochs, and recorded a timestamp after every epoch. The curves are plotted and added in the revised manuscript as Appendix C.3 Objective versus Wall-Clock Time. For each task, the matched budget is the mean wall-clock time at which the fastest first-order or approximate second-order method finishes 100 epochs. A run that has already finished keeps its final best value.

Where the method with the best objective is itself a fast local method, the matched budget selects the same method. Global methods are behind, because the budget buys them on the order of ten steps against a hundred for the local method. L-BFGS still attains the best objective on tight binding and the battery fit. Shampoo still attains the best objective on the fluid design and the mass–spring reach. Adam still attains the best objective on topology, with AdamW unresolved from it. L-BFGS still attains the best objective on atmospheric scattering and flight, ahead of a narrow band of other local methods. On the fluid design, the global methods do reach Shampoo’s plateau, but only after a run many times longer than this budget. On mass–spring, they stay behind even after a full run.

The best method changes when one local method is much more expensive per step. On tire fitting and Gaussian splatting, the matched budget is a fraction of a second and a first-order method attains the best objective, while L-BFGS is still inside its line search. L-BFGS is the best on those two tasks only when it is allowed to finish that longer run. DFT runs the other way: L-BFGS is the first method to finish, and Adam, SGD, and particle swarm are already at numerical zero. Simulated annealing and differential evolution are still far above that floor and reach it only later.

Projectile and the rigid-body setpoint do not separate under either clock. On the projectile problem, all twelve methods sit on the same float32 value within a tenth of a second. On the rigid-body problem, the twelve methods form a narrow band, and the global methods are only a few steps into their search when the local methods have already finished.

AuthorsOpenReview ↗

Official Comment to Reviewer 86sX (Part 2)

W4: Evaluation excludes unsuccessful runs & Q3: Define the criteria for excluding non-converged runs and report failure rates alongside losses

To guarantee fair comparison, we use a fixed hyperparameter scan for all physics simulators and inverse problems. This leads to the result that very unreasonable hyperparameters can be applied to a specific problem, causing convergence failures. But such failure has little correlation with the performance or stability of a specific optimizer on a specific problem. That is the reason why we only report the best hyperparameter configuration with mean±std over 3 random initializations.

In the current procedure, non-converged runs are first identified by humans to observe unreasonable outcome results. More specifically speaking, in the current benchmark and experimental settings, only the topology optimization problem based on the elastic mechanics simulator produces non-converged runs, in which case non-physical negative objectives are generated. Hence in the experiment, we define a run as non-converged if any of the following holds: the optimizer step raises an exception; the best objective is non-finite (NaN or Inf); or, specifically for topology optimization, the compliance becomes a negative value, which is a failed equilibrium solve rather than a better design.

For the topology optimization problem, 20 of 621 runs (3.2%) have a negative best loss: 18/108 differential-evolution runs (16.7%) and 2/36 K-FAC runs (5.6%). No other optimizers on this task have non-converged runs. The other 11 inverse problem tasks have no failures under the same rule. For all problems and optimizers, none of the reported results (3 random initializations of the best hyperparameter configuration) contain non-converged runs.

W5: Limited evaluation of initialization sensitivity

We apologize that the original manuscript is ambiguous because of the notation and the wording. The parameters that affect the random perturbation of the global optimizers were already scanned; see Table 7. Random search and simulated annealing scan sigma, the absolute standard deviation of the proposal drawn around the current point at every iteration. Particle swarm and differential evolution scan init_sigma, the absolute standard deviation of the initial population drawn once around the starting point, over the same candidates. The σinit\sigma_{\mathrm{init}} in Equation 52 applies a small perturbation to the original values of the optimization variables, which can be too trivial (for example, all zeros), causing the optimization process to degrade, and also introduce randomness for all optimizers. We will clarify the meaning of each variable in the modified manuscript.

AuthorsOpenReview ↗

Official Comment to Reviewer 86sX (Part 3)

Q1: Higher numerical precision for Table 5 and 6

Thank you for the comments. This really is a critical point to improve the rigorousness of the experimental results and analysis.

Higher-precision Table 5 (inverse design):

OptimizerProjectile ↑\uparrowFluid ↓\downarrowAtmospheric ↑\uparrowElastic ↓\downarrow
SGD1.6974100×101±01.6974100 \times 10^{1} \pm 06.2431780×10−2±4.58×10−76.2431780 \times 10^{-2} \pm 4.58 \times 10^{-7}7.4780091×10−1±2.02×10−37.4780091 \times 10^{-1} \pm 2.02 \times 10^{-3}1.5359154×100±4.75×10−31.5359154 \times 10^{0} \pm 4.75 \times 10^{-3}
RMSprop1.6974100×101±01.6974100 \times 10^{1} \pm 04.2395562×10−2±1.67×10−74.2395562 \times 10^{-2} \pm 1.67 \times 10^{-7}7.4628963×10−1±9.64×10−57.4628963 \times 10^{-1} \pm 9.64 \times 10^{-5}1.5555815×100±5.53×10−31.5555815 \times 10^{0} \pm 5.53 \times 10^{-3}
Adam1.6974100×101±01.6974100 \times 10^{1} \pm 04.0463197×10−2±1.32×10−74.0463197 \times 10^{-2} \pm 1.32 \times 10^{-7}7.4810408×10−1±6.19×10−47.4810408 \times 10^{-1} \pm 6.19 \times 10^{-4}1.5034566×100±5.21×10−31.5034566 \times 10^{0} \pm 5.21 \times 10^{-3}
AdamW1.6974099×101±8.99×10−71.6974099 \times 10^{1} \pm 8.99 \times 10^{-7}4.1612168×10−2±2.28×10−74.1612168 \times 10^{-2} \pm 2.28 \times 10^{-7}7.4362258×10−1±1.87×10−37.4362258 \times 10^{-1} \pm 1.87 \times 10^{-3}1.4967313×100±6.03×10−3\mathbf{1.4967313 \times 10^{0} \pm 6.03 \times 10^{-3}}
LBFGS1.6974098×101±01.6974098 \times 10^{1} \pm 04.0026218×10−2±2.13×10−84.0026218 \times 10^{-2} \pm 2.13 \times 10^{-8}7.5221533×10−1±4.87×10−8\mathbf{7.5221533 \times 10^{-1} \pm 4.87 \times 10^{-8}}1.5427357×100±4.53×10−31.5427357 \times 10^{0} \pm 4.53 \times 10^{-3}
KFAC1.6974099×101±8.99×10−71.6974099 \times 10^{1} \pm 8.99 \times 10^{-7}4.0034256×10−2±9.78×10−94.0034256 \times 10^{-2} \pm 9.78 \times 10^{-9}7.5215860×10−1±2.08×10−57.5215860 \times 10^{-1} \pm 2.08 \times 10^{-5}1.5563584×100±1.24×10−21.5563584 \times 10^{0} \pm 1.24 \times 10^{-2}
Shampoo1.6974100×101±01.6974100 \times 10^{1} \pm 04.0026035×10−2±0\mathbf{4.0026035 \times 10^{-2} \pm 0}7.4993451×10−1±6.75×10−47.4993451 \times 10^{-1} \pm 6.75 \times 10^{-4}1.5536455×100±8.67×10−31.5536455 \times 10^{0} \pm 8.67 \times 10^{-3}
SOAP1.6974099×101±8.99×10−71.6974099 \times 10^{1} \pm 8.99 \times 10^{-7}4.0075691×10−2±7.31×10−64.0075691 \times 10^{-2} \pm 7.31 \times 10^{-6}7.5210740×10−1±4.75×10−57.5210740 \times 10^{-1} \pm 4.75 \times 10^{-5}1.9110996×100±3.48×10−21.9110996 \times 10^{0} \pm 3.48 \times 10^{-2}
RS1.6974100×101±1.56×10−61.6974100 \times 10^{1} \pm 1.56 \times 10^{-6}4.0031306×10−2±04.0031306 \times 10^{-2} \pm 07.2088854×10−1±4.00×10−57.2088854 \times 10^{-1} \pm 4.00 \times 10^{-5}1.5344769×100±1.61×10−21.5344769 \times 10^{0} \pm 1.61 \times 10^{-2}
SA1.6974100×101±01.6974100 \times 10^{1} \pm 04.0031306×10−2±04.0031306 \times 10^{-2} \pm 07.1578312×10−1±1.98×10−27.1578312 \times 10^{-1} \pm 1.98 \times 10^{-2}2.5621792×100±5.72×10−22.5621792 \times 10^{0} \pm 5.72 \times 10^{-2}
PS1.6974102×101±0\mathbf{1.6974102 \times 10^{1} \pm 0}4.0031306×10−2±04.0031306 \times 10^{-2} \pm 07.4895032×10−1±5.62×10−87.4895032 \times 10^{-1} \pm 5.62 \times 10^{-8}1.5349607×100±6.57×10−31.5349607 \times 10^{0} \pm 6.57 \times 10^{-3}
DE1.6974102×101±0\mathbf{1.6974102 \times 10^{1} \pm 0}4.0031305×10−2±1.76×10−94.0031305 \times 10^{-2} \pm 1.76 \times 10^{-9}7.4871786×10−1±1.27×10−47.4871786 \times 10^{-1} \pm 1.27 \times 10^{-4}2.5396021×100±2.04×10−22.5396021 \times 10^{0} \pm 2.04 \times 10^{-2}

(to be continued)

Comments on the paper

1 comment
AuthorsOpenReview ↗

We have updated the manuscript

Dear Reviewers,

Thank you for your constructive comments and helpful suggestions. We have modified our manuscript accordingly.

  • As proposed by Reviewer 86sX and A7PT, we plotted the Objective versus Wall-Clock Time curves and added them in Appendix C.3.
  • As proposed by Reviewer A7PT, we added analysis on modifying problem scales and added it in Appendix C.4.
  • As pointed out by Reviewer 86sX, we enhanced the description of experimental hyperparameters to better distinguish their meanings and differences.
  • As suggested by Reviewer ghjn and A7PT, we provide a flowchart on general optimizer selection guidelines in Appendix D.

Please let us know if you have any further comments or questions.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.