- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a benchmark comprising 12 differentiable physics models and corresponding inverse problems covering parameter identification, inverse design, and optimal control. It compares 12 optimizers from first-order, approximate second-order, and global-search families, reporting objective values, runtime, and memory usage.
The unified implementation is potentially useful, but the current benchmark resembles a collection of small examples more than a validated tool for practical optimizer selection. Several tasks have limited discriminative power, and the evaluation does not sufficiently establish robustness or transfer of the findings to larger problems. A benchmark need not introduce a new algorithm, but it should provide reliable comparisons and meaningful new insights. I do not think the current submission demonstrates these sufficiently.
Critical strengths
-
The unified PyTorch implementation spans several physical domains and could reduce the effort required to test optimizers across different inverse problems.
-
Including runtime and memory alongside objective values is useful, since optimizer iterations can have substantially different computational costs.
Critical weaknesses
-
The benchmark’s practical significance is insufficiently established. Several tasks provide little separation between optimizers, including projectile motion, flight control, and rigid-body control. The problems are deliberately small, with at most 1,100 optimization variables, and their rankings are not validated across increasing problem scales or difficulty levels. Consequently, it is unclear whether the results offer reliable guidance for realistic applications. The conclusions largely reiterate familiar tradeoffs rather than establishing actionable relationships between problem characteristics and optimizer performance.
-
The evaluation protocol limits conclusions about reliability and efficiency. Non-converged runs are excluded before hyperparameter selection and aggregation. Although invalid numerical results should not be treated as successful optimization, excluding failures without incorporating failure rates can obscure robustness. Moreover, objective values and per-step costs alone do not establish performance under comparable total computational budgets, particularly when population methods include additional local searches.
Questions for authors
-
What actionable optimizer-selection guidance follows from this benchmark beyond the observation that different tasks favor different methods?
-
Can the optimizer rankings be shown to persist as resolution, parameter count, or rollout length increases?
-
How are non-converged runs defined, and what are the failure rates for each method? Would including failures change the conclusions?
-
How do the methods compare under matched wall-clock budgets, including all local searches and objective evaluations?
5 comments
Official Comment to Reviewer A7PT (Part 2)
W1 (c) Establishing actionable relationships between problems and optimizers & Q1 Actionable optimizer-selection guidance
Thank you for proposing this suggestion. We structured the optimizer selection guidelines in the flowchart below. Note that this is for reference only, as realistic problems can involve more complicated factors and concerns.
Start: new differentiable inverse problem
|
v
Long rollout?
|
+-- yes --> approximate second-order
|
+-- no
|
v
Ill-conditioned?
|
+-- yes --> approximate second-order
|
+-- no
|
v
High-dimensional?
|
+-- yes --> first-order
|
+-- no, low-dimensional and nonconvex,
| initialization uncertain
| --> global
|
+-- no, low-dimensional, well-conditioned,
not strongly nonconvex
--> first-order
On the other hand, we would also like to emphasize that we think a novel optimizer designed specifically for differentiable physics simulators is called for, which is our original motivation for building this benchmark.
W2 (a) Non-converged runs are excluded & Q3 Definition of non-converged runs and failure rates
To guarantee fair comparison, we use a fixed hyperparameter scan for all physics simulators and inverse problems. This leads to the result that very unreasonable hyperparameters can be applied to a specific problem, causing convergence failures. But such failure has little correlation with the performance or stability of a specific optimizer on a specific problem. That is the reason why we only report the best hyperparameter configuration with mean±std over 3 random initializations.
In the current procedure, non-converged runs are first identified by humans to observe unreasonable outcome results. More specifically speaking, in the current benchmark and experimental settings, only the topology optimization problem based on the elastic mechanics simulator produces non-converged runs, in which case non-physical negative objectives are generated. Hence in the experiment, we define a run as non-converged if any of the following holds: the optimizer step raises an exception; the best objective is non-finite (NaN or Inf); or, specifically for topology optimization, the compliance becomes a negative value, which is a failed equilibrium solve rather than a better design.
For the topology optimization problem, 20 of 621 runs (3.2%) have a negative best loss: 18/108 differential-evolution runs (16.7%) and 2/36 K-FAC runs (5.6%). No other optimizers on this task have non-converged runs. The other 11 inverse problem tasks have no failures under the same rule. For all problems and optimizers, none of the reported results (3 random initializations of the best hyperparameter configuration) contain non-converged runs.
Official Comment to Reviewer A7PT (Part 3)
W2 (b) Per-step costs are not sufficient & Q4 Comparison under matched wall-clock budgets
Thank you for proposing this suggestion. We reran all 12 tasks, 3 seeds, and 100 epochs, and recorded a timestamp after every epoch. The curves are plotted and added in the revised manuscript as Appendix C.3 Objective versus Wall-Clock Time. For each task, the matched budget is the mean wall-clock time at which the fastest first-order or approximate second-order method finishes 100 epochs. A run that has already finished keeps its final best value.
Where the method with the best objective is itself a fast local method, the matched budget selects the same method. Global methods are behind, because the budget buys them on the order of ten steps against a hundred for the local method. L-BFGS still attains the best objective on tight binding and the battery fit. Shampoo still attains the best objective on the fluid design and the mass–spring reach. Adam still attains the best objective on topology, with AdamW unresolved from it. L-BFGS still attains the best objective on atmospheric scattering and flight, ahead of a narrow band of other local methods. On the fluid design, the global methods do reach Shampoo’s plateau, but only after a run many times longer than this budget. On mass–spring, they stay behind even after a full run.
The best method changes when one local method is much more expensive per step. On tire fitting and Gaussian splatting, the matched budget is a fraction of a second and a first-order method attains the best objective, while L-BFGS is still inside its line search. L-BFGS is the best on those two tasks only when it is allowed to finish that longer run. DFT runs the other way: L-BFGS is the first method to finish, and Adam, SGD, and particle swarm are already at numerical zero. Simulated annealing and differential evolution are still far above that floor and reach it only later.
Projectile and the rigid-body setpoint do not separate under either clock. On the projectile problem, all twelve methods sit on the same float32 value within a tenth of a second. On the rigid-body problem, the twelve methods form a narrow band, and the global methods are only a few steps into their search when the local methods have already finished.
Official Comment to Reviewer A7PT (Part 1)
Thank you for your precise assessment and constructive comments. We appreciate your recognition that such a unified benchmark is useful for testing optimizers across different inverse problems. Below, we address the proposed weaknesses and questions one by one.
W1 (a) Several tasks provide little separation
On one hand, we agree with the reviewer on this point to a large extent. We had thought about it and realized this is rooted in a very tricky dilemma, that the famous problems in differentiable programming are the ones that have already been verified solvable by current optimizers, while for some other problems, none of the existing optimization methods can obtain outstanding results, which is the major motivation why we built this benchmark. In the future, we plan to introduce more problems where most or all existing optimizers fail.
On the other hand, we would like to highlight that, with the current set of simulators and problems, different types of optimizers already show performance differences. For the problems where optimizers show similar performances, we gain the knowledge that such tasks are not bottlenecked by the optimizers.
W1 (b) No validation across increasing problem scales & Q2 Optimizer performances across different resolutions, parameter counts, or rollout lengths
Thank you for this suggestion. In principle, the problem remains the same mathematical structure for different sizes, hence the suitable optimizer types should hold. However, this conjecture needs to be examined. We reran the topology optimization problem on different mesh resolutions, and the mass-spring problems on different rollout lengths, with 3 fresh seeds and 100 epochs.
Resolution of the Topology Optimization problem.
From the table, we can see that Adam and AdamW remain the best on every mesh resolution. Global optimizers simulated annealing and differential evolution are significantly worse, even with higher computational cost. The performance of Shampoo and K-FAC degrades significantly as the mesh size grows.
Rollout length of the mass–spring problem.
From the results, we can see that Shampoo basically keeps its leading position. However, for the problem with shorter rollout length, particle swarm and random search show advantages, indicating that for shorter rollout length, the long-rollout problem does not dominate the mathematical characteristics of the problem.