- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed a framework for learning optimization objective through bilevel optimization. Instead of using a fixed objective function, the method learns neural objective components and trains them by differentiating through the optimization process. The authors also use symbolic regression to convert the learned objectives into interpretable mathematical expressions.
Critical strengths
- The idea of learning the optimization objective instead of the optimizer is interesting.
- The symbolic distillation part is interesting, as it makes the learned objective more interpretable.
Critical weaknesses
- I don't quite get the motivation for using bilevel optimization here. Usually, we use bilevel/differentiable optimization when we know some structure of the problem. However, in this paper, the goal is to learn the optimization objective itself. I don't see clear evidence showing why this is better than directly training a neural network to predict the solution. Could the authors add experimental comparisons with direct prediction and learning-to-optimize (L2O) methods? Physical constraints or physics-based regularization could also be included in these baselines for a fair comparison.
- I also feel that the comparison with LISTA/ALISTA is not fully fair. LISTA/ALISTA mainly learns parameters of the optimization updates, such as weights, step sizes, and thresholds, while BOLD can learn a more flexible neural regularizer. Therefore, the improvement might come from the additional modeling capacity rather than learning the objective itself. Could the authors provide a more controlled comparison with similar model capacity and computational budgets?
- All experiments are conducted on synthetic datasets. It would be helpful to include some real-world experiments to show the practical value of the proposed method.
Questions for authors
See weaknesses.
2 comments
3. New Real-World Inverse-Problem Experiments
We evaluated BOLD on two real-world inverse problems and introduced direct-prediction MLP baselines with the same number of trainable neural parameters as BOLD. All neural experiments used three training seeds, the original data splits, and validation-selected checkpoints.
Raman spectral unmixing.
We recover the proportions of five mixture components from measured Raman spectra. BOLD learns unary spectral factors and adjacent-channel coupling factors, and FCLS (Fully Constrained Least Squares) serves as a classical baseline.
| Method | Test RMSE (pp) ↓ |
|---|---|
| FCLS | 4.929 |
| Direct-prediction MLP | 3.014 ± 0.339 |
| BOLD | 1.916 ± 0.054 |
| SR-BOLD | 2.064 ± 0.001 |
UWB localization.
We estimate 3D positions from measured time-difference-of-arrival (TDOA) observations and surveyed anchor geometry. BOLD learns an exchange-symmetric geometric objective, and LS (Least Squares) serves as a classical baseline by minimizing the squared TDOA residuals.
| Method | Test RMSE (m) ↓ |
|---|---|
| LS | 0.651 |
| Direct-prediction MLP | 0.955 ± 0.104 |
| BOLD | 0.440 ± 0.008 |
| Compact SR-BOLD | 0.452 ± 0.004 |
Under these experimental settings, BOLD outperforms the tested parameter-matched MLP baselines and classical objective-based methods. Meanwhile, the compact symbolic objectives retain most of the recovery performance of the learned objectives. Further details regarding the actual inverse problem experiments are presented in the revised paper.
Interpretability.
Across three training seeds, the distilled Raman objectives consistently contain squared spectral residuals and log-rational penalties on adjacent-channel differences. The UWB objectives similarly recover TDOA residual penalties and baseline-dependent geometric corrections. These compact expressions retain much of the neural objectives' recovery performance and provide interpretable empirical structures that can be independently optimized. More details are provided in the revised manuscript.
We thank the reviewer for the constructive feedback. To further address the concerns, we conducted additional experiments.
1. Motivation for Bilevel Objective Learning
We agree that bilevel optimization often exploits known problem structure. If we understand the reviewer's concern correctly, it may relate to the fact that, in many existing applications, the inner objective has a largely prescribed form. A typical bilevel formulation is
\qquad x_\theta^\star\in\arg\min_x E_\theta(x;y),$$ where the inner objective may take the form $$E_\theta(x;y)=\frac12\|Ax-y\|^2+\lambda_\theta R(x).$$ In such settings, learning may focus on regularization weights, prior parameters, or parameters within a predefined regularizer family. BOLD takes a step further by aiming to learn the functional form of the objective itself, rather than only tuning parameters within a fixed functional form. However, learning an unknown objective does not imply that the optimization problem is entirely unstructured. BOLD retains the optimization variables, decomposable factor structure, and inner solver, while learning the functional forms of the scalar objective components. Bilevel optimization enables this by differentiating the recovery loss through the inner solver to update the objective parameters. In this way, the objective is learned directly from its downstream recovery performance. Unlike direct prediction, which learns an input-to-solution mapping, BOLD aims to discover an explicit, decomposable objective that can be independently optimized, analyzed, and further distilled into an interpretable analytic expression. For a more comprehensive comparison, we introduce a direct-prediction multilayer perceptron (MLP) baseline model in the experiments on real-world inverse problems. We also include a learning-based graph recovery baseline model L2G (Pu et al., NeurIPS 2021) for the graph learning experiment. ### 2. Controlled Comparison with Similar Model Capacity We agree that the original ALISTA baseline, with only $2K$ learnable parameters, has substantially lower modeling capacity than LP-ALISTA. This difference is partly by design: while ALISTA primarily learns layerwise step sizes and thresholds, LP-ALISTA introduces the gradient of a learned scalar potential to provide more expressive, state-dependent corrections to the update direction. However, BOLD does not claim that scalar-objective parameterization inherently yields higher reconstruction accuracy than an equally expressive learned update rule. Rather, its central goal is to discover interpretable analytic optimization objectives that retain competitive recovery performance. We constructed a capacity-matched LC-ALISTA variant that directly learns a correction field instead of deriving it from a scalar potential. Both methods share the ALISTA backbone, analytic terms, parameter count, and training protocol. | K / SNR | LC-ALISTA NMSE (dB) ↓ | LP-ALISTA NMSE (dB) ↓ | |---|---:|---:| | 8 / 20 dB | −18.170 | −18.530 | | 8 / 40 dB | −25.397 | −25.301 | | 24 / 20 dB | −20.840 | −20.802 | | 24 / 40 dB | −34.384 | −35.067 | The results show comparable performance between the two approaches. Constraining the correction field to be the gradient of a scalar potential does not inherently guarantee better recovery performance. Nevertheless, compared with directly predicting a correction direction, the scalar-potential formulation provides a more interpretable mathematical structure with greater potential for scientific insight.