- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies an important problem in bilevel optimization and proposes an interesting algorithmic framework based on second-order stationarity. The technical development is non-trivial, and the convergence analysis appears carefully constructed. However, I have concerns about the paper's main contribution and positioning. The claim that the work provides a fundamentally new solution to general nonconvex lower-level bilevel optimization is not sufficiently supported given the recent progress in LLNC-BLO. The proposed SOSP surrogate is a meaningful technical direction, but its novelty relative to existing formulations and its practical significance are not convincingly demonstrated.
Critical strengths
The observation that first-order stationarity is insufficient to distinguish local minima from saddle points in nonconvex lower-level problems is important. The proposed SOSP-based surrogate provides a principled way to incorporate curvature information into the lower-level solution concept.
Critical weaknesses
-
The novelty relative to recent LLNC-BLO literature is unclear. The main claimed contribution is the use of second-order stationarity as a lower-level surrogate for general nonconvex bilevel optimization. However, recent works have already moved beyond traditional convex/FOSP-based formulations by considering meaningful lower-level solution concepts, local solution behavior, and nonconvex lower-level structures. For example, recent approaches based on Moreau-envelope reformulations and correspondence-driven formulations have already addressed important limitations of FOSP-based LLNC-BLO methods. In particular, correspondence-driven approaches consider algorithmically generated lower-level solutions and use second-order optimization techniques to handle nonconvex lower-level behavior. The paper does not provide a sufficiently detailed comparison explaining why the proposed SOSP surrogate is fundamentally different from these recent directions. The distinction appears to be mainly in the specific formulation and algorithmic implementation rather than a clearly new problem perspective. This weakens the central novelty claim of the paper.
-
The theoretical guarantee relies on assumptions whose practical relevance is unclear. The proposed approach relies on second-order stationarity to avoid saddle points. However, the implication from SOSP to local minimum requires additional structural assumptions, such as strict saddle properties. While these assumptions are standard in some nonconvex optimization settings, their validity in modern deep learning models is not established. Therefore, the paper's motivation that SOSP resolves the lower-level optimization difficulty may be stronger than what is actually supported by the theory. A clearer discussion of the gap between the theoretical surrogate and practical deep learning landscapes would be needed.
-
The experiments do not sufficiently validate the claimed mechanism. The experiments mainly compare the final upper-level objective performance against existing LLNC-BLO algorithms. While the results show that PROBE performs competitively, they do not directly evaluate whether the algorithm indeed escapes lower-level saddle points more effectively.
Questions for authors
Recent LLNC-BLO works have proposed alternatives to FOSP-based formulations, including Moreau-envelope based approaches and correspondence-driven formulations. Could the authors provide a more detailed comparison explaining the fundamental differences between these methods and the proposed SOSP-based surrogate?
Is the proposed SOSP formulation mainly motivated as a computational surrogate, or does it aim to approximate the true lower-level local/global optimum? Under what practical conditions can the authors justify that an SOSP solution is sufficient for the original bilevel problem?
The theoretical results rely on strict saddle assumptions to connect SOSP with local minima. Do the authors have empirical evidence that the lower-level problems arising in their applications approximately satisfy this property?
Can the authors provide additional experiments directly measuring whether PROBE escapes saddle points more effectively than existing methods?