- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to reduce the cost of computing hypergradients in bilevel optimization. The main idea is to reuse the adjoint from the previous outer step. The method first rescales the old adjoint with a cheap one-dimensional step.
Critical strengths
The idea is simple and easy to implement, with little extra memory cost.
Critical weaknesses
-
Reusing the previous adjoint as a warm start is already common in bilevel optimization. The main new part is a one-dimensional rescaling step, which seems to be a small change to existing methods. More importantly, the ablation results do not clearly support the main idea. Removing the rescaling step makes almost no difference. The version without adjoint recycling even performs better and faster on ridge regression. This makes me question whether recycling is really the reason for the improvement. The gains may come from early stopping or other implementation choices.
-
The claim of low cost is not fully convincing. The method needs very few steps mainly after it has already reached a good solution. During the early stage of optimization, the cost saving seems much smaller. The experiments are limited to small-scale linear problems with strongly convex lower objectives. There are no tests on neural networks or non-convex lower-level problems. The paper also lacks comparison with related Krylov recycling methods. With more time, other methods seem to catch up. The theory only provides a local error analysis based on standard results, without a convergence guarantee for the full bilevel algorithm. The paper also introduces too many variants and names, making the main contribution hard to follow.
Questions for authors
See wakenesses.