- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the optimization of input-convex neural networks (ICNNs) with nonnegative weight constraints. The authors argue that the commonly used direct softplus parameterization can suffer from “shoulder trapping,” where gradients become very small near the boundary. They propose a lifted, permutation-invariant batch-conditioned parameterization intended to improve the escape dynamics from this region. The paper provides theoretical analysis of this mechanism and evaluates it against direct softplus, projected gradient descent, and related approaches on controlled examples and several ICNN applications.
Critical strengths
- The paper provides a coherent mechanism-driven analysis rather than only reporting final performance. The experiments directly examine shoulder occupancy and escape behavior, which helps connect the empirical results to the proposed optimization explanation.
- The negative controls are useful. In settings where shoulder trapping does not occur, the proposed method provides little advantage, which is consistent with the mechanism claimed by the paper rather than suggesting a universal improvement.
Critical weaknesses
- The methodological novelty limited. The proposed lift appears essentially to be a permutation-invariant, batch-conditioned reparameterization of the nonnegative ICNN weights, while the motivating “shoulder trapping” phenomenon seems closely related to the well-known saturation behavior of softplus. Given the simplicity of the method, I am not convinced that the accompanying analysis establishes a sufficiently new optimization insight. 2.Empirical evidence does not establish broad practical significance. Much of the evaluation uses relatively classical or controlled ICNN settings, and the paper does not convincingly demonstrate that shoulder trapping is a common bottleneck in practically important ICNN applications. The comparison against alternative approaches to positive parameterization/constrained optimization also seems limited.
Questions for authors
- What is the key technical distinction between the proposed “shoulder trapping” phenomenon and the standard saturation/vanishing-gradient behavior of softplus, and which theoretical result would not follow from the latter observation alone?
- Since the constrained ICNN weights depend on the minibatch during lifted training, could the authors clarify the relationship between the lifted optimization problem and the original fixed-parameter ICNN training problem, particularly regarding the final solution being optimized?
- How frequently does shoulder trapping occur in realistic ICNN applications, and is there evidence that it remains an important optimization bottleneck beyond the controlled settings considered here?
- How does the proposed method compare with other reasonable modern approaches to enforcing positivity or optimizing nonnegative constraints beyond direct softplus and PGD?