acceptodds
Under review as a conference paper at ICLR 2027

A lift for input-convex neural net training

Abstract

Input-convex neural nets parametrize the convex potentials of density models and transport maps, and their convexity requires the inter-layer weights to be non-negative. Projected gradient descent enforces this by projecting after each step, and due to mini-batch noise the boundary is re-crossed indefinitely, which leads to an active set the projection never identifies. The differentiable alternative, direct softplus, optimizes a free latent weight through a softplus positivity map whose derivative attenuates the gradient exponentially where the weight is negative—the shoulder—so a coordinate that reaches it stays for an exponentially long time. To keep this unconstrained parametrization without its slow escape, we propose the lift, which replaces the free latent weight by a learnable slack plus an unconstrained network—the body—that takes a permutation-invariant summary of the training batch as input. The latent weight thus varies with the batch before the positivity map, and couples to the gradient formed on it. We show that this coupling enters the variance of the update to the latent weight at first order in the fluctuation, and that the slack, the batch dependence and the shared batch are each needed for it to act. Where the coupling aligns positively with the loss curvature, that variance is larger under the lift than under direct softplus, and a coordinate leaves the shoulder sooner. We compare the lift with the two existing methods on several applications. Where a constrained weight of direct softplus reaches the shoulder and does not leave, the lift fits the target more closely and reaches the same reconstruction about three times sooner. Where almost none reaches it, the methods agree.

Primary area:
optimization
Keywords:
input-convex neural networks, positivity constraints, hypernetworks, stochastic gradient descent dynamics

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YkT5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the optimization of input-convex neural networks (ICNNs) with nonnegative weight constraints. The authors argue that the commonly used direct softplus parameterization can suffer from “shoulder trapping,” where gradients become very small near the boundary. They propose a lifted, permutation-invariant batch-conditioned parameterization intended to improve the escape dynamics from this region. The paper provides theoretical analysis of this mechanism and evaluates it against direct softplus, projected gradient descent, and related approaches on controlled examples and several ICNN applications.

Critical strengths
  1. The paper provides a coherent mechanism-driven analysis rather than only reporting final performance. The experiments directly examine shoulder occupancy and escape behavior, which helps connect the empirical results to the proposed optimization explanation.
  2. The negative controls are useful. In settings where shoulder trapping does not occur, the proposed method provides little advantage, which is consistent with the mechanism claimed by the paper rather than suggesting a universal improvement.
Critical weaknesses
  1. The methodological novelty limited. The proposed lift appears essentially to be a permutation-invariant, batch-conditioned reparameterization of the nonnegative ICNN weights, while the motivating “shoulder trapping” phenomenon seems closely related to the well-known saturation behavior of softplus. Given the simplicity of the method, I am not convinced that the accompanying analysis establishes a sufficiently new optimization insight. 2.Empirical evidence does not establish broad practical significance. Much of the evaluation uses relatively classical or controlled ICNN settings, and the paper does not convincingly demonstrate that shoulder trapping is a common bottleneck in practically important ICNN applications. The comparison against alternative approaches to positive parameterization/constrained optimization also seems limited.
Questions for authors
  1. What is the key technical distinction between the proposed “shoulder trapping” phenomenon and the standard saturation/vanishing-gradient behavior of softplus, and which theoretical result would not follow from the latter observation alone?
  2. Since the constrained ICNN weights depend on the minibatch during lifted training, could the authors clarify the relationship between the lifted optimization problem and the original fixed-parameter ICNN training problem, particularly regarding the final solution being optimized?
  3. How frequently does shoulder trapping occur in realistic ICNN applications, and is there evidence that it remains an important optimization bottleneck beyond the controlled settings considered here?
  4. How does the proposed method compare with other reasonable modern approaches to enforcing positivity or optimizing nonnegative constraints beyond direct softplus and PGD?
Reviewer_oUtAOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper aims to address the training stagnation and slow convergence caused by the vanishing gradients in the shoulder region of the softplus transformation, which is commonly used to enforce non-negative weights in Input Convex Neural Networks (ICNNs).

To overcome this issue, the authors propose Lift, a method that generates latent weights from mini-batches and exploits the coupling between mini batch dependent weight fluctuations and loss gradients. The experimental results demonstrate that Lift facilitates faster escape from the shoulder region while preserving the non-negativity constraints, leading to faster training and improved performance.

Critical strengths
  1. This paper has a clear research objective "ensuring the non-negativity of ICNN weights without compromising training speed or convergence". And the proposed method is simple yet highly effective, more accurate and more faster, in addressing this problem.

  2. Through extensive experiments across diverse applications of ICNNs, the paper demonstrates the effectiveness of the proposed method.

  3. The proposed method is also clearly written and easy to follow.

Critical weaknesses
  1. Some sentences written with em dashes (line13-17, line 17-20, line 260-261 and so on) in the paper lack readability and clarity.
Questions for authors
  1. Could the authors explain why the constrained weights in convex potential flows and transport maps rarely enter the shoulder region, unlike those in the other experimental settings?

  2. Regardless of held-out performance and convergence speed, could the proportion of learned constrained weights falling within the shoulder region differ between direct softplus and Lift depending on the threshold used to define the shoulder region?

  3. Could the performance and convergence speed of Lift differ depending on whether softplus or exponential parameterization is used? If so, I would be interested in understanding what differences might arise and how they would affect the training dynamics.

  4. Is there a particular reason for the frequent use of em dashes throughout the paper? I find that their excessive use negatively affects the readability of some sentences.

Reviewer_tAK6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses an important issue faced while training an input convex neural network. The ICNN requires weights to be positive to ensure that the output is convex with respect to the input. This is done by either using PGD or softplus. Both approaches have their own challenges. The proposed fix, the “lift”, replaces the free latent weight with θ̃ = b + hφ(X). Here b is a learnable slack and hφ is a permutation-invariant hypernetwork that reads the current training minibatch. The latent weight then changes from batch to batch, and that change is correlated with the gradient computed on the same batch. The hypernetwork is used only in training, so the deployed ICNN has the same architecture as under direct softplus. Experiments cover 1-D log-concave EBMs, the POWER, HEPMASS and MiniBooNE tabular benchmarks, CP-Flow 2-D toy problems, a 100-D Darcy posterior transport map, and an ICNN regularizer for FFHQ inpainting. The lift wins clearly where direct softplus gets stuck on the shoulder (the EBMs and tabular data), reaches the same PSNR about 3× sooner on the image regularizer, and ties where almost no weight reaches the shoulder (CP-Flow and Darcy).

Critical strengths
  1. The problem is real and under-discussed. Softplus saturation in ICNN training is something practitioners run into. “Shoulder occupancy” is a useful diagnostic, and I’d expect others to adopt it.

  2. The method is simple and costs nothing at inference: the hypernetwork is dropped after training.

  3. On Darcy and CP-Flow almost no weight reaches the shoulder, and the methods tie, which is what the theory predicts.

  4. The authors say the formal results are machine-checked in Lean and state clearly which steps are modelling assumptions and which are proved.

Critical weaknesses
  1. To be honest, the paper is very hard to understand. It takes many rounds of reading to understand the paper. I strongly recommend that the authors work on the readability of the paper. Explain the main idea of the paper and its reasoning in a better way.

  2. Authors have missed an important related work: The Curious Case of Convex Networks, Sarath Sivaprasad, Ankur Singh, Naresh Manwani, Vineet Gandhi, ECML-PKDD 2021. Authors should compare their experimental results with this paper also.

  3. The proposed approach also adds a large overparametrization (b plus a hypernetwork) and changes the optimization geometry. Under Adam, that alone can change effective step sizes on the shoulder.

  4. The theory analyses SGD, but the experiments use Adam. Assumption A1 models SGD as an Itô process with step η\eta. Every experiment uses Adam (App. B), and Adam’s per-coordinate second-moment normalization largely undoes the ψ′\psi' attenuation that defines the shoulder. That changes both the drift and the diffusion along ebe_b, so the Kramers comparison doesn’t obviously carry over. Either analyse a preconditioned SDE or run the key experiments with SGD.

  5. The paper’s own dwell analysis cuts against the narrative. Section 4.1 says the lift “ends with more weights on the shoulder than direct softplus does”, and that on Beta direct softplus leaves the shoulder as often as the lift does. If the lift’s advantage is faster escape, why does it end with higher occupancy? The paper argues dwell matters rather than occupancy. Then the link to final likelihood needs to be shown more directly.

Questions for authors

Please see the weaknesses and address those concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.