- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a neural network architecture called Feature-wise Lipschitz Neural Networks (FLNNs) which satisfies a Lipschitz constraint only with respect to a predetermined subset of the input variables. The architecture combines a established GroupSort Layer for the constrained variables with a normalized multiplicative interaction with the unconstrained ones. It is proved that the proposed architecture indeed satisfies the desired Lipschitz condition and furthermore allows for a universal approximation result in the class of functions with the same Lipschitz condition. The paper concludes with experiments on synthetic and real-world data sets which illustrate that the proposed architecture allows for small errors while satisfying the partial Lipschitz constraint.
Critical strengths
- The paper is generally well-written and proposes a, in my opinion, novel and interesting neural network architecture.
- The theoretical results are mostly rigorous (I didn't check all proofs).
Critical weaknesses
- The motivation for feature-wise Lipschitz architectures is unclear and no immediate application is mentioned. Also in the experimental section it is just referred to "domain knowledge" for choosing the controlled features.
- The proof of Theorem 2 (universal approximation) is hard to parse and seems to wordy and not sufficiently rigorous.
- The experiments are limited to scalar regression problems on relatively unknown datasets. Having at least one classification example, where Lipschitzness could benefit, e.g., robustness, would be nice.
- The appendix seems to be LLM-generated to a large extent. While I appreciate explanations of mathematical details, the proofs are very wordy and still sometimes hard to process.
Questions for authors
- Section 4.4 is unclear. I understand this is a relaxation of the Lipschitz assumption but why is that necessary? If is a bounded function, it seems like you're getting the same class of functions (with replaced by ).
- In the paragraph Layer-wise interactions could you alreay give an heuristic explanation for why the interaction (16) does not mess up the Lipschitz constraint? Are other interactions with the same property thinkable?
- How can one adapt the proposed architecture to neural networks with vector-valued outputs?
- Can you quantify Theorem 2 in terms of the required depth and width? At least Lemma 2 seems quantifiable and one would have to check Lemma 3, as well.
- How did you choose the controlled features in section 5? And how would one do that without domain knowledge or for other problems, like for example image classification?
6 comments
Response to Reviewer S9Zq — Weakness #2, #4 / Question #4
Weakness 2, 4 / Question 4
Thank you for these comments. We have clarified the three steps of Theorem 2 in the revised manuscript. Lemma 2 uses the Lipschitz bound to construct upper bounds that touch the target at anchors and rise with distance, forming V shapes in one dimension. Their minimum approaches the target as anchors become denser. Lemma 3 approximates the components, and Lemma 4 combines them using GroupSort without amplifying errors.
For anchors, two sign choices per controlled coordinate give components. Let denote the width of the single hidden layer in the shared ReLU coefficient approximator. This width depends on the functions and required accuracy. The condition in (78) gives sufficient width per branch and depth
k\le5+2\bigl(d_S+\lceil\log_2M\rceil\bigr).$$ The depth counts four preparation layers, two GroupSort layers per comparison round, and one output layer. The maxima take $d_S$ rounds in parallel across anchors, followed by $\lceil\log_2M\rceil$ rounds for the minimum. (Appendices B.3.3–B.3.4). --- For example, consider $f^*(x,u)=|x-u|$ on $[0,1]^2$, with $x$ controlled. Midpoint anchors $a_j=(j-\tfrac12)/M$ give $$f_M(x,u)=\min_{j\in[M]}\\{|a_j-u|+|x-a_j|\\},\qquad \\|f_M-f^*\\|_\infty=\frac1M.$$ Every $x$ is within $1/(2M)$ of its nearest anchor, and the Lipschitz argument bounds the error by twice this distance. This bound is attained at $x=u=0$. The error is bounded by twice the maximum anchor distance $1/(2M)$, with equality at $x=u=0$. The identity $|a_j-u|=2\operatorname{ReLU}(u-a_j)-\operatorname{ReLU}(u)+a_j$ represents the coefficient functions exactly using $M$ individual units and one shared unit, giving $H=M+1$. Substitution into the bounds above gives even width $m=6M+2$ and depth $k=7+2\lceil\log_2M\rceil$. Choosing $C=2$ and $B=32M$ in Lemma 3 adds at most $1/(4M)$ product error, so $$\\|\hat f_\theta-f^*\\|_\infty\le\frac1M+\frac1{4M}=\frac5{4M}.$$ Therefore, width $O(\epsilon^{-1})$ and depth $O(\log(1/\epsilon))$ suffice for error below $\epsilon$ in this example.Response to Reviewer S9Zq — Weakness #3 / Question #5
Weakness 3 / Question 5
We have added binary classification experiments on Wisconsin Diagnostic Breast Cancer (WDBC), Heart Disease, Loan Defaulter, and COMPAS datasets to the revised manuscript. Appendix F reports the experimental settings and clean test accuracy and AUROC results.
For the real datasets in Section 5, controlled features were selected using prior domain studies. For classification, we use a data-driven heuristic to select discriminative features while limiting redundancy. Following an existing method for Lipschitz bound estimation [1], we estimate feature-wise sensitivity bounds from a reference MLP and use them to constrain the sensitivity of the FLNN output logit.
These bounds are empirical modeling choices derived from data, rather than established domain requirements. The results demonstrate applicability to tabular classification with competitive predictive performance. More generally, this framework may be useful when prior knowledge specifies different sensitivity requirements across features, allowing tighter bounds for inputs requiring greater prediction stability while leaving other inputs uncontrolled.
For image data, segmentation or object detection could identify regions for partial sensitivity control. With a fixed region assignment, selected pixels or region features could be treated as controlled inputs, allowing bounds on changes in model outputs under perturbations to those inputs. The choice of controlled regions would reflect the desired stability, such as sensitivity to noise within an object or to changes in the background. We consider this a potential extension that requires further implementation and evaluation.
[1] Shi, Zhouxing, et al. "Efficiently computing local lipschitz constants of neural networks via bound propagation." Advances in Neural Information Processing Systems 35 (2022): 2350-2364.
Response to Reviewer S9Zq — Weakness #1
Weakness 1
Thank you for this important comment. FLNN targets regression problems with different sensitivity bounds on selected inputs.
Motivation and application. A relevant application is learning distance functions for robotic collision avoidance. Robot geometry provides joint-specific displacement bounds used in motion planning [1, 2]. Let Rj bound the displacement per unit change in joint coordinate qj for all points on a link and all configurations considered. For a fixed obstacle set, this gives ∣d(q+δej)−d(q)∣≤Rj∣δ∣, where q denotes the joint configuration and d is the minimum distance between the link and the obstacles. The bounds can differ because each joint has a different geometric effect on the link.
Neural distance models are already used for robotic collision avoidance [3]. FLNN could incorporate the geometric bounds into such models, allowing them to learn the distance function while preserving the prescribed sensitivity limit for each joint coordinate.
Controlled features and bounds in the real-data experiments. Since explicit sensitivity bounds were unavailable for the real-world benchmarks, we selected controlled features based on input-output relationships documented in the dataset descriptions and associated publications. We then used surrogate regressors to derive numerical bounds as empirical proxies for quantitative domain knowledge (Appendix C.3). We examined their absolute partial derivatives to identify regions with different sensitivity scales and set each regional bound to the corresponding empirical 95th percentile.
When physical or geometric sensitivity bounds are available, they can instead be supplied directly to FLNN without surrogate-based estimation.
[1] Hauser, Kris. "Fast interpolation and time-optimization with contact." The International Journal of Robotics Research 33.9 (2014): 1231-1250.
[2] Zhang, Duo, et al. "Provably feasible semi-infinite program under collision constraints via subdivision." IEEE Transactions on Robotics 40 (2024): 2602-2619.
[3] Koptev, Mikhail, Nadia Figueroa, and Aude Billard. "Neural joint space implicit signed distance functions for reactive robot manipulator control." IEEE Robotics and Automation Letters 8.2 (2022): 480-487.