acceptodds
Under review as a conference paper at ICLR 2027

Feature-wise Lipschitz Neural Network

Abstract

Existing Lipschitz-constrained neural networks typically impose a single global bound on input sensitivity. When sensitivity requirements differ across features and apply only to a selected subset, a global bound can unnecessarily constrain the remaining features and limit how accurately the target function can be approximated. We therefore propose Feature-wise Lipschitz Neural Networks (FLNN), an architecture that imposes distinct sensitivity bounds on selected features while leaving the remaining features unconstrained. The architecture uses controlled and uncontrolled branches that interact across layers while preserving the prescribed sensitivity bounds throughout training. We establish feature-wise Lipschitz guarantees and prove universal approximation of continuous functions satisfying the same constraints on compact domains. A coordinate-wise input transformation further extends the sensitivity guarantees to bounds that vary across regions within the domain of each controlled feature. Experiments on synthetic and real-world datasets demonstrate constraint satisfaction and assess predictive performance under the prescribed sensitivity bounds.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Lipschitz neural networks, Feature-wise control, Universal approximation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_S9ZqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a neural network architecture called Feature-wise Lipschitz Neural Networks (FLNNs) which satisfies a Lipschitz constraint only with respect to a predetermined subset of the input variables. The architecture combines a established GroupSort Layer for the constrained variables with a normalized multiplicative interaction with the unconstrained ones. It is proved that the proposed architecture indeed satisfies the desired Lipschitz condition and furthermore allows for a universal approximation result in the class of functions with the same Lipschitz condition. The paper concludes with experiments on synthetic and real-world data sets which illustrate that the proposed architecture allows for small errors while satisfying the partial Lipschitz constraint.

Critical strengths
  • The paper is generally well-written and proposes a, in my opinion, novel and interesting neural network architecture.
  • The theoretical results are mostly rigorous (I didn't check all proofs).
Critical weaknesses
  • The motivation for feature-wise Lipschitz architectures is unclear and no immediate application is mentioned. Also in the experimental section it is just referred to "domain knowledge" for choosing the controlled features.
  • The proof of Theorem 2 (universal approximation) is hard to parse and seems to wordy and not sufficiently rigorous.
  • The experiments are limited to scalar regression problems on relatively unknown datasets. Having at least one classification example, where Lipschitzness could benefit, e.g., robustness, would be nice.
  • The appendix seems to be LLM-generated to a large extent. While I appreciate explanations of mathematical details, the proofs are very wordy and still sometimes hard to process.
Questions for authors
  • Section 4.4 is unclear. I understand this is a relaxation of the Lipschitz assumption but why is that necessary? If LiL_i is a bounded function, it seems like you're getting the same class of functions (with LiL_i replaced by max⁡x∥Li(x)∥\max_x\|L_i(x)\|).
  • In the paragraph Layer-wise interactions could you alreay give an heuristic explanation for why the interaction (16) does not mess up the Lipschitz constraint? Are other interactions with the same property thinkable?
  • How can one adapt the proposed architecture to neural networks with vector-valued outputs?
  • Can you quantify Theorem 2 in terms of the required depth and width? At least Lemma 2 seems quantifiable and one would have to check Lemma 3, as well.
  • How did you choose the controlled features in section 5? And how would one do that without domain knowledge or for other problems, like for example image classification?
6 comments
AuthorsOpenReview ↗

Response to Reviewer S9Zq — Weakness #1

Weakness 1

Thank you for this important comment. FLNN targets regression problems with different sensitivity bounds on selected inputs.

Motivation and application. A relevant application is learning distance functions for robotic collision avoidance. Robot geometry provides joint-specific displacement bounds used in motion planning [1, 2]. Let RjR_j bound the displacement per unit change in joint coordinate qjq_j for all points on a link and all configurations considered. For a fixed obstacle set, this gives ∣d(q+δej)−d(q)∣≤Rj∣δ∣,\lvert d(\mathbf q+\delta e_j)-d(\mathbf q)\rvert\leq R_j\lvert\delta\rvert, where q\mathbf q denotes the joint configuration and dd is the minimum distance between the link and the obstacles. The bounds can differ because each joint has a different geometric effect on the link.

Neural distance models are already used for robotic collision avoidance [3]. FLNN could incorporate the geometric bounds into such models, allowing them to learn the distance function while preserving the prescribed sensitivity limit for each joint coordinate.

Controlled features and bounds in the real-data experiments. Since explicit sensitivity bounds were unavailable for the real-world benchmarks, we selected controlled features based on input-output relationships documented in the dataset descriptions and associated publications. We then used surrogate regressors to derive numerical bounds as empirical proxies for quantitative domain knowledge (Appendix C.3). We examined their absolute partial derivatives to identify regions with different sensitivity scales and set each regional bound to the corresponding empirical 95th percentile.

When physical or geometric sensitivity bounds are available, they can instead be supplied directly to FLNN without surrogate-based estimation.

[1] Hauser, Kris. "Fast interpolation and time-optimization with contact." The International Journal of Robotics Research 33.9 (2014): 1231-1250.

[2] Zhang, Duo, et al. "Provably feasible semi-infinite program under collision constraints via subdivision." IEEE Transactions on Robotics 40 (2024): 2602-2619.

[3] Koptev, Mikhail, Nadia Figueroa, and Aude Billard. "Neural joint space implicit signed distance functions for reactive robot manipulator control." IEEE Robotics and Automation Letters 8.2 (2022): 480-487.

AuthorsOpenReview ↗

Response to Reviewer S9Zq — Weakness #2, #4 / Question #4

Weakness 2, 4 / Question 4

Thank you for these comments. We have clarified the three steps of Theorem 2 in the revised manuscript. Lemma 2 uses the Lipschitz bound to construct upper bounds that touch the target at anchors and rise with distance, forming V shapes in one dimension. Their minimum approaches the target as anchors become denser. Lemma 3 approximates the components, and Lemma 4 combines them using GroupSort without amplifying errors.

For MM anchors, two sign choices per controlled coordinate give N=2dSMN=2^{d_S}M components. Let HH denote the width of the single hidden layer in the shared ReLU coefficient approximator. This width depends on the functions and required accuracy. The condition m≥2N+max⁡H,N+1m\ge2N+\max\\{H,N\\}+1 in (78) gives sufficient width per branch and depth

k\le5+2\bigl(d_S+\lceil\log_2M\rceil\bigr).$$ The depth counts four preparation layers, two GroupSort layers per comparison round, and one output layer. The maxima take $d_S$ rounds in parallel across anchors, followed by $\lceil\log_2M\rceil$ rounds for the minimum. (Appendices B.3.3–B.3.4). --- For example, consider $f^*(x,u)=|x-u|$ on $[0,1]^2$, with $x$ controlled. Midpoint anchors $a_j=(j-\tfrac12)/M$ give $$f_M(x,u)=\min_{j\in[M]}\\{|a_j-u|+|x-a_j|\\},\qquad \\|f_M-f^*\\|_\infty=\frac1M.$$ Every $x$ is within $1/(2M)$ of its nearest anchor, and the Lipschitz argument bounds the error by twice this distance. This bound is attained at $x=u=0$. The error is bounded by twice the maximum anchor distance $1/(2M)$, with equality at $x=u=0$. The identity $|a_j-u|=2\operatorname{ReLU}(u-a_j)-\operatorname{ReLU}(u)+a_j$ represents the coefficient functions exactly using $M$ individual units and one shared unit, giving $H=M+1$. Substitution into the bounds above gives even width $m=6M+2$ and depth $k=7+2\lceil\log_2M\rceil$. Choosing $C=2$ and $B=32M$ in Lemma 3 adds at most $1/(4M)$ product error, so $$\\|\hat f_\theta-f^*\\|_\infty\le\frac1M+\frac1{4M}=\frac5{4M}.$$ Therefore, width $O(\epsilon^{-1})$ and depth $O(\log(1/\epsilon))$ suffice for error below $\epsilon$ in this example.
AuthorsOpenReview ↗

Response to Reviewer S9Zq — Weakness #3 / Question #5

Weakness 3 / Question 5

We have added binary classification experiments on Wisconsin Diagnostic Breast Cancer (WDBC), Heart Disease, Loan Defaulter, and COMPAS datasets to the revised manuscript. Appendix F reports the experimental settings and clean test accuracy and AUROC results.

For the real datasets in Section 5, controlled features were selected using prior domain studies. For classification, we use a data-driven heuristic to select discriminative features while limiting redundancy. Following an existing method for Lipschitz bound estimation [1], we estimate feature-wise sensitivity bounds from a reference MLP and use them to constrain the sensitivity of the FLNN output logit.

These bounds are empirical modeling choices derived from data, rather than established domain requirements. The results demonstrate applicability to tabular classification with competitive predictive performance. More generally, this framework may be useful when prior knowledge specifies different sensitivity requirements across features, allowing tighter bounds for inputs requiring greater prediction stability while leaving other inputs uncontrolled.

For image data, segmentation or object detection could identify regions for partial sensitivity control. With a fixed region assignment, selected pixels or region features could be treated as controlled inputs, allowing bounds on changes in model outputs under perturbations to those inputs. The choice of controlled regions would reflect the desired stability, such as sensitivity to noise within an object or to changes in the background. We consider this a potential extension that requires further implementation and evaluation.

[1] Shi, Zhouxing, et al. "Efficiently computing local lipschitz constants of neural networks via bound propagation." Advances in Neural Information Processing Systems 35 (2022): 2350-2364.

Reviewer_KqbLOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is based on a simple and plausible observation that a global L-Lipschitz bound might be too restrictive, and that finer, per-feature constraints (i.e., LiL_i) offer flexibility that may lead to enhanced performance without violating the Lipschitz constraints. The authors validate their approach on four synthetic datasets and four real-world datasets.

My assessment can be summarized as follows: the paper proposes a feature-wise Lipschitz architecture and supports it with a universal approximation under feature-wise bounds. However, the paper’s contributions are not clearly stated, and the experiments leave practical questions open, e.g.: how the approach scales as the number of controlled features dSd_S grows (each feature needs its own bound), and how robust it is to errors in the bounds, which for the real-world datasets are estimated with a surrogate model.

Critical strengths

Strength 1:

The authors not only provide a feature-wise Lipschitz guarantee in Theorem 1, but also prove universal approximation in Theorem 2 to support the expressivity of their method.

Strength 2:

The region-wise extension (Section 4.4) is a nice addition to the framework, and appears to be novel.

Strength 3:

Figure 1 gives a clear illustration of the motivation, and the preliminaries (Section 3) give a clear and concise description of the setup.

Critical weaknesses

Weakness 1:

The paper's contributions are not stated in the paper, and are not clear from its context. Two examples: a. The abstract presents the FLNN architecture as a main contribution, but the design largely combines existing components: (1) the two-branch layout resembles the PICNN of Amos et al. (2017), and (2) without the element-wise products with the uncontrolled branch, the controlled branch is the ℓ1\ell_1 case of the norm-constrained GroupSort networks in Anil et al. (2019). b. Feature-wise control has already been discussed in the literature, which Table 1 reflects (albeit vaguely). Kitouni et al. (2023) [1] noted that per-feature bounds can be imposed by rescaling the inputs, and their subsequent work Nolte et al. (2023) re-iterated this.

[1] Ouail Kitouni, Niklas Nolte, and Mike Williams. Robust and provably monotonic networks. Machine Learning: Science and Technology, 2023.

Weakness 2:

The paper focuses on the potential upside of feature-wise bounds, which - as discussed in Weakness 1.b - is not a new idea. The practical question is how to obtain such bounds, which the paper addresses only through a surrogate heuristic. The cost of specifying LiL_i should at least be acknowledged as a limitation. Furthermore, all datasets have at most three controlled features (dS≤3d_S \leq 3), so the experiments do not show how the difficulty grows with dSd_S.

Weakness 3:

Since the bounds can have estimation errors, I expect the paper to comment on robustness to such errors.

Smaller issues:

  1. The paper excessively uses numbered equations (101 in total), which clutters the important ones.
  2. I found the motivation and examples for the region-wise extension (lines 349-354) to be confusing.
Questions for authors
  1. Which contributions do the authors consider theirs: the implementation of feature-wise bounds, the architecture, or the universal approximation result? The answer from the authors’ perspective is needed to accurately assess their contribution.
  2. In Appendix C.3, the surrogate that defines the bounds is fit on the full dataset, does this include the test split? If so, test information leaks to FLNN through its constraints. Also, since each bound is the 95th percentile of the surrogate’s derivatives, the surrogate itself would violate it on about 5% of the data. Does this mean the CVR is not a reliable metric for measuring constraints’ violations?
6 comments
AuthorsOpenReview ↗

Response to Reviewer KqbL — Weakness #1 / Question #1

Weakness 1 / Question 1

Thank you for the question. Our contributions are the integrated FLNN architecture for partial feature-wise Lipschitz control and proofs of constraint preservation and universal approximation. FLNN combines separately prescribed bounds on selected features with interactions involving the remaining inputs, for which no sensitivity bounds are prescribed.

Table 1 distinguishes three properties:

  1. Feature-specific bounds. Different features may have different prescribed sensitivity limits.
  2. Separate coordinate bounds. Each bound applies individually, without an additional coupled derivative budget. For example, a joint ℓ2\ell_2 bound couples the squared partial derivatives, creating a trade-off between sensitivities in different directions (Figure 1).
  3. Unconstrained features. Bounds apply only to selected inputs, while the remaining inputs have no prescribed sensitivity limits.

Regarding (a), the architectural components. PICNN provides interacting two-branches that preserve convexity in selected inputs [1]. PBLNN uses a PICNN-based construction for partial bi-Lipschitz control, with joint Euclidean bounds over the controlled subspace [2]. FLNN targets separately prescribed upper bounds on individual controlled coordinates, without an additional joint Euclidean constraint.

Our controlled backbone follows the ℓ1\ell_1 case of the norm-constrained GroupSort construction [3]. We extend it with normalized interactions that allow uncontrolled inputs to modulate the controlled response while preserving the prescribed coordinate-wise bounds.

Regarding (b), feature-wise control through rescaling. Input rescaling for heterogeneous feature-wise bounds is established in previous work [4, 5], whose ℓ1\ell_1 formulations also allow separate coordinate bounds. With fixed input rescaling, these networks remain Lipschitz-constrained with respect to the full input vector. FLNN prescribes bounds only on selected inputs. The remaining inputs can contribute directly to the prediction and interact with the controlled branch while the prescribed bounds are preserved.

Scope of the approximation result. Theorem 2 establishes uniform approximation over both controlled and uncontrolled inputs. Under its assumptions, including compactness and strictly positive controlled bounds, FLNN can approximate any target in C(X)∩FctrlC(\mathcal X)\cap\mathcal F_{\mathrm{ctrl}} to arbitrary accuracy while preserving the prescribed bounds. The target must be continuous, but it need not be Lipschitz with respect to the uncontrolled inputs.

Our constructive proof forms component functions with continuous dependence on the uncontrolled inputs, approximates them through normalized interactions, and combines them through GroupSort (Lemmas 2–4). This establishes universal approximation for the complete interacting architecture and its partially constrained function class. The cited works do not establish this combination of partial feature-wise control and approximation guarantees. Section 4.4 additionally extends the sensitivity guarantee to prescribed region-dependent bounds.

In summary, FLNN provides an integrated architecture for partial feature-wise Lipschitz control, together with constraint-preservation guarantees and a new constructive proof of universal approximation.

[1] Amos, Brandon, Lei Xu, and J. Zico Kolter. "Input convex neural networks." International conference on machine learning. PMLR, 2017.

[2] Kinoshita, Yuri, and Taro Toyoizumi. "A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness." Advances in Neural Information Processing Systems 37 (2024): 85241-85309.

[3] Anil, Cem, James Lucas, and Roger Grosse. "Sorting out Lipschitz function approximation." International conference on machine learning. PMLR, 2019.

[4] Kitouni, Ouail, Niklas Nolte, and Mike Williams. "Robust and provably monotonic networks." Machine Learning: Science and Technology 4.3 (2023): 035020.

[5] Nolte, Niklas, Ouail Kitouni, and Mike Williams. "Expressive monotonic neural networks." The Eleventh International Conference on Learning Representations. 2022.

AuthorsOpenReview ↗

Response to Reviewer KqbL — Weakness #2

Weakness 2

As discussed in our response on novelty, FLNN belongs to the line of work on constructing neural networks that satisfy prescribed Lipschitz constraints. Norm-constrained GroupSort and Sandwich networks pursue this objective under global Lipschitz bounds [1, 2]. FLNN focuses on enforcing sensitivity bounds that can vary across selected input features and regions.

FLNN takes prescribed sensitivity bounds as inputs to the learning problem, and we acknowledge that suitable bounds must be specified for each application. These can be determined using domain knowledge or analytical models when available, or estimated from data otherwise. Estimating the Lipschitz constant of a given network is a related analysis problem [3].

In our real-world experiments, we used data-derived sensitivity bounds as a proxy for quantitative domain knowledge. For each controlled feature and region, we set the bound to the empirical 95th percentile of the magnitude of the fitted surrogate’s partial derivative with respect to that feature, evaluated within the region. FLNN uses these bounds to guide learning toward an accurate fit while ensuring by construction that the learned predictor respects the prescribed sensitivity limits.

We also added an ablation varying the number of controlled features. Please see Appendix E.2, "Effect of the number of controlled features" for the experimental setup and results. In a 40-dimensional variant of Constant, increasing dSd_S from 1 to 20 leaves mean test RMSE and training time relatively stable, with a fixed parameter count and test CVR remaining at 0.00% across all settings.

[1] Anil, Cem, James Lucas, and Roger Grosse. "Sorting out Lipschitz function approximation." International conference on machine learning. PMLR, 2019.

[2] Wang, Ruigang, and Ian Manchester. "Direct parameterization of lipschitz-bounded deep networks." International Conference on Machine Learning. PMLR, 2023.

[3] Virmaux, Aladin, and Kevin Scaman. "Lipschitz regularity of deep neural networks: analysis and efficient estimation." Advances in neural information processing systems 31 (2018).

AuthorsOpenReview ↗

Response to Reviewer KqbL — Weakness #3 / Question #2

Weakness 3 / Question 2

Bound specification. FLNN considers function approximation under sensitivity bounds assumed to be known in advance. In the original experiments, we used surrogate regressors fitted to the full datasets, including the test split, to construct empirical sensitivity requirements for this setting. To remove the influence of test labels on these requirements, we have re-estimated the bounds and regional boundaries using only the training and validation splits and repeated the experiments, reserving the test split for final evaluation. The resulting performance remains close to the original results. Please see Table 2 of the main paper and our response to Reviewer xB7D, Weakness 1 & Question 1.

Robustness to bound estimation errors. FLNN guarantees satisfaction of the prescribed bounds, while the choice of those bounds can affect predictive performance. Underestimated bounds can exclude the target function from the admissible function class and introduce approximation error, whereas overestimated bounds relax the sensitivity constraints. We have added a sensitivity analysis to Appendix D "Sensitivity to Prescribed Bounds". Both RMSE and its standard deviation remain relatively stable as the prescribed bounds are progressively increased over the tested range. These results suggest that FLNN’s predictive performance is relatively insensitive to the bound variations examined in these experiments. Please also see our response to Reviewer xB7D, Weakness 4 & Question 3.

The 95th-percentile rule and CVR. The 95th-percentile rule specifies the bounds; it does not allow FLNN to violate them on 5% of the data. Although some surrogate derivatives may exceed the selected bounds, FLNN is required to satisfy the prescribed constraints, including at points where the surrogate would exceed them. CVR measures the proportion of evaluated sample–feature pairs that violate these constraints, subject to the stated numerical tolerance. It measures compliance with the specified bounds, rather than the accuracy of the bound estimates. The surrogate’s potential violations therefore do not invalidate CVR as a measure of constraint violations.

Smaller issues :

  1. Thank you for this helpful observation. We understand that the extensive use of equation numbering may make the manuscript appear somewhat cluttered. However, we have retained the equation numbers to help readers follow the step-by-step derivations, particularly in the appendix. They also allow us to refer precisely to individual steps when explaining or clarifying the proofs, including in our responses to reviewer questions. We believe this approach helps readers follow and verify the mathematical arguments.

  2. Feature-wise control assigns different sensitivity bounds to different inputs, but the appropriate bound for a given input may also depend on its value. The region-wise extension addresses this variation within each controlled feature. In the ALD example, the same increase in exposure duration produces a larger change in surface coverage at shorter durations than near saturation. This motivates a tighter sensitivity bound at longer durations while allowing a stronger response at shorter durations. The aim is to incorporate this regional knowledge into the predictor, rather than apply the same sensitivity limit over the entire range of that feature.

    Accordingly, Section 4.4 replaces the constant bound LiL_i for each controlled feature with a prescribed bound function Li(xi)L_i(x_i). Please see our response to Reviewer S9Zq, Question 1 for the mathematical distinction between these regional bounds and a uniform bound obtained by taking the maximum over the range of the same feature.

Reviewer_xB7DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Feature-wise Lipschitz Neural Networks (FLNNs) to enforce different sensitivity bounds on selected inputs while leaving the remaining features unconstrained. The architecture combines controlled and uncontrolled branches through normalized interactions and coordinate-wise input transformations. The authors prove constraint preservation and universal approximation under constant feature-wise bounds, extend constraint enforcement to region-dependent bounds, and evaluate predictive accuracy and constraint satisfaction on synthetic and real-world regression datasets.

Critical strengths
  1. The formulation addresses heterogeneous sensitivity requirements and selectively constrains relevant features, avoiding unnecessary restrictions on the remaining inputs.
  2. The prescribed bounds hold for all trainable parameter values throughout training, rather than depending on penalty terms or constraint checks at sampled inputs.
  3. The universal-approximation theorem establishes that the architecture can approximate continuous targets satisfying the prescribed positive constant bounds on compact product domains.
  4. Integrating the prescribed sensitivity functions provides an interpretable way to enforce region-dependent bounds without introducing additional architectural constraints.
  5. FLNN reports zero observed constraint violations across all four real-world and four synthetic datasets while maintaining competitive predictive accuracy.
Critical weaknesses
  1. Appendix C.3 states that surrogate regressors are fitted to each full dataset to determine sensitivity bounds and regional boundaries. This appears to let held-out labels influence the constraints incorporated into FLNN, requiring clarification and training-only re-evaluation.
  2. Competing models’ Lipschitz parameters are selected using validation RMSE rather than satisfaction of the same feature-wise requirements. Consequently, the reported comparisons do not fully establish predictive superiority under equivalent constraints.
  3. The experiments lack ablations separating the benefits of input transformation, two-branch decomposition, and layer-wise interactions. Comparisons with transformed-input baselines and a simpler, selectively constrained GroupSort architecture would clarify the proposed design’s incremental value.
  4. In Constant, Region, and LinearRegion, the bounds match the coordinate-response slopes. Integrating these bounds therefore substantially simplifies the controlled responses, limiting evidence for more complex interactions or imperfectly specified bounds.
  5. FLNN has higher RMSE than the unconstrained MLP on Airfoil and CCPP, while its AutoMPG improvement is small relative to the reported variability. The results support competitive accuracy with guaranteed constraints more strongly than general predictive superiority.
Questions for authors
  1. Were validation or test labels used when fitting the surrogate models that determine sensitivity bounds and regional boundaries? How do the results change when this procedure uses training data only?
  2. How much improvement comes from the normalized layer-wise interactions rather than the input transformation? Could you provide component ablations and comparisons against simpler architectures using the same transformations and sensitivity constraints?
  3. How robust is FLNN to overestimated or underestimated sensitivity bounds, and do its advantages persist on synthetic targets with nonmonotone responses and stronger interactions between controlled features?
  4. See weakness too.
11 comments
AuthorsOpenReview ↗

Response to Reviewer xB7D — Weakness #1 / Question #1

Weakness 1 / Question 1

We agree with your concern and thank you for the helpful suggestion. In the original experiments, we fitted the surrogate regressors to the full datasets, including both validation and test labels, to estimate the sensitivity bounds and regional boundaries.

In the setting we study, we assume that the sensitivity requirements are known in advance when the relevant domain knowledge is available. We hope this clarifies the premise of the original experiments, in which we used surrogate regressors to construct empirical sensitivity requirements for the real-world benchmarks. We agree with your suggestion to hold out the test split when estimating these requirements from data. We therefore re-estimated the bounds and regional boundaries using only the data available at training time (the training and validation splits) and reserved the test split for final evaluation.

The revised sensitivity bounds are comparable in magnitude to the original bounds, and the number of regions per controlled feature is the same under both protocols.

datasetOriginal bounds (full dataset)Revised bounds (training + validation)
airfoil[0.05, 0.012], [0.25][0.037, 0.014], [0.25]
autompg[0.015], [0.2], [0.15, 0.07][0.011], [0.25], [0.097, 0.044]
ccpp[2.5][2.8]
concrete[2.7, 1.1, 0.7, 0.03], [0.2][3.5, 1.0, 0.65, 0.023], [0.21]

We have updated Table 2 in the main text with the results obtained under the revised bounds. FLNN’s performance was stable across all four datasets, with test RMSE within approximately 1.5% of the original values and CVR remaining at 0.00% on every dataset. The revised results continue to support our conclusion that FLNN achieves competitive predictive accuracy while satisfying the prescribed sensitivity constraints.

AuthorsOpenReview ↗

Response to Reviewer xB7D — Weakness #2, #5

Weakness 2, 5

Our experimental objective is to evaluate predictive accuracy together with satisfaction of the prescribed feature-wise sensitivity requirements. We clarify the original model selection procedure and present an additional comparison that explicitly incorporates constraint satisfaction.

The prescribed bounds apply only to controlled inputs and do not determine an appropriate global Lipschitz constant for compared models that constrain the full input vector. A small global constant may unnecessarily restrict dependence on uncontrolled inputs. We therefore considered larger constants, with search ranges informed by the largest prescribed controlled-feature bound for each dataset, and selected the final parameters by validation RMSE.

The additional comparison under constraint requirements is reported in Table 3 for real-world datasets and Table 12 for synthetic datasets, and reproduced below. Keeping trained weights fixed, we select one common Lipschitz parameter for each compared model and dataset that minimizes mean validation RMSE among candidates with no observed validation violations across all five seeds. We then report test RMSE and CVR, with FLNN evaluated in its original configuration.

Under this procedure, FLNN achieves the lowest mean test RMSE on most datasets and zero observed test CVR on all datasets. Several compared models exhibit substantial increases in RMSE relative to the predictive performance results in Table 2, and some still exhibit test violations despite satisfying the validation constraints.

Although the MLP achieves lower RMSE on Airfoil and CCPP in Table 2, it also violates the prescribed requirements. The additional results demonstrate the value of directly incorporating these requirements into the architecture. FLNN preserves predictive accuracy while enforcing the specified bounds by construction. These findings support competitive accuracy with guaranteed constraint satisfaction, rather than general predictive superiority. Thus, the additional experiment distinguishes empirical feasibility on validation samples from the architectural constraint guarantee provided by FLNN.

Real-world datasets

MethodAirfoilAutoMPGCCPPConcrete
RMSECVRRMSECVRRMSECVRRMSECVR
SN-ℓ1\ell_19.37 ±\pm 0.180.00 ±\pm 0.004.17 ±\pm 2.440.00 ±\pm 0.004.20 ±\pm 0.050.00 ±\pm 0.0036.33 ±\pm 0.980.00 ±\pm 0.00
SN-ℓ2\ell_211.28 ±\pm 0.180.00 ±\pm 0.008.57 ±\pm 1.360.00 ±\pm 0.0025.51 ±\pm 0.410.00 ±\pm 0.0036.90 ±\pm 0.990.00 ±\pm 0.00
Cayley17.93 ±\pm 0.240.00 ±\pm 0.0015.29 ±\pm 1.210.00 ±\pm 0.0025.47 ±\pm 0.400.00 ±\pm 0.0037.20 ±\pm 0.990.00 ±\pm 0.00
AOL20.09 ±\pm 0.270.00 ±\pm 0.0013.71 ±\pm 1.180.00 ±\pm 0.0015.61 ±\pm 0.290.00 ±\pm 0.0036.28 ±\pm 0.980.00 ±\pm 0.00
Sandwich10.07 ±\pm 1.170.00 ±\pm 0.009.09 ±\pm 0.920.11 ±\pm 0.2512.47 ±\pm 0.760.00 ±\pm 0.0031.97 ±\pm 5.060.00 ±\pm 0.00
LMN4.91 ±\pm 1.140.13 ±\pm 0.302.95 ±\pm 0.450.00 ±\pm 0.0014.04 ±\pm 1.310.00 ±\pm 0.0028.62 ±\pm 3.480.00 ±\pm 0.00
PBLNN6.01 ±\pm 0.290.00 ±\pm 0.004.43 ±\pm 0.730.00 ±\pm 0.007.08 ±\pm 0.110.00 ±\pm 0.0036.04 ±\pm 1.330.00 ±\pm 0.00
FLNN2.04 ±\pm 0.200.00 ±\pm 0.002.97 ±\pm 0.380.00 ±\pm 0.003.91 ±\pm 0.100.00 ±\pm 0.005.39 ±\pm 0.400.00 ±\pm 0.00

Synthetic datasets

MethodConstantRegionALDLinearRegion
RMSECVRRMSECVRRMSECVRRMSECVR
SN-ℓ1\ell_13.853 ±\pm 0.3150.00 ±\pm 0.007.696 ±\pm 0.3360.00 ±\pm 0.000.149 ±\pm 0.0450.00 ±\pm 0.0051.172 ±\pm 3.1840.00 ±\pm 0.00
SN-ℓ2\ell_25.805 ±\pm 0.4430.04 ±\pm 0.108.762 ±\pm 0.3810.00 ±\pm 0.000.086 ±\pm 0.0080.00 ±\pm 0.0050.786 ±\pm 3.3740.00 ±\pm 0.00
Cayley10.449 ±\pm 0.8170.00 ±\pm 0.008.828 ±\pm 0.3850.00 ±\pm 0.000.093 ±\pm 0.0140.00 ±\pm 0.0038.732 ±\pm 2.5530.04 ±\pm 0.10
AOL9.290 ±\pm 0.7290.00 ±\pm 0.008.762 ±\pm 0.3800.00 ±\pm 0.000.023 ±\pm 0.0040.00 ±\pm 0.0047.395 ±\pm 3.1680.00 ±\pm 0.00
Sandwich8.756 ±\pm 0.6090.00 ±\pm 0.009.206 ±\pm 0.7940.00 ±\pm 0.000.018 ±\pm 0.0050.00 ±\pm 0.0062.457 ±\pm 5.2280.00 ±\pm 0.00
LMN5.079 ±\pm 0.6770.00 ±\pm 0.007.035 ±\pm 0.8220.00 ±\pm 0.000.012 ±\pm 0.0040.00 ±\pm 0.0041.542 ±\pm 1.4370.00 ±\pm 0.00
PBLNN10.231 ±\pm 1.5780.00 ±\pm 0.008.947 ±\pm 0.6790.00 ±\pm 0.000.138 ±\pm 0.0140.00 ±\pm 0.0023.650 ±\pm 5.0210.00 ±\pm 0.00
FLNN0.107 ±\pm 0.0170.00 ±\pm 0.000.087 ±\pm 0.0140.00 ±\pm 0.000.016 ±\pm 0.0020.00 ±\pm 0.001.176 ±\pm 0.4620.00 ±\pm 0.00
AuthorsOpenReview ↗

Response to Reviewer xB7D — Weakness #3 / Question #2

Weakness 3 / Question 2

Thank you for these comments. The input transformation, two-branch decomposition, and normalized interactions serve different roles in FLNN. We clarify these roles below and provide two architectural ablations that retain the same input transformation and prescribed feature-wise requirements.

Input transformation. The transformation incorporates the prescribed bounds into the controlled inputs. Constant bounds correspond to coordinate-wise rescaling, while region-dependent bounds are incorporated through a transformation with the prescribed slope in each region. Together with the constrained network, this converts sensitivity bounds in the transformed coordinates into the required bounds in the original coordinates.

Removing the transformation without replacing its role in constraint enforcement would change the imposed requirements. We therefore retain it in both ablations to examine the additional contributions of the architecture under the same prescribed requirements.

Two-branch decomposition. Separate branches allow sensitivity constraints to apply to the controlled inputs without imposing such requirements on the remaining inputs. We compare FLNN with a single branch constrained GroupSort network that receives both the transformed controlled inputs and the uncontrolled inputs through concatenation. In this baseline, both input groups are subject to the same norm constraints, which can unnecessarily limit how the prediction responds to the uncontrolled inputs. This comparison examines the benefit of separating constrained and unconstrained computations.

Normalized interactions. We compare FLNN with a variant without layer-wise interactions that retains the same input transformation and two branches but removes all multiplicative gates. This comparison examines the contribution of interactions between the controlled and uncontrolled inputs beyond the shared input transformation.

Connection to universal approximation. The two-branch decomposition and normalized interactions also play explicit roles in the constructive proof of Theorem 2. Lemma 3 uses the uncontrolled branch and normalized interactions to approximate component functions with continuous dependence on the uncontrolled inputs while preserving the prescribed sensitivity bounds. GroupSort then combines these components through finite minima and maxima. Under the assumptions of Theorem 2, this construction establishes universal approximation for FLNN, providing a theoretical basis for the architecture alongside the empirical contributions examined by the ablations.

As shown in the tables below, FLNN achieves lower mean RMSE than the single-branch variant on all eight datasets. Compared with the variant without layer-wise interactions, the improvements are most pronounced on the synthetic datasets and Airfoil, whereas the differences are smaller on CCPP and Concrete, and this variant performs slightly better on AutoMPG. Since all variants use the same input transformation, these results demonstrate the additional contribution of the architectural components, with benefits that vary across datasets. We have added these ablation results to Appendix E.1 of the revised manuscript.

Real-world datasetsAirfoilAutoMPGCCPPConcrete
w/o interaction4.64 ±\pm 0.242.95 ±\pm 0.423.94 ±\pm 0.135.55 ±\pm 0.48
single branch2.46 ±\pm 0.113.09 ±\pm 0.534.03 ±\pm 0.066.04 ±\pm 0.49
FLNN2.01 ±\pm 0.212.99 ±\pm 0.393.93 ±\pm 0.075.28 ±\pm 0.52
Synthetic datasetsConstantRegionALDLinearRegion
w/o interaction0.714 ±\pm 0.0220.551 ±\pm 0.0250.042 ±\pm 0.0076.577 ±\pm 0.886
single branch0.117 ±\pm 0.0110.100 ±\pm 0.0110.020 ±\pm 0.0053.441 ±\pm 0.659
FLNN0.107 ±\pm 0.0170.087 ±\pm 0.0140.016 ±\pm 0.0021.176 ±\pm 0.462

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Discussion Summary

Dear AC and Reviewers,

We sincerely thank all reviewers for their feedback and discussion. We summarize our core contribution and the additional analyses and revisions addressing the concerns raised during review.

Core contribution. FLNN combines separate sensitivity bounds on selected features with interactions involving inputs that have no prescribed bounds. Our contribution comprises this integrated architecture and its guarantees of constraint preservation and constructive universal approximation.

1. Contribution and theoretical scope

The revised Introduction and related-work discussion clarify these contributions and their distinction from existing rescaling approaches and architectures with joint sensitivity constraints. Our approximation guarantee covers continuous targets satisfying prescribed positive constant bounds on compact domains, without requiring Lipschitz continuity in uncontrolled inputs. A coordinate-wise transformation additionally extends constraint enforcement to region-dependent bounds.

2. Comparisons with existing methods

When sensitivity constraints are treated as requirements, most compared methods exhibit substantial increases in test RMSE compared with prioritizing predictive accuracy alone. FLNN satisfies these constraints while achieving markedly lower mean test RMSE on most datasets. Comparisons with LMN further show that increasing bounds on uncontrolled inputs can reduce predictive accuracy, with FLNN achieving lower mean test RMSE on most datasets.

3. Architectural ablations and extended evaluation

We added architectural ablations using the same input transformations, together with experiments on nonmonotone responses with stronger controlled-feature interactions, increasing numbers of controlled features, and binary classification. The ablations demonstrate benefits from branch separation and interactions beyond the shared input transformation, with gains that vary across datasets.

4. Bound specification

Additional analyses examine bound estimation using only data available at training time, variability across data splits, and sensitivity to the prescribed bounds. The revised manuscript also explains how bounds can be supplied through process knowledge or physical relations. The main findings remain consistent when bounds are estimated without test data. Performance remains stable under overestimated bounds within the tested range, whereas overly restrictive bounds can reduce accuracy.

We respectfully ask that these clarifications, additional analyses, and manuscript revisions be considered in the final assessment. We sincerely thank you for your time, thoughtful feedback, and engagement throughout the rebuttal period, which have helped us improve the clarity and quality of our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.