acceptodds
Under review as a conference paper at ICLR 2027

Accelerating Global optimisation with Piecewise-Linear Kolmogorov-Arnold Network Surrogates

Abstract

Neural surrogates can provide highly accurate approximations of complex functions, yet this alone does not guarantee reliable and efficient downstream global optimisation. When neural networks are embedded into mathematical programs, their architecture and representation directly affect formulation size, relaxation strength, and ultimately the optimisation runtime. This paper focuses on continuous piecewise-linear Kolmogorov–Arnold networks (CPWL-KAN), whose learnable edge functions are parametrised as linear combinations of ReLU functions and admit exact mixed-integer programming (MIP) representations. Three CPWL-KAN variants are considered, which correspond to different weight combinations and expressivity properties. Numerical experiments conducted across three benchmark functions and an engineering optimisation problem show that CPWL-KANs can provide predictive accuracy comparable to, and sometimes better than, that of conventional ReLU MLPs of similar complexity; but the computational runtime with MIP solvers to certify global optimality of the surrogates can be significantly lower with CPWL-KANs embedded. Overall, these results demonstrate that functional parametrisation of neural surrogates is as important as predictive accuracy when their intended use is in downstream optimisation.

Primary area:
optimization
Keywords:
Kolmogorov-Arnold Networks, Global Optimisation, Piecewise Linear, Surrogate Modelling, Mixed-Integer Linear Programming

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LxwNOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The present manuscript investigates the use of certain Kolmogorov-Arnold networks (CPWL-KANs) as surrogate models for global optimization goals. The authors introduce KANs with different expressivity and develop exact mixed-integer formulations. Through theoretical analysis and experiments on benchmarks and an engineering problem (hydrogen production), they demonstrate that CPWL-KANs can achieve competitive predictive accuracy while also reducing downstream optimization runtimes compared to conventional MLPs. The central findings are particularly interesting: (1) the internal representation of a learned function, rather than predictive accuracy alone, matters for global optimization performance, as demonstrated with CPWL-KANs; and (2) the surrogate models with the highest predictive accuracy are not necessarily the most computationally efficient to optimize.

Critical strengths
  1. The paper addresses an important but relatively underexplored aspect of surrogate modeling: how neural-network representations and their parameterizations affect downstream global optimization. The connection between representational structure, relaxation strength, and solver efficiency is compelling, particularly for scientific/engineering applications.

  2. The analysis of expressivity and relaxation strength provides a clear explanation for the observed computational improvements. In particular, exploiting the univariate piecewise-linear structure of KAN edges through joint SOS2 formulations is a smart approach, and the experiments show meaningful improvements in optimization runtime.

Critical weaknesses

If I understand correctly, the architectural advantage of KANs is not yet fully isolated. The main computational improvements appear to arise from exploiting univariate piecewise-linear representations and stronger SOS2 relaxations, rather than from a fundamental advantage of KANs themselves. Since related CPWL-KAN and ReLU MLP representations can be mathematically equivalent, it remains unclear how much of the observed speedup is attributable specifically to the KAN architecture versus the choice of optimization formulation.

Questions for authors
  1. Given the known representational connections between CPWL-KANs and ReLU MLPs, could the authors clarify whether comparable gains might be obtained by applying stronger piecewise-linear formulations to equivalent MLP representations?
  2. How do the improvements scale with problem complexity? In particular, would the authors expect similar advantages for higher-dimensional or deeper networks that may be required for more complex problems?
2 comments

We appreciate the reviewer’s suggestions in enhancing the surrogate modelling ability of CPWL-KANs.

1. Relation to equivalent ReLU-ANN representations.

CPWL-KANs enable the grouping of multiple ReLU terms that share a single scalar input within each edge function – this grouping is not possible in general for a ReLU-ANN as the ReLU terms in any given layer will depend on multiple inputs (the affine layer being applied prior to the ReLU activation). Exact conversions preserve the underlying input dependencies: KAN-to-ANN conversions retain the shared-input grouping, whereas the reverse conversion retains the affine aggregation preceding each activation (Schoots et al., 2025). Our contribution concerns how CPWL-KAN parametrisations explicitly represent the univariate CPWL edge function structure and how the corresponding formulations exploit it.

With CPWL-KANs, the separate big-M relaxation already gives the convex hull of each individual ReLU term over a common scalar input domain. Joint encoding goes further by capturing their dependence through the complete CPWL edge graph, yielding a no-weaker relaxation (Proposition 3.5). Our formulations exploit this shared-input structure for joint convexification, bound propagation and breakpoint pruning. The computational results of CPWL-KANs over the corresponding ReLU_ANNs (Section 4.2; Appendix, Table 14) confirm that the observed improvements do not depend solely on the encoding approaches but from the unique network structure.

Although other stronger formulations for ReLU-ANNs, such as the convex-hull and partition-based formulations, may improve the optimisation performance of ReLU-ANNs, these typically come at the expense of additional auxiliary variables and preprocessing efforts. Furthermore, the dependence among inputs is inherent to ReLU-ANNs, so the unique advantages afforded by univariate CPWL edge function in CPWL-KANs does not readily translate to ReLU-ANNs in general. We added an explanation of this key structural difference at the end of Section 3 (Page 6) in the revision.

2. Scalability with input dimension and network depth.

We have added scaling experiments in Appendix C.3.5 (Tables 16–17), covering 4D, 8D, and 12D Ackley and Rosenbrock problems, and Hartmann-6D networks of increasing depth, with a 600 s time limit.

For Ackley-12D, KAN-TITO and KAN-TIFO solve all 128 instances under both encodings, with mean runtimes of 0.5–2.2 s, while ANN-TW and ANN-FW solve 2.3% and 26.6%, respectively. For Rosenbrock-12D, KAN-TIFO solves all instances under both encodings; KAN-TITO solves all with separate big-M and 82.0% with joint SOS2 encoding.

Increasing depth is more challenging as the network complexity increases significantly. At depth 3, KAN-TIFO with separate big-M solves all ten instances in 22.4 s on average, compared with 35.9 s for ANN-FW. At depth 4, KAN-TIFO solves 20% and 40% with joint SOS2 and separate big-M, respectively, while the other models solve none. Bound overestimation can accumulate across layers, weakening network relaxations despite strong individual edge formulations.

These results show that computational advantages persist in the higher-dimensional problems, while benefits for deeper networks depend more strongly on the parametrisation and encoding. We discuss these trends and limitations in the revised appendix.

I thank the authors for clarifying how the CPWL-KAN parameterization exposes shared-input structure and for adding the scaling experiments. The new results address my scalability question: the gains persist on the higher-dimensional benchmarks tested, while the depth experiments show where those gains become less reliable.

My main concern is partly addressed. The rebuttal explains why the parameterization makes the shared-input structure easy to exploit, but it remains unclear whether an exactly equivalent ReLU network, formulated to exploit the same grouping, would achieve comparable runtimes. The results support the practical value of the parameterization and formulation together, but do not fully isolate their respective contributions. I am therefore keeping my original score of 3 (weak acceptance).

Reviewer_2XdUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the representability of Kolmogorov-Arnold networks (KANs) as Mixed Integer Programs. KANs differ from standard neural networks by applying the (learnable) univariate activation function on each edge of the computational graph, rather than at vertices. Three parameterizations of the univariate activation function, which combines an affine function and ReLUs, are proposed; they differ by the number of parameters and their expressivity. Then, two MIP formulations for the learnable activation functions are proposed, involving big-M or SOS2 techniques; the SOS2 has a tighter relaxation than the big-M method. Experiments on synthetic and an industrial test case compare the behavior of the KAN variants and more standard feed forward neural networks, in terms of time to solve the MIP and validation error.

While the paper is clearly written, and the main content has a good quality, I am not sure about how relevant the question studied here is. I detail this below, and will update my assessment with this information.

Critical strengths
  • the approach to solve the proposed problem (finding good MIP formulations for KANs) is reasonable; the theoretical results characterize the relevant properties.
Critical weaknesses
  • It is unclear from the introduction what kind of tasks require solving an optimisation problem featuring a neural network to global optimality. It is therefore unclear if (i) there exists alternatives to solving MIPs to that problem, and (ii) whether global optimality is really required. For instance on (i), it is known that SGD iterates will visit arbitrary neighborhoods of global minimizers (see e.g. [1]); it would then be relevant to compare the two approaches. For instance on (ii), for statistical purposes, it is generally enough to minimize empirical risk up to some threshold which corresponds to the mismatch between the empirical risk and the true risk. Indeed, this mismatch is a price that is paid anyway in the overall risk of the learned predictor.

  • The numerical section lacks a comprehensive exposition of the problems solved. In particular, what are the MIP being solved? What are the tasks at hand for the synthetic instances? for the hydrogen production case? (e.g., regression of a function)

[1] The Global Convergence Time of Stochastic Gradient Descent in Non-Convex Landscapes: Sharp Estimates via Large Deviations (icml 2025)

Questions for authors

Any material on the two weaknesses above would be helpful in refining my assessment of the paper.

4 comments

We appreciate the reviewer’s constructive feedback in clarifying the motivation and experimental descriptions. We have replaced the hydrogen case with a more complicated HDA flowsheet optimisation problem to better demonstrate the ability of CPWL-KANs.

1. Downstream optimisation and the role of global optimality.

We have revised the Introduction (Pages 1 and 2) to clarify the engineering setting and relevance: trained surrogates replace expensive process simulations when optimising operating decisions under production and quality constraints. Network parameters remain fixed; the optimisation variables are operating conditions, and the objective is process cost or energy consumption, rather than training loss. The revised Introduction also explains the purpose of global certification. A feasible operating solution and a global lower bound quantify how much further improvement remains possible and support termination at a prescribed optimality tolerance. Although some local and stochastic search approaches are useful in practical applications, developing efficient methods for global optimisation is always desired to achieve the best process performance, either from the perspective in saving energy or increasing profits. Given the scale of chemical processes, small percentage improvements can enable significant financial and environmental gains.

Beyond the traditional application of neural surrogates in simulation (e.g. for function evaluation), when embedding these surrogates into optimisation problems, the prediction error alone is no longer sufficient to decide what an acceptable optimisation gap might be. When the surrogate is sufficiently accurate for the intended decision task, global bounds help quantify the remaining potential for improvement. This motivates considering not only predictive accuracy but also downstream optimisation tractability when designing surrogate models. We demonstrate that the CPWL-KAN can achieve comparable predictive ability to common ReLU-ANNs when trained using a similar approach, while significantly speeding up the optimisation performance (see results and discussion sections). This provides an alternative approach to deploying data-driven networks in mathematical optimisation, enlarging the application area.

While local and stochastic search could also be applied to optimise the surrogates, these methods cannot certify global optimality of the design and operational decisions in general. This distinction matters in process optimisation: a feasible solution or a local optimum alone does not quantify the remaining potential for cost reduction, whereas MIP solvers provide this information through the incumbent objective value and a global lower bound.

To illustrate this, we applied SGD to the KAN-TITO formulation for HDA using 25 initial points, solved in parallel and subject to the same time budget for each point (between 10-600 s). Constraint satisfaction was checked a posteriori, and the objective statistics (in million USD/year) include feasible solutions only. The results illustrate the high sensitivity to SGD initialisation and confirm that, even with a significantly larger time budget, none of the SGD runs can identify the global solution with objective value of 0.40.

MethodBudget per start (s)Feasible runsObjective: mean ± SDMedianMin–max
SGD10.008/250.42 ± 0.010.420.41–0.43
SGD60.008/250.42 ± 0.010.420.41–0.43
SGD300.008/250.42 ± 0.010.420.41–0.43
SGD600.008/250.42 ± 0.010.420.41–0.43
MIP reference60.421/10.40——

2. Definition of the experimental optimisation tasks.

All surrogate models are trained for regression tasks to predict the objective values or process metrics. The models are then embedded into the problems with algebraic encoding approaches described in the paper for optimisation.

We have expanded Section 4 to clarify the downstream optimisation setting and present its general formulation in Eq. (4.1). Section 4.3 describes the objective and constraints of the HDA flowsheet problem. Appendix B.5 provides the analytical benchmark definitions, HDA decision variables and bounds (Table 8), the HDA optimisation formulation (Eq. (B.7)), and instance generation procedures. Data preparation, training, and solver settings are detailed in Appendices B.1–B.4.

Analytical benchmarks. For Ackley, Rosenbrock and Hartmann, we first train a surrogate f^\hat{f} to approximate the corresponding function. With the trained parameters fixed, we solve

\min_{\boldsymbol{x}} \quad & \hat{f}(\boldsymbol{x}) \\\\ \mathrm{s.t.} \quad & \underline{\boldsymbol{x}}^{(k)} \leq \boldsymbol{x} \leq \overline{\boldsymbol{x}}^{(k)}. \end{aligned}$$ Here, $k$ indexes the optimisation instance. Varying the input bounds generates 128 instances per benchmark. **HDA flowsheet optimisation.** The surrogate $\hat{f}_{\mathrm{HDA}}$ maps seven operating variables $\boldsymbol{x}$ to four process outputs $\boldsymbol{z}$. Following Eq. (B.7), the optimisation problem is $$\begin{aligned} \min_{\boldsymbol{x},\boldsymbol{z}} \quad & z_1 \\\\ \mathrm{s.t.} \quad & \boldsymbol{z} = \hat{f}_{\mathrm{HDA}}(\boldsymbol{x}), \\\\ & z_2 \geq F_{\min}, \\\\ & z_3 \geq p_{\min}, \\\\ & z_4 \leq L_{\max}, \\\\ & \underline{\boldsymbol{x}} \leq \boldsymbol{x} \leq \overline{\boldsymbol{x}}. \end{aligned}$$ The outputs $z_1,z_2,z_3,z_4$ represent the predicted annual cost, distillate flowrate, benzene purity, and flash overhead benzene loss fraction, respectively. The 165 instances are generated by varying the minimum flowrate $F_{\min}$ and purity $p_{\min}$ requirements, while keeping the operating bounds and maximum benzene-loss fraction $L_{\max}$ fixed.

I thank the authors for the detailed response and the revision.

My second question is answered by your comment and the revision.

My first question is partly answered: I now understand the motivation of this work, and that the paper does not consider global optimization of the KAN training problem, but rather global optimization of an operational problem featuring a trained KAN. I was thrown off by the amount of detail given on the quality of KAN training (MSE plot in experiments, corresponding section among other things). I believe the revision suitably clarifies the paper's positioning in that regard. With that new understanding, I would like to ask new questions:

  • the quality of the global solution is only as good as the approximation quality of the KAN relative to the industrial process at hand. In the HDA rebuttal experiment, the best multisart SGD reaches 0.41, while the MIP reaches 0.40 (~2.5% improvement, although more decimals would be helpful here). What is the KAN prediction error on z1z_1? Could you evaluate both solutions' objective value and constraint satisfaction on the actual HDA simulation software? In that case, does the MIP solution remains the best one? By how much?
  • Some more details on SGD are necessary to interpret the result:
    • where does the stochasticity comes from in the SGD iteration? can you characterize it?
    • what is the stepsize schedule used?
    • how are the constraints handled? (sgd does unconstrained minimization, perhaps a mehod from constrained ML would be useful here?)
    • the results are identical over the widening time budgets. This suggests that SGD iterates settle to a local minimizer very quickly, and thus stray away from the setting of [1] from my initial review. Adding more noise may be helpful here.
    • The multistart with 10s reaches 0.41 best loss value, while the MIP needs 60s to find a 0.40 incumbent and certify its optimality. These are two qualitatively different settings, which is why I think that it is important that SGD be in the regime of [1].
    • This is behavior on 1 instance of HDA out of 165. How did you choose the instance? Is this representative of the behavior on other instances?

Answers to these points would help me finalize my assessment.

We appreciate the reviewer’s assessment on the revised paper.

We would first like to reemphasize that the main contribution of our paper is on improving the tractability of deterministic global optimisation (DGO) over neural surrogates by exploiting the univariate CPWL structure of KANs. In contrast to DGO which provides a certificate of global optimality within a preset ε>0\varepsilon>0 tolerance in finite time, local search methods (e.g. gradient descent) and stochastic search methods likewise are unable to produce such certificates in finite time or iterations [1]. While we agree with the reviewer that it is perfectible, our previous comparison between DGO and SDG (representing local-search methods) was aimed at illustrating this gap.

[1] Neumaier, A. (2004) Complete Search in Continuous Global Optimization and Constraint Satisfaction. Acta Numerica, 13, 271-369. https://doi.org/10.1017/s0962492904000194

1. Predictive error for surrogate modelling.

We agree with the reviewer that predictive accuracy is a foremost concern in surrogate-based optimisation, as it is only meaningful to optimise over surrogates that yield sufficiently high accuracy. This is also the reason why we investigated both the predictive performance (Section 4.1) and the optimisation efficiency (Sections 4.2 & 4.3). In particular, we confirmed that all of the trained CPWL-KANs are in excellent agreement with the sampled ground truth (Table 11), with a normalised test MSE of (5.29 ± 1.01) × 10⁻⁵ (mean ± SD), thereby providing a suitable basis for optimisation. For the HDA process benchmark, we furthermore fixed all seven operating variables to the solutions returned by either DGO or SGD (with a fixed step size), and simulated them in the IDAES platform:

MethodFeasible/startsBest objectiveLower boundMin–MaxIDAES costConstraints met?
SGD8/250.4074463—0.4074463‑0.43316730.4081804Yes
SGD-F8/250.4071295—0.4071295‑0.43317320.4078804Yes
MIP/DGO—0.39933600.3993360—0.3991872Yes

All the final solutions satisfy physical constraints in IDEAS due to the high accuracy and DGO remains the best one.

Naturally, we also acknowledge that even with high predictive accuracy, the surrogate solution can sometimes violate physical constraints. Approaches for improving constraint satsifaction in surrogate-based optimisation include the adddition of safety margins [2], but we consider it to be beyond the scope of our paper.

[2] Petsagkourakis P, Sandoval I O, Bradford E, et al. (2022) Chance constrained policy optimization for process control and optimization. Journal of Process Control, 111: 35-45.

2. Details and claims on SGD.

The implementation details are provided below at the reviewer’s request.

Implementation details of the SGDs.

SettingSGDSGD-F
Learning rate at iteration k0.005/(1+k/2000)0.60.005/(1+k/2000)^{0.6}0.005 (fixed)
Starting points25 sampled from [0,1]7[0,1]^7Same
Gradient noiseN(0,0.052)\mathcal{N}(0, 0.05^2)Same

Q1-Q3: The stochasticity arises from the specification of multiple starting points. Constraint satisfaction is only checked after the optimisation, so adding a constraint violation penalty to the loss function could help improve the solution feasibility.

Q4-Q5: Reference [3] characterises the expected iterations for constant-step SGD to enter a neighbourhood of a global minimiser. Its analysis assumes a twice continuously differentiable objective. Our trained CPWL-KAN is piecewise affine in operating variables and is generally nonsmooth at region boundaries. The HDA problem also includes explicit process constraints. The theorem therefore does not directly apply, even to our fixed-step runs. And again, such a local search method is unable to certify global optimality.

[3] Azizian W, Iutzeler F, Malick J, Mertikopoulos P (2025) Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:1982-2044.

Q6: We selected this instance from the IDAES flowsheet example [4]. It lies within the range covered by the 165 instances, with detailed constraints in Appendix B.5.3. Figure 3 reports the distribution of MIP runtimes across all instances and both KAN-TITO encodings reached the prescribed optimality gap in every case. For SGD, the objective range from different start points above show that the best feasible cost varied across random starts and noise realisations. Close behavior for these methods is therefore expected for other instances.

[4] Lee A, Ghouse J H, Eslick J C, et al. The IDAES process modeling framework and model library—Flexibility for process simulation and optimization[J]. Journal of advanced manufacturing and processing, 2021, 3(3): e10095.

Reviewer_7DuFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the use of continuous piecewise-linear Kolmogorov-Arnold networks (CPWL-KANs) as surrogate models embedded into mixed-integer programming formulations for deterministic global optimization. The authors propose three CPWL-KAN parameterizations: KAN-TITO, KAN-FITO, and KAN-TIFO. They analyze their expressive properties and derive two exact mixed-integer formulations for their edge functions: separate big-M and joint SOS2 formulations. The paper also introduces an interval-pruning procedure that identifies unreachable breakpoints through bound propagation and reduces the size of the embedded MIP formulation.

The proposed models are evaluated on the Ackley, Rosenbrock, and Hartmann benchmark functions, as well as on a hydrogen-production process. The experiments examine predictive error, continuous relaxation gaps, the number of effective piecewise-linear intervals, model-generation time, optimization runtime, and objective-value discrepancies. The results indicate that KAN-TITO generally provides better predictive accuracy and objective-value agreement, while KAN-TIFO often yields more compact effective representations and faster optimization. The paper’s main argument is that neural-network surrogates for downstream global optimization should be designed with consideration for predictive performance, piecewise-linear representation structure, and the computational characteristics of the embedded optimization model.

Critical strengths

The paper addresses a practically relevant issue. Once a neural-network surrogate is embedded into a mathematical program, predictive error cannot be the only evaluation criterion; the piecewise-linear structure, continuous relaxation, and MIP size can also affect global optimization performance.

The paper also provides a relatively systematic theoretical and computational analysis of three CPWL-KAN parameterizations, exact MIP formulations, and interval pruning. This analysis may serve as a useful reference for understanding how different CPWL-KAN structures affect embedded optimization models.

Critical weaknesses

1.The paper does not propose a training or model-selection mechanism that implements an active trade-off between predictive accuracy and downstream optimization tractability. Although this trade-off is presented as the central motivation, the training objective, architecture search, and model selection remain primarily driven by MSE, while optimization tractability is evaluated only after training. The paper does not provide a joint loss, complexity regularizer, MIP-aware architecture search procedure, relaxation-strength constraint, or optimization-time-based selection criterion. Consequently, it does not explain how the proposed method achieves the balance it claims to target.

2.The paper also does not demonstrate that CPWL-KANs resolve a specific unresolved problem in mixed-integer formulations of ReLU MLPs. Existing work has already considered exact ReLU encodings, bound tightening, partition-based formulations, convex-hull relaxations, and structured preprocessing. The paper does not clearly describe the remaining limitations of these methods or explain why CPWL-KANs are necessary to address them. Joint SOS2 encoding, univariate edge functions, bound propagation, and interval pruning may be the main sources of the reported improvements, but the paper does not separate these factors from the KAN architecture itself.

Questions for authors

1.The paper should clarify exactly how the claimed balance between predictive accuracy and downstream optimization tractability is achieved. The current training and model-selection procedures are primarily based on MSE, while formulation size, relaxation strength, effective interval count, and optimization runtime do not enter any explicit objective. The authors should provide a concrete joint objective, regularization method, architecture-search criterion, or model-selection rule that explains how the model trades off these potentially conflicting metrics.

2.The paper should also precisely identify the unresolved bottleneck in existing ReLU MLP mixed-integer optimization methods. It is not sufficient to state that separate neuron-wise big-M formulations may produce weak relaxations, because prior work has already proposed bound tightening, partition-based formulations, convex-hull relaxations, and network preprocessing. The authors should explain the limitations of these methods and why the capability provided by CPWL-KANs cannot be obtained through improved MLP representations and formulations.

3.The paper should further separate the contribution of the KAN architecture from the effects of the accompanying optimization techniques. Can joint SOS2 encoding, univariate edge representations, bound propagation, and interval pruning be applied to strengthened ReLU MLP formulations as well? If the same encoding, bound propagation, and pruning procedures are applied, does CPWL-KAN still have an independent advantage? Without such comparisons, the reported improvements cannot be attributed specifically to KAN rather than to the associated MIP modeling techniques.

4.For KAN-TIFO, the authors should clarify whether its optimization-speed advantage comes from the convexity of its edge functions or primarily from the smaller number of effective intervals after pruning. For KAN-TITO, the authors should clarify whether its predictive and objective-value advantages arise from trainable inner weights themselves or from differences in initialization, training trajectories, and breakpoint distributions.

2 comments

We appreciate the reviewer’s insightful and invaluable comments on this work.

1. Clarifying the accuracy–tractability trade-off.

We agree that the current method does not explicitly optimise an accuracy–tractability trade-off during training or model selection. The models are determined heuristically based on the MSE metrics. The CPWL-KAN hidden width is selected by cross-validation, while interval counts are chosen near the MSE elbow to avoid increasing formulation size for only marginal accuracy gains (Appendix B.2–B.3).

Our objective is to investigate how these two different neural networks (and their variants) perform when used in the context of deterministic global optimisation. And we use the same training methodology for all the networks and use comparable network sizes. Our focus is thus on how CPWL parameterisations and exact formulations affect predictive and computational performance under this training process. Specifically, we assess whether the use of KAN with univariate edge functions can reduce downstream optimisation costs while maintaining comparable or lower prediction errors than the ReLU-ANN baselines. Tractability is addressed through the representation, encoding (Proposition 3.5) and breakpoint pruning (Remark 3.4).

Accordingly, the reported “balance” describes empirical performance across the cases and it is not automatically optimised by the training. Adding tractability into training is a complementary direction, rather than the contribution claimed here. We have clarified the distinct scope and contribution in Introduction (Page 2), and the corresponding statements in the results.

2. Relation to ReLU-ANN formulations.

We agree that additional machinery, such as bound tightening, convex-hull and partition-based formulations, can improve tractability of ReLU-ANNs, although it usually comes with increased model complexity and preprocessing effort (Tsay et al., 2021). Several of these techniques would also likely improve the tractability of the CPWL-KANs and this would be an interesting direction for future work. We are focused here on understanding the inherent strengths and weaknesses of these CPWL-KANs and ReLU-ANNs. A remaining challenge in this area is to capture dependencies among the inputs to the ReLU activations when controlling formulation size and preprocessing effort, which has already been 4discussed in the Introduction. We have added a more precise explanation, on Page 2.

CPWL-KANs enable the grouping of multiple ReLU terms that share a single scalar input within each edge function – this grouping is not possible in general for a ReLU-ANN as the ReLU terms in any given layer will depend on multiple inputs (the affine layer being applied prior to the ReLU activation). Over a common input domain, the separate encoding in Proposition 3.5 already gives the convex hull of each individual scalar ReLU term. Joint encoding captures their dependence through the complete edge graph, providing a relaxation that is no weaker and often much tighter for wide networks. The potential improvement of CPWL-KANs then comes from retaining the joint dependence, rather than merely tightening individual bounds.

We focus on how CPWL-KAN parameterisations expose this structure and how the corresponding formulations exploit it. Additional bound tightening can also be combined with our formulations to enhance their tractability. We have clarified the advantages of CPWL-KANs at the end of Section 3 (Page 6).

3. Structural contribution of CPWL-KANs.

We examine the structural contribution of CPWL-KANs through complementary comparisons. Previous work (Schoots et al., 2025) has established exact conversions between CPWL-KANs and ReLU-ANNs. While converting a CPWL-KAN into a ReLU-ANN preserves groups of ReLU terms sharing one scalar input, the reverse conversion retains the multivariate affine aggregation preceding each nonlinear activation. Thus, functional equivalence alone is not automatically advantageous in terms of optimisation.

Under big-M encoding, both CPWL-KANs and ReLU-ANNs use interval arithmetic for bound propagation. Interval pruning is performed to both CPWL-KANs and ReLU-ANNs by dropping the always-inactive terms and simplifying always-active terms with affine expressions. Then with big-M encoding, the computational advantages compared to ReLU-ANNs in reducing runtimes, as reported in Table 14 in Appendix, are observed, revealing the structural contribution of CPWL-KANs.

Leveraging the CPWL-KAN structure, multiple ReLU terms sharing a common scalar input on a given edge can be grouped and jointly relaxed. Conventional MLP neurons generally depend on different affine input combinations, so the same joint encoding and the associated bound propagation are not automatically available. With this joint encoding, we compare the potential relaxation improvements as well as computational benefits derived from the CPWL edge function.

4. Optimisation contribution of CPWL-KAN variants.

As mentioned in Appendix B, all CPWL-KAN variants are initialised with the same strategy to obtain a uniform distribution of breakpoints for each layer. Then, with different parameters fixed, the CPWL-KANs undergo different training trajectories and result in different surrogates.

KAN-TIFO: Even though the CPWL edge functions are convex in KAN-TIFO, the network itself may be nonconvex since the composition of convex functions needs not be convex. Interval pruning can cut the number of breakpoints by about half in either layers (Table 4), which leads to improvement in SOS2 encoding without changing convexity in the edge functions, while pruning remains limited for the separate big-M encoding (Figure 2). The embedded convexity in KAN-TIFO (owing to nonnegative outer weights) ensures that the separate big-M and joint SOS2 encodings yield the same convex relaxation over the edge function. This could avoid the additional relaxation loss that can arise from separately encoding nonconvex CPWL edges. And convexity reduces the required runtimes compared to the other variants.

KAN-TITO: Proposition 3.1 shows that TITO and FITO are equally expressive. The additional inner weights in TITO change the optimisation parametrisation, and also affect the gradient updates during training. This can lead to different training trajectories and learned CPWL representations. Compared with KAN-FITO, KAN-TITO can more easily move the breakpoints as any changes in both inner and outer weights during training now result in new CPWL functions. This can help the networks achieve higher predictive accuracy and more compact representation after pruning relative to KAN-FITO, as illustrated in Section 4.1 (Figure 1) and Section 4.2 (Table 4), respectively.

We have clarified these mechanisms and their supporting evidence in Sections 4.1 and 4.2, with the initialisation procedure detailed in Appendix B.2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.