- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The present manuscript investigates the use of certain Kolmogorov-Arnold networks (CPWL-KANs) as surrogate models for global optimization goals. The authors introduce KANs with different expressivity and develop exact mixed-integer formulations. Through theoretical analysis and experiments on benchmarks and an engineering problem (hydrogen production), they demonstrate that CPWL-KANs can achieve competitive predictive accuracy while also reducing downstream optimization runtimes compared to conventional MLPs. The central findings are particularly interesting: (1) the internal representation of a learned function, rather than predictive accuracy alone, matters for global optimization performance, as demonstrated with CPWL-KANs; and (2) the surrogate models with the highest predictive accuracy are not necessarily the most computationally efficient to optimize.
Critical strengths
-
The paper addresses an important but relatively underexplored aspect of surrogate modeling: how neural-network representations and their parameterizations affect downstream global optimization. The connection between representational structure, relaxation strength, and solver efficiency is compelling, particularly for scientific/engineering applications.
-
The analysis of expressivity and relaxation strength provides a clear explanation for the observed computational improvements. In particular, exploiting the univariate piecewise-linear structure of KAN edges through joint SOS2 formulations is a smart approach, and the experiments show meaningful improvements in optimization runtime.
Critical weaknesses
If I understand correctly, the architectural advantage of KANs is not yet fully isolated. The main computational improvements appear to arise from exploiting univariate piecewise-linear representations and stronger SOS2 relaxations, rather than from a fundamental advantage of KANs themselves. Since related CPWL-KAN and ReLU MLP representations can be mathematically equivalent, it remains unclear how much of the observed speedup is attributable specifically to the KAN architecture versus the choice of optimization formulation.
Questions for authors
- Given the known representational connections between CPWL-KANs and ReLU MLPs, could the authors clarify whether comparable gains might be obtained by applying stronger piecewise-linear formulations to equivalent MLP representations?
- How do the improvements scale with problem complexity? In particular, would the authors expect similar advantages for higher-dimensional or deeper networks that may be required for more complex problems?
2 comments
I thank the authors for clarifying how the CPWL-KAN parameterization exposes shared-input structure and for adding the scaling experiments. The new results address my scalability question: the gains persist on the higher-dimensional benchmarks tested, while the depth experiments show where those gains become less reliable.
My main concern is partly addressed. The rebuttal explains why the parameterization makes the shared-input structure easy to exploit, but it remains unclear whether an exactly equivalent ReLU network, formulated to exploit the same grouping, would achieve comparable runtimes. The results support the practical value of the parameterization and formulation together, but do not fully isolate their respective contributions. I am therefore keeping my original score of 3 (weak acceptance).
We appreciate the reviewer’s suggestions in enhancing the surrogate modelling ability of CPWL-KANs.
1. Relation to equivalent ReLU-ANN representations.
CPWL-KANs enable the grouping of multiple ReLU terms that share a single scalar input within each edge function – this grouping is not possible in general for a ReLU-ANN as the ReLU terms in any given layer will depend on multiple inputs (the affine layer being applied prior to the ReLU activation). Exact conversions preserve the underlying input dependencies: KAN-to-ANN conversions retain the shared-input grouping, whereas the reverse conversion retains the affine aggregation preceding each activation (Schoots et al., 2025). Our contribution concerns how CPWL-KAN parametrisations explicitly represent the univariate CPWL edge function structure and how the corresponding formulations exploit it.
With CPWL-KANs, the separate big-M relaxation already gives the convex hull of each individual ReLU term over a common scalar input domain. Joint encoding goes further by capturing their dependence through the complete CPWL edge graph, yielding a no-weaker relaxation (Proposition 3.5). Our formulations exploit this shared-input structure for joint convexification, bound propagation and breakpoint pruning. The computational results of CPWL-KANs over the corresponding ReLU_ANNs (Section 4.2; Appendix, Table 14) confirm that the observed improvements do not depend solely on the encoding approaches but from the unique network structure.
Although other stronger formulations for ReLU-ANNs, such as the convex-hull and partition-based formulations, may improve the optimisation performance of ReLU-ANNs, these typically come at the expense of additional auxiliary variables and preprocessing efforts. Furthermore, the dependence among inputs is inherent to ReLU-ANNs, so the unique advantages afforded by univariate CPWL edge function in CPWL-KANs does not readily translate to ReLU-ANNs in general. We added an explanation of this key structural difference at the end of Section 3 (Page 6) in the revision.
2. Scalability with input dimension and network depth.
We have added scaling experiments in Appendix C.3.5 (Tables 16–17), covering 4D, 8D, and 12D Ackley and Rosenbrock problems, and Hartmann-6D networks of increasing depth, with a 600 s time limit.
For Ackley-12D, KAN-TITO and KAN-TIFO solve all 128 instances under both encodings, with mean runtimes of 0.5–2.2 s, while ANN-TW and ANN-FW solve 2.3% and 26.6%, respectively. For Rosenbrock-12D, KAN-TIFO solves all instances under both encodings; KAN-TITO solves all with separate big-M and 82.0% with joint SOS2 encoding.
Increasing depth is more challenging as the network complexity increases significantly. At depth 3, KAN-TIFO with separate big-M solves all ten instances in 22.4 s on average, compared with 35.9 s for ANN-FW. At depth 4, KAN-TIFO solves 20% and 40% with joint SOS2 and separate big-M, respectively, while the other models solve none. Bound overestimation can accumulate across layers, weakening network relaxations despite strong individual edge formulations.
These results show that computational advantages persist in the higher-dimensional problems, while benefits for deeper networks depend more strongly on the parametrisation and encoding. We discuss these trends and limitations in the revised appendix.