acceptodds
Under review as a conference paper at ICLR 2027

Parameter-Efficient Learning with Hybrid Volterra Neural Networks

Abstract

Despite the flexibility of multi-layer perceptrons (MLPs) and convolutional neural networks (CNNs), they learn nonlinear frequency interactions indirectly through successive layers, which can increase parameter requirements. For non-polynomial pointwise activations such as ReLU in these networks, a globally exact finite-order polynomial representation is generally unavailable. To address this limitation, we propose the Hybrid Volterra Neural Network (HVNN), which assigns selected spectral components to an activation-free Volterra branch and sends the remaining spectrum to a neural branch. The Volterra branch directly parameterizes nonlinear responses through finite-order polynomial expansions in the frequency domain, making nonlinear frequency coupling explicit. We further propose spectral interaction sparsification to restrict computation to a budgeted set of valid sum- and difference-frequency pairs, thereby limiting evaluated nonlinear interactions. Thus, HVNN constrains model complexity, targeting improved predictive performance with a modest parameter count. HVNN-MLP and HVNN-CNN variants are evaluated on four nonlinear system-identification, three image-denoising and two image-classification datasets to demonstrate their ability across diverse predictive tasks and to achieve a favorable accuracy–parameter trade-off. Our HVNN-CNN reaches PSNR values of 30.7351 and 26.8766 on MNIST and Fashion-MNIST denoising with 65.6K parameters, compared with 24.1188 and 22.3718 with 79.0K parameters for CNN. The code is available at this link.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Volterra Series, Parameter-Efficient Learning, Nonlinear System Identification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_19XSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hybrid Volterra Neural Networks for parameter-efficient nonlinear prediction. The architecture retains a full-spectrum linear path, applies low-rank Volterra interactions to frequencies selected by training-input energy, and processes the complementary frequencies with a compact MLP or CNN. Sparse Pair limits the number of evaluated second-order sum- and difference-frequency interactions. Experiments on system identification, image denoising, and image classification examine the trade-offs among predictive accuracy, trainable parameters, and evaluated interaction counts.

Critical strengths
  1. Sparse Pair provides explicit control over the nonlinear interaction budget.
  2. Some reported results show a promising accuracy–model-size trade-off.
Critical weaknesses
  1. The parameter accounting is inconsistent with the stated model configuration. Table 2 reports 41,973 parameters for Sparse Pair and only 34,950 for HVNN-CNN, making both the parameter-efficiency comparison and the contribution of the complement difficult to interpret.

  2. The comparisons do not sufficiently establish the advantage of the proposed design. The main baselines are MLP, CNN, and KAN, without reported results for parameter-matched neural models or the time-domain and full-spectrum Volterra controls described in Appendix B. The standalone CNN also uses pooling whereas the complement CNN preserves positions, making the denoising gap difficult to attribute specifically to the proposed design.

Questions for authors
  1. Why does adding the CNN complement reduce the parameter count in Table 2? Are the feature widths, selected support size, ranks, active pair set, and task head identical between the two models? Please provide the corresponding configurations and module-level parameter counts.

  2. Were the time-domain and full-spectrum Volterra controls described in Appendix B evaluated? Can their results clarify the separate benefits of support selection, pair sparsification, and the neural complement?

Reviewer_efvyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a hybrid neural network architecture combining a sparse Volterra network with nonlinear neural networks, such as MLPs or CNNs. Because managing the high-order interaction terms in regular Volterra networks requires a substantial number of parameters, the authors impose sparsity to: (1) reduce the parameter space, (2) potentially improve generalization performance, and (3) provide better interpretability. The core idea is well-presented, and some experimental results show that the approach is promising. However, the current experiments do not clearly demonstrate that the performance improvements stem from the paper's proposed hybrid design.

Critical strengths
  • Strong empirical performance against baselines in some experiments.

  • The underlying concept is natural and well-motivated. Aside from the numerical experiments section, the paper is well-written and clearly conveys the main ideas.

Critical weaknesses
  • A primary limitation of this paper is that it combines several existing concepts into a single framework. Although it introduces some novel components, it remains unclear which specific component drives the performance gains.

  • Pairing a classical, interpretable model with a highly expressive nonlinear model is a well-established paradigm. Furthermore, both standard and sparse Volterra network architectures have been explored in prior work. Consequently, it is unclear whether the superior benchmark performance is genuinely due to the proposed hybrid framework, or simply stems from the underlying sparse Volterra network (even if the exact implementation differs slightly from prior work).

Questions for authors
  1. In Table 2, the standalone Sparse Pair Volterra model already achieves the best (or near-best) results without the hybrid architecture. The only case where the hybrid architecture yields a clear improvement is the Tank experiment. Is this interpretation correct, or am I missing something?

  2. Regarding Table 6, the paper states: "Within the ablation runs, ρ=0.01 reduces the evaluated Pair count from 11,935 to 120 on WH, while retaining or improving NMSE on Duffing, Van der Pol, and Tanks." This drastic reduction without performance degradation is quite surprising. Does this imply that the meaningful high-order interactions in Volterra neural networks are inherently very sparse for these tasks?

Reviewer_6S5POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Hybrid Volterra Neural Networks, combining a sparse frequency-domain Volterra model with an MLP or CNN operating on the complementary spectrum. Input-energy ranking selects frequencies and second-order interaction pairs, while low-rank factors control parameter storage. Experiments evaluate accuracy–parameter trade-offs on system identification, image denoising, and classification.

Critical strengths
  1. Explicit spectral interactions provide a sensible inductive bias for compact nonlinear models. The Sparse Pair results on Wiener–Hammerstein and denoising suggest that the structured representation has practical potential.
  2. The pair-budget experiment demonstrates a concrete trade-off: evaluated pairs decrease from 11,935 to 120 while retaining similar performance on several datasets. The paper also reports the accuracy penalty on WH and appropriately distinguishes interaction count from parameter storage.
Critical weaknesses

The parameter counts conflict with the stated architecture. Appendix B states that HVNN retains the corresponding Sparse Pair model’s width, support, ranks, pair set, and task head, then adds the neural branch and fusion parameters. Its parameter count should therefore increase. Yet Table 2 reports 34,950 parameters for HVNN-CNN versus 41,973 for Sparse Pair, and Table 4 reports 50,723 versus 61,906. The actual configurations or accounting must differ from the description. This directly undermines the central parameter-efficiency comparison and the interpretation of hybridization as a controlled addition.

The experiments support the spectral model more clearly than the hybrid routing. The CNN complement adds only about 0.04–0.05 dB on grayscale denoising, slightly worsens CIFAR-10 denoising, and produces a VdP improvement smaller than the reported run variation. Tanks is a meaningful exception, but the broader benefit remains unclear. Input-energy ranking also does not directly measure nonlinear response importance, and no fixed low-frequency routing control is provided. The limited visual benchmarks and pooled CNN comparator leave insufficient evidence that the proposed routing improves over simpler compact alternatives.

The visual evaluation remains largely at the proof-of-concept level. Classification is restricted to MNIST and Fashion-MNIST, while denoising adds only low-resolution CIFAR-10 under Gaussian corruption. Together with the shallow neural baselines, this provides insufficient evidence for a broadly useful, parameter-efficient architecture at the ICLR main-track level. At least one more demanding visual benchmark and competitive compact baselines are needed to establish the claimed accuracy–efficiency advantage.

Questions for authors

What configuration changes explain why HVNN-CNN is smaller than its supposedly unchanged Sparse Pair backbone? Please provide per-module parameter counts and reconcile Tables 2 and 4 with Appendix B. At matched parameter budgets, does hybrid routing outperform standalone Sparse Pair or a full-input neural complement? At matched support size, does Energy Top-K outperform a fixed low-frequency split and Random-K? How does the pooled CNN reconstruct full-resolution residuals? Do the denoising gains persist against a similarly sized resolution-preserving CNN, and what is the variability across runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.