acceptodds
Withdrawn from ICLR 2027

Gradient-based training of probabilistic neural network with class-feature smoothing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

This paper proposes a gradient-based training procedure for probabilistic neural network with class- and feature-dependent smoothing parameters. The parameters are updated using analytically derived gradients defined for the normalized class outputs. Three initialization strategies are considered: class-related feature variance, median pairwise distance, and interquartile range. The method is evaluated on 20 classification datasets using repeated stratified 10-fold cross-validation. The variance-based initialization obtains the best mean rank, although the relative performance of the initialization methods is dataset-dependent. Statistical tests do not indicate significant differences between the considered initialization strategies.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
probabilistic neural network, smoothing parameter, gradient-based training

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PJLQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a training procedure for probabilistic neural networks (PNNs) with class- and feature-specific smoothing parameters. It derives updates based on quadratic errors of normalized class outputs, applies mini-batch gradient descent, and compares three bandwidth initialization strategies on 20 classification datasets.

Critical strengths

None

Critical weaknesses

W1. PNNs are old and well-studied models that do not constitute a mainstream practical choice.

W2. The paper lacks novelty. The proposed parameterization already exists as well as gradient-based training approaches.

W3. There is no comparison to other PNN training methods.

Questions for authors

Q1. It is typical to consider cross-entropy as a loss function in classification problems. In this work the squared error is used as a loss function. The authors should justify their choice and present experimental results using cross-entropy loss.

Q2. What precisely distinguishes the proposed method from existing gradient-based PNN training? Why comparison with such approaches is not included in the experiments?

Reviewer_SiE3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a gradient-based training procedure for Probabilistic Neural Network (PNN) with class-dependent and feature-dependent smoothing parameters. The parameters are updated using analytically derived gradients defined for the normalized class outputs. Three initialization strategies are considered: class-related feature variance, median pairwise distance, and interquartile range.

The paper is interesting and well written. However, I not figure out the contributions. Using a gradient descent with appropriate initializations is not novel.

Critical strengths

The paper is very clear to read.

The numerical experiments exploit a lot of datasets.

Using a statistical test to measure the differences due to the initializations is a relevant idea.

Critical weaknesses

The contributions of the paper are not obvious. In particular, the authors should better explain the limitations of existing training methods and clarify how their proposed approach addresses these limitations.

The paper lacks of theoretical studies. The algorithm is well described together with its initializations but there are no theoretical analyses.

Experiments are not convincing. The comparison with published PNN results is limited to the appendix. Since the motivation of the work is not clear enough, the comparison with the state of the art should be more elaborated.

The paper should also compare the proposed approach to more recent trends for probability density estimate like Neural Autoregressive Distribution Estimation or Normalizing Flows.

Questions for authors

Clarification requests:

  • Why do the authors think that the algorithm initialization is the main problem? Modifying the whole training procedure could be a better approach, couldn't it?

  • Why do the authors choose PNN over other neural network architectures designed for probability density estimation, such as Neural Autoregressive Distribution Estimation and many other ones ?

Reviewer_9b3BOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  1. The paper proposes a method for learning class- and feature-specific smoothing parameters of the probabilistic neural network (PNN) using analytically derived gradients.
  2. Three initialization strategies for the smoothing-parameter matrix Σ\Sigma are presented: class-related feature variance (V), median pairwise distance (MP), and interquartile range (Q).
  3. The method is evaluated on 20 standard UCI-type machine learning datasets using repeated stratified 10-fold cross-validation. V is reported to obtain the best mean rank, but the statistical tests do not indicate significant differences among the initialization strategies.
Critical strengths
  1. The gradient derivation is presented explicitly in Appendix A, which clarifies the learning procedure.
  2. The performance of V, MP, and Q and the epoch-wise learning behavior are reported in a unified format across all datasets, which helps illustrate how the proposed method behaves during training.
  3. Section 2 provides a clear overview of the historical development of PNN training methods.
Critical weaknesses
  1. Limited contribution. Despite the explicit derivation and experiments on 20 datasets, the paper concludes only that performance varies across datasets depending on the initialization strategy, and the reported differences are not statistically significant (Section 5.3). This conclusion is largely expected and does not constitute a substantial methodological contribution.

  2. Insufficient justification of the three initialization strategies. At the end of Section 3.3, the authors state that the initialization strategies serve the subsequent gradient-based optimization of Σ\Sigma. However, the paper does not explain why these three particular strategies were selected, nor why they are appropriate for this objective.

  3. Unclear validity of the experimental protocol and results.

    • Eqs. (18)–(20) appear to select the optimal epoch based on the error of the held-out fold, i.e., the test set. This constitutes test-set model selection and is not appropriate from a standard machine learning perspective.
    • A recent related study (Kowalski et al., 2025) appears to follow a similar protocol. However, earlier PNN studies (Georgiou et al., 2006; 2008) select parameters on a separate validation set and evaluate on an independent test set. The protocol is therefore not a settled convention in this literature.
    • Moreover, for several datasets the CV error increases as training proceeds (e.g., BT, DBC, E, IO, L, PID, SE in Appendix B). Combined with the lack of statistical significance, this makes it unclear whether the proposed gradient-based training under the three initialization strategies works as intended.
Questions for authors
  1. What is the difference between the CV error of the initialized but untrained PNN and the error after training? Instead of reporting only the minimum error, please report the CV error at fixed epochs (e.g., 0, 50, 100, 150, and 200).

  2. What is the rationale for choosing these three initialization strategies? Many other data-dependent initialization schemes exist. Why do the authors consider these three to be suitable for the gradient-based optimization of Σ\Sigma?

  3. Prior studies (e.g., Kusy & Zajdel, 2015; Kowalski et al., 2025) compare against a range of existing methods. The authors state that such comparisons were omitted due to differences in evaluation protocols. However, initialization is a component that can be attached to various PNN training methods. Applying V, MP, and Q to the methods listed in Table 2 (Appendix C) under an identical protocol would show how the proposed initialization strategies behave across PNN variants. What differences arise when the proposed initialization is combined with these methods?

  4. Why do the errors on CH (MP) and HV remain close to chance level, with nearly flat trajectories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.