- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Traditional neural networks usually seek a single optimal set of parameters during training. However, when the number of parameters exceeds the number of training samples, this approach struggles to find a truly optimal solution. This paper proposes a machine learning framework inspired by statistical physics (SML). Instead of seeking a single set of parameters, it makes predictions by integrating over all possible parameters to compute expected values. By maximum the entropy in the parameter spaces, a new framework called SML is proposed. A concrete formulation of this method (SML-PN) is implemented on power-law networks (PN). It is experimentally demonstrated that SML-PN has strong resistance to label noise, useless features, and data contamination.
Critical strengths
-
The method is highly suitable for high-dimensional and small-sample problems. When training samples are very few, SML-PN performs better than most traditional models and can predict results more stably.
-
This method leverages the average of probability distributions, so when facing noisy data or incorrect labels, it can remain stable without additional complex denoising steps.
Critical weaknesses
-
Currently, this method has only achieved a closed-form solution on power-law networks (PN). If it is to be applied to other deeper and more complex neural networks, the mathematical integration seems to be difficult, limiting the scope of this method.
-
When the importance of different features in the data differs greatly (as in some tests), SML-PN may perform worse than some tree models that can automatically adjust weights (such as XGBoost).
Questions for authors
-
In the process of solving equations (11) and (12), the authors introduced some additional optimization variables (such as parameter bounds, scaling factors, and intercepts) and jointly tuned them using the Adam optimizer. Do these adjustments weaken the method's claim of "not requiring hyperparameter search"?
-
For some real datasets with very complex features or highly uneven distributions, will this simple product network structure limit the model's ability to fit complex patterns? What's the approximation capacity of PN compared to modern NN and transformer? Does SML can be extended to modern NN architectures?
-
All baselines are classical ML models, can you give insights on how it contribute to modern ML community? e.g. are the highest-probability parameters computed by this method can be used as an initialization for ordinary deep learning models to speedup training?
6 comments
(2/4): Weakness 2
Weakness 2 (uneven feature importance). We fully agree that this is where tree-based models have a structural advantage. Tree ensembles select features by construction, and TabPFN has learned to do so through pretraining, so they cope well when a few inputs carry most of the signal, as in Griewank. SML can potentially do similar things, but in the present work (the first investigation on SML), it does not change the architecture it is applied to, only the way its parameters are determined, and SML-PN, like ridge-type estimators, spreads its weight over all inputs. This shows mainly on wide tables: on OpenML, SML-PN ranks first for l ≤ 5 and second for 6 ≤ l ≤ 10, and tree ensembles and TabPFN gain ground mainly for l > 20 (Table S26). It is not an effect of dimension as such, since on VLSE, where all coordinates carry signal, SML-PN ranks first at every dimension up to l = 500 (Table S19). The revised Discussion now clarifies this limitation (a limitation to the specific formulation of PN, not to the general concept of SML) and names input-specific reference measures as a way to encode relevance (Sec. 7); an architecture that favors sparse relevance is another natural extension.
Changes in the manuscript: Sec. 5.3; Sec. 7, third paragraph.
(3/4): Question 1
Question 1 (trained bounds, scale and intercept). In our view, they do not weaken the claim, but our wording should indeed have been clearer. The bounds, the scale T and the intercept b are fitted on the training data together with the Lagrange multipliers βⱼ: they are fitted quantities of the realization, not hyperparameters, and no setting is searched or selected per dataset. The validation set is used only to choose the stopping point, as with early stopping in conventional networks, and every baseline receives the same validation rows. All other settings (Table S2) are fixed once and used unchanged in every evaluation cell of SML-PN; the solver test of SM Sec. 3.4 changes only the setting under test. The revised manuscript now clarifies that "no per-dataset hyperparameter search" (Reproducibility statement; likewise Sec. 5), states that the bounds, the intercept and the scale factor are fitted together with βⱼ (Sec. 4), and explains how the fixed settings were chosen directly after Table S2 (SM Sec. 2.2).
Changes in the manuscript: Sec. 4, last paragraph; Sec. 5; Reproducibility statement; paragraph after Table S2 (SM Sec. 2.2).
(1/4): Weakness 1
We thank the reviewer for the detailed assessment and encouragement. The questions on the scope of the closed form, on uneven feature importance and on the role of the fitted quantities greatly helped us sharpen a number of statements in the paper.
Weakness 1 (closed form only for PN). Section 4 of this paper is indeed specific to PN (Section 3, the general formula of SML, is applicable to any NN whose parameters range over a bounded domain). We see PN as the first realization of a general idea rather than as the limit of SML. The idea of the paper comes from statistical physics: rather than searching for one optimal parameter configuration {wᵢ}ₒ, SML predicts with the ensemble average over all configurations weighted by the maximum-entropy distribution that the training data constrain, based on the principle of maximum entropy. For a given reference box, the probability distribution ρ is determined by n Lagrange multipliers βⱼ (j=1,2…n), each corresponding to one training sample; that is, ρ explicitly contains information of all training samples. The number of unknown parameters (βⱼ) is the same as the number of training samples n (i.e., the amount of useful information), regardless of the size of the network. What depends on the NN architecture is how the expectations are evaluated.
Our goal is to show the feasibility of SML, rather than to promote a particular NN architecture. PN was the first network we tried, chosen because its ensemble average remains simple after integration. We initially planned to quickly move on to other more complicated NNs, but were surprised by the (relatively) impressive performance of PN (with its very simple configuration and very small size); so we studied PN in depth and compared it with a broad set of tuned models. In the conventional methods, the same function class (e.g., tuned ridge and LLR) is clearly weaker.
Beyond PN, we have identified two families of NNs for which the ensemble averages also have analytical forms (through integral wavelet transforms), including a few relatively commonly studied NNs. More general NNs may need Monte Carlo sampling, yet through a hybrid method (coupled with integral wavelet transforms) the required computational overhead may be drastically reduced (compared to fully random sampling). In fact, we suspect that the hybrid approach may not be necessary, as the analytical methods of the two NN families have shown encouraging preliminary results (thanks to the recent progress of AI tools). We are currently working in this area (not only on PN but also on the new NN families) and hope to obtain presentable results by the end of this academic year (next summer). The results will be published separately. We hope that this paper (if accepted) will attract broader interest and encourage other researchers to explore this concept in machine learning.
Changes in the manuscript: Sec. 4; Sec. 7, last paragraph.