acceptodds
Under review as a conference paper at ICLR 2027

A Statistical-Physics-Inspired Framework of Machine Learning

Abstract

While the statistical and probabilistic aspects of neural networks (NNs) have been extensively studied, conventional NN training typically seeks a single optimal configuration of internal parameters, which is fundamentally underdetermined when the number of parameters exceeds the number of training samples. In this paper, we investigate a framework of statistical-physics-inspired machine learning (SML). Predictions are obtained as expectation values by integrating over all possible parameter configurations, rather than relying on the optimal parameter configuration. Such an algorithm allows regular system-level behavior to emerge naturally, without being overwhelmed by the underlying irregular parameter-space landscape. The number of unknowns scales with the training-data size, not the NN size, making the approach attractive to the high-dimensional low-sample-size regime. Realized in closed form on a power-law network (PN), SML ranks first on standardized function benchmarks against 19 high-performance reference model variants, and outperforms most of them on 848 real-world datasets. SML-PN remains quite robust to label noise, irrelevant features, and contamination.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
ensemble average, expectation value, maximum entropy, tabular regression, small data, high-dimensional low-sample-size

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_azDAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Traditional neural networks usually seek a single optimal set of parameters during training. However, when the number of parameters exceeds the number of training samples, this approach struggles to find a truly optimal solution. This paper proposes a machine learning framework inspired by statistical physics (SML). Instead of seeking a single set of parameters, it makes predictions by integrating over all possible parameters to compute expected values. By maximum the entropy in the parameter spaces, a new framework called SML is proposed. A concrete formulation of this method (SML-PN) is implemented on power-law networks (PN). It is experimentally demonstrated that SML-PN has strong resistance to label noise, useless features, and data contamination.

Critical strengths
  1. The method is highly suitable for high-dimensional and small-sample problems. When training samples are very few, SML-PN performs better than most traditional models and can predict results more stably.

  2. This method leverages the average of probability distributions, so when facing noisy data or incorrect labels, it can remain stable without additional complex denoising steps.

Critical weaknesses
  1. Currently, this method has only achieved a closed-form solution on power-law networks (PN). If it is to be applied to other deeper and more complex neural networks, the mathematical integration seems to be difficult, limiting the scope of this method.

  2. When the importance of different features in the data differs greatly (as in some tests), SML-PN may perform worse than some tree models that can automatically adjust weights (such as XGBoost).

Questions for authors
  1. In the process of solving equations (11) and (12), the authors introduced some additional optimization variables (such as parameter bounds, scaling factors, and intercepts) and jointly tuned them using the Adam optimizer. Do these adjustments weaken the method's claim of "not requiring hyperparameter search"?

  2. For some real datasets with very complex features or highly uneven distributions, will this simple product network structure limit the model's ability to fit complex patterns? What's the approximation capacity of PN compared to modern NN and transformer? Does SML can be extended to modern NN architectures?

  3. All baselines are classical ML models, can you give insights on how it contribute to modern ML community? e.g. are the highest-probability parameters computed by this method can be used as an initialization for ordinary deep learning models to speedup training?

6 comments
AuthorsOpenReview ↗

(1/4): Weakness 1

We thank the reviewer for the detailed assessment and encouragement. The questions on the scope of the closed form, on uneven feature importance and on the role of the fitted quantities greatly helped us sharpen a number of statements in the paper.

Weakness 1 (closed form only for PN). Section 4 of this paper is indeed specific to PN (Section 3, the general formula of SML, is applicable to any NN whose parameters range over a bounded domain). We see PN as the first realization of a general idea rather than as the limit of SML. The idea of the paper comes from statistical physics: rather than searching for one optimal parameter configuration {wᵢ}ₒ, SML predicts with the ensemble average over all configurations weighted by the maximum-entropy distribution that the training data constrain, based on the principle of maximum entropy. For a given reference box, the probability distribution ρ is determined by n Lagrange multipliers βⱼ (j=1,2…n), each corresponding to one training sample; that is, ρ explicitly contains information of all training samples. The number of unknown parameters (βⱼ) is the same as the number of training samples n (i.e., the amount of useful information), regardless of the size of the network. What depends on the NN architecture is how the expectations are evaluated.

Our goal is to show the feasibility of SML, rather than to promote a particular NN architecture. PN was the first network we tried, chosen because its ensemble average remains simple after integration. We initially planned to quickly move on to other more complicated NNs, but were surprised by the (relatively) impressive performance of PN (with its very simple configuration and very small size); so we studied PN in depth and compared it with a broad set of tuned models. In the conventional methods, the same function class (e.g., tuned ridge and LLR) is clearly weaker.

Beyond PN, we have identified two families of NNs for which the ensemble averages also have analytical forms (through integral wavelet transforms), including a few relatively commonly studied NNs. More general NNs may need Monte Carlo sampling, yet through a hybrid method (coupled with integral wavelet transforms) the required computational overhead may be drastically reduced (compared to fully random sampling). In fact, we suspect that the hybrid approach may not be necessary, as the analytical methods of the two NN families have shown encouraging preliminary results (thanks to the recent progress of AI tools). We are currently working in this area (not only on PN but also on the new NN families) and hope to obtain presentable results by the end of this academic year (next summer). The results will be published separately. We hope that this paper (if accepted) will attract broader interest and encourage other researchers to explore this concept in machine learning.

Changes in the manuscript: Sec. 4; Sec. 7, last paragraph.

AuthorsOpenReview ↗

(2/4): Weakness 2

Weakness 2 (uneven feature importance). We fully agree that this is where tree-based models have a structural advantage. Tree ensembles select features by construction, and TabPFN has learned to do so through pretraining, so they cope well when a few inputs carry most of the signal, as in Griewank. SML can potentially do similar things, but in the present work (the first investigation on SML), it does not change the architecture it is applied to, only the way its parameters are determined, and SML-PN, like ridge-type estimators, spreads its weight over all inputs. This shows mainly on wide tables: on OpenML, SML-PN ranks first for l ≤ 5 and second for 6 ≤ l ≤ 10, and tree ensembles and TabPFN gain ground mainly for l > 20 (Table S26). It is not an effect of dimension as such, since on VLSE, where all coordinates carry signal, SML-PN ranks first at every dimension up to l = 500 (Table S19). The revised Discussion now clarifies this limitation (a limitation to the specific formulation of PN, not to the general concept of SML) and names input-specific reference measures as a way to encode relevance (Sec. 7); an architecture that favors sparse relevance is another natural extension.

Changes in the manuscript: Sec. 5.3; Sec. 7, third paragraph.

AuthorsOpenReview ↗

(3/4): Question 1

Question 1 (trained bounds, scale and intercept). In our view, they do not weaken the claim, but our wording should indeed have been clearer. The bounds, the scale T and the intercept b are fitted on the training data together with the Lagrange multipliers βⱼ: they are fitted quantities of the realization, not hyperparameters, and no setting is searched or selected per dataset. The validation set is used only to choose the stopping point, as with early stopping in conventional networks, and every baseline receives the same validation rows. All other settings (Table S2) are fixed once and used unchanged in every evaluation cell of SML-PN; the solver test of SM Sec. 3.4 changes only the setting under test. The revised manuscript now clarifies that "no per-dataset hyperparameter search" (Reproducibility statement; likewise Sec. 5), states that the bounds, the intercept and the scale factor are fitted together with βⱼ (Sec. 4), and explains how the fixed settings were chosen directly after Table S2 (SM Sec. 2.2).

Changes in the manuscript: Sec. 4, last paragraph; Sec. 5; Reproducibility statement; paragraph after Table S2 (SM Sec. 2.2).

Reviewer_QEQXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a statistical-physics-inspired machine learning (SML) framework that replaces conventional optimization of a single parameter configuration with ensemble averaging over a maximum-entropy parameter distribution constrained by training data. The framework is designed for high-dimensional, low-sample-size (HDLSS) learning, where the number of model parameters can greatly exceed the number of training samples. The authors further introduce a Power-law Network (PN) that enables analytical evaluation of the parameter-space integrals, making the proposed framework computationally tractable.

The paper provides extensive experimental validation, including synthetic datasets, standardized benchmark functions, and 848 real-world regression datasets. The results demonstrate strong performance in HDLSS settings, outperforming 19 reference model variants on most learnable benchmark functions and achieving competitive results against established methods such as XGBoost and CatBoost on real-world datasets. Additional experiments demonstrate robustness to label noise, irrelevant features, and data contamination. Overall, the work presents an interesting integration of statistical physics principles with machine learning, supported by comprehensive theoretical derivations and experimental evaluations.

Critical strengths
  1. Novel integration of statistical physics principles with machine learning. The paper presents an interesting framework that incorporates the maximum entropy principle and ensemble averaging into machine learning, replacing the optimization of a single parameter configuration with predictions based on a probability distribution over parameter space. The analytical realization through a Power-law Network further connects the theoretical framework to a practical learning algorithm. This approach provides an alternative perspective on learning in high-dimensional, low-sample-size settings and is a key strength supporting my positive assessment.

  2. Comprehensive experimental validation and strong empirical performance. The authors conduct extensive experiments on synthetic data, standardized benchmark functions, and 848 real-world regression datasets, comparing against a broad range of established methods. The proposed SML-PN achieves strong results, particularly in data-scarce settings, and demonstrates competitive performance on real-world datasets. Additional robustness experiments involving label noise, irrelevant features, and data contamination further strengthen the empirical evidence. The breadth of the evaluation and the consistently competitive results are important factors supporting my recommendation for acceptance.

Critical weaknesses
  1. Limited generalizability to more complex neural network architectures. Although the proposed SML framework is theoretically applicable to general neural networks, its practical implementation relies on the special mathematical structure of the Power-law Network, which enables analytical evaluation of parameter-space integrals. For more complex architectures, such as MLPs or Transformers, nonlinear activations and parameter coupling make these integrals difficult to evaluate analytically. Numerical approximations may introduce substantial computational overhead during training and inference. Consequently, the practical applicability and computational efficiency of the proposed framework beyond PN remain unclear, limiting the generality of the contribution.

  2. Insufficient analysis of training efficiency and stability under challenging noise conditions. Although the paper includes robustness experiments involving noisy inputs, corrupted labels, and irrelevant features, it does not sufficiently investigate how increasing noise levels affect optimization convergence, training stability, and computational cost. Since the proposed method relies on training-data constraints to determine the maximum-entropy parameter distribution, it remains unclear whether strongly noisy or inconsistent observations could make the numerical optimization more difficult. A more detailed analysis of convergence behavior and training efficiency under challenging noise conditions would strengthen the practical applicability of the method.

Questions for authors
  1. Generalizability to more complex neural network architectures. The proposed SML framework is theoretically applicable to general neural networks, but its current implementation relies on the special structure of the Power-law Network to analytically evaluate parameter-space integrals. Could the authors clarify how this framework could be practically extended to more complex architectures, such as MLPs or Transformers, where nonlinear activations and parameter coupling make analytical integration difficult? Are there any feasible approaches to approximate these integrals while preserving the advantages of SML?

  2. Computational efficiency during training and inference. For architectures where analytical integration is not possible, numerical methods such as Monte Carlo sampling may be required, potentially introducing substantial computational overhead. Could the authors clarify the expected computational complexity of SML in such cases? Additionally, could the authors provide more details on the training and inference efficiency of the current SML-PN implementation compared with conventional baselines?

  3. Training stability under strong noise and data contamination. The paper demonstrates robustness to several types of noise and data contamination. However, it remains unclear how these perturbations affect the optimization process itself. Could the authors clarify whether increasing noise levels lead to slower convergence, greater optimization instability, or increased training costs? In particular, how does the method behave when noisy training labels introduce conflicting or difficult-to-satisfy constraints on the maximum-entropy distribution?

3 comments
AuthorsOpenReview ↗

(1/2): Weakness 1, Questions 1 and 2

We thank the reviewer for the detailed assessment and the encouragement. The reviewer's questions on scope, computational cost and behavior under noise have helped us to greatly improve the manuscript.

Weakness 1 and Question 1 (MLPs and Transformers). The idea of the paper comes from statistical physics: rather than searching for one optimal parameter configuration {wᵢ}ₒ, SML predicts with the ensemble average over all configurations weighted by the maximum-entropy distribution that the training data constrain, based on the principle of maximum entropy. For a given reference box, the probability distribution ρ is characterized by n Lagrange multipliers βⱼ (j=1,2…n), each corresponding to one training sample; that is, ρ explicitly contains information of all training samples.

Our aim is to show the feasibility of SML, rather than to promote a particular NN architecture. PN was the first network we tried, chosen because its ensemble average remains simple after integration. We initially planned to quickly move on to other more complicated NNs, but were surprised by the (relatively) impressive performance of PN (with its very simple configuration and very small size); so we studied PN in depth and compared it with a broad set of tuned models. In the conventional methods, the same function class (tuned ridge and LLR) is clearly weaker.

The formulation of Sec. 3 does not depend on the NN architecture: for any model whose parameters range over a bounded domain, the maximum-entropy distribution has one Lagrange multiplier (βⱼ) per training sample; the number of unknown parameters (βⱼ) is the same as the number of training samples n (i.e., the amount of useful information), regardless of the size of the network.

Beyond PN, we have identified two other NN families, including a few relatively commonly studied NNs, for which the ensemble averages are in analytical forms (through integral wavelet transform); for more general NN architectures, the expectation values indeed need to be computed numerically. This work is ongoing, its preliminary results are encouraging, and the results will be published elsewhere. The revised Discussion states this extension as the main direction for future work (Sec. 7). We hope that this paper (if accepted) will attract broader interest and encourage other researchers to explore this concept in machine learning.

Changes in the manuscript: Sec. 4; Sec. 7, last paragraph.

Question 2 (complexity and runtime). Indeed, for NN architectures where analytical integration is not possible, numerical methods such as Monte Carlo sampling is required, potentially introducing substantial computational overhead. Therefore, our ongoing research prioritizes NNs for which closed-formed ensemble averages are possible (through integral wavelet transforms). In addition to PN, we have identified two families of NNs that satisfy this requirement, including a few relatively commonly studied NNs. More general NNs may need Monte Carlo sampling, yet through a hybrid method (coupled with integral wavelet transforms) the computational overhead may be drastically reduced. In fact, we suspect that the hybrid approach may not be necessary, as the analytical methods of the two new NN families have shown encouraging preliminary results (thanks to the recent progress of AI tools). We are currently working in this area and hope to obtain presentable results by the end of this academic year (next summer). The results will be published separately.

For SML-PN itself, one epoch costs O(nl) time and memory, the fitted quantities are n + 2l + 2, and a prediction costs O(l). This is now stated in SM Sec. 3.4.

Changes in the manuscript: SM Sec. 3.4 (computational cost).

AuthorsOpenReview ↗

(2/2): Weakness 2 and Question 3

Weakness 2 and Question 3 (efficiency and stability under noise; conflicting constraints). The optimization does not become noticeably slower as the data get noisier. In the robustness experiments of Sec. 6, the training time per run actually decreases as the input noise grows (by ~5% at σ = 0.15) and as the label contamination grows (by ~15% at 50% replaced targets); on average over all perturbation levels, it decreases by 2.5% under input noise and by 9.5% under replaced targets (SM Sec. 7), and the run-to-run spread grows with the perturbation but stays moderate: the interquartile range of test R² over the ten repetitions of a setting is 0.12 at σ = 0.15 and 0.19 at 50% replaced targets (ribbons of Fig. 4a–c). With noisy targets, the constraints of Eq. (11) could still be met exactly (but by fitting the noise) and conflicting targets (or n > l) would require a new formulation for exact constraints. In the present work, SML-PN minimizes their least-squares residual (Eq. S1), which remains well defined in either case, and the retained state is chosen on the validation set. The behavior under contamination shows the effect directly. With 50% of the training targets replaced by outliers, the test R² on clean data stays at 0.80 (median over the four functions and both training sizes): the retained state does not follow the corrupted targets. The revised main text now clarifies that training time does not grow under noise or contamination (Sec. 6), and the new SM Sec. 7 (Table S29) reports test R² at every perturbation level.

To answer this question more directly, we ran two further tests on the same four functions (l = 100, n = 50 and 100, ten repetitions), together with the 18 reference model variants of Sec. 5 (all except NNₘ). First, Gaussian noise was added to the training and validation targets, with a standard deviation up to that of the targets themselves. Second, to create conflicting constraints, up to all training inputs were duplicated with two contradictory targets, two standard deviations of the targets above and below the true value. In both tests SML-PN had no numerical failure, and its training time changed only by a few percent (on average +3% and +4%, paired as in SM Sec. 7). With label noise as large as the signal, the training R² of SML-PN falls to 0.51, i.e., it does not follow the noise, while its median test R² decreases only from 0.89 to 0.81; it has the best mean rank of the 19 variants at every noise level, and its spread over repetitions stays small (interquartile range of test R² at most 0.09). With all inputs duplicated with contradictory targets, where the constraints can not be met at all, the training R² falls to 0.19 but the test R² stays at 0.93: the solution settles at the average of the contradictory targets. Again SML-PN has the best mean rank at every level, with an interquartile range of test R² of at most 0.015. For very strong input noise (σ = 0.3 and 0.5), its test R² falls to 0.37 and 0.09, but it still has the best mean rank (for TabPFN, the runs of this test could be completed in time only on one of the four functions).

Changes in the manuscript: Sec. 6 (last sentence of the noise paragraph); SM Sec. 7 (training time) and Table S29 (test R²).

We thank the reviewer again for the encouraging assessment and the questions, which led to the new tests above. We hope that our replies address the reviewer's questions; if so, we would be grateful if the reviewer would consider raising the rating. We are happy to answer any further questions during the discussion period.

Thank you for your response and the additional clarifications. They have addressed most of my concerns, and I will maintain my original score.

Reviewer_yiSkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes statistical-physics-inspired machine learning (SML). Instead of fitting one set of parameters, it finds the maximum-entropy distribution over parameters such that the expected model output matches each training target, and it predicts with the expected output. This gives one Lagrange multiplier per training sample, so the number of unknowns grows with the data rather than with the model. The only model used is a "power-law network" (PN), whose output is linear in its parameters, so the integrals have a closed form. In the experiments there is one parameter per input feature and the inputs are scaled to [0.9,1.2][0.9, 1.2], so the model is effectively a linear regression on nearly linear log-features. The multipliers, the parameter bounds, a scale and an intercept are fitted with Adam on the training error, with early stopping on a validation set and a schedule that starts with all parameters tied together. The experiments use data generated by the PN itself, 13 standard test functions (VLSE) on [0.1,1]l[0.1, 1]^l, and 848 OpenML regression datasets cut down to at most as many training rows as features, against 19 baseline variants.

Critical strengths
  • The idea is clean and the method is simple and cheap. It needs no per-dataset tuning, yet on OpenML it is statistically tied with tuned CatBoost and ahead of tuned random forests, SVR and TabPFN, especially when training data is scarcest.
  • The authors are open about where the method fails, including Griewank, the excluded test functions and wide real-world tables, and the Griewank control experiment helps explain why.
  • The evaluation is extensive. It covers 848 OpenML datasets and 1.46 million training runs, the baselines are tuned in every run with search spaces taken from prior work, and the dataset lists are pinned so the corpus can be reproduced.
Critical weaknesses

I will prelude these comments by stating that I'm open to updating my scores if the authors address these weaknesses. I'd request special emphasis on the comparison with previous work, better baselines, and a more careful theoretical analysis.

1. The framework is only shown for a linear model, so its claims about neural networks are untested.

  • The paper is motivated by neural networks with more parameters than samples and irregular loss landscapes. But the only model it uses has an output that is linear in its parameters, with one parameter per feature.
  • The inputs are scaled to [0.9,1.2][0.9, 1.2], where log⁡x\log x is almost linear in xx. So in practice SML-PN is a linear regression on scaled features, with an unusual way of choosing the coefficients. The nearly identical results for ridge and log-linear ridge in Tables S14–S15 confirm this.
  • The closed form only exists because the output is linear in the parameters. The paper says extending SML to deeper networks is feasible when the integrals are tractable, but for networks that are nonlinear in their parameters they aren't. Those would need sampling or variational approximations, as in Bayesian neural networks.

2. The benchmark wins are likely explained by a hidden bias toward equal weights.

  • All coefficients start from the same box, centred at −1.2-1.2. Training starts with one shared transform for all parameters, then frees them in contiguous chunks of feature indices. Early stopping on validation loss likely keeps the solution close to this tied start when there are fewer samples than features. Together, this strongly favours solutions where every feature gets roughly the same weight.
  • The benchmarks reward exactly that. The synthetic targets use weights drawn from N(−2,1)\mathcal N(−2, 1), so all coefficients have the same sign and a similar size. On [0.1,1]l[0.1, 1]^l, five of the nine learnable VLSE functions treat all coordinates alike, and three of the others weight them smoothly by index.
  • The one learnable function where SML-PN loses is Griewank, where a few coordinates carry most of the signal. The authors' own control shows that making the coordinates interchangeable brings R2R^2 back to about 0.950.95, and they write that "Griewank penalizes the property of SML-PN that is beneficial elsewhere".
  • I checked this with a quick reimplementation of the VLSE protocol, using the same functions, domain, dimensions, budgets and 30/30 validation/test splits. A two-parameter baseline, y=a+b⋅∑klog⁡xky = a + b\cdot\sum_k \log x_k fitted by least squares gave similar R2R^2 values.
  • On the PN-generated data, the same baseline reaches R2R^2 of about 0.80.8 from n=10−20n = 10-20 in most of the data pools I drew, close to SML-PN's reported 0.76–0.880.76–0.88.
  • If this holds up, the gains come from the solver's bias and the choice of benchmarks, not from maximum entropy. That wouldn't be surprising, since equal-weight linear models are known to be hard to beat with small samples (Dawes, 1979). The paper attributes the gains to "the algorithm rather than the functional form", but the only comparison is with ridge, which shrinks toward zero rather than toward equal weights.

3. On real data, the method doesn't beat the strongest baselines, and the protocol is unusual.

  • On OpenML, SML-PN ranks behind tuned XGBoost and is statistically tied with tuned CatBoost (Table S19). Calling it the "best fixed-configuration model" compares it with library defaults, which were not designed for training sets of 2 to 50 rows.
  • Its rank gets steadily worse as the number of features grows, from 4.0 for at most 5 features to 12.3 for more than 100 (Table S21). That cuts against the high-dimensional, small-sample framing. It does best when the training set has at most five rows.
  • Even tuned XGBoost gets R2>0R^2 > 0 in only 53%53\% of cells, and SML-PN in 45%45\%, so many cells carry little or no learnable signal. The authors dropped the unlearnable VLSE functions because there "rankings merely reflect which model shrinks most strongly toward the training mean". The same argument applies to OpenML, yet those cells are kept, and SML-PN ranks best on the 128 datasets where no model recovers any signal.
  • Training sets can be as small as 2 rows, while every method also gets 30 validation rows. In practice, one would train on all 32 labelled rows, for example with cross-validation. Datasets that are naturally high-dimensional with few samples, such as genomics data with thousands of features, would be a more convincing test than cutting ordinary tables down to n≤ln \leq l.

4. The claims of "no prior" and "no hyperparameters" don't hold.

  • Maximum entropy with a uniform reference on a box is a prior, defined by the box and its starting point at −1.2−1.2. The literature on maximum entropy on the mean (MEM) says this explicitly and gives MEM a Bayesian flavour (Gamboa & Gassiat, 1997, and Le Besnerais et al., 1999). More generally, Bayesian updating is a special case of maximum relative entropy (Caticha & Giffin, 2006). The box bounds are also fitted to the training data along with the multipliers.
  • The "fixed configuration" in Table S2 contains many design choices. They include the initial bounds and multipliers, the input range [0.9,1.2][0.9, 1.2], the target range [−10,10][−10, 10], the tanh⁡\tanh surrogate for the logarithm, the tying schedule, and the epochs, patience and learning rate of every stage. The paper doesn't say how these were chosen, or whether the VLSE functions or the synthetic data were used to choose them.
  • The computed solution doesn't satisfy the maximum-entropy constraints, since training stops early, "before the constraints are met exactly". So the paper's uniqueness argument doesn't apply to what is actually computed, which is an early-stopped least-squares fit in a reparameterised form.

5. Closely related prior work isn't discussed or compared against.

  • The core construction is maximum entropy on the mean (MEM), an established method for linear inverse problems (Gamboa & Gassiat, 1997, and Le Besnerais et al., 1999). MEM picks the maximum-entropy distribution over parameters relative to a reference measure, imposes one expectation constraint per measurement, and uses the mean as the estimate. Le Besnerais et al. (1999) even use a product of uniform densities on bounds as the reference, as in the PN, and they also motivate it through Jaynes and statistical physics.
  • Because the PN output is linear in its parameters, the prediction in Eq. (12) is just the linear model evaluated at the mean weights. What the paper adds is the solver, meaning Adam on the training error with early stopping and the tying schedule, rather than a new framework.
  • In machine learning, Maximum Entropy Discrimination (Jaakkola, Meila & Jebara, 1999) and its regression version (Jebara & Jaakkola, 2000) do the same thing. They use a maximum-entropy distribution over parameters, one expectation constraint and one Lagrange multiplier per training example, and they predict by averaging. RegBayes (Zhu, Chen & Xing, 2014) generalises this.
  • In econometrics, generalized maximum entropy estimation (Golan, Judge & Miller, 1996) applies the same idea to ill-posed linear regression with bounded parameters and little data, which is the same motivation as this paper.
  • Averaging predictions over weight space is also the starting point of the statistical mechanics of learning (Watkin, Rau & Biehl, 1993) and of Bayes point machines (Herbrich, Graepel & Campbell, 2001), while only Seung et al. (1992) is cited.
  • Since none of these is discussed or compared against, the novelty claim doesn't hold as stated.

6. The parameter-matched network is a weak control.

  • NNm\text{NN}_m has only 2 to 3 hidden units. It fails everywhere, which says nothing about neural networks. Matching its size to the number of Lagrange multipliers isn't a meaningful control.

References

Questions for authors
  1. How does SML-PN compare with an equal-weight baseline (an intercept plus a slope on ∑klog⁡xk\sum_k \log x_k), and with ridge that shrinks toward equal weights instead of zero, on all three benchmarks?
  2. What happens if the tying schedule is removed, so all parameters are free from the start, or if the feature order is shuffled before training?
  3. How were the values in Table S2 chosen, and on which data?
  4. How does SML differ from maximum entropy on the mean (Le Besnerais et al., 1999), MED regression (Jebara & Jaakkola, 2000) and generalized maximum entropy estimation (Golan et al., 1996)? Can you compare against at least one of them, and against Bayesian linear regression with a non-zero prior mean?
  5. Can you show results on datasets that are naturally high-dimensional with few samples, and with a protocol where every method can use all n+30n + 30 labelled rows, for example through cross-validation?
  6. Is there any version of SML for a model that is nonlinear in its parameters?
12 comments
AuthorsOpenReview ↗

(1/11): Opening

We thank the reviewer for the careful and constructive review, including an independent implementation of the models. The comments have helped us to greatly improve the manuscript. Before replying point by point, we would like to first summarize the overall idea underlying the paper.

The basic idea of SML is inspired by probabilistic NN and equilibrium statistical physics (particularly, the principle underlying the derivation of Maxwell-Boltzmann, Fermi-Dirac, and Bose-Einstein distributions). For instance, the air in a room has an astronomically large number of microscopic configurations (i.e., the positions and velocities of all molecules, analogous to the complicated phase space of a NN). Yet, thanks to the success of statistical physics, a small number of accurate measurements can determine temperature, pressure, etc (the system-level behavior). There is no need to track the microstates of all molecules (analogous to tracking specific parameter configurations of a NN), which would be computationally prohibitive and, more importantly, unnecessary (waste of information).

SML does not directly “borrow” the Boltzmann factor (characterized by a single inverse-temperature β); rather, it applies the same statistical-physics principle to derive the Boltzmann-factor-like distribution ρ (characterized by n Lagrange multipliers βⱼ) specifically for the available training datasets. Each training sample imposes a constraint on ρ (Eq. 3) (somewhat similar to but not the same as energy conservation in physics). The fundamental objective is not to find a single optimal parameter configuration {wᵢ}ₒ, but to determine the expectation value (like temperature or pressure in physics), following the maximum-entropy principle. At the theoretical level, no loss-function minimization is required. In the present SML-PN implementation, numerical minimization is employed to determine βⱼ and auxiliary quantities, but it is just a part of the solver. Even if we view it as a loss function, it is based on βⱼ (and the parameter bounds), different from the conventional loss functions (based on {wᵢ}).

The role of power-law network (PN): The main purpose of this paper is to report on the feasibility of the concept of SML, instead of the performance of any particular NN or for any particular datasets. PN was the first network architecture we investigated, chosen because its ensemble averages remain simple after integration. We initially planned to move quickly on to other more complex NNs, but were surprised by the (relatively) impressive performance of PN (with its simple architecture). So we investigated PN in greater depth and compared it systematically with a broad range of tuned models. Beyond PN, we have identified two additional network families, including a few relatively commonly studied NNs, for which integral wavelet transforms provide analytical expressions for ensemble averages. This work is ongoing, and the results will be reported elsewhere. We hope that the present paper (if accepted) will attract broader interest and encourage other researchers to explore the idea of SML in machine learning.

AuthorsOpenReview ↗

(2/11): Opening, continued: what is new in SML

Innovation of SML: We fully agree with the reviewer that ensemble averaging and maximum-entropy principle, by themselves, are not new to machine learning. Ensemble averaging has been extensively investigated in Bayesian neural networks, statistical mechanics of learning, Bayes point machines, etc., while maximum entropy on the mean (MEM), generalized maximum entropy (GME), maximum entropy discrimination (MED), and related methods provide important precedents for determining probability distributions under constraints (see our reply to Weakness 5 below for more details). The principal distinctions of SML concern how the ensemble average is formulated for supervised network learning and how it is computed and applied.

  • Specifically, training data are used as expectation constraints rather than an explicit Bayesian update. For a given network architecture, each training sample imposes one constraint (with no margin or slack variables, unlike MED) on the ensemble-averaged network output, associated with one Lagrange multiplier βⱼ. Entropy maximization determines ρ subject to these constraints. Unlike Bayesian neural networks, which construct posterior distributions from priors and likelihoods, SML does not explicitly introduce either; its distribution is determined through constrained entropy maximization relative to a reference measure, which acts as its only prior (in SML-PN it is only used to initialize the parameter bounds, which are then fitted) (see our reply to Weakness 4). Although ρ has an exponential form resembling Gibbs distributions, its n Lagrange multipliers βⱼ are not the inverse temperature β. Unlike conventional neural-network training, SML does not seek a weight configuration that minimizes a conventional loss. This distinction does not imply that Bayesian methods necessarily optimize individual weights, nor that multiple constraint multipliers are unique to SML. The novelty of SML is that it puts these ingredients together in one supervised-learning formulation, which none of the related methods discussed in our reply to Weakness 5 does: SML derives its own Boltzmann-factor-like distribution with one Lagrange multiplier βⱼ per training sample, and each training target is imposed as an equality on the ensemble-averaged output of a parametric network; its only prior is a uniform reference on a bounded parameter domain; and new inputs are predicted directly by the same ensemble average. SML-PN shows that this formulation can be solved in practice, using a coarse-to-fine numerical strategy for the resulting nonlinear system of equations (SM Sec. 3.4).

In the revised manuscript, we have added the equal-weight baseline suggested by the reviewer, ridge regression shrinking toward equal weights, and Bayesian linear regression with an equal-weight prior mean, evaluated on the synthetic task and both benchmarks (Secs. 5.1 and 5.4). We report the effect of removing the tying schedule (SM Sec. 3.4) and permuting input features (SM Sec. 7), compare SML-PN with MEM and GME (Sec. 5.4), clarify its relationship with MED (Sec. 2; SM Sec. 2.3), and present benchmark results both including and excluding the no-signal cases (SM Secs. 5.2 and 6.3). We have also added new references (Sec. 2) and clarified the scope of our claims (Secs. 2-5 and 7).

AuthorsOpenReview ↗

(3/11): Weakness 1 and Question 6

Weakness 1 and Question 6. We fully agree with the reviewer that after its logarithmic activation, PN’s output is linear in its parameters, and on the scaled range [0.9, 1.2] the log-features are close to affine. We regard this as the design of the comparison rather than a weakness of the results, and we have made the scope explicit.

Why a linear function class is relevant here. The main concept of SML is focused on how the ensemble distribution is formulated for supervised network learning and how it is computed and applied. In the n ≤ l regime studied here, the function class is not the bottleneck. With continuous features, n ≤ l training inputs are generically linearly independent, so an unconstrained linear (or nonlinear) model can fit the n targets; its coefficients are underdetermined, and accuracy depends on how they are chosen. Tuned ridge and LLR share essentially the same function class as SML-PN, so comparing them isolates exactly this (please see SM Sec. 3.2). Within that class, SML-PN exceeds tuned ridge and LLR by 0.31 to 0.48 in median R² on 8 of the 9 analysis-set functions (Table S18) and ranks ahead of both on the 848 OpenML datasets (Table S22). As the reviewer notes, tuned ridge and LLR are themselves nearly identical (within 0.030 in median R²), which supports that the gap comes from how SML-PN determines the coefficients rather than from the features; our reply to Weakness 2 shows that, on real data, it is not an equal-weight effect.

As mentioned in the opening remark, the main purpose of this paper is to report on the feasibility of SML, not the performance of PN on any particular datasets. PN is merely a device to demonstrate the concept of SML.

Neural networks; Question 6. PN’s closed form limits the PN realization, not the formulation of SML. The formulation of SML (Sec. 3) applies to any model whose output is continuous in parameters that range over a bounded domain, which keeps the partition function Z (Eq. 5) finite; the bounds need not be fixed in advance and can be fitted, as in SML-PN. For models that are sums of terms each depending on a single parameter, Z still factorizes into one-dimensional integrals that quadrature can evaluate, even when each term is nonlinear in its parameter. For networks with coupled parameters, the expectations no longer reduce to one-dimensional integrals. As mentioned in the opening remark, we found two new families of NN (fundamentally nonlinear) for which integral wavelet transforms give analytical forms; the new NNs may also enable hybrid methods of simplified numerical procedure (e.g., guided Monte Carlo sampling) for broader ranges of NNs. PN is merely the first realization of SML, chosen because all its integrals are explicit and simple; extending SML to architectures with coupled parameters is the subject of our ongoing work and will be reported separately.

Changes in the manuscript: the formulation is stated for any model with bounded parameters and demonstrated with PN (Sec. 4); the factorized case and the extension to architectures with coupled parameters, as the main direction for future work (Sec. 7, last paragraph).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the revision (to all reviewers)

Summary of the revision: what changed and where

We thank the reviewers for careful reading and constructive comments. The revised manuscript addresses all questions and comments; the modified text is shown in blue.

What the reviewers found valuable. We are grateful that all three reviewers found merit in the work. Reviewer yiSk found the idea clean and the method simple and cheap, needing no per-dataset tuning while ranking ahead of tuned random forests, SVR and TabPFN on OpenML, especially when training data are scarcest; the reviewer also appreciated our openness about its limitations and found the evaluation extensive (848 OpenML datasets, 1.46 million training runs, tuned baselines and pinned dataset lists). Reviewer QEQX highlighted the novel integration of statistical physics principles with machine learning, the comprehensive experimental validation and the strong performance, particularly in data-scarce settings. Reviewer azDA found the method highly suitable for high-dimensional small-sample problems, performing better than most traditional models when training samples are very few, and stable under noisy labels without additional denoising steps.

For your convenience, the major changes are listed below.

New experiments (all on the published splits; the ranking of the 20 model variants is unchanged)

  1. Equal-weight controls (yiSk W2, Q1): an equal-weight model, ridge shrinking toward equal weights, and Bayesian linear regression with an equal-weight prior mean, each compared head-to-head with SML-PN on the synthetic task and on both benchmarks. → Secs. 5, 5.1, 5.2, 5.4 and 8; Reproducibility statement; SM Secs. 2.3, 3, 4.3, 5.2 and 6.3; Tables S14, S20 and S23.

  2. Maximum-entropy controls (yiSk W5, Q4): maximum entropy on the mean in six variants, and generalized maximum entropy. → Secs. 5, 5.4 and 8; SM Secs. 2.3, 3, 4.3, 5.2 and 6.3; Tables S14, S20 and S23.

  3. Optimization schedule: its rationale, and the result without it (yiSk Q2). → SM Sec. 3.4, Table S13.

  4. Input columns permuted (yiSk Q2). → SM Sec. 7.

  5. Both benchmarks with and without the no-signal cases (yiSk W3). → Sec. 5.2; SM Secs. 5.2 and 6.3; Table S24.

  6. Computational complexity of SML-PN (QEQX Q2). → SM Sec. 3.4.

  7. Training time under noise and contamination (QEQX W2, Q3). → Sec. 6; SM Sec. 7 (training time) and Table S29 (test R²).

Changes to the text

  1. Related work on maximum entropy on the mean, generalized maximum entropy, maximum entropy discrimination, Bayes point machines and the statistical mechanics of learning, and how SML relates to them (yiSk W5). → Sec. 2.

  2. Prior and likelihood: SML requires no likelihood or kernel; its only prior assumption is a uniform reference on a bounded parameter box (yiSk W4). → Secs. 2 and 3.

  3. Scope: the formulation applies to any model with bounded parameters and is demonstrated here with PN; the extension to other architectures is stated as the main direction for future work (yiSk W1, Q6; QEQX W1, Q1; azDA W1, Q2). → Sec. 4; Sec. 7, last paragraph.

  4. Fitted bounds, validation-based stopping and uniqueness (yiSk W4; azDA Q1). → Sec. 2; Sec. 4, last paragraph; Sec. 7, first paragraph; SM Secs. 3.4 and 4.3.

  5. "No per-dataset hyperparameter search", and how the fixed settings were chosen (yiSk Q3; azDA Q1). → Sec. 5; Reproducibility statement; paragraph after Table S2 (SM Sec. 2.2).

  6. Results on real data stated against the tuned models: third of 20, tied with tuned CatBoost and behind tuned XGBoost (yiSk W3). → Sec. 5.3; Sec. 8; SM Secs. 5.3 and 6.3.

  7. Uneven feature importance and wide tables as a limitation (azDA W2; yiSk W3). → Sec. 5.3; Sec. 7, third paragraph.

  8. Roles of NNₘ and of the tuned MLP (yiSk W6). → SM Sec. 2.1.

  9. Wording: the perturbation of Fig. 4c is input noise, and the abstract and the conclusion now say so (→ Abstract; Sec. 8). "Control" now denotes only the ten models of items 1 and 2; NNₘ and LLR are called baselines (→ Sec. 5; SM Secs. 2.2 and 5.3; captions of Tables S3 and S11 and of Fig. S4; note of Table S17). A few further statements are worded more precisely, with no change in results; all are in blue.

Our replies to each reviewer give the results and refer to these places.

Reviewer_yiSkOpenReview ↗

Response to authors' rebuttal

Thank you for the extensive response and the new controls! They address several most my concerns.

The reworded statements are more precise now, the feature-order test is reassuring, and the OpenML comparisons show that the real-data results are not the equal-weight effect I suspected. Your controls on the published splits are more reliable than my reimplementation, and I defer to them. I have raised my soundness score.

My main concern remains. For the model studied, SML coincides with maximum entropy on the mean with fitted bounds, and the controls on VLSE and the schedule ablation suggest the gains come from the solver and its bias toward equal weights as much as from the maximum-entropy formulation. The feature that distinguishes SML from MEM, constraining the average output of a nonlinear network remains to be shown, and the restriction to n≤ln \leq l should also be stated clearly in the paper. I raise my score while strongly suggesting to the authors to acknowledging this in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.