acceptodds
Under review as a conference paper at ICLR 2027

Neural Boltzmann Equations

Abstract

The dynamics of particles in the early universe are described by Boltzmann equations, which involve high-dimensional phase-space integrals. Classical approaches use quadrature integration and evolve the system on a fixed momentum grid, which scales poorly to complicated systems and parameter scans, severely limiting the complexity of processes that can be studied. We introduce Neural Boltzmann Equations (NBEs), which combine three coupled concepts to overcome these limitations. First, particle properties are encoded in physics-inspired neural distribution functions, with parameters that can be predicted using neural networks, enabling efficient parameter scans. Second, phase-space integrals are evaluated with Monte Carlo, using importance sampling tools from collider physics. Third, we use the natural gradient method to evolve the system. After demonstrating the individual benefits of NBEs, we use the framework to perform the most numerically precise calculation to date of the effective number of relativistic neutrino degrees of freedom, , in the Standard Model.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Boltzmann equation, early universe cosmology, neutrino decoupling, importance sampling, neural quantum states, natural gradient

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kJqbOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a novel approach to the solution of Boltmann equation in the context of astrophysics and cosmology. This integro-differential equation governs the evolution of the density of particle species in the universe, and depends on the type of particles, on the geometry and on the physics ansatz considered. Solving this equation encompasses different issues that can be tackled using modern ML, in particular the the need to estimate an high dimensional integral, how to represent a valid class of solutions, and how to do it iteratively to find a solution of the differential equation. The proposed solution is based on a physics-informed NN that represents a class of solution, optimised via natural gradient optimisation, and on the use of Monte Carlo to estimate the high dimensional collision integral. The authors then present 3 experiments, going from toy examples to neutrino decoupling (computation of effective degrees of freedom of this particle).

Critical strengths

The overall aim of the paper is clear, and well motivated by physical applications. The use of ML appears to be decisive. In general, I really appreciate the tentative to use methods of ML to solve a real practical problem in computational physics.

Critical weaknesses
  • The title: Neural boltzmann eq for what? Apparently mostly astrophysics, while BE is a wider concept and appears also elsewhere. Naming the paper Neural Boltzmann Equation could be misleading in my opinion.
  • The venue: even if the paper present a very advanced application of machine learning to astrophysics and cosmology, in this form it appears to be too much "physics-centered" (I go against my physicist origins....) for ICLR. For example the object that is "neural" is a bit hidden in the text, I would prefer something like: "this is the object of study (BE) and classically we use this algo to solve the problem, and this leads to issues that are solved by our neural formulation".
  • The review: I am not completely sure that a paper where the claim is "to perform the most numerically precise calculation to date of the effective number of relativistic neutrino degrees of freedom, Neff, in the Standard Model." should be sent to a conference, rather than a computational physics journal. In general, I believe that this paper deserves a more accurate and specialised review than the one that can provide someone (like me...) who was a physicist, but not computational astrophysicist.
  • The comparison with SoTA: I would present in a more evident way the comparison (also in terms of computational cost) with SoTA methods used to tackle the same BE problems
  • The algorithmic pipeline: before presenting the details of each step of the pipeline, I would appreciate a meta-algorithmic presentation of the overall procedure (this is the differential equation, this will be neural, this is estimated from data, this is sampled, etc). Also the choice of the symbols could be improved, to better distinguish what is analytic and what is trained, for instance ω\omega and w should not be used in the same context (e.g. eq 11 etc). This would be fundamental for a general purpose venue as ICLR, especially in the application sub-sessions.
Questions for authors
  • Why did you decide to submit this paper to a ML conference rather than a computational cosmology journal?
  • Could you please elaborate more on the limitations and computational costs? This would be important for readers to elaborate possible follow-ups, and does not necessarily affect the goodness of the paper.
3 comments
AuthorsOpenReview ↗

Author rebuttal

Thank you for the thorough and constructive review. We are glad to hear that you appreciate our effort to solve a practical problem in computational physics, and that you found the overall aim of the paper clear and well motivated. Thanks as well for the questions and criticisms, which we address one by one below.

Naming the paper Neural Boltzmann Equation could be misleading in my opinion.

In this work, we specialise to Boltzmann equations in FLRW spacetime. However, besides the cosmology applications in the main paper, NBEs using our gamma mixture neural distribution function ansatz are readily extended to any isotropic, homogeneous kinetic theory problem. We demonstrate this in a new Appendix E, where we use NBEs to simulate a mixture of classical, non-relativistic gases.

Anisotropic or inhomogeneous systems require modified ansätze for the neural distribution function that include some combination of the momentum direction and spatial position of the particles. However, the high-level NBE workflow still applies: predict the parameters of the neural distribution function with a neural network, evaluate the collision operator, and then update the network parameters such that they solve the Boltzmann equation. We chose this title by analogy with neural quantum states and neural wave functions.

even if the paper present a very advanced application of machine learning to astrophysics and cosmology, in this form it appears to be too much "physics-centered" [...] for ICLR.

We tried to write the paper around the method, while giving the minimal physics background to make it self-contained. We extended the related work section to discuss the previous Boltzmann solvers and their limitations in more detail.

I believe that this paper deserves a more accurate and specialised review than the one that can provide someone (like me...) who was a physicist, but not computational astrophysicist.

The paper does not claim to produce any new physics results. In the last sentence of the abstract, we claim the "numerically most precise" calculation, putting emphasis on the solver uncertainty. All expressions in Appendix B on the neutrino decoupling formalism have already been published in an equivalent form. We tried our best to include the minimal amount of physics context in our background and results sections, giving readers from the wider AI-for-science community enough information to understand the problem.

I would present in a more evident way the comparison (also in terms of computational cost) with SoTA methods used to tackle the same BE problems

We extended the related work section to explain the properties of contemporary Boltzmann equation solvers in more detail. Our updated submission includes a 'rapid' neutrino decoupling config that achieves 1e-4 precision in 6min on a laptop, a 10x speedup compared to the 1h runtime for the same precision using the implementation of Bennett et al. on the same machine. We refer to our response to reviewer M7nf for more information.

before presenting the details of each step of the pipeline, I would appreciate a meta-algorithmic presentation of the overall procedure

We added Figure 1 to the updated submission, located at the start of the methods section, to give a high-level overview of the NBE workflow with references to the three method subsections.

Also the choice of the symbols could be improved

We have replaced the label for the latent representation, ω→Θ\omega \to \Theta, to avoid confusion with the gamma mixture weights or natural gradient step weights, ww, and to use theta as a common label for learned objects.

Why did you decide to submit this paper to a ML conference rather than a computational cosmology journal?

This work is a collaboration between machine learning and cosmology researchers. Given the number of ways in which NBEs differ from previous Boltzmann solvers, and the list of benefits that NBEs have, we decided to have a separate methods paper that establishes the approach as a new computational paradigm for this class of problems. NBEs can also readily be applied to standard kinetic theory problems, as we demonstrate in Appendix E. We also believe that the way in which NBEs combine tools from different fields (optimisation, LHC physics, statistical physics) is a meaningful contribution to the wider ML community.

Could you please elaborate more on the limitations and computational costs?

We extended the limitations and computational cost paragraphs with information about the initialisation cost, the benefits of running the code on a GPU, and challenges in conditional training.

We hope we were able to address your questions and look forward to discussing further.

Thank you for the responses. I am convinced to raise the score, but I still hope that, if accepted, the paper will appear also on some computational astrophysics venue.

Thank you for the fast response. We are happy to hear you appreciated these explanations.

We are working on a submission to an astrophysics and cosmology journal, where we showcase the benefits of NBEs on a wider range of early-universe processes including, for instance, dark matter production and leptogenesis.

Reviewer_M7nfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a Neural Network based method for solving systems of Boltzmann equations primarily arising in applications in cosmology. This is optionally coupled with a Neural Importance Sampling framework to allow computations of integral quantities with low variance. The headline result that is presented is a calculation of the effective number of relativistic species in the early universe under the Standard Model to a much lower variance than previously claimed. This is motivated by future calculations where for more complex processes it is possible that the numerical precision would be a dominant uncertainty in the theoretical calculation of these quantities

Critical strengths

The paper is very well presented, clearly written, and uses as far as I can tell minimal AI writing so is easy to read and understand. The submission is aiming for applications to the physical sciences, for papers in this category a clear motivation of the problem and why it is interesting is needed, and in my estimation the authors do a good job of this. The authors provide a code for full replication which I have not run but from a cursory glance looks thorough and easy to reproduce

Critical weaknesses

For an applied physical-sciences submission at ICLR, I would expect either (1) a substantial methodological contribution of interest to the ML community, or (2) evidence that the application of ML enables a qualitatively new scientific capability. As things stand I do not think this submission quite meets either of these criteria sufficiently for acceptance.

For (1) the headline result of 5.3 states “We do not use the MadNIS neural importance sampling refinement for this calculation.”, I therefore dont understand if this part of the procedure is needed, and without this the Machine learning is learned coefficients for a Gamma mixture model as far as I can tell. Whilst establishing a new application for ML in the sciences is interesting and useful, this seems like a fairly simple construction. The applications presented have not had a great deal of prior ML work, and the related areas that have, Einstein Boltzmann solvers for power spectrum estimation do not clearly stand to gain from this.

On (2) I have some prior experience in this area and my impression is that there are much larger issues than the numerical precision of the calculations, and my understanding of the prior literature, including that cited in this paper, is that the existing techniques are largely “good enough” for calculating N_eff for example. Taking (Bennett et al) as cited in this work, the conclusion of that work states that for an uncertainty of 10^-4 “An uncertainty of this magnitude also more than suffices to minimise the total error budget in the inference of cosmological parameters from the forthcoming generation of cosmological observations”. As presented, the main advance appears to be increased numerical precision for an already tractable calculation, and the scientific significance of that additional precision is unclear.

If the NBE approach can unlock new physics results or calculations then I think this would have to be concretely demonstrated, and in that eventuality it would likely be more appropriate to be reviewed in a scientific journal.

Bennett, Jack J., et al. "Towards a precision calculation of $ N_ {\rm eff} $ in the Standard Model II: Neutrino decoupling in the presence of flavour oscillations and finite-temperature QED." arXiv preprint arXiv:2012.02726 (2020).

Questions for authors

The authors position this work as a general framework for Boltzmann equation solving and cite the Cosmology Einstein Boltzmann codes CLASS and CAMB. To my knowledge these are the most high profile usage of Boltzmann solvers in cosmology, and I know that there has been significant development and interest in accelerating these (1) and (2). Does the proposed NBE framework help on these problems? And how would it improve on say a differentiable solver/emulator coupled to a gradient accelerated MCMC as in (1)?

The usage of Neural Importance Sampling in general seems not particularly well motivated. For the use cases of these NBEs I would wonder if something like Sequential Monte Carlo is more effective for computing normalised integrals efficiently? SMC can exploit the time steps being used in the current solve, have the authors considered these approaches? It seems the NIS isn’t even used for the headline result as well, so to what extent is that an important part of the process? The abstract motivates NIS as being one of the 3 components of NBEs.

On a related note, Figure 2. Shows the gain from BEST in particular, the dominant variance reduction is attributed to MadSpace phase-space mappings and distribution-function sampling rather than MadNIS. Could the authors provide a matched baseline using the same phase-space mapping and momentum proposal with a conventional adaptive integrator such as VEGAS? This seems necessary to isolate the contribution of neural importance sampling. As things stand it appears that the dominant gain is just the hand designed phase space mapping

The paper states that training can take up to 5 hours on an NVIDIA A30 GPU. Are the comparisons to non-neural baselines controlled for computational cost? In particular, could the authors provide a more detailed comparison of wall-clock time and computational cost on matched hardware, ideally reporting time-to-fixed-accuracy as well as sample efficiency? This comparison should include any proposal training or adaptation overhead. Without this, it is difficult to assess whether the reported reduction in Monte Carlo variance translates into a practical computational advantage. I am wondering if for example, a less efficient VEGAS integration could be arrived at much more cheaply, and although less efficient give a similar level of accuracy the amount of time the proposed approaches take

(1) Hahn, O., List, F., & Porqueres, N. (2024). DISCO-DJ I: a differentiable Einstein-Boltzmann solver for cosmology. (2) Spurio Mancini, A., Piras, D., Alsing, J., Joachimi, B., & Hobson, M. P. (2022). CosmoPower: emulating cosmological power spectra for accelerated Bayesian inference from next-generation surveys.

4 comments
AuthorsOpenReview ↗

Author rebuttal (part 1/2)

Thank you for the thorough and constructive review. We are glad to hear that you appreciate our motivation of the physics problem and its relevance. We were particularly happy to hear that you found the paper very well presented, clearly written, and easy to read and understand. Thanks as well for the questions and criticisms, which we address one by one below.

For an applied physical-sciences submission at ICLR, I would expect either (1) a substantial methodological contribution of interest to the ML community, or (2) evidence that the application of ML enables a qualitatively new scientific capability. As things stand I do not think this submission quite meets either of these criteria sufficiently for acceptance.

We agree that either (1) or (2) is necessary for a good AI-for-science paper. We argue in the next point that our submission enables a qualitatively new scientific capability, i.e. aspect (2), with ML as an indispensable component of the method.

On (2) I have some prior experience in this area and my impression is that there are much larger issues than the numerical precision of the calculations, and my understanding of the prior literature, including that cited in this paper, is that the existing techniques are largely “good enough” for calculating N_eff for example.

Existing Boltzmann solvers mostly rely on momentum discretisation and analytic reductions for the collision integral. As a result, they share a list of clear limitations: expensive parameter scans; poor scaling to complex processes in the collision integral; limited overall precision; the lack of a modular framework that can easily be applied to other processes. All of these limitations can be overcome using NBEs. These limitations matter most in new physics scenarios, such as dark matter production, leptogenesis, or neutrino decoupling in the presence of new particles with relevant interactions in the MeV epoch. Going beyond order-of-magnitude estimates when simulating these systems requires a full Boltzmann equation solver, which has led to a range of codes that modify the standard approach described above or make physics assumptions to make the calculation feasible. NBEs provide a principled way of performing these calculations, and currently we do not see any limitations besides those of the distribution ansatz. We have already used NBEs to quickly reproduce results of papers on dark matter production and leptogenesis, but decided to focus on standardised applications in this submission. Overall, we believe that NBEs significantly move the frontier of what calculations are computationally feasible for early-universe Boltzmann equation solvers.

The authors position this work as a general framework for Boltzmann equation solving and cite the Cosmology Einstein Boltzmann codes CLASS and CAMB. [...] Does the proposed NBE framework help on these problems?

CLASS/CAMB solve the Einstein-Boltzmann equations for linear perturbations to the spacetime metric and distribution functions, whereas NBEs solve the Boltzmann equation on a fixed background for the full distribution function. The two solvers are complementary: results of an NBE calculation can be used as inputs to CLASS/CAMB. We discuss this in more detail in the new "Einstein-Boltzmann solvers" paragraph in the related work section.

And how would it improve on say a differentiable solver/emulator [(2)] coupled to a gradient accelerated MCMC as in (1)?

References (1) and (2) make CMB analyses using CLASS/CAMB faster and more flexible by (1) providing a differentiable implementation, or (2) training an emulator to interpolate between solutions of earlier CLASS/CAMB runs. NBEs currently solve the Boltzmann equation on a fixed background, and so in their current form are not directly applicable to CMB analyses. This would require a different ansatz for the neural distribution functions that incorporates metric perturbations, and is beyond the scope of our submission. Methodologically, however, NBEs combine the benefits of both: of (1), via a differentiable implementation in pytorch and of (2), via the conditional training option. Compared to Ref. (2), NBEs have the benefit that they simulate the underlying system directly, whereas (2) interpolates between pre-existing solutions and is therefore limited by the resolution of the training data.

AuthorsOpenReview ↗

Author rebuttal (part 2/2)

I would wonder if something like Sequential Monte Carlo is more effective for computing normalised integrals efficiently?

NBEs use neural importance sampling to evaluate Monte Carlo integrals using independent and identically distributed draws, see Eq. (12), without correlations between iterations or time steps. Our understanding is that the reviewer proposes to instead track a fixed pool of momentum configurations across iterations and time, and use SMC to reweight them such that they remain optimal. This approach uses weighted samples and therefore increases the required number of samples for a fixed integral precision. For NBEs the integrand evaluation is the time bottleneck in the collision integral, making the proposed sequential approach slower for fixed accuracy. Markov chain approaches are designed for applications where sample generation is the time bottleneck, such as CMB analyses where a single sample requires a full CLASS/CAMB simulation.

It seems the NIS isn’t even used for the headline result as well, so to what extent is that an important part of the process?

We use NIS for the precision neutrino decoupling calculation in our updated submission, see Figure 9 for the gain. NIS allows us to reduce the number of samples by 10x, which reduces memory consumption by 4x, all while keeping runtime similar and precision unchanged.

Could the authors provide a matched baseline using the same phase-space mapping and momentum proposal with a conventional adaptive integrator such as VEGAS?

We performed an ablation with VEGAS refinement, using the public VEGAS python package. We find that NBE+VEGAS sits between the NBE and NBE+MadNIS precision, as the VEGAS proposal distribution is binned and factorised, and therefore cannot resolve sharp or correlated features.

Phase-space generatorσˉ\bar\sigma (2-to-2)σˉ\bar\sigma (2-to-3)
BEST9.2%6.8%
NBE0.45%0.58%
NBE+VEGAS (10 iterations)0.30%0.27%
NBE+MadNIS0.26%0.25%

We also stress that VEGAS is not compatible with conditional NBE training, i.e. learning distribution functions jointly over model parameter space, as it would require a separate VEGAS call for each parameter value.

could the authors provide a more detailed comparison of wall-clock time and computational cost on matched hardware, ideally reporting time-to-fixed-accuracy as well as sample efficiency?

We compare the runtime of NBEs to the public neutrino decoupling solver of Bennett et al., with the default 1e-4 config taking 1h on our workstation laptop. Froustey et al. use a similar approach and report "a few hours on a single CPU" for the runtime, matching our measurements for the code of Bennett et al.

ApproachMachineRuntime for 1e-4 precisionRuntime for 1e-5 precisionRuntime for 2e-6 precision
Bennett et al.Workstation laptop (CPU, 16 cores)1h (measured)100h (estimated lower bound)--
NBEWorkstation laptop (CPU, 16 cores)6min (measured)50h (scaled)330h (scaled)
NBENVIDIA A30 (GPU)--40min (measured)4h (measured)

For NBEs, we created a new 'rapid' config that reaches 1e-4 precision after only 6 minutes on the same laptop, 10x faster than the code of Bennett et al., and described in Appendix D.6. We also timed 5 iterations with our 1e-5 and 2e-6 configs on the same laptop, and scaled this value by the number of iterations required for the full run, obtaining 50h for 1e-5 and 330h for 2e-6. The same runs take 40min and 4h on a GPU. The GPU and 'rapid' CPU timings are end-to-end, whereas the scaled CPU timings neglect warm-up times which are below 1%.

We make a naive estimate of the required runtime of the Bennett et al. code to reach 1e-5 precision on our laptop. For simplicity, we only focus on the dominant 1e-4 momentum discretisation uncertainty, and neglect the 2e-5 initial condition, 1e-5 compiler/machine uncertainty, and potential other uncertainties that also have to be reduced for this precision target. The cost increase then comes from the collision integral evaluation, which scales as N^4 for N bins. Neglecting the quadrature integration uncertainty, the momentum discretisation uncertainty is due to the trapezoid rule used for the momentum integrals, which scales as 1/N^2. To decrease the uncertainty by 10x, we therefore need sqrt(10)x as many grid points, increasing the number of operations by a factor of sqrt(10)^4=100, and the runtime from 1h to 100h. We expect the real runtime to be significantly larger, as this estimate relies on a series of aggressive assumptions.

We hope we were able to address your questions and look forward to discussing further.

Thanks for the detailed response. Ultimately my primary concern was that the headline result of this work was reducing the variance on a quantity that is measured to a precision of 2.81 ± 0.12 from a solver uncertainty of 10−410^{-4} to 10−610^{-6}, as far as I am aware even the future experiments that will improve this measurement will still be well beyond the regime where this kind of resolution is actually providing much utility.

Beyond the headline result, the applications that would show a new capability remain potential rather than demonstrated, other targets such as dark matter production and leptogenesis are mentioned (l. 133–135) but not shown. The new-physics scan needs only percent-level precision by the manuscript's own statement (l. 1525–1526), and actually appears to be mostly 1 dimensional (nothing substantive changes across rr), so I think this could be constructed with classic approaches. The 2↔32\leftrightarrow3 thermalisation and the processes up to 5→55\rightarrow5 are toy problems, and I'm not sure what realistic new physics model would require a 5→55\rightarrow5 process in this regime. As an aside I find the framing of Einstein–Boltzmann CMB solvers as complementary a little confusing as these just seem to be different problems requiring different methods, which was sort of what I was getting at in my initial question. As in my original review, I think a new capability would need to be concretely demonstrated for me to change my score.

Thank you for the detailed response. We address the individual aspects below.

the applications that would show a new capability remain potential rather than demonstrated

Thanks to the targeted requests by the reviewers, we believe the three method benefits of NBEs are now clearly demonstrated:

  • High-dimensional collision integrals: Using state-of-the-art LHC tools, NBEs achieve around 1000x higher sample efficiency than a previous method designed for this problem, demonstrated on up to 5-to-5 processes. In principle, NBEs can handle any process that an LHC event generator can.
  • Precision and efficiency: Neutrino decoupling is the most established precision calculation for early-universe Boltzmann solvers. NBEs can achieve 50x more precision than existing solvers, and are 10x faster at the equal precision. The modular framework of NBEs carries these gains to other problems.
  • Single-solve parameter scans: We perform a 2d parameter scan for a new physics model with a single solve, with effectively infinite resolution in parameter space after ~13x the time of a single-point solve. The trained network also provides gradients in model parameter space, and the gains increase with the dimensionality of the parameter space.

To our best understanding, the referee would like us to directly demonstrate the new scientific capability unlocked by NBEs in the form of new physics results, as opposed to giving evidence that NBEs enable this capability. As the referee emphasises, new physics results should be reviewed in a physics journal, and not a computer science venue like ICLR.

Choosing reference problems to demonstrate the capabilities of NBEs was a challenge, as the problems that benefit most from NBEs have not been studied quantitatively before, as existing solvers are incapable of solving them efficiently. Below, we address the referee's concerns about the relevance of our individual experiments.

the headline result of this work was reducing the variance on a quantity that is measured to a precision of 2.81 ± 0.12 from a solver uncertainty of 1e-4 to 1e-6

The goal of the N_eff theory program is to keep the theory uncertainty small enough to be safely neglected in any analysis. With upcoming experiments aiming for 1e-2 precision on N_eff, the theory uncertainty should be at the 1e-4 level or below. Within the theory uncertainty, the solver uncertainty should sit below physical effects like higher-order corrections or the precision of experimental input parameters, so that these can be quantified. Previous solvers do not achieve this. Big Bang Nucleosynthesis is even more demanding: the primordial helium fraction is already measured at the 5e-3 level, and depends linearly on neutrino distribution functions from decoupling, so the same arguments call for a solver uncertainty at the 1e-5 level.

In particular, there is an ongoing effort for a full next-to-leading-order calculation of neutrino interactions, with an effect between 7e-4 and 2e-5 based on estimates in previous papers. Once the analytic expressions are available, quantifying their impact on N_eff will require a Boltzmann solver with at least 1e-5 precision.

other targets such as dark matter production and leptogenesis are mentioned (l. 133–135) but not shown

We do not see any new computational challenges in simulating these problems beyond the three points demonstrated above. Therefore, adding these results to the present submission adds little extra evidence regarding the capabilities of the method, while requiring significant physics discussion to introduce the problems.

The new-physics scan needs only percent-level precision by the manuscript's own statement (l. 1525–1526), and actually appears to be mostly 1 dimensional (nothing substantive changes across r)

We find that N_eff varies by up to 0.13 across values of r. Classical methods are too expensive for parameter scans with sub-percent-level precision if they don't rely on specific assumptions.

The 2-to-3 thermalisation and the processes up to 5-to-5 are toy problems, and I'm not sure what realistic new physics model would require a 5-to-5 process in this regime

The 5-to-5 study was purely to demonstrate the capabilities of NBEs. The 2-to-3 case is in the standard SIMP scenario (1402.5143), and 2-to-4 is the dominant process in Z2Z_2 extensions of SIMP (e.g. 1510.08527, 1803.08064, 2212.09759). These papers rely on assumptions to make the computation tractable, which are not required by NBEs.

I find the framing of Einstein–Boltzmann CMB solvers as complementary a little confusing as these just seem to be different problems requiring different methods

Einstein-Boltzmann CMB solvers (CAMB/CLASS) and Boltzmann solvers (e.g. NBEs) are complementary from a physics perspective, not a method perspective. We removed the statement about complementarity of the approaches to avoid this possible confusion, and will upload this in our final submission later this week.

Reviewer_1aKoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a solver for homogeneous, isotropic early-universe Boltzmann equations. Each particle's distribution function is a mixture of 50 gamma distributions, whose parameters can be predicted by a small MLP conditioned on physics parameters, to amortise parameter scans. Collision integrals are computed by Monte Carlo, using existing high-energy-physics phase-space mappings (MadSpace) and, optionally, normalising-flow importance sampling (MadNIS). Time stepping uses implicit multi-step schemes, where each step is a fixed-point iteration fitted with a Gauss–Newton least-squares update, plus a constrained update that preserves conserved moments. Experiments cover fitting equilibrium distributions, a toy thermalisation problem with 2−22-2 and 2−32-3 processes, a precision calculation of effective degrees of freedom, and a two-parameter scan of a new-physics model.

Critical strengths
  • The paper delivers a real scientific result. The NeffN_\text{eff} value agrees with the literature, and the solver uncertainty is estimated.
  • The approach is general. Adding a new process only needs its matrix element, and Monte Carlo integration handles processes like 2-3 that are hard for grid-based solvers.
  • The distribution model can be sampled exactly, so it also serves as the importance sampler for the collision integrals. The update that keeps number and energy density fixed is simple, and Fig. 7 shows that it matters.
  • One conditional training covers a two-parameter scan of a new-physics model, which would otherwise need many separate runs.
  • Every table and figure can be reproduced from the supplementary code with well-documented instructions.
Critical weaknesses

I will prelude this section by saying that these are points of improvement and do not reflect in proportion what I consider the state of the paper; on the other hand, my scores do. I consider it a strong submission and addressing the points below would further improve my scores.

1. The paper overstates the role of the neural networks.

  • The name and abstract suggest that neural networks are central. In practice, a network appears in only two places: the small MLP used in the conditional runs and the optional MadNIS flow. As far as I can tell from Appendix C.3, the precision NeffN_\text{eff} run is unconditional and does not use MadNIS. So the headline result comes from fitting a gamma mixture with about 150 parameters by Gauss–Newton, with Monte Carlo collision integrals.
  • This doesn't weaken the result, but the paper should say clearly which experiments use which learned parts.

2. The baselines are weak.

  • The only optimiser baseline is Adam, on fitting a fixed equilibrium distribution (Section 5.1). Gauss–Newton is expected to beat Adam on a small least-squares problem like this. L-BFGS or Levenberg–Marquardt would be fairer comparisons, and the optimisers are never compared during the actual time evolution.
  • The comparison with BEST (Section 5.2) measures Monte Carlo variance at a fixed number of samples, which shows the benefit of better importance sampling. It doesn't compare wall-clock time or end-to-end accuracy, and the authors changed the baseline's quadrature.
  • For NeffN_\text{eff}, there's no comparison of runtime and accuracy against existing neutrino-decoupling codes. One precision run takes about 4 GPU-hours. According to the supplementary REPRODUCE.md, the uncertainty budget in Table 3 alone needs 74 such runs, so more than 300 GPU-hours. The paper doesn't say how this compares with existing codes at the same precision.

3. The uncertainty budget behind the precision claim is incomplete.

  • The claims of "the most numerically precise calculation to date" and a "50×50\times smaller solver uncertainty" compare the authors' error budget with the ±10−4\pm 10^{-4} of Bennett et al. (2021). The paper credits Froustey et al. (2020) with the same ±10−4\pm 10^{-4}, but Froustey et al. set their tolerances "so as to obtain results with numerical errors below 10−510^{-5}". Against that, the gain is closer to 5×5\times than 50×50\times, and the claim should be toned down.
  • Table 3 varies the Monte Carlo samples, quadrature, starting point, step size and fixed-point convergence. It never varies the model itself, meaning the number of mixture components (always M=50M = 50) or the regularisation used when inverting the Fisher matrix. Yet the paper says these two limit the precision, and it shows that a restricted Fermi–Dirac shape shifts the result. Any bias from the model would be the same in every run, so neither the spread over seeds nor the fast-versus-precise check would catch it.
  • By the authors' own account, the missing higher-order QED corrections are 1010 to 350×350 \times larger than the solver uncertainty. So the better solver doesn't change the physical prediction today.

4. The scalability claims aren't backed by experiments.

  • The abstract and conclusion say that grid-based solvers "scale poorly", that NBEs "scale comfortably to complex processes", and that they bring "high-dimensional parameter spaces within reach".
  • The largest process tested is 2−32-3, a 5−5-dimensional integral. The parameter scans cover one and two parameters, and the universe is homogeneous and isotropic throughout.
  • The conditional training is 10 to 100×100 \times less precise than unconditional runs. It costs 5 GPU-hours against 40 minutes for a fast unconditional run, and the paper doesn't discuss this trade-off.
  • The statement that "for n>2n > 2, Monte Carlo integration is the only efficient integration method" goes too far. Existing codes use analytic reductions, and quasi-Monte Carlo methods exist.

5. Related ML work is missing.

  • Fitting each implicit time step with Gauss–Newton is a known way to train neural models step by step in time. Close precedents are Neural Galerkin schemes (Bruna et al., 2024), implicit time steps solved as fitting problems for neural quantum states (Gutiérrez & Mendl, 2022, and Sinibaldi et al., 2023), and the "discretise-then-optimise" view of these methods (Zhang et al., 2024). The paper should compare its update to these (theoretically, no need for an experiment) and make clear that what's new is the mixture model, the Monte Carlo collision integrals and the moment-preserving update.
  • Neural approaches to Boltzmann collision operators aren't discussed. The closest is Barrera Cabodevila, Kurkela & Lindenbauer (2025), who replace the Monte Carlo collision integral of a particle-physics kinetic equation with a neural network. For the homogeneous Boltzmann equation, see also Xiao & Frank (2021) and Holloway et al. (2021).
  • Neural importance sampling predates MadNIS (Müller et al., 2019, and Bothmann et al., 2020), and solvers conditioned on physical parameters have been studied as hypernetworks (de Avila Belbute-Peres et al., 2021). A short paragraph putting the conditional training in this context would help.

References

  • Barrera Cabodevila, Kurkela, Lindenbauer. Solving the QCD effective kinetic theory with neural networks. Eur. Phys. J. C 85:1304, 2025. https://arxiv.org/abs/2506.19632
  • Bothmann et al. Exploring phase space with Neural Importance Sampling. SciPost Phys. 8:069, 2020.
  • Bruna, Peherstorfer, Vanden-Eijnden. Neural Galerkin schemes with active learning for high-dimensional evolution equations. J. Comput. Phys. 496:112588, 2024. https://arxiv.org/abs/2203.01360
  • de Avila Belbute-Peres, Chen, Sha. HyperPINN: Learning parameterized differential equations with physics-informed hypernetworks. NeurIPS 2021 DLDE Workshop. https://arxiv.org/abs/2111.01008
  • Froustey, Pitrou, Volpe. Neutrino decoupling including flavour oscillations and primordial nucleosynthesis. JCAP 12:015, 2020.
  • Gutiérrez, Mendl. Real time evolution with neural-network quantum states. Quantum 6:627, 2022.
  • Holloway, Wood, Alekseenko. Mathematics 9:1384, 2021 (neural acceleration of the homogeneous Boltzmann collision integral). https://doi.org/10.3390/math9121384
  • Müller, McWilliams, Rousselle, Gross, Novák. Neural Importance Sampling. ACM Trans. Graph. 38, 2019. https://arxiv.org/abs/1808.03856
  • Sinibaldi, Giuliani, Carleo, Vicentini. Unbiasing time-dependent Variational Monte Carlo by projected quantum evolution. Quantum 7:1131, 2023.
  • Xiao, Frank. Using neural networks to accelerate the solution of the Boltzmann equation. J. Comput. Phys., 2021. https://arxiv.org/abs/2010.13649
  • Zhang, Chen, Vanden-Eijnden, Peherstorfer. Sequential-in-time training of nonlinear parametrizations for solving time-dependent partial differential equations. arXiv:2404.01145, 2024.
Questions for authors
  1. Does the precision NeffN_\text{eff} run use any neural network, either the conditional MLP or MadNIS? If not, what do the neural parts add to the main result?
  2. How does Gauss–Newton compare with L-BFGS or Levenberg–Marquardt, for the equilibrium fit and during the time evolution?
  3. Can you add the number of mixture components and the Fisher regularisation to the uncertainty budget, for example by rerunning with M=30M = 30 and M=80M = 80?
  4. How do runtime and precision compare with an existing neutrino-decoupling code at the same precision?
  5. For the 2D2D scan, how do cost and accuracy compare between one conditional run and many unconditional runs?
  6. Can you show a case that backs the scalability claims, such as a process beyond 2−32-3 or a scan over more parameters?
  7. How does your per-step Gauss–Newton fit compare with a Neural Galerkin step or a projected step as in p-tVMC?
6 comments
AuthorsOpenReview ↗

Author rebuttal (part 1/2)

Thank you for the thorough and constructive review. We are glad to hear that you appreciated the generality of the approach as well as the distribution function sampling and the conditional training. We were particularly happy that you appreciate our focus on reproducibility. Thanks as well for the questions and criticisms, which we address in turn below.

In practice, a network appears in only two places: the small MLP used in the conditional runs and the optional MadNIS flow.

Correct. The challenge in NBEs is to train the small MLP on the Boltzmann equation objective to high precision while remaining flexible and efficient.

L-BFGS or Levenberg–Marquardt would be fairer comparisons, and the optimisers are never compared during the actual time evolution.

We applied the L-BFGS and Levenberg-Marquardt optimisers on the same problem. We report the mean absolute error (MAE) on the massless boson and fermion equilibrium distributions after 100 steps below, as well as the CPU time and peak memory per step. The Levenberg-Marquardt optimiser matches the performance of our natural gradient approach after tuning the hyperparameters of all approaches, and L-BFGS is in between natural gradient and Adam.

OptimiserMassless boson MAEMassless fermion MAECPU timePeak memory
Natural gradient (default)4e-64e-87ms3MB
Natural gradient (tuned regularisation)6e-78e-97ms3MB
Levenberg-Marquardt3e-62e-97ms3MB
L-BFGS6e-42e-58ms0.4MB
Adam1e-23e-43ms0.4MB

The time evolution algorithm builds the target distribution function based on the Boltzmann equation, but still uses the same update algorithm, and therefore suffers from the same precision limitations.

The comparison with BEST [...] doesn't compare wall-clock time or end-to-end accuracy, and the authors changed the baseline's quadrature.

We perform a timing study on CPUs using 1e6 samples and the 2-to-2 process. We find that the timings of NBE and BEST are similar, however NBE has a 400x higher sample efficiency.

Generatorσˉ\bar\sigmaEffective sample gain compared to BESTCPU time
BEST9.2%1x306s
NBE0.45%418x264s

Our change to the baseline's quadrature fixes a bug in the BEST paper's implementation, and does not affect the reported σˉ\bar\sigma metrics.

For NeffN_\mathrm{eff}, there's no comparison of runtime and accuracy against existing neutrino-decoupling codes.

We compared runtimes at fixed precision on a laptop, and found that NBEs reach 1e-4 precision 10x faster than Bennett et al. We refer to our detailed response to reviewer M7nf for more information.

Froustey et al. set their tolerances "so as to obtain results with numerical errors below ±10−5\pm 10^{-5}".

This quote refers to the ODE solver uncertainty. Their conclusions say "Results on N_eff [...] with a numerical precision better than 10−410^{-4}", but they do not quantify any uncertainties beyond the ODE solver uncertainty. Bennett et al. use a very similar approach and report momentum discretisation at 1e-4 as dominant uncertainty, so we believe that the 1e-4 uncertainty of Froustey et al. is due to the same effect.

Table 3 [...] never varies the model itself, meaning the number of mixture components (always M=50M=50) or the regularisation used when inverting the Fisher matrix.

We varied the number of mixture components and regularisation parameters in the same way as we tested the other entries of Table 3, and found that the corresponding impact on N_eff is below the 1e-6 level. Our statement "The precision is limited by the restriction to 50 mixture components, and by regularisation effects in the Fisher matrix inversion." explains why the equilibrium distributions in Figure 1 are not learned to machine precision.

By the authors' own account, the missing higher-order QED corrections are 1010 to 350×350\times larger than the solver uncertainty.

The reviewer is correct, however we highlight that we did not claim to have improved upon existing Standard Model predictions beyond numerical precision. Rather, the higher precision that can be achieved by NBEs allows the higher-order corrections to be included meaningfully, whereas previous approaches would struggle to isolate them from numerical noise.

The largest process tested is 2−32-3, a 5−5-dimensional integral.

We added Appendix D.3 with collision integral precision plots for up to 5-to-5 processes, where we demonstrate that NBE+MadNIS always reaches σˉ<1%\bar\sigma < 1\% with 1e6 samples.

The parameter scans cover one and two parameters

We do not see any new challenges in higher-dimensional parameter spaces, provided they are sufficiently smooth such that the network can express the target function.

AuthorsOpenReview ↗

Author rebuttal (part 2/2)

The conditional training is 10 to 100×100\times less precise than unconditional runs. It costs 5 GPU-hours against 40 minutes for a fast unconditional run, and the paper doesn't discuss this trade-off.

Conditional runs solve a harder task, and it is therefore expected that they take longer or achieve lower precision. In contrast to unconditional runs on grids, they solve the problem with effectively infinite resolution in parameter space, and therefore there is always a critical resolution such that conditional runs are more efficient afterwards. This critical resolution depends on the smoothness of the parameter space dependence, and converges to the speed and precision of unconditional runs if there is no dependence on model parameters. We discuss this aspect in the limitations paragraph of our updated submission.

The statement that "for n>2n>2, Monte Carlo integration is the only efficient integration method" goes too far. Existing codes use analytic reductions, and quasi-Monte Carlo methods exist.

Analytic reductions can reduce e.g. 2-to-3 processes to a 7-dimensional integral, which is not feasible with quadrature, and further simplifications require assumptions about the integrand. We consider quasi-Monte Carlo a type of Monte Carlo integration in this context.

Fitting each implicit time step with Gauss–Newton is a known way to train neural models step by step in time.

We agree and have added the reference accordingly. The neural quantum states section discusses how NBEs differ from previous approaches.

Neural approaches to Boltzmann collision operators aren't discussed.

We agree and have added these references alongside a discussion in the related work section.

Neural importance sampling predates MadNIS (Müller et al., 2019, and Bothmann et al., 2020), and solvers conditioned on physical parameters have been studied as hypernetworks (de Avila Belbute-Peres et al., 2021).

We agree and have added the NIS references.

We hope we were able to address your questions and look forward to discussing further.

Reviewer_1aKoOpenReview ↗

Response to Rebuttal

Thank you for the detailed response and the new experiments. The added uncertainty studies, the runtime comparisons with BEST and Bennett et al., and the comparison with L-BFGS and Levenberg–Marquardt address my main technical concerns.

However, the headline NeffN_\text{eff} result uses direct mixture parameters and no learned component, and Levenberg–Marquardt matches the natural-gradient update. So the neural network's contribution rests on the conditional parameter scan, where its advantage over repeated unconditional runs is argued but not measured. A cost and accuracy comparison for the two-parameter scan, against unconditional runs on grids of increasing resolution, would make this concrete. I keep my overall score at weak accept.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Author rebuttal and updated submission

We would like to thank all reviewers for their detailed feedback and questions. We hope that we were able to address their questions adequately and look forward to the discussion period.

We incorporate the reviewers' feedback in our updated submission, using the additional page in the main paper and extending the appendices. Below we list the changes individually, and refer to the reviewer responses for additional information.

  • We have extended the related work section with a more detailed discussion of other types of Boltzmann solvers.
  • The new Figure 1 provides a high-level overview of the method, and we changed the notation for the neural distribution function parameters from ω\omega to Θ\Theta.
  • We have extended the paragraphs on computational cost and the limitations of NBEs.
  • We use neural importance sampling (MadNIS) for the neutrino decoupling calculation, which reduced the number of Monte Carlo samples by 10×10\times for the precision calculation, and 40×40\times for the fast variant.
  • The new Appendix D.3 studies the scaling of collision integral precision for many-particle processes, up to 5-to-5.
  • We have extended Figure 9 to include the gain from using MadNIS in the neutrino decoupling calculation, and added a panel comparing the gain at fixed wall time.
  • We have added a new 'rapid' neutrino decoupling config in Appendix D.6 that allows a direct runtime comparison with literature results.
  • In the new Appendix E we show that NBEs can be used to simulate non-relativistic, classical gases.

We updated our submission one more time, now clearly marking every added or edited text compared to the original submission in blue. We also added a citation based on our second response to reviewer 1aKo.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.