- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
This paper introduces DMA an alternative to EP and VMP that formulates factor-to-variable messages directly, bypassing the traditional edge-marginal projection and cavity-message division steps. By enforcing a Dirac input consistency condition and applying Gaussian moment matching, the authors establish Theorem 2.3, an edge local KL divergence inequality for proper messages, and derive explicit update operators for product and leaky-ReLU factors. To handle the improper exact backward message of product factors, the paper proposes a log-normal approximation backed by an error bound in the concentrated input regime. These operators are integrated via Algorithm 1 into a single sweep BNN inference scheme, and benchmarked against Adam, AdamW, IVON, and diagonal Laplace across 1D synthetic regression, a model-mismatch test, and a larger synthetic network.
Critical strengths
- Definition 2.1 provides a clear principle for constructing approximate messages directly. Theorem 2.3 appears valid under its stated assumptions and bounds how much an error in one message can affect the resulting local marginal distribution.
- Propositions 3.1, 3.4, and 3.5 give explicit formulas for the means and variances of the product factor’s forward message and the leaky-ReLU factor’s messages. Algorithm 1 and the Appendix explain the order of message updates, how stored mini-batch messages are replaced, and how messages are parameterised. These details make the method concrete to implement.
Critical weaknesses
- Equation (7) does not match the derivation in Appendix B.4. Its predicted mean is independent of (), whereas the appendix’s formula depends on it. The proof of Theorem 3.3 analyzes a different distribution from the reference defined in Eq. (24). Also, the theorem bounds error against a truncated reference without showing how this controls the actual marginal error. These gaps leave the claimed () guarantee for the proposed update insufficiently justified.
- Two distributions can both shrink toward the same point without their KL divergence approaching zero. This happens for leaky-ReLU when the input concentrates at zero. The smooth case proof also assumes that linear effects dominate nonlinear effects, which can fail when input variances shrink at different rates. Standard ReLU places positive probability exactly at zero, while a continuous Gaussian does not, making their forward KL infinite. Table 2 therefore incorrectly lists this case as covered by Theorem 2.3.
- The evaluation focuses on small synthetic regression problems, with the larger network having only 1,932 weights and no direct EP baseline available. Comparisons are made against Adam and AdamW, which partly highlight the advantages of modeling uncertainty rather than specifically demonstrating the benefits of DMA. DMA achieves predictive NLL and calibration comparable to diagonal Laplace on the small synthetic task. This supports its competitiveness, with the main reported advantage being computational efficiency rather than improved predictive quality. However, the small synthetic experiment does not establish that this advantage generalises.
Questions for authors
- While local message errors are well-bounded, feedforward neural networks act as highly composition heavy loopy structures when message passing interacts across multiple layers and mini-batch epochs. Without a global convergence guarantee or a compositional error bound across layers, what empirical or theoretical evidence suggests that local errors do not compound exponentially or lead to drift as network depth increases?
- Equation (7) and the log-normal derivation in Appendix B.4 give different moments. Which formula is intended and used in the implementation? Please provide a consistent derivation and validate the approximation numerically against a clearly specified proper reference distribution or normalized marginal.
- What additional assumptions or restrictions are needed for the KL convergence claimed in Corollary 2.4? Please address the leaky-ReLU kink at zero and the smooth example (), with independent centered Gaussian inputs having standard deviations () and () as (). In the latter case, the quadratic term dominates despite the nonzero gradient.
4 comments
Response to Reviewer kNeb, part 2 of 2
R4.4 (error over depth). These are two important aspects, and the revision addresses both. Over depth, we measure the error of each layer's marginal against a Monte Carlo reference, since with two or more hidden layers the factor graph has cycles even for one example and a tree argument does not apply: the error grows at most linearly with depth, not exponentially. Over epochs, the new Lemma D.2 gives how far one replacement of a batch's message moves the beliefs, and we measure the drift it causes.
Changes.
- Added the divergence of each layer's marginal from a Monte Carlo reference, with the weight beliefs held fixed, for networks of width 16 and up to 12 layers (Appendix C.5). In the first two layers the propagated moments are exact; from the third on, where units share inputs, the error of the propagated moments grows at most linearly with depth, by about nats per layer for well-determined weights and more slowly for prior-like ones, not exponentially. For prior-like weights the larger part of the remaining error is the non-Gaussianity of the true marginal, which no Gaussian message removes.
- Measured and stated the behavior of the batch iteration over epochs (R3.2).
- Derived how far one replacement of a batch's message moves the beliefs, and measured the drift over an epoch: with randomly ordered data the means stay within to standard deviations of their values at the start of the epoch, for every number of batches (Lemma D.2, Table 3).
R4.5 (derivation of Eq. (7)). Eq. (7) is the intended formula, and it was implemented correctly and used in all experiments; we corrected the inconsistent derivation in Appendix B.4 (R4.1).
Changes.
- Validated it by quadrature against the truncated message over and $r_y/r_z\in\lbrace 1/2,1,2\rbrace $: times the divergence stays below and tends to , and ; most of it is the message's own non-Gaussianity, which no Gaussian removes, and for Eq. (7) is within % of the best Gaussian (Appendix C.4). The formula of the submitted derivation, with both inputs log-normal, is about as accurate at (Figure 6, dashed).
- Validated it against the normalized marginal below and at zero mean (R1.1).
R4.6 (assumptions of Corollary 2.4). Thank you for the comment. Corollary 2.4 needs two additional assumptions: the inputs concentrate at a common rate, with fixed , under which the linear term is and the quadratic term ; and the point is one where the factor is differentiable with a nonzero gradient. In the reviewer's example the linear term's share of is , so tends to the shape of the scaled and to a positive constant; at the leaky-ReLU kink is the constant of R4.2 for every .
Changes.
- Corollary 2.4 now states both assumptions and gives explicit bounds for every factor of the paper (Appendix B.3).
Dear Reviewer kNeb,
Thanks again for your time and effort. As the deadline for the discussion period is approaching, we would greatly appreciate it if you could kindly let us know whether our responses solve your concerns and/or there are any further questions.
Thank you so much for your attention to our work.
Warmest regards,
Authors
Thank you for your reply. I consider q1–q3 addressed and have therefore raised my soundness score. The two weight example shows that EP can stall, while DMA learns. Given DMA’s positioning as an alternative to EP, a matched comparison on the main BNN benchmarks would help establish its practical gains.
Response to Reviewer kNeb, part 1 of 2
Summary. We thank the reviewer for a careful technical check, which pointed us to details in our proofs that needed correction and helped to improve the paper. All of them are corrected; the message formula and the released code are unchanged (G1).
R4.1 (Eq. (7) and its proof). We agree on all three points, and all three are resolved.
Changes.
R4.2 (Corollary 2.4). We agree on all three points, and all three are resolved. At the leaky-ReLU kink the output of N(x;0,σ2) is σReLUα(U), a scale family, so δ does not depend on σ and does not vanish; with different rates the quadratic term can dominate, as the reviewer's example shows (R4.6); and the standard ReLU's forward message has an atom at zero, so δ=∞.
Changes.
R4.3 (evaluation and EP). We agree with this reading: in the submission DMA is comparable to diagonal Laplace in predictive quality, and its advantages are its cost and the absence of tuning. Whether this generalizes is what the new experiments test, with every method on the same data and batches, from 83 to 1 932 and 52K weights (G3). In fit, every method's mean, DMA's included, is close to the truth at plot scale; DMA's error is at the noise level. In uncertainty, its band stays reasonable over the epochs at every size, while PBP's shrinks with every pass and IVON's is narrow outside the data; at 1 932 weights diagonal Laplace with the exact Hessian has the better band. In runtime, that Laplace needs 6 to 10 minutes per seed at 1 932 weights, against DMA's seconds, and is out of reach at 52K weights, where DMA's band covers the true function at 0.62 of the points outside the training range, against 0.26 for PBP and 0.20 for IVON. The advantage thus grows with the network: the exact methods drop out, and the cheap ones lose their uncertainty (Table 7). The tasks are synthetic by design: with the true function drawn from the model's own prior, the exact posterior is the target and an HMC reference can measure every method against it, which real data with an unknown truth would not allow (R1.4).
Changes.