- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper treats the summary network in NPE as a channel with rate , and measures distortion as the expected KL from the full-data posterior to the summary posterior. By the standard information-bottleneck identity this frontier is the IB curve of the Bayesian experiment, written in posterior-error coordinates. Sections 2–3 collect bounds on the resulting intrinsic rate (a parametric converse via Clarke–Barron, posterior quantisation, a small-ball converse, the exact Gaussian frontier). The main result, Theorem 4.2, shows that under mutual absolute continuity of the sampling laws, stochastic summaries achieve strictly lower distortion than any deterministic summary or time-sharing mixture of deterministic summaries, at every interior rate. The proof uses "hidden relabelling": with small probability , replace the code's output by an independent draw from its own marginal, which saves order nats of at a distortion cost of . Empirically, the authors insert a 128-bit Bernoulli bottleneck into NPE, estimate each model's rate, and compare at matched rate: a stochastic channel beats the exact deterministic hull on an enumerable Beta–Binomial example, stochastic bottlenecks beat annealed deterministic ones by 0.17–0.62 nats NLL on four simulators, and encoder rankings change with rate.
Critical strengths
- Theorem 4.2 is the real contribution and it is done well. The hidden-relabelling construction is elegant, the uniform control of the collision gap avoids any attainment assumption, and the result holds for general standard-Borel experiments, which is cleaner than the usual finite-alphabet arguments.
- The scoping is careful. Appendix D gives examples showing exactly where each assumption is needed (no gap for deterministic targets, no gap for overlapping-posterior erasure models, the difference between an entropy constraint and a rate constraint), and Prop. B.8 rules out the gap coming from different initial slopes. The limitations paragraph is candid that this is a one-shot result.
- The distinction between summary dimension, code cardinality and information rate (Thm 3.2 vs Thm 4.2) is a useful one for the SBI community, as is the remark that posterior fidelity alone cannot distinguish from when is irrelevant.
- The Beta–Binomial benchmark is a good controlled check: all 21,147 count partitions are enumerated, so the deterministic hull is exact, and the decoder error ( nats) is reported and sits well below the measured gap.
Critical weaknesses
-
The paper never says what I(Y;S) is the cost of, so it is unclear what the main theorem buys in practice. The whole result turns on measuring the summary's "budget" as mutual information rather than as output entropy. In a one-shot setting, producing a stochastic summary actually costs more than , whereas a deterministic code costs H(T) up to a bit, so the two frontiers compared in Theorem 4.2 are not measuring the same thing. And in NPE there is no channel at all: the decoder sees S directly. The introduction's motivation, that the information budget confounds comparisons between summary architectures, is never demonstrated. Standard NPE summaries are deterministic and continuous, so their is infinite (or H(Y) at float precision), and the paper never measures an "effective budget" for an ordinary encoder.
-
The simulator experiments mix the theoretical effect with differences in how the two arms are trained. The stochastic arm is trained and deployed the same way. The deterministic arm is trained stochastically, annealed to temperature 0.1, then hard-thresholded with straight-through gradients, and its rate penalty is, for hard codes, a loose factorised stand-in for the true entropy H(B). So the deterministic models are optimised against the wrong objective. The size of the gaps is telling: 0.17–0.62 nats on the simulators, against at most 0.036 nats in the one case where the gap can be computed exactly. A difference ten times larger than the theory produces is more likely to come from training than from Theorem 4.2. A deterministic baseline trained against its actual entropy (with a learned prior on B, as neural compression does) and the lower convex hull over would make H1 convincing.
-
"Matched rate" depends on rate estimates that are shaky, and the results have no error bars. The Monte Carlo rate estimate is capped at nats, so every operating point sits below about 10 nats and everything above the cap is thrown away. In Fig. 2, the gap between the MC and MADE estimates for the same model is, at higher rates, about as large as the horizontal shift between the stochastic and deterministic curves, and the paper does not say which estimate it uses to match. There are three seeds, the headline H1 statistic comes without a confidence interval, and the Fig. 6 diagnostic is a single seed.
Questions for authors
- What resource does I(Y;S) represent in NPE, where the decoder receives S directly? Is there any downstream benefit (generalisation from a finite simulation budget, robustness, calibration) from operating at lower I(Y;S) at fixed NLL?
- App. K.2 says the stochastic NLL "averages eight deployed code draws per observation". Is this the average of , or of the averaged density? If the latter, the stochastic arm is being scored as an ensemble and Fig. 2 should be re-reported with single-draw NLL.
- Which rate estimate (MC or MADE) is used to match rates in Figs. 2–3, and do the H1 differences keep their sign and size under the other one? Could you add confidence intervals across seeds?
- Can you add a deterministic baseline trained against its actual entropy, and report the lower convex hull of the deterministic points so that the comparison is really against D^ts as in the theorem?
5 comments
For a stochastic summary, , meaning that it excludes the randomness that remains even when the data are known. For a deterministic summary , so , and hence the same measure is applied to both classes.
Proposition 3.1 further shows that the intrinsic rate can be understood entirely through the random posterior. Any summary channel can be replaced by one depending on the data only through their posterior, with the same posterior error and no greater rate. Dependence on differences between datasets yielding identical posteriors is unnecessary for approaching the optimum. Allowing positive error then lets us ask how much information is required to distinguish the posteriors themselves to the desired accuracy.
To summarize: classical minimal sufficiency identifies which distinctions every exactly sufficient summary must preserve. The intrinsic rate asks the quantitative counterpart: how much information must every summary preserve to achieve a specified posterior accuracy? In these terms, Theorem 4.2 says the intrinsic requirement cannot be met deterministically: at every interior accuracy, a deterministic summary or revealed mixture must retain strictly more information about the data than . Moreover, our matched-rate comparisons then empirically test which summaries preserve the posterior most accurately while retaining the same amount of data information. That characterization is useful independently of whether a smaller rate produces any computational savings. Furthermore, related work of Kawaguchi et al. (2023), Fischer and Alemi (2020) and Alemi et al., 2017) demonstrate generalization, robustness and calibration gains from limiting this information in settings like classification; we leave testing this for NPE as future work.
Q2. Eight code draws per observation
We apologise for the confusion. We compute average of over the eight draws. This is a Monte Carlo estimate of the single-draw loss that appears in Lemma 2.2. Densities are not averaged, so the stochastic arm is not scored as an ensemble. We will clarify this in Appendix K.2 of the updated manuscript.
W3 and Q3. Width of the rate bracket, and H1 under the adverse assignment
The rate bracket. In the submission, the upper rate estimate uses a fitted MADE as the marginal in the variational bound of Eq. (26). To address the reviewer's concern, we have rerun the comparison with a different marginal: the encoder's own channel averaged over about million independent simulated observations, . The bound is the same identity, , which holds for any fixed independently of the scored observations. This is similar to the leave-one-out bound of Poole et al. (2019) evaluated on a held-out dataset. The same rule is used for both stochastic and deterministic codes, for all trained models. The lower estimate is the Monte Carlo estimate of Section 5.1, as before. The new bracket has a median width of nats ( in the submission) and is at most nats (); for stochastic models, it is at most nats.
We now also report the stochastic vs deterministic comparison under two assignments of rate. In the point assignment, both codes are placed at the lower estimate. In the adverse assignment, each stochastic model is placed at its upper estimate and each deterministic model at its lower estimate. This is the pairing least favourable to our hypothesis. The table gives the matched-rate NLL difference, stochastic minus deterministic ( less is better) in nats, averaged over the three training seeds, with 95% intervals across seeds. (the t-interval with two degrees of freedom.)
| Simulator | Encoder | Adverse | Point |
|---|---|---|---|
| Bernoulli GLM | Deep Sets | -0.582 [-0.739, -0.424] | -0.589 [-0.747, -0.432] |
| Bernoulli GLM | Set Transformer | -0.513 [-0.617, -0.410] | -0.529 [-0.607, -0.451] |
| Bernoulli GLM | BiLSTM | -0.750 [-0.768, -0.732] | -0.759 [-0.778, -0.740] |
| g-and-k | Deep Sets | -0.325 [-0.354, -0.295] | -0.330 [-0.360, -0.300] |
| g-and-k | Set Transformer | -0.337 [-0.395, -0.280] | -0.342 [-0.401, -0.284] |
| g-and-k | BiLSTM | -0.425 [-0.457, -0.393] | -0.431 [-0.463, -0.399] |
| Lotka-Volterra | TCN | -1.155 [-1.257, -1.053] | -1.169 [-1.265, -1.073] |
| Lotka-Volterra | BiLSTM | -1.066 [-1.136, -0.997] | -1.075 [-1.141, -1.010] |
| Ricker | TCN | -0.829 [-0.862, -0.795] | -0.838 [-0.870, -0.805] |
| Ricker | BiLSTM | -0.821 [-0.849, -0.793] | -0.828 [-0.857, -0.799] |
Every interval lies below zero, and the adverse and point gaps differ by at most nats. Each training seed separately favors the stochastic arm in every cell.
We continue our response in the next comment.
W2 and Q4. A deterministic baseline trained against its entropy, compared on lower convex hulls
The baseline. We trained the baseline the reviewer describes. The fixed prior is replaced by a MADE prior on the code, trained jointly with the encoder and decoder. A deterministic model is charged on its transmitted code . For a hard code this is the cross-entropy of the code under the prior, whose average is , so the model is trained against a bound on its own entropy that is tight when the prior fits. A stochastic model is charged , whose average is . We ran this in four simulator-encoder combinations: the Bernoulli GLM with BiLSTM and Set Transformer encoders, and g-and-k with Deep Sets and Set Transformer encoders. We obtain are 28 stochastic and deterministic model over varying values of for each setting.
The comparison. For each setting, we take the lower convex hull, separately for deterministic and stochastic models, of their (rate, NLL) points. This is the time-sharing comparator of Theorem 4.2. We do the comparison under two assignments of rate. In the point assignment, both arms are placed at their lower rate estimate obtained by Monte Carlo (Section 5.1). In the adverse assignment, the stochastic arm is placed at its upper rate estimate and the deterministic arm at its lower estimate. This is the pairing least favourable to H1. The upper estimate is obtained by averageing the encoder's own channel over an independent simulated set as described above. The table gives the NLL difference, stochastic minus deterministic (, lower is better), in nats: its mean over the grid and, for the adverse assignment, also its value closest to zero.
| Cell | Matched rates (nats) | Point, mean | Adverse, mean | Adverse, closest to zero |
|---|---|---|---|---|
| Bernoulli GLM, BiLSTM | 2.34 to 8.38 | -0.55 | -0.47 | -0.38 |
| Bernoulli GLM, Set Transformer | 2.57 to 5.73 | -0.63 | -0.58 | -0.51 |
| g-and-k, Deep Sets | 1.02 to 4.88 | -0.25 | -0.23 | -0.14 |
| g-and-k, Set Transformer | 1.54 to 5.88 | -0.20 | -0.19 | -0.12 |
The difference is negative at every matched rate, in all four cells and under both assignments.
We will reconcile all revised experiments with the methods text, including the rate estimators and reference-set sizes, the H1 aggregation convention and numerical results, and the seed count and protocol for the learned-prior.
Thank you again for your detailed engagement with both the theoretical contribution and the experimental methodology. We particularly appreciate the opportunity to clarify the statistical motivation for information rate and the interpretation of our comparisons. We hope our responses address your concerns; please let us know whether any points require further explanation or supporting evidence.
We thank the reviewer for a precise and constructive review. It has helped us articulate the motivation of our paper more explicitly and we will include the following changes in the revised manuscript.
W1 and Q1. What does I(Y;S) measure, and why compare summaries on this axis?
Our use of “budget” and “cost” suggested that the rate I(Y;S) is a physical resource, such as the cost of sending a message or of computation or storage. That’s useful terminology from the information theory literature, but potentially misleading about why it's our point of focus, for which we apologize. Here, ‘rate’ measures the statistical information that the summary retains about the data; we do not identify it with the one-shot cost of storing or transmitting the summary.
Retaining the entire dataset already incurs no posterior information loss, but tells us nothing about how much information was actually needed. Minimizing retained information I(Y;S) subject to a specified accuracy lets us formally characterize how much data information any adequate summary must preserve. The question matters independently of any savings in computation or storage.
Here’s a simple example for intuition. Suppose we observe a sequence of IID coin flips and want to infer θ=pheads. The total count of heads is a sufficient statistic for θ: once we know this count, conditioning on it gives the same posterior as conditioning on the full sequence. But the whole sequence itself is also sufficient. So is the count accompanied by the first flip, the longest run of heads, or any other feature of the sequence, all of which become redundant for posterior preservation once the count is known. Sufficiency alone does not tell us which distinctions among datasets are necessary to preserve accurate inference. It guarantees that no parameter-relevant information has been lost, but does not prevent a summary from retaining redundant details about the sampling realization.
The count of heads, however, also has a much stronger property of minimality. Every sufficient statistic must contain enough information to recover it. Formally, a minimal sufficient statistic Tmin is a measurable function of every sufficient statistic T, up to null sets. Data processing inequality then gives I(Y;Tmin)≤I(Y;T). Classical minimality thus implies an information-economy principle: preserve the inferential content while retaining no unnecessary data distinctions among datasets. This information-minimal interpretation of sufficiency, and its relaxation through the information bottleneck, are established; see Shamir, Sabato, and Tishby (2010, Section 5). We use this perspective to study the intrinsic requirements of approximate posterior preservation.
Our paper allows for a small posterior error, which raises a quantitative question that exact sufficiency alone does not answer. Exact preservation requires distinguishing any two datasets that yield different posteriors, however small the difference. The issue is especially clear in the Gaussian location model from the paper. The sample mean preserves everything needed for inference about the mean parameter. But how accurately must that sample mean be retained? Two values differing only in the tenth decimal place yield different posteriors, so exact preservation requires distinguishing those two values. But ignoring that difference may have a negligible inferential effect. Moreover, the required precision changes as more data arrive. The posterior narrows, and a rounding error that was previously negligible can become substantial relative to posterior concentration. Therefore, knowing which statistic is sufficient doesn’t by itself determine the precision needed to meet a prescribed posterior accuracy.
The intrinsic rate R⋆(ε)=infΔKL(S)≤εI(Y;S) answers exactly this quantitative question: what is the least information about the data that a summary must retain to keep expected posterior error below ε? Every summary meeting that tolerance must retain at least this much information, regardless of its architecture or construction. Lemma 2.2 further expresses this as the information bottleneck of the Bayesian experiment in posterior-error coordinates.
This also gets to the heart of why we measure retained information by I(Y;S), rather than the entropy or number of possible values of S. We want to measure how much the summary reveals about the data. But measures like entropy and cardinality would count variation in the summary that reveals nothing about the data. For example, appending independent random bits to a summary increases both the entropy and the number of possible summary values, while leaving its information about the data unchanged.
We continue our response in the next comment.