acceptodds
Under review as a conference paper at ICLR 2027

Intrinsic Information Rates of Summaries in Neural Posterior Estimation

Abstract

Neural posterior estimation (NPE) often encodes observations into a fixed-dimensional summary , with the neural posterior conditioning only on . Any such summary imposes a finite information budget on inference, whereas exact preservation requires infinite information whenever posteriors vary continuously with the data. The budget typically emerges from architecture and training rather than an explicit measurement or choice. Better performance may therefore reflect better encoding or simply a larger budget. In this paper, we make the budget explicit and ask two questions: how large must it be, and how best is it spent? To answer them, we model the summary as a (possibly stochastic) channel from to . For the first question, we show that this budget, measured by the mutual information , grows unboundedly as the posterior concentrates: in regular -dimensional models, bounded posterior distortion requires nats, even when a fixed-dimensional sufficient statistic exists. More generally, the required budget is set by the distribution of the random posterior, and we bound it from above by quantizing posteriors and from below by converses that weight each posterior state by its probability. For the second question, we show that a finite budget is best spent stochastically. Under a common-support condition on the simulator, which covers the continuous observation spaces typical of simulators, a deterministic summary necessarily loses something: at every positive budget too small for exact preservation, some stochastic summary achieves _strictly_ lower distortion than every deterministic summary and every time-shared mixture of deterministic summaries within the same budget. Motivated by these results, we insert a binary bottleneck into NPE, measure each trained summary's rate, and compare summaries at matched rate, which removes the budget as a confounder in architecture comparisons. Across four simulator benchmarks and several encoder families, stochastic bottlenecks achieve favorable rate–loss tradeoffs, and architecture rankings change with the budget.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
neural posterior estimation, bayesian computation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_4aZmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats the summary network in NPE as a channel Y→SY \to S with rate I(Y;S)I(Y;S), and measures distortion as the expected KL from the full-data posterior to the summary posterior. By the standard information-bottleneck identity this frontier is the IB curve of the Bayesian experiment, written in posterior-error coordinates. Sections 2–3 collect bounds on the resulting intrinsic rate (a d/2⋅log⁡nd/2\cdot\log n parametric converse via Clarke–Barron, posterior quantisation, a small-ball converse, the exact Gaussian frontier). The main result, Theorem 4.2, shows that under mutual absolute continuity of the sampling laws, stochastic summaries achieve strictly lower distortion than any deterministic summary or time-sharing mixture of deterministic summaries, at every interior rate. The proof uses "hidden relabelling": with small probability tt, replace the code's output by an independent draw from its own marginal, which saves order tlog⁡(1/t)t \log(1/t) nats of I(Y;S)I(Y;S) at a distortion cost of o(tlog⁡1/t)o(t \log 1/t). Empirically, the authors insert a 128-bit Bernoulli bottleneck into NPE, estimate each model's rate, and compare at matched rate: a stochastic channel beats the exact deterministic hull on an enumerable Beta–Binomial example, stochastic bottlenecks beat annealed deterministic ones by 0.17–0.62 nats NLL on four simulators, and encoder rankings change with rate.

Critical strengths
  • Theorem 4.2 is the real contribution and it is done well. The hidden-relabelling construction is elegant, the uniform control of the collision gap avoids any attainment assumption, and the result holds for general standard-Borel experiments, which is cleaner than the usual finite-alphabet arguments.
  • The scoping is careful. Appendix D gives examples showing exactly where each assumption is needed (no gap for deterministic targets, no gap for overlapping-posterior erasure models, the difference between an entropy constraint and a rate constraint), and Prop. B.8 rules out the gap coming from different initial slopes. The limitations paragraph is candid that this is a one-shot result.
  • The distinction between summary dimension, code cardinality and information rate (Thm 3.2 vs Thm 4.2) is a useful one for the SBI community, as is the remark that posterior fidelity alone cannot distinguish UU from (U,V)(U, V) when VV is irrelevant.
  • The Beta–Binomial benchmark is a good controlled check: all 21,147 count partitions are enumerated, so the deterministic hull is exact, and the decoder error (≈4⋅10−4\approx4·10^{-4} nats) is reported and sits well below the measured gap.
Critical weaknesses
  1. The paper never says what I(Y;S) is the cost of, so it is unclear what the main theorem buys in practice. The whole result turns on measuring the summary's "budget" as mutual information rather than as output entropy. In a one-shot setting, producing a stochastic summary actually costs more than I(Y;S)I(Y;S), whereas a deterministic code costs H(T) up to a bit, so the two frontiers compared in Theorem 4.2 are not measuring the same thing. And in NPE there is no channel at all: the decoder sees S directly. The introduction's motivation, that the information budget confounds comparisons between summary architectures, is never demonstrated. Standard NPE summaries are deterministic and continuous, so their I(Y;S)I(Y;S) is infinite (or H(Y) at float precision), and the paper never measures an "effective budget" for an ordinary encoder.

  2. The simulator experiments mix the theoretical effect with differences in how the two arms are trained. The stochastic arm is trained and deployed the same way. The deterministic arm is trained stochastically, annealed to temperature 0.1, then hard-thresholded with straight-through gradients, and its rate penalty is, for hard codes, a loose factorised stand-in for the true entropy H(B). So the deterministic models are optimised against the wrong objective. The size of the gaps is telling: 0.17–0.62 nats on the simulators, against at most 0.036 nats in the one case where the gap can be computed exactly. A difference ten times larger than the theory produces is more likely to come from training than from Theorem 4.2. A deterministic baseline trained against its actual entropy (with a learned prior on B, as neural compression does) and the lower convex hull over β\beta would make H1 convincing.

  3. "Matched rate" depends on rate estimates that are shaky, and the results have no error bars. The Monte Carlo rate estimate is capped at log⁡M≈12.48\log M \approx 12.48 nats, so every operating point sits below about 10 nats and everything above the cap is thrown away. In Fig. 2, the gap between the MC and MADE estimates for the same model is, at higher rates, about as large as the horizontal shift between the stochastic and deterministic curves, and the paper does not say which estimate it uses to match. There are three seeds, the headline H1 statistic comes without a confidence interval, and the Fig. 6 diagnostic is a single seed.

Questions for authors
  1. What resource does I(Y;S) represent in NPE, where the decoder receives S directly? Is there any downstream benefit (generalisation from a finite simulation budget, robustness, calibration) from operating at lower I(Y;S) at fixed NLL?
  2. App. K.2 says the stochastic NLL "averages eight deployed code draws per observation". Is this the average of −log⁡q(θ∣bk)−\log q(\theta|b_k), or −log⁡−\log of the averaged density? If the latter, the stochastic arm is being scored as an ensemble and Fig. 2 should be re-reported with single-draw NLL.
  3. Which rate estimate (MC or MADE) is used to match rates in Figs. 2–3, and do the H1 differences keep their sign and size under the other one? Could you add confidence intervals across seeds?
  4. Can you add a deterministic baseline trained against its actual entropy, and report the lower convex hull of the deterministic points so that the comparison is really against D^ts as in the theorem?
5 comments

We thank the reviewer for a precise and constructive review. It has helped us articulate the motivation of our paper more explicitly and we will include the following changes in the revised manuscript.

W1 and Q1. What does I(Y;S)I(Y;S) measure, and why compare summaries on this axis?

Our use of “budget” and “cost” suggested that the rate I(Y;S)I(Y;S) is a physical resource, such as the cost of sending a message or of computation or storage. That’s useful terminology from the information theory literature, but potentially misleading about why it's our point of focus, for which we apologize. Here, ‘rate’ measures the statistical information that the summary retains about the data; we do not identify it with the one-shot cost of storing or transmitting the summary.

Retaining the entire dataset already incurs no posterior information loss, but tells us nothing about how much information was actually needed. Minimizing retained information I(Y;S)I(Y;S) subject to a specified accuracy lets us formally characterize how much data information any adequate summary must preserve. The question matters independently of any savings in computation or storage.

Here’s a simple example for intuition. Suppose we observe a sequence of IID coin flips and want to infer θ=pheads\theta=p_{\mathrm{heads}}. The total count of heads is a sufficient statistic for θ\theta: once we know this count, conditioning on it gives the same posterior as conditioning on the full sequence. But the whole sequence itself is also sufficient. So is the count accompanied by the first flip, the longest run of heads, or any other feature of the sequence, all of which become redundant for posterior preservation once the count is known. Sufficiency alone does not tell us which distinctions among datasets are necessary to preserve accurate inference. It guarantees that no parameter-relevant information has been lost, but does not prevent a summary from retaining redundant details about the sampling realization.

The count of heads, however, also has a much stronger property of minimality. Every sufficient statistic must contain enough information to recover it. Formally, a minimal sufficient statistic Tmin⁡T_{\min} is a measurable function of every sufficient statistic TT, up to null sets. Data processing inequality then gives I(Y;Tmin⁡)≤I(Y;T)I(Y;T_{\min})\le I(Y;T). Classical minimality thus implies an information-economy principle: preserve the inferential content while retaining no unnecessary data distinctions among datasets. This information-minimal interpretation of sufficiency, and its relaxation through the information bottleneck, are established; see Shamir, Sabato, and Tishby (2010, Section 5). We use this perspective to study the intrinsic requirements of approximate posterior preservation.

Our paper allows for a small posterior error, which raises a quantitative question that exact sufficiency alone does not answer. Exact preservation requires distinguishing any two datasets that yield different posteriors, however small the difference. The issue is especially clear in the Gaussian location model from the paper. The sample mean preserves everything needed for inference about the mean parameter. But how accurately must that sample mean be retained? Two values differing only in the tenth decimal place yield different posteriors, so exact preservation requires distinguishing those two values. But ignoring that difference may have a negligible inferential effect. Moreover, the required precision changes as more data arrive. The posterior narrows, and a rounding error that was previously negligible can become substantial relative to posterior concentration. Therefore, knowing which statistic is sufficient doesn’t by itself determine the precision needed to meet a prescribed posterior accuracy.

The intrinsic rate R⋆(ε)=inf⁡ΔKL(S)≤εI(Y;S)R^\star(\varepsilon)=\inf_{\Delta_{\mathrm{KL}}(S)\le\varepsilon}I(Y;S) answers exactly this quantitative question: what is the least information about the data that a summary must retain to keep expected posterior error below ε\varepsilon? Every summary meeting that tolerance must retain at least this much information, regardless of its architecture or construction. Lemma 2.2 further expresses this as the information bottleneck of the Bayesian experiment in posterior-error coordinates.

This also gets to the heart of why we measure retained information by I(Y;S)I(Y;S), rather than the entropy or number of possible values of SS. We want to measure how much the summary reveals about the data. But measures like entropy and cardinality would count variation in the summary that reveals nothing about the data. For example, appending independent random bits to a summary increases both the entropy and the number of possible summary values, while leaving its information about the data unchanged.

We continue our response in the next comment.

For a stochastic summary, I(Y;S)=H(S)−H(S∣Y)I(Y;S)=H(S)-H(S\mid Y), meaning that it excludes the randomness that remains even when the data are known. For a deterministic summary H(S∣Y)=0H(S\mid Y) = 0, so I(Y;S)=H(S)I(Y;S) = H(S), and hence the same measure is applied to both classes.

Proposition 3.1 further shows that the intrinsic rate can be understood entirely through the random posterior. Any summary channel can be replaced by one depending on the data only through their posterior, with the same posterior error and no greater rate. Dependence on differences between datasets yielding identical posteriors is unnecessary for approaching the optimum. Allowing positive error then lets us ask how much information is required to distinguish the posteriors themselves to the desired accuracy.

To summarize: classical minimal sufficiency identifies which distinctions every exactly sufficient summary must preserve. The intrinsic rate asks the quantitative counterpart: how much information must every summary preserve to achieve a specified posterior accuracy? In these terms, Theorem 4.2 says the intrinsic requirement cannot be met deterministically: at every interior accuracy, a deterministic summary or revealed mixture must retain strictly more information about the data than R⋆(ϵ)R^\star(\epsilon). Moreover, our matched-rate comparisons then empirically test which summaries preserve the posterior most accurately while retaining the same amount of data information. That characterization is useful independently of whether a smaller rate produces any computational savings. Furthermore, related work of Kawaguchi et al. (2023), Fischer and Alemi (2020) and Alemi et al., 2017) demonstrate generalization, robustness and calibration gains from limiting this information in settings like classification; we leave testing this for NPE as future work.

Q2. Eight code draws per observation

We apologise for the confusion. We compute average of −log⁡qϕ(θ∣bk)-\log q_\phi(\theta\mid b_k) over the eight draws. This is a Monte Carlo estimate of the single-draw loss E(−log⁡qϕ(θ∣B))\mathbb E(-\log q_\phi(\theta\mid B)) that appears in Lemma 2.2. Densities are not averaged, so the stochastic arm is not scored as an ensemble. We will clarify this in Appendix K.2 of the updated manuscript.

W3 and Q3. Width of the rate bracket, and H1 under the adverse assignment

The rate bracket. In the submission, the upper rate estimate uses a fitted MADE as the marginal rr in the variational bound of Eq. (26). To address the reviewer's concern, we have rerun the comparison with a different marginal: the encoder's own channel averaged over about 4.24.2 million independent simulated observations, r(b)=1N∑jq(b∣yj′)r(b)=\frac1N\sum_{j} q(b\mid y_j'). The bound is the same identity, E(−log⁡r(B))−H(B∣Y)=I(Y;B)+KL(pB∥r)≥I(Y;B)\mathbb E(-\log r(B)) - H(B\mid Y) = I(Y;B) + \mathrm{KL}(p_B\Vert r) \ge I(Y;B), which holds for any rr fixed independently of the scored observations. This is similar to the leave-one-out bound of Poole et al. (2019) evaluated on a held-out dataset. The same rule is used for both stochastic and deterministic codes, for all trained models. The lower estimate is the Monte Carlo estimate of Section 5.1, as before. The new bracket has a median width of 0.0100.010 nats (0.350.35 in the submission) and is at most 0.390.39 nats (2.362.36); for stochastic models, it is at most 0.120.12 nats.

We now also report the stochastic vs deterministic comparison under two assignments of rate. In the point assignment, both codes are placed at the lower estimate. In the adverse assignment, each stochastic model is placed at its upper estimate and each deterministic model at its lower estimate. This is the pairing least favourable to our hypothesis. The table gives the matched-rate NLL difference, stochastic minus deterministic (↓\downarrow less is better) in nats, averaged over the three training seeds, with 95% intervals across seeds. (the t-interval with two degrees of freedom.)

SimulatorEncoderAdversePoint
Bernoulli GLMDeep Sets-0.582 [-0.739, -0.424]-0.589 [-0.747, -0.432]
Bernoulli GLMSet Transformer-0.513 [-0.617, -0.410]-0.529 [-0.607, -0.451]
Bernoulli GLMBiLSTM-0.750 [-0.768, -0.732]-0.759 [-0.778, -0.740]
g-and-kDeep Sets-0.325 [-0.354, -0.295]-0.330 [-0.360, -0.300]
g-and-kSet Transformer-0.337 [-0.395, -0.280]-0.342 [-0.401, -0.284]
g-and-kBiLSTM-0.425 [-0.457, -0.393]-0.431 [-0.463, -0.399]
Lotka-VolterraTCN-1.155 [-1.257, -1.053]-1.169 [-1.265, -1.073]
Lotka-VolterraBiLSTM-1.066 [-1.136, -0.997]-1.075 [-1.141, -1.010]
RickerTCN-0.829 [-0.862, -0.795]-0.838 [-0.870, -0.805]
RickerBiLSTM-0.821 [-0.849, -0.793]-0.828 [-0.857, -0.799]

Every interval lies below zero, and the adverse and point gaps differ by at most 0.0160.016 nats. Each training seed separately favors the stochastic arm in every cell.

We continue our response in the next comment.

W2 and Q4. A deterministic baseline trained against its entropy, compared on lower convex hulls

The baseline. We trained the baseline the reviewer describes. The fixed prior is replaced by a MADE prior rωr_\omega on the code, trained jointly with the encoder and decoder. A deterministic model is charged β(−log⁡rω(b))\beta(-\log r_\omega(b)) on its transmitted code bb. For a hard code this is the cross-entropy of the code under the prior, whose average is H(B)+KL(pB∥rω)H(B)+\mathrm{KL}(p_B\Vert r_\omega), so the model is trained against a bound on its own entropy that is tight when the prior fits. A stochastic model is charged βKL(q(⋅∣y)∥rω)\beta \mathrm{KL}(q(\cdot\mid y)\Vert r_\omega), whose average is I(Y;B)+KL(pB∥rω)I(Y;B)+\mathrm{KL}(p_B\Vert r_\omega). We ran this in four simulator-encoder combinations: the Bernoulli GLM with BiLSTM and Set Transformer encoders, and g-and-k with Deep Sets and Set Transformer encoders. We obtain are 28 stochastic and deterministic model over varying values of β\beta for each setting.

The comparison. For each setting, we take the lower convex hull, separately for deterministic and stochastic models, of their (rate, NLL) points. This is the time-sharing comparator of Theorem 4.2. We do the comparison under two assignments of rate. In the point assignment, both arms are placed at their lower rate estimate obtained by Monte Carlo (Section 5.1). In the adverse assignment, the stochastic arm is placed at its upper rate estimate and the deterministic arm at its lower estimate. This is the pairing least favourable to H1. The upper estimate is obtained by averageing the encoder's own channel over an independent simulated set as described above. The table gives the NLL difference, stochastic minus deterministic (↓\downarrow, lower is better), in nats: its mean over the grid and, for the adverse assignment, also its value closest to zero.

CellMatched rates (nats)Point, meanAdverse, meanAdverse, closest to zero
Bernoulli GLM, BiLSTM2.34 to 8.38-0.55-0.47-0.38
Bernoulli GLM, Set Transformer2.57 to 5.73-0.63-0.58-0.51
g-and-k, Deep Sets1.02 to 4.88-0.25-0.23-0.14
g-and-k, Set Transformer1.54 to 5.88-0.20-0.19-0.12

The difference is negative at every matched rate, in all four cells and under both assignments.

We will reconcile all revised experiments with the methods text, including the rate estimators and reference-set sizes, the H1 aggregation convention and numerical results, and the seed count and protocol for the learned-prior.

Thank you again for your detailed engagement with both the theoretical contribution and the experimental methodology. We particularly appreciate the opportunity to clarify the statistical motivation for information rate and the interpretation of our comparisons. We hope our responses address your concerns; please let us know whether any points require further explanation or supporting evidence.

Reviewer_8osiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the information budget required for summaries in neural posterior estimation (NPE). The information retained by a summary is measured by the rate I(Y;S)I(Y;S), while information loss is quantified by the expected KL divergence between the full-data and summary-based posteriors. The authors study two quantities: the intrinsic rate R⋆(ε)R^\star(\varepsilon) required to achieve a given distortion tolerance and the minimum distortion D⋆(R)D^\star(R) under a fixed information budget.

For regular dd-dimensional parametric models, the authors establish a lower bound R⋆(ε)≥dlog⁡(n)/2−O(1)R^\star(\varepsilon) \ge d\log(n)/2-O(1). They further characterize information requirements using posterior quantization and probability-weighted converse bounds. Their main theoretical result shows that, under mutual absolute continuity and finite, positive Bayesian information, stochastic summaries can achieve strictly lower distortion than the deterministic time-sharing envelope.

To validate these findings, the authors introduce binary bottlenecks into NPE and compare stochastic and deterministic summaries at matched estimated information rates. Experiments on four simulator benchmarks demonstrate favorable rate–NLL trade-offs for stochastic summaries and illustrate the importance of accounting for information budgets when comparing architectures.

Critical strengths
  1. The main theoretical contribution provides a meaningful characterization of the advantage of stochastic summaries. In particular, Theorem 4.2 establishes a strict separation between stochastic summaries and the deterministic time-sharing frontier under a general common-support assumption. This provides a rigorous theoretical motivation for considering probabilistic summaries under mutual information constraints.
  2. The information-theoretic perspective offers a useful framework for evaluating summaries by distinguishing representation dimensionality from the information actually retained. The posterior quantization and converse results further clarify how the distribution of possible posterior distributions determines the required information rate.
Critical weaknesses
  1. The connection between the general theoretical results and the proposed BNPE architecture could be strengthened. Theorem 4.2 establishes a strict advantage of stochastic summarization over the deterministic time-sharing frontier by considering unrestricted stochastic channels. However, BNPE restricts the encoder to a finite-dimensional, conditionally independent Bernoulli channel. It remains unclear whether the stochastic channels underlying Theorem 4.2 can be represented within this restricted architecture. Moreover, while the theorem guarantees a strict separation, it does not generally quantify the magnitude of the gap. Clarifying whether the theoretical advantage persists under the architectural constraints of BNPE, and to what extent it is practically meaningful, would strengthen the connection between theory and experiments.

Minor comments: There appear to be typos in L126 (“by.”) and L281 (“,in”). In addition, the source code was not provided as a supplement.

Questions for authors
  1. As shown in Lemma 2.2, the evaluation NLL can be decomposed into information loss ΔKL(B)\Delta_{\mathrm{KL}}(B) and decoder approximation error, apart from the common conditional entropy term. Could the authors examine the respective contributions of these two terms to the observed NLL improvements? In particular, it would be helpful to understand whether the gains arise primarily from better information preservation or reduced decoder approximation error.

  2. This paper establishes a strict theoretical advantage of stochastic summarization under mutual information constraints. Rubenstein et al. (2018) also discuss the benefits of probabilistic encoders, although their motivation concerns latent dimensionality mismatch and aggregate posterior consistency rather than posterior information preservation. Could the authors elaborate on how these two perspectives are related? In particular, it would be interesting to understand whether the proposed theoretical framework offers insights into the design of stochastic encoders for autoencoders, such as Wasserstein Autoencoders, and what additional assumptions or modifications to the objective would be needed to extend the analysis to this setting.

    Reference: Rubenstein et al. (2018), On the Latent Space of Wasserstein Auto-Encoders, arXiv:1802.03761.

2 comments

We thank the reviewer for the detailed feedback and for the questions, which helped us tighten the link between the theory and the BNPE protocol.

W1, first part. Can the stochastic channels of Theorem 4.2 be represented by the BNPE channel?

We would like to clarify that Theorem 4.2 is an existence result over all channels. Our proof gives one stochastic construction that improves upon the entire deterministic time-sharing envelope. It need not be the only one, and therefore the fact that this construction is not a conditionally independent Bernoulli channel does not rule out a gain within BNPE. We introduce the BNPE protocol so that estimating the information rate and comparing summaries at matched rates becomes possible. We also agree that Theorem 4.2 does not by itself say that a conditionally independent Bernoulli channel in BNPE attains the gain, and therefore we argue that it in fact complements Theorem 4.2 to show that even within the restricted BNPE class, stochastic summaries are better.

The count benchmark shows this both analytically and empirically: the explicit witness of Appendix C is a one-bit Bernoulli channel, which is inside the BNPE class, and it lies below the exact envelope over all 21,147 deterministic encoders (0.1545 against 0.1623 at 0.5 nats). The trained stochastic BNPE models in Figure 1 also lie below that exact envelope. On the simulators, where the full envelope cannot be enumerated, the stochastic BNPE models have a lower loss than the trained deterministic ones at every matched rate.

W1, second part. Magnitude of the separation

While we agree that Theorem 4.2 itself is qualitative, we do quantify the gap in Corollary B.7 of the submission when the lautum information is finite:

A lower bound on the gap (Corollary B.7). For finite Lautum information L:=∫KL(π∥Πy)PY(dy)\mathcal L := \int \textrm{KL}(\pi \| \Pi_y) P_Y(dy) of the experiment (Definition B.3), Corollary B.7 gives: for every t∈(0,1/2]t\in(0,1/2] with log⁡(1/t)≥2(I+L)/c\log(1/t)\ge 2(I+L)/c,

D⋆(R)≤Dts(R)−c2tlog⁡(1/t),D^\star(R) \le D^{\mathrm{ts}}(R) - \frac{c}{2} t\log(1/t),

where I=I(θ;Y)I=I(\theta;Y) and c>0c\gt 0 is the constant constructed in that proof of Theorem 4.2. The condition holds in the Gaussian experiment. The bound certifies a quantitative positive gap.

Count benchmark. The benchmark further quantifies the practical gap. The largest gap is 0.0360.036 nats, about 12 percent of I(θ;C)=0.293I(\theta;C)=0.293 nats.

Q1. Contribution of the decoder error

We agree with the reviewer that the difference in NLL does include the decoder error. For a code BB with decoder qϕq_\phi, Lemma 2.2 gives

L=H(θ∣Y)+ΔKL(B)+EKL(ΠB∥qϕ(⋅∣B)).L = H( \theta\mid Y) + \Delta_{\mathrm{KL}}(B) + \mathbb E \mathrm{KL}(\Pi_B \Vert q_\phi(\cdot\mid B)).

To address this, we present the evidence from our count and Gaussian benchmarks presented in the paper, showing that the emprical gains of the stochastic code are indeed not confounded by the decoder error. We also ran experiments with larger decoders to isolate this effect further for the simulated datasets.

  1. Exact evidence from the count benchmark. Our count benchmark computes the information loss in closed form and obtains the deterministic time-sharing envelope exactly by enumerating all 21,147 deterministic count encoders. We show that the stochastic channel lies below it by up to 0.0360.036 nats, which is about 12 percent of the available information I(θ;C)=0.293I(\theta;C)=0.293 nats. Furthermore, the decoder gap of our trained models is at most 4×10−44\times 10^{-4} nats. Please see the new Figure 6. Therefore, this experimentis a direct test of our Theorem 4.2.

  2. Evidence under a restricted decoder family (Gaussian diagnostic, Appendix K.4). While the true summary-conditioned posterior is a mixture of Gaussians in this benchmark, we restrict our decoder to a Gaussian with known posterior variance. Equation (27) then splits the loss exactly into a term that depends on the code alone and the extra loss of the learned decoder from the analytic optimal decoder. The new Figure 7(b) in Appendix K.4 shows that the decoder error term is at most 0.0330.033 nats for any of the trained models, while the smallest gap between the stochastic and deterministic codes is 0.450.45 nats. The gain in this experiment is therefore a property of the code.

  3. Experiments on trained families with a larger decoder. To further isolate the error contributions due to limited decoder capacity, we froze our trained encoders and bottlenecks in the simulated experiments, and refit only the decoder with 35×35 \times more parameters. Freezing the bottleneck fixes the code law, so the rate and the information loss ΔKL(B)\Delta_{\mathrm{KL}}(B) cannot change. By Lemma 2.2, any change in NLL can be directly attributed to the decoder error term. We report this for 18 BNPE models: both stochastic and deterministic models at a low, a middle and a high rate, for each of the Bernoulli GLM, g-and-k and Ricker. Please see the new Figure 8.

We continue our response in the next comment.

The table compares stochastic and deterministic codes with the original and with the large decoder. Entries are NLL differences, stochastic minus deterministic (↓\downarrow less is better), in nats.

Three rows, marked †\dagger, match the rate exactly by time-sharing the deterministic summaries (Proposition A.3).

Simulator (encoder)Rate, stochasticRate, deterministicOriginal decoderLarge decoder
Bernoulli GLM (BiLSTM)4.804.33-1.06-1.01
Bernoulli GLM (BiLSTM)4.804.80 †^\dagger-0.81-0.75
Bernoulli GLM (BiLSTM)8.598.28-0.98-0.88
g-and-k (Deep Sets)3.553.85-0.32-0.30
g-and-k (Deep Sets)3.553.55 †^\dagger-0.47-0.45
g-and-k (Deep Sets)6.866.42-0.19-0.18
Ricker (BiLSTM)4.854.89-0.86-0.82
Ricker (BiLSTM)4.854.85 †^\dagger-0.89-0.86
Ricker (BiLSTM)8.528.40-1.03-0.81

The stochastic code keeps a lower loss under the large decoder in all nine comparisons. At exactly matched rate, the difference is −0.75-0.75 nats for the Bernoulli GLM, −0.45-0.45 for g-and-k and −0.86-0.86 for Ricker. Hence, the gains largely do not come from limited decoder capacity.

Q2. Relation to Rubenstein et al. (2018) and autoencoders

Rubenstein et al. motivate stochastic WAE encoders through latent-dimensionality mismatch: when the intrinsic dimension of the data manifold is smaller than the latent dimension of the encoded code, randomness can populate redundant directions and improve matching between the aggregate code distribution and the prior. However, their log-variance regularization, introduced to discourage collapse toward deterministic encoders, can be formulated as an information-rate regularization term under additional assumptions.

A possible connection via their regularizers. In their notation, let XX denote the input (our YY), ZZ its code (our summary SS). Denote by q(z∣X)q(z\mid X) the encoder, qZq_Z its aggregate distribution, and pZp_Z a reference prior. The variational identity underlying our training penalty is

EXKL(q(⋅∣X)∥pZ)=I(X;Z)⏟retained information+KL(qZ∥pZ)⏟aggregate mismatch.\mathbb E_X\mathrm{KL}(q(\cdot\mid X)\Vert p_Z) = \underbrace{I(X;Z)} _{\text{retained information}} + \underbrace{\mathrm{KL}(q_Z\Vert p_Z)} _{\text{aggregate mismatch}}.

Thus our penalty combines information-rate regularization with aggregate KL matching. Rubenstein et al.’s WAE-MMD objective instead penalizes aggregate mismatch through MMD, supplemented by their log-variance penalty.

For their diagonal-Gaussian encoder family, denote the conditional variances by σi2(x)\sigma_i^2(x). Under exact aggregate matching, i.e., qZ=pZ=N(0,Id)q_Z=p_Z=\mathcal N(0,I_d), and finite entropy expressions, we obtain

I(X;Z)=−12EX∑i=1dlog⁡σi2(X).I(X;Z)=-\frac12\mathbb E_X\sum_{i=1}^{d}\log\sigma_i^2(X).

If additionally 0<σi2(X)≤10\lt\sigma_i^2(X)\le 1, their L1L_1 log-variance penalty satisfies

λ1EX∑i∣log⁡σi2(X)∣=2λ1I(X;Z).\lambda_1\mathbb E_X\sum_i\lvert\log\sigma_i^2(X)\rvert = 2\lambda_1 I(X;Z).

This derivation gives their regularizer an information-rate interpretation.

Reconstruction versus posterior preservation. There is also an important distinction worth mentioning. Their objective concerns reconstruction of the input, however, ours preserves the posterior of a parameter θ\theta inferred from observations YY. Our loss measures the parameter information discarded when replacing YY with a summary SS. Thus the two works provide complementary motivations for stochastic encoding: aggregate-prior consistency and reconstruction on one hand, and efficient posterior preservation on the other.

Extensions and future work. Applying our theorem to an autoencoder would require a fixed latent target θ\theta with a noisy observation model (satisfying common support), thereby defining the posterior-preservation loss. These connections motivate controlling retained information alongside reconstruction and aggregate matching could help probabilistic encoders, which is also consistent with the information-theoretic rate–distortion perspective on VAEs developed by Alemi et al. (2018). Establishing the corresponding guarantees and evaluating their effects on generation and representation learning are promising directions for future work. We thank the reviewer for suggesting this connection and we will add the discussion to the final manuscript.

References

Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., & Tucker, G. (2019, May). On variational bounds of mutual information. In International conference on machine learning (pp. 5171-5180). PMLR.

Rubenstein et al. (2018), On the Latent Space of Wasserstein Auto-Encoders, arXiv:1802.03761.

Thank you again for your thoughtful review and for suggesting the connection to stochastic autoencoders. We will happily release the code along with the camera ready manuscript upon acceptance. We would also welcome any further questions and would be happy to clarify any points that remain unresolved.

Comments on the paper

3 comments

We thank the reviewer for the careful reading and for the positive assessment of Theorem 4.2 and of the count experiment. We address the reviewer's concerns about the simulator experiments below.

W1. Contribution of the decoder error

We agree with the reviewer that the difference in NLL may include the decoder error. For a code BB with decoder qϕq_\phi, Lemma 2.2 gives

L=H(θ∣Y)+ΔKL(B)+EKL(ΠB∥qϕ(⋅∣B)).L = H( \theta\mid Y) + \Delta_{\mathrm{KL}}(B) + \mathbb E \mathrm{KL}(\Pi_B \Vert q_\phi(\cdot\mid B)).

To address this, we present the evidence from our count and Gaussian benchmarks presented in the paper, showing that the emprical gains of the stochastic code are indeed not confounded by the decoder error. We also ran experiments with larger decoders to isolate this effect further for the simulated datasets.

  1. Exact evidence from the count benchmark. Our count benchmark computes the information loss in closed form and obtains the deterministic time-sharing envelope exactly by enumerating all 21,147 deterministic count encoders. We show that the stochastic channel lies below it by up to 0.0360.036 nats, which is about 12 percent of the available information I(θ;C)=0.293I(\theta;C)=0.293 nats. Furthermore, the decoder gap of our trained models is at most 4×10−44\times 10^{-4} nats. Please see the new Figure 6. Therefore, this experimentis a direct test of our Theorem 4.2.

  2. Evidence under a restricted decoder family (Gaussian diagnostic, Appendix K.4). While the true summary-conditioned posterior is a mixture of Gaussians in this benchmark, we restrict our decoder to a Gaussian with known posterior variance. Equation (27) then splits the loss exactly into a term that depends on the code alone and the extra loss of the learned decoder from the analytic optimal decoder. The new Figure 7(b) in Appendix K.4 shows that the decoder error term is at most 0.0330.033 nats for any of the trained models, while the smallest gap between the stochastic and deterministic codes is 0.450.45 nats. The gain in this experiment is therefore a property of the code.

  3. Experiments on trained families with a larger decoder. To further isolate the error contributions due to limited decoder capacity, we froze our trained encoders and bottlenecks in the simulated experiments, and refit only the decoder with 35×35 \times more parameters. Freezing the bottleneck fixes the code law, so the rate and the information loss ΔKL(B)\Delta_{\mathrm{KL}}(B) cannot change. By Lemma 2.2, any change in NLL can be directly attributed to the decoder error term. We report this for 18 BNPE models: both stochastic and deterministic models at a low, a middle and a high rate, for each of the Bernoulli GLM, g-and-k and Ricker. Please see the new Figure 8.

The table compares stochastic and deterministic codes with the original and with the large decoder. Entries are NLL differences, stochastic minus deterministic (↓\downarrow, lower is better), in nats.

Three rows, marked †\dagger, match the rate exactly by time-sharing the deterministic summaries (Proposition A.3).

Simulator (encoder)Rate, stochasticRate, deterministicOriginal decoderLarge decoder
Bernoulli GLM (BiLSTM)4.804.33-1.06-1.01
Bernoulli GLM (BiLSTM)4.804.80 †^\dagger-0.81-0.75
Bernoulli GLM (BiLSTM)8.598.28-0.98-0.88
g-and-k (Deep Sets)3.553.85-0.32-0.30
g-and-k (Deep Sets)3.553.55 †^\dagger-0.47-0.45
g-and-k (Deep Sets)6.866.42-0.19-0.18
Ricker (BiLSTM)4.854.89-0.86-0.82
Ricker (BiLSTM)4.854.85 †^\dagger-0.89-0.86
Ricker (BiLSTM)8.528.40-1.03-0.81

The stochastic code keeps a lower loss under the large decoder in all nine comparisons. At exactly matched rate, the difference is −0.75-0.75 nats for the Bernoulli GLM, −0.45-0.45 for g-and-k and −0.86-0.86 for Ricker. Hence, the gains largely do not come from limited decoder capacity.

We continue our response in the next comment.

W2. Stability of the matched-rate comparison across the two rate estimates

The rate bracket. In the original submission, the upper rate estimate uses a fitted MADE as the marginal rr in the variational bound of Eq. (26). To address the reviewer's concern, we have rerun the comparison with a different marginal: the encoder's own channel averaged over about 4.24.2 million independent simulated observations, r(b)=1N∑jq(b∣yj′)r(b)=\frac1N\sum_{j} q(b\mid y_j'). The bound is the same identity, E(−log⁡r(B))−H(B∣Y)=I(Y;B)+KL(pB∥r)≥I(Y;B)\mathbb E(-\log r(B)) - H(B\mid Y) = I(Y;B) + \mathrm{KL}(p_B\Vert r) \ge I(Y;B), which holds for any rr fixed independently of the scored observations. This is similar to the leave-one-out bound of Poole et al. (2019) evaluated on a held-out dataset. The same rule is used for both stochastic and deterministic codes, for all trained models and we have updated Figure 2 in the main paper accordingly. The lower estimate is the Monte Carlo estimate of Section 5.1, as before. The new bracket has a median width of 0.0100.010 nats (0.350.35 in the submission) and is at most 0.390.39 nats (2.362.36); for stochastic models, it is at most 0.120.12 nats.

We now also report the stochastic vs deterministic comparison under two assignments of rate. In the point assignment, both codes are placed at the lower estimate. In the adverse assignment, each stochastic model is placed at its upper estimate and each deterministic model at its lower estimate. This is the pairing least favourable to our hypothesis. The table gives the matched-rate NLL difference, stochastic minus deterministic (↓\downarrow less is better) in nats, averaged over the three training seeds, with 95% intervals across seeds. (the t-interval with two degrees of freedom.)

SimulatorEncoderAdversePoint
Bernoulli GLMDeep Sets-0.582 [-0.739, -0.424]-0.589 [-0.747, -0.432]
Bernoulli GLMSet Transformer-0.513 [-0.617, -0.410]-0.529 [-0.607, -0.451]
Bernoulli GLMBiLSTM-0.750 [-0.768, -0.732]-0.759 [-0.778, -0.740]
g-and-kDeep Sets-0.325 [-0.354, -0.295]-0.330 [-0.360, -0.300]
g-and-kSet Transformer-0.337 [-0.395, -0.280]-0.342 [-0.401, -0.284]
g-and-kBiLSTM-0.425 [-0.457, -0.393]-0.431 [-0.463, -0.399]
Lotka-VolterraTCN-1.155 [-1.257, -1.053]-1.169 [-1.265, -1.073]
Lotka-VolterraBiLSTM-1.066 [-1.136, -0.997]-1.075 [-1.141, -1.010]
RickerTCN-0.829 [-0.862, -0.795]-0.838 [-0.870, -0.805]
RickerBiLSTM-0.821 [-0.849, -0.793]-0.828 [-0.857, -0.799]

Every interval lies below zero, and the adverse and point gaps differ by at most 0.0160.016 nats. Each training seed separately favours the stochastic arm in every cell.

Sensitivity of the lower estimate to the reference-set size. The lower estimate is biased downward for a finite reference set of MM observations, and the bias shrinks as MM grows. We therefore re-estimated the rate of all trained models with twenty times as many references, M=20⋅218M=20\cdot 2^{18} in place of 2182^{18}, with the models, channels and the NLL values held fixed.

  • Below 6 nats, the estimate does not depend on MM. The median shift is within 0.0100.010 nats for stochastic models and within 0.0020.002 nats for deterministic ones.
  • Between 6 and 10 nats, it rises with MM, and more for the deterministic arm. The median shift is +0.014+0.014 nats at 6 to 8 nats and +0.081+0.081 nats at 8 to 10 nats for deterministic models, against +0.004+0.004 and +0.024+0.024 nats for stochastic ones.
  • Direction. The deterministic rates rise more with increasing MM, so the smaller reference set in the paper placed deterministic models at a smaller rate than actual, already favoring deterministic models in the comparison.
  • Effect on the stochastic vs deterministic. With the larger reference set, the matched-rate gaps change by at most 0.0260.026 nats, with a median absolute change of 0.00250.0025 nats. Every gap stays negative and favors stochastic models.

We will reconcile all revised experiments with the methods text in the final manuscript.

References

Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., & Tucker, G. (2019, May). On variational bounds of mutual information. In International conference on machine learning (pp. 5171-5180). PMLR.

Thank you again for your careful reading and constructive feedback. We hope our responses clarify the relationship between the theoretical separation and the empirical evidence. Please let us know whether any concerns remain and we will be happy to answer :)

As the author/reviewer discussion period comes to an end, please let us know if our response addresses your questions. We would be happy to provide any further clarification if needed. :)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.