- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether a compressed representation optimized for prediction preserves information needed to evaluate future measurements. It considers a particularly interesting setting: the epistemic quantity is recoverable from the complete predictive distribution, so any loss arises during compression.
Critical strengths
- The central distinction is clear and meaningful. Separating information absent from the predictive marginal from information lost during compression sharpens the paper’s contribution. The Type-II assumption makes the result stronger than simply showing that prediction and inference require different information.
- The quantitative Gaussian analysis is convincing. Lemma 10 bounds conditional context information for every feasible encoder near the predictive optimum. The exact joint rate in Theorem 12, together with achievability, provides a complete characterization within the stated regime. The corresponding proofs appear internally consistent.
- The natural-KL analysis substantially strengthens the argument. A weighted coordinate distortion could build the desired mismatch into the objective. Section 6 instead explains how marginalizing target uncertainty creates the mismatch under a predictive likelihood loss. The separate global argument in Theorem 17 appropriately avoids treating a local quadratic approximation as globally exact.
- The empirical protocol is careful about selection and interpretation. Validation-only configuration locking, ten independent replication seeds, stochastic-latent evaluation, and separate rate diagnostics improve credibility. The authors also correctly acknowledge that probe failure does not establish unrestricted decoder impossibility, and that posterior-to-prior KL is not mutual information.
Critical weaknesses
-
The incremental theoretical novelty needs sharper justification. The exact Gaussian solution relies on established reverse water-filling, Gaussian rate–distortion converses, and independent Gaussian test channels. The general separation criterion is useful, but its proof follows directly from comparing its two defining information quantities. The most substantive contributions appear to be the all-encoder stability bound and the global natural-KL bound. The paper would benefit from organizing its novelty claim around these results and explaining more precisely what their analysis adds beyond classical compression with multiple fidelity constraints.
-
The likelihood-based theorem remains restricted in an important way. Theorem 17 permits arbitrary encoders but restricts predictive reconstructions to product Gaussians. Unrestricted forward-KL reconstruction generally permits mixture distributions, which can change the predictive rate–distortion boundary. The paper explicitly acknowledges this limitation, but the bound on does not itself establish the same excess-rate theorem for unrestricted decoders. Extending the result, or quantifying the effect of this restriction, would strengthen its relevance to learned predictive models.
-
The experiments do not isolate the empirical cost of epistemic fidelity. In Table 1, epistemic-aware models achieve predictive distortion around 0.046–0.049, whereas predictive-selected models achieve approximately 0.079–0.087. Both satisfy the threshold, but the former use additional rate while delivering substantially better prediction. Consequently, the reported information gap cannot be attributed solely to epistemic preservation. The authors acknowledge this; a comparison at matched predictive distortion, using denser operating-point searches or constrained optimization, would make the empirical evidence more informative.
-
The empirical scope is narrow relative to the motivation. All executed neural experiments use the same static, two-variable Gaussian construction. Varying target uncertainty tests the proposed mechanism, but does not establish robustness to correlated sources, nonlinear observation models, higher-dimensional contexts, or sequential learning. The binary sensing task also explicitly makes the optimal action depend on the sign of the context with a fixed utility margin. This is a useful controlled demonstration, but provides limited evidence about naturally arising sensing decisions.
-
Decoder and reproducibility controls could be stronger. The epistemic results rely on a prescribed neural probe. Stronger probe families or an independent conditional-mean estimate would help distinguish information loss from readout limitations. Appendix E specifies many implementation details, but does not clearly provide training/validation/test sample counts, maximum training duration, or the complete probe-training protocol. Appendix G reproduces analytical checks rather than the neural study. These omissions limit independent reproduction.
Questions for authors
Please see the weakness part.
Response on Near-Boundary Stability, Empirical Confounds, and Reproducibility
We thank the reviewer for the detailed technical assessment and for recognizing the all-encoder stability result and the global natural-KL analysis. We have sharpened the theoretical contribution and directly tested three major empirical alternative explanations.
1. Theoretical contribution beyond classical rate-distortion
Our novelty claim is not reverse water-filling or the mere existence of competing distortion criteria, but quantitative epistemic-information rigidity near the predictive rate-distortion boundary.
For every admissible stochastic encoder satisfying DP∗(Z)≤D and I(H;Z)≤RP(D)+ϵ, we establish
\qquad D_E^*(Z)\ge c^2\sigma_V^2e^{-2\epsilon/\kappa_\lambda}.$$ These results apply to all stochastic encoders under the stated Gaussian assumptions, not merely Gaussian test channels. The exact joint rate-distortion theorem further provides achievability and quantifies the required additional rate. A second substantive contribution is the global natural-KL analysis: marginalizing target uncertainty suppresses predictive sensitivity to sensing quality under a likelihood-based objective. Its finite-$M$ excess-rate bounds assume product-Gaussian reconstructions; we do not claim an unrestricted finite-$M$ decoder theorem. ### 2. Robustness to mixture decoders We trained 2-, 4-, and 8-component GMM predictive decoders (18 models). All satisfy $D_P\le0.1$; all nine predictive-only models fail $D_E\le0.16$, whereas all nine epistemic-aware models satisfy both thresholds. Mean epistemic MSEs are 0.635–0.648 versus 0.128–0.129. This directly tests whether the observed separation depends on unimodal Gaussian prediction. ### 3. Matched predictive fidelity We addressed the original Table 1 confound using a validation-selected three-seed comparison with the original single-Gaussian decoder at $M=32$. | Metric | Predictive-only | Epistemic-aware | |---|---:|---:| | $D_P$ | 0.08482 | 0.08359 | | $D_E$ | 0.63820 | 0.13644 | | Action error | 0.4981 | 0.1460 | | Variational rate | 1.6115 | 2.4136 | **Predictive distortion differs by only 0.00123, while epistemic MSE differs by 0.50176.** The separation therefore persists at matched predictive fidelity. An additional matched 8-GMM experiment provides consistent evidence. The different variational rates are consistent with the capacity mechanism but are not identified with the unrestricted theoretical $\Delta R_E$. ### 4. Stronger epistemic readouts We froze the matched-$D_P$ encoders and trained 64×2, 128×3, and 256×3 MLP probes, plus a nonparametric kNN conditional-mean estimator. Across three seeds, predictive-only epistemic MSE remains 0.6346–0.6367, near the zero-context benchmark of 0.64, whereas epistemic-aware MSE is 0.1348–0.1420. Thus, the separation persists under stronger readouts, although these estimates do not certify Bayes-optimal epistemic distortion. ### 5. Reproducibility and scope The additional GMM controls use independently generated validation/test sets of 5,000/6,000 samples per seed. Stronger-probe experiments use 35,000/6,000/12,000 training/validation/test samples. These controls use 650 optimization steps, distinct from the original early-stopping protocol. The revised manuscript documents the executed procedures and per-seed results. Neural predictive KL uses five-node numerical quadrature, whereas the theory uses exact forward KL. We distinguish variational KL, mutual information, and unrestricted excess rate. Our experiments isolate the mechanism within a controlled Gaussian sensing source; they do not establish generality across non-Gaussian sources, nonlinear observations, higher-dimensional contexts, or sequential policies. ### Conclusion **The principal theoretical and empirical concerns have now received direct, substantive responses.** The all-encoder rigidity and native-KL results clarify the scientific contribution, while matched-$D_P$, mixture-decoder, and stronger-probe controls address the principal alternative explanations. We respectfully request reconsideration of the initial assessment based on the additional evidence and clarified scope.