- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Discrete Bridge Mutual Information estimation (DBMI) for discrete random variables. It constructs two reciprocal processes sharing the same reference bridges but having joint and independent endpoint distributions, respectively. Their path-space KL divergence equals the mutual information and decomposes into expected KL divergences between conditional Markov transitions. The practical estimator learns these transitions through conditional bridge matching, using a shared network and a factorized parameterization. Experiments cover factorized categorical distributions, DNA sequences with correlated synthetic tags, and binary images generated from correlated rectangle parameters.
Critical strengths
- A useful bridge-based formulation for discrete MI estimation. The shared-bridge construction provides a clear connection between endpoint mutual information and transition-level KL divergences. Sampling intermediate states directly from reference bridges makes training and estimation possible without simulating complete trajectories. This is a useful extension of bridge-based information estimation to categorical data.
- Promising empirical stability on controlled benchmarks. DBMI achieves competitive estimation errors and relatively low variability across training seeds in several settings, including the DNA-tag and image experiments. The known ground-truth MI makes these comparisons interpretable. The use of comparable backbones for DBMI and Info-SEDD also strengthens the experimental evaluation.
Critical weaknesses
- The justification of the practical factorized estimator is incomplete.
Proposition 2 concerns exact transition distributions, whereas the implementation uses the factorization in Eq. (14). The claim in lines 482–485 that the resulting bias vanishes as the number of steps increases does not follow under the fixed per-step noise parameter in Eq. (15).
For example, let both endpoint vectors equal (B, B), where B is a uniformly distributed binary variable, and choose two categories with a per-step switching probability of 1/2. The reference process then independently resamples every coordinate at each step. For any number of intermediate steps, the joint and independent bridges have identical intermediate transitions. At the final transition, the joint endpoint is determined by the starting point, while the independent endpoint is uniform over the two vectors (0, 0) and (1, 1). Its optimal coordinate-factorized approximation is uniform over all four binary vectors. Substituting these factorized transitions into Eq. (10) therefore gives twice the true mutual information: approximately 1.386 rather than 0.693 nats, regardless of the number of steps. A suitable vanishing-step-noise regime might address this issue, but that regime and its assumptions need to be stated and justified.
There are also two specific inconsistencies requiring clarification. First, Eq. (5) should generally average over the endpoint posterior conditioned on both the current intermediate state and the starting point, rather than conditioned only on the starting point. Second, Algorithm 2 should use an N+1 multiplier when uniformly sampling the N+1 terms indexed from 0 through N. These issues do not themselves invalidate the exact KL decomposition, but their implications for the implementation are unclear.
- The empirical evidence does not yet establish a sufficiently strong advance over the closest prior approaches.
The central construction closely follows InfoBridge, with existing categorical bridge-matching machinery supplying the discrete modeling framework. Consequently, the practical advantage over existing discrete estimators is particularly important. However, the categorical benchmark factorizes across dimensions, the image benchmark has only four informative latent variables, and the DNA dependence comes from synthetic tags at fixed positions. These are useful controlled tests but provide limited evidence about more general dependence structures. The extreme overestimation by InfoSEDD-J on images also warrants diagnosis before attributing the gap to the underlying approaches.
Additionally, Table 8 contains numerical inconsistencies that affect confidence in the reporting. For the two-category setting, the ground truth is 3.63 and the f-DIME-KL mean is 3.97, yet its reported RMSE is 0.10, below the absolute mean error of 0.34. The f-DIME-H entry has the same problem. These entries should be checked against the individual seed results.
Questions for authors
The following are clarification and validation requests that could affect my assessment:
-
What assumptions on the reference process, particularly how the per-step noise parameter changes with the number of steps, justify vanishing factorization bias? How does the claim accommodate the counterexample above? Evidence separating factorization error from model-fitting error would help establish the practical estimator's reliability.
-
Are Eq. (5) and the N multiplier in Algorithm 2 manuscript errors only? Please clarify the actual posterior conditioning and normalization used in the implementation, and whether correcting them changes any reported results.
-
Can the authors reconcile the inconsistent entries in Table 8 with the per-seed estimates and explain the severe failure of InfoSEDD-J on the image benchmark? In particular, what checks were performed on its convergence and numerical estimation procedure?
Related-work suggestion: I recommend considering the following papers on information-theoretic diffusion and mutual information estimation:
- Information-Theoretic Diffusion (Kong et al., ICLR 2023).
- Information-Theoretic Discrete Diffusion (Jeon et al., NeurIPS 2025).
- Beyond Normal: On the Evaluation of Mutual Information Estimators (Czyż et al., NeurIPS 2023).
- MMG: Mutual Information Estimation via the MMSE Gap in Diffusion (Yu et al., UAI 2026).
4 comments
Rebuttal
2. Eq. (5) endpoint posterior
Reviewer quote
Eq. (5) should generally average over the endpoint posterior conditioned on both the current intermediate state and the starting point, rather than conditioned only on the starting point.
Are Eq. (5) and the N multiplier in Algorithm 2 manuscript errors only? Please clarify the actual posterior conditioning and normalization used in the implementation, and whether correcting them changes any reported results.
Answer
We thank the reviewer for catching this mistake. In Eq. (5), the expectation should be taken with respect to the endpoint posterior , rather than . We have corrected this equation and its accompanying explanation in the revised manuscript.
This correction does not change the optimization objectives in Eqs. (6)–(7). Eq. (7) already samples from , under which , conditional on and , follows precisely the endpoint posterior in the corrected Eq. (5). The same argument applies to Eq. (6) for fixed . Thus, this correction requires no changes to the optimization objectives, training Algorithm 1, or the implementation.
3. Algorithm 2 normalization
Reviewer quote
Algorithm 2 should use an N+1 multiplier when uniformly sampling the N+1 terms indexed from 0 through N. These issues do not themselves invalidate the exact KL decomposition, but their implications for the implementation are unclear.
Are Eq. (5) and the N multiplier in Algorithm 2 manuscript errors only? Please clarify the actual posterior conditioning and normalization used in the implementation, and whether correcting them changes any reported results.
Answer
We thank the reviewer for catching this discrepancy. Indeed, because the estimator uniformly samples time indices and averages their KL terms, the scaling factor should be rather than . This off-by-one factor was also present in our evaluation code, introducing a small downward bias of on the Low-Dimensional benchmarks and on the Image benchmark. Since this factor enters purely as a multiplicative constant, we rescaled all affected metrics in closed form and updated the corresponding tables and figures in the revised manuscript (highlighted in blue). Notably, in most cases, applying this correction reduces the discrepancy with the ground truth rather than increasing it.
Importantly, these updates do not change any method rankings across the benchmarks, nor do they affect any of our contributions, conclusions, or experimental discussions.
4. Practical positionining w.r.t. other methods
Reviewer quote
The empirical evidence does not yet establish a sufficiently strong advance over the closest prior approaches.
The central construction closely follows InfoBridge, with existing categorical bridge-matching machinery supplying the discrete modeling framework. Consequently, the practical advantage over existing discrete estimators is particularly important.
Answer
Overall, DBMI achieves leading or shared-leading performance across all evaluated benchmarks, with a substantial margin over competing methods on the DNA Tag benchmark. Because few existing discrete MI benchmarks push estimators to their limits, we introduced the DNA Tag and image-based benchmarks to help fill this gap.
The key advantage of DBMI is that it is stable, robust, and performs reliably across all originally presented settings, making it a versatile MI estimator for discrete data. By contrast, each competing estimator exhibits a distinct weakness: InfoSEDD-J performs well on the synthetic benchmark but deteriorates on the DNA Tag and Image benchmarks, InfoSEDD-C underperforms DBMI across nearly all settings, and -DIME struggles in high-MI regimes. Because DBMI remains stable across all of these regimes, it offers the most reliable practical choice for general discrete MI estimation, with its main practical advantage being stability and fewer failure cases.
5. Dependence structure in benchmarks
Reviewer quote
the categorical benchmark factorizes across dimensions, the image benchmark has only four informative latent variables, and the DNA dependence comes from synthetic tags at fixed positions. These are useful controlled tests but provide limited evidence about more general dependence structures.
Answer
Thank you for raising this concern. We agree that our controlled benchmarks have limitations. Reliable comparison of MI estimators benefits from benchmarks with known ground-truth MI, which constrains how the joint distributions can be constructed. Existing benchmarking approaches typically rely on analytically tractable dependence structures or transformations that preserve MI [1, 2].
Rebuttal
In addition, benchmarking MI estimation in discrete spaces remains relatively underexplored. In our work we have contributed DNA-tag and discrete rectangle-image benchmarks as two controlled settings that combine known MI with complex observations. In the revised manuscript’s Future Work discussion, we now note that current benchmarks, including ours, cover a limited range of dependence structures and identify developing richer discrete benchmarks with known ground-truth MI as an important research direction.
6. Table 8 statistics
Reviewer quote
Additionally, Table 8 contains numerical inconsistencies that affect confidence in the reporting. For the two-category setting, the ground truth is 3.63 and the f-DIME-KL mean is 3.97, yet its reported RMSE is 0.10, below the absolute mean error of 0.34. The f-DIME-H entry has the same problem. These entries should be checked against the individual seed results.
Answer
Thank you for catching these inconsistencies. At , the DBMI entry should agree with Table 2. For f-DIME-KL and f-DIME-H, the errors were in the reported means and standard deviations and we have corrected entries in the revised version.
7. InfoSEDD-J image overestimation
Reviewer quote
The extreme overestimation by InfoSEDD-J on images also warrants diagnosis before attributing the gap to the underlying approaches.
explain the severe failure of InfoSEDD-J on the image benchmark? In particular, what checks were performed on its convergence and numerical estimation procedure?
Answer
Thank you for raising this concern. InfoSEDD [3] is a discrete diffusion counterpart to MINDE [4], and both offer joint formulations. Substantial overestimation by these joint estimators has been reported in prior work. In its text-consistency experiment, InfoSEDD [3, Table 11] reports an InfoSEDD-J estimate of versus ground-truth MI , and versus an approximate reference range of nats. InfoBridge [5, Tables 16–19] reports MINDE-J estimates of , , , and , respectively, versus ground-truth MI . It also reports severe overestimation on protein embeddings, with a MINDE-J MAE of nats (Section 5.3, Table 14). We observe overestimation by InfoSEDD-J on our DNA-tag benchmark as well.
Our hypothesis is that two factors contribute to the observed overestimation. First, InfoSEDD-J requires learning the joint diffusion process [3], which may be more challenging than learning a conditional or bridge process on the rectangles benchmark. Second, its MI estimator sums two contributions [3, Algorithm 1] corresponding to transitions in each variable, both involving approximate joint and marginal scores, which can accumulate in the final estimate, potentially causing substantial overestimation especially in low GT MI cases.
We used the official implementation with the setup described in our paper, under which InfoSEDD-C performs well. Limited hyperparameter variations did not yield substantial improvements, but we did not conduct an extensive search. We therefore interpret this result as a failure under the evaluated configuration.
We agree that the severe overestimation requires diagnosis before attributing it to the estimator itself. We will qualify the comparison as an observation under the evaluated configuration.
8. Suggested related work
Reviewer quote
Related-work suggestion: I recommend considering the following papers on information-theoretic diffusion and mutual information estimation: ....
Answer
Thank you for suggesting these references. We have incorporated Kong et al., Jeon et al., and Yu et al. into the Related Work section and cited Czyż et al. [1] in the benchmark discussion on MI invariance under bijective mappings.
We hope that our answers, the new theoretical analysis of coordinate-factorized transitions (Section 4.3, Proposition 3), the clarified advancement over competitor methods and the updated manuscript have thoroughly addressed your concerns. Please, in the remaining time let us know if any questions remain, and we kindly ask if you would consider adjusting your score in light of these revisions.
References
[1] Czyż et al., Beyond Normal: On the Evaluation of Mutual Information Estimators, NeurIPS 2023.
[2] Butakov et al., Mutual Information Estimation via Normalizing Flows, NeurIPS 2024.
[3] Foresti et al., Information Estimation with Discrete Diffusion, 2026.
[4] Franzese et al., MINDE: Mutual Information Neural Diffusion Estimation, ICLR 2024.
[5] Kholkin et al., InfoBridge: Mutual Information Estimation via Bridge Matching, 2026.
Thank you for the detailed response. The added factorization-bias analysis under explicit temporal-refinement assumptions, together with the posterior-conditioning clarification and numerical corrections, substantially addresses my main concerns. Some reservations remain about the positivity assumption’s applicability to sparse-support data and the limited diversity of benchmark dependence structures. Overall, I am raising my rating from weak rejection to weak acceptance.
Rebuttal
We thank Reviewer ibgC for the thoughtful and detailed review. We are glad that you found our bridge-based formulation to be a useful extension of mutual information estimation to categorical data, enabling trajectory-free training and estimation, and that you appreciated DBMI's strong empirical accuracy, stability across seeds, and fair evaluation. Below, we address each of your comments and questions in detail. We have also uploaded a revised version of the manuscript with all changes highlighted in blue.
1. Factorizaton transitions and MI estimation vanishing bias clarification
Reviewer quote
Answer
Thank you for pointing out the gap in our argument. Your counterexample is correct and shows that increasing the number of transitions alone does not guarantee vanishing factorization bias when the per-step stochasticity is fixed. However, our original statement did not specify how the per-step stochasticity should change as the number of transitions increases. The intended setting is temporal refinement with a fixed reference endpoint transition distribution qref(x1∣x0):
qref(x1∣x0)=∏d=1D[e−β1{x1d=x0d}+S1−e−β],β>0.
In this setting, increasing the number of intermediate time points requires decreasing the per-step stochasticity. Specifically, we consider
αN=SS−1(1−e−β/(N+1)),β>0,
which preserves the reference endpoint transition distribution for every N.
To formalize this setting and connect it directly to MI estimation, we have added Proposition 3 in Section 4.3 of the revised manuscript (with its proof in Appendix A.3). Under a strict-positivity assumption on the endpoint distribution, it establishes an O((N+1)−1) bound on the factorization bias when the coordinate transition marginals are known exactly:
Proposition 3 (Vanishing Factorization Bias). Assume that π(x0,x1)>0 for all (x0,x1)∈X2. Fix β>0, and let rπ,Njoint and rπ,Nind be the exact reciprocal processes defined in Eqs. (8)–(9), using the reference process with stochasticity parameter αN. For a∈{joint,ind}, define
rˉπ,Na(xtn+1∣xtn,x0)≜∏d=1Drπ,Na(xtn+1d∣xtn,x0),
where each factor is the exact coordinate marginal of the corresponding conditional transition. Define the population factorized MI functional by
INfact≜∑n=0NErπ,Njoint(x0,xtn)[KL(rˉπ,Njoint(⋅∣xtn,x0)∥rˉπ,Nind(⋅∣xtn,x0))].
Then there exists a finite constant C=C(π,β,D,S), independent of N, such that
INfact−I(X0;X1)≤N+1C.
This adressed the counter example, which uses a fixed switching probability α=1/2 (so coordinates are fully resampled at every step rather than refined with αN=O((N+1)−1)) and a degenerate endpoint distribution that violates π(x0,x1)>0.