- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Product-CMI, a leave-one-out plug-in estimator of conditional mutual information for categorical and continuous variables. Categorical coordinates use exact matching, while continuous coordinates use Gaussian similarities. A differentiable version replaces categorical matches with inner products of probability vectors. The paper claims consistency across all eight type assignments, gives an error decomposition in a representative mixed setting, and bounds the soft–hard discrepancy when the soft representations uniformly approach their observed one-hot counterparts. Experiments examine estimation accuracy, local support, relaxation error, and optimization toward constructed conditional-independence solutions.
Critical strengths
- The estimator is simple and clearly specified. The cancellation of the kernel-volume factors in the four-mass CMI ratio is useful, and the soft categorical construction recovers the hard estimator exactly at one-hot inputs.
- Subject to adequate uniform local support, the soft–hard argument identifies a meaningful property: both the perturbation and the hard mass have the same kernel-volume scale, avoiding a spurious inverse-bandwidth factor in their relative error.
- The empirical reporting is candid. The paper acknowledges that Mesner–Shalizi has lower pooled MAE, distinguishes local rare-stratum error from global CMI error, and reports the supported subset separately for Marx et al. The known-optimum experiments evaluate progress using both parameter error and estimators outside the gradient computation.
Critical weaknesses
- The assumptions do not justify the uniform relative approximation used by the main proofs. A3 on page 5 bounds the densities away from zero only on a fixed region , but Lemma 1 asserts a relative approximation uniformly over all sample points. In Appendix A.1, equation (11) controls an absolute fluctuation; obtaining equation (12) requires a lower bound on for every evaluation point. No condition ensuring that all observations lie in , or a treatment of observations outside it, is stated. Gaussian variables illustrate the issue: positivity on each fixed bounded region does not provide a uniform density lower bound at sample extremes. Theorem 1 then uses this uniform approximation in equation (13), and Theorem 2 again uses it when dividing by the hard masses in equation (16). A4 controls the population log ratio but does not by itself control estimation errors in low-density regions. This is a gap in the stated proof, rather than a claim that no consistent estimator of this form can exist. A corrected support/boundary formulation or an explicit tail/trimming argument is needed.
- The main benchmark does not instantiate the claimed collection of continuous-distribution settings. Appendix B.1 generates U and W as Bernoulli or finite multiclass variables and subsequently changes whether the estimator treats them as categorical or numerical. Consequently, the configurations labeled CCN, CNN, NNC, and NNN contain atomic variables in positions labeled numerical. The appendix explicitly acknowledges this distinction, but the pooled six-configuration result is therefore not direct evidence for the corresponding continuous cases of Theorem 1. Genuine continuous data are used in the separate dimension and sparsity diagnostics; these do not resolve the interpretation of the main comparison. This also makes the large pooled CMIh error difficult to interpret without configuration-level results.
- The demonstrated optimization benefit is narrower than the motivating contribution. Theorem 2 controls estimator values near the observed one-hot representations; it does not control gradients or a learned, non-hard representation. The categorical optimization experiments instead use a fixed temperature , without establishing that their representations satisfy the theorem’s vanishing maximum one-hot error. Moreover, only Product-CMI is optimized in the known-optimum study, and Appendix E is a one-dimensional repair-grid check. These are useful sanity checks, but they do not establish the comparative value of the proposed differentiable objective. Appendix D also describes class-specific logits and a smooth structural model without giving their full formulas in the supplied PDF. A direct differentiable comparator and an explicit account of the target at nonzero temperature would materially improve the evidence. I recognize that Section 6 already disclaims optimizer convergence; my concern is the remaining gap between the guarantee and the principal use case.
Questions for authors
- What condition on makes Lemma 1 uniform over the complete sample? Please identify the precise assumption or supply the missing treatment of low-density observations and boundaries, and state how it changes Theorems 1 and 2.
- Can you disaggregate the main accuracy results by configuration and clarify which continuous-distribution cases are actually evaluated? A focused rerun with genuinely continuous U or W would help distinguish estimator behavior from numerical treatment of atomic variables.
- Please provide the exact categorical logits/structural models in Appendix D. What population objective does the fixed-temperature relaxation target, and how do its gradient and parameter recovery compare with a suitable existing differentiable estimator or objective on the same controlled model?
3 comments
I am raising my score from Weak Reject to Weak Accept
Thank you for the substantial revisions and additional experiments. The corrected support and boundary analysis, genuinely continuous benchmark, and clearer fixed-temperature analysis largely address my main concerns. While reservations remain about the scope of the optimization guarantees and the methodological novelty, I am raising my score from Weak Reject to Weak Accept.
Thank You for Reviewing Our Revision
Thank you very much for taking the time to review our revised manuscript and reconsider your assessment. We really appreciate your constructive feedback, which helped us strengthen both the theoretical analysis and experimental evaluation of the paper.
Response to Reviewer 7puY: Strengthened Theory and Experimental Evidence
Thank you for your detailed review. You correctly identified issues in our original assumptions, benchmark, and optimization presentation. We have made substantive revisions (highlighted in blue):
1. Uniform approximation and low-density observations (Weakness/Question 1).
You are right that our original fixed-region density bound did not suffice for uniform approximation on unbounded samples. We corrected A3 (Sec. 4.1): within each admissible categorical stratum, the entire continuous support is a common compact Cartesian rectangle, and relevant mixed marginals are uniformly bounded above and away from zero throughout it. Every sample point therefore falls within the stated support.
Lemma 1 and App. A.1 explicitly account for Gaussian boundary effects through uniformly positive factors qS. Cartesian support ensures qUVWqW=qUWqVW, so these factors cancel in the four-mass ratio (Sec. 4.2; App. A.2). App. A.3 controls boundary contributions to the rate. Theorems 1–2 and Proposition 1 now hold under corrected A1–A5, with Theorem 2 additionally requiring δn=Op(an). Separately, Proposition 3 (App. A.8) proves consistency for a specified unbounded finite Gaussian-mixture family by treating the bulk and tails, without an invalid uniform density lower bound. We thus address both the proof gap and an important unbounded setting.
2. Genuinely continuous benchmark (Weakness/Question 2).
We agree that numerical coding of Bernoulli variables does not make them continuous. We replaced the main benchmark: every coordinate marked N now has a genuinely continuous generator (App. B.1). Six symmetry-distinct configurations, two dependence levels, five sample sizes (n=500–10,000), and five seeds yield 300 datasets; all four estimators completed every run. The generators and independently calculated reference CMIs are specified in App. B.1, and App. C.1 (Tables 7–11) provides per-configuration results.
The new benchmark shows competitive mixed-type estimation accuracy: pooled MAE is 0.0102 for Product-CMI, versus 0.0108 for Mesner–Shalizi, 0.0126 for Marx, and 0.0198 for CMIh (Sec. 5.1; Fig. 1). We also report the sample-size trends and distinguish these Gaussian experiments from the compact-support assumptions of the rate theorem.
3. Fixed-temperature target, models, and comparator (Weakness/Question 3).
We agree that the original manuscript needed explicit optimization equations and a clearer distinction between estimator guarantees and optimization behavior. Appendix D now specifies the models (Eqs. 39–41): categorical V uses five-class logits bi,k+4θui(k−2), where bi,k=0.3Wi+0.15k+0.1ϵi,k and τ=2; continuous V uses Vθ,i=0.5Wi+θui+0.5ϵV,i. Inputs and noise remain fixed during optimization.
Appendix D.1 (Eqs. 42–48) defines the fixed-temperature, fixed-bandwidth population surrogate and derives its gradient through both pairwise kernel arguments. We clarify how it differs from sampled categorical CMI, using explicit categorical and Gaussian examples. Proposition 4 (App. D.2) provides a complementary positive guarantee: in a specified balanced binary model at any fixed temperature, the surrogate and its finite-sample counterpart (under the group-count condition) have the correct independence minimizer and negative-gradient descent direction. Empirical gradients converge uniformly on compact intervals to the surrogate population gradient at Op(n−1/2).
We also add 360 paired runs against differentiable residualized conditional HSIC (CHSIC), using the same 100-step Adam budget and independent hard-CMI evaluation (Sec. 5.3; App. D.3). Product-CMI attains 82.2% parameter-recovery success, with competitive hard-CMI reductions (0.5226 versus 0.5288 for CHSIC) and 2.45× lower measured method time (3.75 s versus 9.19 s). CHSIC attains 100% recovery; all configuration-level outcomes are reported. This directly addresses the requested comparator while establishing what is supported by the fixed-temperature analysis.
We hope these corrections and added comparisons address your substantive concerns.