Differentiable Estimation of Conditional Mutual Information for Mixed-Type Data

Conditional mutual information (CMI) is a fundamental measure of statistical dependence, but using it as an optimization objective for mixed categorical and continuous data has been limited by the lack of estimators that are both consistent and differentiable. In this article, we devise a product-kernel CMI estimator that combines exact categorical matching with Gaussian kernel smoothing over continuous coordinates. We prove its consistency for a canonical mixed setting under standard bandwidth conditions and give an explicit error decomposition into smoothing bias, local-kernel stochastic error, and empirical-averaging error. We further introduce a differentiable relaxation that replaces hard categorical matching with soft probability-vector similarities compatible with gradient-based optimization. We show that, when the relaxed representations approach their hard one-hot counterparts at rate $a_n$, the soft estimator deviates from the hard one by $O_p(a_n)$; thus, consistency of the hard estimator carries over to its differentiable counterpart. Finally, we show consistency for both the hard and differentiable estimators across all eight possible categorical-continuous configurations of the three variables involved in a CMI query. Our results provide theoretical guarantees for using CMI as a differentiable regularizer in gradient-based procedures for mixed-type data, such as constrained data imputation, fairness-aware learning, and invariant representation learning.