acceptodds
Under review as a conference paper at ICLR 2027

Differentiable Estimation of Conditional Mutual Information for Mixed-Type Data

Abstract

Conditional mutual information (CMI) is a fundamental measure of statistical dependence, but using it as an optimization objective for mixed categorical and continuous data has been limited by the lack of estimators that are both consistent and differentiable. In this article, we devise a product-kernel CMI estimator that combines exact categorical matching with Gaussian kernel smoothing over continuous coordinates. We prove its consistency for a canonical mixed setting under standard bandwidth conditions and give an explicit error decomposition into smoothing bias, local-kernel stochastic error, and empirical-averaging error. We further introduce a differentiable relaxation that replaces hard categorical matching with soft probability-vector similarities compatible with gradient-based optimization. We show that, when the relaxed representations approach their hard one-hot counterparts at rate , the soft estimator deviates from the hard one by ; thus, consistency of the hard estimator carries over to its differentiable counterpart. Finally, we show consistency for both the hard and differentiable estimators across all eight possible categorical-continuous configurations of the three variables involved in a CMI query. Our results provide theoretical guarantees for using CMI as a differentiable regularizer in gradient-based procedures for mixed-type data, such as constrained data imputation, fairness-aware learning, and invariant representation learning.

Primary area:
learning theory
Keywords:
conditional mutual information, mixed-type data, differentiable estimation, kernel methods, statistical estimation, gradient-based optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7puYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Product-CMI, a leave-one-out plug-in estimator of conditional mutual information for categorical and continuous variables. Categorical coordinates use exact matching, while continuous coordinates use Gaussian similarities. A differentiable version replaces categorical matches with inner products of probability vectors. The paper claims consistency across all eight type assignments, gives an error decomposition in a representative mixed setting, and bounds the soft–hard discrepancy when the soft representations uniformly approach their observed one-hot counterparts. Experiments examine estimation accuracy, local support, relaxation error, and optimization toward constructed conditional-independence solutions.

Critical strengths
  1. The estimator is simple and clearly specified. The cancellation of the kernel-volume factors in the four-mass CMI ratio is useful, and the soft categorical construction recovers the hard estimator exactly at one-hot inputs.
  2. Subject to adequate uniform local support, the soft–hard argument identifies a meaningful property: both the perturbation and the hard mass have the same kernel-volume scale, avoiding a spurious inverse-bandwidth factor in their relative error.
  3. The empirical reporting is candid. The paper acknowledges that Mesner–Shalizi has lower pooled MAE, distinguishes local rare-stratum error from global CMI error, and reports the supported subset separately for Marx et al. The known-optimum experiments evaluate progress using both parameter error and estimators outside the gradient computation.
Critical weaknesses
  1. The assumptions do not justify the uniform relative approximation used by the main proofs. A3 on page 5 bounds the densities away from zero only on a fixed region X0X_{0}, but Lemma 1 asserts a relative approximation uniformly over all sample points. In Appendix A.1, equation (11) controls an absolute fluctuation; obtaining equation (12) requires a lower bound on gS(XS,i)g_S(X_{S,i}) for every evaluation point. No condition ensuring that all observations lie in X0X_0, or a treatment of observations outside it, is stated. Gaussian variables illustrate the issue: positivity on each fixed bounded region does not provide a uniform density lower bound at sample extremes. Theorem 1 then uses this uniform approximation in equation (13), and Theorem 2 again uses it when dividing by the hard masses in equation (16). A4 controls the population log ratio but does not by itself control estimation errors in low-density regions. This is a gap in the stated proof, rather than a claim that no consistent estimator of this form can exist. A corrected support/boundary formulation or an explicit tail/trimming argument is needed.
  2. The main benchmark does not instantiate the claimed collection of continuous-distribution settings. Appendix B.1 generates U and W as Bernoulli or finite multiclass variables and subsequently changes whether the estimator treats them as categorical or numerical. Consequently, the configurations labeled CCN, CNN, NNC, and NNN contain atomic variables in positions labeled numerical. The appendix explicitly acknowledges this distinction, but the pooled six-configuration result is therefore not direct evidence for the corresponding continuous cases of Theorem 1. Genuine continuous data are used in the separate dimension and sparsity diagnostics; these do not resolve the interpretation of the main comparison. This also makes the large pooled CMIh error difficult to interpret without configuration-level results.
  3. The demonstrated optimization benefit is narrower than the motivating contribution. Theorem 2 controls estimator values near the observed one-hot representations; it does not control gradients or a learned, non-hard representation. The categorical optimization experiments instead use a fixed temperature τ=2\tau = 2, without establishing that their representations satisfy the theorem’s vanishing maximum one-hot error. Moreover, only Product-CMI is optimized in the known-optimum study, and Appendix E is a one-dimensional repair-grid check. These are useful sanity checks, but they do not establish the comparative value of the proposed differentiable objective. Appendix D also describes class-specific logits and a smooth structural model without giving their full formulas in the supplied PDF. A direct differentiable comparator and an explicit account of the target at nonzero temperature would materially improve the evidence. I recognize that Section 6 already disclaims optimizer convergence; my concern is the remaining gap between the guarantee and the principal use case.
Questions for authors
  1. What condition on X0X_{0} makes Lemma 1 uniform over the complete sample? Please identify the precise assumption or supply the missing treatment of low-density observations and boundaries, and state how it changes Theorems 1 and 2.
  2. Can you disaggregate the main accuracy results by configuration and clarify which continuous-distribution cases are actually evaluated? A focused rerun with genuinely continuous U or W would help distinguish estimator behavior from numerical treatment of atomic variables.
  3. Please provide the exact categorical logits/structural models in Appendix D. What population objective does the fixed-temperature relaxation target, and how do its gradient and parameter recovery compare with a suitable existing differentiable estimator or objective on the same controlled model?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 7puY: Strengthened Theory and Experimental Evidence

Thank you for your detailed review. You correctly identified issues in our original assumptions, benchmark, and optimization presentation. We have made substantive revisions (highlighted in blue):

  • Uniform-support gap: Strengthened A3, explicit boundary analysis, and a separate unbounded Gaussian-mixture result (Sec. 4; App. A).
  • Atomic variables labeled continuous: A new genuinely continuous benchmark with configuration-level results (Sec. 5.1; Apps. B–C).
  • Fixed-temperature optimization: Exact models, a population-surrogate analysis, a gradient result, and a matched CHSIC comparison (Sec. 5.3; App. D).

1. Uniform approximation and low-density observations (Weakness/Question 1).

You are right that our original fixed-region density bound did not suffice for uniform approximation on unbounded samples. We corrected A3 (Sec. 4.1): within each admissible categorical stratum, the entire continuous support is a common compact Cartesian rectangle, and relevant mixed marginals are uniformly bounded above and away from zero throughout it. Every sample point therefore falls within the stated support.

Lemma 1 and App. A.1 explicitly account for Gaussian boundary effects through uniformly positive factors qSq_S. Cartesian support ensures qUVWqW=qUWqVWq_{UVW}q_W=q_{UW}q_{VW}, so these factors cancel in the four-mass ratio (Sec. 4.2; App. A.2). App. A.3 controls boundary contributions to the rate. Theorems 1–2 and Proposition 1 now hold under corrected A1–A5, with Theorem 2 additionally requiring δn=Op(an)\delta_n=O_p(a_n). Separately, Proposition 3 (App. A.8) proves consistency for a specified unbounded finite Gaussian-mixture family by treating the bulk and tails, without an invalid uniform density lower bound. We thus address both the proof gap and an important unbounded setting.

2. Genuinely continuous benchmark (Weakness/Question 2).

We agree that numerical coding of Bernoulli variables does not make them continuous. We replaced the main benchmark: every coordinate marked N now has a genuinely continuous generator (App. B.1). Six symmetry-distinct configurations, two dependence levels, five sample sizes (n=500n=500–10,00010{,}000), and five seeds yield 300 datasets; all four estimators completed every run. The generators and independently calculated reference CMIs are specified in App. B.1, and App. C.1 (Tables 7–11) provides per-configuration results.

The new benchmark shows competitive mixed-type estimation accuracy: pooled MAE is 0.0102 for Product-CMI, versus 0.0108 for Mesner–Shalizi, 0.0126 for Marx, and 0.0198 for CMIh (Sec. 5.1; Fig. 1). We also report the sample-size trends and distinguish these Gaussian experiments from the compact-support assumptions of the rate theorem.

3. Fixed-temperature target, models, and comparator (Weakness/Question 3).

We agree that the original manuscript needed explicit optimization equations and a clearer distinction between estimator guarantees and optimization behavior. Appendix D now specifies the models (Eqs. 39–41): categorical VV uses five-class logits bi,k+4θui(k−2)b_{i,k}+4\theta u_i(k-2), where bi,k=0.3Wi+0.15k+0.1ϵi,kb_{i,k}=0.3W_i+0.15k+0.1\epsilon_{i,k} and τ=2\tau=2; continuous VV uses Vθ,i=0.5Wi+θui+0.5ϵV,iV_{\theta,i}=0.5W_i+\theta u_i+0.5\epsilon_{V,i}. Inputs and noise remain fixed during optimization.

Appendix D.1 (Eqs. 42–48) defines the fixed-temperature, fixed-bandwidth population surrogate and derives its gradient through both pairwise kernel arguments. We clarify how it differs from sampled categorical CMI, using explicit categorical and Gaussian examples. Proposition 4 (App. D.2) provides a complementary positive guarantee: in a specified balanced binary model at any fixed temperature, the surrogate and its finite-sample counterpart (under the group-count condition) have the correct independence minimizer and negative-gradient descent direction. Empirical gradients converge uniformly on compact intervals to the surrogate population gradient at Op(n−1/2)O_p(n^{-1/2}).

We also add 360 paired runs against differentiable residualized conditional HSIC (CHSIC), using the same 100-step Adam budget and independent hard-CMI evaluation (Sec. 5.3; App. D.3). Product-CMI attains 82.2% parameter-recovery success, with competitive hard-CMI reductions (0.5226 versus 0.5288 for CHSIC) and 2.45× lower measured method time (3.75 s versus 9.19 s). CHSIC attains 100% recovery; all configuration-level outcomes are reported. This directly addresses the requested comparator while establishing what is supported by the fixed-temperature analysis.

We hope these corrections and added comparisons address your substantive concerns.

Reviewer_7puYOpenReview ↗

I am raising my score from Weak Reject to Weak Accept

Thank you for the substantial revisions and additional experiments. The corrected support and boundary analysis, genuinely continuous benchmark, and clearer fixed-temperature analysis largely address my main concerns. While reservations remain about the scope of the optimization guarantees and the methodological novelty, I am raising my score from Weak Reject to Weak Accept.

AuthorsOpenReview ↗

Thank You for Reviewing Our Revision

Thank you very much for taking the time to review our revised manuscript and reconsider your assessment. We really appreciate your constructive feedback, which helped us strengthen both the theoretical analysis and experimental evaluation of the paper.

Reviewer_SW1POpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper builds Product-CMI, a leave-one-out product-kernel estimator of CMI for mixed data, proves consistency, and adds a differentiable categorical relaxation with an O_p(a_n) soft-to-hard transfer theorem.

Critical strengths

The differentiable relaxation with a statistical transfer guarantee (Theorem 2) is new.

Critical weaknesses
  1. The hard estimator is a known construction specialized to mixed data, and the precedent is not cited: with all variables continuous, the estimator I^\hat{I} is exactly a leave-one-out KDE plug-in of the CMI density ratio. Related references should be added.
  2. Kernel methods are limited to relatively low dimensional data.
Questions for authors
  1. How is Product-CMI's continuous case different from a leave-one-out KDE plug-in CMI-ratio estimator, beyond the mixed-type generalization?
  2. Beyond the tested ranges (q <= 8, K <= 4), how does performance degrade empirically as the continuous dimension q or the number of categories K grows at fixed n: given that (7) prices these through h^d and π_min ≈ 1/(K_U K_W)? Would per-dimension bandwidths (as in Li–Racine) or a kNN-baseline comparison change the picture? Please also state the fixed-dimension, fixed-K scope of A1–A5 explicitly.
3 comments
AuthorsOpenReview ↗

Response to Reviewer SW1P: Clarifying Novelty and Extending the Evaluation

Thank you for your review and for recognizing the novelty of our differentiable relaxation and transfer guarantee. We agree that its relationship to established kernel methods needed clearer presentation. The revisions (in blue) address each concern:

  • KDE precedent and novelty: We explicitly acknowledge the standard KDE special case and sharpen the contribution statement (Secs. 2–3, 6).
  • Dimension/cardinality sensitivity: We clarify the assumptions and add controlled tests up to q=12q=12 and K=16K=16 (Secs. 4–5; App. E).
  • Bandwidths and kNN: We add a coordinatewise-bandwidth theorem and pilot, plus a full-conditioning, paired Mesner–Shalizi comparison (Apps. A.4, E.2–E.3).

1. KDE precedent and the exact novelty (Weakness/Question 1).

We agree that in the all-continuous case, hard Product-CMI is exactly the standard leave-one-out KDE plug-in estimator of the four-density CMI ratio. We should have stated this connection explicitly. Section 3.1 now does so; Section 2 cites established mixed-data kernel work, including Li and Racine (2003), Hall et al. (2004), and Racine and Li (2004). We have also revised the Introduction and Discussion to avoid presenting the product-kernel construction itself as novel.

The principal contribution is the differentiable categorical relaxation and its statistical link to the hard mixed-type estimator. The probability-vector inner product recovers exact categorical matching at one-hot inputs and allows gradients for soft representations. Crucially, Theorem 2 (App. A.6) establishes more than differentiability: when soft representations approach their observed one-hot labels at rate ana_n, the soft–hard CMI discrepancy is Op(an)O_p(a_n) without an extra inverse-bandwidth factor. Consistency and rates now cover all eight type assignments (Theorem 1; Proposition 1). Proposition 4 (App. D.2) proves the correct independence minimum and negative-gradient direction for a specified binary surrogate at fixed temperature. These are the differentiability and statistical properties our work adds to the established kernel foundation.

2. Larger dimensions and category counts (Weakness/Question 2).

We agree that performance beyond the original ranges needed direct evaluation. Section 4.1 now explicitly states the fixed-dimension and fixed-finite-support scope of A1–A5, including positive relevant cell probabilities, common compact rectangular continuous supports, and deterministic shrinking bandwidths. The general stochastic rate depends on hnDh_n^D, and the representative mixed-case expression (Eq. 11) makes the πmin⁡\pi_{\min} factor explicit. Thus we state precisely which dimensional and cardinality regimes the theory covers.

We added two focused stress tests:

  • Continuous dimension: At n=2000n=2000, mean Product-CMI MAE changes from 0.0310 at q=1q=1 to approximately 0.100 at q=8q=8 (Fig. 2b), with non-monotone intermediate values. A separate paired Gaussian comparison extends to q=12q=12 (App. E.2).
  • Controlled cardinality: Holding population CMI fixed, increasing the number of categories of WW alone from 2 to 16 changes moderate-dependence MAE only modestly, 0.0162 to 0.0184 at n=1000n=1000 (App. E.1). Increasing all three alphabets creates much sparser joint cells; at K=16K=16, Table 16 reports MAE 12.9843 and approximately 79% empty cells. We identify leave-one-out matching sparsity and numerical flooring as the causes, clarifying when large alphabets require more care.

3. Coordinatewise bandwidths and kNN comparison (Question 2).

Proposition 2 (App. A.4) extends the convergence-rate guarantee to deterministic per-coordinate bandwidths reused across the four local masses. In the heteroscaled sensitivity experiment, coordinatewise bandwidth selection improves independence MAE from 0.0043 to 0.0004 (App. E.3). We also report the moderate-dependence oversmoothing and unequal tuning budgets so the comparison remains accurately scoped.

We now compare against Mesner–Shalizi using the same full qq-dimensional conditioning vector WW for both methods, on paired Gaussian datasets (App. E.2, Table 17). Under moderate dependence, Product-CMI has lower MAE at q=8q=8 and q=12q=12 (0.0507 versus 0.0958 at q=12q=12). Mesner–Shalizi is more accurate in some other settings. The results establish competitive estimation accuracy alongside Product-CMI's differentiable mixed-type objective and additional statistical guarantees.

We hope these revisions address both the novelty framing and the requested evidence on dimensionality, categorical support, and bandwidths.

The raised questions have been partially addressed in the revision. I also appreciate the hard work the authors spent in proving that the proposal is promising. However, I am still seeing some essential limitations of the proposal (such as generalizability to higher dimensional data). No change in my rating.

AuthorsOpenReview ↗

Follow-up on the Main Contribution and Remaining Concerns

Thank you for taking another look at our revised manuscript and for acknowledging the additional theoretical and experimental results. We appreciate your continued concern about generalizability to higher-dimensional data.

We would like to clarify an important distinction regarding the contribution of our work. Our goal is not to introduce a dimension-free CMI estimator or claim to overcome the curse of dimensionality. Rather, we address a different challenge: making mixed-type CMI estimation differentiable with respect to learned categorical representations while establishing a rigorous connection to the original CMI estimator.

As you recognized in your original review, the differentiable relaxation and its statistical transfer guarantee are new. We have also clarified that the all-continuous hard estimator coincides with standard leave-one-out KDE. However, standard KDE plug-in estimators do not, by themselves, provide a differentiable objective over learned categorical representations or a statistical guarantee connecting that objective to the original mixed-type CMI. This is the specific gap our work addresses.

Importantly, our contribution goes beyond simply making the KDE construction differentiable. Theorem 2 establishes that, under the stated approximation condition, the soft estimator differs from the hard mixed-type estimator by Op(an)O_p(a_n), without an additional inverse-bandwidth penalty. Proposition 4 provides a complementary fixed-temperature result: in a specified binary model, the differentiable objective has the correct independence minimizer and gradient direction even when categorical representations remain soft. These results provide statistical and optimization-relevant guarantees that do not follow directly from standard KDE.

Regarding dimensionality, we have explicitly stated the fixed-dimensional scope of our guarantees and extended our evaluation to q=12q=12 and K=16K=16, including controlled sensitivity analyses, coordinatewise bandwidths, and a fair comparison with Mesner–Shalizi using the full conditioning vector. We acknowledge the remaining limitations of kernel-based estimation in higher dimensions, but believe the differentiable relaxation and its theoretical analysis make a meaningful contribution within the stated scope.

Since you mentioned that your concerns were only partially addressed, could you help us understand which limitations remain most important to your assessment? In particular, is your main concern the absence of theoretical guarantees for growing dimensions, the need for stronger empirical evidence in higher-dimensional mixed-type optimization settings, or another limitation?

We would greatly appreciate any clarification and would be happy to address any specific remaining concerns as far as possible during the discussion period.

Reviewer_L9ytOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposed a kernel-based conditional mutual information estimator for mixed categorical and continuous data, called Product-CMI. The authors prove the consistency for canonical mixed setting under standard bandwidth conditions, and provide error decomposition for smoothing bias, local kernel stochastic error, and empirical averaging error.

Critical strengths
  1. The paper identifies an important and real open problem, where mixed data CMI estimators lack differentiability for use as optimization regularizers.
Critical weaknesses
  1. The theoretical guarantee is incomplete, i.e., explicit error rates derived for only one configuration, the remaining seven only has consistency, no convergence rate is given.

  2. Theorem 2 provides the key theoretical result of this paper and is the foundation of the proposed estimator. Only when the condition, δn→0\delta_n \rightarrow 0 is satisfied, consistency and convergence can be achieved. However, there is no guarantee that the optimization process will make δn→0\delta_n \rightarrow 0. The paper says nothing when the condition fails and how it would affect the feasibility of the method.

  3. Complexity of the method scales quadratically with sample size. Large dataset compatibility is compromised, and this is a substantial limitation for a method depicted as a practical regularizer for modern learning.

Questions for authors
  1. Can you provide theoretical analysis for the other configurations? Is consistency and convergence available for these configurations?

  2. The consistency result in Theorem 2 relies heavily on a strong assumption that cannot be guaranteed in practice. What happens if this core assumption, δn→0\delta_n \rightarrow 0, is violated? Does your method breakdown? What remedy can you propose?

  3. Can the authors provide alternative kernel approximation or sparse sampling strategies to address the large datasets and scale Product CMI? Prolonged runtime substantially impair the applicability to larger datasets. How would this impact the optimization performance and would your statistical consistency guarantee break?

3 comments
AuthorsOpenReview ↗

Response to Reviewer L9yt: Addressing All Three Main Concerns

Thank you for your careful review and for emphasizing the importance of differentiable CMI estimation for mixed data. Your comments helped us identify where the original manuscript needed stronger analysis and clearer explanations. We have made substantial revisions (highlighted in blue):

  • Rates beyond one configuration: We now give consistency and convergence-rate bounds for all eight type assignments (Sec. 4.2; Prop. 1).
  • The soft-to-hard assumption: We analyze its violation, establish a sufficient margin–temperature condition, and add a complementary fixed-temperature gradient result (Apps. A.7, D.2).
  • Quadratic runtime: We introduce comparison-partner subsampling, evaluate its estimation and optimization behavior, and establish a consistency condition (Sec. 5.4; Apps. E.4–E.5).

1. Rates for all type assignments (Weakness/Question 1).

We agree that giving an explicit rate only for a representative mixed case was insufficient. Proposition 1 (Sec. 4.2; App. A.3) now establishes rates for all eight categorical/continuous assignments. With DD the total continuous dimension, the hard-estimator error is Op(n−1/2)O_p(n^{-1/2}) for D=0D=0 and Op(hn2+log⁡n/(nhnD)+n−1/2)O_p(h_n^2+\sqrt{\log n/(nh_n^D)}+n^{-1/2}) for D>0D>0. Theorem 2 yields the corresponding soft-estimator rate with an additional Op(an)O_p(a_n) when δn=Op(an)\delta_n=O_p(a_n). We now explicitly specify the scope: fixed dimensions and categorical supports, regular mixed densities bounded away from zero on common compact rectangular supports, and deterministic bandwidths satisfying nhnD/log⁡n→∞nh_n^D/\log n\to\infty. Complete proofs are in Apps. A.1–A.3. Proposition 2 extends rates to deterministic coordinatewise bandwidths; Proposition 3 separately establishes consistency for a specified unbounded Gaussian-mixture family.

2. The one-hot condition and optimization (Weakness/Question 2).

You are right that optimization alone does not guarantee δn→0\delta_n\to0; we should have distinguished this more clearly from the theorem's conclusion. Theorem 2 guarantees estimator-value consistency under an explicit approximation condition, not convergence of an optimization algorithm. Appendix A.7 now provides a counterexample showing that, if the condition is violated, the soft estimator may remain well-defined but target a quantity different from categorical CMI. We also derive the sufficient Softmax bound δn≤2(Kmax⁡−1)e−γn/τn\delta_n\le2(K_{\max}-1)e^{-\gamma_n/\tau_n}, where γn\gamma_n is the minimum observed-label logit margin. This gives a concrete margin–temperature condition for recovering the one-hot limit when the logits are aligned with the observed labels.

Importantly, we also establish a positive optimization-relevant result without requiring one-hot convergence. Proposition 4 (App. D.2) proves that, in a specified balanced binary model, the fixed-temperature surrogate has the correct independence minimizer for every τ>0\tau>0, and its negative gradient points toward that minimizer. Under the stated group-count condition, the exact finite-sample objective has the same minimizer and descent direction; its gradient converges uniformly on compact parameter intervals to the population surrogate gradient at Op(n−1/2)O_p(n^{-1/2}). Section 5.3 adds a 360-paired-run comparison with differentiable conditional HSIC. Together, these additions answer what the assumption does and does not guarantee while providing independent evidence for optimization at fixed temperature.

3. Scalable evaluation and its guarantees (Weakness/Question 3).

We agree that exact O(n2L)O(n^2L) pairwise work is a practical limitation. Our uniform comparison-partner subsampling reduces work to O(nmL)O(nmL) by using mm partners per observation, shared across the four kernel masses. At n=10,000n=10{,}000 and m=256m=256, estimation time decreases from 4.2 s to 0.27 s (15.7×), with mean absolute disagreement 0.0057 relative to exact evaluation (App. E.4; excluding shared preprocessing). In a five-seed optimization pilot (n=1000n=1000, m=128m=128), both exact and subsampled objectives reach the known recovery threshold in all runs, while mean wall time decreases from about 25 s to 16 s.

Proposition 5 (App. E.5) additionally proves consistency of the unfloored hard subsampled estimator under A1–A5 when mnhnD/log⁡n→∞m_nh_n^D/\log n\to\infty, showing that subquadratic evaluation and consistency are compatible. We also analyze sampled-versus-exact gradients at a fixed parameter under additional conditions. As specified in the revision, the fixed-mm pilot is empirical evidence, not a claim of general optimizer-trajectory equivalence.

We hope these new proofs, explicit conditions, and scaling results fully address the main concerns raised in your review.

AuthorsOpenReview ↗

Follow-up on the One-Hot Assumption and Fixed-Temperature Result

Thank you again for your thoughtful review. Following our detailed response addressing all three of your concerns, we wanted to highlight one additional theoretical result that directly relates to your question about the one-hot assumption during optimization.

We agree that minimizing soft Product-CMI does not automatically guarantee convergence to one-hot representations, and we have clarified this distinction in Theorem 2.

Importantly, Proposition 4 (Appendix D.2) establishes a complementary positive result without requiring one-hot convergence. For a specified balanced binary model, we prove that the soft objective has the correct independence minimizer at any fixed temperature, and its negative gradient points toward that minimizer. Under the stated group-count condition, the finite-sample objective also has the correct minimizer and descent direction. Furthermore, its empirical gradient converges uniformly on compact parameter intervals to the population surrogate gradient at Op(n−1/2)O_p(n^{-1/2}).

This provides a rigorous example in which Product-CMI retains the desired optimization properties even when the categorical representations remain soft, complementing the soft-to-hard consistency analysis in Theorem 2.

Would this additional fixed-temperature result help address your concern about optimization when one-hot convergence is not guaranteed? If any concerns remain, we'd be happy to clarify.

Thank you for providing additional theoretical analysis. My concerns have been partially addressed, but there are still some concerns remaining in the proposed method. Therefore, the current score will be maintained.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.