- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Prompt-CMI, an in-context conditional mutual information (CMI) estimator that eliminates the need for dataset-specific training. The model is pretrained on synthetic distributions using joint and conditionally independent samples, together with known CMI values for calibration. At inference, a context encoder summarizes the input dataset, and a frozen information decoder estimates CMI through context-conditioned density-ratio scores. Experiments demonstrate competitive estimation accuracy and substantial computational efficiency across synthetic benchmarks, with additional validation on real-world downstream tasks.
Critical strengths
-
Meaningful methodological contribution: Extending pretrained information estimation to CMI is a relevant and challenging problem. The proposed context-conditioned density-ratio estimation and synthetic reference generation provide a reasonable framework for avoiding dataset-specific optimization.
-
Strong empirical results and efficiency: Prompt-CMI achieves lower estimation errors than conventional statistical and neural CMI estimators across diverse synthetic settings, while substantially reducing inference time. The downstream experiments further demonstrate its potential practical utility.
Critical weaknesses
-
Insufficient validation of generalization to unseen dependence mechanisms: The main contribution is a reusable CMI estimator pretrained on synthetic distributions. However, the evaluation does not clearly establish whether the model generalizes to dependence mechanisms excluded from pretraining. Since the effectiveness of this approach depends heavily on the coverage of the synthetic task distribution, this is a central concern regarding its general applicability.
-
Limited validation of the proposed design and comparison with pretrained MI estimators: The paper does not sufficiently isolate the contributions of important components, particularly the calibration loss and copula preprocessing. Moreover, although related pretrained MI estimators such as InfoNet and InfoAtlas are discussed, the paper lacks a direct comparison with CMI estimation obtained through the MI chain rule using these models. Such comparisons would help establish the benefits of directly learning a context-conditioned CMI estimator.
Questions for authors
-
Generalization to unseen mechanisms: Can the authors evaluate Prompt-CMI on dependence mechanisms explicitly excluded from synthetic pretraining, for example through a leave-one-mechanism-out experiment? This would be important for assessing whether the model learns a generalizable CMI estimation procedure rather than relying on patterns represented during pretraining.
-
Comparison with pretrained MI estimation: Can the authors compare Prompt-CMI with a CMI estimator constructed from pretrained MI models such as InfoNet or InfoAtlas using the identity (I(X;Y\mid Z)=I(X;Y,Z)-I(X;Z))? This would clarify the practical advantage of directly estimating CMI.
-
Component contributions: Can the authors provide ablation results isolating the effects of the calibration loss and copula preprocessing on estimation accuracy?
-
I would be willing to raise my rating if the rebuttal provides convincing evidence of generalization to previously unseen dependence mechanisms and clarifies the advantages over existing pretrained MI-based alternatives.
2 comments
As the discussion deadline approaches, We would be grateful to hear whether our responses have adequately addressed your concerns! Feel free to let us know if you have any other concerns. Thank you again for your time and consideration !
The authors
Reply to Reviewer 3f2Y
We sincerely thank Reviewer 3f2Y for the thoughtful review and for recognizing our methodological contribution, estimation accuracy, and computational efficiency. Following your suggestions, we have evaluated generalization to distributions excluded from pretraining, compared against MI-based baselines, and assessed the training objectives and copula preprocessing.
W1. Evidence for Prompt-CMI’s generalization
To provide further evidence of generalization, we have evaluated four additional distributions explicitly excluded from synthetic pretraining—Gamma (log), Beta, Logistic, and Gumbel—using frozen Prompt-CMI. All four settings use (dX,dY,dZ)=(20,20,8). Prompt-CMI gives the closest mean estimate to the true CMI in all four cases.
These results, together with the main paper’s real-world feature-selection, dependence-ordering, and conditional-independence experiments, support practical generalization. We have included these generalization experiments in Appendix H of the revised manuscript.
\begin{array}{lccccccc} \hline \textbf{Metric} & \textbf{Distribution} & I^\star & \textbf{Prompt-CMI} & \textbf{KNN} & \textbf{CCMI} & \textbf{CMINE} & \textbf{DINE} \\\\ \hline \text{Est. CMI} & \text{Gamma (log)} & 4 & \mathbf{3.627\pm 0.525} & 0.227\pm 0.019 & 0.998\pm 0.295 & 2.540\pm 1.658 & 3.299\pm 0.140 \\\\ & \text{Beta} & 4 & \mathbf{4.267\pm 0.367} & 0.224\pm 0.029 & 0.842\pm 0.198 & 2.362\pm 0.987 & 3.297\pm 0.137 \\\\ & \text{Logistic} & 4 & \mathbf{3.837\pm 0.593} & 0.244\pm 0.023 & 0.930\pm 0.212 & 2.905\pm 1.607 & 3.466\pm 0.142 \\\\ & \text{Gumbel} & 4 & \mathbf{3.584\pm 0.545} & 0.241\pm 0.026 & 0.928\pm 0.225 & 2.204\pm 1.264 & 3.282\pm 0.172 \\\\ \hline \end{array}$$ **W2. Comparison with MI baselines and component ablations** We have compared Prompt-CMI with CMI baselines constructed from MIST, InfoAtlas, and KSG via MI identities across **all experimental tasks in the main paper**. **Prompt-CMI performs best on all summary metrics below**. We have included these comparisons in Appendix E of the revised manuscript. $$\def\arraystretch{1.25} \begin{array}{lcccccc} \hline \textbf{Metric} & (d_X,d_Y,d_Z) & I^\star & \textbf{Prompt-CMI} & \textbf{MIST} & \textbf{InfoAtlas} & \textbf{KSG-chain} \\\\ \hline \text{MAE}\\,\downarrow & (20,20,8) & 4 & \mathbf{0.9825} & 3.7692 & \text{—} & 3.8931 \\\\ & (4,4,8) & 0.8 & \mathbf{0.0799} & 0.5466 & 0.2677 & 0.5981 \\\\ \text{FS: NRMSE}\\,\downarrow & \text{—} & \text{—} & \mathbf{0.1963} & 0.3157 & 0.3220 & 0.3155 \\\\ \text{Order: IR (\\%)}\\,\downarrow & \text{—} & \text{—} & \mathbf{1.04} & 52.06 & 8.90 & 3.72 \\\\ \text{CI: AUROC}\\,\uparrow & \text{—} & \text{—} & \mathbf{0.8048} & 0.5301 & 0.6852 & 0.7034 \\\\ \hline \end{array}$$ In addition, to assess the components you highlighted, we have compared classification-only and calibration-only training and removed copula preprocessing. As shown below, **the full Prompt-CMI model achieves the lowest MAE in every setting**. We have included these ablation experiments in Appendix F of the revised manuscript. $$\def\arraystretch{1.25} \begin{array}{lcccccc} \hline \textbf{Metric} & (d_X,d_Y,d_Z) & I^\star & \textbf{Prompt-CMI} & \textbf{GT-only} & \textbf{BCE-only} & \textbf{Copula-off} \\\\ \hline \text{MAE}\\,\downarrow & (20,20,8) & 4 & \mathbf{0.9825} & 1.5277 & 3.2251 & 2.2131 \\\\ & (4,4,8) & 0.8 & \mathbf{0.0799} & 0.1581 & 0.4543 & 0.2471 \\\\ \hline \end{array}$$ **Q1. Generalization to mechanisms excluded from pretraining** Please see our response to your W1 above. **Q2. CMI estimation via the MI chain rule using pretrained MI models** Please see our response to your W2 above. **Q3. Contributions of the calibration loss and copula preprocessing** Please see our response to your W2 above.