acceptodds
Under review as a conference paper at ICLR 2027

Prompt-CMI: In-Context Estimation of Conditional Mutual Information

Abstract

Conditional mutual information (CMI) measures the dependence between two variables that remains after conditioning on others, and it is the quantity behind conditional independence testing, feature selection and dependence discovery. Statistical estimators lose the signal as dimension grows, and neural estimators must be trained anew on every dataset. We introduce Prompt-CMI, a network pretrained once on synthetic tasks that estimates CMI on unseen tables with frozen parameters. Prompt-CMI reads a context of observations from the table and assigns every observation an information score, trained to approximate the log-density ratio between the joint law and its conditionally independent reference; the average score is the CMI estimate. A context encoder summarizes the context into a fixed-size memory for any sample size and variable dimensions, and a synthetic task generator supplies exact reference samples and known CMI for pretraining. On synthetic benchmarks, Prompt-CMI attains the lowest error across seven dependence mechanisms, up to 60 dimensions and CMI levels from 0 to 6 nats, at over a hundredfold lower cost than a conventional neural estimator, and its frozen scores serve for weak-dependence detection, feature selection and conditional independence testing on real data.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conditional mutual information, tabular foundation model, in-context learning, density-ratio estimation, conditional independence testing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3f2YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Prompt-CMI, an in-context conditional mutual information (CMI) estimator that eliminates the need for dataset-specific training. The model is pretrained on synthetic distributions using joint and conditionally independent samples, together with known CMI values for calibration. At inference, a context encoder summarizes the input dataset, and a frozen information decoder estimates CMI through context-conditioned density-ratio scores. Experiments demonstrate competitive estimation accuracy and substantial computational efficiency across synthetic benchmarks, with additional validation on real-world downstream tasks.

Critical strengths
  • Meaningful methodological contribution: Extending pretrained information estimation to CMI is a relevant and challenging problem. The proposed context-conditioned density-ratio estimation and synthetic reference generation provide a reasonable framework for avoiding dataset-specific optimization.

  • Strong empirical results and efficiency: Prompt-CMI achieves lower estimation errors than conventional statistical and neural CMI estimators across diverse synthetic settings, while substantially reducing inference time. The downstream experiments further demonstrate its potential practical utility.

Critical weaknesses
  • Insufficient validation of generalization to unseen dependence mechanisms: The main contribution is a reusable CMI estimator pretrained on synthetic distributions. However, the evaluation does not clearly establish whether the model generalizes to dependence mechanisms excluded from pretraining. Since the effectiveness of this approach depends heavily on the coverage of the synthetic task distribution, this is a central concern regarding its general applicability.

  • Limited validation of the proposed design and comparison with pretrained MI estimators: The paper does not sufficiently isolate the contributions of important components, particularly the calibration loss and copula preprocessing. Moreover, although related pretrained MI estimators such as InfoNet and InfoAtlas are discussed, the paper lacks a direct comparison with CMI estimation obtained through the MI chain rule using these models. Such comparisons would help establish the benefits of directly learning a context-conditioned CMI estimator.

Questions for authors
  • Generalization to unseen mechanisms: Can the authors evaluate Prompt-CMI on dependence mechanisms explicitly excluded from synthetic pretraining, for example through a leave-one-mechanism-out experiment? This would be important for assessing whether the model learns a generalizable CMI estimation procedure rather than relying on patterns represented during pretraining.

  • Comparison with pretrained MI estimation: Can the authors compare Prompt-CMI with a CMI estimator constructed from pretrained MI models such as InfoNet or InfoAtlas using the identity (I(X;Y\mid Z)=I(X;Y,Z)-I(X;Z))? This would clarify the practical advantage of directly estimating CMI.

  • Component contributions: Can the authors provide ablation results isolating the effects of the calibration loss and copula preprocessing on estimation accuracy?

  • I would be willing to raise my rating if the rebuttal provides convincing evidence of generalization to previously unseen dependence mechanisms and clarifies the advantages over existing pretrained MI-based alternatives.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer 3f2Y

We sincerely thank Reviewer 3f2Y for the thoughtful review and for recognizing our methodological contribution, estimation accuracy, and computational efficiency. Following your suggestions, we have evaluated generalization to distributions excluded from pretraining, compared against MI-based baselines, and assessed the training objectives and copula preprocessing.

W1. Evidence for Prompt-CMI’s generalization

To provide further evidence of generalization, we have evaluated four additional distributions explicitly excluded from synthetic pretraining—Gamma (log), Beta, Logistic, and Gumbel—using frozen Prompt-CMI. All four settings use (dX,dY,dZ)=(20,20,8)(d_X,d_Y,d_Z)=(20,20,8). Prompt-CMI gives the closest mean estimate to the true CMI in all four cases.

These results, together with the main paper’s real-world feature-selection, dependence-ordering, and conditional-independence experiments, support practical generalization. We have included these generalization experiments in Appendix H of the revised manuscript.

\begin{array}{lccccccc} \hline \textbf{Metric} & \textbf{Distribution} & I^\star & \textbf{Prompt-CMI} & \textbf{KNN} & \textbf{CCMI} & \textbf{CMINE} & \textbf{DINE} \\\\ \hline \text{Est. CMI} & \text{Gamma (log)} & 4 & \mathbf{3.627\pm 0.525} & 0.227\pm 0.019 & 0.998\pm 0.295 & 2.540\pm 1.658 & 3.299\pm 0.140 \\\\ & \text{Beta} & 4 & \mathbf{4.267\pm 0.367} & 0.224\pm 0.029 & 0.842\pm 0.198 & 2.362\pm 0.987 & 3.297\pm 0.137 \\\\ & \text{Logistic} & 4 & \mathbf{3.837\pm 0.593} & 0.244\pm 0.023 & 0.930\pm 0.212 & 2.905\pm 1.607 & 3.466\pm 0.142 \\\\ & \text{Gumbel} & 4 & \mathbf{3.584\pm 0.545} & 0.241\pm 0.026 & 0.928\pm 0.225 & 2.204\pm 1.264 & 3.282\pm 0.172 \\\\ \hline \end{array}$$ **W2. Comparison with MI baselines and component ablations** We have compared Prompt-CMI with CMI baselines constructed from MIST, InfoAtlas, and KSG via MI identities across **all experimental tasks in the main paper**. **Prompt-CMI performs best on all summary metrics below**. We have included these comparisons in Appendix E of the revised manuscript. $$\def\arraystretch{1.25} \begin{array}{lcccccc} \hline \textbf{Metric} & (d_X,d_Y,d_Z) & I^\star & \textbf{Prompt-CMI} & \textbf{MIST} & \textbf{InfoAtlas} & \textbf{KSG-chain} \\\\ \hline \text{MAE}\\,\downarrow & (20,20,8) & 4 & \mathbf{0.9825} & 3.7692 & \text{—} & 3.8931 \\\\ & (4,4,8) & 0.8 & \mathbf{0.0799} & 0.5466 & 0.2677 & 0.5981 \\\\ \text{FS: NRMSE}\\,\downarrow & \text{—} & \text{—} & \mathbf{0.1963} & 0.3157 & 0.3220 & 0.3155 \\\\ \text{Order: IR (\\%)}\\,\downarrow & \text{—} & \text{—} & \mathbf{1.04} & 52.06 & 8.90 & 3.72 \\\\ \text{CI: AUROC}\\,\uparrow & \text{—} & \text{—} & \mathbf{0.8048} & 0.5301 & 0.6852 & 0.7034 \\\\ \hline \end{array}$$ In addition, to assess the components you highlighted, we have compared classification-only and calibration-only training and removed copula preprocessing. As shown below, **the full Prompt-CMI model achieves the lowest MAE in every setting**. We have included these ablation experiments in Appendix F of the revised manuscript. $$\def\arraystretch{1.25} \begin{array}{lcccccc} \hline \textbf{Metric} & (d_X,d_Y,d_Z) & I^\star & \textbf{Prompt-CMI} & \textbf{GT-only} & \textbf{BCE-only} & \textbf{Copula-off} \\\\ \hline \text{MAE}\\,\downarrow & (20,20,8) & 4 & \mathbf{0.9825} & 1.5277 & 3.2251 & 2.2131 \\\\ & (4,4,8) & 0.8 & \mathbf{0.0799} & 0.1581 & 0.4543 & 0.2471 \\\\ \hline \end{array}$$ **Q1. Generalization to mechanisms excluded from pretraining** Please see our response to your W1 above. **Q2. CMI estimation via the MI chain rule using pretrained MI models** Please see our response to your W2 above. **Q3. Contributions of the calibration loss and copula preprocessing** Please see our response to your W2 above.

As the discussion deadline approaches, We would be grateful to hear whether our responses have adequately addressed your concerns! Feel free to let us know if you have any other concerns. Thank you again for your time and consideration !

The authors

Reviewer_vPrXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an 'in-context' estimator of conditional mutual information, based on a pre-trained attention based setup for predicting the pointwise CMI for each 'query' point given the 'context' which is also obtained from another pre-trained network(s) after processing a subset of the input observations. The final CMI is computed as the average of the pointwise CMIs. The pre-training is done on a bunch of synthetic tasks where the product and reference distributions are known and the network is trained to predict a sample from one or the other via a binary sigmoidal output. On 7 synthetic distribution benchmarks prompt-CMI does better than some other general CMI estimation counterparts, and the critical point is that it takes hundred times less compute to do so. The utility of prompt-CMI is also demonstrated in some other real world tasks like feature selection and CI testing.

Critical strengths
  • In context estimation of MI or CMI is not an easy task due to arbitrary dependencies, so the standardization elements of this work was nice to see. The paper is striving to solve quite a challenging problem in that regard.
  • An immediate benefit of pulling this off would be the significant compute reduction as re-training is avoided, which is demonstrated in this work, and is the strongest selling point of the proposed approach.
Critical weaknesses

Clarity and writing are my biggest concerns with this. Both need significant levels of improvement throughout the paper.

Isn't taking a conditioning on a subset not using the full sampling information from P and also subset dependent?

The pre-processing map psi_C is suddenly introduced, Also, keeping both the rank Gaussian and whitened co-ordinates is weird, it is hard to see what is the benefit from keeping both info. As CMI is invariant to this, isn't this more information than needed? As in, ZCA whitening is invertible, doing this does not lose information, so I'm not sure why the authors are also keeping the rank-Gaussian transformed co-ordinates.

The synthetic part of the experiments has the risk of being quite circular: The generating "laws" in the pre-training itself are not well-explained and it is unclear what degree of overlap is there between the 7 different types of distributions and the "laws" and "cores" in the synthetic data generation process. Some of the choices in the generating process seem ad-hoc as well. If there is overlap there is a risk of the better performance coming from memorization/overfitting.

The method's description is quite obscure and requires effort to make sense of. Mainly, the axial encoding can be described a bit better with more precision regarding the precise attention mechanisms with clear examples, and subsequently, the role pooling's connection to the output of this encoding must also be clarified as it is not clear currently. It is also not clear how the 'condensation' part happens for compressing from nC to Kr number of vectors, and also it is not clear what the 'memenc' and the 'pool' parts in (7) are actually doing. The description of the method and its details must be significantly improved.

In the feature selection experiments, again the writing is very obscure and it is hard to see what exactly is being done here to arrive at the selected features. The same applies for the order preservation experiments. Please describe what actually are X,Y and Z in all these real world tasks, what they represent. What are "donor" rows? The description of the real world experiments are also not proper and need significant more exposition.

Furthermore, the results in the CMI feature selection experiments aren't very significant, best results from the prompt-cmi approach only happen for certain number of feature groups. So it is hard to see the actual significance of the results in Table 2.

There is no theoretical element to this work except for minor intuitive ones such as the symmetry of CMI to X,Y swaps. Therefore it is hard to truly gauge the effectiveness of the proposed approach. Furthermore, there are no examples of their proposed framework, in a simplified setting, to truly see that in context CMI estimation could even be possible. Some toy theoretical examples would be useful.

Smaller points:

reference for cmine?

The analysis is up to cumulative dimensions of 60 (x+y+z). Would be interesting to see if this can scale up beyond this number as in most real world tasks the dimensionality would be higher.

Questions for authors

Please see weaknesses

4 comments
AuthorsOpenReview ↗

Reply to Reviewer vPrX - Part 1/3

We sincerely thank Reviewer vPrX for the careful review and for recognizing our standardization design and computational efficiency. To address these concerns, we further clarify the architecture, data generation, and evaluation protocols, and add a consistency analysis and high-dimensional results.

W1. Analysis of information utilization in the Context-Query architecture

Holding queries out of their context prevents self-matching and the resulting overfitting bias. Each query is therefore scored using a context formed only from other observations, so the scorer relies on the distributional information they provide rather than matching the query to itself. This separation does not discard any data: with four-fold readout, every observation is scored once as a query and used three times as context, and the final estimate averages the held-out scores over all observations, so we make full use of all sample information without relying on a single fixed context subset.

W2. Rationale for retaining both rank-Gaussian and whitened coordinates

Thank you for your suggestion. We have adjusted where the map ΨC\Psi_C is introduced and clarified the role of each view in Section 3.1 of the revised manuscript. Rank-Gaussian normalization unifies the marginals, and role-wise ZCA whitening further decorrelates variables within each role. Retaining both complementary views lets the model capture variable-aligned dependencies alongside covariance-normalized geometry, together providing more effective input preprocessing and making dependencies easier for the model to identify. The input-view ablation below reports mean absolute error (MAE).

\begin{array}{lccccc} \hline \textbf{Metric} & (d_X,d_Y,d_Z) & I^\star & \textbf{Prompt-CMI} & \textbf{Gaussian-only} & \textbf{ZCA-only} \\\\ \hline \text{MAE}\\,\downarrow & (20,20,8) & 4 & \mathbf{0.9825} & 2.3457 & 2.5415 \\\\ & (4,4,8) & 0.8 & \mathbf{0.0799} & 0.2740 & 0.1186 \\\\ \hline \end{array}$$ **W3. Further clarification on data generation** We appreciate the concern about memorization. We have clarified our data-generation process in Section 4.2 of the revised manuscript. Our generator follows the prior-design philosophy of tabular foundation models such as TabPFN-3 (Grinsztajn et al., 2026) and aims to produce diverse statistical worlds that capture structural patterns of real-world data. **Pretraining across these worlds encourages the model to learn a transferable CMI estimation procedure rather than fit a particular dataset.** A core defines a latent dependence mechanism, a law specifies its concrete configuration, and each table contains observations drawn independently from that law. Laws and tables are sampled on the fly during pretraining rather than drawn from a fixed dataset collection. The generator’s design is guided by coverage of diverse dependence structures, **providing a principled basis for our generation choices**. **Our pretraining was not tailored to the specific benchmark distributions, although some overlap remains between them and our pretraining core families.** To further assess generalization, we evaluated the same Prompt-CMI on distributions generated from four additional dependence cores—Gamma (log), Beta, Logistic, and Gumbel—which were explicitly excluded from pretraining. All four settings use $(d_X,d_Y,d_Z)=(20,20,8)$, and the table below reports the estimated CMI. The main paper’s real-world experiments also provide complementary evidence of generalization. $$\def\arraystretch{1.25} \begin{array}{lccccccc} \hline \textbf{Metric} & \textbf{Distribution} & I^\star & \textbf{Prompt-CMI} & \textbf{KNN} & \textbf{CCMI} & \textbf{CMINE} & \textbf{DINE} \\\\ \hline \text{Est. CMI} & \text{Gamma (log)} & 4 & \mathbf{3.627\pm 0.525} & 0.227\pm 0.019 & 0.998\pm 0.295 & 2.540\pm 1.658 & 3.299\pm 0.140 \\\\ & \text{Beta} & 4 & \mathbf{4.267\pm 0.367} & 0.224\pm 0.029 & 0.842\pm 0.198 & 2.362\pm 0.987 & 3.297\pm 0.137 \\\\ & \text{Logistic} & 4 & \mathbf{3.837\pm 0.593} & 0.244\pm 0.023 & 0.930\pm 0.212 & 2.905\pm 1.607 & 3.466\pm 0.142 \\\\ & \text{Gumbel} & 4 & \mathbf{3.584\pm 0.545} & 0.241\pm 0.026 & 0.928\pm 0.225 & 2.204\pm 1.264 & 3.282\pm 0.172 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Reply to Reviewer vPrX - Part 2/3

W4. Further explanation of the model architecture

Thank you for this feedback. We have added further details on the attention operations and tensor shapes, together with a concrete example, to make the model architecture clearer in Section 3.2. In brief:

  • Axial encoding produces a token grid VC∈RnC×D×dV_C\in\mathbb{R}^{n_C\times D\times d}, one token per observation–variable pair, by alternating row attention among each observation’s DD tokens and column attention among each variable’s nCn_C tokens via KcK_c inducing vectors.
  • Role pooling collapses the variable axis: for each row, learned queries attend to its XX, YY, ZZ, and all tokens, and the four outputs, with X/YX/Y combined symmetrically, form HC∈RnC×4dH_C\in\mathbb{R}^{n_C\times4d}.
  • Eq. (7): Pool\mathrm{Pool} cross-attends KrK_r learned queries to the nCn_C rows of HCH_C, compressing them into KrK_r vectors, which MemEnc\mathrm{MemEnc} refines by self-attention into MC∈RKr×4dM_C\in\mathbb{R}^{K_r\times4d}.

For example, with d=128d=128 and Kr=512K_r=512, a 768-row, 4-variable context maps as 768×4×128→768×512→512×512768\times4\times128\to768\times512\to512\times512. To assess the contribution of role pooling, we have also included an ablation study in our response to Reviewer HGGV (W2.2).

W5. Clearer definitions of real-world tasks

Thank you for highlighting the need for clearer real-world task definitions and protocols. We have refined the descriptions of the real-world experiments in Section 5.2 to clarify the experimental setups and evaluation protocols. In brief:

  • Feature selection. XX is a candidate feature group, YY the target, and ZZ the background variables plus previously selected groups. At each step, we add the group with the largest estimated I(X;Y∣Z)I(X;Y\mid Z) on training data, then train identical MLPs on the selected features and report test NRMSE.
  • Order preservation. A donor row is another observation matched to the recipient on ZZ alone. Swapping in the donor’s XX keeps Y,ZY,Z intact but breaks the XX–YY link given ZZ, so CMI should decrease as more rows are swapped. The inversion rate measures how often an estimate increases instead.
  • Conditional independence. For each variable pair (X,Y)(X,Y), ZZ contains all remaining variables. Pairs are ranked by 1−p1-p, and AUROC is computed against the reference network.

Strength of the feature-selection results. Under the protocol above, Table 2 uses test NRMSE to evaluate the downstream predictive utility of the selected feature groups. Using a single frozen model without refitting, Prompt-CMI consistently outperforms neural CMI baselines and remains competitive with dedicated feature selectors as the conditioning set grows.

AuthorsOpenReview ↗

Reply to Reviewer vPrX - Part 3/3

W6. Theoretical analysis and consistency proof for Prompt-CMI

We have established consistency of in-context CMI estimation for a finite family of distinguishable tasks. Under the assumptions detailed in Appendix A.1, the population-optimal in-context BCE estimator (Eqs. 8–9) satisfies

\mathbb{E}\\!\bigl[(\widehat I_{m,n}-I)^2\bigr] \le \tfrac{C_1}{n}+C_2\beta^m, \quad 0<\beta<1.$$ where $m$ and $n$ are the context and query sizes. The $\beta^m$ term vanishes exponentially as the context identifies the task, and $C_1/n$ is the held-out averaging error, giving $O(N^{-1})$ MSE under four-fold cross-fitting. Thus, **a single scorer shared across tasks consistently estimates CMI from context alone**. We have added the full conditions, proof, and an explicit example with both zero and positive CMI to Appendix A.1 of the revised manuscript. **W7. Reference for CMINE** Thank you for requesting clarification. Our CMINE baseline uses the log-density-ratio (LDR) estimator of Molavipour et al., *“Neural Estimators for Conditional Mutual Information Using Nearest Neighbors Sampling”* (IEEE TSP, 2021). We have made this attribution explicit at the first mention of CMINE in Section 2. **W8. Prompt-CMI’s performance on high-dimensional data** We have tested both higher observed dimensions and strong role imbalance using frozen Prompt-CMI with role-wise PCA. Results are shown below. $$\def\arraystretch{1.25} \begin{array}{lccccccc} \hline \textbf{Metric} & (d_X,d_Y,d_Z) & N & I^\star & \textbf{Prompt-CMI} & \textbf{CCMI} & \textbf{CMINE} & \textbf{DINE-HD} \\\\ \hline \text{MAE}\\,\downarrow & (170,170,172) & 1024 & 2 & \mathbf{0.4672} & 1.0120 & 1.2726 & 0.9145 \\\\ & (508,508,8) & 2048 & 2 & \mathbf{0.4996} & 0.6876 & 0.6018 & 0.6756 \\\\ & (4,4,504) & 2048 & 0.8 & \mathbf{0.0826} & 0.2362 & 0.1671 & 0.2171 \\\\ & (756,4,8) & 1024 & 0.8 & \mathbf{0.1036} & 0.3042 & 0.3963 & 0.2544 \\\\ & (4,4,1016) & 1024 & 0.8 & \mathbf{0.0770} & 0.2489 & 0.2230 & 0.3047 \\\\ & (256,256,256) & 2048 & 0 & \mathbf{0.0149} & 0.0796 & 0.0163 & 0.0863 \\\\ \hline \end{array}$$ We have also evaluated conditional image–text dependence on COCO, with $X,Y$ as 512-dimensional CLIP image and caption embeddings and $Z$ as a 12-dimensional one-hot encoding of the main object's supercategory. Using class-centered PCA, Prompt-CMI achieves the lowest zero-CMI MAE and the best ordering correlations under both pairing and noise perturbations. We have included the full COCO results in Appendix G.2 of the revised manuscript. $$\def\arraystretch{1.25} \begin{array}{lccccc} \hline \textbf{Metric} & \textbf{Setting} & \textbf{Prompt-CMI} & \textbf{CCMI} & \textbf{CMINE} & \textbf{DINE} \\\\ \hline \text{MAE}\\,\downarrow & \text{Zero CMI} & \mathbf{0.00099} & 0.59049 & 0.01710 & 0.04754 \\\\ \text{Spearman}\\,\rho\\,\text{}\\,\uparrow & \text{Pairing} & \mathbf{1.00} & 0.98 & 0.93 & 1.00 \\\\ & \text{Added noise} & \mathbf{0.99} & 0.56 & 0.90 & 0.98 \\\\ \hline \end{array}$$

As the discussion deadline approaches, We would be grateful to hear whether our responses have adequately addressed your concerns! Feel free to let us know if you have any other concerns. Thank you again for your time and consideration !

The authors

Reviewer_HGGVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

1 Paper Summary

Prompt-CMI estimates conditional mutual information from a finite tabular sample in a single forward pass without test-time parameter optimization [1]. Conditional mutual information measures the residual dependence between two target variable groups given a conditioning group, providing the foundation for conditional independence testing, causal structure learning, and feature selection. Estimating conditional mutual information from finite data is challenging because the conditional independence reference distribution is unknown, and classical local permutation or nearest-neighbor schemes degrade as the conditioning dimension grows [2, 3]. Existing neural estimators train a variational or classifier critic on each dataset from scratch, which incurs high optimization variance and slow per-table runtime [4, 5, 6].

Prompt-CMI replaces per-dataset training with amortized in-context inference over frozen transformer weights [7, 8]. A subset of rows serves as the context split, and the network scores held-out query rows by predicting a pointwise log-density-ratio witness between the joint distribution and the conditionally independent reference distribution. The method combines four stages:

  1. Context-fitted copula preprocessing: Exploiting the invariance of conditional mutual information to monotone marginal transforms and role-wise invertible maps, the pipeline applies an empirical rank-Gaussian transform to continuous columns followed by regularized role-wise ZCA whitening fitted on the context split. Discrete columns are represented using context-support frequency summaries and equality-group prototypes.
  2. Bi-axial context encoding and memory compression: Embedded table cells pass through alternating variable-axis self-attention and sample-axis Set Transformer inducing-point attention [9, 10, 11], caching variable-level keys and values in a variable memory. A role-pooling module aggregates variable tokens by role (the two target groups, the conditioning group, and the full row) into a fixed-width row descriptor. To keep query evaluation cost independent of context size, an inducing-point pooler and self-attention stack compress all context row descriptors into a fixed-size table memory [12].
  3. Information decoding and four-fold cross-fitting: Each query row is transformed using the context-fitted preprocessor, encoded along the variable axis by reading the cached variable memory without updating it, pooled by role, and scored by cross-attending to the compressed table memory. Dataset-level conditional mutual information is computed by averaging out-of-fold query scores across a four-fold partition of the dataset.
  4. Synthetic pretraining with exact conditional references: The model is pretrained offline across synthetic structural causal models with known ground-truth conditional mutual information. For each conditioning state, drawing two independent target pairs from the known conditional generator produces an exact joint sample and an exact conditionally independent reference sample without nearest-neighbor approximation error. Training minimizes a sum of balanced binary cross-entropy on joint versus reference queries and a Huber calibration loss on the average joint score.
Critical strengths

2 Critical strength

  1. Important problem domain Conditional mutual information estimation is a foundational problem with far reaching applications across subfields of science and engineering.
  2. Decoupled context-memory architecture and cross-fitted evaluation: Compressing the context into fixed-size variable and table memories makes query scoring independent of the context sample size and enables sub-second four-fold cross-fitted evaluation on datasets of up to ten thousand rows. Scoring held-out folds against frozen context memories avoids the in-sample critic memorization that biases standard neural mutual information estimators [4, 13].
Critical weaknesses

3 Critical weakness

  1. Lack of citing and comparing against prior work on amortized information estimation: The paper claims that existing estimators require per-dataset optimization and frames amortized zero-shot mutual information estimation as largely unaddressed beyond bivariate Gaussian InfoNet [14]. This framing omits directly relevant prior work published more than six months before submission. In particular, MIST [15] already formulates zero-shot mutual information estimation by pretraining a two-dimensional Set Transformer / Perceiver architecture on synthetic copula and normalizing-flow meta-distributions; DEMINE [16] introduced meta-learned neural mutual information estimation across synthetic task families using context and query splits; AVICI [10] introduced alternating variable-by-sample axial attention pretrained on synthetic structural causal models for amortized conditional independence structure learning; and amortized variational mutual information estimators are well established in Bayesian experimental design and dynamic feature selection [17, 18, 19]. Furthermore, any pretrained zero-shot mutual information estimator immediately defines a zero-shot conditional mutual information baseline via the chain rule by subtracting two mutual information estimates [5, 20], yet this baseline is neither cited nor evaluated.
  2. Low-dimensional evaluation and weak experimental protocol: All experiments are restricted to low-dimensional, symmetric settings with at most 60 total variables (target dimensions equal and at most 26, conditioning dimension fixed at 8, and latent dependence rank capped at 8). Neural conditional mutual information estimators are primarily motivated by high-dimensional settings (such as 512 or more dimensions) and strongly imbalanced role dimensions (for example, scalar targets conditioned on hundreds of covariates, or high-dimensional representations conditioned on a low-dimensional variable), none of which are evaluated. Even at target dimension 16 and 20, Prompt-CMI collapses on the regime-switching benchmark (Figure 4 and Table 4). Moreover, the experimental protocol omits ablations isolating the classification loss from the direct scalar calibration loss, the copula preprocessing, and the role-pooling heuristic, while the conditional independence test in Section 5.4 still relies on 199 local nearest-neighbor permutations in the conditioning set (requiring 800 context-encoding passes per pair and reintroducing nearest-neighbor degradation in the conditioning dimension) [3].
  3. Differentiability of the estimator is not investigated: A central motivation for neural mutual information and conditional mutual information estimators over classical non-parametric estimators is differentiability with respect to upstream model parameters, enabling gradient-based representation learning, conditional information bottlenecks, and causal discovery [4, 17, 21]. The paper neither develops a differentiable path through the estimator nor evaluates gradient quality or downstream optimization performance, restricting the method to static post-hoc tabular estimation where classical non-parametric estimators already apply.
Questions for authors

4 Questions for author

  1. Missing prior work and chain-rule amortized baselines: Why were MIST [15], DEMINE [16], AVICI [10], and amortized experimental design estimators [17, 18] omitted from the related work, and how does Prompt-CMI compare against applying the chain rule of conditional mutual information using pretrained zero-shot mutual information estimators such as MIST or InfoNet [14]?
  2. High-dimensional scaling, role imbalance, and experimental ablations: How does Prompt-CMI perform when the total dimension scales to 512 or higher, or when the target and conditioning dimensions are strongly imbalanced? In addition, what are the ablation results separating the binary classification loss from the scalar calibration loss, and why does Prompt-CMI fail on the regime-switching mechanism once the target dimension reaches 16?
  3. Differentiability and downstream gradient-based use cases: Can Prompt-CMI be made differentiable with respect to upstream representations or input distributions (for example, by replacing or relaxing the empirical rank transformation), and how accurate are the resulting gradients when optimizing a downstream objective?

5 References

[1] Anonymous, “Prompt-CMI: In-context estimation of conditional mutual information,” /Under review as a conference paper at ICLR 2027/, 2027.

[2] R. Sen, A. T. Suresh, K. Shanmugam, A. G. Dimakis, and S. Shakkottai, “Model-powered conditional independence test,” in /Advances in neural information processing systems/, vol. 30, p. 2951–2961, 2017.

[3] J. Runge, “Conditional independence testing based on a nearest-neighbor estimator of conditional mutual information,” in /Proceedings of the 21st international conference on artificial intelligence and statistics/, PMLR, vol. 84, p. 938–947, 2018.

[4] M. I. Belghazi, A. Baratin, S. Rajeshwar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm, “Mutual information neural estimation,” in /Proceedings of the 35th international conference on machine learning/, PMLR, vol. 80, p. 531–540, 2018.

[5] S. Mukherjee, H. Asnani, and S. Kannan, “CCMI: Classifier based conditional mutual information estimation,” in /Proceedings of the 36th conference on uncertainty in artificial intelligence/, PMLR, vol. 124, p. 1083–1093, 2020.

[6] S. Molavipour, G. Bassi, and M. Skoglund, “Neural estimators for conditional mutual information using nearest neighbors sampling,” /IEEE transactions on signal processing/, vol. 69, p. 2672–2685, 2021.

[7] N. Hollmann, S. Müller, L. Purucker, A. Krishnakumar, M. Körfer, S. B. Hoo, R. T. Schirrmeister, and F. Hutter, “Accurate predictions on small data with a tabular foundation model,” /Nature/, vol. 637, p. 319–326, 2025.

[8] J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan, “TabICLv2: A better, faster, scalable, and open tabular foundation model,” /arXiv preprint arXiv:2602.11139/, 2026.

[9] J. Lee, Y. Lee, J. Kim, A. R. Kosiorek, S. Choi, and Y. W. Teh, “Set transformer: A framework for attention-based permutation-invariant neural networks,” in /Proceedings of the 36th international conference on machine learning/, PMLR, vol. 97, p. 3744–3753, 2019.

[10] L. Lorch, S. Sussex, J. Rothfuss, A. Krause, and B. Schölkopf, “Amortized inference for causal structure learning,” in /Advances in neural information processing systems/, vol. 35, p. 13104–13118, 2022.

[11] M. Bouadi, P. Seth, A. Tanna, and V. K. Sankarapu, “Orion-Bix: Bi-axial attention for tabular in-context learning,” /arXiv preprint arXiv:2512.00181/, 2025.

[12] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, O. Hénaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira, “Perceiver IO: A general architecture for structured inputs and outputs,” in /International conference on learning representations/, 2022.

[13] J. Song and S. Ermon, “Understanding the limitations of variational mutual information estimators,” in /International conference on learning representations/, 2020.

[14] Z. Hu, K. Song, Q. Zeng, Z. Wang, and K. Huang, “InfoNet: Neural estimation of mutual information without test-time optimization,” in /Proceedings of the 41st international conference on machine learning/, PMLR, vol. 235, p. 19437–19462, 2024.

[15] G. Gritsai, M. Richards, M. Meloux, K. Cho, and M. Peyrard, “MIST: Mutual information estimation via supervised training,” /arXiv preprint arXiv:2511.18945/, 2025.

[16] L. Xiao, I. Sur, S. A. Nastase, A. Split, N. B. Turk-Browne, U. Hasson, and M. Kapadia, “Data-efficient mutual information neural estimator,” /arXiv preprint arXiv:1905.03319/, 2019.

[17] A. Foster, D. R. Ivanova, I. Malik, and T. Rainforth, “Deep adaptive design: Amortizing sequential Bayesian experimental design,” in /Proceedings of the 38th international conference on machine learning/, PMLR, vol. 139, p. 3384–3395, 2021.

[18] D. R. Ivanova, A. Foster, S. Kleinegesse, M. U. Gutmann, and T. Rainforth, “Implicit deep adaptive design: Policy-based experimental design without likelihoods,” in /Advances in neural information processing systems/, vol. 34, p. 25785–25798, 2021.

[19] I. Covert, W. Qiu, M. Lu, N. Kim, N. White, and S.-I. Lee, “Learning to maximize mutual information for dynamic feature selection,” /Proceedings of the 40th international conference on machine learning/, vol. 202, p. 6424–6447, 2023.

[20] G. Franzese, M. Bounoua, and P. Michiardi, “MINDE: Mutual information neural diffusion estimation,” in /International conference on learning representations/, 2024.

[21] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker, “On variational bounds of mutual information,” in /Proceedings of the 36th international conference on machine learning/, PMLR, vol. 97, p. 5171–5180, 2019.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer HGGV - Part 1/2

We sincerely thank Reviewer HGGV for the thorough and constructive review and for recognizing the importance of CMI estimation, the strengths of our context-memory architecture and cross-fitted evaluation. In the responses below, we expand the baseline and high-dimensional evaluations, assess key model components, and clarify the Switch results, conditional-independence procedure, and differentiability.

W1. Missing related work and comparison with MI baselines

Thank you for pointing out these missing references. We have added these references and refined the related-work discussion in Section 6 of the revised manuscript. And we have also compared Prompt-CMI with CMI baselines constructed from MIST, InfoAtlas, and KSG via MI identities across all experimental tasks in the main paper. Prompt-CMI performs best on all summary metrics below.

\begin{array}{lcccccc} \hline \textbf{Metric} & (d_X,d_Y,d_Z) & I^\star & \textbf{Prompt-CMI} & \textbf{MIST} & \textbf{InfoAtlas} & \textbf{KSG-chain} \\\\ \hline \text{MAE}\\,\downarrow & (20,20,8) & 4 & \mathbf{0.9825} & 3.7692 & \text{—} & 3.8931 \\\\ & (4,4,8) & 0.8 & \mathbf{0.0799} & 0.5466 & 0.2677 & 0.5981 \\\\ \text{FS: NRMSE}\\,\downarrow & \text{—} & \text{—} & \mathbf{0.1963} & 0.3157 & 0.3220 & 0.3155 \\\\ \text{Order: IR (\\%)}\\,\downarrow & \text{—} & \text{—} & \mathbf{1.04} & 52.06 & 8.90 & 3.72 \\\\ \text{CI: AUROC}\\,\uparrow & \text{—} & \text{—} & \mathbf{0.8048} & 0.5301 & 0.6852 & 0.7034 \\\\ \hline \end{array}$$ **W2.1. Prompt-CMI’s performance in high-dimensional and role-imbalanced settings** We have tested both higher observed dimensions and strong role imbalance using frozen Prompt-CMI with role-wise PCA, and have included the high-dimensional experiments in Appendix G.1 of the revised manuscript. The table below reports mean absolute error (MAE). **Prompt-CMI achieves the lowest MAE across all settings.** $$\def\arraystretch{1.25} \begin{array}{lccccccc} \hline \textbf{Metric} & (d_X,d_Y,d_Z) & N & I^\star & \textbf{Prompt-CMI} & \textbf{CCMI} & \textbf{CMINE} & \textbf{DINE-HD} \\\\ \hline \text{MAE}\\,\downarrow & (170,170,172) & 1024 & 2 & \mathbf{0.4672} & 1.0120 & 1.2726 & 0.9145 \\\\ & (508,508,8) & 2048 & 2 & \mathbf{0.4996} & 0.6876 & 0.6018 & 0.6756 \\\\ & (4,4,504) & 2048 & 0.8 & \mathbf{0.0826} & 0.2362 & 0.1671 & 0.2171 \\\\ & (756,4,8) & 1024 & 0.8 & \mathbf{0.1036} & 0.3042 & 0.3963 & 0.2544 \\\\ & (4,4,1016) & 1024 & 0.8 & \mathbf{0.0770} & 0.2489 & 0.2230 & 0.3047 \\\\ & (256,256,256) & 2048 & 0 & \mathbf{0.0149} & 0.0796 & 0.0163 & 0.0863 \\\\ \hline \end{array}$$ We have also evaluated conditional image–text dependence on COCO, with $X,Y$ as 512-dimensional CLIP image and caption embeddings and $Z$ as a 12-dimensional one-hot encoding of the main object's supercategory. **Using class-centered PCA, Prompt-CMI achieves the lowest zero-CMI MAE and the best ordering correlations under both pairing and noise perturbations.** We have included the full COCO results in Appendix G.2 of the revised manuscript. $$\def\arraystretch{1.25} \begin{array}{lccccc} \hline \textbf{Metric} & \textbf{Setting} & \textbf{Prompt-CMI} & \textbf{CCMI} & \textbf{CMINE} & \textbf{DINE} \\\\ \hline \text{MAE}\\,\downarrow & \text{Zero CMI} & \mathbf{0.00099} & 0.59049 & 0.01710 & 0.04754 \\\\ \text{Spearman}\\,\rho\\,\text{}\\,\uparrow & \text{Pairing} & \mathbf{1.00} & 0.98 & 0.93 & 1.00 \\\\ & \text{Added noise} & \mathbf{0.99} & 0.56 & 0.90 & 0.98 \\\\ \hline \end{array}$$ **W2.2. Ablation results for the binary classification loss, scalar calibration loss, copula preprocessing, and role pooling** To assess the components you highlighted, we have compared classification-only and calibration-only training, removed copula preprocessing, and replaced role-aware pooling (RP) with global pooling. As shown below, **the full Prompt-CMI model achieves the lowest MAE in every setting**. $$\def\arraystretch{1.25} \begin{array}{lccccccc} \hline \textbf{Metric} & (d_X,d_Y,d_Z) & I^\star & \textbf{Prompt-CMI} & \textbf{GT-only} & \textbf{BCE-only} & \textbf{Copula-off} & \textbf{RP-off} \\\\ \hline \text{MAE}\\,\downarrow & (20,20,8) & 4 & \mathbf{0.9825} & 1.5277 & 3.2251 & 2.2131 & 2.4553 \\\\ & (4,4,8) & 0.8 & \mathbf{0.0799} & 0.1581 & 0.4543 & 0.2471 & 0.2446 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Reply to Reviewer HGGV - Part 2/2

W2.3. Prompt-CMI’s failure on the Switch mechanism

The Switch failure can be understood through two properties that become more challenging as dimension grows. In Switch, Cov⁡(X,Y∣Z)=sign⁡(Z1),ρBXBY⊤\operatorname{Cov}(X,Y\mid Z)=\operatorname{sign}(Z_1)\\,\rho B_XB_Y^\top, where BX∈RdX×4B_X\in\mathbb{R}^{d_X\times4} and BY∈RdY×4B_Y\in\mathbb{R}^{d_Y\times4} have orthonormal columns. First, the sign flip makes the correlation direction depend on sign⁡(Z1)\operatorname{sign}(Z_1), while the conditional cross-covariance averages to zero over ZZ. Second, dense mixing spreads the four signal directions across all coordinates, weakening the correlation carried by each coordinate pair as dXd_X and dYd_Y grow. Recovering such regime-dependent, diluted interactions from finite samples is therefore difficult, a bottleneck also reflected in all four baselines severely underestimating Switch throughout Table 4.

W2.4. Limitations of the conditional-independence testing on CI procedure

Thank you for prompting this clarification: the 199 nearest-neighbor resamples in Section 5.2 are used only to calibrate pp-values, not to estimate CMI, and are shared across CMI-based methods. For the edge-ranking evaluation in Table 3, Prompt-CMI’s CMI estimates can also be used directly, without nearest-neighbor resampling. This direct-ranking approach also achieves the highest AUROC among all the tasks.

\begin{array}{lccccccc} \hline \textbf{Metric} & \textbf{Prompt-CMI} & \textbf{KNN} & \textbf{CCMI} & \textbf{CMINE} & \textbf{KCI} & \textbf{RCIT} & \textbf{Fisher-Z} \\\\ \hline \text{Mean AUROC}\\,\uparrow & \mathbf{0.7734} & 0.6963 & 0.5850 & 0.4969 & 0.6366 & 0.6681 & 0.7224 \\\\ \hline \end{array}$$ **W3. Differentiability analysis of Prompt-CMI** Simply replacing the hard rank with a sigmoid-based soft rank ([Qin et al.](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/tr-2008-164.pdf)) can make Prompt-CMI **end-to-end differentiable**: the fixed-fold estimator becomes locally Lipschitz in its continuous inputs and is thus differentiable almost everywhere by [Rademacher’s theorem](https://web.math.utk.edu/~freire/teaching/m448s17/Rademacher.pdf). We have designed a series of synthetic tasks with analytically known CMI optima and used Prompt-CMI gradients to iteratively update their parameters. For example, our Gaussian task uses $(d_X,d_Y,d_Z)=(4,4,8)$ and four tunable mixing angles $\boldsymbol{\alpha}$ controlling the signal–noise mixture in $X$, with analytically known CMI, $$\textstyle I_{\mathrm{true}}(\boldsymbol{\alpha}) =-\tfrac{1}{2}\sum\nolimits_{j=1}^{4}\log(1-0.8^2\cos^2\alpha_j), \quad \alpha_j\in[0.05,\pi/2-0.05].$$ Keeping Prompt-CMI frozen, we have backpropagated through its soft-rank CMI estimate to jointly update these four parameters by gradient ascent. The table reports gradient cosine similarity against the analytic gradient at initialization, and the final true CMI as a percentage of the analytic optimum. $$\def\arraystretch{1.25} \begin{array}{lccccc} \hline \textbf{Metric} & \textbf{Gaussian} & \textbf{Nonlinear marginals} & \textbf{Heteroscedastic} & \textbf{Student-t} & \textbf{Correlation}\\,\rho \\\\ \hline \text{Gradient cosine}\\,\uparrow & \mathbf{0.869} & \mathbf{0.966} & \mathbf{0.935} & \mathbf{0.923} & \mathbf{1.000} \\\\ \text{CMI / optimum}\\,\uparrow & \mathbf{99.10\\%} & \mathbf{99.09\\%} & \mathbf{99.30\\%} & \mathbf{97.74\\%} & \mathbf{100.00\\%} \\\\ \hline \end{array}$$ **Q1. Missing prior work and MI-chain-rule baselines** Please see our response to your W1 above. **Q2. High-dimensional scaling, role imbalance, ablations, and Switch performance** Please see our responses to your W2.1–W2.3 above. **Q3. Differentiability and gradient-based optimization** Please see our response to your W3 above.

As the discussion deadline approaches, We would be grateful to hear whether our responses have adequately addressed your concerns! Feel free to let us know if you have any other concerns. Thank you again for your time and consideration !

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We sincerely thank the AC for handling our submission and the reviewers for their careful evaluation and constructive feedback. Below, we summarize the strengths highlighted by the reviewers and our responses to their main concerns.

We have submitted our detailed rebuttal on November 15 but have not yet received any follow-up feedback from reviewers. We would appreciate the AC’s consideration of our responses and revised manuscript in the remaining evaluation.

Strengths highlighted

The reviewers highlighted the following strengths:

  • Our work provides a tabular foundation-style method for estimating CMI, which is a difficult-to-estimate quantity with broad scientific and engineering applications (HGGV, vPrX, 3f2Y).
  • Methodological innovations in context-conditioned density-ratio estimation and substantial computational savings (HGGV, vPrX, 3f2Y).
  • Strong estimation accuracy across diverse synthetic settings and promising performance on real-world tasks (3f2Y).

Main concerns and our responses

We have addressed all major concerns with clear experimental evidence, theoretical analysis, and targeted manuscript revisions. The main concerns and our clarifications are summarized below.

  • Comparison with MI baselines. Prompt-CMI outperforms MI-derived MIST, InfoAtlas, and KSG baselines on all reported aggregate metrics (Appendix E).
  • High-dimensional evaluation. With PCA, frozen Prompt-CMI achieves the lowest MAE in high-dimensional and role-imbalanced synthetic experiments, and the best zero-CMI error and ordering correlations in real-world experiments on COCO (Appendix G).
  • Component ablations. Prompt-CMI achieves lower MAE than its ablated variants, confirming the contributions of the training losses, copula preprocessing, and role pooling (Appendix F).
  • Generalization. Frozen Prompt-CMI gives the closest estimates on four additional synthetic families, which were not included in the pretraining (Appendix H).

Other concerns. We have also addressed the remaining concerns about differentiability analysis, gradient-based optimization experiments and theoretical support, and have refined our presentation.

Once again, thank you for your time and consideration.

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.