Prompt-CMI: In-Context Estimation of Conditional Mutual Information

Conditional mutual information (CMI) measures the dependence between two variables that remains after conditioning on others, and it is the quantity behind conditional independence testing, feature selection and dependence discovery. Statistical estimators lose the signal as dimension grows, and neural estimators must be trained anew on every dataset. We introduce Prompt-CMI, a network pretrained once on synthetic tasks that estimates CMI on unseen tables with frozen parameters. Prompt-CMI reads a context of observations from the table and assigns every observation an information score, trained to approximate the log-density ratio between the joint law and its conditionally independent reference; the average score is the CMI estimate. A context encoder summarizes the context into a fixed-size memory for any sample size and variable dimensions, and a synthetic task generator supplies exact reference samples and known CMI for pretraining. On synthetic benchmarks, Prompt-CMI attains the lowest error across seven dependence mechanisms, up to 60 dimensions and CMI levels from 0 to 6 nats, at over a hundredfold lower cost than a conventional neural estimator, and its frozen scores serve for weak-dependence detection, feature selection and conditional independence testing on real data.