acceptodds
Under review as a conference paper at ICLR 2027

ALICE: In-context, Zero-shot, Mutual Information Estimation

Abstract

Estimating mutual information (MI) from samples is a central objective in a variety of scientific fields. Modern neural estimators are accurate in the large-data regime, but they fall short when data is scarce, and each must be fit anew for every distribution under study. Current estimators are moreover tied to specific data types. These constraints limit their adoption in many applications where per-distribution training is impractical and sample sizes are small. We present ALICE, a foundation model that removes per-distribution training, while achieving competitive estimation accuracy. Trained exclusively on a broad family of synthetic distributions, ALICE acts as an in-context estimator of rectified-flow velocity fields: conditioned on samples of an unseen distribution, it estimates that distribution's velocity field without any explicit training. MI is then obtained through a fixed identity that integrates the squared difference between the joint and conditional fields. We validate ALICE on a standard, challenging benchmark and showcase its applications in three domains, biology, genetics, and neuroscience, whose data the model has never seen.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation Models, Mutual Information Estimation, Scientific Discovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UZaSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a novel zero-shot MI estimator. This is a foundation model for in-context estimation of velocity fields that can be used at any joint width and context length. In contrast to modern neural estimators, which require training on the distribution of interest, the proposed ALICE is a transformer network, pre-trained on a variety of synthetic distributions, to predict their rectified-flow velocity fields, in the spirit of amortized in-context predictors for tabular data.

Critical strengths

A single frozen model estimates MI with no per-distribution training. On the Czyż benchmark the authors report that ALICE (Base variant) has the lowest MAE among the estimators they compare, with the clear margin at low number of samples. Amortized estimators of this kind, including ALICE and InfoAtlas, are an important direction for further research.

Critical weaknesses

I think the reported benchmark does not fully test the generalization claim. Most of the Czyż tasks are based on Gaussians or a Student's tt-distribution. Copula mixtures of both are the training family that the manuscript fully specifies, and the inference copula is designed to remove monotone marginal maps. So it is not clear to me to which extent your model does “generalize to distributions and data types that the corpus does not contain.” Of course, utilizing your method on the respective three scientific applications with no ground-truth MI can be considered as a potential application, but not as a validation of the method.

Questions for authors
  1. Please add a held-out suite with known MI whose dependence is not a Gaussian or Student's copula and not a monotone image of one. You are free to choose the setup, just for example some laws with additive uniform noise, provided this suite is not sampled in pretraining. On that suite, at the same budgets, like N∈{103,104}N \in \{10^3, 10^4\}, compare frozen ALICE with neural estimators trained from scratch on the target samples, and report the error separately from the Gaussian and Student tasks.
  2. Could the authors compare their method with cited in the introduction FMMI and InfoBridge, both are open-source and often outperform existing neural estimators? I believe this extended comparison of presented results in Figure 2 would strengthen the paper.
  3. Eq. (3) holds for true velocity fields. In the experimental setup they are replaced with vθ(⋅,C)v_{\theta}(\cdot, C). Is there any theoretical guarantee on the error in this case? Maybe one can bound it in terms of context size, model width, or distance from the training corpus?
4 comments

I think the reported benchmark does not fully test the generalization claim. [....] Please add a held-out suite with known MI whose dependence is not a Gaussian or Student's copula and not a monotone image of one. [....] compare frozen ALICE with neural estimators trained from scratch on the target samples, and report the error separately from the Gaussian and Student tasks.

We ran one additional experiment that addresses this concern and the similar concern of Reviewer H2uh; we report the same setup and results in both threads.

We trained an ALICE Base (85M parameters) from initialization on Gaussian distributions only, with randomly sampled dimensions. Relative to this restricted corpus, we distinguish nine near-OOD tasks (sparse Gaussian dependence or transformations of Gaussian data) from 25 far-OOD tasks (Student-t and additive-noise families, together with transformed sparse-Gaussian tasks).

The table below reports MAE across tasks with the experimental setup of the submitted paper.

Total samplesNear OOD: Gaussian-onlyNear OOD: full corpusFar OOD: Gaussian-onlyFar OOD: full corpus
1,0000.1440.0620.3460.100
5,0000.0610.0370.2580.067
10,0000.0630.0310.2550.066

At 10,000 samples, the ALICE Base pretrained on Gaussians only achieves a lower MAE in both OOD groups than CCA, KSG, and LNN, which are fitted directly to the target samples, and than InfoAtlas, which is evaluated zero-shot. MINDE, MINE, InfoNCE, D-V, and NWJ are trained and tuned separately on each target distribution; therefore they are never, by definition, tested in an out-of-distribution regime, as shown in the table below:

MethodNear OOD group: MAEFar OOD group: MAE
ALICE Base (full corpus)0.0310.066
MINDE0.0360.080
InfoNCE0.0240.119
MINE0.0230.160
NWJ0.0220.195
D-V0.0240.217
ALICE Base (Gaussian-only)0.0630.255
CCA0.0660.259
KSG0.1350.279
InfoAtlas0.1720.296
LNN2.7834.690

Could the authors compare their method with [....] FMMI and InfoBridge, both are open-source and often outperform existing neural estimators? [....]

The neural baselines of the paper (MINE, InfoNCE, NWJ, D-V, MINDE) are evaluated under a common protocol: the 40 tasks of Czyż et al. at 1,000, 5,000, and 10,000 training samples. InfoBridge (Kholkin et al., ICLR 2026) reports the 40 tasks of Czyż et al. with a different training and testing setup and FMMI (Butakov et al., 2025) reports only the high-dimensional Gaussian and uniform suite of Butakov et al., so neither set of published numbers can be used in our tables. A fair comparison requires running anew both codebases on the 40 tasks at 1,000, 5,000, and 10,000 samples with hyperparameters tuned for these budgets, which is not feasible within the rebuttal period. Nonetheless, we note that the results in Table 21 of Infobridge (Kholkin et al) indicate that MINDE (which we do compare against) is a very strong baseline.

Eq. (3) holds for true velocity fields. [....] Is there any theoretical guarantee on the error in this case? Maybe one can bound it in terms of context size, model width, or distance from the training corpus?

The estimation error has an expression in terms of the velocity error, the difference between the true and paramaetric vector fields. In a similar spirit to the MINDE work (Sec 3.1), it is possible to show that the estimate is neither an upper nor a lower bound of the MI, so the estimator is not subject to the sample-complexity limits of McAllester and Stratos (2020) for variational lower bounds.

How the velocity error behaves is, in general, an open problem:

For a given distribution, Zhu, Locatello, and Cevher (NeurIPS 2023, Section 4, Theorem 2) study the time-averaged score error of a standard deep ReLU network trained by SGD on the denoising objective with explicit dependence on training samples nn, the data dimension, the data radius, and the covering number of the network class (width and depth). This result can help deriving a rate for the non-amortized version of Eq. (3). But this is not yet enough for in-context nature of ALICE, for which three further sources of error should be considered: the finite context, the distance of the target law from the training corpus, and the in-context approximation error of the trained network (training points, width, depth).

The context size plays the role of the sample size of the implicit estimator and controls a bias term, and the distance from the corpus enters through the prior of that posterior. Whether, and how fast, a trained transformer realizes this posterior-predictive map is the subject of the in-context learning theory we build on (Garg et al., 2022; Akyürek et al., 2023; von Oswald et al., 2023; Bai et al., 2023; Xie et al., 2022; Zhang et al., 2025; Xie et al., 2025), whose quantitative guarantees are, to date, restricted to linear or Gaussian settings. The result closest to our problem is Smart, Bietti, and Sengupta (ICML 2025) on in-context denoising.

At the moment, we do not have a bound with meaningful constants for the three terms jointly, for a deep transformer on a corpus as heterogeneous as ours, and we consider it a question for an interesting new study.

I have seen the responses given by the authors to my and other reviews. I appreciate their work. I am planning to keep my score the same.

AuthorsOpenReview ↗

Thank you for your feedback

Dear Reviewer,

Thank you for reading our responses to you and the other reviewers.

We understand that you currently plan to keep your score unchanged. Could you share which concern carries the most weight in that decision? We would welcome the chance to discuss any remaining concern before the discussion closes.

Reviewer_H2uhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a method to compute mutual information (ALICE) based on two elements: (1) it can be computed from the "velocity fields" that relate the joint and the marginal PDFs to Gaussians using the transforms given by diffusion processes (their Eq. 3), and (2) a transformer that is trained to predict these "velocity fields" from a corpus of PDFs.
On the 40-task "Beyond Normal" benchmark (Czyż et al., 2023), ALICE reports the lowest MAE at budgets of 1k, 5k and 10k samples, with the largest margin at 1k samples. The paper also presents three applications without ground truth, and claims a new neuroscience finding in the last one.

Critical strengths
  1. Relevant problem and convincing headline result. MI estimation in the small-sample regime is a real bottleneck in the experimental sciences, and an estimator that needs no per-dataset training or tuning would be useful. The 1k-sample results (at least 2× lower MAE than every trained competitor) are the strongest part of the paper.

  2. Clean estimator design. Training on velocities requires no MI labels. One mask-conditioned network provides all the fields needed. I (as reader working in the field) appreciate Eq. 3 [which I dindt know!, Thanks!] as derived from Franzese et al. (2024) and Wang et al. (2026).

  3. Size-agnostic architecture. Unlike the closest amortized competitor (InfoAtlas), ALICE handles arbitrary joint widths and context sizes. This shows clearly on the width-50/100 tasks (0.086 vs 1.0 nats MAE).

  4. Generality of the read-out. The same checkpoint provides the fields for MI, per-dose divergences / channel capacity, and TC/DTC/O-information (App. I). This is a genuine practical advantage of amortizing the fields.

  5. Transparency. Detailed appendices, per-task results (Table 4), an honest error analysis (spiral and dense-multinormal failures), a code release.

Critical weaknesses

W1. The theoretical framework has a long history, and the Gaussianization line of work is missing

I find expression 3 very interesting although most credit is from Franzese et al. (2024) and Wang et al. (2026).

The authors do not mention other non-parametric mutual information estimates (that do not require pre-distribution training either) and that are also based on transfroming the original PDFs (joint and marginals) into Gaussians, namely Laparra et al. IEEE TNN 2011 and Laparra et al. IEEE TPAMI 2025, with their "Rotation-Based Iterative Gaussianization" (RBIG). In particular, RBIG is made of a series of steps (or layers) consisting of marginal Gaussianization for all coordinates and a rotation. This is repeated (layers stacked) again and again, and the original PDF is turn into a N(0,I). For mutual information, if RBIG is applied three times (to the joint PDF and to the two marginals) the MI is easy to compute (Laparra et al. 25).

The preprocessing step in ALICE is very similar to the first RBIG step. Algorithm 1, step 1, and Appendix E.2 apply a “coordinate-wise copula map”: each marginal goes through its empirical CDF and then to N(0,1), at both training and inference. That is marginal Gaussianization. The authors also say this is why the half-cube, asinh and normal-CDF tasks are “absorbed” (F, l. 1375). So part of the benchmark is solved by Gaussianization before the transformer sees the data. The estimator itself is a Gaussianization-based information measure. The velocity field defines a transport from the data to N(0, I), and MI is read off by comparing the joint transport with the block-wise ones. That is the same logic as the RBIG estimator, I(X;Y) = T([X,Y]) − T(X) − T(Y), where total correlation is accumulated along a Gaussianizing transform. Both approaches depend on two shared facts: MI is invariant to bijections within each block. ALICE uses this explicitly in its training corpus (within-block couplings, plane rotations) and in Appendix I. Dependence vanishes in the Gaussian domain.

The main differences are that ALICE uses a stochastic noising path with learned, amortized fields (learning fields is a big difference) whereas RBIG uses a deterministic iterative transform with closed-form marginal negentropy terms. However, the above similarities should be discussed, as RBIG does not require a per-distribution training either (and this is one of the conceptual advantages of ALICE).

Laparra, Camps & Malo, Rotation-Based Iterative Gaussianization IEEE TNN 2011 Laparra, Johnson, Camps, Santos & Malo, "Estimating Information Theoretic Measures via Multidimensional Gaussianization" IEEE TPAMI 2025.

I ask the authors to (i) discuss this line of work and position ALICE's velocity-gap read-out relative to the total-correlation read-out of Gaussianization (both measure how much dependence is removed along a transport to N(0, I)); (ii) if possible, include RBIG-based MI (public Python/Matlab toolbox) and Gaussian-copula MI (GCMI) as baselines on the Czyż benchmark at 1k/5k/10k samples; and (iii) report CCA/GCMI on the same copula-normalized data. Together these would separate how much of the advantage comes from the transformer and how much from the marginal Gaussianization that is applied only to ALICE's inputs. Since RBIG also yields total correlation, and hence O-information (TC, and DTC via entropies), it would also be a natural non-amortized cross-check for the analysis.

W2. "Zero-shot" generalization is not cleanly separated from overlap between corpus and benchmark

The training corpus (App. E.1) is dominated by Gaussian and Student-t copula mixtures with low-rank, sparse paired (X_i ↔ Y_i) and equicorrelated covariances, plus MI-preserving within-block bijections. The Czyż tasks are largely multinormal and Student-t models (sparse "2-pair" and dense) under monotone or spiral transforms. After copula normalization, most benchmark tasks are therefore close to the training distribution. The largest errors occur on the spiral tasks (−0.43/−0.44 nats), the transformation least represented in the corpus, which is consistent with this concern. I would like to see:

  • a held-out-family evaluation, i.e. a retrained model with a family removed from the corpus (e.g. no Student-t or no sparse-pair covariances);
  • results on benchmarks built differently from Czyż, e.g. those used by InfoBridge or MIENF, or the high-dimensional image-based tasks of Butakov et al.

This is consistent with using the testbed in Laparra 25 as suggested above.

W3. Only low-MI regimes are tested

Estimation is known to be hard at high MI, and that regime (several nats or more) is never tested. Authors could (i) add tasks with MI in the range of 3–10 nats (e.g. the high-MI multinormal settings used by MINDE, InfoBridge or Laparra 25) and (ii) rephrase the claim.

W4. No consistency, uncertainty or out-of-distribution diagnostics

The estimator is a deterministic function of a learned network, and the paper gives no argument that it converges as context size grows. Empirically, ALICE Small degrades with more context on wide tasks (0.197 → 0.241 nats, Table 3). Contexts of 5k and 10k samples are beyond the 2048-sample training window. Authors concede the risk on out-of-distribution data but offers no tool to detect it. Useful additions would be a self-consistency check (e.g. invariance of the estimate under random within-block rotations or bijections, or agreement between the product form Eq. 8, the conditional form Eq. 9 and the joint-vs-conditional form Eq. 3, which coincide for the true fields), and confidence intervals beyond the spread over context draws.

W5. Scientific claims in the applications outrun the validation

There is no ground truth in any of the three applications, and several details call for caution:

For instance, the Neuroscience (Sec. 4.3) App. shows that the estimate depends on context size. With unmatched contexts, a spurious change-minus-non-change difference of +0.12 nats (p = 2·10⁻⁸) appeared before the visual response. Pooled contexts raise the estimates and remove most of the familiar/novel difference. The novel/familiar sessions also differ by −0.16 nats in the first window, before the visual response, which again points to a non-stimulus confound. Given an estimator whose bias depends on context size and composition, the claim of a "new finding" (experience-dependent redundancy) needs a synthetic control with matched dimension (6), matched n (128) and known O-information, showing that the estimator's bias is small compared with the reported effects (≈0.4–0.5 nats). An independent, non-amortized estimate of the same O-information (e.g. a Gaussianization-based estimate, or the score-based estimator of Bounoua et al. fitted per session where sample size allows) would make the claim much more convincing.

Questions for authors
  1. How does ALICE MI compare with RBIG-based MI? (Laparra et al., 2025) and with Gaussian-copula MI on the Czyż benchmark at 1k/5k/10k samples? How do CCA and KSG perform when given the same copula-normalized inputs as ALICE? Conceptually, how do you relate the velocity-gap read-out to the total-correlation read-out of a Gaussianizing transform, given that both quantify the dependence removed along a transport to N(0, I)?

  2. What happens with a held-out dependence family (retraining without one corpus family)?

  3. What is the performance on tasks with MI of 3–10 nats?

  4. Can you include tests used in other publications with comparable methods were tested?

  5. Can you provide additional checks (use of other methods) to support the claim in the neuroscience application?

** I am happy to consider a rise in the score if these questions are addressed because non-parametric mutual information measures from samples are important for the ICLR community

3 comments

I ask the authors to [....] discuss this line of work and position ALICE's velocity-gap read-out relative to the total-correlation read-out of Gaussianization [....]; [....] include RBIG-based MI [....] and Gaussian-copula MI (GCMI) as baselines on the Czyż benchmark at 1k/5k/10k samples; [....] How do CCA and KSG perform when given the same copula-normalized inputs as ALICE?

We thank the reviewer for pointing us to the Gaussianization line of work, including Laparra et al. (2011, 2025), which is important. We will add the positioning below to the related work.

The copula normalization of ALICE is the first marginal Gaussianization step of RBIG. RBIG then alternates deterministic marginal Gaussianizations and rotations until the sample is Gaussian, and sums the marginal negentropies of the rotated data over the layers (Laparra et al. 2025, Eq. 8 and 9); the MI is the total correlation of the two separately Gaussianized blocks (Eq. 15). ALICE moves the joint and the conditionals along a rectified-flow path with amortized velocity fields; the MI is a time integral of squared velocity differences (Eq. 3 of the paper). Both quantities are the KL divergence between the joint and the product of the marginals, evaluated through a transport to the standard normal.

Next, we provide results on new experiments with RBIG. We re-implemented Eq. 7, 8, 9, and 15 of Laparra et al. (2025) with the configuration of the authors' toolbox (PCA rotation, histogram entropies with the Miller-Madow correction, the acceptance threshold and the stopping rule of the toolbox), and we ran the authors' implementation as a cross-check.

The table below reports MAE across tasks at the three sample budgets.

Method1,0005,00010,000
ALICE 1.0 Base (submitted evaluation)0.0920.0630.060
ALICE 1.1 Base (updated evaluation)0.0620.0480.051
RBIG (equations of Laparra et al. 2025)0.1680.1270.120
RBIG (toolbox of the authors)0.2020.1470.137
GCMI = CCA on copula-normalized inputs0.2270.1710.166
CCA (raw inputs)0.1950.1920.181
KSG (raw inputs)0.2880.2400.219

Although RBIG is the most accurate classic estimator at every budget, and at 1,000 samples it is more accurate than every trained neural estimator (MINDE: 0.353), ALICE Base outperforms it and has about half of its error at every budget. RBIG needed a per-dataset tuning over the rotation family (PCA, random, ICA), the null-bias correction, the acceptance threshold, the T([x,y]) - T(x) - T(y) form, the univariate entropy estimator, the patience and the maximum number of layers, while ALICE runs everything from a frozen checkpoint with no per-dataset choices.

After copula normalization, most benchmark tasks are therefore close to the training distribution. [....] What happens with a held-out dependence family (retraining without one corpus family)?

In this rebuttal phase, we trained a new ALICE Base (85M parameters) from scratch on Gaussian distributions only. Relative to this restricted corpus, we distinguish nine near-OOD tasks (sparse Gaussian dependence or transformations of Gaussian data) from 25 far-OOD tasks (Student-t and additive-noise families, together with transformed sparse-Gaussian tasks).

The table below reports MAE across tasks with the experimental setup of the submitted paper.

Total samplesNear OOD: Gaussian-onlyNear OOD: full corpusFar OOD: Gaussian-onlyFar OOD: full corpus
1,0000.1440.0620.3460.100
5,0000.0610.0370.2580.067
10,0000.0630.0310.2550.066

The table below reports all results of this new experiment compared with the competitors at 10,000 samples (which by definition are never tested in their OOD regime).

MethodNear OOD group: MAEFar OOD group: MAE
ALICE Base (full corpus)0.0310.066
MINDE0.0360.080
InfoNCE0.0240.119
MINE0.0230.160
NWJ0.0220.195
D-V0.0240.217
ALICE Base (Gaussian-only)0.0630.255
CCA0.0660.259
KSG0.1350.279
InfoAtlas0.1720.296
LNN2.7834.690

This experiment shows that despite being trained on Gaussian distributions only, ALICE is at least as good as classical estimators in all settings (near and far OOD), and the gap with respect to neural estimators trained on the target distributions is contained.

results on benchmarks built differently from Czyż, e.g. those used by InfoBridge or MIENF, or the high-dimensional image-based tasks of Butakov et al. [....]

We evaluated ALICE on the second benchmark of BMI-v2 Czyż et al. (TMLR 2025), which has 26 tasks. Thirteen of them are new Bend-and-Mix tasks (X, AI, Galaxy, Waves, concentric Gaussians, and inlier mixtures) whose dependence is neither a Gaussian or Student-t copula nor a monotone image of one. We follow its experimental setup and compare with the published results of the competitors (and we include RBIG as a new baseline).

The table below reports MAE across the 26 tasks and is ordered by increasing MAE.

MethodMAE (nats)
ALICE 1.1 Base (updated checkpoint)0.153
ALICE 1.0 Base (submitted evaluation)0.173
InfoNCE0.211
D-V0.225
MINE0.228
KSG0.244
RBIG (equations of Laparra et al. 2025)0.286
NWJ0.322
CCA0.405

Interestingly, note that an oracle that could pick the best published estimator on each task would achieve a MAE of 0.122. Thus, our model ALICE is very close to such best-case performance. We will add the full per-task table to the appendix. Due to compute and time limitations, we could not set up the embedding pipeline required by the image-based tasks of Butakov et al.

Estimation is known to be hard at high MI, and that regime (several nats or more) is never tested. Authors could [....] add tasks with MI in the range of 3–10 nats [....] and [....] rephrase the claim.

The largest MI in the Czyż benchmark is 1.71 nats, so we run a new experiment where we evaluate tasks with a prescribed MI of 0, 1, 3, 5, and 10 nats. We used Gaussian distributions with known high MI, block dimensions 5+5 and 10+10, and 10,000 context samples, for the submitted checkpoint and for the new model ALICE 1.1.

The table below reports the mean MI estimate, with the absolute error of the mean estimate in parentheses, in nats.

Block dimensionsTrue MIALICE 1.0 Base (paper)ALICE 1.1 Base
5+500.004 (0.004)0.004 (0.004)
5+510.975 (0.025)0.925 (0.075)
5+533.316 (0.316)3.025 (0.025)
5+555.413 (0.413)5.079 (0.079)
5+51010.065 (0.065)10.117 (0.117)
10+1000.007 (0.007)0.007 (0.007)
10+1010.838 (0.162)0.888 (0.112)
10+1033.221 (0.221)3.030 (0.030)
10+1055.763 (0.763)5.277 (0.277)
10+101011.196 (1.196)10.400 (0.400)

For a true MI of 3 to 10 nats, the mean estimates of the submitted checkpoint have maximum relative errors of 10.5% (5+5) and 15.3% (10+10). ALICE 1.1 improves five of these six cases and reduces the worst case to 1.6% and 5.5%, respectively. We will rephrase the claim accordingly, and report our new results.

No consistency, uncertainty or out-of-distribution diagnostics [....] Useful additions would be a self-consistency check...

For the agreement tests between Eq. 8 (product form) and the two conditional forms of Eq. 9, we report the following results on the Czyż benchmark with ALICE under the same experimental protocol:

SamplesMAE Eq. 3MAE Eq. 8MAE Eq. 9 (X given Y)MAE Eq. 9 (Y given X)mean abs. difference to Eq. 3 (Eq. 8 / Eq. 9 X / Eq. 9 Y)
1,0000.1080.1070.1240.1190.039 / 0.033 / 0.034
10,0000.0580.0640.0750.0540.034 / 0.033 / 0.020

The four forms agree to within 0.02 to 0.04 nats on average.

Given an estimator whose bias depends on context size and composition, the claim of a "new finding" [....] needs a synthetic control with matched dimension (6), matched n (128) and known O-information [....]. An independent, non-amortized estimate of the same O-information per session [....] would make the claim much more convincing.

We computed, for every session, window, and flash type, four classic non-amortized estimators: the closed-form O-information of a Gaussian fit, the same Gaussian fit after copula transformation, the KSG estimator applied to the mutual-information decomposition of the O-information, and RBIG. The table reports the median over sessions for change flashes, in nats.

Estimator0-50 ms50-100 ms100-150 ms150-200 ms200-250 ms
ALICE 1.0 Base (paper)0.240.460.700.540.29
Gaussian0.170.410.680.520.24
Gaussian copula0.190.420.730.510.25
RBIG (Laparra et al.)0.260.570.830.600.32
KSG0.330.890.900.710.34

Over the 675 combinations of recording session, time window, and flash type, we find in general good agreement between ALICE and the non-amortized estimators.

AuthorsOpenReview ↗

Feedback on the rebuttal

Dear Reviewer,

Your review guided several new experiments.

We added the RBIG and Gaussian-copula baselines you requested, retrained ALICE on Gaussian data alone to test generalization, and evaluated it on distributions with known mutual information up to 10 nats.

Finally we added four independent estimators for the neuroscience application.

Given these results, would you reconsider your score before the discussion closes? If a concern still weighs on your assessment, we would appreciate knowing which one so we can address it directly.

Thank you for suggesting these experiments.

Reviewer_V1APOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ALICE, a pretrained model that estimates velocity fields from a sample context and uses them to estimate mutual information without per-distribution training. It evaluates the method on the BMI benchmark and three scientific applications.

Critical strengths
  1. I find the way the paper uses a velocity-based loss for in-context MI estimation novel and interesting.

  2. MI estimation is an important problem. Using one pretrained model instead of fitting a new estimator for each distribution is a useful direction.

Critical weaknesses
  1. Although I find the loss design interesting, the overall novelty seems limited. The idea of zero-shot MI estimation was already explored in InfoAtlas. The architecture also seems to draw on existing tabular foundation models such as TabPFN, and the copula preprocessing is similar to that used in InfoAtlas.

  2. The direct comparison is limited to the BMI benchmark. I would like to see comparisons on more datasets, or add other suitable baselines to the three scientific applications.

Questions for authors
  1. Is it necessary to use disjoint context and query sets? What benefit does this split provide compared with reusing the same samples for both roles, especially when the sample budget is small?

  2. Have you tried adding auxiliary supervision from distributions with known MI, such as independent data with MI = 0? I wonder whether this could improve calibration and estimation accuracy.

  3. TabPFN and TabICL use SCM-based generators to create rich dependency structures. Have you explored similar generators for ALICE, and how do they compare with the current pretraining distributions?

5 comments

Although I find the loss design interesting, the overall novelty seems limited...

We thank the reviewer for recognizing the interest of the loss design. We agree that InfoAtlas established zero-shot MI estimation and that our method builds on existing attention mechanisms and copula normalization, nonetheless we bring the reviewer’s attention to the following point: ALICE is the first zero-shot MI estimator that matches or outperforms the best classic and neural estimators.

Our contribution is an in-context formulation that amortizes joint and conditional velocity-field estimation through a single masked flow-matching objective and combines these fields into an MI estimate. Instead, InfoAtlas uses a hypernetwork to sidestep per-distribution training, but it suffers from a non-negligible penalty in terms of accuracy.

ALICE conditions a single fixed velocity model directly on context samples. Note that ALICE architecture allows the same model to process varying dimensionalities, including dimensions absent from training, without padding the variables with independent Gaussian noise. Instead, while padding (which is used in InfoAtlas) preserves the population MI; the behavior of a learned estimator under the additional noisy coordinates depends on its training and approximation accuracy.

ALICE also accommodates varying context sizes and scales beyond its training window. Figure 3 shows that ALICE Base benefits from contexts larger than those used during training. Across the configurations evaluated in that figure, every InfoAtlas configuration is Pareto-dominated by an ALICE configuration with both lower MAE and fewer inference FLOPs.

Copula preprocessing normalizes the marginal distributions and retains the dependence structure that determines the MI, which makes it a natural choice for pretraining across heterogeneous distributions.

The direct comparison is limited to the BMI benchmark. I would like to see comparisons on more datasets, or add other suitable baselines to the three scientific applications.

We thank the reviewer for the suggestion. For the rebuttal, we ran a new benchmark, two new classic baselines on the existing and new benchmarks, several new tests, and non-amortized baselines for the O-information application.

We summarize our new results here; the full tables will be in the revised appendix.

On the BMI benchmark we additionally include the RBIG (Laparra et al. 2011, 2025, see Reviewer H2uh) and the Gaussian-copula MI estimators at 1,000, 5,000, and 10,000 samples. RBIG is the most accurate classic estimator at every budget (MAE 0.168 / 0.127 / 0.120), whereas ALICE 1.0 Base has about half of RBIG error (0.092 / 0.063 / 0.060).

Furthermore, we evaluated ALICE on a fresh new benchmark BMI-v2 from Czyż et al. (TMLR 2025), which has 26 tasks at 5,000 samples. Thirteen tasks are new Bend-and-Mix tasks (X, AI, Galaxy, Waves, concentric Gaussians, inlier mixtures) whose dependence is neither a Gaussian or Student-t copula nor a monotone image of one. We follow the original protocol and compare with the results published by the benchmark authors in the table below:

MethodMAE over 26 tasks (nats)
ALICE 1.1 Base (updated checkpoint)0.153
ALICE 1.0 Base (submitted checkpoint)0.173
InfoNCE0.211
D-V0.225
MINE0.228
KSG0.244
RBIG (Laparra et al. 2025)0.286
NWJ0.322
CCA0.405

ALICE is the most accurate estimator on the Student-t tasks, on the sparse normals at 25 dimensions, and on the inlier mixtures at 25 dimensions, where the variational estimators collapse.

For the remaining last point, we added new baselines to the O-information on mouse visual-cortex recordings. For this study, we recomputed every per-session estimate with four estimators that involve no neural network (closed-form Gaussian, Gaussian copula, RBIG, KSG) at the same matched context of 128 rows. Here we report the results:

Estimator0-50 ms50-100 ms100-150 ms150-200 ms200-250 ms
ALICE 1.0 Base (paper)0.240.460.700.540.29
Gaussian copula0.190.420.730.510.25
Gaussian0.170.410.680.520.24
RBIG (Laparra et al.)0.260.570.830.600.32
KSG0.330.890.900.710.34

As expanded in the table above, we find good agreement between ALICE and the non-amortized estimators.

The other two applications already compare with the estimators of their reference works: the NF-κB channel-capacity analysis with the classifier-based estimator of Jetka et al. (2019), whose findings ALICE reproduces without per-analysis fitting, and the promoter localization with Info-SEDD (Foresti et al. 2026), the neural estimator trained on that dataset, whose bias floor is substantially higher.

Is it necessary to use disjoint context and query sets? What benefit does this split provide compared with reusing the same samples for both roles, especially when the sample budget is small?

During training, disjoint context and query sets avoid learning distributions concentrated on the context samples. If the clean query sample belongs to the context, the flow-matching objective targets the empirical distribution, a mixture of Dirac deltas at those samples. For a conditional query, the clean conditioning coordinates can identify the corresponding context row, which lets the model recover the remaining coordinates and collapse the conditional distribution to a Dirac delta. Independent query samples train the model to predict the distribution of fresh observations given the context.

At inference, the trained model can reuse context samples as queries. The table below reports MAE in nats on the Czyż benchmark, with relative changes of reusing queries measured against disjoint queries. The changes are small, which indicates an even more efficient use of the available samples. We thank the reviewer for the suggestion, and we will add this to the revised version.

ModelSample budgetDisjoint queriesReused queriesMAE change
ALICE 1.01,0000.0930.093+0.02%
ALICE 1.05,0000.0650.065−0.11%
ALICE 1.010,0000.0640.060−5.55%
ALICE 1.11,0000.0620.062+0.60%
ALICE 1.15,0000.0480.049+1.40%
ALICE 1.110,0000.0510.051−0.24%

Have you tried adding auxiliary supervision from distributions with known MI, such as independent data with MI = 0? I wonder whether this could improve calibration and estimation accuracy.

During development we explored a similar approach: computing the loss against the exact velocity fields for the distributions where these are available. This reduced the variance across training batches and led to slightly faster convergence, with similar final performance.

We retained the standard flow-matching objective for implementation simplicity and computational efficiency. Combining losses complicates batch construction and introduces conditional execution that triggers compute-graph recompilation and lowers GPU efficiency. Given the modest convergence benefit, we kept the simpler training procedure.

TabPFN and TabICL use SCM-based generators to create rich dependency structures. Have you explored similar generators for ALICE, and how do they compare with the current pretraining distributions?

We have not yet evaluated SCM-based generators for ALICE. They are compatible with our training objective, which requires only samples, without ground-truth densities or MI values. We will add these generators in a new release of ALICE and evaluate their contribution relative to the current pretraining distributions.

AuthorsOpenReview ↗

Feedback on the rebuttal

Dear Reviewer,

We would appreciate hearing whether our response clarified how ALICE differs from InfoAtlas and whether the additional benchmark and baseline comparisons addressed your concerns.

If a concern still weighs on your assessment, could you let us know which one while there is time to discuss it? If our response has resolved your main concerns, we would be grateful if you would consider raising your score. Thank you for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment to all reviewers

We thank the reviewers for their time and constructive feedback. We address every comment in the corresponding thread; the point below concerns all reviewers.

Updated ALICE checkpoint

Since submission we have continued the training of ALICE, which led to a new checkpoint that we will release publicly (ALICE 1.1). It is a training continuation of the submitted ALICE 1.0 checkpoint, with the same architecture and parameter count. With the evaluation setup of the submitted paper at 10,000 samples, the overall MAE decreases by 20.5%, with a lower error on 23 of the 40 tasks.

The table below compares all estimators at 10,000 samples per task and is ordered by increasing MAE.

MethodMAE (nats)
ALICE 1.1 Base (updated checkpoint)0.051
ALICE 1.0 Base (submitted evaluation)0.064
MINDE0.065
InfoNCE0.083
ALICE 1.0 Small (submitted evaluation)0.103
MINE0.108
NWJ0.130
D-V0.144
CCA0.181
KSG0.219
InfoAtlas0.276
LNN4.970

The same comparison at the three sample budgets is below:

Total samplesSubmitted 1.0 MAEUpdated 1.1 MAEReductionTasks improved
1,0000.0930.06233.8%29/40
5,0000.0650.04825.5%24/40
10,0000.0640.05120.5%23/40

The improvement is largest at the smallest sample budget. In the rebuttal we report both checkpoints where relevant: ALICE 1.0 for the submitted model and ALICE 1.1 for the improvements since submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.