- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a novel zero-shot MI estimator. This is a foundation model for in-context estimation of velocity fields that can be used at any joint width and context length. In contrast to modern neural estimators, which require training on the distribution of interest, the proposed ALICE is a transformer network, pre-trained on a variety of synthetic distributions, to predict their rectified-flow velocity fields, in the spirit of amortized in-context predictors for tabular data.
Critical strengths
A single frozen model estimates MI with no per-distribution training. On the Czyż benchmark the authors report that ALICE (Base variant) has the lowest MAE among the estimators they compare, with the clear margin at low number of samples. Amortized estimators of this kind, including ALICE and InfoAtlas, are an important direction for further research.
Critical weaknesses
I think the reported benchmark does not fully test the generalization claim. Most of the Czyż tasks are based on Gaussians or a Student's -distribution. Copula mixtures of both are the training family that the manuscript fully specifies, and the inference copula is designed to remove monotone marginal maps. So it is not clear to me to which extent your model does “generalize to distributions and data types that the corpus does not contain.” Of course, utilizing your method on the respective three scientific applications with no ground-truth MI can be considered as a potential application, but not as a validation of the method.
Questions for authors
- Please add a held-out suite with known MI whose dependence is not a Gaussian or Student's copula and not a monotone image of one. You are free to choose the setup, just for example some laws with additive uniform noise, provided this suite is not sampled in pretraining. On that suite, at the same budgets, like , compare frozen ALICE with neural estimators trained from scratch on the target samples, and report the error separately from the Gaussian and Student tasks.
- Could the authors compare their method with cited in the introduction FMMI and InfoBridge, both are open-source and often outperform existing neural estimators? I believe this extended comparison of presented results in Figure 2 would strengthen the paper.
- Eq. (3) holds for true velocity fields. In the experimental setup they are replaced with . Is there any theoretical guarantee on the error in this case? Maybe one can bound it in terms of context size, model width, or distance from the training corpus?
4 comments
Eq. (3) holds for true velocity fields. [....] Is there any theoretical guarantee on the error in this case? Maybe one can bound it in terms of context size, model width, or distance from the training corpus?
The estimation error has an expression in terms of the velocity error, the difference between the true and paramaetric vector fields. In a similar spirit to the MINDE work (Sec 3.1), it is possible to show that the estimate is neither an upper nor a lower bound of the MI, so the estimator is not subject to the sample-complexity limits of McAllester and Stratos (2020) for variational lower bounds.
How the velocity error behaves is, in general, an open problem:
For a given distribution, Zhu, Locatello, and Cevher (NeurIPS 2023, Section 4, Theorem 2) study the time-averaged score error of a standard deep ReLU network trained by SGD on the denoising objective with explicit dependence on training samples , the data dimension, the data radius, and the covering number of the network class (width and depth). This result can help deriving a rate for the non-amortized version of Eq. (3). But this is not yet enough for in-context nature of ALICE, for which three further sources of error should be considered: the finite context, the distance of the target law from the training corpus, and the in-context approximation error of the trained network (training points, width, depth).
The context size plays the role of the sample size of the implicit estimator and controls a bias term, and the distance from the corpus enters through the prior of that posterior. Whether, and how fast, a trained transformer realizes this posterior-predictive map is the subject of the in-context learning theory we build on (Garg et al., 2022; Akyürek et al., 2023; von Oswald et al., 2023; Bai et al., 2023; Xie et al., 2022; Zhang et al., 2025; Xie et al., 2025), whose quantitative guarantees are, to date, restricted to linear or Gaussian settings. The result closest to our problem is Smart, Bietti, and Sengupta (ICML 2025) on in-context denoising.
At the moment, we do not have a bound with meaningful constants for the three terms jointly, for a deep transformer on a corpus as heterogeneous as ours, and we consider it a question for an interesting new study.
I have seen the responses given by the authors to my and other reviews. I appreciate their work. I am planning to keep my score the same.
Thank you for your feedback
Dear Reviewer,
Thank you for reading our responses to you and the other reviewers.
We understand that you currently plan to keep your score unchanged. Could you share which concern carries the most weight in that decision? We would welcome the chance to discuss any remaining concern before the discussion closes.
I think the reported benchmark does not fully test the generalization claim. [....] Please add a held-out suite with known MI whose dependence is not a Gaussian or Student's copula and not a monotone image of one. [....] compare frozen ALICE with neural estimators trained from scratch on the target samples, and report the error separately from the Gaussian and Student tasks.
We ran one additional experiment that addresses this concern and the similar concern of Reviewer H2uh; we report the same setup and results in both threads.
We trained an ALICE Base (85M parameters) from initialization on Gaussian distributions only, with randomly sampled dimensions. Relative to this restricted corpus, we distinguish nine near-OOD tasks (sparse Gaussian dependence or transformations of Gaussian data) from 25 far-OOD tasks (Student-t and additive-noise families, together with transformed sparse-Gaussian tasks).
The table below reports MAE across tasks with the experimental setup of the submitted paper.
At 10,000 samples, the ALICE Base pretrained on Gaussians only achieves a lower MAE in both OOD groups than CCA, KSG, and LNN, which are fitted directly to the target samples, and than InfoAtlas, which is evaluated zero-shot. MINDE, MINE, InfoNCE, D-V, and NWJ are trained and tuned separately on each target distribution; therefore they are never, by definition, tested in an out-of-distribution regime, as shown in the table below:
Could the authors compare their method with [....] FMMI and InfoBridge, both are open-source and often outperform existing neural estimators? [....]
The neural baselines of the paper (MINE, InfoNCE, NWJ, D-V, MINDE) are evaluated under a common protocol: the 40 tasks of Czyż et al. at 1,000, 5,000, and 10,000 training samples. InfoBridge (Kholkin et al., ICLR 2026) reports the 40 tasks of Czyż et al. with a different training and testing setup and FMMI (Butakov et al., 2025) reports only the high-dimensional Gaussian and uniform suite of Butakov et al., so neither set of published numbers can be used in our tables. A fair comparison requires running anew both codebases on the 40 tasks at 1,000, 5,000, and 10,000 samples with hyperparameters tuned for these budgets, which is not feasible within the rebuttal period. Nonetheless, we note that the results in Table 21 of Infobridge (Kholkin et al) indicate that MINDE (which we do compare against) is a very strong baseline.