W1: Marginalisation Consistency. The authors pay no reference to the concept of marginalisation consistency, defined in PFNs/neural processes as
π(y1∣x1,dn)≡∫π(y1,y2∣x1,x2,dn)q(x2)dx2dy2for any valid density q (including delta function distributions such that x2 is fixed arbitrarily). It is a different property to separability, but is nonetheless closely related. If a model exhibits marginalisation consistency, then it's predictions obtained through inductive and transductive protocols are the same. PFN architectures are designed so that marginalisation consistency is hard-coded as an inductive bias. In PFNs (and TNPs) this is typically achieved through masking so that each query token only attends to context tokens, and each context token only attends to other context tokens. Obeying marginalisation consistency means it is impossible for a PFN’s prediction at a given query to depend on the other queries.
Given how closely this is related to the concept of separability, the authors should add a line or two somewhere to delineate the two concepts and explain how they relate to each other. The final paragraph of section 3 should be updated to mention marginalisation consistency. The authors should also be more clear about the fact that they consider a single query in their analysis framework.
The marginalisation consistency of a PFN can be destroyed by removing the masking between query tokens. Of course, such a PFN would need to be mask-free during both training and testing in order to learn the mechanism by which the query tokens are updated by each other. I would appreciate a comment from the authors about mask-free PFNs. For example, if a practitioner knows their downstream data generating process is non-separable, should they use a SemiTabICL architecture or something like a mask-free PFN?
W2: Martingale Checks. In the paragraph starting on line 360, the authors point out that the martingale property should hold for a PFN trained under a separable prior, but not for one trained under a non-separable prior. Perhaps I have misunderstood, but the results in Figure 4 demonstrate that in practice neither PFN exhibits the martingale property$\textemdash$why? Also, equation 3 implies the martingale property should be tested by averaging predictions obtained from a length-1 autoregressive rollout, yet in Figure 4 the averaging seems to have been performed over an autoregressive rollout of length 1000$\textemdash$why? In its current state I struggle to fully grasp and therefore appreciate the value of this section, and would appreciate some further clarity from the authors.
Further Review Fields
Since there are no boxes for reviewers to enter minor correction suggestions or ask minor clarification questions, I use this box for that purpose as well. None of the below points should be interpreted as criticism; they are intended to help the authors in revising their manuscript for camera-ready or resubmission. The AC doesn't need to read the below.
Minor corrections/editorial suggestions
C1 In the abstract, the colon construction of “: for the priors” is unnecessarily stilted. My recommendation is to replace the string “: “ with “ because “, “ as “, or “ since “.
C2 In the abstract, the sentence starting with “A second difficulty is conceptual” actually would benefit from a (semi-)colon construction for better readability$\textemdash$my recommendation is to swap the string “, as ” with “; “.
C3 In the abstract, the comma in “separable prior, under which” should be removed.
C4 In the abstract, the final two sentences both being tricolons detracts from the readability and flow. It feels as though one of the sentences is a placeholder that the authors forgot to finalise. The same applies to the fact that “characterize” is used in both sentences$\textemdash$this leaves a sour taste in readers’ mouths and distracts from the actual content of the words. The final two sentences read as a list of six pursuits that the authors engage in without imparting the reader with any meaningful narrative or concrete takeaways. When re-writing these sentences, the authors should focus on the fact that readers of the abstract want to know: what research questions the authors pose, why these questions are interesting or important, what the findings are, and what the implications are.
C5 In the introduction, “A common data prior choice are structural causal models” should be altered to have the verb in the singular form “is” rather than the plural “are” (by analogy, when discussing lunchtime meal options, “a common choice are sandwiches” feels less correct than “a common choice is sandwiches”, though I would argue neither feel totally right). My recommendation would be to sneakily rewrite the sentence in a way such as “Structural causal models (SCMs) represent a common choice for the data prior”, where the question of verb plurality is pushed out of view.
C6 In machine learning, input-output pairs are usually referred to as “x-y pairs” and the mathematical notation typically follows this ordering convention (even though it is arbitrary). Is there any particular reason why the datapoints are consistently referred to in the opposite order in this submission? i.e., the authors introduce a context dataset as dn=yi,xi{ni=1}. This is highly pedantic of me, and I fully understand if the authors choose to ignore this very minor point, but to me it seems like an unnecessary (albeit very minor) hinderance on the paper’s readability.
C7 On line 41, the authors should avoid using the word “prior” here and should replace it with something like “antecedent”, “anterior”, “bigger”, “more fundamental”, “retrograde”, or so on. The use of a different meaning of the word “prior” in a paper about Bayesian priors could confuse non-native speakers (actually, it confused me, a native speaker, since I interpreted “prior question” to mean “question about the prior”).
C8 In the paragraph beginning on line 79, I believe I2 should be emboldened throughout as I2 since it is a matrix.
C9 In the Model and notation paragraph of section 2, the sentence “Given a task p∼Π, let X1,…,Xn denote the context features and X the query feature.” is unnecessary and should be removed (i.e. given the subsequent sentences).
C10 There is an inconsistency of notation between line 35 and line 118 where dn is defined as yi,xiI=1n and (yi,xi){i=1}n. I assume the former is correct, but regardless, this inconsistency should be ironed-out one way or another. If braces are to be used to refer to the collection of points, then Dn=(Yi,Xi){i=1}n, which is also on line 118, should be changed accordingly.
C11 On line 145, “like” should be swapped in favour of “such as” as its comparatively informal tone is not so appropriate in scientific writing.
C12 On line 234, “vanishes with the sample size” implies the gap decreases as the sample size decreases, which is wrong. I would change it to “vanishes with increasing sample size”.
C13 On line 421/422 (the first line of that paragraph), “a range of labeled training rows n” should instead be something like “a range of the number of labeled training rows n”. The clunkiness of this sentence can also be reduced by swapping “added as” with “of”, as has already been done in Figure 5’s caption.
C14 In Figure 5, it is my view that n should be ordered from 16 on the lowest row to 96 on the highest row; the current layout feels weird. Also, a colourbar would really help the readability of this figure—it is a super figure with interesting results, but because the results are so different for the two TabICL variants, it initially seems like two separate colourmaps have been used and so the difference is less stark.
Minor Questions
Q1 In definition 1, what operation does ⊗ represent?
Q2 Under a non-separable prior, when would the Martingale property hold under stochastic auxiliary features? In other words, for what q does the following hold?
EX{n+1}∼q(⋅), Y{n=1}∼π(⋅∣x{n+1},dn)[π(A∣x,D{n+1})]≡π(A∣x,dn) Would q need to match the marginal distribution of x under the prior?
We welcome the reviewer’s assertion of the sufficiency of theoretical and empirical evidence in our submission and appreciate the positive review.
In the following, we will address the reviewer’s points:
We would like to stress that only the experiments that need an exact reference use n=2. This is due to the fact that these experiments require a finite task space so that we can exactly compute π and πˉ. The practical evidence is our SemiTabICL model on the TALENT benchmark as well as UCI datasets, which provides direct evidence that non-separability matters in practice.
We thank the reviewer for this question. We cannot directly assess the gap between the two predictives, as non-separable priors are in general not amenable to MCMC sampling. However, we extended our experiments with a more realistic prior based on a Bayesian neural network, as also used in the pretraining of the original TabPFN. For this, we mix two separable prior regimes to obtain a non-separable version together with a separable one, where the marginal feature and conditional label distributions are the same. The only difference between the two priors is thus their dependence structure.
We train a TabICL model on each prior and find that the difference, as expected, vanishes for large context sizes and only the PFN pretrained on the non-separable prior can use the query feature as evidence at small n to identify which of the regimes it is tasked with. A figure and a detailed experimental description is available in the new Appendix C.4.
Our SemiTabICL experiment further shows that the difference matters in practice and enables substantial performance gains on well-established benchmarks, like the TALENT benchmark.
We thank the reviewer for this interesting question. Our results imply that a PFN failing the martingale check is expected, because the exact Bayes predictor fails it too under a non-separable prior.
For a separable prior, we know that the approximation target of a PFN passes the martingale condition. To check whether this is also the case for a PFN trained on a separable prior, we conduct a martingale property check for the mixed prior setup discussed above, where we expect (more) violation for the PFN trained on the non-separable prior.
Results (Appendix C.5) show a small drift of the PFN pretrained on the separable prior, which is due to its approximation error. Compared against this baseline, the PFN pretrained on the non-separable prior shows substantially more drift. As expected, this drift stagnates as the sequential addition of features helps to identify the two different prior regimes in the mixture more and more.