Prior-Data Fitted Networks (PFNs) perform approximate Bayesian inference in a single forward pass, and tabular foundation models (TFMs) built on them are now widely used. To understand what networks infer internally, recent mechanistic studies of TFMs locate where predictions form, but treat these models as tabular predictors rather than as PFNs. It therefore remains unknown whether PFNs represent the spectral content of their context, the quantity that specifies a stationary kernel, and whether this content can be read out as an explicit kernel. We answer both questions. First, across seven PFNs, including four pretrained TFMs and a model trained only on a decision-tree prior, a linear probe on the residual stream recovers the frequency of the context with R2≥0.95. This structure is led by a single principal direction. Second, activation and subspace patching show that the network uses the structure through a low-dimensional subspace, where a few spectral directions move predictions far more than random ones. This holds even for the decision-tree model, so a spectral training prior is not required. On real datasets with up to 499 features, 64 of the 192 directions of TabPFN, chosen without labels, carry 85 to 95% of the causal effect of the context in all but one pair. Third, we introduce a Filter Bank Decoder that turns frozen PFN representations into an explicit stationary kernel through Bochner's theorem. Without any test-time optimization, the decoded kernel supports Gaussian process regression competitive with deep kernel learning and random Fourier features at about 200× lower cost. PFN latents therefore hold spectral structure that is causally used and recoverable as a portable kernel.
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.Reviewer_L3tKOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
This paper investigates whether Prior-Data Fitted Networks (PFNs) internally represent the spectral content of the context and whether that content can be extracted as an explicit stationary kernel. It covers seven models, including VA-PFN and DVA-PFN trained on spectral-mixture GP priors, four released TFMs, and a nanoTabPFN trained only on a decision-tree prior.
Critical strengths
This study covers a wide range of models and shows interesting findings for models like decision-tree nanoTabPFN.
This study also includes multiple decoder controls, e.g., random-PFN latents, H-only/V-only inputs, and a raw-input decoder, and a comparison to classical periodogram-based spectral-mixture fits.
Critical weaknesses
From Table 17, the raw-input decoder matches the latent decoder on Ker-MSE. Does the decoder need the PFN?
The proof assumes unit-norm normalisation, while the preprocessing in A.2 standardises and then applies a sigmoid. After the sigmoid, the data are no longer Gaussian, and the unbiasedness assumption in Proposition 1 does not hold for the normalised data.
Questions for authors
From Table 17, the raw-input decoder matches the latent decoder on Ker-MSE. Does the decoder need the PFN?
The proof assumes unit-norm normalisation, while the preprocessing in A.2 standardises and then applies a sigmoid. After the sigmoid, the data are no longer Gaussian, and the unbiasedness assumption in Proposition 1 does not hold for the normalised data. Please check if there is a preprocessing mismatch.
1. Does the decoder need the PFN? PFN is not needed for the information but it facilitate efficient learning significantly. A decoder trained on the raw context can learn the same kernels, which is expected, since everything the PFN knows comes from the context. The decoder asks a different question: whether the frozen PFN has already computed this information and holds it in a readable form. A head on the frozen latent matches the raw-input decoder while training fewer parameters, and it trained stably, whereas two of three raw-input restarts collapsed to a constant.
We also want to highlight that section 5 ("What the decoder reads") and Appendix D.3 of original manuscript that a decoder trained end to end on the raw (x, y) context reaches similar kernel error (Table 17), given enough capacity. Section 5 introduces it as a constructive test of whether the structure found in Sections 3 and 4 can specify a full covariance function.
We have now trained a decoder on random PFN initialization at matched budget (see reply to reviewer eZ4S). It shows that compared to random PFN, it is significantly more computationally efficient to train a decoder on trained PFN. This finding matches Section 3.2 and Fig. 6. An MLP probe, which has enough capacity, recovers frequency even from the latents of a randomly initialized PFN (R² ≈ 1.0), while a linear probe succeeds only on the trained network (0.998 against 0.18).
2. Normalization Apologies for the confusion caused. As the PFN and the GP stage of the decoder use different preprocessing, and each result applies to its own stage, there is no mismatch
The pipeline in Appendix A.2, which standardizes each context and then applies sigmoid(0.75·z), is the PFN's input preprocessing only. Theorem 1 is about what the PFN can see. Standardization and the unit-norm normalization used in the proof both remove the global scale of the function values, since they differ only by a constant factor. A fixed pointwise map applied afterwards, such as the sigmoid, cannot restore that scale. Rescaling the GP by any positive constant c therefore leaves the PFN's input unchanged, which is exactly the invariance the proof relies on, so the conclusion of Theorem 1 holds for the A.2 pipeline.
Proposition 1 is about the GP stage, which does not use the sigmoid. The scale estimate α̂ and the GP regression are computed on the context targets without the sigmoid, so the setting of Proposition 1 applies directly. Its proof also uses only the second moment E[yyᵀ] = αK, so it does not rely on Gaussianity. We will state in Section 5 and Appendix D.1 which preprocessing each stage uses.
Many thanks for your thoughtful review. We attempted to address all your concerns. Could we ask if you could kindly get back and let us know if our rebuttal addressed all your concerns. We will be really grateful if you could let us know about that.
If you have any remaining concerns, please do not hesitate to ask. We will be more than happy to respond.
Note that we addressed all the discussions in the submission.
This paper studies whether Prior-Data Fitted Networks (PFNs) encode spectral information about their context datasets, whether the information affects the predictions, and whether it can be summarized into an explicit covariance kernel. On seven PFNs including pretrained tabular foundation models (TFMs) and a PFN trained on a decision-tree prior, the authors find that simple spectral quantities are linearly decodable from pooled hidden representations.
The paper also introduces a supervised Filter Bank Decoder that maps frozen DVA-PFN representations to stationary spectral-mixture kernels. Validation includes kernel reconstruction, GP regression, comparisons with iterative and amortized baselines, and Bayesian optimization using a kernel extracted once.
Critical strengths
Interesting studies that provide evidences that general PFN training (across frozen architectures and different types of priors) can make specific spectral quantities (e.g. frequency) linearly accessble.
The paper goes one step further beyond showing that spectral quantities can be probed. It constructs stationary kernels from frozen representations and evaluates if they can be useful to downstream predictions. Experiments show useful GP regression performance.
Critical weaknesses
The central mechanistic evidence is not sufficiently established. Linear decodability of frequency and a causal effect from patching frequency-correlated subspaces do not establish that frequency is the information driving the predictions: the same directions may also carry other prediction-relevant information. It's also unclear how the frequency-correlated subspaces are decided.
The kernel-decoding experiments is problematic. The random-PFN control reuses a decoder fitted on representations from a trained PFN. Its poor performance therefore cannot distinguish insufficient information in the random representations from a mismatch with the decoder’s training distribution. The experiments do not isolate whether the benefit comes from extracting kernel structure already present in the pretrained PFN or from learning kernel inference through the additional supervised decoder.
Questions for authors
How do you distinguish the causal use of frequency information from the use of other prediction-relevant information carried by the same subspace?
Does the performance gap persist when the same decoder architecture is trained separately on frozen random-PFN and trained-PFN representations, using matched data and optimization budgets?
Figure 13 cannot be seen, can the authors provide another version or describe what it's shown there?
There is already work (e.g. TabPFN-TS [1]) on the understanding and application of spectral relationships in TFM. What is the authors' specific positioning for this work -- is it intended to guide the construction of new synthetic priors, or does it focus on optimizing existing TFM predic tions? How the future work could be done?
[1] Hoo, S. B., Müller, S., Salinas, D., & Hutter, F. (2025). From tables to time: Extending tabpfn-v2 to time series forecasting. arXiv preprint arXiv:2501.02945.
Reply to Reviewer eZ4S: W1/Q1 & W2/Q2: Two New Experiments to Show Frequency Causality & Random v/s Trained PFN decoder at matched budget
W1 / Q1 We agree with the reviewer that the frequency-identified subspace also carries other information, such as phase. The results in the paper also indicate this. Hˉ linearly encodes the mixing weight a as well as frequency (Table 1, R2≥0.93). We also stated in Sec. 6 states that Hˉ carries information beyond the quantities we probe. The remaining question is whether frequency itself is used or not. Our current experiment cannot answer this, because the donor B differs from the base A in both frequency and phase.
Frequency Causality (new exp). We add a third signal C with B's frequency and A's phase, so that A and C differ only in frequency. We use the same CE and the same spectral, non-spectral and random subspaces as in Section 4 (50 triples). See Appendix C.5 and section 4 of revised paper.
Model
k/d
Spectral subspace (frequency change, A→C)
Non-spectral subspace
Random subspace
Spectral subspace (phase change, C→B)
DVA-PFN, block 2
8/128
0.998
0.000
0.070
0.998
DVA-PFN, block 2, noise σ = 0.3
4/128
0.972
0.000
0.036
0.968
TabPFN v2.5, block 12
32/192
0.897
0.018
0.024
0.795
-- A. Frequency is used causally. Patching only the spectral subspace moves the prediction from A to C (CE > 0.9), while non-spectral and random subspaces has maximum CE 0.07. Since A and C differ only in frequency, the transferred information shows that frequency is used causally for making predictions.
-- B. Frequency finds a causal subspace which also carries other causal information. Patching a phase-only change through the same subspace also transfers it (CE 0.80–1.00). Therefore, frequency and phase are entangled in these directions, as in our existing results too. We thank the reviewer for raising this distinction. We do not claim that the identified directions encode frequency exclusively. Rather, frequency correlation provides an interpretable criterion for identifying directions that are causally relevant to prediction.
Subspace Choosing Process: The procedure for choosing the frequency-correlated subspaces was described in Section 4 ("Targeted Subspace Patching") of original manuscript. "We collect Hˉ ... a baseline." Also, Appendix C.2 gives the intervention block for each model, and Table 11 gives the settings for the GP experiments. App. C.3 applies the same procedure to the tree-prior nanoTabPFN. Real data has no frequency to rank by, so Section 4.1 and App. C.4 order the directions by explained variance alone.
W2/Q2 In the original manuscript, the random-PFN row of Table 17 was meant to check that the decoder's output depends on the structure of the trained latent, and is not a default kernel that the head produces whatever its input. A separately trained control is already in the paper at the probe level (Section 3.2, "What happens when weights are random?", and Fig. 6). The same linear probe, trained from scratch on latents of a randomly initialized DVA-PFN with the identical protocol, drops from R² = 0.998 to 0.18 for frequency and from 0.99 to 0.64 for the mixing weight, while an MLP probe still recovers both. The spectral content is present in random latents, but it becomes directly readable only after training. We see this effect in the new reviewer-suggested experiment below.
To test random PFN control at the decoder level, we trained the same decoder from scratch on two inputs with matched data and optimization budget as suggested. Both use the same training data (random Fourier feature signals with 1 to 4 spectral peaks), the same curriculum (1-peak signals for the first 10k steps, then 2-peak), and the same optimizer and batch size, and run for 20,000 steps with one seed per variant. We track bin F1, the agreement between predicted and true active frequency bins, on a fixed held-out set of 2,000 signals with 1 to 4 peaks. A decoder that ignores its input scores 0.095.
Trained PFN
Random PFN
Trainable parameters
468k
468k
Steps until the decoder bin F1 above 0.095
250
≈ 10,000
In-distribution loss, end of 1-peak phase (10k steps)
3.96
4.87
In-distribution loss, end of 2-peak phase (20k steps)
5.02
5.20
With trained-PFN latents, the decoder bin F1 moves above 0.095 at 250 steps, while with random latents it needs about 10,000 steps, about 40 times longer. The trained-PFN decoder also has the lowest in-distribution loss at every step. We also note that with enough training budget, random latents become usable too. On the held-out set, which includes 3- and 4-peak mixtures that no variant had trained on by 20k steps, bin F1 is comparable across two. This shows that trained PFN latents have organized information, which mirrors the probe result we reported in the original manuscript. We have updated the description of the random-PFN row in Section 5 accordingly.
Q3 We checked the submitted manuscript via openreview portal and Fig. 13 is visible in both in browser as well as upon downloading it and viewing on mac or windows pdf viewers. It is in Appendix C.1. The text describing it is at lines 1059 to 1063, and its caption is at lines 1072 to 1075 (all of original manuscript). It shows three representative RBF-GP signal pairs from the activation-patching experiment.
Q4 In our understanding, the two works ask different questions. TabPFN-TS supplies spectral structure from outside the model. It detects seasonal periods with a Fourier transform, adds their sine and cosine as input features, and studies the model by changing these inputs and observing the forecasts. It concludes that TabPFN-TS "behaves like a learned kernel or nearest-neighbor regressor". We on the other hand ask what the network computes from raw inputs. We ask whether the frequency content of the context is represented in the hidden states, whether predictions causally depend on it, and whether it can be read out as an explicit kernel. Our work is therefore primarily about understanding, and it turns the "learned kernel" description into a kernel that can actually be extracted.
Relevance. Our results describe how the PFNs we study, including four released TFMs, summarize their context. Below we separate what we observe, what we infer from it, and future directions.
What we observe
The context is summarized early in depth. Sec. 3.3.
The summary is low-dimensional and led by one direction. Table 1, Sec. 3.2, & App. B.4.
Predictions depend on a few of these directions. Sec. 4 & App. C.5.
A spectral training prior is not needed. Sec. 3.2, & Sec. 4.1.
On real data, the context acts through a small subspace. Sec. 4.1, Table 2.
Trained latents make the kernel easy to learn to read. New result, Sec. 5 & App. D.3.1
What we infer
Taken together, in the PFNs we study the context is reduced early in depth to a compact summary. Its leading direction tracks how fast the target varies with the input, and the prediction depends causally on a small subspace of it. We hypothesize that this happens because any amortized regressor must infer how far it can borrow strength across context points (Sec. 3.2). This offers a candidate internal mechanism for the behaviour that Hoo et al. describe from the outside: "TabPFN-TS behaves like a learned kernel or nearest-neighbor regressor, interpolating effectively in feature space."
Future work this enables
Real data. Test whether the frequency direction found on synthetic signals lies inside the label-free real-data subspace, and whether moving along it changes how smooth TabPFN's predictions are, for example how far a change in one context label spreads to nearby predictions.
Prior design.A linear probe is a cheap test of what a prior teaches, as the decision-tree case shows. Can be used to distinguish priors.
Non-stationary priors. A single global rate is not enough for these, so the network would need to keep a local one.
Thank you for the efforts putting into preparing the additional experiments and the rebuttal. There are many techinical details and I will need to read the manuscript again. I had a glance at the manuscript again and can see Figure 13 now. It seemed to be an issue with my Chrome pdf reader (and still broken), but I can see it now with another pdf reader
This paper analyses Prior-Data fitted networks via tools from mechanistic interpretability to study whether their representations encode the spectral density of the context. Furthermore, the analysis is extended to use the frozen embeddings of PFNs for kernel discovery, decoding the representations into explicit kernels usable for GP regression.
In a first step, using 500 ground-truth 1D sinusoids $y(t) = sin(2 \pi f_i t + \phi) $ with known frequencies fi, the authors establish that the latent space of a number of different PFNs contains frequency information that can be read out linearly and is concentrated on a low-dimensional subspace. For more complex data-generating processes, spectral information remains present but weakens, especially for higher-dimensional tasks.
In a second step, the authors verify that the spectral information present in the embeddings is actually used by the model for predictions via activation patching on synthetic datasets and to some extent on real-world data as well.
Lastly, these findings are applied to kernel discovery via a multi-stage pipeline, called the Filter Bank Decoder, building on top of the frozen embeddings of a PFN. In synthetic experiments, for some kernel families, the proposed method performs on par with other kernel discovery methods at competitive inference speeds.
Critical strengths
The paper identifies an interesting and important gap in the literature on Prior Data fitted networks around the topics of (mechanistic) interpretability and our understanding of how PFNs perform Bayesian inference. The analysis angle via spectral structures is interesting and (to my knowledge) novel.
The paper is generally well written and follows a clear and understandable arc. The appendix is comprehensive and provides interesting additional experiments as well as visualizations.
Critical weaknesses
Reproducibility: To my understanding, there are several issues regarding the reproducibility of the stated results. While none of these overturn any of the key claims of the paper, they do raise questions regarding the overall validity and quality of the findings, as examplified by the follwoing two issues: The supplement seemingly omits code for several headline experiments (the tree-prior nanoTabPFN, three of the four pretrained TFMs, and all real-data patching experiments), and running the provided exp_G_subspace_patching.py reproduces the random baseline of Fig. 1 / Table 10 but only about half the reported spectral-subspace causal effect (e.g. 0.30 vs. 0.59 at k = 2).
While some of the findings are interesting on their own right, to my understanding, the paper stops short of showing how these insights are of wider relevance for the PFN literature. This concern is formulated in more detail in the questions to the authors below.
Questions for authors
Regarding the reproducibility concern:
Can the authors provide additional material or information (or point to the material if existing in the supplementals) that supports the findings for the tree-prior nanoTabPFN, the four pretrained TFMs as well as the real-data experiments?
Furthermore, it is unclear to me which PFN is used per experiment. The abstract promises seven different PFNs, yet most experiments seemingly are only based on TabPFNv2.5 and DVA-PFN. For example, for the experiments on the low-rank structure in section 3.2 (producing tables 5 and 6) or the experiments of section 4 (detailed in appendix C), are results for the other 5 models available, and on what basis did the authors choose which PFN to use per experiment?
Regarding the relevance concern:
The experiments are restricted largely to synthetic experiments, and some of the findings for these don't appear to be very surprising to me. For example, in section 3.1, the fact that the representation of the frequency of the sinusoids is (a) present and (b) confined to a low-dimensional subspace makes perfect sense because the PFNs were explicitly (and in the case of VA- and DVA-PFN exclusively) trained on such data and in this perfect, noise-free setting, fi is the only 1D parameter that the PFN needs to identify from the data and represent in its embeddings. For more complex settings (section 3.3.), the readouts become less reliable, as the authors note themselves. Can the authors explain more intuitively what the reader might learn from these experiments beyond the fact that (trained) PFNs organize the spectral information in a linearly decodable manner in some settings?
Furthermore, the experiments with real-world datasets (section 4.1) are seemingly only loosely connected to the spectral analysis of the rest of the paper, as these only confirm that the information that is dominantly used to form the prediction is in fact concentrated on a low-dimensional subspace of the representation and altering this part of the representation also changes the predictions. Is this reading correct, or are there further insights for real datasets/prediction tasks that the spectral analysis reveals?
In general, (mechanistic) transparency is rightfully identified as an open challenge for PFNs, but it remains unclear to me how the findings of this paper actually help in this regard. Kernel discovery is presented as a possible application, with the authors stating that their approach "supports GP regression competitive with iterative baselines at a fraction of their cost," though at the same time, they also state that "the decoder is not meant to compete with kernel discovery methods". To this end, can the authors explain in more detail how their findings address "1) the lack of mechanistic transparency" or the "2) absence of explicit, portable representations"? For example, for a real-world prediction problem where a PFN is applied, can the authors explain how the insights or methods from this work help to better understand how the PFN arrived at its prediction?
The appendix calls the architecture of DVA and VA PFN a "Transformer-based Conditional Neural Process", yet nowhere in the paper any method from the neural processes literature is cited, e.g., Garnelo et al., 2018 or Kim et al., 2019. Can the authors explain more clearly how their work relates to this field?
Reply to Reviewer 1PmB: New Code Shared in Supplementary; Clarification on PFNs used and What Reader Learns
Reproducibility Concerns We sincerely thank the reviewer for running our code and pointing this out. The supplementary archive was an earlier snapshot of our working code, and some settings were updated after it was taken. In the particular case the reviewer reports (experiment G, subspace patching), the snapshot's script uses 200 probe signals, while Fig. 1 and Table 10 were produced with 500. With 500 signals, the script reproduces the reported value at k = 2. Supplementary has updated codes covering all experiments of original and updated paper.
On Which PFN is used where: All seven PFNs enter the representation tests of Section 3 (Table 1 has 6 versions and the nanoTabPFN probes). Further, the deeper analyses use a subset: the low-rank analysis (Table 6) uses TabPFN, the pooling ablation (Table 7) DVA-PFN, the interventions of Section 4 DVA-PFN, VA-PFN, TabPFN and nanoTabPFN, and the decoder DVA-PFN.
The choice follows each model's role in our analysis. VA- and DVA-PFN are trained by us on a known stationary prior, thus becoming first choice to probe for spectral information. Among public TFMs, we picked TabPFN as it is the most widely used released TFM, and nanoTabPFN which we could train ourselves and then test for a prior without spectral content. Since patching needs model-specific hooks, the other three released TFMs (TabPFN 3.0, TabICLv2, TabSwift) appear only in the representation tests, to show that the effect is not specific to one model.
Relevance
Several of our existing results pointed to one simple picture of how a PFN predicts. First, frequency is represented even by a nanoTabPFN trained only on a decision-tree prior, so it is not inherited from a spectral training prior. Second, in DVA-PFN spectral structure is formed by the first attention step, and later blocks keep it. Third, in our architecture sweep (Appendix E), a model with two blocks and width 64 comes within 2.5× of the error of a model with 12× more parameters. Fourth, on real data the context acts on the prediction through a small subspace. Together these suggest a two-stage computation is done by PFNs: the network first estimates from the context how quickly the target changes, and we hypothesize that this estimate sets how far the network borrows strength across context points.
Q1 (what the reader learns). We agree that a PFN trained on sinusoids is expected to represent their frequency. What the experiments add is the following.
(i) The representation does not depend on the prior. The decision-tree nanoTabPFN recovers frequency with R² = 0.95 (0.09 with random weights), and the four public TFMs with R² ≥ 0.99. As every regressor must decide how far to borrow information across context points, so learning how fast the target varies is forced by the task, not by the prior.
(ii) In a new frequency-only test (Sec 4), we compare against a third signal that differs from the original in frequency alone. Patching only the frequency-ranked directions moves the prediction to that signal. The causal effect is 0.998 on DVA-PFN and 0.90 on TabPFN, where 1 means a complete move. Non-spectral and random directions of the same size move it by at most 0.07.
(iii) These results have design consequences for PFNs. Because the estimate forms in one attention step and occupies few directions, small and shallow PFNs carry most of the performance (Appendix E).
Q2 (real datasets). First, we agree that on real data we locate the channel, not a spectral quantity, as stated in Limitations section of the paper. What holds on real data is the entire "low-dimensional channel" discussion: with 5 to 499 features, 64 of 192 directions, chosen without labels, carry 85–95% of the context's effect in all but one pair, against 6–35% for random directions. Our work therefore shows that tabular models also arrange the context's information in a low-dimensional channel, which can inform the design of lighter TFMs.
Q3 (transparency & portability). For transparency, we are able to answer where the structure sits, what it encodes, and whether it is used to make the prediction. For portability, we show a possibility: the structure can be read out as an explicit kernel, and this can be refined further into a cheap replacement for per-task GP kernel fitting.
Q4 (neural processes). PFNs and conditional neural processes share the same setup of a network which takes a context set and predicts at query points in a single forward pass, after training across many tasks. We used "Transformer-based Conditional Neural Process" only to describe this architecture. The difference among these two is that a PFN is trained only on synthetic datasets drawn from an explicit prior, so its output is read as an approximation of the PPD under that prior (Müller et al., 2022). Our analysis uses nothing specific to neural processes. We call the models PFNs in the appendix and cite Garnelo et al. (2018) and Kim et al. (2019) in Sec 2.1.
I want to thank the authors for their detailed response and the work put into the revision of both the code and the manuscript. The supplementary material is now significantly better organized and allows for a more thorough analysis than the first version. At first sight, several of my earlier points regarding reproducibility are resolved, like the experiment for Table 1. Given the extent of the changes made in response to all reviewers, I will read the revised paper again in more detail before finalising my assessment.
Though after a quick first pass, I tend to remain unconvinced regarding my earlier relevance criticism, especially regarding questions Q2 and Q3, mostly because some experiments don't align with some of claims made or a more thorough experimental setup leads to different conclusions, at least as I understand them.
For example, the decoded kernel from the PFN latents is offered as a cheap replacement for per-task kernel fitting, but once you train the baselines (e.g. DKL) properly using your own experiment code (1000 instead of only 100 Adam steps) its median MSE is lowered more than 40-fold to below your encoder's values.
Furthermore, to my understanding, the experiments resulting in Figure 14, which are supposed to transfer the findings of Figure 1 from the toyish sinusoidal setting to a more "serious" setup with RBF-GP functions, fail at exactly this and actually contradict their own caption, "The spectral subspace dominates at every k for both architectures, demonstrating that causal structural information is compactly organised beyond the sinusoidal training distribution," which clearly is not the case looking at the plot.
As already stated, I will read the entire manuscript in more detail again, hence, this does not constitute my final assessment. At this point in time, I don't expect the authors to provide any further experiments or further updates to the paper or supplementals, though I want to give them the chance to clarify any potential misconceptions.
On a minor note:
Some of the text passages quoting earlier versions of the figures and tables seemingly are not yet updated
In the paper you report that you use TabPFNv2.5, while in the code you seemingly use TabPFNv2.6
Reply to further questions: Updates on subspace patching with clarification on others
We thank the reviewer for the feedback and, more importantly, for the effort put into improving the quality of the paper. We reply to the remaining questions below and request the reviewer to consider these during their final read.
1. Decoder vs. per-task fitting. Yes, per-task fitting with more optimization will beat our decoder. As stated in Sec. 5 and the Limitations, we present the decoder as a read-out test, i.e., evidence that kernel information is present in the PFN latents and can be extracted without test-time optimization. By "cheap" in our earlier reply we meant a better cost–quality trade-off at small budgets. We showed that, at a fraction of the cost, the decoder is competitive with per-task fitting run for 100 steps (Table 4) where it has the lower mean MSE on three of five families and the lower median MSE on all five, while on RBF and SM (Q = 4) DKL and RFF are already better at 100 steps. 1,000 Adam steps take roughly 10× the wall-clock time of the 100 steps in Table 4, which puts DKL and RFF at about 450× and 3,000× the decoder's 3.9 ms per task (extrapolating linearly). We also ask the reviewer to consider the matched-training control we added in the last round (Appendix D.3.1). A decoder trained on the latents of a randomly initialized PFN eventually reaches a comparable held-out level, but needs far more training steps to beat the input-ignoring baseline. This indicates that pretrained PFN latents hold the kernel information in an organized form that is much easier to read out. We therefore present the decoder not as a replacement for per-task fitting (we have updated the abstract, contributions and Limitations to state the 100-step budget), but as evidence that the information is present and cheap to decode, and we hope it becomes a direction the community explores further.
2. Figure 14. Your question is valid, and the figure needed an update to support our claimed findings. We respectfully present new results showing that the conclusion holds, and that the correct subspace must be identified for the conclusions to be valid. We keep the old subspace selection (re-run with the new script for fair comparison) and show how it differs from the updated selection method of interventional PCA. Updated Script: spectral_pfn_code/interventional_pca/run_interventional_pca.py
— New result: A change in the lengthscale (the spectral parameter of the RBF kernel) acts through a small subspace: in all four models (TabPFN 2.6, TabPFN 3.0, TabICLv2 and DVA-PFN), 4 directions transfer at least 0.90 of its effect, against at most 0.03 for 4 random directions. This subspace also carries changes in the realization, i.e., which particular function is drawn at a given lengthscale. This is the same pattern we reported for sinusoids in our first reply (Table 2), where the subspace found through frequency also transfers a phase-only change (CE 0.80–1.00). It is also consistent with the common observation in mechanistic interpretability that learned directions are often "polysemantic", carrying several variables at once (Olah et al., 2020; Elhage et al., 2022).
— Why the old figure had an issue: The main limitation of the old Figure 14 was how we selected the subspace. It ranked principal components by their Pearson correlation with the lengthscale. For a direction v with score s=v⊤h, this correlation is r=Cov(s,logℓ)/(sd(s)sd(logℓ)). It is normalized by the spread of the scores, i.e., scaling all scores along v by a factor c leaves r unchanged. However, a patch along v moves the hidden state by (sB−sA)v, and the same scaling multiplies this change by c. Correlation therefore loses the information about how far the hidden state moves along a direction, so a highly correlated direction can carry little of the change, and correlation ranking can miss the causal subspace.
The reviewer did not ask for new experiments, but this analysis clarified the limits of our own finding, and we thank the reviewer for prompting it. The new Figure 14, the protocol (Table 13) and a script with built-in checks are in Appendix C.2 and the supplement. A summary is in the new Section 4 paragraph "Beyond sinusoids".
3. Stale passages. We have corrected the remaining passages and made a full pass over quoted values (for example, the TabPFN probe R² in Sec. 3.2 and the ratio to the oracle GP in Sec. 5), and will repeat it for the final version.
4. TabPFN version. Thank you for spotting this. The package-default regressor we load is TabPFN 2.6 (24 blocks; tabpfn 7.1.1), which we had mislabelled as 2.5. All experiments we had labelled TabPFN 2.5 use it, except the frequency-only test of Appendix C.5, which loads the TabPFN 2.5 checkpoint (18 blocks). Appendix A.4 now names the checkpoint and package version.
The authors perform a thorough empirical analysis of whether and in what way the representations of trained prior-data fitted networks store spectral information about the context data. The results point unanimously towards the fact that spectral information is vital for the proper functioning of a PFN. In light of this, the authors leverage Bochner's theorem to distill (potentially huge) pretrained PFNs into an extremely lightweight spectral mixture covariance function (which is universally approximating in the space of stationary covariance functions) that can be freely plugged into a GP.
Critical strengths
S1 The writing quality is generally excellent.
S2 The rigour and thoroughness of empirical investigation is very high; the authors ask interesting research questions and carefully design their experiments to ensure the answers are made plain in the results.
S3 I find the topic of the paper to be very interesting and I am confident that it will inspire further fruitful research from the community.
Critical weaknesses
None of the weaknesses that I identified were substantial enough to diminish my assessment of the paper. More minor weaknesses are listed below as "Further Requests" from me.
Questions for authors
Further Requests
R1 In Table 1 and the corresponding text in Section 3.2, the max∣Δ∣ column should be replaced with a sign(Δ)max∣Δ∣ column so that readers can see which of the linear or nonlinear probe outperforms the other in each case. Furthermore, the "A linear read-out is enough" paragraph introduces the weight (a) estimation probe task but not the frequency (f) estimation probe task. This should be added. And finally, the standard error bounds for the R2 results columns are currently missing and should be added.
R2 In Section 3.2's "Analysis with a non-spectral prior" paragraph, the difference between the nanoTabPFN that achieves an R2 of 0.951 and the "six-block regression variant of nanoTabPFN" that achieves a slightly higher R2 needs to be made clearer$\textemdash$currently there is no apparent difference between the two nanoTabPFN setups and yet one somehow achieves a higher R2.
R3 In the paragraph on lines 315-322, the authors should mention the results of the V-patching intervention as well. In the previous paragraph they introduce the three interventions they perform, but they currently only discuss the results of two of them (with the v-patch results deferred to an appendix). This should be changed.
R4 In Section 4.1's experiment and the corresponding Table 2, all of the k50 results are multiples of 8. Did the authors perform a dense sweep over k's, or a strided sweep (e.g. with stride 8)? If the latter, could they repeat this experiment under a dense sweep (stride=1), or if the former, could they explain the unusual nature of the results?
R5 Rahimi and Recht (2007)'s random Fourier features method is devised for the purpose of scaling GP regression to larger datasets and circumventing the O(n3) time complexity of marginal likelihood and inference computations. It is not a method for learning a covariance function. However, there are later works that exploit the spectral form of RFF to use it as a kernel-learning method, e.g. this paper or most notably this one here. Throughout the experiments section, the authors cite the Rahimi and Recht paper when referring to the RFF-based kernel-learning baseline, which is not the right paper to cite. They should make it clearer exactly what method they are using for this baseline and update the citations accordingly.
R6 Bochner's Theorem allows us to express any stationary covariance function through its spectral density, which the authors exploit to obtain a portable stationary covariance function (expressed as a mixture of spectral components) that corresponds to any trained PFN. However, no PFNs are translation equivariant (except for a few examples in the neural process literature such as this or this), which means their representation learning mechanism corresponds to different covariance functions at different points across the input domain. As a thought experiment, consider a PFN trained on RBF covariance function GP data where the lengthscale is 10.0 for x<0.0 and 0.1 for x>0.0. Clearly, this data-generating process does not correspond to a stationary stochastic process, so what sort of stationary covariance function would the authors' Filter Bank Decoder recover from the trained PFN? Perhaps something like an RBF covariance function with lengthscale of 1.0? Or perhaps something less sensible? More importantly, many of the SCM-based prior data generating processes on which PFNs are trained do not correspond to stationary processes, so how can we be sure that the "stationary-ified" version of the process that the Filter Bank Decoder recovers is a meaningful reproduction of the ground-truth version? I would appreciate it if the authors could comment on this, and if they could add a comment in their manuscript that acknowledges the non-stationarity of PFNs in general, and the possible limitations this engenders for their work.
R7 A rather trivial baseline that the authors have not included in their experiments is a GP which receives the latents (/final layer representations) of a trained PFN as features$\textemdash$a DKL method in which the trainable MLP is replaced with the (pre-)trained PFN backbone. The authors should include such a baseline as it would help delineate the relative merits and drawbacks of the authors' "stationary-ifying" Filter Bank Decoder method against a more straightforward "use the PFN's representations in a GP" approach. When they include this baseline, they should of course highlight that their approach allows for discarding the PFN entirely, whereas this baseline requires keeping the PFN and performing costly GPU-based forward passes for each prediction.
R8 Table 3 needs some tender loving care from the authors. The "×10−⋅" should be removed from each table entry, either by factorising it out and reporting the MSE in thousandths, or by reporting RMSE instead, which would make all the numbers larger and displayable as decimals rather than in standard form (e.g. 4.6×10−4=0.021). Furthermore, the emboldening is currently a bit of a mess$\textemdash$why is the entire row of average prediction times emboldened? My recommendation is that the authors partition the baselines into a "fair comparison" and an "unfair comparison" section, where the former contains the authors' method, DKL, RFF, and the baseline proposed in R7, while the latter contains the PFN (which isn't a GP at all), the oracle GP (which receives privileged information), and Amor-struct. (which also received privileged information), and they should embolden the best performing baseline in the "fair" section in each case.
Editorial Suggestions
E1 On line 62, "turns" should be used in the plural to yield "none of these analyses turn...".
E2 On lines 63-65, the sentence beginning with "Although, " is not a proper sentence. The final phrase "yet kernel is not read from a trained PFN" needs to be corrected, and the comma after though potentially needs to be removed (depending on what the final phrase is changed to).
E3 On line 76/77, "We study total seven PFNs" should rather be "We study seven PFNs in total".
E4 On line 76/77 ", there is no reason for the first letter of "Four" to be capitalised.
E5 On line 79/80, "decision-tree prior" should be preceded by an indefinite article ("a") to yield "a decision-tree prior".
E6 On lines 82-83, "In the four PFNs we intervene on" implies that the reader should already be aware that the authors perform an intervention study. This should be replaced with something like "We also perform an intervention study on four PFNs...".
E7 On line 87/88, the phrase "the organization" is not quite appropriate here. Something like "this phenomenon"/"this behaviour" would be better.
E8 On line 98, the comma in "best, predictive" should be removed.
E9 On lines 109-110, the word "loss" should not be split into two separate words "lo ss" as it currently is.
E10 On line 111/112, "used" should be in the present tense, "use", in order to maintain consistency of tense with the rest of the section.
E11 On line 147 the comma in "used, by" should be removed.
E12 On line 201, "with" should be changed to "and".
E13 On line 202, "if" should be changed to "is", and the "is" near the end of the line should be removed.
E14 In page 14's final sentence, there should be a comma before "not the existence.".
E15 The first sentence of Section 3.2 needs to be rewritten as it currently makes little sense. My recommendation is as follows: "The geometric alignment tests of Section 3.1 show that H contains frequency information, but not necessarily that any spectral quantities can be directly read out from it."
E16 On line 232, "probes and frequency and weights" should be replaced with something like "probes, and both frequency (f) and weights (a)".
E17 On line 254, "the organization" should be changed to something like "linear accessibility of frequency information in H".
E18 In the final sentence of page 5, the authors refer to linear probing for a1 and a2, but these quantities were never introduced or used previously in any exposition; only a and 1−a. The authors should correct this, presumably by changing "a1,a2)" to "a)".
E19 On line 309 (and beyond), the authors need to define what they mean by "t-grid".
E20 On lines 319-320, the phrase "and k random orthogonal directions a baseline." makes little sense in the context$\textemdash$perhaps there is a missing word (e.g. "as a baseline")? Regardless, please can the authors clarify what they mean by this.
E21 On line 356 (as well as 88/89), the authors should clarify what they mean by "eight pairs of real datasets". Why/how are the eight real datasets arranged in pairs?
E22 In the paragraph corresponding to lines 398-407, the authors should clarify what they mean by a "realization". A dataset/task, or something else?
E23 On line 413/414, "Sec. 2.2" should be in parentheses.
E24 On line 475, "does escapes" should be changed to "does escape".
E25 On line 481, "TFMS" should be changed to "TFMs".
Reply to Reviewer E2AZ: Thanks, all suggestions accepted and manuscript updated.
We thank the reviewer for encouraging comments and more importantly for careful reading of our paper. Below is the list of updates and revisions requested by the reviewer. The revised manuscript has all the updates in magenta color fonts.
R1 (Table 1). Thank you. We have made all three changes.
Signed gap. We replaced max∣Δ∣ with the signed gap Δ=RMLP2−Rlinear2, taken at the target with the larger ∣Δ∣.
Frequency probe. The paragraph "A linear read-out is enough" (Sec. 3.2) now introduces both targets: the frequency f of single sinusoids and the mixing weight a of two-component signals.
Standard errors. The R2 columns now show mean ± standard error over five seeds.
R2 (nanoTabPFN). Apologies for the confusion. In the original submission, the 0.951 ± 0.022 result comes from the 3-block classifier and the 0.989 ± 0.011 result from the 6-block regressor. The two therefore differ in task (classification vs. regression), in the tree prior and in model size. The configurations behind these two numbers were described in Appendix A.4 ("Additional PFNs", paragraph "nanoTabPFN on a decision-tree prior").
Now we update section 3.2 and add results of a 3-block regressor as well. It has the same size as the classifier and reaches 0.951 ± 0.012 (same mean as classifier). Therefore, the higher value comes with the larger 6-block regressor.
R3 (V-patching). We have added the V-patch result to the main-text paragraph on activation patching. Replacing V transfers the prediction completely (CE = 0.999 ± 0.002 over the 50 pairs), as expected for a positive control.
R4 (Table 2, k50). The original sweep actually used the grid k∈{1,4,8,16,24,32,48,64,96,192}, so each reported k50 was the upper end of the grid interval in which the 50% crossing fell. As suggested by the reviewer, we have rerun the sweep at every k from 1 to 96, with 24 draws per pair, and updated Table 2 and the appendix table with these values. The new block-12 k50 values, in Table 2 order, are 15, 21, 18, 28, 16, 25, 32, 44 and 60, so k50 now ranges from 15 to 60 across the eight real pairs (previously 24 to 64). None of the discssion or inference require any change as all numbers are inside or at the border of earlier reported cells.
R5 (RFF baseline). We thank the reviewer for the references. Our baseline uses 128 RFF cos(ωx+b) with the frequencies ω=w/ℓ, with w∼N(0,1) drawn once and held fixed, and the phases b are drawn uniformly and also held fixed. A GP with a linear kernel on these features is fit per task. This is the fixed-spectral-point variant of the sparse-spectrum GP of Lázaro-Gredilla et al. (2010, Sec. 5.2), with one cosine and a random phase per frequency (Rahimi and Recht, 2007). It is therefore an approximation of an RBF-kernel GP with fitted hyperparameters, not a method that learns the spectrum. We have renamed the baseline, rewritten its description in the main text and appendix, and now cite all three works (Rahimi and Recht, Willson et.al. and Lázaro-Gredilla) clearly.
R6 (non-stationarity). We agree and have expanded the limitation in Section 6. We also acknowledge that stationary kernel restrictions do not apply to PFNs and they can represent covariance that changes with location. On the thought experiment (lengthscale 10 for x<0 and 0.1 for x>0): We think the decoder will return one stationary kernel for the whole context, so it cannot indicate where each lengthscale applies. One possible outcome is a kernel with both a broad and a narrow spectral component, weighted by how much of the context falls on each side. We have not evaluated this case, and leave making a non-stationary kernel decoder for future work.
R7. Thank you, done. We added a GP on the frozen DVA-PFN's final-layer latents (DKL with the trainable network replaced by the pretrained PFN) to the fair group of Table 4; the protocol is in Appendix D.4. It also needs a PFN forward pass for every new query point (7.16 s per task, against 3.9 ms for the decoder, which uses the PFN once per context).
R8. Thank you, done. Table 4 (formerly Table 3) and the NLPD table (Table 21) are rebuilt as requested:
Units: MSE in units of 10⁻³, stated once in the caption; mean ± SE over 100 tasks.
Groups: fair, i.e. context only (Decoder, DKL, RFF, GP on PFN latents), and reference (PFN, Amor-struct., Oracle GP).
Bold: only the lowest fair-group mean is bold; the time row is not bold.
Label: the SM row is corrected to Q = 2.
Updated details are in Appendix D.4
Editorial Changes:
E1-E25: We have updated the manuscript with all the suggested changes and have done a careful reading to iron-out typos and grammatical mistakes.
I thank the authors for their detailed response. I am pleased to confirm that all of my concerns have been sufficiently addressed and reflected in the new revision. I particularly appreciate their response to R6 which offers some lovely insight that I hadn't thought of myself.
Reply to Reviewer L3tK
We thank the reviewer for the comments.
1. Does the decoder need the PFN? PFN is not needed for the information but it facilitate efficient learning significantly. A decoder trained on the raw context can learn the same kernels, which is expected, since everything the PFN knows comes from the context. The decoder asks a different question: whether the frozen PFN has already computed this information and holds it in a readable form. A head on the frozen latent matches the raw-input decoder while training fewer parameters, and it trained stably, whereas two of three raw-input restarts collapsed to a constant.
We also want to highlight that section 5 ("What the decoder reads") and Appendix D.3 of original manuscript that a decoder trained end to end on the raw (x, y) context reaches similar kernel error (Table 17), given enough capacity. Section 5 introduces it as a constructive test of whether the structure found in Sections 3 and 4 can specify a full covariance function.
We have now trained a decoder on random PFN initialization at matched budget (see reply to reviewer eZ4S). It shows that compared to random PFN, it is significantly more computationally efficient to train a decoder on trained PFN. This finding matches Section 3.2 and Fig. 6. An MLP probe, which has enough capacity, recovers frequency even from the latents of a randomly initialized PFN (R² ≈ 1.0), while a linear probe succeeds only on the trained network (0.998 against 0.18).
2. Normalization Apologies for the confusion caused. As the PFN and the GP stage of the decoder use different preprocessing, and each result applies to its own stage, there is no mismatch
The pipeline in Appendix A.2, which standardizes each context and then applies sigmoid(0.75·z), is the PFN's input preprocessing only. Theorem 1 is about what the PFN can see. Standardization and the unit-norm normalization used in the proof both remove the global scale of the function values, since they differ only by a constant factor. A fixed pointwise map applied afterwards, such as the sigmoid, cannot restore that scale. Rescaling the GP by any positive constant c therefore leaves the PFN's input unchanged, which is exactly the invariance the proof relies on, so the conclusion of Theorem 1 holds for the A.2 pipeline.
Proposition 1 is about the GP stage, which does not use the sigmoid. The scale estimate α̂ and the GP regression are computed on the context targets without the sigmoid, so the setting of Proposition 1 applies directly. Its proof also uses only the second moment E[yyᵀ] = αK, so it does not rely on Gaussianity. We will state in Section 5 and Appendix D.1 which preprocessing each stage uses.