- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focus on prior-data fitted network (FPN) that are designed for data generation. It observes that the FPN can deviate from the true posterior they are approximating on OOD data. This can make sense, since the OOD data distribution itself can deviate from the training data. In that instance, FPN can provide better generalization than the true posterior they were trained to optimize on. Knowing this, the authors decided to train FPN by increasing diversity and complexity of the data even if it hurt the true bayesian posterior prediction capabilities. They analyze the trade-off between complexity and diversity over the FPN performances. The authors highlight that FPN does not perform exact bayesian inference and that they are learning to leverage inductive bias that are shaped by the training distribution.
One interesting experimental result is that having diverse priors can outperform prior that are less diverse but closer to the downstream task. However, this might come at a cost, since performance could become a bit worse on the training task. They also empirically show that complexity of the training data can also be a predictor of the FPN performance on real world data.
The authors conclude that their experiments allow them to say that FPN are mostly shaped by their architectures and training data, and that they might deviate from Bayesian inference. The authors argue that their experiment will help practitioners designing better prior.
Critical strengths
Good experimental ablations. The experimental results are interesting and they are replicated across different model capacities.
Critical weaknesses
Mostly 1D tasks for the experiments (except one 2D experiment), not sure what the results might be on more complex data distributions. Also, the paper is about more principled data generation, but it seems that the main recommendations of the paper are adding more diverse data and parameters to the model, which is kind of well known. The paper provide an interesting view around what's the effect of more data and more parameters on the FPNs, but I am not sure about the novelty criteria there.
Questions for authors
- You claim in the abstract that design of synthetic data remains largely heuristic. However I am not sure that this paper make it less heuristic since there are additional dimension a practioner might need to play with: Data diversity and model complexity. So how do you reconcile this idea of providing guided principle while showing this in fact require empirical evaluations?
Thank you for your questions. We’re glad you found the experimental results interesting and appreciated the controlled ablations and replication across model capacities.
We appreciate the importance of testing these findings beyond simple settings. In Section 5, both the synthetic SCM tasks and UCI experiments use 5-dimensional inputs, and we additionally evaluate on multivariate TabArena regression datasets which go up to 80 dimensions. Our complexity findings are therefore tested directly on more complex synthetic and real-world data.
For the experiments in Sections 3 and 4, we intentionally use low-dimensional settings where the Bayesian predictive distribution can be computed efficiently.
The natural intuition is that diversity helps by bringing the training distribution closer to the test distribution. Our novel finding is that increasing diversity can improve PFN generalization even when it moves the training distribution farther from the downstream task and makes exact Bayesian prediction worse (Section 4.1). The same change to the prior therefore has opposite effects on the PFN and what we would expect from a true Bayesian predictor. This demonstrates that diversity can benefit the learned predictor beyond improving distribution alignment.
This finding changes how synthetic priors should be evaluated: a prior that is better matched to the downstream task can nevertheless train a worse PFN. Understanding this distinction is especially valuable for synthetic data generation, where the training distribution itself is a design choice.
To address your feedback, we’ve added a new Section 6 to the updated manuscript that explicitly provides guidance for practitioners. We examine two ways to use the proxies before committing to the full training procedure:
Use early proxy measurements to screen candidate priors. We adapt the complexity and diversity proxies to partial training curves and compare their rankings with final UCI performance. Using only the first 10% of logged training observations, the complexity and diversity proxies achieve Spearman correlations of 0.94 and 0.74, respectively, with final downstream performance. Both proxies identify the three best-performing priors among 18 candidates, showing that early measurements can guide prior selection well before training finishes (Figures 8a and A.11, left).
Use smaller models to guide prior selection for larger architectures. We compare the rankings of our complexity and diversity proxies from a smaller model (2 layers, width 128) with the rankings from a larger model (8 layers, width 512) across various priors. Using the full available training curves, the rankings agree closely across architectures, with Spearman correlations of 0.98 for complexity and 0.93 for diversity. These results support using a smaller model to identify priors that remain highly ranked by either proxy in a larger architecture (Figures 8b and A.11, right).
We hope these additions clarify how practitioners can narrow the search before committing to full training and downstream evaluation.