- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce LinearPFN, a prior-data fitted transformer that amortizes Bayesian variable selection in linear regression models including pairwise interactions. Trained on synthetic datasets generated from a spike-and-slab prior, LinearPFN predicts posterior inclusion probabilities, coefficient means, and predictive distributions in a single forward pass. The method is validated against exact posterior inference in small problems and compared with classical variable-selection methods on semi-synthetic benchmarks using real predictor matrices.
Critical strengths
The paper proposes an efficient approach to Bayesian variable selection under linear models with pairwise interactions. Its main strength is amortizing inference under a spike-and-slab prior, providing posterior inclusion probabilities and coefficient estimates in a single forward pass, without per-dataset fitting. The explicit prior specification also enables validation against the exact posterior in tractable settings.
Critical weaknesses
My main concern is that the paper provides limited understanding of when and why LinearPFN should be preferred over existing methods. The main benchmark generates outcomes from the same prior used for pretraining, naturally favoring LinearPFN over methods with different assumptions. Although the authors consider some departures from this prior, these experiments do not sufficiently characterize the limits of its applicability. In particular, it remains unclear which observable properties of a new dataset would allow practitioners to anticipate whether LinearPFN is likely to outperform conventional alternatives. This limits the practical significance of the reported improvements, beyond the demonstrated ability to approximate a specified Bayesian posterior.
Questions for authors
Q1: Figure 4 examines performance across several dataset characteristics, but can the authors identify concrete regimes in which LinearPFN is preferable to existing methods, and, importantly, how practitioners could anticipate this from observable dataset properties before selecting a method?
Q2: The robustness experiments modify coefficient distributions, interaction strength, and noise, while preserving the linear structure and strong heredity. How sensitive is LinearPFN to more fundamental violations of these assumptions? In particular, would the reported advantage persist when interactions violate strong heredity?
5 comments
The new robustness evaluations meaningfully strengthen the paper and largely address my second concern.
My main reservation remains the practical applicability of the results. While the additional experiments provide convincing evidence within the tested regimes, the evaluation still relies predominantly on synthetic outcomes under controlled assumptions. I therefore remain uncertain about how reliably these advantages would translate to real-world applications, and whether practitioners can identify when LinearPFN should be preferred.
Unless I am overlooking something in the interpretation of the results, I believe more extensive empirical validation is needed to establish broader practical applicability. I recognize that this likely goes beyond what can reasonably be addressed during rebuttal, and therefore maintain my original rating.
A new real-outcome benchmark and when to prefer LinearPFN
We thank the reviewer for the follow-up and are glad that the new robustness evaluations address the second concern. For the remaining one, we added a benchmark on real outcomes and updated the PDF (App. H.13, changes for this follow-up in green).
Why selection is usually scored on simulated outcomes
To score a variable selection, one has to know which effects are truly active. With a real outcome this is unknown, for every method. Variable-selection methods are therefore usually evaluated on real predictors with simulated effects, as for SuSiE (Wang et al., 2020) and in our main benchmark.
A benchmark with real outcomes
To address the reviewer's concern that our evaluation relies predominantly on synthetic outcomes, we evaluated all six methods on a further real-outcome benchmark. The new benchmark keeps the real outcome and instead adds predictors that are known to be irrelevant (Wu et al., 2007; Hu et al., 2018). For each predictor of a table, we add a copy whose values are shuffled across the rows (for a random subset when the table would otherwise exceed 30 predictors). The copy has the same values as the original, but no relation to the outcome. Any selected effect that involves a copy, its main effect or one of its interactions, is therefore a false selection. We applied this to 49 tables with their real outcomes from two public benchmark collections, OpenML-CTR23 and PMLB. For each method, the table gives the false selection rate (FSR), following Hu et al. (2018) and, for models with interactions, Crews et al. (2011): the share of its selected effects that involve a copy and are therefore false, per table and averaged over the 49 tables.
| FSR | All effects | Main effects | Interactions |
|---|---|---|---|
| LinearPFN | 1.4 % | 1.4 % | 0.5 % |
| SuSiE | 7.5 % | 2.6 % | 12.5 % |
| Stability selection | 8.0 % | 4.8 % | 10.9 % |
| hierNet | 45.1 % | 38.9 % | 50.0 % |
| glinternet | 45.9 % | 38.0 % | 51.2 % |
| Lasso | 53.5 % | 22.0 % | 62.9 % |
What the results show
LinearPFN has the lowest FSR, 1.4 % against 7.5 % to 53.5 % for the baselines. The gap is largest for interactions, the focus of the paper: 0.5 % against 10.9 % to 62.9 %.
In numbers: of the 275 effects LinearPFN selects, 3 involve a copy. For the baselines it is 46 to 1,392.
LinearPFN still selects effects among the original predictors. It selects 205 of their main effects, more than SuSiE (177) and stability selection (184), and 67 of their interactions. SuSiE and stability selection select more interactions (156 and 129), but 39 and 46 of the interactions they select involve a copy, against 1 for LinearPFN. LinearPFN is thus cautious without being empty: it almost never selects an effect that is known to be false, and it still selects more main effects than the two baselines that come closest to it.
When to use LinearPFN
A practitioner who wants a parsimonious model of main effects and pairwise interactions, with few wrongly selected effects, should prefer LinearPFN. Where the true effects are known, its selections are the most precise of the six methods. On the main benchmark, on average 92 % of the effects it selects are truly active, against 39 % to 84 % for the baselines. On the simulation design of Hastie et al. (2020), which is not drawn from our prior, it is 84 %, against 33 % to 78 %. LinearPFN also has the highest mean F1 in every panel (App. H.7). On real outcomes, only false selections can be counted, and they tell the same story: its selections contain the fewest certainly false effects of all six methods, overall, for main effects, for interactions and in each of the three bins of p.
Revision: The results (§4) now report the real outcomes. The paragraph on when to prefer LinearPFN, added in the first revision, now recommends it when the goal is a parsimonious model with few wrongly selected effects and reports its precision. The contributions (§1), the evaluation (§3.3) and the conclusion (§5) name the new benchmark, and App. H.13 gives the details (Tables 24 and 25).
We hope that this benchmark addresses the reviewer's remaining concern and would be grateful if they reconsidered their rating.
References.
- Crews et al. (2011), Computational Statistics & Data Analysis 55(6), 2026–2037. doi:10.1016/j.csda.2011.01.009
- Hastie, Tibshirani and Tibshirani (2020), Statistical Science 35(4). doi:10.1214/19-STS733
- Hu et al. (2018), Variable selection using pseudo-variables. arXiv:1804.01201
- Wang et al. (2020), JRSS Series B 82(5), 1273–1300. doi:10.1111/rssb.12388
- Wu et al. (2007), Journal of the American Statistical Association 102(477), 235–243. doi:10.1198/016214506000000843
Response to Reviewer JVqk: when to prefer LinearPFN and violations of the prior
We thank the reviewer for their review and for recognizing our efficient approach to Bayesian variable selection. We updated the PDF (changes in blue) to address the reviewer's questions.
Q1. When to prefer LinearPFN
We agree that a reader should be able to tell from observable properties of a dataset when to prefer which method. The two most obvious candidates are the number of rows n and the number of predictors p, which are known before a method is chosen and do not depend on the outcome. We split the 64,880 datasets of all eight simulated panels by n and p: the main benchmark, its five variants, the weak-signal panel and the design of Hastie et al. (§4, App. G.3, H.2–H.5).
Every n and p. In every cell of n and p (p ≤ 30), LinearPFN is the best method in F1 more often than any baseline, on 38 % to 79 % of the datasets, the strongest baseline on 16 % to 64 % (App. H.7).
Every panel. Averaged over each panel, LinearPFN has the highest mean F1 in all eight panels, ahead of the best baseline by +0.035 to +0.163 (App. H.7).
Very weak signal. On the 1,600 datasets of the eight panels with a true R² below 0.05, hierNet selects slightly better (mean F1 0.47 against 0.45, App. H.7). The true R² is not observable to a practitioner. It could only be estimated by first fitting another model, with no guarantee that the estimate is close to the true R².
We therefore argue that, within the tested range of n and p, a practitioner can generally prefer LinearPFN for variable selection.
Revision: In the results section (§4), we now state when to prefer LinearPFN for selection. App. H.7 gives the results by n and p.
Q2. More fundamental violations
We agree that a lead on outcomes drawn from our own prior is a statement under the assumed model, and that the three modifications in the initial submission are only moderate violations of the prior. We now report three further experiments outside the prior: (1) the seminal variable-selection simulation design of Hastie, Tibshirani and Tibshirani (2020), (2) interactions that violate strong heredity and (3) nonlinear main effects.
A design not drawn from our prior (App. H.4). With pairwise interactions added, we draw 2,880 datasets in 288 cells that cross predictor correlation, p, n, coefficient pattern, interactions and signal-to-noise ratio. Strong heredity and linearity still hold in it. LinearPFN has the highest AUC in 268 of the 288 cells and the highest mean F1, 0.68 against 0.60 for stability selection, the best baseline. At the lowest signal-to-noise ratio (0.05), hierNet (F1 0.35) and glinternet (0.33) select better than LinearPFN (0.30).
Interactions violating strong heredity (App. H.2). In each main-benchmark dataset, we move every active interaction, with its coefficient, to a pair of predictors of which at least one has no active main effect. The main effects stay as they were. For 26.5 % of the moved interactions, neither predictor has an active main effect. This changes 5,196 of the 10,000 datasets. The others have no interaction or no inactive main effect, so nothing can move. LinearPFN keeps a higher AUC and F1 than every baseline on the changed datasets and leads SuSiE by +0.014 [+0.009, +0.019] in F1. On the 452 changed datasets small enough for the exact posterior (p ≤ 5), LinearPFN's F1 falls by 0.093 against the same datasets before the move and that of the exact posterior under the same prior by 0.104. At p ≤ 5, the loss is therefore the cost of the heredity assumption in the prior.
Nonlinear main effects (App. H.3). Each active main effect enters through a monotone transform of its predictor (convex, concave or a step) under the same coefficient. LinearPFN keeps a higher AUC (0.95) and F1 (0.72) than every baseline, +0.163 [+0.159, +0.167] in F1 above the best one, SuSiE. Its F1 falls by 0.056 against the same datasets without the transform, SuSiE's by 0.101. On the 2,299 datasets small enough for the exact posterior, LinearPFN's F1 falls by 0.045 and that of the exact posterior under the same prior by 0.071. Most of the loss comes from falsely selected interactions. LinearPFN's interaction precision, the share of selected interactions that are truly active, falls from 0.83 to 0.71, SuSiE's from 0.62 to 0.43.
Revision: We adjusted the abstract and the results (§4). The limitations now state where hierNet does better.
Reference. Hastie, Tibshirani and Tibshirani (2020), Statistical Science 35(4). doi:10.1214/19-STS733