acceptodds
Under review as a conference paper at ICLR 2027

LinearPFN: Amortized Variable Selection for Linear Models with Interactions

Abstract

Spike-and-slab regression is a standard Bayesian formulation of variable selection: it returns a posterior distribution over which candidate effects are active rather than a single selected subset, so that every candidate effect carries an inclusion probability. Its cost grows exponentially with the number of candidate effects, so the posterior can be enumerated exactly only when the number of predictors is small. Beyond that reach, the posterior has to be approximated, typically by Markov chain Monte Carlo over the model space, which requires a fresh run for every dataset and, within a fixed budget of steps, may fail to converge. We present LinearPFN, a prior-data fitted transformer network that amortizes spike-and-slab inference for linear models with main effects and pairwise interactions. The network is pretrained once on synthetic datasets, drawn from an explicitly specified prior, and a single forward pass over a new dataset returns posterior inclusion probabilities, posterior-mean coefficients and posterior predictive distributions with no per-dataset fitting. The prior is conjugate by design, so that the posterior for each fixed set of active effects has a closed form, and wherever the exact posterior is still computable by enumeration we verify the network's outputs against it. On real predictor matrices from published social-science datasets, with outcomes drawn from the prior so that the true active set is known, LinearPFN attains a higher per-dataset selection AUC and a higher under the median probability model rule than five classical baselines. The lead holds when the coefficients, the interactions or the noise depart from the prior.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
amortized Bayesian inference, prior-data fitted networks, spike-and-slab regression, variable selection, posterior inclusion probabilities, in-context learning, tabular foundation models, transformers, social science

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JVqkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce LinearPFN, a prior-data fitted transformer that amortizes Bayesian variable selection in linear regression models including pairwise interactions. Trained on synthetic datasets generated from a spike-and-slab prior, LinearPFN predicts posterior inclusion probabilities, coefficient means, and predictive distributions in a single forward pass. The method is validated against exact posterior inference in small problems and compared with classical variable-selection methods on semi-synthetic benchmarks using real predictor matrices.

Critical strengths

The paper proposes an efficient approach to Bayesian variable selection under linear models with pairwise interactions. Its main strength is amortizing inference under a spike-and-slab prior, providing posterior inclusion probabilities and coefficient estimates in a single forward pass, without per-dataset fitting. The explicit prior specification also enables validation against the exact posterior in tractable settings.

Critical weaknesses

My main concern is that the paper provides limited understanding of when and why LinearPFN should be preferred over existing methods. The main benchmark generates outcomes from the same prior used for pretraining, naturally favoring LinearPFN over methods with different assumptions. Although the authors consider some departures from this prior, these experiments do not sufficiently characterize the limits of its applicability. In particular, it remains unclear which observable properties of a new dataset would allow practitioners to anticipate whether LinearPFN is likely to outperform conventional alternatives. This limits the practical significance of the reported improvements, beyond the demonstrated ability to approximate a specified Bayesian posterior.

Questions for authors

Q1: Figure 4 examines performance across several dataset characteristics, but can the authors identify concrete regimes in which LinearPFN is preferable to existing methods, and, importantly, how practitioners could anticipate this from observable dataset properties before selecting a method?

Q2: The robustness experiments modify coefficient distributions, interaction strength, and noise, while preserving the linear structure and strong heredity. How sensitive is LinearPFN to more fundamental violations of these assumptions? In particular, would the reported advantage persist when interactions violate strong heredity?

5 comments
AuthorsOpenReview ↗

Response to Reviewer JVqk: when to prefer LinearPFN and violations of the prior

We thank the reviewer for their review and for recognizing our efficient approach to Bayesian variable selection. We updated the PDF (changes in blue) to address the reviewer's questions.

Q1. When to prefer LinearPFN

We agree that a reader should be able to tell from observable properties of a dataset when to prefer which method. The two most obvious candidates are the number of rows n and the number of predictors p, which are known before a method is chosen and do not depend on the outcome. We split the 64,880 datasets of all eight simulated panels by n and p: the main benchmark, its five variants, the weak-signal panel and the design of Hastie et al. (§4, App. G.3, H.2–H.5).

  1. Every n and p. In every cell of n and p (p ≤ 30), LinearPFN is the best method in F1 more often than any baseline, on 38 % to 79 % of the datasets, the strongest baseline on 16 % to 64 % (App. H.7).

  2. Every panel. Averaged over each panel, LinearPFN has the highest mean F1 in all eight panels, ahead of the best baseline by +0.035 to +0.163 (App. H.7).

  3. Very weak signal. On the 1,600 datasets of the eight panels with a true R² below 0.05, hierNet selects slightly better (mean F1 0.47 against 0.45, App. H.7). The true R² is not observable to a practitioner. It could only be estimated by first fitting another model, with no guarantee that the estimate is close to the true R².

We therefore argue that, within the tested range of n and p, a practitioner can generally prefer LinearPFN for variable selection.

Revision: In the results section (§4), we now state when to prefer LinearPFN for selection. App. H.7 gives the results by n and p.

Q2. More fundamental violations

We agree that a lead on outcomes drawn from our own prior is a statement under the assumed model, and that the three modifications in the initial submission are only moderate violations of the prior. We now report three further experiments outside the prior: (1) the seminal variable-selection simulation design of Hastie, Tibshirani and Tibshirani (2020), (2) interactions that violate strong heredity and (3) nonlinear main effects.

  1. A design not drawn from our prior (App. H.4). With pairwise interactions added, we draw 2,880 datasets in 288 cells that cross predictor correlation, p, n, coefficient pattern, interactions and signal-to-noise ratio. Strong heredity and linearity still hold in it. LinearPFN has the highest AUC in 268 of the 288 cells and the highest mean F1, 0.68 against 0.60 for stability selection, the best baseline. At the lowest signal-to-noise ratio (0.05), hierNet (F1 0.35) and glinternet (0.33) select better than LinearPFN (0.30).

  2. Interactions violating strong heredity (App. H.2). In each main-benchmark dataset, we move every active interaction, with its coefficient, to a pair of predictors of which at least one has no active main effect. The main effects stay as they were. For 26.5 % of the moved interactions, neither predictor has an active main effect. This changes 5,196 of the 10,000 datasets. The others have no interaction or no inactive main effect, so nothing can move. LinearPFN keeps a higher AUC and F1 than every baseline on the changed datasets and leads SuSiE by +0.014 [+0.009, +0.019] in F1. On the 452 changed datasets small enough for the exact posterior (p ≤ 5), LinearPFN's F1 falls by 0.093 against the same datasets before the move and that of the exact posterior under the same prior by 0.104. At p ≤ 5, the loss is therefore the cost of the heredity assumption in the prior.

  3. Nonlinear main effects (App. H.3). Each active main effect enters through a monotone transform of its predictor (convex, concave or a step) under the same coefficient. LinearPFN keeps a higher AUC (0.95) and F1 (0.72) than every baseline, +0.163 [+0.159, +0.167] in F1 above the best one, SuSiE. Its F1 falls by 0.056 against the same datasets without the transform, SuSiE's by 0.101. On the 2,299 datasets small enough for the exact posterior, LinearPFN's F1 falls by 0.045 and that of the exact posterior under the same prior by 0.071. Most of the loss comes from falsely selected interactions. LinearPFN's interaction precision, the share of selected interactions that are truly active, falls from 0.83 to 0.71, SuSiE's from 0.62 to 0.43.

Revision: We adjusted the abstract and the results (§4). The limitations now state where hierNet does better.

Reference. Hastie, Tibshirani and Tibshirani (2020), Statistical Science 35(4). doi:10.1214/19-STS733

The new robustness evaluations meaningfully strengthen the paper and largely address my second concern.

My main reservation remains the practical applicability of the results. While the additional experiments provide convincing evidence within the tested regimes, the evaluation still relies predominantly on synthetic outcomes under controlled assumptions. I therefore remain uncertain about how reliably these advantages would translate to real-world applications, and whether practitioners can identify when LinearPFN should be preferred.

Unless I am overlooking something in the interpretation of the results, I believe more extensive empirical validation is needed to establish broader practical applicability. I recognize that this likely goes beyond what can reasonably be addressed during rebuttal, and therefore maintain my original rating.

AuthorsOpenReview ↗

A new real-outcome benchmark and when to prefer LinearPFN

We thank the reviewer for the follow-up and are glad that the new robustness evaluations address the second concern. For the remaining one, we added a benchmark on real outcomes and updated the PDF (App. H.13, changes for this follow-up in green).

Why selection is usually scored on simulated outcomes

To score a variable selection, one has to know which effects are truly active. With a real outcome this is unknown, for every method. Variable-selection methods are therefore usually evaluated on real predictors with simulated effects, as for SuSiE (Wang et al., 2020) and in our main benchmark.

A benchmark with real outcomes

To address the reviewer's concern that our evaluation relies predominantly on synthetic outcomes, we evaluated all six methods on a further real-outcome benchmark. The new benchmark keeps the real outcome and instead adds predictors that are known to be irrelevant (Wu et al., 2007; Hu et al., 2018). For each predictor of a table, we add a copy whose values are shuffled across the rows (for a random subset when the table would otherwise exceed 30 predictors). The copy has the same values as the original, but no relation to the outcome. Any selected effect that involves a copy, its main effect or one of its interactions, is therefore a false selection. We applied this to 49 tables with their real outcomes from two public benchmark collections, OpenML-CTR23 and PMLB. For each method, the table gives the false selection rate (FSR), following Hu et al. (2018) and, for models with interactions, Crews et al. (2011): the share of its selected effects that involve a copy and are therefore false, per table and averaged over the 49 tables.

FSRAll effectsMain effectsInteractions
LinearPFN1.4 %1.4 %0.5 %
SuSiE7.5 %2.6 %12.5 %
Stability selection8.0 %4.8 %10.9 %
hierNet45.1 %38.9 %50.0 %
glinternet45.9 %38.0 %51.2 %
Lasso53.5 %22.0 %62.9 %

What the results show

LinearPFN has the lowest FSR, 1.4 % against 7.5 % to 53.5 % for the baselines. The gap is largest for interactions, the focus of the paper: 0.5 % against 10.9 % to 62.9 %.

In numbers: of the 275 effects LinearPFN selects, 3 involve a copy. For the baselines it is 46 to 1,392.

LinearPFN still selects effects among the original predictors. It selects 205 of their main effects, more than SuSiE (177) and stability selection (184), and 67 of their interactions. SuSiE and stability selection select more interactions (156 and 129), but 39 and 46 of the interactions they select involve a copy, against 1 for LinearPFN. LinearPFN is thus cautious without being empty: it almost never selects an effect that is known to be false, and it still selects more main effects than the two baselines that come closest to it.

When to use LinearPFN

A practitioner who wants a parsimonious model of main effects and pairwise interactions, with few wrongly selected effects, should prefer LinearPFN. Where the true effects are known, its selections are the most precise of the six methods. On the main benchmark, on average 92 % of the effects it selects are truly active, against 39 % to 84 % for the baselines. On the simulation design of Hastie et al. (2020), which is not drawn from our prior, it is 84 %, against 33 % to 78 %. LinearPFN also has the highest mean F1 in every panel (App. H.7). On real outcomes, only false selections can be counted, and they tell the same story: its selections contain the fewest certainly false effects of all six methods, overall, for main effects, for interactions and in each of the three bins of p.

Revision: The results (§4) now report the real outcomes. The paragraph on when to prefer LinearPFN, added in the first revision, now recommends it when the goal is a parsimonious model with few wrongly selected effects and reports its precision. The contributions (§1), the evaluation (§3.3) and the conclusion (§5) name the new benchmark, and App. H.13 gives the details (Tables 24 and 25).

We hope that this benchmark addresses the reviewer's remaining concern and would be grateful if they reconsidered their rating.

References.

  1. Crews et al. (2011), Computational Statistics & Data Analysis 55(6), 2026–2037. doi:10.1016/j.csda.2011.01.009
  2. Hastie, Tibshirani and Tibshirani (2020), Statistical Science 35(4). doi:10.1214/19-STS733
  3. Hu et al. (2018), Variable selection using pseudo-variables. arXiv:1804.01201
  4. Wang et al. (2020), JRSS Series B 82(5), 1273–1300. doi:10.1111/rssb.12388
  5. Wu et al. (2007), Journal of the American Statistical Association 102(477), 235–243. doi:10.1198/016214506000000843
Reviewer_AEtHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LinearPFN is a PFN that amortizes spike-and-slab variable selection with pairwise interactions. It is pretrained on synthetic data (p ≤ 30, n ≤ 2,000) from a conjugate prior calibrated to social-science data. It matches the exact posterior for p ≤ 5 and outperforms five baselines in selection on 10,000 semi-synthetic datasets.

Critical strengths

The conjugate prior allows direct validation against the exact posterior (p ≤ 5), using six criteria fixed before training, which is unusually rigorous for amortized inference.

Critical weaknesses

1.No theoretical justification for the amortized posterior.

The only argument is that the losses are minimized in expectation by the posterior quantities, which assumes unlimited capacity and in-distribution data. There is no approximation guarantee, and no evidence that the network's approximate posterior concentrate as n grows.

2.Unclear benefit of amortization at this scale.

All experiments use p ≤ 30, and the pair heads scale as O(p²). If the method cannot handle large p, what is the advantage of pretraining? In this range, MCMC already takes under a minute.

3.Unfair speed comparison.

The headline 0.03 s is GPU time, compared against CPU baselines. On a single CPU thread, LinearPFN (7.4 s) is slower than all baselines except hierNet.

4.Limited novelty.

The method is essentially TabPFN with variable-selection heads added. The architecture and training follow existing PFN work, and the main contribution is the prior design, which seems lack of novelty.

Questions for authors

See weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer AEtH: validation, speed against MCMC and novelty

We thank the reviewer for their review and for naming the validation against the exact posterior as a strength. We updated the PDF (changes in blue) to address each point.

W1. Theoretical justification

The reviewer asks for guarantees that the trained network approximates the exact posterior and that its approximate posterior concentrates as the sample size grows. These requirements go beyond the guarantees available for the practical Bayesian alternatives in our comparison, MCMC and SuSiE. The remaining baselines return point estimates or selection frequencies, so posterior approximation guarantees do not apply to them.

MCMC is only guaranteed to converge asymptotically, so finite runs are assessed with convergence diagnostics (Vehtari et al., 2021). In our experiments, the sampler met its convergence criteria on 71 % of runs at p = 21–30. SuSiE has a guarantee of convergence to a stationary point of its variational objective under stated conditions (Wang et al., 2020, Proposition 2), but this does not establish posterior approximation accuracy or asymptotic selection consistency. Without such guarantees, empirical tests are the practical basis for validating any of these methods. Whatever happens inside the network, its output is an ordinary linear regression model with explicit coefficients. It can be checked on held-out data like a model fitted by any other method, and in simulations we also compare its estimates with the known effects.

We assess posterior fidelity through checks against the exact posterior, on six criteria fixed before training. We now also show that, with more rows, the network's PIPs move toward 0 or 1, as the exact ones do. Their mean entropy falls from 0.426 to 0.134 bits between n = 32 and 2,000, against 0.433 to 0.118 for the exact posterior. Beyond the training range (p = 3, 5, n up to 8,000), the network keeps concentrating, and the two select the same effects on at least 98.2 % of candidates. This is empirical evidence over the tested range, not a proof of asymptotic selection consistency.

Revision: App. H.8.

W2. The benefit of amortization at p ≤ 30

Our goal is variable selection for linear models with interactions as used in the social sciences. Of the 383 eligible social-science datasets, 368 have at most 30 predictors, and 281 of them more than 5.

Above p = 10, MCMC is slow and does not always converge. "Under a minute" holds for the sampler's median over all p. With one CPU thread, the sampler takes 57 s at the median and 866 s at the 90th percentile, and LinearPFN 6.3 s and 42.2 s on the same datasets. At p = 21–30, the sampler takes 1,072 s at the median and 3,715 s at the 90th percentile, and 71 % of its runs meet the convergence criteria, against 16.4 s and 87.3 s for LinearPFN on the same thread and 0.068 s and 0.378 s on one GPU.

Revision: App. H.9.

W3. The speed comparison

We agree. The 0.03 s is GPU time, and on one CPU thread, the hardware of the baselines, LinearPFN takes 7.40 s at the median and is slower than every baseline except hierNet. Table 2, with the fastest method in bold, included the relevant information, but our conclusion then indeed compared GPU to CPU. The conclusion now compares LinearPFN and MCMC on the same CPU thread and gives the GPU time separately. Its speed matters against MCMC, which targets the same posterior. On the same datasets and thread, LinearPFN is 9.1 times faster at the median and 21 times faster at the 90th percentile.

Revision: the conclusion, App. H.9.

W4. Novelty

Bayesian variable selection with pairwise interactions requires inference over a model space that quickly becomes too large to enumerate. We address this problem by designing an architecture with dedicated pair inputs to account for interactions, constructing synthetic training datasets based on the characteristics of social-science data, and testing the model extensively against the exact posterior, five established baselines, and departures from the training prior. LinearPFN meets all six posterior-validation criteria fixed before training. In selection, it has the highest mean F1 on all eight simulated panels we tested (64,880 datasets, App. G.3, H.2–H.7).

Furthermore, the method is useful in practice: it returns inclusion probabilities and model-averaged coefficients for main effects and pairwise interactions up to p = 30, without per-dataset optimization or tuning. For social-science applications, these outputs allow researchers to assess uncertainty about which effects to include alongside their estimated sizes. On the same CPU thread, it is 9.1 times faster than MCMC at the median and 65 times faster at p = 21–30 (W2, W3). This is particularly useful when analyzing many datasets or repeated subsamples, since each analysis requires only a forward pass.

References.

  1. Vehtari et al. (2021), Bayesian Analysis 16(2). doi:10.1214/20-BA1221
  2. Wang et al. (2020), JRSS Series B 82(5), 1273–1300. doi:10.1111/rssb.12388

Thank you for the detailed response and the added experiments. I have a few further questions.

W1. The concentration analysis in App. H.8 covers only p ≤ 5. Is there a similar experiment for p > 5 ?

W2. I am not fully convinced by the argument that p ≤ 30 covers social-science data. Also, this paper does not state social science as its target. The abstract, introduction and conclusion present LinearPFN as a general method, and social science enters only through the prior calibration and the benchmark data. The figure of 368 out of 383 tables rests on RDATASETS, which consists mostly of reduced textbook examples. These counts are also taken after cleaning that drops every categorical column with more than two levels (App. F.1).

Applications of Bayesian model averaging in the social sciences often exceed this range even without interactions. Fernández et al. (2001) use 41 candidate regressors, and this is the application behind Ley & Steel (2009), which the paper cites for its prior. Sala-i-Martin et al. (2004) use 67. I therefore do not find the restriction to p ≤ 30 convincing, even for social-science applications.

References

[1] Fernandez, Carmen, Eduardo Ley, and Mark FJ Steel. "Model uncertainty in cross‐country growth regressions." Journal of applied Econometrics 16.5 (2001): 563-576.

[2] Sala-i-Martin, Xavier, Gernot Doppelhofer, and Ronald I. Miller. "Determinants of long-term growth: A Bayesian averaging of classical estimates (BACE) approach." American economic review 94.4 (2004): 813-835.

AuthorsOpenReview ↗

Response to follow-up: concentration at p > 5 and a network for up to 100 predictors

We thank the reviewer for the further questions. We ran the experiments below and updated the PDF (changes for this follow-up in orange).

W1. Concentration at p > 5

Yes. Above p = 5 the exact posterior cannot be enumerated, so the reference is the MCMC sampler of App. D.2, which targets the same posterior. We drew datasets from the prior at p = 10, 20 and 30 and n = 128, 512, 2,000, 4,000 and 8,000 and fitted the network on 100 datasets per cell and the sampler on 20 of them. Of the 300 sampler runs (3 values of p × 5 values of n × 20 datasets), 272 met the convergence criteria. For the comparisons we use only those runs.

Both posteriors concentrate as n grows. Between n = 128 and 8,000, the sampler's mean PIP entropy falls from 0.162 to 0.034 bits at p = 10 and from 0.090 to 0.012 at p = 30, and the network's from 0.178 to 0.052 and from 0.100 to 0.023. In every cell, the network and the sampler select the same effects on at least 98.5 % of candidates.

Revision: App. H.8.

W2. Scope and the restriction to p ≤ 30

Fernández et al. (2001) and Sala-i-Martin et al. (2004) select among 41 and 67 main effects. We designed LinearPFN for the linear model with pairwise interactions, where the number of candidate effects grows quadratically with the number of predictors. At p = 30, LinearPFN selects among 30 main effects and 435 interactions, 465 candidate effects in all, 11.3 and 6.9 times as many as in the two cited applications.

We trained the paper's model (LinearPFN-30) for p ≤ 30. To show that this is not a limit of the method, we trained a second model, LinearPFN-100, for larger p. It covers up to p = 100, or 5,050 candidate effects (123.2 and 75.4 times as many as in the two cited applications), uses the paper's prior with p up to 100 and the same number of training steps, and starts from LinearPFN-30 with a ten times smaller learning rate. Against the exact posterior it meets all six validation criteria (App. H.10). At p = 40 to 100, its PIPs correlate with those of converged sampler runs at a median of 0.991 over cells (App. H.10). We benchmarked it in two settings (App. H.11). First, on the simulation design of Hastie et al. (2020) at p = 100. Second, on the tables of our two benchmark pools with more than 30 predictors, which we had to cut to 30 columns, at their full width of up to 100 columns (549 datasets built as for the main benchmark, 544 scored). In both, its mean F1 exceeds that of every baseline (0.57 against at most 0.45 on the Hastie design, 0.57 against at most 0.36 on the wide tables). The same holds on the wide tables under each of the five variants of the outcome model, which change the coefficients, the interaction load, the noise, the heredity of the interactions or the linearity of the main effects.

Target domain. LinearPFN is not restricted to social-science data. The first round already added the simulation design of Hastie et al. (2020), which comes neither from our prior nor from social-science data (App. H.4). We added a benchmark of 352 predictor matrices from life science (152), medicine (126) and other fields (74), with 1,000 datasets constructed as for the main benchmark (App. H.12). LinearPFN-30 has the highest mean AUC and F1 overall and in each of the three groups, 0.96 and 0.75 against 0.96 and 0.77 on the main benchmark. We built the main benchmark on social-science data because there the linear model with interactions is the standard form of a quantitative claim (Cohen et al., 2013; Aiken et al., 1991), so we expect the approach to matter most there. LinearPFN-100 also covers the 41 and 67 predictors of the two cited applications. We use Rdatasets for the predictor matrices only. They give the benchmark real correlations and marginal distributions, which a purely synthetic design cannot provide, and the simulated outcomes give a known ground truth against which selection can be scored.

Categorical predictors. LinearPFN and four of the five baselines accept only numeric input, and only glinternet handles factors natively. To give all methods the same columns, App. F.1 drops factors with more than two levels. If all factors were coded as dummy columns instead, 334 of the 383 tables would still have at most 30 predictors, against 368 now, and at least 370 would have at most 100, within the range of LinearPFN-100. In the released code, LinearPFN will accept factors directly, dummy-code them and leave out interactions between levels of the same factor.

Revision: Results, Limitations, App. F.1, H.10 to H.12.

We hope that these results address the reviewer's remaining concerns, and we would be grateful if they reconsidered their score.

References.

  1. Aiken et al. (1991), Sage. ISBN 978-0-7619-0712-1
  2. Cohen et al. (2013), Routledge. doi:10.4324/9780203774441
  3. Hastie et al. (2020), Statistical Science 35(4), 579–592. doi:10.1214/19-STS733
Reviewer_W7KKOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LinearPFN, a prior-data fitted network for Bayesian variable selection in linear models with pairwise interactions. Trained on synthetic data from a spike-and-slab prior, it predicts posterior inclusion probabilities, coefficients, and predictive distributions in one forward pass. The authors validate its outputs against exact inference and evaluate it on real predictor matrices with simulated outcomes, alongside seven datasets with real outcomes. Overall, the method is well motivated and the evaluation is substantial, although some baseline settings and robustness claims need further clarification.

Critical strengths
  1. The probabilistic formulation is clear, and validation against the exact posterior provides strong evidence that the model learns the intended inference task. This makes the reported probabilities more convincing than selection accuracy alone.

  2. The experiments cover diverse predictor matrices, multiple baselines, and several forms of prior misspecification. Fast GPU inference makes the method potentially useful for repeated analyses.

Critical weaknesses
  1. SuSiE is limited to 20 single effects (Appendix F.3), while some evaluation datasets contain substantially more active effects. Increasing this limit or reporting sensitivity analyses would help establish whether the comparison is fair. This matters because the selection advantage is a central empirical result.

  2. The robustness evaluation does not cover several important failure regimes. All variants preserve strong heredity, while the main benchmark excludes null datasets and cases with R^2 < 0.1. Evaluating interactions without active parent main effects and false selections under null or weak signals would better establish the method's practical scope.

Questions for authors
  1. How sensitive is the comparison with SuSiE to increasing its maximum number of single effects?

  2. Are results available for violations of strong heredity and for null or weak-signal datasets? These would help clarify the scope of the robustness claims.

  3. Table 8 reports a maximum PIP discrepancy of 0.600 on the body-fat dataset. What explains this discrepancy, and does it change the variable-selection decision?

3 comments
AuthorsOpenReview ↗

Response to Reviewer W7KK: SuSiE settings, heredity, null signal and the body-fat case

We thank the reviewer for the positive assessment and the precise questions. We updated the PDF (changes in blue) to address the reviewer's questions.

Q1. SuSiE with more single effects

We agree that the comparison is only fair if SuSiE's limit of 20 single effects does not hold it back. We reran SuSiE with L = 10, 50 and 100 on all 10,000 main-benchmark datasets. LinearPFN's lead in F1 stays between +0.118 and +0.134 for every L.

LF1AUCF1, the 864 datasets with > 20 active effectsLinearPFN minus SuSiE, F1
100.650.850.33+0.127 [+0.123, +0.131]
20 (paper)0.660.850.46+0.118 [+0.114, +0.122]
500.650.860.51+0.126 [+0.122, +0.131]
1000.640.860.52+0.134 [+0.130, +0.139]
  1. All datasets. L = 20, the paper's setting, gives SuSiE its highest F1. A larger L raises its F1 on the datasets with more than 20 active effects and lowers it on the others through more false selections.

  2. The 864 datasets with more than 20 active effects. LinearPFN has F1 0.65 on them. It leads SuSiE at L = 100 by +0.126 [+0.116, +0.136] and is ahead on 78.7 % of them.

Revision: App. H.6.

Q2. Strong heredity, null and weak signal

We agree that these regimes belong in the evaluation. We added three panels on the same real predictor matrices.

  1. Interactions without their parents. In each main-benchmark dataset, we move every active interaction, with its coefficient, to a pair of predictors of which at least one has no active main effect. The main effects stay as they were. For 26.5 % of the moved interactions, neither predictor has an active main effect. This changes 5,196 of the 10,000 datasets. The others have no interaction or no inactive main effect, so nothing can move. On the 5,196 changed datasets, LinearPFN keeps a higher AUC and F1 than every baseline and leads SuSiE by +0.014 [+0.009, +0.019] in F1. Its F1 falls by 0.112 against the same datasets before the move. On the 452 changed datasets small enough for the exact posterior (p ≤ 5), LinearPFN's F1 falls by 0.093 and that of the exact posterior under the same prior by 0.104. At p ≤ 5, the loss is therefore the cost of the heredity assumption in the prior.

  2. No active effect. In 1,000 new datasets, the outcome is an intercept plus noise. LinearPFN returns the empty model on 93.7 % of them, with 0.08 false selections per dataset. SuSiE does so on 81.2 % (2.16 false selections), the lasso on 58.3 % (1.84) and hierNet on 7.0 % (3.93).

  3. Weak signal. In 2,000 datasets, R² lies between 0.02 and 0.1. LinearPFN has a higher AUC than every baseline (0.95) and 0.09 false selections per dataset, against 1.68 for SuSiE and 4.08 for hierNet. Its F1 is 0.57, against 0.53 for hierNet, the best baseline (paired difference +0.035 [+0.018, +0.053]). Within this panel, below R² = 0.05, hierNet reaches F1 0.52 and LinearPFN 0.51.

Revision: App. H.2 and H.5. The limitations now state that hierNet selects better at very low signal.

Q3. The PIP gap on the body-fat dataset

One case explains most of the gap. The selection differs on 4 of the 91 candidate effects, and both selected models fit about equally well.

  1. Where the gap is. It is on the main effect of height: 0.893 for LinearPFN and 0.293 for the sampler run of Table 8. The sampler is stable. Four further runs, each four times as long, agree within 0.017 on every effect and give height a mean PIP of 0.304.

  2. What drives it. Case 42 has a recorded height of 29.5 inches, 11.1 standard deviations below the mean. Without it, the sampler gives height 0.701 and LinearPFN 0.961, and the largest gap on any effect falls from 0.589 to 0.260 (both against the longer runs).

  3. Effect on the selection. LinearPFN selects height and hip, and the sampler selects weight and thigh. Weight, hip and thigh correlate at 0.87 to 0.94, and the two models fit about equally well (OLS R² 0.727 against 0.730). Without case 42, the two selections differ on 2 effects.

Revision: new App. G.5 with the PIPs with and without the case.

I thank the authors for addressing my comments. I will maintain my score and increase my confidence in the assessment.

AuthorsOpenReview ↗

Thank you for raising the confidence

We thank the reviewer for the follow-up and for increasing their confidence in the assessment.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

Summary of the revision

We thank the three reviewers for their reviews and the follow-up discussion. The revised PDF marks the changes in four colours: blue for the first revision in response to all three reviewers, orange for the follow-up of Reviewer AEtH, green for the follow-up of Reviewer JVqk and purple for the final reply to Reviewer AEtH. Most of the new material is in a new Appendix H, and each reply points to the sections that answer it. The main changes are:

First revision (blue)

  • Outcomes outside the prior (App. H.2–H.5). Two further variants of the main benchmark break strong heredity (H.2) and make the main effects nonlinear (H.3). We added the simulation design of Hastie, Tibshirani and Tibshirani (2020), which is not drawn from our prior (H.4), and panels with no or only weak signal (H.5).
  • When to prefer LinearPFN from observable properties (§4, App. H.7). We split the 64,880 datasets of all eight simulated panels by the observable n and p. In every cell of n and p, LinearPFN is the best method in F1 more often than any baseline. It also has the highest mean F1 in each of the eight panels. Only at very weak signal, on datasets with a true R² below 0.05, does hierNet select slightly better. However, a practitioner cannot observe the true R². We therefore conclude that, judged by observable properties and within the tested range of n and p, LinearPFN can generally be preferred for variable selection.
  • SuSiE with more single effects (App. H.6). With L = 10, 50 and 100 beside the paper's L = 20, LinearPFN's lead in F1 stays between +0.118 and +0.134.
  • Concentration and cost (App. H.8, H.9). As n grows, LinearPFN's PIPs move toward 0 or 1, as the exact ones do. The cost and convergence of the MCMC sampler are now reported by number of predictors, and the conclusion compares LinearPFN and MCMC on the same CPU thread, with the GPU time given separately.
  • Further checks (App. H.1, G.5). Two further training runs with new seeds also meet all six validation criteria. The largest PIP gap on the real-outcome tables, on the body-fat data, comes mostly from one extreme case.
  • The limitations now also state where hierNet selects better and the range of n and p the network was trained on.

Follow-up of Reviewer AEtH (orange)

  • Concentration at p > 5 (App. H.8). Against the MCMC sampler at p = 10, 20 and 30 and n up to 8,000, both posteriors concentrate as n grows, and the two select the same effects on at least 98.5 % of candidates.
  • A network for up to 100 predictors (App. H.10, H.11). LinearPFN-100 meets all six validation criteria. Its mean F1 exceeds that of every baseline on the design of Hastie et al. at p = 100 and on the benchmark tables with more than 30 predictors.
  • Predictor matrices from other fields (App. H.12). On 352 matrices from life science, medicine and other fields, LinearPFN-30 has the highest mean AUC and F1 overall and in each group.
  • Categorical predictors (App. F.1). The appendix now reports how many tables stay within p ≤ 30 and p ≤ 100 if every factor is dummy-coded.

Follow-up of Reviewer JVqk (green)

  • Real outcomes (§4, App. H.13). On 49 tables with their real outcomes from OpenML-CTR23 and PMLB, permuted copies of the predictors show how often each method selects effects that are certainly false. LinearPFN has the lowest false selection rate, 1.4 % against 7.5 % to 53.5 % for the baselines.
  • When to prefer LinearPFN (§4). The paragraph now recommends LinearPFN when the goal is a parsimonious model with few wrongly selected effects, and reports its precision.

Final reply to Reviewer AEtH (purple)

  • Beyond 100 predictors without retraining (§4, App. H.14). On the design of Hastie et al. at p = 200, 500, 1,000 and 2,000, with up to two million candidate effects, LinearPFN-100 keeps a mean F1 of 0.61 to 0.67. The lasso over all pairwise interactions reaches 0.22 at p = 200 and 0.16 at p = 500. After screening at p = 2,000, LinearPFN-100 reaches 0.57 against 0.22 for the lasso.
  • LinearPFN-30 beyond p = 30 (App. H.11). The paper's network, trained for p ≤ 30, has the highest mean F1 on the benchmark tables with more than 30 predictors, under each of the five outcome variants, and on the design of Hastie et al. at p = 100.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.