acceptodds
Under review as a conference paper at ICLR 2027

GeneICL: A Tabular Foundation Model for Bulk Transcriptomics

Abstract

Gene expression is widely measured in biomedicine, yet clinical outcome prediction remains challenging due to high dimensionality, strong feature correlations, and limited labeled data. Large self-supervised transcriptomic foundation models often fail to outperform simple supervised baselines. Tabular foundation models offer an alternative through in-context learning, but are typically pretrained on generic synthetic data rather than transcriptomic structure. We ask whether transcriptomics-aware pretraining, rather than merely scale, is the missing ingredient. Towards this end, we introduce **GeneICL**, a 4.2M-parameter tabular foundation model that combines a semi-synthetic pretraining prior built from experimentally measured bulk expression profiles with a parameter-efficient weight-tied recurrent architecture. We further enable right-censored survival prediction through a training-free reduction to regression using Cox partial-likelihood residuals. We evaluate GeneICL on a benchmark of 80 clinical outcome-prediction tasks spanning classification, regression, and survival. Tabular foundation models consistently outperform self-supervised transcriptomic models, while GeneICL achieves the best overall rank among all evaluated foundation models and tuned baselines. GeneICL does so with up to 387 fewer parameters, no gradient updates at inference, and inference within seconds on a consumer laptop CPU.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
bulk transcriptomics, gene expression, tabular foundation models, in-context learning, foundation models, semi-synthetic pretraining, recurrent architectures, survival analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7N2EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author proposed a 4.24M looped transformer model called GeneICL. GeneICL uses VAE, PCA and a table transformer for pre-trained in-context learning. The authors evaluated GeneICL on classification, regression and survival outcome prediction. GeneICL outperformed other generic and transcriptomics foundation models. Ablation studies examined model architecture, pretraining distribution and inference-time ensembling.

Critical strengths
  1. The authors proposed an interesting training strategy combining synthetic task generation and in-context learning on modified TabICLv2.
  2. GeneICL achieves better performance with fewer parameters and an efficient architecture, compared with other gene foundation models.
Critical weaknesses
  1. The main novelty is the gene expression prior (VAE+PCA), but the ablation study is not sufficient. It is unclear why the learned VAE latent representations cannot be used directly. Comparisons with simpler generative priors and alternative input representations are missing.
  2. The authors leave cross-batch and platform as limitations. However, experiments within a single cohort are not useful in a real scenario.
  3. In Appendix A, the authors deleted targets with low TabPFN-3 score, which may introduce selection bias.
Questions for authors
  1. Why did the author use task-specific PCA rather than the learned VAE latent representations? An ablation study is needed. How does this choice affect the model's ability to transfer biological information across tasks?

  2. How does GeneICL performance change in terms of the number and diversity of synthetic pre-training tasks?

  3. Since there are relatively small improvements over elastic-net, what task can GeneICL achieve a significant improvement on? An experiment on cross-batch prediction can be helpful.

  4. How would the results change if no tasks are excluded by TabPFN-3?

5 comments

We thank the reviewer for their constructive feedback and for recognizing the potential of combining transcriptomics-specific pretraining with parameter-efficient in-context learning. We agree that further ablations and evaluations are important to establish GeneICL's contributions and practical applicability.

In response, we have substantially extended our experiments, including additional pretraining and representation ablations, cross-batch evaluations, and an analysis of potential benchmark selection bias. We summarize the main findings below.

W1 & Q1: Expression prior and input representation

We agree that the contribution of the VAE-based expression prior and task-specific PCA deserves further investigation. We therefore conducted additional ablations (Appendix H.1, Table 18).

Expression prior. We compare our VAE-based prior against independent Gaussian and low-rank multivariate Gaussian distributions fitted to ARCHS4, as well as the generic TabICLv2 prior and real ARCHS4 profiles.

Pretraining priorAUROC ↑Pearson ↑R² ↑
Independent Gaussian0.627-0.014-0.007
Multivariate Gaussian0.7310.5110.264
TabICLv2 prior0.7350.5030.231
Real ARCHS4 profiles0.7390.5090.259
VAE (ours)0.7410.5190.272

The independent Gaussian performs substantially worse, indicating that modeling marginal gene distributions alone is insufficient. While the multivariate Gaussian recovers much of this performance, the VAE prior achieves significantly higher Pearson and Spearman correlations (Holm-corrected paired Wilcoxon tests, p<0.05p<0.05). This suggests that the VAE captures useful structure beyond second-order correlations.

Input representation. We use support-fitted PCA to obtain a compact representation adapted to each downstream dataset, without requiring a fixed pretrained feature extractor. We additionally compare against VAE latent representations and the 2,000 most highly variable genes.

Input representationAUROC ↑Macro-F1 ↑Pearson ↑
VAE latents0.6680.533-0.047
Top-2,000 HVGs0.6850.5340.501
Task-specific PCA (ours)0.7410.6050.519

Task-specific PCA substantially outperforms both alternatives. We hypothesize that, unlike fixed VAE representations optimized for reconstruction, support-fitted PCA better preserves the variation relevant to each downstream cohort.

Together, these experiments provide empirical support for both our transcriptomics-grounded pretraining prior and task-adaptive input representation.

Q2: Number and diversity of synthetic pretraining tasks

GeneICL is pretrained on 60,000 independently generated tasks, with diversity arising from randomized feature subsets, nonlinear label functions, class structures, signal-to-noise ratios, and batch perturbations.

To investigate the importance of label-function diversity, we trained additional models with restricted function priors (Appendix H.1, Table 18).

Label-function priorAUROC ↑Pearson ↑R² ↑
Linear only0.7410.5100.261
Two-layer MLP only0.7350.5180.272
Full prior (ours)0.7410.5190.272

Restricting the prior to linear functions significantly reduces regression performance, whereas two-layer MLPs recover performance close to the full prior. This suggests that nonlinear task structure is important, while additional function diversity yields limited gains on the present benchmark.

We have clarified the task-generation process in Appendix F.2.

W2 & Q3: Cross-batch generalization and improvements over elastic net

We agree that cross-batch generalization is essential for clinical applications. To address this limitation, we extended GeneICL's architecture and pretraining prior to explicitly account for batch effects, introducing synthetic batch shifts and query-context attention.

Improvements over elastic net. We performed paired Wilcoxon tests across the 52-task main benchmark (Appendix A.2, Table 9). Compared with tuned elastic net, ensembled GeneICL significantly improves four of seven metrics after Holm correction: balanced accuracy (+0.031, p=0.0054p=0.0054), Pearson (+0.028, p<10−4p<10^{-4}), Spearman (+0.028, p=0.0015p=0.0015), and R² (+0.022, p=0.0084p=0.0084). Differences in AUROC, AUPRC, and macro-F1 are not statistically significant.

Cross-batch evaluation. We evaluate this extension on three clinical prediction tasks spanning ten clinical cohorts, including different profiling platforms, and ten BeatAML drug-response tasks across four sequencing batches. Each cohort or batch is held out in turn, with the remaining batches forming the support set (Appendix A.1).

ModelAUROC ↑Bal. Acc. ↑Macro-F1 ↑
Elastic net0.6920.5810.540
TabPFN-30.6770.5320.470
TabFM0.6860.5610.498
GeneICL (query attention off)0.7140.6040.567
GeneICL (query attention on)0.7170.6650.615

GeneICL achieves the best performance across all reported classification metrics, with particularly substantial improvements over elastic net in balanced accuracy (+0.084) and macro-F1 (+0.075). Exploratory paired Wilcoxon tests across the ten held-out cohorts yield uncorrected pp-values of 0.004 and 0.020, respectively.

Disabling query-context attention reduces balanced accuracy from 0.665 to 0.604 and macro-F1 from 0.615 to 0.567, demonstrating its contribution under distribution shift. GeneICL also achieves the best cross-batch regression performance (Pearson 0.488, Spearman 0.483, R² 0.229).

These results suggest that GeneICL's advantages become more pronounced under distribution shift, particularly for classification metrics sensitive to cohort-dependent decision thresholds. Full results are provided in Table 5.

W3 & Q4: Potential selection bias from TabPFN-3 filtering

We thank the reviewer for highlighting this concern. Our original motivation for excluding tasks with low TabPFN-3 performance was to focus on targets exhibiting sufficient predictive signal. However, we acknowledge that model-based filtering can introduce selection bias.

We therefore re-evaluated the benchmark including all eight previously excluded tasks (seven classification and one regression), expanding it to 34 classification and 26 regression tasks.

ModelAUROC ↑Bal. Acc. ↑Pearson ↑R² ↑
Elastic net0.6870.5910.4830.247
TabPFN-30.6740.5550.4700.236
TabFM0.6690.5540.4900.256
GeneICL (ensembled)0.6910.6160.5130.272

GeneICL retains its leading performance on the expanded benchmark. On the excluded tasks alone, tabular foundation models and conventional baselines achieve mean AUROC values between approximately 0.45 and 0.52, while every evaluated method obtains a negative R² on the regression task.

Although model-based filtering can introduce selection bias, the expanded benchmark preserves GeneICL's leading overall performance, suggesting that our main conclusions are robust to including these eight previously excluded tasks. Full results are provided in Appendix A.3, Tables 10–12.

Thank you!

We thank the reviewer again for their constructive suggestions. We believe the additional ablations, cross-batch experiments, and unfiltered benchmark evaluation substantially address the concerns raised and strengthen the empirical evidence for GeneICL. We would be happy to address any remaining questions during the discussion period.

Thank you for these substantial experiments. I appreciate the effort to address my concerns. However, the revised model introduces a lot of changes relative to the original submission, and these changes raise my additional questions.

In partiquery, with query-set attention, the prediction for a given patient can depend on the other supporting set patients. How sensitive are predictions to its size and composition?

I also noticed that there are now two separate classification and regression checkpoints, which should be noted in Figure 1. Besides, the strongest results use inference-time ensembling, where the reported CPU time of 7.5s was measured without ensembling. Could you also report the runtime of ensemble configuration?

Reviewer_j6LiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents GeneICL, a tabular foundation model that was pretrained on semi-synthetic data. The authors leverage a VAE to generate noisy samples of bulkRNA gene expression data that is used as pretraining data to train the tabular model. The authors then present several experiments on outcome prediction tasks. The authors claim that GeneICL achieves best overall rank among other tabular foundation models and other gene expression models.

Critical strengths
  1. The paper adds on to similar findings from several previous and concurrent works (Kedzierska et al., Boiarsky et al.) that simple linear models often beat self-supervised foundation models on downstream tasks for single-cell tasks. The paper confirms that similar findings hold over for bulkRNA tasks as well.

  2. I appreciate that the authors released their codebase allowing for inspecting the actual architecture and confirming some of their results.

Critical weaknesses
  1. The results do not support the claim that GeneICL significantly improves over baselines. The proposed pretraining approach shows little to no margin on classification tasks, and minor improvements on the regression tasks over the Elastic

  2. While the parameters are supposed to be 387x, the overall number of flops are similar to baselines with the ablations not supporting the weight-tying or looped transformer approaches. The primary latency improvements appear to be more a consequence of the PCA which reduces token lengths.

Questions for authors
  1. Please add paired tests over tasks (Wilcoxon or sign) and bootstrap CIs for the comparisons in Table 1. The reported SD is over pretraining seeds, which measures reproducibility, not whether the benchmark separates methods.

  2. It would also be great to add results of training an elastic-net on raw genes (glmnet-style), which is the standard baseline in this setting. The survival track already runs ridge on raw genes and ties GeneICL; the main benchmark has no raw-gene linear row.

3 comments

Scientific contribution and significance

We sincerely thank the reviewer for their thoughtful assessment. As the reviewer correctly highlights, recent work has exposed an important challenge for transcriptomic foundation models: despite extensive self-supervised pretraining, they often fail to outperform simple classical predictors on downstream tasks.

We believe GeneICL represents a significant step toward addressing this challenge by exploring a different pretraining paradigm. Rather than learning transcriptomic representations through self-supervision, GeneICL learns to solve supervised prediction tasks directly through in-context learning, without requiring task-specific weight updates.

Our results provide evidence that this direction is promising: GeneICL significantly outperforms existing transcriptomic foundation models, achieves meaningful improvements over generic tabular foundation models, and reaches competitive performance with carefully tuned classical methods.

We therefore view GeneICL not merely as an improvement in benchmark performance, but as an important step toward building transcriptomic foundation models that deliver genuine downstream predictive utility, while providing insights to guide future research.

W1 & Q1: Performance improvements and statistical significance

We agree that our original analysis can only improve from increased statistical rigor. We have therefore conducted paired Wilcoxon signed-rank tests across downstream tasks, with Holm correction across baselines within each metric, alongside 95% bootstrap confidence intervals for aggregate performance (500 resamples).

The results provide strong evidence for GeneICL's effectiveness:

  1. Transcriptomic FMs: GeneICL significantly outperforms all five evaluated self-supervised transcriptomic FMs across all seven primary metrics. This demonstrates substantial and consistent advantages across classification and regression tasks.

  2. Tabular FMs: GeneICL significantly outperforms all eight competing tabular FMs on balanced accuracy and macro-F1, with additional significant improvements on other metrics depending on the baseline. On the remaining metrics, GeneICL either significantly outperforms the baselines or shows no statistically significant differences with the best-performing baselines.

  3. Classical baselines: Importantly, GeneICL significantly outperforms the carefully tuned Elastic Net on four of seven metrics (balanced accuracy, Pearson, Spearman, and R²), while no statistically significant differences are detected on the remaining metrics. This is particularly encouraging given that GeneICL requires no task-specific supervised optimization.

Table R1: GeneICL versus tuned Elastic Net

MetricΔHolm p
Balanced accuracy+0.0310.0054
Pearson+0.028< 10⁻⁴
Spearman+0.0280.0015
R²+0.0220.0084

Δ denotes the mean paired improvement of ensembled GeneICL over Elastic Net. Holm correction is applied across baselines within each metric. Complete statistical comparisons and aggregate bootstrap confidence intervals are provided in Appendix A.2 (Tables 8–9).

Cross-batch generalization. We additionally evaluated GeneICL under distribution shift, using held-out clinical cohorts and sequencing batches (Appendix A.1, Table 5). GeneICL achieves substantially larger improvements over Elastic Net than on the within-cohort benchmark, particularly in balanced accuracy (+0.084) and macro-F1 (+0.075), providing further evidence of its practical utility.

These findings demonstrate meaningful progress toward building more effective transcriptomic foundation models, establishing substantial advantages over existing foundation-model approaches while achieving statistically supported improvements over strong classical predictors. We have revised the manuscript to reflect these findings and clarify the scope of our claims.

Q2: Raw-gene Elastic Net baseline

We thank the reviewer for this suggestion. We have extended our benchmark to include Elastic Net trained directly on all 20,021 genes without PCA, using the same cross-validation splits and task-specific tuning protocol.

ModelAUROC ↑Bal. Acc. ↑Pearson ↑R² ↑
Elastic Net (raw genes)0.7320.6130.4850.245
Elastic Net (PCA)0.7350.6210.5000.262
GeneICL0.7410.6400.5190.272
GeneICL (ensembled)0.7450.6520.5280.285

The new baseline has been added to Appendix Table 13.

W2: Parameter efficiency versus computational efficiency

We agree with the reviewer that parameter efficiency and computational efficiency should be distinguished more clearly. Our primary motivation for weight tying is parameter efficiency without reducing computational depth. Indeed, our untied-block ablation increases the parameter count from 4.15M to 19.15M, while yielding only marginal differences in downstream performance (see Table 2).

Thank you!

We sincerely thank the reviewer again for their constructive feedback, which has helped us strengthen both the empirical analysis and the presentation of our contributions. In light of the revisions, we would be grateful if the reviewer would consider revisiting their assessment and score.

We thank the reviewer again for their feedback to our submission. As the discussion period closes in about 48 hours, we would be glad to address any remaining questions or provide additional analyses the reviewer would like to see. We would also appreciate the reviewer’s feedback on whether our response has addressed their concerns.

Reviewer_kfQNOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper develops GeneICL, a 4.2M-parameter tabular-like foundation model specifically designed for bulk transcriptomics data and inspired by TabPFN and TabICL models. The model is made of a first encoder module alternating sample- and feature-wise attention across 3 transformer blocks, and a tied, recursive ICL block. GeneICL operates on the scores of the PCA components explaining 90% of the variance of the input data. It is trained on transcriptomics profiles generated with a VAE trained on true +500k bulk samples. For each sample, a classification and a regression labels are generated synthetically with randomly initialized MLPs.

GeneICL is evaluated on a set of 52 classification and regression tasks spanning drug-sensitivity, prognostic and diagnostic tasks. On these tasks, GeneICL reaches the best overall rank compared to existing tabular foundation models and pre-trained transcriptomic foundation models. Crucially, GeneICL does so with up to 387x fewer parameters, and with an inference running in a few seconds on a CPU.

Also, by reformulating survival outcome prediction into a regression task, they show their GeneICL adaptation beats other adaptations, baselines, and a survival specific tabular model (SurvPFN), while GeneICL has not been pretrained on survival-specific task.

Critical strengths
  1. A key contribution in the field competing with existing models. This paper is, to my knowledge, the first paper developing a tabular like foundation model specifically designed for bulk transcriptomics data. This is a strong contribution in a field where most of the methods use the self-supervised learning paradigm. Besides this novelty, GeneICL performs, in average, better than existing TFMs and transcriptomics foundation models on the evaluated tasks, while being much smaller and faster. To me, this paper represents a strong first step towards exploring the usage of tabular models specifically designed for transcriptomics data.

  2. A broad benchmark with clinically relevant tasks. The authors do not evaluate their model on commonly used tasks in the field (like cell-type annotation or tissue prediction) which mostly assess the information quality but not the usability of such a model on a real clinical problem. Instead, they build a benchmark on 27 classification and 25 regression tasks spanning response to drugs, prognosis and diagnosis. They also show GeneICL can adapt to survival outcome task without changing the architecture but by reformulating the task.

Critical weaknesses
  1. Unsupported or inaccurate claims. These claims lessen the overall credibility of the paper and could be addressed with a more careful presentation of the results. For example:
  • in the abstract: “Tabular foundation models consistently outperform self-supervised transcriptomic models” is false given TabDPT-turbo is performing worse than scFoundation and BulkFormer.
  • in the abstract: “We evaluate GeneICL on a benchmark of 80 clinical outcome-prediction tasks”. 10 of these tasks are cell-line drug screens, which Appendix A itself calls ‘preclinical’ (l.775). ‘80 clinical outcome-prediction tasks’ (l.22) is inaccurate.
  • l.418: “Replacing the single recurrent ICL block with eight untied blocks increases the parameter count from 4.24M to 19.25M, yet does not improve any of the evaluated metrics”. Table 2 shows an increase in balanced accuracy. This claim does not hold on the holdout dataset either (Table 4).
  • l.421: “replacing the bottleneck feature attention with full self-attention in the encoder increases the feature-axis complexity from O(n) to O(n2) with matched or lower performance.” Table 2 shows the best R^2 with full attention setting.
  1. Reported standard deviations underestimate variability. The reported standard deviation per task category (in Table 1 and 2) are computed over 3 pretraining seeds of GeneICL. They do not take into account the intrinsic variability of small tasks (n = 40) evaluated under 5-CV fold.
Questions for authors

Clarifications

  1. Missing design justification: The regression head is trained with MSE and returns a point estimate, while TabICLv2 predicts 999 quantiles. Why did you make this choice? Can you run an ablation to compare MSE with quantile prediction, as you did for the ranking loss?
  2. Incorrect claims or missing important information: Related to weakness n°1, can you rephrase the unsupported claims I listed? Please find below additional claims or missing information that I’d like to see rephrased:
  • l.245: “As in TabICLv2 […], the encoder alternates between column- and row- attention”. TabICLv2 runs them sequentially (column then row).
  • at the beginning of section 3.2, you cite architectural changes from TabICLv2. If your starting point is really TabICLv2, using an encoder that alternates column and row attention should be mentioned. This list omits the use of an MSE head instead of a quantile head.
  • l.56: the question you ask is rather “is self-supervised representation learning the right paradigm?”. The distinction is important, because many different pretraining objectives can be and are used for transcriptomics FMs trained with SSL.
  • Could you report how many samples GeneICL was trained on? I only found the number of samples used to train the VAE.
  1. Evaluation strategy. Related to weakness n°2. You use 5-fold CV for small tasks (n=40) which results in n_test = 8. Can you report the standard deviations for these specific tasks in the appendix? Can you propagate these SDs to the the aggregated per-category scores shown in Table 1 and 2?
  2. Working with PCs scores:
    • Can you clarify in average how many PCs are left when you keep 90% of the variance?
    • Can you compare the PCA-90% method with the raw gene expressions of 2000 highly variable genes?
    • Using PCs scores is an argument for removing RoPE and feature grouping (used in tabular models to prevent the collapse in feature representation): could you run an ablation with and without to show it did not prove efficient?
  3. Minor clarity suggestions:
  • you could define the “ensemble” method outside of Table 1 or make it explicit the definition lies in the legend of the Table.

Suggestions for future work

  1. Translational perturbation tasks are missing from this benchmark while they are one of the main challenges in translational research. It would be interesting to see how GeneICL performs on these tasks, and if this needs an architectural change or a task reformulation for GeneICL to handle them.
  2. Translational tasks often require extracting signal from a dataset and apply it to another one. Evaluating GeneICL’s robustness to such a distribution shift would be interesting. Distribution shifts could include: technology, tissue, species, or simply batch effect.
6 comments

We thank the reviewer for the constructive assessment and for recognizing GeneICL as a promising step toward transcriptomics-specific tabular foundation models. We agree that several claims were too strong and that our original standard deviations did not capture downstream sampling variability.

We have revised the identified claims, added uncertainty analyses and paired significance tests, and conducted the requested ablations. All results were regenerated using an updated model incorporating query-set attention and synthetic batch shifts for cross-cohort prediction (discussed below), so some values differ slightly from the original submission.

W1 & Q2: Unsupported or inaccurate claims

"Tabular foundation models consistently outperform self-supervised transcriptomic models." We identified a preprocessing issue affecting TabDPT-Turbo: its internal standardization removed the variance ordering of our principal components before applying PCA. Passing the leading 128 components directly and reevaluating 32 affected tasks improves AUROC from 0.69 to 0.72 and regression Spearman from 0.43 to 0.52. Since TabDPT-Turbo still does not outperform transcriptomic models on every metric, we changed "consistently outperform" to "often outperform" in the abstract.

"80 clinical outcome-prediction tasks." We corrected this to "80 translational outcome-prediction tasks" throughout the manuscript to account for the preclinical drug-screening tasks.

Untied blocks and full feature attention. We repeated both ablations and performed Holm-corrected paired Wilcoxon tests across tasks (Table 18).

VariantParamsAUROCBal. Acc.PearsonR²
GeneICL4.15M0.7410.6400.5190.272
Untied (8 blocks)19.15M0.7420.6420.5160.271
Full feature attention3.95M0.7370.6320.5140.262

Untied blocks increase parameters by 4.6× without statistically significant improvements. Full feature attention performs worse across all evaluated metrics, significantly so for Pearson and R². We revised the corresponding claims in Section 4.3.

Further clarifications (Q2). We corrected the attention attribution to TabFM (l.245), explicitly listed alternating attention and the MSE head as departures from TabICLv2 (Section 3.2), and changed the motivating question to "Is self-supervised representation learning the right pretraining paradigm?"

Each checkpoint is pretrained on 60,000 synthetic tasks, comprising approximately 17 million VAE-generated profiles (32–1,024 per task). The VAE was trained on 547,829 experimentally measured bulk profiles (Appendix F).

W2 & Q3: Variability of small tasks

We agree that the original standard deviations quantify variability across pretraining seeds, not finite-sample uncertainty. We therefore added three complementary analyses (Appendix A.2).

1. Per-task fold variability. Tables 6–7 report performance and five-fold SDs for all 52 tasks. For the six smallest classification tasks (40–46 samples), AUROC fold SDs range from 0.11 to 0.19, compared with a median of 0.075 across all classification tasks and 0.020 for tasks with more than 300 samples.

2. Aggregate uncertainty. We estimate 95% bootstrap intervals by resampling observations within tasks and, separately, both tasks and observations (Table 8).

MetricMeanWithin-task 95% CIHierarchical 95% CI
AUROC0.745[0.735, 0.757][0.721, 0.773]
Bal. Acc.0.652[0.642, 0.662][0.627, 0.678]
Pearson0.528[0.518, 0.537][0.495, 0.562]
R²0.285[0.270, 0.290][0.241, 0.316]

The hierarchical intervals additionally capture heterogeneity across benchmark tasks. Table 8 reports the remaining metrics and discusses bootstrap limitations for AUPRC.

3. Paired significance tests. We performed two-sided paired Wilcoxon tests across tasks, with Holm correction separately for each metric (Table 9). Compared with tuned elastic net, ensembled GeneICL is significantly better on four of seven metrics: balanced accuracy (+0.031, adjusted p=0.0054p=0.0054), Pearson (+0.028, (\p<10^{-4})), Spearman (+0.028, p=0.0015p=0.0015), and R² (+0.022, p=0.0084p=0.0084). Differences in AUROC, AUPRC, and macro-F1 are not significant.

These additions explicitly quantify downstream variability and distinguish statistically supported improvements from numerical differences.

Q1: MSE vs. quantile regression

We thank the reviewer for this suggestion. We chose MSE because our evaluation focuses on point predictions, for which MSE directly targets the conditional mean without requiring 999 quantile outputs. Following the reviewer's suggestion, we trained a quantile-regression variant with TabICLv2's 999-quantile pinball loss (three seeds, Table 18).

Regression headPearsonSpearmanR²
Quantile (999 levels)0.5150.5290.264
MSE (ours)0.5190.5320.272

The quantile head performs slightly worse across all three metrics, significantly so for Pearson and R². We now explicitly document the MSE head as an architectural departure from TabICLv2 (Section 3.2).

Q4: Working with PC scores

Number of components. Retaining 90% of support-set variance yields a median of 134 PCs (range 20–499), corresponding to a median of 52% of the available ns−1n_s-1 components (Appendix Figure 4).

Comparison with 2,000 highly variable genes. We retrained GeneICL with support-selected HVGs instead of PCA (Table 18). PCA achieves AUROC 0.741 vs. 0.685, balanced accuracy 0.640 vs. 0.568, and R² 0.272 vs. 0.231, significantly outperforming HVGs on five of seven metrics while using far fewer feature tokens.

RoPE and feature grouping. We independently reintroduced both mechanisms (Table 18). Neither significantly improves performance; feature grouping additionally reduces Pearson and R² significantly. These findings support omitting both components in our PCA-based architecture, although they do not establish that representation collapse is absent.

Minor: Ensemble definition

We now define the ensemble in Table 1's caption and Appendix H.3: six inference configurations of the same checkpoint, each averaging three input encodings, weighted by inner cross-validated support-set performance.

Suggestions for future work

Distribution shift. We incorporated this suggestion into the revision by adding query-set attention, synthetic batch shifts during pretraining, and a leave-one-cohort-out benchmark covering three clinical classification tasks across ten cohorts and ten BeatAML regression tasks split by sequencing wave (Appendix A.1, Table 5).

ModelAUROCBal. Acc.PearsonR²
Elastic net0.6920.5810.4420.152
TabFM0.6860.5610.4320.165
TabPFN-30.6770.5320.4190.174
GeneICL (ensembled)0.7170.6650.4880.229

GeneICL achieves the highest mean performance across all reported metrics, providing encouraging evidence for cross-cohort generalization. Tissue and species shifts remain future work.

Perturbation tasks. We agree that perturbation prediction is important. Predicting high-dimensional expression responses rather than scalar outcomes requires a task reformulation or architectural extension, which we leave for future work.

Thank you!

We thank the reviewer again for the detailed feedback, which has helped us strengthen the evaluation and clarify the paper's contributions. We would be happy to address any remaining questions.

We thank the reviewer again for their feedback to our submission. As the discussion period closes in about 48 hours, we would be glad to address any remaining questions or provide additional analyses the reviewer would like to see. We would also appreciate the reviewer’s feedback on whether our response has addressed their concerns.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. The reviewers recognized GeneICL's parameter-efficient approach (kfQN, 7N2E), the clinical relevance of our benchmark (kfQN), and the availability of our code for reproducing the results (j6Li). The main concerns were statistical rigor, generalization across cohorts, and the contribution of individual design choices. We addressed them as follows:

  1. Updated model (Sections 3.2–3.4). Since submission, we made four changes to GeneICL. To enable cross-batch prediction, we (i) apply synthetic batch shifts to the query set in 20% of pretraining tasks, and (ii) add query-set attention, through which query samples can detect shared shifts. We also found that (iii) training with class-balanced cross-entropy and (iv) pretraining separate classification and regression checkpoints improve performance. All results were regenerated with this model.

  2. Statistical evaluation (j6Li, kfQN). We added paired Wilcoxon signed-rank tests across tasks with Holm correction, bootstrap confidence intervals for aggregate performance, and per-task fold variability (Appendix A.2, Tables 6–9). The hierarchical bootstrap additionally captures variation across benchmark tasks. Ensembled GeneICL significantly outperforms all five transcriptomic foundation models on all seven metrics, and tuned Elastic Net on balanced accuracy, Pearson, Spearman, and R².

  3. Cross-cohort prediction (7N2E, kfQN). We added a leave-one-cohort-out benchmark: three clinical tasks pooled from ten cohorts across microarray and RNA-seq platforms, and ten drug-response tasks split by sequencing wave (Appendix A.1, Tables 4–5). GeneICL achieves the best result on every metric, with larger margins than within cohorts (balanced accuracy +0.084 over Elastic Net).

  4. Ablations (7N2E, kfQN, j6Li). We trained additional variants covering pretraining priors, label-function classes, input representations, and TabICLv2 components (Appendix H.1, Table 18). Modeling gene–gene dependencies is important, and the VAE prior matches or exceeds real-data pretraining while enabling unlimited generation of synthetic profiles. Our ablations also support task-specific PCA and weight tying.

  5. Baselines, benchmark, and presentation (all reviewers). An Elastic Net trained on all 20,021 genes performs slightly worse than Elastic Net on principal components (Table 13). Including all eight previously filtered tasks preserves GeneICL's leading benchmark performance (Appendix A.3, Tables 10–12). We also revised the claims flagged by reviewers and report the number of principal components per task (Appendix Figure 4).

We believe these additions substantially strengthen the paper and thank the reviewers again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.