- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The author proposed a 4.24M looped transformer model called GeneICL. GeneICL uses VAE, PCA and a table transformer for pre-trained in-context learning. The authors evaluated GeneICL on classification, regression and survival outcome prediction. GeneICL outperformed other generic and transcriptomics foundation models. Ablation studies examined model architecture, pretraining distribution and inference-time ensembling.
Critical strengths
- The authors proposed an interesting training strategy combining synthetic task generation and in-context learning on modified TabICLv2.
- GeneICL achieves better performance with fewer parameters and an efficient architecture, compared with other gene foundation models.
Critical weaknesses
- The main novelty is the gene expression prior (VAE+PCA), but the ablation study is not sufficient. It is unclear why the learned VAE latent representations cannot be used directly. Comparisons with simpler generative priors and alternative input representations are missing.
- The authors leave cross-batch and platform as limitations. However, experiments within a single cohort are not useful in a real scenario.
- In Appendix A, the authors deleted targets with low TabPFN-3 score, which may introduce selection bias.
Questions for authors
-
Why did the author use task-specific PCA rather than the learned VAE latent representations? An ablation study is needed. How does this choice affect the model's ability to transfer biological information across tasks?
-
How does GeneICL performance change in terms of the number and diversity of synthetic pre-training tasks?
-
Since there are relatively small improvements over elastic-net, what task can GeneICL achieve a significant improvement on? An experiment on cross-batch prediction can be helpful.
-
How would the results change if no tasks are excluded by TabPFN-3?
5 comments
2/2
W2 & Q3: Cross-batch generalization and improvements over elastic net
We agree that cross-batch generalization is essential for clinical applications. To address this limitation, we extended GeneICL's architecture and pretraining prior to explicitly account for batch effects, introducing synthetic batch shifts and query-context attention.
Improvements over elastic net. We performed paired Wilcoxon tests across the 52-task main benchmark (Appendix A.2, Table 9). Compared with tuned elastic net, ensembled GeneICL significantly improves four of seven metrics after Holm correction: balanced accuracy (+0.031, ), Pearson (+0.028, ), Spearman (+0.028, ), and R² (+0.022, ). Differences in AUROC, AUPRC, and macro-F1 are not statistically significant.
Cross-batch evaluation. We evaluate this extension on three clinical prediction tasks spanning ten clinical cohorts, including different profiling platforms, and ten BeatAML drug-response tasks across four sequencing batches. Each cohort or batch is held out in turn, with the remaining batches forming the support set (Appendix A.1).
| Model | AUROC ↑ | Bal. Acc. ↑ | Macro-F1 ↑ |
|---|---|---|---|
| Elastic net | 0.692 | 0.581 | 0.540 |
| TabPFN-3 | 0.677 | 0.532 | 0.470 |
| TabFM | 0.686 | 0.561 | 0.498 |
| GeneICL (query attention off) | 0.714 | 0.604 | 0.567 |
| GeneICL (query attention on) | 0.717 | 0.665 | 0.615 |
GeneICL achieves the best performance across all reported classification metrics, with particularly substantial improvements over elastic net in balanced accuracy (+0.084) and macro-F1 (+0.075). Exploratory paired Wilcoxon tests across the ten held-out cohorts yield uncorrected -values of 0.004 and 0.020, respectively.
Disabling query-context attention reduces balanced accuracy from 0.665 to 0.604 and macro-F1 from 0.615 to 0.567, demonstrating its contribution under distribution shift. GeneICL also achieves the best cross-batch regression performance (Pearson 0.488, Spearman 0.483, R² 0.229).
These results suggest that GeneICL's advantages become more pronounced under distribution shift, particularly for classification metrics sensitive to cohort-dependent decision thresholds. Full results are provided in Table 5.
W3 & Q4: Potential selection bias from TabPFN-3 filtering
We thank the reviewer for highlighting this concern. Our original motivation for excluding tasks with low TabPFN-3 performance was to focus on targets exhibiting sufficient predictive signal. However, we acknowledge that model-based filtering can introduce selection bias.
We therefore re-evaluated the benchmark including all eight previously excluded tasks (seven classification and one regression), expanding it to 34 classification and 26 regression tasks.
| Model | AUROC ↑ | Bal. Acc. ↑ | Pearson ↑ | R² ↑ |
|---|---|---|---|---|
| Elastic net | 0.687 | 0.591 | 0.483 | 0.247 |
| TabPFN-3 | 0.674 | 0.555 | 0.470 | 0.236 |
| TabFM | 0.669 | 0.554 | 0.490 | 0.256 |
| GeneICL (ensembled) | 0.691 | 0.616 | 0.513 | 0.272 |
GeneICL retains its leading performance on the expanded benchmark. On the excluded tasks alone, tabular foundation models and conventional baselines achieve mean AUROC values between approximately 0.45 and 0.52, while every evaluated method obtains a negative R² on the regression task.
Although model-based filtering can introduce selection bias, the expanded benchmark preserves GeneICL's leading overall performance, suggesting that our main conclusions are robust to including these eight previously excluded tasks. Full results are provided in Appendix A.3, Tables 10–12.
Thank you!
We thank the reviewer again for their constructive suggestions. We believe the additional ablations, cross-batch experiments, and unfiltered benchmark evaluation substantially address the concerns raised and strengthen the empirical evidence for GeneICL. We would be happy to address any remaining questions during the discussion period.
Thank you for these substantial experiments. I appreciate the effort to address my concerns. However, the revised model introduces a lot of changes relative to the original submission, and these changes raise my additional questions.
In partiquery, with query-set attention, the prediction for a given patient can depend on the other supporting set patients. How sensitive are predictions to its size and composition?
I also noticed that there are now two separate classification and regression checkpoints, which should be noted in Figure 1. Besides, the strongest results use inference-time ensembling, where the reported CPU time of 7.5s was measured without ensembling. Could you also report the runtime of ensemble configuration?
1/2
We thank the reviewer for their constructive feedback and for recognizing the potential of combining transcriptomics-specific pretraining with parameter-efficient in-context learning. We agree that further ablations and evaluations are important to establish GeneICL's contributions and practical applicability.
In response, we have substantially extended our experiments, including additional pretraining and representation ablations, cross-batch evaluations, and an analysis of potential benchmark selection bias. We summarize the main findings below.
W1 & Q1: Expression prior and input representation
We agree that the contribution of the VAE-based expression prior and task-specific PCA deserves further investigation. We therefore conducted additional ablations (Appendix H.1, Table 18).
Expression prior. We compare our VAE-based prior against independent Gaussian and low-rank multivariate Gaussian distributions fitted to ARCHS4, as well as the generic TabICLv2 prior and real ARCHS4 profiles.
The independent Gaussian performs substantially worse, indicating that modeling marginal gene distributions alone is insufficient. While the multivariate Gaussian recovers much of this performance, the VAE prior achieves significantly higher Pearson and Spearman correlations (Holm-corrected paired Wilcoxon tests, p<0.05). This suggests that the VAE captures useful structure beyond second-order correlations.
Input representation. We use support-fitted PCA to obtain a compact representation adapted to each downstream dataset, without requiring a fixed pretrained feature extractor. We additionally compare against VAE latent representations and the 2,000 most highly variable genes.
Task-specific PCA substantially outperforms both alternatives. We hypothesize that, unlike fixed VAE representations optimized for reconstruction, support-fitted PCA better preserves the variation relevant to each downstream cohort.
Together, these experiments provide empirical support for both our transcriptomics-grounded pretraining prior and task-adaptive input representation.
Q2: Number and diversity of synthetic pretraining tasks
GeneICL is pretrained on 60,000 independently generated tasks, with diversity arising from randomized feature subsets, nonlinear label functions, class structures, signal-to-noise ratios, and batch perturbations.
To investigate the importance of label-function diversity, we trained additional models with restricted function priors (Appendix H.1, Table 18).
Restricting the prior to linear functions significantly reduces regression performance, whereas two-layer MLPs recover performance close to the full prior. This suggests that nonlinear task structure is important, while additional function diversity yields limited gains on the present benchmark.
We have clarified the task-generation process in Appendix F.2.