acceptodds
Under review as a conference paper at ICLR 2027

Exploring How LLM-Guided Scientific Discovery Can Find Solutions to Long-Standing Challenges in Disease Risk Prediction from Genetic Data

Abstract

Predicting a person’s risk of disease from their genome works best for people whose genetic ancestry is well represented in the data used to build the model, and worst for everyone else. Because data from underrepresented ancestry groups are scarce, genetic risk models in precision medicine routinely borrow data from other, better-represented groups. How much to borrow, and from whom, is decided by data-borrowing schemes that are designed by hand and apply one global rule to every genetic variant and every person. We ask whether LLM-guided scientific discovery can find a better scheme. An LLM writes candidate schemes as short Python functions that decide, for each person and each variant, how far through continuous genetic ancestry space to borrow labeled subjects. A fixed evaluator scores each candidate on real prostate cancer data, and the best-scoring scheme is locked before any validation data are touched. With East Asian ancestry as the underrepresented (focal) group and discovery performed on one cohort, the locked scheme achieved the highest held-out discrimination (AUC) in three independent validation settings drawn from three different data sources, compared with four established borrowing schemes (Mixture Learning, Independent Learning, and two transfer-learning schemes). Pooling all 112 held-out focal subjects, one-sided paired tests favored the discovered scheme over every comparator (p < 0:05 for all four). An ablation shows that weakening the evaluator during discovery degrades the held-out performance of the resulting scheme, indicating that the evaluator, not the LLM alone, drives what is found.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
precision medicine, LLM guided scientific discovery, equitable machine learning, healthcare & bioinformatics applications, precision oncology

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Bo1SOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper uses LLM-guided program search to select subject- and variant-specific neighborhoods in genetic ancestry space. Local SNP effects estimated from labeled reference subjects are used to construct weights for prediction. The problem is important, but the method proposed in this paper is not practical for polygenic risk prediction commonly used. In addition, LLM is very narrowly used, so it is questionable whether LLM-based scientific discovery is convincingly shown.

Critical strengths

• The paper consider the important problem of disease-risk prediction in underrepresented populations. • It explores an interesting approach to adaptive borrowing across genetic backgrounds.

Critical weaknesses
  • The contribution of LLM search is not established. Neighborhood selection could use cross-validated nearest neighbors, kernel bandwidth selection, or conventional optimization.
  • The method requires individual level genotype-phenotype data for test sample prediction, so it cannot be used for most of use cases in polygenic risk prediction. Proper use case should be demonstrated.
  • The authors used very small scale data for the demonstration. Scalability of the method to large dataset is not shown.
  • The stability of local SNP-effect estimation is unclear. Restricting estimation to local neighborhoods can increase uncertainty in already noisy genetic effects.
Questions for authors

• Does LLM search outperform conventional neighborhood-selection methods under the same evaluator and a comparable search budget? • What are the distributions of selected neighborhood sizes, and how stable are local effect estimates and final predictions under resampling? • How many SNPs are used in each cohort? Can the method scale to genome-wide prediction, and how does it handle LD as the variant set expands? • For admixed individuals, global-PC proximity may not capture locus-specific ancestry similarity. How might this limitation affect neighborhood selection and prediction?

3 comments
AuthorsOpenReview ↗

New Results and Experiments Addressing Reviewers’ Questions and Comments (1)

Does LLM search outperform conventional neighborhood-selection methods under the same evaluator and a comparable search budget?

The contribution of LLM search is not established. Neighborhood selection could use cross-validated nearest neighbors...

The LLM-discovered scheme beats non-LLM neighborhood search tuned on the same data, with the same evaluator and the same budget, on all three independent cohorts

What we did. We kept everything fixed and replaced only the LLM with a conventional grid search: the evaluator, the Discovery Cohort and its folds, the locking rule and the validation. Two neighborhood-borrowing rules were searched:

  • Tuned kNN (number of subjects): borrow the M nearest reference subjects in ancestry space. M is one global value.
  • Tuned novelty-adaptive kNN: borrow a fraction of the reference subjects. The fraction grows or shrinks with how sparse the reference data are around each subject; the scale and the strength of this adaptation are tuned.

Each rule used the same 56 evaluator calls as the LLM search and was locked before any validation data were used.

Results. AUC on held-out focal subjects, with a 90% bootstrap CI for OncoArray. The p-value is from the one-sided paired test over all 112 held-out focal subjects.

SchemeOncoArray (n=63)Held-out MEC (n=19)AoU (n=30)p-value, one-sided paired t-test: LLM scheme better than this baseline (all 112 held-out subjects)
Discovered by LLM0.808 [0.712, 0.890]0.6710.73—
Tuned kNN (number of subjects)0.661 [0.546, 0.776]0.4000.710.002
Tuned novelty-adaptive kNN0.785 [0.692, 0.876]0.4570.700.11

About the non-significant p = 0.11 against the novelty-adaptive baseline.

  1. The novelty-adaptive baseline was inspired by the scheme the LLM discovered for prostate cancer. We took its most visible ingredient (borrow more reference subjects when the target's ancestry support is sparse), dropped all of its per-variant logic, and tuned the result with a deterministic grid search. So part of this baseline's strength comes from the LLM discovery itself.
  2. The one-sided p = 0.112 over n = 112 subjects corresponds to t ≈ 1.22, a standardized paired effect of d ≈ 0.12. At that effect size, roughly 200 held-out subjects would be needed just to reach p < 0.05, so the test is underpowered rather than evidence that the two schemes perform the same.

(Revised PDF: Section 4.3, Figure 3, Appendix B.)

---------------------------------------------------------------

how stable are local effect estimates and final predictions under resampling?

New experiment: the advantage of the discovered scheme survives perturbation of the reference cohort in 10 out of 10 resamples

What we did. Every scheme borrows from the reference portion, so its local effect estimates and predictions depend on which subjects that portion contains. We drew 10 bootstrap resamples of the OncoArray reference portion (23,030 subjects, 453 focal). Subjects were sampled with replacement within each ancestry group and case/control status, so group sizes and case counts were kept. For each resample we reran the locked scheme and the four established schemes from scratch: neighborhoods, local effect estimates, features, the risk model and the predictions. The calibration and held-out portions were not changed, so every resample is scored on the same 63 held-out focal subjects.

Results. Held-out AUC over the 10 resamples:

SchemeMean AUC (SD)RangeHighest AUC (of 10)
Discovered by LLM0.803 (0.019)0.772–0.82710
Mixture Learning0.751 (0.013)0.735–0.7700
TL-PR0.746 (0.027)0.708–0.7800
TL-GDES0.744 (0.014)0.722–0.7650
Independent Learning0.677 (0.042)0.629–0.7640

The discovered scheme ranked first in every resample, and its lowest AUC (0.772) is above the mean of every established scheme. No established scheme was consistently second (TL-PR 7 times, Mixture Learning 2, TL-GDES 1).

(Revised PDF: new paragraph in Section 4.2, "The advantage survives perturbation of the reference portion"; Appendix E, Table 6.)

AuthorsOpenReview ↗

New Results and Experiments Addressing Reviewers’ Questions and Comments (2)

How many SNPs are used in each cohort? how does it handle LD as the variant set expands?

Number of variants. Each cohort went through quality control (QC) before the evaluator saw it: candidate variants, then missingness filters, then LD pruning.

CohortCandidate variantsVariants after QC (used)
MEC200110
OncoArray20073
AoU505

How LD is handled. LD pruning is a QC step before the evaluator. Within each chromosome, a variant is dropped if it is strongly correlated with a higher-ranked variant that is kept. The correlations are computed on all subjects of the cohort together, across ancestry groups, without labels. The discovered scheme then works on one variant at a time, so it runs unchanged on any variant set.

New experiment: larger variant sets. We reran the OncoArray QC with 2–5× more candidate variants and kept everything else fixed: the same subjects, the same splits, the same LD threshold and the same locked scheme. LD pruning kept 121–230 variants, and each larger set contains the original 73. AUC on the 63 held-out focal subjects, with 90% (both side) bootstrap CIs:

Candidate variantsVariants after QC (used)Discovered by LLMMixture LearningIndependent LearningTL-GDESTL-PR
20073 (original)0.808 [0.712, 0.890]0.738 [0.632, 0.837]0.725 [0.604, 0.841]0.747 [0.639, 0.849]0.760 [0.658, 0.853]
4001210.823 [0.729, 0.905]0.785 [0.683, 0.876]0.721 [0.595, 0.835]0.796 [0.697, 0.881]0.796 [0.696, 0.884]
6001520.778 [0.673, 0.875]0.763 [0.654, 0.864]0.681 [0.550, 0.802]0.775 [0.669, 0.868]0.772 [0.669, 0.870]
8001980.792 [0.689, 0.881]0.766 [0.661, 0.864]0.597 [0.462, 0.723]0.779 [0.677, 0.866]0.779 [0.683, 0.873]
10002300.802 [0.711, 0.888]0.763 [0.659, 0.860]0.610 [0.481, 0.737]0.762 [0.652, 0.852]0.781 [0.684, 0.873]

The discovered scheme stays first at every size and does not degrade as variants are added.

(Revised PDF: Appendix C, "Quality control and variant selection" and Table 4; new last paragraph of Section 4.2, "The advantage persists on larger variant sets"; Appendix D, Table 5.)

------------------------------

The authors used very small scale data for the demonstration. Scalability of the method to large dataset is not shown.

New: a second disease with a much larger validation cohort. We applied the full discovery process to lung cancer (focal group: Asian ancestry; see "Reviewer vof2: another disease / focal ancestry group"). The OncoArray lung cancer validation has 840 held-out focal subjects, more than 13× the 63 held-out focal subjects of the prostate OncoArray validation, and the locked scheme was also validated on All of Us. The discovered scheme ranks first on both validation cohorts, and all eight one-sided p-values are below 0.01, including when it must beat each baseline by at least δ = 0.01 AUC.

---------------------------------------

Can the method scale to genome-wide prediction

Our current evaluation focuses on panels of established prostate cancer risk variants, an established and clinically relevant prediction setting. Such panels have been evaluated prospectively for risk-guided screening (McHugh et al., 2025; DOI: 10.1056/NEJMoa2407934), and comparative studies found that the genome-wide methods evaluated did not outperform established-variant scores in their prostate cancer datasets (Yu et al., 2020; DOI: 10.1002/pros.23920; Darst et al., 2023; DOI: 10.1016/j.ajhg.2023.05.010). Genome-wide scaling is therefore not required for the prediction setting studied here, and we did not evaluate it.

AuthorsOpenReview ↗

Addressing Additional Comments

For admixed individuals, global-PC proximity may not capture locus-specific ancestry similarity. How might this limitation affect neighborhood selection and prediction?

Global PCs are an established representation of the genetic ancestry continuum (Ding et al., 2023, Nature; DOI: 10.1038/s41586-023-06079-4), and our scheme borrows along that continuum. We agree that for recently admixed individuals, subjects close in global-PC space may be less similar at particular loci, so the borrowed reference data there may be less informative and the benefit of borrowing smaller. The borrowing radius already varies per subject and per variant, so the formulation is a natural basis for a local-ancestry-aware distance (e.g., using inferred local ancestry at each variant). This requires extending the neighborhood definition and a separate validation, which is outside the scope of this revision.

(Revised PDF: new paragraph in Section 6, Limitations.)

-------------------------

The method requires individual-level genotype-phenotype data for test sample prediction, so it cannot be used for most use cases in polygenic risk prediction. A proper use case should be demonstrated.

Individual-level genotype–phenotype data are standard inputs for supervised genetic prediction, including ancestry-aware and transfer-learning approaches evaluated in large biobanks (Chen et al., 2026; DOI: 10.1038/s41592-026-03235-2; Wu et al., 2026; DOI: 10.1038/s41467-026-68696-7). Our method estimates predictive relationships from labeled reference individuals and uses the target individual's genotype and ancestry coordinates as prediction inputs. The target's phenotype is never used to construct its prediction. The intended use case is prediction within a biobank or authorized research environment maintaining a labeled reference cohort, as represented by our MEC, OncoArray, and All of Us experiments.

Reviewer_vof2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses an LLM to search for rules that decide whose genetic data to borrow for each person and genetic variant when predicting prostate cancer risk. The selected rule is fixed before validation and achieves the highest or tied-highest AUC against four baselines across three datasets, with East Asian ancestry as the focal group.

Critical strengths
  • Flexible data borrowing. Choosing which people’s data to use separately for each person and genetic variant is a useful alternative to applying one fixed rule to everyone. The downstream prediction model remains unchanged, making comparisons easier to interpret.
  • Separate discovery and validation. The rule is fixed before testing on held-out participants and two external datasets. This provides a stronger test than evaluation on the discovery data alone.
Critical weaknesses
  • Very small test sets. The three held-out sets contain only 19, 63, and 30 participants. Figure 2 reports AUC values without confidence intervals, making it difficult to judge whether the gains are reliable or sensitive to a few participants. The pooled statistical tests assess correct-label probability, not AUC differences.
  • Missing comparisons to simpler alternatives. The baselines cover pooling, focal-only learning, and transfer learning, but do not establish the added value of LLM-guided search. Comparisons against a tuned nearest-neighbor or radius-based borrowing rule, and a non-LLM search using the same evaluation budget, would help determine whether the gains require the proposed approach.
  • Limited disease and ancestry coverage. The experiments cover only prostate cancer and one focal ancestry group, East Asian. Testing across three datasets is helpful, but does not establish effectiveness for other diseases or underrepresented groups. The results support a proof of concept rather than broad generalization.
Questions for authors
  • Does the discovered rule outperform simple neighborhood-based borrowing and non-LLM search when both are tuned using the same discovery data and a comparable evaluation budget?
  • What are the confidence intervals for the AUC differences against the strongest baseline in each dataset?
  • Are any results available for another disease or another focal ancestry group that support broader applicability?
2 comments
AuthorsOpenReview ↗

New Results and Experiments Addressing Reviewers’ Questions and Comments

Are any results available for another disease or another focal ancestry group that support broader applicability?

The approach generalizes to a second disease (lung cancer, focal ancestry: Asian ancestry)

What we did. We applied the full discovery process for lung cancer, with Asian ancestry as the focal group. The Discovery Cohort came from the OncoArray lung cancer study, and the search was seeded with the scheme found for prostate cancer. The locked scheme was validated on two independent cohorts: held-out focal subjects from OncoArray lung cancer (n = 840) and from AoU (n ≤ 25).

Results. Held-out AUC (90% bootstrap CI for OncoArray). The p-values come from the one-sided paired t-test over the held-out focal subjects of both cohorts, of whether the discovered scheme beats each baseline by more than a margin δ (effect size).

SchemeOncoArray AUC (n=840)AoU AUC (n≤25)p (δ=0)p (δ=0.01)
Discovered by LLM0.628 [0.595, 0.660]0.53——
Independent Learning0.548 [0.513, 0.581]0.422.9e-76.4e-4
Mixture Learning0.617 [0.585, 0.650]0.493.1e-71.2e-3
TL-GDES0.529 [0.497, 0.561]0.488.3e-57.1e-3
TL-PR0.519 [0.487, 0.553]0.488.8e-56.1e-3

The discovered scheme ranks first on both cohorts and beats every baseline at both margins.

(Revised PDF: Section 4.4, Table 1.)

--------------------------------------------------

Does the discovered rule outperform simple neighborhood-based borrowing and non-LLM search when both are tuned using the same discovery data and a comparable evaluation budget?

The LLM-discovered scheme beats non-LLM neighborhood search tuned on the same data, with the same evaluator and the same budget, on all three independent cohorts

What we did. We kept everything fixed and replaced only the LLM with a conventional grid search: the evaluator, the Discovery Cohort and its folds, the locking rule and the validation. Two neighborhood-borrowing rules were searched:

  • Tuned kNN (number of subjects): borrow the M nearest reference subjects in ancestry space. M is one global value.
  • Tuned novelty-adaptive kNN: borrow a fraction of the reference subjects. The fraction grows or shrinks with how sparse the reference data are around each subject; the scale and the strength of this adaptation are tuned.

Each rule used the same 56 evaluator calls as the LLM search and was locked before any validation data were used.

Results. AUC on held-out focal subjects, with a 90% bootstrap CI for OncoArray. The p-value is from the one-sided paired test over all 112 held-out focal subjects.

SchemeOncoArray (n=63)Held-out MEC (n=19)AoU (n=30)p-value, one-sided paired t-test: LLM scheme better than this baseline (all 112 held-out subjects)
Discovered by LLM0.808 [0.712, 0.890]0.6710.73—
Tuned kNN (number of subjects)0.661 [0.546, 0.776]0.4000.710.002
Tuned novelty-adaptive kNN0.785 [0.692, 0.876]0.4570.700.11

About the non-significant p = 0.11 against the novelty-adaptive baseline.

  1. The novelty-adaptive baseline was inspired by the scheme the LLM discovered for prostate cancer. We took its most visible ingredient (borrow more reference subjects when the target's ancestry support is sparse), dropped all of its per-variant logic, and tuned the result with a deterministic grid search. So part of this baseline's strength comes from the LLM discovery itself.
  2. The one-sided p = 0.112 over n = 112 subjects corresponds to t ≈ 1.22, a standardized paired effect of d ≈ 0.12. At that effect size, roughly 200 held-out subjects would be needed just to reach p < 0.05, so the test is underpowered rather than evidence that the two schemes perform the same.

(Revised PDF: Section 4.3, Figure 3, Appendix B.)

AuthorsOpenReview ↗

New Results and Experiments Addressing Reviewers’ Questions and Comments .... Continued

What are the confidence intervals for the AUC differences against the strongest baseline in each dataset?

We expect AUC to vary a lot between samples. Genetic markers explain only part of disease risk; the rest comes from factors the variants do not capture, such as environment, socioeconomic status, access to healthcare, and differences in heritability between subpopulations. These factors move every scheme's cohort-level AUC up or down together. So we do the subject-paired t-test (Eq. 9; Fig. 2d). It compares schemes within each individual, so this shared variation cancels out. It shows that even when an individual's genetic risk signal is weak, the order in which the schemes predict that individual stays stable. Across all 112 held-out subjects, the discovered scheme beats every baseline (p = 0.0006, 0.0001, 0.045 and 0.049 against Independent Learning, Mixture Learning, TL-GDES and TL-PR).

What we added. CIs for the AUC difference over all three cohorts (new Appendix F). To use all 112 held-out subjects, Appendix F bootstraps the held-out subjects within each of the three cohorts, scores every scheme on the same resamples, and reports the cohort-weighted AUC difference with a 90% CI. The discovered scheme's AUC is strictly higher than the comparator's in 92.3% of bootstrap replicates against Independent Learning, 90.6% against Mixture Learning, 89.8% against TL-GDES and 90.0% against TL-PR. The dAUC values and their 90% CIs are in Appendix F, Table 7.

Figure 2a now shows 90% bootstrap confidence intervals (CIs) on OncoArray, the largest held-out cohort (63 focal subjects, 2,000 bootstrap resamples): Held-out MEC (19 subjects) and AoU (30 subjects) are too small for useful bootstrap CIs, so we report only point estimates there.

(Revised PDF: Figure 2a, its caption, the new paragraph in Section 4.2 on overlapping confidence intervals, and Appendix F, Table 7.)

Reviewer_cYL1OpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether LLM-guided scientific discovery can identify improved data-borrowing strategies for genetic disease-risk prediction in underrepresented ancestry populations. Existing approaches typically rely on fixed pooling, ancestry-specific modeling, or transfer-learning rules that apply globally across individuals and genetic variants. The authors instead formulate data borrowing as an executable Python function that determines how broadly to borrow labeled genetic information for each target individual and each genetic variant using continuous ancestry coordinates. An LLM generates candidate functions, while a fixed evaluator assesses their performance on prostate cancer data. The best scheme is locked before independent validation. The study focuses on East Asian ancestry and conducts discovery using the Multiethnic Cohort (MEC), followed by evaluation on held-out MEC, OncoArray, and All of Us datasets. The discovered scheme achieves AUCs of 0.671, 0.808, and 0.733, respectively, outperforming or tying four established borrowing strategies. Subject-paired analyses across 112 held-out focal-ancestry participants favor the discovered method over the comparators.

Critical strengths
  • The paper addresses ancestry-related performance disparities in genetic risk prediction and converts a methodological design problem into an automatically evaluated program-search task. This is a potentially valuable direction for scientific AI.
  • Unlike studies that repeatedly optimize on evaluation data, this work reports locking the discovered borrowing function before testing it on held-out subjects from MEC, OncoArray, and All of Us. The use of different sources and variant sets provides useful, although statistically limited, evidence of transportability.
Critical weaknesses
  • The evaluation uses 19 focal participants from held-out MEC, 63 from OncoArray, and 30 from All of Us. These sample sizes limit precision, especially for comparing AUC differences across methods.
  • The two transfer-learning comparisons have reported one-sided p-values of approximately 0.045 and 0.049. These are marginal and would not remain significant under a straightforward four-comparison Bonferroni threshold. Moreover, the reported subject-paired tests measure differences in correct-label probabilities, not directly differences in AUC; they therefore do not independently establish statistically significant AUC superiority.
  • The study does not convincingly isolate the contribution of LLM-guided discovery relative to classical optimization of the same candidate-function space. Calibration, uncertainty, clinical utility, and generalization to other ancestry groups and diseases also remain insufficiently established.
Questions for authors

The most important technical limitation is that the discovered data-borrowing rule is optimized for a particular evaluator, disease, and focal ancestry group, making its broader applicability uncertain. Although the authors use continuous ancestry coordinates, genetic ancestry alone does not capture environmental, socioeconomic, clinical, and healthcare-access factors affecting prostate cancer risk. The search may also exploit idiosyncrasies of the discovery cohort or evaluator, even when the final function is locked before validation. Experimentally, the small held-out datasets create substantial uncertainty in the reported AUC advantages, and the nominally significant subject-paired tests require careful interpretation because of multiple comparisons and their different statistical target. Further, the methodology is evaluated against four established borrowing schemes rather than stronger adaptively parameterized or automated-search alternatives. The study should report confidence intervals for AUC differences, direct paired AUC tests, calibration metrics, subgroup-specific risk errors, sensitivity to ancestry principal components, and the reproducibility of the discovery process across multiple independent LLM search runs. An additional limitation is the absence of evidence that improved ranking translates into better screening decisions or patient outcomes.

Here are the key questions to be addressed. Q1: What are the bootstrap confidence intervals for the AUC differences against each comparator on each validation dataset and on pooled predictions? Q2: Do the reported improvements remain statistically significant under two-sided hypothesis tests and appropriate correction for four pairwise comparisons? Q3: Why are paired tests of correct-label probability used to support AUC comparisons? Can the authors provide direct paired AUC analyses? Q$: Does LLM-guided discovery outperform Bayesian optimization, evolutionary search, random search, symbolic regression, or differentiable tuning of the same borrowing-function family under equal evaluation budgets? Q5: What happens when the discovered function is applied to African, Hispanic/Latino, and other underrepresented ancestry populations, or to diseases other than prostate cancer? Q6: How stable is the discovered rule across independent LLM runs, search seeds, discovery-cohort resampling, and different evaluator configurations? Q7: Does the method improve absolute-risk calibration, subgroup performance, and worst-group error, or only discrimination for the focal ancestry population? Q8: How were overlapping subjects, related individuals, variant-selection leakage, cohort-specific ascertainment effects, and genomic data-use restrictions addressed?

2 comments
AuthorsOpenReview ↗

Clarifications and New Data Addressing Reviewers’ Comments, Questions, and Concerns

Do the reported improvements remain statistically significant under two-sided hypothesis tests and appropriate correction for four pairwise comparisons?

Why one-sided. A two-sided test asks whether the discovered scheme's accuracy is different from that of an established scheme, in either direction. Our claim is directional: the discovered scheme has better predictive power than the baselines. The one-sided test (H1: the discovered scheme is better) tests exactly this claim, so it is the appropriate test here.

-----------------------

The two transfer-learning comparisons have reported one-sided p-values of approximately 0.045 and 0.049. These are marginal and would not remain significant under a straightforward four-comparison Bonferroni threshold.

Our claim does not need a multiplicity correction. Whether a correction is needed depends on how the claim is built:

  • "Better than at least one baseline" needs a correction: any single win is enough, so with more tests a winner is more likely to be noise (e.g., screening many drug candidates against a standard).
  • "Better than every baseline" does not: it holds only if all four tests reject, each at level α, so the Type I error stays at or below α (Berger, 1982, DOI: 10.2307/1267823; FDA, Multiple Endpoints in Clinical Trials, 2022).

Our claim is the second kind: one locked scheme is better than each of the four established schemes. All four one-sided p-values are below 0.05, so the joint claim holds at α = 0.05.

Sample size. We agree that the p-values against the two transfer-learning baselines are borderline. The prostate cancer validation has only 112 held-out focal subjects across the three cohorts. We reported the raw p-values for transparency.

Lung cancer: highly significant with a larger validation cohort. For lung cancer, the validation cohort is larger (OncoArray n = 840, AoU n ≤ 25; see "Reviewer vof2: another disease / focal ancestry group"). All eight one-sided p-values are significant at p < 0.01 (**) or lower, including when the test requires the discovered scheme to beat each baseline by at least δ = 0.01 AUC:

Baselinep (δ=0)p (δ=0.01)
Independent Learning2.9e-7 ***6.4e-4 ***
Mixture Learning3.1e-7 ***1.2e-3 **
TL-GDES8.3e-5 ***7.1e-3 **
TL-PR8.8e-5 ***6.1e-3 **

* p < 0.05, ** p < 0.01, *** p < 0.001.

(Revised PDF: Section 4.4, Table 1.)

--------------------------

Why are paired tests of correct-label probability used to support AUC comparisons? Can the authors provide direct paired AUC analyses?

What are the bootstrap confidence intervals for the AUC differences against each comparator on each validation dataset and on pooled predictions?

Why a subject-paired test. We agree that a paired comparison is the right tool. Genetic risk signals vary a lot between individuals, because genetic markers explain only part of disease risk. The rest comes from factors the variants do not capture, such as environment, socioeconomic status, access to healthcare and differences in heritability between subpopulations (as the reviewer also notes). These factors move every scheme's cohort-level AUC up or down together. The subject-paired t-test (Eq. 9; Fig. 2d) compares schemes within each individual, so this shared variation cancels out. It shows that even when an individual's genetic risk signal is weak, the order of the schemes for that individual stays stable.

Why correct-label probability rather than AUC. AUC is a cohort-level measure: it counts how often cases are ranked above controls, so it cannot be computed for one individual and compared across schemes subject by subject. The correct-label probability depends only on that subject's own prediction, so the per-subject differences are independent and a paired t-test is valid. Paired AUC tests do exist, e.g. the DeLong test (DeLong et al., 1988, Biometrics; DOI: 10.2307/2531595), it adjusts for bootstrap sample correlation, this is unreliable for our smallest cohort.

Direct paired AUC analysis (new). We now also compare the schemes directly on AUC. We draw 10,000 bootstrap resamples of the held-out subjects, score every scheme on the same resample, and compute dAUC = AUC(LLM) − AUC(comparator), weighted across the three validation cohorts (details in Appendix F).

The discovered scheme's AUC is strictly higher than the comparator's in 92.3% of bootstrap replicates against Independent Learning, 90.6% against Mixture Learning, 89.8% against TL-GDES and 90.0% against TL-PR. The full table (cohort-weighted AUCs, dAUC and 90% CIs) is in Appendix F, Table 7 of the revised PDF.

(Revised PDF: new Appendix F, Table 7.)

AuthorsOpenReview ↗

Flag For Ethics Review

Flag For Ethics Review: Yes

This paper raises important ethics questions because it uses sensitive genomic information and explicitly models ancestry-dependent disease risk. The research objective, improving predictions for an underrepresented group, is socially valuable, but the methodology requires careful consideration of genetic privacy, ancestry-based discrimination, and possible misuse of genetic risk estimates. Important questions include whether the source datasets authorize this analysis, whether all data-use and consent requirements are satisfied, whether individual genomic information could be reidentified, and whether the discovered rule introduces systematic performance disadvantages for groups outside its target population. The manuscript should distinguish continuous genetic ancestry from socially defined racial identity and avoid treating ancestry as a comprehensive representation of biological or clinical risk. It should also discuss legal and governance considerations relevant to genetic information and clinical decision support.

We thank the reviewer for these questions. We have expanded the Ethics Statement to clarify the information shown to the LLM, ancestry interpretation, and the limits of evaluation across ancestry groups. The statement also describes the approved controlled-access mechanisms and permitted environments used for the analyse

Authorization, consent, and data use. MEC and OncoArray data used in the prostate- and lung-cancer analyses were obtained through approved dbGaP data access requests. These analyses fall within the approved research purposes and the data-use limitations of the consent groups accessed. All of Us data were analyzed within the Researcher Workbench Controlled Tier under the Data User Code of Conduct. Where required, relevant IRB documentation was submitted and reviewed as part of the controlled-access approval process. The study analyzes existing coded data and involves no contact with participants. All analyses were performed within the permitted environments.

Privacy and re-identification. The discovery process is designed so that NO genetic data is ever passed to the LLM: neither individual-level genotypes nor population-level statistics of genetic data. Likewise, NO subject identifiers, subject ancestries, individual-level phenotype data, ancestry coordiantes or population-level summaries of any of these are ever passed to the LLM. This manuscript does NOT disclose subject identifiers or any individual-level or population-level genomic data. See Ethics statement in the paper for further information.
We do NOT train or fine-tune a generative AI model. See the the Ethics Statement in main paper.

Performance in other ancestry groups. The choice of focal group for prostate cancer was guided by data availability: the selected group was sparsely represented in the processed Discovery Cohort’s reference data, while held-out subjects were available across MEC, OncoArray, and All of Us. For lung cancer, Asian ancestry was used as the focal group, with discovery in OncoArray and held-out validation in OncoArray and All of Us. These choices do not reflect a judgment about which groups most need improved prediction. Performance outside the evaluated focal groups remains unknown. The framework can be applied to subjects in the focal group while retaining existing prediction models for other groups, but applying a discovered scheme to another group requires separate validation. The revised Ethics Statement explicitly acknowledges this limitation.

Genetic ancestry and race. Neighborhood construction uses continuous genetic ancestry coordinates rather than treating discrete group labels as complete descriptions of genetic similarity. Source-specific group labels are used to define the focal cohorts and established comparison schemes. Genetic ancestry is distinct from socially defined race and ethnicity. As stated in the Limitations section, ancestry coordinates do not capture all biological, environmental, social, or clinical determinants of disease risk.

Misuse, governance, and clinical use. This study evaluates a research method. Clinical implementation would require additional external and prospective validation, assessment of calibration and clinical utility across intended populations, and any applicable regulatory and governance review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for the discussion and comments. We hope we have addressed your concerns!

We thank all the reviewers for reading and evaluating our paper, and for sharing their views on the merits and shortcomings of our results. Guided by this feedback, we ran several new experiments and share the results below. We have also substantially expanded the data and ethics sections. We hope the reviewers will take them into account in their re-evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.