acceptodds
Under review as a conference paper at ICLR 2027

PhenoBench: Mapping What a Deeply Phenotyped Human Cohort Can Tell Us

Abstract

Deeply phenotyped cohorts combine clinical, imaging, molecular, and wearable observations across timescales from seconds to years. This breadth can reveal which measurements inform which health-related questions, but heterogeneous analyses are not directly comparable. We present PhenoBench, an executable benchmark built around the Human Phenotype Project, in which more than 13,000 participants have completed the initial visit. Each question fixes the target, eligible population, timing, and allowed information; its evaluation contract specifies the split, metric, baseline, and claim boundary. The benchmark defines 90 clinically grounded tasks across 15 domains and 26 input modalities. Measurements showed question- and representation-dependent predictive value, including positive, near-zero, and negative changes in held-out performance relative to matched baselines. We used PhenoBench to evaluate emerging tabular foundation models across 160 matched regression comparisons spanning 52 tasks. These models ranked above standard task-specific models in aggregate but, averaged across the three pretrained models within each cell, improved on ridge by a median of only 0.004 R² (95% CI, 0.002–0.006). We then used the same cohort data and evaluation contracts to evaluate 14 language models, collectively covering 40 tasks spanning phenotype recovery, classification, follow-up forecasting, and participant ordering. Without cohort-specific fitting, language models made informative predictions on some tasks, but showed task-specific capability gaps, shared failures of scale, and rarely surpassed models fitted on the same fields. PhenoBench turns a multimodal longitudinal cohort into a versioned, auditable evaluation system where new questions, measurements, and models can be added without redefining existing comparisons.

Primary area:
datasets and benchmarks
Keywords:
benchmark, health foundation models, clinical evaluation, tabular foundation models, LLM evaluation, deeply phenotyped cohort

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_c5BXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work introduces PhenoBench, a broad benchmark covering 90 different clinical tasks across 15 domains with 26 input modalities, built around the Human Phenotype Project. Each PhenoBench task specifies a prediction target, eligible population, timing, evaluation metrics, and relevant baselines through a human-readable Task Card and an executable evaluation contract. The tasks support mapping measurements to questions in two ways, either fixing the question or the measurement. The experiments compare ten tabular methods across 160 matched regression settings covering 52 tasks, examine frozen time-series representations against engineered features, and evaluate 14 LLMs on up to 40 tasks. The authors report modest improvements from tabular foundation models over ridge regression and generally stronger performance from task-trained traditional models than from LLMs using zero-shot direct inference.

Critical strengths
  1. The benchmark encapsulates diverse cross-domain tasks, with a span of 26 input modalities linked within the same cohort. It could serve as a useful resource, enabling comparisons of information and evaluations of predictions over potentially shared populations.
  2. The evaluation coverage is broad. The experiments include a wide range of linear, tree-based, neural tabular, time-series foundation-model embeddings, and LLM methods.
Critical weaknesses
  1. The manuscript lacks a dedicated “Related Work” section, which makes its novelty and level of contribution difficult to assess. I would like to see a focused comparison clarifying what is new in the task collection, method, evaluation, and resulting scientific insights relative to existing clinical and tabular benchmarks.
  2. The manuscript reports 90 registered tasks, but it does not provide a complete list or supplementary material specifying each task’s target construction, eligibility, timing, sample counts, and available evaluations. It is also unclear why Section 2.3 only evaluates 52 tasks, while the LLM suite covers 40.
  3. Beyond the above concerns, I am uncertain what substantial new research capability or scientific insight PhenoBench provides to the field. Its central promise is to map which measurements can inform which questions, but the explicit matched comparison across distinct measurement modalities in Figure 2B only uses two targets and four modalities. The broader measurement-first examples use different cohorts and outcomes, so they do not provide an equivalent controlled comparison. Additionally, another potential contribution of the benchmark is a consolidation of HPP measurements into a reusable task suite, but the manuscript does not demonstrate clearly which questions become newly tractable or easier to study through this resource. The empirical results currently offer limited insight into the conditions under which different measurements, representations, or model families succeed.

Minor:

  1. The references, page headers, and citation formats do not seem to adhere to the ICLR 2027 style.
  2. The figures in the manuscript can be better presented. Limited examples include: Figure 1 (Some texts overflow from their textboxes, or overlap with shapes and arrows. The icons in C4 appear malformed), Figure 4 (Text overlaps with diagram), and Supplementary Figure 3 & 4 (Lots of textual and graphical overlap. Some plots are also too dense to read).
Questions for authors
  1. What is the principal research capability that PhenoBench adds beyond existing multimodal biomedical studies and clinical prediction benchmarks?
  2. Can you provide a description of all 90 registered tasks, with their target definitions, cohort sizes, and evaluation coverage?
  3. Beyond the two targets in Figure 2B, does the current benchmark contain additional matched comparisons across distinct measurement modalities?
2 comments
AuthorsOpenReview ↗

Thanks. The main contribution is making the cohort reusable for comparing measurements and models on clearly defined clinical questions. We’ve strengthened the related-work comparison, added the full task inventory and supporting materials, and repaired figure labels and formatting. We also now state explicitly that the fully matched measurement comparison covers two targets.

Thank you for separating broad evaluation coverage from the question of what reusable research capability the resource adds.

Related work and contribution: The named Related Work subsection groups the PhysioNet, EHRSHOT, MedAgentBench and MedHELM comparisons and adds a concrete UK Biobank survival-benchmarking reference. The contribution is a reusable cohort-based task collection with versioned target, eligibility, timing, allowed information, comparator and metric, supporting linked comparisons of measurements and methods.

Task descriptions and coverage: Appendix G indexes the 90 registered clinical targets (and other details, please see other responses).

Principal research capability: PhenoBench allows a researcher to change the measurement route for a fixed question or change the model for fixed information while retaining comparison conditions and provenance. The existing paper demonstrates this through the tabular grid, representation comparisons and two-target measurement comparison. It reduces the need to reconstruct target, eligibility, timing, inputs, split and comparator independently for every model. This is a reusable evaluation capability rather than evidence of a new clinical mechanism.

Thanks for your response. I read through the other reviews and responses, as well as the revised manuscript. However, my concerns in W3 remain unresolved, and many figures still have overlapping texts and shapes. I will keep my current rating.

Reviewer_qfemOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The manuscript presents PhenoBench, a versioned set of 90 clinician-reviewed prediction tasks built on the Human Phenotype Project. It uses the benchmark to compare measurement routes, pretrained tabular models against tuned classical baselines, zero-shot Chronos-2 embeddings against engineered features, and 14 language models given participant-level evidence packets.

Critical strengths

The authors also report null and negative results, which many benchmark papers omit.

Critical weaknesses
  1. Figure 1 appears to be generated using AI. Many figures have low quality. The paper is hard to read. The main text is dense with hedges, interval qualifiers and descriptive-versus-inferential caveats, and the key design information sits across a long appendix. I'm not against using LLM to write papers, but the authors did not carefully revise their paper.
  2. Individual-level data are access controlled. External evaluation runs only through maintainer-executed containers on private bundles. Under these conditions PhenoBench works as an internal evaluation system for one group's cohort rather than a public benchmark. The paper also does not position itself against existing biobank-scale prediction resources, such as UK Biobank-derived benchmarks, which raise the same task-definition problems at larger scale with broader access.
  3. The headline "90 tasks" exceeds what is actually evaluated. Tabular models are compared on 52 tasks, language models on 40, and the measurement maps show a handful of selected examples. No table lists all 90 tasks with their cohort sizes, prevalences, baseline performance and coverage by each analysis. The domain distribution is also strongly skewed, with 21 cardiovascular and lipid tasks against one ageing task, so aggregate cross-task summaries are weighted toward a few clinical areas.
  4. The clinical review behind the tasks is thinly documented. Section 3.2 states in one sentence that all Task Cards were reviewed by "a physician". A single reviewer covering 15 domains is not specialist adjudication. No agreement measure, review rubric or record of rejected or revised tasks is reported.
  5. The language-model evaluation is framed so that its main conclusion is largely predetermined. Zero-shot models receive serialized summary statistics and must output values in HPP-specific units and scales. They are then compared with ridge models fitted on thousands of participants from the same cohort. Not surpassing a fitted in-distribution probe is expected, and it says little about biomedical reasoning. The paper also does not examine contamination. HPP results and reference values appear in published papers that may be in the training data.
Questions for authors

See weaknesses above.

3 comments
AuthorsOpenReview ↗

Thanks. We’ve clarified what’s in the benchmark - what we actually tested, how others can use it... The task definitions and existing results are now attached (and we’ve repaired crowded figure labels). We also explain the access limits and the purpose of the LLM comparison more directly: the gap between zero-shot LLMs and cohort-trained models alone doesn’t establish a gap in medical reasoning.

Thanks for the detailed review! We have clarified the contribution and design, made coverage inspectable, and corrected specific presentation defects.

Presentation: The Introduction now has a named Related Work subsection and an earlier contribution statement; the main Methods explains coverage and external execution. We retained qualifiers that bound scientific claims and removed a repeated coverage sentence. We repaired label and annotation collisions in Figure 1, Figure 4B and the supplementary LLM figures. Figure 1 is a conceptual schematic; the AI-use statement now explicitly records its AI-assisted artwork.

Public resource and access: The release exposes task definitions, aggregate evidence, prompts, schemas and synthetic development inputs. Its external container route is operated manually and inference-only (as documented). Participant data and the full fitted-model execution environment are access controlled (for privacy related reasons). Of course, this limits independent reproduction. Researchers can request governed HPP data access through the HPP data-access route. UK Biobank is a larger research-cohort resource, and our contribution does not depend on claiming greater scale or access. PhenoBench supplies versioned task contracts for the measurements evaluated here, including CGM, home sleep recordings, microbiome and repeated deep-phenotyping visits, and links measurement and method comparisons within that setting.

Registered targets and evaluated coverage: Appendix G indexes all 90 registered clinical targets. 52 enter the ten-method regression grid, and 30 underlie the 40 LLM formulations; 16 are in both analyses and 24 in neither. 15 targets have no result row in the released catalog, including 14 without a released Track. Three of those targets: VAT and the two BMD targets, are evaluated in the separate LLM suite. Appendix G and the companion formulation table distinguish their LLM sample sizes from the catalog counts. The companion archive contains the Task Cards, catalog, task index and existing matched-regression rows so that definitions and available results can be inspected. Class prevalences are not comprehensively tabulated in the released records.

LLM comparison: The aim is to formulate relevant biomedical questions and examine what direct use of an off-the-shelf LLM recovers from participant measurements without cohort-specific fitting. A probe trained on the same cohort has an expected advantage and that gap is not a test of biomedical reasoning. The probe provides a reference for the signal learnable from the input fields. Existing results also use scale-free endpoints.

Contamination: Appendix F now distinguishes possible exposure to published HPP findings or reference distributions from participant-level evaluation leakage (such pretraining exposure cannot be ruled out). Private evaluation records and a shuffled-packet control do not establish that published material was absent from training.

Thanks to the authors for their response. The revised materials partly addressed my concerns. However, some of my major concerns remains, including

  1. task card review involves only one physician for all 15 domains.
  2. the clinical meaning of small tabular gains.
  3. the stated but missing subgroup analyses.

I'll raise my score accordingly but I believe there is still a gap to accept.

AuthorsOpenReview ↗

tldr - thank you! corrected the requested changes and added subgroup analysis.

Thanks for taking another look!! on the three remaining points:

  1. One physician reviewed every Task Card for clinical rationale and specification: target, units, timing, eligibility, comparators and leakage risks, using source literature and HPP documentation (see revised Methods 3.2) - the same physician also examined the OpenEvidence material used to develop the cards and so on. the goal was to assess whether each task made clinical sense, drawing on clinical judgment and supporting evidence (without claiming specialist expertise in every domain, honestly - to develop these types of tasks).

  2. we do not claim clinical benefit from the small tabular gains. for scale, in the grip-strength example, TabPFN 3.5 improves R square over ridge by 0.012 and reduces MAE from 5.35 to 5.28 kg. ridge remains competitive under this protocol, and downstream decisions and patient outcomes were not evaluated (such as net benefit analysis). the lack of huge advantage for general tabFMs is consistent with Shaktah et al. (see ref 17 from the manuscript), who found limited gains from TabPFN over optimized baselines across twelve binary clinical prediction tasks. our results extend that evidence to six pretrained models across 52 matched regression tasks, showing where pretraining adds predictive value and how large that benefit is under the tested conditions.

  3. we located the residual diagnostics saved with the original runs (omitted by mistake, thanks for letting us know!) Appendix B.4 now shows recorded sex (female/male) and age bands for grip strength, whose Task Card requests these checks. at ages 40–59, covering 80% of this test cohort, the models' MAE differs by 0.01–0.03 kg. at ages 30–39 and 60–69, mean overprediction falls from 4.04 to 2.09 kg and from 1.45 to 0.69 kg with TabPFN 3.5. these are point estimates from one comparison, without subgroup intervals; the 30–39 band has 43 participants. the small age bands are pooled, and the full diagnostic tables remain private to avoid exposing small-group errors (no models were rerun).

Reviewer_73T1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PhenoBench, an executable multimodal clinical benchmark constructed from the Human Phenotype Project cohort (>13,000 participants).

Spanning 90 tasks across 15 domains and 26 modalities, PhenoBench formalizes task definitions through rigid evaluation contracts to prevent target leakage and enable standardized comparisons.

The authors evaluate emerging tabular foundation models (TFMs) across 52 tasks and 14 language models (LLMs) across 40 tasks.

The empirical results reveal that TFMs yield marginal gains over simple ridge baselines, while zero-shot LLMs consistently lag behind lightweight domain-fitted models.

Critical strengths

1 The introduction of strict "evaluation contracts" systematically specifies temporal boundaries, cohorts, and metric thresholds, addressing chronic comparability and leakage issues in clinical ML.

2 Grounding 90 clinically relevant tasks on deeply phenotyped longitudinal data (wearables, molecular, imaging) provides a valuable, unified testbed.

3 The honest evaluation demonstrates that tabular foundation models and general LLMs currently offer marginal or inconsistent utility over standard task-specific baselines.

Critical weaknesses

1 The Human Phenotype Project originates from a geographically and demographically restricted cohort, raising questions about whether benchmark conclusions translate to diverse global populations.

2 While the paper documents the minimal advantage of TFMs over ridge regression, it provides limited architectural or feature-level analysis explaining why pretraining failed to transfer effectively.

3 Deeply phenotyped clinical and genomic cohorts often come with severe data governance and privacy restrictions, which may hinder seamless open-source execution by the broader community.

Questions for authors

Given data governance restrictions around deeply phenotyped human cohorts, how will external researchers reproduce the benchmark or execute new models against PhenoBench?

2 comments
AuthorsOpenReview ↗

Thanks for the review. We’ve clarified how others can submit models for evaluation (while participant data stay protected), and attached the task definitions and existing results.

Thank you for identifying the evaluation contracts and linked multimodal task collection as strengths. We have clarified external execution, the model comparisons, and population scope.

External evaluation and reproduction: Methods 3.4 now describes the submission route: researchers develop against synthetic, schema-faithful bundles, submit a digest-pinned OCI image, and request evaluation (which is manual at this point). The documented "contract" supplies prediction inputs without labeled HPP training rows (for privacy/IP reasons). It therefore supports predictors that can run without cohort-specific fitting inside the submitted container (fitting methods on HPP training rows is not part of the published interface). Independent reproduction of the fitted-model results requires governed data access and the private execution environment. Researchers may request HPP access through the HPP data-access route. The companion review-materials archive contains the Task Cards, catalog, existing aggregate comparison rows and prediction interface documentation.

Population scope: The results describe HPP’s recruitment and measurement conditions. Versioned contracts expose those conditions but do not replace independent-cohort validation; the generalizability boundary remains explicit in the Limitations and Appendix F.

Interpreting the tabular-model gains: The pretrained models ranked above the four evaluated task-specific baselines in aggregate; the absolute gain over ridge was small under the tested single-estimator, bounded-tuning protocol. Existing Appendix B analyses attribute more variation in the six-FM advantage to task identity (η² = 0.54) than feature-set identity (0.08) (with no clear association with cohort size or the demographic baseline). The separate Chronos2 comparison reports the high-dimensional ridge-head setting and sample-size dependence.

Thanks for your response. I intend to keep my rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

thanks for the feedback - addressed the points that were raised

Thanks for the feedback! we’ve uploaded the revised paper and supporting materials, including the Task Cards and catalog of all 90 registered targets. we’ve clarified what PhenoBench adds, what was evaluated, and how others can use it, including the access limits. following the discussion, we’ve also added a sex- and age-stratified error example and explained the small tabular gains in native units. we clarified the scope of physician review and repaired figure labels and formatting. our detailed replies are below.

we appreciate that the overall assessment weighs several dimensions. we were unsure how some of the written feedback, particularly descriptions of soundness concerns as minor or fixable, translated into the overall scores. we may be missing how the different concerns are being weighed, and would appreciate hearing which remain decisive after the revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.