- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This work introduces PhenoBench, a broad benchmark covering 90 different clinical tasks across 15 domains with 26 input modalities, built around the Human Phenotype Project. Each PhenoBench task specifies a prediction target, eligible population, timing, evaluation metrics, and relevant baselines through a human-readable Task Card and an executable evaluation contract. The tasks support mapping measurements to questions in two ways, either fixing the question or the measurement. The experiments compare ten tabular methods across 160 matched regression settings covering 52 tasks, examine frozen time-series representations against engineered features, and evaluate 14 LLMs on up to 40 tasks. The authors report modest improvements from tabular foundation models over ridge regression and generally stronger performance from task-trained traditional models than from LLMs using zero-shot direct inference.
Critical strengths
- The benchmark encapsulates diverse cross-domain tasks, with a span of 26 input modalities linked within the same cohort. It could serve as a useful resource, enabling comparisons of information and evaluations of predictions over potentially shared populations.
- The evaluation coverage is broad. The experiments include a wide range of linear, tree-based, neural tabular, time-series foundation-model embeddings, and LLM methods.
Critical weaknesses
- The manuscript lacks a dedicated “Related Work” section, which makes its novelty and level of contribution difficult to assess. I would like to see a focused comparison clarifying what is new in the task collection, method, evaluation, and resulting scientific insights relative to existing clinical and tabular benchmarks.
- The manuscript reports 90 registered tasks, but it does not provide a complete list or supplementary material specifying each task’s target construction, eligibility, timing, sample counts, and available evaluations. It is also unclear why Section 2.3 only evaluates 52 tasks, while the LLM suite covers 40.
- Beyond the above concerns, I am uncertain what substantial new research capability or scientific insight PhenoBench provides to the field. Its central promise is to map which measurements can inform which questions, but the explicit matched comparison across distinct measurement modalities in Figure 2B only uses two targets and four modalities. The broader measurement-first examples use different cohorts and outcomes, so they do not provide an equivalent controlled comparison. Additionally, another potential contribution of the benchmark is a consolidation of HPP measurements into a reusable task suite, but the manuscript does not demonstrate clearly which questions become newly tractable or easier to study through this resource. The empirical results currently offer limited insight into the conditions under which different measurements, representations, or model families succeed.
Minor:
- The references, page headers, and citation formats do not seem to adhere to the ICLR 2027 style.
- The figures in the manuscript can be better presented. Limited examples include: Figure 1 (Some texts overflow from their textboxes, or overlap with shapes and arrows. The icons in C4 appear malformed), Figure 4 (Text overlaps with diagram), and Supplementary Figure 3 & 4 (Lots of textual and graphical overlap. Some plots are also too dense to read).
Questions for authors
- What is the principal research capability that PhenoBench adds beyond existing multimodal biomedical studies and clinical prediction benchmarks?
- Can you provide a description of all 90 registered tasks, with their target definitions, cohort sizes, and evaluation coverage?
- Beyond the two targets in Figure 2B, does the current benchmark contain additional matched comparisons across distinct measurement modalities?
2 comments
Thanks for your response. I read through the other reviews and responses, as well as the revised manuscript. However, my concerns in W3 remain unresolved, and many figures still have overlapping texts and shapes. I will keep my current rating.
Thanks. The main contribution is making the cohort reusable for comparing measurements and models on clearly defined clinical questions. We’ve strengthened the related-work comparison, added the full task inventory and supporting materials, and repaired figure labels and formatting. We also now state explicitly that the fully matched measurement comparison covers two targets.
Thank you for separating broad evaluation coverage from the question of what reusable research capability the resource adds.
Related work and contribution: The named Related Work subsection groups the PhysioNet, EHRSHOT, MedAgentBench and MedHELM comparisons and adds a concrete UK Biobank survival-benchmarking reference. The contribution is a reusable cohort-based task collection with versioned target, eligibility, timing, allowed information, comparator and metric, supporting linked comparisons of measurements and methods.
Task descriptions and coverage: Appendix G indexes the 90 registered clinical targets (and other details, please see other responses).
Principal research capability: PhenoBench allows a researcher to change the measurement route for a fixed question or change the model for fixed information while retaining comparison conditions and provenance. The existing paper demonstrates this through the tabular grid, representation comparisons and two-target measurement comparison. It reduces the need to reconstruct target, eligibility, timing, inputs, split and comparator independently for every model. This is a reusable evaluation capability rather than evidence of a new clinical mechanism.