acceptodds
Under review as a conference paper at ICLR 2027

MEDS-Tab: Scalable Tabularization for Reproducible MEDS Baselines

Abstract

Classical tabular models are widely recognized as strong, often competitive baselines for structured prediction tasks on electronic health record (EHR) data, making them an important evaluation target for new machine learning models in this space. However, producing such baselines for longitudinal EHR datasets remains difficult: before any downstream model can be trained, researchers must convert irregular longitudinal event streams into temporal leakage-safe, task-aligned, fixed-dimensional feature matrices. While modern AutoML and hyperparameter optimization tools are incredibly effective at tuning models once such a matrix exists, they largely assume that this upstream tabularization problem has already been solved; however, this is often not the case. To help address this challenge, we introduce MEDS-Tab, a scalable tabularization system for datasets represented in the Medical Event Data Standard (MEDS). Given a MEDS event-stream-formatted dataset and task labels, MEDS-Tab constructs sparse, sharded feature matrices over heterogeneous clinical codes, lookback windows, and aggregation functions, with optional utilities for training downstream tabular baselines. MEDS-Tab scales to datasets with hundreds of millions of events and tens of thousands of codes. In our scalability experiments, MEDS-Tab completes all runs, while competing tools take much longer, and use substantially more memory or run out of memory. MEDS-Tab is also usable as shared infrastructure: it provides public documentation and a MIMIC-IV tutorial. It has received 17,836 PyPI downloads as of September 25, 2026 and has been used to generate baselines in five external studies. Across these studies, MEDS-Tab provides strong tabular baselines, achieving higher reported AUROC than every conventional baseline in 90.2% of study-dataset-task settings and outperforms all models, including neural and foundation models, in 62.7% of settings. By filling the infrastructure gap between standardized EHR event streams and reproducible classical ML baselines, MEDS-Tab improves the ease, scalability, and reproducibility of benchmarking across clinical prediction tasks and publications.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
electronic health records, clinical prediction, tabular baselines, tabularization, benchmark infrastructure, sparse feature matrices, Medical Event Data Standard

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_j1XCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors introduce a PyPI package, MEDS-TAB, an automated framework that tabularize longitudinal EHR data to produce feature tables suitable for standard tabular models such as XGBoost. The method is introduced as a more efficient tabularization method, in terms of computational resources (time and space), compared to their baselines. It takes as input an EHR longitudinal table (in MEDS format) with multiples events of different subjects and different timestamps. The method produces a sparse features table, in which each row corresponds to a prediction task instance indexed by a subject and prediction time, and each column corresponds to a code–aggregation–window triple. Ones of their main contributions is that temporal leakage is prevented by using only observations recorded before each prediction time. They compare their method with tsfresh and CaTabRa and provide evidence of its strength in community led studies as external validation.

Critical strengths
  1. I find the motivation behind the idea very interesting. EHR data is very noisy and every clinical ML paper reimplements its tabularization by hand. A standardized and reusable way to perform it across all downstream models is necessary. They show that in their community experiments where they compare baselines using the proposed tabularization method.
  2. They tackle the issue of leakage safety systematically. That means that is impossible for a user to make a mistake in the feature construction and jeopardize the results of the downstream models.
  3. The utilization of sparsity to generate rows only at valid prediction times is a great engineering choice and they show its contribution by comparing it in terms of computational time and space.
Critical weaknesses
  1. The authors only compare MEDS-TAB with non-healthcare specific tabularization methods and ignore more domain specific comparisons (see first question).
  2. In my opinion the dataset-agnostic is an overstatement, since the user needs to transform the data into the MEDS format before using the package. Even if it is a easy thing to do, it requires preprocessing from the user in order for their package to be dataset-agnostic, a fact they acknowledge in their limitations.
  3. I find that the max of 500 subjects is not a sufficient stress test. I think more comparisons for larger number of subjects is needed to see the capacity of the subject or an explanation is needed of why this is not possible.
  4. CaTabRa software, that the authors compare their method to, offers explainability as an output of the package, but the authors chose not to compare or state anything about this property of their method.
Questions for authors
  1. Why the authors chose not to compare their package with other packages tailored for healthcare data like FIDDLE, FEMR, OHDSI? I would like to see the respective comparison with them or an explanation of how the proposed package is different and why a direct comparison is not applicable.
  2. I would like to see an experiment to show the strength of sparsity compared to more dense representations, which in my understanding is the main contribution of the proposed tabularization method. Is it only used to save space and time and parallelize jobs?
  3. I would like to see more stress tests for larger number of subjects or at least explain what do they expect to see as the number of subjects increase at the scale of thousands.
  4. How does it compare to other methods based on explainability? Its something the authors don't address where other packages care about. Is there a trader-off between MEDS-TAB and explainability?
4 comments

Response to Reviewer j1XC

We are encouraged that the reviewer recognizes the need for standardized, reusable EHR featurization. Based on your feedback we expanded the baselines we compare against to additionally include FIDDLE, FEMR, OHDSI, and Featuretools. We provided runtime and memory profiling experiments across significantly higher patient subgroups, and consistently observe that MEDS-Tab can generate more features faster. We stress-tested MEDS-Tab on the full MIMIC-IV cohort of 364,627 patients, containing 881,400,075 observations and generating 95,603,126 patient–timestamp feature vectors. We show that for each benchmarked aggregation MEDS-Tab takes under 10 minutes to generate nearly 100 million feature vectors on the full dataset. We provide an ablation of sharding and parallelism as well, and show that the sparse aggregations MEDS-Tab uses achieve 35.21–358.94× speed improvements and 6.35–8.27× less memory compared to the best dense implementation tested.

W1 / Q1: Healthcare-specific comparisons

The authors only compare MEDS-Tab with non-healthcare specific tabularization methods and ignore more domain specific comparisons (see first question).
Why the authors chose not to compare their package with other packages tailored for healthcare data like FIDDLE, FEMR, OHDSI?

We appreciate the reviewer highlighting additional packages that we can compare to. We have now added FEMR, FIDDLE, OHDSI, and Featuretools, alongside tsfresh and CaTabRa. In W3 below we expanded our compute time and memory benchmarks to compare against these methods, and additionally to compare time, memory, and predictive performance on a benchmark of real clinical prediction tasks.

One reason some of these packages were not originally benchmarked was that they do not compute the same features. See in the table below a checkmark indicates the method supported the same aggregation and we observe the same output as MEDS-Tab, while "—" indicates that the method does not support that aggregation, and we note that FIDDLE and OHDSI do not include a code/count aggregation, but provide binary presence indicators 0 or 1 outputs instead so we compare the runtime for those aggregations even though it does not exactly match. OHDSI also did not have value/max or static/present in its default pipeline so we used its last/value and Demographics aggregation methods for comparison.

Methodcode/countvalue/countvalue/maxstatic/present
MEDS-Tab✓✓✓✓
CaTabRa✓✓✓✓
tsfresh✓✓✓✓
FEMR✓———
FIDDLEPresence*✓✓—
OHDSI FeatureExtractionPresence*—Last value*Demographics*
Featuretools✓✓✓✓

W3 / Q3: Scaling beyond 500 patients

I find that the max of 500 subjects is not a sufficient stress test.
I would like to see more stress tests for larger number of subjects or at least explain what do they expect to see as the number of subjects increase at the scale of thousands.

We thank the reviewer for this comment. As noted in our response to Reviewer CfQN, the computational cost of tabularization depends mainly on the number of feature rows (one per patient–prediction-time pair) and codes, rather than on the number of patients alone. To show how MEDS-Tab scales well beyond the original 500-patient benchmark, we report results on full clinical task cohorts containing up to 218,060 patients and 533,874 feature rows (Appendix H). In our results tables, the columns are ordered from the smallest task (fewest labels) on the left to the largest task (most labels) on the right. The number of labels, and therefore feature rows, in each task is:

LeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
# labels34,10646,12747,22652,62152,74459,76661,126113,604149,030331,664533,874533,874
# patients26,76035,61436,56840,42340,46345,65646,87983,62494,500157,532218,060218,060

We see much better runtime for MEDS-Tab (units are h:mm:ss):

MethodLeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
MEDS-Tab0:01:400:02:060:02:000:02:080:02:260:02:320:02:360:01:310:02:090:07:050:10:210:10:23
FEMR0:04:260:07:420:03:280:04:010:06:270:06:500:07:230:03:320:03:420:23:490:22:030:21:45
OHDSI0:02:110:03:090:02:520:03:120:03:070:03:410:04:040:04:450:05:550:18:010:25:080:25:20
CaTabRa5:55:2810:43:038:23:499:54:5210:01:0711:22:2913:37:187:11:507:59:3915:19:589:56:099:59:21
FIDDLEOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
tsfreshOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
Featuretools (code/count)OOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM

And peak memory usage (GB):

MethodLeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
MEDS-Tab1.111.411.181.221.11.311.070.881.22.763.433.49
FEMR2.473.124.95.233.13.143.162.725.798.6914.7714.73
OHDSI6.26.326.437.557.428.197.268.1511.5323.0942.4838.04
CaTabRa12.7917.817.4619.5419.5123.226.2136.3347.39115.65163.48180.67
FIDDLEOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
tsfreshOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
Featuretools (code/count)OOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM

MEDS-Tab achieves the shortest feature-generation runtime and lowest peak memory in every completed comparison, while constructing a broad set of features across multiple lookback windows. Its default workflow computes six temporal aggregations (code count, numeric-value count, sum, sum of squares, min, and max) over seven windows: 2 hrs, 12 hrs, 1 day, 7 days, 30 days, 365 days, and full history. Together with static presence features, these form 43 feature blocks. We use the default tabularization workflows provided by competitors (details are found in Appendix H):

  • FEMR: performs code/count aggregation on a full look back window.
  • OHDSI: Computes demographic features, clinical-concept code presence indicators over 30-day and 365-day lookback windows, measurement-range categories, and comorbidity scores.
  • CaTabRa: Computes the following aggregations over a 48 hour lookback window: value/min, value/max, value/count and additionally uses mean, standard deviation, first, and last values and first and last observation times, and the 10th and 90th percentile values. FIDDLE, tsfresh, and Featuretools could not complete the evaluated pipelines because they ran out of memory (OOM). Across tasks the 12 tasks, FEMR generated a min and max of 67,457–131,797 features respectively, OHDSI 36,179–76,870, and CaTabRa 20,933–23,287, while MEDS-Tab generated 547,822–1,002,892 features across task. Even though it provided over 10 times more features than any other method, MEDS-Tab was consistently the fastest and most memory efficient method for every task. We additionally see strong AUROC improvements from this wider set of features which we discuss in detail in CfQN.W2/Q3.

We additionally stress-tested MEDS-Tab on the full MIMIC-IV cohort of 364,627 patients, containing 881,400,075 observations and generating 95,603,126 patient–timestamp feature vectors. MEDS-Tab efficiently tabularized these and scales to the full dataset with modest memory usage and blazing speed (we only ran this experiment one time so there is no standard deviation):

Lookback windowAggregationFeature matrix dimensions (rows × columns)Runtime (min:sec)Peak PSS (GB)
30 dayscode/count95,603,126 × 148,1913:39129.08
30 daysvalue/count95,603,126 × 2,3850:5051.53
30 daysvalue/max95,603,126 × 2,3851:4754.8
Full historycode/count95,603,126 × 148,1917:11199.99
Full historyvalue/count95,603,126 × 2,3850:5964.66
Full historyvalue/max95,603,126 × 2,3852:5168.01
No windowstatic/present364,627 × 2†0:1817.19

Note that each of these aggregations takes under 10 minutes on the full dataset, even when generating nearly 100 million feature vectors.

Q2: What do sparsity, sharding and parallelism contribute?

I would like to see an experiment to show the strength of sparsity compared to more dense representations, which in my understanding is the main contribution of the proposed tabularization method.
Is it only used to save space and time and parallelize jobs?

Thank you for your feedback. We added a direct ablation of MEDS-Tab with and without sharding. We took a random sample of 10 thousand patients, and we compared processing them in 1 shard vs partitioning them into 16 separate shards, and having 1 worker vs having 24 parallel workers (on a machine with 24 cores). We profiled memory usage and runtime for generating the following aggregations, when varying sharding and parallelism. The features generated are:

  • At Seven lookback windows: 2 hours, 12 hours, 1 day, 7 days, 30 days, 365 days and full history.
  • Compute Six temporal aggregations (at every lookback window): code/count, value/count, value/sum, value/sum_sqd, value/min and value/max.
  • Additionally we compute Static/present features, computed once per shard.

Thus, we generated 43 feature matrices totaling 179,615 features across 2,670,555 unique patient–observation-time pairs, thus a 2,670,555 x 179,615 feature matrix. We observed the following run times and memory profiles as we varied sharding and parallelism

ShardsWorkers (Parallelism)Time (m:ss)Avg PSS (GB)Peak PSS (GB)
1116:4612.4375.09
1244:55183.38299.06
16128:350.332.65
16243:4615.1539.97

We observe that parallelism significantly reduces runtime and memory usage, and sharding reduces memory usage but increases runtime. Combining both we nearly half memory utilization while achieving a 4.45× speed gain. More details are in Appendix I.

We want to note that the purpose of Table 1 is to demonstrate the speed benefit from sparsity. In that experiment we compare MEDS-Tab to Catabra and TSFresh for a set of 10, 100, and 500 patients, all of which are loaded as a single shard in these experiments, so efficiency gains are precisely from MEDS-Tab having an efficient sparse implementation, while Catabra and TSFresh rely on dense representations for computing these aggregations. We expanded these results to include the additional baselines that were provided.

We evaluate runtime and memory profiles across multiple aggregation functions at the full and 30 day lookback windows, and we see consistent efficiency gains from MEDS-Tab. We show more results in CfQN.W1/Q1, and here we focus on comparing MEDS-Tab's sparse vs alternative dense aggregation implementations. Catabra, TSFResh, FIDDLE, and featuretools use dense representations, while OHDSI forms aggregations as SQL queries (which use sparsity techniques internal) and FEMR uses a sparse representation, so we exclude those two from this comparison. We compare speed ups at 10 and 100 patients of MEDS-Tab's sparse aggregation (using a single shard and worker) against the dense approaches and we observe the following speed and memory improvements vs dense approaches:

We provide a summary table below of how many times faster MEDS-Tab is than the fastest alternative. CaTabRa is the fastest dense alternative for all three aggregations at 10 patients and for value/count and value/max at 100 patients. FIDDLE’s code/present is the fastest alternative to code/count at 100 patients (providing a binary indicator of whether a code occurred rather than the count) under both lookback windows. Results are below:

MEDS-Tab runtime in seconds (speedup over the fastest matching dense competitor)

PatientsLookback windowcode/count*value/countvalue/max
1030 days0.93 (31.21×)0.93 (3.69×)0.93 (3.76×)
10Full history0.93 (36.38×)0.92 (4.02×)0.93 (4.02×)
10030 days1.03 (244.69×)0.98 (35.21×)1.00 (35.48×)
100Full history1.09 (358.94×)0.99 (39.30×)1.01 (38.65×)

Across these aggregations at 100 patients, MEDS-Tab is 35.21–358.94× faster than the fastest dense alternative across the evaluated temporal aggregations and windows.

We also provide a comparison of how many times higher the peak memory was of the best alternative (lowest memory using alternative). FIDDLE’s code/present also uses the least peak memory among dense alternatives to code/count at both cohort sizes. For value/count and value/max, CaTabRa uses the least memory at 10 patients and FIDDLE at 100 patients. The results are below:

MEDS-Tab peak PSS in GB (lowest-memory matching dense competitor ÷ MEDS-Tab)

PatientsLookback windowcode/count*value/countvalue/max
1030 days0.428 (4.16×)0.433 (1.92×)0.432 (2.11×)
10Full history0.431 (4.37×)0.416 (1.83×)0.429 (1.70×)
10030 days0.587 (8.27×)0.495 (6.60×)0.501 (6.50×)
100Full history0.653 (8.10×)0.520 (6.40×)0.523 (6.35×)

Across these aggregations at 100 patients, the lowest-memory dense alternative uses 6.35–8.27× more peak memory than MEDS-Tab, which requires only 0.495–0.653 GiB.

W2: Scope of dataset independence

In my opinion the dataset-agnostic is an overstatement, since the user needs to transform the data into the MEDS format before using the package.

We thank the reviewer for this comment and would like to clarify our claim. By dataset-agnostic, we mean that the same functional tabularization and baseline fitting pipeline can be run on any EHR dataset, as MEDS-Tab only depends on MEDS format. The limitations section acknowledges that conversion of raw EHR data to MEDS format is a limitation: "Finally, MEDS-Tab requires MEDS format datasets. Converting a dataset to MEDS is relatively simple, and adoption of MEDS format is growing (18; 14), with multiple datasets publicly available in MEDS format and multiple tools aiding in this preprocessing step (18; 3); nonetheless, dependence on this format is a limitation."

W4 / Q4: Explainability

CaTabRa software, that the authors compare their method to, offers explainability as an output of the package, but the authors chose not to compare or state anything about this property of their method.
How does it compare to other methods based on explainability? Its something the authors don't address where other packages care about. Is there a trader-off between MEDS-Tab and explainability?

We thank the reviewer for raising this point, which we did not address in the submission. We do not believe MEDS-Tab introduces a tradeoff with explainability. Every MEDS-Tab feature retains its clinical code, aggregation and lookback window (e.g., LAB//creatinine/7d/value/max). As a result, any feature attribution from the downstream model maps directly to a well-defined clinical summary, such as “the maximum creatinine value in the past 7 days.” Standard attribution methods, such as TreeSHAP for the XGBoost models MEDS-Tab trains, can be applied directly to the resulting predictors. CaTabRa incorporates explanation tools into its workflow, whereas MEDS-Tab provides interpretable features and leaves the choice of attribution method to the user. To facilitate using alternative models or explainability methods, MEDS-Tab also provides a `TabularDataset` class for loading tabular features from disk into a scipy sparse matrix.

Reviewer_MN4qOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MEDS-Tab, a framework for converting longitudinal electronic health records (EHRs) in the Medical Event Data Standard (MEDS) format into sparse, fixed-dimensional feature representations for conventional machine learning models. The framework uses clinical-code aggregations over predefined lookback windows, sparse matrices, subject-level sharding, and parallel processing to improve computational efficiency. It also provides command-line tools and optional downstream model training utilities. The authors evaluate MEDS-Tab through computational benchmarks against tsfresh and CaTabRa on MIMIC-IV and eICU, reporting substantial improvements in runtime and memory consumption. They further demonstrate the framework through clinical prediction tasks and aggregate results from five external studies, reporting competitive AUROC performance against conventional and neural models.

Critical strengths

I appreciate that the paper addresses an important practical challenge in healthcare ML: constructing reproducible, task-aligned tabular baselines from heterogeneous longitudinal EHR data. The proposed MEDS-native framework, together with its documentation and end-to-end workflow, could reduce repetitive feature-engineering efforts and facilitate more consistent comparisons between conventional and neural prediction models. I consider this a meaningful practical contribution to the research community.

I find the reported computational efficiency improvements over CaTabRa and tsfresh encouraging, particularly in settings where competing implementations run out of memory. The use of sparse representations, subject-level sharding, and parallel processing appears well suited to high-dimensional clinical data. Although these techniques are established, their integration into a reusable framework could offer substantial practical value, especially given the reported community adoption and use in external studies.

Critical weaknesses

Limited technical novelty and insufficient differentiation from existing work. My main concern is that the paper does not convincingly establish a sufficiently distinctive methodological or systems-level contribution. The primary components, including sparse representations, subject-level sharding, parallel processing, retrospective window aggregation, and timestamp-based leakage prevention, are well-established techniques. Existing frameworks such as FIDDLE, OHDSI FeatureExtraction, Cardea, Featuretools, CaTabRa, and tsfresh already address related aspects of automated feature extraction, temporal aggregation, and scalable data processing. While I appreciate the practical value of integrating these approaches within the MEDS ecosystem, the paper does not sufficiently explain which components constitute new technical contributions or how the proposed system differs fundamentally from existing approaches. Furthermore, the feature-engineering procedure relies on a relatively limited set of aggregation functions that may not adequately capture clinically relevant temporal characteristics, such as event ordering, trends, or irregular sampling patterns. I am therefore uncertain whether the paper demonstrates sufficient scientific novelty and technical depth for ICLR, beyond its value as a software infrastructure contribution.

Insufficient experimental validation and unclear attribution of the reported improvements. Although the paper reports substantial efficiency gains, the computational evaluation is restricted to a single aggregation function, one lookback window, and subsets of at most 500 subjects. This does not fully substantiate the broader claims of scalability to hundreds of millions of events and tens of thousands of codes under realistic multi-window, multi-aggregation workloads. Moreover, the paper does not adequately isolate the contributions of sparse representations, sharding, parallelization, and window indexing, nor does it provide a detailed computational complexity or bottleneck analysis. I am also concerned about the interpretation of the predictive comparisons. MEDS-Tab is primarily a feature-processing framework rather than a predictive model, yet the reported AUROC comparisons involve potentially different feature representations, model architectures, and optimization strategies. In particular, the external-study results aggregate heterogeneous published experiments, making it difficult to attribute the reported improvements specifically to MEDS-Tab. The clinical prediction results in the appendices also provide limited evidence of statistical robustness and generalizability. Consequently, I find the experimental evidence insufficient to establish the distinct advantages of the proposed framework beyond its demonstrated practical utility.

Lack of statistical robustness and uncertainty analysis. I am concerned that the experimental evaluation does not sufficiently assess the stability and statistical reliability of the reported results. In particular, the scalability experiments use subsets of only 10, 100, and 500 subjects, without reporting variability across independently sampled subject subsets or repeated runs. It is therefore unclear how representative these results are of the broader datasets and how sensitive the measured speedups and memory consumption are to subject selection, differences in event density, or workload skewness. I would appreciate repeated experiments across multiple random samples and seeds, with appropriate measures of variability, such as standard deviations, confidence intervals, or bootstrap-based uncertainty estimates. Similarly, the predictive evaluation would benefit from confidence intervals or statistical comparisons to help distinguish meaningful performance improvements from differences that may arise by chance.

Presentation and organization need improvement. I found several parts of the manuscript difficult to follow due to abrupt transitions, insufficient introductory explanations, and relatively brief methodological descriptions. For example, Section 3 on page 4 begins directly with three numbered points without an introductory paragraph explaining their relationship to the proposed system. Figures 1 and 2 are not sufficiently integrated into the surrounding discussion, and some elements, such as "Parameter Sweep over Tabular Data" in Figure 1 and the operations illustrated in Figure 2(b), are not adequately explained. Additionally, Section 4.2 refers to the "first stage" despite following the sharding stage in Section 4.1. I believe that clearer transitions, more detailed methodological explanations, and stronger connections between the figures and the text would substantially improve the readability and technical clarity of the paper.

Questions for authors

Could the authors clarify which specific components of MEDS-Tab constitute novel technical contributions beyond established sparse representations, sharding, parallelization, and retrospective window aggregation? I would appreciate a side-by-side comparison with relevant existing frameworks, including FIDDLE, OHDSI FeatureExtraction, Cardea, Featuretools, CaTabRa, and tsfresh, highlighting the technical differences and limitations addressed by MEDS-Tab.

Could the authors explain the rationale behind selecting the six aggregation functions? In particular, why are other potentially clinically informative statistics, such as median, mean, standard deviation, and distributional measures, not explicitly included? Since mean and standard deviation can potentially be derived from the existing aggregation statistics, it would be helpful to clarify whether these are computed downstream. I would also appreciate an explanation of why more explicitly temporal features, such as time since the last observation, temporal slopes, inter-event intervals, and sequence-aware statistics, were not considered.

Could the authors clarify whether MEDS-Tab, CaTabRa, and tsfresh produce numerically equivalent feature outputs in the computational comparisons? Additionally, why is the efficiency evaluation restricted to a single code-count aggregation and one lookback window? I would appreciate results for the complete feature-extraction pipeline across larger datasets and different computational environments, including multiple aggregation functions and lookback windows.

How much of the reported computational improvement is attributable to sparse representation alone, compared with subject-level sharding, parallelization, and window indexing? I suggest component-level ablations and scaling experiments varying worker counts, event counts, code cardinality, feature dimensions, sparsity levels, and missingness rates. It would also be valuable to investigate workload imbalance caused by skewed event distributions across patients or shards, along with a detailed analysis of computational complexity, processing bottlenecks, and storage overhead.

Could the authors provide statistical evidence supporting the robustness of the computational results? In particular, were the 10-, 100-, and 500-subject subsets randomly sampled, and were experiments repeated using independently selected subsets and random seeds? I would appreciate variability estimates, such as confidence intervals or standard deviations, to determine whether the reported efficiency gains are consistent across subject samples and workload characteristics. Repeated timing measurements on identical inputs would also help quantify run-to-run variability separately from sampling variability.

How do the authors justify comparisons between MEDS-Tab-based predictors and conventional or neural models when their feature representations, preprocessing procedures, model architectures, and tuning strategies may differ? Are these comparisons intended to evaluate feature-engineering approaches or complete prediction pipelines? I would appreciate a controlled comparison using identical downstream models, data splits, prediction timestamps, and hyperparameter optimization budgets, varying only the feature-engineering strategy. This would help determine how much of the predictive benefit is attributable specifically to MEDS-Tab.

Could the authors clarify the intended purpose and robustness of the clinical prediction experiments in Appendix C? Were the results evaluated across multiple random seeds, repeated runs, or data splits? I would also appreciate an explanation of the substantial performance differences between prediction times and the high AUROC values reported for certain length-of-stay tasks, together with information about cohort definitions and statistical uncertainty.

How were the five external studies selected, and how should the pooled AUROC win rates be interpreted given the differences in datasets, prediction tasks, models, and evaluation protocols? Since individual studies contribute unequal numbers of comparisons, I would appreciate within-study performance differences and an analysis of the sensitivity of the overall conclusions to study weighting. Could the authors also clarify why MEDS-TAB-large was selected as the representative configuration and explain how it differs from MEDS-TAB-tiny and MEDS-TAB-LR?

How does MEDS-Tab handle missing observations, numerical zeros, undefined aggregation values, normalization, heterogeneous coding systems, and differences in measurement units across institutions? Given its emphasis on dataset-agnostic feature construction, I would appreciate clarification of which issues are handled internally and which require upstream preprocessing. Stronger evidence across different datasets and clinical settings would help establish the generalizability of the framework.

Although several clinical prediction tasks are presented in the appendices, could the authors more clearly highlight the framework's applicability across different task categories in the main text? For example, how does the same feature-construction configuration generalize between longitudinal-history-dependent tasks, such as mortality and readmission prediction, and shorter-term clinical tasks, such as infection or incident diagnosis prediction? To what extent are task-specific modifications required? Additionally, is facilitating fair comparisons between conventional non-sequential models and neural sequential models an explicit objective of the framework?

Could the authors provide a more rigorous mathematical formulation of the complete feature-construction process, including static and temporal features, lookback windows, matrix dimensions, missing-value semantics, and feature concatenation? I would also appreciate a clearer explanation of the workflow and figures, particularly the meaning of "Parameter Sweep over Tabular Data" in Figure 1 and the operations shown in Figure 2(b). Finally, could the authors distinguish more clearly between the scientific contributions of MEDS-Tab and the software usability improvements provided by its end-to-end CLI workflow?

8 comments

Thank you for recognizing the value of reproducible tabular baselines. We appreciate your careful reading of our work and detailed questions and comments. We address them below.

W1/Q1: Systems contribution and related frameworks

My main concern is that the paper does not convincingly establish a sufficiently distinctive methodological or systems-level contribution. The primary components, including sparse representations, subject-level sharding, parallel processing, retrospective window aggregation, and timestamp-based leakage prevention, are well-established techniques. Existing frameworks such as FIDDLE, OHDSI FeatureExtraction, Cardea, Featuretools, CaTabRa, and tsfresh already address related aspects of automated feature extraction, temporal aggregation, and scalable data processing. While I appreciate the practical value of integrating these approaches within the MEDS ecosystem, the paper does not sufficiently explain which components constitute new technical contributions or how the proposed system differs fundamentally from existing approaches…I am therefore uncertain whether the paper demonstrates sufficient scientific novelty and technical depth for ICLR, beyond its value as a software infrastructure contribution.
Could the authors clarify which specific components of MEDS-Tab constitute novel technical contributions beyond established sparse representations, sharding, parallelization, and retrospective window aggregation? I would appreciate a side-by-side comparison with relevant existing frameworks, including FIDDLE, OHDSI FeatureExtraction, Cardea, Featuretools, CaTabRa, and tsfresh, highlighting the technical differences and limitations addressed by MEDS-Tab.
Finally, could the authors distinguish more clearly between the scientific contributions of MEDS-Tab and the software usability improvements provided by its end-to-end CLI workflow?

We agree that the individual techniques underlying MEDS-Tab are well established. Our contribution is not a new featurization algorithm, but a scalable, end-to-end system for constructing and optimizing tabular baselines on longitudinal EHR data. MEDS-Tab addresses a practical gap in existing tooling: obtaining strong, reproducible tabular baselines directly from MEDS-formatted data, without extensive custom feature-engineering pipelines. We believe this constitutes a meaningful systems contribution, consistent with ICLR's explicit inclusion of "infrastructure, software libraries" in its call for papers

To substantiate our contribution, we added FEMR, FIDDLE, OHDSI FeatureExtraction and Featuretools, alongside CaTabRa and tsfresh to our evaluations. Featuretools is Cardea’s feature-engineering backend, so comparing to it essentially covers Cardea. MEDS-Tab requires very low data harmonization effort, leverages sparsity and sharding to achieve high speed and a low memory footprint, and supports generating a massive set of various numeric and count based features, no currently available framework provides all of these capabilities in one, and the table below illustrates this:

FrameworkLow data harmonization effortSparsity maintained throughout computationSparse outputOn-disk shardingFlexible features
MEDS-Tab✓✓✓✓✓
FEMR✓✓✓––
FIDDLE✓–✓–✓
OHDSI FeatureExtraction–✓✓✓–
Featuretools✓–––✓
tsfresh✓–––✓
CaTabRa✓–––✓

The value of MEDS-Tab using sharding, parallelism, and sparsity to generate massive diverse feature sets across any MEDs format dataset enables users to quickly generate tabular baselines that consume all available codes, instead of users having to settle for only modeling a subset of codes and remove potentially predictive signal from their set of features. MEDS-Tab allows users to generate vectors faster for clinical tasks than any existing approach (see "W2–3/Q3, Q5: Expanded computational evaluation and equivalence" for results demonstrating this), and we show fitting models on these gives us higher AUROC (see "W2–3/Q6 Predictive attribution and uncertainty" for those results).

Q2: Aggregation choice and temporal information

Furthermore, the feature-engineering procedure relies on a relatively limited set of aggregation functions that may not adequately capture clinically relevant temporal characteristics, such as event ordering, trends, or irregular sampling patterns.
Could the authors explain the rationale behind selecting the six aggregation functions?

We thank the reviewer for this question. The goal is to support generating a large set of features across all codes on any EHR dataset. Using slower and less memory efficient pipelines leads to users often having to filter down which codes the model, and with MEDS-Tab users can comfortably scale to many different look back windows and leverage our supported aggregations. The current six temporal aggregations were chosen because they can be efficiently computed. These are not a comprehensive set of aggregations that provide complementary predictive signal, these aggregations are empirically performant as evidenced by MEDS-Tab strong performance as a baseline in external studies (Table 2) and our internal predictive performance comparisons to other feature generation pipelines (see "W2–3/Q6 Predictive attribution and uncertainty" for those results). We will add a statement that we leave as future work exploration of additional features that further improve MEDS-Tabs performance.

W2–3/Q3, Q5: Expanded computational evaluation and equivalence

Although the paper reports substantial efficiency gains, the computational evaluation is restricted to a single aggregation function, one lookback window, and subsets of at most 500 subjects. This does not fully substantiate the broader claims of scalability to hundreds of millions of events and tens of thousands of codes under realistic multi-window, multi-aggregation workloads.
Additionally, why is the efficiency evaluation restricted to a single code-count aggregation and one lookback window? I would appreciate results for the complete feature-extraction pipeline across larger datasets and different computational environments, including multiple aggregation functions and lookback windows.

We thank the reviewer for this comment. As noted in our response to Reviewer CfQN, the computational cost of tabularization depends mainly on the number of feature rows (one per patient–prediction-time pair) and codes, rather than on the number of patients alone. To show how MEDS-Tab scales well beyond the original 500-patient benchmark, we report results on full clinical task cohorts containing up to 218,060 patients and 533,874 feature rows (Appendix H). In our results tables, the columns are ordered from the smallest task (fewest labels) on the left to the largest task (most labels) on the right. The number of labels, and therefore feature rows, in each task is:

LeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
# labels34,10646,12747,22652,62152,74459,76661,126113,604149,030331,664533,874533,874
# patients26,76035,61436,56840,42340,46345,65646,87983,62494,500157,532218,060218,060

We see much better runtime for MEDS-Tab (units are h:mm:ss):

MethodLeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
MEDS-Tab0:01:400:02:060:02:000:02:080:02:260:02:320:02:360:01:310:02:090:07:050:10:210:10:23
FEMR0:04:260:07:420:03:280:04:010:06:270:06:500:07:230:03:320:03:420:23:490:22:030:21:45
OHDSI0:02:110:03:090:02:520:03:120:03:070:03:410:04:040:04:450:05:550:18:010:25:080:25:20
CaTabRa5:55:2810:43:038:23:499:54:5210:01:0711:22:2913:37:187:11:507:59:3915:19:589:56:099:59:21
FIDDLEOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
tsfreshOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
Featuretools (code/count)OOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM

And peak memory usage (GB):

MethodLeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
MEDS-Tab1.111.411.181.221.11.311.070.881.22.763.433.49
FEMR2.473.124.95.233.13.143.162.725.798.6914.7714.73
OHDSI6.26.326.437.557.428.197.268.1511.5323.0942.4838.04
CaTabRa12.7917.817.4619.5419.5123.226.2136.3347.39115.65163.48180.67
FIDDLEOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
tsfreshOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
Featuretools (code/count)OOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM

MEDS-Tab achieves the shortest feature-generation runtime and lowest peak memory in every completed comparison, while constructing a broad set of features across multiple lookback windows. Its default workflow computes six temporal aggregations (code count, numeric-value count, sum, sum of squares, minimum, and maximum) over seven windows: 2 hours, 12 hours, 1 day, 7 days, 30 days, 365 days, and full history. Together with static presence features, these form 43 feature blocks. We use the default tabularization workflows provided by the competitors, (more details are found in Appendix H):

  • FEMR: performs code/count aggregation on a full look back window.
  • OHDSI: Computes demographic features, clinical-concept code presence indicators over 30-day and 365-day lookback windows, measurement-range categories, and comorbidity scores.
  • CaTabRa: Computes the following aggregations over a 48 hour lookback window: value/min, value/max, value/count and additionally uses mean, standard deviation, first, and last values and first and last observation times, and the 10th and 90th percentile values.

FIDDLE, tsfresh, and Featuretools could not complete the evaluated pipelines because they ran out of memory (OOM). Across tasks the 12 tasks, FEMR generated a min and max of 67,457–131,797 features respectively, OHDSI 36,179–76,870, and CaTabRa 20,933–23,287, while MEDS-Tab generated 547,822–1,002,892 features across task. Even though it provided over 10 times more features than any other method, MEDS-Tab was consistently the fastest and most memory efficient method for every task. We additionally see strong AUROC improvements from this wider set of features which we discuss in detail in CfQN.W2/Q3.

We additionally stress-tested MEDS-Tab on the full MIMIC-IV cohort of 364,627 patients, containing 881,400,075 observations and generating 95,603,126 patient–timestamp feature vectors. MEDS-Tab efficiently tabularized these and scales to the full dataset with modest memory usage and blazing speed (we only ran this experiment one time so there is no standard deviation):

Lookback windowAggregationFeature matrix dimensions (rows × columns)Runtime (min:sec)Peak PSS (GB)
30 dayscode/count95,603,126 × 148,1913:39129.08
30 daysvalue/count95,603,126 × 2,3850:5051.53
30 daysvalue/max95,603,126 × 2,3851:4754.8
Full historycode/count95,603,126 × 148,1917:11199.99
Full historyvalue/count95,603,126 × 2,3850:5964.66
Full historyvalue/max95,603,126 × 2,3852:5168.01
No windowstatic/present364,627 × 20:1817.19

Notice that each of these aggregations takes under 10 minutes on the full dataset, even when generating nearly 100 million feature vectors.

We also would like to clarify the scalability tests in the original submission. Although the largest comparison had only 500 patients, this was already a substantial featurization workload: it processed 795,368 observations across 8,115 unique codes and produced 75,595 feature vectors (one feature vector at every unique patient observation time). So the output of the code/count tabularization in Table 1 was a matrix of size 8,115 x 75,595. The size of a tabularization workload depends on the number of prediction times and codes, not only on the number of patients. We improved this analysis by providing confidence intervals over resampling these patients five separate times and running on each sample, and we expanded from running only code/count on full look back windows to evaluating code/count, value/count, value/max on both the full lookback window and on a 30 day lookback window. Additionally, we compare to a broader set of baselines (FEMR, CaTabRa, tsfresh, Featuretools, FIDDLE, OHDSI).

Reviewer_CfQNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper's MEDS-Tab aims at making healthcare ML benchmarks easier to reproduce. It converts longitudinal electronic health records to sparse feature tables, summarizing clinical events over time windows for models such as XGBoost. It shows computational savings and predictive results across 5 external studies. It focuses on the feature-construction infrastructure rather than a new prediction algorithm.

Critical strengths
  • Addresses a real bottleneck.
  • Convincing efficiency gains in the tested settings.
  • Practical usability and evidence of reuse.
  • Transparent performance aggregation.
Critical weaknesses
  • The direct scalability tests are narrower than the headline suggests.
  • The source of predictive improvements is not isolated.
  • Leakage protection remains conditional.
Questions for authors
  • Can you benchmark the complete default workflow on full datasets?
  • How were computational comparisons made equivalent?
  • How robust are the predictive win rates?
  • What automated checks validate leakage prevention?
5 comments

We thank the reviewer for their time and thoughtful review. Based on your feedback we have added three appendix sections with extended computational results, clinical prediction results and a controlled sharding/parallelism ablation. We have expanded our computation comparisons to compare against all mentioned methods from reviewers: (FEMR, CaTabRa, tsfresh, Featuretools, FIDDLE, OHDSI). Below we address each of your concerns as well.

W2 / Q3: Attribution and robustness of predictive performance

The source of predictive improvements is not isolated.
How robust are the predictive win rates?

In our original submission, we report win rates across external benchmarks, measuring how often MEDS-Tab outperforms every compared conventional baseline on a given task. Across Studies 1, 2, 3, and 5, these win rates are 100%, 78.6%, 66.7% and 95.0%, respectively, demonstrating strong performance across independent evaluations. Study 4 includes only neural comparators and therefore does not contribute to this classical-baseline comparison.

To complement this analysis, we added direct comparisons against classical tabularization pipelines using their default feature-construction and model-fitting procedures. Study 5 is the most recent benchmarking paper, and it selects a diverse set of nontrivial challenging clinical prediction tasks to benchmark forecasting methods. We chose to adopt those tasks and benchmark AUROC and feature generation timing of MEDS-Tab and of classical tabularization competitors (FEMR, CaTabRa, tsfresh, Featuretools, FIDDLE, OHDSI) to highlight MEDS-Tab predictive and efficiency gains on real world tasks. We see the following AUROC (SD estimated using bootstrapping with 1,000 resamples). Experiments are run on a machine with 512 GB of RAM. We sort the columns in our results going from the smallest task (least number of labels) on the right to largest task (most labels) on the right. The tasks have this many labels (and thus this many feature rows):

LeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
34,10646,12747,22652,62152,74459,76661,126113,604149,030331,664533,874533,874

Here are the AUROC results:

MethodLeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
MEDS-Tab / XGBoost0.817 ± 0.0100.875 ± 0.0080.896 ± 0.0100.870 ± 0.0120.906 ± 0.0280.880 ± 0.0210.859 ± 0.0080.815 ± 0.0070.932 ± 0.0040.822 ± 0.0030.924 ± 0.0030.786 ± 0.004
FEMR / XGBoost0.701 ± 0.0130.834 ± 0.0100.864 ± 0.0120.826 ± 0.0130.812 ± 0.0340.838 ± 0.0260.773 ± 0.0110.714 ± 0.0090.769 ± 0.0090.779 ± 0.0030.899 ± 0.0040.764 ± 0.005
FEMR / LR0.545 ± 0.0140.487 ± 0.0150.471 ± 0.0190.448 ± 0.0230.517 ± 0.0390.450 ± 0.0340.451 ± 0.0130.695 ± 0.0090.618 ± 0.0100.719 ± 0.0040.730 ± 0.0070.686 ± 0.007
OHDSI / LASSO0.745 ± 0.0120.847 ± 0.0090.857 ± 0.0130.825 ± 0.0150.823 ± 0.0350.830 ± 0.0290.777 ± 0.0110.743 ± 0.0080.806 ± 0.0080.777 ± 0.0030.918 ± 0.0030.745 ± 0.005
OHDSI / GBM0.752 ± 0.0110.844 ± 0.0090.875 ± 0.0120.839 ± 0.0130.823 ± 0.0340.828 ± 0.0240.785 ± 0.0110.749 ± 0.0080.807 ± 0.0080.791 ± 0.0030.920 ± 0.0030.752 ± 0.005
CaTabRa0.811 ± 0.0100.871 ± 0.0080.897 ± 0.0100.846 ± 0.0130.864 ± 0.0340.500 ± 0.0000.833 ± 0.0090.778 ± 0.0070.825 ± 0.007Fail*Fail*Fail*
FIDDLEOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
tsfreshOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
Featuretools (code/count)OOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM

*CaTaBra uses auto-sklearn to fit models on the features with a default of a 1 minute time-out for model fitting. We raised this to be 120 minutes and it ended up failing to fit models for the 3 largest tasks within this time.

We see much better runtime for MEDS-Tab (units are h:mm:ss):

MethodLeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
MEDS-Tab0:01:400:02:060:02:000:02:080:02:260:02:320:02:360:01:310:02:090:07:050:10:210:10:23
FEMR0:04:260:07:420:03:280:04:010:06:270:06:500:07:230:03:320:03:420:23:490:22:030:21:45
OHDSI0:02:110:03:090:02:520:03:120:03:070:03:410:04:040:04:450:05:550:18:010:25:080:25:20
CaTabRa5:55:2810:43:038:23:499:54:5210:01:0711:22:2913:37:187:11:507:59:3915:19:589:56:099:59:21
FIDDLEOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
tsfreshOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
Featuretools (code/count)OOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM

And peak memory usage:

MethodLeukocytosisICU mortalityResp. failureVasopressorsCirc. failureVentilationAKIHyponatremiaThrombocytopeniaLOS90-day mortality30-day readmission
MEDS-Tab1.111.411.181.221.11.311.070.881.22.763.433.49
FEMR2.473.124.95.233.13.143.162.725.798.6914.7714.73
OHDSI6.26.326.437.557.428.197.268.1511.5323.0942.4838.04
CaTabRa12.7917.817.4619.5419.5123.226.2136.3347.39115.65163.48180.67
FIDDLEOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
tsfreshOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM
Featuretools (code/count)OOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOMOOM

MEDS-Tab achieves the shortest feature-generation runtime and lowest peak memory in every completed comparison, while constructing a broad set of features across multiple lookback windows. Its default workflow computes six temporal aggregations (code count, numeric-value count, sum, sum of squares, minimum, and maximum) over seven windows: 2 hours, 12 hours, 1 day, 7 days, 30 days, 365 days, and full history. Together with static presence features, these form 43 feature blocks.

The competitor default workflows we compare against construct features as follows:

  • FEMR: We followed their default pipeline, which computes code counts over the full patient history.
  • OHDSI: We followed their default pipeline, which computes demographics, clinical-concept code presence indicators over 30-day and 365-day lookback windows, measurement-range categories, and comorbidity scores.
  • CaTabRa: We followed their github "examples/Longitudinal.ipynb" tutorial on synthetical longitudinal time series data in their github which uses eleven aggregations over a 48 hour lookback window: value/min, value/max, value/count and additionally uses mean, standard deviation, first, and last values and first and last observation times, and the 10th and 90th percentile values.
  • FIDDLE: We followed their default pipeline, which computes features over the past 48 hours in hourly bins (so for each code it computes each aggregation for each of the 48 separate 1 hour bins). FIDDLE defines frequently measured as having on average one observation per hour (at least 48 observations in 48 hours). For frequently measured numerical codes, each bin contains observation presence (0 or 1), time since the last observation, the most recent value, and the minimum, maximum, and mean, with forward-fill imputation. For less frequently measured numerical codes and categorical codes, it retains the most recent value in each bin. Numerical features are subsequently discretized and categorical features encoded using training-fitted transformations. It OOMed during feature generation for all our tasks.
  • tsfresh: We followed the tsfresh github tutorial "notebooks/02 sklearn Pipeline.ipynb", which uses "RelevantFeatureAugmenter". It takes every numerical code, selects code-aggregation pairs statistically associated with the training labels. It OOMed in our runs.
  • Featuretools: We followed their default pipeline on our meds-format schema, which outputs a pooled mean numeric value across all codes, and max numeric value across codes (only two features). It supports doing code/count aggregations, so we tried just this however it OOMed.

Appendix H, Tables 30–34, provides the experimental settings and complete results. MEDS-Tab achieves the highest AUROC point estimate on 11 of 12 tasks, with CaTabRa slightly higher on respiratory failure, and the shortest feature-generation runtime and lowest peak memory among completed comparisons. Under a one-sided t-test with alpha .05, MEDS-Tab improve AUROC by a statistically significant margin on 9/12 of the tasks, and these 9 tasks are the tasks with 52,621 or more labels. This would suggest that the wide-feature set that MEDS-Tab provides for tasks achieves AUROC gains on tasks with sufficiently many labels.

W1 / Q1: Workload size and the complete workflow

The direct scalability tests are narrower than the headline suggests.
Can you benchmark the complete default workflow on full datasets?

We agree that the scope of the original scalability tests should have been described more clearly, and we appreciate the opportunity to do so. Although the largest comparison in the submission used 500 patients, this was already a substantial featurization workload: it processed 795,368 observations across 8,115 unique codes and produced 75,595 feature vectors (one feature vector at every unique patient observation time). So the output of the code/count tabularization in Table 1 was a matrix of size 8,115 x 75,595. The size of a tabularization workload depends on the number of prediction times and codes, not only on the number of patients. We improved this analysis by providing confidence intervals over resampling these patients five separate times and running on each sample, and we expanded from running only code/count on full look back windows to evaluating code/count, value/count, value/max on both the full lookback window and on a 30 day lookback window. Additionally, we compare to a broader set of baselines (FEMR, CaTabRa, tsfresh, Featuretools, FIDDLE, OHDSI). Lastly, we ran MEDS-Tab on the full dataset and showed that one can generate a feature vector for every patient in the dataset with any of these aggregations in under 10 minutes (for reference, on the full dataset the code and value aggregations are generating 95,603,126 feature vectors).

Note to keep consistency with the end to end AUROC results we shared above, we switched from the MIMIC MEDS ETL we previously used, which processed MIMIC-IV 2.0 to the MIMIC-IV 3.1 MEDS ETL used for those experiments, and we use the latest release of MEDS-Tab v0.3.0. We note that this leads to there being significantly more rows and columns in the generated feature matrices. Below we show the size of the generated feature matrices for each of the 5 randomly sampled patients we benchmarked on. The rows vary because the number of unique event-times for each patient will vary, and the columns vary because for code/count it is the number of unique codes in the sample, and for value aggregations it is the number of unique codes with at least one numeric value in the sample. We provide and mean and standard deviation for the rows and columns below:

PatientsRows (mean ± SD)code/count columns (mean ± SD)value/count columns (mean ± SD)value/max columns (mean ± SD)
103,295 ± 2,0372,122 ± 605446 ± 45446 ± 45
10023,276 ± 1,4846,503 ± 150818 ± 40818 ± 40
500128,337 ± 11,55614,396 ± 3131,232 ± 371,232 ± 37

We additionally benchmark the static/present aggregation; there are only 2 features in the mimic-iv MEDS ETL that do not have timestamps and are thus static (Gender male and Gender female), and the number of rows is just the number of patients for static feature aggregations. Full results are in Appendix G, Tables 18–29, and we provide a summary below.

For a concise example, full-history code-count runtimes are presented here. We provide mean and standard deviations of the runtimes, in seconds:

Method10 patients100 patients500 patients
MEDS-Tab0.93 ± 0.031.09 ± 0.052.39 ± 0.50
FEMR3.00 ± 1.356.73 ± 1.5130.53 ± 15.78
CaTabRa33.84 ± 32.251081.85 ± 137.83>21600
tsfresh221.15 ± 182.681547.52 ± 253.62Running
Featuretools1765.50 ± 1644.95OOMOOM
FIDDLE93.50 ± 77.85392.45 ± 134.55Running
OHDSI5.37 ± 2.4022.08 ± 3.25121.68 ± 26.10

Comments on the paper

1 comment

As the author–reviewer discussion period concludes today, November 18, we would like to share additional 500-patient benchmark results and follow up on any remaining concerns.

At 500 patients, MEDS-Tab achieves approximately 11–368× speedups and 4.30–9.64-fold reductions in peak memory relative to the best alternative pipeline (among FEMR, OHDSI, CaTabRa, FIDDLE, tsfresh, and Featuretools) across the aggregations and lookback windows evaluated below. The fastest alternatives are FEMR for code/count and CaTabRa for value/count and value/max; the lowest-memory alternatives are FEMR for code/count and FIDDLE for value/count and value/max. Although these benchmarks use only 500 patients, we generate a feature vector at every distinct observation time for each patient, yielding large feature matrices that average 128,337 rows with 14,396 features for code/count and 1,232 features for value/count and value/max.

MEDS-Tab mean runtime in seconds (speedup over the fastest alternative pipeline)

PatientsLookback windowcode/countvalue/countvalue/max
1030 days0.93 (2.67×)0.93 (3.69×)0.93 (3.76×)
10Full history0.93 (3.23×)0.92 (4.02×)0.93 (4.02×)
10030 days1.03 (5.45×)0.98 (35.21×)1.00 (35.48×)
100Full history1.09 (6.16×)0.99 (39.30×)1.01 (38.65×)
50030 days1.87 (11.04×)1.33 (290.80×)1.53 (257.80×)
500Full history2.39 (12.77×)1.44 (368.08×)1.73 (309.95×)

MEDS-Tab mean peak PSS in GiB (lowest-memory alternative pipeline ÷ MEDS-Tab)

PatientsLookback windowcode/countvalue/countvalue/max
1030 days0.428 (4.55×)0.433 (1.92×)0.432 (2.11×)
10Full history0.431 (4.57×)0.416 (1.83×)0.429 (1.70×)
10030 days0.587 (5.05×)0.495 (6.60×)0.501 (6.50×)
100Full history0.653 (4.84×)0.520 (6.40×)0.523 (6.35×)
50030 days1.101 (5.34×)0.612 (9.64×)0.635 (9.29×)
500Full history1.596 (4.30×)0.692 (9.24×)0.697 (9.16×)

For each patient count (10, 100, or 500), we randomly sampled five patient subsets of that size and used the same subsets across methods. For each aggregation and lookback window, we ran each method once on each subset. The tables report the mean runtime and mean peak PSS across these five runs. Standard deviations and complete baseline results appear in Appendix G.

We hope our rebuttal and these additional results address your concerns. We welcome any remaining questions and would appreciate your consideration of this evidence in your final evaluation.

Thank you again for your time and constructive feedback.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.