- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The authors introduce a PyPI package, MEDS-TAB, an automated framework that tabularize longitudinal EHR data to produce feature tables suitable for standard tabular models such as XGBoost. The method is introduced as a more efficient tabularization method, in terms of computational resources (time and space), compared to their baselines. It takes as input an EHR longitudinal table (in MEDS format) with multiples events of different subjects and different timestamps. The method produces a sparse features table, in which each row corresponds to a prediction task instance indexed by a subject and prediction time, and each column corresponds to a code–aggregation–window triple. Ones of their main contributions is that temporal leakage is prevented by using only observations recorded before each prediction time. They compare their method with tsfresh and CaTabRa and provide evidence of its strength in community led studies as external validation.
Critical strengths
- I find the motivation behind the idea very interesting. EHR data is very noisy and every clinical ML paper reimplements its tabularization by hand. A standardized and reusable way to perform it across all downstream models is necessary. They show that in their community experiments where they compare baselines using the proposed tabularization method.
- They tackle the issue of leakage safety systematically. That means that is impossible for a user to make a mistake in the feature construction and jeopardize the results of the downstream models.
- The utilization of sparsity to generate rows only at valid prediction times is a great engineering choice and they show its contribution by comparing it in terms of computational time and space.
Critical weaknesses
- The authors only compare MEDS-TAB with non-healthcare specific tabularization methods and ignore more domain specific comparisons (see first question).
- In my opinion the dataset-agnostic is an overstatement, since the user needs to transform the data into the MEDS format before using the package. Even if it is a easy thing to do, it requires preprocessing from the user in order for their package to be dataset-agnostic, a fact they acknowledge in their limitations.
- I find that the max of 500 subjects is not a sufficient stress test. I think more comparisons for larger number of subjects is needed to see the capacity of the subject or an explanation is needed of why this is not possible.
- CaTabRa software, that the authors compare their method to, offers explainability as an output of the package, but the authors chose not to compare or state anything about this property of their method.
Questions for authors
- Why the authors chose not to compare their package with other packages tailored for healthcare data like FIDDLE, FEMR, OHDSI? I would like to see the respective comparison with them or an explanation of how the proposed package is different and why a direct comparison is not applicable.
- I would like to see an experiment to show the strength of sparsity compared to more dense representations, which in my understanding is the main contribution of the proposed tabularization method. Is it only used to save space and time and parallelize jobs?
- I would like to see more stress tests for larger number of subjects or at least explain what do they expect to see as the number of subjects increase at the scale of thousands.
- How does it compare to other methods based on explainability? Its something the authors don't address where other packages care about. Is there a trader-off between MEDS-TAB and explainability?
4 comments
| Method | Leukocytosis | ICU mortality | Resp. failure | Vasopressors | Circ. failure | Ventilation | AKI | Hyponatremia | Thrombocytopenia | LOS | 90-day mortality | 30-day readmission |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MEDS-Tab | 0:01:40 | 0:02:06 | 0:02:00 | 0:02:08 | 0:02:26 | 0:02:32 | 0:02:36 | 0:01:31 | 0:02:09 | 0:07:05 | 0:10:21 | 0:10:23 |
| FEMR | 0:04:26 | 0:07:42 | 0:03:28 | 0:04:01 | 0:06:27 | 0:06:50 | 0:07:23 | 0:03:32 | 0:03:42 | 0:23:49 | 0:22:03 | 0:21:45 |
| OHDSI | 0:02:11 | 0:03:09 | 0:02:52 | 0:03:12 | 0:03:07 | 0:03:41 | 0:04:04 | 0:04:45 | 0:05:55 | 0:18:01 | 0:25:08 | 0:25:20 |
| CaTabRa | 5:55:28 | 10:43:03 | 8:23:49 | 9:54:52 | 10:01:07 | 11:22:29 | 13:37:18 | 7:11:50 | 7:59:39 | 15:19:58 | 9:56:09 | 9:59:21 |
| FIDDLE | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
| tsfresh | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
| Featuretools (code/count) | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
And peak memory usage (GB):
| Method | Leukocytosis | ICU mortality | Resp. failure | Vasopressors | Circ. failure | Ventilation | AKI | Hyponatremia | Thrombocytopenia | LOS | 90-day mortality | 30-day readmission |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| MEDS-Tab | 1.11 | 1.41 | 1.18 | 1.22 | 1.1 | 1.31 | 1.07 | 0.88 | 1.2 | 2.76 | 3.43 | 3.49 |
| FEMR | 2.47 | 3.12 | 4.9 | 5.23 | 3.1 | 3.14 | 3.16 | 2.72 | 5.79 | 8.69 | 14.77 | 14.73 |
| OHDSI | 6.2 | 6.32 | 6.43 | 7.55 | 7.42 | 8.19 | 7.26 | 8.15 | 11.53 | 23.09 | 42.48 | 38.04 |
| CaTabRa | 12.79 | 17.8 | 17.46 | 19.54 | 19.51 | 23.2 | 26.21 | 36.33 | 47.39 | 115.65 | 163.48 | 180.67 |
| FIDDLE | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
| tsfresh | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
| Featuretools (code/count) | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM | OOM |
MEDS-Tab achieves the shortest feature-generation runtime and lowest peak memory in every completed comparison, while constructing a broad set of features across multiple lookback windows. Its default workflow computes six temporal aggregations (code count, numeric-value count, sum, sum of squares, min, and max) over seven windows: 2 hrs, 12 hrs, 1 day, 7 days, 30 days, 365 days, and full history. Together with static presence features, these form 43 feature blocks. We use the default tabularization workflows provided by competitors (details are found in Appendix H):
- FEMR: performs code/count aggregation on a full look back window.
- OHDSI: Computes demographic features, clinical-concept code presence indicators over 30-day and 365-day lookback windows, measurement-range categories, and comorbidity scores.
- CaTabRa: Computes the following aggregations over a 48 hour lookback window: value/min, value/max, value/count and additionally uses mean, standard deviation, first, and last values and first and last observation times, and the 10th and 90th percentile values. FIDDLE, tsfresh, and Featuretools could not complete the evaluated pipelines because they ran out of memory (OOM). Across tasks the 12 tasks, FEMR generated a min and max of 67,457–131,797 features respectively, OHDSI 36,179–76,870, and CaTabRa 20,933–23,287, while MEDS-Tab generated 547,822–1,002,892 features across task. Even though it provided over 10 times more features than any other method, MEDS-Tab was consistently the fastest and most memory efficient method for every task. We additionally see strong AUROC improvements from this wider set of features which we discuss in detail in CfQN.W2/Q3.
We additionally stress-tested MEDS-Tab on the full MIMIC-IV cohort of 364,627 patients, containing 881,400,075 observations and generating 95,603,126 patient–timestamp feature vectors. MEDS-Tab efficiently tabularized these and scales to the full dataset with modest memory usage and blazing speed (we only ran this experiment one time so there is no standard deviation):
| Lookback window | Aggregation | Feature matrix dimensions (rows × columns) | Runtime (min:sec) | Peak PSS (GB) |
|---|---|---|---|---|
| 30 days | code/count | 95,603,126 × 148,191 | 3:39 | 129.08 |
| 30 days | value/count | 95,603,126 × 2,385 | 0:50 | 51.53 |
| 30 days | value/max | 95,603,126 × 2,385 | 1:47 | 54.8 |
| Full history | code/count | 95,603,126 × 148,191 | 7:11 | 199.99 |
| Full history | value/count | 95,603,126 × 2,385 | 0:59 | 64.66 |
| Full history | value/max | 95,603,126 × 2,385 | 2:51 | 68.01 |
| No window | static/present | 364,627 × 2† | 0:18 | 17.19 |
Note that each of these aggregations takes under 10 minutes on the full dataset, even when generating nearly 100 million feature vectors.
Q2: What do sparsity, sharding and parallelism contribute?
I would like to see an experiment to show the strength of sparsity compared to more dense representations, which in my understanding is the main contribution of the proposed tabularization method.
Is it only used to save space and time and parallelize jobs?
Thank you for your feedback. We added a direct ablation of MEDS-Tab with and without sharding. We took a random sample of 10 thousand patients, and we compared processing them in 1 shard vs partitioning them into 16 separate shards, and having 1 worker vs having 24 parallel workers (on a machine with 24 cores). We profiled memory usage and runtime for generating the following aggregations, when varying sharding and parallelism. The features generated are:
- At Seven lookback windows: 2 hours, 12 hours, 1 day, 7 days, 30 days, 365 days and full history.
- Compute Six temporal aggregations (at every lookback window): code/count, value/count, value/sum, value/sum_sqd, value/min and value/max.
- Additionally we compute Static/present features, computed once per shard.
Thus, we generated 43 feature matrices totaling 179,615 features across 2,670,555 unique patient–observation-time pairs, thus a 2,670,555 x 179,615 feature matrix. We observed the following run times and memory profiles as we varied sharding and parallelism
| Shards | Workers (Parallelism) | Time (m:ss) | Avg PSS (GB) | Peak PSS (GB) |
|---|---|---|---|---|
| 1 | 1 | 16:46 | 12.43 | 75.09 |
| 1 | 24 | 4:55 | 183.38 | 299.06 |
| 16 | 1 | 28:35 | 0.33 | 2.65 |
| 16 | 24 | 3:46 | 15.15 | 39.97 |
We observe that parallelism significantly reduces runtime and memory usage, and sharding reduces memory usage but increases runtime. Combining both we nearly half memory utilization while achieving a 4.45× speed gain. More details are in Appendix I.
We want to note that the purpose of Table 1 is to demonstrate the speed benefit from sparsity. In that experiment we compare MEDS-Tab to Catabra and TSFresh for a set of 10, 100, and 500 patients, all of which are loaded as a single shard in these experiments, so efficiency gains are precisely from MEDS-Tab having an efficient sparse implementation, while Catabra and TSFresh rely on dense representations for computing these aggregations. We expanded these results to include the additional baselines that were provided.
We evaluate runtime and memory profiles across multiple aggregation functions at the full and 30 day lookback windows, and we see consistent efficiency gains from MEDS-Tab. We show more results in CfQN.W1/Q1, and here we focus on comparing MEDS-Tab's sparse vs alternative dense aggregation implementations. Catabra, TSFResh, FIDDLE, and featuretools use dense representations, while OHDSI forms aggregations as SQL queries (which use sparsity techniques internal) and FEMR uses a sparse representation, so we exclude those two from this comparison. We compare speed ups at 10 and 100 patients of MEDS-Tab's sparse aggregation (using a single shard and worker) against the dense approaches and we observe the following speed and memory improvements vs dense approaches:
We provide a summary table below of how many times faster MEDS-Tab is than the fastest alternative. CaTabRa is the fastest dense alternative for all three aggregations at 10 patients and for value/count and value/max at 100 patients. FIDDLE’s code/present is the fastest alternative to code/count at 100 patients (providing a binary indicator of whether a code occurred rather than the count) under both lookback windows. Results are below:
MEDS-Tab runtime in seconds (speedup over the fastest matching dense competitor)
| Patients | Lookback window | code/count* | value/count | value/max |
|---|---|---|---|---|
| 10 | 30 days | 0.93 (31.21×) | 0.93 (3.69×) | 0.93 (3.76×) |
| 10 | Full history | 0.93 (36.38×) | 0.92 (4.02×) | 0.93 (4.02×) |
| 100 | 30 days | 1.03 (244.69×) | 0.98 (35.21×) | 1.00 (35.48×) |
| 100 | Full history | 1.09 (358.94×) | 0.99 (39.30×) | 1.01 (38.65×) |
Across these aggregations at 100 patients, MEDS-Tab is 35.21–358.94× faster than the fastest dense alternative across the evaluated temporal aggregations and windows.
We also provide a comparison of how many times higher the peak memory was of the best alternative (lowest memory using alternative). FIDDLE’s code/present also uses the least peak memory among dense alternatives to code/count at both cohort sizes. For value/count and value/max, CaTabRa uses the least memory at 10 patients and FIDDLE at 100 patients. The results are below:
MEDS-Tab peak PSS in GB (lowest-memory matching dense competitor ÷ MEDS-Tab)
| Patients | Lookback window | code/count* | value/count | value/max |
|---|---|---|---|---|
| 10 | 30 days | 0.428 (4.16×) | 0.433 (1.92×) | 0.432 (2.11×) |
| 10 | Full history | 0.431 (4.37×) | 0.416 (1.83×) | 0.429 (1.70×) |
| 100 | 30 days | 0.587 (8.27×) | 0.495 (6.60×) | 0.501 (6.50×) |
| 100 | Full history | 0.653 (8.10×) | 0.520 (6.40×) | 0.523 (6.35×) |
Across these aggregations at 100 patients, the lowest-memory dense alternative uses 6.35–8.27× more peak memory than MEDS-Tab, which requires only 0.495–0.653 GiB.
W2: Scope of dataset independence
In my opinion the dataset-agnostic is an overstatement, since the user needs to transform the data into the MEDS format before using the package.
We thank the reviewer for this comment and would like to clarify our claim. By dataset-agnostic, we mean that the same functional tabularization and baseline fitting pipeline can be run on any EHR dataset, as MEDS-Tab only depends on MEDS format. The limitations section acknowledges that conversion of raw EHR data to MEDS format is a limitation: "Finally, MEDS-Tab requires MEDS format datasets. Converting a dataset to MEDS is relatively simple, and adoption of MEDS format is growing (18; 14), with multiple datasets publicly available in MEDS format and multiple tools aiding in this preprocessing step (18; 3); nonetheless, dependence on this format is a limitation."
W4 / Q4: Explainability
CaTabRa software, that the authors compare their method to, offers explainability as an output of the package, but the authors chose not to compare or state anything about this property of their method.
How does it compare to other methods based on explainability? Its something the authors don't address where other packages care about. Is there a trader-off between MEDS-Tab and explainability?
We thank the reviewer for raising this point, which we did not address in the submission. We do not believe MEDS-Tab introduces a tradeoff with explainability. Every MEDS-Tab feature retains its clinical code, aggregation and lookback window (e.g., LAB//creatinine/7d/value/max). As a result, any feature attribution from the downstream model maps directly to a well-defined clinical summary, such as “the maximum creatinine value in the past 7 days.” Standard attribution methods, such as TreeSHAP for the XGBoost models MEDS-Tab trains, can be applied directly to the resulting predictors. CaTabRa incorporates explanation tools into its workflow, whereas MEDS-Tab provides interpretable features and leaves the choice of attribution method to the user. To facilitate using alternative models or explainability methods, MEDS-Tab also provides a `TabularDataset` class for loading tabular features from disk into a scipy sparse matrix.
Response to Reviewer j1XC
We are encouraged that the reviewer recognizes the need for standardized, reusable EHR featurization. Based on your feedback we expanded the baselines we compare against to additionally include FIDDLE, FEMR, OHDSI, and Featuretools. We provided runtime and memory profiling experiments across significantly higher patient subgroups, and consistently observe that MEDS-Tab can generate more features faster. We stress-tested MEDS-Tab on the full MIMIC-IV cohort of 364,627 patients, containing 881,400,075 observations and generating 95,603,126 patient–timestamp feature vectors. We show that for each benchmarked aggregation MEDS-Tab takes under 10 minutes to generate nearly 100 million feature vectors on the full dataset. We provide an ablation of sharding and parallelism as well, and show that the sparse aggregations MEDS-Tab uses achieve 35.21–358.94× speed improvements and 6.35–8.27× less memory compared to the best dense implementation tested.
W1 / Q1: Healthcare-specific comparisons
We appreciate the reviewer highlighting additional packages that we can compare to. We have now added FEMR, FIDDLE, OHDSI, and Featuretools, alongside tsfresh and CaTabRa. In W3 below we expanded our compute time and memory benchmarks to compare against these methods, and additionally to compare time, memory, and predictive performance on a benchmark of real clinical prediction tasks.
One reason some of these packages were not originally benchmarked was that they do not compute the same features. See in the table below a checkmark indicates the method supported the same aggregation and we observe the same output as MEDS-Tab, while "—" indicates that the method does not support that aggregation, and we note that FIDDLE and OHDSI do not include a code/count aggregation, but provide binary presence indicators 0 or 1 outputs instead so we compare the runtime for those aggregations even though it does not exactly match. OHDSI also did not have value/max or static/present in its default pipeline so we used its last/value and Demographics aggregation methods for comparison.
W3 / Q3: Scaling beyond 500 patients
We thank the reviewer for this comment. As noted in our response to Reviewer CfQN, the computational cost of tabularization depends mainly on the number of feature rows (one per patient–prediction-time pair) and codes, rather than on the number of patients alone. To show how MEDS-Tab scales well beyond the original 500-patient benchmark, we report results on full clinical task cohorts containing up to 218,060 patients and 533,874 feature rows (Appendix H). In our results tables, the columns are ordered from the smallest task (fewest labels) on the left to the largest task (most labels) on the right. The number of labels, and therefore feature rows, in each task is:
We see much better runtime for MEDS-Tab (units are h:mm:ss):