- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors attempts to address the problem of transferring knowledge between EHR models across different institutions under no raw patient records scenario. The method involves training each client's timeline generator separately and uploads it once. The server then samples synthetic timeline to train a global model. The authors conclude that synthetic timelines can greatly improve a small recipients's model but simply pooling doesn't reliably help. However, this work provides only limited comparisons agains its own configurations.
Critical strengths
- This paper identifies a practical and important question: how to transfer clinical knowledge between institutions without pooling records
- The tasks span hospital, ICD and ED care, which is comprehensive. The representation and scoring rules are clear as well.
Critical weaknesses
- There are many possible baselines that the paper did not compare against: DENSE, FedGen, FedAvg... In addition, a simply fine-tune on the shared donor model using recipients' data is good to compare against.
- The experiment is not really across institutions. The authors describe that three 'clients' are random patient shards of one institution. Moreover, the tokenizer and vocabulary are shared. Thus, it introduces leakage across clients and removes the real problem to solve. (which is different subpopulation and clinical practices)
- The evaluation is far from sufficient. The model is a small 3-layer GPT-2 and every datapoint is generated from one dataset. The checks are not sufficient as well, fidelity checks are only code frequency diagnostics. The authors should add temporal checks in addition.
Questions for authors
- How does FTS compare with fine-tuning the shared big generator directly on the recipient’s data?
- How would the results change with different sites, e.g using MIMIC-IV and eICU
- Why does adding the little source’s synthetic data hurt every task? There seems to be a lack of explanation for it.