- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Hospitals want to train models that generate synthetic patient records, but they cannot share their data. With federated training, there is a trade-off: a model can fit one hospital's patients well (personalization) or work well for all hospitals together (generalization). Training each hospital alone gives the best personalization; pooling all data gives the best generalization. The authors draw a straight line between these two extremes (the "frontier") and show that the standard approach (FedAvg plus fine-tuning) lands right on it.
Their method, FEDLIBRA, tries to beat this line. It (1) adds a penalty (IRM) so the shared model learns patterns that hold at every hospital, uses a best-of-K training trick so different parts of the model can specialize to different types of patients, and (3) fine-tunes a small adapter for each hospital. Experiments use eight hospitals from the eICU dataset and the HALO generator. Quality is mainly measured by how well the frequency of medical codes in synthetic data matches the real data.
Critical strengths
-
The paper draws a line between training each hospital alone (best for that hospital) and pooling all data (best for everyone), and shows that the standard approach lands right on this line. This turns a vague trade-off into a clear, testable target that other work can adopt, and it is the main reason I would like to see the paper accepted once the problems are fixed.
-
By reporting R-squared alongside Pearson correlation, the paper shows that several existing methods get the order of medical code frequencies right but the actual frequencies badly wrong. Pearson correlation alone hides this. This is a real lesson for how synthetic health data should be evaluated.
Critical weaknesses
-
The abstract says the method improves both personalization and generalization, but this holds only on R-squared. On Pearson correlation it is worse than simpler methods on both axes. In addition, the pooled-data reference that defines the frontier was replaced with a modified version after the original performed very badly, which weakens the claim of going beyond the frontier. This is the main reason for my negative score.
-
No repeated runs or confidence intervals are reported, so many of the differences between methods are too small to tell apart from noise. One result is also clearly wrong: training on each hospital’s own real data scores below random guessing (AUROC 0.420 vs. 0.500), which suggests a problem in the evaluation and reduces trust in the results.
Questions for authors
- Can the claim of improving both axes be restated as an R-squared-only claim?
- Why is upweighting used only for the pooled reference? Please retrain all methods with it.
- Please report multiple runs with intervals for FEDLIBRA, FedAvg+FT, FedRep and Ditto.
- Why does local-only training score below chance (AUROC 0.420)?
- Can you add a sequence-level quality measure (e.g. number of visits, code co-occurrence, likelihood on held-out data)?
- How do results and cost change with K? Is there evidence that different latents capture different patient types?
- Why does best-of-K hurt alone but help together with IRM?
- Did IRM actually work? Can a classifier still tell which hospital a sample comes from?
- Why these eight hospitals? What happens with more or fewer, or with another dataset such as MIMIC-III?
- Have you tested attacks on the model updates shared during training?
Reply to Reviewer Zda4
Reply to Reviewer Zda4
We thank Reviewer Zda4 for recognizing the value of our personalization–generalization framing and the importance of $R^2$ alongside Pearson $r$.
"Both axes" claims should be restricted to $R^2$ (W1, Q1). Agreed! We revised the Abstract and Results so simultaneous improvement claims are stated explicitly in terms of $R^2$, with Pearson $r$ reported alongside (General Response §4). In our temperature sweep (Section 4, Appendix B.2), one knob moves $R^2$ by 2.94 while keeping Pearson $r$ within 0.03, showing the two metrics are nearly independent here.
Rare-code upweighting was applied only to the pooled reference (Q2). We retrained all baselines with rare-code upweighting and early stopping across three seeds (General Response §1, Table 1), with per-method effects in Appendix A.3.
Missing multi-seed intervals and below-chance Local-Only AUROC (W2, Q3, Q4). Table 1 now reports 3-seed mean ± SD for all methods. In Table 2, adding $\ell_2$-regularized logistic regression (0.606 AUROC) and gradient boosting (0.573 AUROC) confirms the 0.412 RNN score was small-cohort overfitting (142 training patients at the smallest hospital; General Response §3). RNN and logistic-regression entries report 3-draw averages (SD $\le 0.03$); gradient boosting is a single draw.
Missing sequence-level quality measures (Q5). We report codes per visit against the real rate (General Response §2) and added a full code co-occurrence and sequence structure evaluation in Appendix B.4. Co-occurrence is a much harder target than marginal prevalence—every method scores below zero on co-occurrence $R^2$—though FedLibra attains the best generalized co-occurrence $R^2$ ($-0.01 \pm 0.01$ unweighted, $-0.13$ reweighted) and codes-per-visit Wasserstein distance closer to real ($1.25$ vs. $1.60$ real training data, $3.74$ local-only, $3.11$ centralized). Visit counts are near-degenerate in this cohort (median 1 visit per patient at every client; Appendix A.2), so visit-length distributions carry little signal.
Unclear $K$ sensitivity and why best-of-$K$ hurts alone but helps with IRM (Q6, Q7). We use $K=4$, and sweeping $K \in {1, 2, 4, 8}$ (Appendix B.3) shows $R^2$ scores are flat within $0.007$ (inside seed-to-seed variance) while training time scales roughly linearly with $K$ ($3\text{m }34\text{s}$ at $K=1$, $8\text{m }07\text{s}$ at $K=4$
$≈2.3×$, and $14\text{m }05\text{s}$ at $K=8$
$≈4×$) and generation cost is unchanged ($\approx 4\text{m }11\text{s}$). Without IRM, best-of-$K$ over-generates codes (worsening with $K$); IRM anchors the shared representation (General Response §2, Appendix B.1–B.3).
No domain-classifier test for IRM invariance (Q8). Testing a hospital-origin discriminator on learned representations is a direct invariance probe that we plan to include in the camera-ready.
Hospital selection and lack of MIMIC-III results (Q9). The 8 eICU hospitals span a 13× cohort-size range (202–2,617 patients) across teaching and non-teaching centers, with 8× variation in documentation depth uncorrelated with size (Appendix A.2, Table 5). Compute limits prevented adding MIMIC-III during the rebuttal window, though the framework is generator- and dataset-agnostic.
No evaluation of attacks on shared model updates (Q10). This is an interesting direction! Our privacy evaluation focuses on output-level memorization (NNAAR and MIA on synthetic records), following prior synthetic EHR work. While formal differential privacy is often very punitive in clinical generation, evaluating empirical attacks on shared model updates ($\Delta \theta_k$) is a practical and important complement. In particular, we can consider update-inversion attacks (whether a server can reconstruct multi-hot visit vectors or rare diagnosis codes from client updates) or update-level membership inference (whether parameter deltas reveal individual patient participation, especially at small hospitals). We now note update-level attack evaluation in Section 6 (Limitations and future work) (alongside Section 1).