- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes PORTER, a summarize-then-embed pipeline that uses a clinical schema to guide an LLM to produce textual summaries stripped of site-specific variation, followed by site-adaptive fine-tuning. Transfer is evaluated on cross-site laboratory test prediction and forecasting tasks, and a few-shot experiment is used to argue for the effect of policy distillation.
Critical strengths
The paper correctly attributes cross-site performance degradation to shifts in the data-generating process rather than to downstream model capacity, and the resulting summarize-then-embed framework is a clearly motivated and reasonable design direction.
Critical weaknesses
The transfer evaluation is organized over the thirty task columns of the six transfer directions, and the paper does not verify that the compared quantities share clinical semantics across sites, nor does it provide any same-site (source=target) reference against which the transfer cost could be judged; the central transfer claim therefore cannot be assessed.
Moreover, the forecasting task includes the values to be predicted in its own input, making it a reconstruction exercise rather than forecasting, and the gains in the few-shot experiment cannot be separated from ordinary supervised target adaptation.
Questions for authors
What is the clinical justification for the column-position alignment across sites? Does the forecasting input contain the target window? Can you report a same-site (source=target) upper bound to quantify the true cost of transfer?
4 comments
Please let us know if you have any follow-up questions to our response. We will be glad to clarify before the discussion closes on Nov. 18.
Thank you for your time and effort!
Thank you for the response, which addresses most of my original concerns. However, the magnitude and consistency of the gains under name-matched evaluation remain unclear, as does whether the few-shot gains specifically reflect learned site policies. I have raised my rating from 1 to 2, although I still find the evidence insufficient to support acceptance.
Thank you so much for taking the time to follow up and for raising your score, we truly appreciate your reassessment and support! We are glad to clarify these two remaining points:
Re: name-matched evaluation
We’ve already put the name-matched transfer result in Appendix H in the revised manuscript. For the name-matched (feature aligned) evaluation results:
On the forecasting task: PORTER's MSE is lower than all other prior methods. Compared with the three text-based (Record2Vec, Template and Markdown) baselines, PORTER’s MSE is lower than them by a median of 1% to 4%. Compared with other non-text baselines, PORTER’s MSE is lower than them by 2% to 25% taken over all summarizers, heads and directions. This advantage holds under any choice of summarizing LLM which shows that PORTER consistently outform prior works under the name-matched experiment setup.
On the lab task: name-matching experiment setup does not affect transfer results for all the methods which also shows the consistent performance of PORTER compared with other methods.
The above clarification is shown in our appendix and shows that PORTER provides consistent gain for transfer learning tasks.
Re: few-show gains
On the few-shot gains, we tested different ablations of the policy vector and found that our distilled policy works better than all other ablations (random, false site, learned constant). This suggests that the distilled policy vector captures latent site policy affecting the outcome. We included these results in Figure 4 and Table 24 of the manuscript.
- We tested a random vector, a learned constant vector (requested by Reviewer
Qzt7), and a false policy vector for few-shot ablations. - We found that the distilled policy helps the most on the lab task, and using the other three ablations resulted in significantly decreases in performance.
Again, we sincerely thank the reviewer’s feedback and comment on how we have addressed your original concerns and raised your rating accordingly.
Response to Review
[W2] "forecasting task ... adaptation"
We appreciate the reviewer pointing this out. To clarify, our actual pipeline does not perform reconstruction, and no target information is leaked into the forecasting input. In both our code and all reported experiments, the model takes the embedding of the first 48 hours and forecasts the subsequent, non-overlapping 24-hour window (24×F). The description resembling a reconstruction setup in Appendix D.1.3 was strictly a manuscript writing mistake. All experimental numbers remain unaffected, and we have corrected Appendix D.1.3 and Table 3 in the revised manuscript.
[Q1] What is the clinical justification for the column-position alignment across sites?
We thank the reviewer for this question. Comparing the targets by position is an evaluation choice rather than a clinical claim. Each site keeps its own variable list, as a deployed model would face at a new hospital, and the evaluation does not have to decide which site-specific variables count as the same; recent work likewise bridges differences between sites' vocabularies in the representation, reducing the need to harmonize them (Hegselmann et al., 2026; Kirchler et al., 2026; Guo et al., 2026). The same targets are used for every method, so the comparison between methods stays fair. We further repeated the evaluation with the targets matched by name on the variables shared by all three sites. The conclusions hold: in the setting of Table 1, the lab ranking is unchanged, and on forecasting the 24 hours after the input window, PORTER moves from second to first.
[Q2] Does the forecasting input contain the target window?
We thank the reviewer for catching this writing error in our manuscript. No, the forecasting input does not contain the target window. The input is the embedding of a stay's first 48 hours, and the output and target are all F charted variables for the next 24 hours, a 24×F matrix that does not overlap with the input. The MSE is computed over the observed values. Our experiments, results and code all follow this definition, and the error was only in the writing of Appendix D.1.3, which described the target as the input window. We have corrected Appendix D.1.3 and Table 3.
[W1 Q3] "transfer evaluation ... cannot be assessed. Can you report a same-site (source=target) upper bound to quantify the true cost of transfer?"
We thank the reviewer for raising this. We checked each target against the data at every site. Drugs and labs are defined the same way at all three sites. The lab and forecasting targets are, however, aligned by column position, so we also scored them with targets matched by variable name across the three sites. The conclusions do not change, as shown in the new Appendix H.
For the same-site upper bound, we trained the same pipeline with each site as both source and target, using the same settings as the transfer runs, and tested it on the same test rows as every transfer into that site. On mortality and drug, where the two settings use the same targets, every method except the controls stays below this bound after transfer, averaged over the target sites. In the setting of Table 1, transfer costs PORTER about 4% of its same-site AUROC on mortality and 12% on drug, against 9% to 10% and 18% for Template and Markdown, the strongest methods within a site. Within a site and averaged over the three sites, PORTER is the best method on drug and forecasting in that setting, and on drug in twelve of the sixteen summarizer and head settings. The new Appendix I reports these in-distribution results.