- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the SCT-Bench which is a benchmark used for zero-shot external-control construction in single-arm clinical trials. 9,163 oncology trials in seven tasks are included in the benchmark. TAEC is also proposed in this paper. It is LLM framwork with skill library that chose the trial-specific construction elements, partial optimal transport for patient weighting, and outcome-free verification to improve the construction. TAEC is tested using MarketScan and MIMIC-IV and shows control-outcome and treatment-effect errors lower than several baselines. Other experiment like component ablations and time-zero transfer to existing statistical pipelines are also included.
Critical strengths
-
This is a meaningful work in the field of clinical trial evaluation. TAEC considers trial-specific rather than focusing on patient matching. SCT-Bench also provides several tasks for evaluating external-control construction.
-
Generally, the experiment shows the evidence for the designed framework. Multiple baselines and component ablations are evaluated in this paper.
Critical weaknesses
1.The origin of the gains is not clear: The authors do not show the frequency of TAEC use for each outcome readout as well as the median-only test. As a result of that, some gains have the possiblility that comes from the change of readout. Additionally, MarketScan measures follow-up duration, not time to death. So the comparsion with trial outcomes is not very clear.
2.The time-zero and baseline comparisons are not sufficient enough. The median-ratio adjustment is not directly compared with resetting each patient’s follow-up start. Seven of eight confidence intervals in Table 13 include or touch zero. A simple rule performs better than TAEC on claims subgroups (28.3% vs. 33.4% APE). However, it is not included in the main comparisons.
Questions for authors
1.I wonder how often does TAEC use each outcome readout? Does the conclusion still hold when using the median only? Do the endpoints of MarketScan and the randomized references is the same?
2.Could you compare TAEC with the deterministic rule on the main tasks? How does the median-ratio adjustment compare with directly calculating follow-up from each patient's selected start time? Also, what is the treatment for abstentions?