acceptodds
Withdrawn from ICLR 2027

Zero-Shot External-Control Construction for Single-Arm Clinical Trials

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Oncology single-arm clinical trials (SCTs) evaluate treatments without a control group. External controls constructed from real-world data (RWD) enable comparative evaluation, but require trial-specific elements governing patient inclusion, follow-up initiation, and historical borrowing. Existing methods largely leave these elements to manual specification. Additionally, the absence of a dedicated benchmark has restricted prior evaluations to individual trials or single cancer settings, leaving unclear whether the methods can be transferred to unseen trials. To address these gaps, we introduce SCT-Bench, the first benchmark for this setting, comprising 9,163 clinical trials organized into seven complementary tasks that evaluate protocol extraction, feature ascertainability, control-arm similarity, outcome recovery, survival-curve recovery, treatment-effect recovery, and drug-efficacy recovery. We formulate the SCT evaluation task as zero-shot external-control construction: selecting construction elements for an unseen SCT using only observable evidence. Withheld randomized control arms provide reference outcomes and populations for evaluating both trial-specific element determination and outcome recovery. Such zero-shot construction requires transferring prior methodological knowledge into executable trial-specific actions. Building on this idea, we propose Task-Adaptive External-Control Construction (TAEC), an LLM-based knowledge-to-action framework. Skill-guided planning converts such prior knowledge, together with the target protocol and RWD context, into ten structured construction elements. Data-grounded construction identifies candidate donors and assigns weights through partial optimal transport. Outcome-free refinement verifies the construction without target-trial outcomes, revises unsupported elements and their supporting skills, and updates the skill library within the current target trial. Across SCT-Bench evaluations with two RWD sources, TAEC consistently improves external-control construction and downstream outcome recovery over existing baselines. Further experiments show that transferring TAEC's selected time-zero choice into otherwise unchanged baseline pipelines yields large performance gains. These results identify upstream task instantiation as a major source of generalization beyond estimator choice.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Single arm clinical trial, Zero-shot, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mNkEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the SCT-Bench which is a benchmark used for zero-shot external-control construction in single-arm clinical trials. 9,163 oncology trials in seven tasks are included in the benchmark. TAEC is also proposed in this paper. It is LLM framwork with skill library that chose the trial-specific construction elements, partial optimal transport for patient weighting, and outcome-free verification to improve the construction. TAEC is tested using MarketScan and MIMIC-IV and shows control-outcome and treatment-effect errors lower than several baselines. Other experiment like component ablations and time-zero transfer to existing statistical pipelines are also included.

Critical strengths
  1. This is a meaningful work in the field of clinical trial evaluation. TAEC considers trial-specific rather than focusing on patient matching. SCT-Bench also provides several tasks for evaluating external-control construction.

  2. Generally, the experiment shows the evidence for the designed framework. Multiple baselines and component ablations are evaluated in this paper.

Critical weaknesses

1.The origin of the gains is not clear: The authors do not show the frequency of TAEC use for each outcome readout as well as the median-only test. As a result of that, some gains have the possiblility that comes from the change of readout. Additionally, MarketScan measures follow-up duration, not time to death. So the comparsion with trial outcomes is not very clear.

2.The time-zero and baseline comparisons are not sufficient enough. The median-ratio adjustment is not directly compared with resetting each patient’s follow-up start. Seven of eight confidence intervals in Table 13 include or touch zero. A simple rule performs better than TAEC on claims subgroups (28.3% vs. 33.4% APE). However, it is not included in the main comparisons.

Questions for authors

1.I wonder how often does TAEC use each outcome readout? Does the conclusion still hold when using the median only? Do the endpoints of MarketScan and the randomized references is the same?

2.Could you compare TAEC with the deterministic rule on the main tasks? How does the median-ratio adjustment compare with directly calculating follow-up from each patient's selected start time? Also, what is the treatment for abstentions?

Reviewer_PzGYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses external-control construction for oncology single-arm clinical trials (SCTs) using real-world data (RWD). It argues that existing pipelines automate only downstream matching/weighting while leaving upstream design choices (e.g., time-zero, eligibility, feature selection) to manual specification. The paper introduces two contributions:

  1. SCT-Bench: A benchmark of 9,163 oncology trials organized into seven tasks where randomized control arms are replayed/withheld as reference truth (evaluated on 11 patient-level RCTs and trial-level ClinicalTrials.gov summaries).
  2. TAEC: An LLM-based planner that selects predefined operators for 10 design elements using a skill library, executed via partial optimal transport and an outcome-free verifier. Evaluated on MarketScan and MIMIC-IV, TAEC reports lower control-outcome median absolute percent errors (APE) than classical baselines, with its learned time-zero transfer substantially improving traditional estimators.
Critical strengths
  • Valuable Benchmark Framework: Replaying RCTs as single-arm trials with withheld control arms as ground truth to evaluate both upstream design decisions and downstream outcomes provides a useful open-source benchmark direction.
  • Important Empirical Insight: The time-zero transfer experiments (Tables 13–14) convincingly demonstrate that aligning the follow-up start (time-zero) reduces outcome errors across diverse downstream estimators, proving that upstream design choices dominate estimator selection.
Critical weaknesses
  1. Unjustified LLM Machinery & Omission of Established Paradigm:
    • Rule-Based Sufficiency: The main performance gains stem almost entirely from time-zero adjustment, which reduces to a two-clock ratio determined by the trial’s line of therapy (Appendix B.2, F, Table 16). In fact, a simple deterministic rule (28.3%) outperforms the full TAEC iterative loop (33.4%) on claims strata (Table 15). Furthermore, applying TAEC's time-zero to standard entropy balancing (21.0–21.7%) or IPTW (22.4–24.2%) matches or beats TAEC (24.2%).
    • Missing Target Trial Emulation (TTE) Baselines: The paper ignores the standard non-LLM methodology for this problem: target trial emulation (Hernán & Robins, 2016) and established protocol-driven RCT replications from RWD (e.g., RCT-DUPLICATE, Carrigan et al., 2020). Classical baselines are unaligned strawmen, whereas protocol-aligned TTE practices are never benchmarked against.
  2. Evaluation Bottlenecks & Data Contamination Risks:
    • Small Sample & High Variance: End-to-end patient evaluation relies on only 11 RCTs (10 for primary metrics) without confidence intervals or significance testing. Large swings in MIMIC-IV ablations (41.8% to 183.1%, Table 12) indicate severe variance.
    • Outcome Mismatch: On MarketScan, outcome recovery measures time-to-end-of-enrollment against reported OS, rendering "outcome recovery" ambiguous (especially given TAEC's flexible read-out quantile).
    • Potential Contamination/Leakage: The paper does not guarantee that benchmark trials were excluded when constructing the skill library, nor does it address whether pre-trained LLM backbones memorized control-arm medians from ClinicalTrials.gov.
Questions for authors
  1. Target Trial Emulation (TTE) Benchmark: Can you compare TAEC against a standard Target Trial Emulation baseline where time-zero and eligibility are determined from protocol fields using standard epidemiological rules (e.g., index at qualifying line of therapy start) combined with classical estimators? (Demonstrating clear superiority over TTE is essential to justify the complex LLM planning setup.)
  2. Time-Zero Source Clarification: In the transfer experiment (Table 13), was time-zero selected autonomously by TAEC or manually constructed from protocol history by the authors (as suggested in Appendix E.5)? How does r∗r^* relate to TAEC's selected scale ss (given mean ∣log⁡s−log⁡r∗∣=0.60|\log s - \log r^*| = 0.60 in Table 16)?
  3. Data Contamination & Leakage Checks:
    • What is the source of MpubM_{\text{pub}} in the public-prior quantile operator, and how often is it selected?
    • Were benchmark RCTs strictly excluded when distilling the historical skill library?
    • Have you tested whether the LLM backbones can memorize/recall control-arm median survival for the 11 evaluation trials?
Reviewer_2T3uOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies zero-shot external-control construction for single-arm clinical trials. It introduces SCT-Bench, containing 9,163 trial instances across seven tasks, and TAEC, an LLM-based framework combining retrieved methodological skills, adaptive planning, partial optimal transport, and outcome-free refinement. Evaluation on two real-world data sources reports improvements over statistical baselines.

Critical strengths

The paper addresses an important clinical problem and highlights the influence of upstream design choices, particularly time zero. The benchmark and withheld randomized-control evaluations provide a useful framework for studying these choices.

Critical weaknesses
  1. Transparency is particularly important when combining single-arm trials with external controls. Unmeasured confounding cannot generally be resolved from the observed data alone. Design assumptions therefore require explicit justification and expert scrutiny. Although TAEC produces structured plans, the paper does not establish that its retrieved skills, inferred features, and iterative revisions provide clinically auditable reasoning or support meaningful expert review. Outcome-free verification does not resolve this limitation.

  2. The methodological contribution appears limited to integrating established LLM planning, retrieval, verification, and transport components. The experiments do not clearly establish a generalizable machine-learning advance beyond this application, making the contribution less compelling for ICLR.

Questions for authors
  1. How are consequential design assumptions and unsupported clinical inferences exposed to experts for review? Is there evidence that experts can reliably audit the generated plans?

  2. How does TAEC communicate residual unmeasured confounding and distinguish a supported construction plan from a credible causal comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.