- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies how to use large language models to predict the outcomes of prospective clinical trials. This paper proposes DeepImagine, a framework that starts from a historical clinical trial with observed results and decomposes the differences between the historical and target trials into three factors, including enrollment count, study arm, and outcome measure. At each step, the model changes one factor while keeping the others fixed and predicts the corresponding outcome based on the previous result. This process continues until all three factors have been updated. They also train specialized models using natural and synthetic counterfactual data and aggregate predictions from multiple historical trials to improve performance. Experimental results show that DeepImagine outperforms direct prediction and RAG baselines across several LLMs when using multiple historical anchors.
Critical strengths
-
The paper decomposes a complex clinical trial outcome prediction task into a sequence of single-factor local predictions, providing an intuitive and interesting approach to this challenging problem.
-
The authors construct natural and synthetic counterfactual data to train specialized local prediction models, extending the proposed framework from LLM inference to model training.
-
The paper evaluates the framework across multiple LLMs and provides ablation studies on historical anchors and factor modification orders to further investigate its effectiveness.
Critical weaknesses
My major concerns are about the effectiveness of the proposed method.
W1. Error Propagation. Although Proposition 1 analyzes the greedy approximation error, it mainly concerns the discrepancy between a single greedy pathway and the full model marginal distribution. It does not characterize how errors in intermediate predictions relative to ground-truth clinical outcomes propagate or accumulate across successive steps. The sequential local prediction is central to DeepImagine, but this paper lacks a direct empirical analysis of the numerical reliability of intermediate predictions and their error propagation.
W2. Limited Validation in More Complex Clinical Settings. I understand that the authors acknowledge that considering only three factors is a limitation of this work. However, real-world clinical trials involve many more complex and interacting factors. Given the potential error propagation in sequential predictions (as discussed in W1), it remains unclear whether the framework can maintain reliable performance under more complex trial conditions with additional factors. The current experiments provide limited evidence in this regard.
Questions for authors
Could the authors provide further analysis of the reliability of sequential local predictions to demonstrate the effectiveness of the proposed framework under more complex clinical trial conditions?