acceptodds
Under review as a conference paper at ICLR 2027

DeepImagine: Clinical Trial Outcome Prediction via Stepwise Local Counterfactual Imaginations

Abstract

Predicting the outcomes of prospective clinical trials remains a major challenge. Clinical trial outcomes result from complex interactions among experimental factors such as drug interventions, participant demographics, and protocols. Here, we introduce DeepImagine, a framework that predicts target trial outcomes through stepwise counterfactual imagination anchored on historical trials with observed results. Starting from a relevant historical trial, DeepImagine sequentially modifies one differing experimental factor at a time. With each step a large language model (LLM) is posed a local counterfactual: how would the current imagined outcome change with this single perturbation? The updated result is carried forward as the input to the next step, until the historical configuration matches the target more closely, yielding the final prediction. Empirically, DeepImagine consistently outperforms traditional clinical trial outcome prediction methods and direct one-step prediction across several off-the-shelf LLMs, with further gains when multiple imagination pathways, initiated from different historical anchors, are aggregated. We also construct natural counterfactuals augmented with synthetic reasoning traces and train a family of specialized language models, each dedicated to learning one factor's local counterfactual transition. Integrating these learned local operators into DeepImagine yields substantial improvements over general-purpose LLM baselines. Our findings position stepwise counterfactual imagination, distinct from both correlational prediction and explicit structural causal modeling, as a promising direction for clinical trial outcome prediction. All code, training scripts, and evaluation scripts are available at https://github.com/deepimagine-counterfactual/DeepImagine.

Primary area:
foundation or frontier models, including LLMs
Keywords:
counterfactual reasoning, clinical trial outcome prediction, AI for drug development, synthetic data generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gZ4vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies how to use large language models to predict the outcomes of prospective clinical trials. This paper proposes DeepImagine, a framework that starts from a historical clinical trial with observed results and decomposes the differences between the historical and target trials into three factors, including enrollment count, study arm, and outcome measure. At each step, the model changes one factor while keeping the others fixed and predicts the corresponding outcome based on the previous result. This process continues until all three factors have been updated. They also train specialized models using natural and synthetic counterfactual data and aggregate predictions from multiple historical trials to improve performance. Experimental results show that DeepImagine outperforms direct prediction and RAG baselines across several LLMs when using multiple historical anchors.

Critical strengths
  1. The paper decomposes a complex clinical trial outcome prediction task into a sequence of single-factor local predictions, providing an intuitive and interesting approach to this challenging problem.

  2. The authors construct natural and synthetic counterfactual data to train specialized local prediction models, extending the proposed framework from LLM inference to model training.

  3. The paper evaluates the framework across multiple LLMs and provides ablation studies on historical anchors and factor modification orders to further investigate its effectiveness.

Critical weaknesses

My major concerns are about the effectiveness of the proposed method.

W1. Error Propagation. Although Proposition 1 analyzes the greedy approximation error, it mainly concerns the discrepancy between a single greedy pathway and the full model marginal distribution. It does not characterize how errors in intermediate predictions relative to ground-truth clinical outcomes propagate or accumulate across successive steps. The sequential local prediction is central to DeepImagine, but this paper lacks a direct empirical analysis of the numerical reliability of intermediate predictions and their error propagation.

W2. Limited Validation in More Complex Clinical Settings. I understand that the authors acknowledge that considering only three factors is a limitation of this work. However, real-world clinical trials involve many more complex and interacting factors. Given the potential error propagation in sequential predictions (as discussed in W1), it remains unclear whether the framework can maintain reliable performance under more complex trial conditions with additional factors. The current experiments provide limited evidence in this regard.

Questions for authors

Could the authors provide further analysis of the reliability of sequential local predictions to demonstrate the effectiveness of the proposed framework under more complex clinical trial conditions?

Reviewer_YZJBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DeepImagine, a method for predicting clinical trial outcomes. Instead of predicting the target result directly, it starts from a historical trial with an observed result. It then changes three factors one at a time (enrollment size, study arm, and outcome measure), carrying the predicted result forward at each step. The authors also fine-tune three Gemma-3-12B models, one per factor. On the 701-question CT Open benchmark, DeepImagine beats direct prediction, RAG, and conventional trial prediction methods.

Critical strengths
  • The main idea of breaking a hard prediction into a chain of single-factor updates is interesting. Starting from an observed historical result and keeping the intermediate results explicit gives a more structured alternative to direct prediction or plain RAG.

  • With seven distinct anchors, DeepImagine beats both RAG and Direct for all three backbones. The trained Gemma operators also come close to GPT-5, which suggests the approach can work with smaller models.

Critical weaknesses
  1. The most important comparison, DeepImagine vs. the fine-tuned one-step baselines in Table 2, is confounded. DeepImagine uses three separately trained Gemma-3-12B models, 274k training tuples, and seven anchors. A1 and A2 use one model, 127k pairs, and at most one anchor. The seven-anchor setup also needs 42 operator calls per question, not counting retrieval and the final comparison. So the jump from 63.2 to 78.7 cannot be credited to factorized reasoning specifically. The authors acknowledge this in Section 5.3, which I appreciate. However, the introduction still interprets this result as suggesting that single-factor transitions are much easier to learn than the joint transition. The single-anchor results in Table 1 provide some support for the benefit of stepwise reasoning, but they also use more model calls than RAG, and the paper does not report token or latency costs.

  2. The numerical validity of synthetic intermediate results is not established. The enrollment operator is trained entirely on GPT-5-generated transitions. The expert review of 90 explanations is helpful, but it checks whether the explanations are plausible. It does not check whether the generated intermediate numbers are correct.

Questions for authors
  1. Can you report factor-specific accuracy on held-out natural transitions for the arm and outcome operators?

  2. For the enrollment operator, can you provide an independent validation of the generated numerical results, e.g., through simulation?

  3. Could you run the one-step Gemma predictor with seven distinct anchors, numerical aggregation of the predictions, and a comparable training and inference budget? This would show how much of the improvement comes from factorization rather than from extra supervision and ensembling.

Reviewer_bpVzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

At its core, this paper presents a new LLM prompting technique (DeepImagine) to predict clinical trial outcomes. Given an existing trial and a new trial, it progressively mutates the existing trial input towards the new trial and asks the model to predict the outcome at each stage.

DeepImagine is evaluated on the CT-Open benchmark (n = 700), where it improves over zero-shot and RAG prompting by ~5% for GPT-5 but with no improvement for o3-mini. The technique also beats supervised baselines such as XGBoost. Finally, the paper also finetunes Gemma3 and achieves comparable performance with GPT-5.

Critical strengths
  • Good motivation. Clinical trials are expensive to run, thus the paper's goal to predict outcomes is interesting and can help stakeholders prior to funding or approving a trial.

  • Leakage-proof evaluation. Prior work has shown that models memorize benchmarks [1]. This paper uses models with cutoffs before the tested trials, which prevents contamination by design.

[1] https://arxiv.org/abs/2402.03927

Critical weaknesses

Please refer to questions. I have phrased each weakness as a question that can be addressed, to avoid duplication.

Questions for authors

1. Missing baselines. Since the gains are small (~5% on an n = 700 dataset size) and there is no gain over o3-mini, please run stronger baselines.

    (i) Agent w/ Web Search. The original CT-Open paper shows gains by simply using an agent with web search. Why was this not tried?
    (ii) Self-consistency. At the same compute budget, simply prompt the Direct version multiple times (with non-zero temperature) and average the answers.
    (iii) Reasoning effort. Increase the reasoning effort? Use the same budget as DeepImagine for fair comparison
    (iv) Newer LLMs. o3-mini and GPT-5) are now over a year old. Please use modern LLMs. If cutoff is an issue, why not use a fresher version of CT-Open?
    (v) 2-stage prompt for Direct and RAG. Currently, DeepImagine predicts the raw outcome for each arm separately, but Direct & RAG produce a yes/no answer directly. The performance gain could simply be coming from a two-step prompt. Concretely:
        a) Prompt for arm 1's predicted result on the target endpoint, e.g. median PFS
        b) Prompt for arm 2's predicted result
        c) Compare the two with the same final step that DeepImagine uses

2. Report accuracy metric and confusion matrix too, alongside macro-F1. On its own, macro-F1 doesn't show whether more questions were answered correctly. It averages the F1 of the two classes with equal weight, so on this imbalanced cohort (442 yes / 259 no) a correct "no" improves the score more than a correct "yes". A method can therefore raise macro-F1 just by predicting "no" more often, without getting any more questions right. For example, a model that trades 96 correct "yes" answers for 96 correct "no" answers still gets 523 right (74.6% accuracy), yet its macro-F1 would rise from 72.4 to 74.6.

3. Does the benchmark really require predicting biology? A trial exists because no one knows whether the drug will work. Yet the LLM reaches ~75 macro-F1. I suspect some of the performance is due to leaked clues. I actually looked at a few samples in CT-Open and ran some experiments to check. First, it seems most questions test an already-validated drug v/s placebo; most such studies result in statistically significant (i.e. "yes") results. For instance, look at Figure 1. It considers Keytruda, which is the most successful cancer drug ever, and has consistently shown strength across cancer types. Most trials with Keytruda v/s placebo would be statistically significant. Conversely, studies that compare doses of the same drug tend to be statistically null.

Finally, for example, on the Superiority split, the single rule "yes if the lead sponsor is industry" scores 73.1 macro-F1 / 74.2% accuracy. That is similar to Direct and RAG and beats every trained baseline in Table 1.

Can you add 2-3 simple rules that checks for these? Please report the macro-F1 stratifie by sponsor type, clinical trial type and drug type (already-validated v/s novel).

4. Step Ablation. The number of counterfactuals is always fixed at 3 (Number of Patients, Arm, Outcome). Please try an ablation where either N, A or O are dropped. Could you also clarify why changing the order (N -> A -> O) drops performance? This part of the paper was unclear.

5. (minor) CT-Open-Superiority has 662 questions, but your paper has 701 qns. Where the additional questions from?

6. (minor) Was the boostrap done on the question-level or the trial-level? In CT-Open, multiple questions come from the same trial, so question-level boostraps would be incorrect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.