- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper defines "knowledge contextualization" as converting broad biomedical prior knowledge into scenario-grounded propositions, each made of a search direction, an executable conjunctive rule, and an evidence record. The method, SCENE, is a bi-level loop. LLM agents (Qwen3.5-9B) propose search directions from knowledge records and ground them to dataset features. An NSGA-II-style evolutionary search then optimizes rules of length at most 3 for effect and support, and a deterministic feedback rule links the two levels. The main evaluation is treatment-benefit subgroup discovery on 6 task frames from 2 randomized trials (breast cancer and PCOS), using 50 paired 75/25 splits and a Best-1 rule replayed on held-out patients, against 7 baselines. Further experiments cover context discovery for 3 L1000 target genes and few-shot LLM classification with SCENE "knowledge cards" in a clinical and an L1000 scene. The paper claims the best held-out absolute risk reduction (ARR) in all 6 clinical frames and 100% Best-1 direction consistency.
Critical strengths
- Turning broad prior knowledge into executable, testable rules for a specific dataset is an interesting problem framing, and the outputs can be inspected and replayed.
- The split protocol is careful: the Best-1 rule is fixed before held-out replay, split IDs are paired across methods, and leakage fields are blocked
- The data are real (two randomized trials and LINCS L1000), and the appendix includes targeted controls with paired bootstrap intervals.
Critical weaknesses
- The causal reading of the trajectory (TR) frames is unclear. In the TR frames, rules may use on-treatment variables, such as "Higher peak creatinine (cycles 1-4)". A subgroup defined by a variable the treatment can change is no longer a randomized comparison, so the within-subgroup ARR is not a treatment-benefit estimand. This applies to all methods in these frames, and App. B.8 itself says such rules "should be read as predictive subgroup hypotheses". Yet the main text and abstract present all six frames as treatment-benefit subgroup discovery, and SCENE's largest margin is on a TR frame. The claims should be separated by frame type.
- The superiority claims go beyond the paper's own evidence. The main text says SCENE outperforms existing methods on B-D-BL and P-M-BL, but the paired 95% CIs for SCENE − MaxTE include zero on both (Table 23), and the main tables give no intervals. SCENE also has lower AUPRC and much lower usable support than MaxTE-Context, which undercuts the claim that it balances effect and support. The abstract reports only the metrics where SCENE wins.
- The contributions of the LLM and of prior knowledge are not separated, and leakage is possible. Knowledge records are "statements derived from publications, trial records" (Sec. 3.1), and the PCOS trial's own publication (Wen et al., 2022) is in the reference list, so the LLM may encode results from the same cohort. The LLM backend is also selected using held-out benefit on P-M-TR. In addition, there is no baseline that uses the same evolutionary search and feature library without LLM planning, so the gain from the "contextualization" step over the search itself is unknown.
- The evidence is narrow for a general claim, and the 100% direction consistency needs explanation. The clinical results come from 2 cohorts, the L1000 results from 3 genes, and the 50 splits reuse the same patients. Best-1 direction consistency is 100% (300/300), yet the top-5 archive gives 84.9% and held-out subgroups are small. That can happen, but the paper should report the held-out arm sizes and the ordering that defines the SCENE row in the main results.
Questions for authors
- What does Table 1 show if rules may use only pre-randomization covariates? How should a reader interpret the TR-frame rules, as treatment-benefit subgroups or as predictive hypotheses?
- What does knowledge contextualization add beyond knowledge-guided feature selection followed by rule search? Can you add a baseline with the same evolutionary search budget and feature library but no LLM planning?
- Which knowledge records were supplied for each trial, and were any derived from publications on the same trials? Why was the backend selected on held-out results, and do the results hold for the other backends?
- What are the held-out arm sizes of the Best-1 subgroups per frame, and can the ordering that defines the SCENE row in Table 12 depend on validation outcomes?
5 comments
Response to Reviewer QZKf (2/3)
Q1. Results using only pre-randomization covariates
SCENE's clinical advantage does not depend on treatment-course information. With pretreatment covariates alone, it leads all four controls on every metric under the same endpoints and paired evaluation (Table R2). Search also performs worse when we keep the initial LLM plan fixed or remove LLM-generated features and planning. The trajectory analyses additionally generate predictive hypotheses about observed treatment courses (Apps. A.8 and B.8). This serves the paper's stated goal of producing inspectable hypotheses for follow-up validation.
Table R2. Pretreatment-only controls across six frames and ten seeds per frame.
| Condition | ARR | P-ARR | Usable support | Direction consistency |
|---|---|---|---|---|
| Full SCENE | 0.248 | 0.089 | 85.0% | 100.0% |
| Initial LLM plan only | 0.095 | 0.038 | 71.7% | 68.3% |
| Shared library, no later planning | 0.151 | 0.051 | 61.7% | 68.3% |
| Shared library, random restarts | 0.166 | 0.062 | 75.0% | 71.7% |
| No LLM features or planning | 0.061 | 0.026 | 71.7% | 71.7% |
Q2. Planning beyond knowledge-guided feature selection
Continued planning and feedback are a key source of SCENE's contextualization gains, not a redundant step after feature selection. Given SCENE's complete final training-derived features and thresholds, both shared-library controls still fall short under the same evolutionary search and rule-evaluation budget. SCENE reaches ARR 0.263, compared with 0.158 without later planning and 0.169 with random restarts, and leads on all other metrics (Table R1). Search also performs worse with the initial LLM plan fixed or with LLM-generated features and planning removed.
Table R1. Clinical controls across six frames and ten seeds per frame.
| Condition | ARR | P-ARR | Usable support | Direction consistency |
|---|---|---|---|---|
| Full SCENE | 0.263 | 0.089 | 80.0% | 95.0% |
| Initial LLM plan only | 0.083 | 0.036 | 51.7% | 66.7% |
| Shared library, no later planning | 0.158 | 0.049 | 63.3% | 73.3% |
| Shared library, random restarts | 0.169 | 0.056 | 65.0% | 83.3% |
| No LLM features or planning | 0.055 | 0.041 | 51.7% | 63.3% |
Response to Reviewer QZKf (3/3)
Q3. Trial-source records and backend selection
SCENE uses trial-related prior knowledge to guide search, not to certify subgroup findings. The LLM proposes directions, while discovery data determine each rule's effect and support before the selected rule is frozen for held-out replay (Apps. A.5 and A.11).
SCENE establishes subgroup evidence through data-based validation of LLM-proposed hypotheses. We tested Qwen3.5-9B with full citations for the PCOS and breast-cancer trials. None of six responses supplied a verifiable subgroup finding for the relevant treatment comparison.
The supplied records contain trial-protocol context, task guidance and admissibility policies, not published treatment effects or responder-subgroup findings. Their sources are NCT00174655 and NCT02491333 (Supplementary Material, configs/tasks.json), while Wen et al. and Dryad identify PCOS data provenance (App. A.10). Removing these records from all LLM requests retains ARR 0.251 and P-ARR 0.082 with the model fixed (Table R5).
Table R5. Removal of supplied trial-source records.
| Condition | ARR | P-ARR | Usable support | Direction consistency |
|---|---|---|---|---|
| Supplied records | 0.263 | 0.089 | 80.0% | 95.0% |
| Supplied records removed | 0.251 | 0.082 | 75.0% | 98.3% |
Held-out results are not used to select the benchmark LLM. Figure 7 measures backend sensitivity on P-M-TR under a fixed protocol, with positive mean held-out ARR for the discovery-ranked Top-10 rules from all five LLMs (App. A.14). Qwen3.5-9B has the shortest runtime, while Qwen3.5-27B and DeepSeek-V3.2 give larger contrasts.
Q4. Held-out arm sizes and Best-1 ordering
Best-1 is ranked and selected entirely on discovery data, without using held-out outcomes. SCENE applies validity and minimum-support checks before effect-support Pareto ranking, so a large effect must also meet the discovery-support requirement (Table 7). Effects that are large relative to their variation across splits can change in magnitude without reversing sign. Figure 9 shows the corresponding high-effect, low-variability profile, making the observed direction stability plausible. The 100% rate covers all 300 committed Best-1 outputs, whereas 84.9% covers the broader Top-5 archive (App. A.11). In Table 12, 'first validation row' refers to evaluating the training-selected rule, not selecting a rule by held-out performance. The 100% direction consistency does not result from excluding small held-out subgroups. Table R6 reports arm sizes for all 300 outputs, including those with fewer than five patients in an arm.
Table R6. Held-out arm sizes of the original 300 Best-1 outputs used in Table 1 of the paper.
| Frame | Runs | Treatment n, median [Q1, Q3] | Control n, median [Q1, Q3] | Minimum n, treatment/control |
|---|---|---|---|---|
| B-D-BL | 50 | 53 [39, 68] | 54 [40, 69] | 7 / 7 |
| B-D-TR | 50 | 44 [31, 60] | 45 [32, 61] | 5 / 5 |
| P-M-BL | 50 | 11 [8, 13] | 10 [7, 13] | 3 / 4 |
| P-M-TR | 50 | 10 [7, 12] | 10 [7, 13] | 3 / 3 |
| P-A-FPG | 50 | 11 [8, 13] | 11 [8, 14] | 4 / 4 |
| P-A-AUC | 50 | 10 [8, 13] | 10 [7, 13] | 4 / 4 |
Response to Reviewer QZKf (1/3)
We thank the reviewer for recognizing the inspectable outputs and careful split protocol.
W1. Interpretation of trajectory-defined subgroups
SCENE's clinical advantage does not depend on treatment-course information. With pretreatment covariates alone, it leads all four controls on every metric under the same endpoints and paired evaluation (Table R2 in Q1 of Part 2). Search also performs worse when we keep the initial LLM plan fixed or remove LLM-generated features and planning. The trajectory analyses additionally generate predictive hypotheses about observed treatment courses (Apps. A.8 and B.8). This serves the paper's stated goal of producing inspectable hypotheses for follow-up validation.
W2. Clinical uncertainty and the L1000 effect-support trade-off
SCENE has the highest held-out ARR point estimate in all six clinical tasks, with statistically significant paired improvements over MaxTE in four (Table 23).
SCENE uses Pareto optimization to balance response strength with evidential support, rather than maximize any single metric. Table R7 directly verifies this design, showing higher response concentration and usable support together with a smaller train-holdout gap than scalar selection. Against MaxTE-Context, SCENE more than doubles strong-response concentration (41.5% versus 18.5%) and reduces the gap from 0.228 to 0.063. This is a trade-off between stronger, more stable response contexts and the higher AUPRC and usable-rule fraction of MaxTE-Context (Table 2 of the paper). The contribution is the joint effect-support search and its demonstrated response and stability gains, which do not require ranking first on every diagnostic.
Table R7. L1000 Pareto-selection ablation, reproduced from Table 2 of the submission.
W3. Prior knowledge, backend selection, and planning gains
SCENE evaluates trial-related suggestions using discovery data rather than treating recalled conclusions as answers, and held-out results do not select the LLM backend. The closed-book probe and source-removal control address direct answer recovery and explicit-record dependence, respectively (Q3 in Part 3). The matched-library comparison separately establishes the contribution of continued planning beyond feature availability (Table R1 in Q2 of Part 2).
W4. Generality, repeated splits, and direction consistency
Best-1 is ranked and selected entirely on discovery data, without using held-out outcomes. Support checks and effect-support Pareto ranking favor supported effects rather than isolated extremes. Effects that are large relative to their variation across splits can change in magnitude without reversing sign. Figure 9 shows the corresponding high-effect, low-variability profile. Q4 in Part 3 explains the selection order and reports held-out arm sizes for all 300 committed outputs.
SCENE's gains extend across diseases, interventions and clinical endpoints, and from patient-level outcomes to molecular perturbation responses. The six clinical tasks cover breast-cancer chemotherapy schedules and PCOS responses to metformin or acupuncture, using different survival or metabolic endpoints and baseline or trajectory views (Table 11). L1000 instead evaluates target-specific responses in gene-expression perturbation profiles. The same SCENE planning and feedback mechanism improves both settings with each adapter held fixed (Table R1 in Part 2 and Table 2 of the paper). This reuse across distinct scientific settings supports the framework's cross-scenario applicability, rather than merely repeating experiments on similar datasets. SCENE knowledge cards also improve downstream classification in both settings (Table 3). The 50 clinical splits assess robustness to patient allocation within each cohort.