acceptodds
Under review as a conference paper at ICLR 2027

Can Broad Biomedical Knowledge Be Contextualized into Scenario-Grounded Knowledge?

Abstract

Biomedical discovery often requires reconciling broad biomedical knowledge with specific experimental or clinical data. While background knowledge suggests useful biological mechanisms, it is typically too general to map directly onto concrete dataset variables. Conversely, data-driven patterns are often dataset-specific and lack mechanistic insight, creating a gap between abstract principles and concrete evidence. We formulate the bridging of this missing link as knowledge contextualization: transforming broad biomedical knowledge into evidence-supported, scenario-grounded propositions, with the goal of allowing domain experts to efficiently inspect, replay, and validate candidate hypotheses. To achieve this, we propose SCENE, a bi-level multi-agent framework that implements knowledge contextualization as an iterative search process. The upper level translates broad knowledge into search directions and maps them onto the dataset schema. The lower level then executes these grounded directions using multi-objective optimization to find concrete propositions that balance evidential strength with sufficient data support. A feedback loop between the two levels progressively refines these search directions. We evaluate SCENE in two distinct settings: discovering patient subgroups with heterogeneous treatment benefits in clinical trial scenarios, and identifying context-specific biological responses in LINCS L1000 studies. In clinical trial scenarios, SCENE outperforms existing baselines by discovering highly specific, strongly supported subgroups. Furthermore, SCENE uniquely enables the discovery of perturbational contexts with strong target-response matching and high positive rates in LINCS L1000 studies. These results demonstrate that SCENE effectively bridges the gap between broad knowledge and scenario-specific evidence, providing traceable and inspectable hypotheses for follow-up validation.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
knowledge contextualization, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QZKfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper defines "knowledge contextualization" as converting broad biomedical prior knowledge into scenario-grounded propositions, each made of a search direction, an executable conjunctive rule, and an evidence record. The method, SCENE, is a bi-level loop. LLM agents (Qwen3.5-9B) propose search directions from knowledge records and ground them to dataset features. An NSGA-II-style evolutionary search then optimizes rules of length at most 3 for effect and support, and a deterministic feedback rule links the two levels. The main evaluation is treatment-benefit subgroup discovery on 6 task frames from 2 randomized trials (breast cancer and PCOS), using 50 paired 75/25 splits and a Best-1 rule replayed on held-out patients, against 7 baselines. Further experiments cover context discovery for 3 L1000 target genes and few-shot LLM classification with SCENE "knowledge cards" in a clinical and an L1000 scene. The paper claims the best held-out absolute risk reduction (ARR) in all 6 clinical frames and 100% Best-1 direction consistency.

Critical strengths
  1. Turning broad prior knowledge into executable, testable rules for a specific dataset is an interesting problem framing, and the outputs can be inspected and replayed.
  2. The split protocol is careful: the Best-1 rule is fixed before held-out replay, split IDs are paired across methods, and leakage fields are blocked
  3. The data are real (two randomized trials and LINCS L1000), and the appendix includes targeted controls with paired bootstrap intervals.
Critical weaknesses
  1. The causal reading of the trajectory (TR) frames is unclear. In the TR frames, rules may use on-treatment variables, such as "Higher peak creatinine (cycles 1-4)". A subgroup defined by a variable the treatment can change is no longer a randomized comparison, so the within-subgroup ARR is not a treatment-benefit estimand. This applies to all methods in these frames, and App. B.8 itself says such rules "should be read as predictive subgroup hypotheses". Yet the main text and abstract present all six frames as treatment-benefit subgroup discovery, and SCENE's largest margin is on a TR frame. The claims should be separated by frame type.
  2. The superiority claims go beyond the paper's own evidence. The main text says SCENE outperforms existing methods on B-D-BL and P-M-BL, but the paired 95% CIs for SCENE − MaxTE include zero on both (Table 23), and the main tables give no intervals. SCENE also has lower AUPRC and much lower usable support than MaxTE-Context, which undercuts the claim that it balances effect and support. The abstract reports only the metrics where SCENE wins.
  3. The contributions of the LLM and of prior knowledge are not separated, and leakage is possible. Knowledge records are "statements derived from publications, trial records" (Sec. 3.1), and the PCOS trial's own publication (Wen et al., 2022) is in the reference list, so the LLM may encode results from the same cohort. The LLM backend is also selected using held-out benefit on P-M-TR. In addition, there is no baseline that uses the same evolutionary search and feature library without LLM planning, so the gain from the "contextualization" step over the search itself is unknown.
  4. The evidence is narrow for a general claim, and the 100% direction consistency needs explanation. The clinical results come from 2 cohorts, the L1000 results from 3 genes, and the 50 splits reuse the same patients. Best-1 direction consistency is 100% (300/300), yet the top-5 archive gives 84.9% and held-out subgroups are small. That can happen, but the paper should report the held-out arm sizes and the ordering that defines the SCENE row in the main results.
Questions for authors
  1. What does Table 1 show if rules may use only pre-randomization covariates? How should a reader interpret the TR-frame rules, as treatment-benefit subgroups or as predictive hypotheses?
  2. What does knowledge contextualization add beyond knowledge-guided feature selection followed by rule search? Can you add a baseline with the same evolutionary search budget and feature library but no LLM planning?
  3. Which knowledge records were supplied for each trial, and were any derived from publications on the same trials? Why was the backend selected on held-out results, and do the results hold for the other backends?
  4. What are the held-out arm sizes of the Best-1 subgroups per frame, and can the ordering that defines the SCENE row in Table 12 depend on validation outcomes?
5 comments
AuthorsOpenReview ↗

Response to Reviewer QZKf (1/3)

We thank the reviewer for recognizing the inspectable outputs and careful split protocol.

W1. Interpretation of trajectory-defined subgroups

SCENE's clinical advantage does not depend on treatment-course information. With pretreatment covariates alone, it leads all four controls on every metric under the same endpoints and paired evaluation (Table R2 in Q1 of Part 2). Search also performs worse when we keep the initial LLM plan fixed or remove LLM-generated features and planning. The trajectory analyses additionally generate predictive hypotheses about observed treatment courses (Apps. A.8 and B.8). This serves the paper's stated goal of producing inspectable hypotheses for follow-up validation.

W2. Clinical uncertainty and the L1000 effect-support trade-off

SCENE has the highest held-out ARR point estimate in all six clinical tasks, with statistically significant paired improvements over MaxTE in four (Table 23).

SCENE uses Pareto optimization to balance response strength with evidential support, rather than maximize any single metric. Table R7 directly verifies this design, showing higher response concentration and usable support together with a smaller train-holdout gap than scalar selection. Against MaxTE-Context, SCENE more than doubles strong-response concentration (41.5% versus 18.5%) and reduces the gap from 0.228 to 0.063. This is a trade-off between stronger, more stable response contexts and the higher AUPRC and usable-rule fraction of MaxTE-Context (Table 2 of the paper). The contribution is the joint effect-support search and its demonstrated response and stability gains, which do not require ranking first on every diagnostic.

Table R7. L1000 Pareto-selection ablation, reproduced from Table 2 of the submission.

ConditionConnectivity ↑Strong-Pos. ↑AUPRC ↑Usable support ↑Gap ↓
Full SCENE0.18841.5%0.26831.4%0.063
w/o Pareto0.18418.8%0.27330.0%0.081

W3. Prior knowledge, backend selection, and planning gains

SCENE evaluates trial-related suggestions using discovery data rather than treating recalled conclusions as answers, and held-out results do not select the LLM backend. The closed-book probe and source-removal control address direct answer recovery and explicit-record dependence, respectively (Q3 in Part 3). The matched-library comparison separately establishes the contribution of continued planning beyond feature availability (Table R1 in Q2 of Part 2).

W4. Generality, repeated splits, and direction consistency

Best-1 is ranked and selected entirely on discovery data, without using held-out outcomes. Support checks and effect-support Pareto ranking favor supported effects rather than isolated extremes. Effects that are large relative to their variation across splits can change in magnitude without reversing sign. Figure 9 shows the corresponding high-effect, low-variability profile. Q4 in Part 3 explains the selection order and reports held-out arm sizes for all 300 committed outputs.

SCENE's gains extend across diseases, interventions and clinical endpoints, and from patient-level outcomes to molecular perturbation responses. The six clinical tasks cover breast-cancer chemotherapy schedules and PCOS responses to metformin or acupuncture, using different survival or metabolic endpoints and baseline or trajectory views (Table 11). L1000 instead evaluates target-specific responses in gene-expression perturbation profiles. The same SCENE planning and feedback mechanism improves both settings with each adapter held fixed (Table R1 in Part 2 and Table 2 of the paper). This reuse across distinct scientific settings supports the framework's cross-scenario applicability, rather than merely repeating experiments on similar datasets. SCENE knowledge cards also improve downstream classification in both settings (Table 3). The 50 clinical splits assess robustness to patient allocation within each cohort.

AuthorsOpenReview ↗

Response to Reviewer QZKf (2/3)

Q1. Results using only pre-randomization covariates

SCENE's clinical advantage does not depend on treatment-course information. With pretreatment covariates alone, it leads all four controls on every metric under the same endpoints and paired evaluation (Table R2). Search also performs worse when we keep the initial LLM plan fixed or remove LLM-generated features and planning. The trajectory analyses additionally generate predictive hypotheses about observed treatment courses (Apps. A.8 and B.8). This serves the paper's stated goal of producing inspectable hypotheses for follow-up validation.

Table R2. Pretreatment-only controls across six frames and ten seeds per frame.

ConditionARRP-ARRUsable supportDirection consistency
Full SCENE0.2480.08985.0%100.0%
Initial LLM plan only0.0950.03871.7%68.3%
Shared library, no later planning0.1510.05161.7%68.3%
Shared library, random restarts0.1660.06275.0%71.7%
No LLM features or planning0.0610.02671.7%71.7%

Q2. Planning beyond knowledge-guided feature selection

Continued planning and feedback are a key source of SCENE's contextualization gains, not a redundant step after feature selection. Given SCENE's complete final training-derived features and thresholds, both shared-library controls still fall short under the same evolutionary search and rule-evaluation budget. SCENE reaches ARR 0.263, compared with 0.158 without later planning and 0.169 with random restarts, and leads on all other metrics (Table R1). Search also performs worse with the initial LLM plan fixed or with LLM-generated features and planning removed.

Table R1. Clinical controls across six frames and ten seeds per frame.

ConditionARRP-ARRUsable supportDirection consistency
Full SCENE0.2630.08980.0%95.0%
Initial LLM plan only0.0830.03651.7%66.7%
Shared library, no later planning0.1580.04963.3%73.3%
Shared library, random restarts0.1690.05665.0%83.3%
No LLM features or planning0.0550.04151.7%63.3%
AuthorsOpenReview ↗

Response to Reviewer QZKf (3/3)

Q3. Trial-source records and backend selection

SCENE uses trial-related prior knowledge to guide search, not to certify subgroup findings. The LLM proposes directions, while discovery data determine each rule's effect and support before the selected rule is frozen for held-out replay (Apps. A.5 and A.11).

SCENE establishes subgroup evidence through data-based validation of LLM-proposed hypotheses. We tested Qwen3.5-9B with full citations for the PCOS and breast-cancer trials. None of six responses supplied a verifiable subgroup finding for the relevant treatment comparison.

The supplied records contain trial-protocol context, task guidance and admissibility policies, not published treatment effects or responder-subgroup findings. Their sources are NCT00174655 and NCT02491333 (Supplementary Material, configs/tasks.json), while Wen et al. and Dryad identify PCOS data provenance (App. A.10). Removing these records from all LLM requests retains ARR 0.251 and P-ARR 0.082 with the model fixed (Table R5).

Table R5. Removal of supplied trial-source records.

ConditionARRP-ARRUsable supportDirection consistency
Supplied records0.2630.08980.0%95.0%
Supplied records removed0.2510.08275.0%98.3%

Held-out results are not used to select the benchmark LLM. Figure 7 measures backend sensitivity on P-M-TR under a fixed protocol, with positive mean held-out ARR for the discovery-ranked Top-10 rules from all five LLMs (App. A.14). Qwen3.5-9B has the shortest runtime, while Qwen3.5-27B and DeepSeek-V3.2 give larger contrasts.

Q4. Held-out arm sizes and Best-1 ordering

Best-1 is ranked and selected entirely on discovery data, without using held-out outcomes. SCENE applies validity and minimum-support checks before effect-support Pareto ranking, so a large effect must also meet the discovery-support requirement (Table 7). Effects that are large relative to their variation across splits can change in magnitude without reversing sign. Figure 9 shows the corresponding high-effect, low-variability profile, making the observed direction stability plausible. The 100% rate covers all 300 committed Best-1 outputs, whereas 84.9% covers the broader Top-5 archive (App. A.11). In Table 12, 'first validation row' refers to evaluating the training-selected rule, not selecting a rule by held-out performance. The 100% direction consistency does not result from excluding small held-out subgroups. Table R6 reports arm sizes for all 300 outputs, including those with fewer than five patients in an arm.

Table R6. Held-out arm sizes of the original 300 Best-1 outputs used in Table 1 of the paper.

FrameRunsTreatment n, median [Q1, Q3]Control n, median [Q1, Q3]Minimum n, treatment/control
B-D-BL5053 [39, 68]54 [40, 69]7 / 7
B-D-TR5044 [31, 60]45 [32, 61]5 / 5
P-M-BL5011 [8, 13]10 [7, 13]3 / 4
P-M-TR5010 [7, 12]10 [7, 13]3 / 3
P-A-FPG5011 [8, 13]11 [8, 14]4 / 4
P-A-AUC5010 [8, 13]10 [7, 13]4 / 4
Reviewer_wxjDOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates knowledge contextualization as the problem of converting broad biomedical knowledge into evidence-supported, scenario-grounded propositions. Each proposition consists of a search direction, an executable rule, and an evidence record. The proposed method, SCENE, works at two levels: Upper level: an LLM planner proposes search directions and maps them onto the dataset schema. Lower level: a multi-objective evolutionary search optimizes candidate rules for effect and support. Feedback from the frontier revises the directions in later rounds. The evaluation covers treatment-benefit subgroup discovery in two RCTs (six task frames) and perturbational context discovery in LINCS L1000. It also tests downstream few-shot classification using knowledge cards built from the discovered rules.

Critical strengths
  1. The problem is well motivated. Bridging generic biomedical priors and dataset-specific evidence is a real gap, and treating the output as a proposition with provenance, rather than as a bare rule, is a sensible design choice.
  2. The engineering is careful: a) LLM outputs are treated as proposals that must pass deterministic validation; b) leakage controls are explicit; c) the experiments use paired splits and fix the Best-1 rule before held-out replay; d) every run records a manifest.
  3. The appendix is thorough, and the limitations section is candid about the exploratory nature of the findings.
Critical weaknesses
  1. The contribution of knowledge-guided, closed-loop search is not separated from the effect of a larger search budget. a) SCENE runs a fixed number of rounds T, each with up to 20 generations of evolutionary search. The baselines, by contrast, search once. The value of T used in the experiments is not reported, and its effect is not analyzed. b) Upper-level stagnation triggers a change of direction rather than early termination, and propositions that pass the reporting checks accumulate across rounds (Algorithm 1, line 9). The Best-1 rule is then selected from this union. SCENE may therefore benefit from a larger candidate pool and from repeated restarts, independently of the knowledge guidance. c) I appreciate the controls in Appendix B.9: MaxTE + Grounding, the removal of source records (“No Sources”), and the one-shot proposal control on L1000. However, none of them matches the total search budget. d) A random-restart evolutionary search without the LLM, run for the same number of rounds and generations, would directly test whether knowledge-guided planning outperforms additional search at equal cost.
  2. No direct comparison with simpler strategies that use existing quantitative knowledge. The post-hoc clinical audits rely on explicit risk thresholds (HOMA-IR ≥ 2.5, waist ≥ 80 cm, TG ≥ 1.7 mmol/L). The paper never evaluates these thresholds as predefined rules, as candidate threshold values, or as seeds for the search. I recognize that such thresholds describe risk strata and need not correspond to subgroups with heterogeneous treatment benefit. Precisely for that reason, however, such a comparison would clarify the incremental value of SCENE over simpler knowledge-driven strategies. The paper also does not report how many knowledge records each task uses or what they contain, and the examples given are qualitative statements.
  3. The cross-scenario value of the shared planning and feedback machinery is not demonstrated. The introduction presents scenario-dependent grounding (Challenge 2) as a core difficulty. In SCENE, everything that defines a valid finding in each domain is specified manually in the scenario adapter (Table 5): the effect estimator, the support function, the validity and leakage rules, the candidate grammar, and the split design. Task-specific objectives and constraints are not a problem in themselves; many general frameworks require them. The concern is different. The paper does not show that the shared upper-level planning and feedback deliver consistent incremental value across scenarios: the w/o Upper ablation is reported only on L1000, and the two experiments use different knowledge and different tasks. Nor does the paper quantify the manual effort needed to port the framework to a new scenario. I suggest the authors: a) separate domain-agnostic components from scenario-specific ones; b) report the effort required to build a new adapter, or explore LLM-assisted adapter construction; c) either temper the framing of Challenge 2 or add an experiment that grounds the same prior in both settings. Relatedly, Appendix A.6 states that thresholds come from predeclared grids fit on the training partition. However, the construction rule, the number of grid points per feature, and the sensitivity of the results to grid granularity are not reported. The same applies to the support floors, the controller cutoffs, the endpoint thresholds, and the L1000 filters and blocklists.
Questions for authors
  1. What value of T is used in the experiments? How sensitive are the results to T, and from which rounds do the Best-1 rules come?
  2. How are the threshold grids constructed, and how many grid points does each feature have? How does this granularity compare with that of the baselines?
  3. Can you report a budget-matched random-restart evolutionary search without the LLM, and a clinical-setting ablation that removes the upper level or the feedback loop?
  4. Can you add a baseline that uses guideline or literature thresholds, as predefined rules or as seeds, and report the knowledge records used for each task?
4 comments
AuthorsOpenReview ↗

Response to Reviewer wxjD (1/2)

We thank the reviewer for the careful assessment and concrete suggestions on search budgets, quantitative priors, and cross-scenario evaluation.

W1. Planning gains versus additional search

SCENE's advantage comes from using search feedback to continually revise its search directions, rather than from a larger search budget or repeated random restarts. To isolate continued planning, we gave both shared-library controls SCENE's final training-derived features and thresholds while matching the evolutionary algorithm and rule-evaluation budget, including repeated evaluations across restarts. SCENE leads both controls on all four metrics, reaching ARR 0.263 versus 0.158 without later planning and 0.169 with random restarts (Table R1). Search also performs worse with the initial LLM plan fixed or with LLM-generated features and planning removed.

Table R1. Clinical controls across six frames and ten seeds per frame.

ConditionARRP-ARRUsable supportDirection consistency
Full SCENE0.2630.08980.0%95.0%
Initial LLM plan only0.0830.03651.7%66.7%
Shared library, no later planning0.1580.04963.3%73.3%
Shared library, random restarts0.1690.05665.0%83.3%
No LLM features or planning0.0550.04151.7%63.3%

W2. Fixed clinical thresholds and supplied knowledge

SCENE's gains extend beyond applying established clinical thresholds. The training-selected combination of the requested cutoffs reaches ARR 0.082 versus 0.301 for SCENE, which improves all four metrics (Table R4 in Q4 of Part 2). Q4 also gives the clinical and L1000 knowledge-record contents.

W3. Cross-scenario planning, adapter work, and settings

The shared planning and feedback mechanism improves both patient-level clinical discovery and molecular perturbation analysis beyond what their fixed adapters provide. The clinical shared-library controls and L1000 upper-layer ablation isolate these gains with each adapter held fixed (Table R1 and Table 2 of the paper). Table 5 separates scenario-specific evidence functions and validity constraints from the reusable planner, evolutionary search and feedback controller. Table 9 illustrates how the same metabolic-dysregulation prior maps to patient measurements or L1000 signature summaries. Porting requires data preparation and specification of the schema, admissible features, effect and support functions, and split policies, rather than redesigning the discovery algorithm.

Table 8 specifies controller settings and discovery support floors. Table 11 defines the clinical endpoints. App. A.12 specifies the L1000 filters and blocklists. The same endpoint and assay definitions apply to all methods. Table 18 examines replay reliability under stricter held-out support requirements. Q2 in Part 2 gives the grids, and Table R3 in Part 2 tests grid and budget sensitivity.

AuthorsOpenReview ↗

Response to Reviewer wxjD (2/2)

Q1. Number of rounds and sensitivity to the search budget

Table R3 shows little additional ARR gain beyond the four planning rounds used in our added comparison. Each round contains five search generations, so T = 2, 4 and 6 correspond to 10, 20 and 30 generations in total. On the same paired splits with the five-point grid fixed, ARR is 0.218, 0.263 and 0.265, respectively. At 20 generations with five grid points, Best-1 first appears in rounds 3 to 4 (mean 3.83) and generations 15 to 20 (mean 18.00).

Table R3. Sensitivity to quantile-grid resolution and total search generations.

Quantile pointsTotal generationsARRP-ARRUsable supportDirection consistency
3200.2580.09386.7%95.0%
5200.2630.08980.0%95.0%
9200.2730.09180.0%100.0%
5100.2180.07581.7%86.7%
5300.2650.08678.3%100.0%

Q2. Threshold-grid construction and baseline granularity

Threshold granularity does not explain the added planning gain. The tested grids use equally spaced training quantiles over [0.15, 0.85], with duplicate or inadmissible thresholds removed. All controls in Table R1 in Part 1 share the five-point grid. Threshold grids are fitted on training data and frozen before held-out replay (App. A.11). CURLS uses four thresholds, SIDES up to five per feature, and RuAG up to three Gini-derived cut points. Tree methods learn their splits (App. A.11, Table 13).

Q3. Random restarts and clinical planning ablations

Neither random restarts nor a single initial LLM plan matches SCENE on any of the four metrics (Table R1 in W1 of Part 1).

Q4. Clinical-threshold baseline and knowledge records

SCENE's subgroup search improves on fixed clinical risk rules. It leads on all four metrics in the paired pretreatment PCOS comparison (Table R4). The baseline selects the eligible rule with the highest training ARR among all seven nonempty conjunctions of HOMA-IR ≥ 2.5, waist ≥ 80 cm, and triglycerides ≥ 1.7 mmol/L.

Table R4. SCENE versus fixed clinical thresholds on 40 paired PCOS runs.

ConditionARRP-ARRUsable supportDirection consistency
Full SCENE0.3010.11177.5%100.0%
Training-selected fixed thresholds0.0820.03527.5%67.5%

Each clinical frame receives three records covering trial context, task guidance, and admissibility policies. Each L1000 target receives one study-context record attributed to GEO GSE70138 and two internal policies defining admissible features and excluding shortcuts. The record lists are in configs/tasks.json and src/scene/pipeline.py in Supplementary Material.

I appreciate the authors' thorough and constructive rebuttal, particularly the budget-matched random-restart controls, clinical-threshold baseline, and sensitivity analyses. These additional experiments directly address my main concerns and provide stronger evidence for the contribution of iterative planning beyond increased search effort. I also appreciate the clarification of the knowledge records and scenario-specific configurations.

Some limitations remain regarding the manual effort required to adapt SCENE to new scenarios and the extent to which the gains can be attributed specifically to external biomedical knowledge rather than feedback-guided search.

Nevertheless, I consider these limitations relatively minor in light of the new evidence and am satisfied with the authors' response. I have increased my Informativeness score to 3 and will maintain my positive overall assessment of the paper.

AuthorsOpenReview ↗

Follow-up to Reviewer wxjD

We thank the reviewer for the constructive discussion and positive assessment. We will incorporate the additional controls and sensitivity analyses reported in our responses into the final version. We will also include the clarifications on knowledge inputs and scenario-specific configurations.

Reviewer_HSyEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SCENE, a framework for “knowledge contextualization” with the help of LLM agents and evolutionary search bridging abstract biological knowledge with collected medical data. The agents take the role of planners and critics, to set boundaries and evaluate search on the collected data. The authors demonstrate the effectiveness of SCENE by reporting their findings on various clinical subgroup discoveries and mechanism identification.

Critical strengths
  • The authors created an auditable workflow that translates biological knowledge into executable rules and adaptively updates further search queries based on evidence from the search results. Also, they created deterministic checks to curb LLM format hallucination or goal misspecification.
  • The authors did extensive testing and identified potential blind spots for previous methodologies in finding effective treatments especially when the findings were applied to subpopulation/subgroups with unique identifiers.
  • Lastly the method is valuable because it helps identify cause and effect in a clearer fashion than prior art.
Critical weaknesses
  • I am convinced that LLM hallucination or misinterpretation of biological knowledge is adequately mitigated as no LLM finetuning was reported, nor extra ablation studies were presented to show that agent misspecification is avoided and agent performance is evaluated.
  • The study reports post randomization subgroup selection like peak creatinine-based selection that breaks randomization.
  • Reproducibility concerns arise based on the level of the reported specification available in the publication a link to the exact workflow repository could solve this problem.
  • The authors say that datasets are included and will be released upon acceptance, so we do not get access to these during review, and can hence not verify things. This is a major issue.

Minor:

  • SCENE abbreviation is only expanded in figure 2. caption.
  • The authors do not clearly report statistical significance in the main text.
  • The authors have potentially underreported AI use in the writeup and text formalisation; the repetitive text and meta commentary do not seem to facilitate human understanding, while the text lacks certain points that would help human comprehension. These point to a wider use for text generation compared to the statement.
Questions for authors
  1. Have you considered wider ablation studies for the higher level LLM functions to verify the actual performance benefit this step brings?
  2. How does the LLM part of the framework differ from other agent-based methodologies like Popper (https://arxiv.org/abs/2502.09858) or Co-Scientist (https://arxiv.org/abs/2502.18864)?
  3. What measures were taken to avoid fitting to subgroups that already showed signs of benefitting from the treatments, i.e. identifying individuals that are already on the right trajectory and thus skew true results of the method?
  4. To what extent have the human experts reviewed the results, and how did their expertise cover the biomedical knowledge required for evaluating the LLM panel’s performance?
4 comments
AuthorsOpenReview ↗

Response to Reviewer HSyE (1/2)

We thank the reviewer for recognizing the auditable workflow and the value of grounding biomedical knowledge in executable rules.

W1. Validation of LLM proposals

SCENE checks LLM proposals against the schema and data so that hallucinated assertions cannot substitute for evidence. The controller rejects invalid or leakage-prone inputs and verifies that generated features are executable (Apps. A.2 and A.3). Of all proposed generated-feature instances, 11.6% pass the evidence gate and 6.5% appear in validated rules (Fig. 17). These checks apply to every backend, so changing the LLM does not allow its claims to bypass data-based evaluation. The planning ablations are reported in Q1.

W2. Interpretation of post-randomization subgroups

SCENE's gains do not require selecting patients who already show a favorable treatment trajectory. In the pretreatment-only comparison, SCENE leads all four controls on all four metrics under the same endpoints and paired splits (Table R2 in Q3). The peak-creatinine example illustrates how SCENE grounds a biomedical prior in an executable trajectory hypothesis (Fig. 18 and App. B.8).

W3. Workflow implementation and reproducibility

The implementation is available during review in Supplementary Material, with configurations, README commands, and L1000 download and preprocessing scripts. Apps. A.11 and A.12 specify the evaluation protocols.

W4. Dataset access during review

The submitted Supplementary Material includes the processed clinical datasets under data/clinical/. For L1000, the package provides a download and preprocessing workflow for the public GEO releases GSE70138 and GSE92742: under scripts/, L1000_DATA.md specifies the source files and acquisition instructions, fetch_l1000.py downloads them, and prepare_l1000.py constructs the modeling tables. These resources are already available to reviewers.

W5. SCENE acronym

We will expand SCENE at first mention.

W6. Statistical significance

SCENE has the highest held-out ARR point estimate in all six clinical tasks, with statistically significant paired improvements over MaxTE in four (Table 23). For L1000, paired intervals also support higher connectivity, greater strong-response concentration, and a smaller train-holdout gap than the one-shot LLM control (Table 25). We will summarize these paired-comparison results in Sec. 4.6 (Further Analysis) and cite Tables 23 and 25 explicitly.

W7. AI-use disclosure and presentation

The AI Use Statement on p. 10 discloses language polishing of author-written text and the authors' review of AI-assisted edits. It also identifies LLM use in SCENE and the report-quality evaluation. For readability, we will use the worked example in App. A.4 to make Algorithm 1 easier to follow.

AuthorsOpenReview ↗

Response to Reviewer HSyE (2/2)

Q1. Ablations of upper-level LLM planning

SCENE's advantage comes not only from LLM-selected features, but from using search feedback to continually revise its search directions. To isolate continued planning, we gave both shared-library controls SCENE's final training-derived features and thresholds while matching the search algorithm and number of rule evaluations. SCENE leads both controls on all four metrics, with ARR 0.263 versus 0.158 without later planning and 0.169 with random restarts (Table R1). Search also performs worse with the initial LLM plan fixed or with LLM-generated features and planning removed.

Table R1. Clinical controls across six frames and ten seeds per frame.

ConditionARRP-ARRUsable supportDirection consistency
Full SCENE0.2630.08980.0%95.0%
Initial LLM plan only0.0830.03651.7%66.7%
Shared library, no later planning0.1580.04963.3%73.3%
Shared library, random restarts0.1690.05665.0%83.3%
No LLM features or planning0.0550.04151.7%63.3%

Removing the upper layer in L1000 also reduces strong-response concentration and worsens held-out stability (Table 2 of the paper).

Q2. Differences from Popper and Co-Scientist

SCENE discovers executable patient subgroups and perturbation contexts, a conditional discovery task not directly addressed by the published Popper or Co-Scientist workflows. Popper's sequential tests assess a supplied hypothesis, whereas Co-Scientist's tournaments refine research proposals. These workflows do not directly search for the measurable conditions under which the proposed relationship holds in the current dataset. SCENE addresses that gap by using search evidence to revise both the scientific direction and its grounding in available variables (Sec. 3). This converts a broad idea into a rule whose applicability and evidence can be checked in the given data. The shared-library controls show that this feedback contributes beyond the final features and thresholds alone (Table R1).

Q3. Controls using only pretreatment covariates

SCENE's gains do not require selecting patients who already show a favorable treatment trajectory. In the pretreatment-only comparison, SCENE leads all four controls on all four metrics under the same endpoints and paired splits (Table R2).

Table R2. Pretreatment-only controls across six frames and ten seeds per frame.

ConditionARRP-ARRUsable supportDirection consistency
Full SCENE0.2480.08985.0%100.0%
Initial LLM plan only0.0950.03871.7%68.3%
Shared library, no later planning0.1510.05161.7%68.3%
Shared library, random restarts0.1660.06275.0%71.7%
No LLM features or planning0.0610.02671.7%71.7%

Q4. Human expertise and evaluation scope

A medical doctor assessed all 48 clinical reports using the same four criteria as the two AI evaluators (App. B.8). These include clinical coherence and the sufficiency of the supporting evidence. SCENE receives the highest mean human rating on every criterion (Table 22).

AuthorsOpenReview ↗

Kind Request to Review Our Responses

Dear Reviewer HSyE,

Thank you for your time and thoughtful feedback. We have posted our point-by-point responses, including additional analyses and clarifications. We would greatly appreciate it if you could review these updates when convenient. We hope these responses address your concerns, and we would be happy to provide any further clarification if helpful.

Best regards, The Authors

AuthorsOpenReview ↗

Follow-up on Our Responses

Dear Reviewer HSyE,

We are writing to follow up on our responses to your review. We would appreciate your feedback when you have a chance. If any questions remain, we would be glad to discuss them further.

Thank you for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.