- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors proposed an information theoretic framework for guiding LLM agents in scientific exploration. While traditional RAG methods have Markovian memory their proposed graph memory mechanism permits a sequential accumulation of evidence. To evaluate their approach the authors construct a custom TCGA benchmark for binary classification and compare their methods against several other popular agentic frameworks as well ablations of their own method.
Critical strengths
The problem of studying scientific exploration with information theoretic approaches for agents is timely and well-motivated.
Critical weaknesses
The chosen benchmark seems very atypical for evaluating information retrieval and scientific exploration.
The method seems like a straightforward application of standard information theoretic approaches.
The reporting of the results is very confusing and I am not convinced the method achieves SOTA, as well as a poor overall performance of all methods on the benchmark.
The authors only evaluate their approach using a single LLM backbone on a single benchmark.
Questions for authors
Could the authors please run additional benchmarks for the approach including more common benchmarks for information retrieval/scientific exploration such as SciGym explain why their setting is more suitable.
Why were 15 tumour types excluded and why is the number of samples capped at 200?
What were the criteria for manual selection of the negatives per type?
Can the authors please report AUROC and AUPRC.
I am very confused about the authors intention when showing Table 2 as under the shown numbers AlphaApollo is the best method but the text argues extensively why the metric is flawed so why include it at all?
Can the authors please run a more sensitivity analysis of the hyperparameters in the ablations as opposed to deactivated a certain component fully. Since the results if any of the terms are removed are random there is a concern that the current system is just overfitted on the benchmark. Furthermore, is the development set excluded from the metrics being reported?
Can the authors please test a general classifier e.g. XGBoost for the task with cross validation to establish how much predictive information is available in the dataset as a potential upper bound.
Section 2.4 the authors state that traditional EIG requires a generative model but then proceed to use a LLM for that task. Could you please elaborate the novelty in this section.
All the experiments use bulk RNA settings, which inevitably mixes tumout types and surrounding tissues and as such is ill suited for the task at hand. Can the authors please run experiments with scRNA sequencing methods and discuss the limitation of their current experiments more candidly.
Minor Comments: Figure 1 is very small please make it full width