- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EviMem+CAVE adds memory to an LLM-based gene-ranking system. It combines verified gene-trait lookups with retrieval of similar past cases. Tests on held-out and cross-ancestry loci show fewer LLM calls and higher accuracy, mainly through reuse of known relationships. The paper also identifies a benchmark shortcut caused by how correct genes are added to candidate lists.
Critical strengths
- Practical savings. The system avoids 31.7% of LLM calls on within-ancestry test loci, with 92.5% accuracy on those cases. This is useful for repeated analyses involving known associations.
- Useful benchmark analysis. The missingness-only baseline shows that models can exploit missing evidence to identify manually added correct genes. This exposes a shortcut that can make benchmark results misleading.
Critical weaknesses
- Limited benefit beyond a simple lookup. The full system achieves 61.0% accuracy versus 61.1% for a gene-identity lookup in Table 1. Nearly all avoided calls come from direct CAVE lookup, while fingerprint retrieval and retrieved hints contribute little. The results therefore do not establish the value of the additional components.
- The cleaner evaluation remains inconclusive. The natural-gold subset contains only 59 loci and uses evidence availability, rather than actual retrieval records, to identify naturally retrieved genes. The accuracy improvement over the memory-free agent is statistically inconclusive on this subset, leaving the benefit without the candidate-insertion artifact uncertain.
Questions for authors
- How does the full system compare with a direct CAVE lookup using the same trait-name key and LLM fallback, but no fingerprint retrieval or hints, in accuracy and calls saved?
- Can the natural-gold subset be checked against actual candidate-insertion records? Does the accuracy gain persist on confirmed naturally retrieved cases?