- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper develops an LLM-based scientific knowledge discovery approach based on the proposed schema. It first extracts evidence-linked functional tuples from scientific literature and then uses the tuples to build scientific knowledge recombination and rank unreported combinations that appear in the future literature. The idea itself sounds plausible, but the methodological justification, novelty, and validation present a few concerns from my perspective.
Critical strengths
- This paper looks at scientific knowledge recombination from a functional perspective, which is interesting and has the potential to help researchers identify combination so established relationships even when experimental evidence is not appearing.
- The study adopts a temporal evaluation strategy by freezing historical features at each cutoff to investigate future literature outcomes. This is a practical and functional design.
Critical weaknesses
- The general idea of extracting structured relations from literature and combing them to identify knowledge gaps is not entirely new. The paper does not sufficiently differentiate its contributions from the existing approaches or establish the novelty of the proposed representation and evidence-linked tuples.
- The main experiments primarily just compare direct LLM generation, single-stage fine-tuning, and relatively simple ranking baselines. The comparison with more recent approaches, particularly knowledge graph link prediction and path-ranking approaches, are needed. Also, more baseline and ablation studies should be in place.
- The reliability of terminology normalization and trajectory assembly is not fully validated, which may lead to error propagation in downstream tasks.
Questions for authors
- Why is the functional schema specifically defined as MS3? Is this choice made based on an established theoretical framework or and empirical analysis of this domain? The authors should explain why these four roles are appropriate and ordered like the current form.
- The proposed framework adopts three supervised adaptors, but individual performance and contributions are not evaluated, reporting this is important for the initial extraction and normalization stages, as errors introduced early can propagate into following analyses.
- How were the 100 papers used for initial annotation and 200 evaluation papers samples? The authors also need to provide inter-annotator agreement stats and additional robustness analysis. Also, although I understand that annotation is a time and resource consuming task, the small number of samples may not be significant enough to justify and generalize the findings.
- Given that future publication is used as the outcome for evaluating candidate trajectories, how do the authors distinguish literature evidence from real scientific feasibility in practical use?