- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents GI-TraceBench, a gastrointestinal oncology benchmark with 10,282 questions. It covers provenance QA, knowledge discovery, and hypothesis evaluation. The authors also propose FindLink. It extracts source-bound premises, builds constrained support plans, and generates answers from supplied evidence. A deterministic compiler checks plan structure and source bindings, but not scientific entailment. On a fixed-evidence test set of 1,545 questions, FindLink improves the balanced family macro score from 67.95 for answer-level supervised fine-tuning to 87.25.
Critical strengths
-
The problem is well motivated. The paper correctly distinguishes tracing individual findings to their sources from establishing whether those findings jointly support a new connection. This distinction is consequential for scientific use of literature-based systems.
-
The shared evidence representation is useful. Linking provenance, discovery, and hypothesis-evaluation questions to Paper, Claim, and Evidence Span records makes it possible to inspect the evidence behind an answer.
Critical weaknesses
-
The reliability of the reference answers is insufficiently established. The construction pipeline derives candidate associations and hypotheses from source records and includes silver-label supervision, but the provided description does not clearly establish the extent of independent expert validation. Traceable sources alone do not establish that a cross-paper association is valid, leaving uncertainty about the benchmark's scientific correctness.
-
The independence of the training and test sets needs stronger evidence. Topic-cluster splitting and checks on identifiers and normalized questions help reduce overlap, but may miss different versions of the same study or semantically equivalent claims across splits. Such overlap could inflate performance and weaken the evidence for generalization to unseen scientific connections.
-
Structural validation does not establish scientific validity. The compiler checks plan structure and source bindings, but explicitly does not verify scientific entailment. A plan could therefore pass validation while combining findings from incompatible study conditions. The described checks alone do not establish whether FindLink reliably detects these invalid connections.
Questions for authors
-
Reference-answer validation: What proportion of candidate associations and hypotheses was independently checked by domain experts, and what were the inter-annotator agreement and error rates?
-
Split independence: Did the leakage checks cover duplicate study versions, overlapping evidence spans, and semantically equivalent claims across training and test sets? How were these checks performed?
-
Scientific validity: Is there a targeted evaluation in which sources and plan structure are valid, but study conditions are incompatible? If so, how accurately does FindLink identify the invalid connections?
-
Experimental fairness: FindLink uses additional supervised adaptation stages and separate decoding budgets for grounding, planning, and answering. Its gains over Default SFT may therefore partly reflect additional training and inference resources. Does FindLink still outperform Default SFT when training and inference budgets are matched?
3 comments
Follow-up on Our Response to Reviewer fXud
Dear Reviewer fXud,
Thank you again for your thoughtful comments. We would like to follow up on our rebuttal, which includes an expanded train–test overlap audit, reevaluation after excluding identified overlaps, and further evidence on scientific-condition assessment and budget controls. We would be grateful for your feedback on these additions and any concerns that remain. Please let us know if further clarification would be helpful.
Thank you for your time and consideration.
Thank you for the authors' detailed response. While the rebuttal clarifies some of my concerns, the current manuscript still suffers from substantial issues regarding experimental rigor and missing details. Given the extent of these issues, I do not believe the paper in its current form meets the acceptance bar of ICLR. Therefore, I will maintain my original score.
Response to Reviewer fXud: Overlap Controls and Scientific Assessment
Reviewer fXud:
Thank you for asking how source overlap and incompatible study conditions affect the evidence for FindLink.
Q1. Reference audit and answer ratings (W1&Q1).
A1: Two reviewers with computing backgrounds checked all reference labels in a 20% sample of each split against the supplied passages and task specifications (Appendix A.2). Their review covered source bindings and whether a candidate’s conditions and uncertainty were preserved. The references were frozen after review. Separate rates for independent biomedical review of associations and hypotheses, pre-adjudication agreement, and label corrections are unavailable.
The answer-rating audit covers four systems on the same 24 questions. Expert re-review found no scores requiring revision. The two-rater records give the following agreement on model answers.
ICC(2,1) remains 0.983 on the 82 non-prefilled cases. This measures reproducibility of answer ratings. Reference-label validation is limited to the sampled audit above.
Q2. Train–test overlap in documents, spans, and targets (W2&Q2).
A2: The original split assigns 42 topic clusters to training and eight to testing. Appendix A.2 checks normalized questions and identifiers for papers, claims, spans, associations, and hypotheses, including nested references. Distinct identifiers can still refer to overlapping source material. Our expanded audit covered the actual supervision for SFT, Premise Grounding, and Structured Reasoning. It found 14 shared source documents and 48 matched span pairs, affecting 157 unique test questions.
We kept predictions fixed, excluded those 157 questions, and then applied a further semantic-target risk filter. The last step removed 399 questions, leaving 989. The table gives BFM (0–100).
On the same 1,388 questions, FindLink leads SFT by 19.62 points under original scoring and 14.20 after uniform failure handling and removal of empty–empty credit for Premise, Relation, and Boundary. On the remaining 989, it leads SFT by 5.08 and w/o B by 2.95. The gains survive the identified overlaps. The audit does not establish that every alternate study version or semantically equivalent claim has been detected.
Q3. Recognizing unestablished connections under incompatible conditions (W3&Q3).
A3: FindLink identifies 43 of 70 unestablished connections (61.4%), compared with 30 of 70 (42.9%) for Default SFT. We added 121 expert-reviewed condition-challenge pairs, comprising 20 cross-paper pairs and 101 single-paper contrasts. The 18.6-percentage-point gain is exact not_established recall under a shared four-class readout. On refuted cases, the counts are 2/51 vs. 26/51; on 20 cross-paper negative cases, exact status accuracy is 5/20 vs. 11/20 (FindLink/SFT).
This decision is separate from the compiler’s checks of structure and source bindings (§4.2). Tables 1 and 11 give complementary results on all 286 Discovery questions. The values below use the original scoring on a 0–100 scale.
w/o B achieves the highest Premise F1. FindLink gains 12.54 Relation, 2.62 Boundary, and 17.69 Sufficiency F1 points over that model. These scores measure recovery of the reference assessment. The condition challenge tests whether the final decision respects the limits of the evidence.
Q4. Performance under controlled token budgets.
A4: FindLink has higher diagnostic BFM at both generation caps. Both methods used Qwen3.5-9B and the same 100 parent training questions. SFT and FindLink processed 9,499,757 and 9,552,027 training tokens, respectively, a difference of about 0.55%. We test 120 balanced questions; each cap covers all generated tokens.
Diagnostic BFM gains are 2.31/2.54 points, computed before rounding. Original scoring favors SFT; both diagnostic bootstrap intervals include zero. Supervision, calls, and decoding constraints still differ.
We would be grateful if you would reconsider your evaluation in light of the gains after source and target exclusions and the improved recognition of unestablished connections.