acceptodds
Under review as a conference paper at ICLR 2027

FindLink: Learning to Assess Scientific Connections from Source-Grounded Premises

Abstract

Scientific discovery from the literature requires assessing connections among findings obtained under different study conditions. Source attribution makes individual findings traceable, but does not determine what they jointly support. We introduce **GI-TraceBench**, a benchmark of 10,282 questions in gastrointestinal oncology that unifies provenance question answering, knowledge discovery, and hypothesis evaluation through shared *Paper, Claim, and Evidence Span* records. We further propose **FindLink**, a two-stage framework for provenance-grounded scientific reasoning. *Premise Grounding* constructs source-bound premises with explicit study context and support status. *Structured Reasoning* organizes these premises into support plans that represent both supported relations and unresolved conditions, then generates evidence-conditioned answers using deterministic compiler feedback. The compiler verifies structural and source-binding consistency, but does not assess scientific entailment. On the 1,545-question fixed-evidence test set, FindLink improves the balanced family macro score from 67.95 to 87.25 over direct supervised fine-tuning of the same backbone. A retrained variant without Structured Reasoning achieves slightly higher premise-identifier F1, but substantially lower relation and evidence-sufficiency F1. These results indicate that scientific reasoning requires not only identifying grounded premises, but also assessing what connections those premises support.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scientific question answering, evidence-grounded reasoning, provenance attribution, cross-document reasoning, knowledge discovery, hypothesis evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_fXudOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents GI-TraceBench, a gastrointestinal oncology benchmark with 10,282 questions. It covers provenance QA, knowledge discovery, and hypothesis evaluation. The authors also propose FindLink. It extracts source-bound premises, builds constrained support plans, and generates answers from supplied evidence. A deterministic compiler checks plan structure and source bindings, but not scientific entailment. On a fixed-evidence test set of 1,545 questions, FindLink improves the balanced family macro score from 67.95 for answer-level supervised fine-tuning to 87.25.

Critical strengths
  1. The problem is well motivated. The paper correctly distinguishes tracing individual findings to their sources from establishing whether those findings jointly support a new connection. This distinction is consequential for scientific use of literature-based systems.

  2. The shared evidence representation is useful. Linking provenance, discovery, and hypothesis-evaluation questions to Paper, Claim, and Evidence Span records makes it possible to inspect the evidence behind an answer.

Critical weaknesses
  1. The reliability of the reference answers is insufficiently established. The construction pipeline derives candidate associations and hypotheses from source records and includes silver-label supervision, but the provided description does not clearly establish the extent of independent expert validation. Traceable sources alone do not establish that a cross-paper association is valid, leaving uncertainty about the benchmark's scientific correctness.

  2. The independence of the training and test sets needs stronger evidence. Topic-cluster splitting and checks on identifiers and normalized questions help reduce overlap, but may miss different versions of the same study or semantically equivalent claims across splits. Such overlap could inflate performance and weaken the evidence for generalization to unseen scientific connections.

  3. Structural validation does not establish scientific validity. The compiler checks plan structure and source bindings, but explicitly does not verify scientific entailment. A plan could therefore pass validation while combining findings from incompatible study conditions. The described checks alone do not establish whether FindLink reliably detects these invalid connections.

Questions for authors
  1. Reference-answer validation: What proportion of candidate associations and hypotheses was independently checked by domain experts, and what were the inter-annotator agreement and error rates?

  2. Split independence: Did the leakage checks cover duplicate study versions, overlapping evidence spans, and semantically equivalent claims across training and test sets? How were these checks performed?

  3. Scientific validity: Is there a targeted evaluation in which sources and plan structure are valid, but study conditions are incompatible? If so, how accurately does FindLink identify the invalid connections?

  4. Experimental fairness: FindLink uses additional supervised adaptation stages and separate decoding budgets for grounding, planning, and answering. Its gains over Default SFT may therefore partly reflect additional training and inference resources. Does FindLink still outperform Default SFT when training and inference budgets are matched?

3 comments
AuthorsOpenReview ↗

Response to Reviewer fXud: Overlap Controls and Scientific Assessment

Reviewer fXud:

Thank you for asking how source overlap and incompatible study conditions affect the evidence for FindLink.

Q1. Reference audit and answer ratings (W1&Q1).

A1: Two reviewers with computing backgrounds checked all reference labels in a 20% sample of each split against the supplied passages and task specifications (Appendix A.2). Their review covered source bindings and whether a candidate’s conditions and uncertainty were preserved. The references were frozen after review. Separate rates for independent biomedical review of associations and hypotheses, pre-adjudication agreement, and label corrections are unavailable.

The answer-rating audit covers four systems on the same 24 questions. Expert re-review found no scores requiring revision. The two-rater records give the following agreement on model answers.

Reliability checkResult
ICC(2,1), all 96 cases0.998
ICC(2,1), 82 non-prefilled cases0.983
Mean absolute rating difference (0-5)0.082

ICC(2,1) remains 0.983 on the 82 non-prefilled cases. This measures reproducibility of answer ratings. Reference-label validation is limited to the sampled audit above.

Q2. Train–test overlap in documents, spans, and targets (W2&Q2).

A2: The original split assigns 42 topic clusters to training and eight to testing. Appendix A.2 checks normalized questions and identifiers for papers, claims, spans, associations, and hypotheses, including nested references. Distinct identifiers can still refer to overlapping source material. Our expanded audit covered the actual supervision for SFT, Premise Grounding, and Structured Reasoning. It found 14 shared source documents and 48 matched span pairs, affecting 157 unique test questions.

We kept predictions fixed, excluded those 157 questions, and then applied a further semantic-target risk filter. The last step removed 399 questions, leaving 989. The table gives BFM (0–100).

Evaluation settingNDefault SFTw/o BFindLink
Original test, original scoring1,54567.9583.5787.25
Source-overlap exclusions, original scoring1,38868.585.2888.12
Same subset, strict diagnostic scoring1,38863.8575.3378.05
Additional target-risk exclusions, strict scoring98972.2174.3477.29

On the same 1,388 questions, FindLink leads SFT by 19.62 points under original scoring and 14.20 after uniform failure handling and removal of empty–empty credit for Premise, Relation, and Boundary. On the remaining 989, it leads SFT by 5.08 and w/o B by 2.95. The gains survive the identified overlaps. The audit does not establish that every alternate study version or semantically equivalent claim has been detected.

Q3. Recognizing unestablished connections under incompatible conditions (W3&Q3).

A3: FindLink identifies 43 of 70 unestablished connections (61.4%), compared with 30 of 70 (42.9%) for Default SFT. We added 121 expert-reviewed condition-challenge pairs, comprising 20 cross-paper pairs and 101 single-paper contrasts. The 18.6-percentage-point gain is exact not_established recall under a shared four-class readout. On refuted cases, the counts are 2/51 vs. 26/51; on 20 cross-paper negative cases, exact status accuracy is 5/20 vs. 11/20 (FindLink/SFT).

This decision is separate from the compiler’s checks of structure and source bindings (§4.2). Tables 1 and 11 give complementary results on all 286 Discovery questions. The values below use the original scoring on a 0–100 scale.

MethodPremise F1Relation F1Boundary F1Sufficiency F1
Default SFT59.9358.6961.6362.06
w/o B88.176.4489.7460.71
FindLink87.6988.9892.3678.4

w/o B achieves the highest Premise F1. FindLink gains 12.54 Relation, 2.62 Boundary, and 17.69 Sufficiency F1 points over that model. These scores measure recovery of the reference assessment. The condition challenge tests whether the final decision respects the limits of the evidence.

Q4. Performance under controlled token budgets.

A4: FindLink has higher diagnostic BFM at both generation caps. Both methods used Qwen3.5-9B and the same 100 parent training questions. SFT and FindLink processed 9,499,757 and 9,552,027 training tokens, respectively, a difference of about 0.55%. We test 120 balanced questions; each cap covers all generated tokens.

Generation capSFT diagnostic BFMFindLink diagnostic BFM
4,0963032.3
8,19230.4432.98

Diagnostic BFM gains are 2.31/2.54 points, computed before rounding. Original scoring favors SFT; both diagnostic bootstrap intervals include zero. Supervision, calls, and decoding constraints still differ.

We would be grateful if you would reconsider your evaluation in light of the gains after source and target exclusions and the improved recognition of unestablished connections.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer fXud

Dear Reviewer fXud,

Thank you again for your thoughtful comments. We would like to follow up on our rebuttal, which includes an expanded train–test overlap audit, reevaluation after excluding identified overlaps, and further evidence on scientific-condition assessment and budget controls. We would be grateful for your feedback on these additions and any concerns that remain. Please let us know if further clarification would be helpful.

Thank you for your time and consideration.

Thank you for the authors' detailed response. While the rebuttal clarifies some of my concerns, the current manuscript still suffers from substantial issues regarding experimental rigor and missing details. Given the extent of these issues, I do not believe the paper in its current form meets the acceptance bar of ICLR. Therefore, I will maintain my original score.

Reviewer_JaowOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces GI-TraceBench, 10,282 templated questions in gastrointestinal oncology across three task families (Provenance, Knowledge Discovery, Hypothesis Evaluation). All three are built from shared Paper/Claim/Evidence Span records via deterministic transformations on a cluster-disjoint split. The authors also propose FindLink, a Qwen3.5-9B LoRA pipeline, first grounding evidence into typed premises, then building a constrained support plan checked by a non-learned compiler, then answering conditioned on the compiler output. FindLink reaches 87.25 BFM against 67.95 for same-backbone SFT, while untuned open, closed, and adapted systems score below 30.

Critical strengths

S1 The benchmark design separates support status ("not established" vs. "refuted") from polarity or claim origin, while being very diligent with cluster-disjoint splits, overlap checks, and providing the full prompts.

Critical weaknesses

W1 The method is the title’s main contribution, but it is evaluated only on the authors’ own single-domain dataset with one backbone and one seed. This cannot support the general promise of “provenance-grounded scientific reasoning". The authors can address this issue by providing additional experiments on multiple datasets and backbones, though verifying their validity without the experiment code or precise description of the experiments during the rebuttal period will be difficult. Just scoping claims accordingly is not sufficient to warrant an increase in scope, since this paper would then add limited novelty and impact for the broader ICLR community.

W2 Label validity in this medical domain is not established, since the dataset was built with generative AI and silver labels and audited only by two non-medical reviewers reporting 100% confirmation. The human evaluation raises similar concerns, as its reported agreement (ICC 0.997, core-error κ = 1.000) is implausibly high for subjective ratings, and the raters’ expertise is not disclosed.

Given the two major contributions of the paper are currently not sufficiently addressed, the score will sit at clear reject.

Questions for authors

No questions that relate to the 2 weaknesses mentioned above are repeated in here, though a response of the authors to these points is appreciated.

Q1 How are the concepts used in your paper different from established frameworks like Swanson’s ABC model of literature-based discovery? Is there any related literature you could cite to position this more accurately among established related work for your definitions?

Q2 Have you done additional research to what extent SFT (and additional training procedures) improves schema-adherence over actual scientific reasoning? BERT is the only metric in which untrained baselines largely compare with SFT-only scores, while most other baselines show huge gaps. How would results of large proprietary models (e.g., Sonnet 4.6, GPT 5.4 mini) shift, if given the same XGrammar constrained decoding and a few in-context training examples? More generally, what evidence beyond assumption attributes the performance increase to improved reasoning?

Q3 Given your hardware, what were your recorded training times? Which training phase uses the most resources, and which training phase provides the highest metric-improvement-per-GPU-hour?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jaow: Method Contributions and Training Cost

Reviewer Jaow:

Thank you for asking what structured reasoning adds once a model identifies the premises, and at what cost.

Q1. Contribution of structured reasoning and scope of generalization (W1&Q2).

A1: w/o B leads on premise identification. FindLink leads on relation, boundary, and sufficiency assessment. All configurations share Qwen3.5-9B and the same evidence. BFM covers all 1,545 questions, and the four F1 scores cover all 286 Discovery questions (Tables 1 and 11, original scoring, 0–100).

MethodBFMPremise F1Relation F1Boundary F1Sufficiency F1
Default SFT67.9559.9358.6961.6362.06
w/o A81.3886.0187.5489.2565.79
w/o B83.5788.176.4489.7460.71
FindLink87.2587.6988.9892.3678.4

Relative to w/o B, FindLink gains 12.54 Relation, 2.62 Boundary, and 17.69 Sufficiency F1 points, despite slightly lower Premise F1. Stage B trains the model to connect premises, retain conditions, and answer from the compiled plan and its evidence. Table 9 compares complete, retrained configurations. The gain over Default SFT is 19.30 BFM points.

After uniform failure handling and removal of empty–empty credit for Premise, Relation, and Boundary, FindLink reaches 58.47 DRS and 77.43 BFM, versus 47.45 and 63.50 for SFT. Under a shared four-class readout, FindLink correctly labels 43/70 not_established cases vs. SFT’s 30/70; on refuted cases, the counts are 2/51 vs. 26/51. The evaluation covers held-out GI oncology clusters, one backbone, and one seed. Cross-domain and cross-backbone transfer and matched few-shot/XGrammar comparisons for proprietary models remain untested.

Q2. Reference validity and reliability of answer ratings (W2).

A2: GI-TraceBench binds reference claims to passages in identified papers. Two reviewers with computing backgrounds checked every reference label in samples covering 20% each of Train and Test (Appendix A.2). All sampled labels were confirmed before release. The 100% figure describes this post-audit sample. The records provide no independent biomedical error rate for the full benchmark.

For four systems on 24 shared questions (eight per family), expert re-review of all 96 two-rater records identified no scores requiring revision.

Reliability checkResult
ICC(2,1), all 96 cases0.998
ICC(2,1), 82 non-prefilled cases0.983
Cases with different overall ratings46/96
Mean absolute rating difference (0-5)0.082

The raters disagreed on 46/96 overall ratings, with a mean absolute difference of 0.082 across all cases. ICC remains 0.983 on the 82 non-prefilled cases. Both classes occur in the separate binary core-error judgment.

The full archive contains two rating matrices. We will correct the description of three complete independent rating sets. Removing 123 prefilled zeros from 456 cases reduces ordinal α from 0.836 to 0.574 and ICC(2,1) from 0.997 to 0.989. Biomedical qualifications for these raters are undocumented.

Q3. Positioning relative to Swanson’s ABC model (Q1).

A3: Swanson’s ABC model starts with two literatures linked through an intermediate concept B and proposes a candidate A–C connection [1, 2]. FindLink starts with that candidate and asks what the cited studies jointly establish. For example, every premise in Figure 1 may be grounded in its original study, yet cellular and mouse findings still leave human efficacy unestablished. FindLink can retain those observations, specify their conditions, and identify the unsupported step in the proposed connection. We will make this distinction explicit in Section 2.

[1] D. R. Swanson. Undiscovered Public Knowledge. 1986.
[2] D. R. Swanson. Migraine and Magnesium: Eleven Neglected Connections. 1988.

Q4. Training time and return per GPU-hour (Q3).

A4: Stage B used 69.65 allocated GPU-hours, or 64.39% of the successful training run. We computed allocation as GPU count multiplied by logged wall time for each stage.

StageTime (h:mm:ss)GPUsGPU-hours
S: shared answer-level SFT6:45:15427.02
A: Premise Grounding3:50:12311.51
B: Structured Reasoning23:13:01369.65
Total33:48:28Per stage108.18

Relative to shared SFT, A+B adds 81.16 GPU-hours and yields 19.30 BFM and 27.14 DRS points, or approximately 0.24 BFM and 0.33 DRS points per additional GPU-hour. A separate efficiency ranking for A and B would require scores for the successive checkpoints under a common inference interface. In particular, w/o B trains a separate direct-answer adapter on the premise-adapted checkpoint (Table 9). Its score cannot be assigned to Stage A alone.

We would be grateful if you would reconsider your evaluation in light of what the support plan adds on the same backbone and evidence, and the measured cost of that gain.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer Jaow

Dear Reviewer Jaow,

Thank you again for reviewing our work. We would like to follow up on our response, which clarifies the contribution of structured reasoning beyond premise identification and provides supporting evaluation results and recorded training costs. We would appreciate your thoughts on these additions and their implications for your assessment of the contribution. We would be happy to address any remaining questions.

Thank you for your time and consideration.

I thank the authors for the detailed response, particularly for the transparent corrections regarding the rating sets, prefilled zeros, and empty-output credit. The clarification on Swanson’s ABC model and the reported training costs address my Q1 and Q3.

Two main concerns remain, (i, W1) as the authors acknowledge, cross-domain, cross-backbone, and multi-seed evaluation, as well as constrained-decoding/few-shot comparisons for proprietary models, remain untested. Ablations show that Stage B adds value within this setup, yet they cannot support the general claims of the method; (ii, W2) the benchmark labels still lack biomedical validation, and the corrected agreement statistics (alpha dropping from 0.836 to 0.574) suggest that the original reliability claims were overstated.

I would also appreciate clarification on two points. First, how do 46/96 differing ratings yield a mean absolute difference of 0.082 on a 0–5 scale? Second, could the authors confirm the order of the refuted-case counts (2/51 vs. 26/51)? If FindLink correctly labels only 2/51 refuted cases, this would undermine the support-status distinction I considered a key strength.

For the reasons stated above I will therefore maintain my score for now.

Reviewer_SYUkOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a scientific reasoning framework called FindLink alongside a benchmark named GI-TraceBench to improve LLMs' ability to reason across scientific literature. FindLink first uses Premise Grounding to extract source-linked evidence while preserving study context. It then applies Structured Reasoning to organize relationships between findings. A deterministic compiler checks the structural validity and consistency of evidence references. The framework aims to distinguish supported scientific findings from unverified connections and identify evidence gaps that require further investigation.

Critical strengths

1. Clear Distinction Between Evidence Attribution and Scientific Reasoning

I think one strength of this paper is that it distinguishes identifying supporting evidence from determining whether multiple findings establish a scientific connection. GI-TraceBench evaluates this distinction through provenance QA, knowledge discovery, and hypothesis evaluation. In my view, the emphasis on study context and unresolved evidence gaps makes the evaluation more meaningful than simply checking whether an answer cites the correct sources.

2. Informative Component Ablation Studies

I find the component ablation studies particularly useful. The authors retrain variants without Premise Grounding (w/o A) and Structured Reasoning (w/o B) rather than simply removing these components during inference. The results show that removing Structured Reasoning slightly improves premise identification but reduces relation and evidence sufficiency performance. This helps clarify the different contributions of the two stages and provides more specific evidence for the framework's design.

Critical weaknesses

1. Insufficient Validation of Benchmark Quality

GI-TraceBench relies partly on automatically generated silver labels, with only 20% of the training and test sets manually audited. The reviewers' biomedical expertise is not clearly established. I believe that domain-expert validation is particularly important for a benchmark involving scientific relationships and hypothesis evaluation. Without stronger validation, I am concerned about the reliability of the reference answers and the credibility of the reported improvements.

2. Limited Evidence of Genuine Scientific Reasoning

The deterministic compiler verifies structural validity and source bindings but does not assess scientific entailment. Furthermore, several evaluation metrics rely on matching structured reference fields. As a result, higher benchmark scores may not necessarily reflect better scientific reasoning. I think more direct evidence is needed to demonstrate that FindLink can reliably assess scientific connections rather than simply reproduce expected structured outputs.

Questions for authors
  1. How were the silver labels validated, and were biomedical domain experts involved in the process?

  2. Would FindLink maintain its performance improvements under matched training and inference budgets compared with Default SFT?

  3. How do the authors ensure that FindLink performs genuine scientific reasoning rather than simply matching structured reference answers?

2 comments
AuthorsOpenReview ↗

Response to Reviewer SYUk: Reference Validation and Scientific Assessment

Reviewer SYUk:

We thank you for asking whether a structurally valid answer also makes the right scientific judgment when the studies support different conclusions.

Q1. Silver-label validation and reviewer expertise (W1&Q1).

A1: Two reviewers with computing backgrounds audited 20% of Train and 20% of Test before release (Appendix A.2). They checked every reference label in the sampled records against the source evidence and task specification, including the distinction between an observed finding, the conditions under which it applies, and a proposed connection that remains unverified. The benchmark uses the references confirmed at the end of that audit.

The documented review does not establish separate biomedical-specialist coverage or a pre-adjudication error rate for candidate associations and hypotheses. We will state the reviewers’ backgrounds, unit of sampling, and scope of confirmation. The two-expert review in A3 concerns model answers.

Q2. Performance under shared training and inference budgets.

A2: On 120 balanced test questions, FindLink scores higher under diagnostic scoring at both generation caps. Both methods used Qwen3.5-9B and the same 100 parent training questions. Default SFT processed 9,499,757 training tokens and FindLink 9,552,027, a difference of approximately 0.55%. Each cap limits total generated tokens across all calls.

Generation capSFT diagnostic BFMFindLink diagnostic BFM
4,0963032.3
8,19230.4432.98

The diagnostic BFM gains are 2.31 and 2.54 points, computed before rounding. Original scoring favors SFT; both diagnostic bootstrap intervals include zero. This policy retains failed cases and removes empty–empty credit for Premise, Relation, and Boundary. These data control processed training tokens closely and hold generation caps fixed. The methods still differ in supervision, call structure, and decoding constraints.

Q3. Scientific judgments under incompatible study conditions (W2&Q3).

A3: On 70 cases where the evidence left the proposed connection unestablished, FindLink recognized 43 (61.4%) and Default SFT recognized 30 (42.9%). The gain is 18.6 percentage points in recall, calculated from the counts. This challenge comprises 121 expert-reviewed pairs, including 20 cross-paper pairs and 101 single-paper condition contrasts. Under the shared four-class readout, this is exact not_established recall, not binary withholding of support. The gain does not extend to refuted cases or the cross-paper subset.

The compiler checks plan structure, identifier resolution, and source bindings (§4.2). The model must decide whether the cited findings, with their original context and polarity, jointly support the candidate. We examined this distinction in two further evaluations.

(1) Scientific content. Two experts checked two existing sets of AI-generated assessments against the original model answers and paper evidence, considering relation correctness, evidence entailment, boundary preservation, and unsupported inference. Within the historical 120-question pool, the table reports FindLink minus SFT on the 0–4 scale, using only questions with valid scores for both models on that dimension.

Reviewed record setRelation correctness differencePaired questionsBoundary preservation differencePaired questions
Set 1-0.028635043
Set 2-0.108137-0.125040

Each row is an expert-reviewed AI score set. FindLink scores at or below SFT on these two dimensions. The paired coverage differs by dimension and is nonrandom. These results do not establish a general advantage in scientific correctness.

(2) Structured recovery. On the same 140 Discovery questions with nonempty reference relations and boundaries, FindLink scores 79.29/87.31 in instance-mean Relation/Boundary F1, compared with 70/78.06 for SFT (0–100). Thus the evidence supports better recovery of the specified relations and conditions as well as higher recall of unestablished connections in the challenge.

We would be grateful if you would reconsider your evaluation in light of the condition challenge and the results obtained under shared generation caps.

Thank you for the detailed response and the additional experiments. The matched-budget comparisons and expert evaluations help clarify the strengths and limitations of FindLink.

I appreciate the authors' efforts. While some concerns about benchmark validation and scientific reasoning remain, I acknowledge the additional evidence and will maintain my original positive rating.

Good luck!

Reviewer_9GHgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies provenance-grounded scientific connection assessment: determining not only which source passages support individual scientific claims, but also what multiple source-grounded premises jointly establish under their respective study contexts (population, model, assay, endpoint) and which evidential links remain unresolved. The authors introduce:

  1. GI-TraceBench (Section 3, Appendix A), a gastrointestinal-oncology benchmark comprising 50 topic clusters, 1,614 paper records, 5,475 claims, 4,968 evidence spans, and 10,282 questions split by topic cluster into 8,737 training and 1,545 test instances across three task families: Provenance QA (scored by PGS, Eq. (16)), Knowledge Discovery QA (scored by DRS, Eq. (17)), and Hypothesis Evaluation QA (scored by HSS, Eq. (18)), summarized by the Balanced Family Macro score BFM=13(PGS+DRS+HSS)\mathrm{BFM} = \frac{1}{3}(\mathrm{PGS} + \mathrm{DRS} + \mathrm{HSS}) (Eq. (5) / Eq. (19)).
  2. FindLink (Section 4, Appendix B), a two-stage framework trained via sequential LoRA adaptation (S→SA→SABS \to SA \to SAB, Eqs. (6)–(9)) on Qwen3.5-9B. Stage A performs source-bound Premise Grounding (Eqs. (1)–(2)), and Stage B performs Constrained Support Planning (Eqs. (3)–(4), Algorithm 1) checked by a deterministic compiler ΓG\Gamma _ {\mathcal{G}} (Eqs. (11)–(12)) before generating the final structured response under XGrammar schema constraints.

On the 1,545-question test set (Table 1), FindLink achieves 87.25 BFM (85.46 PGS, 90.02 DRS, 86.26 HSS), outperforming same-backbone Default SFT (67.95 BFM), four adapted retrieval/attribution systems (44.89–45.32 BFM), and 11 zero-shot open/closed LLM baselines (17.61–32.42 BFM).

Critical strengths
  1. Valuable task decomposition for source-grounded scientific reasoning. GI-TraceBench moves beyond single-claim citation verification to jointly evaluate source localization (Provenance QA), cross-premise connection synthesis (Knowledge Discovery QA), and mechanistic/clinical hypothesis auditing (Hypothesis Evaluation QA) under explicit study-context boundaries (population, model, assay, endpoint) and cluster-disjoint splits (Section 3, Tables 2–5).

  2. Auditable intermediate state with transparent compiler scope. Decomposing generation into an explicit premise inventory AA (Eq. (2)), a DAG support plan TT (Eq. (3)), and a deterministic compiler record C=ΓG(A,T)C = \Gamma _ {\mathcal{G}}(A, T) (Eq. (12), Algorithm 1) makes intermediate failures inspectable, and the paper is careful not to overclaim ΓG\Gamma _ {\mathcal{G}} as a theorem prover of scientific entailment.

Critical weaknesses
  1. Potential target leakage / benchmark circularity via structured input records, coupled with schema-bound metric artifacts (63.69 constant scores).

    • Input–target overlap: Section 3.1 (p. 4, lines 188–201) and Appendix A.1 (p. 14, lines 717–719, Eq. (10)) state that the evidence packet EE provided to the model may contain pre-extracted Claim records, candidate association records (which store candidate relations and contextual boundaries), and hypothesis cards (which store predicted observations, verification requirements, and refutation criteria). Meanwhile, Section 3.2 (lines 207–215) and Appendix A.2 (line 731) state that the reference answers yrefy^{\mathrm{ref}} and intermediate supervision (DA,Dplan,Danswer\mathcal{D} _ A, \mathcal{D} _ {\mathrm{plan}}, \mathcal{D} _ {\mathrm{answer}}) are generated from those same upstream records via deterministic transformations. Indeed, Figure 3 (p. 5) shows that Knowledge Discovery and Hypothesis Evaluation have identical Single-paper (10.49%, 1,079 questions) and Cross-paper (7.87%, 809 questions) shares, reflecting 1-to-1 template expansion. Without a passage-only control (providing only raw paper passages without candidate cards or pre-extracted claims), it is unclear whether FindLink learns scientific reasoning or deterministic record-field recovery.
    • Dynamic ID enumeration and empty-set scoring artifacts: Premise F1 (Appendix C.2, p. 20, line 1044) requires matching internal premise_claim_ids. FindLink enforces per-item valid ID enums via XGrammar (Appendix B.5, line 967; Appendix E, line 1168) to reach 87.69%, whereas all 11 general LLM baselines (Claude Sonnet 4.6, Gemini 3.5 Flash, GPT-5.4-mini, Qwen3.6-Plus, etc.) score 0.00%–5.87% on Premise F1 (Figure 5, p. 9). Worse, Appendix C.2 (line 1050) specifies that "The set scorer assigns 1 when both extracted sets are empty." As a result, all four adapted methods (Attribute First, OpenScholar, PaperQA2, VeriCite)—which score 0.00% on Premise F1—achieve the exact same constant 63.69 on Relation F1 and 63.69 on Boundary F1 (8 identical cells in Figure 5 and Table 11) simply by outputting empty sets on the 63.69% of test items whose reference sets are empty. Back-solving the undefined fifth DRS component m‾gap-coverage\overline{m} _ {\mathrm{gap\text{-}coverage}} from Eq. (17) similarly yields 0.00% for Gemini 3.5 Flash and 3.94%–3.98% for the four adapted methods versus 92.27% for FindLink.
  2. Confounded same-backbone comparisons, single-backbone/single-seed evaluation, and selective omission of fine-tuned baselines in human evaluation.

    • Unmatched training and inference budgets: Default SFT trains for 2 LoRA epochs on DSFT\mathcal{D} _ {\mathrm{SFT}} and uses a single 4,096-token call, whereas FindLink trains for 2+1+1=42 + 1 + 1 = 4 LoRA epochs (with doubled Stage-B data Dplan∪Danswer\mathcal{D} _ {\mathrm{plan}} \cup \mathcal{D} _ {\mathrm{answer}}, Eqs. (6)–(9), Table 8) and uses 3 sequential calls (2,048 + 2,048 + 4,096 = 8,192 tokens, Eq. (15)) plus deterministic compiler overrides. In Table 11, w/o B already reaches 83.57 BFM and slightly beats full FindLink on Premise F1 (88.10 vs. 87.69), HSS (86.44 vs. 86.26), and BERTScore (91.02 vs. 90.96).
    • Model scale and seed gaps: Only one backbone (Qwen3.5-9B) and one random seed (seed = 42, Table 8 line 958) are evaluated; the backbone and training setup for the four "in-domain trained" adapted baselines (PaperQA2, OpenScholar, VeriCite, Attribute First) and the identity of the anonymous LLM Judge (Appendix C.4 line 1069) are undisclosed.
    • Omission in Table 12 and Figure 6: Although Appendix C.1 (p. 19, line 1014) and Appendix D.2 (p. 22, line 1134) state that 3 human raters scored all 19 systems across 24 questions (456 outputs), Table 12 and Figure 6 report only 12 systems, omitting Default SFT, w/o A, w/o B, and all 4 adapted methods! Moreover, Appendix D.2 (lines 1134–1137) reports 3-rater Krippendorff's alpha = 0.836 alongside single-measure ICC(2,1) = 0.997 and pairwise Cohen's kappa = 1.000, which requires methodological clarification.
Questions for authors
  1. Task-by-task input schema and passage-only control (decisive for benchmark validity): Which exact fields of the pre-extracted Claim records, candidate association records, and hypothesis cards in V(E)V(E) (Eq. (10)) are visible in the prompt for Provenance QA, Knowledge Discovery QA, and Hypothesis Evaluation QA? What is the performance of Default SFT, w/o B, and FindLink under a strict passage-only setting where only raw paper passages (and paper IDs) are provided in EE?

  2. Empty-set scoring artifact (63.69), undefined m‾gap-coverage\overline{m} _ {\mathrm{gap\text{-}coverage}}, and matched-budget baselines: Can you confirm whether the eight identical 63.69 scores for Attribute First, OpenScholar, PaperQA2, and VeriCite on Relation F1 and Boundary F1 (Figure 5, Table 11) arise from empty predicted sets receiving 1.0 on empty reference sets (Appendix C.2, line 1050)? Please provide the exact mathematical definition of m‾gap-coverage\overline{m} _ {\mathrm{gap\text{-}coverage}} in Eq. (17), report scores when empty-vs-empty pairs are excluded or macro-averaged over positive instances, and compare FindLink against a 4-epoch Default SFT baseline and a 3-call chain-of-thought SFT baseline under identical per-item XGrammar ID constraints across at least 3 seeds.

  3. Omitted 7 systems in human evaluation (Table 12 / Figure 6) and undisclosed model details: Why were Default SFT, w/o A, w/o B, and the 4 adapted methods omitted from Table 12 and Figure 6 despite being evaluated in the 456 human-rated outputs (Appendix D.2, line 1134)? Please report their human and LLM-judge scores on that 24-question subset, disclose the exact model/version used as the LLM Judge and the backbone/training epochs used for the 4 adapted baselines, and explain how ICC(2,1) = 0.997 and 3-rater Cohen's kappa = 1.000 were computed alongside Krippendorff's alpha = 0.836.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9GHg: Evaluation Fairness and Scoring Analysis

Reviewer 9GHg:

Thank you for tracing the repeated scores to the empty-set rule.

Q1. Visible inputs and target-field recovery (W1.1&Q1).

A1: All tasks receive the question, task family, and public evidence, with the reference withheld (§3.1). Eq. (10) lists paper ID, claim ID, span, position, and availability. Appendix E.2 illustrates the target claim (Provenance), candidate chain (Discovery), and proposed hypothesis (Hypothesis Evaluation).

Packets can include extracted claims and candidate records. References derive from upstream associations (relations and boundaries) and hypothesis cards (predictions, verification requirements, and refutation criteria). Figure 3 reports subtype counts. SFT and FindLink received identical packets on all 1,545 questions. Without a per-item field inventory or passage-only scores on identical evidence for SFT, w/o B, and FindLink, field recovery remains unresolved.

Q2. Empty-set credit and nonempty-reference performance (W1.2&Q2).

A2: Yes. All eight 63.69 scores in Figure 5 arise from empty-set credit. Empty extracted sets earn full credit on the 146/286 questions (51.05%) without extractable reference relations or boundaries. Topic-cluster averaging gives 63.69. The four adaptations score 26.88, 27.61, 27.10, and 26.74 on Premise F1. Extraction can miss content expressed in prose.

The first two scores are instance means over 140 nonempty references. Diagnostic DRS covers all 286 Discovery questions and BFM all 1,545 test questions. We handle confirmed failures uniformly and remove empty–empty credit for Premise, Relation, and Boundary only (0–100).

MethodRelation F1Boundary F1Diagnostic DRSDiagnostic BFM
Default SFT7078.0647.4563.5
w/o B46.2281.550.1273.86
FindLink79.2987.3158.4777.43

FindLink gains 9.29 Relation and 9.25 Boundary F1 over SFT on nonempty references. Pooled Relation micro-F1 across all 286 questions is 82.24 vs. 82.50. Diagnostic DRS gains are 11.02 over SFT and 8.35 over w/o B.

Gap Coverage is the fifth equal-weight term in Eq. (17). Nonempty reference gaps score set F1 against predicted gaps. Gap-free abstention references score 1 for predicted abstention and 0 otherwise. Other gap-free references score 1 for empty predicted gaps and 0.5 otherwise. We average within and then across clusters. Actual Gap scores are 92.29 for FindLink and 3.97 for each adaptation. We will add the rule to Appendix C.2.

Q3. Budget controls and ablation trade-offs (W2.1&W2.2&Q2).

A3: Original SFT uses two epochs and one 4,096-token call. FindLink adds two one-epoch stages and uses 2,048/2,048/4,096-token calls (§5.1). The control uses 100 shared training parent QAs and 120 balanced test QAs. SFT/FindLink processed 9,499,757/9,552,027 tokens (0.55% difference), with shared generation caps.

Generation capSFT diagnostic BFMFindLink diagnostic BFM
4,0963032.3
8,19230.4432.98

Diagnostic BFM gains are 2.31/2.54 points, computed before rounding. Original scoring favors SFT; both diagnostic bootstrap intervals include zero. We use Qwen3.5-9B and seed 42. Four-epoch SFT and three-call CoT SFT with identical XGrammar constraints across three seeds remain untested. A valid ID can still cite irrelevant evidence.

w/o B leads FindLink on Premise F1 (88.10 vs. 87.69), HSS (91.93 vs. 89.79), and BERTScore-F1 (94.18 vs. 93.04). FindLink leads on DRS, 87.94 vs. 79.22 (Tables 1 and 11).

Q4. Evaluation coverage and rating reliability (W2.2&W2.3&Q3).

A4: We aligned all 19 systems on 24 shared questions. Below are the seven omitted from Table 12/Figure 6 and FindLink (0–5; prefilled zeros retained).

MethodJudge meanArchived A/B mean
Default SFT3.153.17
w/o A3.63.87
w/o B4.084.80
Attribute First2.152.70
OpenScholar2.233.33
PaperQA24.043.54
VeriCite2.853.31
FindLink4.194.40

Archived human means favor w/o B over FindLink, approximately 4.80 vs. 4.40. Two FindLink final-answer failures contribute 0.4125 points to that gap. On the other 22 questions FindLink leads 4.8015 to 4.7879, offsetting 0.0125. Both failures count.

Two rating matrices underlie the archived means. We will correct the three-rater account in Eq. (20) and Appendices C.4 and D.2. Removing 123 prefilled zeros leaves 333/456 records. Ordinal α falls from 0.836 to 0.574 and absolute-agreement ICC(2,1) from 0.997 to 0.989. Core-error κ = 1.000 measures a separate binary judgment. Human–Judge Pearson correlation is 0.932 across 19 system means, versus 0.958 across 12.

Appendix E.4 gives the Judge prompt. Its exact version and the four local adaptations’ full backbone/epoch settings remain unavailable.

We would be grateful if you would reconsider your evaluation in light of the nonempty-reference and budget-controlled results.

AuthorsOpenReview ↗

Follow-up on Our Response to Reviewer 9GHg

Dear Reviewer 9GHg,

Thank you again for your careful review. We would like to follow up on our rebuttal, which includes nonempty-reference results, clarifications of the scoring rules, and expanded evaluation reporting. We would be grateful for your feedback on this additional evidence and any remaining concerns. Please let us know if there are specific points you would like us to clarify further.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.