acceptodds
Under review as a conference paper at ICLR 2027

Preserving Exposure and Outcome Context in Automated Cancer Evidence Synthesis

Abstract

Scientific evidence verification must preserve a claim's chemical, exposure setting, species, cancer site, endpoint, and evidence type. We study whether a deterministic toxicology scope gate transfers to general biomedical claim verification. Motivated by solvent-cancer synthesis, we combine a retrospective retrieval audit with a frozen experiment on 662 SciFact claim-abstract pairs from the public development split and an original-test citation subset. A lexical field extractor gates a TF-IDF verifier; linked source families are excluded from training, and policies receive identical complete stored abstracts. Strict compatibility retains no substantive answers because the extractor recognizes no chemical values. Mismatch-only filtering leaves development support decisions unchanged and removes two correct supports from one source family in the secondary subset, without removing any reference support-errors. At 89 accepted supports, confidence filtering retains 34 correct decisions versus 32 for mismatch filtering. Separately, we reanalyze published TCE-metabolite transcript estimates across 22 matched-dose pairs in five human cell models. At 62.5 micromolar, both metabolites induce large renal NRF2-associated transcript responses, whereas two neuronal models respond much more strongly to the cysteine conjugate, consistent with the source study. We release frozen protocols, predictions, probe-level estimates, and sensitivity analyses. The results locate the observed verification failures in missing fields and erroneous extraction; the biological analysis supports hypothesis development without measuring malignant transformation or establishing a new cancer mechanism.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
scientific claim verification, evidence synthesis, selective prediction, toxicology, reproducibility, extraction errors

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_HRt5OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Research Question

An automated scientific evidence synthesis pipeline can return a citation that is real and on topic but supports a different claim from the one presented. The paper asks whether a deterministic toxicology scope gate can transfer to general biomedical claim verification. The gate checks six fields: chemical, exposure setting, species, cancer site, endpoint, and evidence type.

A TF-IDF logistic-regression verifier produces predictions for each claim–abstract pair, after which the gate determines which substantive predictions to retain. The strict policy retains a SUPPORT or CONTRADICT prediction only when all required scope fields are compatible and the required provenance is available. Otherwise, the substantive prediction is withheld, and existing NOT_ENOUGH_INFO predictions are preserved. The relaxed mismatch-only policy rejects detected mismatches but permits unknown fields.


Their Work

1. Retrieval audit. The authors re-examine logs from an earlier automated literature pipeline. All six passage inspection responses contain empty passage arrays, so the archive cannot verify that passage-level verification occurred. Among the 68 records, 41 stored abstracts contain ellipses, suggesting possible truncation. One record whose title reports no association was tagged as reporting one, and 241 expansion rows map to 238 merged records.

2. Transfer experiment. The gate is tested on 662 claim–abstract pairs: 339 from SciFact’s development set and 323 from a recovered subset of its original test set. A TF-IDF logistic-regression verifier is trained on 384 claims after excluding 425 of 809 for overlap with the evaluation sets through linked source/claim families.

The strict gate retains no substantive answers because its eight-category chemical dictionary recognizes no chemical values. The relaxed policy leaves development SUPPORT decisions unchanged. In the secondary test subset, it removes two correct supports and no incorrect SUPPORT predictions. At the same acceptance count of 89, confidence-based selection retains 34 correct supports versus 32 for the gate, without using evaluation labels for selection. A post hoc diagnostic finds zero compatible pairs even after removing the chemical requirement.

3. Biological reanalysis. Separately, the authors reanalyze published, processed gene-expression estimates for two TCE metabolites, DCVC and DCVG, comparing 22 dose-matched pairs across five human cell models.


Conclusion

The authors reach a negative conclusion: the gate does not improve support selection. The strict gate eliminates substantive answers, while the relaxed policy discards supported claims due to extraction errors. The authors attribute the failure to field extraction rather than to the verifier.

Critical strengths

1. The paper investigates the cause of failure. The analysis shows that missing extracted fields prevent the strict gate from retaining substantive answers, so changing the verifier alone would not solve the problem. This provides a useful direction for future work.

2. The experiment includes useful controls and transparent reporting. The authors compare the gate with confidence-based selection at the same acceptance count. They also preserve the frozen experimental results and clearly disclose the extraction error.

Critical weaknesses

1. The transfer experiment provides limited insight into the scope-gating approach. The narrow lexical extractor leaves required fields unavailable, making substantive acceptance impossible under the strict policy. Although this establishes that the current implementation fails in this setting, it does not show whether scope gating would be useful with reliable and appropriate field inputs. The domain-specific evaluation outlined in Section 8 remains future work.

2. The schema lacks independent field-level validation. Although the experiment uses external claim-verification labels, it provides no independent annotations of the extracted fields or scope compatibility. The 24 constructed cases establish software conformance but do not demonstrate generalization. In addition, strict equality may miss semantic or hierarchical compatibility. These limitations make it difficult to assess whether the gate reliably identifies meaningful differences between claims and evidence.

Questions for authors

Questions


  1. What fraction of the evaluation claims or abstracts actually mention one of the eight dictionary chemicals, based on an independent check?

  2. Species has the highest two-sided field coverage. Using the frozen ledger, how would a species-only mismatch filter perform in terms of risk and coverage?

  3. On a small subset with independently annotated fields, does the strict gate retain substantive answers, and does it improve support selection?

Suggestions


  1. Title. Automated Evidence Synthesis overstates the work: no synthesis system is built or evaluated, and the experiment is limited to three-way claim classification using a TF IDF model.

  2. Related work. Discuss the relationship between scope compatibility and GRADE’s concept of indirectness [1]. Clarify what the proposed framework adds and why a binary gate is appropriate for this task.

  3. Section 3 lacks its premise. The paper never says what the archived collections are or why they exist, leaving the reader to infer that they are logs from the authors' own earlier retrieval run.

  4. The three works do not form a coherent paper. They share neither data nor methods, and the paper itself treats them as separate.

[1]: Guyatt, Gordon H., et al. "GRADE guidelines: 8. Rating the quality of evidence—indirectness." Journal of clinical epidemiology 64.12 (2011): 1303-1310.

Reviewer_iSLpOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The core question in this paper is whether a scientific study supports a particular claim in the context of cancer evidence synthesis. The authors focus on checking whether the claim and study concern the same chemical, exposure conditions, species, and outcome. Those checks alone cannot establish support, but they could identify reasons a citation is inappropriate.

Critical strengths

The paper addresses an important problem and reports its limitations.

Critical weaknesses

Despite reading the paper several times and making a careful effort to understand the approach, I remain unclear about the methodology. In particular, I could not clearly follow how the retrieval audit, compatibility-checking experiment, and biological reanalysis fit together, or which component establishes the paper’s central contribution. This lack of clarity limits my ability to assess the work.

My main concern, based on my understanding of the paper, is that the verification experiment does not adequately evaluate the proposed idea. The strict filter retains no substantive answers because the extractor fails to supply the required fields. The relaxed filter removes correct supported evidence without removing any incorrect ones. These outcomes diagnose implementation limitations, but do not establish whether accurate context checking would improve scientific verification.

The biological reanalysis does not resolve this limitation, and its role in evaluating the computational method remains unclear. 
 I am not convinced that the paper establishes a sufficient methodological or empirical contribution.

Questions for authors

Please provide any answers that might address comments in critical weaknesses.

Reviewer_UnQFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The manuscript investigates whether a deterministic toxicology scope gate can be transferred to general biomedical claim verification. It reports experiments on SciFact claim-abstract pairs, comparing strict compatibility and mismatch-based filtering strategies, and separately reanalyzes published transcript-response estimates for TCE metabolites in human cell models. However, despite the methodological and experimental components described in the manuscript, I found the presentation difficult to follow and I was unable to establish a sufficiently clear understanding of how the different components support the overall contribution.

Critical strengths

The manuscript addresses the relevant problem of preserving the scope and context of biomedical claims during evidence verification. The experimental setup suggests an effort to evaluate the verification approach systematically, and the authors appear to acknowledge the limitations of their biological analysis. However, the clarity of the manuscript limits my ability to assess the rigor and significance of these methodological choices and to determine the extent to which the reported results support the proposed contribution.

Critical weaknesses

My main concern is the manuscript's lack of clarity, coherence, and organizational structure. Although the abstract identifies two distinct research lines, the rationale connecting them is not sufficiently clear in the manuscript. Technical concepts and methodological choices are introduced without adequate explanation, and the relationships between the research objectives, experimental design, reported results, and conclusions are difficult to follow. The reported findings may be relevant, but the current presentation makes it difficult to assess their interpretation, significance, and implications. The severity of these presentation issues raises concerns about whether the manuscript meets the minimum standards of clarity required for a meaningful scientific evaluation. In its current form, I cannot reliably assess the paper's technical merits or scientific contribution.

Questions for authors

My concerns are not limited to specific technical questions that could be addressed individually, but to the difficulty in understanding the overall narrative of the paper. Addressing these concerns would require a substantially clearer and more coherent presentation of the work as a whole.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.