acceptodds
Under review as a conference paper at ICLR 2027

Same Paper, Different Decision: A Controlled Diagnostic of Task-Conditioned Scientific Evidence Readiness

Abstract

Scientific papers are not intrinsically ready or unready for information extraction: their usability depends on the evidence required by the downstream task. We formulate task-conditioned scientific evidence readiness, in which a system maps a paper–task pair to one of three operational actions: READY, REVIEW_REQUIRED, or ABSTAIN. To isolate task conditioning from document variation, we construct a paired diagnostic set from eight Ti–6Al–4V laser powder bed fusion papers and three extraction tasks that differ only in their material-subtype requirement, yielding 24 correlated evaluations. Holding the paper fixed, the frozen AI-assisted author-reference action changes across tasks for six of eight papers. We compare four frozen prompting and control configurations. Direct prompting matches the reference action on 16/24 pairs, while the full structured configuration matches 14/24, indicating that additional structure does not consistently improve decisions in this testbed. An evidence-grounded configuration makes no false READY decisions among its 18 valid outputs but has execution failures on 6/24 pairs, exposing a safety–coverage tradeoff. Error analysis further shows that an output can record a task-relevant evidence conflict yet select an inconsistent operational action. These errors motivate a stage-wise diagnostic perspective separating evidence-state recognition from task-dependent action mapping; the stages are not independently evaluated here. Our results motivate pair-conditioned evaluation for scientific extraction systems whose outputs trigger downstream use or human review.

Primary area:
datasets and benchmarks
Keywords:
task-conditioned readiness, scientific information extraction, paired diagnostic evaluation, evidence grounding, abstention

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_wj73OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces task-conditioned scientific evidence readiness, where a paper–task pair is classified as READY, REVIEW_REQUIRED, or ABSTAIN depending on whether the paper provides sufficient evidence for a specific extraction task. Using eight Ti–6Al–4V papers and three controlled task variants, the authors show that readiness can change even when the paper is fixed and compare four prompting configurations for making these decisions.

Critical strengths

None

Critical weaknesses
  1. Very limited scale. The evaluation contains only eight independent papers and 24 paper–task pairs, which makes it difficult to draw conclusions beyond this small diagnostic set.

  2. Weak ground truth. The reference labels are AI-assisted author annotations rather than independent expert gold, with no external adjudication or inter-annotator agreement study, and no any calibration. This makes the reported agreement numbers difficult to interpret as true readiness accuracy.

  3. Narrow setting and limited novelty. The study only varies one task dimension, material subtype, within a very specific Ti–6Al–4V extraction setting. The paper also explicitly states that sufficiency, abstention, and the three-way action space are not new, so the main contribution is primarily the paired diagnostic setup.

Questions for authors

Please see Critical Weaknesses section.

Reviewer_QmBROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether a scientific paper provides sufficient evidence for a specified extraction task. It pairs eight materials-science papers with three variants of a material-subtype requirement and compares four prompting configurations on decisions to proceed, request human review, or abstain.

Critical strengths
  1. Holding the paper fixed while changing the extraction requirement provides a straightforward way to examine task-dependent decisions.
  2. The paper reports execution failures alongside decision agreement and explicitly acknowledges the limitations of its small dataset and AI-assisted reference labels.
Critical weaknesses
  1. Writing and presentation. I find the abstract overloaded with definitions, experimental details, metrics, and numerical results, such as 16/24 and 14/24 agreement counts. This level of compression might be easier to follow for a familiar problem with a shared background, but the paper introduces a specialized setting that needs more explanation. I would have liked the abstract to first make the practical problem, motivation, and purpose of the study clear enough to understand on their own. Instead, the reader encounters many technical details before developing a clear understanding of why the study is needed. This concern extends to the presentation beyond the abstract. In my view, the manuscript still needs substantial development and polishing before it is ready for submission.

  2. Limited experimental scope. I do not find the evaluation sufficiently comprehensive. It covers only eight papers from one materials-science setting, three task variants that change the material-subtype requirement, and one model configuration. I recognize that the authors position this as a controlled diagnostic study, but the evidence remains too limited for me to assess the broader usefulness of the proposed formulation. Table 2 and Figures 1–3 summarize the outcomes, but I would also have liked a worked illustration showing the source evidence, the changed task requirement, and why the resulting decision should change. This would make the experimental setting and its implications easier to understand.

Overall, I do not think the paper is ready for acceptance. The question is understandable, but both the presentation and the empirical study need substantial further development. In its current form, I see this as preliminary work that needs more preparation before submission.

Questions for authors

no specific questions

Reviewer_Yq1YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines the use of LLMs for three information extraction by examining a controlled set of 8 papers discussing titanium alloy material science papers and their applicability to (non-) or extra-low-interstitial (ELI) tasks.

Critical strengths

None.

Critical weaknesses

The paper is not well structured at all. The second paragraph of the introduction does not introduce the pertinent material science aspects necessary for a reviewer with ICLR expertise to understand the task, its importance or the contribution.
The current work is not understandable.

Questions for authors

I have no questions for the authors since this paper is not sufficiently well written for me to understand it well enough to ask questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.