acceptodds
Under review as a conference paper at ICLR 2027

SciEngram: Evidence-Grounded Scientific Memory for Cumulative Research Reasoning

Abstract

Scientific agents that continually read the literature must maintain traceable and revisable scientific beliefs. Existing scientific question answering, retrieval-augmented generation, knowledge graphs, and general-purpose agent memory mainly evaluate query-time retrieval and answer correctness. They do not test whether the evidence structure behind an answer persists across queries and changes with new evidence. We introduce SciEngram, which formalizes scientific memory as an evaluable and implementable computational problem. SciEngram-Benchmark evaluates incremental scientific question answering through 401 questions in six task families, answered as papers arrive within literature groups. An ordered-evidence audit design specifies complementary revision and retention checks. It evaluates answers along eight dimensions, including correctness, evidence quality, source traceability, relational reasoning, scope qualification, and abstention. SciEngram-Core implements these functions with provenance-bound paper memory, scope-aware consolidation, dependency correction, and factorized belief states. Under the main reader configuration, SciEngram-Core scores 8.87, exceeding the strongest BM25 baseline at 8.65, and improves source traceability by 1.09 points. Removing hierarchical evidence organization or dependency correction lowers the score by more than one point. The gains therefore come primarily from sustained organization of evidence sources, relations, and scope.

Primary area:
datasets and benchmarks
Keywords:
SciEngram, Memory, Agentic System

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1d5MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SciEngram, casting cumulative scientific memory as an evaluable problem where evidence should be provenance-traceable, consolidatable by scope, and revisable or retainable as new evidence arrives. Contributions include an incremental scientific-QA benchmark answered in literature-arrival order with a multi-dimensional scoring protocol, and a system (SciEngram-Core) that organizes paper materials into structured memory packages. Experiments compare retrieval-style baselines and ablate hierarchy, scope admission, and dependency-related components; validation relies mainly on an LLM judge and group-level statistics.

Critical strengths
  1. Incremental scientific QA plus a structured multi-dimensional rubric (especially provenance, scope, and abstention) targets blind spots in existing SciQA, RAG, and general agent-memory evaluations, so the evaluation contract itself is useful.
  2. Ablations of hierarchy and dependency-related components help argue that performance is not solely from better prompting or longer context, giving the architecture some testable content.
Critical weaknesses
  1. Claims and main experiments are misaligned. The paper emphasizes revision and retention dynamics of cumulative memory, yet the main comparison appears to focus on incremental answering; without quantitative main-result evidence for those dynamics, the “revisable scientific memory” narrative is hard to support.
  2. Empirical advantages look weak and not robust enough. Gains over the strongest retrieval baseline appear limited and partly concentrated in provenance-related dimensions; performance is also unstable across readers, and scoring depends heavily on an LLM judge.
Questions for authors
  1. What are the quantitative revision, retention, and interference results versus baselines? When new evidence conflicts with prior conclusions, how does SciEngram-Core’s correct-revision rate compare with that of the baseline methods?
  2. Relative to the strongest retrieval baseline, do advantages remain after removing provenance-related bonuses or penalties, or after task-family splits?
  3. Is there an independent judge outside the extraction and question-generation model family? If not, please add a cross-model-family judge comparison. Are there human-agreement results on a sample?
Reviewer_pMLqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SciEngram addresses scientific question answering as literature accumulates, motivated by the need to preserve provenance, distinguish study scopes, and revise scientific assessments appropriately. It introduces a 401-question benchmark spanning six task families and a reference architecture combining paper-local evidence, scope-aware consolidation, dependency correction, and factorized belief states. The main comparison reports 88.67/100 for Core versus 86.53 for BM25-Chunk, with the largest improvement in provenance. Three component ablations reduce answering scores, while a complementary checkpoint-based revision and retention audit is specified.

Critical strengths

[Scientifically Meaningful Evidence Architecture]

  • The architecture preserves source records while organizing cross-paper claims by scope and evidence dependence. Requiring retrieved structures to resolve to source passages provides a useful foundation for auditing scientific conclusions. The provenance improvement over BM25-Chunk—92.63 versus 81.75—is encouraging (Table 2).

[Useful Benchmark Construction]

  • The benchmark includes cross-paper relations, premise correction, claim-status assessment, and insufficient-evidence handling. Each accepted question received two human reviews, with disagreements adjudicated by a third reviewer.
Critical weaknesses

[Revision and Retention Remain Unevaluated]

  • The reported experiments score questions independently at their associated paper-arrival states. They do not report whether the same scoped assessment changes correctly after decisive evidence or remains stable after irrelevant, dependent, or incomparable evidence. The checkpoint audit is described as complementary, and Figures 3/S3 label it a “Planned Dynamic Track.” No transition-level revision, retention, or interference results are reported. Thus, the experiments establish incremental evidence-grounded answering, while the central capability of reliable cumulative scientific memory remains insufficiently tested.

[Limited Demonstrated Headroom for Future Methods]

  • Strong retrieval baselines already achieve high scores: BM25-Chunk obtains 86.53/100 in the main setting, and BM25-Public reaches 9.17/10 with Claude (Tables 2–3). These starting points raise concern about the benchmark’s ability to distinguish future advances in scientific memory. The paper provides no aggregate task-family or difficulty breakdown identifying which capabilities remain challenging; Table 4 contains selected examples. Although these scores do not establish saturation, harder evaluation subsets and demonstrated separation among strong methods are needed to substantiate the benchmark’s longer-term research value.
Questions for authors
  1. Are completed checkpoint-audit results available beyond the reported experiments? Please provide transition counts and accuracy for justified revision and retention, including responses to opposing, irrelevant, and dependent evidence. These results would directly address the central evaluation gap.
  2. Which exact configurations produced Core’s 8.867/10 main result, 8.7167 ablation Full result, and 9.29 DeepSeek result? Please clarify reader versions, additional retrieval/read steps, citation instructions, and answer review.
Reviewer_nKaiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on evidence-grounded scientific memory that enables research agents to accumulate, organize, and revise scientific knowledge as new papers arrive. This paper introduces SciEngram-Benchmark to evaluate scientific reasoning across 401 questions from 228 papers, and SciEngram-Core, a memory architecture that maintains provenance-aware evidence, cross-paper relationships, and revisable belief states. Experimental results show that SciEngram-Core outperforms several retrieval and memory baselines.

Critical strengths
  1. The paper proposes a benchmark that goes beyond conventional scientific QA by evaluating evidence tracing, cross-paper reasoning, claim assessment, and uncertainty handling under sequential paper ingestion.

  2. The author proposes SciEngram-Core, which explicitly models scientific claims, experimental scope, evidence dependencies, and belief states, providing a structured approach to maintaining traceable and revisable scientific knowledge.

  3. Writing is good and easy to follow.

Critical weaknesses
  1. BM25-Chunk achieves 86.53, only 2.14% below SciEngram-Core. However, the paper does not report computational costs. Given the additional complexity of SciEngram-Core, BM25 may offer a better cost-performance trade-off. What unique advantages does SciEngram-Core provide over a much simpler BM25-based approach?

  2. Single paper factual questions account for 40.4% of the benchmark, while the paper does not report a comprehensive transition-level evaluation of belief revision and retention across successive evidence updates. So, the experiments do not directly demonstrate the claimed ability to maintain and revise scientific beliefs over time, which is one of the paper's central motivations.

  3. The authors use DeepSeek Pro for benchmark construction and reference-answer generation, as well as for evaluating model responses. This overlap may introduce model-specific evaluation biases.

  4. The fact that a simple BM25-Chunk baseline already achieves a score of 86.53 suggests that the benchmark may not be sufficiently challenging for modern LLMs. I believe a more challenging benchmark would be more valuable.

Questions for authors

See Weaknesses.

Comments on the paper

1 comment
Area_Chair_V8pCOpenReview ↗

Please Respond to the Authors’ Rebuttal

Dear Reviewers,

The Author–Reviewer Discussion period is now nearing its end and will close on 18 November. The authors have submitted their response and, in some cases, a revised manuscript.

Could you please take a moment to:

  • Read the authors’ rebuttal and any updated version of the paper;
  • Leave a reply to the authors on the points you raised, indicating whether your concerns have been addressed.

Even if your evaluation remains unchanged, a short reply confirming your final position is very helpful, as it allows me to track each reviewer’s stance and write an accurate meta-review.

If you are unable to participate further, please let me know as soon as possible.

Thank you for your time and careful reviewing.

Best regards,

Area Chair, ICLR 2027

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.