- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SciEngram, casting cumulative scientific memory as an evaluable problem where evidence should be provenance-traceable, consolidatable by scope, and revisable or retainable as new evidence arrives. Contributions include an incremental scientific-QA benchmark answered in literature-arrival order with a multi-dimensional scoring protocol, and a system (SciEngram-Core) that organizes paper materials into structured memory packages. Experiments compare retrieval-style baselines and ablate hierarchy, scope admission, and dependency-related components; validation relies mainly on an LLM judge and group-level statistics.
Critical strengths
- Incremental scientific QA plus a structured multi-dimensional rubric (especially provenance, scope, and abstention) targets blind spots in existing SciQA, RAG, and general agent-memory evaluations, so the evaluation contract itself is useful.
- Ablations of hierarchy and dependency-related components help argue that performance is not solely from better prompting or longer context, giving the architecture some testable content.
Critical weaknesses
- Claims and main experiments are misaligned. The paper emphasizes revision and retention dynamics of cumulative memory, yet the main comparison appears to focus on incremental answering; without quantitative main-result evidence for those dynamics, the “revisable scientific memory” narrative is hard to support.
- Empirical advantages look weak and not robust enough. Gains over the strongest retrieval baseline appear limited and partly concentrated in provenance-related dimensions; performance is also unstable across readers, and scoring depends heavily on an LLM judge.
Questions for authors
- What are the quantitative revision, retention, and interference results versus baselines? When new evidence conflicts with prior conclusions, how does SciEngram-Core’s correct-revision rate compare with that of the baseline methods?
- Relative to the strongest retrieval baseline, do advantages remain after removing provenance-related bonuses or penalties, or after task-family splits?
- Is there an independent judge outside the extraction and question-generation model family? If not, please add a cross-model-family judge comparison. Are there human-agreement results on a sample?
Please Respond to the Authors’ Rebuttal
Dear Reviewers,
The Author–Reviewer Discussion period is now nearing its end and will close on 18 November. The authors have submitted their response and, in some cases, a revised manuscript.
Could you please take a moment to:
Even if your evaluation remains unchanged, a short reply confirming your final position is very helpful, as it allows me to track each reviewer’s stance and write an accurate meta-review.
If you are unable to participate further, please let me know as soon as possible.
Thank you for your time and careful reviewing.
Best regards,
Area Chair, ICLR 2027