acceptodds
Under review as a conference paper at ICLR 2027

Stored But Not Delivered: Benchmarking and Repairing Research-State Discontinuity in Scientific Agents

Abstract

In scientific agents, each hypothesis, experiment, and result conditions later decisions, and a decision can depend on evidence produced many steps earlier. An agent can store a hypothesis and the result that refutes it, yet retrieve one without the other, and so repeat a direction it has already ruled out. We formulate this as research-state discontinuity: the context delivered for a decision can omit relationships that stored history still holds, and inspecting the store or final task success does not reveal the omission. To assess it we construct the Executable Scientific-Agent Memory Benchmark (ESMB), a controlled diagnostic suite that fixes histories, checkpoints and authored action effects so that only memory varies, and scores a memory interface returning labeled evidence separately from a planner acting on it. This separation localizes the failure to two operations. Text baselines confine candidates to the current branch, so the decisive record is dropped before ranking, and a record that does reach ranking can still be displaced once the budget fills. We then propose Research State Memory (RSM), which gives every item a branch identity, stores contradiction and supersession as explicit edges, and delivers an item together with the evidence bearing on it. RSM raises joint successes from 6/15 to 12/15 on the cued suite, and solves 60/102 externally authored ScienceAgentBench tasks against repair summary’s 46/102.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scientific agent, LLM agent, agent memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RymaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

The paper studies research-state discontinuity: an agent retains relevant evidence in its memory but fails to receive the evidence or its relationships when making a later decision. For example, retrieving a hypothesis without the experiment that refuted it may lead the agent to repeat an unsuccessful direction. The proposed Executable Scientific-Agent Memory Benchmark, ESMB, separates memory delivery from planner behavior through controlled histories, checkpoints, and authored action effects. It distinguishes evidence excluded before ranking from evidence displaced during ranking or context assembly. Research State Memory, RSM, stores typed scientific items with branch identities and explicit relations, including hypothesis and related evidence.

Critical strengths
  • Interesting problem scenario and failure diagnosis: Separating evidence exclusion, displacement, and planner misuse explains failures that final task success alone cannot localize.
  • Authors also propose simple yet novel memory mechanism RSM to link a hypothesis to results that support, contradict, or replace it, and retrieves these records together. RSM solves 60 of 102 ScienceAgentBench tasks, compared with 46 for repair summary, and the reported confidence interval supports a positive improvement.
Critical weaknesses
  • The strongest controlled results rely on hints that identify relevant evidence. RSM uses these hints, while the main simple baselines ignore some of them and restrict retrieval to the current research branch. When both memory and planner cues are removed under the stricter interface, RSM versus summary drops from 27/30 versus 12/30 successes to 3/30 versus 1/30.
  • The experiments do not yet show that RSM works reliably in long, naturally evolving research projects.
Questions for authors
  • When you remove the evidence hints, what else changes in the experiment? Do the existing results show whether RSM benefits from its memory design or mainly from those hints and the baselines' restriction to one branch? In ScienceAgentBench, did both methods receive equally recent and complete code and error messages within the same actual context size? Could differences in this information explain the improvement?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Ryma (1/2)

We thank the reviewer for the careful review, and for recognising the value of separating exclusion, displacement and planner misuse, the novelty of linking a hypothesis to the results that bear on it, and the ScienceAgentBench [1] improvement. In this rebuttal, we:

  • Q (other changes): list every change in the strict interface;
  • W1 / Q (design or hints): analyse where the cued gains come from, using a deterministic delivery check;
  • Q (ScienceAgentBench): compare the context the two methods received;
  • W2: explain the role of authored histories.

Q (other changes). "When you remove the evidence hints, what else changes in the experiment?"

The strict interface is the paper's joint cue-removal test: it removes memory-side and planner-side cues together rather than isolating one factor. Besides the memory-side annotations (focus IDs, required labels, phase, expected kind), it:

  • removed from the planner prompt:
    • the labels on each fragment;
    • each action's listed label requirements and the present/missing support summary;
    • the list of labels observed so far;
    • the phase and expected-kind lines;
    • descriptive action identifiers, replaced by opaque ones;
    • the task description;
  • added a 4,096-byte context cap and a 256-token output cap.

The text baselines kept their branch filter in both conditions. Because several factors change at once, the joint test cannot attribute the drop to one of them; we will add a factorized analysis that removes them one at a time.

W1 / Q (design or hints). "Do the existing results show whether RSM benefits from its memory design or mainly from those hints and the baselines' restriction to one branch?"

Because ESMB fixes the histories and actions, we can check deterministically, without a planner, which decisive records each memory delivers. The check shows:

  • With the hints, RSM delivers every decisive record on all three planner suites, and on Replay it does so with zero cross-branch contamination.
  • The branch filter is the main reason the baselines miss records, and it is a deliberate safeguard: it keeps cross-branch contamination at zero on Replay. On these short histories, removing only the filter lets the summary baseline also deliver every decisive record, but a large share of what it then delivers comes from other branches.
  • Without the hints, RSM's base retrieval stays on the request branch and its cross-branch attachments are rarely triggered. This explains the drop under the stricter interface (27/30 vs 12/30 → 3/30 vs 1/30).

The cued 6/15 → 12/15 comparison therefore reflects both RSM's delivery and the baselines' branch scoping, and does not by itself isolate the contribution of RSM's representation; we will present it accordingly. The analysis also exposes the underlying trade-off between excluding cross-branch evidence and contaminating the context, which branch identities and typed relations are designed to resolve. Making cross-branch delivery independent of hints is the key improvement we will make.

AuthorsOpenReview ↗

Response to Reviewer Ryma (2/2)

Q (ScienceAgentBench). "In ScienceAgentBench, did both methods receive equally recent and complete code and error messages within the same actual context size? Could differences in this information explain the improvement?"

Both methods received the same prompt outside memory and the same 6,000-token memory cap, but they allocate that cap differently:

  • Repair summary reserves a guaranteed slot for the latest program and its execution result, truncated at head and tail to fit.
  • RSM has no guaranteed slot, but returns up to two recent results, each bundled with the program that produced it, in full.

Neither allocation dominates: repair summary always sees the latest attempt, whereas RSM more often sees two complete attempts, so part of the gain may come from this difference. The gain itself is consistent at the task level (22 of 30 discordant tasks favour RSM; exact McNemar p = 0.0161; positive in all four domains) and uses no annotation cues. We will add a control that gives repair summary the same two attempts, and report per-call context sizes for both methods.

W2. "The experiments do not yet show that RSM works reliably in long, naturally evolving research projects."

The controlled histories are authored by design: this makes the decisive record known, so a delivery failure can be attributed exactly. ScienceAgentBench complements them with histories that arise from the agent's own repair attempts. We agree that long, naturally evolving research projects are the important next test, and we will state this scope explicitly in the main claims.

We thank the reviewer again for these precise questions, and hope this response addresses the reviewer's concerns.

References

  • [1] Ziru Chen, Shijie Chen, Yuting Ning, et al. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. arXiv:2410.05080, 2024.
Reviewer_8qHuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates research-state discontinuity in scientific agents, where stored evidence and its relationships may be lost during memory retrieval, leading to poor decisions. The authors introduce ESMB, a benchmark designed to diagnose such failures, and propose Research State Memory (RSM), a structured memory system that preserves relationships between hypotheses, experiments, and results. Experiments on ESMB and ScienceAgentBench demonstrate improvements over several memory baselines.

Critical strengths

The paper identifies an interesting and practically relevant problem in scientific-agent memory and introduces a benchmark that separates memory retrieval failures from planning failures. The proposed RSM achieves promising improvements on both controlled evaluations and ScienceAgentBench, providing some evidence of its practical effectiveness.

Critical weaknesses
  1. Small benchmark scale. The main ESMB evaluations contain only five tasks, each repeated three times (15 trials per setting). Such a small number of independent tasks raises serious concerns about the robustness and generalizability of the reported improvements.

  2. Limited technical novelty and model diversity. The proposed method is entirely harness-level, without any model training or architectural innovation. The main evaluations involve only two LLM planners (GPT-5.2 and GPT-4.1), limiting the generalizability of the findings.

  3. Weak and insufficiently validated baselines. On ScienceAgentBench, RSM is compared only against the authors' repair-summary baseline, without established memory methods such as A-MEM or StructMem. It is unclear whether these methods can be successfully adapted to this benchmark, and unlike RSM, they were not specifically optimized for it, potentially making the comparisons unfair.

  4. Lack of evidence for necessity. The paper demonstrates effectiveness but fails to establish whether the proposed memory architecture is necessary. In particular, it lacks comparisons with full-history prompting using strong long-context models, which might eliminate the need for such specialized memory mechanisms.

Questions for authors

Could the authors clarify how they ensured fair baseline comparisons, particularly given that existing memory methods were not specifically adapted or optimized for ScienceAgentBench?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8qHu (1/2)

We thank the reviewer for the careful review, and for recognising the practical relevance of the problem and the value of a benchmark that separates memory-retrieval failures from planning failures. In this rebuttal, we:

  • W1: explain the suite design and the evaluation beyond the five Fixed tasks;
  • W2: explain why the method is harness-level, and discuss planner diversity;
  • W3 / Q: describe the fairness controls on ScienceAgentBench, including budget allocation;
  • W4: discuss necessity relative to full-history prompting.

W1. "The main ESMB evaluations contain only five tasks, each repeated three times (15 trials per setting)."

Each ESMB task is hand-authored so that the decisive record and the effect of every action are known. This lets ESMB attribute a failure to exclusion, displacement or planner misuse rather than only report end-task success, but makes each task costly to build. The five Fixed tasks, one per task family, are complemented by:

  • TaskVariant: five variants of the Fixed tasks;
  • LongHorizon: 11 tasks × 5 executions, with longer write–use delays;
  • Replay: 17 context requests in six traces, planner-free and deterministic;
  • ScienceAgentBench [1]: 102 tasks with the native evaluator, where RSM solves 60 versus 46 (22 of 30 discordant tasks favour RSM; exact McNemar p = 0.0161).

We agree that repeated executions are not independent tasks. We will report design-level results with sign tests alongside the execution-level counts, and increase the number of task designs.

W2. "The proposed method is entirely harness-level … The main evaluations involve only two LLM planners (GPT-5.2 and GPT-4.1) …"

  • Harness level is a deliberate choice. Research-state discontinuity arises between storage and the planner: the evidence exists but is not delivered. A harness-level method addresses it where it occurs and works with any planner, including closed models that cannot be fine-tuned, as do established memory systems such as MemGPT [2] and A-MEM [3]. The contribution lies in the representation (branch identities, typed epistemic relations, evidence bundling) and in a benchmark that localizes delivery failures.
  • Planner diversity. Replay's delivery metrics involve no planner, so those findings do not depend on the planner. For end-to-end success, GPT-5.2 and GPT-4.1 span two model generations; we agree that planners from more model families would strengthen the conclusions, and we will add them.
AuthorsOpenReview ↗

Response to Reviewer 8qHu (2/2)

W3 / Q. "Could the authors clarify how they ensured fair baseline comparisons, particularly given that existing memory methods were not specifically adapted or optimized for ScienceAgentBench?"

  • Same conditions. No method was tuned on ScienceAgentBench; both memories used their ESMB configuration. The wrapper, everything in the prompt outside memory, the 6,000-token memory cap and the 2,000-token feedback cap were shared, no ESMB annotations were used, and no campaign version changed either method.
  • Budget allocation. Within the same cap, repair summary reserves a guaranteed slot for the latest program and execution result (truncated at head and tail to fit), whereas RSM has no guaranteed slot but returns up to two recent results, each bundled with its program, in full. Each method thus has an advantage the other lacks, and part of RSM's gain may come from seeing two complete attempts. We will add a control that gives repair summary the same two attempts.
  • Established baselines. On ESMB we compared against the official A-MEM [3] and adapted StructMem [4] under public requests: RSM ties A-MEM on Fixed (3/15 each), leads on TaskVariant (3/15 vs 0/15), and exceeds StructMem by 33.3 points on both suites. Both call their own memory-side models (A-MEM made 338 such calls in one ESMB panel), which changes cost and adds models; we will add them to ScienceAgentBench with these calls reported separately.

W4. "… it lacks comparisons with full-history prompting using strong long-context models …"

The submitted suites were designed to isolate delivery, so their histories are short enough to fit in the context window, and full-history prompting delivers every decisive record on them. The case for bounded memory concerns long-running projects whose history grows with every experiment: prompt cost then grows with history length, and long-context models are known to under-use evidence placed in the middle of long inputs [5]. We agree that a direct comparison with strong long-context models is needed to establish necessity, and we will add it on histories long enough to decide the question.

We thank the reviewer again for these comments, and hope this response addresses the reviewer's concerns.

References

  • [1] Ziru Chen, Shijie Chen, Yuting Ning, et al. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. arXiv:2410.05080, 2024.
  • [2] Charles Packer, Sarah Wooders, Kevin Lin, et al. MemGPT: Towards LLMs as Operating Systems. arXiv:2310.08560, 2023.
  • [3] Wujiang Xu, Zujie Liang, Kai Mei, et al. A-MEM: Agentic Memory for LLM Agents. arXiv:2502.12110, 2025.
  • [4] Buqiang Xu, Yijun Chen, Jizhan Fang, et al. StructMem: Structured Memory for Long-Horizon Behavior in LLMs. In ACL (Volume 2: Short Papers), pages 122–146, 2026.
  • [5] Nelson F. Liu, Kevin Lin, John Hewitt, et al. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.
Reviewer_QmCAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper formalizes research-state discontinuity, a failure mode in LLM scientific agents where prior research evidence is retained in long-term storage but omitted from the context delivered to the planner at decision time, leading agents to revisit previously refuted hypotheses or repeat unproductive experiments. To isolate memory delivery from planner execution, the authors introduce the Executable Scientific-Agent Memory Benchmark (ESMB), which localizes memory delivery failures into two mechanisms: exclusion before ranking (hard same-branch candidate filters that drop decisive cross-branch evidence) and displacement after candidate selection (decisive records outranked or cut under tight context budgets). As a solution, the paper proposes Research State Memory (RSM), a branch-aware relational memory that organizes hypotheses, experiments, and results as typed items connected by eight explicit scientific relations (e.g., contradicts, supersedes). During retrieval, RSM bundles related evidence directly alongside primary items within fragment slots, doubling joint decision success on the annotation-cued ESMB benchmark (from 6/15 to 12/15 under GPT-5.2) and transferring to native program repair on ScienceAgentBench by solving 60/102 tasks compared to 46/102 for a repair summary baseline (+13.7 percentage points).

Critical strengths

S1. Timely and Consequential Problem Formulation for Autonomous Scientific Agents The paper identifies and formalizes research-state discontinuity—a critical yet overlooked failure mode where scientific LLM agents retain hypotheses, experiments, and results in storage, but fail to deliver qualifying negative evidence or supersession history to the planner at decision time. As language-model agents are increasingly deployed for iterative scientific discovery and automated experimentation, standard benchmarks that evaluate only final execution success mask underlying memory delivery failures. This leads agents to waste computational or physical experiment budgets by repeatedly revisiting previously ruled-out lines of inquiry. Formalizing the gap between memory retention and decision-time context delivery addresses an urgent, highly practical bottleneck for the growing field of agentic AI for science.

Critical weaknesses

W1. Unclear Architectural Motivation and Unsubstantiated Distinction from Existing Structured Memory Frameworks The paper does not provide a clear theoretical or empirical motivation for the specific design choices in Research State Memory (RSM), nor does it sufficiently establish how RSM fundamentally differs from prior graph/relational agent memory systems. While RSM introduces branch identities, seven typed items, eight explicit relation types (e.g., contradicts, supersedes), and a six-stage context assembly sequence, the authors offer little justification for why these specific relations or retrieval heuristics were selected over simpler relational structures. Furthermore, existing structured memory frameworks like StructMem, A-MEM, and TierMem also organize past interactions using graph edges, hierarchical events, or evolving notes. In public-request comparisons where memory-side guidance is removed, RSM yields inconsistent results against these baselines (e.g., losing to Lossless Summary in the A-MEM panel), making it difficult to discern whether RSM offers a generalizable architectural advance or relies on benchmark-aligned engineering rules.

Questions for authors

Please see Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer QmCA (1/2)

We thank the reviewer for the careful review and for recognising research-state discontinuity as a timely and consequential problem for autonomous scientific agents. In this rebuttal, we:

  • W1 (motivation): explain the motivation for each design choice;
  • W1 (positioning): state how RSM differs from A-MEM [1], StructMem [2] and TierMem [3];
  • W1 (public-request results): analyse the panel results;
  • Presentation: list the changes we will make.

W1 (motivation). "… the authors offer little justification for why these specific relations or retrieval heuristics were selected over simpler relational structures."

Each element targets a failure mode that ESMB's task families instantiate:

  • Branch identity. Research forks into alternative hypotheses. Evidence must stay attributed to the branch that produced it, so that results from another line are neither mistaken for current evidence nor lost when they become relevant (branch switching, failed-experiment recovery).
  • Typed, directed relations. The decision-relevant fact is often a relation rather than a record: a result contradicts a hypothesis, or a rerun supersedes an earlier result. A similarity link can connect two records but cannot say whether one confirms, refutes or replaces the other (conflicting evidence, supersession). The eight relation types cover these epistemic relations (supports, contradicts, supersedes) and provenance (e.g., derived-from, has-artifact).
  • Evidence bundling. The core failure is delivering a hypothesis without the result that changes its interpretation. Bundling places them in the same fragment, so a budget cut cannot separate them.

We agree that the submission motivated these choices mainly conceptually, and we will add ablations that compare each element with a simpler alternative.

W1 (positioning). "… existing structured memory frameworks like StructMem, A-MEM, and TierMem also organize past interactions using graph edges, hierarchical events, or evolving notes."

We thank the reviewer for these references.

  • A-MEM [1] links notes by model-judged similarity and evolves them.
  • StructMem [2] organizes events hierarchically with cross-event synthesis.
  • TierMem [3] separates a summary tier from a provenance-checked evidence tier.

RSM differs in three respects:

  • every item carries a research-branch identity;
  • relations are directed and typed, including epistemic ones: one record supports, contradicts or supersedes another;
  • an item is delivered together with the evidence that bears on it.

RSM also needs no memory-side model calls, whereas A-MEM made 338 such calls in one ESMB panel. We will add an explicit comparison table.

AuthorsOpenReview ↗

Response to Reviewer QmCA (2/2)

W1 (public-request results). "In public-request comparisons where memory-side guidance is removed, RSM yields inconsistent results against these baselines (e.g., losing to Lossless Summary in the A-MEM panel) …"

Against the two published systems, RSM does not lose in either panel: it ties A-MEM on Fixed (3/15 each), leads on TaskVariant (3/15 vs 0/15), and exceeds StructMem by 33.3 points on both suites, remaining ahead after removing any task family. The inconsistency concerns lossless summary, and we see two contributing factors:

  • Panel variance. The same adapters score differently across panels: lossless summary reaches 6/15 on Fixed in the A-MEM panel but 2/15 in the StructMem panel, where RSM leads it on both suites (6/15 and 7/15 vs 2/15 and 3/15). With 15 executions per suite, these between-panel differences are comparable to the within-panel margins.
  • Mechanism. Without memory-side guidance, RSM's base retrieval stays on the request branch, as lossless summary's does, and its cross-branch attachments are rarely triggered. Lossless summary, which renders each selected record in full, can then match or exceed it.

This identifies cross-branch retrieval without request guidance as the component to improve, and we will address it.

We thank the reviewer again for these comments, and hope this response addresses the reviewer's concerns.

References

  • [1] Wujiang Xu, Zujie Liang, Kai Mei, et al. A-MEM: Agentic Memory for LLM Agents. arXiv:2502.12110, 2025.
  • [2] Buqiang Xu, Yijun Chen, Jizhan Fang, et al. StructMem: Structured Memory for Long-Horizon Behavior in LLMs. In ACL (Volume 2: Short Papers), pages 122–146, 2026.
  • [3] Qiming Zhu, Shunian Chen, Rui Yu, et al. From Lossy to Verified: A Provenance-Aware Tiered Memory for Agents. arXiv:2602.17913, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.