- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
The paper studies research-state discontinuity: an agent retains relevant evidence in its memory but fails to receive the evidence or its relationships when making a later decision. For example, retrieving a hypothesis without the experiment that refuted it may lead the agent to repeat an unsuccessful direction. The proposed Executable Scientific-Agent Memory Benchmark, ESMB, separates memory delivery from planner behavior through controlled histories, checkpoints, and authored action effects. It distinguishes evidence excluded before ranking from evidence displaced during ranking or context assembly. Research State Memory, RSM, stores typed scientific items with branch identities and explicit relations, including hypothesis and related evidence.
Critical strengths
- Interesting problem scenario and failure diagnosis: Separating evidence exclusion, displacement, and planner misuse explains failures that final task success alone cannot localize.
- Authors also propose simple yet novel memory mechanism RSM to link a hypothesis to results that support, contradict, or replace it, and retrieves these records together. RSM solves 60 of 102 ScienceAgentBench tasks, compared with 46 for repair summary, and the reported confidence interval supports a positive improvement.
Critical weaknesses
- The strongest controlled results rely on hints that identify relevant evidence. RSM uses these hints, while the main simple baselines ignore some of them and restrict retrieval to the current research branch. When both memory and planner cues are removed under the stricter interface, RSM versus summary drops from 27/30 versus 12/30 successes to 3/30 versus 1/30.
- The experiments do not yet show that RSM works reliably in long, naturally evolving research projects.
Questions for authors
- When you remove the evidence hints, what else changes in the experiment? Do the existing results show whether RSM benefits from its memory design or mainly from those hints and the baselines' restriction to one branch? In ScienceAgentBench, did both methods receive equally recent and complete code and error messages within the same actual context size? Could differences in this information explain the improvement?
2 comments
Response to Reviewer Ryma (2/2)
Q (ScienceAgentBench). "In ScienceAgentBench, did both methods receive equally recent and complete code and error messages within the same actual context size? Could differences in this information explain the improvement?"
Both methods received the same prompt outside memory and the same 6,000-token memory cap, but they allocate that cap differently:
- Repair summary reserves a guaranteed slot for the latest program and its execution result, truncated at head and tail to fit.
- RSM has no guaranteed slot, but returns up to two recent results, each bundled with the program that produced it, in full.
Neither allocation dominates: repair summary always sees the latest attempt, whereas RSM more often sees two complete attempts, so part of the gain may come from this difference. The gain itself is consistent at the task level (22 of 30 discordant tasks favour RSM; exact McNemar p = 0.0161; positive in all four domains) and uses no annotation cues. We will add a control that gives repair summary the same two attempts, and report per-call context sizes for both methods.
W2. "The experiments do not yet show that RSM works reliably in long, naturally evolving research projects."
The controlled histories are authored by design: this makes the decisive record known, so a delivery failure can be attributed exactly. ScienceAgentBench complements them with histories that arise from the agent's own repair attempts. We agree that long, naturally evolving research projects are the important next test, and we will state this scope explicitly in the main claims.
We thank the reviewer again for these precise questions, and hope this response addresses the reviewer's concerns.
References
- [1] Ziru Chen, Shijie Chen, Yuting Ning, et al. ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery. arXiv:2410.05080, 2024.
Response to Reviewer Ryma (1/2)
We thank the reviewer for the careful review, and for recognising the value of separating exclusion, displacement and planner misuse, the novelty of linking a hypothesis to the results that bear on it, and the ScienceAgentBench [1] improvement. In this rebuttal, we:
The strict interface is the paper's joint cue-removal test: it removes memory-side and planner-side cues together rather than isolating one factor. Besides the memory-side annotations (focus IDs, required labels, phase, expected kind), it:
The text baselines kept their branch filter in both conditions. Because several factors change at once, the joint test cannot attribute the drop to one of them; we will add a factorized analysis that removes them one at a time.
Because ESMB fixes the histories and actions, we can check deterministically, without a planner, which decisive records each memory delivers. The check shows:
The cued 6/15 → 12/15 comparison therefore reflects both RSM's delivery and the baselines' branch scoping, and does not by itself isolate the contribution of RSM's representation; we will present it accordingly. The analysis also exposes the underlying trade-off between excluding cross-branch evidence and contaminating the context, which branch identities and typed relations are designed to resolve. Making cross-branch delivery independent of hints is the key improvement we will make.