acceptodds
Under review as a conference paper at ICLR 2027

Continual Graph Memory Agent for AI Research Replication

Abstract

Replicating modern AI research is often a multi-hour, error-prone process that hinges on locating implementation-critical details dispersed across text, code, and execution logs. Recent methods on AI research replication expose a recurring bottleneck: agents repeatedly rediscover the same reusable implementation patterns across papers. To address this challenge, we present the Congram — a continual graph memory agent for paper replication that accumulates reusable "recipes" from papers and generated code, allowing continual experience collection and reuse for future code generation. Evaluated on papers from PaperBench, Congram demonstrates significant improvements in token usage (on average, 7763 tokens less for prompt and 1724 tokens less for completion) for code generation compared to the zero-shot setting while preserving comparable replication quality scores.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Memory, Continual, LLM Agent, Research Replication

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_ezwfOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AI paper replication repeatedly rebuilds reusable implementation components. Congram stores paper-derived patterns and linked code in a growing graph, retrieves them for new papers, and refines submissions using rubric feedback. Across 22 target papers, it reports average scores of 38.5% and 32.4% on two backbones, exceeding its GoldenMem baseline by 35.6 and 24.6 percentage points.

Critical strengths
  1. Reusing data processing, training, and evaluation components across replication attempts is a sensible direction for reducing repetitive implementation work.

  2. The reported improvements are substantial. Congram achieves reported average scores of 38.5% and 32.4%, exceeding GoldenMem by 35.6 and 24.6 percentage points, respectively.

Critical weaknesses
  1. The final scoring procedure needs further clarification. Congram receives target-rubric feedback during generation, whereas the imported original PaperBench baselines do not, preventing a controlled comparison. The authors should also clarify whether the reported scores are derived from the full evaluation including execution and provide the corresponding configurations.

  2. The refinement ablation may also change retrieval. In the supplementary code, the refinement path uses lexical Jaccard matching, while the ordinary generation path uses Contriever embeddings. If these entry points produced Table 4, the score differences cannot be attributed solely to refinement.

  3. The no-graph and no-card scores of 4.7% and 4.0% should be compared with the matched no-refinement score of 5.5%. Without a vector-RAG baseline using the same memory content, refinement procedure, and budget, the contribution of graph structure remains unclear.

  4. Evidence for continual accumulation and resistance to forgetting is limited. Using just one bootstrap paper and a fixed task order, with no frozen-initial-memory control condition, cannot separate the gains brought by later updates. Likewise, a non-significant difference observed when revisiting only a single paper is not enough to rule out catastrophic forgetting.

  5. The efficiency analysis does not cover the complete system. Reported token savings concern single-pass generation without refinement and do not cover the full costs of judging, repair, and memory updates. The efficiency of the complete pipeline achieving the main scores therefore remains unclear.

  6. Minor: (1) Page 5, line 256: “plan paper text” should be “plain paper text.” (2) Page 11, Figure 4 caption, line 552: “Mmeory” should be “Memory.”

Questions for authors

See weaknesses

1 comment
AuthorsOpenReview ↗

Author Response

We thank the reviewer for highlighting how reusing data-processing, training, and evaluation components can reduce repeated implementation work across papers, and for noting the improvements in replication scores. We address the weaknesses in the answer below.

W1. The reported main scores are full PaperBench replication scores obtained after executing the selected submission, with code_only=false and skip_reproduction=false; refinement instead uses only the depth-3 Code Development subtree for up to five repair iterations, without receiving final-evaluation results.

W2. The supplementary refinement driver uses the shared hybrid retriever with lexical overlap as its relevance signal, a supported configuration that avoids requiring embedding inference. Although the reviewer correctly identifies different defaults across entry points, this does not establish that refinement changes retrieval: disabling refinement within the same driver (max_refine_iters=0) preserves retrieval and prompt construction, allowing refinement to be isolated.

W3. The appropriate submitted single-pass comparison is indeed 5.5% for Congram without refinement versus 4.7% without graph-based memory and 4.0% without cards, so we will make that matching explicit. We have additionally completed refinement-enabled no-graph and no-cards experiments, scoring 36.3% and 31.5% over 22 papers and three runs.

W4. The paper provides evidence for the value of accumulated experience: removing cross-paper memory reuse lowers the score from 38.5% to 16.1% while retaining rubric-guided refinement. Revisiting an earlier paper yields no statistically significant degradation on either backbone, supporting retention in the tested setting. Regarding frozen-initial-memory control condition, in No cumulative memory ablation we exactly satisfy the suggested condition: we forbid memory updates during the sequential processing of the papers. 5. The reported token savings measure prompt and completion tokens for single-pass code generation without refinement, isolating the generation-stage benefit of memory reuse. We will label this scope explicitly and distinguish it from the full pipeline's cost.

W6. We thank the reviewer for suggestions, and we will fix the misprints in the paper text.

Reviewer_Miq1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a memory framework for continual learning in the research paper reproduction task. It shows that with such a memory framework, the performance on PaperBench is better than several baselines of independent evaluation without any experience accumulation, including the one with official repository as reference. Then the ablation study is conducted, and it shows that both the memory component and the refinement step plays important roles.

Critical strengths

Paper reproduction is an important task in general.

Critical weaknesses
  1. The setting of this paper does not have enough significance. The idea of this paper is to do continual learning on paper production task. Paper reproduction itself is an important task, but the practical motivation of continually learn to reproduce papers is insufficient. Different from common agentic tasks or conversation tasks, which is frequent in real deployment and the users expect the agent can continually improve, paper reproduction is relatively infrequent. Thus, for continual learning community, paper reproduction is a very narrow application area.

  2. The baselines are all independent evaluation, and hence it is obvious that the proposed method, which includes experience accumulation, can outperform these baselines. It can provide more insights if the authors show that the proposed method is better than other common memory framework or other continual learning methods.

  3. From the ablation study, experience accumulation is even not the most important component. In the ablation study, it is clear that the improvement compared with baselines are mostly due to the refinement step, instead of the memory.

  4. The conclusion that the agent with memory can even outperform the agent with access to the official repository. This result is very counter-intuitive and the authors do not investigate the reason. Generally in the PaperBench dataset, more similar the topic or techniques of different papers are, more useful the experience from another paper would be. However, I think no experience can have better similarity than the official repository in terms of paper reproduction. So I think the problem probably lies in the method of utilizing the official repository. It needs more supportive evidence to conclude that the experience accumulation in the dataset can outperform the agent with access to the official repo.

Based on the reasons above, I do not recommend the publication of this paper in ICLR. The main reason is that the setting / topic is relative narrow and of little practical significance.

Questions for authors

I have included my comments in the strengths and weaknesses part.

1 comment
AuthorsOpenReview ↗

Author Response

We thank the reviewer for the recognition of the importance of the paper reproduction task. We address the weaknesses below.

W1. Research replication is a demanding testbed for persistent agents because research workflows repeatedly require reusable training, data-processing, and evaluation components across papers, baselines, and ablations. Congram targets reuse of this accumulated engineering experience across related research tasks, making its practical value the avoidance of repeated costly implementation rather than assuming that every end user frequently reproduces complete papers.

W2. The study includes alternatives that also accumulate experience: the no-graph ablation uses continual flat-code memory, while the no-cards ablation retains the cumulative graph but replaces card-based retrieval with flat triplet/code packing. We have additionally evaluated both with refinement enabled under updated prompts, obtaining 36.3% and 31.5%, respectively, over 22 target papers and three runs.

W3. With rubric-guided refinement retained, removing cross-paper cumulative memory and retrieved code assets (No cumulative memory ablation) reduces the submitted score from 38.5% to 16.1%, a substantial 22.4-percentage-point difference showing that refinement alone does not explain the full pipeline's performance. Removing refinement produces an even larger drop to 5.5%, supporting the importance of both components rather than an additive attribution of the total improvement to either one alone.

W4. GoldenMem retrieves author-code fragments through the graph/card linking and prompt-packing pipeline, rather than giving an unrestricted repository-browsing agent direct access to the complete official implementation, so topical similarity alone does not guarantee that the selected fragments can be integrated and executed successfully. The reported advantage is therefore a comparison of the complete Congram pipeline against this specific single-pass author-code baseline, not evidence that accumulated experience is intrinsically superior to official code under matched refinement and access conditions; we will make that scope explicit.

Reviewer_hex1OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Congram, a continual graph memory agent for AI research replication. It separates paper-specific information from reusable implementation patterns, organizes them into cards, and retrieves relevant memory cards and linked code assets to construct prompts for code generation. Generated submissions are refined by an LLM reviewer using a pruned PaperBench requirement tree, and the selected implementation is incorporated into the memory graph for future reuse. On Qwen3-Next and DeepSeek-R1, the paper reports improvements in cross-paper PaperBench scores of 35.6 and 24.6 percentage points, respectively, over GoldenMem, a single-pass baseline using the original paper implementations. The paper also analyzes the effects of requirement-tree depth and reports token savings in a separate setting without refinement.

Critical strengths
  1. The problem formulation is well motivated. Treating research replication as a sequence of tasks in which implementation patterns accumulate and are reused addresses repeated implementation work in agents that treat each paper independently. This framing is the paper’s main conceptual contribution and may be useful beyond PaperBench.
  2. The reported gains are consistent across papers. Congram achieves higher mean replication scores than GoldenMem on all 22 evaluated papers with both backbones (Tables 2–3).
Critical weaknesses
  1. The evaluation protocol differs from the original PaperBench setting. Congram uses PaperBench's grading rubric during refinement, whereas the original PaperBench protocol explicitly withholds it from agents to prevent overfitting (Starace et al., 2025, Sec. 2.1). Nevertheless, Table 1 includes o3-mini-high scores from the original PaperBench as reference baselines. Since rubric access and evaluated paper sets differ, these results are not directly comparable.
  2. Refinement feedback is not matched across methods. Congram uses PaperBench requirement-tree feedback, whereas IterativeAgent uses compile checks (L203–212, L228–231). Matching refinement iterations does not control for this difference, so the gains over IterativeAgent cannot be attributed to memory alone.
  3. The graph’s contribution is not sufficiently isolated. Table 4 suggests that both refinement and retrieved context contribute, but graph removal is tested only without refinement (5.5% versus 4.7%). A flat-memory baseline with the same code pool, reviewer feedback, and refinement budget is needed to establish the graph’s added value in the full pipeline.
  4. The memory baselines are ambiguously defined. NoMem is described as using refinement in Section 4 but excluding it in Section 4.1. Also, “No cumulative memory” removes retrieved code assets without clarifying whether accumulated memory cards and updates remain. These ambiguities make the contribution of continual accumulation difficult to interpret.
  5. Baseline comparisons are incomplete. Table 1 does not report BasicAgent and IterativeAgent results for DeepSeek-R1, making it difficult to assess Congram's improvements over standard paper-text-only baselines on this backbone.
Questions for authors
  1. How sensitive are the results to the choice of bootstrap paper P0P_0 and the order of subsequent papers?
  2. Please fix the figure placement on the References page (L540–554). Figure 4 currently appears among the reference entries.
1 comment
AuthorsOpenReview ↗

Author Response

We thank the reviewer for recognizing the motivation for studying paper replication as a sequence of related tasks. We appreciate the observation that reusing implementation work across tasks could also be useful beyond PaperBench. We address each reviewer’s concern and question in the answer below.

W1. Refinement uses only the Code Development subtree of the PaperBench rubric, pruned to depth 3, whereas the reported final scores use the full rubric after executing the selected submission, without feeding final-evaluation results back into refinement. We will explicitly identify this as a rubric-guided continual replication setting and remove the externally sourced o3-mini-high results from the main comparison table.

W2. The IterativeAgent comparison measures the complete pipeline, while Table 4 provides a comparison with rubric-guided refinement enabled in both Congram and the "No cumulative memory" ablation. Their scores of 38.5% and 16.1%, respectively, support the contribution of the cumulative-memory and retrieved-code configuration beyond rubric-guided refinement alone, rather than attributing the entire IterativeAgent gap to memory.

W3. Table 4 compares graph-based memory with flat code memory while disabling refinement in both configurations, obtaining 5.5% versus 4.7%, and separately tests removal of cards at 4.0%. We performed the evaluations of flat-memory no graph and no cards ablations with the same code pool, reviewer feedback, and refinement budget, obtaining full replication scores of 36.3% and 31.5%, respectively, averaged over three runs on the 22 target papers.

W4. NoMem in the submitted tables uses independent, single-pass generation from paper cards without memory cards or retrieved code assets, and the Section 4 statement that it includes refinement will be corrected. "No cumulative memory" retains each paper's own memory cards and rubric-guided refinement but excludes linked code assets, cross-paper memory accumulation, and memory updates; we will state these retained and removed components explicitly.

W5. The DeepSeek-R1 experiments establish the reported comparison against graph-based baselines, including GoldenMem, while BasicAgent and IterativeAgent were evaluated only with Qwen3-Next. We will make this coverage explicit and limit backbone-specific comparative claims to the evaluated configurations, rather than implying that the missing entries are measurements.

Q1. In preliminary experiments with Gemini 2.0 Flash, we varied both the bootstrap paper and the order of subsequent papers. The differences between the resulting average replication scores were statistically insignificant (Friedman test, p=0.58) the sequences we tested, so we kept one paper order choice for the main experiments.

Q2. Thank you for your suggestion. We will fix the figure placement.

Reviewer_GuR1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Congram, a continual graph-memory agent for paper-to-code research replication. The authors evaluate Congram on 23 PaperBench papers over three runs with Qwen3-Next and DeepSeek-R1, reporting average scores of 38.5% and 32.4%, respectively, along with per-paper results, ablations, memory-growth analysis, and token-cost comparisons.

Critical strengths
  1. The motivation and memory representation are well targeted at a real bottleneck. Research-replication agents often repeatedly rediscover generic implementation patterns such as training loops, data loaders, evaluation utilities, and repository scaffolds. Linking typed graph “cards” to concrete code assets is more interpretable than an opaque retrieval cache.

  2. The empirical scope is substantial. The paper evaluates over all 23 PaperBench papers, uses two backbone models, reports three-run averages and per-paper results, and includes ablations for cumulative memory, refinement, graph structure, card grouping, requirement-tree depth, and token cost.

Critical weaknesses
  1. The method’s review-and-repair loop uses the “pruned code development part of the PaperBench requirement tree,” but the reported final scores are PaperBench replication scores. The headline improvement may be dominated by benchmark-aligned feedback rather than by continual graph memory.

  2. The experiments use a fixed sequence of papers drawn entirely from PaperBench, with no randomized order, held-out paper stream, or external benchmark evaluation. The full-memory experiment therefore risks measuring in-benchmark transfer rather than transfer to genuinely future or unseen research. In addition, the DeepSeek-R1 BasicAgent/IterativeAgent baselines are missing.

Questions for authors
  1. Does the LLM reviewer used during refinement see the same PaperBench requirement tree that is later used for final scoring?

  2. Is Table 1 reporting the full PaperBench replication score or only a code-development sub-score?

1 comment
AuthorsOpenReview ↗

Author Response

We thank the reviewer for highlighting the interpretability of linking memory cards to concrete code and for recognizing the breadth of our evaluation across papers, backbones, and ablations. We address the concerns and specific questions below.

W1. During refinement, Congram uses feedback from only the Code Development portion of the PaperBench rubric, pruned to depth 3, to improve the candidate implementation. Final evaluation separately executes the selected submission and grades it against the full rubric, including Code Development, Execution, and Result Match requirements, without feeding that final evaluation back into refinement.

W2. We used Qwen3-Next as a main backbone for our experiments and performed both memory-augmented and memory-less baselines evaluation with it. DeepSeek-R1 was used as an additional backbone with evaluations focused mainly on memory-related baselines.

Q1: The refinement reviewer sees only the Code Development subtree of the same underlying PaperBench rubric, pruned to depth 3, whereas final scoring uses the full requirement tree, including Execution and Result Match requirements.

Q2: Table 1 reports the full PaperBench replication score obtained after executing the selected submission, not the code-development sub-score used during refinement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.