- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AI paper replication repeatedly rebuilds reusable implementation components. Congram stores paper-derived patterns and linked code in a growing graph, retrieves them for new papers, and refines submissions using rubric feedback. Across 22 target papers, it reports average scores of 38.5% and 32.4% on two backbones, exceeding its GoldenMem baseline by 35.6 and 24.6 percentage points.
Critical strengths
-
Reusing data processing, training, and evaluation components across replication attempts is a sensible direction for reducing repetitive implementation work.
-
The reported improvements are substantial. Congram achieves reported average scores of 38.5% and 32.4%, exceeding GoldenMem by 35.6 and 24.6 percentage points, respectively.
Critical weaknesses
-
The final scoring procedure needs further clarification. Congram receives target-rubric feedback during generation, whereas the imported original PaperBench baselines do not, preventing a controlled comparison. The authors should also clarify whether the reported scores are derived from the full evaluation including execution and provide the corresponding configurations.
-
The refinement ablation may also change retrieval. In the supplementary code, the refinement path uses lexical Jaccard matching, while the ordinary generation path uses Contriever embeddings. If these entry points produced Table 4, the score differences cannot be attributed solely to refinement.
-
The no-graph and no-card scores of 4.7% and 4.0% should be compared with the matched no-refinement score of 5.5%. Without a vector-RAG baseline using the same memory content, refinement procedure, and budget, the contribution of graph structure remains unclear.
-
Evidence for continual accumulation and resistance to forgetting is limited. Using just one bootstrap paper and a fixed task order, with no frozen-initial-memory control condition, cannot separate the gains brought by later updates. Likewise, a non-significant difference observed when revisiting only a single paper is not enough to rule out catastrophic forgetting.
-
The efficiency analysis does not cover the complete system. Reported token savings concern single-pass generation without refinement and do not cover the full costs of judging, repair, and memory updates. The efficiency of the complete pipeline achieving the main scores therefore remains unclear.
-
Minor: (1) Page 5, line 256: “plan paper text” should be “plain paper text.” (2) Page 11, Figure 4 caption, line 552: “Mmeory” should be “Memory.”
Questions for authors
See weaknesses
Author Response
We thank the reviewer for highlighting how reusing data-processing, training, and evaluation components can reduce repeated implementation work across papers, and for noting the improvements in replication scores. We address the weaknesses in the answer below.
W1. The reported main scores are full PaperBench replication scores obtained after executing the selected submission, with code_only=false and skip_reproduction=false; refinement instead uses only the depth-3 Code Development subtree for up to five repair iterations, without receiving final-evaluation results.
W2. The supplementary refinement driver uses the shared hybrid retriever with lexical overlap as its relevance signal, a supported configuration that avoids requiring embedding inference. Although the reviewer correctly identifies different defaults across entry points, this does not establish that refinement changes retrieval: disabling refinement within the same driver (max_refine_iters=0) preserves retrieval and prompt construction, allowing refinement to be isolated.
W3. The appropriate submitted single-pass comparison is indeed 5.5% for Congram without refinement versus 4.7% without graph-based memory and 4.0% without cards, so we will make that matching explicit. We have additionally completed refinement-enabled no-graph and no-cards experiments, scoring 36.3% and 31.5% over 22 papers and three runs.
W4. The paper provides evidence for the value of accumulated experience: removing cross-paper memory reuse lowers the score from 38.5% to 16.1% while retaining rubric-guided refinement. Revisiting an earlier paper yields no statistically significant degradation on either backbone, supporting retention in the tested setting. Regarding frozen-initial-memory control condition, in No cumulative memory ablation we exactly satisfy the suggested condition: we forbid memory updates during the sequential processing of the papers. 5. The reported token savings measure prompt and completion tokens for single-pass code generation without refinement, isolating the generation-stage benefit of memory reuse. We will label this scope explicitly and distinguish it from the full pipeline's cost.
W6. We thank the reviewer for suggestions, and we will fix the misprints in the paper text.