- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The reviewer uses "human" to denote my hand-written opinions in the review and the reviewer uses "Opus" to denote Opus 5.5. The reviewer uses the incognito mode so no data will be shared to Claude.
Human:
This paper proposes a harness that incorporates continual graph memory, which involves multiple workers and verifiers. The authors propose curators to organize the graph, revisits stale annotations and record structured lessons. The scoped recall retrieves fact statements, and other findings with provance. The proposed system reports a 10/10 on FirstProof B2.
Critical strengths
Human:
-
This paper shows good self-review audit.
-
The proposed system demonstrates good performance in open problems.
Critical weaknesses
-
Human: the provided link cannot be accessed.
-
Human: Experiments have major flaws.
-
The grading schemes on First Proof B2 are not clear. It's not clear whether the authors invited expert mathematicians, and it's not clear how many of them are hired to grade each problem. Also, it's not clear whether the invited graders have the same standard as the official evaluation.
-
The proposed results on First Proof B2 (10/10) are too good to be true, especially with the weak model GPT 5.6.
-
The contribution states "An auditable empirical study of persistent mathematical research." But no proofs on First Proof B2 are submitted anywhere. The supplementary is empty. The reviewer is farmilar with all proofs in First Proof B2, but he cannot audit the proofs because they are not provided.
-
Minor revisions are not shown in the Table 1.
-
The compute may not be matched so it's an unfair comparison. The proposed method is a multi-worker system, so it might use significantly more tokens than baseline.
-
No enough ablations to isolate each part of the proposed system.
-
Danus cannot reach 7/10 as reported. The reviewer tried Danus but it could be much worse than 7/10 in First Proof B2. Detailed settings must be provided to support this results.
-
Opus also raised similar concerns which are verified by the reviewer:
The grading protocol for the headline 10/10 is not described, and may not match the baselines'. The official team results in Table 1 come from the benchmark's expert review. The Ansatz, Danus, Codex, and plain GPT-5.6 columns are labeled "Our reported results," but the paper never says who graded them. It doesn't say whether expert mathematicians reviewed them or under what rubric.The wording elsewhere suggests these may be self-assessed. The abstract and introduction say Ansatz "reports closure" or "local closure," and §3.2 defines "accepted" as accepted by the system's own LLM verifier. If the ✓ marks for Ansatz reflect internal acceptance rather than independent expert grading on the benchmark's scale, the main comparison is not valid. The authors should state exactly who graded each of the 40 new submissions, whether graders were blinded to the system, and release the final proofs.
Baseline comparisons are confounded by model and compute. All of the authors' runs use GPT-5.6 Sol, while the official teams used GPT-5.5 Pro or Gemini 3.1 Pro. So "substantially outperforms the official participating teams" largely reflects a newer backbone. Among the same-backbone systems, no compute is reported: tokens, cost, wall-clock time, number of worker sessions. Ansatz runs five worker lanes, a fresh verifier per submission, a curator, and strategy consultation. A gap from 4/10 (plain) to 10/10 may simply reflect orders of magnitude more inference. The introduction promises "matched resource and memory diagnostics from the earlier configuration," but these do not appear anywhere in the paper. Each system is also run once per task, with no indication of variance.
- Human: novelties might not be significant. The proposed system contains graph-based verifications, which has already been included in Danus, it's not sure what is the new contribution.
Opus also has concerns about the paper's novelty and ablation study:
The ablations do not support the memory contribution, which is the paper's title and stated novelty. Several problems compound here: Ablation A (flat notebook instead of the fact graph) shows no difference, 7/7 vs. 7/7. The authors acknowledge that graph structure of facts gives no measurable benefit. Ablation B (no exploration graph or curator) differs by a single task in a single run. Ablations C and D test the independent verifier and strategy consultation, which are not memory components. The independent verifier is inherited from Danus. The A/B task subset is exactly the set of tasks Danus already solves: {01, 02, 05, 06, 07, 08, 09}. The three tasks where Ansatz goes beyond Danus (03 Bernoulli, 04 Dilation, 10 von Neumann) are excluded from A and B. Task 04, the hardest and the one only Ansatz solves, is excluded from every ablation. So the ablations say nothing about which component produces the gains the paper highlights. The "continual" and "cross-problem" mechanisms are never evaluated. These are scoped recall and lessons. No experiment compares recall off vs. on, measures transfer across a sequence of projects, or shows that lessons change curator behavior. The scope setting used in the First Proof runs is not even reported.
Human: due to severe concerns, the reviewer decides to give a one to this paper.
Questions for authors
Human:
-
Who graded those submissions in Table 1, and using what protocol? What's the expertise and focus domains of each grader?
-
Since First Proof B2 solutions are already released, how can the authors prevent data contamination?
-
What is the key novelty against Danus and why is it useful?