- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Authors argue that current methods are optimizing the evaluator verdict without actually interpreting it and governing it in a right way. In this regard, they introduce a new harness/scaffold that tries to improve on this limitation. They empirically validate their method on a set of well-known benchmarks in an open-ended discovery domain and show its superior performance over a set of baselines. Furthermore, authors study several cases of novel discoveries their system produced.
Critical strengths
The paper’s most compelling contribution is a portfolio of ten verifiable results across algebraic complexity, selection, reversible circuits and combinatorics (e.g., eleven improved border-rank bounds, a 54-comparison median-of-23 procedure, a 14.1% T-count reduction for AES, a Lean-checked matroid counterexample family). Each result comes with explicit certificates, and the paper is careful to scope what is and is not proven (Table 4).
Critical weaknesses
The empirical evaluation does not adequately establish the incremental contribution of EPOCH's evidence-governance architecture.
The paper reports encouraging aggregate benchmark results and an interesting portfolio of discovery outcomes. However, the evaluation lacks several simple but important baselines that would establish whether these improvements result from EPOCH's proposed mechanisms rather than from the underlying model, evaluator, or search budget.
This concern is supported by recent empirical findings. Gideoni et al. (2025) show that random LLM program sampling can match sophisticated code-evolution methods on several tasks. Wang et al. (2026) demonstrate that automatic harness evolution does not consistently outperform simple test-time scaling under matched inference and feedback budgets. More directly, Brilliantov et al. (2026) find that a minimal, single-session coding agent can match sophisticated autonomous ML-engineering harnesses under controlled comparisons.
These results make it important to include (i) best-of-N independent proposals, (ii) multiple short autonomous agent runs, and (iii) a single long-running coding agent, using the same underlying model and comparable budgets.
Moreover, the comparison is based on one reported model configuration and equalized primarily through semantic-call counts, without comprehensive inference-cost accounting. This is important because recent studies show that harness choice can substantially affect cost without producing commensurate performance gains (Pan et al., 2026; Liu & Han, 2026). A model–harness quality–cost Pareto analysis would provide a substantially stronger evaluation.
Finally, Tables 1–3 report aggregate means without sufficient uncertainty characterization, while the task-level win/tie/loss comparisons are mixed. Reporting seed-level dispersion, paired confidence intervals, and matched-budget comparisons is necessary to assess the robustness of the claimed performance advantages.
Why this is consequential: The ablation results (Table 3) provide evidence that EPOCH's components improve aggregate performance relative to their removal on the six AgentHPOBench tasks. However, these experiments do not establish that evidence governance provides an advantage over alternative search strategies or simpler autonomous agents. Consequently, the reported benchmark gains cannot be attributed specifically to the proposed governance mechanisms rather than other differences in search strategy, computational expenditure, or implementation. Given recent evidence that simple sampling and minimally orchestrated agents can be highly competitive, matched-budget comparisons against these alternatives are essential. Without them, the paper demonstrates that its components are useful within EPOCH, but not that the proposed architecture offers a meaningful advantage over simpler approaches.
Insufficient attribution of the discovery results to EPOCH's proposed architecture.
The paper's strongest contribution is arguably its portfolio of ten mathematical and algorithmic discovery cases. These are interesting results, and I appreciate the effort to provide verifiable artifacts and distinguish established claims from remaining proof obligations. However, the experiments do not establish how much the proposed evidence-governance architecture contributes to obtaining these discoveries.
In particular, the paper does not provide a sufficiently detailed account of the model, computational budget, number of attempts, and human involvement for each discovery. More importantly, none of the discovery problems is evaluated with a comparable baseline, making it difficult to distinguish the contribution of EPOCH from the capabilities of the underlying model, extensive search, or human-guided experimentation.
This is especially consequential given prior demonstrations of AI-assisted mathematical discovery. Georgiev et al. (2025) already demonstrated AlphaEvolve's ability to produce mathematical advances across a portfolio of 67 problems. More recently, general-purpose frontier models have also demonstrated substantial mathematical discovery capabilities, including an autonomous disproof of a longstanding Erdős conjecture (OpenAI, 2026). Thus, demonstrating that an AI-based system can produce mathematical discoveries does not, by itself, establish the significance of a particular agent architecture.
Why this matters: Because the discovery portfolio is central to the paper's contribution, the absence of controlled comparisons substantially limits what can be concluded about EPOCH's effectiveness. At minimum, I would expect detailed per-discovery provenance and matched-budget plain-agent baselines on a representative subset of these problems. A comparison showing that EPOCH enables less expensive models to achieve discoveries that otherwise require substantially more capable or costly agents would provide an additional, compelling contribution. As presented, the paper demonstrates interesting discovery outcomes but does not establish the marginal value of EPOCH in producing them.
Questions for authors
Q1. Simple baselines and the incremental value of evidence governance.
Have the authors evaluated EPOCH against simpler baselines, such as best-of-N independent proposals, multiple short autonomous agent runs, or a single long-running coding agent, using the same underlying model and comparable inference budgets? If such experiments have been conducted but are not reported, could the authors share the results, including computational costs? These comparisons would help clarify whether EPOCH's governance mechanisms provide a meaningful advantage over simpler approaches.
Q2. Discovery provenance and attribution.
Could the authors provide more details about how the ten reported discoveries were obtained? In particular, which model(s) were used, what were the computational budgets and numbers of attempts per problem, and how much human intervention was involved in problem formulation, search, candidate selection, and verification? Were these discoveries produced through autonomous EPOCH runs, or did they involve additional manual experimentation and refinement? Clarifying this process, ideally with representative search trajectories and success rates across attempts, would help assess how much of the reported discovery capability can be attributed to EPOCH itself.