- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes an evolutionary procedure to design an agent/prover interface (an MCP server). A frontier model (Claude Fable 5) proposes one mutation at a time (new tool, output format, config), small "tester" models (Haiku 4.5, Sonnet 5) are run on a fixed problem set under tight budgets to evaluate a mutation. Starting from a compile-only CONTROL server, 14 Phase 1 mutations and 3 Phase 2 mutations yield ROCQ-MCP-EVOLVE (stateful session, try, auto close, hints, preloaded automation, verify). On the miniF2F-Rocq test split with four models it beats CONTROL and the existing ROCQ-MCP on accuracy, cost and wall time; the gain holds on five synthetic project tasks (Phase 2 mutations were all rejected, so these gains come from Phase 1 tools). A Lean port improves cost and time but not accuracy over LEAN-LSP-MCP.
Critical strengths
- The evaluation of the resulting interface is solid: held-out test split, four models from two families, an anti-cheating verifier, and the full mutation ledger including reverted mutations.
- Negative findings are reported honestly (e.g. search is heavily used but gives no accuracy gain).
- The released server is a practical, cheaper artifact for Rocq.
Critical weaknesses
The contribution is ambiguous: the title and abstract sell the evolutionary method, but the evaluation only evaluates the resulting interface. Nothing tests the method itself: one trajectory, no seeds, and no comparison with ungated, all-at-once, or human-designed alternatives. The work applies a general method (LLM-driven evolution under a measured gate) to an arbitrary, under-optimized niche problem. Prover interfaces are not optimized for efficiency today, so almost any systematic tuning should show gains; this does not show that evolution is what produced them. There is no comparison with a proper baseline, such as a general iterative program/harness design framework . The related work only argues the differences (one feature per step, small tool set) in prose. It is not shown that these choices matter, so the novelty is largely the application domain.
Questions for authors
How does your evolution-based method compare against other iterative harness improvement methods (e.g. Meta-Harness, Agentic Harness Engineering, which you cite) at equal budget?
2 comments
We evaluated the server produced by the second evolution trajectory with Haiku and Sonnet. As described in our previous response, the orchestrator took a different approach in this run: it focused on giving more information rather than on making computation more efficient. The results are consistent with those presented in the paper. The resulting server, rocq-mcp-evolve-2, outperforms both CONTROL and rocq-mcp on accuracy, cost per solve and wall time per solve. It comes close to rocq-mcp-evolve on accuracy, and has a lower cost and wall time per solve when used by Sonnet. The detailed results are given in the table below:
| Model | Server | Accuracy | Cost / solve ($) | Time / solve (s) |
|---|---|---|---|---|
| Haiku | CONTROL | .17 | .061 | 49 |
| Haiku | rocq-mcp | .33 | .052 | 32 |
| Haiku | rocq-mcp-evolve | .48 | .031 | 17 |
| Haiku | rocq-mcp-evolve-2 | .46 | .032 | 26 |
| Sonnet | CONTROL | .50 | .240 | 96 |
| Sonnet | rocq-mcp | .72 | .160 | 44 |
| Sonnet | rocq-mcp-evolve | .80 | .112 | 35 |
| Sonnet | rocq-mcp-evolve-2 | .77 | .050 | 30 |
We thank the reviewer for their comments.
One trajectory. We ran a second, independent evolution of a Rocq MCP server, starting again from CONTROL with the same brief, validation rule, tester model and budgets, and with no access to the first run. It produced a different server. The first run built a persistent proof session. The second run stayed stateless, like CONTROL: the agent submits complete files, and all the improvements are in what the checking tool returns. The second run also kept a search tool, which the first run reverted. Both runs found two features: explanations of non-Rocq syntax, and suggestions of existing lemma names close to an unknown one. The evaluation of the second server on the miniF2F test split is running; we will post the results in this thread when it completes.
Ungated, all-at-once and human-designed alternatives. The paper already contains this comparison. rocq-mcp is an open-source MCP server for Rocq whose design was driven by its human authors. Its code was written by Claude Opus and refined over several campaigns and projects, with contributions from other users. Its tools were never validated one at a time against a measurement. rocq-mcp-evolve outperforms it on every model (Table 2).
What the evolution brings over a one-shot design. First, every tool of the final server is backed by its own measurement (Table 1), and these measurements rejected several plausible designs. A one-shot server would need a separate ablation study to obtain the same information. Second, the process can resume from any point, for example when a new model or a new kind of task appears. A one-shot design offers no such path.
Our measurements do not support the claim that almost any tuning shows gains. Of the 14 mutations proposed in Phase 1, 5 were reverted, and all 3 mutations of Phase 2 were reverted. Each was a plausible improvement proposed by a frontier model. One is a search tool, which most existing prover interfaces provide: the agents used it heavily and accuracy did not improve. In addition, rocq-mcp is itself the result of tuning an interface from the logs of a prior experiment, and rocq-mcp-evolve outperforms it on every model.
On the choice of the problem: prover interfaces determine the cost of every machine-checked proof that an agent produces, and models are now trained against specific interfaces (Leanstral).
The reviewer is right that we do not show this, and we could not run the comparison during the rebuttal period. Neither framework provides an environment for a proof assistant. Running either at equal budget means porting it to an MCP server that drives Rocq, together with our anti-cheating verification, which is a project of its own.
We therefore do not claim that our method is better than these frameworks, and we will revise the abstract and the related work to say so. Our method belongs to the same family. What the paper shows is that a loop of this kind, with one feature per step and small tester models distinct from the designer, produces a prover interface whose gains hold on a held-out split, across four models from two families and on project tasks, and that its record of kept and reverted mutations tells interface designers which features matter. The second evolution reported above is a first test of the method itself.