acceptodds
Under review as a conference paper at ICLR 2027

Growing an Agent/Prover Interface: Evolutionary Tool Design for Cost-Efficient Theorem Proving in Rocq and Lean

Abstract

Recent achievements in AI-assisted mathematics require intensive interaction of agents with proof assistants to generate machine-checked proof certificates. Agents interact with proof assistants such as Rocq or Lean through MCP servers that control what the agent receives from the prover and the cost of these interactions. Today, these interfaces are adapted from tools designed for humans and not optimized for agents. We propose an evolutionary method where a frontier model incrementally proposes new features and only keeps the ones that improve the overall performance of smaller models. We demonstrate the effectiveness of our method by growing on a curated set of mathematical problems a new MCP server for the Rocq prover: rocq-mcp-evolved. On the held-out test split of miniF2F-Rocq, an agent equipped with rocq-mcp-evolved outperforms both the baseline that only exposes the Rocq compiler and an established MCP server, across four models from two families, in success rate, cost per solve, and time per solve. Although evolved for Rocq, the resulting server transfers to Lean, improving cost and time per solve on a subset of PutnamBench. We release rocq-mcp-evolved and its port to Lean.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Agents, Theorem proving, Rocq, Lean, MCP servers, Tools

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RiF7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes an evolutionary procedure to design an agent/prover interface (an MCP server). A frontier model (Claude Fable 5) proposes one mutation at a time (new tool, output format, config), small "tester" models (Haiku 4.5, Sonnet 5) are run on a fixed problem set under tight budgets to evaluate a mutation. Starting from a compile-only CONTROL server, 14 Phase 1 mutations and 3 Phase 2 mutations yield ROCQ-MCP-EVOLVE (stateful session, try, auto close, hints, preloaded automation, verify). On the miniF2F-Rocq test split with four models it beats CONTROL and the existing ROCQ-MCP on accuracy, cost and wall time; the gain holds on five synthetic project tasks (Phase 2 mutations were all rejected, so these gains come from Phase 1 tools). A Lean port improves cost and time but not accuracy over LEAN-LSP-MCP.

Critical strengths
  • The evaluation of the resulting interface is solid: held-out test split, four models from two families, an anti-cheating verifier, and the full mutation ledger including reverted mutations.
  • Negative findings are reported honestly (e.g. search is heavily used but gives no accuracy gain).
  • The released server is a practical, cheaper artifact for Rocq.
Critical weaknesses

The contribution is ambiguous: the title and abstract sell the evolutionary method, but the evaluation only evaluates the resulting interface. Nothing tests the method itself: one trajectory, no seeds, and no comparison with ungated, all-at-once, or human-designed alternatives. The work applies a general method (LLM-driven evolution under a measured gate) to an arbitrary, under-optimized niche problem. Prover interfaces are not optimized for efficiency today, so almost any systematic tuning should show gains; this does not show that evolution is what produced them. There is no comparison with a proper baseline, such as a general iterative program/harness design framework . The related work only argues the differences (one feature per step, small tool set) in prose. It is not shown that these choices matter, so the novelty is largely the application domain.

Questions for authors

How does your evolution-based method compare against other iterative harness improvement methods (e.g. Meta-Harness, Agentic Harness Engineering, which you cite) at equal budget?

2 comments

We thank the reviewer for their comments.

Critical Weaknesses: The contribution is ambiguous: the title and abstract sell the evolutionary method, but the evaluation only evaluates the resulting interface. Nothing tests the method itself: one trajectory, no seeds and no comparison with ungated, all-at-once, or human-designed alternatives.

One trajectory. We ran a second, independent evolution of a Rocq MCP server, starting again from CONTROL with the same brief, validation rule, tester model and budgets, and with no access to the first run. It produced a different server. The first run built a persistent proof session. The second run stayed stateless, like CONTROL: the agent submits complete files, and all the improvements are in what the checking tool returns. The second run also kept a search tool, which the first run reverted. Both runs found two features: explanations of non-Rocq syntax, and suggestions of existing lemma names close to an unknown one. The evaluation of the second server on the miniF2F test split is running; we will post the results in this thread when it completes.

Ungated, all-at-once and human-designed alternatives. The paper already contains this comparison. rocq-mcp is an open-source MCP server for Rocq whose design was driven by its human authors. Its code was written by Claude Opus and refined over several campaigns and projects, with contributions from other users. Its tools were never validated one at a time against a measurement. rocq-mcp-evolve outperforms it on every model (Table 2).

What the evolution brings over a one-shot design. First, every tool of the final server is backed by its own measurement (Table 1), and these measurements rejected several plausible designs. A one-shot server would need a separate ablation study to obtain the same information. Second, the process can resume from any point, for example when a new model or a new kind of task appears. A one-shot design offers no such path.

The work applies a general method (LLM-driven evolution under a measured gate) to an arbitrary, under-optimized niche problem. Prover interfaces are not optimized for efficiency today, so almost any systematic tuning should show gains; this does not show that evolution is what produced them.

Our measurements do not support the claim that almost any tuning shows gains. Of the 14 mutations proposed in Phase 1, 5 were reverted, and all 3 mutations of Phase 2 were reverted. Each was a plausible improvement proposed by a frontier model. One is a search tool, which most existing prover interfaces provide: the agents used it heavily and accuracy did not improve. In addition, rocq-mcp is itself the result of tuning an interface from the logs of a prior experiment, and rocq-mcp-evolve outperforms it on every model.

On the choice of the problem: prover interfaces determine the cost of every machine-checked proof that an agent produces, and models are now trained against specific interfaces (Leanstral).

There is no comparison with a proper baseline, such as a general iterative program/harness design framework . The related work only argues the differences (one feature per step, small tool set) in prose. It is not shown that these choices matter, so the novelty is largely the application domain.

Questions For Authors: How does your evolution-based method compare against other iterative harness improvement methods (e.g. Meta-Harness, Agentic Harness Engineering, which you cite) at equal budget?

The reviewer is right that we do not show this, and we could not run the comparison during the rebuttal period. Neither framework provides an environment for a proof assistant. Running either at equal budget means porting it to an MCP server that drives Rocq, together with our anti-cheating verification, which is a project of its own.

We therefore do not claim that our method is better than these frameworks, and we will revise the abstract and the related work to say so. Our method belongs to the same family. What the paper shows is that a loop of this kind, with one feature per step and small tester models distinct from the designer, produces a prover interface whose gains hold on a held-out split, across four models from two families and on project tasks, and that its record of kept and reverted mutations tells interface designers which features matter. The second evolution reported above is a first test of the method itself.

We evaluated the server produced by the second evolution trajectory with Haiku and Sonnet. As described in our previous response, the orchestrator took a different approach in this run: it focused on giving more information rather than on making computation more efficient. The results are consistent with those presented in the paper. The resulting server, rocq-mcp-evolve-2, outperforms both CONTROL and rocq-mcp on accuracy, cost per solve and wall time per solve. It comes close to rocq-mcp-evolve on accuracy, and has a lower cost and wall time per solve when used by Sonnet. The detailed results are given in the table below:

ModelServerAccuracyCost / solve ($)Time / solve (s)
HaikuCONTROL.17.06149
Haikurocq-mcp.33.05232
Haikurocq-mcp-evolve.48.03117
Haikurocq-mcp-evolve-2.46.03226
SonnetCONTROL.50.24096
Sonnetrocq-mcp.72.16044
Sonnetrocq-mcp-evolve.80.11235
Sonnetrocq-mcp-evolve-2.77.05030
Reviewer_9NkdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a pipeline that evolves the tools agents use to interact with a proof assistant. A frontier model proposes changes to the available tools, and smaller models evaluate whether those changes improve performance. The resulting interface enables agents to outperform an existing Rocq MCP server. The authors also port the tools to Lean, where they reduce cost and time on solved problems.

Critical strengths

The approach successfully develops a compiler-only interface into a richer toolset, substantially improving proof accuracy and reducing resource usage on commonly solved Rocq problems.

Critical weaknesses
  • The compiler-only CONTROL baseline does not represent a practical modern coding agent with Bash access. Although the paper also compares against an existing MCP server, it is unclear whether evolving specialized interfaces has benefits over directly using a modern coding agent.
  • The paper reports changes in accuracy and efficiency after each evolution step, but it is unclear why individual changes help or hurt performance. Some analysis of this would help readers better understand how each tool helps the system.
  • Only one evolution trajectory is evaluated. It is unclear whether different evolution runs would consistently produce similar improvements. In a long evolution process, each step can introduce noise, so only one evolution trajectory is not enough to prove that the pipeline can consistently improve the interface.
Questions for authors

I don't have additional questions. If the authors can provide clarification or more results addressing the concerns in weaknesses section, I'm willing to increase my score.

3 comments

We thank the reviewer for their comments.

Critical Weaknesses: The compiler-only CONTROL baseline does not represent a practical modern coding agent with Bash access. Although the paper also compares against an existing MCP server, it is unclear whether evolving specialized interfaces has benefits over directly using a modern coding agent.

CONTROL is Claude Code restricted to the compiler; the other configurations are Claude Code restricted to their MCP server. This setup isolates the effect of the interface, but we agree that it leaves the reviewer's question open, so we ran new experiments where the baseline is a coding agent with bash access.

They use a 4-hour limit per problem on 46 competition problems (7 easy / 19 medium / 20 hard; 22 from PutnamBench, 24 from CombiBench) stated in Rocq. Every agent has bash access (no internet), with or without rocq-mcp-evolve. With open-weight models served locally, the server raises the number of problems solved (out of 46):

Modelbash only (B)bash + server (E)gained / lostGPU-hours, B / E
DeepSeek V4 Flash27325 / 0177 / 138
GLM-5.323296 / 01,101 / 983

For GLM-5.3, the run with the server also used a prompt that describes the tools and shows the clock; for DeepSeek V4 Flash both runs used it.

The bash agents run Python code for about 10 % of their time, but 80 % of the bash calls of the DeepSeek bash-only agent are compiler runs. So a coding agent with bash mostly does what CONTROL does, and the evolved interface still helps it.

The paper reports changes in accuracy and efficiency after each evolution step, but it is unclear why individual changes help or hurt performance. Some analysis of this would help readers better understand how each tool helps the system.

Reason for each verdict (to be added as a new column of Table 8):

MutationVerdictWhy
step, rollback, statekeptNo whole-file recompilation; raised accuracy on medium and hard problems.
trykeptUp to eight candidates tested in one call; helped most on easy problems.
compact renderingrevertedAccuracy dropped: the agent spent extra calls to fetch the hidden part of the proof state.
searchrevertedHeavily used (324 calls, in 66 of 120 attempts); took budget from other calls with no gain in accuracy.
Lean-ism hintskeptModels consistently write Lean syntax in Rocq; the hint turns a wasted call into a corrected one.
auto_closekeptRaised accuracy in every difficulty bucket.
near-miss hintskeptReal lemma names in unknown-reference errors raised accuracy on easy and medium problems.
preloadkeptSome correct proofs were rejected for adding a Require; preloading removed these rejections.
team of threerevertedRarely used, with little gain when used.
false-winner fixkeptBug fix.
real arithmetic helpkeptAutomatic closing of goals succeeds more often.
checkkeptRaised Sonnet's accuracy more than it lowered Haiku's; kept to avoid overfitting to Haiku.
SSReflect hintsrevertedNo substantial effect on accuracy.
exemplar retrievalrevertedNeutral to negative effect on accuracy.
open, build, verifyadded, not evaluatedNeeded for the project tasks.
hole visibilityrevertedRemaining admits were mostly due to the time limit, so showing them did not help.
finisher in openrevertedClosed only one goal in 70 open calls.
compact rendering (2nd)revertedAccuracy unchanged, but cost and input tokens rose.

Overall, the kept mutations give the agent more prover information in calls it already makes. The reverted ones either made the agent spend extra calls (search, team of three) or hid information from it (compact rendering).

Only one evolution trajectory is evaluated. It is unclear whether different evolution runs would consistently produce similar improvements. In a long evolution process, each step can introduce noise, so only one evolution trajectory is not enough to prove that the pipeline can consistently improve the interface.

We agree, so we ran a second, independent evolution, starting again from CONTROL with the same brief, validation rule, tester model and budgets, and with no access to the first run.

It produced a different server. The first run built a persistent proof session; the second stayed stateless, like CONTROL: the agent submits complete files, and all improvements are in what the checking tool returns. The second run also kept a search tool, which the first run reverted. Both runs found two features: explanations of non-Rocq syntax, and suggestions of existing lemma names close to an unknown one.

The evaluation of the second server on the miniF2F test split is running; we will post the results here when it completes.

Thank you for providing more results on bash-only agent and why each mutation is reverted or kept. I will decide my final score after you finish the complete evaluation on your second evolution run.

We evaluated the server produced by the second evolution trajectory with Haiku and Sonnet. As described in our previous response, the orchestrator took a different approach in this run: it focused on giving more information rather than on making computation more efficient. The results are consistent with those presented in the paper. The resulting server, rocq-mcp-evolve-2, outperforms both CONTROL and rocq-mcp on accuracy, cost per solve and wall time per solve. It comes close to rocq-mcp-evolve on accuracy, and has a lower cost and wall time per solve when used by Sonnet. The detailed results are given in the table below:

ModelServerAccuracyCost / solve ($)Time / solve (s)
HaikuCONTROL.17.06149
Haikurocq-mcp.33.05232
Haikurocq-mcp-evolve.48.03117
Haikurocq-mcp-evolve-2.46.03226
SonnetCONTROL.50.24096
Sonnetrocq-mcp.72.16044
Sonnetrocq-mcp-evolve.80.11235
Sonnetrocq-mcp-evolve-2.77.05030

The table below lists the mutations of the second trajectory, with their verdict and the reason for it:

MutationVerdictWhy
explained errors in checkkeptMost of Haiku's calls failed on a syntax error, mostly Lean syntax.
other errors of the file in checkrevertedFewer syntax errors, but the extra compilations took time.
proof state before the failing sentencerevertedNo measurable effect on accuracy.
searchkeptRaised accuracy on easy and medium problems and lowered cost.
closest lemma names in checkrevertedNo measurable effect once search exists.
search answers queries that find nothingkeptNearby queries and equivalent names instead of an empty answer.
division help for lra / nrakeptIt explains why these tactics fail on quotients.
definitions in searchrevertedLonger search results cost time.
check refuses a modified theorem statementrevertedFew corrections in practice, mostly Lean syntax.
fix of a false "unknown tactic" messagekeptBug fix.
fix of search reading warnings as lemmaskeptBug fix.
Reviewer_MHyxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies an evolutionary self-improvement loop in which a frontier model incrementally develops an MCP interface that helps smaller models solve formal theorem-proving tasks in Rocq. The resulting interface improves proof accuracy while reducing cost and wall time. When transferred to Lean, it reduces cost and wall time but does not improve proof accuracy over the compiler-only baseline.

Critical strengths

The paper is clearly written and easy to follow, with well-documented experimental and implementation details. The evolved Rocq interface consistently improves proof accuracy while reducing cost and wall-clock time across multiple models.

Critical weaknesses

While the MCP clearly improves model's performance in Rocq, the improvements do not transfer convincingly to Lean, where the evolved interface reduces cost and runtime but fails to improve accuracy. Moreover, the evaluation does not establish whether the approach can help models solve genuinely difficult problems, since no hard PutnamBench problems are solved and many miniF2F problems can be handled by standard automation.

Questions for authors
  1. How many failures, particularly on hard problems, result from reaching the 300-second timeout or 200-tool-call limit? Could the authors evaluate a larger budget to determine whether performance is limited by the evaluation constraints?

  2. Could the authors report accuracy after removing the 52 problems solvable by auto_close alone, or provide an ablation disabling auto_close and preloaded tactics? This would better isolate the contribution of agent–prover interaction.

  3. Cost and runtime are primarily calculated over successful attempts, while Table 2 further restricts evaluation to problems solved by all interfaces. Could the authors report total cost and runtime including failed attempts?

  4. The evolutionary process is nondeterministic, but the paper reports only one trajectory. Do independent runs discover similar tools and achieve comparable improvements? As a complementary robustness test, the authors could evolve an interface for Lean and transfer the resulting tools back to Rocq, revealing which features are general and which are prover-specific.

4 comments

We thank the reviewer for their remarks and suggestions. The new results below will be integrated in the revised paper.

Critical Weaknesses: While the MCP clearly improves model's performance in Rocq, the improvements do not transfer convincingly to Lean, where the evolved interface reduces cost and runtime but fails to improve accuracy. Moreover, the evaluation does not establish whether the approach can help models solve genuinely difficult problems, since no hard PutnamBench problems are solved and many miniF2F problems can be handled by standard automation.

Lean. We agree, and the paper reports this as a negative result. lean-mcp-evolve is a port of the tools evolved for Rocq; no evolution was run for Lean. It matches CONTROL on accuracy (.33), stays below lean-lsp-mcp (.42), and has the lowest cost and wall time of the three servers. An accuracy gain in Lean would require running the evolution for Lean, the next step stated in our conclusion.

Difficult problems. We ran new experiments with a 4-hour limit per problem on 46 competition problems (7 easy / 19 medium / 20 hard; 22 from PutnamBench, 24 from CombiBench), each stated in Rocq (Stdlib, MathComp). The model receives the statement ending with Admitted and returns a complete file. An independent checker rejects any change to the theorem or to the definitions it uses, and accepts only a closed proof. Sonnet reaches an accuracy of 0.76, and rocq-mcp-evolve lets two open-weight models solve 5 and 6 more problems than bash alone (details in our answer to question 1).

Standard automation. Our answer to question 2 shows that, on the problems that automation alone does not solve, the ordering of the three servers is unchanged.

  1. How many failures, particularly on hard problems, result from reaching the 300-second timeout or 200-tool-call limit?

No attempt reached the tool-call limit. Percentage of attempts stopped by the 300 s limit on the miniF2F test split (easy / medium / hard):

ModelCONTROLrocq-mcprocq-mcp-evolve
Haiku65 / 92 / 9039 / 80 / 8629 / 60 / 77
Sonnet32 / 63 / 838 / 35 / 408 / 26 / 31
Opus40 / 72 / 8712 / 43 / 569 / 40 / 47
Terra7 / 5 / 15 / 8 / 11 / 3 / 4

With the Claude models, the time limit ends 63–92 % of CONTROL's medium and hard attempts, against 26–77 % with rocq-mcp-evolve. Terra rarely reaches the limit: it stops by itself. We will add this table to the appendix.

Could the authors evaluate a larger budget to determine whether performance is limited by the evaluation constraints?

Yes. We ran the 46 competition problems described above with a 4-hour limit per problem, one attempt per problem and no call limit. In these runs the agent is a coding agent with bash access (no internet), with or without rocq-mcp-evolve.

Sonnet. The two runs solve the same number of problems (0.76 accuracy), but the run with rocq-mcp-evolve takes half the time and costs less:

Per problemrocq-mcp-evolvebash onlyratio
Wall time45.7 min93.6 min2.0×
Cost$18.74$25.901.4×
Tool calls2092681.3×
Output tokens216 k294 k1.4×
Input tokens (almost all cache reads)70.2 M107.2 M1.5×

Open-weight models served locally. Here rocq-mcp-evolve also raises the number of problems solved (out of 46):

Modelbash only (B)bash + rocq-mcp-evolve (E)gained / lostsolved within 300 s, B / EGPU-hours, B / E
DeepSeek V4 Flash27325 / 06 / 5177 / 138
GLM-5.323296 / 04 / 31,101 / 983

For GLM-5.3, the run with the server also used a prompt that describes the tools and shows the clock; for DeepSeek V4 Flash both runs used that prompt (as a measure of noise, two identical bash-only runs differed by 3 problems).

So the 300 s budget does limit absolute accuracy: within 300 s, both configurations solve only 3 to 6 of these problems. The problems gained with the server were solved after 71 to 240 minutes, and with Sonnet the server halves the time to reach the same accuracy.

  1. Could the authors report accuracy after removing the 52 problems solvable by auto_close alone, or provide an ablation disabling auto_close and preloaded tactics? This would better isolate the contribution of agent–prover interaction.

The table gives the accuracy on the 192 problems of the miniF2F test split that remain after removing the 52 problems solvable by auto_close alone. On these 192 problems the agent can still call auto_close inside an interactive proof: this is a subset analysis, not the ablation.

ModelServerAccuracy (easy / medium / hard)
HaikuCONTROL.05 (.11 / .01 / .00)
Haikurocq-mcp.17 (.32 / .07 / .04)
Haikurocq-mcp-evolve.34 (.52 / .25 / .12)
SonnetCONTROL.39 (.54 / .32 / .15)
Sonnetrocq-mcp.64 (.81 / .49 / .54)
Sonnetrocq-mcp-evolve.75 (.86 / .66 / .66)
OpusCONTROL.30 (.42 / .26 / .10)
Opusrocq-mcp.64 (.82 / .53 / .44)
Opusrocq-mcp-evolve.69 (.85 / .58 / .53)
TerraCONTROL.43 (.56 / .37 / .25)
Terrarocq-mcp.74 (.88 / .64 / .62)
Terrarocq-mcp-evolve.85 (.95 / .79 / .71)

The ordering is the same as on the full split for every model. rocq-mcp-evolve gains .29 to .42 over CONTROL and .05 to .17 over rocq-mcp, so the gains do not come from the problems that automation solves alone.

  1. Cost and runtime are primarily calculated over successful attempts, while Table 2 further restricts evaluation to problems solved by all interfaces. Could the authors report total cost and runtime including failed attempts?

The table gives, for every model and server, the totals over all 488 attempts of the miniF2F test split (244 problems × 2 runs), failed attempts included. "Per solve" is the total divided by the number of solved attempts, so failures are charged to the solves.

ModelServerAcc.Total cost ($)Total time (h)Cost / attempt ($)Time / attempt (s)Cost / solve ($)Time / solve (s)
HaikuCONTROL.1718334.60.372552.201499
Haikurocq-mcp.3319928.70.412121.23639
Haikurocq-mcp-evolve.4815924.90.331840.68380
SonnetCONTROL.5013427.10.272000.55401
Sonnetrocq-mcp.7216317.50.331290.47180
Sonnetrocq-mcp-evolve.8012215.40.251140.31142
OpusCONTROL.438828.70.182110.42489
Opusrocq-mcp.7217819.50.371440.51201
Opusrocq-mcp-evolve.7613417.90.271320.36175
TerraCONTROL.544518.60.091370.17253
Terrarocq-mcp.803012.60.06930.08117
Terrarocq-mcp-evolve.882310.00.05740.0584

The conclusions of Table 2 hold when failures are included. For every model, rocq-mcp-evolve has the lowest total time, cost per solve and time per solve. It also has the lowest total cost, with one exception: CONTROL with Opus spends less, because 278 of its 488 attempts are stopped by the 300 s limit after few calls.

  1. The evolutionary process is nondeterministic, but the paper reports only one trajectory. Do independent runs discover similar tools and achieve comparable improvements? As a complementary robustness test, the authors could evolve an interface for Lean and transfer the resulting tools back to Rocq, revealing which features are general and which are prover-specific.

We ran a second, independent evolution of a Rocq MCP server, starting again from CONTROL with the same brief, validation rule, tester model and budgets, and with no access to the first run.

Do independent runs discover similar tools? Only in part. The first run built a persistent proof session (step, rollback, state, try). The second run stayed stateless, like CONTROL: the agent submits complete files, and all the improvements are in what the checking tool returns. The second run also kept a search tool, which the first run reverted. Both runs found two features: explanations of non-Rocq syntax, and suggestions of existing lemma names close to an unknown one.

Do they achieve comparable improvements? The evaluation of the second server on the miniF2F test split is running; we will post the results in this thread when it completes.

Evolving for Lean and transferring back. This requires a full evolution for Lean, which we could not complete during the rebuttal period. It is the next step stated in our conclusion, and we agree that it is the right test to separate general features from prover-specific ones.

We evaluated the server produced by the second evolution trajectory with Haiku and Sonnet. As described in our previous response, the orchestrator took a different approach in this run: it focused on giving more information rather than on making computation more efficient. The results are consistent with those presented in the paper. The resulting server, rocq-mcp-evolve-2, outperforms both CONTROL and rocq-mcp on accuracy, cost per solve and wall time per solve. It comes close to rocq-mcp-evolve on accuracy, and has a lower cost and wall time per solve when used by Sonnet. The detailed results are given in the table below:

ModelServerAccuracyCost / solve ($)Time / solve (s)
HaikuCONTROL.17.06149
Haikurocq-mcp.33.05232
Haikurocq-mcp-evolve.48.03117
Haikurocq-mcp-evolve-2.46.03226
SonnetCONTROL.50.24096
Sonnetrocq-mcp.72.16044
Sonnetrocq-mcp-evolve.80.11235
Sonnetrocq-mcp-evolve-2.77.05030

To further address question 2, we reran CONTROL with the automation tactics preloaded and announced in its tool documentation. This removes from the comparison any gain that comes only from access to these tactics. This other baseline performs better, but the results remain consistent with what was stated in the paper: rocq-mcp-evolve still outperforms CONTROL and rocq-mcp on accuracy, cost per solve and wall time per solve. The only exception is that Sonnet equipped with CONTROL has a lower cost per solve than with rocq-mcp-evolve. Results for all models and servers are given in the table below:

ModelServerAccuracyCost / solve ($)Time / solve (s)
HaikuCONTROL (w/ preloading).30.06044
Haikurocq-mcp.33.05533
Haikurocq-mcp-evolve.48.03822
Haikurocq-mcp-evolve-2.46.03729
SonnetCONTROL (w/ preloading).74.10065
Sonnetrocq-mcp.72.22465
Sonnetrocq-mcp-evolve.80.16153
Sonnetrocq-mcp-evolve-2.77.09055
OpusCONTROL (w/ preloading).64.21870
Opusrocq-mcp.72.28870
Opusrocq-mcp-evolve.76.20861
TerraCONTROL (w/ preloading).75.08864
Terrarocq-mcp.80.04460
Terrarocq-mcp-evolve.88.02749
Reviewer_MHyxOpenReview ↗

Response

Dear Authors, Thank you for the detailed response. I appreciate the additional comparison tables, which confirms the proposed method achieves higher accuracy at lower computational and time costs compared to the non-evolution/bash baselines. However, I find the proposed evolution method fairly standard, as it largely relies on the LLM's existing knowledge and capabilities. I do not see a sufficiently novel methodological contribution beyond applying this existing approach to the problem. Therefore, I maintain my original score due to the limited methodological novelty.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.