acceptodds
Under review as a conference paper at ICLR 2027

Interactive Proofs as a Unified Framework for Evaluating and Leveraging Verifiable Proof Generation by LLM Agents

Abstract

Recent advances in LLM-assisted theorem proving and program verification have attracted widespread attention. Beyond determining whether a proposition is true, LLM agents must produce evidence that can be independently checked by humans or formal verification tools. To evaluate the underlying, tool-agnostic ability of LLM agents to construct verifiable proofs—rather than their proficiency with any particular proof language or verification tool—we introduce Merlin-Bench. Drawing on interactive proof systems from computational complexity theory, we cast an LLM agent as the prover: under a designated protocol, the agent interacts with a formal verifier to substantiate its claimed answer to a problem instance. We develop an automated framework for constructing test cases and use it to create 100 cases across 15 problem families, each formulated within this prover-verifier framework. Experiments with 10 frontier LLMs across multiple agent harnesses show that reliably producing verifiable proofs in accordance with specified protocols remains a substantial challenge. Our case studies further provide actionable insights into improving agents’ ability to generate verifier-accepted proofs: tool use is essential for completing practical proof tasks; different forms of guidance, ranging from engineering instructions to problem-specific knowledge, substantially affect success rates on interactive-proof tasks; and proof-solving experience distilled from abstract tasks can transfer to concrete, real-world verification tasks.

Primary area:
datasets and benchmarks
Keywords:
LLM agent, Benchmark, Interactive proof

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T27iOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper, if I understand correctly, attempts to strip away the implementation-specific factors of Interactive Theorem Proving (ITP) across different systems (e.g., Lean) by returning to the underlying theoretical structure of ITP, and thereby, from the perspective of Theoretical Computer Science (TCS), proposes a way to measure the "tool-agnostic ability of LLM agents to construct verifiable proofs."

The benchmark itself is organized into three tiers: (1) implementing six classes of classical, widely studied algorithms (like network flow); (2) solving six classes of canonical Interactive Proof (IP) problems from computational complexity theory (like those used in crypotograph); and (3) in line with existing ITP and ATP benchmarks, a set of proof tasks for Lean, Dafny, and DRAT. Tier 3 by itself is fairly conventional.

Critical strengths
  1. To me, it is novel to use notions from TCS (e.g., the IP class) to systematically construct a benchmark.
  2. The authors' goal of measuring the "tool-agnostic ability of LLM agents to construct verifiable proofs" is meaningful from a theoretical standpoint.
Critical weaknesses
  1. The authors' methodology is questionable with respect to achieving the goal they set out.

It should be noted that research in the ITP community draws its tools mainly from proof theory (a branch of mathematics), rather than being based on the IP class. Research in the ITP field rarely relies on the notion of IP, let alone uses it directly. The six problem families the authors present in Tier 2 are generally not regarded as representative proof challenges within the ITP community. The reviewer thus finds that the paper lacks sufficient justification for the claim that algorithm implementation (Tier 1) and IP-class problems (Tier 2) can represent the "ability of LLM agents to construct verifiable proofs" as that ability is understood in day-to-day ITP practice.

  1. It is unclear exactly which capability of LLMs the authors intend their benchmark to measure.

The "tool-agnostic ability of LLM agents to construct verifiable proofs" is too broad, because ITP is used across many different application domains: some benchmarks measure the ability to do frontier mathematical research (e.g., FATE, PutnamBench), some measure program verification ability (those based on Dafny and Verus), and some measure LLMs' reasoning ability on logical puzzles. Could the authors clarify which specific LLM capability their benchmark is meant to measure, and further justify why algorithm implementation (Tier 1) and IP-class problems (Tier 2) indeed represent the capability they target?

Many of the problems in Tier 1 and Tier 2 have been extensively studied in the literature. An LLM agent under evaluation could achieve high scores simply by reproducing existing state-of-the-art results; going beyond those scores, on the other hand, would amount to making progress on classical problems that are each of considerable research value in their own right, which is extremely difficult and, if achieved, would itself be a remarkable breakthrough in the field of Theoretical Computer Science (TCS). This further confuses the reviewer as to exactly which capability of the agent the benchmark is meant to measure, and whether it can represent, as the authors mention at the outset, theorem proving for frontier mathematics and program verification.

  1. The paper's comparison with prior work (Table 1) is not really defensible.

The paper criticizes MiniF2F for not being "agent-oriented," yet much recent work tackles MiniF2F precisely with agents (Delta Prover, AoA, Numina-Lean-Agent); on SATBench and similar benchmarks, one could take SAT/SMT solvers as baselines and require LLM agents to invoke SAT/SMT solvers in order to outperform the scores of the solvers alone. The paper criticizes SATBench, etc provide no "rigorous certificate," but they are verified by SAT solvers, despite differences in the trusted base that are acceptable in most cases. The paper emphasizes "scalable synthesis," but the authors' benchmark is likewise generated from only a handful of problem families, while using LLMs to synthesize miniF2F-like data has already become standard practice.

Questions for authors

I would invite the authors to respond my concerns in the weakness section.

6 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for the detailed feedback and valuable perspective. We would first like to clarify our interpretation of the issues you raise.

W1. The authors' methodology is questionable with respect to achieving the goal they set out.

We first clarify the distinction between interactive proofs and ITP. In interactive theorem proving, a prover establishes a proposition in a language such as Lean. The prover interacts with the system of the proof assistant. The system presents the current goal, the prover supplies the tactics to apply next, and the system updates the goal until it becomes ⊤\top (or no goal remains, which indicates the proof term passes the kernel's type-checking). Lean, for example, implements a deterministic algorithm based on the Calculus of Constructions, whose soundness follows directly from that of the underlying type theory. In the language of TCS, when the verifier in an interactive proof protocol is deterministic, the complexity class of problems it can verify is analogous to DIP=NP⊆IP\textsf{DIP} = \textsf{NP}\subseteq \textsf{IP}.

We define unified proof ability in item 1 of our response to all reviewers. The interactive proof framework contains but is not limited to the ITP protocols you describe and other forms of verification. Consider the claim that an LLM agent holds the private key of a public-key cryptosystem. The verifier repeatedly sends the prover a random ciphertext c=Enc(m)c = \textsf{Enc}(m), asks it to compute Dec(c)\textsf{Dec}(c), and checks whether the result equals mm. We believe our IP-based framework is better suited than ITP to formalizing this kind of practical interaction.

As you point out, our Tier 1 and Tier 2 problems are not standard proof challenges in the ITP community. However, this is consistent with our motivation. TCS offers a wide range of IP problems for which our framework can generate instances in bulk. We therefore ask whether these sources of problems can provide a suitable dataset both for benchmarking and, potentially, for post-training. We designed the transfer experiment in Case Study 3 (Section 5.3) to test this idea. After we distilled lessons from proof transcripts on Tiers 1 and 2, the models' performance on Tier 3 also improved.

W2. It is unclear exactly which capability of LLMs the authors intend their benchmark to measure.

We evaluate unified proof ability as we define it. Conducting frontier mathematics research, constructing mathematical objects, and writing code are distinct capabilities, and we caution against interpreting unified proof ability as a combination of those capabilities. We gave the following definition in our response to all reviewers.

Unified proof ability. By unified proof ability, we mean the ability of an LLM agent to convince a human verifier, through a specified protocol and with high confidence, that a claim is true when the verifier does not know the answer in advance.

The verifier is free to specify the protocol, which can extend beyond a particular formal verification system such as Lean. In practice, the human verifier and the LLM agent indeed agree on a protocol Π\Pi and interact according to it. If the interaction meets the agreed conditions, the verifier accepts the claim. The protocol can cover rigorous mathematical proofs, formal verification code, and tasks that allow small completeness and soundness errors (such as our GNI task). Bounding the number of rounds and the verification cost of Π\Pi by low-degree polynomials gives the setting of our benchmark. Our benchmark therefore evaluates this ability.

Regarding the concern in your second paragraph, asking an LLM to reproduce state-of-the-art results from TCS research would mischaracterize our evaluation goal. We can select benchmark instances with particular mathematical structures. Even when a solver cannot handle a problem at the chosen scale, an LLM can examine the structure of the instance and devise a proof strategy tailored to it. Identifying that structure is itself a necessary and nontrivial part of solving problems and developing verification strategies in practice. Case Study 2 (Section 5.2) provides evidence for this point. When we gave the LLM agents a hint identifying the hidden structure, their scores improved substantially.

AuthorsOpenReview ↗

Rebuttal (2/2)

W3. The paper's comparison with prior work (Table 1) is not really defensible.

  • Regarding whether MiniF2F is agent-oriented, we would emphasize the word oriented. In our response to all reviewers, we give two criteria.

    (1) A bare LLM, without a harness or tools, has difficulty solving the benchmark problems. (2) When LLM agents can write code and search the web, the benchmark still distinguishes their performance.

    MiniF2F was originally designed without agents in mind and appears to fall short of these criteria. In our Case Study 1, we show that a bare LLM can solve many MiniF2F-style problems. And when we allow LLM agents to search the web for answers, they can solve nearly all of them.

  • Regarding rigorous certification in SATBench, the original paper requires the LLM being evaluated to provide its reasoning for an UNSAT instance. An LLM judge checks that reasoning. This step falls short of our definition of rigorous certification.

    Subsequent studies using SATBench, such as VeriTrans (Liu et al. 2026), use a somewhat different evaluation setup from the original paper. SATBench generates test instances in three steps. (1) It generates UNSAT instances containing contradictory clauses. (2) It removes some clauses from an UNSAT instance and retains the result as a SAT instance if a SAT solver confirms it is satisfiable. (3) It converts all instances into natural language puzzles to test the model's reasoning ability. Since a SAT solver can distinguish the resulting instances, an agent can translate the natural language puzzle into CNF and call the same solver. This departs from the benchmark's original aim of evaluating the model's reasoning ability. Our assessment of the first three papers highlights what we see as a fundamental limitation of using only NP problems to evaluate models. When obtaining answers is too costly, agent-oriented evaluation, rigorous certification, and scalable synthesis cannot all be achieved at once. This motivates our use of IP\textsf{IP} for model evaluation.

  • Regarding scalable synthesis, our benchmark contains only 15 problem families, but it supports synthesis at two levels. (1) Each test case specifies a way to generate a series of instances with a given property. In this sense, each test case offers data generation capabilities comparable to those of a benchmark such as SATBench. (2) The data construction pipeline in Section 3.2 can generate test cases at scale within a single family. The original MiniF2F draws on a finite collection of mathematics olympiad problems. Synthesizing further problems also entails a trade-off between generating answers, so that the designer knows a test statement is true, and retaining the benchmark's ability to distinguish among agents. For these reasons, we consider the original MiniF2F to fall short of our scalable synthesis criterion.

Thank you again for your detailed feedback. We welcome any further questions you may have. If you feel that your questions have been addressed, we hope you will consider raising your score.

[1] Liu et al. 2026. VeriTrans: Fine-Tuned LLM-Assisted NL-to-PL Translation via a Deterministic Neuro-Symbolic Pipeline.

I do not think one of my major concerns has been addressed. In their rebuttal, the authors attempt to clarify the distinction between interactive proofs (IP) and interactive theorem proving (ITP). However, the main claimed novelty, the unified proof ability, rests solely on the former, while the abstract and introduction motivate the work almost entirely through the latter, ITP and program verification (see, e.g., the opening sentence of the abstract: “Recent advances in LLM-assisted theorem proving and program verification…”). This creates a mismatch between how the work is framed and what it actually contributes. In particular, it remains unclear why IP-based benchmark cases can be representative of day-to-day ITP practice. To be candid, the framing gives the impression that the paper leverages the current momentum around ITP and program verification to motivate a benchmark that, in practice, serves a different purpose.

Reviewer_KrYoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Merlin-Bench: 100 generator-defined cases across 15 families for evaluating agents that construct certificates or interact with executable verifiers. Experiments compare agents and study tools, prompting, and transfer.

Critical strengths

The diverse tasks, explicit protocols, and executable verifiers form a potentially useful resource for evaluating complete proof-producing agent systems.

Critical weaknesses

The contribution is more as a benchmark than as a new evaluation paradigm. Amit et al. (2025) already formulate learned provers interacting with sound executable verifiers. A common interface does not establish that Merlin-Bench measures a unified, tool-independent proof ability rather than a mixture of reasoning, coding, and execution skills. Evidence for transfer is limited: study-guide prompting yields 2–4 additional successes on 24 generated tasks using real verification tools, relative to zero-trace guide controls. No repeated-run uncertainty is reported, and these are not external real-world verification benchmarks.

Questions for authors
  1. What is new in the proposed framework beyond the benchmark itself, and what evidence supports interpreting its score as a unified proof capability?
  2. Do the transfer gains persist across repeated runs and on independent verification benchmarks?
4 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

Thank you for reading our paper carefully. We address your questions below.

Q1.1 What is new in the proposed framework beyond the benchmark itself

Both our work and Amit et al. (2025) draw on the concept of interactive proofs. To make model outputs more trustworthy, Amit et al. define a self-proving model as one that both outputs an answer and completes an interaction with VV. They focus on how to train a self-proving model for a given VV.

We use theoretical properties of the complexity class IP\textsf{IP} to establish why interactive proofs are suited to model evaluation for the first time. We also show how familiar problems in IP\textsf{IP} from theoretical computer science can be used to build large datasets. Amit et al. use IP for modeling and do not examine the properties of the class or the problems it contains in comparable depth. Two points illustrate this distinction.

  • We address how to organize an evaluation when the label of an instance is unknown. Of the two algorithms proposed by Amit et al. (2025), TL requires a labeled dataset containing both answers and proofs. RLVF requires the verifier to receive an acceptable proof of the correct answer with appreciable probability before training. Their full pipeline hence run TL first, and RLVF subsequently. In this sense, their training data depend on access to answers in the TL step. For evaluation, the procedure in Appendix C.2 suffices to handle instances whose answers are unknown. This makes model evaluation a particularly natural application of IP.
  • Amit et al. (2025) use a fixed verifier during training. If we use RLVF or a similar verifier-based method for LLM post-training, which VV should we choose to obtain both enough data and improvements that generalize? Our Tier 1 and Tier 2 constructions show that familiar problems in theoretical computer science provide ample data. Case Study 3 suggests that capabilities learned on instances of these problems could also improve performance on several real-world tasks. This suggests a promising direction for LLM training.

Q1.2 what evidence supports interpreting its score as a unified proof capability

Unified proof ability should not be interpreted as a combination of capabilities such as conducting mathematics research and writing code. We gave the following definition in our response to all reviewers.

Unified proof ability. By unified proof ability, we mean the ability of an LLM agent to convince a human verifier, through a specified protocol and with high confidence, that a claim is true when the verifier does not know the answer in advance.

The verifier is free to specify the protocol, which can extend beyond a particular formal verification system such as Lean. In practice, the human verifier and the LLM agent indeed agree on a protocol Π\Pi and interact according to it. If the interaction meets the agreed conditions, the verifier accepts the claim. The protocol can cover rigorous mathematical proofs, formal verification code, and tasks that allow small completeness and soundness errors (such as GNI task). Bounding the number of rounds and the verification cost of Π\Pi by low-degree polynomials gives the setting of our benchmark. Our benchmark therefore evaluates this ability.

We'll explain this ability in more detail in the revised version.

AuthorsOpenReview ↗

Rebuttal (2/2)

Q2. Do the transfer gains persist across repeated runs and on independent verification benchmarks?

We conducted additional experiments to address your questions.

  • Stability of the results in Case Study 3. We ran three evaluations on the original 24 Tier 3 test cases. We also used our automated data synthesis system to double the number of Tier 3 cases and ran three independent evaluations on the full set of 48. The table reports the mean scores and sample standard deviations.

    SettingLean (8)Dafny (8)DRAT (8)Total (24, %)Lean (16)Dafny (16)DRAT (16)Total (48, %)
    Sol + 05.3 ± 0.64.7 ± 2.55.7 ± 0.665.28 ± 10.4911.3 ± 0.67.7 ± 2.512.0 ± 1.064.58 ± 5.51
    Sol + Sol5.0 ± 1.06.0 ± 1.76.3 ± 1.572.22 ± 13.3912.3 ± 0.610.0 ± 3.512.7 ± 1.572.92 ± 9.55
    Terra + 03.7 ± 0.65.7 ± 1.22.3 ± 1.248.61 ± 2.415.3 ± 1.27.7 ± 1.24.3 ± 0.636.11 ± 4.34
    Terra + Terra3.3 ± 0.65.7 ± 0.64.7 ± 0.656.94 ± 2.415.7 ± 1.58.0 ± 1.77.0 ± 1.743.06 ± 1.20
    Terra + Sol4.0 ± 1.05.7 ± 0.65.3 ± 0.662.50 ± 4.177.3 ± 1.58.7 ± 0.68.7 ± 1.251.39 ± 6.36
  • Performance on external benchmarks. We tested our method on tasks from CombiBench (Liu et al. 2025) and AlgoVeri (Zhao et al. 2026), which evaluate Lean and Dafny, respectively. For each target benchmark, we asked the model to extract relevant lessons from existing Tier 1 and Tier 2 trajectories in Merlin-Bench (as TCS guide).

    SettingAlgoVeri·DafnyCombiBench
    Sol + 084.4%54.0%
    Sol + Sol83.1%58.0%
    Terra + 074.0%44.0%
    Terra + Terra75.3%48.0%
    Terra + Sol77.9%49.0%

We will add these results to the revised version. Thank you for your valuable suggestions. Please let us know if you have any further questions, and we sincerely hope you may improve the rating if you find our clarifications helpful.

[1] Amit et al. 2025. A Theory for Worst-Case vs. Average-Case Guarantees for LLMs.

[2] Liu et al. 2025. CombiBench: Benchmarking LLM Capability for Combinatorial Mathematics.

[3] Zhao et al. 2026. AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms.

Thank you for your answer.

Re my first question (Q1-2), I do not see how your definition applies to the GNI task as evaluated: each pair is built with a 'simple' invariant (bipartiteness, 4-clique...) so the challenge for the agent is to find which invariant applies. I do not see what interaction adds in this case.

I do not understand what you mean by "For each target benchmark, we asked the model to extract relevant lessons from existing Tier 1 and Tier 2 trajectories in Merlin-Bench (as TCS guide).

Thank you for your response.

  1. Regarding the definition, there is no contradiction: here, "interaction" refers to the process by which a model presents its reasoning in a way that is convincing to humans. In the case of GNI, such interaction is essential for arriving at conclusions that humans can accept with high probability (proved by TCS theory, Multi-turn is necessary). In more general cases, though models need to identify which invariant applies, the ability to employ an appropriate protocol to communicate this in a human-convincing manner serves as a demonstration of "unified proof ability."

  2. Regarding the transfer experiments: our core argument is that synthetic data generated via TCS can be valuable for real-world tasks (as a dataset & benchmark paper). Therefore, our trajectories were obtained through rollouts performed on our own datasets. However, in our initial experiments, the extracted information inadvertently included details specific to the evaluation harness (e.g., the model might have "learned" constraints like the 10-minute time limit, as the same evaluation framework was used for both Tier 3 and Tiers 1/2). Such information would be inappropriate for external benchmarks. Therefore, we filtered out these elements to ensure a more rigorous and valid evaluation when applying our approach to the external benchmarks you mentioned. So extract refers to the process of filtering out these unsuitable elements from the original case study 3.

Thank you very much for your time. Please let us know if you have any further questions?

Reviewer_JVJtOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission proposes a benchmarking tool addressing limitations of the current systems. In particular, the tool is intended for agent-based use, scalable, enabling certified outcomes, and verifying unified proof ability. The tool currently spans 100 test cases, and it's application is illustrated with 3 case studies.

Critical strengths

The benchmark uses rigorous, automatically checked verification, which avoids relying on unreliable model-based judging.

Critical weaknesses

The benchmark as presented is relatively small and its results may not be reliable enough.

Questions for authors

The paper does not define “unified proof capabilities”. Please define this capability, and provide evidence that the benchmark actually measures it. Also please explain why and how this benchmark could be used to produce reliable outcomes in testing agents given its relatively small size.

2 comments

Thank you for your questions and positive feedback. We address your questions below.

Q1. Please define this capability, and provide evidence that the benchmark actually measures it.

In our response to all reviewers, we address your first question about how we define unified proof ability and why our benchmark evaluates it.

Unified proof ability. By unified proof ability, we mean the ability of an LLM agent to convince a human verifier, through a specified protocol and with high confidence, that a claim is true when the verifier does not know the answer in advance.

The verifier is free to specify the protocol, which can extend beyond a particular formal verification system such as Lean. In practice, the human verifier and the LLM agent indeed agree on a protocol Π\Pi and interact according to it. If the interaction meets the agreed conditions, the verifier accepts the claim. The protocol can cover rigorous mathematical proofs, formal verification code, and tasks that allow small completeness and soundness errors (such as GNI task). Bounding the number of rounds and the verification cost of Π\Pi by low-degree polynomials gives the setting of our benchmark. Our benchmark therefore evaluates this ability.

We'll describe this ability in more detail in the revised version.

Q2. why and how this benchmark could be used to produce reliable outcomes in testing agents given its relatively small size.

Intuitively, "proof" is more reliable than only "answer" because it not only provides the answer but also a way to verify it. 100100 counts the distinct test case types. Each test case specifies a data generator that can produce a variety of instances, so the pool of instances is considerably larger than this count suggests. We also ran GPT 5.6 Sol and GPT 5.6 Terra on the original 100 test cases in three independent runs each. The table reports the means and standard deviations.

TierGPT 5.6 Sol HighGPT 5.6 Terra High
Tier 1 (36)25.0 ± 1.019.7 ± 0.6
Tier 2 (40)21.0 ± 2.69.3 ± 0.6
Tier 3 (24)16.0 ± 2.012.3 ± 3.1
Total (100)62.0 ± 2.041.3 ± 2.9

The relatively small standard deviations indicate that our evaluation procedure yields reliable results. We'll add this table to the revised version. Thanks for your suggestion and hope you can keep the positive rating.

Thank you for the response. My comments have been addressed now.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Clarification of Definitions

We thank all three reviewers for their careful reading and constructive feedback. The reviews suggest that we and the reviewers have understood the four merits introduced in the paper somewhat differently. We therefore clarify what we mean by unified proof ability, agent-oriented evaluation, rigorous certification, and scalable synthesis, and why we believe our benchmark has all four merits.

  1. Unified proof ability. By unified proof ability, we mean the ability of an LLM agent to convince a human verifier, through a specified protocol and with high confidence, that a claim is true when the verifier does not know the answer in advance. In practice, the human verifier and the LLM agent agree on a protocol Π\Pi and interact according to it. If the interaction meets the agreed conditions, the verifier accepts the claim. The protocol can cover rigorous mathematical proofs, formal verification code, and tasks that allow small completeness and soundness errors (see our GNI task). Bounding the number of rounds and the verification cost of Π\Pi by low-degree polynomials gives the setting of our benchmark.
  2. Agent-oriented evaluation. We evaluate the complete agent system, including the LLM, its harness, and its tools. We therefore want the benchmark to have two properties. (1) A bare LLM, without a harness or tools, has difficulty solving the benchmark problems. (2) When LLM agents can write code and search the web, the benchmark still distinguishes their performance. Case Study 1 (Section 5.1) was designed to demonstrate these properties.
  3. Rigorous certification. We want the procedure for checking a claim to have explicit completeness and soundness bounds. We want to be able to check claims rigorously even on sizable instances of problems that lack a low-degree polynomial-time decision procedure. Methods such as using an LLM as a judge offer no theoretical guarantee of correctness. Appendix E gives proofs of the completeness and soundness bounds for every protocol in our benchmark.
  4. Scalable Synthesis. We mean that we can quickly generate test cases at scale that meet the three conditions above. Scale refers both to the number of test cases and to the size of the individual instances. One important difference from prior work is that, by building Merlin-Bench around interactive proofs (IP), we can evaluate an LLM agent even when the benchmark designer does not know whether a claim is true (see Appendix C.2). This makes it straightforward to include a wide range of decision problems in IP\textsf{IP}. Although our test cases span only 15 families, our dataset construction agent can identify a variety of special properties within these families and use them to generate diverse test cases.

We encourage all reviewers to read Appendix C.2. It presents the theoretical rationale for an IP-based evaluation paradigm. Moving to the complexity class IP allows the benchmark designer to evaluate an agent without knowing in advance whether the claim is true. We placed this procedure in the appendix and linked to it from the introduction because, for problems with known answers, the experimental procedure in the main text requires only one evaluation instead of two.

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.