acceptodds
Under review as a conference paper at ICLR 2027

DatalogBench: Evaluating Large Language Models on Text-to-Datalog Synthesis

Abstract

Datalog underpins reasoning tasks such as program analysis, but its programs are hard to write. Existing synthesizers automate this task but require users to state their intent as input-output examples. Large language models (LLMs) suggest a more natural route, text-to-Datalog synthesis from a natural-language question, yet how well they do so has not been systematically evaluated. We present DatalogBench, a benchmark of 136 text-to-Datalog synthesis tasks curated from existing Datalog-based artifacts. Synthesized programs are graded by execution on held-out inputs against an oracle validated by mutation analysis. Across six LLMs and four prompting configurations, exact match peaks at 68.4%, and relation descriptions or an input-output example have only modest, model-dependent effects. Under direct prompting, most failures occur at compile time, typically because a model invents auxiliary predicates that it never declares or types consistently. Two coding agents reach up to 83.8% and eliminate nearly all such failures, leaving mostly semantic errors concentrated in recursive tasks. DatalogBench thus identifies recursive reasoning and decomposition as open challenges for current LLMs and agents, and offers a reliable, execution-grounded measure of both.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neurosymbolic Reasoning, Program Synthesis, Code Generation, Datalog, Benchmarks

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_uV2jOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DatalogBench, a benchmark of 136 natural-language-to-Soufflé Datalog synthesis tasks, and evaluates six model configurations and two coding agents. Generated programs are evaluated by execution on held-out inputs, with compilation failures distinguished from output mismatches. The agent settings substantially improve task pass rates, but semantic errors remain, and tasks whose reference programs contain both recursion and auxiliary predicates are considerably harder.

Critical strengths
  • The problem is well motivated, particularly for program analysis and relational queries requiring recursive inference.
  • The evaluation separates demonstration instances from held-out tests and reports compilation success, execution accuracy, and declaration-only repair results separately, providing useful diagnostic information.
  • The agents achieve substantial improvements over their corresponding direct-prompting configurations, supported by paired statistical analyses. Candidates that pass the demonstration instance but fail held-out tests also illustrate the importance of independent evaluation.
  • The appendices provide detailed dataset, implementation, and experimental information, and acknowledge limitations concerning dialect coverage, test adequacy, and attribution of agent gains.
Critical weaknesses
  • Test adequacy is validated within the existing mutation family. Test selection and the 96.4% mutation score rely on the same three operators, leaving coverage of other errors, including negation, aggregation, and grouping errors, unclear. With 58 tasks having only one evaluation instance, additional independent tests are needed to assess the robustness of EX.

  • Representativeness of practical Datalog workloads remains unclear. Despite useful structural and scale statistics, tasks come from four public artifacts, target only Soufflé, exclude complex types and some language components, and mostly involve small programs and inputs. The selection process, source and task-family distributions, and implications of excluded workloads need further clarification.

  • Interactive requirements elicitation is outside the evaluated scope. The agents receive compilation and execution feedback but cannot clarify requirements with users or handle requirement revisions. The results therefore primarily concern synthesis from fixed specifications. User interaction would be a valuable extension; otherwise, this limitation on practical applicability should be explicit.

  • Independent validation of specification sufficiency is not reported. The authors conducted an internal audit, but do not report an independent review without access to reference implementations. It remains unclear whether model-visible information sufficiently determines the intended behavior, particularly regarding implicit domain knowledge and boundary conventions. This is an unresolved risk, not a demonstrated annotation error.

  • The stability of experimental results across runs remains unclear. Each direct-generation configuration and agent was run only once. Task-level bootstrapping and paired tests do not directly capture variability across repeated runs. Multiple independent runs would help assess the stability of the reported results and main comparisons.

Questions for authors
  • Could the authors add new test cases designed independently of the three mutation operators, especially for the 58 tasks with only one evaluation instance? Do programs that pass the current tests still pass these additional tests, and do the main results change?

  • How many tasks were considered and retained from each source, and why were others excluded? How were similar tasks handled? Which types of practical Datalog tasks are not covered by this benchmark?

  • Could the authors include a small set of tasks where the model can ask users to clarify unclear requirements or revise the program when requirements change? If this is outside the paper’s scope, please explain how this limits its relevance to interactive use.

  • Could someone who has not seen the reference program understand exactly what a task asks the program to do, using only the information given to the model? Could independent reviewers check a sample of tasks from different domains and difficulty levels, and report any unclear wording or missing assumptions?

  • Could the authors run the experiments multiple times under the same settings and report the results from each run, along with the mean and standard deviation? Do the main findings remain consistent across runs?

6 comments

We thank the reviewer for the careful and constructive review. This comment and the next answer Q2, Q3, and Q4 (with W3 and W4). Our answers to Q1, Q5, W1, and W2 report new experiments and will follow in later comments with the revised manuscript.

Q2. How many tasks were considered and retained from each source, and why were others excluded? How were similar tasks handled? Which types of practical Datalog tasks are not covered by this benchmark?

We traced every task back to its source and counted the candidates at each stage.

Counts per source. For each source, the table gives the candidates we considered, those we kept after sanitization, those we kept before the specification audit, and the final tasks.

SourceCandidatesSanitizedPre-auditFinal
Soufflé123857268
EGS86502525
GenSynth47171616
CodeFuse-Query31312727
Total287183140136

Why others were excluded. Exclusions included unsupported language features, engine tests, cross-source copies, simple SQL-style selection or join tasks, and scope-based selection. We also consolidated variant families and removed tasks whose intended outputs could not be stated unambiguously; the revised appendix gives the details.

Similar tasks. We kept a task that appears in several sources only once, and we merged families of near-identical variants into one task. We kept different algorithms for the same problem as separate tasks in selected cases. Where one upstream program defines several outputs, we split it into several tasks.

Not covered. The benchmark does not cover whole-program analyses at the scale of Doop (hundreds of rules, millions of facts), programs that need complex types, components, or user-defined functors, or dialects other than Soufflé.

Q3. Could the authors include a small set of tasks where the model can ask users to clarify unclear requirements or revise the program when requirements change? If this is outside the paper’s scope, please explain how this limits its relevance to interactive use.

We did not add interactive tasks. Our evaluation fixes the specification so that every system is graded against the same intended output. An interactive setting needs a simulated user and a different definition of correctness, which is a separate benchmark design. For interactive use, this means three things are outside what our results show:

  • Incomplete requests. Each task description is written to state the full requirement. Interactive use often starts from an incomplete request, and our results do not show whether a system notices the gap and asks.
  • Revision. Every task is solved from scratch. No task asks a system to change an existing program after a requirement changes.
  • Feedback from a user. Our agents get feedback by compiling and running their programs on a demonstration instance. They get no feedback from a user, such as a remark that an output tuple is wrong.

The results cover the step interactive use still needs once the requirement is settled: writing a correct program for it. EX on our benchmark is the success rate of this one step, and an interactive session involves the other three as well.

Q4. Could someone who has not seen the reference program understand exactly what a task asks the program to do, using only the information given to the model? Could independent reviewers check a sample of tasks from different domains and difficulty levels, and report any unclear wording or missing assumptions?

We completed an independent specification review and targeted follow-up checks. Evaluation uses a frozen benchmark version, shared reference outputs, and a common execution-based grading protocol.

Three non-author reviewers, none of whom had seen the reference programs or expected outputs, independently assessed 24 unmodified task specifications, stratified across four domains and three levels of program complexity. Six additional items with a key sentence removed served as hidden controls. In the initial review, they received the question, relation signatures and descriptions, and any existing knowledge note; no reference programs or input–output examples were provided. They recorded their interpretation and any assumptions needed to determine the output.

In the initial review, all three reviewers judged 11 of the 24 specifications self-contained. Two further specifications received at least one domain-knowledge-dependence judgment and no underdetermination judgment; the remaining 11 were judged underdetermined by at least one reviewer. Across the three reviewers, 6/6, 6/6, and 3/6 deletion controls were labeled underdetermined. We report these judgments as a quality check.

The same reviewers then completed a targeted follow-up on 13 items using paired interpretations and small input examples. We retained the original independent responses separately from the follow-up responses; the follow-up did not replace the initial labels.

Q1. Could the authors add new test cases designed independently of the three mutation operators, especially for the 58 tasks with only one evaluation instance? Do programs that pass the current tests still pass these additional tests, and do the main results change?

Yes. Across all three runs of each configuration, we tested every accepted program on an additional pool for the 58 tasks that had one evaluation variant when the scope was fixed, before input generation.

  • New tests. We generated 493 inputs using generic (373) and structured (120) generators; the latter covers 12 tasks with structured inputs. These inputs satisfy constraints encoded from the task statements and are sampled without mutant or candidate scores. Six hand-specified comparison-boundary inputs for three tasks brought the total to 499.
  • Do passing programs still pass? Of the 3434 generated programs that passed the evaluation variants on those 58 tasks, 3318 pass the additional inputs. We inspected all 116 rejected programs: 6 are wrong programs missed by the evaluation variants; 65 fail on inputs for which the task statement leaves the answer undefined, and 45 concern two readings of the wording.
  • Do the main results change? The six wrong programs all come from DeepSeek V4 Flash without thinking. Counting them as failures lowers a configuration’s mean EX by at most 1.0 percentage point and leaves every other model unchanged. Counting all 116 rejections as failures gives a maximum reduction of 2.5 points. The gains from agents and an input–output example, and the recursion and auxiliary-predicate gaps, retain their direction. The small descriptions-versus-signatures difference still has an interval including zero.

W1. Test adequacy is validated within the existing mutation family. Test selection and the 96.4% mutation score rely on the same three operators, leaving coverage of other errors, including negation, aggregation, and grouping errors, unclear. With 58 tasks having only one evaluation instance, additional independent tests are needed to assess the robustness of EX.

We tested the frozen variants against five additional mutation operators targeting errors beyond the original three:

OperatorDistinguished/compilableRate
Negation removal69/7295.8%
Aggregate-function replacement25/2696.2%
Grouping-variable change21/2295.5%
Constant change501/55690.1%
Comparison-bound change77/12462.1%
Total693/80086.6%

For comparison, the original operators distinguish 1440/1521 compilable mutants (94.7%) on the frozen variants. Surviving mutants have not been individually checked for equivalence, so the reported discrimination rates are lower bounds.

Reviewer_VKa3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Datalog is widely used in reasoning tasks such as program analysis due to its expressive recursion and predictable semantics. However, existing benchmarks do not systematically evaluate LLM ability to synthesize semantically correct Datalog programs from natural language queries, particularly those requiring recursive reasoning and predicate invention. Addressing this gap, the authors introduce DatalogBench, an execution-grounded benchmark consisting of 136 text-to-Datalog synthesis tasks curated from existing Datalog-based artifacts across 4 reasoning domains: program analysis, graph analytics, formal reasoning, and knowledge discovery. The authors evaluate six LLMs and two coding agents using held-out evaluation inputs. The results show that direct prompting achieves up to 68.4% exact match, while coding agents reach 83.8%, with recursive reasoning and predicate invention remaining key challenges.

Critical strengths
  1. The paper addresses a clear research gap. While prior work on Datalog synthesis has primarily relied on input-output examples as specifications, this paper introduces a benchmark for systematically evaluating text-to-Datalog synthesis. The problem formulation is well-defined, and the benchmark covers program analysis, graph analytics, formal reasoning, and knowledge discovery.
Critical weaknesses
  1. The evaluation approach offers limited novelty. Despite focusing on Datalog-specific challenges, the benchmark relies on conventional execution-based evaluation (compilation, test execution, and output comparison). It is unclear how this evaluation approach meaningfully differs from those used in existing Datalog (or code) generation benchmarks.

  2. The validity and reliability of the evaluation remain uncertain. The lower performance on tasks involving recursion and auxiliary predicates may reflect overall program complexity rather than these specific challenges. Moreover, the benchmark uses only 247 evaluation variants across 136 tasks, with 58 tasks having just one variant. Although mutation analysis is conducted, it covers only three mutation operators, leaving uncertainty about the evaluation's ability to detect diverse semantic errors.

Questions for authors
  1. The authors argue that existing benchmarks do not adequately evaluate recursive reasoning and predicate invention. However, the paper does not directly compare DatalogBench with existing benchmarks. What new evaluation insights does DatalogBench provide?

  2. The authors categorize task difficulty based on whether the reference programs use recursion or auxiliary predicates. However, the use of these structures in a reference program does not necessarily mean that all correct implementations require them. How accurately does this classification reflect the intrinsic difficulty of the tasks?

  3. The evaluation settings differ across models. GPT, Claude, and DeepSeek are given an output token budget of 32768, while Gemini is limited to 4096. This raises concerns about the fairness of model comparisons, especially since some Gemini responses were truncated before completing the generated programs. What motivated this difference?

  4. The benchmark only evaluates Soufflé. Could the authors also evaluate other Datalog engines, such as Datomic or LogicBlox, to see whether the findings hold more broadly?

  5. Unlike text-to-SQL, which has a clear use case in natural language database querying, the practical demand for text-to-Datalog synthesis is less evident. Although it could help automate rule development in domains such as program analysis, the paper provides limited evidence of actual user demand. Could the authors further clarify the practical importance and relevance of text-to-Datalog synthesis?

5 comments

We thank the reviewer for the detailed questions. These comments are answers to W1, Q1, Q4, and Q5. Our answers to W2, Q2, and Q3 report new experiments and will follow in later comments, with the revised manuscript.

Q1. The authors argue that existing benchmarks do not adequately evaluate recursive reasoning and predicate invention. However, the paper does not directly compare DatalogBench with existing benchmarks. What new evaluation insights does DatalogBench provide?

Comparison with existing benchmarks. We compare the specification interfaces, structural coverage, and evaluation criteria of related benchmarks and synthesis settings. The main distinction is what is specified and evaluated:

  • Text-to-SQL benchmarks: the published construct inventories of Spider and BIRD do not list recursive queries, while Spider 2.0 describes CTE-based workflows without a separate recursion-specific evaluation. SQL supports intermediate results through subqueries or named CTEs. These evaluations do not directly report the reference-based recursion and auxiliary-predicate slices used in DatalogBench.
  • Symbolic Datalog synthesis settings: tasks are specified through examples, output constraints, or an oracle, often with templates or candidate rules that constrain the search. Some settings support recursion and intermediate-predicate invention. They do not evaluate deriving these structures from a natural-language question.

New insights. Because tasks are labeled by structure and compile failures are reported separately from semantic errors, DatalogBench shows the following:

  1. Under direct prompting, most failures happen before execution. Models introduce auxiliary predicates but do not declare or type them consistently, and supplying the missing declarations fixes few of these programs.
  2. Coding agents remove nearly all compile failures, but their remaining errors are semantic and lie mainly in recursive tasks. Execution feedback narrows the recursion gap without closing it.
  3. For both LLMs and agents, tasks that combine recursion with auxiliary predicates remain much harder than tasks with neither.

W1. The evaluation approach offers limited novelty. Despite focusing on Datalog-specific challenges, the benchmark relies on conventional execution-based evaluation (compilation, test execution, and output comparison). It is unclear how this evaluation approach meaningfully differs from those used in existing Datalog (or code) generation benchmarks.

The answer to Q1 above compares what the benchmarks measure. In how a program is graded, our evaluation differs from both kinds of benchmarks as follows.

Symbolic Datalog synthesis. The compared settings use examples, output constraints, or an oracle to guide synthesis; some also report held-out generalization. In our direct LLM and agent evaluations, candidates are graded on input instances withheld throughout generation and feedback.

Function-level code generation. HumanEval and EvalPlus also grade programs by executing tests. We compare complete output relations across held-out input instances and report compilation pass separately from exact output agreement, with separate analyses of compilation failures and compiled-but-wrong programs.

Q4. The benchmark only evaluates Soufflé. Could the authors also evaluate other Datalog engines, such as Datomic or LogicBlox, to see whether the findings hold more broadly?

We have not evaluated a second engine, and the paper states the single-dialect scope as its first limitation. Datalog has no standard syntax, and engines differ in typing, aggregation, and negation, so adding an engine means translating all 136 references and re-validating every oracle. Of the two engines named, LogicBlox is a commercial system, and Datomic queries are data structures over entity–attribute–value facts, not programs over a relational schema. We chose Soufflé for three reasons:

  1. Prior work on Datalog synthesis runs on it. ProSynth, GenSynth, and Mobius evaluate their candidate programs with Soufflé, and LogicLoc executes LLM-written Datalog with it.
  2. Our sources target it. Soufflé's own programs supply 68 of the 136 tasks, and the GenSynth tasks were run on Soufflé by their authors.
  3. It is open source, so anyone can reproduce the grading, and it powers analysis tools in several areas: Doop (points-to analysis for Java), DDisasm (binary disassembly), and Gigahorse and Securify (smart-contract analysis).

Checking how the findings transfer to another dialect will be promising in future work.

Q5. Unlike text-to-SQL, which has a clear use case in natural language database querying, the practical demand for text-to-Datalog synthesis is less evident. Although it could help automate rule development in domains such as program analysis, the paper provides limited evidence of actual user demand. Could the authors further clarify the practical importance and relevance of text-to-Datalog synthesis?

The demand shows in two ways: Datalog is used well beyond program analysis, and several recent systems already have LLMs write Datalog-family queries. Datalog is used in network verification, database systems, distributed systems, and large-scale graph analytics. Demand for automatic Datalog generation is already visible: LLMs are being used to write Datalog and Datalog-based analysis queries in systems built because such queries are hard to write by hand. LogicLoc has an LLM that writes Soufflé Datalog programs from natural-language questions to localize code. QLCoder and QLPro have LLMs write CodeQL queries for vulnerability detection. CodeQL's query language compiles to Datalog, and QLCoder motivates its work by noting that writing such queries is challenging and requires diverse expertise. These rule languages are used extensively in industry. CodeFuse-Query, an industrial Datalog-based analysis system, reports processing up to 10 billion lines of code daily across more than 300,000 analysis tasks. Before LLMs, the same need motivated a line of work on synthesizing Datalog from input–output examples (ALPS, ProSynth, EGS, GenSynth). The revised introduction states this motivation with the CodeFuse-Query and LogicLoc examples.

Q2. The authors categorize task difficulty based on whether the reference programs use recursion or auxiliary predicates. However, the use of these structures in a reference program does not necessarily mean that all correct implementations require them. How accurately does this classification reflect the intrinsic difficulty of the tasks?

We checked whether the reference-based labels correspond to the structures used by successful solutions and to differences in observed difficulty.

Across 6277 programs accepted by the held-out evaluation, 98.3% of programs on recursive-labeled tasks are recursive, compared with 5.7% on the other tasks. For auxiliary-predicate labels, the corresponding proportions are 63.5% and 5.4%. On 25 of the 59 auxiliary-labeled tasks, at least one accepted program uses no auxiliary predicate.

The performance differences also remain after adjustment for program complexity. Across the Direct configurations, the adjusted EX gaps are −27.8 points for recursion (95% CI [−38.9, −17.1]) and −18.1 for auxiliary predicates [−29.9, −5.4].

These results support using the labels to characterize structural task groups and their observed difficulty. We describe them as properties of reference programs, without assuming that every correct implementation must use the same structures.

Q3. The evaluation settings differ across models. GPT, Claude, and DeepSeek are given an output token budget of 32768, while Gemini is limited to 4096. This raises concerns about the fairness of model comparisons, especially since some Gemini responses were truncated before completing the generated programs. What motivated this difference?

We initially used a 4096-token output cap. We raised it to 32768 for GPT, Claude, and DeepSeek after early requests exhausted the cap without producing answer text. Gemini returned non-empty text, so we did not raise its cap, although 38 of its 544 responses (7.0%) were incomplete.

In the revised evaluation, we requested all six models with the same 32768-token output cap. We evaluated all four Gemini configurations in three runs through the official API using Gemini 3.8 Flash, as Gemini 3.7 Flash was no longer available at this time.

All 1632 Gemini candidates in the reported main experiment were scored without filtering on format flags. Fifteen responses (0.9%) ended with MAX_TOKENS; they contained answer text, and all failed compilation under normal grading. EX (%), reported as mean ± sample SD over three runs, is 65.7 ± 1.1 with signatures and no example, 67.2 ± 2.4 with descriptions and no example, 72.8 ± 1.9 with signatures and one example, and 72.3 ± 1.9 with descriptions and one example.

Reviewer_DqEMOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes 136 text to datalog tasks spanning 4 domains. The authors evaluate 6 language models and 2 coding agents. The paper shows that compilation remains a challenge for one shot performance of language models and in an agentic setting, semantics become a challenge, specifically reasoning about recursive reasoning and decomposition.

Critical strengths

The paper is well presented. It provides a text-to-datalog benchmark using shuffle, that measures the ability of language models to reason about datalog. It further measures the effect of providing descriptions and i/o examples in the prompt to the LLM/Agent.

Critical weaknesses

None

Questions for authors

Clarification

  1. Why do you use a bound of 4 steps for the agents? would additional steps help?
  2. Would providing additional examples improve the performance of LLMs / agents for semantic correctness?

Suggestions fro future work

  1. Could we see the performance of a few open-weight / open-source coding models ?
2 comments

We thank the reviewer for the positive assessment and the concrete questions.

Q1. Why do you use a bound of 4 steps for the agents? Would additional steps help?

Why four. We use a fixed four-turn budget—one initial generation followed by up to three feedback-guided revisions—to evaluate execution feedback under a limited interaction budget. We also tested an eight-turn budget to check sensitivity to this choice.

Would more turns help? We ran three trials on all 136 tasks with up to eight turns. Four-turn results use the first four turns of the same trajectories, with the same candidate-selection rule based on demonstration-instance feedback. The turn budget is not stated in the prompts, and held-out evaluation results are not used for selection.

AgentFour-turn mean EXEight-turn mean EXChange
Codex83.1%84.6%+1.5 points
Claude Code86.8%86.8%0.0 points

Sessions stop once a candidate exactly matches the demonstration instance. On average, 7.0 Codex and 3.7 Claude Code sessions per run continued beyond four turns. Under this protocol, extending the budget to eight turns gives Codex a modest additional gain and leaves Claude Code’s EX unchanged.

Q2. Would providing additional examples improve the performance of LLMs / agents for semantic correctness?

We directly compared one versus two input–output examples on 40 tasks sampled across domains and program-complexity tiers. Both conditions use relation signatures and the same tasks, run indices, and held-out evaluation variants. The added examples are distinct from the demonstration and evaluation inputs, and their outputs are computed by the reference programs.

ModelOne-example EXTwo-example EXChange
GPT67.5%66.7%−0.8 points
Claude76.7%78.3%+1.7 points
Gemini75.8%78.3%+2.5 points
DeepSeek Pro55.0%60.8%+5.8 points
DeepSeek Flash, thinking56.7%57.5%+0.8 points
DeepSeek Flash, non-thinking49.2%49.2%0.0 points
Mean of six models63.5%65.1%+1.7 points

Results are means over the reported runs; changes are computed before rounding. We average paired differences over runs within each task and then equally over models. The average change is +1.7 percentage points, with a task-bootstrap 95% interval of [−1.5, 4.9]. Thus, this experiment does not establish an average improvement from adding a second example; each model’s interval also includes zero.

Could we see the performance of a few open-weight / open-source coding models?

We evaluated three additional open-weight models on all 136 tasks using relation signatures and no input–output example, with the same held-out evaluation variants and grading procedure:

ModelTasks passedEX
GLM-5.380/13658.8%
MiniMax-M3.1-Flash73/13653.7%
Kimi K2.7-Code70/13651.5%

These are single-evaluation results for each model. In this zero-shot signature setting, the three open-weight models achieve 51.5–58.8% EX, compared with GPT-5.6 Sol’s three-run mean of 53.2%. We left reasoning effort unspecified in our requests. Official defaults differ: GLM-5.3 and MiniMax-M3.1-Flash default to max, whereas GPT-5.6 defaults to medium; Kimi K2.7 Code uses mandatory thinking.

Thank you for the new results. I would like to maintain my score.

Comments on the paper

1 comment

We thank the reviewers and the AC for their feedback. We have uploaded a revised PDF. Key changes made in response to the reviews are highlighted in blue. The main responses and revisions are:

  • Clarified the four-turn protocol (Section 5.1), compared four/eight turns and one/two examples (Appendices N, C.4), and evaluated three additional open-weight models. Eight turns add 1.5 points for Codex and 0.0 for Claude Code; on the 40 sampled tasks, the second example gives no clear average gain. (DqEM)
  • Clarified practical motivation (Section 1) and distinctions from related benchmarks (Section 2), with a comparison table (Appendix Q). (VKa3)
  • Documented source selection, task-family coverage, and limits on workload scale, dialects, and interactive use (Appendices D, E, O). (uV2j, VKa3)
  • Added independent specification review and targeted follow-up checks (Appendix C.5). (uV2j)
  • Extended test-adequacy checks with 499 additional inputs on 58 tasks (Appendix G.4) and five additional mutation operators (Section 4.3; Appendix G.3); clarified survivor accounting (Appendix G.2). (uV2j, VKa3)
  • Clarified reference-based structural labels (Section 3, 4.2) and checked accepted programs (Appendix K); Direct performance gaps remain after complexity adjustment (Section 6.3). (VKa3)
  • Reported per-run EX in the responses, updated repeated-run results and their interpretation (Section 6.1–6.2), clarified run-level means and sample SD (Section 5.1; Appendix C.3), and aligned requested output caps at 32,768 tokens (Appendix C.2). (uV2j, VKa3)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.