- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DatalogBench, a benchmark of 136 natural-language-to-Soufflé Datalog synthesis tasks, and evaluates six model configurations and two coding agents. Generated programs are evaluated by execution on held-out inputs, with compilation failures distinguished from output mismatches. The agent settings substantially improve task pass rates, but semantic errors remain, and tasks whose reference programs contain both recursion and auxiliary predicates are considerably harder.
Critical strengths
- The problem is well motivated, particularly for program analysis and relational queries requiring recursive inference.
- The evaluation separates demonstration instances from held-out tests and reports compilation success, execution accuracy, and declaration-only repair results separately, providing useful diagnostic information.
- The agents achieve substantial improvements over their corresponding direct-prompting configurations, supported by paired statistical analyses. Candidates that pass the demonstration instance but fail held-out tests also illustrate the importance of independent evaluation.
- The appendices provide detailed dataset, implementation, and experimental information, and acknowledge limitations concerning dialect coverage, test adequacy, and attribution of agent gains.
Critical weaknesses
-
Test adequacy is validated within the existing mutation family. Test selection and the 96.4% mutation score rely on the same three operators, leaving coverage of other errors, including negation, aggregation, and grouping errors, unclear. With 58 tasks having only one evaluation instance, additional independent tests are needed to assess the robustness of EX.
-
Representativeness of practical Datalog workloads remains unclear. Despite useful structural and scale statistics, tasks come from four public artifacts, target only Soufflé, exclude complex types and some language components, and mostly involve small programs and inputs. The selection process, source and task-family distributions, and implications of excluded workloads need further clarification.
-
Interactive requirements elicitation is outside the evaluated scope. The agents receive compilation and execution feedback but cannot clarify requirements with users or handle requirement revisions. The results therefore primarily concern synthesis from fixed specifications. User interaction would be a valuable extension; otherwise, this limitation on practical applicability should be explicit.
-
Independent validation of specification sufficiency is not reported. The authors conducted an internal audit, but do not report an independent review without access to reference implementations. It remains unclear whether model-visible information sufficiently determines the intended behavior, particularly regarding implicit domain knowledge and boundary conventions. This is an unresolved risk, not a demonstrated annotation error.
-
The stability of experimental results across runs remains unclear. Each direct-generation configuration and agent was run only once. Task-level bootstrapping and paired tests do not directly capture variability across repeated runs. Multiple independent runs would help assess the stability of the reported results and main comparisons.
Questions for authors
-
Could the authors add new test cases designed independently of the three mutation operators, especially for the 58 tasks with only one evaluation instance? Do programs that pass the current tests still pass these additional tests, and do the main results change?
-
How many tasks were considered and retained from each source, and why were others excluded? How were similar tasks handled? Which types of practical Datalog tasks are not covered by this benchmark?
-
Could the authors include a small set of tasks where the model can ask users to clarify unclear requirements or revise the program when requirements change? If this is outside the paper’s scope, please explain how this limits its relevance to interactive use.
-
Could someone who has not seen the reference program understand exactly what a task asks the program to do, using only the information given to the model? Could independent reviewers check a sample of tasks from different domains and difficulty levels, and report any unclear wording or missing assumptions?
-
Could the authors run the experiments multiple times under the same settings and report the results from each run, along with the mean and standard deviation? Do the main findings remain consistent across runs?
6 comments
Q3. Could the authors include a small set of tasks where the model can ask users to clarify unclear requirements or revise the program when requirements change? If this is outside the paper’s scope, please explain how this limits its relevance to interactive use.
We did not add interactive tasks. Our evaluation fixes the specification so that every system is graded against the same intended output. An interactive setting needs a simulated user and a different definition of correctness, which is a separate benchmark design. For interactive use, this means three things are outside what our results show:
- Incomplete requests. Each task description is written to state the full requirement. Interactive use often starts from an incomplete request, and our results do not show whether a system notices the gap and asks.
- Revision. Every task is solved from scratch. No task asks a system to change an existing program after a requirement changes.
- Feedback from a user. Our agents get feedback by compiling and running their programs on a demonstration instance. They get no feedback from a user, such as a remark that an output tuple is wrong.
The results cover the step interactive use still needs once the requirement is settled: writing a correct program for it. EX on our benchmark is the success rate of this one step, and an interactive session involves the other three as well.
Q4. Could someone who has not seen the reference program understand exactly what a task asks the program to do, using only the information given to the model? Could independent reviewers check a sample of tasks from different domains and difficulty levels, and report any unclear wording or missing assumptions?
We completed an independent specification review and targeted follow-up checks. Evaluation uses a frozen benchmark version, shared reference outputs, and a common execution-based grading protocol.
Three non-author reviewers, none of whom had seen the reference programs or expected outputs, independently assessed 24 unmodified task specifications, stratified across four domains and three levels of program complexity. Six additional items with a key sentence removed served as hidden controls. In the initial review, they received the question, relation signatures and descriptions, and any existing knowledge note; no reference programs or input–output examples were provided. They recorded their interpretation and any assumptions needed to determine the output.
In the initial review, all three reviewers judged 11 of the 24 specifications self-contained. Two further specifications received at least one domain-knowledge-dependence judgment and no underdetermination judgment; the remaining 11 were judged underdetermined by at least one reviewer. Across the three reviewers, 6/6, 6/6, and 3/6 deletion controls were labeled underdetermined. We report these judgments as a quality check.
The same reviewers then completed a targeted follow-up on 13 items using paired interpretations and small input examples. We retained the original independent responses separately from the follow-up responses; the follow-up did not replace the initial labels.
Q1. Could the authors add new test cases designed independently of the three mutation operators, especially for the 58 tasks with only one evaluation instance? Do programs that pass the current tests still pass these additional tests, and do the main results change?
Yes. Across all three runs of each configuration, we tested every accepted program on an additional pool for the 58 tasks that had one evaluation variant when the scope was fixed, before input generation.
- New tests. We generated 493 inputs using generic (373) and structured (120) generators; the latter covers 12 tasks with structured inputs. These inputs satisfy constraints encoded from the task statements and are sampled without mutant or candidate scores. Six hand-specified comparison-boundary inputs for three tasks brought the total to 499.
- Do passing programs still pass? Of the 3434 generated programs that passed the evaluation variants on those 58 tasks, 3318 pass the additional inputs. We inspected all 116 rejected programs: 6 are wrong programs missed by the evaluation variants; 65 fail on inputs for which the task statement leaves the answer undefined, and 45 concern two readings of the wording.
- Do the main results change? The six wrong programs all come from DeepSeek V4 Flash without thinking. Counting them as failures lowers a configuration’s mean EX by at most 1.0 percentage point and leaves every other model unchanged. Counting all 116 rejections as failures gives a maximum reduction of 2.5 points. The gains from agents and an input–output example, and the recursion and auxiliary-predicate gaps, retain their direction. The small descriptions-versus-signatures difference still has an interval including zero.
W1. Test adequacy is validated within the existing mutation family. Test selection and the 96.4% mutation score rely on the same three operators, leaving coverage of other errors, including negation, aggregation, and grouping errors, unclear. With 58 tasks having only one evaluation instance, additional independent tests are needed to assess the robustness of EX.
We tested the frozen variants against five additional mutation operators targeting errors beyond the original three:
| Operator | Distinguished/compilable | Rate |
|---|---|---|
| Negation removal | 69/72 | 95.8% |
| Aggregate-function replacement | 25/26 | 96.2% |
| Grouping-variable change | 21/22 | 95.5% |
| Constant change | 501/556 | 90.1% |
| Comparison-bound change | 77/124 | 62.1% |
| Total | 693/800 | 86.6% |
For comparison, the original operators distinguish 1440/1521 compilable mutants (94.7%) on the frozen variants. Surviving mutants have not been individually checked for equivalence, so the reported discrimination rates are lower bounds.
We thank the reviewer for the careful and constructive review. This comment and the next answer Q2, Q3, and Q4 (with W3 and W4). Our answers to Q1, Q5, W1, and W2 report new experiments and will follow in later comments with the revised manuscript.
Q2. How many tasks were considered and retained from each source, and why were others excluded? How were similar tasks handled? Which types of practical Datalog tasks are not covered by this benchmark?
We traced every task back to its source and counted the candidates at each stage.
Counts per source. For each source, the table gives the candidates we considered, those we kept after sanitization, those we kept before the specification audit, and the final tasks.
Why others were excluded. Exclusions included unsupported language features, engine tests, cross-source copies, simple SQL-style selection or join tasks, and scope-based selection. We also consolidated variant families and removed tasks whose intended outputs could not be stated unambiguously; the revised appendix gives the details.
Similar tasks. We kept a task that appears in several sources only once, and we merged families of near-identical variants into one task. We kept different algorithms for the same problem as separate tasks in selected cases. Where one upstream program defines several outputs, we split it into several tasks.
Not covered. The benchmark does not cover whole-program analyses at the scale of Doop (hundreds of rules, millions of facts), programs that need complex types, components, or user-defined functors, or dialects other than Soufflé.