- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLMs can automatically translate natural-language domain knowledge into propositional constraints that can be used by Neuro-Symbolic (NeSy) predictors. The authors introduce auto-nesy-bench, which evaluates the full pipeline from constraint formalization to downstream NeSy prediction. They compare different LLMs, prompting strategies, output formats, and detailed vs. non-detailed task specifications, and then integrate the generated constraints into Semantic Probabilistic Layers (SPL) and Semantic Loss. The results show that strong models can generate constraints close to expert-provided ones under suitable configurations, while different types of formalization errors have substantially different downstream effects, with overly strict constraints generally being more harmful than permissive ones.
Critical strengths
-
The end-to-end evaluation provides meaningful practical validation. The paper goes beyond standalone natural-language-to-constraint evaluation by actually integrating LLM-generated constraints into SPL and Semantic Loss. This allows the authors to study whether and how formalization errors affect downstream NeSy performance. The finding that strict, permissive, and incomparable errors have different downstream consequences shows that formula-level similarity alone is insufficient to evaluate practical utility, and I view this as the most important contribution beyond existing constraint-generation benchmarks.
-
The benchmark and evaluation methodology are broad and carefully designed. The paper systematically compares multiple LLMs, prompting strategies, five output formats, and different levels of specification detail. It also evaluates formulas semantically through model counting and introduces metrics such as REL, MC-R, and consistency to characterize different types of errors. Overall, the experimental coverage and reproducibility are strong.
Critical weaknesses
-
The remaining headroom of auto-nesy-bench on current frontier models is unclear. The paper already shows that, under suitable prompting and output configurations, the strongest evaluated open-weight models achieve nearly saturated performance on auto-nesy-bench (e.g., F1 scores around 0.99). For a benchmark paper, it is therefore important to understand whether stronger current frontier models can already solve most of these tasks almost perfectly. I understand that a full evaluation of expensive frontier models may not be practical, but even a small-scale sanity check on a representative subset would help establish the benchmark’s remaining difficulty, discriminative power, and longer-term value.
-
The scientific role of RQ2, and in particular which conclusions are LLM-specific, is not fully disentangled. RQ2 is useful for validating the practical utility of auto-formalization, but once a generated constraint is fixed, its downstream effect depends on the constraint itself rather than on whether it was produced by GPT-OSS, Qwen, a human expert, or another process. Thus, findings about strict, permissive, and incomparable constraints are closer to general observations about imperfect constraints in NeSy predictors than direct insights into LLM capability. The paper could more clearly position RQ1 as an LLM capability evaluation and RQ2 as downstream utility validation. If the goal is to characterize different LLMs, it would be useful to analyze whether models systematically differ in the types of errors they produce and the corresponding downstream risks.
Questions for authors
See Weakness
4 comments
As we stated in our latest response to reviewer S2jn, we report here the findings from the evaluation of GPT-5.5 on auto-nesy-bench, as promised.
The results are as follows:
| Model | F1 (↑) | Precision (↑) | Recall (↑) | Syntax ER (↓) |
|---|---|---|---|---|
| GPT-5.5 | 0.927 ± 0.049 | 0.926 ± 0.050 | 0.962 ± 0.027 | 0.0% ± 0.0% |
Compared with GPT-6 Astra, GPT-5.5 achieves slightly lower overall performance (0.927 vs 0.930 f1 score). However, it produces no syntax errors (0%), making it the most reliable LLM evaluated so far in terms of syntactic correctness.
To complement the quantitative results, we examined some of the generated formulas that are not equivalent to the ground truth:
- GPT-6-astra on chx (strict, MC-R = 0.94, detailed): In chx, the goal is to assign exactly one severity code per chest X-ray, i.e., healthy, green, yellow, orange, or red. GPT formula correctly encodes all severity codes except for healthy.
- GPT-6-astra on cebab (permissive, MC-R = 1024, detailed): The task consists in mapping five aspect scores (food, service, ambiance, noise, price) from a restaurant review to an overall sentiment label (one of five classes). GPT formula enforces the one-hot constraint on the output label only, omitting the one-hot constraints on the five aspect scores.
- GPT-5.5 on mn-mul-bin (incomparable, MC-R = 1.0, detailed): GPT formula has exactly 100 valid assignments (the same count as the ground truth) yet four assignments yield incorrect products. Specifically, the generated formula encodes 3 × 7 = 19 (should be 21), 5 × 8 = 41 (should be 40), 7 × 6 = 41 (should be 42) , and 7 × 7 = 63 (should be 49).
We will report these findings in the manuscript as well.
Dear Reviewer,
Thank you again for your review.
With the discussion period coming to an end in roughly 48 hours, we would be grateful to know whether our response resolves your concerns. If any points remain unclear, we would be happy to address them.
Thank you for your time and your constructive feedback.
Best regards,
Authors
Thank you for your responses and the additional experiments. My main concerns have been addressed, and I will maintain my recommendation.
We thank the reviewer for the constructive feedback, for recognizing the value of our findings on strict, permissive, and incomparable formulas, for appreciating the careful construction of the benchmark and pipeline, and for acknowledging our experimental effort.
We address the weaknesses and questions below.
First, our goal is to evaluate LLMs of different sizes and capabilities, helping practitioners on a budget understand how much they can obtain from small and medium-sized models.
Second, we would like to clarify the current level of saturation. While individual configurations reach very high F1 on some tasks, the best and largest (117B) open-weight model achieves an average F1 of 0.786 on auto-nesy-bench across different configurations (Table 1), showing that auto-formalization per se is not fully solved on these tasks yet. Third, following the reviewer’s suggestion, we are evaluating frontier models. Preliminary results for GPT-6 Astra with detailed descriptions (see our response to Reviewer S2jn) show an average F1 of 0.930, beating the best open weight model. We will add results concerning non-detailed descriptions as soon as we have them. We want to remark that our plan in the future is to extend the benchmark with novel, unseen tasks at multiple levels of description detail.
We agree. RQ1 (Section 5.1, Tables 1–3) evaluates formalization capability, isolating the effect of the LLM, prompt, and output format, while RQ2 (Section 5.2, Table 4, Section I.2.3) evaluates the downstream utility of the generated formulas. In the revision, we will state this distinction explicitly in the contributions and in Sections 5.1 and 5.2, and clarify that the RQ2 findings on strict, permissive, and incomparable formulas hold for imperfect constraints regardless of their source.
Models do differ systematically. Syntax error rates range from 3.9% (gemma4-31b) to 62.8% (mistral-nemo) (Section 5.1), with errors characterized by our taxonomy (Appendix C). Figure 13 gives a good overview of the syntactic errors made by the different LLMs. As for semantic errors, recall exceeds precision for all LLMs (Table 1), indicating that wrong formulas tend to be permissive. The downstream effect of permissive formulas is largely benign (Section 5.2, Table 4). Pipeline failures never occur for the best and largest gpt-oss-120b, but affect 6/18 and 7/18 tasks for olmo3-32b-think and qwen3-8b, respectively (Section 5.2, Table 4; details in Table 24).
Qualitatively, as we answered reviewer S2jn, we have observed the following mistakes:
qwen3-8b on kand-logic-2 (strict, MC-R ≈ 0.015, detailed). The formula accepts only pairs where objects share both shape and color, ignoring the other valid patterns. E.g., two images with a red and a blue square are valid for the target (same shape pattern) but rejected by the generated formula.
olmo3-32b-think on warcraft (permissive, MC-R = 1.75, detailed). The formula encodes the local path constraints (source, target, connected edges) but misses the explicitly stated no-cycle condition, so paths containing a cycle are wrongly accepted.
olmo3-32b-think on mn-add-bin (incomparable, MC-R = 2.56, non detailed). The formula correctly models most binary digit additions but oversimplifies the carry rule, triggering only when both digits are ≥ 8. E.g., 7 + 9 = 16:
is not a valid assignment for the predicted formula.
This insight is what is measured by our REL and MC-R metrics. We will report these findings in the manuscript.