acceptodds
Withdrawn from ICLR 2027

Generation Is Not Selection: Neuro-Symbolic Natural Language Formalization of Temporal Logic

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Natural-language-to-temporal-logic translation is challenging because a requirement can admit multiple plausible formalizations. We study whether neuro-symbolic systems can select the intended formalization when a correct candidate is already present. We present Pendulum, a pipeline separating neural candidate synthesis from symbolic analysis and final selection. Symbolic analysis provides solver-backed equivalence and behavioral evidence, which a downstream judge uses before deterministic finalization. We evaluate Pendulum on , a 306-instance future-time LTL dataset derived from the Little Tricky Logic benchmark. An audit of all 306 references identifies 55 confirmed reference errors and 48 ambiguous specifications, leaving 258 determinate cases for primary evaluation. A correct candidate is present for 92.2% of these cases, but the final system selects one correctly for 87.2%, yielding a 5.0-percentage-point coverage-to-selection gap. Controlled changes to synthesis, semantic analysis, and selection do not consistently eliminate this gap. Solver-backed behavioral evidence likewise does not reliably resolve selection when multiple semantically distinct candidates remain plausible given the natural-language requirement.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Natural Language Processing, Temporal Logic, Neuro-Symbolic Reasoning, Formal Methods, Program Synthesis, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iAYhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Pendulum, a neuro-symbolic framework that separates candidate generation, symbolic analysis, and selection for natural language-to-LTL translation. It also audits the DStricky benchmark and identifies a measurable gap between candidate coverage and final selection accuracy.

Critical strengths
  1. The distinction between candidate generation and selection provides a useful perspective on neuro-symbolic formalization.
  2. The benchmark audit identifies substantial reference errors and ambiguous specifications.
  3. Solver-backed semantic equivalence provides a rigorous evaluation of formalization correctness.
  4. Controlled experiments systematically investigate selection failures and component-level effects.
Critical weaknesses
  1. The primary evaluation relies on a single benchmark with only 258 determinate instances.
  2. Direct performance comparisons with recent neuro-symbolic LTL translation methods are insufficient.
  3. The proposed pipeline does not consistently improve candidate selection, limiting its demonstrated methodological contribution.
  4. The benchmark corrections and ambiguity judgments involve subjective interpretation, potentially affecting evaluation reliability.
Questions for authors
  1. Can the coverage–selection gap be validated on additional NL-to-LTL benchmarks?
  2. How does Pendulum compare with NeuroSTAR and NeuroNL2LTL under matched evaluation settings?
  3. What is the incremental benefit of symbolic evidence over simpler candidate selection strategies?
  4. Can independent expert annotations further validate the benchmark corrections and ambiguity classifications?
Reviewer_mWXJOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents Pendulum, a pipeline for natural-language-to-LTL translation that separates candidate generation, symbolic analysis, and final selection. The authors audit an existing benchmark and evaluate primarily on 258 specifications classified as determinate. Their main configuration generates a correct candidate for 238 instances but selects one correctly for 225, leaving 13 selection failures. Additional experiments investigate whether changes to synthesis, symbolic evidence, and selection reduce this gap.

Critical strengths
  • Separating candidate coverage from final accuracy is a useful diagnostic for identifying where a formalization pipeline fails.
  • Evaluation uses logical equivalence rather than textual matching, appropriately accounting for semantically equivalent LTL formulas.
  • The benchmark audit addresses an important evaluation concern. Corrected references and ambiguity annotations could be useful resources if made accessible and independently validated.
  • The authors report unsuccessful interventions and account for regressions rather than reporting only newly solved examples.
Critical weaknesses
  1. The novelty is insufficiently established. The central observation-that a system may generate a correct candidate without selecting it-is expected in any imperfect generate-and-rank pipeline. The coverage–selection distinction is useful, but the paper does not adequately explain what new methodological or conceptual contribution it provides beyond measuring oracle candidate accuracy and selected-answer accuracy separately. Pendulum combines existing synthesis, compilation, solver, and LLM-judging components, without clearly establishing a novel selection mechanism.

  2. The empirical findings have limited scope. The headline result rests on 13 selection failures in one audited benchmark. This establishes that selection is imperfect in the evaluated system, but does not by itself provide a substantial new explanation of why selection fails or how to improve it. The appendix contains numerous interventions, so the issue is not simply the number of experiments; it is their ability to support a broader conclusion. Appendix B.9 briefly mentions a second corpus for orchestration comparisons, but the central coverage–selection analysis is not systematically replicated across datasets. Additional evaluation across requirement styles, candidate-pool sizes, and independent benchmarks would strengthen the contribution.

  3. The interpretation conflates unresolved ambiguity with selector error. The primary evaluation excludes specifications whose intended formalization cannot be determined. Nevertheless, the discussion repeatedly explains the remaining selection failures as requiring missing information about the intended reading. For genuinely determinate cases, the relevant information should already be available in the requirement; the model may simply fail to use it correctly. The experiments do not distinguish this failure from actual underspecification. Similarly, unsuccessful interventions do not establish that additional symbolic evidence cannot help.

  4. The benchmark audit requires stronger validation. The authors describe model-assisted screening and author adjudication, but do not clearly report independent human annotation, agreement rates, or a reproducible adjudication protocol. These details matter because the corrected references define correctness for the main result. Some examples also warrant closer examination: Appendix E associates “If x1 is true at the beginning then x2 is true at the end” with G(x1 -> F x2), which imposes a recurring obligation and does not directly represent an initial-state condition or an explicit endpoint under infinite-trace semantics.

  5. Several reported quantities need reconciliation. Appendix B.6 associates the increase from 221/258 to 225/258 with paired discordant counts of b = 8, c = 0; those counts imply eight additional successes, whereas the reported totals differ by four. Section 4.2 also attributes an increase in coverage from 237 to 238 to filtering, merging, and deduplication, without explaining how these operations introduce a newly correct candidate. These discrepancies undermine confidence in the quantitative reporting.

  6. The practical benefit and reproducibility remain unclear. Appendix B.7 reports that a cross-source agreement selector matches the headline pipeline’s 225/258 accuracy, albeit with a different candidate pool. A comparison using matched pools and inference budgets would help establish the value of the judge and symbolic-evidence stages. In addition, I could not access the linked artifact, preventing inspection of the corrected benchmark, prompts, and per-instance results.

Questions for authors

See weaknesses.

Reviewer_o1iyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies NL-to-LTL translation by separating candidate coverage (any-of, whether a semantically equivalent formula is in the candidate pool) from candidate selection (top-1, whether the final chosen formula is correct). The authors propose Pendulum, a neuro-symbolic pipeline combining two synthesis paths, solver-backed analysis (BLACK), and an LLM judge. They also audit the DStricky benchmark by fixing reference errors and removing ambiguous cases, and run controlled ablations showing that while compiler prompt guidance helps synthesis, extra symbolic evidence or selection tweaks do not reliably close the coverage-to-selection gap.

Critical strengths
  1. Separating candidate coverage from final selection is a clean and useful way to diagnose NL-to-LTL systems. It highlights an important boundary for verifier-guided generation: a formal solver like BLACK can check equivalence or generate distinguishing traces between formulas, but it cannot tell which non-equivalent formula matches an underspecified natural language sentence.
  2. Auditing the benchmark to fix broken ground-truth labels and separate genuinely ambiguous requirements is a valuable contribution to the community, and the honest reporting of negative results across the selection interventions is refreshing.
Critical weaknesses
  1. The empirical scope is narrow. Everything is evaluated on a single small dataset, and in the main configuration the gap between coverage and top-1 selection is only 13 examples. Running a dozen ablations on such a small residual error set makes it hard to tell whether the interventions fail in general or if performance is simply hitting a ceiling on this dataset. There is also no direct experimental comparison with existing neuro-symbolic NL-to-LTL systems on the audited benchmark.
  2. The main text is missing many basic details and relies too heavily on the appendix. Section 2 does not explain how the LLM judge scores candidates or how the deterministic finalizer ranks them, Table 3 summarizes twelve ablations purely with text labels ("no consistent gain") without showing any numbers, and the main paper never shows a concrete example of an NL requirement alongside its competing LTL candidates. Also, the benchmark audit relies on author adjudication without reporting inter-annotator agreement.
Questions for authors
  1. How exactly does the deterministic finalizer rank candidates from the LLM judge output, and can you include the actual quantitative numbers for Table 3 in the main text?
  2. During the benchmark audit, how many annotators independently reviewed the flagged examples, and what was the inter-annotator agreement when deciding whether a specification was erroneous versus ambiguous?
  3. In Table 2, why does candidate coverage (any-of) increase from 85.2% to 88.6% and why does the denominator change when moving from Gemma 31B to Gemma 12B?
  4. How do recent neuro-symbolic baselines (such as those listed in Table 4) perform on the audited 258-example benchmark in terms of coverage and selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.