acceptodds
Under review as a conference paper at ICLR 2027

CPUNeSy: Controlling Model Writes for Reliable Neuro-Symbolic Reasoning

Abstract

Large language models are highly effective at recalling statistical patterns, but their accuracy degrades sharply when answers must be derived rather than recalled, especially on deep multi-hop chains. Delegating derivation to deterministic symbolic executors shifts the reliability question to whether model-generated premises are supported by the source. We introduce CPUNeSy (Certified Predicate-Unit Neuro-Symbolic Networks), a serving architecture that controls model writes to symbolic state through a task-defined predicate interface and a certificate gate, and abstains when grounding passes disagree. A component analysis separates deterministic execution from restricted grounding, agreement, and source rechecking. Our experiments show that deterministic execution accounts for most of the accuracy recovery on derivation-heavy tasks, whereas controlled writes primarily improve selective reliability by withholding unsupported or inconsistent answers at the cost of coverage. On controlled multi-hop stress tests in law and formal mathematics, deterministic execution recovers most of the accuracy gap over chain-of-thought and retrieval baselines, with full-pool gains up to 35.0 points. Certification behaves as a selective-serving control rather than an accuracy mechanism: with grounding traces held fixed on ContractNLI, source rechecking removes a quarter of DeepSeek's wrong answers that survive two-vote agreement, at a measurable coverage cost. When abstention is expensive, routing withheld cases to an explicitly uncertified same-model fallback raises full-pool accuracy on MedCalc-Bench Verified by and points for Seed and DeepSeek; these gains do not belong to the certified serving channel. On LeanDojo Benchmark, kernel-restricted pools match BM25 recall@ (). Gains depend on the grounding model's error regime: bias-dominated grounders benefit less, consistent with the voting bound we state formally. The resulting certificates guarantee derivational validity relative to admitted premises; semantic faithfulness to natural-language sources remains conditional on the source checker, and prospective validation of the deployment protocol is future work.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-symbolic reasoning, Large language models, Formal reasoning, Certified grounding, Symbolic execution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tahVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

It is difficult for me to provide a summary of the paper, as it is so heavily LLM written, with vague and confusing terms (e.g., "architecture's contract," "adversarial mid-link break negatives," "learned retriever," "certificate gate," etc.) that it is almost impossible to understand what the authors have in mind. Nevertheless, I will do my best, without spending days trying to decipher the paper.

The authors propose CPUNeSy, an architecture consisting of a large language model, which maps a query and a context (e.g., "Does Alice own the book?") into a symbolic program, grounds the symbols, and executes the resulting program using a solver. Before the formula is passed to a solver, it undergoes a "deploy" (formalization) and "serve" stage, during which the formalization is corrected (although it is not clear to me against what or according to which criteria). It is also unclear to me in which language the formalization takes place and which solver is used for grounding.

Critical strengths

I was not able to fully understand the paper due to the unclear and overly AI generated writing. There may nevertheless be some interesting ideas here, perhaps particularly around auto-formalization for Neuro-Symbolic AI, which is an active and relevant research direction. However, in its current form, I find it extremely difficult to assess the paper fairly. Research papers should be written to be understandable to human readers, rather than requiring an LLM to either interpret or reconstruct the intended meaning.

Critical weaknesses

The paper is extremely difficult to understand in its current form. The writing is heavily AI generated, often vague and imprecise, making it difficult to determine what the authors are actually proposing, what the main claims are, and how the different components of the approach fit together.

Below, you will find some insights into what I understood from the paper, along with an attempt to suggest ways to improve it.

The proofs are very vague. In particular, the bound (Lemma 1) is not clearly described, and it is unclear whether it is useful or what insight it provides.

The plots are similarly difficult to interpret. Their purpose and main takeaways are unclear, and they are not sufficiently discussed in the text.

Finally, for what I understood, the overall approach seems somewhat close in spirit to Stein et al. (2025), "Neuro-symbolic programming in the age of foundation models: Pitfalls and opportunities". If that is the case, this work should be discussed.

Questions for authors

See the main weaknesses.

Reviewer_e3XbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The main problem this paper addresses is that, in neuro-symbolic systems, the LLM may write facts into the premises that are not in the source text, which affects the solver's reasoning results. The authors therefore propose adding a gate between the LLM and the solver, which checks whether the facts proposed by the LLM are supported by the source text and restricts the predicates the LLM is allowed to write. The authors run experiments on five test suites and conduct component ablation studies.

Critical strengths
  1. The paper targets a real problem: an LLM may write facts that are not supported by the source text into the solver's premises. The solver will then derive wrong answers through otherwise valid reasoning.
Critical weaknesses
  1. The models used in the experiments are never given their full names in the main text, and the paper provides no citation for the identity or version of any model. There is also a basic error in which one label refers to two different models: "DeepSeek" in Table 2 is deepseek-flash, whereas "DeepSeek" in Table 4 is deepseek-chat.

  2. The provenance of the benchmarks in the main table is unclear, and the caption "public benchmarks" does not match what is actually reported. The LD-MH questions and labels are produced by the authors themselves, so it's not a public benchmark. ContractNLI is not cited, and the two ContractNLI columns in Table 2 seem not evaluated on the same test set: the Seed column uses the dev set (1,037 items) and the DeepSeek column uses the test set (2,091 items). MedCalc-Bench Verified is likewise not cited and its source and construction are not described.

  3. The implementation of the gate does not match its definition, and it is not the same thing across tasks. The method section defines the certificate gate as a unified admission check, but in the experiments it is several mutually different checks. The central "source support" check has no independent verifier and no reliability guarantee; on ContractNLI, the LLM recheck with the Seed model actually withholds more correct answers than wrong ones.

  4. The certificate and the derivation check exist only on paper. In Figure 2 and in the experiments, the derivation is produced by a deterministic solver, in which case having the gate check the solver's derivation seems redundant. Also the paper acknowledges that "Complete per-query certificate archival was not instrumented in every adapter".

  5. The theorems are disconnected from the paper's setting and claims: their assumptions are not satisfied in the experiments, or their parameters cannot be estimated, and the authors themselves acknowledge this in several places. These theorems should not be listed as a contribution.

  6. The seven baselines in the main table are not comparable. In the appendix the authors themselves state that these results are not used for comparison, yet the main text claims superiority on the basis of them; the two statements thus directly contradict each other. The authors acknowledge that they do not know the experimental conditions of these baselines, so it cannot be verified that their setting matches the main experiments. Moreover, these results can only have been run by the authors themselves, since the benchmarks involved include ones the authors constructed. Some baselines clearly did not run properly (on the three-way ContractNLI: Logic-LM 0.0 / 0.0, MenTaL 1.8 / 0.0, LINC 16.0 / 0.2), and no reason is given.

  7. The only contribution of the newly added mechanism is a small reduction in the error rate on the answered subset, which however is not compared against any selective-prediction baseline at matched coverage. Its cost is lower overall accuracy and a doubling of model calls. In all 13 comparisons reported in the paper, the full method seems to have lower overall accuracy than its simplified variant.

  8. Figures 3 and 4 cannot be interpreted and support no conclusion. Readers can hardly tell what is being plotted, and the authors themselves acknowledge that the conclusions do not depend on them. Neither figure explains what the "log features" are; no definition appears anywhere in the paper. Nor is it stated what is being clustered or which clustering algorithm is used. The four panels of Figure 3 have nearly identical shapes, even though they correspond to four different tasks and use two different models. The caption of Figure 3 mentions ellipses for "bootstrap 95% centroid regions", but no such ellipses seem to appear in the figure.

  9. The writing is very disorganized, with many symbols and terms that are not defined in context, e.g. "historical runs" (pp. 2, 4, 8), "adapter" (p. 4, Table 1), "mid-link-break negatives" (pp. 2, 6, 7), "leaf-Ω" and "the protocol's τ = 0.5" (p. 9), DdcertD_d^{\mathrm{cert}} in Theorem 1 (p. 6), and "(δ, f_c, B) configuration" in Figure 2. The figures are carelessly made: several use fonts that are too small, the main figure repeatedly contains symbols that are hard to understand, and font sizes are inconsistent in many places.

Questions for authors
  1. What experimental settings were used for each of the seven baselines in Table 2? Can they be rerun under matched conditions as the main experiments? What caused the near-zero scores on ContractNLI?
  2. Why do the Seed and DeepSeek columns for ContractNLI in Table 2 appear to use different data splits (dev and test)?
  3. In Appendix G.2, the explicit admission check is only evaluated by offline replay on L1-HARD: in the recorded runs all proposals were already within the allowed vocabulary, and the 120 rejected items were injected artificially. In any run with a model in the loop, was the explicit gate actually executed, and did it reject anything the model proposed?
  4. Could the authors add comparisons with existing abstention methods, for example sampling CoT multiple times and abstaining on disagreement, evaluated at matched coverage?
Reviewer_EJeFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CPUNeSy is a neuro-symbolic serving architecture that controls which LLM-proposed premises may enter symbolic state. The LLM only extracts leaf facts through a task-defined predicate interface. A certificate gate checks their source support, a deterministic Horn-rule executor derives the answer, and the system abstains when grounding passes disagree. On legal, mathematical, and medical benchmarks, it outperforms chain-of-thought and retrieval baselines on multi-hop tasks, mainly due to deterministic execution, while the gate trades coverage for fewer unsupported answers.

Critical strengths

Promising direction. Delegating derivation to a Datalog-style Horn executor makes multi-hop reasoning deterministic and inspectable, and it scales well given correct rules and good autoformalization.

Strong results on derivation-heavy tasks. The method clearly outperforms CoT and RAG baselines on L1-HARD, M-HARD, and LD-MH, with paired significance tests.

Critical weaknesses

Overclaimed novelty. Section 2 claims “whether the written premises are supported by the source is never asked..” This overlooks closely related uncited work like https://openreview.net/pdf?id=zHuV3Vatov, https://arxiv.org/pdf/2511.09008.

Rule sets are central but their origin and quality are unreported. Every answer comes from executing a fixed rule set per task, yet the paper never states who built the rules, how, or at what cost for L1-HARD and ContractNLI. Please describe rule construction and effort and give examples.

Grounding, i.e., the step where the LLM extracts facts from the source as the main source of error in the system, but it is underspecified and not directly evaluated. The paper provides little detail on the grounding prompts or how the predicate vocabularies and schemas were constructed for each task. Grounding accuracy is also not measured against human annotations on the real benchmarks.

Undefined terms. Core terms are used before definition or never defined: “Horn rule,” the basic representation unit, is neither defined nor cited. The argument in the introduction relies on term like “rule-head predicate” before they are defined. Cn(·), S_O, and δ appear before their definitions.

The theory does not support the reliability claims where they matter. The formal results are correct but largely definitional. Can the authors better explain the purpose of the theory here?

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.