acceptodds
Withdrawn from ICLR 2027

Logic-Aware Policy Hallucination Detection through Text-to-Graph Encoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Translating policy text into an executable program can omit conditions needed for the correct decision. As a result, policies that should produce different decisions for the same question and context may produce identical outcomes after translation. This loss of semantic sensitivity contributes to hallucination when the system asserts an unsupported conclusion. Successful symbolic execution alone cannot establish source support, so automatic detection must connect source meaning with the completed claim. We learn continuous, query-conditioned graph embeddings to recognize such failures in supplied verifier runs. A graph-only branch predicts candidate execution, while a source-aware branch aligns policy units with program nodes and scores each asserted claim without requiring a comparison partner. A paired branch uses symmetric and directional features to classify changes in semantic sensitivity. Logic-derived supervision trains these differentiable representations without sending gradients through the compiler or symbolic evaluator. Source-grounded repair and replay assess contribution separately. On 10,845 held-out controlled pairs from 600 policy families, sensitivity accuracy is 63.89%. Individual-claim detection achieves 81.24% F1 and 91.88% average precision on 4,635 evaluable assertions, against a 65.46% constant-score baseline. On a structurally broader benchmark, average precision was 56.56% among scored claims; a subsequent capacity check accommodated all 3,974 evaluable claims and yielded 56.44%. These single-seed results establish a detection signal in controlled logical settings alongside substantial transfer failures. Outcome recoverability, executable fidelity, and source-supported claims remain distinct.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
policy-program embeddings, hallucination detection, neuro-symbolic learning, semantic sensitivity, graph representation learning, formal verification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_apYiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines policy hallucination as lack of source support for an individual claim, distinguishing it from candidate-program execution and information recoverability. It trains a graph/text/source-alignment detector on controlled policy constructions, with individual unsupported-claim and paired-sensitivity supervision, and explores development-only checkpoint and repair diagnostics. The original controlled test achieves AP 91.88% and F1 81.24%. A structurally broader standalone test transfers poorly: AP is 56.56% versus a 66.26% prevalence baseline, and raising graph capacity does not resolve the failure. The paper carefully describes the bounded semantics and these limitations.

Critical strengths
  1. The semantic distinctions are careful and consequential. Missing permission is not treated as prohibition; conflicting support, unresolved evaluation, and information loss are separated; and a structural defect need not invalidate the current claim (Sections 3–4; Appendices A–C). This provides a useful specification for source-grounded detection.

  2. The experimental reporting is unusually candid about coverage, operating-point transfer, family dependence, and the limits of development-only diagnostics (Section 7; Appendices E–G). Reporting the broader-test failure and the unsuccessful capacity intervention makes the negative generalization result informative and prevents the original-test accuracy from being mistaken for established deployment performance.

Critical weaknesses
  1. There is no access-matched learned baseline or ablation demonstrating the value of the graph/logic architecture. Appendix G explicitly leaves graph versus serialized structure, source alignment, and checkpoint access without matched comparisons. Prevalence and readiness baselines do not distinguish benefits of logical structure from simpler source/answer cues or template regularity. This is central given that standalone AP remains below prevalence even after capacity repair (56.44% versus 65.53%); the present evidence does not explain which design choice accounts for the original-test signal or transfer failure.

  2. The central empirical result cannot be independently reproduced from the described release. The reproducibility statement (PDF p.10) excludes source policies, the training corpus, full prediction records, and trained weights; aggregate arithmetic checks cannot substitute for rerunning the learned detector. Because controlled data construction determines the labels and structural regularities, the unavailable data or complete generation recipe materially limits evaluation of the method. A single training seed further leaves the stability of the reported representation comparison unresolved.

Questions for authors
  1. Under identical information access, splits, and compute, how does the detector compare with a learned source/text/program baseline and graph/source-alignment ablations? Which component produces the gain on individual unsupported claims? These comparisons could materially change the publication case.

  2. Can the complete controlled-data generator or equivalent corpus, model checkpoint, and per-example predictions be released so the original and standalone evaluations can be rerun? Please clarify the obstacle to releasing these synthetic artifacts.

  3. How stable are the original and standalone results across training seeds and warm starts? A reproducible rerun would help distinguish a representation limitation from a particular training outcome.

Reviewer_VPuvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper trains a neural detector to catch cases where translating natural-language policies into programs drops a condition and leads to unsupported answers. It works on data similar to training but falls below a constant-score baseline on structurally different data.

Critical strengths

I don't find any strength in this paper, as it looks very much like an AI generation without any supervision and insights.

Critical weaknesses

This paper is extremely difficult to follow and has a bunch of text and figures that are irrelevant to the method it proposes. According to paper "Among scored claims, F1 is 35.52%; AP is 56.56%, below the 66.26% constant-score baseline.", the method cannot beat a baseline that scores equally on all samples.

Questions for authors

How does a simple LLM-as-judge baseline perform (e.g., prompting a strong LLM with the policy text, program, and answer to judge support)? Without this, it is unclear whether the learned graph detector offers any advantage.

On the standalone benchmark, AP (56.56%) falls below the constant-score baseline (66.26%). What do the authors take as the main contribution given this result, and why should the method be considered effective?

Reviewer_idqNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies hallucination detection in systems that translate policy text into executable programs. The authors distinguish whether a representation retains enough information to recover the correct outcome, whether the program executes faithfully, and whether the final claim is supported by the source policy. The proposed detector combines query-conditioned graph representations, source-to-node alignment, and verifier-trace features to assess individual claims. Logic-derived labels supervise the neural components, while program construction and symbolic execution remain fixed. A separate repair-and-replay procedure examines whether correcting a defect changes the proof or the emitted claim.

The paper evaluates the detector on constructed policy corpora with controlled language and deliberately varied programs, queries, and answers. The experiments report a detection signal on the original held-out corpus, but substantial performance degradation on a benchmark with broader logical structures, where average precision falls below the constant-score baseline. Additional analyses examine scoring coverage, threshold transfer, and the effects of specified repairs on proof support and claim emission.

Critical strengths

S1. The paper makes a useful distinction between executable fidelity and support for the claim actually asserted. This gives the detection task a more precise target than program disagreement alone and helps avoid labeling defects that do not affect the current claim as unsupported answers.

S2. The repair-and-replay procedure provides an explicit way to examine whether a suspected defect affects the answer. This distinction is useful because correcting the program’s justification does not necessarily remove an unsupported assertion from the final answer.

Critical weaknesses

W1. My main concern is that the detector does not demonstrate reliable generalization beyond the original controlled corpus. On the structurally broader benchmark, average precision falls below the constant-score baseline, while claim-detection F1 drops substantially. Increasing the input capacity restores scoring coverage but does not resolve the detection failure. Since both benchmarks use constructed policies and controlled language, these results leave substantial uncertainty about the method’s applicability to unfamiliar logical structures and naturally occurring policy-verification errors. The original-test results establish a detection signal, but do not provide sufficient evidence for a broadly useful detector.

W2. The contribution of the proposed architecture is not sufficiently established. The detector combines graph encoders, source-to-node alignment, checkpoint features, and several prediction heads, but the evaluation lacks matched comparisons isolating these components. In particular, there is no comparison with a simpler detector using serialized programs and the same source, query, and trace information. The supplementary configurations also differ in training procedures and do not provide controlled architectural comparisons. Consequently, the reported detection performance does not establish whether the graph representation, alignment mechanism, or checkpoint modeling provides an advantage over simpler alternatives.

Questions for authors

Q1. What accounts for the performance degradation on the structurally broader benchmark? Could the authors provide a breakdown by logical category, graph structure, and defect type to clarify which changes cause the detector to fail?

Q2. Could the authors compare the detector with a serialized-input model given the same source, program, query, and trace information under matched training conditions? What additional benefit comes from graph encoding, source-to-node alignment, and checkpoint modeling individually?

Q3. How does the information-preservation analysis in Section 4 inform the proposed detector? Could the authors provide empirical evidence connecting recoverability or information loss to detection performance, beyond the exact constructions?

Q4. What trusted information is required to construct and validate the repair alternatives in Section 6? Could the authors clarify how the learned detector and the repair-and-replay procedure would be used together when a trusted reference program is unavailable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.