- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper defines policy hallucination as lack of source support for an individual claim, distinguishing it from candidate-program execution and information recoverability. It trains a graph/text/source-alignment detector on controlled policy constructions, with individual unsupported-claim and paired-sensitivity supervision, and explores development-only checkpoint and repair diagnostics. The original controlled test achieves AP 91.88% and F1 81.24%. A structurally broader standalone test transfers poorly: AP is 56.56% versus a 66.26% prevalence baseline, and raising graph capacity does not resolve the failure. The paper carefully describes the bounded semantics and these limitations.
Critical strengths
-
The semantic distinctions are careful and consequential. Missing permission is not treated as prohibition; conflicting support, unresolved evaluation, and information loss are separated; and a structural defect need not invalidate the current claim (Sections 3–4; Appendices A–C). This provides a useful specification for source-grounded detection.
-
The experimental reporting is unusually candid about coverage, operating-point transfer, family dependence, and the limits of development-only diagnostics (Section 7; Appendices E–G). Reporting the broader-test failure and the unsuccessful capacity intervention makes the negative generalization result informative and prevents the original-test accuracy from being mistaken for established deployment performance.
Critical weaknesses
-
There is no access-matched learned baseline or ablation demonstrating the value of the graph/logic architecture. Appendix G explicitly leaves graph versus serialized structure, source alignment, and checkpoint access without matched comparisons. Prevalence and readiness baselines do not distinguish benefits of logical structure from simpler source/answer cues or template regularity. This is central given that standalone AP remains below prevalence even after capacity repair (56.44% versus 65.53%); the present evidence does not explain which design choice accounts for the original-test signal or transfer failure.
-
The central empirical result cannot be independently reproduced from the described release. The reproducibility statement (PDF p.10) excludes source policies, the training corpus, full prediction records, and trained weights; aggregate arithmetic checks cannot substitute for rerunning the learned detector. Because controlled data construction determines the labels and structural regularities, the unavailable data or complete generation recipe materially limits evaluation of the method. A single training seed further leaves the stability of the reported representation comparison unresolved.
Questions for authors
-
Under identical information access, splits, and compute, how does the detector compare with a learned source/text/program baseline and graph/source-alignment ablations? Which component produces the gain on individual unsupported claims? These comparisons could materially change the publication case.
-
Can the complete controlled-data generator or equivalent corpus, model checkpoint, and per-example predictions be released so the original and standalone evaluations can be rerun? Please clarify the obstacle to releasing these synthetic artifacts.
-
How stable are the original and standalone results across training seeds and warm starts? A reproducible rerun would help distinguish a representation limitation from a particular training outcome.