acceptodds
Under review as a conference paper at ICLR 2027

A Formal Guarantee at the Individual Decision

Abstract

An accuracy rate says nothing about whether the decision in front of you is right. We show a construction that does by routing a probabilistic system’s decisions through a knowledge property graph (KPG), a graph that holds an authority’s own claims, each anchored to its source and marked with who stands behind it and whether it still binds. The KPG decides by descent and eliminates provisions where the authority licenses it. It takes a verdict on each surviving element in three values, and commits only where every way of settling the open elements reaches the same outcome. The guarantee is a theorem. If the KPG represents the authority and the settled verdicts are correct, every committed decision is the authority’s decision. Every decision carries a witness, a certificate a checker verifies without the model or the system that produced it, and if a premise fails, the witness names the claim or the verdict, or bounds the claim the KPG lacks. A decision is therefore as right as the KPG and the authority it answers to, and every shortfall is located, so an organization corrects its KPG and every later decision inherits the correction. The construction is exercised on five authorities. On 700 variant classifications under the ACMG/AMP standard as one expert panel’s four specifications refine it, with the KPG’s conditions enforced on every recorded criterion, no commitment deviates from the authority’s decision. Two disagree with the curator’s label, and each is located in the label. On 537 curations from a second panel, with the same code, the same holds, and with the curator’s criteria taken as settled, the one commitment that disagrees with a label is located in the label. Authority enters the record by two witnessed paths. Attestation adds or amends a claim to the KPG itself. Adjudication supplies a fact to one abstention and can change only that one. Without the KPG’s certification, injected intake faults produce 615 silent wrong answers. With it, every fault is caught before it reaches a decision, and named so it can be corrected.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
knowledge property graph, witness, certificate, proof-carrying code, attestation, adjudication, decision correctness, formal guarantees, selective prediction, abstention, certified decisions, neurosymbolic reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fv9BOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper associates a model's decisions with formal guarantees, which can be checked without referring to the model, through a knowledge property graph (KPG) built from an authority's own claims. The effectiveness of the approach is demonstrated on synthetic and real authorities, and the guarantees are tested under injected intake faults, tampered witnesses, and supplied facts.

Critical strengths
  • The framing identifies a genuine gap between accuracy and derivation, i.e., a decision that carries the derivation establishing it is a different object from one that carries a probability.
Critical weaknesses
  • The paper reads like an AI-handwaved draft without clear contextual support. It is not self-contained and provides no running example to demonstrate its motivation and methodology; the formal notations and the theorems are overly conceptual and lack solid semantic grounding; and the results are reported without the details of the evaluation protocol or clear explanations of the figures.
  • It is rather confusing what role ground truths have to play in the methodology. A derivation showing how a decision is reached neither has to assume nor necessarily implies that the decision is correct. However, the whole work is motivated from the perspective of accuracy, while the experimental evaluation also heavily relies on ground truths to establish the validity of the methodology itself.
  • The structures and semantics of KPGs are not well defined, and it is not clear where a KPG may come from. Moreover, a KPG may not be sound, nor complete. Thus, the resulting guarantee is issued with correctness left unjustified.
  • The three closest lines of work discussed in Section 4 are not thoroughly compared in the evaluation, especially considering conversational machine reading also allows facts to be supplied.
Questions for authors
  1. Is an abstention a report that the case is genuinely undecidable on the authority, or a report that the given KPG does not settle it?
  2. Could the probability of a decision also be machine-checked, given a KPG or other mechanism? Would it be possible to lift the derivations to the probability distribution of the outcome?
  3. Do the Lean proofs generalize for any given authority, with the coarse definitions of the two theorems? How are the evaluated authorities defined within the theorems and proofs? In particular, is the correspondence between the descent as formalized and the descent as implemented checked, and by what means?
  4. How would the methodology deal with the discrepancy between recorded criteria and recorded labels, and the ambiguity case where a specification admits more than one descent, as somehow indicated in Appendices I and J?
Reviewer_KtheOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a knowledge property graph (KPG) for decisions governed by written authorities, including regulations and clinical-variant classification standards. The KPG represents provisions, factual elements, combination rules, and precedence relations extracted from an authority. A probabilistic model may be used to construct the KPG and read individual case elements, but the governing provisions and final outcome are computed without a model. The system commits only when every completion of the undetermined elements produces the same outcome; otherwise, it abstains and names the blocking elements. Each commitment includes a witness that an independent checker can replay. The paper proves conditional soundness and localization results, assuming that the KPG faithfully represents the authority and that the settled element verdicts are correct. Experiments cover a synthetic regulatory register, two legal tasks, and two clinical-variant expert panels. The paper reports no observed wrong commitments relative to the applicable KPG and record, the successful discharge of 129 case-level abstentions using named facts, and the conversion of injected intake faults from silent wrong answers into named abstentions.

Critical strengths
  1. Replayable decision-specific certificates.
    The paper draws a useful distinction between probabilistic confidence and a replayable certificate for an individual decision. Separating model-based reading from symbolic decision-making is conceptually clean and relevant to high-accountability applications.

  2. Actionable abstentions.
    The system identifies the open elements preventing a commitment rather than returning generic uncertainty. For example, 40 of 41 FRE 801 abstentions are discharged using label-independent benchmark annotations (§5 and Appendix G).

  3. Explicit assumptions and trust boundaries.
    The paper acknowledges that correctness is relative to the KPG, the governing authority, the case record, and the correctness of settled element verdicts (§6–7). This makes the scope of the guarantee more transparent.

  4. Informative negative controls.
    Removing precedence produces 46 wrong Meridian commitments, while allowing a model to rank the remaining candidates produces nine wrong decisions (§5.5). These experiments illustrate why authority completeness and the separation of reading from deciding matter.

  5. Independent witness verification.
    The checker is independent of the model that produced the readings and is subjected to systematic tampering tests. It certifies 1,237 commitments and rejects 7,418 of 7,422 modified artifacts; the four accepted changes concern premises already declared conditional (§5.3 and Appendix F).

  6. Evidence of cross-panel transfer.
    The Familial Hypercholesterolemia experiment applies unchanged decision code to a different expert-panel specification. This is the paper’s strongest evidence that parts of the implementation transfer beyond the principal development setting (Appendix L).

Critical weaknesses
  1. The main theorem assumes the hardest semantic property.
    Theorem 1 is mathematically sound as stated, but its central assumption is strong. The paper defines a KPG as representing an authority over region RR only when its claims are faithful to the authority and its decision function satisfies Φ(G∣R,⋅)=Φ∗(A,⋅)\Phi(G|_R,\cdot)=\Phi^*(A,\cdot) (§2.6). Given this equality, correct settled verdicts, and a singleton version space, the conclusion follows directly, as reflected in the short proof in Appendix B. The automatic intake checks establish textual anchoring, document coverage, and structural well-formedness, but they do not establish semantic equivalence between the KPG and the authority. Therefore, the theorem verifies reasoning relative to a correctly represented authority but does not itself guarantee that the text-to-KPG translation is correct.

  2. The localization theorem provides a candidate set rather than necessarily identifying the exact error.
    Theorem 2 states that a wrong commitment implies at least one incorrect settled verdict named by the witness, a cited KPG claim that the authority does not make, or an authority claim missing from the KPG and bounded to region RR. This is a valid localization result, but it does not necessarily identify which named verdict or claim is wrong, and the missing-claim case may only narrow the search to an entire region. The practical strength of the result therefore depends on the typical number of premises in a witness and the size of RR, neither of which is quantified.

  3. The evaluation does not independently establish semantic representation.
    The experiments provide strong evidence of internal consistency, replayability, and agreement with recorded labels. However, label agreement alone cannot establish that the KPG faithfully represents the governing authority because a label and the constructed KPG could share an error. The paper audits the few commitments that disagree with their labels, but it does not report an independent domain-expert audit of a representative sample of agreeing commitments. Appendix Q also states that the FRE 801 and platelet ACMG pipelines were developed on the sets on which they are evaluated, while §3 describes development subsets. This makes it difficult to determine which semantic claims and design decisions were fixed before evaluation.

  4. Many FRE 801 certificates remain conditional on an unverified premise.
    Appendix F reports that 27 of the 38 certified FRE 801 commitments are conditional on one element—whether the declarant was testifying—that was read from the case text without a recorded reading. The checker correctly lists this premise as conditional, so this does not invalidate the checker. However, these certificates do not independently establish the correctness of that premise and therefore do not provide an unconditional guarantee of the resulting decisions. The headline count of certified commitments should distinguish unconditional certificates from certificates whose soundness still depends on an unverified case-level fact.

  5. Evidence for the correctness of model-settled verdicts is limited.
    The formal guarantee assumes that every settled element verdict is correct. The paper directly studies this assumption on the decisive FRE 801 element for 35 cases, where five readings exhibit correlated errors but all nine unanimous cases are correct (§5.4 and Appendix G). This is useful evidence, but the sample is small, and analogous robustness analyses are not reported for the other model-read elements and registers. Thus, the experiments do not yet establish how reliably unanimity or the other extraction procedures satisfy the theorem’s correct-verdict premise under distribution shift or shared model bias.

Questions for authors
  1. How is semantic representation established in practice?
    What operational procedure establishes that a KPG “represents” an authority beyond textual anchoring, document coverage, and structural validation? Who performed the semantic attestations for each evaluated KPG, what relevant expertise did they have, and how were ambiguous or disputed interpretations handled? The answer would clarify whether the theorem’s principal assumption can be established reproducibly in deployment.

  2. How should conditional certificates be interpreted in the headline results?
    Appendix F states that 27 of the 38 FRE 801 certificates are conditional on whether the declarant was testifying because no reading supporting that premise was recorded. Are these 27 decisions included in the headline claim that every commitment is certified and right? Could the authors separately report unconditional and conditional commitments and explain what additional evidence would convert the latter into unconditional certificates?

  3. How precise is the localization guarantee in practice?
    When Theorem 2 applies, does the witness identify a particular faulty premise, or only a collection containing at least one faulty premise? Please report the distribution of witness sizes, the number of named premises, and the size of region RR across the evaluated commitments. These quantities would show how much human investigation is required after localization.

  4. How were development and evaluation separated?
    Appendix Q states that the FRE 801 and platelet ACMG pipelines were developed on the sets on which they are evaluated, whereas §3 describes separate development examples. Which prompts, decomposition rules, KPG claims, attestations, and implementation choices were frozen before the authors examined the evaluation cases and their labels? This distinction is important for interpreting the reported zero observed errors.

  5. Can the two semantic premises be evaluated independently?
    Can the authors provide an independent domain-expert audit of a randomly selected sample of label-agreeing commitments, including both the KPG representation and the model-settled element verdicts? Such an audit would test the two assumptions required by Theorem 1 rather than relying primarily on agreement with dataset labels.

1 comment

We thank you again for your careful questions. We have addressed your central concerns about the assumptions underlying the formal theorems, the separation of verification cases, and the independence of the held-out evidence in the revised paper. Theorem 1 no longer assumes the semantic property. Theorem 2 now separates five cases, each with its own check. The distinction between development-set results and held-out evidence has also been clarified in response to your feedback. On the conditions-enforced runs, the checker verifies every condition and value reading against the KPG, without labels.

If this resolves your concerns, we would be grateful if you would consider updating your assessment. If anything remains unclear, we would welcome the chance to address it before the discussion closes.

Reviewer_ZUwcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a neuro-symbolic framework for ensuring that individual decisions comply with an authoritative body of rules. The approach relies on a Knowledge Property Graph (KPG), a procedure for identifying relevant provisions, three-valued factual verdicts, and a decision mechanism that commits only when all completions of undetermined facts yield the same outcome.

Critical strengths

The paper addresses an important problem concerning the reliability of AI-assisted decision-making. The combination of symbolic decision procedures, explicit abstention, and independently checkable certificates is appealing. The experimental evaluation is relatively extensive and includes useful fault-injection experiments.

Critical weaknesses

My main concern is the lack of mathematical precision in the presentation of the proposed formal framework, particularly in Sections 2.2–2.4. Several fundamental notions and symbols are introduced without adequate formal definitions. For example, the mathematical structure of the knowledge property graph G is not specified, the notion of an intention q is left informal, and the functions λᵢ and ρ(G,q) are not precisely defined. Consequently, the meaning of the descent equation in Section 2.2 and the conditions under which provisions can legitimately be eliminated remain unclear.

In particular, the condition λᵢ(p) ∈ G only expresses membership of a claim in the KPG. It does not establish that the claim actually justifies eliminating provision p in the relevant context. The paper does not provide a formal semantics or consequence relation specifying when such eliminations are warranted.

Section 2.3 introduces strong Kleene three-valued logic, but without defining a logical language for representing the authority's provisions, their applicability conditions, and precedence relations. The relationship between this three-valued framework, the descent procedure, and the combination function Φ introduced in Section 2.4 is insufficiently explained.

These shortcomings are particularly problematic because the paper's main contribution is presented as a formal correctness guarantee. The soundness theorem assumes that the KPG faithfully represents the authority and that the settled factual verdicts are correct. Under these assumptions, the conclusion follows directly from the singleton version-space condition. The localization theorem is essentially its contrapositive. Thus, the theoretical results appear relatively straightforward, while the central reasoning mechanisms and their semantic foundations remain insufficiently specified.

Overall, the paper relies on established logical ideas without providing a sufficiently precise logical characterization of its own framework. This makes it difficult to assess both the correctness of the proposed reasoning procedure and the novelty of its formal contributions.

Questions for authors
  1. Could the authors provide precise mathematical definitions of the KPG G, the intention q, and the functions λᵢ and ρ, including their domains and codomains?

  2. Why is no explicit logical language and semantics provided for representing the authority's provisions, applicability conditions, and precedence relations?

  3. How exactly are the strong Kleene three-valued semantics and the combination function Φ integrated into the reasoning procedure?

1 comment

We thank you again for your review. Your central concern about insufficient formal justification and the completeness of the soundness argument is addressed. A licensed removal is now defined semantically, the narrowing invariant is proved, and Soundness follows from it. The terms you flagged are defined there, and the relation between strong Kleene evaluation and Φ is stated as a proposition machine-checked in Lean. The checker's decision core is also proved in Lean against the same definitions (Appendix B).

If this resolves your concerns, we would be grateful if you would consider updating your assessment. If anything remains unclear, we would welcome the chance to address it before the discussion closes.

Comments on the paper

3 comments
AuthorsOpenReview ↗

To the Area Chair, Program Chairs, and all reviewers

We thank the AC and every reviewer for a discussion that has substantially improved this work.

On the latest revision. The paper's introduction was edited to further reinforce the intuition, and make it explicitly clear about the work's focus on individual decisions, and the rationale and broader impacts behind it. The edit also contains a more sanitary ordering and structure of the appendix (7 sections, down from 19). No claim, theorem, or result changed.

The discussion confirmed that soundness holds. The theorems withstood the strongest challenges raised against them and are more solid for it. The claims we make hold, and the evidence we present, including the reproducible artifacts, shows that the KPG is a contribution the community should know about for any deployment that requires oversight.

Regarding reviewer fv9B. We believe the review reflects a misunderstanding over the distinction between prediction and verification. The reviewer evaluates KPG as a predictor whose outputs should be scored against labels. However, KPG is a checker that certifies whether a decision follows from the authority under its stated premises. Labels are used only to identify disagreements and locate their causes. We accept part of the responsibility for that reading, and the revised introduction addresses exactly this.

Across the review and subsequent discussion, we have not seen a substantive engagement with this distinction. Despite multiple requests, they have not identified any incorrect theorem, proof, or result to invalidate any of our claims. We therefore respectfully object to the reviewer's stated familiarity with this line of work and the use of that expertise and confidence to dismiss the soundness of our contributions without cause.

Furthermore, several of the reviewer's criticisms are inconsistent with the record. For example, the reviewer stated that the evaluation protocol was not reported, although it was included in the original submission. They wrote that "the whole work is motivated from the perspective of accuracy." Nothing could be further from the truth. The paper is motivated by the impossibility for accuracy to discern correct from incorrect individual outcomes. They also questioned the role of ground truth, only to later state that its use "was never objected to".

We are grateful for the opportunity to participate in this discussion and hope to present our work at ICLR 2027.

On professionalism and misstatement. The reviewer denies evaluating the KPG as a predictor. Their latest comment does exactly that: it says the premises and conclusion of Theorem 1 are "validated based on ground truths." A theorem is not validated by ground truth. It is proved, and Theorem 1 is proved. Judging the method by agreement with labels is how a predictor is evaluated. That was our statement, and their own comment supports it.

On Theorem 1. The evaluation does not validate Theorem 1, and the paper never says it does. Labels only count agreement and locate disagreements. Where the label is the authority's own decision, it cannot depart from the authority. On the Meridian test split, 128 of 130 probes agree, none disagree, and 2 abstain. On held-out FH, with conditions enforced, no commitment disagrees. The checker certifies all 1,237 commitments and rejects all 7,422 tampered copies without reading a label.

On Theorem 2. The reviewer says a missing provision cannot be named by a witness. Theorem 2 says so itself, in case (c). A missing provision can only be tested by removing a known one. The ablation does this, and its result is what the theorem predicts: 37 commitments replay cleanly and no witness names the cause. This is why the checks sit at intake. Across 69 injected faults, including dropped provisions, the unchecked KPG gives 364 silent wrong answers; with the intake checks it gives none, and each affected decision abstains and names the claim that failed.

On accuracy. The paper is titled A Formal Guarantee at the Individual Decision. It has argued against accuracy as the measure from its first sentence. Nothing was rewritten to hide it.

On the protocol. It is stated in the main text, in §3.

On Lean. The authority's text is not formalized, and the paper says so. The proved checker instantiates all three kinds of combination the registers use, each with its own theorem.

On Kthe. We said the premise was split into premises checked locally. It was. Faithfulness over R is premise (iii).

The reviewer has not identified any error in any theorem, proof, or number to justify a soundness score of 1 with a stated confidence of 3. Our objection stands.

On the design. The reviewer asks for one formal guarantee that holds for every real-world authority. The paper rejects that design on purpose. No single specification can stand in for the authorities that govern real decisions, and the only defensible guarantee is one that holds relative to the authority a deployment chooses. A statute, a clinical standard, and an organization's own policy each define what is correct for their domain, and they do not reduce to one another. The same choice runs through economics: one plan for everyone, or each stakeholder specifying their own ends. The KPG is built for the second. It does not decide what is correct. It holds each authority to its own claims, names those claims as premises, and abstains where the authority is silent. Responsibility for what is correct stays with the authority, where it belongs, not with a model or a universal specification.

The results support the design. A deployment may begin by abstaining on the majority of decisions, with each abstention naming what would settle it. The results also show that one or two turns of attestation or adjudication can close the gap.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.