- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper associates a model's decisions with formal guarantees, which can be checked without referring to the model, through a knowledge property graph (KPG) built from an authority's own claims. The effectiveness of the approach is demonstrated on synthetic and real authorities, and the guarantees are tested under injected intake faults, tampered witnesses, and supplied facts.
Critical strengths
- The framing identifies a genuine gap between accuracy and derivation, i.e., a decision that carries the derivation establishing it is a different object from one that carries a probability.
Critical weaknesses
- The paper reads like an AI-handwaved draft without clear contextual support. It is not self-contained and provides no running example to demonstrate its motivation and methodology; the formal notations and the theorems are overly conceptual and lack solid semantic grounding; and the results are reported without the details of the evaluation protocol or clear explanations of the figures.
- It is rather confusing what role ground truths have to play in the methodology. A derivation showing how a decision is reached neither has to assume nor necessarily implies that the decision is correct. However, the whole work is motivated from the perspective of accuracy, while the experimental evaluation also heavily relies on ground truths to establish the validity of the methodology itself.
- The structures and semantics of KPGs are not well defined, and it is not clear where a KPG may come from. Moreover, a KPG may not be sound, nor complete. Thus, the resulting guarantee is issued with correctness left unjustified.
- The three closest lines of work discussed in Section 4 are not thoroughly compared in the evaluation, especially considering conversational machine reading also allows facts to be supplied.
Questions for authors
- Is an abstention a report that the case is genuinely undecidable on the authority, or a report that the given KPG does not settle it?
- Could the probability of a decision also be machine-checked, given a KPG or other mechanism? Would it be possible to lift the derivations to the probability distribution of the outcome?
- Do the Lean proofs generalize for any given authority, with the coarse definitions of the two theorems? How are the evaluated authorities defined within the theorems and proofs? In particular, is the correspondence between the descent as formalized and the descent as implemented checked, and by what means?
- How would the methodology deal with the discrepancy between recorded criteria and recorded labels, and the ambiguity case where a specification admits more than one descent, as somehow indicated in Appendices I and J?
We thank you again for your careful questions. We have addressed your central concerns about the assumptions underlying the formal theorems, the separation of verification cases, and the independence of the held-out evidence in the revised paper. Theorem 1 no longer assumes the semantic property. Theorem 2 now separates five cases, each with its own check. The distinction between development-set results and held-out evidence has also been clarified in response to your feedback. On the conditions-enforced runs, the checker verifies every condition and value reading against the KPG, without labels.
If this resolves your concerns, we would be grateful if you would consider updating your assessment. If anything remains unclear, we would welcome the chance to address it before the discussion closes.