- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper audits prover certified process rewards in which a language model supplies cited premises, an additional assumption, and a conclusion, after which a formalizer and theorem prover certify phi(P) AND phi(A) entails phi(C). The central claim is that the certificate can be laundered: when cited premises do not entail the conclusion, the model can add an assumption that makes the obligation provable, so the prover validates a model written axiom rather than the reasoning step. The authors propose a provenance audit that checks the obligations again after removing the assumption and checks the assumption against the symbolic program generating ProntoQA or ProofWriter. On 15,733 rollout steps, 87% of accepted certificates on locally unsupported steps are carried by the model assumption. Restricting assumptions to program licensed statements improves informedness, while a premises only rule is portable and mainly useful as a high precision filter. The paper also identifies a separate proper name compilation defect caused by the released prompt demonstrations and evaluates a repair using three additional demonstrations.
Critical strengths
- The paper identifies a concrete worst case degeneracy: if the candidate can assume the conclusion, thresholding the prover verdict cannot restore discrimination (Observation 1, Section 4.1).
- The provenance audit separates premise entailment, assumption carrying, reference licensing, conclusion validity, inherited error, and compiler mistranslation instead of treating one prover verdict as ground truth.
- The empirical support is substantial: 15,733 policy rollouts, a held out constructed audit, two symbolic corpora, multiple policy and temperature controls, two formalizers, cross implementation checks, and a second verification design.
Critical weaknesses
- The strongest licensing conclusions require ProntoQA or ProofWriter symbolic programs. On the FOLIO dataset, the e SNLI dataset, the ProverQA dataset, and the Multi LogiEval dataset, the authors can only provide upper bounds because no trusted reference decides assumption licensing or compiler faithfulness. The deployment claim should be scoped more explicitly.
- The reported 87% laundering statistic is conditional on a deterministic reader and mapped steps; the reader covers only 79.9%/76.1% of ProofWriter carrying assumptions on correct/incorrect steps. The paper reports bounds, but the main claim would be easier to evaluate if these coverage limits were prominent in the primary results.
- The matched LoRA DPO experiment has 188 pairs per arm and no power to distinguish selection rules, so the paper does not establish that the proposed filter improves a trained policy.
Questions for authors
- Can the authors state the intended deployment claim separately for settings with a trusted symbolic/reference program and settings without one?
- How sensitive are the 87% and informedness results to alternative assumption readers beyond the reported regex reader, especially on unparsed ProofWriter cases?
- Which claims about the proper name defect are intended to generalize beyond the released LogicReward prompt, and what evidence supports that scope?
- Can the authors clarify whether any downstream policy improvement is claimed? Appendix D.2 appears explicitly underpowered to answer that question.
- Please distinguish more explicitly between an assumption entailed by a generator program and an assumption that should count as an acceptable citation in a deployed verifier.