acceptodds
Under review as a conference paper at ICLR 2027

Assumption Laundering in Prover-Certified Process Rewards

Abstract

Recent work supervises language-model reasoning with a theorem prover: each step is translated into first-order logic, the prover checks that the step's premises entail its conclusion, and the verdict is used as a step-level reward. When the cited premises fall short, some pipelines let the model add an assumption to close the gap. We show that this makes the certificate weaker than it looks, because the model being supervised writes its own obligation: a proof from the model's own axiom is reported as a proof of the model's step, a failure we call assumption laundering. We audit it on 15,733 steps sampled from the base model of a released pipeline on ProntoQA and ProofWriter, whose symbolic generators let us decide both whether a step is valid and whether an assumption is entailed by the problem. Where the cited premises do not entail the conclusion, 87% of accepted certificates hold only because of an assumption the model added, and the certificate cannot tell an assumption the problem entails from one it does not. A model that assumes what it needs to prove is accepted on every step, so no threshold on the verdict can repair this; the assumption has to be licensed by a reference the model does not control. Enforcing this condition raises informedness (acceptance of valid minus acceptance of invalid steps) on ProntoQA from 0.09 to 0.41, and to 0.49 after we also repair a separate compiler defect that we trace to the demonstrations in the released prompt; repairing the compiler alone leaves it at 0.09. A second pipeline that constrains assumptions differently behaves as the condition predicts. Where no reference exists, disallowing assumptions altogether does not give a usable step reward, but it does give a high-precision data filter: 13% of responses kept at 93% answer accuracy against a 72% base rate, ahead of a learned process reward model at the same yield.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
process supervision, process reward models, automated theorem proving, neuro-symbolic reasoning, verifier reliability, reward hacking, formalization faithfulness, LLM reasoning, data filtering, PrOntoQA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s2LdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper audits prover certified process rewards in which a language model supplies cited premises, an additional assumption, and a conclusion, after which a formalizer and theorem prover certify phi(P) AND phi(A) entails phi(C). The central claim is that the certificate can be laundered: when cited premises do not entail the conclusion, the model can add an assumption that makes the obligation provable, so the prover validates a model written axiom rather than the reasoning step. The authors propose a provenance audit that checks the obligations again after removing the assumption and checks the assumption against the symbolic program generating ProntoQA or ProofWriter. On 15,733 rollout steps, 87% of accepted certificates on locally unsupported steps are carried by the model assumption. Restricting assumptions to program licensed statements improves informedness, while a premises only rule is portable and mainly useful as a high precision filter. The paper also identifies a separate proper name compilation defect caused by the released prompt demonstrations and evaluates a repair using three additional demonstrations.

Critical strengths
  1. The paper identifies a concrete worst case degeneracy: if the candidate can assume the conclusion, thresholding the prover verdict cannot restore discrimination (Observation 1, Section 4.1).
  2. The provenance audit separates premise entailment, assumption carrying, reference licensing, conclusion validity, inherited error, and compiler mistranslation instead of treating one prover verdict as ground truth.
  3. The empirical support is substantial: 15,733 policy rollouts, a held out constructed audit, two symbolic corpora, multiple policy and temperature controls, two formalizers, cross implementation checks, and a second verification design.
Critical weaknesses
  1. The strongest licensing conclusions require ProntoQA or ProofWriter symbolic programs. On the FOLIO dataset, the e SNLI dataset, the ProverQA dataset, and the Multi LogiEval dataset, the authors can only provide upper bounds because no trusted reference decides assumption licensing or compiler faithfulness. The deployment claim should be scoped more explicitly.
  2. The reported 87% laundering statistic is conditional on a deterministic reader and mapped steps; the reader covers only 79.9%/76.1% of ProofWriter carrying assumptions on correct/incorrect steps. The paper reports bounds, but the main claim would be easier to evaluate if these coverage limits were prominent in the primary results.
  3. The matched LoRA DPO experiment has 188 pairs per arm and no power to distinguish selection rules, so the paper does not establish that the proposed filter improves a trained policy.
Questions for authors
  1. Can the authors state the intended deployment claim separately for settings with a trusted symbolic/reference program and settings without one?
  2. How sensitive are the 87% and informedness results to alternative assumption readers beyond the reported regex reader, especially on unparsed ProofWriter cases?
  3. Which claims about the proper name defect are intended to generalize beyond the released LogicReward prompt, and what evidence supports that scope?
  4. Can the authors clarify whether any downstream policy improvement is claimed? Appendix D.2 appears explicitly underpowered to answer that question.
  5. Please distinguish more explicitly between an assumption entailed by a generator program and an assumption that should count as an acceptable citation in a deployed verifier.
Reviewer_BSWVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper identifies a weakness in prover-based process verification, assumption laundering, through an audit of LogicReward[1]. A prover may be hacked by model-generated assumptions that are unsupported by the original problem. This paper intorduces provenance audit to detect assumption laundering, evaluates restrictions on supplementary assumptions, and conduct experiments to validate the effcectiveness.

[1] LogicReward: Incentivizing LLM Reasoning via Step-Wise Logical Supervision. ICLR 2026. https://openreview.net/forum?id=IRhYVOKFe0

Critical strengths

None.

Critical weaknesses
  1. This paper is very hard to follow. I remain confused about the main methods and insights of this paper despite hours of reading.
  2. This contribution is narrowly limited to a particular, unpopular design. To my best knowledge, allowing models to add assumptions when cited premises fall short is not a general approach, and the paper's empirical evidence primarily concerns LogicReward[1].

[1] LogicReward: Incentivizing LLM Reasoning via Step-Wise Logical Supervision. ICLR 2026. https://openreview.net/forum?id=IRhYVOKFe0

Questions for authors
  1. Can the methods and insights of this paper generalize to approaches that do not allow model-generated supplementary assumptions?
Reviewer_55t6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies a particular weakness in proof-based rewards for AI reasonings. The core idea is that, a proof checker may approve a step that depends on an unjustified assumption. The authors systematically measure this problem and test two restrictions on added assumptions: necessary assumptions only if a trusted reference supports it, and another, requiring the cited premises alone to support the conclusion. The restrictions, while improving some verification results, does not establish any practical training improvement. Moreover, the writing of the paper requires substantial improvements, given the writing is often too dense to follow and sometimes slightly less formal.

Critical strengths
  1. The paper targets a specific issue in proof based reward mechanisms in AI reasoning - a valid proof often does not establish that all its assumptions are justified. Therefore, it helps uncover that unrestricted assumptions can be challenging. Additionally, the authors show that solving this problem is also not very trivial given only changing the approval threshold does not solve it.

  2. Across almost 16K generated steps, the authors distinguish unjustified assumptions from inherited errors and translation errors. Some controlled prompt changes also show that examples with named individuals reduce translation errors that may cause conclusion approval about the wrong person.

Critical weaknesses
  1. The presentation is a major weakness for this paper, and the writing should be revised substantially. Several long paragraphs are extremely difficult to follow as they often combine definitions, exceptions, references to other sections in a dense paragraph. Overall, this writing structure makes the evidence difficult to assess. The authors should rewrite and reorganized the paper with normal sentence structure, and maybe develop one complete example before introducing the formal analysis.

  2. The evidence of the practical benefits seem to be quite limited. The training experiment does not establish any noticeable improvement, even with reference based training labels.

Questions for authors

If we train directly with premises-only rewards, would that improve the reasoning, or would it simply give preferences for shorter responses to pass the checker?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.