acceptodds
Under review as a conference paper at ICLR 2027

Auditing Chain-of-Thought Register Effects in Lean 4 Proof Generation

Abstract

Does the register of a chain-of-thought (CoT) prompt matter when a language model writes Lean 4 proofs? We evaluate five prompt conditions on 11 models and analyzed model–condition–problem cells from a 500-problem LeanWorkbook draw, and report two findings that appear only once proof scoring is itself audited. First, the scoring pipeline carried two defects: the verifier read Lean's sorry warning from the wrong stream, counting unfinished proofs as successes, and the extractor handed Lean text the model never offered as its proof. Screening every stored success removes that had closed with sorry, and re-verifying the cells whose proof text the repair changes recovers real ones (panel successes ). A further cells were truncated before storage and cannot be re-scored; we retain their old verdicts and bound the uncertainty this creates; the adversarial bound includes no effect. Second, on the re-scored panel NL CoT alone has a panel-average interval excluding (OR , HDI, a highest-density interval, ; raw panel rates ), while Lean pseudocode, symbolic math and tactic plans point above with intervals containing it. The average does not determine the effect for an untested model: heterogeneity persists ( –) and the predicted interval for a new model spans in every register. After re-scoring, neither self-rewriting nor an independent renderer establishes a pooled presentation effect (OR and ); with five samples per problem on two models, no register's paired gain over Direct excludes zero. We report a small average benefit for one register; these data do not establish a prompting rule for an untested model.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.