Auditing Chain-of-Thought Register Effects in Lean 4 Proof Generation
Abstract
Does the register of a chain-of-thought (CoT) prompt matter when a language model writes Lean 4 proofs? We evaluate five prompt conditions on 11 models and analyzed model–condition–problem cells from a 500-problem LeanWorkbook draw, and report two findings that appear only once proof scoring is itself audited. First, the scoring pipeline carried two defects: the verifier read Lean's sorry warning from the wrong stream, counting unfinished proofs as successes, and the extractor handed Lean text the model never offered as its proof. Screening every stored success removes that had closed with sorry, and re-verifying the cells whose proof text the repair changes recovers real ones (panel successes ). A further cells were truncated before storage and cannot be re-scored; we retain their old verdicts and bound the uncertainty this creates; the adversarial bound includes no effect. Second, on the re-scored panel NL CoT alone has a panel-average interval excluding (OR , HDI, a highest-density interval, ; raw panel rates ), while Lean pseudocode, symbolic math and tactic plans point above with intervals containing it. The average does not determine the effect for an untested model: heterogeneity persists ( –) and the predicted interval for a new model spans in every register. After re-scoring, neither self-rewriting nor an independent renderer establishes a pooled presentation effect (OR and ); with five samples per problem on two models, no register's paired gain over Direct excludes zero. We report a small average benefit for one register; these data do not establish a prompting rule for an untested model.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.