- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper evaluates 30 language models on terminating primitive-recursion rewrite systems, separating termination verdicts, mathematical proof validity, construction under a bespoke 'rule-derived' requirement, and recognition of supplied proofs. Its reference mathematics shows why nonnegative additive symbol weights fail on the duplicating recursor and gives successful alternatives. Across 4,160 sessions, it reports low construction success despite stronger supplied-proof recognition, plus a copy-removal control that improves mathematical proof success from 85/240 to 117/240. The main interpretive question is whether these outcomes measure inability to construct a valid proof or sensitivity to the study's additional scoring conventions.
Critical strengths
-
The benchmark has a concrete mathematical foundation. The additive-weight obstruction is established by a ground substitution rather than failed search, and successful reference proofs distinguish the inadequacy of a proof class from nontermination (Section 2; Appendix C). This makes specific observed proof errors falsifiable.
-
Separating verdict, proof construction, and supplied-proof assessment is useful. The broad model panel and paired copy-removal contrast provide evidence that a correct termination answer need not come with a valid proof. The +13.33 percentage-point construction improvement under copy removal, with reported interval [7.1, 19.6], is an informative result within the stated scoring regime (Table 1; Appendix H).
Critical weaknesses
-
The headline 'rule-derived' construction rate does not directly measure whether an answer constructs a valid requested proof. Appendix B.4/Table 9 (PDF p.17) removes credit even when a correct rule-derived proof accompanies a chosen comparison mentioned only hypothetically, rejected, or withdrawn. This scores the entire exposition's purity and can penalize successful final constructions. Moreover, selecting a verified precedence or polynomial is normally proof-witness construction rather than adding object-level assumptions; the open prompts and clarified menu do not express the same restriction equally clearly. The 2/240 and 0/480 figures therefore need final-proof/existential rescoring before supporting the stated construction-recognition gap.
-
Test 01 appears to score a stronger rewrite relation than the displayed task. The retained Lean prompt (Appendix I, PDF pp.44–47) defines Step with eight root-rule constructors and no context-closure constructor, while Appendix B's proof checks require decrease at arbitrary rewrite positions. Termination of the literal root relation and of its contextual closure are different obligations. Without an explicit mapping from the text shown to models to the scored relation, failures on this arm cannot cleanly establish the claimed representation-shift failure. Together these issues require more than a wording correction to the central benchmark conclusion.
Questions for authors
-
Please report valid-proof existence and final-endorsed-proof rates separately from whole-answer purity. How many failures arise solely from hypothetical, rejected, or withdrawn alternatives, and how much of the construction-recognition gap remains?
-
Was contextual closure explicitly included in the instructions shown to every Test 01 model? Please provide that text, or rescore against the literal Lean Step relation and separately report results for contextual rewriting.
-
Can independent termination experts audit a stratified sample of the semantic transcriptions and scoring decisions? This could materially change confidence in the extreme success rates, particularly given the 166/1,200 mathematical-score changes reported in Appendix B.5.
Author Response to Reviewer 7fqd
Thank you very much for your thorough review. The review reads the scoring rulebook and the prompt files closely, and it is right on several points. The revision reports each count that the review asks for, and we give them here.
First, the core of the paper in one line. The two rules do the same thing as the factorial of everyday arithmetic, n! (not to be confused with the factorial system in the paper). Four factorial becomes four times three factorial: the line gets longer, but the calls count down to 0, and there the run halts. The rule of the paper has the same countdown and copies one item at each call.
We are asking frontier LLMs: does the everyday factorial n! eventually produce an answer?
It is an extremely simple question, and the failure holds under every reading that the review asks for.
Weakness 1 and Question 1, the scoring. We agree that a checked ranking of the symbols or a checked polynomial is ordinary proof construction, and the paper gives it full mathematical credit. The rule-derived score is a second, separate property: the proof compares the counter at the recursive call and picks no ranking and no coefficients. Table 10 of the revision gives the readings that the review asks for. On the two-rule recursor (30 models, 8 runs each, 240 answers), a valid proof is present in 85 answers as a whole, in 86 with the main proof alone, and in 125 with any proof anywhere in the answer. A rule-derived proof is present in 2 answers as a whole, in 20 with the final proof alone, and in 41 anywhere. On the eight-rule system the valid-proof counts are 80, 65 and 99 of 480, and the rule-derived counts are 0, 0 and 6. Ignoring alternatives that are hypothetical, rejected, withdrawn or quoted adds no pass on either system. An answer that asserts a false proof beside a true one leaves the choice to the reader, and the whole-answer rule was fixed before the submission (entries of 30 June and 9 August 2026 in the scoring record). On a list of five valid methods, 452 of 480 answers judge the counter proof right, so the gap between writing that proof and recognizing it is 93 points under the paper's score and 77 points under the widest reading. We agree that the open prompt and the clarified list state the restriction with unequal clarity, and the paper reports the effect of the clearer wording.
Weakness 2 and Question 2, the eight-rule prompt. The answer is no: that prompt did not state the closure under contexts (its text is in Appendix I.1). Its Lean text defines Step by eight root constructors, and our scores used the rewrite relation that those rules generate, closed under contexts. The reviewer's reading is right. Section 2 and Appendix B.9 of the revision say so, and Table 12 scores both relations. The answers saw it too: 168 of the 480 state that the displayed relation has no rule for steps inside a term. Under the literal root relation, 85 to 177 of the 480 answers hold a valid proof, against 80 under the closure, and 109 to 122 answers give a wrong termination verdict under both relations. The root relation cannot test recursion: the recursive rule fires once at the root and stops, since no rule starts with the wrapper, and even the rule without its counter, F(x, y) → G(y, F(x, y)), stops after one root step. In factorial terms, once 4! has become 4 × 3!, a root step can no longer reach 3!, since the multiplication is at the root. Our main result does not depend on that prompt: 3,480 of the 4,160 sessions of the paper do not depend on that reading. The two-rule recursor and its control print plain rewrite rules in the TPDB format, and rewriting inside terms is the standard meaning of that format.
Question 3, the audit. No outside expert has audited a sample yet, and we welcome such an audit. The supplement holds every answer, its recorded reading with verbatim quotes, its score and the rule that decided it, so an expert can grade a sample blind and compare. For the final version we can have outside termination experts grade a sample stratified by task and by outcome. The 166 changed scores come from the move from earlier recorded judgments to the checker program, and 130 of them remove credit (Appendix B.5). Our own re-audit has already corrected one reader defect, and three control answers regained credit (Appendix B.6). We will post the reading, the counterexample and the deciding rule of any session that the reviewer names.
The bottom line. Set the eight-rule prompt aside, set the rule-derived score aside, and credit any valid proof anywhere in an answer. On the two-rule recursor, 115 of 240 answers still hold no valid proof, for two rules that a standard termination tool proves in 0.025 seconds. This is the everyday factorial, and it is the failure that the paper reports.