- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents FYAN, a multi-agent harness for document-level Lean formalization: a source is split into dependency-linked theorem units, and each unit passes through a reasoner (built on QED), a planner, and a formalizer (a rewritten fork of ARCHON), with proved units reused downstream and human/orchestrator advice allowed but unable to change protected artifacts. The central component is a semantic audit: an LLM extracts a quoted source formula tree, code slices both source and Lean statements into objects/hypotheses/claims, the LLM labels each paired slot (same / trivial equivalent / mutated / unaligned), and deterministic code checks polarity, coverage and omissions and assigns a minimum grade; statements below the threshold are repaired, and admitted "modified" statements require a Lean-checked bridge to a source-facing statement (with a compositional transfer theorem in App. E). Experiments with DeepSeek-V4.1-Flash report 86 vs. 69 FormalTCS formal proofs and 0.851 vs. 0.501 natural-language proof score against a general agent harness; higher recall than a single LLM judge on 300 ConsistencyCheck items (0.873 vs. 0.820 on original labels, 0.777 vs. 0.636 on author-revised labels); and ODENumLib, a 9,355-line Lean library for numerical ODE analysis.
Critical strengths
Semantic faithfulness of autoformalized statements is an important and under-addressed problem, especially at the document level where an unfaithful statement propagates to every downstream proof. The audit design is thoughtful: the source is extracted before any Lean is shown, omissions are tracked explicitly as unpaired slots, polarity-aware admissibility is decided by code rather than the LLM, and each rejection is localized to a specific hypothesis, claim, or scope, which makes repairs actionable. The analysis of BEq+'s limitations (App. C.1) is also a useful observation for the community.
Critical weaknesses
1. The experimental comparisons do not support the claims.
- The FormalTCS baseline is not specified. It is described only as "a general-purpose agent harness" / "general-purpose agent using the benchmark prompts verbatim" (Sec. 3.1, Table 6); the agent, its tools, and its iteration or compute budget are never given. FYAN's formalizer, by contrast, allows up to 50 proving iterations and 4 concurrent provers (Table 14), and its natural-language proofs come from a multi-agent reasoner. No tokens, cost, or wall-clock time are reported, so the +17 proofs and the T2NP gain cannot be attributed to the harness design rather than to more compute.
- There is no comparison with similar pipelines (e.g., ARCHON, which FYAN's formalizer is forked from, or other document-level/agentic formalization systems cited in App. A), and published FormalTCS/ProofNet results are explicitly not compared.
- There are no ablations of any component (reasoner, planner, curator, orchestrator, audit, bridges); the paper states that curator and orchestrator effects are not evaluated (Sec. 4, App. F.4). It is therefore unclear which parts of the system matter.
- The audit is compared against a single direct-judge call at low reasoning effort, while costing ~21x more per item (0.15 vs. 0.007 CNY; App. B.3); there is no compute-matched judge baseline (e.g., majority voting, higher effort, or a checklist prompt).
- The "human–AI" framing in the title and abstract is not supported. The human role is an optional advice channel: users can post guidance through a dashboard that enters the prover context as non-executable comments and cannot change statements or stage transitions (Sec. 2.1, App. F.4). No experiment reports whether humans intervened, how often, or with what effect — including ODENumLib — and the paper states that this supervision channel is not evaluated (App. F.4). As presented, FYAN is an automated pipeline with an optional comment interface.
2. The central contribution is not evaluated in research math setting
- The paper motivates document-level formalization of research papers, textbooks, and similar collections (Sec. 1), but the evaluation never covers that setting. FormalTCS, ConsistencyCheck (drawn from miniF2F competition problems and ProofNet undergraduate textbook exercises), and ProofNet# are all per-theorem benchmarks at competition, textbook, or course level; the only document-level evidence is ODENumLib, a single project built from course lecture notes, with no baseline to compare against.
- The semantic audit is presented as the gate that keeps unfaithful statements out of a document-level library, but it is evaluated only stand-alone as a classifier on ConsistencyCheck. It is not listed as part of the FormalTCS runs (Table 6), and ODENumLib was built before the audit existed, with statements accepted on compilation alone (App. D) — a fact not mentioned in the main text, which presents ODENumLib as a FYAN result (Sec. 3.4). App. D in fact shows 9 of 19 units underwent substantive statement changes, some weakening the result. The bridge mechanism is likewise never exercised in any experiment.
- Even as a classifier, the advantage depends on relabeling: under the original labels the audit has lower precision and accuracy (0.771 vs. 0.939; 0.807 vs. 0.883), and its recall edge largely disappears if failed runs are not counted as rejections (0.827 vs. 0.820; App. C.2). The "verified labels" were produced by Claude Opus 5.5 screening plus author confirmation, under a criterion the paper says the audit was designed to encode (Sec. 3.3), with no independent annotators or agreement statistics.
Questions for authors
Since the code is released, the following would substantially strengthen the paper:
- Fully specify the FormalTCS control (agent, tools, iteration/time budget) and report tokens, cost, and wall-clock time for both arms; ideally compare under matched compute.
- Compare against at least one comparable pipeline (e.g., ARCHON or another agentic formalization system) on the same model and budget.
- Ablate the main components (reasoner, planner, audit, curator, orchestrator) to show which contribute.
- Evaluate the audit end-to-end: run the full pipeline with and without the audit (and with bridges enabled) on a document-level task, and report how often statements are rejected, repaired, or bridged, and the effect on downstream faithfulness. Rebuilding ODENumLib with the audit enabled would be a natural experiment.
- Have the ConsistencyCheck relabeling done by independent annotators blind to both methods, with inter-annotator agreement, and add a compute-matched direct-judge baseline.
- Was any human guidance used in the reported experiments, and if so, how much and where?
- Show promise or applicability on research math level formalization problems.
5 comments
Response to W1.3 and Q3
Scope of the added experiments. Time and resource constraints during rebuttal prevented controlled component ablations on document-level tasks. We instead prioritized an ARCHON comparison using the same model and Lean environment (W2.1/Q7), stronger audit baselines (W1.4/Q5), and detailed component traces from the Sidorenko run.
Component roles in the added run. The Reasoner develops mathematical arguments, the Planner decomposes them into formal obligations before Lean search, and the semantic audit localizes statement discrepancies for repair. In Sidorenko, the Reasoner ran on two of three proof units whose source proofs were absent or partial, and was skipped for the complete source proof; the Planner produced one blueprint per unit.
Curator turns final Lean code and logs into notes on lemmas, tactics, decompositions, and failed attempts. Proving agents receive summaries and can retrieve relevant notes; referenced lemma names are checked against the live Lean environment before retrieval, and subsequent proofs face the same completion checks. Sidorenko produced 8 strategy notes for later projects, recorded as sketches. These traces document component use without establishing isolated gains.
Response to W1.4 and Q5
We added 23-vote majority, five-round LeanScorer, and independent expert review. Within each model, all methods receive the same 300 source–Lean pairs. The direct judge makes one assessment; voting repeats it in independent calls with other answers withheld, targeting a cost comparable to the audit. LeanScorer is our reimplementation of its published rubric, scoring assumptions and conclusions at threshold 0.6; five rounds average five independent scores. Costs include all calls and invalid-output reruns.
Counts are missed inconsistent statements / correct statements rejected, under original and author-revised labels. DeepSeek uses 300 pairs; every GLM-5.3 method uses the common 297 pairs with valid FYAN verdicts. Costs cover all 300 DeepSeek pairs.
| Method | DeepSeek, original | DeepSeek, revised | GLM-5.3, original | GLM-5.3, revised | Cost (CNY, DeepSeek) |
|---|---|---|---|---|---|
| Single direct judgment | 24 / 10 | 71 / 1 | 56 / 3 | 109 / 0 | 1.2 |
| Majority voting, 23 judgments | 24 / 11 | 69 / 0 | 64 / 2 | 118 / 0 | 21.9 |
| LeanScorer, five rounds | 16 / 29 | 47 / 4 | 36 / 15 | 79 / 2 | 25.9 |
| FYAN | 10 / 41 | 34 / 9 | 20 / 52 | 41 / 17 | 31.0 |
FYAN has the highest accuracy under the revised labels: 85.7%/80.5% (DeepSeek/GLM), versus 76.0%/63.3% for direct judgment, 77.0%/60.3% for voting, and 83.0%/72.7% for LeanScorer. Gains over direct judgment and voting remain significant after multiple-comparison correction. Voting changes DeepSeek accuracy little and lowers it on GLM. Against LeanScorer, the uncorrected 95% interval includes zero on DeepSeek and excludes zero on GLM; the GLM difference is not significant after Holm correction over 16 comparisons (p = 0.128).
Cost and practical value. The audit costs CNY 31 versus 26 for five-round LeanScorer, a CNY 5 difference across 300 pairs. A further audit after repair costs about 0.1 CNY; a missed discrepancy can lead to proof work (median FT2FP cost about 4 CNY) and downstream rework. At preset thresholds, FYAN misses the fewest inconsistencies in all 16 comparisons in Appendix B.5; uncorrected 95% paired intervals on missed statements exclude zero in 14. For the assumed cost ρ × misses + false rejections, FYAN is favored throughout when ρ exceeds about 2.3; under revised labels, the threshold is below 0.4. Test-label threshold sweeps are separate posthoc analyses, with optima varying by setting.
Audit feedback for repair. Localized findings identify the object, assumption, conclusion, or conversion to repair; accepted statements stay protected for proving and downstream reuse. An offline OR of the historical direct-judge and audit rejections reduces revised-label misses from 46 to 32, with false rejections unchanged at 10 (original-label misses: 19 to 6). Using the direct judge as an early filter would leave its rejections without local audit findings; this execution strategy remains untested. FYAN retains the full audit to guide repair. The independent review and invalid-verdict sensitivity analysis are detailed in W2.3 and Appendix C.4.
Response to W1.5 and Q6
The benchmarks and ODENumLib used no interactive proof guidance. Their bounded tasks and supplied context, including numerical-analysis lecture notes for ODENumLib, supported automatic evaluation.
The human role includes both mathematical guidance and specification decisions. Users can supply lemmas, insights, conventions, and proof routes to reduce unproductive search and reflect their goals. Guidance remains advice; reviewed statements stay protected, and acceptance requires semantic and Lean checks. When an audit cannot certify a statement, the workflow stops and records its findings. A human can correct the specification, amend the target, or authorize continuation with the statement flagged; this decision is recorded separately from proof hints and code edits. Sidorenko includes such a continuation decision (W2.1).
Appendix F.4 explains this role, and the revision states each experiment's human-intervention conditions.
Response to W2.1 and Q7
We added a connected research-paper formalization and a comparable pipeline. FormalTCS already contains 143 research-derived theorems from 2025–2026 STOC, FOCS, SODA, and COLT papers (benchmark paper). The new experiment starts directly from the Sidorenko graph-homomorphism paper.
Targets were fixed before either run: edge-removal inequality, recursive homomorphism count, leaf redistribution, Theorem 1, the three parts of Corollary 2, and optionally the weighted AM–GM proof. Both systems used the same model and Lean environment.
| Method | Proof/build result | Setup | Recorded time |
|---|---|---|---|
| ARCHON v0.2.0 | All formal statements proved; build passed | No semantic audit | Later proving rounds, iterations 5–7: 2 h 30 min 50 s |
| FYAN (preliminary run) | All three proof units proved, including Theorem 1; build passed; no sorry, standard axioms only | Statement-first: statements frozen before proving | Proving after statements were frozen: 80 min |
Efficient proof construction. FYAN wrote shared definitions and statements once, froze them in one module, and completed the three proof units in parallel. The resulting 3,787-line development has no duplicated definitions. ARCHON also performed proof work before iterations 5–7; these phase timings do not establish an end-to-end speedup.
The statement phase involved four audit rounds and three repairs. All three proof units passed the final Lean checks. The recorded audit status was preserved: one certified source unit and three flagged units continued under human authorization (Appendix D.2).
Response to W2.2 and Q4
ODENumLib predates semantic auditing. The revision makes this explicit in §3.4 and Appendix D. Its 9,355 lines and 20 completed units demonstrate project construction and dependency reuse; completion required compilation without sorry or new axioms in the original environment. Appendix D records substantive changes in nine of 19 units with retained initial statements, including repairs to drafts and weakened source claims. This motivates reviewing meaning before proving and protecting reviewed statements.
FormalTCS NT2FT used an audit acceptance gate, now specified in revised Table 6. NT2FT translates statements, FT2FP proves benchmark-supplied Lean statements, and T2NP generates natural-language proofs.
Within the rebuttal's time and resource limits, we prioritized Sidorenko to evaluate document-level coordination and research mathematics together (W2.1). An audited ODENumLib rebuild remains a follow-up. No bridge was triggered in the recorded runs. A bridge proves S1 → S0 from a modified statement to a source-checked original claim; amending the source or proving a corrected target does not establish this implication to an unchanged original.
Source extraction was independently checked: a mathematics PhD student blindly reviewed 50 random source trees, seeing only source and tree, and judged 47 correct, 3 incorrect, none unresolved. The expert also recognized useful ambiguity notes, including unspecified zero-based versus one-based indexing. Further details of the review protocol, error analysis, and examples appear in our response to Reviewer fAD7, W2/Q2, under “Reading the original statement: 47/50 judged correct.”
We thank the reviewer for recognizing FYAN's actionable semantic feedback and BEq⁺ analysis. The revision adds stronger controls, independent human review, and a research-paper case study, and clarifies the configurations and limits of each experiment.
Response to W1.1 and Q1
FYAN proves 17 more FormalTCS theorems at approximately twice the recorded FT2FP cost. With the same model, it proves 86/143 versus 69/143, with 22 solved only by FYAN versus 5 only by the control (exact paired test, p = 0.0015). The natural-language proof score rises from 0.501 to 0.851.
Actual configurations and resources. Table 14 in the original submission (Table 24 in the revision) lists defaults. The reported runs used at most five outer iterations, with proving subagents and automatic supervision disabled. Both arms shared the evaluator loop, model, Lean environment, graders, and at most one final scored output per item. Below are the settings and recorded resources for formal proof construction (FT2FP), natural-language proof generation (T2NP), and statement translation (NT2FT), including internal calls, retries, and failures.
Costs use the same DeepSeek Flash off-peak price (per 1M tokens: cache hit 0.02, miss 1, output 4 CNY). Missing usage records make FYAN's figures lower bounds.
What the extra cost buys. FT2FP caps were matched, but spending differed: 628 versus 310 CNY, or 7.3 versus 4.5 CNY per verified proof and about 19 CNY per additional proof. Within the nominal six-hour limit per run, the control reached its turn cap on only 3 items and submitted 50 of its 74 failures with
sorrystill present. FYAN continued unfinished proofs through Lean checks and repairs. These traces suggest that completion checks and continued work on unresolved obligations contribute to the gain.Response to W1.2 and Q2
Our Formalizer began as a fork of ARCHON v0.2.0 and has since been substantially rewritten (Appendix F.2–F.3). Its document-level controls include:
sorryor disallowed axioms, intact reviewed statements, and required Lean bridges.FYAN connects proof construction to source comparison, repair, and protected reuse across a document. W2.1/Q7 reports the added ARCHON comparison on the same model and Lean environment, with explicit timing scopes.