acceptodds
Under review as a conference paper at ICLR 2027

Fyan: A Human–AI Harness with Semantic Auditing for Document-Level Formalization

Abstract

We present Fyan, a human–AI harness for document-level mathematical formalization. Rather than treating theorems in isolation, Fyan coordinates an end-to-end workflow spanning specification, proof planning, logical review, Lean proof construction, knowledge curation, and validation, with support for independent supervision and human guidance. A central component is evidence-grounded semantic auditing, which assesses whether formal statements faithfully preserve their informal specifications. A language model constructs structured evidence over local correspondences, omissions, scope, and logical relations, while a deterministic validator checks this evidence and produces reproducible judgments. When a substantive but admissible deviation is accepted, Fyan requires an explicit proof-transfer obligation connecting the formal statement back to a source-facing interpretation. With the same model (DeepSeek-V4.1-Flash) in every stage, Fyan proves 86 of 143 FormalTCS theorems under a strict Lean check, against 69 for a general agent harness, and raises the natural-language proof score from 0.501 to 0.851. On ConsistencyCheck, its semantic audit catches more inconsistent statements than a direct LLM judge, both on labels verified against the source (recall 0.777 vs. 0.636) and on the original labels (0.873 vs. 0.820), and localizes each mismatch it reports to a specific hypothesis, conclusion, or scope. Fyan also built ODENumLib, a 9,355-line Lean library for the numerical analysis of ordinary differential equations.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
formal mathematics, human–AI collaboration, Lean, semantic auditing, theorem proving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FTXfOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents FYAN, a multi-agent harness for document-level Lean formalization: a source is split into dependency-linked theorem units, and each unit passes through a reasoner (built on QED), a planner, and a formalizer (a rewritten fork of ARCHON), with proved units reused downstream and human/orchestrator advice allowed but unable to change protected artifacts. The central component is a semantic audit: an LLM extracts a quoted source formula tree, code slices both source and Lean statements into objects/hypotheses/claims, the LLM labels each paired slot (same / trivial equivalent / mutated / unaligned), and deterministic code checks polarity, coverage and omissions and assigns a minimum grade; statements below the threshold are repaired, and admitted "modified" statements require a Lean-checked bridge to a source-facing statement (with a compositional transfer theorem in App. E). Experiments with DeepSeek-V4.1-Flash report 86 vs. 69 FormalTCS formal proofs and 0.851 vs. 0.501 natural-language proof score against a general agent harness; higher recall than a single LLM judge on 300 ConsistencyCheck items (0.873 vs. 0.820 on original labels, 0.777 vs. 0.636 on author-revised labels); and ODENumLib, a 9,355-line Lean library for numerical ODE analysis.

Critical strengths

Semantic faithfulness of autoformalized statements is an important and under-addressed problem, especially at the document level where an unfaithful statement propagates to every downstream proof. The audit design is thoughtful: the source is extracted before any Lean is shown, omissions are tracked explicitly as unpaired slots, polarity-aware admissibility is decided by code rather than the LLM, and each rejection is localized to a specific hypothesis, claim, or scope, which makes repairs actionable. The analysis of BEq+'s limitations (App. C.1) is also a useful observation for the community.

Critical weaknesses

1. The experimental comparisons do not support the claims.

  • The FormalTCS baseline is not specified. It is described only as "a general-purpose agent harness" / "general-purpose agent using the benchmark prompts verbatim" (Sec. 3.1, Table 6); the agent, its tools, and its iteration or compute budget are never given. FYAN's formalizer, by contrast, allows up to 50 proving iterations and 4 concurrent provers (Table 14), and its natural-language proofs come from a multi-agent reasoner. No tokens, cost, or wall-clock time are reported, so the +17 proofs and the T2NP gain cannot be attributed to the harness design rather than to more compute.
  • There is no comparison with similar pipelines (e.g., ARCHON, which FYAN's formalizer is forked from, or other document-level/agentic formalization systems cited in App. A), and published FormalTCS/ProofNet results are explicitly not compared.
  • There are no ablations of any component (reasoner, planner, curator, orchestrator, audit, bridges); the paper states that curator and orchestrator effects are not evaluated (Sec. 4, App. F.4). It is therefore unclear which parts of the system matter.
  • The audit is compared against a single direct-judge call at low reasoning effort, while costing ~21x more per item (0.15 vs. 0.007 CNY; App. B.3); there is no compute-matched judge baseline (e.g., majority voting, higher effort, or a checklist prompt).
  • The "human–AI" framing in the title and abstract is not supported. The human role is an optional advice channel: users can post guidance through a dashboard that enters the prover context as non-executable comments and cannot change statements or stage transitions (Sec. 2.1, App. F.4). No experiment reports whether humans intervened, how often, or with what effect — including ODENumLib — and the paper states that this supervision channel is not evaluated (App. F.4). As presented, FYAN is an automated pipeline with an optional comment interface.

2. The central contribution is not evaluated in research math setting

  • The paper motivates document-level formalization of research papers, textbooks, and similar collections (Sec. 1), but the evaluation never covers that setting. FormalTCS, ConsistencyCheck (drawn from miniF2F competition problems and ProofNet undergraduate textbook exercises), and ProofNet# are all per-theorem benchmarks at competition, textbook, or course level; the only document-level evidence is ODENumLib, a single project built from course lecture notes, with no baseline to compare against.
  • The semantic audit is presented as the gate that keeps unfaithful statements out of a document-level library, but it is evaluated only stand-alone as a classifier on ConsistencyCheck. It is not listed as part of the FormalTCS runs (Table 6), and ODENumLib was built before the audit existed, with statements accepted on compilation alone (App. D) — a fact not mentioned in the main text, which presents ODENumLib as a FYAN result (Sec. 3.4). App. D in fact shows 9 of 19 units underwent substantive statement changes, some weakening the result. The bridge mechanism is likewise never exercised in any experiment.
  • Even as a classifier, the advantage depends on relabeling: under the original labels the audit has lower precision and accuracy (0.771 vs. 0.939; 0.807 vs. 0.883), and its recall edge largely disappears if failed runs are not counted as rejections (0.827 vs. 0.820; App. C.2). The "verified labels" were produced by Claude Opus 5.5 screening plus author confirmation, under a criterion the paper says the audit was designed to encode (Sec. 3.3), with no independent annotators or agreement statistics.
Questions for authors

Since the code is released, the following would substantially strengthen the paper:

  1. Fully specify the FormalTCS control (agent, tools, iteration/time budget) and report tokens, cost, and wall-clock time for both arms; ideally compare under matched compute.
  2. Compare against at least one comparable pipeline (e.g., ARCHON or another agentic formalization system) on the same model and budget.
  3. Ablate the main components (reasoner, planner, audit, curator, orchestrator) to show which contribute.
  4. Evaluate the audit end-to-end: run the full pipeline with and without the audit (and with bridges enabled) on a document-level task, and report how often statements are rejected, repaired, or bridged, and the effect on downstream faithfulness. Rebuilding ODENumLib with the audit enabled would be a natural experiment.
  5. Have the ConsistencyCheck relabeling done by independent annotators blind to both methods, with inter-annotator agreement, and add a compute-matched direct-judge baseline.
  6. Was any human guidance used in the reported experiments, and if so, how much and where?
  7. Show promise or applicability on research math level formalization problems.
5 comments

We thank the reviewer for recognizing FYAN's actionable semantic feedback and BEq⁺ analysis. The revision adds stronger controls, independent human review, and a research-paper case study, and clarifies the configurations and limits of each experiment.

Response to W1.1 and Q1

FYAN proves 17 more FormalTCS theorems at approximately twice the recorded FT2FP cost. With the same model, it proves 86/143 versus 69/143, with 22 solved only by FYAN versus 5 only by the control (exact paired test, p = 0.0015). The natural-language proof score rises from 0.501 to 0.851.

Actual configurations and resources. Table 14 in the original submission (Table 24 in the revision) lists defaults. The reported runs used at most five outer iterations, with proving subagents and automatic supervision disabled. Both arms shared the evaluator loop, model, Lean environment, graders, and at most one final scored output per item. Below are the settings and recorded resources for formal proof construction (FT2FP), natural-language proof generation (T2NP), and statement translation (NT2FT), including internal calls, retries, and failures.

Run configuration and resourcesGeneral-purpose controlFYAN
Agent/version; tools and promptsDeepSeek Harness 0.1.6-alpha.1; stock file and shell tools, no network; benchmark prompts verbatim (one sentence added to NT2FT/T2NP stating that no network is available)FYAN as submitted; Formalizer with Lean tools
Model and samplingDeepSeek-V4.1-Flash, T = 0.6, top-p 0.9DeepSeek-V4.1-Flash
Outer loopShared evaluator: nominally 3 turns per run; interrupted runs resumedSame evaluator; within a turn at most 5 Formalizer iterations
Time/budget caps and stopping rules7,200 s per FT2FP turn; 1,800 s per T2NP turn; 2,400 s per NT2FT turn; at most one final scored outputSame FT2FP and T2NP caps; NT2FT in three batches at 3,600 / 2,400 / 2,400 s per turn; strict completion checks
Proving subagents and automatic supervisionNot applicableDisabled
Components, audit version, and bridge policyNoneNT2FT used an audit acceptance gate; no bridge was triggered in these runs
FT2FP tokens / cost / time1.10 B input (1.02 B cached), 52.8 M output; 310 CNY; 60.3 h span5.95 B input (5.83 B cached), 98.6 M output; 628 CNY; 37.4 h active
T2NP tokens / cost / timeNot recorded for the reported run3.77 B input, 105.7 M output; 585 CNY; 59.1 h active
NT2FT tokens / cost / time0.16 B input, 8.8 M output; 47 CNY; 6.1 h1.65 B input, 69.8 M output; 362 CNY; three batches, ≥ 22.4 h

Costs use the same DeepSeek Flash off-peak price (per 1M tokens: cache hit 0.02, miss 1, output 4 CNY). Missing usage records make FYAN's figures lower bounds.

What the extra cost buys. FT2FP caps were matched, but spending differed: 628 versus 310 CNY, or 7.3 versus 4.5 CNY per verified proof and about 19 CNY per additional proof. Within the nominal six-hour limit per run, the control reached its turn cap on only 3 items and submitted 50 of its 74 failures with sorry still present. FYAN continued unfinished proofs through Lean checks and repairs. These traces suggest that completion checks and continued work on unresolved obligations contribute to the gain.

Response to W1.2 and Q2

Our Formalizer began as a fork of ARCHON v0.2.0 and has since been substantially rewritten (Appendix F.2–F.3). Its document-level controls include:

  • Execution controlled by code. A Pi-based runtime advances stages from persisted artifacts and validation results, keeping unfinished work active for repair.
  • Protected statements. Reviewed theorem signatures and source inputs stay protected during proving; changes invalidate approval.
  • Strict completion checks. Downstream reuse requires a successful build, no sorry or disallowed axioms, intact reviewed statements, and required Lean bridges.

FYAN connects proof construction to source comparison, repair, and protected reuse across a document. W2.1/Q7 reports the added ARCHON comparison on the same model and Lean environment, with explicit timing scopes.

Response to W1.3 and Q3

Scope of the added experiments. Time and resource constraints during rebuttal prevented controlled component ablations on document-level tasks. We instead prioritized an ARCHON comparison using the same model and Lean environment (W2.1/Q7), stronger audit baselines (W1.4/Q5), and detailed component traces from the Sidorenko run.

Component roles in the added run. The Reasoner develops mathematical arguments, the Planner decomposes them into formal obligations before Lean search, and the semantic audit localizes statement discrepancies for repair. In Sidorenko, the Reasoner ran on two of three proof units whose source proofs were absent or partial, and was skipped for the complete source proof; the Planner produced one blueprint per unit.

Curator turns final Lean code and logs into notes on lemmas, tactics, decompositions, and failed attempts. Proving agents receive summaries and can retrieve relevant notes; referenced lemma names are checked against the live Lean environment before retrieval, and subsequent proofs face the same completion checks. Sidorenko produced 8 strategy notes for later projects, recorded as sketches. These traces document component use without establishing isolated gains.

Response to W1.4 and Q5

We added 23-vote majority, five-round LeanScorer, and independent expert review. Within each model, all methods receive the same 300 source–Lean pairs. The direct judge makes one assessment; voting repeats it in independent calls with other answers withheld, targeting a cost comparable to the audit. LeanScorer is our reimplementation of its published rubric, scoring assumptions and conclusions at threshold 0.6; five rounds average five independent scores. Costs include all calls and invalid-output reruns.

Counts are missed inconsistent statements / correct statements rejected, under original and author-revised labels. DeepSeek uses 300 pairs; every GLM-5.3 method uses the common 297 pairs with valid FYAN verdicts. Costs cover all 300 DeepSeek pairs.

MethodDeepSeek, originalDeepSeek, revisedGLM-5.3, originalGLM-5.3, revisedCost (CNY, DeepSeek)
Single direct judgment24 / 1071 / 156 / 3109 / 01.2
Majority voting, 23 judgments24 / 1169 / 064 / 2118 / 021.9
LeanScorer, five rounds16 / 2947 / 436 / 1579 / 225.9
FYAN10 / 4134 / 920 / 5241 / 1731.0

FYAN has the highest accuracy under the revised labels: 85.7%/80.5% (DeepSeek/GLM), versus 76.0%/63.3% for direct judgment, 77.0%/60.3% for voting, and 83.0%/72.7% for LeanScorer. Gains over direct judgment and voting remain significant after multiple-comparison correction. Voting changes DeepSeek accuracy little and lowers it on GLM. Against LeanScorer, the uncorrected 95% interval includes zero on DeepSeek and excludes zero on GLM; the GLM difference is not significant after Holm correction over 16 comparisons (p = 0.128).

Cost and practical value. The audit costs CNY 31 versus 26 for five-round LeanScorer, a CNY 5 difference across 300 pairs. A further audit after repair costs about 0.1 CNY; a missed discrepancy can lead to proof work (median FT2FP cost about 4 CNY) and downstream rework. At preset thresholds, FYAN misses the fewest inconsistencies in all 16 comparisons in Appendix B.5; uncorrected 95% paired intervals on missed statements exclude zero in 14. For the assumed cost ρ × misses + false rejections, FYAN is favored throughout when ρ exceeds about 2.3; under revised labels, the threshold is below 0.4. Test-label threshold sweeps are separate posthoc analyses, with optima varying by setting.

Audit feedback for repair. Localized findings identify the object, assumption, conclusion, or conversion to repair; accepted statements stay protected for proving and downstream reuse. An offline OR of the historical direct-judge and audit rejections reduces revised-label misses from 46 to 32, with false rejections unchanged at 10 (original-label misses: 19 to 6). Using the direct judge as an early filter would leave its rejections without local audit findings; this execution strategy remains untested. FYAN retains the full audit to guide repair. The independent review and invalid-verdict sensitivity analysis are detailed in W2.3 and Appendix C.4.

Response to W1.5 and Q6

The benchmarks and ODENumLib used no interactive proof guidance. Their bounded tasks and supplied context, including numerical-analysis lecture notes for ODENumLib, supported automatic evaluation.

The human role includes both mathematical guidance and specification decisions. Users can supply lemmas, insights, conventions, and proof routes to reduce unproductive search and reflect their goals. Guidance remains advice; reviewed statements stay protected, and acceptance requires semantic and Lean checks. When an audit cannot certify a statement, the workflow stops and records its findings. A human can correct the specification, amend the target, or authorize continuation with the statement flagged; this decision is recorded separately from proof hints and code edits. Sidorenko includes such a continuation decision (W2.1).

Appendix F.4 explains this role, and the revision states each experiment's human-intervention conditions.

Response to W2.1 and Q7

We added a connected research-paper formalization and a comparable pipeline. FormalTCS already contains 143 research-derived theorems from 2025–2026 STOC, FOCS, SODA, and COLT papers (benchmark paper). The new experiment starts directly from the Sidorenko graph-homomorphism paper.

Targets were fixed before either run: edge-removal inequality, recursive homomorphism count, leaf redistribution, Theorem 1, the three parts of Corollary 2, and optionally the weighted AM–GM proof. Both systems used the same model and Lean environment.

MethodProof/build resultSetupRecorded time
ARCHON v0.2.0All formal statements proved; build passedNo semantic auditLater proving rounds, iterations 5–7: 2 h 30 min 50 s
FYAN (preliminary run)All three proof units proved, including Theorem 1; build passed; no sorry, standard axioms onlyStatement-first: statements frozen before provingProving after statements were frozen: 80 min

Efficient proof construction. FYAN wrote shared definitions and statements once, froze them in one module, and completed the three proof units in parallel. The resulting 3,787-line development has no duplicated definitions. ARCHON also performed proof work before iterations 5–7; these phase timings do not establish an end-to-end speedup.

The statement phase involved four audit rounds and three repairs. All three proof units passed the final Lean checks. The recorded audit status was preserved: one certified source unit and three flagged units continued under human authorization (Appendix D.2).

Response to W2.2 and Q4

ODENumLib predates semantic auditing. The revision makes this explicit in §3.4 and Appendix D. Its 9,355 lines and 20 completed units demonstrate project construction and dependency reuse; completion required compilation without sorry or new axioms in the original environment. Appendix D records substantive changes in nine of 19 units with retained initial statements, including repairs to drafts and weakened source claims. This motivates reviewing meaning before proving and protecting reviewed statements.

FormalTCS NT2FT used an audit acceptance gate, now specified in revised Table 6. NT2FT translates statements, FT2FP proves benchmark-supplied Lean statements, and T2NP generates natural-language proofs.

Within the rebuttal's time and resource limits, we prioritized Sidorenko to evaluate document-level coordination and research mathematics together (W2.1). An audited ODENumLib rebuild remains a follow-up. No bridge was triggered in the recorded runs. A bridge proves S1 → S0 from a modified statement to a source-checked original claim; amending the source or proving a corrected target does not establish this implication to an unchanged original.

Source extraction was independently checked: a mathematics PhD student blindly reviewed 50 random source trees, seeing only source and tree, and judged 47 correct, 3 incorrect, none unresolved. The expert also recognized useful ambiguity notes, including unspecified zero-based versus one-based indexing. Further details of the review protocol, error analysis, and examples appear in our response to Reviewer fAD7, W2/Q2, under “Reading the original statement: 47/50 judged correct.”

Reviewer_QCjsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FYAN, a human–AI harness for document-level mathematical formalization. Its workflow spans natural-language proof generation, decomposition into proof steps and auxiliary lemmas, Lean statement generation, semantic auditing, Lean proof construction and repair, and completion validation. A central component is evidence-grounded semantic auditing, which assesses whether generated formal statements faithfully preserve their informal source specifications. The paper evaluates FYAN’s semantic auditing on ConsistencyCheck and its statement translation, natural-language proof generation, and formal proof construction capabilities on FormalTCS.

Critical strengths
  1. The paper addresses an important problem: transforming PDF, TeX, and Markdown sources into a dependency-aware Lean project.
  2. The experimental results are promising, with the number of theorems passing strict Lean verification on FormalTCS increasing from 69/143 to 86/143.
Critical weaknesses
  1. Although the paper aims to transform PDF, TeX, and Markdown sources into a dependency-aware Lean project, it lacks a quantitative end-to-end evaluation of how accurately the resulting project preserves the source statements and their dependencies.
  2. The reliability of the re-annotation in Section 3.3 requires further justification. Although the authors confirmed the model-generated classifications, were independent human experts involved?
  3. The evaluation uses only one underlying model. Experiments with additional models would help establish whether the observed improvements generalize across models.
  4. The paper lacks comparisons with closely related approaches, particularly the LeanScorer semantic evaluation framework introduced in Mathesis [1]. [1] Mathesis: Towards Formal Theorem Proving from Natural Languages
Questions for authors

Please refer to the weaknesses discussed above.

5 comments

We thank the reviewer for the thoughtful feedback and for recognizing the importance of document-level formalization. We provide new independent human-review evidence and supplementary comparisons below to address your concerns.

Response to W1

Thank you for asking us to evaluate how faithfully the complete project preserves both the statements and their dependencies. This captures FYAN's main use case: formalizing related theorems together, so that reviewed and proved results can support later proofs. We want the evaluation to show both whether the original mathematics is preserved and how earlier results are reused.

Clarifying ODENumLib. ODENumLib was generated from numerical-analysis lecture notes and contains 9,355 lines of Lean, with 20 completed theorem units, 34 theorems, and 118 supporting lemmas (§3.4). It covers linked results such as discrete Grönwall inequalities, error bounds for θ-methods, and the consistency conditions of linear multistep methods. It was built before semantic auditing was added: completion required compilation and proofs without sorry or new axioms in the original environment. The revision states this setting in the main text.

Appendix D complements these project-scale results with a unit-level record of statement changes. Nine of the 19 units with retained initial Lean statements underwent substantive revisions, including repairs to initial drafts and changes that weakened the source claim. This record helps distinguish useful corrections from deviations requiring further review. These observations motivate the current workflow's semantic audit before proof search and Lean-checked proof transfer for admitted substantive changes. The revision count describes changes to initial Lean statements, not fidelity of the final statements to the original text.

How dependencies support formalization. FYAN's document-level dependency graph records which results each unit relies on. In the statement-first configuration used for the new experiment, all statements and shared definitions are written and reviewed first, then frozen in one module; each unit then proves its own statement while importing the frozen statements of its prerequisites. Units can therefore be proved in parallel, and earlier results are available for reuse without repeated proof work. During proof search, the agent may discover a more efficient route, use an existing library result, or reuse a prerequisite through an intermediate lemma. A valid final proof may therefore use only a subset of the recorded prerequisites. Module imports establish availability; references in the final proofs establish actual reuse.

Evaluating the statement-first workflow. To evaluate statement preservation and dependencies with semantic auditing included, we added an experiment on the Sidorenko graph-homomorphism paper, starting from the paper itself. The checklist of target statements was fixed before any run: the edge-removal inequality, the recursive count, leaf redistribution, Theorem 1 (the main result), the three parts of Corollary 2, and optionally the weighted AM–GM proof. The agents receive the original document, without this checklist or reference Lean code.

Preliminary result. With the same model and Lean environment as ARCHON v0.2.0, FYAN completed the proofs of all three proof units, including Theorem 1; the project builds with no sorry and standard axioms only. The shared definitions and all statements were written once, in one module, before any proof. The final development has 3,787 lines with no duplicated definitions, and parallel proving completed in 80 minutes after statement freezing.

All three proof units passed the final Lean checks. The recorded audit status was preserved: one certified source unit and three flagged units continued under human authorization (Appendix D.2).

Actual reuse. The run's dependency graph has seven edges; six involve the shared statement or definition units, and one links two proof units: the refined tree ordering depends on tree-count extremality. The final proof of the refined ordering uses both components of that upstream theorem (tree_count_extremality.proved.1 and .proved.2) through local lemmas: zero direct uses, one indirect use, and no unused prerequisite. The revision's Sidorenko appendix lists these references.

Response to W2

Thank you for highlighting the need for expert validation. An independent mathematics PhD student reviewed 86 source–Lean pairs. We distinguish the original benchmark labels from our stricter revised labels reported in the paper. The sample includes all 56 pairs whose labels we revised and 15 randomly selected unchanged pairs from each class (consistent and inconsistent).

The review had two stages. In the initial review, with both label sets, method predictions, and model explanations withheld, the expert agreed with our revised labels on 54 of 80 decisive pairs (67.5%). During final adjudication, 23 judgments changed. The table compares the expert's final judgments with our revised labels.

Benchmark label → our revised labelPairsExpert agrees with our revised labelExpert recommends correctionNeed context
Consistent → Inconsistent5639710
Consistent → Consistent151500
Inconsistent → Inconsistent151500
Total8669710

After adjudication, 69 of the 76 decisive cases agree with our revised labels (90.8%). The expert recommends reversing seven of our 56 changes; ten cases require further source context.

Why we revised the benchmark labels. Our re-annotation prompt checks even small changes in definitions, domains, assumptions, quantifier scope, and conclusions. We distinguish a direct restatement from a mathematically equivalent reformulation requiring an argument. This keeps changes and their justifications explicit before a theorem is reused.

Example 1: a benchmark “consistent” label correctly revised to “inconsistent.” The source asks us to prove that two permutations are both identities when their composition is the identity and they “disturb no common element.” This last condition means that no element is moved by both:

Source: for every a, σ(a) = a OR τ(a) = a
Lean:   for every a, σ(a) = a IFF τ(a) ≠ a

The source allows both permutations to fix an element; Lean requires exactly one to move it. For example, setting both permutations to the identity on {1, 2} satisfies all source assumptions but violates Lean's condition. Lean excludes a valid source case, and the expert confirmed our revised inconsistent label.

Example 2: an overly strict revision corrected by the expert. The source says that 40 calories are 2%2\% of a person's daily requirement xx and asks us to show that x=2000x=2000:

Source: (2/100) × x = 40  →  x = 2000
Lean:   x / 50 = 40      →  x = 2000

Since 2/100=1/502/100=1/50, the premises differ only by immediate arithmetic simplification; the conclusion is unchanged. The expert judged this directly acceptable and recommended restoring the benchmark's consistent label, correcting our revised inconsistent label.

Thus, independent expert review supports the labeling standard while also identifying corrections. The revision lists the expert's corrections and unresolved cases (Appendix C.4) and keeps results under the original benchmark labels.

Response to W3

Thank you for asking whether the improvements carry over to other underlying models. We repeated the semantic-checking comparison with a second model, GLM-5.3, on the same 300 source–Lean pairs: FYAN, a single direct judge, majority voting over 23 judgments, and LeanScorer. Invalid outputs were rerun under the same settings for every method; FYAN returned a verdict on 297 of 300 pairs; the three others still had no valid verdict after two engineering reruns, and all GLM-5.3 figures below use these common 297 pairs for every method.

The advantage under the revised labels carries over, and grows. With GLM-5.3, FYAN's accuracy is 80.5%, against 63.3% for one direct judgment, 60.3% for 23-vote majority, and 72.7% for five-round LeanScorer. Its recall on inconsistent pairs is 83.5% with DeepSeek and 79.8% with GLM-5.3, while the direct judge's falls from 65.5% to 46.3%. The comparison table in W4 reports both models.

Response to W4

Thank you for pointing us to LeanScorer in Mathesis (§3.2). Its assessment of individual assumptions and conclusions makes it a particularly relevant comparison for FYAN. We add a direct empirical comparison and a clearer explanation of the mechanisms.

Direct comparison. All checkers receive the same 300 source–Lean pairs and underlying model. LeanScorer is our reimplementation of its published rubric: components are rated and aggregated into a score, accepted at the published threshold 0.6. We report it as published (one pass) and averaged over five independent rounds. Results under the original benchmark labels and our revised labels in the paper are reported separately.

How the controls are set up. The single direct judge assesses each source–Lean pair once. The majority-vote baseline repeats the same assessment in independent calls, using the same model, prompt, and sampling settings, and takes the majority verdict. We used 23 votes, which with DeepSeek costs 21.9 CNY for the 300 pairs against 31.0 CNY for the full audit.

Missed inconsistent statements / correct statements rejected (DeepSeek: 300 pairs; GLM-5.3: the common 297 pairs):

MethodDeepSeek, originalDeepSeek, revisedGLM-5.3, originalGLM-5.3, revised
Single direct judgment24 / 1071 / 156 / 3109 / 0
Majority voting, 23 judgments24 / 1169 / 064 / 2118 / 0
LeanScorer, one pass16 / 3048 / 649 / 1592 / 2
LeanScorer, five rounds16 / 2947 / 436 / 1579 / 2
FYAN10 / 4134 / 920 / 5241 / 17

Costs for the 300 pairs with DeepSeek: one direct judgment 1.2 CNY, 23-vote majority 21.9, LeanScorer 5.3 (one pass) and 25.9 (five rounds), FYAN 31.0. Reporting both missed errors and false rejections shows how each method balances catching changed statements with accepting faithful ones.

What the numbers show. Under our revised labels, which the independent review supports (W2), FYAN is the most accurate method on both models: 85.7% with DeepSeek and 80.5% with GLM-5.3, against at most 83.0% and 72.7% for the baselines. The gap to five-round LeanScorer is +2.7 points with DeepSeek (uncorrected 95% CI [−1.3, +7.0]) and +7.7 with GLM-5.3 (uncorrected CI [+2.0, +13.5]; not significant after Holm correction over 16 comparisons, p = 0.128). FYAN also misses the fewest inconsistent statements in all four settings. Its extra rejections are deliberate: a rejected statement returns to repair for one more audit, while a missed one is proved and built upon.

What FYAN adds. LeanScorer splits a statement into assumptions and conclusions, rates their matches, and combines those ratings into an overall score. FYAN connects such comparisons to whether a theorem may proceed to proof construction and become available to later theorems, through three checks:

  1. Check every part. Each object, assumption, and conclusion in the recorded source and Lean statement must have a corresponding part or be marked as missing. If the source assumes both A and B but Lean contains only A, the missing B remains an explicit item to check.
  2. Check direction and available assumptions. To recover the original claim, Lean's conclusion must imply the source conclusion, while the source assumptions must suffice for Lean's assumptions. For example, proving A → (G ∧ H) gives A → G, but proving A → G alone does not generally recover the missing H. Code checks these directions and which variables and premises may be used at each step.
  3. Protect statements and prove admitted changes. Reviewed statements and the inputs used to review them are protected during proof construction. An admitted substantive change also requires a Lean-checked bridge from the modified theorem to a statement of the original claim. The bridge's target is itself checked against the source. This retains a useful modified theorem while also providing the original result for later proofs. The model still judges whether a routine rewrite is justified.

The source trees, recorded matches, and conversion reasons make decisions traceable and help guide precise repairs. LeanScorer folds its component ratings into one score for evaluation; FYAN returns each finding, naming the part and the conversion, to the statement repair step.

Thank you again for your thoughtful feedback. The new case study, independent human review, and baseline comparisons provide further evidence for FYAN’s role in document-level formalization. We hope these additions address your concerns and clarify the value of connecting traceable semantic checks to statement repair and subsequent proofs.

Reviewer_fAD7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Fyan is a human–AI harness for document-level mathematical formalization. It constructs dependency-linked theorem units from PDF, TeX, or Markdown sources, organizes informal reasoning, formalization plans, and Lean proofs, and supports human guidance through a dashboard. Its main focus is structured semantic auditing: comparing local source–Lean correspondences, checking coverage, scope, and polarity, protecting accepted statements, and requiring bridges for admitted substantive changes. The authors evaluate it through stage-wise FormalTCS tasks, ConsistencyCheck, and an ODE development, reporting an increase in FormalTCS proof passes from 69/143 for a general-purpose agent to 86/143.

Critical strengths
  1. The semantic audit addresses an important practical failure mode: a valid formal proof may establish a statement that has drifted from its source. Fyan records local correspondences for objects, hypotheses, conclusions, and definitions, connecting them to coverage, scope, polarity, statement protection, and bridge obligations where required. The records help locate discrepancies, while freezing and rechecking help prevent reviewed interfaces from changing during proof construction. This careful organization of source fidelity and formal-proof acceptance is the most valuable part of the work.

  2. The system integrates document-level input with controlled human interaction. It constructs theorem units, dependencies, plans, and Lean scaffolds from PDF/TeX/Markdown sources and lets experts inspect blockers and provide mathematical or strategic guidance during execution. This broadens its interface relative to workflows that require a prepared blueprint or start from an isolated target. Routing human advice through a channel that cannot directly modify protected statements is a useful engineering design for combining interaction with acceptance control.

Critical weaknesses
  1. The evaluation does not establish the workflow's resource–quality trade-off or the scales at which it is worthwhile. FormalTCS includes a general-purpose-agent control and reports proof passes increasing from 69/143 to 86/143, but sufficient accounting of tokens and cost, cumulative agent execution time, end-to-end wall-clock time under concurrency, effective budgets, and scale-stratified outcomes is missing for the two arms. This makes it difficult to judge when the additional workflow is justified relative to a simpler agent. The semantic subtask already shows substantial overhead: Appendix B.3 estimates model costs of about 0.15 CNY per item versus 0.007 CNY for the direct judge; this does not establish the corresponding proving costs on FormalTCS. The approximately 9,355-line ODE case predates the new semantic audit and lacks a controlled same-project comparison, leaving the complete current workflow's relative efficiency on large dependency-linked developments unestablished.

  2. Default semantic acceptance still depends on unverified model judgments, with limited direct evidence about the reliability of these stages. A model extracts the source tree and another call reviews it, but a separate human evaluation of the trees' semantic accuracy is not reported. The model also assigns trivial_equivalent: deterministic code checks permitted conversion names, structure, and required evidence fields without proving that the conversion holds. Appendix C.2 reports that all 46 missed inconsistencies under the authors' verified labels were grade-2 acceptances. Those labels also count mathematical equivalence requiring a substantive argument as inconsistent, so the 46 are not all necessarily logically inequivalent. They nevertheless show that the default gate can admit candidates that fail the stated fidelity criterion. Appendix E acknowledges the guarantee boundaries, but the contributions of source interpretation, pairing, and conversion classification to these errors remain unclear, limiting the assessment of audit reliability.

Questions for authors
  1. Can you provide resource accounting for the existing FormalTCS runs of both arms, including input/output tokens and cost, cumulative agent execution time, and end-to-end wall-clock time under the actual concurrency settings, including unsuccessful runs? Please clarify effective budgets and accounting boundaries and how success and cost vary with task scale. Further comparisons of quality, cost, and elapsed time for the complete current workflow on dependency-linked paper- or textbook-scale projects would help establish its range of applicability.

  2. Were the source trees themselves independently checked by humans for semantic fidelity, and what was found? For the 46 grade-2 misses, can you distinguish errors in source interpretation or slot pairing, substantive equivalences classified as trivial, and genuinely invalid conversions? This would help identify the principal failure modes of current semantic acceptance.

  3. Please compare Fyan's structured semantic auditing more concretely with Aria/AriaScorer, Mathesis/LeanScorer, and LoC-Decomp/ASCC, explaining which additional checks or guarantees come from two-sided coverage, scope/polarity rules, and bridge integration. Which existing conversion categories could use model-proposed restricted transformation paths or local certificates checked by a rule system and Lean, and which still require semantic judgment? A mechanism-level discussion would help locate the contribution and identify which acceptance evidence could be made more verifiable.

5 comments

Thank you for recognizing FYAN's semantic auditing and controlled workflow. We added independent human reviews and comparisons addressing resource use, semantic reliability, and verifiable evidence.

Response to W1 and Q1

FYAN proves 17 more FormalTCS theorems with the same model: 86/143 versus 69/143. We added resource accounting, results by input size, and a document-level comparison.

Resources and settings. Both methods used DeepSeek-V4.1-Flash, Lean 4.32.2/Mathlib, and the same evaluator: nominally 3 turns per run, 7,200 s per FT2FP turn, interrupted runs resumed, and at most one final scored output per item. FYAN used at most five outer iterations, without proving subagents or automatic supervision. The table covers formal proofs (FT2FP), natural-language proofs (T2NP), and statement translation (NT2FT), including failures and retries.

TaskMethodInput tokens (cached)Output tokensCost (CNY)Time
FT2FPFYAN5.95 B (5.83 B)98.6 M62837.4 h active, 4 workers
FT2FPGeneral-purpose agent1.10 B (1.02 B)52.8 M31060.3 h span, mean concurrency 1.9
T2NPFYAN3.77 B (3.68 B)105.7 M58559.1 h active, 4 workers
NT2FTFYAN1.65 B (1.59 B)69.8 M362three batches (3,600 / 2,400 / 2,400 s per turn), ≥ 22.4 h
NT2FTGeneral-purpose agent0.16 B (0.15 B)8.8 M476.1 h generation

Both arms use the same off-peak price (CNY per 1M tokens: cache hit 0.02, miss 1, output 4).

At matched FT2FP caps, FYAN costs 628 versus 310 CNY: 7.3 versus 4.5 per verified proof, and about 19 per additional proof. The control reached its turn cap on only 3 items and submitted 50 of its 74 failures with sorry, while FYAN continued repairs within its allowance. The paired gain is 22 FYAN-only versus 5 control-only proofs (exact McNemar p = 0.0015).

Outcomes by input size. We sorted all 143 statements plus definition contexts by cl100k_base token length, breaking ties by item ID, into three nearly equal groups shared by both methods:

Group (ranks)Input length (tokens)TasksFYAN provedControl provedFYAN mean cost (CNY)FYAN mean time per task
Small (1–48)197–7464834 (70.8%)26 (54.2%)3.4170 min
Medium (49–96)751–1,4464832 (66.7%)25 (52.1%)4.5775 min
Large (97–143)1,449–5,3014720 (42.6%)18 (38.3%)5.2290 min
All197–5,30114386 (60.1%)69 (48.3%)4.3978 min

Gains are +16.7, +14.6, and +4.3 points on small, medium, and large inputs. On larger inputs, both methods prove fewer tasks and FYAN's cost rises. Control per-task costs and times were not recorded; its 310 CNY total averages 2.2 CNY per task. Input size does not measure mathematical difficulty.

Audit cost and stronger controls. On 300 DeepSeek pairs, the audit costs 31.0 CNY (0.10 per pair), versus 21.9 for 23 independent votes and 1.2 for direct judgment (0.004 per pair), including all calls and invalid-output reruns. Voting targets a cost comparable to the audit.

Voting changes DeepSeek accuracy little: 88.3% vs 88.7% for direct judgment under original labels, and 77.0% vs 76.0% under revised labels. On the common 297 GLM-5.3 pairs, voting lowers accuracy: 77.8% vs 80.1% and 60.3% vs 63.3%, respectively. Under revised labels, the DeepSeek audit misses 34 inconsistencies versus 69 for voting and 71 for direct judgment, with 9 false rejections versus 0 and 1 (Q3).

Document-level efficiency. The revision clarifies that ODENumLib (9,355 lines, numerical ODEs) predates semantic auditing and demonstrates theorem reuse in the earlier workflow. We added a comparison on the Sidorenko paper, with targets fixed before either run: edge-removal inequality, recursive count, leaf redistribution, Theorem 1, all three parts of Corollary 2, and an optional weighted AM–GM proof.

With the same model and Lean environment as ARCHON v0.2.0, FYAN proved all three proof units, including Theorem 1, in parallel in 80 minutes after statement freezing. Its 3,787-line project builds without sorry and with standard axioms only. Shared definitions and statements were written once in one module, supporting parallel work and reuse without duplicated definitions.

ARCHON produced 2,466 lines; its later proving rounds (iterations 5–7) took 2 h 30 min 50 s (whole run: 5 h 57 min). Earlier rounds also included proving, so these phase timings do not establish an end-to-end speedup.

All three proof units passed the final Lean checks. The recorded audit status was preserved: one certified source unit and three flagged units continued under human authorization (Appendix D.2).

Response to W2 and Q2

An independent mathematics PhD student, uninvolved in the project, reviewed source trees and all 46 historical acceptances to assess extraction accuracy and distinguish faithful restatement from mathematical equivalence.

Reading the original statement: 47/50 judged correct

The source tree records objects, definitions, assumptions, quantifiers, and conclusions, linked to quotations and extracted before seeing the Lean candidate.

For a fixed random 50 of the 300 evaluation pairs, the expert saw only the source text and tree; Lean candidates, system verdicts, and prior labels were withheld.

Expert judgment of the source treeItems
Correctly represents the source47/50 (94%)
Incorrect3/50
Cannot determine0/50

The expert also valued notes on unclear source wording:

  • An unspecified starting index is flagged as potentially zero-based or one-based.
  • The ambiguous term zn/n is preserved, with possible readings z^n/n and z·n/n flagged for clarification.

Each Lean object, assumption, or conclusion can be checked against a specific source entry; quotations and ambiguity notes expose the chosen reading and help reviewers locate disagreements or missing context.

The three errors concerned omitted requests for all possible values or uniqueness, and a function whose occurrences were not linked to its quantifier.

The expert suggested that supplying answers alongside questions may obscure task requirements. A problem asks for nn satisfying i+2i2+⋯+nin=48+49ii+2i^2+\cdots+ni^n=48+49i, then says “Show that it is D,” with D = 97. The tree recorded 97 satisfies the equation, but omitted every admissible solution equals 97.

This explanation remains untested. Follow-up checks could separate answer verification from finding all solutions and check quantified functions' bindings. The reviewed sample, judgments, and reasons accompany the revision.

Checking the 46 accepted statements

These 46 historical grade-2 acceptances were inconsistent under our stricter labels, which scrutinize changes in definitions, domains, assumptions, and quantifier scope. Models may differ on whether a reformulation is “obvious.” The same expert reviewed all 46 using source text, Lean statements, source trees, recorded matches, and relabeling reasons:

Review outcomeCases
Immediately equivalent; directly acceptable3
Equivalent, requiring an argument24
Stronger, weaker, or non-equivalent14
Unresolved without source context5

27/46 are mathematically equivalent under a broader criterion. Three warrant corrections (e.g., 2%2\% and 1/501/50 are interchangeable); the other 24 require an argument and remain inconsistent under the direct-restatement criterion. For example:

  • Improper integrals: evenness of xsin⁡x/(x2+a2)x\sin x/(x^2+a^2), a>0a>0, makes its integral over [−y,y][-y,y] twice that over [0,y][0,y], connecting Lean's symmetric limit to the source's two one-sided limits.
  • Sums of two squares: absolute values replace integer square witnesses by natural ones; nonnegative sums and the connection between infinitude and unboundedness justify the changed formulation.

The records flag 36 issues in judging rewrites, 4 in matching text to Lean, and 2 in source trees, with overlap. One tree issue concerns an incorrect explanatory note; one further tree judgment is unresolved.

The shared-eigenvalue and local-compactness judgments retain their source conventions: finite dimensionality and “each point has a compact neighborhood,” respectively.

Source trees, matches, and conversion reasons make decisions traceable, locating the definition, correspondence, or argument needing attention and prioritizing Lean-checked conversion proofs (Q3). The revision lists corrections and unresolved cases alongside original-label results.

Under our revised labels, FYAN misses the fewest inconsistencies among the methods in Q3. Recall remains 83.5% with DeepSeek and 79.8% with GLM-5.3, versus 65.5% and 46.3% for direct judgment (GLM: common 297 valid pairs).

Response to Q3

FYAN connects semantic checking to permission to prove a theorem and reuse it downstream. The following comparison distinguishes existing mechanisms from proposed extensions.

ApproachMain mechanism
Mathesis / LeanScorer, §3.2Decomposes assumptions and conclusions, rates their match to Lean, and combines the ratings.
Aria / AriaScorer, §3.2Retrieves Lean terms' meanings to compare mathematical definitions.
LoC-Decomp / ASCCTranslates Lean segments back to natural language, compares with the source, and refines formalizations using semantic/compiler feedback.
FYANChecks omissions and implication directions, protects reviewed statements, and requires proofs connecting admitted changes to the source claim.

FYAN adds three checks:

  1. Two-sided coverage. Every recorded object, assumption, and conclusion on both sides must be matched or marked missing. If the source assumes A and B but Lean includes only A, the missing B remains explicit.
  2. Polarity and scope. Lean's conclusion must imply the source conclusion; source assumptions must suffice for Lean's assumptions. For example, A → (G ∧ H) gives A → G, but the reverse need not hold. Code checks these directions and the available variables/premises.
  3. Freezing and proof transfer. Reviewed statements and review inputs are protected. Under the default policy, grades 2–3 proceed to proving; an admitted grade-1 change requires a Lean-checked bridge S1 → S0 from the modified theorem to the source-facing original claim in Lean. The audit checks S0 against the text. Both the useful modification and the source result are retained; grade-2 rewrites still depend on semantic judgment.

Proposed extensions: models could propose restricted transformation paths or local certificates for rules and Lean to check:

Conversion categoriesProposed check
Renaming, reordering independent arguments, grouping/splitting assumptions, unfolding definitionsCheck substitutions, typing, scope, and rewrite rules; supply Lean equality/equivalence proofs where needed.
Changing indices, encoding conditions in types, replacing local definitions with Mathlib definitionsSupply representation maps and local Lean proofs, including bounds, side conditions, and source–Lean object correspondences.
Implicit textbook assumptions or ambiguous terminologyConsult the source to fix the intended meaning; Lean then verifies its formal consequences.

The 46-case analysis prioritizes domain and definition changes for these extensions.

Empirical comparison. On the same 300 pairs and model, we compared FYAN with direct judgment, 23-vote majority, and our reimplementation of LeanScorer's rubric at its published threshold 0.6, in one pass and averaged over five rounds. Invalid outputs were rerun under the same settings for every method: all returned valid DeepSeek verdicts on 300 pairs; FYAN returned 297 with GLM-5.3. Counts below are misses / false rejections; costs cover all 300 pairs.

Method (DeepSeek-V4.1-Flash)Original labelsOur revised labelsCost (CNY)
Single direct judgment24 / 1071 / 11.2
23-vote majority24 / 1169 / 021.9
LeanScorer, one pass16 / 3048 / 65.3
LeanScorer, five rounds16 / 2947 / 425.9
FYAN10 / 4134 / 931.0

FYAN has the highest accuracy under our revised labels: 85.7% with DeepSeek and 80.5% with GLM-5.3, versus at most 83.0% and 72.7% for the baselines (GLM: common 297 pairs). The gap to five-round LeanScorer is +2.7 points with DeepSeek (uncorrected 95% CI [−1.3, +7.0]) and +7.7 with GLM (uncorrected 95% CI [+2.0, +13.5]; not significant after Holm correction over 16 comparisons, p = 0.128).

Across the 16 comparisons (two models, two labelings, four baselines), FYAN misses the fewest inconsistencies; uncorrected 95% paired intervals on misses exclude zero in 14. False rejections require repair and another audit (about 0.1 CNY for the audit); missed discrepancies can propagate into later proofs. In the conditional cost analysis, FYAN has lower weighted error throughout when a miss costs more than about 2.3 false rejections.

Thank you again for your constructive feedback. The added comparisons and human review clarify FYAN’s practical benefits, resource costs, and semantic safeguards. We hope these results and clarifications address your concerns and better demonstrate the value of checking source fidelity before discrepancies propagate into subsequent proofs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.