acceptodds
Under review as a conference paper at ICLR 2027

The Future Is the Test: Library Abstractions for Mathematics Not Yet Written

Abstract

The value of a mathematical abstraction lies in the mathematics it supports next. On major Mathlib refactors, adopted designs complete 60/60 use attempts against 29/60 for their predecessors, with no losses. We formulate future-use generalization (FUG) as evaluating abstractions on withheld mathematical uses in Lean. FUSE turns failures and recurring proof work on observed uses into revisions of shared statements. FutureUseBench connects historical designs, authored use graphs, and real-library replay to test these components separately. Across eight authored development targets, exposing use descriptions increases unseen declaration completion . With accepted interfaces fixed, feedback on observed uses removes 81% of remaining declaration failures. Among jointly completed, source-screened uses, exposure reduces downstream source tokens by 16–27%, a proxy for adaptation burden. Proof dependence changes method rankings, and shared revisions can both enable reuse and lose interface acceptance. Reliable transfer to unseen proofs after revision remains open. These findings establish later mathematics as an executable test and a concrete design signal for abstraction.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
autoformalization, mathematical abstraction, formal mathematical reasoning, library learning, execution-guided synthesis, future-use generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bZ9pOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

judge a maths defination by the future maths it would make possible

question it suggestes is: eval maths by what we can build on top of it

Proposes FUG metrics, FUSE, FutureUseBench

experiments around mathlib, showcasing results. Takeaways: feedback fixing failures, and shorter isnt better always

Critical strengths
  • idea is strong
  • well framed
  • Proposes FUG metrics, FUSE, FutureUseBench
Critical weaknesses
  • this split of tests is based on ID. thus better to call random train/test split rather than, "future" test split
  • sample size is small: 5 redesigns × 4 uses × 3 trials.
  • same AI family models in GPT series
  • cost of ai usage is not reported
  • data contamination: as mathlib info would be in training data
  • aligning related works: ReGAL (Stengel-Eskin et al., 2024), TroVE (Wang, Neubig and Fried, 2024) and LATM, "LLMs as Tool Makers" (Cai et al., 2023).
  • also it contradicts the findings from "Large Language Models Cannot Self-Correct Reasoning Yet" (Huang et al., ICLR 2024) and Kamoi et al. (2024), and should be mentioned within the paper on how its still justified
  • the work misses on how FUSE could point towards gains apart from any general repair, and beat accross unseen proofs
Questions for authors
1 comment
AuthorsOpenReview ↗

Response to Reviewer bZ9p on evaluation, cost and downstream feedback

Thank you for highlighting the mathematics a definition makes possible and for asking how we measured reuse, feedback and cost.

Common setup. Direct sees the target alone. Visible also sees descriptions of public uses. FUSE-C keeps the upstream statement fixed and repairs the downstream code that uses it. FUSE can change both after seeing the results of running that code.

1. How are public and held-out uses split within each target? Within each of eight development targets, we split by use ID. The model saw the target and public use descriptions. Other uses were reserved for evaluation and could belong to the same family. The separate proof comparison covers seven targets.

2. What are the sample sizes and experimental units? For each of five Mathlib changes, we froze statements generated from descriptions of the designs before and after refactoring. Four uses per target give 20 distinct target–use pairs. Three trials per pair give 60 attempts per version. The two versions completed 29/60 and 60/60, respectively.

panelopportunitiesresult
providing use descriptions, 8 targets624Direct 112/624 → Visible 255/624; +22.92 percentage points
repair on the same starting files84 initial failures among 300 opportunities with an accepted statementbinary 35/84, detailed 66/84
proof transfer, 7 targets × 30 uses × 3 repeats630initial 167/630; binary feedback 189/630

In the repair panel, 100 further opportunities were blocked at the upstream check. Counting all 400 opportunities gives 251/400 final completions with binary feedback and 282/400 with detailed feedback.

3. Which models handle generation, repair, feedback, and judgment? All executors are GPT-family models. Main runs use GPT-5.4/high for generation and GPT-5.6-sol/high for semantic checking. Historical calibration uses GPT-5.6-luna/low for solving and GPT-5.6-sol/high for the final check. Matched diagnostics use GPT-5.4/high for repair and GPT-5.6-sol/high for feedback and final judgment. We fix the Mathlib version, prompts, use sets, and execution settings across runs.

4. What is the token cost of each workflow?

workflowcompleted / 800tokens
Direct1555.60M
Visible3496.59M
Visible + binary4189.47M
Visible + detailed4318.97M

Both repair rows count the shared Visible starting stage (6.593M tokens) once. Their added stages use 2.878M and 2.376M tokens. In the separate experiment starting from identical files, detailed feedback repaired 31 more failures for 1.01% more tokens in the repair stage.

5. How might prior exposure to Mathlib affect the comparisons? Within a target, we use the same model and Mathlib version across methods. Absolute completion rates can still reflect pretraining, and the two historical designs may differ in familiarity. The 29/60 and 60/60 figures calibrate the measure on those five changes.

6. How does FUG relate to previous library-learning and theorem-proving work? ReGAL, TroVE, and LATM study refactoring using execution results and reusable tools. Stitch and LILO learn program libraries through compression and synthesis. TacMiner discovers tactic libraries. FUG evaluates whether a faithful Lean statement supports held-out constructions and proofs. The historical calibration also shows that a shorter statement does not automatically provide better support for later uses.

7. How does repair with downstream feedback relate to work on self-correction? Huang et al. (2024) study revision without external feedback. Kamoi et al. (2024) distinguish settings with reliable external feedback. Here both arms receive downstream results on the same 84 failures. Binary feedback repairs 35, detailed feedback 66. Both compile 300/300 candidates. At least 26 of the 31 additional completions come from checks of meaning and use. GPT-5.6-sol gives both the feedback and the final semantic judgment.

8. What do the held-out proofs and cases of mathematical reuse show beyond repair? Across seven targets, initial statements gave 167/630 held-out proof completions and binary-feedback endpoints gave 189/630 (+3.49 percentage points). Two endpoints retained the initial statement. On the separate observed uses, FUSE-C and FUSE completed 1308/6000 and 1119/6000 before checking proof dependencies, and 899/6000 and 901/6000 after that check. Some configurations share candidates. In the Ruzsa case, several branches reused the same conditional construction and function of a reference pair. Before the check, FUSE-C and FUSE completed 15/50 and 22/50 uses. After it, they completed 8/50 and 22/50. In the group presentation case, a later proof used the shared presentation and a quotient isomorphism to transport the result.

Reviewer_v8jPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes evaluating a Lean abstraction by whether it supports later mathematical uses. It defines future-use generalization (FUG) as completion of withheld uses, introduces FUSE algorithm to revise a shared abstraction using failures and repeated proof work on observed uses. The paper also introduces FutureUseBench to evaluate the idea, and shows FUSE improves the completion rate.

Critical strengths

The paper suggests using Future-use compatibility or FUG as the main evaluation metric instead of the conventionally used compression rate

The paper introduces FutureUseBench from Mathlib, and FUSE improves completion rate on FutureUseBench.

Critical weaknesses

Idea of paper is already presented in previous works. While later works on library-learning uses compression rate as the main evaluation metric, maximizing the probability of downstream corpus is very initial idea suggested in the DreamCoder paper.

The paper lacks comparison with similar works (e.g., DreamCoder https://arxiv.org/abs/2006.08381 , Stitch https://arxiv.org/abs/2211.16605, LILO https://arxiv.org/abs/2310.19791, TacMiner https://arxiv.org/abs/2503.24036)

FUG is a metric that depends on downstream autoformalizer, but table is only presented for a single model

The description of the FUSE algorithm is vague and lacks formal rigor, relying on imprecise phrasing such as 'the proposal is supported' and 'a hypothesis produced s'

Questions for authors

How the idea of FUG is different to probability maximization of DreamCoder?

How much does the FUSE algorithm improve over previous methods?

How important is the downstream autoformalizer to the overall pipeline? Specifically, how does the completion rate change when an LLM is asked to find reusable generalizations of the corpus without testing on the downstream autoformalizer?

Could you please provide formal definitions for the key concepts and steps in the FUSE algorithm?

3 comments
AuthorsOpenReview ↗

Response to FUG, DreamCoder and FUSE

Thank you for asking how FUG builds on library learning and what downstream execution adds to statement design.

Q1. How does FUG differ from DreamCoder’s probability objective for learning a library? DreamCoder learns a library that makes programs solving observed tasks more probable. In minimum description length terms, a routine justifies its cost when it saves enough bits across those solutions. DreamCoder also measures the learned library’s ability to solve new tasks.

For example, root permutations preserving algebraic relations over the base field form a Galois group. Once cancellation (ax=ayax=ay implies x=yx=y) is proved from the group laws, the same proof applies to those permutations and to invertible matrices. Their multiplication is implemented differently.

Future-use generalization (FUG) asks which statement for one fixed mathematical target supports more later work. We freeze each faithful candidate. The same executor, with the same budget, attempts the same held-out constructions and proofs, rebuilding their prerequisites. A law may save no space in the observed solutions. If exposing it enables one more held-out construction and preserves the earlier successes, FUG rises even when the statement grows longer.

Q2. How does FUSE compare with prior methods, and what gains do the Lean experiments measure? DreamCoder, Stitch and LILO learn libraries of program abstractions. TacMiner learns proof tactics. FUSE uses later proofs to guide the design of a shared Lean statement. A direct numerical comparison would require adapting those systems to produce statements for the same Lean targets.

In our group presentation case, the initial statement let a proof transport generators. The revised structure requires relations that account for every equality between words in the generators. Downstream code uses this condition to derive an isomorphism from the quotient by those relations to the original group. The transport proof can then use the whole presentation.

We quantified the value of seeing use descriptions during statement design. Across eight development targets, the statement passed its checks in 4/16 Direct trials and 9/16 Visible trials. Completed held-out declarations rose from 112/624 to 255/624, or 143 more with public use descriptions.

Q3. What does downstream execution add beyond seeing use descriptions? In Visible, the model sees the target and public descriptions of its intended uses. It can revise the draft up to three times using checks on the statement itself. No downstream use is run to guide these edits. Direct also omits the use descriptions.

In a separate experiment with 800 opportunities, we fixed the Visible statement before running its uses. Both repair conditions started from the same files and received feedback on the uses scored below.

ConditionDeclarations completedRate
Direct, target alone155/80019.38%
Visible, target and use descriptions349/80043.63%
Visible + pass/fail feedback418/80052.25%
Visible + detailed feedback431/80053.88%

Detailed feedback adds 82 completions over Visible (+10.25 percentage points) with the statement fixed. GPT-5.6-sol supplies feedback and final semantic judgments. These rates describe repair with that executor. FUSE-S separately tests the final FUSE statement with fresh downstream files.

Q4. How does FUSE define, test and select a statement revision?

  1. For target xx, define held-out uses UxU_x, executor EE, and budget BB. Let cE,B(s,u)∈{0,1}c_{E,B}(s,u)\in\{0,1\} indicate whether statement ss completes use uu.
  2. A run scores ∣Ux∣−1∑u∈UxcE,B(s,u)|U_x|^{-1}\sum_{u\in U_x}c_{E,B}(s,u), or zero for an invalid statement. FUG averages runs within each target, then gives each target equal weight.
  3. Define a proposed revision ht=(et,δt,Tt,Pt)h_t=(e_t,\delta_t,T_t,P_t) from allowed evidence, an allowed shared edit, checks to run again, and uses that must remain valid.
  4. Before editing, verify the evidence and edit are allowed, the tests TtT_t can be run, and the protected uses PtP_t previously succeeded.
  5. The revision model AA produces st+1=Revise⁡A(x,st,ht)s_{t+1}=\operatorname{Revise}_A(x,s_t,h_t). We store the proposal with the changes it produced so the result of each test can be traced back to the proposed edit.
  6. After editing, run TtT_t and recheck PtP_t. Among accepted statements with a complete fresh evaluation, select the one completing the most public uses. The formal protocol resolves ties by a predeclared rule over extra conversions and computation.
MethodInputWhat the method can change or test
Directtargetinitial statement
Visibletarget + use requirementsdraft + required checks
FUSE-Cobserved usesdownstream code; upstream statement fixed
FUSEobserved execution tracesupstream statement + downstream code
FUSE-Sfinal FUSE statementfresh downstream workspace

Q1. While DreamCoder prefers the smallest term in the version space, its minimization objective is not term size; rather, its cost function is defined in terms of probability. It remains unclear how the core idea of FUG differs from that of DreamCoder Q2. I agree that adapting existing library-learning tools to dependently typed programs is non-trivial Q3 & Q4. The responses to Q3 and Q4 helped clarify the algorithm, but the novelty of the paper remains my main concern

AuthorsOpenReview ↗

How does FUG differ from DreamCoder’s objective?

Thank you for clarifying DreamCoder’s probabilistic objective. Both approaches can be expressed in terms of optimization. DreamCoder scores a library using a prior on the library and its weights together with the likelihood of the observed tasks. Its abstraction step refactors programs it has already found. FUSE, our design procedure, revises a faithful Lean statement of one fixed mathematical target using downstream attempts, including failed ones. FUG (future-use generalization), our evaluation measure, asks what fraction of held-out constructions and proofs a fixed solver can complete using that statement. DreamCoder, Eq. 1 and Fig. 2

Compressing existing solutions and supporting future uses can favor different abstractions. To adapt the polygon example from LAPS, suppose every training polygon has side length 3. Both polygon(n), with length fixed at 3, and polygon(n, length) can draw them all. The first saves an argument at every call. If a later task asks for a hexagon of side length 5, the second can be reused directly as polygon(6, 5). A choice that compresses the observed programs can fix precisely the parameter that a later task needs to vary. LAPS, §4.3

If DreamCoder later solves the larger-hexagon task, that program becomes available for refactoring. FUSE can revise a statement before the attempted construction succeeds. It tries a candidate statement on uses visible during design. A construction with no successful solution yet can expose a missing operation or a parameter fixed too early. FUSE can edit the shared statement and rerun the affected uses, including earlier successes. In one Ruzsa-distance run, the first attempt assumed the triangle inequality it was meant to prove. After pass/fail feedback, FUSE revised the shared statement and the downstream proofs. The next round derived the inequality by summing simpler bounds with nonnegative weights. After selection, we freeze the statement and test it on uses withheld during design. A use counts if the solver completes it using the statement, either directly or through an earlier result that depends on it.

Does knowing some future requirements help with uses that remain unseen? DreamCoder evaluates transfer to new synthesis tasks. Our experiment varies the information available to the statement designer while holding the target, background library, scoring rule and test uses fixed. Direct sees the target and library. Visible also sees descriptions of a separate set of public uses. Across eight development targets with two trials each, held-out declaration completion before downstream repair rose from 112/624 to 255/624 (+22.92 percentage points). When a statement fails the initial interface check, its held-out uses contribute zero completions and remain in the denominator. Direct passed this check in 4/16 trials and Visible in 9/16. Within their respective accepted trials, completion was similar (Direct 112/154, or 72.7%, and Visible 255/350, or 72.9%). The main observed gain from seeing the descriptions was that more statements met the required interface.

Reviewer_3mwQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Future-Use Generalization (FUG), a framework for evaluating mathematical abstractions through their ability to support unseen downstream constructions and proofs in Lean. It proposes FUSE, which uses downstream execution feedback to revise shared mathematical statements, and FutureUseBench, combining Mathlib refactors, authored dependency graphs, and real-library evaluations.

Experiments show that exposing downstream use descriptions improves held-out declaration completion from 17.95% to 40.87%, while execution feedback improves completion on observed uses. However, the full FUSE pipeline and its generalization to independent mathematical tasks remain unverified.

Critical strengths
  1. Meaningful problem formulation. Evaluating abstractions through downstream mathematical uses addresses an important limitation of existing autoformalization evaluations. The distinction between interface acceptance, proof completion, and proof dependencies is valuable.

  2. Useful empirical analysis. The Mathlib refactor comparisons and controlled experiments provide concrete evidence that abstraction design affects downstream usability. The authors also explicitly acknowledge important limitations in their measurements.

Critical weaknesses
  1. Missing end-to-end validation of the proposed method. Appendix E explicitly states that the complete FUSE algorithm has not been evaluated. The held-out experiments in Table 7 use development targets with potentially overlapping use families, while the larger benchmark remains unevaluated. Consequently, the central claim of generalization to genuinely unseen mathematical uses is not sufficiently supported.

  2. No convincing advantage from joint abstraction revision. Table 29 reports nearly identical source-screened completion for FUSE and the fixed-interface FUSE-C baseline (901/6000 vs. 899/6000). Moreover, the held-out proof improvement in Table 7 is statistically inconclusive (+3.49 percentage points, with a confidence interval crossing zero). The source-based dependency screening also does not certify full proof closure. These results leave the incremental benefit of the paper's core revision mechanism unclear.

Questions for authors
  1. Can the authors provide an end-to-end comparison of FUSE against FUSE-C and matched self-refinement baselines on untouched targets with whole-family held-out uses? This is essential for establishing the proposed method's generalization benefit.

  2. What explains the negligible aggregate improvement over FUSE-C in Table 29? Can the authors report fully proof-closed results and quantify how interface regressions and evaluation variability affect this comparison?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3mwQ on held-out proofs and Table 29

Thank you for asking which held-out proofs Lean confirmed and why FUSE and FUSE-C finish so close in Table 29.

Q1. Can the authors provide an end-to-end comparison of FUSE against FUSE-C and matched self-refinement baselines on untouched targets with whole-family held-out uses?

We compared FUSE, FUSE-C and matched self-refinement on two development targets, FUB030 and FUB018. For each target, the methods saw the same three uses during revision, and we held out the other 47. We split by individual use, so uses from the same family could fall on both sides.

TargetFUSEFUSE-CMatched self-refinement
FUB03013/2314/239/23
FUB0184/223/226/22

Before seeing the new scores, we fixed the first 12 numbered held-out uses on each target and registered the laws each use required. This gave 23 proof obligations on FUB030 and 22 on FUB018. Each numerator counts obligations Lean confirmed. On FUB018 the FUSE count is final. The other entries give confirmed lower bounds.

The methods differ in what they can revise before that evaluation. FUSE can propose and test an edit to the shared statement. FUSE-C keeps the statement fixed and repairs downstream code. Matched self-refinement can revise the statement using the same kinds of public evidence as FUSE, without a separate proposal step. Each method freezes its choice before the held-out uses are revealed.

Within each target, all three methods selected the same upstream statement. The columns show what their downstream workflows proved using it.

Q2. Why are FUSE and FUSE-C so close in Table 29? What do proof closure, statement regressions and repeated runs tell us?

Table 29 counts uses on which the methods received feedback. After checking proof dependencies, FUSE completed two more uses overall. Grouping trials by their final outcomes shows how the counts add up. FUSE completed 78 more uses where both upstream statements passed, and 76 fewer in the remaining trials. All attempts are included below.

Uses across five configurationsFUSE-CFUSE
Both statements passed, after checking proof dependencies823/2450901/2450
Remaining trials, after the same check76/35500/3550
All trials after the check899/6000901/6000
All trials before the check1308/60001119/6000

What can improve when both proofs already pass? When a few uses each rebuild the same argument, it can be worth refactoring the library definition they share. Mathlib’s convex spaces illustrate why. Several proofs may need to flatten a weighted average of averages. Giving the shared definition one law for that step lets each proof cite it. A proof that derives the step locally and one that cites the law each earn one completion. Across the five Table 29 configurations, both methods completed the same use and passed the source check in 724 matched pairs. Their downstream files totaled 86,503 source tokens with FUSE-C and 76,913 with FUSE (11.1% fewer). With FUSE, 122 files were shorter, 39 longer and 563 equal in length. We counted downstream source only. FUSE could edit both the shared statement and the downstream proofs.

What the proof check adds. The original count allows a downstream proof to use an upstream proposition whose proof was left open. The source check looks for such dependencies and can miss implicit ones. Q1 counts individual obligations verified by Lean. Checking a whole module would also require Lean to verify every prerequisite.

What changed when FUSE revised a statement? In the configuration with four trials per target, 3 of 15 initially accepted FUSE statements lost acceptance. None of the 17 initially rejected statements became accepted. FUSE could therefore change which trials entered the group where both statements passed. It could also edit downstream code, and in this configuration it received feedback on whether the statement passed. The 78 extra completions describe the combined workflows in that group. Before the dependency check, the full-cohort totals favored FUSE-C (1308 versus 1119).

How much did repeated evaluation vary? In a separate proof experiment, eight comparisons repeated evaluation with the same upstream statement. The differences in proof completion had a standard deviation of 8.55 percentage points. Several comparisons shared an initial evaluation. This gives a scale for variation when downstream proofs are attempted again. Table 29 itself also includes later runs that continued from earlier candidates, so the five configurations should be read together as a record of these runs.

In the Ruzsa case, several downstream proofs reused the same conditional construction and the same function of a reference pair. FUSE-C and FUSE completed 15/50 and 22/50 uses before the dependency check, and 8/50 and 22/50 after it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.