- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
judge a maths defination by the future maths it would make possible
question it suggestes is: eval maths by what we can build on top of it
Proposes FUG metrics, FUSE, FutureUseBench
experiments around mathlib, showcasing results. Takeaways: feedback fixing failures, and shorter isnt better always
Critical strengths
- idea is strong
- well framed
- Proposes FUG metrics, FUSE, FutureUseBench
Critical weaknesses
- this split of tests is based on ID. thus better to call random train/test split rather than, "future" test split
- sample size is small: 5 redesigns × 4 uses × 3 trials.
- same AI family models in GPT series
- cost of ai usage is not reported
- data contamination: as mathlib info would be in training data
- aligning related works: ReGAL (Stengel-Eskin et al., 2024), TroVE (Wang, Neubig and Fried, 2024) and LATM, "LLMs as Tool Makers" (Cai et al., 2023).
- also it contradicts the findings from "Large Language Models Cannot Self-Correct Reasoning Yet" (Huang et al., ICLR 2024) and Kamoi et al. (2024), and should be mentioned within the paper on how its still justified
- the work misses on how FUSE could point towards gains apart from any general repair, and beat accross unseen proofs
Response to Reviewer bZ9p on evaluation, cost and downstream feedback
Thank you for highlighting the mathematics a definition makes possible and for asking how we measured reuse, feedback and cost.
Common setup. Direct sees the target alone. Visible also sees descriptions of public uses. FUSE-C keeps the upstream statement fixed and repairs the downstream code that uses it. FUSE can change both after seeing the results of running that code.
1. How are public and held-out uses split within each target? Within each of eight development targets, we split by use ID. The model saw the target and public use descriptions. Other uses were reserved for evaluation and could belong to the same family. The separate proof comparison covers seven targets.
2. What are the sample sizes and experimental units? For each of five Mathlib changes, we froze statements generated from descriptions of the designs before and after refactoring. Four uses per target give 20 distinct target–use pairs. Three trials per pair give 60 attempts per version. The two versions completed 29/60 and 60/60, respectively.
In the repair panel, 100 further opportunities were blocked at the upstream check. Counting all 400 opportunities gives 251/400 final completions with binary feedback and 282/400 with detailed feedback.
3. Which models handle generation, repair, feedback, and judgment? All executors are GPT-family models. Main runs use GPT-5.4/high for generation and GPT-5.6-sol/high for semantic checking. Historical calibration uses GPT-5.6-luna/low for solving and GPT-5.6-sol/high for the final check. Matched diagnostics use GPT-5.4/high for repair and GPT-5.6-sol/high for feedback and final judgment. We fix the Mathlib version, prompts, use sets, and execution settings across runs.
4. What is the token cost of each workflow?
Both repair rows count the shared Visible starting stage (6.593M tokens) once. Their added stages use 2.878M and 2.376M tokens. In the separate experiment starting from identical files, detailed feedback repaired 31 more failures for 1.01% more tokens in the repair stage.
5. How might prior exposure to Mathlib affect the comparisons? Within a target, we use the same model and Mathlib version across methods. Absolute completion rates can still reflect pretraining, and the two historical designs may differ in familiarity. The 29/60 and 60/60 figures calibrate the measure on those five changes.
6. How does FUG relate to previous library-learning and theorem-proving work? ReGAL, TroVE, and LATM study refactoring using execution results and reusable tools. Stitch and LILO learn program libraries through compression and synthesis. TacMiner discovers tactic libraries. FUG evaluates whether a faithful Lean statement supports held-out constructions and proofs. The historical calibration also shows that a shorter statement does not automatically provide better support for later uses.
7. How does repair with downstream feedback relate to work on self-correction? Huang et al. (2024) study revision without external feedback. Kamoi et al. (2024) distinguish settings with reliable external feedback. Here both arms receive downstream results on the same 84 failures. Binary feedback repairs 35, detailed feedback 66. Both compile 300/300 candidates. At least 26 of the 31 additional completions come from checks of meaning and use. GPT-5.6-sol gives both the feedback and the final semantic judgment.
8. What do the held-out proofs and cases of mathematical reuse show beyond repair? Across seven targets, initial statements gave 167/630 held-out proof completions and binary-feedback endpoints gave 189/630 (+3.49 percentage points). Two endpoints retained the initial statement. On the separate observed uses, FUSE-C and FUSE completed 1308/6000 and 1119/6000 before checking proof dependencies, and 899/6000 and 901/6000 after that check. Some configurations share candidates. In the Ruzsa case, several branches reused the same conditional construction and function of a reference pair. Before the check, FUSE-C and FUSE completed 15/50 and 22/50 uses. After it, they completed 8/50 and 22/50. In the group presentation case, a later proof used the shared presentation and a quotient isomorphism to transport the result.