- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TACO uses a fixed 27B conjecturer to generate training problems for a 2B prover. Separate prover copies are trained on candidate problem sets with GRPO, and their target-set performance determines prover updates and subsequent curriculum seeds. Across four domain-specific runs, held-out pass@32 coverage increases from 10/128 to 43/128 problems.
Critical strengths
- Selecting generated problems by their measured effect on prover performance is well motivated. It connects curriculum construction directly to the target learning objective rather than relying only on difficulty or relevance heuristics.
- The improvements extend to held-out problems in all four domains. The random-selection ablations also provide useful evidence about the contribution of curriculum selection.
Critical weaknesses
- The main improvement is measured against the initial SFT model rather than a comparable training baseline. In Table 2, random iterative seed selection with aligned initial seeds already solves 38/128 problems, compared with 43/128 for TACO. Every ablation retains target-based selection of the best prover. The experiments therefore do not isolate the value of the proposed selection mechanism from synthetic-data training and checkpoint selection.
- Appendix E evaluates each candidate curriculum after one training epoch. This favors immediate improvement, which need not identify the best longer-term curriculum. Ranking stability across training horizons is not tested, and one run per configuration leaves selection variability unresolved.
Questions for authors
- Does TACO outperform pooled or randomly selected synthetic curricula using the same 27B generator and total compute budget, without selecting among separately trained prover copies?
- How do curriculum rankings change with longer training horizons? Please report learning curves for fixed candidate sets and repeat the main comparison with independent runs.
3 comments
Thank you for the new experiments and the revised manuscript. The pooled single-Prover baseline in Appendix F.1 addresses an important part of my original question. On Radical Algebra, the gap of 16/32 versus 3/32 is substantial and provides useful evidence for the full TACO procedure on that domain.
However, after considering the rebuttal, I am leaning toward keeping my original rating. The new baseline strengthens the paper, but I am still unsure how much of the gain comes from target-based Prover selection and whether it holds under a comparable end-to-end compute budget.
My main remaining question is what drives this gap and how much it costs. The comparison matches the number of training conjectures and rollouts, but the methods differ in seed selection, which Prover updates are retained, and some GRPO settings. Could you report the end-to-end compute for both runs, including target-set evaluation? A comparison that holds the candidate-training procedure and seed-selection rule fixed while varying only the Prover-update rule would help isolate the value of target-based Prover selection. Table 2 already gives a useful comparison of seed-selection rules under a shared Prover-update rule; the aggregate coverage difference there is 43/128 versus 38/128.
The additional Geometry: Area run is also helpful. Both runs improve on the initial Prover, although their final coverage and per-attempt success rates differ. The resampling analysis in Appendix F.7 addresses sensitivity to which target problems are included. I would be interested in the planned three-epoch results, particularly whether the candidates favored after one epoch remain competitive after three.
Thank you for this suggestion. We evaluated all 16 candidate Provers after one, two and three epochs in the first two iterations of an additional Geometry: Area run. More epochs do not consistently improve average target performance: the mean score (m=12) decreases from 0.792 to 0.654 in the first iteration, while increasing from 1.709 to 2.114 in the second. One possible explanation for the declines is that repeated updates on only 64 conjectures encourage over-specialisation to those problems, reducing transfer to the target set. The candidates ranked highest after one epoch do not consistently remain the strongest after three.
What is the end-to-end compute, including target-set evaluation? Appendix E estimates approximately 60 GPU-hours per ten-iteration TACO run, including conjecture generation, training all candidate Provers, their target-set evaluations, and seed and Prover selection on one NVIDIA GH200 GPU. For the pooled Radical Algebra baseline, Slurm accounting records 40.73 GPU-hours. This measured total includes initialisation, generation, training, target/test evaluations, and failed or restarted jobs, while excluding queue waiting.
Thank you for the very thoughtful review! Your feedback was very helpful in improving our manuscript, and we are glad you found that selecting generated problems by their measured effect on prover performance is well motivated. We address your concerns and questions below and have updated our paper accordingly, marking the changes in blue for easy detection. If you have further concerns, please let us know. Otherwise, we would sincerely appreciate it if you would consider updating your review in light of our responses below.
Q1 / W1. How does TACO compare to training a single Prover on pooled conjectures?
We ran this comparison on the Radical Algebra domain, using the same initial Prover and fixed Conjecturer. At each iteration, we generate 64 conjectures from each of the 16 seeds and train a single Prover on all 16 x 64 = 1,024 conjectures using GRPO. We sample the 16 seeds for the next iteration uniformly from eligible conjectures: those proved in 1-7 of 8 attempts. We repeat this process for 10 iterations, retaining every update. The GRPO optimization settings were slightly adjusted for stable training.
Both methods have a training budget of 10,240 conjectures and 81,920 proof attempts over 10 iterations. TACO solves more problems and produces more successful attempts. These results suggest that its selection process brings benefits beyond training a single Prover on all generated conjectures. One possible explanation is that the pooled Prover retains updates that may not benefit the target set, whereas TACO retains only updates that improve its measured target score.
These results and the experimental protocol have been added to Appendix F.1.
Q2a / W2. How do curriculum rankings change with longer training?
We are currently running an experiment on the Geometry: Area domain using all 16 candidate sets generated from the 16 seeds of the first iteration. Each set is used to train a copy of the same initial Prover for three epochs. All 16 candidate Provers are evaluated after epochs 1, 2 and 3 to compare the target-score curves and rank correlations. We will report the results as soon as they are available.
Q2b / W2. Do improvements recur across training runs?
We compare the original TACO run with one additional run on the Geometry: Area domain, using target-aligned seeds and the same initial models and target and test sets. We compare results after 10 iterations.
Both runs improve over their initial Prover evaluations, while their final coverage and number of successful attempts differ. This illustrates variability across runs: conjecture generation, GRPO training, and seed selection all involve randomness, and their effects can accumulate across iterations.
These results and the experimental protocol have been added to Appendix F.5.