- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies a trade-off: prioritising a predicted tactic family can make the first suggestion more likely to succeed but the shortlist less useful. Rankers are judged by candidate exposure, meaning whether the top-k list contains any tactic Lean accepts, rather than by matching the recorded human tactic. Candidates are training records retrieved by TF–IDF cosine similarity. A logistic-regression family prediction enters either as a bounded additive bonus (soft prior) or as a strict family ordering (hard routing).
Contributions. A bonus of weight λ preserves within-family order and reverses only pairs whose similarities differ by at most λ. This gives a sufficient top-k retention condition under any family prediction (Thms 1–2, Cor. 3). Theorem 4 gives the exact worst-case rank over all family distributions. Under hard routing, a candidate is displaced by the full size of every family ranked ahead of it (Prop. 5).
Validation. On 3,723 steps from 1,702 mathlib4 theorems, hard routing halves trace exact@5 (0.169 → 0.082). The fixed top-5 lists were re-scored on 497 states with corrected Lean outcomes. Accept@5 is 0.412 (unguided), 0.408 (soft) and 0.342 (hard; −7.0 pp, 95% CI [−11.8, −2.6]). Hard routing leads at k = 1 but trails by k = 3.
Critical strengths
None
Critical weaknesses
Missing LLM baseline. The only policy is TF–IDF/cosine retrieval plus a logistic-regression family classifier. Similarity-based tactic selection has precedents (e.g., TacticToe’s k-NN) but is uncommon in contemporary provers. The classifier is also weak: Table 1’s hard-routing Family@5 of 0.13 implies its top prediction is right on at most 13% of states. An easy, ICLR-relevant baseline is missing: an LLM (an older open model would do) as the family predictor in Eq. (3), or as a reranker. Since the bounds hold for every prediction, this would show how far realistic behaviour sits from the worst case. The paper never measures that gap, because the retention conditions are not computed on data (App. G). Why is such a baseline absent, when App. G mentions later experiments with larger classifiers and proof search?
Limited theoretical novelty. The theory specialises known geometry for ranking with uncertain linear scoring functions, a literature the paper does not cite. The soft score is a weighted sum with weights (1, λp), so a rival overtakes a candidate only when the weights cross the pair’s switching hyperplane. Thm 2 and Cor. 3 parallel the stability radius of Soliman et al. (SIGMOD 2011). Bounding the worst-case rank by k asks whether the family simplex lies inside the record’s top-k region (Tang, Mouratidis & Yiu, SIGMOD 2017). Thm 1 and Props 5–6 follow directly from the definitions; only Thm 4’s closed form goes beyond this. The paper itself concedes that its invariance and local bounds apply to any bounded group-constant bonus (Sec. 6). Changing the domain to Lean may add empirical value, but it does not establish theoretical novelty.
Questions for authors
None