- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MLDerive, a benchmark of 1,292 machine learning problems that links textbook exercises to related variants. To assess complete derivations rather than final answers alone, it proposes an evaluator combining tool verification, checks of claim dependencies, and semantic adjudication. Experiments with six language models examine performance across tasks, knowledge areas, and related problem families, while human validation compares the evaluator with direct LLM judgment.
Critical strengths
-
The paper addresses the important problem of evaluating complete machine learning reasoning and contributes problems with reviewed reference solutions and links between source exercises and variants.
-
The experiments cover multiple models and reasoning tasks, and the human validation shows higher agreement for the proposed evaluator than for direct LLM judgment on the evaluated subset.
Critical weaknesses
-
The benchmark offers limited headroom for the strongest evaluated models, with three models exceeding 95% overall pass rate and the best reaching 96.83%. Although some task categories remain challenging, the high overall scores limit its ability to distinguish stronger models and measure further progress in ML reasoning. Including results from smaller or open models could better demonstrate the benchmark’s value across a broader range of model capabilities.
-
The dataset provides reference derivations, but assessing complete solutions requires a complex evaluation pipeline involving claim extraction, dependency analysis, and semantic adjudication, which introduces uncertainty into the judgments. Providing explicit numerical answers with tolerances, symbolic expressions with assumptions, and Lean 4 statements and proof scripts would offer more reliable verification materials. Related benchmarks such as FormalML (Yang et al., 2026), cited in Related Work, already use Lean 4 for formal proof subgoal completion in machine learning theory. Such annotations would support more reproducible automatic evaluation.
Questions for authors
-
How do smaller or open models perform on MLDerive, and what differences in reasoning capability does the benchmark reveal?
-
Could the dataset provide explicit numerical or symbolic answers and Lean 4 statements and proof scripts to support automatic verification?