- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AstroMathBench, a benchmark designed to evaluate the mathematical and computational reasoning capabilities of large language models (LLMs) in astronomy. The benchmark comprises 554 expert-verified seed problems across five task categories: Formula Derivation (FD), Formula Calculation (FC), Unimodal Computation (UC), Multimodal Computation (MC), and Code-based Calculation (CC). To mitigate data contamination, the authors propose a code-based mutation framework that generates numerical variants of existing problems while preserving their underlying reasoning structures. Additionally, the benchmark incorporates five cognitive evaluation dimensions: problem comprehension, condition robustness, knowledge consistency, reverse reasoning, and anti-interference.
The authors evaluate ten LLMs, including general-purpose and astronomy-specialized models, under different prompting and reasoning settings. The evaluation combines final-answer accuracy with process-level scoring using an LLM-as-a-judge framework. Experimental results indicate that even the strongest models exhibit limitations in astronomical computation, with the highest reported accuracy reaching 76%. The authors further validate their evaluation framework through contamination tests, human evaluation of 301 sampled responses, comparisons across multiple LLM judges, and stability experiments on generated question variants. The results highlight differences in model performance across task categories and reasoning settings, as well as challenges in numerical accuracy, domain-specific reasoning, and robustness.
Critical strengths
The paper focuses on mathematical and computational reasoning in astronomy, which requires integrating domain knowledge, physical principles, mathematical operations with astronomical constants, coordinate systems, celestial mechanics and numerical computations. Unlike existing astronomy benchmarks that primarily emphasize conceptual understanding and auxiliary skills, AstroMathBench targets computational reasoning across five task categories. This is a meaningful contribution, as it provides a more specialized framework for assessing LLM capabilities in scientific problem-solving.
The authors present a systematic benchmark construction approach and multidimensional evaluation. The authors curate 554 seed problems through a multi-stage annotation and expert-verification process and introduce code-based parameter perturbations to generate question variants. The inclusion of five cognitive evaluation dimensions, alongside conventional accuracy and process-level metrics, enables a more detailed examination of model behavior beyond final-answer correctness. I also appreciate the additional human evaluation, contamination analysis, and variant-stability experiments on the benchmark.
Critical weaknesses
A central contribution of the paper is its code-based mutation framework, which is claimed to mitigate data contamination while preserving problem difficulty. However, changing numerical constants might not necessarily eliminate contamination, since models may have previously encountered the original problems and memorized their underlying solution strategies. The evidence supports mitigation of exact-instance memorization, but not the stronger assertion that the benchmark is uncontaminated.
There are few other concerns with the evaluation. Multimodal computation is evaluated on a relatively small subset, with only 28 multimodal problems shown in the dataset distribution. It is difficult to support broad claims about visual reasoning from such limited coverage. The experimental evaluation is not equally comprehensive across the five tasks. Table 1 provides detailed results under multiple prompting settings for FD, FC, and UC, whereas Table 2 reports only selected metrics for MC and CC. Furthermore, CC is evaluated only for UC and FC, while MC is evaluated only for FD and UC, without sufficient justification for these omissions.
Questions for authors
What proportion of the benchmark comes from already-public Olympiad, textbook, and benchmark problems? Can the authors compare models on original problems, numerical variants, and structurally novel problems?
The paper presents the following instruction - "But, if you think the problem is not unsolvable, just return ‘Unsolvable‘." The wording instructs the model to output Unsolvable when it believes a problem is solvable. This seems incorrect.
We appreciate the reviewer’s suggestion.
To W1: We clarify that our code-based mutation involves both numerical substitutions and variable-name modifications. Preserving the original reasoning structure is intentional, as arbitrary structural changes could introduce uncontrolled difficulty variations and compromise benchmark fairness.
We acknowledge that exposure to similar problem templates during pre-training cannot be entirely ruled out. Nevertheless, no model encounters an identical problem instance more than once during evaluation. Moreover, approximately 80 problems (14.4%) were independently constructed, while 294 (53.1%) lacked valid solution procedures in their original benchmark sources and required independent annotation or correction, reducing the likelihood of direct solution memorization.
We further report accuracy comparisons between original problems and their generated variants (Section 3.6, Table 6). The largely consistent results suggest reasonable reasoning stability under controlled numerical and variable-name variations. However, we emphasize that this consistency does not establish the absence of pre-training contamination; rather, it supports the effectiveness of our mutation strategy in assessing reasoning stability while maintaining comparable difficulty.
These design considerations and limitations have been clarified in the revised manuscript (Section 3.4 and Section 3.6).
To W2: We acknowledge that the evaluation protocols are not fully uniform, as they are intentionally tailored to the characteristics of each task.
First, FD, FC, and UC are evaluated under multiple prompting settings to examine model performance and prompting sensitivity, whereas MC and CC report the key metrics most relevant to their objectives. AstroMathBench aims to provide a high-quality assessment of astronomical reasoning, rather than exhaustively enumerate results across all possible settings; additional evaluations remain open to users.
Second, the evaluation coverage reflects methodological applicability. CC emphasizes code-assisted logical reasoning and numerical computation, which is not necessarily suitable for FD's formula-derivation objective. Similarly, MC involves multimodal computational reasoning, whereas FC relies on established astronomical formulas and given physical quantities, making multimodal evaluation less relevant.
We will clarify these task-specific rationales in the revised manuscript (Section 3.1 EXPERIMENTAL SETUP and Appendix N Prompts in AstroMathBench Evaluation) to avoid ambiguity regarding the evaluation design.
To Q1: We agree that clarifying data provenance and examining model performance across different problem variants are important for assessing benchmark reliability.
(1) Data provenance. We have clarified the benchmark composition in the revised Appendix. Specifically, the 554 problems comprise 88 Olympiad problems (15.9%), 92 textbook problems (16.6%), 294 existing benchmark problems (53.1%), and 80 self-built problems (14.4%). Notably, the existing benchmark problems underwent manual verification and correction, as many originally lacked solutions or contained incomplete or incorrect answers. We have also added a detailed breakdown across ten astronomical subfields. Since some problems span multiple subfields, these counts are not mutually exclusive. The corresponding revisions are provided in Appendix D (DATASET INFORMATION).
(2) Original problems versus numerical variants. We have extended Table 5 to include results on original problems and their generated numerical variants. All five models achieve 2–7 percentage points higher accuracy on the variants, indicating no observable performance degradation under controlled numerical perturbations. However, this does not rule out potential training-data contamination, as the underlying reasoning structures remain largely unchanged. The corresponding revisions are provided in Section 3.6 (Table 6).
(3) Structural novelty and benchmark fairness. We considered the possibility of introducing structurally novel problems as an additional evaluation setting. However, independently constructed problems may differ substantially in domain knowledge requirements, computational complexity, and reasoning difficulty. Without a rigorous difficulty-calibration procedure, direct comparisons across such problem sets could introduce confounding factors and compromise the fairness and validity of benchmark evaluation. We therefore retain the existing benchmark design rather than introducing a structurally novel subset that cannot be reliably calibrated against the original problems.
To Q2: Thanks for the suggestions. We has corrected them in the revised manuscript.