acceptodds
Under review as a conference paper at ICLR 2027

AstroMathBench: Evaluating the Mathematical Reasoning Capabilities of LLMs for Astronomical Problems

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in advanced mathematical reasoning tasks. However, computational reasoning in astronomy with LLMs presents unique challenges yet has received much less attention. Existing benchmarks often fail to capture LLMs' cross-disciplinary reasoning abilities and are prone to data contamination. To address these problems, we propose AstroMathBench, a benchmark for evaluating LLMs' computational reasoning capabilities in astronomy. AstroMathBench covers five task types in astronomical reasoning: formula calculation, formula derivation, unimodal computation, multimodal computation, and code-based calculation. Beyond task-level evaluation, we decompose the reasoning process into five cognitive dimensions, enabling fine-grained assessment of LLMs' capability boundaries. Moreover, we design a cross-validation-based pipeline to mitigate data contamination by separating private and public data and employing isomorphic variant generation. The resulting dataset contains 554 seed problems that have been manually verified and checked for contamination, with 90.1% fully code-formalized. Our evaluation of 10 leading LLMs finds that the highest accuracy reaches only 76%, highlighting the significant challenges of computational reasoning in astronomy and the need for LLMs with stronger cross-disciplinary reasoning abilities. In general, we hope that AstroMathBench will drive advances in AI for astronomy.

Primary area:
datasets and benchmarks
Keywords:
benchmarking, automatic creation and evaluation of language resources, evaluation methodologies, evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_M9a2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AstroMathBench, a benchmark designed to evaluate the mathematical and computational reasoning capabilities of large language models (LLMs) in astronomy. The benchmark comprises 554 expert-verified seed problems across five task categories: Formula Derivation (FD), Formula Calculation (FC), Unimodal Computation (UC), Multimodal Computation (MC), and Code-based Calculation (CC). To mitigate data contamination, the authors propose a code-based mutation framework that generates numerical variants of existing problems while preserving their underlying reasoning structures. Additionally, the benchmark incorporates five cognitive evaluation dimensions: problem comprehension, condition robustness, knowledge consistency, reverse reasoning, and anti-interference.

The authors evaluate ten LLMs, including general-purpose and astronomy-specialized models, under different prompting and reasoning settings. The evaluation combines final-answer accuracy with process-level scoring using an LLM-as-a-judge framework. Experimental results indicate that even the strongest models exhibit limitations in astronomical computation, with the highest reported accuracy reaching 76%. The authors further validate their evaluation framework through contamination tests, human evaluation of 301 sampled responses, comparisons across multiple LLM judges, and stability experiments on generated question variants. The results highlight differences in model performance across task categories and reasoning settings, as well as challenges in numerical accuracy, domain-specific reasoning, and robustness.

Critical strengths

The paper focuses on mathematical and computational reasoning in astronomy, which requires integrating domain knowledge, physical principles, mathematical operations with astronomical constants, coordinate systems, celestial mechanics and numerical computations. Unlike existing astronomy benchmarks that primarily emphasize conceptual understanding and auxiliary skills, AstroMathBench targets computational reasoning across five task categories. This is a meaningful contribution, as it provides a more specialized framework for assessing LLM capabilities in scientific problem-solving.

The authors present a systematic benchmark construction approach and multidimensional evaluation. The authors curate 554 seed problems through a multi-stage annotation and expert-verification process and introduce code-based parameter perturbations to generate question variants. The inclusion of five cognitive evaluation dimensions, alongside conventional accuracy and process-level metrics, enables a more detailed examination of model behavior beyond final-answer correctness. I also appreciate the additional human evaluation, contamination analysis, and variant-stability experiments on the benchmark.

Critical weaknesses

A central contribution of the paper is its code-based mutation framework, which is claimed to mitigate data contamination while preserving problem difficulty. However, changing numerical constants might not necessarily eliminate contamination, since models may have previously encountered the original problems and memorized their underlying solution strategies. The evidence supports mitigation of exact-instance memorization, but not the stronger assertion that the benchmark is uncontaminated.

There are few other concerns with the evaluation. Multimodal computation is evaluated on a relatively small subset, with only 28 multimodal problems shown in the dataset distribution. It is difficult to support broad claims about visual reasoning from such limited coverage. The experimental evaluation is not equally comprehensive across the five tasks. Table 1 provides detailed results under multiple prompting settings for FD, FC, and UC, whereas Table 2 reports only selected metrics for MC and CC. Furthermore, CC is evaluated only for UC and FC, while MC is evaluated only for FD and UC, without sufficient justification for these omissions.

Questions for authors

What proportion of the benchmark comes from already-public Olympiad, textbook, and benchmark problems? Can the authors compare models on original problems, numerical variants, and structurally novel problems?

The paper presents the following instruction - "But, if you think the problem is not unsolvable, just return ‘Unsolvable‘." The wording instructs the model to output Unsolvable when it believes a problem is solvable. This seems incorrect.

1 comment

We appreciate the reviewer’s suggestion.

To W1: We clarify that our code-based mutation involves both numerical substitutions and variable-name modifications. Preserving the original reasoning structure is intentional, as arbitrary structural changes could introduce uncontrolled difficulty variations and compromise benchmark fairness.

We acknowledge that exposure to similar problem templates during pre-training cannot be entirely ruled out. Nevertheless, no model encounters an identical problem instance more than once during evaluation. Moreover, approximately 80 problems (14.4%) were independently constructed, while 294 (53.1%) lacked valid solution procedures in their original benchmark sources and required independent annotation or correction, reducing the likelihood of direct solution memorization.

We further report accuracy comparisons between original problems and their generated variants (Section 3.6, Table 6). The largely consistent results suggest reasonable reasoning stability under controlled numerical and variable-name variations. However, we emphasize that this consistency does not establish the absence of pre-training contamination; rather, it supports the effectiveness of our mutation strategy in assessing reasoning stability while maintaining comparable difficulty.

These design considerations and limitations have been clarified in the revised manuscript (Section 3.4 and Section 3.6).

To W2: We acknowledge that the evaluation protocols are not fully uniform, as they are intentionally tailored to the characteristics of each task.

First, FD, FC, and UC are evaluated under multiple prompting settings to examine model performance and prompting sensitivity, whereas MC and CC report the key metrics most relevant to their objectives. AstroMathBench aims to provide a high-quality assessment of astronomical reasoning, rather than exhaustively enumerate results across all possible settings; additional evaluations remain open to users.

Second, the evaluation coverage reflects methodological applicability. CC emphasizes code-assisted logical reasoning and numerical computation, which is not necessarily suitable for FD's formula-derivation objective. Similarly, MC involves multimodal computational reasoning, whereas FC relies on established astronomical formulas and given physical quantities, making multimodal evaluation less relevant.

We will clarify these task-specific rationales in the revised manuscript (Section 3.1 EXPERIMENTAL SETUP and Appendix N Prompts in AstroMathBench Evaluation) to avoid ambiguity regarding the evaluation design.

To Q1: We agree that clarifying data provenance and examining model performance across different problem variants are important for assessing benchmark reliability.

(1) Data provenance. We have clarified the benchmark composition in the revised Appendix. Specifically, the 554 problems comprise 88 Olympiad problems (15.9%), 92 textbook problems (16.6%), 294 existing benchmark problems (53.1%), and 80 self-built problems (14.4%). Notably, the existing benchmark problems underwent manual verification and correction, as many originally lacked solutions or contained incomplete or incorrect answers. We have also added a detailed breakdown across ten astronomical subfields. Since some problems span multiple subfields, these counts are not mutually exclusive. The corresponding revisions are provided in Appendix D (DATASET INFORMATION).

(2) Original problems versus numerical variants. We have extended Table 5 to include results on original problems and their generated numerical variants. All five models achieve 2–7 percentage points higher accuracy on the variants, indicating no observable performance degradation under controlled numerical perturbations. However, this does not rule out potential training-data contamination, as the underlying reasoning structures remain largely unchanged. The corresponding revisions are provided in Section 3.6 (Table 6).

(3) Structural novelty and benchmark fairness. We considered the possibility of introducing structurally novel problems as an additional evaluation setting. However, independently constructed problems may differ substantially in domain knowledge requirements, computational complexity, and reasoning difficulty. Without a rigorous difficulty-calibration procedure, direct comparisons across such problem sets could introduce confounding factors and compromise the fairness and validity of benchmark evaluation. We therefore retain the existing benchmark design rather than introducing a structurally novel subset that cannot be reliably calibrated against the original problems.

To Q2: Thanks for the suggestions. We has corrected them in the revised manuscript.

Reviewer_HH5uOpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AstroMathBench, a benchmark built from 554 seed problems for evaluating LLMs’ astronomical computational reasoning across five task types: formula calculation, formula derivation, text-only computation, multimodal computation, and code-based calculation. It addresses the limitations of existing mathematical benchmarks in capturing the specific requirements of astronomy, aiming to provide an evaluation setting closer to real-world scientific computation. The paper also evaluates model behavior across five cognitive dimensions using input perturbations and reasoning-completion tasks, and analyzes the resulting performance differences and failure modes.

Critical strengths

Beyond introducing the benchmark and reporting model performance, the paper provides a detailed analysis across five cognitive dimensions and examines common failure modes. The study evaluates ten LLMs and includes human assessments, comparisons across evaluator models, and tests of evaluation stability across question variants.

Critical weaknesses

Although the paper outlines its intended contributions and discusses related benchmarks, a more explicit comparison of task coverage and evaluation design is needed to clarify its added value over existing astronomy benchmarks. Also, the strongest model evaluated here is GPT-5.6-Luna. Results for newer leading models, such as GPT-6.1-Sol, Opus-5.5, and DeepSeek-V4.1-Flash-1017, are not included. Consequently, it remains unclear how well the best currently available models perform on these tasks. By the way, in the conclusion, the benchmark is referred to as “AstroBench”, which is inconsistent with the name used elsewhere in the paper and appears to be a typographical error.

Questions for authors

I have no additional questions beyond the points raised above.

2 comments

To Critical Weaknesses: We sincerely thank the reviewer for these valuable suggestions. Regarding newer model evaluations, our manuscript was initially submitted in September 2026, and some most recent models, such as DeepSeek-V4.1-Flash-1017, were not available or suffered from restricted API access. Nevertheless, we will make efforts to evaluate these newer models and release the results through our GitHub repository when experiments are completed. Furthermore, we provide a tabular comparison in Appendix (Table 7, Extended Related Work) to explicitly highlight the differences between our benchmark and existing astronomy and math benchmarks in terms of task coverage and evaluation design. Finally, the inconsistent use of "AstroBench" has been corrected in the revised manuscript.

Comparison between AstroMathBench and Existing Benchmarks

BenchmarkAstronomical KnowledgeMathematical/Physical ReasoningAstronomical ComputationMLLM Visual UnderstandingCode ExecutionCognitiveAgent Planning and Execution
Astro-QA√√√————
AstroMLab 1√√—————
IOAA√√√√———
AstroMMBench√√—√———
AstroVisBench√√√√—√—
Astro2Method√√—————
AstroAgentBench√√√—√—√
GSM8K / MATH—√—————
Omni-MATH—√———√—
OlympiadBench√√√√———
SciBench√√√—√√—
TPBench√√√—√——
MathVista—√—√———
AgentBench—√—√√√√
CORE-Bench—√√√√√√
ScienceAgentBench√√√√√√√
AstroMathBench√√√√√√—

Thank you for the detailed response and the added comparison table, which clarifies the positioning of AstroMathBench relative to existing benchmarks. My concerns have been largely addressed.

Regarding the newer models, I apologize, as some of the models I mentioned were released after the submission deadline, and it would be unfair to expect them in the paper. My intention was simply to suggest that including stronger models, where possible, would better show how the benchmark challenges current state-of-the-art systems. I appreciate your plan to release these results.

Overall, I remain positive about this work and will keep my score.

Reviewer_9Ce9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AstroMathBench, a benchmark for evaluating LLMs' computational reasoning abilities in astronomy. The dataset contains 554 manually verified seed problems collected from existing benchmarks, astronomy Olympiads, textbooks, and expert-written questions. The problems cover five task types: formula calculation, formula derivation, text-only computation, multimodal computation, and code-based calculation. To reduce data contamination, the authors formalize most solutions as executable code and generate new problem instances by perturbing numerical constants. The seed dataset remains private, while the evaluation code and generated variants are publicly available. Inspired by CogMath, the paper also evaluates models along five cognitive dimensions: problem comprehension, condition robustness, knowledge consistency, reverse reasoning, and anti-interference. The authors evaluate 10 closed- and open-source models, with the highest reported accuracy reaching 76% on unimodal computation under a particular prompting setting. The results suggest that cross-disciplinary computational reasoning in astronomy remains challenging for current LLMs.

Critical strengths
  1. Clear motivation and practical significance: The paper addresses a relevant gap in evaluating LLMs' mathematical reasoning abilities in astronomy. Existing benchmarks focus more on general mathematical problems or domain-specific knowledge, leaving computational reasoning in astronomy relatively underexplored.
  2. Thorough data pipeline: Collection from multiple sources, filtering, step-level annotation, difficulty classification, and expert verification are all described clearly. About 90% of the problems are formalized as code, allowing new evaluation instances to be generated automatically.
  3. Comprehensive benchmark design: The benchmark covers five task types, including formula calculation, formula derivation, and multimodal reasoning. It also considers several cognitive dimensions, going beyond simply evaluating final answer accuracy. This provides a relatively broad perspective on LLMs' reasoning capabilities.
Critical weaknesses
  1. Methodological Novelty: The paper brings together several useful components into a comprehensive benchmark. However, Most components, including domain-specific task construction, input perturbations, and LLM-based evaluation, are based on existing approaches. It would be helpful to further clarify what distinguishes the proposed framework from existing benchmarks, beyond its application to astronomy.
  2. Data Contamination: The proposed code-based mutation mainly changes numerical values while preserving the original problem structure and reasoning steps. But since it mainly changes numerical values while preserving the original problem structure and reasoning steps, models may still benefit from having seen similar problem templates or solutions during training. It would be helpful to further discuss this limitation or provide additional evidence of the strategy's effectiveness.
  3. Evaluation Validity: The connection between five cognitive dimensions and their corresponding evaluation tasks could be better justified, as certain tests may also reflect robustness to input perturbations rather than the intended cognitive abilities. Further discussion or validation would help strengthen the evaluation framework.
  4. Interpretation of Results: The experimental results provide some interesting observations, but a few interpretations could benefit from further discussion. For example, the authors suggest that the lower accuracy on formula calculation (FC) compared to unimodal computation (UC) reflects limitations in symbolic reasoning, although differences in task difficulty or evaluation criteria might also contribute to this result. Further analysis would help better support these interpretations.
Questions for authors
  1. Mutation Strategy: How does the proposed code-based mutation strategy differ from existing approaches such as GSM-Symbolic? In particular, are there any astronomy-specific design considerations or advantages that distinguish the proposed method?
  2. Data Contamination: Since the code-based mutation mainly changes numerical values while preserving the original reasoning structure, could the authors report how model performance differs between the original problems and their generated variants? Would such a comparison provide additional insights into potential memorization or data contamination?
2 comments

We appreciate the reviewer’s suggestion.

To W1: We added a comparison table in Appendix (Table 7, Extended Related Work) to highlight AstroMathBench's distinctions from existing astronomy and related mathematical benchmarks in terms of task coverage and evaluation dimensions. We further highlight its methodological novelty in our response to Q1.

To W3: On the boundary between “cognitive dimensions” and “robustness tests”: We agree with the reviewer that settings such as word-order shuffling or conditional perturbations can, on the surface, also be viewed as robustness tests. However, we would like to emphasize that there is no strict boundary between cognitive evaluation and robustness evaluation; in many cases, the two are inherently coupled. From a cognitive science perspective, human problem comprehension fundamentally involves maintaining stable semantic representations under superficial changes in expression, such as syntactic or ordering variations. Therefore, our goal in designing sentence-order / word-order perturbations is not merely to test noise robustness, but to distinguish whether the model relies on surface patterns or truly forms abstract semantic representations. In other words, if a model degrades substantially under perturbations that preserve semantics, this is not only evidence of limited robustness, but also suggests that its problem understanding depends on shallow cues rather than semantic modeling, which in turn affects downstream reasoning. For this reason, we place these tests under Problem Comprehension rather than treating them as pure robustness evaluations. In the revision, we will further clarify the distinction between engineering robustness—stability to noisy inputs—and cognitive robustness—stability of representation under meaning-preserving transformations. Our focus in this work is on the latter.

To W4: We agree that the lower FC accuracy cannot be attributed solely to limitations in symbolic reasoning, as task difficulty, computational complexity, and evaluation criteria may also contribute. We have thus revised our discussion to frame this result as suggesting potential challenges in formula-based calculations, rather than definitive evidence of deficient symbolic reasoning. The relevant revisions have been incorporated into Section 3.2 of the revised manuscript.

More comprehensive error analyses are provided in Appendix G (Details of Error Analysis). Furthermore, as detailed in the Appendix (Examples of Generated Reports by Analysis Agent), we introduce an analysis agent, provide illustrative model-generated analyses, and release its implementation in our public GitHub repository to facilitate result interpretation and further investigation. These analyses are intended to characterize general performance patterns and potential limitations, rather than exhaustively explain the underlying reasoning mechanisms. More fine-grained interpretations remain dependent on model behaviors and evaluation contexts and warrant further investigation.

To Q1: We agree that both approaches generate isomorphic variants through numerical perturbations while preserving the underlying reasoning structure, and do not claim numerical perturbation itself as novel. Our contribution lies in adapting this principle to executable astronomical computations through physics-aware parameter sampling and execution-based verification.

Specifically, GSM-Symbolic relies on manually designed symbolic templates with predefined variable slots, sampling ranges, and mathematical constraints, generating variants primarily through variable substitution. In contrast, AstroMathBench represents eligible problems as executable functions or ordered computational procedures, perturbs numerical inputs, and re-executes them to recompute intermediate and final results.

This adaptation is essential in astronomy, where valid parameter ranges are governed by physical constraints rather than generic numerical bounds. For example, geographic latitude and stellar declination must lie within [-90°, 90°], whereas hour angle lies within [0°, 360°]. As detailed in Appendix B, we incorporate such domain-specific constraints into parameter sampling and validate candidate ranges using 20 random samples, retaining only those that produce successful execution and physically valid results.

We acknowledge that this mechanism is not universally applicable. Problems without independently perturbable numerical parameters (e.g., purely FD problems) retain their original forms. Overall, variants are generated for 90.1% of the 554 seed problems, while the remaining 9.9% remain unchanged.

Thus, our distinction from GSM-Symbolic is not numerical perturbation itself, but its adaptation to astronomical computations via executable representations, physics-aware sampling, and execution-based verification. We will clarify this distinction and acknowledge the related prior work in the revision. We have added two comparison tables in the Appendix (Table 7 and 8, Extended Related Work) to clarify.

To W2 and Q2: Our code-based mutation strategy involves both numerical substitutions and variable-name modifications. Preserving the original computational structure and reasoning steps is intentional, as arbitrary structural changes could introduce uncontrolled difficulty variations and undermine benchmark comparability.

Regarding data contamination, we acknowledge that exposure to similar problem templates during pre-training cannot be entirely ruled out. However, no model encounters an identical problem instance more than once during evaluation. Furthermore, 14.4% of the problems were independently constructed, while 53.1% lacked valid solutions in their original sources; we independently annotated or corrected these solutions, further reducing the risk of direct solution memorization.

To investigate this concern, we extended Table 5 with comparisons between original problems and generated variants in both accuracy and process-level scores. All five models perform slightly better on generated variants, with accuracy gains of 2–7 percentage points and process-score gains of 0.07–0.28. This indicates that controlled numerical and variable-name perturbations do not degrade model performance.

Importantly, these results do not establish the absence of pre-training contamination, since the preserved computational structures may still allow models to leverage previously learned knowledge and reasoning patterns. Rather, they provide complementary evidence of robustness to controlled problem variations under comparable difficulty. We have added these results to Table 6 and clarified the corresponding discussion and limitations in Section 3.4 (DATA POLLUTION TESTING) and Section 3.6 (EVALUATION STABILITY ON DIFFERENT QUESTION VARIANTS).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.