acceptodds
Withdrawn from ICLR 2027

AstroAlertBench: Evaluating the Accuracy, Reasoning, and Honesty of Multimodal LLMs in Astronomical Classification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Modern astronomical observatories generate a massive volume of multimodal data, creating a critical bottleneck for expert human review. While multimodal Large Language Models (LLMs) have shown promise in interpreting complex visual and textual inputs, their ability to perform specialized scientific classification while providing interpretable reasoning remains understudied. We introduce AstroAlertBench, a comprehensive multimodal benchmark designed to evaluate LLMs’ performance in astronomical event review along a three-stage logical chain: metadata grounding, scientific reasoning, and hierarchical classification over five categories. We utilize a pilot sample of 1,500 real-world alerts from the Zwicky Transient Facility (ZTF), a wide-field survey that scans the northern sky to detect transient astronomical events. On this dataset, we benchmark 13 frontier closed-source and open-weight LLMs supporting visual input. Our results reveal that high accuracy does not always align with model rationale self-assessment, impacting its reliability as a real-world assistant. Together, AstroAlertBench provides a framework for developing calibrated and interpretable astronomical assistants.

Primary area:
datasets and benchmarks
Keywords:
multimodal large language models, scientific benchmarking, astronomical classification, scientific reasoning, model calibration, model honesty, time-domain astronomy

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_9bphOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is focused on the use of (M)LLMs to address the following problem from astronomy: classification of transient astronomical events detected by scans of the sky. In particular, the authors are interested on the scans of northern sky from the Zwicky Transient Facility (ZTF). The authors mention that, given the large amount of data, the classification cannot be done manually but already exist methods that are based on machine learning, which can perform the classification with high accuracy. The weakness of these methods stems from the fact that they do not provide the reasoning leading to the reported classification results. In order to address this, the authors construct AstroAlertBench that contains 1500 real-world alerts from ZTF and use it to evaluate (M)LLMs in a three-stage logical chain: Metadata Grounding, Scientific Rationale, and Staged Classification. The results of this evaluation are discussed in detail in the paper.

Critical strengths

S1. The benchmark can be useful for other researchers working in the field.

S2. The idea of having and interpretable astronomical assistant is highly valuable.

Critical weaknesses

W1. The methodological contribution of the paper is limited, but the application is important. This makes the paper more suitable for publication in an astronomy journal/conference, where the audience can really appreciate if the explanations on how the classification was performed are more important than the actual classification results. In its current form, the paper does not indicate the level of classification accuracy expected to be achieved so that the system will be used in practical applications and if that level of accuracy is indeed achieved by the model configurations which were evaluated.

W2. In the experiments, the evaluated models are asked to self-assign quality scores (within the same prompt step). As this approach potentially suffers from self-favouring bias, it can lead to score inflation.

W3. According to the definition provided in the paper, the metadata grounding consists in requiring the models to map the raw numerical parameters to their scientific definitions using a mapping key provided in the prompt. According to the reported results, all 13 model configurations achieve 100% accuracy on this subtask, which means that the subtask does not provide any diagnostic differentiation.

Questions for authors

Q1. Taking into consideration that the ZTF alert data are publicly available, the authors should provide complete information about the specific procedures conducted to check for data contamination in the pretraining corpora. How sensitive are the benchmark rankings to alternative prompt templates or few-shot setups?

Q2. (related to W2) Why the authors did not adopt an independent LLM-as-a-Judge protocol or a blind expert human sampling as the main evaluation mechanism for scientific reasoning?

Q3. (related to W3) How do the authors plan to make the perception grounding more challenging?

Reviewer_VnSdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper constructed AstroAlertBench that converts real world data of astronomy alert classification into evaluation following three directions: metadata grounding, scientific rationale generation and self-assessment, and hierarchical classification. They evaluate 13 MLLMs including open-source ones and closed-source ones, and analyze classification performance, self-assessment reliability, and second-pass error correction.

Critical strengths

1. Evaluation of multiple MLLMs on a specialized scientific task.

The paper provides a systematic comparison of 13 MLLM configurations on astronomical alert classification, and the benchmark is created from real world data which is closely related to applications in this field.

2. The analysis and observation of self-score VS. true accuracy is interesting but can be further developed.

The paper instructs MLLMs to score its own reasoning and finds an inverse relation between the model's true accuracy in classification comparing to its self-score. Demonstrating that models that are having higher accuracy scores tends to score themselves lower using the same criteria. It would be more interesting if authors can allow them to cross-evaluate each others' reasoning traces to find out if the inverse relation came from a tendency to score everyone lower for better-performance models.

Critical weaknesses

1. Limited novelty in benchmark construction and the evaluation design.

The paper is essentially constructing the benchmark from existing data and just converting them into three parts of an evaluation prompt with data selection and balancing and preprocessing. Reusing existing data is not necessarily an inherent problem for novelty, but the paper's claim over the three evaluation directions are not established the same as the authors claimed to be, which is detailed illustrated in weakness 2.

2. The three evaluation contributions in this paper are not adequately evaluating their claimed aspects of the MLLMs.

Part A metadata grounding:

The task is essentially inputting the metadata with its raw names from the original data and then specify the naming mapping with real concepts in the prompt, and ask the model to output the mapped correct concepts for the input data. This task is essentially testing the model's ability to copy data and apply the mapping provided in the prompt. This does not require substantial understanding in astronomy, and it is unclear why it should be an important stage in the evaluation design of this benchmark.

Part B scientific rationale generation and self-assessment:

In part b questions the paper asks models to generate scientific rationales behind the final 5-way classification. This is essentially a big contribution claimed by the paper to distinguish itself from just constructing the 5-way classification task. But the main quantitative analysis then relies on scores assigned by the models themselves. Section 5 investigates the relationship between these self-ratings and classification correctness, but this measures the reliability of model self-assessment rather than directly evaluating the scientific correctness of the generated explanations. And the analysis would be benefited more if the models can cross-evaluate each other on the same set of reasoning traces to reveal possible bias/tendency in scoring rationales. The authors did provide one alert example of expert evaluation in the appendix, but this is very limited comparing to the 1500 examples in this benchmark. Essentially, the performance of the models are still judged through the classification accuracy and the self-scores, not necessarily from the scientific reasoning quality. This deviates from the title's claim of "Evaluating the Accuracy, Reasoning, and Self-assessment(Honesty? Mismatch title wording) of Multimodal LLMs".

Part C staged classification:

The authors instruct the models to output different classification answers in the prompt with one call to the model. So the staged output is generated within one single model response. This does not establish whether the model is actually following this hierarchy internally. Although the authors report stage-wise classification accuracy, the additional value of explicitly requiring staged outputs is still unclear, as similar stage-level analyses could be derived from direct five-class predictions. The paper also lacks a controlled comparison demonstrating why this staged design should be considered a distinct evaluation contribution.

Questions for authors

No additional questions. My main concerns are detailed in the Critical Weaknesses section.

Reviewer_YBKMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces AstroAlertBench, a benchmark of 1,500 balanced ZTF alerts for evaluating 13 multimodal LLM configurations. It combines image cutouts and alert metadata to assess metadata grounding, scientific rationale, hierarchical classification, and model self-assessment. The best reported end-to-end five-class accuracy is 60.6%.

Critical strengths

The task is relevant, and combining astronomical image cutouts with structured alert metadata provides a useful setting for evaluating multimodal models. The public benchmark and prompts could support reproducible comparisons.

Critical weaknesses

The main concern is the reliability of the evaluation. Appendix C.3 defines Stage-3 conditional accuracy as the accuracy on examples where Stages 1 and 2 are both correct. However, the values in Table 1 appear to match the mean end-to-end class accuracy over supernovae, AGNs, and variable stars in Table 9 for several models (e.g., GPT-5.4 high-think: 47.11%). This raises a substantial concern about whether the metric is implemented as defined. Furthermore, Appendix D uses an independence-based approximation for reasoning-mode comparisons on the same alerts, while truncated outputs in the Qwen thinking configurations further confound the comparison. Consequently, the reported conclusions about reasoning-mode effectiveness are not sufficiently supported.

The benchmark also provides limited evidence for the broader claims about real-world reliability. It uses a balanced, high-confidence sample of 1,500 alerts, and the supernova labels are derived from ALeRCE's high-confidence broker stream rather than an independently verified reference. Moreover, the rationale self-assessment results receive limited independent validation, with the human-grading comparison covering only one representative alert. These limitations weaken the claims about generalization and the reliability of model self-assessment.

Questions for authors

Can the authors clarify the exact implementation and denominator of Stage-3 conditional accuracy, and report paired statistical comparisons with truncated or invalid outputs analyzed separately? These clarifications could materially affect confidence in the main findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.