- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is focused on the use of (M)LLMs to address the following problem from astronomy: classification of transient astronomical events detected by scans of the sky. In particular, the authors are interested on the scans of northern sky from the Zwicky Transient Facility (ZTF). The authors mention that, given the large amount of data, the classification cannot be done manually but already exist methods that are based on machine learning, which can perform the classification with high accuracy. The weakness of these methods stems from the fact that they do not provide the reasoning leading to the reported classification results. In order to address this, the authors construct AstroAlertBench that contains 1500 real-world alerts from ZTF and use it to evaluate (M)LLMs in a three-stage logical chain: Metadata Grounding, Scientific Rationale, and Staged Classification. The results of this evaluation are discussed in detail in the paper.
Critical strengths
S1. The benchmark can be useful for other researchers working in the field.
S2. The idea of having and interpretable astronomical assistant is highly valuable.
Critical weaknesses
W1. The methodological contribution of the paper is limited, but the application is important. This makes the paper more suitable for publication in an astronomy journal/conference, where the audience can really appreciate if the explanations on how the classification was performed are more important than the actual classification results. In its current form, the paper does not indicate the level of classification accuracy expected to be achieved so that the system will be used in practical applications and if that level of accuracy is indeed achieved by the model configurations which were evaluated.
W2. In the experiments, the evaluated models are asked to self-assign quality scores (within the same prompt step). As this approach potentially suffers from self-favouring bias, it can lead to score inflation.
W3. According to the definition provided in the paper, the metadata grounding consists in requiring the models to map the raw numerical parameters to their scientific definitions using a mapping key provided in the prompt. According to the reported results, all 13 model configurations achieve 100% accuracy on this subtask, which means that the subtask does not provide any diagnostic differentiation.
Questions for authors
Q1. Taking into consideration that the ZTF alert data are publicly available, the authors should provide complete information about the specific procedures conducted to check for data contamination in the pretraining corpora. How sensitive are the benchmark rankings to alternative prompt templates or few-shot setups?
Q2. (related to W2) Why the authors did not adopt an independent LLM-as-a-Judge protocol or a blind expert human sampling as the main evaluation mechanism for scientific reasoning?
Q3. (related to W3) How do the authors plan to make the perception grounding more challenging?