acceptodds
Under review as a conference paper at ICLR 2027

Virtual Meteorologist’s Exam: Benchmarking Meteorological Expertise in Large Language Models

Abstract

Recently, the growing demand for virtual meteorologist assistants in operational scenarios has driven the incorporation of meteorological corpora into the training of large language models (LLMs). However, due to the scale and diversity, existing benchmarks fail to provide a comprehensive assessment of LLMs’ understanding and reasoning capabilities in meteorology. To this end, we introduce a benchmark, namely Virtual Meteorologist's Exam (VME), to evaluate whether existing large language models have professional-level meteorological expertise equivalent to that of humans. To be concrete, our VME comprises nearly 6,500 question–answer pairs across four formats, making it four times the size of comparable datasets. It covers both academic knowledge and operational scenarios. Meanwhile, it is also organized into 5 primary categories and 25 fine-grained subcategories, enabling a detailed analysis of existing models’ strengths and weaknesses across meteorological subdomains. Based on that, we employ numerous state-of-the-art large language models, either general-purpose or meteorologically specialized, to examine their capability ceilings and preferences. Our results reveal critical weaknesses in current LLMs across specific meteorological subdomains, particularly meteorological observation, indicating key directions for further improvement. We hope that VME will serve as a standardized testbed for tracking progress and guiding the development of meteorological LLMs.

Primary area:
datasets and benchmarks
Keywords:
Meteorology, Benchmark, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QVNBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a new meteorological dataset to evaluate the capabilities of different large models in the field of meteorology. It analyzes the limitations of existing meteorological datasets and proposes a more comprehensive dataset to address these limitations. The paper provides a detailed description of the dataset construction process and the characteristics of the final dataset. It also evaluates the performance of different models on the new dataset.

Critical strengths
  1. Evaluating the capabilities of large language models across different domains is indeed important, and each domain requires more specialized datasets for evaluation.

  2. The paper provides a detailed and clear description of both the dataset construction process and the characteristics of the final dataset.

  3. The paper evaluates a wide range of large language models on the newly constructed dataset and compares their performance.

Critical weaknesses
  1. Based on the information in Lines 143–144, the original data source of the proposed dataset appears to be examinations from China. I could not find any other data sources mentioned elsewhere in the paper, so I assume that all the data come from Chinese examinations. In my opinion, this is insufficient for a domain-specific evaluation dataset. Examinations in different countries may focus on different aspects of the field. Therefore, the authors should try to collect examination questions from more countries to enable a more objective evaluation of model performance.

  2. Although I do believe that evaluation datasets for specialized domains are important, I think the ICLR main conference places greater emphasis on innovations in learning methods and model mechanisms. While this paper compares and analyzes the performance of different models, it provides relatively limited insights into model characteristics or guidance on how to improve model performance. Therefore, I think this paper may be more suitable for another conference or journal.

Questions for authors

I would like the authors to address the concerns raised in the Weaknesses section, particularly regarding the specific sources of the data, how the objectivity of the dataset is ensured, and how this work could contribute to further advances in learning methods and model mechanisms.

1 comment
AuthorsOpenReview ↗

Response to Reviewer QVNB

W1: Based on the information in Lines 143–144, the original data source of the proposed dataset appears to be examinations from China. I could not find any other data sources mentioned elsewhere in the paper, so I assume that all the data come from Chinese examinations. In my opinion, this is insufficient for a domain-specific evaluation dataset. Examinations in different countries may focus on different aspects of the field. Therefore, the authors should try to collect examination questions from more countries to enable a more objective evaluation of model performance.

Response to W1: Thank you for raising this important concern. We would like to clarify that the primary objective of VME is to evaluate the domain-specific knowledge and professional competencies of LLMs in meteorology, rather than to comprehensively represent examination systems across different countries. First, Chinese meteorological examinations provide a valuable source of professionally grounded questions, covering various aspects of meteorology and relevant operational practices. Second, although examinations in different countries may emphasize different aspects of the field due to differences in regional climates and operational practices, many fundamental meteorological principles and professional concepts are shared across countries. Therefore, we believe that questions from Chinese examinations can still provide a meaningful basis for evaluating models' meteorological capabilities.


W2: Although I do believe that evaluation datasets for specialized domains are important, I think the ICLR main conference places greater emphasis on innovations in learning methods and model mechanisms. While this paper compares and analyzes the performance of different models, it provides relatively limited insights into model characteristics or guidance on how to improve model performance. Therefore, I think this paper may be more suitable for another conference or journal.

Response to W2: Thank you for your comments. We would like to clarify that our work was submitted to the Dataset and Benchmark Track, whose primary objective differs from that of conventional methodology-oriented papers. First, VME provides broad coverage of meteorological knowledge and enables systematic evaluation of both general-purpose and domain-specific models, offering a comprehensive assessment of their capabilities in this specialized domain. Second, through extensive evaluations and comparative analyses, we identify important limitations of existing models in meteorological knowledge and professional competencies, providing insights into their current capability gaps and limitations. Finally, the goal of our work is not to propose a new model architecture or learning mechanism, but to establish an objective and systematic benchmark that can help guide and inform future model development in the meteorological domain. We believe that such benchmarks are essential for identifying existing gaps, tracking progress, and guiding future research.

Reviewer_97B2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Virtual Meteorologist's Exam (VME), a benchmark of 6,508 questions collected from Chinese meteorological professional skill competitions. It covers four question formats and both academic and operational topics. The authors evaluate 11 general-purpose and domain-specialized LLMs and analyze their performance across question formats, meteorological subdomains, and empirical difficulty levels. The results show that general-purpose models generally outperform specialized models, with relatively weak performance on operational data processing and quality-control questions.

Critical strengths

The benchmark is constructed from professional meteorological examinations and covers practical topics such as observation procedures, instrumentation, data reporting, and quality control. This operational focus is useful because these topics are relevant to real-world meteorological assistance and are less emphasized in many scientific QA benchmarks.

Critical weaknesses
  1. Limited contribution beyond dataset scale.

K-MetBench already evaluates meteorological expertise using professional examination questions, while ClimaQA provides text-based scientific QA evaluation. VME mainly extends the scale and coverage of existing resources, but the paper does not clearly establish which important evaluation gaps are addressed by the additional questions. The analyses largely consist of topic distributions, word clouds, model rankings, and accuracy comparisons. They provide a descriptive overview rather than substantive new insights into model capabilities. The paper therefore falls short of demonstrating the significance of its incremental contribution.

  1. Insufficient evidence for the claimed capability gaps.

The paper interprets higher true/false accuracy than cloze accuracy as evidence that models are better at understanding meteorological knowledge than reasoning with it. However, these question formats differ in guessing probability, content, response requirements, and scoring, so the comparison does not isolate reasoning ability.

Similarly, the lower performance on Data Files and Quality Control questions does not distinguish missing operational knowledge from difficulty applying known rules. The category-level results also do not control for question-format composition. These distinctions matter because they lead to different conclusions about model limitations and different directions for improvement. The current experiments identify where models score poorly, but provide limited evidence about why they fail.

Questions for authors
  1. Incremental value: What important operational competencies does VME evaluate that are not adequately covered by existing benchmarks? Can the authors provide representative examples or quantitative evidence demonstrating this additional coverage?

  2. Capability analysis: Can the authors distinguish missing domain knowledge from failures in applying known rules? For example, on a small expert-verified subset, how does performance change when the relevant official rules are provided, and can models correctly apply these rules when numerical values or conditions are modified?

  3. Evaluation validity: Please provide the human evaluation protocol, including participant counts, qualifications, testing conditions, and scoring procedures. What quantitative quality audit was conducted on the final dataset, and what residual error rate was observed? Additionally, please clarify the discrepancy in the Claude-Opus-4.8 aggregate score in Table 2 (approximately 59.2% when calculated from the reported per-format scores, versus 59.08% reported) and the mismatched DeepSeek response in Appendix Case 2.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 97B2

W1: Limited contribution beyond dataset scale.

Response to W1: 1) We would like to clarify that, just as scaling is important for the scaling laws of models, the scale of evaluation benchmarks also matters. As the scale of a benchmark increases, the assessment of a model's actual capabilities may also change. A large-scale, high-quality benchmark can reveal capabilities that smaller benchmarks may fail to capture, offering greater evaluation stability and broader coverage of diverse evaluation dimensions. Therefore, we believe that the scale and coverage of an evaluation benchmark are important factors that should not be overlooked. In terms of size, our dataset is approximately 10 times larger than ClimaQA and 5 times larger than K-MetBench. Furthermore, our dataset covers a much broader range of dimensions, with a more fine-grained categorization of meteorological subdomains. The improvements in scale and coverage represent substantial contributions rather than merely incremental advances. 2)Rather than devoting extensive space to describing the dataset itself, our primary goal is to use this benchmark to identify the capability boundaries of existing LLMs. Therefore, we only provide an objective description of the dataset, while placing greater emphasis on analyzing how current models perform on our benchmark and what statistical patterns their performance reveals. Specifically, we investigate the subdomains in which models perform well or poorly and explore the potential reasons underlying these differences via extensive experiments.


W2: Insufficient evidence for the claimed capability gaps.

Response to W2 1) We apologize for not explicitly stating the basis for this conclusion in our work. Previously, we calculated the average response length for true/false and close questions and found that true/false responses were longer than close responses (approximately 125 vs. 89 words). We considered this difference to suggest that TF questions may involve a greater degree of reasoning. Following your suggestion, we have incorporated this evidence into our revision. 2) Thank you for your suggestion. When considering the possible reasons, we previously focused only on the lack of relevant data. Indeed, it is also possible that models perform poorly in these areas because they have difficulty applying existing known rules. To ensure the completeness of our conclusions, we have incorporated this possibility into the revised manuscript. 3) We report the accuracy of each question type separately to maintain consistency with existing benchmarks. Other existing benchmarks typically report accuracy for each question type independently to decouple model accuracy from the influence of question-format composition. This allows readers to directly assess model performance without potential confounding effects introduced by differences in question-format proportions. Therefore, we adopt this reporting strategy in our work for clear illustration.


Q1: Incremental value.

Response to Q1: Thank you for your suggestion. Following your suggestion, we conduct an analysis to examine the overlap and differences in category coverage between our dataset and existing datasets. Since K-MetBench is in Korean, we exclude it from the comparison with our English-language datasets. Specifically, we compare our VME dataset with all other datasets. To this end, we first map the questions in each dataset into TF-IDF embeddings and then calculate their distances to the embeddings of our 25 predefined second-level categories to determine the most relevant category for each question. As illustrated in the figure8, our dataset shares substantial coverage with existing datasets, particularly in the two major categories of CRV (Clouds, Rain, and Visibility) and WSA (Weather Systems and Airflow). However, VME also covers several subcategories that are not adequately represented in existing datasets, including Measurement Sensors and Data Collectors, Missing Records and Time-Series Gaps, Warning Signals and Information, and etc.


Q2: Capability analysis.

Response to Q2: Due to time constraints, it is difficult for us to arrange expert annotation. We also acknowledge that the models' poor performance may indeed be attributable to their insufficient understanding of the relevant rules. Indeed, it is also possible that models perform poorly in these areas because they have difficulty applying existing known rules. To ensure the completeness of our conclusions, we have incorporated this possibility into the revised manuscript.

Q3: Evaluation validity,

Response to Q3: Thank you for your suggestion. For evaluation details, please refer to our response to the second question in Review FGIP (Response to W2 and Q2) . Thank you for your reminder, we have fixed the aforementioned errors in our revision.

Reviewer_FG1POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the VME, a meteorological benchmark containing 6,508 questions collected from real meteorological operational skill competitions, covering both academic and operational knowledge. The authors evaluate 11 general-purpose and specialized LLMs on this benchmark and analyze their performance across different question types, knowledge categories, and difficulty levels. The results show that general-purpose models generally outperform specialized meteorological models, but current models still have clear limitations in areas such as meteorological observation and data quality control.

Critical strengths

S1: This paper introduces a large-scale meteorological benchmark with 6,508 questions, covering four question types and multiple meteorological subdomains. It offers advantages in dataset size and knowledge coverage.

S2: The paper provides a systematic evaluation of various general-purpose and specialized LLMs, analyzing their performance across question types, knowledge categories, and difficulty levels. These results provide useful insights into the capabilities and limitations of current LLMs in meteorology.

Critical weaknesses

W1: The dataset is entirely based on questions from meteorological operational skill competitions and mainly evaluates models' ability to answer static text-based questions. However, real-world meteorological operations often involve analyzing multiple data sources, processing changing information, and making practical decisions.

W2: The paper compares model performance with that of human meteorological professionals, but provides limited details about the human evaluation setup. Important information, such as the number of participants, their professional backgrounds, the test questions, and the scoring procedure, is missing.

W3: The general-purpose and specialized models differ considerably in their base models, parameter sizes, model generations, and training data, which may affect the performance comparison. Therefore, the current experiments cannot clearly determine whether the better performance of general-purpose models comes from stronger base model capabilities or differences between general-purpose and domain-specific training.

W4: The dataset is collected from meteorological competition exams published between 2016 and 2024, and some questions may have been publicly available and included in the evaluated models' pretraining data. However, the paper does not provide any data contamination analysis.

Questions for authors

Q1: Could the authors clarify to what extent performance on static exam questions reflects models' capabilities in real-world meteorological operations?

Q2: Could the authors provide more details about the human baseline evaluation? Were the human participants and models evaluated on the same questions under comparable conditions to ensure a fair comparison?

Q3: Could the authors further analyze how differences in base model capabilities, parameter sizes, and model generations affect the results? Would it be possible to include a comparison of the same base model before and after domain-specific fine-tuning to better understand the actual benefits of meteorological training?

Q4: Have the authors checked for possible overlap between the original competition questions and the training data of the evaluated models? Could the authors provide data contamination analysis or evaluate the models on a subset with a lower risk of contamination to verify the robustness of the main findings?

1 comment
AuthorsOpenReview ↗

Response to Reviewer FG1P

W1: The dataset is entirely based on questions from meteorological operational skill competitions and mainly evaluates models' ability to answer static text-based questions. However, real-world meteorological operations often involve analyzing multiple data sources, processing changing information, and making practical decisions.

Q1: Could the authors clarify to what extent performance on static exam questions reflects models' capabilities in real-world meteorological operations?

Response to W1 and Q1: Thank you for your suggestion. The evaluation of large language models' meteorological knowledge and capabilities in performing meteorological tasks lags considerably behind that in general-purpose domains. Current evaluations in the meteorological domain still focus primarily on static benchmarks. In assessing meteorological knowledge, we consider not only academic knowledge but also the understanding of operational meteorological practices and real-world applications. Evaluating a model's ability to leverage existing data and perform data-driven analysis is more challenging, as this falls into the scope of agent evaluation and requires substantially more sophisticated testing. We appreciate your suggestion and will consider exploring this direction in future work.


W2: The paper compares model performance with that of human meteorological professionals, but provides limited details about the human evaluation setup. Important information, such as the number of participants, their professional backgrounds, the test questions, and the scoring procedure, is missing.

Q2: Could the authors provide more details about the human baseline evaluation? Were the human participants and models evaluated on the same questions under comparable conditions to ensure a fair comparison?

Response to W2 and Q2: Thank you for your suggestion. We randomly select 25 single-choice questions (SCQs), 30 multiple-choice questions (MCQs), 25 true-or-false questions (TFs), and 20 cloze questions to construct our question bank. We invite three general meteorological practitioners, all of whom hold either a master's or a doctoral degree, as well as one meteorological expert from a meteorological bureau, to participate in the evaluation. We compare the participants' answers with the corresponding ground-truth answers to measure their accuracy. We found that the general meteorological practitioners achieved an average accuracy of approximately 63%, while the meteorological expert achieved an accuracy of 81%.


W3 and Q3 : The general-purpose and specialized models differ considerably in their base models, parameter sizes, model generations, and training data, which may affect the performance comparison. Therefore, the current experiments cannot clearly determine whether the better performance of general-purpose models comes from stronger base model capabilities or differences between general-purpose and domain-specific training.

Response to W3 and Q3: Thank you for your suggestion. As we can see, the general-purpose models we selected are all state-of-the-art (SOTA) models, several generations ahead of their specialized counterparts. This indeed makes it difficult to disentangle the source of the performance gains achieved by general-purpose models. Furthermore, we are also unable to fine-tune these models, as many of them are closed-source, while fine-tuning others requires substantial computational resources and meteorological data at this stage. We hope to investigate this factor in the future when we are capable.


W4 and Q4: Have the authors checked for possible overlap between the original competition questions and the training data of the evaluated models? Could the authors provide data contamination analysis or evaluate the models on a subset with a lower risk of contamination to verify the robustness of the main findings?

Response to W4 and Q4: Sorry for the confusion. We use proprietary internal data, and the questions are not necessarily publicly available on the Internet. Moreover, the original questions were written in Chinese, whereas the evaluation is conducted in English. Therefore, these questions are unlikely to have been directly encountered by the models during pre-training. Moreover, Fenghe is a Chinese meteorological model trained on a large corpus of domain-specific meteorological data. However, we observe that it does not perform particularly well on this benchmark, suggesting that our evaluation data may not have been included in its training corpus. Similarly, most of the evaluated models achieve relatively unsatisfactory results on this benchmark, providing further evidence that our data are unlikely to have been contaminated by their training data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.