- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a new meteorological dataset to evaluate the capabilities of different large models in the field of meteorology. It analyzes the limitations of existing meteorological datasets and proposes a more comprehensive dataset to address these limitations. The paper provides a detailed description of the dataset construction process and the characteristics of the final dataset. It also evaluates the performance of different models on the new dataset.
Critical strengths
-
Evaluating the capabilities of large language models across different domains is indeed important, and each domain requires more specialized datasets for evaluation.
-
The paper provides a detailed and clear description of both the dataset construction process and the characteristics of the final dataset.
-
The paper evaluates a wide range of large language models on the newly constructed dataset and compares their performance.
Critical weaknesses
-
Based on the information in Lines 143–144, the original data source of the proposed dataset appears to be examinations from China. I could not find any other data sources mentioned elsewhere in the paper, so I assume that all the data come from Chinese examinations. In my opinion, this is insufficient for a domain-specific evaluation dataset. Examinations in different countries may focus on different aspects of the field. Therefore, the authors should try to collect examination questions from more countries to enable a more objective evaluation of model performance.
-
Although I do believe that evaluation datasets for specialized domains are important, I think the ICLR main conference places greater emphasis on innovations in learning methods and model mechanisms. While this paper compares and analyzes the performance of different models, it provides relatively limited insights into model characteristics or guidance on how to improve model performance. Therefore, I think this paper may be more suitable for another conference or journal.
Questions for authors
I would like the authors to address the concerns raised in the Weaknesses section, particularly regarding the specific sources of the data, how the objectivity of the dataset is ensured, and how this work could contribute to further advances in learning methods and model mechanisms.
Response to Reviewer QVNB
W1: Based on the information in Lines 143–144, the original data source of the proposed dataset appears to be examinations from China. I could not find any other data sources mentioned elsewhere in the paper, so I assume that all the data come from Chinese examinations. In my opinion, this is insufficient for a domain-specific evaluation dataset. Examinations in different countries may focus on different aspects of the field. Therefore, the authors should try to collect examination questions from more countries to enable a more objective evaluation of model performance.
Response to W1: Thank you for raising this important concern. We would like to clarify that the primary objective of VME is to evaluate the domain-specific knowledge and professional competencies of LLMs in meteorology, rather than to comprehensively represent examination systems across different countries. First, Chinese meteorological examinations provide a valuable source of professionally grounded questions, covering various aspects of meteorology and relevant operational practices. Second, although examinations in different countries may emphasize different aspects of the field due to differences in regional climates and operational practices, many fundamental meteorological principles and professional concepts are shared across countries. Therefore, we believe that questions from Chinese examinations can still provide a meaningful basis for evaluating models' meteorological capabilities.
W2: Although I do believe that evaluation datasets for specialized domains are important, I think the ICLR main conference places greater emphasis on innovations in learning methods and model mechanisms. While this paper compares and analyzes the performance of different models, it provides relatively limited insights into model characteristics or guidance on how to improve model performance. Therefore, I think this paper may be more suitable for another conference or journal.
Response to W2: Thank you for your comments. We would like to clarify that our work was submitted to the Dataset and Benchmark Track, whose primary objective differs from that of conventional methodology-oriented papers. First, VME provides broad coverage of meteorological knowledge and enables systematic evaluation of both general-purpose and domain-specific models, offering a comprehensive assessment of their capabilities in this specialized domain. Second, through extensive evaluations and comparative analyses, we identify important limitations of existing models in meteorological knowledge and professional competencies, providing insights into their current capability gaps and limitations. Finally, the goal of our work is not to propose a new model architecture or learning mechanism, but to establish an objective and systematic benchmark that can help guide and inform future model development in the meteorological domain. We believe that such benchmarks are essential for identifying existing gaps, tracking progress, and guiding future research.