acceptodds
Under review as a conference paper at ICLR 2027

RadarVL: From Model-Generated Candidates to Expert-Verified Supervision for Domain-Specialized VLMs

Abstract

Specializing general-purpose vision-language models (VLMs) to scientific visual tasks with high domain expertise requirements relies on large-scale, high-quality, and reliable domain supervision. However, constructing such supervision entirely through manual annotation by domain experts is costly, while directly using descriptions generated by strong VLMs as training labels may introduce erroneous supervision that is inconsistent with the original observations. To address this challenge, we focus on weather-radar understanding, an interdisciplinary task that requires professional meteorological knowledge, temporal visual reasoning, and multimodal interpretation, and propose an expert-verified synthetic supervision approach. Based on this approach, we construct Radar-9K, containing 9,089 expert-verified convective events and 94,825 MRMS composite-reflectivity frames, and further specialize an 8B VLM into the weather-radar vision-language model RadarVL-8B. Experiments show that verified supervision substantially improves radar-domain understanding. On the MRMS test set, RadarVL-8B improves the score from 64.8 to 77.1. On unseen European OPERA radar observations, its expert score improves from 61.7 to 71.5 without additional fine-tuning. These results demonstrate that candidate annotations generated by strong VLMs can effectively support the specialization of weather-radar models, while reliability verification is critical for obtaining high-quality supervision and robust downstream performance.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models (VLMs), Weather Radar, Synthetic Supervision, Expert Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zbw2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Radar-9K, containing 9,089 MRMS events and 94,825 frames, and a pipeline for improving radar annotation quality. Qwen3-VL-32B generates candidate annotations, audits the observations without seeing those candidates, and cross-verifies the results before expert review. LoRA fine-tuning of Qwen3-VL-8B on these data produces RadarVL-8B. Its reported Radar Score improves from 64.8 to 77.1 on MRMS and from 61.7 to 71.5 on 400 unseen OPERA events. Ablations examine the verification stages and the use of candidate-blind auditing.

Critical strengths
  1. A useful task and a well-motivated annotation process. The paper addresses the difficulty of obtaining reliable supervision for a task that requires domain expertise. The distinction between what reflectivity sequences can support and what they cannot is particularly useful (Sections 2.2-2.4). Checking descriptions against the original observations, followed by expert review, is a sensible way to reduce unsupported scientific claims. Expert evaluation of factual correctness is also appropriate for this task.
  2. Substantial improvements with evidence of transfer. RadarVL-8B improves over its base model by 12.3 points on MRMS and 9.8 points on OPERA. Table 2 provides useful evidence for the verification process: automatic verification improves the score from 69.8 to 73.9 before expert correction, which then raises it to 77.1. The authors report that the backbone, training-set size, and training configuration are controlled. Performance on unseen OPERA data strengthens the results, and the 8B model reaches a level comparable to the larger 32B model. These are meaningful findings for a domain-specialization study.
Critical weaknesses
  1. The reliability of the expert evaluation is difficult to assess. The main results rely on expert judgments, but the paper does not report the number and relevant experience of the evaluators, whether model identities were hidden, or the degree of agreement between evaluators. It is also unclear whether evaluation was conducted separately from training-data annotation. Section 3.1 gives a scoring rubric, which is helpful, but these additional details are needed to judge how consistently it was applied. This is a key concern because the same evaluation supports both the overall performance claims and the supervision ablations. The large gains are encouraging; the missing information limits confidence in their interpretation rather than showing that they are invalid.
  2. The advantage of candidate-blind auditing needs clearer evidence. On the same 500 events, Table 3 reports an error rate of 7.8% for candidate-blind auditing versus 10.6% for candidate-aware auditing. This supports the design, but the metric is computed on the remaining claims. Without knowing how much correct information each method retains, it is difficult to separate better verification from more conservative deletion. Table 2 supports the benefit of the automatic verification stage as a whole, but does not establish how much of the downstream improvement comes specifically from hiding the candidate. A comparison at similar information coverage would make this claim more convincing. Some evidence on expert editing effort would also help assess the practical benefit of the full process, although a complete cost study is not essential to the main result.
Questions for authors
  1. Could you clarify the expert evaluation protocol, including the number and relevant experience of evaluators, whether model identities were hidden, and whether evaluators also annotated the training data? Please report inter-rater agreement and, where available, uncertainty estimates for the main score differences.
  2. For the two auditing methods in Table 3, how many claims and correct claims are retained? Does the candidate-blind advantage remain when information coverage is comparable? If available, downstream results comparing the two auditing methods would help connect this analysis to model performance.
  3. What evidence is available that automatic auditing reduces expert editing effort or improves the final annotations compared with experts directly correcting raw candidates? Editing-time or correction-rate statistics would be helpful; a small matched comparison would also be informative.
Reviewer_wNqdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this work, author propose to leverage VLM for a weather radar problem. the input is MRMS data for a series of events. the output is a text description of those events (like motion, dBz, strength over time). to achieve this, the author propose to use qwen to generate the labels. In specific, they generate labels with same model to do sanity check and cleaning. Then let expert to further do sanity check and cleaning, which are cheaper than direct human label. As a result, they collect Radar-9K datasets for smaller VLM fine tuning since it has cleaner labels. the resulting model has improved performance

Critical strengths

(+) the topic and exploration is interesting as levering the VLM in weather field. the candidate-blind audit is helpful. the collected dataset could be helpful for the community. the zero-shot evaluation on other data shows sort of transfer or generalized capability

Critical weaknesses

(-) the motivation and problem settings sound not too challenging enough to justify the usage of VLM. most of the output stuff like position / intensity / motion / and strength can be got from the outputs of simple tracking algorithms or codes. For example, why not using author's data cleaning algorithm to get part of the output. Meanwhile, use another simple vision based method to get rest of those. my main concern is that this paper doesn't explicitly show what unique and necessary capability in VLM should be brought to this problem settings. Meanwhile, most of the reasoning capability that a VLM could bring got banned by author, which makes the necessity of the VLM be more confusing. Author should compare combination of current methods (as baseline) to prove the necessity of the VLM.

(-) the method is sort of distillation from a foundation model. meanwhile, human cleaning also included for better labels. Overall, it not very insightful as weather-specific prior in data cleaning procedure is expected. In other word, what uninterpretable stuff in weather really needs VLM's help. Currently, it reads like trying standard distillation in a simple problem setting in weather field.

(-) Moreover, all baselines are VLMs. comparison on inference speed, computational cost are necessity to be presented. Also, there is no comparison with combination of non-vlm methods for same goal.

Questions for authors

it is not very clear of using a VLM for such problem settings, which raises my concern about the value of this work or motivation's value. Can authors explicitly explain or show that this input and output setting requires a VLM's unique capability? In current settings, part of the output seem does not challenge the combination of traditional methods at all.

Reviewer_MiGGOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2(−1)
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper addresses the scarcity of expert supervision for weather-radar understanding. A strong VLM (Qwen3-VL-32B) generates candidate descriptions of convective events, which are then verified through candidate-blind auditing, observation-grounded cross-verification, and domain-expert correction, yielding the Radar-9K dataset (9,089 events). LoRA fine-tuning of Qwen3-VL-8B on this supervision produces RadarVL-8B, evaluated by domain experts on five radar-interpretation dimensions (Radar Score). RadarVL-8B improves the Radar Score from 64.8 to 77.1 on held-out MRMS events and generalizes to unseen OPERA observations (61.7 to 71.5) without additional fine-tuning. Ablations indicate that supervision reliability, rather than mere domain exposure, drives these gains.

Critical strengths
  1. Quantitative validation of the verification pipeline on both the data side and the model side. The supervision ablation (Table 2) shows that training on raw, audit-verified, and expert-verified annotations yields progressively higher Radar Scores (69.8, 73.9, and 77.1), indicating that reliability control rather than mere domain exposure drives the gains. The audit-protocol comparison (Table 3, same 500 sampled events) isolates the value of information independence: candidate-blind auditing leaves a residual atomic-claim error rate of 7.8%, compared with 10.6% under candidate-aware auditing, suggesting that an auditor who cannot see the candidate is less likely to inherit its errors.
  2. Honest evaluation methodology. Appendix C uses a controlled case study (two peer-reviewed expert analyses of the same Zhengzhou event, ROUGE-L 0.1347) to show that lexical-overlap metrics are invalid for this task, justifying the expert-rubric Radar Score; GPT Score is explicitly auxiliary and does not replace expert evaluation.
Critical weaknesses
  1. Marginal advantage over the untuned 32B model without cost or significance analysis. On MRMS, RadarVL-8B (77.1) exceeds the untuned Qwen3-VL-32B (76.8) by only 0.3 points, and on OPERA (71.5 compared with 70.8) by 0.7 points. The paper itself describes this as "slightly exceeds". With no inference-cost comparison and no significance analysis, the central value proposition, namely domain-specific SFT of an 8B model compared with direct deployment of the untuned 32B model, remains unquantified.
  2. The reliability of the main metric is not established. The number of experts M behind Radar Score is never stated, and no inter-annotator agreement, variance, or significance is reported anywhere. Sec 2.4 states that inter-expert agreement "is independently assessed on a sampled subset", yet no result of this assessment is reported. Without these quantities, score gaps of 0.3 to 0.7 points cannot be distinguished from rating noise.
Questions for authors
  1. Sec 2.4 states that inter-expert agreement is assessed on a sampled subset via second-expert review and third-expert adjudication. Please report the actual agreement value (e.g., weighted kappa or percent agreement), the number of experts M involved in the Radar Score evaluation, and the variance or confidence intervals of the reported scores.
  2. What is the inference cost (latency, throughput, memory) of RadarVL-8B compared with the untuned Qwen3-VL-32B? Under what deployment conditions do the observed Radar Score margins (0.3 points on MRMS and 0.7 points on OPERA) justify domain-specific SFT over deploying the larger untuned model?
  3. How sensitive is the pipeline to the choice of generator and auditor? If a weaker open-source VLM replaces Qwen3-VL-32B-Instruct for candidate generation, auditing, and cross-verification, how do the atomic-claim error rate and the final model performance change?
  4. What was the total expert workload, in expert-hours and number of experts, for constructing Radar-9K, and how is it distributed across the verification stages? In particular, the supervision ladder in Table 2 shows that the final expert-verification stage contributes a gain of 3.2 points over audit and cross-verification alone; what does this stage cost, and how much of it could be recovered by cheaper reliability controls (e.g., self-consistency, majority voting, model-based adjudication) prior to expert review? This determines the reproducibility of the pipeline in lower-resource domains.
  5. RadarQA and WeatherQA are cited as motivation but not compared. Please either state explicitly why their tasks (forecast-quality assessment; severe-weather prediction QA) are incompatible with event interpretation, or report a comparison on any usable subset.
  6. All supervision and the primary evaluation derive from 2025 MRMS observations, split at the event level into 6,362/1,363/1,364 training/validation/test events, with the stated safeguard that frames of the same tracked event do not cross splits. Consequently, fragments of the same storm, or distinct events from the same day and synoptic environment, may appear on both sides of the split. Please clarify whether the split was random or chronological.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.