- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents Radar-9K, containing 9,089 MRMS events and 94,825 frames, and a pipeline for improving radar annotation quality. Qwen3-VL-32B generates candidate annotations, audits the observations without seeing those candidates, and cross-verifies the results before expert review. LoRA fine-tuning of Qwen3-VL-8B on these data produces RadarVL-8B. Its reported Radar Score improves from 64.8 to 77.1 on MRMS and from 61.7 to 71.5 on 400 unseen OPERA events. Ablations examine the verification stages and the use of candidate-blind auditing.
Critical strengths
- A useful task and a well-motivated annotation process. The paper addresses the difficulty of obtaining reliable supervision for a task that requires domain expertise. The distinction between what reflectivity sequences can support and what they cannot is particularly useful (Sections 2.2-2.4). Checking descriptions against the original observations, followed by expert review, is a sensible way to reduce unsupported scientific claims. Expert evaluation of factual correctness is also appropriate for this task.
- Substantial improvements with evidence of transfer. RadarVL-8B improves over its base model by 12.3 points on MRMS and 9.8 points on OPERA. Table 2 provides useful evidence for the verification process: automatic verification improves the score from 69.8 to 73.9 before expert correction, which then raises it to 77.1. The authors report that the backbone, training-set size, and training configuration are controlled. Performance on unseen OPERA data strengthens the results, and the 8B model reaches a level comparable to the larger 32B model. These are meaningful findings for a domain-specialization study.
Critical weaknesses
- The reliability of the expert evaluation is difficult to assess. The main results rely on expert judgments, but the paper does not report the number and relevant experience of the evaluators, whether model identities were hidden, or the degree of agreement between evaluators. It is also unclear whether evaluation was conducted separately from training-data annotation. Section 3.1 gives a scoring rubric, which is helpful, but these additional details are needed to judge how consistently it was applied. This is a key concern because the same evaluation supports both the overall performance claims and the supervision ablations. The large gains are encouraging; the missing information limits confidence in their interpretation rather than showing that they are invalid.
- The advantage of candidate-blind auditing needs clearer evidence. On the same 500 events, Table 3 reports an error rate of 7.8% for candidate-blind auditing versus 10.6% for candidate-aware auditing. This supports the design, but the metric is computed on the remaining claims. Without knowing how much correct information each method retains, it is difficult to separate better verification from more conservative deletion. Table 2 supports the benefit of the automatic verification stage as a whole, but does not establish how much of the downstream improvement comes specifically from hiding the candidate. A comparison at similar information coverage would make this claim more convincing. Some evidence on expert editing effort would also help assess the practical benefit of the full process, although a complete cost study is not essential to the main result.
Questions for authors
- Could you clarify the expert evaluation protocol, including the number and relevant experience of evaluators, whether model identities were hidden, and whether evaluators also annotated the training data? Please report inter-rater agreement and, where available, uncertainty estimates for the main score differences.
- For the two auditing methods in Table 3, how many claims and correct claims are retained? Does the candidate-blind advantage remain when information coverage is comparable? If available, downstream results comparing the two auditing methods would help connect this analysis to model performance.
- What evidence is available that automatic auditing reduces expert editing effort or improves the final annotations compared with experts directly correcting raw candidates? Editing-time or correction-rate statistics would be helpful; a small matched comparison would also be informative.