- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a deployment-risk-aware evaluation protocol for wafer-map VQA, focusing on false alarms, missed defects, and diagnostic consistency. The authors construct a benchmark from existing wafer-map data and evaluate VLM training strategies. The study examines how aggregate accuracy and coherent diagnostic responses can mask prediction errors and shortcut learning.
Critical strengths
The paper highlights a relevant problem in industrial VQA evaluation, where aggregate accuracy can conceal critical diagnostic failures. The analysis of false alarms, missed defects, and shortcut learning offers useful insights into the limitations of conventional evaluation protocols.
Critical weaknesses
The paper has several significant weaknesses that limit its contribution and practical relevance. The proposed evaluation protocol largely combines established metrics and consistency checks, offering limited methodological novelty. The benchmark is derived from an existing wafer-map dataset and can be largely solved using conventional classification followed by deterministic template generation, raising serious concerns about its ability to assess genuine multimodal reasoning. The morphology descriptions and process-cause hypotheses lack expert verification, further weakening the reliability of the benchmark. The experimental evaluation is also limited in scope, with insufficient evidence of generalization across industrial datasets and model architectures. Although the authors claim to release the dataset and implementation, the provided anonymous repository is currently inaccessible, preventing independent verification of the reported results. These issues substantially weaken the paper's novelty, benchmark validity, reproducibility, and overall significance.
Questions for authors
The authors should clarify the methodological contributions of the proposed evaluation protocol beyond existing class-conditional metrics, consistency checks, and shortcut-learning analyses. The benchmark's validity as a multimodal reasoning task requires stronger justification, particularly since a conventional classifier combined with deterministic templates outperforms the evaluated VLMs. The authors should explain how the benchmark distinguishes genuine image-grounded reasoning from label prediction and template matching. The reliability of the morphology descriptions and process-cause hypotheses also needs to be established through independent domain-expert validation. Additional experiments on independent wafer-map datasets and other industrial inspection tasks would help demonstrate whether the identified failure patterns generalize beyond the current benchmark. The authors should also clarify the limited model and seed coverage and provide stronger evidence that the observed training instability is not specific to particular configurations. The anonymous repository is currently inaccessible, so the authors should ensure that the dataset annotations, evaluation scripts, training code, and experimental configurations are available for independent verification.