acceptodds
Under review as a conference paper at ICLR 2027

Beyond Aggregate Accuracy: A Deployment-Aware Dataset and Benchmark for Mixed-Defect Wafer-Map VQA

Abstract

Industrial diagnosis with vision-language models is hierarchical: decide whether a part is abnormal, name every co-occurring defect, localise each one, and explain it. We release Mixed-WM38 VQA, a benchmark that links these stages: 190,075 question-answer instances over 38,015 wafer maps and 38 normal, single-defect and mixed-defect combinations, with answers derived deterministically from labels or pixels, fixed splits, and a full-corpus audit that removed a shortcut in which location was a relabelling of defect type. The benchmark is built to expose what aggregate scores hide. Only 2.66% of the evaluation wafers are normal, so an always-abnormal gate is 97.34% accurate, and on a 4B backbone every standard recipe we tried (SFT, minority replication, domain-QA pretraining, chain-of-thought, three GRPO rewards) sits at that score while rejecting all 212 normal wafers and then narrating a coherent, image-unsupported diagnosis. The failure is not perceptual: a compact CNN separates normal from abnormal wafers perfectly on the same set, whereas in the VLM the correct "No" is the second-ranked token, about 18 times less likely than "Yes". Threshold calibration does not repair it: the false-alarm-minimising threshold still misses 52% of defective wafers on held-out data. The training interventions that do move the gate, a margin loss and a selective GRPO reward, trade false alarms for fatal misses in a seed-dependent way: one recipe yields 7.6% or 30.6% fatal misses depending only on the seed, and another declares every wafer normal in two of three seeds. A prompt that looks curative under teacher forcing (74% correct-answer probability) produces 94% fatal misses under generation. Cross-stage consistency is no safeguard either: chain-of-thought raises internal agreement to 87-100% while jointly correct gate-type-location tuples fall to 8.6%. We also audit the benchmark against itself: a classifier-to-template pipeline beats the VLM on every metric, which bounds how much of the protocol needs open-ended reasoning and gives future work a mandatory floor. Aggregate accuracy, class-conditional risk, cross-stage consistency and end-to-end correctness are distinct quantities that can move in opposite directions; hierarchical VQA systems should be selected on all four, with denominators and run-to-run variation reported.

Primary area:
datasets and benchmarks
Keywords:
visual question answering, vision-language models, benchmark, industrial inspection, wafer map, class imbalance, deployment risk, affirmative bias, GRPO, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wWanOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a deployment-risk-aware evaluation protocol for wafer-map VQA, focusing on false alarms, missed defects, and diagnostic consistency. The authors construct a benchmark from existing wafer-map data and evaluate VLM training strategies. The study examines how aggregate accuracy and coherent diagnostic responses can mask prediction errors and shortcut learning.

Critical strengths

The paper highlights a relevant problem in industrial VQA evaluation, where aggregate accuracy can conceal critical diagnostic failures. The analysis of false alarms, missed defects, and shortcut learning offers useful insights into the limitations of conventional evaluation protocols.

Critical weaknesses

The paper has several significant weaknesses that limit its contribution and practical relevance. The proposed evaluation protocol largely combines established metrics and consistency checks, offering limited methodological novelty. The benchmark is derived from an existing wafer-map dataset and can be largely solved using conventional classification followed by deterministic template generation, raising serious concerns about its ability to assess genuine multimodal reasoning. The morphology descriptions and process-cause hypotheses lack expert verification, further weakening the reliability of the benchmark. The experimental evaluation is also limited in scope, with insufficient evidence of generalization across industrial datasets and model architectures. Although the authors claim to release the dataset and implementation, the provided anonymous repository is currently inaccessible, preventing independent verification of the reported results. These issues substantially weaken the paper's novelty, benchmark validity, reproducibility, and overall significance.

Questions for authors

The authors should clarify the methodological contributions of the proposed evaluation protocol beyond existing class-conditional metrics, consistency checks, and shortcut-learning analyses. The benchmark's validity as a multimodal reasoning task requires stronger justification, particularly since a conventional classifier combined with deterministic templates outperforms the evaluated VLMs. The authors should explain how the benchmark distinguishes genuine image-grounded reasoning from label prediction and template matching. The reliability of the morphology descriptions and process-cause hypotheses also needs to be established through independent domain-expert validation. Additional experiments on independent wafer-map datasets and other industrial inspection tasks would help demonstrate whether the identified failure patterns generalize beyond the current benchmark. The authors should also clarify the limited model and seed coverage and provide stronger evidence that the observed training instability is not specific to particular configurations. The anonymous repository is currently inaccessible, so the authors should ensure that the dataset annotations, evaluation scripts, training code, and experimental configurations are available for independent verification.

Reviewer_FpV1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an evaluation protocol for hierarchical wafer-map VQA that separates aggregate accuracy, class-conditional screening risk, cross-stage consistency, and complete diagnostic correctness. The motivation is clear and strong, and the contribution is clearly positioned in methodology auditing and benchmark design. While there are some issues that should be clarified, this is a good trial.

Critical strengths
  1. Strong motivation
  2. Extensive results with both good and bad trials.
Critical weaknesses
  1. Benchmark versions are inconsistent throughout the experiment
  2. most evaluation principles are existing ones
Questions for authors
  1. After rebuilding Q3, its conditional entropy given Q2 is reported as 3.64 bits. How can a mapper described as predicting only the 38-way class and then applying deterministic lookup achieve Q3 F1 of 0.976?

  2. The consistency-versus-correctness comparisons use version-1 labels, where location was determined by defect type, while the v2 SFT tuple rate is reported as only 0.44%. Can the full comparison be repeated on v2, including CoT and intervention checkpoints? Or any specific reason for the inconsistent versions?

  3. Concerning the VLM failure, any specific analysis on the rationale behind? Is it on probes, or image encodings, or classifcation heads?

Reviewer_ZP96OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper introduces a VQA protocol over the wafer-map dataset, which includes false alarms, missed defects, cross-stage consistency, and joint diagnostic issues
  • Using this reporting protocol, the authors study accuracy under class imbalance the extend to which classification w/ answer templates can solve the benchmark
  • The authors find that standard fine-tuning recipes collapse to predicting "abnormal" and a CNN+template approach outscores the VLM
Critical strengths
  • The authors address a practical concern related to failures hidden by aggregate accuracy in VQA settings
  • The ablations (question-only, blank-image, shuffled-image) show whether the CLM's language scores depend on images
Critical weaknesses
  • The proposed reporting protocol repackages standard class-conditional error rates, which is not a novel contribution beyond common concerns w/ class imbalance and benchmark shortcuts
  • The benchmark adds little information beyond the existing class labels from the dataset. Q1/Q2 re-use those labels and Q4/Q5 leverage its reasoning
  • The protocol does not show that the multi-modal support outperforms other methods
Questions for authors
  • Clarification - which annotation version and scorer are used in Table 15?
  • Suggestion - can the consistency and joint-correctness scores be reported using updated annotations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.