- The paper is poorly organized and the writing is unclear. The paper jumps to details without giving enough context for the reader or explaining important terminology. For example, "axis" is used repeatedly in the text without stating explicitly what it means. Similarly, "defect family" term is used although it's unclear whether it's just a synonym for the "defect class" or something else. Multiple repeated statements about the benchmark's diagnostic benefits occupy the space in the paper but don't add any value. Section 3 immediately goes into details listing some specific labels without properly explaining what they mean or giving a clear overview of how the dataset and the task look like. In the current form, it's hard to understand and evaluate the paper's contributions.
- The motivation for this paper is unclear. The proposed questions ask about the presence of the defect, its class, count, location and area. These can be typically answered with defect segmentation/classification followed by filtering, counting and comparison rules. What is the benefit of adding the explicit language interface to this task? Does it provide any benefit to the defect diagnostic? Does it create opportunity for the new practical use case? The experiments show that LoRA adaptation improves accuracy over zero-shot inference, which is an expected outcome, but they do not explain what is the benefit of end-to-end VQA approach.
- The novelty is limited and the paper overstates its contribution. The benchmark is built on top of the existing dataset so the main contribution is the construction of questions and answers from the existing annotations. Such programmatic question generation and compositional visual reasoning has been explored extensively in such datasets as CLEVR [1], CLEVR_HYP [2], GQA [3]. These benchmark have also established the filtering rules and checks to reject ambiguous or ineffective questions. The paper does not explain how its construction rules go beyond adapting these principles to defect annotations. The paper also claims the "evaluation that separates visual benefit from benchmark priors" as one of the contributions, but this is also a well-researched problem in VQA field. Prior works (VQA v2 [4], GQA [3], VQA-CP [5], RUBi [6], VQA-CE [7]) have identified and tried to reduce the reliance on language priors and answer biases. The paper does not sufficiently distinguish its evaluation from this literature or establish what new methodological contribution or broader insight it provides.
[1] Johnson, Justin, et al. "Clevr: A diagnostic dataset for compositional language and elementary visual reasoning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[2] Sampat, Shailaja Keyur, et al. "CLEVR_HYP: A challenge dataset and baselines for visual question answering with hypothetical actions over images." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.
[3] Hudson, Drew A., and Christopher D. Manning. "Gqa: A new dataset for real-world visual reasoning and compositional question answering." 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019.
[4] Goyal, Yash, et al. "Making the v in vqa matter: Elevating the role of image understanding in visual question answering." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[5] Agrawal, Aishwarya, et al. "Don't just assume; look and answer: Overcoming priors for visual question answering." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018.
[6] Cadene, Remi, et al. "Rubi: Reducing unimodal biases for visual question answering." Advances in neural information processing systems 32 (2019).
[7] Dancette, Corentin, et al. "Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering." 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021.