acceptodds
Withdrawn from ICLR 2027

SolarVQA: A Diagnostic Benchmark for Visual Reasoning in Photovoltaic Inspection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual inspection of a solar cell involves more than detecting a defect. Questions about location, count and covered area require different properties to be combined. We introduce SOLARVQA, a diagnostic visual question answering benchmark with 106,555 questions over 16,322 electroluminescence images. Seven core tasks and six compositional tasks are generated from existing region annotations through image-integrity checks, consistent geometry, explicit ambiguity rules and controls for selected answer shortcuts. Across eleven zero-shot vision-language models, the highest task-macro score remains below a training-fitted nonvisual prior. However, supervised adaptation reveals substantial learnable visual signal. After one epoch of LoRA training, Qwen2.5-VL-7B reaches 64.49% test accuracy, compared with 41.64% before adaptation. With shuffled images, the corresponding scores are 38.02% and 37.08%. The benefit of correct image pairing therefore increases by 21.91 percentage points, with a paired 95% image-cluster interval of [20.86, 23.02]. Compositional performance also improves, although questions that change a simpler answer expose persistent failures. We further show that counting options can inflate ordinal agreement without improving exact accuracy. SOLARVQA supports both learning and diagnosis: its controls identify visual progress while revealing what aggregate gains leave unresolved.

Primary area:
datasets and benchmarks
Keywords:
visual question answering, vision-language models, photovoltaic defect inspection, electroluminescence imaging, multimodal benchmarking, solar cells, industrial inspection, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3SV6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This submission introduces SolarVQA, a benchmark for visual question answering for solar cell defects. The dataset encompasses 106,555 questions over 16,322 electroluminescence images sourced from UCF-EL-Defect, and breaks down into 13 multiple-choice tasks, including 6 compositional ones. The paper benchmarks 11 open-source models (<7B params), and additionally tests improvements of Qwen2.5-VL-7B under one epoch of LoRA adaptation.

Critical strengths
  • The dataset is from an unusual domain and tests model recognition beyond the usual natural test cases
  • The questions encompass normal and compositional tasks
  • Answer options are balanced
  • Adaptation experiment one one model
Critical weaknesses
  • The dataset is extremely niche and likely only relevant to a tiny fraction of the ICLR community; furthermore, the only contribution seems to be the creation of the questions and the eval, as the images are sampled from UCF-EL-Defect dataset.
  • It is unclear how the questions were generated or verified. A human baseline is missing.
  • The evaluation is limited to small open-source models that are fairly outdated. E.g., Qwen2.5-VL-7B, whereas the latest Qwen family is 3.8. It is not clear to me why any model should be able to answer these questions zero-shot, especially given the cryptic wording (e.g., "Comparing the top and bottom halves, are the centers of the largest visible Crack_Closed region and the largest visible Crack_Resistive region in the same half?"). The low performance is therefore hardly surprising.
  • The LoRA experiment shows an improvement in performance in just one epoch, which is likely due to domain adaptation, but this seems expected. Tasks like counting do not improve, but this is well known, too and expected.
  • Multiple sections are hard to follow and feel very AI-generated.

Minor:

  • Colors in Fig. 2 are barely distiguishable
Questions for authors

n/a

Reviewer_rS5COpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SolarVQA - a visual question answering (VQA) dataset for electroluminescence (EL) images of solar cells. It is built on top of existing UCF-EL-Defect dataset by constructing direct and compositional questions from the image annotations. The paper tests several vision and language models on the proposed benchmark in a zero-shot setting and further shows that LoRA adaptation can significantly improve the accuracy.

Critical strengths

The paper proposes a VQA benchmark for photovoltaic defect inspection - this is an underexplored application of VQA and the dedicated benchmark could support further research in this area.

Critical weaknesses
  1. The paper is poorly organized and the writing is unclear. The paper jumps to details without giving enough context for the reader or explaining important terminology. For example, "axis" is used repeatedly in the text without stating explicitly what it means. Similarly, "defect family" term is used although it's unclear whether it's just a synonym for the "defect class" or something else. Multiple repeated statements about the benchmark's diagnostic benefits occupy the space in the paper but don't add any value. Section 3 immediately goes into details listing some specific labels without properly explaining what they mean or giving a clear overview of how the dataset and the task look like. In the current form, it's hard to understand and evaluate the paper's contributions.
  2. The motivation for this paper is unclear. The proposed questions ask about the presence of the defect, its class, count, location and area. These can be typically answered with defect segmentation/classification followed by filtering, counting and comparison rules. What is the benefit of adding the explicit language interface to this task? Does it provide any benefit to the defect diagnostic? Does it create opportunity for the new practical use case? The experiments show that LoRA adaptation improves accuracy over zero-shot inference, which is an expected outcome, but they do not explain what is the benefit of end-to-end VQA approach.
  3. The novelty is limited and the paper overstates its contribution. The benchmark is built on top of the existing dataset so the main contribution is the construction of questions and answers from the existing annotations. Such programmatic question generation and compositional visual reasoning has been explored extensively in such datasets as CLEVR [1], CLEVR_HYP [2], GQA [3]. These benchmark have also established the filtering rules and checks to reject ambiguous or ineffective questions. The paper does not explain how its construction rules go beyond adapting these principles to defect annotations. The paper also claims the "evaluation that separates visual benefit from benchmark priors" as one of the contributions, but this is also a well-researched problem in VQA field. Prior works (VQA v2 [4], GQA [3], VQA-CP [5], RUBi [6], VQA-CE [7]) have identified and tried to reduce the reliance on language priors and answer biases. The paper does not sufficiently distinguish its evaluation from this literature or establish what new methodological contribution or broader insight it provides.

[1] Johnson, Justin, et al. "Clevr: A diagnostic dataset for compositional language and elementary visual reasoning." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[2] Sampat, Shailaja Keyur, et al. "CLEVR_HYP: A challenge dataset and baselines for visual question answering with hypothetical actions over images." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

[3] Hudson, Drew A., and Christopher D. Manning. "Gqa: A new dataset for real-world visual reasoning and compositional question answering." 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019.

[4] Goyal, Yash, et al. "Making the v in vqa matter: Elevating the role of image understanding in visual question answering." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

[5] Agrawal, Aishwarya, et al. "Don't just assume; look and answer: Overcoming priors for visual question answering." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018.

[6] Cadene, Remi, et al. "Rubi: Reducing unimodal biases for visual question answering." Advances in neural information processing systems 32 (2019).

[7] Dancette, Corentin, et al. "Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering." 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021.

Questions for authors

I'd appreciate if authors could address my concerns about the motivation and novelty, but I don't think that I will change my score since I believe the paper can't be accepted without the major rewrite.

Reviewer_MRL5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduces SOLARVQA, which is a diagnostic visual question-answering benchmark containing 106,555 multiple-choice questions generated programmatically from existing region annotations across 16,322 electroluminescence solar-cell images. Seven core tasks and six compositional tasks are generated from existing region annotations through image-integrity checks, consistent geometry, explicit ambiguity rules and controls for selected answer shortcuts. Evaluations across 11 open-weight vision-language models (VLMs) show that all zero-shot models underperform a simple training-fitted nonvisual prior. The authors found that one epoch of LoRA training with Qwen2.5-VL-7B significant improved the performance. However, fine-grained diagnostics reveal that adapted models still fail on compositional subtasks.

Critical strengths
  1. Good control of shortcuts. Different from many VQA benchmarks that simply report aggregate accuracy, the authors carefully isolate nonvisual priors.
  2. Good compositional analysis. The paper tracks whether applying a spatial restriction or class exclusion actually changes the answer compared to the global image question. This reveals a interesting failure mode: while LoRA adaptation improves overall Excluded-class area accuracy, accuracy on the 60 cases where removing a class changes the winning half actually drops significantly, thus, showing the model learns global shortcuts rather than executing the compositional operation.
  3. Good control of data. The authors perform pixel-level and annotation-level deduplication, enforce a 5% minimum margin on area comparisons to avoid noisy border ties, and compute 95% confidence intervals via 5,000 paired image-cluster bootstrap replicates rather than treating correlated questions from the same image as independent.
Critical weaknesses
  1. The benchmak collects no new images and no new human annotations. It only converts polygon masks from the existing UCF-EL-Defect semantic segmentation dataset into 13 fixed question templates. Furthermore, as the authors acknowledge in Section 3.2, a region is just a source polygon annotation rather than a distinct physical defect, and coverage reflects pixel mask area rather than electrical severity.
  2. Since every single question in SolarVQA is computed deterministically from 9 semantic segmentation classes with simple set/area operations (Equation 1), the most straightforward baseline is to train a lightweight segmentation model (e.g., U-Net or Mask R-CNN) on the exact same UCF-EL-Defect training split and run the deterministic Python rules on the predicted masks.
Questions for authors
  1. After filtering, several compositional tasks have very small test sets, for example, only 196 test questions for Count/area agreement. Do you think this will impact the practical value of the benchmark?
  2. Do you think expecting zero-shot general-purpose VLMs to know the exact visual boundaries of UCF-EL-Defect's internal taxonomy without visual in-context examples makes the evaluation unfair?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.