- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses a practical yet underexplored problem in visual defect inspection using vision-language models (VLMs): comprehensive defect localization while maintaining restraint on defect-free images. Unlike conventional defect detection tasks that focus primarily on determining whether an image contains a defect, real-world inspection requires a model to identify all visible defects while avoiding false alarms on clean images. The paper argues that recognizing the presence of a defect is fundamentally different from conducting a comprehensive inspection. To investigate this problem, the authors introduce InspectBench, a benchmark comprising 1,039 images, including 718 defective images and 321 clean controls, with 1,983 annotated defect regions. The paper also proposes an ownership-based region-set scoring protocol to evaluate defect localization. Unlike conventional one-to-one matching, the proposed protocol allows a single prediction to cover multiple neighboring ground-truth regions while ensuring that each ground-truth region is credited to at most one prediction. It distinguishes among missed regions, weak overlaps, accepted detections, duplicate predictions, and false positives. Using InspectBench, the authors evaluate 17 model and inference-procedure configurations. The paper contributes primarily through benchmark construction, evaluation protocol design, and systematic empirical analysis, rather than through the development of a fundamentally new VLM architecture.
Critical strengths
1 Well-Motivated Task Definition with High Practical Relevance:
The paper sharply identifies a blind spot in current VLM-based generated image evaluation and defect detection—namely, that image-level flagging does not equal fine-grained inspection. This holds significant practical value for image editing, AIGC quality control, and retouching.
2 Solid and Multi-Source Heterogeneous Dataset Construction:
INSPECTBENCH integrates generated images (e.g., GPT-image-2, SenseNova-U1.5) with public datasets (ArtiBench, SalArt-VQA, etc.), encompassing 1,983 fine-grained defect region annotations across text and visual modalities. It underwent rigorous multi-annotator cross-review and arbitration, ensuring high data quality.
3 Meticulous and Comprehensive Evaluation Protocol:
Unlike traditional simple Box mAP, the proposed region-set scoring protocol balances flawed-image scores and clean-image restraint using the weight parameter. Its lexicographic maximum mechanism cleverly resolves multi-region overlapping, ownership assignment, and geometric penalties.
4 Extensive Empirical Evaluation and Rich Comparison Dimensions:
The paper evaluates 17 configurations (spanning commercial closed-source models, base open-source models, multiple SFT strategies, and multi-stage pipelines) and provides thorough bootstrap confidence intervals and ablation analyses, along with complete supplementary materials for reproducibility.
Critical weaknesses
1 Trade-off Between Performance Gains and Computational Cost:
While multi-stage pipelines (e.g., Full-Seq, Full-Sched, Selective) show improvements on certain metrics, they introduce complex agentic procedures (multiple crops, global/local scouting, candidate verification), leading to exceptionally high computational and time overheads without achieving Pareto dominance across both recall and false-positive axes on all checkpoints.
2 Limitations in the Definition and Distribution of Clean Controls:
As acknowledged by the authors in the limitations, clean images originate from three specific sources and are not strictly content-matched with flawed images. Part of the high clean FPR observed in some models may stem from sensitivity to unknown natural or art styles (mistaking artistic styles for defects); the diversity of styles could be further enhanced.
3 Subjective Biases in Geometric and Ownership Protocols:
Hyperparameters in the scoring protocol are fixed conventions. Although justified by the authors, such hardcoded geometric penalties and ownership assignments may still occasionally conflict with human expert intuition in extreme scenarios involving dense small objects or complex text clumping.
Questions for authors
See Weakness
2 comments
The authors partially address my concerns. Considering the novelty and contribution of the paper, I will keep my score.
Thank you for recognizing the benchmark's value and the distinction between image flagging and comprehensive inspection. We address the three limitations with analyses of existing predictions. We have uploaded a revised version of our paper, with all major changes highlighted in blue.
W1. Cost: procedure gains are checkpoint-dependent trade-offs.
We agree that multi-stage inspection is expensive and does not universally dominate Direct. Selective changes clean FPR by -33.33 percentage points for Trajectory-R2, but +11.21 for Workflow-SFT and +15.26 for Decision-SFT. This supports the paper's central distinction: adding an inspection procedure does not reliably improve coverage and restraint together.
Paired comparisons quantify the trade-off. For Workflow-SFT, Full-Sched versus Direct raises recall by 9.03 points (95% CI [6.55,11.57]), while FPR changes by +3.12 points [-1.25,7.48]. Selective versus Full-Sched lowers recall by 2.37 points [-4.25,-0.44] and raises FPR by 8.10 points [3.43,12.46]. The latter is worse on both axes for this checkpoint. These intervals use 10,000 paired image-bootstrap replicates conditional on fixed predictions (Appendix H.4).
Historical Full-Seq/Full-Sched/Selective runs require 51.70/26.39/14.97 requests per image and 11.18/3.69/1.72 batch hours (H.5). They quantify overhead without establishing an advantage at matched compute. The contribution is the benchmark and diagnosis of these operating points; component-level causal or universal efficiency gains are not established.
W2. Clean controls: high FPR also occurs on generated images.
For Qwen3.6-27B/Direct, clean FPR is 98.20% on GPT-image-2, 95.00% on SenseNova-U1.5 and 93.64% on natural controls. Thus high false-alarm rates are not confined to the natural subset. For Workflow-SFT/Direct -> Decision-SFT/Direct, the corresponding rates are 43.24% -> 9.01%, 54.00% -> 21.00%, and 10.91% -> 3.64%. Removing natural controls still gives 48.34% -> 14.69% FPR on 211 generated clean images (H.3).
This rules out natural-image recognition as the sole explanation for the reduction, but does not establish content-matched generalization. Recall changes by source too: 9.02% -> 5.64% on GPT-image-2, 30.63% -> 36.38% on U1.5, and 24.69% -> 20.52% on Public-4. We therefore interpret the result as source-dependent coverage/restraint changes. Style sensitivity and suppression on particular subsets remain possible. FPR is conditional on cross-reviewed reference labels.
W3. Geometry: sensitivity checks support the coverage gap, not a universal human-utility metric.
We confirmed that an enclosing box can take ownership from a tight prediction and sharply lower its credit. On actual outputs, quality-first ownership changes recall by at most 0.66 percentage points and S by 0.180, while preserving all 17 S ranks. Alternative class-agnostic one-to-one matching gives maximum recall 40.39% at IoU >=0.1 and 19.92% at >=0.5 (H.1). The broad coverage deficit therefore survives alternative matching.
Changing the area coefficient from 4 to 2/8 gives maximum recall 37.67%/30.26%; grouping-exponent variants give 30.96-34.54%. S-rank correlations remain >=0.998 for these variants. However, removing the area penalty raises maximum recall to 53.86%, and changing clean/flawed aggregation weights changes the winning configuration (H.2). The precise recall ceiling and scalar ranking are therefore protocol-dependent; S is one explicit utility convention alongside recall/FPR.
The strongest spatial conclusion is independent of those conventions: every configuration leaves at least 43.97% of annotated regions with no predicted overlap. Dense-object and text grouping may still disagree with expert judgment. We have corrected the wording that implied the 0.1 quality threshold was uniformly permissive.