- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes DECAR, a framework for fine-grained vision–language alignment scoring. Instead of relying on isolated image–text pairs, DECAR constructs image-centered clusters of captions with different alignment levels, including positive, weak-positive, and counterfactual-negative examples, and uses element-level evidence to provide additional supervision. The method combines a distributional scoring objective with within-image ranking constraints, cross-level coupling, and evidence-based regularization to improve the consistency and accuracy of continuous alignment scores. The element-level branch is used only during training and removed at inference. The authors construct a dataset of 32,717 image records across 2,991 prompt groups and 20 text-to-image generators, and evaluate the method using Qwen2.5-VL-7B and frozen CLIP-based controls. The results show improvements in rank correlation and prediction error compared with the proposed ablation variants.
Critical strengths
- Practical dual-branch training design with global-only inference: The proposed framework effectively incorporates local element grounding and within-image ranking as auxiliary supervision signals during training to regularize the global scorer, while retaining only the global scoring head at test time so that inference remains lightweight and requires no element extraction.
- Well-motivated image-centered clustered supervision. Organizing captions into image-centered clusters—including anchors, positives, weak positives, and counterfactual negatives—provides a structured way to jointly learn relative caption ordering and continuous alignment scores, going beyond supervision from isolated image–text pairs.
Critical weaknesses
- Unvalidated pseudo-label quality and insufficient annotation details. Approximately 90% of captions are scored by GLM-4.5V using in-context examples, while human reviewers only correct 7%–13% of propagated annotations for clear errors. The paper neither quantifies GLM-4.5V's agreement with multi-rater human scores nor reports model performance separately on the strictly human-scored subset. Thus, subtle scoring errors and model-specific biases may remain in the majority of the training and evaluation labels. Furthermore, neither the main text nor Appendix specifies the human scoring rubric for the scale or the criteria for revising clear errors, and Appendix B omits key details about caption generation, the scoring prompts and demonstrations, and the extraction and annotation of the 165,224 element phrases, limiting reproducibility.
- Evaluation restricted to the self-constructed dataset and flawed experimental design: First, all experiments are conducted exclusively on the authors' self-constructed dataset without evaluating generalization on any external fine-grained vision–language benchmarks or demonstrating downstream utility. Second, the main Qwen2.5-VL-7B experiments (Sections 4 and 5.1, Table 3) have no held-out test set—they only use a 90/10 train/validation split.
- Poor readability and presentation. Many quantitative results discussed in the experimental sections are difficult to trace to their corresponding tables, making the evidence supporting the authors' claims harder to verify. For example, Sections 5.1, 5.2, 5.3, and 5.5 discuss experimental numbers without ever citing Tables 3, 4, 5, and 6, respectively. There is also inconsistent equation numbering where single multi-line equations are assigned two separate numbers (e.g., Eqs. 4–5 on page 4, and Eqs. 13–14 and Eqs. 15–16 in Appendix A.1).
Questions for authors
- Table 7 only evaluates coarse ordering across the four macro caption roles (, while omitting ). Could you also evaluate whether the model correctly ranks captions within the same category/role (e.g., among multiple or multiple variants of the same image) and report the within-image SRCC across all captions of the same image, where captions differ only in subtle details? This would better demonstrate DECAR’s ability to distinguish fine-grained alignment differences.
- Could you provide the explicit scoring criteria or rubric used by both human annotators and GLM-4.5V?