acceptodds
Withdrawn from ICLR 2027

DECAR: Evidence-Grounded Clustered Calibration for Fine-Grained Vision–Language Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained image–text scoring becomes difficult when captions differ only in a local attribute, relation, count, or factual detail. A model may rank such captions correctly while still assigning scores that disagree with the reference scale. We introduce DECAR, a continuous vision–language scoring framework trained on image-centered groups of graded captions together with element-level support. DECAR combines a distributional global score head, element-level prediction, within-image ranking constraints, and soft evidence bounds that act on implausible global scores during training. The element branch is removed at inference, leaving only the image–caption branch and global score head. On the Qwen2.5-VL-7B instantiation, DECAR achieves 0.7583 SRCC and 0.5864 MAE. In a separate prompt-disjoint frozen-CLIP study with matched training settings, clustered labels increase mean SRCC from 0.4660 to 0.5601 across three seeds. Target-shuffle and gap-aware controls further show that caption-specific graded labels mainly improve correlation, whereas additional ranking constraints reduce numerical error. Additional analyses examine paraphrase stability, counterfactual ordering, residual errors, and performance across semantic subsets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language Alignment, Fine-Grained Image-Text Matching, Clustered Supervision, Distributional Prediction, Evidence-Grounded Calibration, Counterfactual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_3mdXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DECAR, a framework for fine-grained vision–language alignment scoring. Instead of relying on isolated image–text pairs, DECAR constructs image-centered clusters of captions with different alignment levels, including positive, weak-positive, and counterfactual-negative examples, and uses element-level evidence to provide additional supervision. The method combines a distributional scoring objective with within-image ranking constraints, cross-level coupling, and evidence-based regularization to improve the consistency and accuracy of continuous alignment scores. The element-level branch is used only during training and removed at inference. The authors construct a dataset of 32,717 image records across 2,991 prompt groups and 20 text-to-image generators, and evaluate the method using Qwen2.5-VL-7B and frozen CLIP-based controls. The results show improvements in rank correlation and prediction error compared with the proposed ablation variants.

Critical strengths
  1. Practical dual-branch training design with global-only inference: The proposed framework effectively incorporates local element grounding and within-image ranking as auxiliary supervision signals during training to regularize the global scorer, while retaining only the global scoring head at test time so that inference remains lightweight and requires no element extraction.
  2. Well-motivated image-centered clustered supervision. Organizing captions into image-centered clusters—including anchors, positives, weak positives, and counterfactual negatives—provides a structured way to jointly learn relative caption ordering and continuous alignment scores, going beyond supervision from isolated image–text pairs.
Critical weaknesses
  1. Unvalidated pseudo-label quality and insufficient annotation details. Approximately 90% of captions are scored by GLM-4.5V using in-context examples, while human reviewers only correct 7%–13% of propagated annotations for clear errors. The paper neither quantifies GLM-4.5V's agreement with multi-rater human scores nor reports model performance separately on the strictly human-scored subset. Thus, subtle scoring errors and model-specific biases may remain in the majority of the training and evaluation labels. Furthermore, neither the main text nor Appendix specifies the human scoring rubric for the [1,5][1, 5] scale or the criteria for revising clear errors, and Appendix B omits key details about caption generation, the scoring prompts and demonstrations, and the extraction and annotation of the 165,224 element phrases, limiting reproducibility.
  2. Evaluation restricted to the self-constructed dataset and flawed experimental design: First, all experiments are conducted exclusively on the authors' self-constructed dataset without evaluating generalization on any external fine-grained vision–language benchmarks or demonstrating downstream utility. Second, the main Qwen2.5-VL-7B experiments (Sections 4 and 5.1, Table 3) have no held-out test set—they only use a 90/10 train/validation split.
  3. Poor readability and presentation. Many quantitative results discussed in the experimental sections are difficult to trace to their corresponding tables, making the evidence supporting the authors' claims harder to verify. For example, Sections 5.1, 5.2, 5.3, and 5.5 discuss experimental numbers without ever citing Tables 3, 4, 5, and 6, respectively. There is also inconsistent equation numbering where single multi-line equations are assigned two separate numbers (e.g., Eqs. 4–5 on page 4, and Eqs. 13–14 and Eqs. 15–16 in Appendix A.1).
Questions for authors
  1. Table 7 only evaluates coarse ordering across the four macro caption roles (T0,Tp,Tw,TnT_0, T_p, T_w, T_n, while omitting Tp>TwT_p > T_w). Could you also evaluate whether the model correctly ranks captions within the same category/role (e.g., among multiple TwT_w or multiple TnT_n variants of the same image) and report the within-image SRCC across all captions of the same image, where captions differ only in subtle details? This would better demonstrate DECAR’s ability to distinguish fine-grained alignment differences.
  2. Could you provide the explicit scoring criteria or rubric used by both human annotators and GLM-4.5V?
Reviewer_t6r9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies continuous image-text alignment scoring when captions differ in local details. DECAR trains on image-centered groups of graded captions and combines a global distributional score head, an auxiliary element-support head, within-image ranking losses, and evidence-based score bounds. The paper reports improved SRCC and MAE with Qwen2.5-VL and uses a frozen-CLIP study to test the effect of clustered labels.

Critical strengths
  1. The paper separates caption ordering from matching a continuous reference score. Both matter in fine-grained image-text alignment.
  2. The training method uses graded caption groups and local element support.
  3. The matched frozen-CLIP controls share features and training settings, and the paper reports three seeds. The within-image target shuffle is a useful check on the role of caption-specific scores.
Critical weaknesses
  1. Table 3 appears to report Qwen results on the 10% validation split used for checkpoint selection.

  2. Only about 10% of caption scores are assigned directly by humans. Performance against these targets does not establish agreement with independent human judgments.

  3. The claim that correct caption-specific labels drive the correlation gain is stronger than the matched controls support. In Table 5, shuffling still raises SRCC from 0.4660 to 0.5422, while correct assignments raise it further to 0.5601.

  4. The evidence-bound term lowers Qwen MAE from 0.6512 to 0.5864, but the paper does not compare it with a simple calibration of the dual-level scores. It is therefore unclear how much of this error reduction requires element-based bounds.

Questions for authors

Please address the above weaknesses and answer the following questions.

  1. Can you evaluate the selected Qwen checkpoint on a prompt-disjoint test set that was not used for model selection? Please report all four metrics for the full model and its ablations.

  2. How does DECAR perform on a directly human-scored test subset or an independent human-rated dataset? Please report the number of items and the level of agreement among raters.

  3. What are the paired confidence intervals for full cluster-soft versus within-image shuffling across the three seeds? Does restoring the correct caption-score assignments give a clear gain in SRCC and PLCC?

Reviewer_nNJQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DECAR, a framework for fine-grained image–text alignment scoring that jointly considers semantic ranking and numerical score calibration. The method organizes captions into image-centered clusters containing positive, weak-positive, and counterfactual descriptions. It combines a distributional global scoring head, auxiliary element-level evidence prediction, within-cluster ranking constraints, and confidence-aware evidence bounds. The element branch is removed during inference. Experiments on Qwen2.5-VL-7B achieve 0.7583 SRCC and 0.5864 MAE, while controlled frozen-CLIP experiments demonstrate the benefits of clustered supervision and caption-specific score assignments.

Critical strengths
  1. Meaningful Problem Formulation and Integrated Framework. The paper addresses the important distinction between fine-grained semantic ranking and numerical alignment accuracy. Integrating image-centered caption clusters with element-level evidence provides a coherent framework for jointly optimizing these complementary objectives.
  2. Carefully Designed Supervision Controls. The matched frozen-CLIP experiments and within-image target-shuffling controls help distinguish the benefits of additional caption supervision from those of correct caption–score assignments.
  3. Useful Ablation and Error Analyses. The progressive ablation in Table 6 demonstrates the contributions of element supervision, cross-level coupling, and evidence bounds. Additional analyses of paraphrase stability, counterfactual ordering, and domain-specific errors provide useful insights into the method's behavior and limitations.
Critical weaknesses
  1. Limited Reliability of Reference Score Annotations. Only approximately 10% of captions are directly scored by humans, while the remaining scores are generated by GLM-4.5V and subsequently reviewed. Without separate evaluation against independently collected human judgments, it remains unclear whether the reported improvements reflect genuine perceptual alignment or agreement with the automated annotator.
  2. Insufficient Independent Evaluation of the Main Model. The Qwen experiments use a 90/10 validation protocol, with checkpoints selected according to validation SRCC, but no separate untouched test evaluation is established. Moreover, comparisons with supervised alternatives using the same backbone and training budget are limited, making the full model's generalization and relative advantages difficult to assess.
  3. Incomplete Validation of the Evidence-Based Calibration Mechanism. Although evidence bounds substantially reduce MAE, their advantage over simpler calibration or regularization strategies is not established. Furthermore, clustered supervision increases RMSE from 0.9421 to 0.9743 in the matched frozen-CLIP comparison, suggesting that improved ranking does not consistently translate into better numerical scoring.
Questions for authors

Q1. Reliability of Human-Reviewed Labels. Can the authors report performance separately on directly human-scored and GLM-propagated subsets? An evaluation using independently annotated human reference scores would help establish whether the improvements generalize beyond the automated scoring process.

Q2. Independence of the Qwen Evaluation. Are the results in Table 3 obtained on the same validation partition used for checkpoint selection?

Q3. Necessity of Evidence-Grounded Calibration. How does the proposed evidence-bound mechanism compare with simpler alternatives, such as post-hoc score calibration or conventional score regularization, under matched training settings? Would perturbing or shuffling element evidence significantly reduce its reported benefits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.