- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces CAPEval, a caption evaluation framework designed to investigate how different aspects of caption quality affect downstream multimodal understanding and image generation.
The authors construct a benchmark containing 300 images and approximately 15K human-verified atomic facts. Caption quality is decomposed into Coverage and Precision, measuring the completeness of visual descriptions and their factual accuracy, respectively.
Using captions generated by 10 captioners, the authors conduct controlled training experiments across two vision-language understanding pipelines and two image generation pipelines. Regression analysis reveals that Coverage is more strongly associated with understanding performance, while Precision is more strongly associated with image generation performance.
Critical strengths
-
High-quality fine-grained evaluation dataset. CAPEval contains 300 high-resolution images and approximately 15K human-verified atomic facts spanning eight semantic dimensions, providing a valuable resource for fine-grained caption evaluation.
-
Systematic downstream validation. The paper conducts controlled experiments across 10 captioners and four downstream model pipelines, connecting caption quality with actual training utility. The finding that understanding and generation favor different caption properties offers practical insights for multimodal data curation.
Critical weaknesses
Incomplete measurement of factual Precision. The paper defines Precision as the proportion of correct factual claims in a caption, but its actual computation only considers a predefined GT checklist. Additional hallucinated objects, attributes, or relationships that are not covered by the checklist cannot be directly penalized. Consequently, the metric does not fully capture the factual accuracy of all claims in a caption, which may affect the paper's central conclusion regarding the relationship between Precision and image generation performance.
Questions for authors
-
Since Precision is computed only from the predefined GT checklist, how does CAPEval detect and penalize additional hallucinated content that falls outside the checklist? Could the authors introduce an independent verification of all factual claims in each caption and examine whether the main conclusions still hold?
-
Could the authors report the correlations between caption length and Coverage or Precision, and examine whether the main findings remain consistent after controlling for caption length?
-
The paper uses Qwen2.5-72B as the automatic checklist evaluator. Could the authors validate its reliability through human evaluation on a randomly sampled subset or comparisons with other judge models?
4 comments
Q2: Correlation of caption length with Coverage and Precision, and results after controlling for length.
A2: We report correlations between mean caption length and C/P scores in figure 6, Appendix E.
The figure shows that Coverage increases modestly with caption length, while Precision remains largely unchanged. A comprehensive study controlling for caption length would require regenerating approximately 1.2 million captions for each of the sixteen captioners and retraining the downstream models, which is not feasible within the short rebuttal period and we will finish this in the future work. We therefore conducted a pilot experiment to examine how training on longer versus shorter captions relates to downstream performance:
Specifically, we selected InternVL3.5-1B and LLaVA-OV-8B as caption sources. For each captioner, we constructed separate training subsets containing longer or shorter captions, using 500,000 image–caption pairs per subset for SigLIP-Qwen3 and 50,000 per subset for Z-Image. We then trained the corresponding downstream models on these subsets and compared their understanding and generation benchmark scores. We regard DPG-Bench as our main metric, as this benchmark tests instruction-following capabilities under dense prompts.
| Setting | Token length range | C | P | DPG score | understanding score |
|---|---|---|---|---|---|
| InternVL-1B short | 127~344 | 41.1 | 71.7 | 84.07 | 66.19 |
| InternVL-1B long | 344~496 | 46.6 | 71.5 | 84.06 | 66.74 |
| LLaVA-8B short | 105~187 | 42.3 | 43.2 | 83.35 | 66.69 |
| LLaVA-8B long | 187~269 | 45.4 | 43.2 | 83.44 | 66.71 |
The results show that models trained on longer captions achieve higher mean understanding scores. For generation, longer captions yield higher overall mean scores for both caption sources. However, the model trained on shorter captions achieves a comparable DPG score. These pilot results are broadly consistent with our main findings. More details are provided in Appendix E.
Q3: Reliability of the Qwen2.5-72B as judge.
A3: Following your suggestions, we re-evaluate the scores from Qwen2.5-72B using GPT-6 Astra and two annotators who independently labeled 1,600 caption–checklist pairs for comparison with the judges. The table below reports agreement with human annotations. This high agreement supports the reliability of the automated checklist judgments. Further validation details are provided in Appendix B.
| Judge | Qwen2.5-72B | GPT-6 Astra | Human agreement |
|---|---|---|---|
| Qwen2.5-72B | - | 96% | 95% |
| GPT-6 Astra | 96% | - | 98% |
We then examine whether the downstream associations are consistent between Qwen2.5-72B and GPT-6 Astra. For each judge, we fit separate C+P regressions using its own Coverage and Precision scores on the original ten and expanded sixteen captioners.
| Judge | n | CLIP-Vicuna | SigLIP-Qwen3 | SD3.5M | Qwen-Image | Z-Image |
|---|---|---|---|---|---|---|
| Qwen2.5-72B | 10 | C: +0.218 (0.044), P: -0.049 (0.440) | C: +0.121 (0.022), P: -0.035 (0.255) | C: +0.027 (0.334), P: +0.191 (<0.001) | C: -0.076 (0.170), P: +0.234 (<0.001) | C: +0.023 (0.627), P: +0.091 (0.021) |
| Qwen2.5-72B | 16 | C: +0.114 (0.028), P: +0.019 (0.695) | C: +0.080 (0.002), P: -0.014 (0.528) | C: -0.008 (0.757), P: +0.194 (<0.001) | C: -0.032 (0.323), P: +0.199 (<0.001) | C: +0.011 (0.607), P: +0.088 (<0.001) |
| GPT-6 Astra | 10 | C: +0.197 (0.017), P: -0.241 (0.154) | C: +0.094 (0.022), P: -0.109 (0.196) | C: +0.078 (0.010), P: +0.394 (<0.001) | C: -0.007 (0.856), P: +0.538 (<0.001) | C: +0.052 (0.191), P: +0.170 (0.085) |
| GPT-6 Astra | 16 | C: +0.119 (0.005), P: -0.054 (0.659) | C: +0.070 (<0.001), P: -0.057 (0.297) | C: +0.053 (0.087), P: +0.335 (0.004) | C: +0.027 (0.386), P: +0.376 (0.003) | C: +0.034 (0.061), P: +0.159 (0.014) |
With sixteen captioners, both Qwen2.5-72B and GPT-6 Astra show positive Coverage–understanding associations in both pipelines and Precision–generation associations across all three generators.
We hope our responses address your concerns about the scope of Precision, caption length and the reliability of the judge. We are happy to clarify any remaining points.
Thanks for your response, I will raise my score.
We greatly appreciate your prompt and constructive feedback, and we are delighted to learn that our responses have adequately addressed your concerns. We sincerely thank you for raising your evaluation score to 4: clear acceptance. We will carefully refine our manuscript in full accordance with all points discussed above.
We thank Reviewer T854 for the valuable time and constructive feedback. We are truly grateful for your description of CAPEval as a "high-quality fine-grained evaluation dataset" and "a valuable resource for fine-grained caption evaluation", and for your recognition of our "systematic downstream validation" and its "practical insights for multimodal data curation". Point-to-point responses are provided below.
W1 & Q1: Incomplete measurement of factual Precision.
A1: We appreciate the reviewer's concern regarding hallucinated claims outside the checklist. We agree that the original Precision score measures correctness relative to the GT checklist, rather than the factual correctness of every claim in a caption. The checklist provides a common, human-verified reference for comparing captioners, but claims outside it are not penalized.
To be honest, systematically examining this hallucination is extremely difficult, and we have no good idea of how to quantify hallucination in a human-verifiable way during the short rebuttal period. Therefore, we present a very pilot study here. We use GPT-6 Astra to extract claims outside the checklist and assess their factuality against the images, with human auditing on a sample to revise the judging prompt.
For each image–caption pair, GPT-6 Astra returns a list of claims not covered by the GT checklist and an image-based correctness judgment for each claim, rather than a new overall Precision score. We combine these judgments with the original checklist judgments and compute extended Precision for each caption as Pnew=100×Nmentioned,check+NextraNcorrect,check+Ncorrect,extra. The numerator counts correctly judged checklist items and additional claims and the denominator counts all mentioned checklist items and extracted additional claims. We keep Coverage unchanged and refit the regressions.
10 captioners denotes the original captioners, and expanded fits use all 16 captioners for every pipeline pleasereferto∗∗fuEx′sA1∗∗fordetaileddefinitionofthisexpandedcaptionerset.
In the expanded fits, Coverage remains positively associated with understanding and extended Precision shows a consistent association with generation performance. Details are provided in Appendix E.