acceptodds
Under review as a conference paper at ICLR 2027

CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

Abstract

Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a *decoupled* caption evaluation benchmark, **CAPEval** (**C**overage **A**nd **P**recision **Eval**uation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into *Coverage* and *Precision* . The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream *end-to-end* experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent *task-dependent* dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.

Primary area:
datasets and benchmarks
Keywords:
image captioning evaluation, caption quality, multimodal evaluation, vision-language models, text-to-image generation, benchmarks

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_T854OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CAPEval, a caption evaluation framework designed to investigate how different aspects of caption quality affect downstream multimodal understanding and image generation.

The authors construct a benchmark containing 300 images and approximately 15K human-verified atomic facts. Caption quality is decomposed into Coverage and Precision, measuring the completeness of visual descriptions and their factual accuracy, respectively.

Using captions generated by 10 captioners, the authors conduct controlled training experiments across two vision-language understanding pipelines and two image generation pipelines. Regression analysis reveals that Coverage is more strongly associated with understanding performance, while Precision is more strongly associated with image generation performance.

Critical strengths
  1. High-quality fine-grained evaluation dataset. CAPEval contains 300 high-resolution images and approximately 15K human-verified atomic facts spanning eight semantic dimensions, providing a valuable resource for fine-grained caption evaluation.

  2. Systematic downstream validation. The paper conducts controlled experiments across 10 captioners and four downstream model pipelines, connecting caption quality with actual training utility. The finding that understanding and generation favor different caption properties offers practical insights for multimodal data curation.

Critical weaknesses

Incomplete measurement of factual Precision. The paper defines Precision as the proportion of correct factual claims in a caption, but its actual computation only considers a predefined GT checklist. Additional hallucinated objects, attributes, or relationships that are not covered by the checklist cannot be directly penalized. Consequently, the metric does not fully capture the factual accuracy of all claims in a caption, which may affect the paper's central conclusion regarding the relationship between Precision and image generation performance.

Questions for authors
  1. Since Precision is computed only from the predefined GT checklist, how does CAPEval detect and penalize additional hallucinated content that falls outside the checklist? Could the authors introduce an independent verification of all factual claims in each caption and examine whether the main conclusions still hold?

  2. Could the authors report the correlations between caption length and Coverage or Precision, and examine whether the main findings remain consistent after controlling for caption length?

  3. The paper uses Qwen2.5-72B as the automatic checklist evaluator. Could the authors validate its reliability through human evaluation on a randomly sampled subset or comparisons with other judge models?

4 comments

We thank Reviewer T854 for the valuable time and constructive feedback. We are truly grateful for your description of CAPEval as a "high-quality fine-grained evaluation dataset" and "a valuable resource for fine-grained caption evaluation", and for your recognition of our "systematic downstream validation" and its "practical insights for multimodal data curation". Point-to-point responses are provided below.

W1 & Q1: Incomplete measurement of factual Precision.

A1: We appreciate the reviewer's concern regarding hallucinated claims outside the checklist. We agree that the original Precision score measures correctness relative to the GT checklist, rather than the factual correctness of every claim in a caption. The checklist provides a common, human-verified reference for comparing captioners, but claims outside it are not penalized.

To be honest, systematically examining this hallucination is extremely difficult, and we have no good idea of how to quantify hallucination in a human-verifiable way during the short rebuttal period. Therefore, we present a very pilot study here. We use GPT-6 Astra to extract claims outside the checklist and assess their factuality against the images, with human auditing on a sample to revise the judging prompt.

For each image–caption pair, GPT-6 Astra returns a list of claims not covered by the GT checklist and an image-based correctness judgment for each claim, rather than a new overall Precision score. We combine these judgments with the original checklist judgments and compute extended Precision for each caption as Pnew=100×Ncorrect,check+Ncorrect,extraNmentioned,check+NextraP_{\mathrm{new}}=100\times\frac{N_{\mathrm{correct,check}}+N_{\mathrm{correct,extra}}}{N_{\mathrm{mentioned,check}}+N_{\mathrm{extra}}}. The numerator counts correctly judged checklist items and additional claims and the denominator counts all mentioned checklist items and extracted additional claims. We keep Coverage unchanged and refit the regressions.

10 captioners denotes the original captioners, and expanded fits use all 16 captioners for every pipeline pleasereferto∗∗fuEx′sA1∗∗fordetaileddefinitionofthisexpandedcaptionersetplease refer to **fuEx's A1** for detailed definition of this expanded captioner set.

In the expanded fits, Coverage remains positively associated with understanding and extended Precision shows a consistent association with generation performance. Details are provided in Appendix E.

PipelineScores10 captioners16 captioners
βC pCpCβP pPpPβC pCpCβP pPpP
CLIP-VicunaOriginal+0.218 00˙440\.044-0.049 04˙400\.440+0.114 00˙280\.028+0.019 06˙950\.695
Extended+0.165 00˙720\.072-0.022 09˙110\.911+0.113 00˙250\.025+0.063 06˙550\.655
SigLIP-Qwen3Original+0.121 00˙220\.022-0.035 02˙550\.255+0.080 00˙020\.002-0.014 05˙280\.528
Extended+0.108 00˙170\.017-0.097 02˙820\.282+0.092 <00˙01<0\.001-0.086 01˙530\.153
SD3.5MOriginal+0.027 03˙340\.334+0.191 <00˙01<0\.001-0.008 07˙570\.757+0.194 <00˙01<0\.001
Extended+0.113 00˙640\.064+0.480 00˙060\.006+0.043 03˙430\.343+0.396 00˙120\.012
Qwen-ImageOriginal-0.076 01˙700\.170+0.234 <00˙01<0\.001-0.032 03˙230\.323+0.199 <00˙01<0\.001
Extended+0.036 06˙460\.646+0.565 00˙160\.016+0.013 07˙800\.780+0.436 00˙080\.008
Z-ImageOriginal+0.023 06˙270\.627+0.091 00˙210\.021+0.011 06˙070\.607+0.088 <00˙01<0\.001
Extended+0.084 01˙430\.143+0.163 02˙210\.221+0.042 01˙480\.148+0.148 00˙980\.098

Q2: Correlation of caption length with Coverage and Precision, and results after controlling for length.

A2: We report correlations between mean caption length and C/P scores in figure 6, Appendix E.

The figure shows that Coverage increases modestly with caption length, while Precision remains largely unchanged. A comprehensive study controlling for caption length would require regenerating approximately 1.2 million captions for each of the sixteen captioners and retraining the downstream models, which is not feasible within the short rebuttal period and we will finish this in the future work. We therefore conducted a pilot experiment to examine how training on longer versus shorter captions relates to downstream performance:

Specifically, we selected InternVL3.5-1B and LLaVA-OV-8B as caption sources. For each captioner, we constructed separate training subsets containing longer or shorter captions, using 500,000 image–caption pairs per subset for SigLIP-Qwen3 and 50,000 per subset for Z-Image. We then trained the corresponding downstream models on these subsets and compared their understanding and generation benchmark scores. We regard DPG-Bench as our main metric, as this benchmark tests instruction-following capabilities under dense prompts.

SettingToken length rangeCPDPG scoreunderstanding score
InternVL-1B short127~34441.171.784.0766.19
InternVL-1B long344~49646.671.584.0666.74
LLaVA-8B short105~18742.343.283.3566.69
LLaVA-8B long187~26945.443.283.4466.71

The results show that models trained on longer captions achieve higher mean understanding scores. For generation, longer captions yield higher overall mean scores for both caption sources. However, the model trained on shorter captions achieves a comparable DPG score. These pilot results are broadly consistent with our main findings. More details are provided in Appendix E.

Q3: Reliability of the Qwen2.5-72B as judge.

A3: Following your suggestions, we re-evaluate the scores from Qwen2.5-72B using GPT-6 Astra and two annotators who independently labeled 1,600 caption–checklist pairs for comparison with the judges. The table below reports agreement with human annotations. This high agreement supports the reliability of the automated checklist judgments. Further validation details are provided in Appendix B.

JudgeQwen2.5-72BGPT-6 AstraHuman agreement
Qwen2.5-72B-96%95%
GPT-6 Astra96%-98%

We then examine whether the downstream associations are consistent between Qwen2.5-72B and GPT-6 Astra. For each judge, we fit separate C+P regressions using its own Coverage and Precision scores on the original ten and expanded sixteen captioners.

JudgenCLIP-VicunaSigLIP-Qwen3SD3.5MQwen-ImageZ-Image
Qwen2.5-72B10C: +0.218 (0.044), P: -0.049 (0.440)C: +0.121 (0.022), P: -0.035 (0.255)C: +0.027 (0.334), P: +0.191 (<0.001)C: -0.076 (0.170), P: +0.234 (<0.001)C: +0.023 (0.627), P: +0.091 (0.021)
Qwen2.5-72B16C: +0.114 (0.028), P: +0.019 (0.695)C: +0.080 (0.002), P: -0.014 (0.528)C: -0.008 (0.757), P: +0.194 (<0.001)C: -0.032 (0.323), P: +0.199 (<0.001)C: +0.011 (0.607), P: +0.088 (<0.001)
GPT-6 Astra10C: +0.197 (0.017), P: -0.241 (0.154)C: +0.094 (0.022), P: -0.109 (0.196)C: +0.078 (0.010), P: +0.394 (<0.001)C: -0.007 (0.856), P: +0.538 (<0.001)C: +0.052 (0.191), P: +0.170 (0.085)
GPT-6 Astra16C: +0.119 (0.005), P: -0.054 (0.659)C: +0.070 (<0.001), P: -0.057 (0.297)C: +0.053 (0.087), P: +0.335 (0.004)C: +0.027 (0.386), P: +0.376 (0.003)C: +0.034 (0.061), P: +0.159 (0.014)

With sixteen captioners, both Qwen2.5-72B and GPT-6 Astra show positive Coverage–understanding associations in both pipelines and Precision–generation associations across all three generators.

We hope our responses address your concerns about the scope of Precision, caption length and the reliability of the judge. We are happy to clarify any remaining points.

Thanks for your response, I will raise my score.

We greatly appreciate your prompt and constructive feedback, and we are delighted to learn that our responses have adequately addressed your concerns. We sincerely thank you for raising your evaluation score to 4: clear acceptance. We will carefully refine our manuscript in full accordance with all points discussed above.

Reviewer_dmncOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CAPEval, a caption evaluation benchmark that separates Coverage and Precision using human-verified checklists. Through downstream training experiments, it examines how these metrics relate to multimodal understanding and text-to-image generation performance.

Critical strengths
  1. Separating Coverage and Precision provides a useful perspective on caption quality beyond a single aggregate score.

  2. The downstream training experiments connect caption evaluation with practical utility across both understanding and generation, while keeping images and training settings fixed within each pipeline.

Critical weaknesses
  1. Precision appears to measure correctness relative to the GT checklist, rather than the factual correctness of all claims in a caption. It is unclear how hallucinated claims outside the checklist are penalized, which limits the interpretation of this metric as overall factual reliability.

  2. As a minor comment, Qwen2.5-72B is a relatively older model.

Questions for authors
  1. How well do the judge’s labels agree with human annotations, and do captioner rankings and downstream associations remain stable with a different judge?

  2. Can CAPEval predict the downstream utility of unseen captioners or guide caption modifications that improve downstream performance?

5 comments

We thank Reviewer dmnc for the insightful feedback. We are grateful that you found the separation of Coverage and Precision "a useful perspective on caption quality beyond a single aggregate score", and that our downstream experiments "connect caption evaluation with practical utility across both understanding and generation". Point-to-point responses are provided below.

W1: Precision is measured relative to the GT checklist.

A1: We appreciate the reviewer's concern regarding hallucinated claims outside the checklist. We agree that the original Precision score measures correctness relative to the GT checklist, rather than the factual correctness of every claim in a caption. The checklist provides a common, human-verified reference for comparing captioners, but claims outside it are not penalized.

To be honest, systematically examining this hallucination is extremely difficult, and we have no good idea of how to quantify hallucination in a human-verifiable way during the short rebuttal period. Therefore, we present a very pilot study here. We use GPT-6 Astra to extract claims outside the checklist and assess their factuality against the images, with human auditing on a sample to revise the judging prompt.

For each image–caption pair, GPT-6 Astra returns a list of claims not covered by the GT checklist and an image-based correctness judgment for each claim, rather than a new overall Precision score. We combine these judgments with the original checklist judgments and compute extended Precision for each caption as Pnew=100×Ncorrect,check+Ncorrect,extraNmentioned,check+NextraP_{\mathrm{new}}=100\times\frac{N_{\mathrm{correct,check}}+N_{\mathrm{correct,extra}}}{N_{\mathrm{mentioned,check}}+N_{\mathrm{extra}}}. The numerator counts correctly judged checklist items and additional claims and the denominator counts all mentioned checklist items and extracted additional claims. We keep Coverage unchanged and refit the regressions.

10 captioners denotes the original captioners, and expanded fits use all 16 captioners for every pipeline pleasereferto∗∗fuEx′sA1∗∗fordetaileddefinitionofthisexpandedcaptionersetplease refer to **fuEx's A1** for detailed definition of this expanded captioner set.

In the expanded fits, Coverage remains positively associated with understanding and extended Precision shows a consistent association with generation performance. Details are provided in Appendix E.

PipelineScores10 captioners16 captioners
βC pCpCβP pPpPβC pCpCβP pPpP
CLIP-VicunaOriginal+0.218 00˙440\.044-0.049 04˙400\.440+0.114 00˙280\.028+0.019 06˙950\.695
Extended+0.165 00˙720\.072-0.022 09˙110\.911+0.113 00˙250\.025+0.063 06˙550\.655
SigLIP-Qwen3Original+0.121 00˙220\.022-0.035 02˙550\.255+0.080 00˙020\.002-0.014 05˙280\.528
Extended+0.108 00˙170\.017-0.097 02˙820\.282+0.092 <00˙01<0\.001-0.086 01˙530\.153
SD3.5MOriginal+0.027 03˙340\.334+0.191 <00˙01<0\.001-0.008 07˙570\.757+0.194 <00˙01<0\.001
Extended+0.113 00˙640\.064+0.480 00˙060\.006+0.043 03˙430\.343+0.396 00˙120\.012
Qwen-ImageOriginal-0.076 01˙700\.170+0.234 <00˙01<0\.001-0.032 03˙230\.323+0.199 <00˙01<0\.001
Extended+0.036 06˙460\.646+0.565 00˙160\.016+0.013 07˙800\.780+0.436 00˙080\.008
Z-ImageOriginal+0.023 06˙270\.627+0.091 00˙210\.021+0.011 06˙070\.607+0.088 <00˙01<0\.001
Extended+0.084 01˙430\.143+0.163 02˙210\.221+0.042 01˙480\.148+0.148 00˙980\.098

W2 & Q1: Choice and reliability of the judge.

A2: Following your suggestions, we re-evaluate the scores with GPT-6 Astra and two annotators independently labeled 1,600 caption–checklist pairs for comparison with the judges. The table below reports agreement with human annotations. Further validation details are provided in Appendix B.

JudgeQwen2.5-72BGPT-6 AstraHuman agreement
Qwen2.5-72B-96%95%
GPT-6 Astra96%-98%

We then examine whether the downstream associations are consistent between Qwen2.5-72B and GPT-6 Astra. For each judge, we fit separate C+P regressions using its own Coverage and Precision scores on the original ten and expanded sixteen captioners.

JudgenCLIP-VicunaSigLIP-Qwen3SD3.5MQwen-ImageZ-Image
Qwen2.5-72B10C: +0.218 (0.044), P: -0.049 (0.440)C: +0.121 (0.022), P: -0.035 (0.255)C: +0.027 (0.334), P: +0.191 (<0.001)C: -0.076 (0.170), P: +0.234 (<0.001)C: +0.023 (0.627), P: +0.091 (0.021)
Qwen2.5-72B16C: +0.114 (0.028), P: +0.019 (0.695)C: +0.080 (0.002), P: -0.014 (0.528)C: -0.008 (0.757), P: +0.194 (<0.001)C: -0.032 (0.323), P: +0.199 (<0.001)C: +0.011 (0.607), P: +0.088 (<0.001)
GPT-6 Astra10C: +0.197 (0.017), P: -0.241 (0.154)C: +0.094 (0.022), P: -0.109 (0.196)C: +0.078 (0.010), P: +0.394 (<0.001)C: -0.007 (0.856), P: +0.538 (<0.001)C: +0.052 (0.191), P: +0.170 (0.085)
GPT-6 Astra16C: +0.119 (0.005), P: -0.054 (0.659)C: +0.070 (<0.001), P: -0.057 (0.297)C: +0.053 (0.087), P: +0.335 (0.004)C: +0.027 (0.386), P: +0.376 (0.003)C: +0.034 (0.061), P: +0.159 (0.014)

With sixteen captioners, both Qwen2.5-72B and GPT-6 Astra show positive Coverage–understanding associations in both pipelines and Precision–generation associations across all three generators.

Q2: Predicting unseen captioners and guiding caption modification.

A3: We appreciate this question. We now evaluate predictive transfer to six additional captioners in a retrospective train–test analysis. For each pipeline, we fit a regression on Coverage and Precision using only the ten submitted captioners, then apply it to the six additional captioners without including them in coefficient estimation. Each cell below gives the predicted / actual score:

CLIP-VicunaSigLIP-Qwen3SD3.5MQwen-ImageZ-Image
ScoreRankScoreRankScoreRankScoreRankScoreRank
seed-2.0-lite58.56 / 55.131 / 269.11 / 67.631 / 270.46 / 69.912 / 275.16 / 77.054 / 366.27 / 65.972 / 2
seed-2.0-pro55.94 / 55.124 / 367.66 / 66.844 / 470.16 / 69.584 / 476.10 / 77.061 / 266.01 / 65.564 / 3
seed-2.1-lite57.61 / 54.652 / 468.58 / 67.182 / 370.56 / 69.911 / 175.73 / 77.113 / 166.28 / 66.271 / 1
deepseek-v4.1-flash56.79 / 55.833 / 168.12 / 68.403 / 170.46 / 69.663 / 376.02 / 76.702 / 466.19 / 65.423 / 4
gemma3-12b-it51.80 / 53.195 / 565.48 / 65.715 / 566.39 / 68.935 / 573.96 / 75.895 / 563.97 / 64.215 / 5
kimi-vl-16b-a3b50.79 / 48.516 / 665.05 / 64.906 / 662.90 / 64.486 / 670.60 / 72.596 / 662.21 / 63.256 / 6
Spearman rho0.7140.8291.0000.7140.943

Each score cell reports predicted / observed performance.Regressions are fitted only on the original ten captioners. Predictions are evaluated on all six additional captioners.

Then we use Spearman's rank correlation coefficient, ρ\rho, to assess whether the predicted scores preserve the observed ordering of captioners. The coefficient ranges from −1-1 to 11: 11 indicates identical ordering, −1-1 indicates reversed ordering.

Across the six test captioners, ρ\rho computed from the reported Rank columns is 0.714 for CLIP-Vicuna, 0.829 for SigLIP-Qwen3, 1.000 for SD3.5M, 0.714 for Qwen-Image, and 0.943 for Z-Image. These results indicate strong ranking agreement across all five pipelines, suggesting that the predictions largely preserve relative captioner performance.

We hope these responses address your concerns about the scope of Precision, the choice of judge, and the predictive use of CAPEval. We are happy to discuss any point further.

Dear Reviewer dmnc,

As the discussion period is approaching its end, we are writing to follow up on our rebuttal and sincerely hope to hear your feedback on whether our responses have addressed your concerns. We highly value your detailed and constructive review, and have invested substantial experimental effort to resolve the issues you raised. We would greatly appreciate any further comments from you, and we are ready to provide additional clarifications if needed.

Thank you again for your time and careful review!

Best regards,

The Authors

Reviewer_NNYBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAPEval evaluates caption quality along two dimensions, Coverage and Precision, using human-written reference captions and human-verified atomic checklists. The downstream study compares ten captioners across two understanding and two generation pipelines. The reported results suggest task-dependent associations: Coverage is more closely associated with understanding performance, while Precision is more closely associated with generation performance.

Critical strengths
  • The paper addresses a useful and practically relevant question. Connecting caption quality to downstream understanding and generation can inform the selection of captions for multimodal training.
  • The empirical study is substantial. The experiments cover multiple captioner families and downstream training pipelines. The controlled comparisons provide convincing support for the observed task-dependent associations between caption quality and downstream performance.
  • The paper is well organized and easy to follow. The evaluation framework and experimental findings are clearly presented.
Critical weaknesses
  • The distinction from CapMAS needs clarification. CapMAS [1] already evaluates factuality and coverage separately, so the claim in Appendix A that previous frameworks do not distinguish these dimensions seems too broad. The downstream study is valuable, but it is less clear what CAPEval adds in terms of metric design.
  • The causal interpretation is not fully supported. The comparisons support captioner-level associations, but captioners also differ in caption length, style, and information density. The regressions therefore do not by themselves establish the separate causal effects of Coverage and Precision, as claimed in the Conclusion.
Questions for authors
  1. How does CAPEval differ from CapMAS in its metric definitions and evaluation capabilities?
  2. What evidence supports the claim that CAPEval isolates the causal contribution of each quality dimension? If the findings are intended as associations, could the authors revise the conclusion accordingly?

I would consider raising my rating if the response clarifies the contribution relative to CapMAS and addresses my concerns about the interpretation of the downstream results.

References

[1] Saehyung Lee, Seunghyun Yoon, Trung Bui, Jing Shi, and Sungroh Yoon. Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage. ICML, 2025.

4 comments

We thank Reviewer NNYB for the valuable time and constructive feedback. We are grateful that you found our question "useful and practically relevant", the empirical study "substantial", and the paper "well organized and easy to follow". We particularly appreciate your observation that the controlled comparisons "provide convincing support for the observed task-dependent associations", which is the claim we now state explicitly in the revision. Point-to-point responses are provided below.

W1 & Q1: Distinction from CapMAS.

A1: We thank the reviewer for pointing us to CapMAS [1] and apologize for overlooking it in the related work section. We actually have compared our benchmark statistics with CapMAS in Figure 4. We agree that CapMAS makes a valuable contribution to caption evaluation by assessing factuality and coverage separately. However, our core contribution is not only the decoupled evaluation, but the link between upstream evaluation scores and downstream performance.

More specifically, we focus on caption utility as training data: we train understanding and generation models on captions from different captioners and analyze how Coverage and Precision relate to downstream performance through regression analysis. Our results suggest that Coverage is the stronger predictor of understanding, while Precision is more associated with generation. This empirical link between caption quality and downstream performance is our unique contribution.

In the revision, we have discussed CapMAS in the Introduction and related work (Appendix A).

W2 & Q2: Causal interpretation of the downstream results.

A2: We appreciate the reviewer's concern regarding the caption length, information density, and caption style.

To be honest, we have not independently controlled caption style, which remains a potential confound. The matched natural-language and structured-prompt comparisons in Chen et al. [3] provide a useful direction for studying how caption organization affects downstream learning. A corresponding test in our setting would construct paired natural-language and JSON captions like [3] from the same factual content, measure any changes in Coverage, Precision, and length, and retrain the downstream models under matched settings. Building these paired corpora across captioners and completing the training runs is beyond what we can finish within the short rebuttal period. We will conduct this controlled comparison in future work.

We tried our best to examine sensitivity to caption length and information density below.

Inspired by the logarithmic length penalty in the conciseness metric of Parekh et al. [2], we define a log-length-penalized Coverage proxy, D=C/ln⁡(L)D=C/\ln(L), where L is each captioner’s mean caption length in tokens and ln is the natural logarithm. Unlike their content-word metric, D uses our checklist Coverage.

Since density itself is actually a deterministic nonlinear function of coverage density=coverage/log⁡(length)\text{density} = \text{coverage} / \log(\text{length}), which would induce multicollinearity, we replace C with D and fit Y=β0+βDD+βPPY=\beta_0+\beta_D D+\beta_P P, keeping Precision unchanged. We compare this model with C/P on the same captioners and downstream outcomes to assess sensitivity to a logarithmic length penalty. Further details are provided in Appendix E.

PipelinenModelβC or βD ppβP pp
CLIP-Vicuna10C+P+0.218 00˙440\.044-0.049 04˙400\.440
CLIP-Vicuna10C/lnLL+P+1.580 00˙530\.053+0.019 06˙740\.674
CLIP-Vicuna16C+P+0.114 00˙280\.028+0.019 06˙950\.695
CLIP-Vicuna16C/lnLL+P+0.730 00˙430\.043+0.058 01˙450\.145
SigLIP-Qwen310C+P+0.121 00˙220\.022-0.035 02˙550\.255
SigLIP-Qwen310C/lnLL+P+0.915 00˙190\.019+0.002 09˙260\.926
SigLIP-Qwen316C+P+0.080 00˙020\.002-0.014 05˙280\.528
SigLIP-Qwen316C/lnLL+P+0.531 00˙030\.003+0.012 04˙710\.471
SD3.5M10C+P+0.027 03˙340\.334+0.191 <00˙01<0\.001
SD3.5M10C/lnLL+P+0.204 03˙220\.322+0.199 <00˙01<0\.001
SD3.5M16C+P-0.008 07˙570\.757+0.194 <00˙01<0\.001
SD3.5M16C/lnLL+P-0.133 04˙470\.447+0.198 <00˙01<0\.001
Qwen-Image10C+P-0.076 01˙700\.170+0.234 <00˙01<0\.001
Qwen-Image10C/lnLL+P-0.596 01˙460\.146+0.212 <00˙01<0\.001
Qwen-Image16C+P-0.032 03˙230\.323+0.199 <00˙01<0\.001
Qwen-Image16C/lnLL+P-0.277 02˙020\.202+0.194 <00˙01<0\.001
Z-Image10C+P+0.023 06˙270\.627+0.091 00˙210\.021
Z-Image10C/lnLL+P+0.195 05˙810\.581+0.098 00˙020\.002
Z-Image16C+P+0.011 06˙070\.607+0.088 <00˙01<0\.001
Z-Image16C/lnLL+P+0.067 06˙360\.636+0.091 <00˙01<0\.001

n = 10 denotes the original captioners, and expanded fits use all sixteen captioners n=16n=16 for every pipeline pleasereferto∗∗fuEx′sA1∗∗fordetaileddefinitionofthisexpandedcaptionersetplease refer to **fuEx's A1** for detailed definition of this expanded captioner set.

From the sixteen-captioner fits, we can find that: 11 Density remains positively associated with understanding in both pipelines p=00˙43and00˙03p = 0\.043 and 0\.003, while Precision is not, and 22 for generation, Precision is significant in all available fits p≤00˙36p ≤ 0\.036, but Density is not. In conclusion, the main association patterns remain consistent under the logarithmic length penalty.

We hope our responses address your concerns about the contribution relative to CapMAS and the interpretation of the downstream results. We are happy to clarify any remaining points.

[1] Saehyung Lee, et al. Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coverage. ICML, 2025.

[2] Jash Parekh, Pengcheng Jiang, and Jiawei Han. 2025. Structure-Augmented Reasoning Generation. arXiv:2506.08364v3, Section 4.3.1, Eq. 1212. The cited version defines conciseness using content-word proportion divided by ln1+wordcount1 + word count, followed by max normalization.

[3]Chen, Deng, Li, Yuan & Fan 20262026. Scaling properties of text conditioning in visual generation. arXiv:2607.29679.

Thank you to the authors for the detailed response and the revisions. The added comparison clarifies the contribution relative to CapMAS, and my concerns regarding the interpretation of the downstream results have been satisfactorily addressed. As promised, I have raised my rating from 2 to 3.

We really appreciate your prompt feedback, and we are glad to hear that your concerns have been well addressed. We will carefully revise our paper based on the discussion above.

Reviewer_fuExOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  1. A valuable benchmark: dense human references, about 50 checklist items per image, and high-resolution images.
  2. A clean split: dividing caption quality into Coverage and Precision is easy to interpret.
  3. A controlled end-to-end study: only the caption source changes, and each task has two pipelines. Studies like this are expensive and rare.
Critical strengths
  • A valuable benchmark: dense human references, about 50 checklist items per image, and high-resolution images.
  • A clean split: dividing caption quality into Coverage and Precision is easy to interpret.
  • A controlled end-to-end study: only the caption source changes, and each task has two pipelines. Studies like this are expensive and rare.
Critical weaknesses

W1. The understanding result is not robust

Each regression fits two predictors to only ten captioners, leaving 7 degrees of freedom. The predictors are strongly correlated with each other (r = 0.81). The captioners are also not independent: they come from four families that share training data and caption style. I refit the regressions from Table 2 and reproduced the reported numbers, but the Coverage→understanding effect is fragile:

  • On CLIP-Vicuna, dropping any one of six of the ten captioners makes it non-significant (p up to 0.17).
  • Dropping the two LLaVA-OV captioners gives p = 0.11 on CLIP-Vicuna and p = 0.39 on SigLIP-Qwen3.

W2. Coverage may also matter for generation

I agree that Precision should come first for text-to-image training. This matches common practice: T2I caption pipelines put correctness first and often verify captions by hand, and Cheng et al. (2024) report the same finding. However, the paper goes further and concludes that Coverage "contributes little" to generation, and I am not convinced. Recent work on scaling text conditioning (Chen et al., 2026, arXiv:2607.29679) finds that generation keeps improving as captions carry more image-grounded detail. The authors use two measures of how much grounded information a caption carries. Grounded Perplexity Gain (GPG) is how much seeing the image raises a VLM's likelihood of the caption. Effective Detailness (ED) is an attribute-level F0.5 score against image-grounded references. Both closely track converged diffusion loss (r = −0.98 and −0.97), and generation quality and VLM-judged preference rise monotonically as structured prompts add grounded fields. They also find that adding words without adding facts helps little, so what scales is correct detail, that is, more coverage at fixed precision. That is exactly what CAPEval cannot isolate, because its ten captioners vary Coverage and Precision together (r = 0.81). CAPEval's own data are consistent with a real Coverage effect. On SD3.5, the 95% CI for βC\beta_C is [−0.03, +0.09], and its upper end is about half of βP\beta_P. Generation also correlates with Coverage at r = 0.83 on its own. The setup may also hide such an effect. Most of the spread in generation scores comes from GenEval, whose short prompts reward getting a few stated details right. Training truncates text at 512 tokens, and SD3.5's CLIP encoders see only 77, so the extra content in long captions may never reach the model. Fine-tuning is also only about two passes over 50K–100K images. On DPG-Bench, the only dense-prompt benchmark, Precision is no longer significant once the two LLaVA-OV captioners are removed. A fair conclusion is that Precision comes first, while the value of Coverage at fixed Precision is still open.

  • [1] Chen, Deng, Li, Yuan & Fan (2026). Scaling properties of text conditioning in visual generation. arXiv:2607.29679.
  • [2] Cheng, Patel & Yang (2024). Precision or recall? An analysis of image captions for training text-to-image generation model. Findings of EMNLP.
Questions for authors
  • What are the caption lengths per captioner, and what fraction exceed 512 and 77 tokens?
  • How many SFT captions per captioner remain after the token-length filter?
5 comments

We thank Reviewer fuEx for the valuable time and constructive feedback. We are truly grateful for your description of CAPEval as "a valuable benchmark" with "dense human references", for finding the Coverage and Precision split "easy to interpret", and for recognizing our "controlled end-to-end study" as one that is "expensive and rare". We particularly appreciate that you refit the regressions from Table 2, reproduced our reported numbers, and gave concrete, checkable analyses. Point-to-point responses are provided below.

W1: The understanding result is not robust.

A1: We acknowledge that the statistical significance is sensitive to captioner selection. Therefore, we have tried our best to expand our experiments within the very limited rebuttal period. Specifically, we have added six captioners from four new families, including seed-2.0-lite, seed-2.0-pro, seed-2.1-lite, deepseek-v4.1-flash-1017, gemma3-12b-it, and kimi-vl-16b-a3b. This expansion actually requires six captioning passes over 1.2M images and 30 downstream training runs, with 712 H20 GPU-hours required per additional captioner.

We rebuild understanding regression models based on this expanded set, and we expand the statistical significance analysis for both 11 the full set, i.e., all 16 captioners, 22 the subset without two LLaVA-OV captioners, and 33 the worst subset (the worst subset indicates the least significant regression model). Across these three different settings, Coverage remains positively associated with understanding significantly. Therefore, after appropriate expansion, the Coverage→understanding effect is no longer fragile.

Please refer to Appendix C for detailed results.

PipelineCaptionersbeta_CpCbeta_PpP
CLIP-Vicunaall 16+0.1140.028+0.0190.695
CLIP-Vicunadrop one (worst case)+0.0870.096+0.0410.406
CLIP-Vicunadrop two LLaVA-OV+0.0850.110+0.0920.194
SigLIP-Qwen3all 16+0.0800.002-0.0140.528
SigLIP-Qwen3drop one (worst case)+0.0700.007+0.0040.864
SigLIP-Qwen3drop two LLaVA-OV+0.0610.021+0.0260.426

W2 & Q1: Coverage in generation and caption truncation.

A2: We appreciate this constructive suggestion, and we recognize that our generation evaluation setting may have minor issues. Therefore, we refine our setting in three aspects:

  1. For the downstream benchmark, we regard DPG-Bench as our main metric, since this benchmark is designed to test the instruction-following capabilities under dense prompts.

  2. For the caption truncation issue that might occur in SD3.5M, we add Z-Image, which uses Qwen3-4B as the text embedding module.

  3. For statistical sensitivity, we also perform the generation evaluation under the expanded set (similar to the understanding in W1) with 16 captioners in total. Meanwhile, we report the statistical significance before and after dropping two LLaVA-OV captioners.

Main results are provided in the following. Key findings include:

  • Overall, Precision is much more significant across different captioner sets, benchmarks, and generators. For instance, using 16 captioners, the p-value of Precision remains smaller than 0.001 on DPG-Bench for both SD3.5M, Qwen-Image, and Z-Image.

  • Coverage appears to be significant under some specific settings. After dropping two LLaVA-OV captioners on DPG-Bench, the p-value of Coverage becomes 0.048 and 0.032 for SD3.5M and Qwen-Image, respectively. Therefore, Coverage does contribute to text-to-image generation.

Based on our new results, Precision shows a more consistent association with generation performance, but Coverage does contribute sometimes. We therefore revise our conclusion, acknowledging the contribution of Coverage in Section 4.

PipelineBenchmark10 captioners: beta_C pCpC10 captioners: beta_P pPpP16 captioners: beta_C pCpC16 captioners: beta_P pPpP
SD3.5Moverall+0.027 03˙340\.334+0.191 <00˙01<0\.001-0.008 07˙570\.757+0.194 <00˙01<0\.001
SD3.5Moverall, drop two LLaVA-OV+0.039 03˙090\.309+0.163 00˙130\.013+0.017 05˙260\.526+0.138 00˙020\.002
SD3.5MDPG-Bench+0.005 08˙370\.837+0.100 <00˙01<0\.001+0.013 05˙070\.507+0.088 <00˙01<0\.001
SD3.5MDPG-Bench, drop two LLaVA-OV+0.036 02˙850\.285+0.050 02˙400\.240+0.038 00˙480\.048+0.035 01˙670\.167
Qwen-Imageoverall-0.076 01˙700\.170+0.235 <00˙01<0\.001-0.032 03˙230\.323+0.199 <00˙01<0\.001
Qwen-Imageoverall, drop two LLaVA-OV+0.007 08˙450\.845+0.080 01˙360\.136+0.016 03˙870\.387+0.092 00˙040\.004
Qwen-ImageDPG-Bench-0.014 06˙690\.669+0.098 <00˙1<0\.01+0.017 03˙590\.359+0.081 <00˙01<0\.001
Qwen-ImageDPG-Bench, drop two LLaVA-OV+0.040 03˙350\.335+0.005 09˙210\.921+0.040 00˙320\.032+0.030 02˙040\.204
Z-Imageoverall+0.023 06˙270\.627+0.091 00˙210\.021+0.011 06˙070\.607+0.088 <00˙01<0\.001
Z-Imageoverall, drop two LLaVA-OV+0.089 02˙040\.204-0.017 08˙320\.832+0.034 01˙370\.137+0.040 01˙920\.192
Z-ImageDPG-Bench-0.034 03˙680\.368+0.090 00˙070\.007-0.019 02˙960\.296+0.078 <00˙01<0\.001
Z-ImageDPG-Bench, drop two LLaVA-OV+0.013 07˙970\.797+0.017 07˙810\.781-0.004 08˙340\.834+0.049 00˙890\.089

Moreover, based on your suggestion, we analyze the caption truncation issue on SD3.5M. Among the original ten captioners, 99.44% of captions exceed the CLIP limit of 77 tokens, while 8.18% exceed the T5 limit of 512 tokens. Thus, we agree that SD3.5M alone cannot fully assess the value of additional Coverage, and thus we provide additional Z-Image experiments. Detailed numbers are reported below.

Captioner>77 tokens (CLIP tokenizer)>512 tokens (T5 tokenizer)
FractionTruncated TokenFractionTruncated Token
Qwen3-VL-4B99.99%83.24%13.16%0.4477%
Qwen3-VL-8B100.0%83.08%13.39%0.4422%
Qwen3-VL-32B99.98%83.49%16.93%0.6239%
GLM-4.6V-Flash99.99%84.54%37.57%1.5760%
InternVL3.5-1B98.97%72.30%0.0380%0.0012%
InternVL3.5-4B99.79%75.49%0.0740%0.0044%
InternVL3.5-8B99.83%78.26%0.2620%0.0084%
InternVL3.5-38B99.97%79.35%0.2440%0.0076%
LLaVA-OV-4B98.77%61.10%0.008%0.0006%
LLaVA-OV-8B97.08%61.85%0.1040%0.0108%
Overall(Original 10)99.44%78.61%8.179%0.4152%
Seed 2.0 Pro100.0%88.79%87.82%26.65%
Seed 2.0 Lite100.0%90.38%97.92%36.16%
Seed 2.1 Lite99.93%86.49%61.26%16.41%
DeepSeek 4.1 Flash99.99%89.55%96.15%30.78%
Gemma-3-12B-IT99.98%84.97%49.28%8.517%
Kimi-VL-A3B-Instruct99.85%62.12%0.1480%0.3069%

Q2: Number of SFT captions per captioner after the token-length filter.

A3: We apologize for omitting the filtering details. Each of the sixteen captioners provides 1,246,901 captions before filtering. Due to the restriction of GPU memory, we apply sequential length checks because we keep max_model_len=2048 when training understanding models. The table below reports the exact token numbers removed and final training captions retained, excluding the common base SFT data. We have added these filtering details and counts to Appendix D.1.

CaptionerTokens removedCaptions retained
Qwen3-VL-4B0.008%1,246,873
Qwen3-VL-8B0.011%1,246,699
Qwen3-VL-32B0.021%1,246,424
GLM-4.6V-Flash0.022%1,246,458
InternVL3.5-1B01,246,901
InternVL3.5-4B0.001%1,246,898
InternVL3.5-8B0.001%1,246,899
InternVL3.5-38B0.007%1,246,880
LLaVA-OV-4B0.001%1,246,899
LLaVA-OV-8B0.001%1,246,899
Gemma3-12B-IT0.001%1,246,877
Kimi-VL-16B-A3B0.013%1,246,596
DeepSeek-V4.1-Flash0.158%1,243,833
Seed-2.0-Lite0.094%1,245,731
Seed-2.1-Lite0.033%1,246,334
Seed-2.0-Pro0.052%1,246,167

We hope the revised claims now match what our data could support. We are happy to clarify any remaining points.

Comments on the paper

1 comment

We sincerely thank all reviewers for their valuable and constructive feedback. We are delighted by your positive assessments of our work. We thank Reviewer fuEx for describing CAPEval as “a valuable benchmark” with “dense human references,” finding the Coverage and Precision split “easy to interpret,” and recognizing our “controlled end-to-end study” as “expensive and rare”. We also particularly appreciate your independent refitting of the regressions and reproduction of our reported numbers. We thank Reviewer NNYB for finding our question “useful and practically relevant,” our empirical study “substantial,” and the paper “well organized and easy to follow,” and for noting that the controlled comparisons “provide convincing support for the observed task-dependent associations.” We thank Reviewer dmnc for recognizing the Coverage and Precision split as “a useful perspective on caption quality beyond a single aggregate score” and noting that our downstream experiments “connect caption evaluation with practical utility across both understanding and generation.” We thank Reviewer T854 for describing CAPEval as a “high-quality fine-grained evaluation dataset” and “a valuable resource for fine-grained caption evaluation,” and for recognizing our “systematic downstream validation” and its “practical insights for multimodal data curation.” We have conducted additional experiments to address the concerns about statistical robustness, factual Precision, judge reliability, and downstream utility. All revisions are marked in purple. Specifically:

  • @Reviewer fuEx and Reviewer dmnc, we have expanded the downstream study from ten to sixteen captioners byaddingseed−20˙−lite,seed−20˙−pro,seed−21˙−lite,deepseek−v41˙−flash−1017,gemma3−12b−it,andkimi−vl−16b−a3bby adding seed-2\.0-lite, seed-2\.0-pro, seed-2\.1-lite, deepseek-v4\.1-flash-1017, gemma3-12b-it, and kimi-vl-16b-a3b across eight model families, where the Coverage–understanding remains positive association under every single-captioner exclusion and after removing both LLaVA-OV captioners.

  • @Reviewer fuEx, we have added Z-Image generation experiments and provided separate analyses on DPG-Bench. Based on our new results, Precision shows a more significant positive association with generation performance, but Coverage does contribute sometimes.

  • @Reviewer dmnc and Reviewer T854, we have verified caption claims outside the ground-truth checklist and incorporated them into extended Precision, keeping Coverage unchanged. Precision remains positively associated with generation performance under both definitions. These results support the broad consistency of our main conclusions across the original and extended Precision definitions.

  • @Reviewer dmnc and Reviewer T854, to assess the reliability of Qwen2.5-72B as a checklist judge, we have compared it with GPT-6 Astra and validated both judges against human annotations on 1,600 caption–checklist pairs. The high agreement  95\>95% supports the reliability of the automated checklist judgments. We also report regressions across judges, and all associations remain statistically significant.

  • @Reviewer NNYB and Reviewer T854, we have introduced caption information density into our regression analysis. Since density is actually a deterministic nonlinear function of coverage density=coverage/log⁡(length)\text{density} = \text{coverage} / \log(\text{length}), which would induce multicollinearity, we replace coverage with density and use only density and precision as predictors. Our main conclusions hold: coverage (or density) is significant for understanding tasks, while precision is significant for generation tasks.

  • @Reviewer NNYB, we clarify the distinction from CapMAS and the interpretation of our results: our main contribution is linking caption-quality dimensions to downstream utility through end-to-end training experiments, and the regressions characterize associations rather than separate causal effects.

  • @Reviewer dmnc, we evaluated predictive transfer to six additional captioners using regressions fitted only on the original ten. We use Spearman's ρ, computed from the reported Rank columns, which measures agreement between predicted and observed rankings. It is 0.714 for CLIP-Vicuna, 0.829 for SigLIP-Qwen3, 1.000 for SD3.5M, 0.714 for Qwen-Image, and 0.943 for Z-Image. These results indicate strong ranking agreement across all five pipelines, suggesting that the predictions largely preserve relative captioner performance.

  • @Reviewer fuEx, we have provided the token-length filtering rules and exact token removal and caption retention counts for all sixteen captioners.

We welcome any further questions or suggestions and would be happy to provide additional clarification and revise the paper accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.