- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ZoomCap, which uses an initial image caption to guide retrieval, OCR, and local zooming, then incorporates geometric constraints to generate detailed captions. It also introduces FactCycle, which evaluates factuality and recoverable information through atomic claims assessed against the original and reconstructed images. Experiments cover caption quality, human preferences, ablations, and downstream training across multiple backbones and application settings.
Critical strengths
- The overall effectiveness is supported by several types of evaluation. The paper combines automatic evaluation, human preferences, and downstream training, reporting gains across three backbones and both general and medical domains. These results provide useful evidence for the practical value of the generated captions.
- The approach to evidence acquisition and budget control is well motivated. Table 2(b) directly compares the full second-level pyramid with complexity gating. The gated version achieves higher preference scores with fewer crop inputs, supporting selective acquisition of local information.
Critical weaknesses
-
Appendix E’s agreement and sensitivity checks and the DINO results are useful, but they do not directly test the accuracy of FactCycle’s extraction or judgments against either image. PrefBench’s human study evaluates caption preferences on different pairs. Since FCtrans counts facts, the paper should also clarify how facts are split and semantic duplicates are removed.
-
Lines 317–319 describe earlier cycle metrics as measuring only reconstruction-side support and argue that they reward fabricated but rendered facts. However, CLIP-Image-Score and CAMScore both compare original and reconstructed images; CAMScore also considers objects and spatial relations. FactCycle’s fact-level checks and attribution are useful, but the comparison with these methods needs to be more accurate.
Questions for authors
-
How are fact granularity and semantic duplicates handled during extraction? Could the authors provide a small human audit across caption sources and fact categories, checking extraction faithfulness and completeness, as well as judgment accuracy on both images?
-
Lines 295–312 average the judges’ per-claim scores before combining the two anchors, then average rates over images. Appendix E instead pools source-image judgments and uses reconstruction-side majority judgments, excluding disagreements. Its pooled counts also reproduce Table 1’s transferred percentages at the reported precision, e.g., . Please reconcile these descriptions, specifying judge aggregation, macro- versus micro-averaging, and the treatment of disagreements and unverifiable claims. Please also give Table 9’s numerators, denominators, and aggregation rules, and explain their relationship to Table 1.
-
Appendix A provides the shared training recipe, and Table 10 lists image sources. Do ZoomCap and ScaleCap in Table 5 use exactly the same 450K images? Please also report the maximum CPT sequence length, truncation policy, and proportion of truncated samples for each corpus to clarify the scope of this comparison.
3 comments
Human Validation and Interventional Validity with Exact Aggregation – Part2
Q4: One ledger for Section 3.6/Appendix E/Tables 1/9. Truthfulness : two source votes/claim, then image macro-average. : pooled source decisions and reconstruction majority/claim; split/unverifiable rows are excluded from both-decidable denominators.
| Quantity | ZoomCap | ScaleCap | ShareGPT4V |
|---|---|---|---|
| Both-decidable rows | 28,794 | 17,854 | 9,226 |
| Both-supported rows | 26,184 | 15,749 | 7,411 |
| Transfer rate | 90.9% | 88.2% | 80.3% |
| Reconstruction-nondecidable rows dropped (split or unverifiable) | 2,462 | 1,301 | 444 |
| 52.4 | 31.5 | 14.8 |
One estimator. CSV (image_id,claim_id,source_votes,reconstruction_votes,split_flag) and script regenerate Tables 1/9 from 250 FactCycle images, with the pooled denominator (2N=500): and . Strict macro 56.5/33.7/15.6 is sensitivity-only.
Table 9 uses source-decidable : fabrication is 1,594/31,256=5.1%, 1,896/19,155=9.9%, 1,779/9,670=18.4%; fabricated-then-rendered is 1,031/1,594=64.7%, 1,375/1,896=72.5%, 1,567/1,779=88.1%. The rendered set partitions exactly as , where source-unverifiable rendered counts are 970/978/320: 28,185=26,184+1,031+970, 18,102=15,749+1,375+978, 9,298=7,411+1,567+320. Thus true-among-rendered is 26,184/28,185=92.9%, 15,749/18,102=87.0%, 7,411/9,298=79.7%. Source-unverifiable rows enter only , never or ; the script asserts all three partition identities.
Q5: Identical images/truncation. Qwen2.5-7B CPT on the exact 450,000 official ScaleCap images gives 62.74 ZoomCap (62.70 original; 60.48 ScaleCap), max_seq_len=4,096, right truncation, 768 visual/3,328 text tokens. Truncation is 0.00% for ShareGPT4V/DenseFusion/ScaleCap and 0.18% (810/450K) for ZoomCap; mean/P99 text lengths are 831/2,290 tokens.
Related evidence in other responses. Generator-level robustness and corpus-level/downstream analyses are reported in Reviewer yk9V, Q1 and Q3–Q4. Complementary interventional evidence includes the CCG/draft and crossover controls in Reviewer JUgd, Q2–Q3, while human-preference/verbosity analysis and full-FactCycle draft-corruption robustness appear in Reviewer Saef, Q3–Q4.
We hope the human audit, identical-image controls, and reproducible aggregation ledger address your concerns. If these results resolve the corresponding weaknesses, we would be grateful for an updated assessment.
Official Comment by Reviewer Ceoo
Thank you for the detailed response and additional experiments. The fact-level human audit addresses my concern about the lack of direct validation for FactCycle. The clarifications on prior metrics, training-image consistency, and truncation also resolve some of my questions.
My main remaining concern is how an individual fact is defined. Figure 4 provides simple examples, and the authors explain how semantic duplicates are merged, but the specific splitting rules remain unclear. For example, “A white bathtub is present” counts as one fact. Would “A bathtub is present. The bathtub is white” yield the same fact count? Also, the human audit reports only overall extraction completeness, without separate results for ZoomCap, ScaleCap, and ShareGPT4V. It therefore remains unclear whether fact splitting and extraction omissions affect the methods differently.
And the clarification that CLIP-Image-Score and CAMScore both compare the source and reconstructed images addresses my concern about prior metrics. I suggest correcting the relevant statements in the revised manuscript as well.
FactCycle is a central contribution of the paper, but these unresolved counting issues still limit my confidence in the reported improvement in information transfer. The additional experiments strengthen the empirical evidence, but do not sufficiently change my assessment of the overall contribution. I therefore maintain my current score.
Human Validation and Interventional Validity with Exact Aggregation
Dear Reviewer Ceoo,
Thank you. We provide human audit, accurate prior-metric boundary, one reproducible ledger and identical-image CPT controls.
Overview. We address your concerns in five parts: (1) atomic claims and human validation, (2) controlled selection on identical images, (3) the prior-metric boundary, (4) exact aggregation, and (5) identical-image CPT and truncation controls.
Q1: Atomic claims and human validation. Coreference IDs and normalized-triple/BGE-large merging (cosine >0.88 or entailment) consolidate semantic duplicates while preserving negations and counts. We evaluate all 250 FactCycle images and 750 captions across three caption sources and seven fact categories. Five professional annotators participate in the audit, with three independent blind votes per claim and adjudication under a frozen protocol.
For extraction evaluation, annotators independently decompose the caption text into atomic assertions. Extraction faithfulness measures whether an extracted claim is entailed by the caption, while extraction completeness measures the coverage of Human-identified caption assertions by the frozen extractor output. This separation ensures that omissions in the original caption are not incorrectly charged to the extractor.
For image-grounded validation, annotators assess the same claims against the source images and reconstructions. Corpus identities and VLM verdicts remain hidden. The audit does not modify captions, extractor claims, VLM votes, prompts, thresholds, crops, or checkpoints.
The audit indicates that the extractor largely preserves the meaning of the claims it emits, while approximately one in seven Human-identified caption assertions is missed, merged, or otherwise not recovered as an independent atomic claim. Source-image and reconstruction judgments show substantial but non-perfect Human–VLM agreement. We therefore treat the VLM judges as automatic proxies with measurable residual disagreement rather than as infallible ground truth, and report the two anchors separately. Human ratings never alter captions, VLM votes, or denominators; the original 250-image estimator with 500 pooled source rows in the denominator is unchanged.
Q2: Controlled selection on identical images. Each metric selects one of ShareGPT4V/ScaleCap/noisy-crop-concatenation/ZoomCap captions on 100K identical images; Qwen2.5-7B CPT fixes images, steps, optimization and 11-benchmark evaluation. ZoomCap-450K and ScaleCap-450K also use exactly the same 450K images, so image composition is held fixed in both the 450K CPT comparison and this 100K selection study.
Dual-anchor selection improves source/reconstruction-only by +1.14/+0.82 and the strongest alternative by +0.76 on identical images.
Q3: Prior-metric boundary. CLIP-Image-Score/CAMScore compare reconstruction with source; CAMScore checks objects/relations. FactCycle adds claim-level dual-anchor attribution, separating fabricated-but-rendered claims from true-but-unrendered generator loss.