- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies the trade-off between object hallucination and object coverage in image captioning. It compares LLaVA-1.5 with the masked-diffusion model LLaDA-V and proposes COVER, which uses autoregressive captions for broad object proposals and diffusion captions for conservative evidence. COVER builds an object plan from source agreement and real-image versus null-image scores, realizes the plan with visual residual guidance, and appends a bounded number of verified omissions. The main empirical result is a useful high-recall operating point: on MSCOCO, sentence-level CHAIR falls from 0.597 to 0.429 while recall changes from 0.797 to 0.786. The paper also provides a probabilistic decomposition of admission, realization, and recovery, with evaluations on AMBER, Object HalBench, and OpenCHAIR.
Critical strengths
-
The paper takes the coverage side of hallucination mitigation seriously. Reporting recall alongside CHAIR makes the central result more informative than a comparison of hallucination rates alone. The MSCOCO improvement is consistent across the six pools in Table 5, and the AMBER result is also encouraging: COVER increases Cover from 52.4 to 59.3 while reducing CHAIR from 10.3 to 6.8 (Table 2, p. 8; Appendix C, p. 13).
-
The component analyses are more useful than a simple end-to-end ablation. In particular, holding the object plan fixed in Table 12 shows that VRG reduces unsupported realization from 0.7793 to 0.7165 while supported realization changes from 0.9237 to 0.9116. The admission audit and recovery identities likewise make the sources of the trade-off explicit, including the fact that append-only recovery cannot improve sentence-level CHAIR over its scaffold (Appendices E, J.2, and L.3, pp. 14, 20, and 22).
Critical weaknesses
-
The evidence supports the usefulness of this particular pipeline more clearly than the claimed architectural explanation. The comparison uses one autoregressive checkpoint and one diffusion checkpoint, so architecture changes together with training and model capabilities. Table 1 usefully acknowledges that strict LLaVA can outperform direct LLaDA-V on both metrics, but the uncertainty observations do not establish that revisability causes the final gains. COVER also combines multiple proposals, visual verification, sentence filtering, and template recovery. There is no corresponding autoregressive propose/verify/realize/recover control with comparable proposals and inference effort, and the fixed-plan VRG experiment only varies guidance within LLaDA-V. This leaves open how much of the benefit actually requires diffusion (Sections 3–4, pp. 4–7; Table 12, p. 20).
-
The evidence for broadly useful captioning is less complete than the closed-ontology results suggest. The pipeline extracts canonical object sets, deletes sentences with out-of-plan objects, and restores coverage with fixed-template additions; the evaluation therefore needs to establish how well it transfers beyond canonical object inclusion. On OpenCHAIR, COVER is worse than both DoLa and AdaIAT on both reported axes, so there is no demonstrated advantage there (Table 2). Moreover, Section 5.1 specifies only a “released evaluation subset,” without its size or selection, and the AI-use statement says GPT-5.4 evaluates an explicitly labeled OpenCHAIR subset, but I could not locate the subset definition or judging protocol in the appendices. The mapping used for planning/filtering outside COCO and the quality of the resulting full captions also remain unclear (Sections 4.2–4.3 and 5.1, pp. 6–7; AI-use statement, p. 10; Appendices B and I).
Questions for authors
-
Can you test an autoregressive realization/verification control with the same candidate plan, filtering, and recovery budget, and report its hallucination–coverage trade-off together with latency or model-forward cost? This would help determine whether the central advantage comes from masked diffusion or from the surrounding verification pipeline.
-
Please specify the OpenCHAIR subset size and selection, object extraction and support-judging protocol, GPT-5.4 prompt/settings and unresolved-case handling, and how the planning vocabulary and sentence filter operate on OpenCHAIR and AMBER. Does a small blinded comparison of full-caption factuality and usefulness support the coverage gains after template recovery?
-
Were the recovery settings fixed before transfer evaluation? Table 2's Object HalBench result (0.410 sentence-level CHAIR and 0.769 recall) corresponds to the “strict” row of Table 11 rather than its “default” row. Please explain the choice of variant and how thresholds, caps, and the global gain median were selected and transferred across benchmarks.
4 comments
Response 2: OpenCHAIR/AMBER protocol and full-caption quality
Response 2: OpenCHAIR/AMBER protocol and full-caption quality
Thank you for prompting us to clarify the complete OpenCHAIR and AMBER protocols. We address each requested item in turn below.
Subset size and selection. OpenCHAIR requires a separate GPT-5.4 support judgment for every object extracted from a caption, resulting in thousands of calls per method. Evaluating the complete released 4,863-image test split would therefore incur substantial time and token cost. We use the fixed first 500 records of the released test split for every method, without selecting images according to model outputs or evaluation results. The generation prompt is Describe this image in detail.
Object extraction. OpenCHAIR lowercases each caption, extracts spaCy NOUN lemmas, and retains nouns with Brysbaert concreteness greater than 4.5. The protocol marks painting/drawing/photo/picture/portrait/photograph as ignore.
Support judging and GPT-5.4 settings. For each extracted object, GPT-5.4 receives:
Here are a few descriptions of an image: {reference}.
Does the image contain the following object: {object}?
Answer yes/no/unsure.
The answer is:
We use temperature=0 and max_tokens=8. Technical failures are retried, and the latest valid yes/no/unsure/ignore result is retained for each (run, image, object_index, object). Resolved OCH is No/(Yes+No); unsure and ignore cases are reported separately. The same-pass COVER row in Table 2 contains 3521 Yes, 529 No, 1490 Unsure, and 20 Ignore, yielding OCH=529/(3521+529)=0.1306.
Planning vocabulary and sentence filtering. COVER uses COCO-80 as a compact canonical planning space to control the cost of candidate canonicalization, Visual Gain verification, sentence filtering, and recovery. We deliberately retain this planning space on OpenCHAIR rather than expanding or retuning the Object Plan to its open vocabulary. Besides improving computational efficiency, this design allows OpenCHAIR to test the generality of COVER’s unchanged pipeline under broader-vocabulary evaluation rather than evaluating a benchmark-specific version.
The final captions are evaluated by an independent open-vocabulary evaluator, so non-COCO nouns are still extracted and judged. Sentences are split using (?<=[.!?])\s+. A sentence containing an internal COCO object outside the Object Plan is removed; if all sentences are removed, a deterministic Object Plan template is used. Filtering precedes recovery.
For all 1,004 AMBER images, COVER instead uses the AMBER-specific internal object space. Attributes and relations in the AMBER evaluator are downstream evaluation concepts rather than Object Plan inputs. The saved captions exactly match the official evaluator input and reproduce (CHAIR, Cover, Hal, Cog)=(6.8, 59.3, 33.1, 2.3).
Full-caption quality after template recovery. We further compare complete captions before and after bounded recovery:
| Stage | CLIPScore ↑ | SPICE ↑ | CHAIRs ↓ | Recall ↑ |
|---|---|---|---|---|
| Scaffold | 0.83480 | 0.15371 | 0.378 | 0.7400 |
| Final | 0.83488 | 0.15449 | 0.432 | 0.7776 |
Recovery changes 96/500 captions, adds 1.416 words per caption on average, and increases Recall by 3.76 points. COVER preserves the same level of image–text alignment and scene-graph semantic quality after recovery: neither CLIPScore nor SPICE decreases, and no exact duplicate sentence is introduced.
For broader evaluation, we added 1,000 MSCOCO val2014 images disjoint from H-compare500, yielding 1,500 unique images. On this set, COVER obtains a reference-free CLIPScore of 0.8359, the highest among the four compared methods.
It is worth emphasizing that the annotation-based benchmarks already establish COVER’s effectiveness: COVER achieves the highest AMBER Cover, the lowest CHAIRs in the matched-coverage MSCOCO comparison, and the lowest Object HalBench CHAIRs among methods with Recall above 0.76. OPA-DPO attains lower hallucination through preference training but with substantially lower coverage, while DoLa and AdaIAT are currently stronger on OpenCHAIR.
The current OpenCHAIR experiment retains the compact COCO-80 planning space and therefore leaves clear headroom. Expanding this space to OpenCHAIR’s long-tail object categories would allow COVER to verify and recover a broader range of open-vocabulary objects, providing a direct path toward a stronger and potentially leading OpenCHAIR operating point.
Response 3: Object HalBench recovery settings
Response 3: Object HalBench recovery settings
Thank you for noting the correspondence between Table 2 and the variants in Table 11. Table 2 reports the strict configuration. We clarify below why this variant was selected and how its thresholds and caps differ from the other two configurations.
We compared three complete recovery configurations on the same 300 Object HalBench images. They use the same models, Object Plan construction, VRG, sentence filter, and recovery template, while varying only the thresholds and caps controlling candidate admission and recovery volume.
| Variant | Plan threshold / proposer cap | Recovery cap | Global quantile | Low-tier cap | CHAIRs ↓ | Recall ↑ |
|---|---|---|---|---|---|---|
| Default | 0.25 / 3 | 4 | 0.50 | 2 | 0.4167 | 0.7597 |
| Strict | 0.00 / 5 | 5 | 0.70 | 3 | 0.4100 | 0.7692 |
| High recall | 0.00 / 5 | 4 | 0.40 | 2 | 0.4467 | 0.7775 |
- Choice of variant. Table 2 uses strict because it improves both reported axes over default: lower sentence-level CHAIR (0.4100 vs. 0.4167) and higher Recall (0.7692 vs. 0.7597). It therefore performs best on the two metrics shown in Table 2. High recall increases Recall further but also increases hallucination.
- Threshold selection. The plan threshold controls the Visual Gain required for proposer-only candidates to enter the Object Plan. Default uses 0.25; strict and high recall use 0.00, allowing more proposer-only candidates to enter the Object Plan and become eligible for bounded recovery.
- Cap selection. The proposer cap bounds proposer-only objects in the Object Plan, the recovery cap bounds all recovered objects, and the low-tier cap separately bounds proposer-only recovery. Default, strict, and high recall respectively use
(3,4,2),(5,5,3), and(5,4,2). - Global gain threshold. This threshold is computed from the Visual Gain distribution of low-tier candidates across all images and is candidate-weighted. Default uses the median (
q=0.50), strict uses the 70th percentile, and high recall uses the 40th percentile. - Transfer across benchmarks. The Object HalBench numerical settings were not transferred unchanged from MSCOCO. The strict setting was selected from the three configurations evaluated on the same 300 Object HalBench images; therefore, the reported Object HalBench row uses benchmark-specific settings rather than a recovery policy frozen on MSCOCO. What transfers unchanged is the propose–plan–realize–filter–recover pipeline, the Visual Gain definition, and the role of each threshold and cap. All three configurations are reported in the appendix section J.
Cost of the Same-Plan Realizer Control
We appreciate this practical suggestion, because comparable inference effort is essential for interpreting the same-plan control. We profile the complete image-to-final-caption boundary on an RTX A6000 with batch size 1 and resident models, holding the Object Plan, filtering, and recovery policy fixed as in our earlier response.
| Final realizer | Warm s/image | Cross-validated FLOPs/image | Recall$\uparrow$ | ||
|---|---|---|---|---|---|
| Same-plan AR-P | 26.334 | 0.97--1.06 PFLOPs | 0.394 | 0.1768 | 0.7470 |
| Masked diffusion + VRG | 27.323 | 0.99--1.08 PFLOPs | 0.432 | 0.1338 | 0.7776 |
The two realizers therefore operate at comparable total wall-clock and estimated compute, with an end-to-end difference of only 0.99 s/image. AR-P attains lower sentence-level CHAIR by producing substantially shorter captions, while masked diffusion restores 3.06 Recall points and lowers the unsupported fraction among object mentions by 4.30 points. This strengthens the attribution that masked diffusion reaches its higher-coverage, better mention-grounded operating point without a substantially larger inference-time allocation.
To identify the shared cost in this control, we synchronize CUDA around every pipeline stage, record per-image wall time, and aggregate five 50-image warm trials. Proposal Visual Gain takes 7.74 s/image, CaptionAug 5.26 s, direct diffusion 3.89 s, the LLaVA proposal 3.32 s, VRG realization 5.85 s, and recovery Visual Gain 1.13 s. Thus, the comparison includes the same costly discovery and verification path, while the final realizer changes the resulting hallucination--coverage regime.
Because the measured cost is distributed across separable stages, COVER can expose lighter operating modes through the same stage switches used in our ablations, selecting proposal, verification, and recovery modules for the desired latency--quality budget. Enabled stages can then be optimized independently by parallelizing proposal views, batching real/null scoring, and caching repeated visual features. This modular deployment supports offline, asynchronous, or on-demand verified-description applications.
Response 1: Same-Plan AR Control and Realizer Contribution
Thank you for the valuable and specific comments. We began the requested experiments as soon as we received the review. We have now (i) run a same-plan autoregressive realizer control, including diffusion with and without VRG; (ii) audited the complete OpenCHAIR and AMBER planning/filter/evaluation protocols; (iii) evaluated full-caption quality before and after recovery; and (iv) verified all Object HalBench recovery configurations and their selection. We are also profiling inference cost and will report the completed results before the rebuttal deadline. We answer each question below.
Response 1: Same-Plan AR Control and Realizer Contribution
We construct H(held-out)-compare500 by deterministically selecting 500 images from a disjoint 2,000-image held-out MSCOCO pool without inspecting captions, labels, or metrics. The LLaDA-V realizer and LLaVA-1.5 same-plan autoregressive realizer (AR-P) receive the identical frozen Object Plan and semantic prompt, followed by the same sentence filter, recovery eligibility, gates, caps, and templates. We report diffusion without VRG to separate realization architecture from diffusion-specific guidance.
Applying the shared plan/filter/recovery components through AR-P lowers CHAIRs from the detailed LLaVA operating point of 0.592 to 0.394, showing that the architecture-agnostic pipeline is effective. AR-P pays for this reduction with shorter captions and lower Recall. Replacing the AR final realizer with masked diffusion restores approximately three Recall points and reduces CHAIRi from 0.1768 to 0.1338. The no-VRG comparison shows that this contribution is not solely produced by guidance: with the plan, filtering, and recovery fixed, masked diffusion realizes more supported plan content with a lower hallucinated fraction among mentions.
VRG then adds a uniquely non-autoregressive mechanism. In the fixed-plan realization study (Table 13), VRG reduces unsupported realization from 0.7793 to 0.7165 while preserving most supported realization (0.9237 to 0.9116). It can reorder unresolved positions and revise low-confidence predictions before fixation; standard left-to-right AR cannot use this fixation-priority branch. Thus, shared verification/postprocessing first lowers sentence-level hallucination, masked-diffusion realization restores coverage and improves mention-level grounding, and VRG operationalizes selective commitment.
The complete control is reported in the appendix section J. The common-boundary latency and model-forward profile remains in progress and will be updated before the deadline.