acceptodds
Under review as a conference paper at ICLR 2027

Exploring and Mitigating Object Hallucination with Non-Autoregressive Vision-Language Models

Abstract

Vision-Language Models (VLMs) support a growing range of multimodal applications, yet their generated descriptions remain prone to object hallucination. A conservative caption can avoid unsupported mentions by saying less, but the resulting loss of coverage limits usefulness; this tension appears clearly across representative generation paradigms. Autoregressive VLMs such as LLaVA-1.5 describe more of the scene but hallucinate more objects, whereas non-autoregressive masked-diffusion VLMs such as LLaDA-V achieve much lower hallucination but leave more visible content unmentioned. We trace this contrast to how the two generation architectures resolve uncertainty: autoregressive decoding makes each emitted token an immediate, irreversible part of the context, whereas masked diffusion can revise low-confidence predictions before fixation. We formalize this distinction as *selective object commitment*, separating broad object discovery from the final decision to assert each candidate. Specifically, we decompose final commitment into proposal admission, diffusion realization, and omission recovery, and derive the stagewise selectivity needed to expand coverage without proportionally increasing hallucination risk. Guided by this formulation, we introduce COVER (Cross-view Object Verification and Evidence-guided Recovery), a training-free method that combines autoregressive and diffusion proposal views, builds an object plan from cross-view agreement and object-level real/null visual gain, applies the same contrast through Visual Residual Guided diffusion decoding (VRG) to guide token selection and fixation, and restores verified omissions through bounded recovery. On MSCOCO evaluations, COVER reduces sentence-level CHAIR by 28.1% while retaining 98.7% of the detailed proposer's object Recall. COVER establishes proposal-commitment separation as a practical cross-architecture principle for grounded multimodal generation. *The source code for our method will be made available upon publication.*

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language models, object hallucination, Non-Autoregressive Vision-Language models, hallucination mitigation, masked diffusion, visual grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V82ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the trade-off between object hallucination and object coverage in image captioning. It compares LLaVA-1.5 with the masked-diffusion model LLaDA-V and proposes COVER, which uses autoregressive captions for broad object proposals and diffusion captions for conservative evidence. COVER builds an object plan from source agreement and real-image versus null-image scores, realizes the plan with visual residual guidance, and appends a bounded number of verified omissions. The main empirical result is a useful high-recall operating point: on MSCOCO, sentence-level CHAIR falls from 0.597 to 0.429 while recall changes from 0.797 to 0.786. The paper also provides a probabilistic decomposition of admission, realization, and recovery, with evaluations on AMBER, Object HalBench, and OpenCHAIR.

Critical strengths
  1. The paper takes the coverage side of hallucination mitigation seriously. Reporting recall alongside CHAIR makes the central result more informative than a comparison of hallucination rates alone. The MSCOCO improvement is consistent across the six pools in Table 5, and the AMBER result is also encouraging: COVER increases Cover from 52.4 to 59.3 while reducing CHAIR from 10.3 to 6.8 (Table 2, p. 8; Appendix C, p. 13).

  2. The component analyses are more useful than a simple end-to-end ablation. In particular, holding the object plan fixed in Table 12 shows that VRG reduces unsupported realization from 0.7793 to 0.7165 while supported realization changes from 0.9237 to 0.9116. The admission audit and recovery identities likewise make the sources of the trade-off explicit, including the fact that append-only recovery cannot improve sentence-level CHAIR over its scaffold (Appendices E, J.2, and L.3, pp. 14, 20, and 22).

Critical weaknesses
  1. The evidence supports the usefulness of this particular pipeline more clearly than the claimed architectural explanation. The comparison uses one autoregressive checkpoint and one diffusion checkpoint, so architecture changes together with training and model capabilities. Table 1 usefully acknowledges that strict LLaVA can outperform direct LLaDA-V on both metrics, but the uncertainty observations do not establish that revisability causes the final gains. COVER also combines multiple proposals, visual verification, sentence filtering, and template recovery. There is no corresponding autoregressive propose/verify/realize/recover control with comparable proposals and inference effort, and the fixed-plan VRG experiment only varies guidance within LLaDA-V. This leaves open how much of the benefit actually requires diffusion (Sections 3–4, pp. 4–7; Table 12, p. 20).

  2. The evidence for broadly useful captioning is less complete than the closed-ontology results suggest. The pipeline extracts canonical object sets, deletes sentences with out-of-plan objects, and restores coverage with fixed-template additions; the evaluation therefore needs to establish how well it transfers beyond canonical object inclusion. On OpenCHAIR, COVER is worse than both DoLa and AdaIAT on both reported axes, so there is no demonstrated advantage there (Table 2). Moreover, Section 5.1 specifies only a “released evaluation subset,” without its size or selection, and the AI-use statement says GPT-5.4 evaluates an explicitly labeled OpenCHAIR subset, but I could not locate the subset definition or judging protocol in the appendices. The mapping used for planning/filtering outside COCO and the quality of the resulting full captions also remain unclear (Sections 4.2–4.3 and 5.1, pp. 6–7; AI-use statement, p. 10; Appendices B and I).

Questions for authors
  1. Can you test an autoregressive realization/verification control with the same candidate plan, filtering, and recovery budget, and report its hallucination–coverage trade-off together with latency or model-forward cost? This would help determine whether the central advantage comes from masked diffusion or from the surrounding verification pipeline.

  2. Please specify the OpenCHAIR subset size and selection, object extraction and support-judging protocol, GPT-5.4 prompt/settings and unresolved-case handling, and how the planning vocabulary and sentence filter operate on OpenCHAIR and AMBER. Does a small blinded comparison of full-caption factuality and usefulness support the coverage gains after template recovery?

  3. Were the recovery settings fixed before transfer evaluation? Table 2's Object HalBench result (0.410 sentence-level CHAIR and 0.769 recall) corresponds to the “strict” row of Table 11 rather than its “default” row. Please explain the choice of variant and how thresholds, caps, and the global gain median were selected and transferred across benchmarks.

4 comments
AuthorsOpenReview ↗

Response 1: Same-Plan AR Control and Realizer Contribution

Thank you for the valuable and specific comments. We began the requested experiments as soon as we received the review. We have now (i) run a same-plan autoregressive realizer control, including diffusion with and without VRG; (ii) audited the complete OpenCHAIR and AMBER planning/filter/evaluation protocols; (iii) evaluated full-caption quality before and after recovery; and (iv) verified all Object HalBench recovery configurations and their selection. We are also profiling inference cost and will report the completed results before the rebuttal deadline. We answer each question below.

Response 1: Same-Plan AR Control and Realizer Contribution

We construct H(held-out)-compare500 by deterministically selecting 500 images from a disjoint 2,000-image held-out MSCOCO pool without inspecting captions, labels, or metrics. The LLaDA-V realizer and LLaVA-1.5 same-plan autoregressive realizer (AR-P) receive the identical frozen Object Plan and semantic prompt, followed by the same sentence filter, recovery eligibility, gates, caps, and templates. We report diffusion without VRG to separate realization architecture from diffusion-specific guidance.

Final realizerCHAIRs ↓\downarrowCHAIRi ↓\downarrowRecall ↑\uparrowAvgObj
Diffusion + VRG0.4320.13380.77765.604
Diffusion, no VRG0.4320.13930.78275.644
AR-P0.3940.17680.74702.986

Applying the shared plan/filter/recovery components through AR-P lowers CHAIRs from the detailed LLaVA operating point of 0.592 to 0.394, showing that the architecture-agnostic pipeline is effective. AR-P pays for this reduction with shorter captions and lower Recall. Replacing the AR final realizer with masked diffusion restores approximately three Recall points and reduces CHAIRi from 0.1768 to 0.1338. The no-VRG comparison shows that this contribution is not solely produced by guidance: with the plan, filtering, and recovery fixed, masked diffusion realizes more supported plan content with a lower hallucinated fraction among mentions.

VRG then adds a uniquely non-autoregressive mechanism. In the fixed-plan realization study (Table 13), VRG reduces unsupported realization from 0.7793 to 0.7165 while preserving most supported realization (0.9237 to 0.9116). It can reorder unresolved positions and revise low-confidence predictions before fixation; standard left-to-right AR cannot use this fixation-priority branch. Thus, shared verification/postprocessing first lowers sentence-level hallucination, masked-diffusion realization restores coverage and improves mention-level grounding, and VRG operationalizes selective commitment.

The complete control is reported in the appendix section J. The common-boundary latency and model-forward profile remains in progress and will be updated before the deadline.

AuthorsOpenReview ↗

Response 2: OpenCHAIR/AMBER protocol and full-caption quality

Response 2: OpenCHAIR/AMBER protocol and full-caption quality

Thank you for prompting us to clarify the complete OpenCHAIR and AMBER protocols. We address each requested item in turn below.

Subset size and selection. OpenCHAIR requires a separate GPT-5.4 support judgment for every object extracted from a caption, resulting in thousands of calls per method. Evaluating the complete released 4,863-image test split would therefore incur substantial time and token cost. We use the fixed first 500 records of the released test split for every method, without selecting images according to model outputs or evaluation results. The generation prompt is Describe this image in detail.

Object extraction. OpenCHAIR lowercases each caption, extracts spaCy NOUN lemmas, and retains nouns with Brysbaert concreteness greater than 4.5. The protocol marks painting/drawing/photo/picture/portrait/photograph as ignore.

Support judging and GPT-5.4 settings. For each extracted object, GPT-5.4 receives:

Here are a few descriptions of an image: {reference}.
Does the image contain the following object: {object}?
Answer yes/no/unsure.
The answer is:

We use temperature=0 and max_tokens=8. Technical failures are retried, and the latest valid yes/no/unsure/ignore result is retained for each (run, image, object_index, object). Resolved OCH is No/(Yes+No); unsure and ignore cases are reported separately. The same-pass COVER row in Table 2 contains 3521 Yes, 529 No, 1490 Unsure, and 20 Ignore, yielding OCH=529/(3521+529)=0.1306.

Planning vocabulary and sentence filtering. COVER uses COCO-80 as a compact canonical planning space to control the cost of candidate canonicalization, Visual Gain verification, sentence filtering, and recovery. We deliberately retain this planning space on OpenCHAIR rather than expanding or retuning the Object Plan to its open vocabulary. Besides improving computational efficiency, this design allows OpenCHAIR to test the generality of COVER’s unchanged pipeline under broader-vocabulary evaluation rather than evaluating a benchmark-specific version.

The final captions are evaluated by an independent open-vocabulary evaluator, so non-COCO nouns are still extracted and judged. Sentences are split using (?<=[.!?])\s+. A sentence containing an internal COCO object outside the Object Plan is removed; if all sentences are removed, a deterministic Object Plan template is used. Filtering precedes recovery.

For all 1,004 AMBER images, COVER instead uses the AMBER-specific internal object space. Attributes and relations in the AMBER evaluator are downstream evaluation concepts rather than Object Plan inputs. The saved captions exactly match the official evaluator input and reproduce (CHAIR, Cover, Hal, Cog)=(6.8, 59.3, 33.1, 2.3).

Full-caption quality after template recovery. We further compare complete captions before and after bounded recovery:

StageCLIPScore ↑SPICE ↑CHAIRs ↓Recall ↑
Scaffold0.834800.153710.3780.7400
Final0.834880.154490.4320.7776

Recovery changes 96/500 captions, adds 1.416 words per caption on average, and increases Recall by 3.76 points. COVER preserves the same level of image–text alignment and scene-graph semantic quality after recovery: neither CLIPScore nor SPICE decreases, and no exact duplicate sentence is introduced.

For broader evaluation, we added 1,000 MSCOCO val2014 images disjoint from H-compare500, yielding 1,500 unique images. On this set, COVER obtains a reference-free CLIPScore of 0.8359, the highest among the four compared methods.

It is worth emphasizing that the annotation-based benchmarks already establish COVER’s effectiveness: COVER achieves the highest AMBER Cover, the lowest CHAIRs in the matched-coverage MSCOCO comparison, and the lowest Object HalBench CHAIRs among methods with Recall above 0.76. OPA-DPO attains lower hallucination through preference training but with substantially lower coverage, while DoLa and AdaIAT are currently stronger on OpenCHAIR.

The current OpenCHAIR experiment retains the compact COCO-80 planning space and therefore leaves clear headroom. Expanding this space to OpenCHAIR’s long-tail object categories would allow COVER to verify and recover a broader range of open-vocabulary objects, providing a direct path toward a stronger and potentially leading OpenCHAIR operating point.

AuthorsOpenReview ↗

Response 3: Object HalBench recovery settings

Response 3: Object HalBench recovery settings

Thank you for noting the correspondence between Table 2 and the variants in Table 11. Table 2 reports the strict configuration. We clarify below why this variant was selected and how its thresholds and caps differ from the other two configurations.

We compared three complete recovery configurations on the same 300 Object HalBench images. They use the same models, Object Plan construction, VRG, sentence filter, and recovery template, while varying only the thresholds and caps controlling candidate admission and recovery volume.

VariantPlan threshold / proposer capRecovery capGlobal quantileLow-tier capCHAIRs ↓Recall ↑
Default0.25 / 340.5020.41670.7597
Strict0.00 / 550.7030.41000.7692
High recall0.00 / 540.4020.44670.7775
  • Choice of variant. Table 2 uses strict because it improves both reported axes over default: lower sentence-level CHAIR (0.4100 vs. 0.4167) and higher Recall (0.7692 vs. 0.7597). It therefore performs best on the two metrics shown in Table 2. High recall increases Recall further but also increases hallucination.
  • Threshold selection. The plan threshold controls the Visual Gain required for proposer-only candidates to enter the Object Plan. Default uses 0.25; strict and high recall use 0.00, allowing more proposer-only candidates to enter the Object Plan and become eligible for bounded recovery.
  • Cap selection. The proposer cap bounds proposer-only objects in the Object Plan, the recovery cap bounds all recovered objects, and the low-tier cap separately bounds proposer-only recovery. Default, strict, and high recall respectively use (3,4,2), (5,5,3), and (5,4,2).
  • Global gain threshold. This threshold is computed from the Visual Gain distribution of low-tier candidates across all images and is candidate-weighted. Default uses the median (q=0.50), strict uses the 70th percentile, and high recall uses the 40th percentile.
  • Transfer across benchmarks. The Object HalBench numerical settings were not transferred unchanged from MSCOCO. The strict setting was selected from the three configurations evaluated on the same 300 Object HalBench images; therefore, the reported Object HalBench row uses benchmark-specific settings rather than a recovery policy frozen on MSCOCO. What transfers unchanged is the propose–plan–realize–filter–recover pipeline, the Visual Gain definition, and the role of each threshold and cap. All three configurations are reported in the appendix section J.
AuthorsOpenReview ↗

Cost of the Same-Plan Realizer Control

We appreciate this practical suggestion, because comparable inference effort is essential for interpreting the same-plan control. We profile the complete image-to-final-caption boundary on an RTX A6000 with batch size 1 and resident models, holding the Object Plan, filtering, and recovery policy fixed as in our earlier response.

Final realizerWarm s/image ↓\downarrowCross-validated FLOPs/imageCHAIRS↓\mathrm{CHAIR}_S\downarrowCHAIRI↓\mathrm{CHAIR}_I\downarrowRecall$\uparrow$
Same-plan AR-P26.3340.97--1.06 PFLOPs0.3940.17680.7470
Masked diffusion + VRG27.3230.99--1.08 PFLOPs0.4320.13380.7776

The two realizers therefore operate at comparable total wall-clock and estimated compute, with an end-to-end difference of only 0.99 s/image. AR-P attains lower sentence-level CHAIR by producing substantially shorter captions, while masked diffusion restores 3.06 Recall points and lowers the unsupported fraction among object mentions by 4.30 points. This strengthens the attribution that masked diffusion reaches its higher-coverage, better mention-grounded operating point without a substantially larger inference-time allocation.

To identify the shared cost in this control, we synchronize CUDA around every pipeline stage, record per-image wall time, and aggregate five 50-image warm trials. Proposal Visual Gain takes 7.74 s/image, CaptionAug 5.26 s, direct diffusion 3.89 s, the LLaVA proposal 3.32 s, VRG realization 5.85 s, and recovery Visual Gain 1.13 s. Thus, the comparison includes the same costly discovery and verification path, while the final realizer changes the resulting hallucination--coverage regime.

Because the measured cost is distributed across separable stages, COVER can expose lighter operating modes through the same stage switches used in our ablations, selecting proposal, verification, and recovery modules for the desired latency--quality budget. Enabled stages can then be optimized independently by parallelizing proposal views, batching real/null scoring, and caching repeated visual features. This modular deployment supports offline, asynchronous, or on-demand verified-description applications.

Reviewer_k4bWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of object hallucination in large vision-language models on open-ended image description tasks, focusing on the trade-off between reducing hallucination rates and maintaining object coverage. The authors observe that the autoregressive model LLaVA-1.5 usually describes more objects but has a higher hallucination rate, while the non-autoregressive diffusion model LLaDA-V has a lower hallucination rate but tends to miss real objects. Based on the differences in how the two generation paradigms handle uncertainty, the authors propose the concept of Selective Object Commitment, separating broad object discovery from final object confirmation.

To achieve this, the paper introduces a training-free framework called COVER, which includes four stages. First, three description views are generated using LLaVA-1.5 and LLaDA-V, and candidate objects are extracted. Second, an Object Plan is constructed by combining cross-view consistency and the prediction differences between real images and zero images. Next, Visual Residual Guidance (VRG) is used to adjust token predictions and fix priorities during the diffusion denoising process, filtering out sentences containing unplanned objects.

Finally, missing planned objects are recovered through visual evidence gating and fixed templates. The framework is evaluated on MSCOCO, AMBER, Object HalBench and OpenCHAIR. Experimental results show that COVER can reduce object hallucination rates while maintaining high object recall, and the contributions of different modules are verified through candidate source analysis, VRG ablation and recovery strategy comparisons.

Critical strengths
  1. The research motivation is clear, focusing on the important trade-off between hallucination rate and object coverage. This paper not only looks at reducing object hallucination but also emphasises that being overly conservative in generation might miss real objects. The authors show by comparing CHAIR and Recall under different generation strategies that a lower hallucination rate doesn’t necessarily mean higher quality visual descriptions. This makes the research question more complete than just optimising hallucination metrics and also has practical significance.
  2. The method forms a fairly complete cross-architecture collaborative framework and provides relatively thorough experimental analysis. COVER combines the wide candidate generation capability of autoregressive models with the iterative generation mechanism of diffusion models, controlling the final output through candidate planning, visual residual guidance, and constrained recovery. The overall design has a clear division of modules. In addition, the paper conducts experiments on four benchmarks and provides multiple random subset results, analysis of candidate source reliability, VRG ablation, recovery strategy comparison, and some statistical confidence intervals. These experiments help in understanding the framework’s behaviour and performance trade-offs
Critical weaknesses
  1. The reliability of candidate generation and admission mechanisms is not fully guaranteed.COVER relies on VLMs that are prone to hallucinations for candidate generation, and these erroneous candidates can further affect the Object Plan and the final output. Although the paper attempts to filter hallucinations through cross-view consistency and Visual Gain, the actual admission policy is inconsistent: any object proposed by a LLaDA-V view can directly enter the Object Plan, whereas LLaVA-only candidates need additional verification. Table 6 in Appendix D shows that the precision of Direct-only and CaptionAug-only candidates is only 50.24% and 37.37%, respectively, much lower than multi-view consistent candidates. The paper does not sufficiently explain why these low-reliability candidates can be accepted directly, nor does it analyse how candidate hallucinations propagate in later stages or whether stricter candidate verification could improve the final CHAIR–Recall tradeoff.
  2. There is not enough validation of the candidate generation model and Prompt configuration dependency. COVER's Object Plan fully relies on candidates provided by the three generation views, and real objects not discovered by any view cannot be recovered later. However, the paper only uses a fixed combination of LLaVA-1.5-7B and LLaDA-V, lacking experiments on model scale or Prompt sensitivity. Appendix F shows that different generation setups can significantly affect object Recall, but the paper does not further validate how these changes impact the complete COVER system, so its robustness and generalisation ability remain unclear.
  3. Object-level constraints may harm open-ended generation quality. COVER removes entire sentences containing unplanned objects and restores missing objects using fixed templates. This can also remove correct actions, attributes, and relational information, making the generation process at the object selection level more like constrained candidate filtering and template-based output. Although the paper reports CHAIR and Recall, it lacks direct evaluation of description completeness and fine-grained semantic quality. Moreover, COVER's hallucination rate and correct object mention density on OpenCHAIR are no better than DoLa and AdaIAT, further limiting the persuasiveness of its open-ended generation advantages.
  4. The independent contribution of the non-autoregressive diffusion mechanism has not been fully proven. The paper attributes the generative differences between LLaVA-1.5 and LLaDA-V to the different ways autoregressive and diffusion architectures handle uncertainty, but both models also differ in training and model capabilities, making it difficult to isolate the effect of the architecture itself. Moreover, the paper lacks a control where the diffusion generator is replaced with an autoregressive model under the same Object Plan, Sentence Filter, and Recovery conditions. Therefore, the existing experiments cannot clearly distinguish the contributions of diffusion iterative revision from candidate filtering and rule-based post-processing.
  5. The visual validation reliability of Visual Gain is limited. Visual Gain measures the visual support of objects using the prediction difference between real and zero images, but a positive probability gain does not necessarily mean the object actually exists and may also be affected by visual relevancy or input distribution differences. Appendix E reports an AUROC of only 0.7036, and the filtered LLaVA-only candidates have a precision of 0.4793. The paper lacks analysis of error propagation and different Null Image settings, which limits the persuasiveness of this core validation mechanism. 6. Inference efficiency is insufficiently evaluated. COVER involves multiple model generation processes, candidate-level visual validation, and Real/Null dual-branch forward computation at each diffusion step. Although this complex process could incur significant additional inference costs, the paper does not report complete end-to-end latency, throughput, or memory comparisons. As a result, it is still unclear whether the reduction in hallucination rate and coverage gain is enough to offset the extra computational overhead.
Questions for authors
  1. About asymmetric candidate access strategies. Table 6 in Appendix D shows that the precision of candidates unique to a single LLaDA-V diffusion view is significantly lower than that of multi-view consistent candidates. Why can these candidates still go directly into the Object Plan, while only LLaVA-only candidates require extra Visual Gain verification? If the same or independent visual gating were applied to diffusion-unique candidates, how would the final CHAIR–Recall trade-off change?
  2. About sensitivity to candidate generation conditions. COVER's final coverage is limited by the candidate set, and Appendix F shows that generation settings can significantly alter the coverage of candidate objects. Could the authors further verify whether the quality of the Object Plan and overall COVER performance remain stable under different prompts, generation lengths, or candidate generation models? How much do the current results rely on the specific capabilities and complementarity of the candidate generator?
  3. About the independent contribution of the diffusion generation mechanism. Is it possible, while keeping the same Object Plan, Sentence Filter and Recovery, to replace the Stage 3 diffusion generator with an autoregressive VLM? This comparison would help clarify whether the performance gains come from the iterative revision capability of the diffusion model or mainly from candidate planning and post-processing.
  4. About the quality of open-ended descriptions. COVER removes entire sentences that include unplanned objects and uses fixed templates to recover missing objects. Could the authors provide evaluations for actions, attributes, spatial relations and descriptive completeness to verify that the improvement in object-level hallucination rates does not significantly harm broader open-ended generation quality?
  5. About the reliability of Visual Gain. Visual Gain measures candidate visual support by the prediction difference between real and null images. Could the authors analyse how its misjudgements affect the Object Plan and the final output, and verify whether different Null Image constructions significantly change candidate scoring and overall performance?
7 comments
AuthorsOpenReview ↗

Response 1: Asymmetric Candidate Admission

Thank you for the careful and constructive review. We began the requested experiments as soon as the rebuttal period opened. We have now (i) tested symmetric Visual-Gain gating for diffusion-unique candidates; (ii) propagated prompt and generation-length changes through the complete COVER pipeline; (iii) compared autoregressive and masked-diffusion realizers under the same frozen Object Plan, filtering, and recovery policy; (iv) evaluated complete open-ended captions with CLIPScore, RefCLIPScore, and SPICE; and (v) traced Visual-Gain errors from candidate scoring to the final output and tested multiple null constructions. We are also profiling end-to-end latency, throughput, peak memory, and model-forward counts and will report the completed results before the rebuttal deadline. We answer each question below.

Response 1: Asymmetric Candidate Admission

The submitted source prior was motivated by the complementary operating regimes established in Section 3 and Table 1: detailed LLaVA-1.5 provides broad Recall at higher hallucination, whereas direct LLaDA-V is substantially more conservative, with both lower hallucination and lower Recall. Autoregressive generation fixes every emitted token,

pAR(y∣I)=∏t=1Tp(yt∣y<t,I),p_{\mathrm{AR}}(y\mid I)=\prod_{t=1}^{T}p(y_t\mid y_{<t},I),

whereas masked diffusion iteratively revises unresolved positions,

pMD(y(0)∣I)=∏s=1Spθ(y(s−1)∣y(s),I).p_{\mathrm{MD}}(y^{(0)}\mid I)=\prod_{s=1}^{S}p_\theta(y^{(s-1)}\mid y^{(s)},I).

This motivated treating diffusion proposals as a conservative source prior while requiring additional Visual Gain for LLaVA-only candidates. You identify a finer-grained question: although the diffusion source marginals are reliable, candidates unique to a single diffusion view are less reliable than multi-view candidates.

We therefore keep the same fixed proposal set--the union of canonical COCO-80 objects proposed by LLaVA, direct LLaDA-V, and CaptionAug, as well as the Visual-Gain threshold of 0.25 and every downstream component, and alter only admission. We compare three explicitly defined policies: submitted asymmetric admits candidates proposed only by either diffusion view directly but applies Visual Gain to LLaVA-only candidates; symmetric single-view gating applies Visual Gain to every candidate proposed by only one of the three views, while retaining candidates supported by multiple views; and multi-view agreement only excludes all single-view candidates.

Admission policyPlan precision ↑Plan coverage ↑CHAIRs ↓CHAIRi ↓Recall ↑
Submitted asymmetric0.74120.83860.4320.13380.7776
Symmetric single-view gate0.75110.83510.4100.13000.7718
Multi-view only0.91740.66740.1220.04060.6316

The experiment validates the core strategy used by COVER: multi-view agreement supplies the strongest evidence, but retaining visually supported single-view candidates is essential for high coverage. Removing all single-view candidates loses 14.6 Recall points, confirming that the submitted method was right to preserve this complementary evidence rather than rely on agreement alone. The new symmetric policy further strengthens that strategy: it raises plan precision from 0.7412 to 0.7511 and lowers CHAIRs by 0.022, while Recall changes by only 0.0058 and its paired interval includes zero. Among the tested high-coverage policies, symmetric single-view gating therefore provides the strongest hallucination--coverage balance.

These results are reported in the appendix section O.

AuthorsOpenReview ↗

Response 2: Sensitivity to Candidate-Generation Conditions && Response 3: Independent Contribution of Diffusion Realization

Response 2: Sensitivity to Candidate-Generation Conditions

Thank you for raising this insightful question about how candidate discovery constrains the attainable Recall. To test this dependency directly, we perturb one upstream generation condition at a time, hold the entire downstream pipeline fixed, and measure whether Object Plan quality and final performance change coherently. We compare detailed-128 with detailed-64 to isolate generation length, and detailed-128 with conservative-128 to isolate the proposal prompt. Both LLaDA-V views, object extraction, Visual Gain, Object Plan thresholds, VRG, filtering, recovery, image IDs, and official CHAIR evaluation remain fixed.

Proposer conditionUnion coverage ↑Union precision ↑Plan coverage ↑Plan precision ↑CHAIRs ↓Recall ↑
Detailed, 128 tokens0.87350.64670.83860.74120.4320.7776
Detailed, 64 tokens0.78130.78180.77500.81300.2940.7215
Conservative, 128 tokens0.80570.75610.78830.80110.3020.7279

Although the proposer condition changes the attainable hallucination--coverage point, COVER consistently selects a stronger Object Plan from each proposal set. Object Plan precision rises from 0.6467 to 0.7412 under detailed-128, from 0.7818 to 0.8130 under detailed-64, and from 0.7561 to 0.8011 under conservative-128. At the same time, the plan retains 96.0%, 99.2%, and 97.8% of the respective proposal-set coverage. Thus, across both prompt and length changes, COVER consistently selects the more reliable candidates while preserving nearly all available coverage.

A stage-by-stage count of missed objects, reported in the appendix section O, further shows that the remaining performance change originates mainly from what the three views discover, not from instability in COVER's verification. Ground-truth image--object pairs missed by all three views increase from 181 under detailed-128 to 313 under detailed-64 and 278 under conservative-128, whereas exclusions at the Object Plan gate decrease from 50 to 9 and 25. In other words, a weaker proposal set naturally lowers the achievable Recall, but COVER still reliably selects its higher-quality content. Prompt and length therefore move the complete system along a predictable hallucination--coverage frontier; they do not cause the verification pipeline to fail.

We use this controlled experiment to address prompt and generation-length sensitivity. Replacing the proposer with modern checkpoints of different scales requires additional model-pair evaluations and is left to future work.

Response 3: Independent Contribution of Diffusion Realization

To isolate the contribution of diffusion realization from candidate planning and rule-based postprocessing, we hold the Object Plan, semantic prompt, sentence filter, recovery eligibility, gates, caps, and templates fixed and change only the final realizer. We run this control on H(held-out)-compare500, a deterministic SHA256-selected 500-image subset of a disjoint 2,000-image held-out MSCOCO pool, selected without inspecting captions, labels, or metrics. We additionally report diffusion without VRG so that the realization architecture is separated from diffusion-specific guidance.

Final realizerCHAIRs ↓CHAIRi ↓Recall ↑AvgObj
Diffusion + VRG0.4320.13380.77765.604
Diffusion, no VRG0.4320.13930.78275.644
AR-P0.3940.17680.74702.986

The AR-P result first confirms that the shared COVER components are effective: relative to detailed LLaVA (CHAIRs 0.592), the same planning/filtering/recovery pipeline lowers CHAIRs to 0.394. Its lower sentence-level CHAIR is accompanied by much shorter output and lower Recall. Replacing only the final realizer with masked diffusion, even without VRG, restores 3.57 Recall points and reduces CHAIRi by 3.76 points. Diffusion realizes substantially more of the same plan while producing a lower unsupported fraction among object mentions.

This matched control validates the specific architectural contribution proposed by COVER: our selective-commitment mechanism is not explained by candidate planning and postprocessing alone. Masked-diffusion realization adds high-coverage, mention-level grounding under the same plan. Because standard left-to-right AR has only one next-token position and cannot revise emitted tokens, we design VRG specifically to turn diffusion's unique revisability into selective commitment by reordering unresolved positions and revising low-confidence predictions before fixation.

The complete control is reported in the appendix section J.

AuthorsOpenReview ↗

Response 4: Quality of Open-Ended Descriptions

Response 4: Quality of Open-Ended Descriptions

To test whether object-level filtering and template recovery improve object-level CHAIR and Recall at the expense of broader description quality, we evaluate the complete open-ended captions with three complementary metrics. CLIPScore measures reference-free image--caption alignment; RefCLIPScore additionally incorporates similarity to COCO references; and SPICE compares generated and reference scene graphs, including objects, attributes, and relations.

On the same 500 held-out MSCOCO images, bounded recovery changes 96 captions and adds 1.416 words per caption on average:

StageCLIPScore ↑RefCLIPScore ↑SPICE ↑CHAIRs ↓Recall ↑
Scaffold0.834800.751210.153710.3780.7400
Final0.834880.751000.154490.4320.7776

Recovery raises Recall by 3.76 points while CLIPScore, RefCLIPScore, and SPICE remain at the same level, and it introduces no exact duplicate sentence. We also audit the filtering stage: it removes 80 sentences and 183 object mentions, including 70 unsupported mentions, after which bounded recovery restores verified omissions. The complete result is stronger than an object-list gain: COVER substantially increases Recall while preserving both image--text alignment and scene-graph semantic quality, including the attributes and relations measured by SPICE.

For broader evaluation, we add 1,000 MSCOCO val2014 images disjoint from H-compare500, yielding 1,500 unique images. COVER obtains the highest reference-free CLIPScore among COVER, LLaVA, direct LLaDA-V, and AdaIAT (0.8359). Its SPICE is 0.1571, above direct LLaDA-V (0.1136), while LLaVA and AdaIAT reach 0.1766 and 0.1900. These results demonstrate that COVER improves the object hallucination--coverage trade-off without sacrificing the measured quality of the complete open-ended descriptions.

The broader benchmark results further clarify the scope of this quality advantage. DoLa and AdaIAT are stronger on the two reported OpenCHAIR axes, but COVER is evaluated there as a direct transfer: it retains the compact COCO-80 planning space and uses an independent open-vocabulary evaluator, without OpenCHAIR-specific vocabulary expansion or tuning. Its ability to transfer unchanged to this broader-vocabulary evaluation supports the generality of the pipeline. If COVER's planning space is expanded to OpenCHAIR's long-tail categories, our method has substantial potential to verify and recover more relevant objects and improve this operating point further. COVER already achieves the highest AMBER Cover, the lowest CHAIRs in the matched-coverage MSCOCO comparison, and the lowest Object HalBench CHAIRs among methods with Recall above 0.76.

The full-caption audit is reported in the appendix section N.

Reviewer_7kxWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how to reduce object hallucination without losing the objects that make a caption informative. It compares LLaVA-1.5 and LLaDA-V, introduces a caption-augmentation step for LLaDA-V, and proposes COVER. COVER collects objects from three caption views, constructs an object plan using source agreement and real/null-image scoring, generates a diffusion caption with visual residual guidance (VRG), filters sentences containing out-of-plan objects, and adds selected omissions through templates. The accompanying analysis decomposes object inclusion into admission, realization, and recovery.

Across six MSCOCO pools, COVER reduces CHAIRs from 0.5973 to 0.4293 with Recall of 0.7863 versus 0.7970 for LLaVA-1.5. AMBER and Object HalBench also show useful coverage–hallucination trade-offs. The OpenCHAIR result is less competitive, as the authors acknowledge.

Critical strengths
  1. The paper takes coverage seriously. Tables 1 and 8 include conservative prompts and length controls, and Section 3.2 explicitly notes that strict LLaVA outperforms direct LLaDA-V on both hallucination and Recall. This makes the motivation more convincing than simply comparing a detailed model with a terse one. COVER then preserves most of the proposer's Recall on MSCOCO, while on AMBER it increases Cover from 52.4 to 59.3 and reduces CHAIR from 10.3 to 6.8 (Table 10).

  2. Several component checks are well chosen. Table 12 fixes the object plan and separates logit guidance from fixation guidance. Combined VRG reduces unsupported realization from 0.7793 to 0.7165, with a smaller reduction for supported objects, from 0.9237 to 0.9116. Appendix E compares visual-gain selection against equal-count random selection within each image and reports image-level bootstrap intervals. Appendix L.3 also correctly explains that append-only recovery cannot improve CHAIRs relative to the same scaffold. These checks help distinguish coverage restoration from hallucination suppression.

Critical weaknesses
  1. The headline gains are relative to COVER's own proposer, and several baselines are stronger or non-dominated. OPA-DPO achieves far lower hallucination everywhere (MSCOCO CHAIRs 0.127 vs. 0.429); AdaIAT is not dominated on MSCOCO (0.395/0.119/0.751 vs. 0.429/0.129/0.786); on OpenCHAIR, DoLa and AdaIAT beat COVER on both OCH and supported mentions (Table 2). The claimed advantage holds only inside the self-selected "Recall > 0.78" regime, and no matched-coverage comparison (e.g., running baselines in detailed mode to reach 0.786 Recall) is provided to show COVER's frontier is actually better.

  2. No inference-cost accounting, and no equal-compute comparison. Each image requires three caption generations, per-candidate masked gain scoring, and paired real/null forwards at every one of 128 VRG denoising steps. No latency, FLOPs, or throughput is reported. A natural and missing baseline is to spend the same compute budget on a single stronger VLM (or on best-of-n/self-consistency with the proposer) and compare the resulting trade-off points.

  3. The template-based recovery buys Recall at the direct expense of description quality, and this cost is never measured. Roughly +3.5 Recall points over the scaffold (0.7508 -> 0.7863, Table 9) come from mechanically appending fixed sentences ("The image also shows a suitcase and a car.") to an otherwise finished caption. This is Recall for Recall's sake: the appended objects are verified as present, but their attributes, spatial relations, and interactions — the content that makes a description informative — are discarded, and the caption becomes a stitched object list whose fluency and coherence visibly degrade (the paper's own Figure 2 example has this form). The same append mechanism is available to any baseline, so part of the reported coverage advantage reflects postprocessing rather than better generation. Crucially, the paper never checks the other side of the ledger: no CIDEr/SPICE/CLIPScore, no human preference or fluency evaluation accompanies the Recall gains, so the claim that COVER produces useful high-coverage captions — as opposed to higher-scoring object inventories — is asserted rather than demonstrated.

  4. Validation is confined to captioning. Hallucination is equally prevalent in VQA, multi-turn dialogue, and reasoning settings, none of which is evaluated; the plan-then-realize pipeline with template recovery is also specific to description-style outputs, so it is unclear the approach transfers.

  5. A single, dated model pair. Only LLaVA-1.5 + LLaDA-V (both 7B-era) are tested. Whether the AR/diffusion complementarity survives with stronger modern VLMs — and whether a diffusion verifier remains necessary when the proposer itself is stronger — is unknown.

  6. The architectural explanation is not isolated from verification and postprocessing. CaptionAug changes conditioning and the number of passes rather than isolating revisability, and the fixation diagnostics are correlational. The missing control is an autoregressive realizer (or a direct template rendering) on the same verified plan with the same filtering and recovery: source agreement, plan gating, and template recovery may be doing most of the work regardless of architecture. The probability identities organize the trade-off but do not resolve this attribution.

Questions for authors
  1. Can you provide matched-coverage comparisons — e.g., AdaIAT or the proposer run in a detailed mode to reach Recall ≈ 0.786 — to show COVER's hallucination–coverage frontier is actually better, rather than only better within the self-selected "Recall > 0.78" regime?

  2. How much text quality does template recovery cost? Please report CIDEr/SPICE/CLIPScore or a human fluency/preference evaluation comparing the scaffold before and after template appending, so the Recall gain is weighed against description quality.

  3. What are the end-to-end latency, forward-pass counts, and FLOPs? Given the same compute budget, how does COVER compare against a single stronger VLM, or best-of-n / self-consistency with the proposer?

  4. Can you compare an autoregressive realizer (or direct template rendering) on the same verified plan, with the same filtering and recovery? This would isolate what the diffusion architecture contributes beyond source agreement and postprocessing.

  5. Does the pipeline transfer beyond captioning — e.g., to VQA or multi-turn dialogue, where hallucination is also prevalent? And does the AR/diffusion complementarity still hold with stronger modern VLMs?

4 comments
AuthorsOpenReview ↗

Response 1: Matched-Coverage Hallucination-Coverage Frontier

Thank you for the valuable and detailed comments. We began the requested experiments as soon as we received the review. We have now (i) measured complete hallucination--coverage curves rather than comparing one operating point per method, (ii) evaluated caption quality before and after recovery with CLIPScore, RefCLIPScore, and SPICE, and (iii) compared autoregressive and masked-diffusion realizers under the same frozen Object Plan, filtering, and recovery policy. We are also running end-to-end cost profiling and will report the completed results before the rebuttal deadline. We answer each question below.

Response 1: Matched-Coverage Hallucination-Coverage Frontier

Thank you for suggesting a matched-coverage comparison, which we found helpful for characterizing the complete hallucination-coverage frontier. The submitted comparison evaluated only one operating point per method and therefore did not fully characterize their hallucination-coverage frontiers. We subsequently evaluated every declared setting on H(held-out)-compare500, a deterministic SHA256-selected 500-image subset of a disjoint 2,000-image held-out MSCOCO pool, selected without inspecting captions, labels, or metrics.

For LLaVA-1.5, we fix the detailed prompt and greedy decoding and vary only the 64/96/128-token budget. For AdaIAT, we fix β=0.5\beta=0.5 and the 128-token budget and vary α\alpha over 0/2/4/6/8/10; increasing α\alpha strengthens suppression and jointly lowers hallucination and Recall. For COVER, we retain every tested recovery policy, from no recovery through increasingly permissive bounded recovery. We define the matching interval before comparison as COVER’s submitted Recall ±0.01\pm0.01: [0.7676, 0.7876].

Matched pointCHAIRs ↓CHAIRi ↓Recall ↑
COVER submitted0.4320.13380.7776
LLaVA-1280.5920.17050.7871
AdaIAT α=2\alpha=20.5240.14590.7699

At comparable Recall, LLaVA-128 has 0.160 higher CHAIRs and 0.0367 higher CHAIRi than COVER. AdaIAT α=2\alpha=2 has 0.092 higher CHAIRs, while its CHAIRi is similar to COVER. The same ordering is observed on two additional disjoint 500-image MSCOCO sets evaluated with the identical protocol.

We have added the complete two-panel Recall-CHAIRs/CHAIRi curve to the Discussion and replaced the original Recall > 0.78 textual claim with the matched-coverage analysis. Conservative LLaVA and AdaIAT settings can attain lower hallucination by sacrificing coverage; COVER is the best observed local matched-coverage point among the tested settings.

This result also supports the architectural motivation of COVER. Autoregressive generation supplies broad object discovery but commits each output irreversibly. Masked diffusion keeps uncertain positions revisable before fixation. Combining these roles allows COVER to preserve the broad proposer's coverage while moving the matched-coverage frontier toward lower hallucination, validating selective commitment as the central design contribution.

The broader benchmark results further show where COVER is strongest. OPA-DPO attains substantially lower hallucination through preference training, but at much lower MSCOCO Recall (0.575). COVER obtains the highest AMBER Cover (59.3), the lowest CHAIRs in this matched-coverage MSCOCO comparison, and the lowest Object HalBench CHAIRs among methods with Recall above 0.76. Although DoLa and AdaIAT are stronger on the reported OpenCHAIR axes, we further clarify that OpenCHAIR is also a transfer test: COVER retains its unchanged compact COCO-80 planning space and is evaluated by an independent open-vocabulary evaluator, rather than being optimized for OpenCHAIR. The result shows that the pipeline transfers directly to broader-vocabulary evaluation, while expanding the planning vocabulary to long-tail categories provides clear headroom for further improvement.

AuthorsOpenReview ↗

Response 2: Text Quality Cost of Template Recovery && Response 3: Inference Cost and Equal-Compute Comparison

Response 2: Text Quality Cost of Template Recovery

We first compare the complete captions before and after bounded template recovery on H(held-out)-compare500, the same deterministic 500-image held-out MSCOCO subset used for the curve and same-plan control.

We use three complementary semantic metrics. CLIPScore is reference-free and measures image-caption alignment in CLIP space. RefCLIPScore is reference-aware and additionally incorporates semantic similarity to the COCO reference captions. SPICE compares scene graphs extracted from generated and reference captions, covering objects, attributes, and relations.

StageCLIPScore ↑RefCLIPScore ↑SPICE ↑CHAIRs ↓Recall ↑
Scaffold0.834800.751210.153710.3780.7400
Final0.834880.751000.154490.4320.7776

Recovery changes 96/500 captions, adds 1.416 words per caption on average, and raises Recall by 3.76 points. CLIPScore, RefCLIPScore, and SPICE remain effectively unchanged, and no exact duplicate sentence is introduced. Thus, recovery increases coverage and hallucination exposure while preserving the measured whole-caption semantic quality.

To evaluate complete-caption quality more broadly, we add 1,000 MSCOCO val2014 images disjoint from H-compare500, producing a total of 1,500 unique images.

MethodCLIPScore ↑SPICE ↑
COVER0.835910.15709
LLaVA-1.50.800090.17661
Direct LLaDA-V0.762590.11358
AdaIAT0.796970.19004

COVER achieves the highest reference-free image-caption alignment and maintains strong scene-graph semantic content. Thus, recovery is a controlled coverage-risk intervention rather than Recall for Recall’s sake: it restores verified objects without reducing the measured semantic quality of the complete caption. The paired recovery table is included in the appendix section N.

Response 3: Inference Cost and Equal-Compute Comparison

Thank you for raising this practical issue. We are running repeated profiling under a common image-to-final-caption boundary, recording end-to-end latency, total model-forward counts, and FLOPs under a common image-to-final-caption boundary. We will report the completed results before the rebuttal deadline.

Our central objective is to identify and exploit the contribution of non-autoregressive decoding, rather than to claim that the current implementation is the cheapest use of a fixed compute budget. Stronger autoregressive VLMs still exhibit hallucination and retain irreversible left-to-right commitment; this is precisely the architectural limitation that motivates COVER. COVER treats the AR model as a broad proposer and applies evidence-gated, revisable commitment downstream. The proposer interface is replaceable, so a stronger AR VLM can be incorporated without changing plan construction, diffusion verification, filtering, or recovery.

The equal-compute stronger-VLM/best-of-n suggestion is a useful systems comparison but changes the checkpoint, training data, model capability, and output-selection rule simultaneously. The same-plan AR-P experiment in Response 4 is therefore our primary attribution control: it holds the Object Plan, filtering, and recovery fixed and changes only the final realizer. We use that experiment to address the mechanism question and the ongoing cost profile to address the systems question.

AuthorsOpenReview ↗

Response 4: Same-Plan Autoregressive Realizer && Response 5: Beyond Captioning and Stronger VLMs

Response 4: Same-Plan Autoregressive Realizer

We implemented the requested control on H(held-out)-compare500 introduced in Response 1, i.e., the same deterministic 500-image subset of the independent 2,000-image held-out MSCOCO set.

The LLaDA-V masked-diffusion realizer and a LLaVA-1.5 same-plan autoregressive realizer (AR-P) receive the identical frozen Object Plan and semantic prompt, followed by the same sentence filter, recovery eligibility rules, gates, caps, and templates. To separate the realization architecture from Visual Residual-Guided decoding (VRG), we report diffusion both with and without VRG.

Final realizerCHAIRs ↓CHAIRi ↓Recall ↑AvgObj
Diffusion + VRG0.4320.13380.77765.604
Diffusion, no VRG0.4320.13930.78275.644
AR-P0.3940.17680.74702.986

Relative to the original detailed LLaVA operating point (CHAIRs 0.592, Recall 0.7871), applying the shared plan/filter/recovery pipeline through AR-P lowers CHAIRs to 0.394, demonstrating that the architecture-agnostic components are effective. AR-P pays for this reduction with shorter captions and lower Recall. Replacing the AR final realizer with masked diffusion restores approximately three Recall points and reduces CHAIRi from 0.1768 to 0.1338, showing the additional value of non-autoregressive realization.

The no-VRG diffusion comparison further shows that this gain is not solely produced by VRG: with source agreement, Object Plan, filtering, and recovery fixed, masked diffusion realizes more supported plan content with a lower hallucinated fraction among mentions. VRG then adds a uniquely non-autoregressive mechanism. In the fixed-plan realization study (Table 13), VRG reduces unsupported realization from 0.7793 to 0.7165 while preserving most supported realization (0.9237 to 0.9116). It can reorder unresolved positions and revise low-confidence predictions before fixation; standard left-to-right AR cannot use this fixation-priority branch. Together, the shared components first suppress sentence-level hallucination, masked-diffusion realization restores coverage and improves mention-level grounding, and VRG operationalizes selective commitment.

Response 5: Beyond Captioning and Stronger VLMs

The expected architectural advantage depends on response length. For an autoregressive response of length TT,

pAR(y∣I,q)=∏t=1Tp(yt∣y<t,I,q),p_{\mathrm{AR}}(y\mid I,q)=\prod_{t=1}^{T}p(y_t\mid y_{<t},I,q),

each emitted token becomes irreversible context. A masked-diffusion response instead follows iterative denoising,

$p_{\mathrm{MD}}(y^{(0)}\mid I,q) =\prod_{s=1}^{S}p_\theta(y^{(s-1)}\mid y^{(s)},I,q),$

where multiple unresolved positions remain jointly revisable before fixation. Short-answer VQA usually contains only one to a few tokens and should not exhibit the same long-horizon commitment advantage as detailed captioning.

For multi-turn dialogue, diffusion can revise the current response before it is emitted, but neither AR nor diffusion can revise earlier turns already shown to the user. This work focuses on image captioning. VQA, multi-turn dialogue, and reasoning are valuable extensions requiring dedicated task protocols, which we leave to future work.

COVER exposes replaceable proposer and verifier interfaces. A stronger autoregressive VLM can replace the LLaVA proposer without changing plan construction, verification, realization, or recovery; a stronger masked-diffusion VLM can likewise replace LLaDA-V.

This distinction is also supported by the same-plan control in Response 4. When we replace the masked-diffusion final realizer with AR-P while retaining the Object Plan, filtering, and recovery, AR-P produces shorter captions with lower Recall and higher CHAIRi. Thus, a stronger AR model can replace the broad proposal module, while the final commitment stage still benefits from the revisable decoding process of masked diffusion. VRG is designed around this non-autoregressive property: it uses multiple unresolved positions and pre-fixation revision to implement selective commitment.

We emphasize that our contribution does not depend on treating LLaVA-1.5 as the strongest possible proposer, but on the division of labor between broad autoregressive discovery and revisable evidence-gated commitment. Evaluating stronger modern model pairs is beyond the scope of this work and is left to future work.

AuthorsOpenReview ↗

Cost Accounting and Equal-Latency Control

Thank you for raising this practical systems question. We have now completed common-boundary profiling on an RTX A6000 with batch size 1. The warm boundary starts with the image path and prompt available and ends when the final caption is materialized in host memory; image loading and preprocessing are included, evaluation is excluded, and all required models remain resident.

MethodWarm s/image ↓\downarrowCross-validated FLOPs/imageCHAIRS↓\mathrm{CHAIR}_S\downarrowCHAIRI↓\mathrm{CHAIR}_I\downarrowRecall$\uparrow$
LLaVA-1.5, single3.30510.5--10.8 TFLOPs0.5920.17050.7871
LLaVA self-consistency, N=8N=826.611--0.5100.14250.8069
COVER27.3230.99--1.08 PFLOPs0.4320.13380.7776

The FLOP ranges are cross-validated module-level estimates from PyTorch FlopCounterMode and DeepSpeed FlopsProfiler; the two backends differ by at most 8.31%, so we report their range rather than treating either as exact hardware FLOPs. COVER uses 48,579 MiB peak GPU memory and executes 5 generation calls, 336 language-model forwards, 18.28 vision forwards, 13.28 null-image forwards, and 512 denoising steps per image.

For the requested repeated-proposer control, we use timing only to identify the largest self-consistency configuration within COVER's warm-latency budget. This selects eight sequential LLaVA samples with label-free object-set medoid selection, requiring 26.611 s/image. Repeated decoding substantially improves the proposer, while COVER retains a clear hallucination advantage at essentially the same measured wall-clock budget: CHAIRS\mathrm{CHAIR}_S is lower by 0.078 and CHAIRI\mathrm{CHAIR}_I by 0.0086, while COVER retains 0.778 Recall. Spending the same latency budget on proposer self-consistency alone therefore does not reproduce COVER's operating point. This demonstrates COVER's central advantage: broad autoregressive discovery is converted by evidence-gated, revisable selective commitment into a lower-hallucination operating point.

To identify where the full-system time is spent, we synchronize CUDA before and after every pipeline stage, record per-image wall time, and aggregate five 50-image warm trials. Proposal Visual Gain takes 7.74 s/image, VRG realization 5.85 s, CaptionAug 5.26 s, direct diffusion 3.89 s, the LLaVA proposal 3.32 s, and recovery Visual Gain 1.13 s. This shows that cost is distributed across broad discovery, visual verification, and selective realization rather than concentrated in template recovery.

Because these costs arise from distinct modules, COVER can use the same stage switches evaluated in our ablations to expose lighter deployment modes: proposal views, verification, and recovery can be selectively enabled according to the desired latency--quality budget. Each enabled stage can also be optimized independently: proposal views can run in parallel, real/null candidate scoring can be batched, and repeated visual features can be cached. This modular deployment supports verification-intensive offline or asynchronous applications such as dataset curation, image indexing, cached descriptions generated at ingestion, and on-demand detailed captions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.