acceptodds
Under review as a conference paper at ICLR 2027

EAG-Cap: Evidence-Aware Grounded Caption Revision with Visual Evidence Screening and Caption-Driven Grounding Recovery

Abstract

Grounded vision-language models connect generated descriptions to image regions, but dense grounded proposals mix useful localized content with unsupported or overly specific claims. Conservative visual evidence screening can clean these proposals, yet it introduces a precision–coverage trade-off: valid caption-relevant regions may be pruned together with noisy ones. We present EAG-Cap, a training-free framework that treats proposal phrases as candidate evidence-bearing claims rather than trusted facts. EAG-Cap screens their localized visual support, composes a concise caption under structured evidence constraints, and then uses that final caption to identify groundable concepts that still lack compatible spatial support. Caption-driven grounding recovery selectively re-queries only these missing concepts and merges recovered regions with retained grounding. On the full 5,000-image COCO val2017 split, EAG-Cap obtains 0.839 CIDEr and 18.01% CHAIRs, reducing hallucination relative to Florence-2 Detailed (29.62% CHAIRs) while retaining more annotated content than similarly concise baselines. More importantly, recovery nearly restores the caption-conditioned coverage of raw proposals (90.97% vs. 90.96%) while retaining 84.54% [email protected], compared with 61.45% for raw proposals, and reducing boxes per image from 19.75 to 4.91. A controlled text-only rewrite explains part of the language behavior but provides no caption-aligned grounding output. These results characterize EAG-Cap as a coupled evidence-revision and grounding-recovery framework rather than a caption filter alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Grounded Caption Revision, Vision-Language Models, Visual Grounding, Hallucination Mitigation, Evidence Verification, Grounding Recovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_phyPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents EAG-Cap, a training-free, post-hoc framework for grounded image caption revision. To address the issue that dense region proposals from grounded vision–language models often include unsupported claims while conservative filtering can remove valid spatial support for concepts in the final caption, EAG-Cap processes outputs through four sequential stages using frozen off-the-shelf models: (1) Proposal and Claim Preparation, which collects global captions, detailed captions, object detections, and dense region descriptions from Florence-2 and applies rule-based normalization and risk tagging to form candidate visual claims; (2) Visual Evidence Screening, which queries GroundingDINO on the image for each candidate claim and categorizes the resulting detections into discrete evidence states (strong, weak, coarse, or rejected) based on confidence and bounding-box area; (3) Evidence-Aware Caption Composition, which uses Qwen-3B-Instruct to revise the Florence-2 global caption (with the detailed caption as optional context) under prompt constraints defined by the screened evidence states; and (4) Caption-Driven Grounding Recovery, which filters retained evidence boxes to match the generated caption, extracts remaining caption concepts that lack compatible spatial support, and re-queries GroundingDINO to recover and merge their bounding boxes. The framework is evaluated on the 5,000-image COCO val2017 split and a 1,000-image Flickr30k-Entities subset using standard captioning metrics, object hallucination rates (CHAIR), bounding-box [email protected], and caption-conditioned grounding coverage (CC-Cov.).

Critical strengths
  1. Clear formulation of caption–grounding consistency: The paper identifies a practical trade-off in post-hoc grounded caption revision—namely, that conservative filtering of noisy region proposals improves box precision at the expense of pruning valid spatial support for concepts mentioned in the final caption—and jointly evaluates language faithfulness and caption-conditioned grounding coverage (CC-Cov. and CC-IoU).
  2. Training-free modular pipeline with transparent efficiency profiling: The framework operates entirely with frozen off- the-shelf models (Florence-2, GroundingDINO, and Qwen-3B-Instruct) without task-specific training, and the authors transparently report stage-wise wall-clock latency and GPU memory overhead in Section 4.1 and Appendix A.2 (Table 6).
Critical weaknesses
  1. Insufficient implementation details for rule-based claim preparation and concept extraction (Sections 3.2 & 3.5, Appendices B & E): Converting proposal phrases into calibrated claims ci=ϕ(pi)c_i = \phi(p_i) (safe vs. specific queries) and risk tags ri=ρ(pi)r_i = \rho(p_i) relies on matching "recognized compound heads," removing "nonvisual terms," and flagging "dietary claims" and "fine-grained food terms" (Section 3.2, Appendix B), while concept extraction in Section 3.5 relies on a fixed list of generic lexical equivalences (Appendix E). However, the manuscript does not provide the specific lexicons or transformation rules used in Appendix B, nor does it specify the exact condition under which the safe versus specific query is passed to GroundingDINO. Providing the complete rules, word lists, and concrete examples of ϕ(pi)\phi(p_i) and ρ(pi)\rho(p_i) would significantly improve reproducibility.
  2. Suppression rather than visual verification of non-entity semantics (Sections 3.3–3.5): The actual pipeline lacks a mechanism to visually verify these non-entity semantics. As noted in Section 3.3, GroundingDINO only checks localized entity presence, and the caption composer in Section 3.4 is a text-only LLM (Qwen-3B-Instruct) without visual access to the image. Consequently, non-entity details are handled primarily by suppressing them via prompt rules (Appendix D) and discarding relation/action words as delimiters during concept extraction (Appendix E). This limits the method's capability to coarse entity-level filtering and reduces descriptive richness, rather than genuinely resolving fine-grained attribute or relational hallucinations.
  3. Large drop in caption-conditioned coverage before Stage 4 recovery and missing direct-grounding baseline (Sections 3.3–3.5, Tables 3–4): Although Stage 3 composes the final caption yy under the constraints of Stage 2's Evidence State Store, prior to Stage 4 recovery (w/o Grounding Recovery), CC-Cov. drops sharply to 76.07% on COCO (2.313 vs. 4.907 boxes/image in Table 3) and 44.07% on Flickr30k (vs. 78.66% after recovery in Table 4). It is unclear why so many concepts in the final caption lack spatial support from Stage 2 if caption composition is constrained by the screened evidence, and why the same GroundingDINO model with identical thresholds (0.30 / 0.25) can subsequently recover them in Stage 4.
  4. Uncontrolled prompt differences in the text-only baseline (Section 4.6, Table 5, Appendix D): In Table 5 (COCO-2K), the text-only baseline Qwen Rewrite (Global + Detailed) already achieves most of the hallucination reduction (19.26% vs. 17.79% CHAIRs) and SPICE (0.222 vs. 0.230) without any visual screening. While EAG-Cap shows higher BLEU-4 (0.264 vs. 0.191) and CIDEr (0.868 vs. 0.612), Appendix D reveals that the EAG-Cap prompt explicitly enforces "one natural COCO-style sentence of 8–18 words" and "minimal edits" to the global caption, whereas these constraints are omitted from the Qwen Rewrite prompt. This confounding difference in prompt constraints makes it difficult to attribute the BLEU-4 and CIDEr gains in Table 5 to visual evidence screening rather than to explicit length and style instructions.
Questions for authors
  1. Could you provide the specific lexicons/rules used for matching "recognized compound heads," removing "nonvisual terms," and assigning hard/soft-risk tags, and clarify the exact rule determining when a safe query versus a specific query is passed to GroundingDINO?
  2. why does w/o Grounding Recovery have relatively low CC-Cov. (76.07% in Table 3 and 44.07% in Table 4) after Stage 3 composition—do the ungrounded concepts in yy primarily come from phrases retained from the Florence-2 Global anchor?
  3. Can you provide the two different prompts listed in Appendix D?
4 comments

We thank the reviewer for the detailed methodological concerns.

W1. Claim preparation/query reproducibility

The implementation uses finite English rules, not learned claim calibration: four hard-risk regex groups (uncertainty, subjective, dietary, identity/title), two soft-risk groups (fine-grained food, name-like phrases), an 18-entry compound-normalization dictionary, stop/query-exclusion sets and simple singularization. Both distinct nonempty safe and specific queries may be sent to GroundingDINO; the composer makes the post-detection selection.

W2. Non-entity semantic verification

We agree. GroundingDINO provides object-region compatibility, not verification of actions, relations, attributes, counts or identity. The text-only composer mainly omits/generalizes risky non-entity claims and may reduce informativeness. We will not characterize this as comprehensive semantic hallucination correction.

W3. Recovery, low pre-recovery coverage and the same detector

We evaluated a strict pre-recovery condition with all grounding-field fallbacks disabled, on COCO-5K:

MetricStrict beforeFull
[email protected]88.45%84.54%
CC-Cov.74.99%90.97%
CC-IoU0.69790.8438
Boxes/image2.184.91

The +15.98 pp coverage gain (paired-bootstrap 95% CI [+15.13,+16.90] pp) comes with a 3.91 pp precision loss. Among 6,490 caption-conditioned GT-category units, the full method matches 1,037 more units overall; 586 remain unmatched. The earlier 76.07% pre-recovery figure used a less strict evaluation with grounding fallback; we will distinguish the two.

Re-querying is not a repeat of the same exact query set: Stage 4 extracts short phrases from the final caption, optionally incorporating used evidence and caption-compatible phrases, then queries only concepts missing compatible retained boxes. Query wording/candidate pools can therefore differ even with the same detector thresholds. This mechanism explains how new detections can arise; it does not prove why every Stage 2 concept was absent.

W4. Uncontrolled prompt differences

We agree: the historical Qwen Rewrite lacks EAG-Cap's explicit 8–18-word, minimal-edit, one-added-item and strict JSON-output instructions. The original Table 5 therefore cannot isolate evidence access from prompt design. We have audited the source templates and prepared a prospective COCO-2K prompt-matched text-only control. Its CPU input audit passed on all 2,000 images; GPU inference is underway. No final results are available yet. Historical EAG request-level prompt provenance remains unverified.

Q1. Exact lexical and safe/specific logic

The source matches uncertainty, subjective, dietary_claim, and identity_or_title hard-risk patterns; soft tags include fine_grained_food and named_entity_like. Safe queries use longest recognized compounds or stripped visual heads (e.g., "vegan black bean burger" → "burger"; "Dr. ... headshot" → "person"). Specific queries are suppressed under hard risk or name-like text, with explicit tested exceptions such as "side salad" and "tomato sauce".

The exact selection is: specific iff allowed_to_upgrade and specific_query exists and its detection is detected_strong (score ≥ 0.50, normalized box area ≤ 0.65); otherwise safe iff it has any detected_strong/weak/coarse result; otherwise neither is verified. The detector's box/text thresholds default to 0.30/0.25. Literal regexes and lexical lists are given in our accompanying source-detail comment.

Q2. Are missing concepts mainly copied from the global anchor?

The global caption is explicitly the composer's primary anchor, so it can retain concepts that are not covered by the screened proposal box pool. However, lexical overlap with the anchor is not causal proof that a missing box was copied from it. The logs do not justify a claim that most missing boxes are due to over-pruning. What the paired experiment establishes is improved caption-conditioned localization after re-querying.

Q3. Both Appendix D prompts

The EAG-Cap Anchor uses a system prompt requiring the Global caption anchor, optional Detailed context, evidence constraints, at most one verified addition, 8–18 words, minimal edits and strict JSON; its user message is a per-image JSON evidence table. The historical Qwen Global+Detailed Rewrite uses a single user prompt to concisely rewrite the two supplied captions, avoiding unsupported details, and returns plain text. The complete source-literal prompts are given in our accompanying comment; historical request logs remain unverified.

We will report the rules, exact prompt templates, clear metric definitions and remaining limitations; unfinished GPU comparisons will not be presented as completed results.

Source-literal templates and key lexical resources (current source, not verified historical request logs).

Prompt A: EAG-Cap balanced/anchor system message

You are the writing module of an evidence-aware grounded captioning system.

Your job is to revise a concise global caption into one COCO-style caption under evidence constraints.

Rules:
1. Use the Florence-2 Global caption as the primary sentence anchor if it is not unsafe.
2. Preserve the global caption wording and sentence structure as much as possible.
3. Use the Florence-2 Detailed caption only as optional context, not as the main source.
4. Use verified evidence to correct or lightly enrich the caption.
5. Add at most one additional verified object, attribute, or action beyond the global caption.
6. Do not convert the caption into an object list.
7. Prefer one concise natural sentence of 8 to 18 words.
8. Do not mention rejected evidence or anything in do_not_mention.
9. Do not mention weak or coarse evidence unless it is low-risk and directly supported by the image.
10. Do not mention exact text, logos, identities, brands, counts, or fine-grained categories unless verified.
11. If the global caption is already safe and visually plausible, keep it with minimal edits.
12. The output should be a COCO-style caption, not a detailed scene report.
13. Return strict JSON only:
{
  "caption": "...",
  "used_evidence": ["..."],
  "suppressed_claims": ["..."],
  "revision_operations": ["keep", "simplify", "enrich"]
}

EAG user JSON keys: mode, balanced_profile, global_caption_anchor, detailed_caption_optional_context, allowed_primary_evidence, allowed_weak_evidence, allowed_context_evidence, surface_description_cues, do_not_mention, and writing_rules.

Prompt B: historical Qwen Global+Detailed user message

Write one concise natural image caption.
Use the global caption as the primary linguistic anchor.
Use the detailed caption only to preserve additional information that is compatible with the global caption.
Remove redundant, uncertain, or overly specific details.
Do not introduce information not contained in the two input captions.
Return only the final caption.

Global caption:
{global_caption}

Detailed caption:
{detailed_caption}

Exact preprocessing lexical resources

Hard-risk regex tags:

  • uncertainty: \b(likely|probably|possibly|appears to be|seems to be|may be|might be)\b
  • subjective: \b(delicious|tasty|juicy|succulent|beautiful|nice|dreamy|neatly)\b
  • dietary_claim: \b(vegan|vegetarian|gluten[- ]free)\b
  • identity_or_title: \b(dr\.?|doctor|mr\.?|mrs\.?|ms\.?|prof\.?|professor)\b Soft-risk regex tags:
  • fine_grained_food: \b(falafel|black bean|cheese|tomato sauce|tofu)\b
  • named_entity_like: \b[A-Z][a-z]+(?:\s+[A-Z][a-z]+){1,}\b COMPOUND_NORMALIZATION (all 18 keys): human face → human face; wine glass → wine glass; glass of wine → wine glass; picture frame → frame; wooden frame → frame; advent calendar → calendar; dining room table → table; dining table → table; kitchen table → table; mobile phone → mobile phone; cell phone → mobile phone; land vehicle → vehicle; side salad → salad; tissue paper → napkin; paper towel → napkin; towel napkin → napkin; brown ugg boots → boot; ugg boots → boot. DROP_WORDS: a, an, and, around, at, by, five, for, four, from, in, inside, into, many, near, of, on, onto, or, outside, several, some, the, three, to, two, with. NON_VISUAL_OR_BAD_QUERY: appear, appears, area, background, black, blue, brim, brown, clear, exterior, foreground, gold, golden, gray, green, grey, group, image, interior, item, items, likely, nappet, object, objects, orange, paper, photo, picture, pink, plat, probably, purple, red, scene, setting, sid, side, silver, smile, smiling, towet, view, white, yellow. SINGULAR_MAP: patties→patty, pattie→patty, burgers→burger, sandwiches→sandwich, plates→plate, glasses→glass, carrots→carrot, towels→towel, papers→paper, dishes→dish, people→person, men→man, women→woman, children→child.

Exact branch policy

  1. safe_query prioritizes identity/title→person, longest matching compound, cleaned main head, food-priority head, then last surviving normalized noun. A specific query is suppressed under hard risk, or name-like expressions (except human face, wine glass, side salad), with food-specific exceptions (side salad, tomato sauce, vegetable burger) and default suppression of falafel/black bean.
  2. Both distinct nonempty queries can be sent to DINO (box/text thresholds source defaults 0.30/0.25). After detection, choose specific only if allowed_to_upgrade and specific_query and specific_dino_status==detected_strong; else safe if safe status is strong/weak/coarse; else no verified match. Strong means score≥0.50 and area≤0.65 (source defaults); area>0.65 is coarse even with strong score.

These heuristics do not fully verify non-entity semantics.

AuthorsOpenReview ↗

Gentle Follow-up and Additional Experimental Update

Dear Reviewer,

Thank you again for your detailed and constructive feedback.

As the discussion period is approaching its end, we would like to kindly follow up on our point-by-point response to your concerns (W1–W4 and Q1–Q3).

We have provided additional implementation clarifications, including the claim preparation rules, query selection logic, and prompt differences. We have also conducted further analyses of the coverage loss and grounding recovery mechanisms.

Importantly, the prompt-matched Qwen Rewrite experiment discussed in our response to W4 has now been completed on COCO-2K. With matched writing constraints, EAG-Cap achieves higher CIDEr (0.868 vs. 0.718) and BLEU-4 (0.264 vs. 0.181) than the text-only control, while its CHAIRs is slightly worse (17.79% vs. 17.67%). We acknowledge this mixed result and do not claim superiority across all metrics.

We would be grateful if you could share whether our responses and additional evidence have addressed your main concerns, particularly regarding reproducibility, recovery behavior, and prompt fairness. We would be happy to clarify any remaining issues.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_phyPOpenReview ↗

Official Comment by Reviewer phyP

Thank you for the detailed responses and additional experiments. While the rebuttal has clarified several implementation details and addressed some of my concerns, I remain unconvinced by the paper's core contribution. In particular, the method does not genuinely verify fine-grained non-entity semantics, and the underlying reasons for the low pre-recovery coverage in Stage 2 remain insufficiently explained. The additional analyses improve the clarity of the paper but do not sufficiently address these fundamental limitations. Therefore, I maintain my original rating.

Reviewer_ub64OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introducees a training free framework for improving image caption faithfulness while maintaining spatial grounding for visual concepts mentioned in caption. They basically built a pipeline. First florence 2 generates image level captions and dense region proposal. Candidate description normalized into visual claims and assigned risk. Grounding DINO is used to screen the claims for localization. A frozen Qwen LLM composes concise revised caption using screened evidence. Finally the method identifies caption concepts that lack grounding and use grounding DINO to recover regions. Primary contribution is a system that does caption revision with caption conditioned grounding rekcovery. Experiments done on COCO val2017, Flickr30k. They reported improvement in grounding precision and caption conditioned coverage. However caption quality improvements are mixed. EAG-Cap does ot consistently outperform baselines like Kosmos 2 or florence 2, specially on hallucination and reference based captioning metrics.

Critical strengths

The strong contribution is the anlaysis and resolution of the coverage loss by evidence screening. On COCO 5k caption conditioned coverage increases from 76% to 90.97% without vs with recovery, while grounding precision remains 84.54%. It also reduces the number of boxes per image from 19.75 to 4.9. The corresponding improments on Flicklr30k entities suggest that this behavior is not for one dataset. This method demonstrates a technique fo recovering spatial information.

The method integrates pretrained captioning, grounding, language models without additional fine tuning. So it's basically training free framework.

Critical weaknesses

The prompst frame work primarity an integration of esisitn gpretrained models and heuristic processing stages. The main distinguishing feature is the caption conditioned grounding recovery step. However the paper does not provide a simple control which is directly grounding all concepts in the final EAG-Cap caption using same Grounding DINO model. Also its difficult to establish if the evidence retention and selective recovery procedure provides a big benefit over simpler existing approaches. This concerns paper's novelty.

The paper does not not demonstrate consistent improvments over strong existing captioning methods. On COCO-5k, EAG-Cap achieves better CIDEr and CHAIRs compared to Kosmos-2. Florence-2 also achieves a lower object hallucination rate. On Flickr 30k, Florence outperforms EAG CAP on CIDEr.

CHAIR mainly measure shallucinated object categories, but it claimed there are improvements on attributes, identities, relations, counts etc. Grounding DINO detections do not necessarily check these properties and paper does not have any comprehensive human evaluation.

So the experimental evidence strongly suppors improvements in grounding than general improvements in caption correctness.

Questions for authors
  1. Is the caption driven grounding recovery better than direct regrounding of the final caption ? Can the authors compare the full method against a baseline that extracts all groundable concepts from the final EAG-Cap captions and directly ground them using Grounding DINO? This will help determine whether the proposed retained evidence and selective recovery contributes more than simply grounding the generated caption.

  2. How robust is the approach to the choice of models? The framework presented as training free and broadly applicable, but the main experiments use a single proper, grounding model and LLM composer. Can the authors demonstrate whether main improvements persist with another grounding backbone or provide evidences they are spcific to selected Florence 2 and Grounding DIBO combination? This should show generality of the framework, rather than specific configuration.

2 comments

We thank the reviewer for the thoughtful critique. We agree that the paper supports a stronger claim about caption-aligned grounding than about universal caption correctness.

W1. Existing components and missing direct-grounding control

EAG-Cap combines pretrained Florence-2, GroundingDINO and Qwen with rule-based claim processing; we do not claim to have invented these components. Our proposed contribution is the coupling of evidence screening, constrained caption composition and selective grounding recovery to address screening-induced precision–coverage loss. We have now run the requested same-caption control (Q1); its improvements are measurable but modest, motivating a narrower claim of training-free system-level novelty.

W2. Mixed captioning and hallucination metrics

We agree that EAG-Cap does not consistently outperform strong captioning baselines. On COCO-5K, CHAIRs is 18.01% for EAG-Cap, versus 29.62% for Florence-2 Detailed, 12.55% for Florence-2 Global, and 15.14% for Kosmos-2. Reference-based metrics are likewise mixed across COCO and Flickr30k. Our evidence supports a trade-off relative to dense, noisy proposal captions alongside stronger caption-conditioned grounding, not universal caption-quality or hallucination superiority.

W3. Non-entity semantics and lack of human evaluation

This is a valid limitation. CHAIR primarily assesses recognized object categories. A compatible GroundingDINO box does not prove attributes, relations, counts, identities or actions. Our rule-based risk handling often suppresses or generalizes uncertain non-entity descriptions rather than verifying them visually. We did not perform a comprehensive human correctness evaluation. We will remove broad claims of verified fine-grained semantics and explicitly distinguish localized support from complete factual correctness.

Q1. Does selective recovery beat grounding every final-caption concept?

We evaluated direct grounding of the same final EAG-Cap captions and matched queries using the same detector and evaluation on COCO-5K:

Method[email protected]CC-Cov.CC-IoU
Direct grounding83.27%89.77%0.8323
EAG-Cap Full84.54%90.97%0.8438

EAG-Cap improves precision by 1.26 percentage points, coverage by 1.20 points, and CC-IoU by 0.0115. Paired image-bootstrap 95% CIs (2,000 replicates) exclude zero for all three differences. This shows a small but consistent benefit from retained evidence plus selective recovery when the final caption is held fixed; it does not establish the necessity of every upstream stage.

Q2. Robustness to alternative model choices

We acknowledge that the principal experiments use one proposal model, one grounding backbone and one LLM composer. Results on COCO and Flickr30k-Entities demonstrate replication across datasets, not backbone independence. We have not completed an alternative grounding-backbone experiment and therefore cannot claim that gains persist for other backbones. We will clarify the checkpoint variants and identify cross-backbone robustness as a limitation, rather than infer generality from the training-free architecture alone.

Thank you for helping us distinguish the empirical grounding result from broader, currently unsupported claims.

Thank you for the reply and experiments. I appreciate authors acknowledgment of the limitations and addressing my main concerns. The new same caption grounding comparison is helpful. It show a statistically significant, but modest improvement from the proposed approach. However my main concerns regarding the methodological novelty and incomplete validation of caption faithfulness remain. The lack of component level ablations and evaluation across alternative model backbones also limits the contribution. While the rebuttal improves my assessment of the evidence, I do not believe it sufficiently changes the overall contribution. I will therefore maintain my original rating.

Reviewer_b9LXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EAG-Cap checks Florence-2 phrase proposals with GroundingDINO, rewrites the global caption with a Qwen model, and re-grounds caption concepts selectively. Recovery repairs coverage lost to screening with few, precise boxes, but captioning quality is mixed.

Critical strengths

The pipeline is clean with a sound idea. Table 3 holds the final caption fixed and shows a real effect, and Table 4 repeats the pattern on Flickr30k-Entities (44.07 →\to 78.66%). The drawbacks and limitations are analyzed and stated honestly.

Critical weaknesses

W1. The fixed-caption recovery result is useful, but the evidence does not sufficiently separate the proposed coupling from a simpler caption-then-ground pipeline. At the current state, the grounding comparison has no external baseline or same-caption control. W2. While the engineering effort is clear, the novelty compared to Woodpecker is limited. W3. Many of the model names lack clarity and reproducibility. For instance, Florence-2 has base and large variants. Qwen-3B-Instruct is not the official name. GroundingDINO has Swin-T and Swin-B, the latter was trained on COCO/RefCOCO.

Questions for authors
  1. The reviewer would reconsider the assessment if a new row is added to Table 3 and/or 4 that controls everything in the final EAG-Cap caption but no retained evidence.
  2. The authors should report the raw hallucinating-caption counts / denominator row counts for CHAIRs (if filtered).
2 comments

We thank the reviewer for the constructive and actionable assessment. We address the three weaknesses and two questions below.

W1. Contribution beyond caption-then-ground

We agree the original tables lacked the most direct control. We have now evaluated direct grounding of the identical final EAG-Cap captions using GroundingDINO and the same query inventory, without retained EAG-Cap boxes, on all 5,000 COCO images. Q1 reports the result. The gains are positive but modest, so we do not present this as proof that every component is indispensable.

W2. Novelty relative to Woodpecker

We acknowledge substantial prior work on detector-assisted LLM hallucination correction, including Woodpecker. We do not claim the pretrained models, claim screening, or rewriting in isolation as novel. The contribution is the coupled precision–coverage mechanism: screening removes unsupported proposals; evidence-constrained composition selects caption content; and selective post-composition grounding retains compatible screened boxes while recovering support for remaining caption concepts. We will make this distinction and its limitations explicit without asserting an untested head-to-head advantage over Woodpecker.

W3. Exact model variants and reproducibility

We have audited the available code and run records. Florence-2 generation logs identify microsoft/Florence-2-base (cached snapshot 5ca5edf5bd017b9919c05d08aebef5e4c7ac3bac). The GroundingDINO verification script references a Swin-T OGC configuration/checkpoint; the historical launch arguments are not yet available to independently confirm the weights used. The local Qwen directory is models/Qwen-3B-instruct, whose bundled config specifies Qwen2ForCausalLM (36 layers, hidden size 2048); this alone does not authenticate an official model ID or historical weight hash. We will report confirmed identifiers/versions, source defaults and remaining provenance uncertainty separately rather than present informal labels as official model names.

Q1. Same-final-caption, no-retained-evidence baseline

The direct baseline grounds the same EAG-Cap captions and matched query inventory, under identical evaluation rules:

COCO-5K[email protected]CC-Cov.CC-IoU
Direct, same caption/queries83.27%89.77%0.8323
EAG-Cap Full84.54%90.97%0.8438

The gains are +1.26 percentage points in precision, +1.20 points in caption-conditioned coverage, and +0.0115 in CC-IoU. Image-paired bootstrap (2,000 replicates) yields 95% CIs of [+0.92,+1.60] pp for precision, [+0.85,+1.55] pp for coverage, and [+0.0083,+0.0145] for CC-IoU. We will incorporate this direct baseline in the grounding comparison. These CIs reflect image-resampling uncertainty on this split, not cross-model robustness.

Q2. Raw CHAIRs numerator and denominator

Re-evaluating the 5,000 input captions per method with our original CHAIR implementation gives:

MethodHallucinating / eligibleCHAIRs
Florence-2 Global510 / 4,06312.55%
Florence-2 Detailed1,317 / 4,44729.62%
Kosmos-2605 / 3,99615.14%
EAG-Cap763 / 4,23718.01%

The implementation's CHAIRs denominator is captions containing at least one recognized COCO object mention, not all 5,000 generated captions. We will state this explicitly. EAG-Cap improves upon Florence-2 Detailed, not the concise Florence-2 Global or Kosmos-2 baselines on CHAIRs alone.

We appreciate these suggestions, particularly the explicit same-caption control.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer b9LX,

Thank you again for your constructive review and valuable suggestions.

As the author–reviewer discussion period is approaching its end, we would like to kindly follow up on our rebuttal.

In particular, we have completed the same-caption, same-query direct-grounding control requested in Q1, together with a paired bootstrap analysis on COCO-5K. We have also provided the raw CHAIRs counts and denominators requested in Q2 and clarified the scope and limitations of our contributions.

We would sincerely appreciate it if you could let us know whether these additional experiments and clarifications address your main concerns, or if any questions remain.

Thank you very much for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.