- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents EAG-Cap, a training-free, post-hoc framework for grounded image caption revision. To address the issue that dense region proposals from grounded vision–language models often include unsupported claims while conservative filtering can remove valid spatial support for concepts in the final caption, EAG-Cap processes outputs through four sequential stages using frozen off-the-shelf models: (1) Proposal and Claim Preparation, which collects global captions, detailed captions, object detections, and dense region descriptions from Florence-2 and applies rule-based normalization and risk tagging to form candidate visual claims; (2) Visual Evidence Screening, which queries GroundingDINO on the image for each candidate claim and categorizes the resulting detections into discrete evidence states (strong, weak, coarse, or rejected) based on confidence and bounding-box area; (3) Evidence-Aware Caption Composition, which uses Qwen-3B-Instruct to revise the Florence-2 global caption (with the detailed caption as optional context) under prompt constraints defined by the screened evidence states; and (4) Caption-Driven Grounding Recovery, which filters retained evidence boxes to match the generated caption, extracts remaining caption concepts that lack compatible spatial support, and re-queries GroundingDINO to recover and merge their bounding boxes. The framework is evaluated on the 5,000-image COCO val2017 split and a 1,000-image Flickr30k-Entities subset using standard captioning metrics, object hallucination rates (CHAIR), bounding-box [email protected], and caption-conditioned grounding coverage (CC-Cov.).
Critical strengths
- Clear formulation of caption–grounding consistency: The paper identifies a practical trade-off in post-hoc grounded caption revision—namely, that conservative filtering of noisy region proposals improves box precision at the expense of pruning valid spatial support for concepts mentioned in the final caption—and jointly evaluates language faithfulness and caption-conditioned grounding coverage (CC-Cov. and CC-IoU).
- Training-free modular pipeline with transparent efficiency profiling: The framework operates entirely with frozen off- the-shelf models (Florence-2, GroundingDINO, and Qwen-3B-Instruct) without task-specific training, and the authors transparently report stage-wise wall-clock latency and GPU memory overhead in Section 4.1 and Appendix A.2 (Table 6).
Critical weaknesses
- Insufficient implementation details for rule-based claim preparation and concept extraction (Sections 3.2 & 3.5, Appendices B & E): Converting proposal phrases into calibrated claims (safe vs. specific queries) and risk tags relies on matching "recognized compound heads," removing "nonvisual terms," and flagging "dietary claims" and "fine-grained food terms" (Section 3.2, Appendix B), while concept extraction in Section 3.5 relies on a fixed list of generic lexical equivalences (Appendix E). However, the manuscript does not provide the specific lexicons or transformation rules used in Appendix B, nor does it specify the exact condition under which the safe versus specific query is passed to GroundingDINO. Providing the complete rules, word lists, and concrete examples of and would significantly improve reproducibility.
- Suppression rather than visual verification of non-entity semantics (Sections 3.3–3.5): The actual pipeline lacks a mechanism to visually verify these non-entity semantics. As noted in Section 3.3, GroundingDINO only checks localized entity presence, and the caption composer in Section 3.4 is a text-only LLM (Qwen-3B-Instruct) without visual access to the image. Consequently, non-entity details are handled primarily by suppressing them via prompt rules (Appendix D) and discarding relation/action words as delimiters during concept extraction (Appendix E). This limits the method's capability to coarse entity-level filtering and reduces descriptive richness, rather than genuinely resolving fine-grained attribute or relational hallucinations.
- Large drop in caption-conditioned coverage before Stage 4 recovery and missing direct-grounding baseline (Sections 3.3–3.5, Tables 3–4): Although Stage 3 composes the final caption under the constraints of Stage 2's Evidence State Store, prior to Stage 4 recovery (w/o Grounding Recovery), CC-Cov. drops sharply to 76.07% on COCO (2.313 vs. 4.907 boxes/image in Table 3) and 44.07% on Flickr30k (vs. 78.66% after recovery in Table 4). It is unclear why so many concepts in the final caption lack spatial support from Stage 2 if caption composition is constrained by the screened evidence, and why the same GroundingDINO model with identical thresholds (0.30 / 0.25) can subsequently recover them in Stage 4.
- Uncontrolled prompt differences in the text-only baseline (Section 4.6, Table 5, Appendix D): In Table 5 (COCO-2K), the text-only baseline Qwen Rewrite (Global + Detailed) already achieves most of the hallucination reduction (19.26% vs. 17.79% CHAIRs) and SPICE (0.222 vs. 0.230) without any visual screening. While EAG-Cap shows higher BLEU-4 (0.264 vs. 0.191) and CIDEr (0.868 vs. 0.612), Appendix D reveals that the EAG-Cap prompt explicitly enforces "one natural COCO-style sentence of 8–18 words" and "minimal edits" to the global caption, whereas these constraints are omitted from the Qwen Rewrite prompt. This confounding difference in prompt constraints makes it difficult to attribute the BLEU-4 and CIDEr gains in Table 5 to visual evidence screening rather than to explicit length and style instructions.
Questions for authors
- Could you provide the specific lexicons/rules used for matching "recognized compound heads," removing "nonvisual terms," and assigning hard/soft-risk tags, and clarify the exact rule determining when a safe query versus a specific query is passed to GroundingDINO?
- why does w/o Grounding Recovery have relatively low CC-Cov. (76.07% in Table 3 and 44.07% in Table 4) after Stage 3 composition—do the ungrounded concepts in primarily come from phrases retained from the Florence-2 Global anchor?
- Can you provide the two different prompts listed in Appendix D?
4 comments
Source-literal templates and key lexical resources (current source, not verified historical request logs).
Prompt A: EAG-Cap balanced/anchor system message
You are the writing module of an evidence-aware grounded captioning system.
Your job is to revise a concise global caption into one COCO-style caption under evidence constraints.
Rules:
1. Use the Florence-2 Global caption as the primary sentence anchor if it is not unsafe.
2. Preserve the global caption wording and sentence structure as much as possible.
3. Use the Florence-2 Detailed caption only as optional context, not as the main source.
4. Use verified evidence to correct or lightly enrich the caption.
5. Add at most one additional verified object, attribute, or action beyond the global caption.
6. Do not convert the caption into an object list.
7. Prefer one concise natural sentence of 8 to 18 words.
8. Do not mention rejected evidence or anything in do_not_mention.
9. Do not mention weak or coarse evidence unless it is low-risk and directly supported by the image.
10. Do not mention exact text, logos, identities, brands, counts, or fine-grained categories unless verified.
11. If the global caption is already safe and visually plausible, keep it with minimal edits.
12. The output should be a COCO-style caption, not a detailed scene report.
13. Return strict JSON only:
{
"caption": "...",
"used_evidence": ["..."],
"suppressed_claims": ["..."],
"revision_operations": ["keep", "simplify", "enrich"]
}
EAG user JSON keys: mode, balanced_profile, global_caption_anchor, detailed_caption_optional_context, allowed_primary_evidence, allowed_weak_evidence, allowed_context_evidence, surface_description_cues, do_not_mention, and writing_rules.
Prompt B: historical Qwen Global+Detailed user message
Write one concise natural image caption.
Use the global caption as the primary linguistic anchor.
Use the detailed caption only to preserve additional information that is compatible with the global caption.
Remove redundant, uncertain, or overly specific details.
Do not introduce information not contained in the two input captions.
Return only the final caption.
Global caption:
{global_caption}
Detailed caption:
{detailed_caption}
Exact preprocessing lexical resources
Hard-risk regex tags:
uncertainty:\b(likely|probably|possibly|appears to be|seems to be|may be|might be)\bsubjective:\b(delicious|tasty|juicy|succulent|beautiful|nice|dreamy|neatly)\bdietary_claim:\b(vegan|vegetarian|gluten[- ]free)\bidentity_or_title:\b(dr\.?|doctor|mr\.?|mrs\.?|ms\.?|prof\.?|professor)\bSoft-risk regex tags:fine_grained_food:\b(falafel|black bean|cheese|tomato sauce|tofu)\bnamed_entity_like:\b[A-Z][a-z]+(?:\s+[A-Z][a-z]+){1,}\bCOMPOUND_NORMALIZATION(all 18 keys): human face → human face; wine glass → wine glass; glass of wine → wine glass; picture frame → frame; wooden frame → frame; advent calendar → calendar; dining room table → table; dining table → table; kitchen table → table; mobile phone → mobile phone; cell phone → mobile phone; land vehicle → vehicle; side salad → salad; tissue paper → napkin; paper towel → napkin; towel napkin → napkin; brown ugg boots → boot; ugg boots → boot.DROP_WORDS: a, an, and, around, at, by, five, for, four, from, in, inside, into, many, near, of, on, onto, or, outside, several, some, the, three, to, two, with.NON_VISUAL_OR_BAD_QUERY: appear, appears, area, background, black, blue, brim, brown, clear, exterior, foreground, gold, golden, gray, green, grey, group, image, interior, item, items, likely, nappet, object, objects, orange, paper, photo, picture, pink, plat, probably, purple, red, scene, setting, sid, side, silver, smile, smiling, towet, view, white, yellow.SINGULAR_MAP: patties→patty, pattie→patty, burgers→burger, sandwiches→sandwich, plates→plate, glasses→glass, carrots→carrot, towels→towel, papers→paper, dishes→dish, people→person, men→man, women→woman, children→child.
Exact branch policy
safe_queryprioritizes identity/title→person, longest matching compound, cleaned main head, food-priority head, then last surviving normalized noun. A specific query is suppressed under hard risk, or name-like expressions (excepthuman face,wine glass,side salad), with food-specific exceptions (side salad,tomato sauce,vegetable burger) and default suppression offalafel/black bean.- Both distinct nonempty queries can be sent to DINO (box/text thresholds source defaults 0.30/0.25). After detection, choose specific only if
allowed_to_upgradeandspecific_queryandspecific_dino_status==detected_strong; else safe if safe status is strong/weak/coarse; else no verified match. Strong means score≥0.50 and area≤0.65 (source defaults); area>0.65 is coarse even with strong score.
These heuristics do not fully verify non-entity semantics.
Gentle Follow-up and Additional Experimental Update
Dear Reviewer,
Thank you again for your detailed and constructive feedback.
As the discussion period is approaching its end, we would like to kindly follow up on our point-by-point response to your concerns (W1–W4 and Q1–Q3).
We have provided additional implementation clarifications, including the claim preparation rules, query selection logic, and prompt differences. We have also conducted further analyses of the coverage loss and grounding recovery mechanisms.
Importantly, the prompt-matched Qwen Rewrite experiment discussed in our response to W4 has now been completed on COCO-2K. With matched writing constraints, EAG-Cap achieves higher CIDEr (0.868 vs. 0.718) and BLEU-4 (0.264 vs. 0.181) than the text-only control, while its CHAIRs is slightly worse (17.79% vs. 17.67%). We acknowledge this mixed result and do not claim superiority across all metrics.
We would be grateful if you could share whether our responses and additional evidence have addressed your main concerns, particularly regarding reproducibility, recovery behavior, and prompt fairness. We would be happy to clarify any remaining issues.
Thank you very much for your time and consideration.
Best regards,
The Authors
Official Comment by Reviewer phyP
Thank you for the detailed responses and additional experiments. While the rebuttal has clarified several implementation details and addressed some of my concerns, I remain unconvinced by the paper's core contribution. In particular, the method does not genuinely verify fine-grained non-entity semantics, and the underlying reasons for the low pre-recovery coverage in Stage 2 remain insufficiently explained. The additional analyses improve the clarity of the paper but do not sufficiently address these fundamental limitations. Therefore, I maintain my original rating.
We thank the reviewer for the detailed methodological concerns.
The implementation uses finite English rules, not learned claim calibration: four hard-risk regex groups (uncertainty, subjective, dietary, identity/title), two soft-risk groups (fine-grained food, name-like phrases), an 18-entry compound-normalization dictionary, stop/query-exclusion sets and simple singularization. Both distinct nonempty safe and specific queries may be sent to GroundingDINO; the composer makes the post-detection selection.
We agree. GroundingDINO provides object-region compatibility, not verification of actions, relations, attributes, counts or identity. The text-only composer mainly omits/generalizes risky non-entity claims and may reduce informativeness. We will not characterize this as comprehensive semantic hallucination correction.
We evaluated a strict pre-recovery condition with all grounding-field fallbacks disabled, on COCO-5K:
The +15.98 pp coverage gain (paired-bootstrap 95% CI [+15.13,+16.90] pp) comes with a 3.91 pp precision loss. Among 6,490 caption-conditioned GT-category units, the full method matches 1,037 more units overall; 586 remain unmatched. The earlier 76.07% pre-recovery figure used a less strict evaluation with grounding fallback; we will distinguish the two.
Re-querying is not a repeat of the same exact query set: Stage 4 extracts short phrases from the final caption, optionally incorporating used evidence and caption-compatible phrases, then queries only concepts missing compatible retained boxes. Query wording/candidate pools can therefore differ even with the same detector thresholds. This mechanism explains how new detections can arise; it does not prove why every Stage 2 concept was absent.
We agree: the historical Qwen Rewrite lacks EAG-Cap's explicit 8–18-word, minimal-edit, one-added-item and strict JSON-output instructions. The original Table 5 therefore cannot isolate evidence access from prompt design. We have audited the source templates and prepared a prospective COCO-2K prompt-matched text-only control. Its CPU input audit passed on all 2,000 images; GPU inference is underway. No final results are available yet. Historical EAG request-level prompt provenance remains unverified.
The source matches
uncertainty,subjective,dietary_claim, andidentity_or_titlehard-risk patterns; soft tags includefine_grained_foodandnamed_entity_like. Safe queries use longest recognized compounds or stripped visual heads (e.g., "vegan black bean burger" → "burger"; "Dr. ... headshot" → "person"). Specific queries are suppressed under hard risk or name-like text, with explicit tested exceptions such as "side salad" and "tomato sauce".The exact selection is: specific iff
allowed_to_upgradeandspecific_queryexists and its detection isdetected_strong(score ≥ 0.50, normalized box area ≤ 0.65); otherwise safe iff it has anydetected_strong/weak/coarseresult; otherwise neither is verified. The detector's box/text thresholds default to 0.30/0.25. Literal regexes and lexical lists are given in our accompanying source-detail comment.The global caption is explicitly the composer's primary anchor, so it can retain concepts that are not covered by the screened proposal box pool. However, lexical overlap with the anchor is not causal proof that a missing box was copied from it. The logs do not justify a claim that most missing boxes are due to over-pruning. What the paired experiment establishes is improved caption-conditioned localization after re-querying.
The EAG-Cap Anchor uses a system prompt requiring the Global caption anchor, optional Detailed context, evidence constraints, at most one verified addition, 8–18 words, minimal edits and strict JSON; its user message is a per-image JSON evidence table. The historical Qwen Global+Detailed Rewrite uses a single user prompt to concisely rewrite the two supplied captions, avoiding unsupported details, and returns plain text. The complete source-literal prompts are given in our accompanying comment; historical request logs remain unverified.
We will report the rules, exact prompt templates, clear metric definitions and remaining limitations; unfinished GPU comparisons will not be presented as completed results.