- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes VCap, a reward design for image and video captioning. A frozen multimodal judge uses the visual input and a reference caption to assess correctness, completeness, and text quality. The scores are combined to train a captioning model with GRPO. The paper also presents a hypergeometric analysis of factual checks and regenerates references for a second training round. Experiments include captioning benchmarks, component ablations, human assessment, distillation, and QA transfer.
Critical strengths
- Clear presentation: The paper is generally easy to follow. The roles of the reference caption and visual input are clearly described, and the limitations of the theoretical analysis are acknowledged.
- Broad experimental evaluation: The paper evaluates both image and video captioning and includes reward ablations, human assessment, distillation, and QA transfer. These experiments provide evidence that the overall training recipe is useful.
Critical weaknesses
-
Limited novelty and incomplete positioning: The method mainly combines reference-assisted judging with GRPO, while the hypergeometric analysis is not directly used in reward computation. Recent work such as CCCaption, ClaimDiff-RL, and CapRL++ should be discussed to clarify the differences in completeness, correctness, and factual reward design.
-
Insufficient validation of visual adjudication: The paper does not directly test whether the judge can correct deliberately erroneous references. Human evaluation also checks only model-proposed issues, leaving missed errors unmeasured. Stronger controls and an independent factual audit are needed to support the claimed mechanism.
Questions for authors
- How does VCap differ from the recent captioning methods listed above? Could a matched reference-assisted judge baseline establish the benefit of the proposed formulation?
- Does the judge remain reliable when references contain factual errors or omissions? Could an independent human audit check errors outside the model-proposed list?
5 comments
Response II
Weakness I (b) and Question 1 (a). Positioning against CCCaption, ClaimDiff-RL and CapRL++
We confirm the omission and have added all three to Related Work. We have also run matched comparisons against ClaimDiff-RL-style and CapRL++-style rewards (Q1b below).
All three differ from VCap in where the factual inventory comes from and in what arbitrates. CCCaption derives visual queries from the image with multiple LVLMs and scores a caption by the share it answers, with a separate correctness judge on sub-caption queries; CapRL++ is reference-free, converting the image into a multiple-choice bank that a vision-free LLM answers from the caption alone, with format and length rewards in place of an explicit correctness term. In both, the factual source is the image rather than a reference, no visual arbiter is present at scoring time, and the question set bounds what can be checked. ClaimDiff-RL is the closest prior work: its judge receives the image, the candidate, and a reference from a strong external model (Gemini-3-Pro in the original), enumerates every candidate-reference difference and verifies each against the image. That protocol is more explicit and finer-grained than VCap's three holistic integer scores, and VCap belongs to the same family of reference-anchored comparison with visual adjudication, a family VCap did not introduce. What separates them is that VCap's reference is deliberately weak and self-generated, and serves as a one-directional inspection cue on the candidate rather than as one side of a symmetric comparison.
Across all three, the factual-comparison channel is demonstrated on images only; CapRL++ extends to video, but through a timestamp-format reward. VCap applies the same factual-comparison protocol to video, and gives an account of why the reward works.
Q1b below gives the empirical counterpart: with the reference source held fixed at VCap's weak self-generated caption, difference (1) is not cosmetic, it is where the ClaimDiff-RL-style protocol loses most of its effect.
Question 1 (b). Matched reference-assisted judge baseline
We have added the RL version of the baseline described below: same judge, same images, same reference, same GRPO setup as VCap (e1), but with the judge instruction stripped of VCap's comparison rules, dimension definitions, and "image is ground truth" instruction, replaced by a single overall 0 to 10 score:
| Reward | CapMAS Avg | DecapBench Avg | Factuality | Precision |
|---|---|---|---|---|
| Qwen3-VL-8B-Instr (no RL) | 77.91 | 69.17 | 76.25 | 82.73 |
| Generic reference-assisted judge | 78.80 | 71.27 | 75.43 | 85.43 |
| VCap (e1) | 81.52 | 73.32 | 81.48 | 87.80 |
The generic matched baseline underperforms VCap on every metric, and notably its Factuality (75.43) falls below the untrained backbone (76.25), a reference in the prompt without an explicit comparison protocol and without "the image decides" can make factuality worse than no training at all. This is the matched baseline literally requested, and it separates the proposed formulation from "any judge with a reference in the prompt."
We also completed a matched comparison against the two closest published protocols. Every row uses the same reference caption as VCap (e1) (the weak, self-generated one) not ClaimDiff-RL's original strong external reference or CapRL++'s image-derived question bank, so the only variable changing across rows is the comparison protocol:
| Reward | CLAIR | Coverage | Factuality | Precision | Recall | DCScore |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B-Instr (no RL) | 86.19 | 71.30 | 76.25 | 82.73 | 55.61 | 65.71 |
| Generic reference-assisted judge | 87.06 | 73.90 | 75.43 | 85.43 | 57.11 | 66.20 |
| ClaimDiff-RL-style | 86.35 | 72.26 | 76.60 | 84.43 | 57.61 | 65.87 |
| CapRL++-style (paper default N=8) | 86.78 | 71.84 | 77.49 | 86.79 | 56.37 | 65.27 |
| VCap (e1) | 89.14 | 73.95 | 81.48 | 87.80 | 58.83 | 69.52 |
Under the weakened reference, the ClaimDiff-RL-style reward barely moves CLAIR/Factuality/DCScore off the untrained backbone (all within 0.5 points), which is the regime its pairwise difference-enumeration protocol assumes away: it was built around a strong external anchor. The CapRL++-style reward gets closer on Precision but not on Coverage/Recall/DCScore, with DCScore falling below the untrained backbone, which is what VCap's mechanism reduces to at a small sampled fact count and with no visual arbiter for errors outside the sampled questions. Taken together, the comparison points to VCap being the more robust of the three to both the reference and the reward model. A reference caption can be supplied in whatever form the reward model is invoked, and the amount of factual content it carries is not capped in advance; a VQA bank fixes both how many facts can be checked and the form in which they are checked, so whatever falls outside the sampled questions is unreachable.
Response III
Weakness II (a) and Question 2 (a). Does the judge remain reliable when the reference has errors or omissions?
To settle this, we ran an in-depth independent audit. Using the uncorrected 8B backbone caption as the reference, itself independently audited at 9.43% error rate and 23.24% missed-fact rate, i.e. genuinely "containing factual errors or omissions", we scored VCap (e2) with the 32B judge and stratified its true errors and omissions by whether the reference could help:
| Case | True count | Caught by the judge-assisted protocol | Catch rate |
|---|---|---|---|
| Candidate's inconsistency is also present in the reference | 21 | 14 | 66.7% |
| Inconsistency, reference gives no cue at all | 139 | 105 | 75.5% |
| Inconsistency, reference gives a cue | 42 | 34 | 81.0% |
| Omission, reference also misses it | 634 | 445 | 70.2% |
| Omission, reference does not miss it | 310 | 276 | 89.0% |
The first row is the direct test of your scenario: when the candidate repeats an error the reference also contains, a judge relying on text agreement sees agreement and misses it, while ours still flags two thirds of these cases from the image. Overall detection is 75.7% on inconsistencies and 76.4% on omissions.
Two limits: the shared-error slice is small (n=21), so 66.7% carries a wide interval; and the measurement used a fact-enumerating judge instruction for human verification rather than the three-score training-time reward instruction, so it characterizes the judge's detection capability, not training-time behaviour verbatim.
Neither input is dispensable, though each ablation read alone suggests the other is. The no-cue rows (75.5%, 70.2%) might read as the judge doing nearly as well with no reference; retraining says otherwise, since Table 4's "−reference caption" row lands at 77.65/69.52 CapMAS/DecapBench Avg against the untrained backbone's 77.91/69.17, with Coverage and Recall falling below backbone (70.24 vs. 71.30, 53.24 vs. 55.61). Conversely, Table 4's "−reference image" row costs only 0.48/1.66, which makes visual adjudication look redundant, but that is measured with the reference and the comparison rule still in place, and on most samples the reference carries the signal.
The two are not in conflict because they quantify different things. The table above is recall given that a human-audited error exists at a location; it says nothing about precision when the judge has no reference to bound what is worth checking, and precision is what sets within-group contrast under GRPO. The asymmetry between the axes is the mechanism: an error can be checked against the candidate's own claims, an omission only against some notion of what should have been said. That is where the gap is widest (89.0% versus 70.2%) and exactly the axis that falls below backbone without a reference. The reference supplies the completeness inventory and the image adjudicates correctness, and the minority of cases where the reference is silent or wrong is what determines whether the method degrades gracefully under a weak reference. The revision will report both and scope the visual-grounding claim accordingly. If you consider this evidence insufficient, we will prioritize a larger stratified test with deliberately injected reference errors in the remaining discussion period.
Response of Submission402 for Reviewer Q14r
We thank the reviewer for recognizing that the respective roles of the reference caption and the visual input come through clearly, since that separation of roles is what the rest of the method rests on. We are also grateful for the comments on the relation between the analysis and the implementation, the positioning against related work, judge reliability, and the coverage of our human-evaluation protocol. We did a substantial amount of additional work on these questions, and the findings have improved the paper.
Weakness I (a). Hypergeometric analysis not directly used in reward computation
It is true that no slot extraction is performed and no hypergeometric quantity is computed at training time (Section 2.2). The hypergeometric structure is nonetheless realized in the way VCap's reward is constructed. The model characterizes the statistical structure of the protocol (supplying both visual and partial textual evidence, using the reference as a per-claim inspection cue with the image adjudicating) and that protocol is precisely what the reward implements, whether or not a hypergeometric quantity is computed explicitly at runtime. Section 2.3's Scope paragraph limits the analysis to the reference-guided channel rather than to the judge's full behaviour.
We ran a new experiment to test this directly: degrading exactly this structure (judge compares the candidate directly against the image, with the reference as a weak anchor rather than a per-claim inspection cue) while holding images, GRPO hyperparameters, and judge fixed:
Removing the structure loses ground on all eight metrics, with the correctness axes (Factuality, Precision) and the completeness axes (Coverage, Recall) dropping together, the direction a real mechanism, not a cosmetic framing, would predict.
A second, more direct test of the model's own prediction. The hypergeometric account says the reference opens a detection channel: facts the reference covers should be easier to check than facts it does not. We measured this directly (full table in W2a/Q2a below). On inconsistencies, detection is 81.0% when the reference provides a textual cue versus 75.5% when it provides none; on omissions, 89.0% when the reference covers the fact versus 70.2% when it does not. The channel the analysis describes is visible in the measurement, and the residual detection in the no-cue regime is what the Scope paragraph attributes to direct visual inspection.
We will soften the title's framing in-text, add this ablation to Section 3.4, and revise the title accordingly.