acceptodds
Under review as a conference paper at ICLR 2027

CuRe: Claim-Level Rubric Rewards for Video Captioning

Abstract

Existing reward functions for dense video captioning largely assume reference captions provide complete supervision, overlooking that a single reference can rarely cover all valid visual content in a video. To address this issue, we propose Claim-Level Rubric Rewards (CuRe), a reward framework that decomposes captions into visual claims such as entities and actions, and verifies each claim directly against the video. This claim-level verification enables CuRe to distinguish fine-grained errors from valid details omitted by the reference. We then define a reference-anchored credit weighting scheme that assigns credit based on both visual support and reference alignment: reference-matched claims receive full credit, video-supported claims beyond the reference receive bounded credit, and unsupported claims receive zero credit. This design rewards visually grounded coverage rather than strict reference matching in text, allowing valid details beyond the reference to contribute while keeping unsupported claims penalized. On three public captioning benchmarks, CuRe achieves the strongest results among open-source models. To test whether CuRe-generated captions provide better supervision than the original captions, we re-caption three video-caption corpora (867K samples) and pretrain VLMs with either the original or CuRe-generated captions. CuRe-recaptioned data improves performance on 25 of 27 metrics across six downstream video-understanding benchmarks, demonstrating that CuRe-generated captions provide stronger video-language supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Captioning, Reinforcement Learning, Post-training, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4sM2OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CuRe, a claim-level reward framework for reinforcement learning in detailed video captioning. Rather than evaluating generated captions through holistic judgments or strict reference matching, CuRe decomposes captions into atomic visual claims and verifies their factual support directly against video evidence.

The method introduces reference-anchored credit weighting that assigns full credit to reference-matched claims, partial credit to additional video-supported claims, and zero credit to unsupported claims. Category-wise precision and recall rewards are combined to optimize a video captioner through GRPO.

The authors evaluate CuRe on three video-captioning benchmarks and through caption-based video question answering. They also replace the original captions of three training corpora containing 867K videos with CuRe-generated captions and demonstrate improvements on 25 of 27 downstream evaluation metrics. Controlled reward ablations and human evaluations provide additional evidence of improved factual accuracy and semantic coverage.

Critical strengths
  1. A well-motivated reward formulation addressing incomplete reference supervision. The paper identifies an important limitation of conventional captioning rewards: details absent from a reference caption are not necessarily incorrect. The proposed claim-level verification and reference-anchored credit weighting provide a principled way to distinguish unsupported statements from valid additional visual information. This is a meaningful methodological contribution because it addresses factual reliability and semantic coverage simultaneously, rather than optimizing one at the expense of the other.

  2. Strong validation extending beyond direct captioning benchmarks. The experimental design is a major strength. In addition to direct captioning evaluations, the authors demonstrate that CuRe-generated captions improve subsequent VLM pretraining across three different video-caption corpora. The improvements on 25 of 27 downstream metrics are particularly compelling because they provide evidence that the generated captions are useful as training supervision, not merely better aligned with a particular caption evaluation metric. The reward ablations and human evaluations further strengthen this conclusion.

Critical weaknesses
  1. The reward formulation remains dependent on imperfect model-based visual verification. The reliability of CuRe fundamentally depends on correctly determining whether individual claims are supported by the video. Although Appendix C.5 evaluates the reward modules, the reported verification accuracy of 81.93% and false-support rate of 6.17% indicate that verification errors remain non-negligible. More importantly, the verification analysis in Appendix C.8 reports a conflicting-claim F1 of only 29.8% for the proposed category-based grouping. These results raise concerns about how reliably CuRe penalizes difficult hallucinations. The current validation uses another large VLM as the primary reference for module correctness, so the extent to which these measurements reflect independent human judgments remains uncertain. Since factual grounding is the central motivation of the method, this limitation is consequential to the paper's main claim.

  2. Generalization of the reward framework beyond the selected policy and reference models is not established. The main experiments use a single policy-model backbone and reference captions from Gemini 3.0 Pro. Although the authors evaluate a smaller reward model, they do not demonstrate whether the proposed credit assignment remains effective when reference quality, reference completeness, or the policy architecture changes. This matters because the reference caption determines which claims receive full credit, while additional verified claims receive only partial credit. Consequently, differences in reference detail or systematic reference omissions could alter the reward distribution and potentially affect the learned captioning behavior. The current experiments establish effectiveness under the chosen configuration but provide limited evidence of broader reward portability.

Questions for authors
  1. Given the relatively low F1 for detecting conflicting claims in Appendix C.8, how frequently do unsupported claims receive positive CuRe reward in practice? Could the authors report a human-verified analysis of false-positive reward assignments, particularly for subtle temporal, relational, and attribute-level hallucinations?

  2. How sensitive is CuRe to the completeness and quality of the reference captions? In particular, would the improvements persist if the Gemini-generated reference were replaced by a shorter or less comprehensive reference source? This would help determine whether the reference-anchored weighting scheme generalizes beyond the current supervision setting.

1 comment

I have decided to maintain my score and hope that all the suggested revisions will be properly incorporated into the final version of the paper.

Reviewer_3VWkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Claim-Level Rubric Rewards (CuRe), a reward framework for dense video captioning that addresses the limitations of reference-based evaluation, where a single reference caption may not cover all valid visual content. CuRe decomposes generated captions into fine-grained visual claims, such as entities and actions, and verifies each claim directly against the video. It introduces a reference-anchored credit weighting scheme that rewards reference-matched claims fully, assigns bounded credit to visually supported claims beyond the reference, and penalizes unsupported claims. Experiments on three public captioning benchmarks demonstrate competitive performance, while large-scale re-captioning experiments involving 867K samples show that CuRe-generated captions improve results on 25 of 27 metrics across six downstream video-understanding benchmarks.

Critical strengths
  1. The paper addresses an important limitation of conventional captioning rewards: strict reference matching may penalize correct visual details that are absent from reference captions. The claim-level formulation provides a reasonable way to distinguish visual grounding from textual agreement.

  2. The proposed reference-anchored credit weighting scheme offers a clear balance between rewarding additional valid information and preventing unsupported claims, potentially improving the reliability of caption evaluation and optimization.

  3. The experiments cover multiple public benchmarks and downstream video-understanding tasks. In particular, the large-scale re-captioning and pretraining study provides useful evidence that the proposed framework may benefit not only captioning evaluation but also the quality of video-language training data.

Critical weaknesses
  1. Although the overall idea is reasonable, the methodological novelty appears moderate. Decomposing text into atomic claims, verifying claims against visual evidence, and assigning graded rewards are related to existing fact verification, grounded evaluation, and fine-grained reward modeling approaches. The paper should more clearly establish the conceptual and technical differences between CuRe and these related methods.

  2. The reliability of claim-level verification is central to the framework, but the abstract does not clarify how claims are extracted, how visual support is determined, or how verification errors are controlled. Errors in claim decomposition or video-grounded verification could directly affect reward quality and potentially reinforce incorrect captions.

  3. The empirical results are promising, but the reported aggregate improvements do not fully reveal the robustness of CuRe across different captioning models, video domains, or types of visual claims. Further analysis of the reward scheme, including sensitivity to credit-weighting choices and the quality of the verification process, would strengthen the evidence for the proposed design.

  4. The manuscript is generally understandable, but its presentation could be improved. Some experimental figures and tables may benefit from clearer organization and more informative comparisons, while certain textual descriptions could be made more precise to help readers understand the proposed mechanism and interpret the experimental findings.

Questions for authors
  1. How does CuRe differ technically from existing claim-level fact verification, video-grounded caption evaluation, and fine-grained reward modeling methods? Which specific component constitutes the main methodological contribution?

  2. How are visual claims extracted and verified against the video? What evidence demonstrates the reliability of the verification process, and how does CuRe handle ambiguous actions, temporal relationships, or claims that are only partially supported by the video?

  3. How sensitive are the results to the credit assigned to video-supported claims that are absent from the reference? Have the authors conducted ablation or sensitivity analyses to establish that the proposed reference-anchored weighting scheme is preferable to alternative reward formulations?

  4. Can the authors provide more detailed analyses of the downstream improvements, including performance across different model scales or video-understanding tasks, as well as the computational cost of claim extraction and verification? Such analyses would help clarify the generalizability and practical applicability of CuRe.

Reviewer_m7jdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CuRe, a claim-level reward for RL-based video captioning. An LLM decomposes rollout captions into atomic claims over ten visual categories, a VLM (Qwen3-VL-235B) verifies each claim against the video, and supported claims are matched one-to-one to claims decomposed from a Gemini-3.0-Pro reference caption. A reference-anchored credit scheme gives full credit to reference-matched claims, discounted credit (w_u = 0.5) to supported but unmatched claims, and zero credit to unsupported claims, in both a precision term and a capped recall term. Category-wise scores are aggregated with hand-set priors and a length penalty, and the result is used as the GRPO reward for a Qwen3-VL-30B-A3B captioner warm-started with Gemini-distilled SFT. The paper reports results on EventHallusion, VCapsBench, and DREAM-1K, on downstream VLM pretraining with 867K CuRe-recaptioned samples, and on Prism-style caption-to-QA evaluation.

Critical strengths
  1. The problem that a single reference cannot distinguish false claims from valid omissions is real for dense captioning, and the paper quantifies it: 42.71% of verifier-supported claims are unmatched to the reference.
  2. Factual accuracy improves clearly over reward baselines that share the same SFT start and GRPO budget (VCapsBench IR 13.78 vs. 15.65 for Holistic; 0.97 vs. 4.45 human-counted hallucinations per caption against Continued SFT).
  3. The reporting is broad and detailed: three evaluation settings, analyses of reward-module quality and grouping granularity, and full prompts and runtime configurations.
Critical weaknesses
  1. The novelty of the reward design is limited. Decomposing captions into atomic units and checking them against a reference (AutoDQ) or the visual input (FaithScore) is an established paradigm. Fig. 1 omits the natural alternative, verification-only claim precision (the precision term of Base P/R). Relative to it, Eq. 5 lowers the credit of supported claims beyond the reference, and only Eq. 6 extends credit beyond it. The actual contribution narrows down to a discount factor and a cap on decompose-and-verify, but this distinction is not made clear.
  2. The reward structure works against the coverage it targets. Beyond the discount w_u, extra content is limited by the recall cap δ_k and the claim-ratio penalty Ω. Precision, which carries λ = 0.7 of the reward, is averaged only over non-empty categories (Eq. 7), so dropping a category with below-average precision raises the precision term. Consistent with this, coverage collapses under uniform category weights (recall 27.32 and CR 67.07, both below SFT), which leaves the method dependent on hand-set priors.
  3. The experiments do not show the coverage benefit that motivates the credit weighting. Against Base P/R, which differs only in Eqs. 5–6, CuRe changes DREAM-1K recall by +0.27 and VCapsBench CR by +0.57, while DREAM-1K precision rises by 6.18. Against SFT, recall even drops (36.26 vs. 38.75). The effect is thus on precision rather than coverage, and on DREAM-1K this precision is measured against a single reference, the protocol the paper argues is unreliable. w_u is not swept, the human evaluation omits Base P/R, and no variance is reported.
  4. The gains over external models and in downstream use are not attributed to the reward. The policy is SFT-distilled from Gemini-3.0-Pro, which also produces the references, and SFT alone accounts for most of the VCapsBench CR gain over the base model (71.49 → 77.51, vs. 78.60 for CuRe). Tables 1 and 3 have no SFT-only row. Table 2 compares CuRe recaptions only with the original captions, so its gains may come from Gemini distillation or longer captions; recaptioning with the SFT-only or Holistic-reward model is the necessary control.
Questions for authors

Q1. Could you report a sweep over w_u, including 0 and 1, to show whether the credit weighting itself improves coverage?

Q2. The Reference-matching baseline is a caption-level, text-only judge. How does claim-level reference matching without video verification compare? This would isolate the role of the video.

Q3. How were the category priors chosen, and were evaluation benchmarks used in this process? Could you report per-category claim counts before and after GRPO under the proposed and uniform priors?

Q4. The Holistic baseline does not use Ω (Appendix C.3). Does CuRe’s advantage hold when Ω is applied to it, and across multiple seeds?

Q5. The qualitative examples contain hedged claims such as “head or upper body” and “extends an arm or points”. Could you report the rate of such claims for each model and include Base P/R in the human evaluation?

Q6. For Table 2, what are the results when recaptioning with the SFT-only or Holistic-reward model, and what is the average caption length of each corpus?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.