acceptodds
Under review as a conference paper at ICLR 2027

RubiCap: Rubric-Guided Reinforcement Learning for Dense Image Captioning

Abstract

Dense image captioning is critical for vision-language pretraining and for text-to-image generation, but scaling expert-quality annotations is prohibitively expensive. While synthesizing captions from strong vision-language models (VLMs) is a practical alternative, supervised distillation often yields limited output diversity and weak generalization. Reinforcement learning (RL) could overcome these, but its successes have so far been concentrated in verifiable domains that rely on deterministic checkers—a luxury not available in open-ended captioning. We address this verification bottleneck with RubiCap, an RL framework that derives fine-grained, sample-specific rewards from LLM-written rubrics. For each image, an LLM rubric writer compares captions from a diverse committee of VLMs to identify consensus strengths and diagnose the current policy's deficiencies. These findings are converted into explicit evaluation criteria, enabling an LLM judge to decompose quality assessment and replace coarse scalar rewards with structured, multi-faceted assessments. Across extensive benchmarks, RubiCap achieves the highest win rates on CapArena, outperforming supervised distillation, prior RL methods, human-expert annotations, and GPT-4V-augmented outputs. On CaptionQA, it shows superior word efficiency: our 7B model matches Qwen2.5-VL-32B-Instruct, and our 3B model surpasses its 7B counterpart. Remarkably, using a compact RubiCap-3B as a captioner produces stronger pretrained VLMs than those trained on captions from proprietary models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Language Models, Dense Image Captioning, Rubric-based Evaluation, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eGAROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes RubiCap, a reinforcement learning framework for dense image captioning. It uses captions from a committee of visual models to identify the student's deficiencies and construct image-specific rubrics. A text-only judge checks generated captions against these criteria, and their weighted scores provide the GRPO reward. Experiments evaluate caption quality, word efficiency, retention of general capabilities, and the usefulness of generated captions for VLM pretraining.

Critical strengths
  1. Clear formulation: The paper is easy to follow, and the explicit criteria make the reward more interpretable than an overall quality score.
  2. Broad evaluation: The experiments cover multiple model sizes, component ablations, rubric-augmented SFT, human preferences, and downstream pretraining utility.
Critical weaknesses
  1. The reward has an important coverage gap: The rubrics are fixed offline and cover only the student's initial deficiencies (Section 2.2; Appendix E). New claims outside the checklist, or errors in previously correct content, may leave criterion scores unchanged. The text-only judge cannot independently verify these claims against the image. This gap needs direct evaluation.
  2. Teacher consensus is not sufficient factual validation: Shared teacher errors can enter the rubrics, while correct details outside the consensus may be omitted. The committee-size ablation and overall preference results do not directly establish rubric accuracy or coverage. An independent audit is needed to support the reliability of the reward.
Questions for authors
  1. If a caption satisfies the rubrics but adds a false claim outside them, does its reward decrease? Please test such cases and explain how newly introduced errors are constrained.
  2. Could the authors independently evaluate the factual accuracy and coverage of the synthesized rubrics, including shared teacher errors and content the initial student already described correctly?
2 comments

We are grateful for your feedback and for recognizing that our paper is easy to follow and that our synthetic rubric design is transparent and interpretable. We also thank you for highlighting our wide range of analyses and empirical results, a strength other reviewers noted as well. Our study assesses caption quality across six different evaluation dimensions, including a human annotation study. We address your questions below.

  • On Reward Coverage.
    • We appreciate this concern. First, we clarify that our paper focuses on offline rubric synthesis. As discussed in Appendix E, our teacher committee is queried once, a one-time offline cost. In contrast, online rubric generation re-synthesizes rubrics against the current policy, adding substantial latency to every training step. Online rubrics are also difficult to audit during training, which is directly relevant to the reviewer's concern about rubric correctness; offline synthesis, by contrast, allows human experts to review and verify rubrics before training begins. Our goal is an approach that is both effective and efficient.
    • Second, we acknowledge that criteria not covered by the teacher consensus are excluded from our RL rewards. However, we find that such criteria typically involve minor details whose omission does not harm overall caption quality; in fact, including them can make captions verbose and degrade quality. Leveraging a committee of teacher models captures what matters most and helps close the gap between the teacher and student captioning policies.
  • On Teacher Consensus.
    • We thank the reviewer for this important question. First, we note that dense captioning is an open-ended and subjective task, unlike mathematics and coding, where factual correctness can be guaranteed and used to derive deterministic verifiers from ground truth. Even rubrics written by human experts (which we explored extensively internally) require many iterations on annotation guidelines, quality reviews, and crowd training. They can still be imperfect and biased; most importantly, they are difficult to scale. We discuss these points in our introduction.
    • Second, we acknowledge that gold-standard rubrics are hard to obtain, so our work instead leverages the expertise of multiple models to derive a reliable approximation of high-quality rubrics. In our committee-size study (Appendix J, Table 7), we show that larger committees provide more diverse teacher perspectives, which in turn yield higher-quality captions in win rate and better rubrics for RL training. Notably, adding smaller and weaker models (Gemma-3 and Qwen3-VL in our case) does not introduce noise that degrades performance; instead, these models contribute additional perspectives that help the rubric writer reach a more reliable consensus.
    • Third, during our experiments, we carefully reviewed the generated rubrics with our annotators and iteratively refined our synthesis prompt. Moreover, we validate our captions with human annotators: our human study (Appendix K) shows that RubiCap receives the highest Elo rating among all sources and is chosen by annotators in 70% of the comparisons in which it appears. We further provide direct support for the factual accuracy of our captions with two additional experiments. In our blind-ranking experiment (Section 3.2), GPT-5.4, GPT-5, GPT-4.1, and Sonnet 5 serve as independent rankers, evaluating the generated captions on accuracy, completeness, clarity, and hallucination. Our captions achieve the lowest hallucination rate and the highest accuracy, outperforming even the larger 72B model. In our CaptionQA experiment (Section 3.4), RubiCap-trained models more accurately answer multiple-choice questions derived from factual content, such as object attributes, object counting, and information in documents. Our RubiCap-7B outperforms the 32B model under tighter word budgets, delivering 32B-level caption quality at 7B inference cost. If the policy were exploiting unpenalized hallucinations, we would expect its hallucination rate to be higher, not the lowest among all models.

Dear Reviewer eGAR,

Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.

Reviewer_MSKJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses scalable, detailed image captioning through RL(reinforcement learning) with automatically generated, sample-specific rubrics as training rewards. A committee of VLM(vision-language models) provides candidate captions, while an LLM rubric writer extracts consensus and identifies student weaknesses. These findings become explicit criteria for evaluating objects, attributes, spatial relationships, and hallucinations. The paper reports improved caption quality, information density, and retention of pretrained capabilities, alongside advantages over baselines, competitive performance against larger models, and improved downstream vision-language pretraining.

Critical strengths
  1. The paper is clearly written and easy to follow.
  2. The experiments cover a broad range of model sizes and provide a comprehensive evaluation.
Critical weaknesses
  1. Majority agreement among teacher models does not guarantee factual correctness. The teachers may share biases and collectively overlook challenging visual details. The paper lacks a human audit of the generated rubrics, including an assessment of their factual accuracy.
  2. The paper offers limited novelty, with its contribution primarily amounting to an engineering refinement. Using rubrics as RL rewards has already been explored in prior work; the main distinguishing contribution here is the incorporation of teacher consensus.
Questions for authors

None

4 comments

We thank the reviewer for the thoughtful feedback! We are glad the paper is considered clearly written and easy to follow, and we appreciate the recognition of our hard work on providing comprehensive empirical evaluation. Our study assesses caption quality across six different evaluation dimensions, including a human annotation study. We address your questions below.

  • On Rubric Quality.
    • We thank the reviewer for this important question. First, we note that dense captioning is an open-ended and subjective task, unlike mathematics and coding, where factual correctness can be guaranteed and used to derive deterministic verifiers from ground truth. Even rubrics written by human experts (which we explored extensively internally) require many iterations on annotation guidelines, quality reviews, and crowd training. They can still be imperfect and biased; most importantly, they are difficult to scale. We discuss these points in our introduction.
    • Second, we validate our captions with human annotators: our human study (Appendix K) shows that RubiCap receives the highest Elo rating among all sources and is chosen by annotators in 70% of the comparisons in which it appears. We further provide direct support for the factual accuracy of our captions with two additional experiments. In our blind-ranking experiment (Section 3.2), GPT-5.4, GPT-5, GPT-4.1, and Sonnet 5 serve as independent rankers, evaluating the generated captions on accuracy, completeness, clarity, and hallucination. Our captions achieve the lowest hallucination rate and the highest accuracy, outperforming even the larger 72B model. In our CaptionQA experiment (Section 3.4), RubiCap-trained models more accurately answer multiple-choice questions derived from factual content, such as object attributes, object counting, and information in documents. Our RubiCap-7B outperforms the 32B model under tighter word budgets, delivering 32B-level caption quality at 7B inference cost. If the policy were exploiting unpenalized hallucinations, we would expect its hallucination rate to be higher, not the lowest among all models.
    • Third, during our experiments, we carefully reviewed the generated rubrics with our annotators and iteratively refined our synthesis prompt. We acknowledge that gold-standard rubrics are hard to obtain, so our work instead leverages the expertise of multiple models to derive a reliable approximation of high-quality rubrics. In our committee-size study (Appendix J, Table 7), we show that larger committees provide more diverse teacher perspectives, which in turn yield higher-quality captions in win rate and better rubrics for RL training. Notably, adding smaller and weaker models (Gemma-3 and Qwen3-VL in our case) does not introduce noise that degrades performance; instead, these models contribute additional perspectives that help the rubric writer reach a more reliable consensus.
  • On Novelty.
    • We thank the reviewer for raising this concern. While we acknowledge that using rubrics as rewards has been explored, prior work targets objective domains, such as mathematics, medicine, and science, where ground truth makes deterministic verifiers easily accessible [1]. In contrast, dense captioning, the focus of this paper, is a fundamentally subjective task with no deterministic verifiers to provide reliable rewards, let alone ones that support effective RL training. To our best knowledge, RubiCap is the first rubric-based RL framework for open-ended dense captioning. We show that current captioning evaluation metrics, whether lexical metrics or even VLM judges, lead to suboptimal performance and are prone to reward hacking when used as rewards. By comparison, RubiCap-trained models produce higher-quality captions with greater information density and deliver stronger downstream utility. Our contribution is to automatically convert this ground-truth-free, subjective task into a list of auditable, sample-specific checks by leveraging the collective expertise of teacher models.
    • [1] Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

Dear Reviewer MSKJ,

Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.

Reviewer_MSKJOpenReview ↗

Response

Thank you for the detailed responses. These have addressed part of my concerns, and I will maintain rating.

Which of your concerns can we help clarify in the final few hours?

Reviewer_fSL4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RubiCap, a RL post training-based framework for dense image captioning. For each training image, five strong VLMs write captions. An LLM rubric writer compares the student’s caption with what most teachers agree upon, and turns each of the student’s failures into a binary pass/fail criterion weighted by severity. The student is then trained with GRPO, where a judge checks each rollout against these criteria. The method is evaluated with the CapArena judging protocol and outperforms supervised distillation and other RL baselines.

Critical strengths

1.The cost profile is practical. It uses the expensive model once, offline with around $0.013 per image, with training only needing a 7B text-only judge. Ablation (iv) shows that a 72B VLM judge does not significantly improve the performance, which is an interesting result.

2.With the CapArena judging protocol with GPT-4.1, RubiCap-7B wins 70.8% of comparisons against its own base model on PixMoCap.

3.The problem is clearly defined, and the proposed design is sensible. By evaluating caption quality through per-image binary checks, the paper proposes a natural fix to lexical metrics and opaque, holistic Likert judges. Also, targeting the student VLM’s own failures is a good method to make the reward discriminative within a GRPO group. 4. Ablations are very detailed, covering prompting strategies, rubric granularity, committee size and so on.

Critical weaknesses
  1. Table 7 contains rows marked “chosen”, which should be the same configuration. But their win rates against the base model are shown as 69.0, 67.6, 67.6, 68.6 and 66.4, and their averages range from 40.43 to 41.18. This requires an explanation.

  2. The method section states the framework as an adaptive curriculum, where as the student improves, the rubrics move on to its new weakness. But algorithm and Appendix E states that rubrics are generated once, before training, from the untrained base model’s caption, and aren't updated. This requires correction in wording or running of an online variant.

3.The reward can't see new mistakes. For example, if the image shows a man in a hat and a cake reading “24 CARROT CAKE”, the base caption mentioning the man and the hat but missing the text, leading to rubric only containing “mentions 24 CARROT CAKE”. If the rollout reads “A cake reading 24 CARROT CAKE, next to three dogs”, it would still score a perfect reward.

4.RubiCap receives information from very strong VLMs like GPT-5 and Gemini 2.5 Pro captions, which the baselines generally do not receive. So it must be tested with baselines receiving the same information from these strong models.

5.Section 3.5 pretraining experiment shows RubiCap captions beat GPT-4V caption set but does not show that RubiCap beats its own untrained base model as a recaptioner.

6.Number of criteria M per image, severity distribution, base-model pass rate, fraction of GRPO groups with zero reward variance and criterion-level agreement between 7B judge and humans is not reported. A rule of >= 3 of 5 is mentioned in the prompt but never checked.

  1. Novelty is moderate. The RL framework is GRPO with Rubrics-as-Rewards [34]. Paper adds teacher-consensus grounding and the failure-focused targeting. Related works should be expanded upon, with works like RLAIF-V, “Painting with Words” etc.
Questions for authors

1.RubiCap-7B-PixMoCap vs expert annotations is 67.4 in Table 2 and Figure 2 but 66.0 in Figure 5. Which is correct?

2.The five rows marked “chosen” in Table 7 report different win rates. Are these independent training runs?

3 comments

We appreciate your positive review! Thank you for highlighting our cost profile---a one-time offline API spend for synthesis. We also thank you for acknowledging that our setup is well designed and that our framework is an effective alternative to prior captioning metrics as verifiers. Finally, thank you for recognizing that our experiments are extensive and detailed, as other reviewers noted.

  • On Chosen Rows in Table 7.
    • The highlighted rows in Table 7 come from independent runs conducted during our ablation studies, each initialized with a different random seed. After selecting the best configuration from these ablations, we conducted a separate full-scale run, whose results are reported in the main paper.
  • On Online and Offline Rubric Synthesis.
    • Thank you for this important question! We clarify that our paper focuses on offline rubric synthesis. As discussed in Appendix E, our teacher committee is queried only once, which incurs a low, one-time API cost. In contrast, online rubric generation re-synthesizes rubrics against the current policy, adding substantial latency to every training step. Online rubrics are also difficult to audit during training, which is directly relevant to the reviewer's concern about rubric correctness. Note that our paper does not claim to be a curriculum. We state only that the student is incentivized to address visual details its initial policy overlooked, progressively closing the quality gap with the teacher consensus.
    • Regarding newly introduced mistakes, we find empirically that they are rare. In our blind-ranking experiment, RubiCap-trained models achieve the lowest hallucination rate among all compared models, and they answer factual CaptionQA questions more accurately (Section 3.4). If the policy were exploiting unpenalized hallucinations, we would expect its hallucination rate to be higher, not the lowest among all models.
  • On Baselines.
    • Our work includes two sets of experiments that already grant the baselines access to teacher-level expertise. First, in Section 3.3, we bring our generated rubrics directly into supervised fine-tuning: the distillation targets are rewritten so that all criteria are satisfied, giving the baselines the identical teacher expertise used by RubiCap.
    • Second, in Sections 3.1 and 3.2, we perform supervised fine-tuning on high-quality captioning sources, including human-expert annotations, GPT-enriched captions, and captions generated by a 72B model. The first two are widely regarded as reference standards in prior works, and the third was the strongest open-weight model we considered at the time of our experiments. These supervision sources carry information comparable to, or even richer than, what RubiCap receives. Eventually, RubiCap outperforms both sets of baselines.
  • On Base Model as Re-captioner.
    • We chose the GPT-4V caption set used by LLaVA-NeXT since it is a strong and widely adopted reference for VLM pretraining, to the best of our knowledge, its captions are already of higher quality than those produced by our base model.
  • On Rubric Statistics.
    • We have added these into our current draft. In our experiments, we first remove criteria that lack teacher consensus. Such criteria are rare, accounting for only 1.1--2.3% of generated criteria (Table 1). We then remove criteria that the student model already meets, which account for only 0.09--0.13% of the remaining criteria (Table 2). Tables 3 and 4 report the number of criteria per image and the severity distribution for the final rubric set.

Table 1. Teacher consensus

Student# of CriteriaTeacher Consensus
Qwen2-VL-2B186,908184,874 (98.9%)
Qwen2.5-VL-3B188,122185,825 (98.8%)
Qwen2.5-VL-7B176,508172,432 (97.7%)

Table 2. Base-model pass rate

StudentOverall
Qwen2-VL-2B206 / 184,874 (0.11%)
Qwen2.5-VL-3B169 / 185,825 (0.09%)
Qwen2.5-VL-7B231 / 172,432 (0.13%)

Table 3. Number of criteria per image (mean ± std)

StudentTrainTest
Qwen2-VL-2B3.74 ± 0.723.77 ± 0.70
Qwen2.5-VL-3B3.76 ± 0.753.77 ± 0.73
Qwen2.5-VL-7B3.48 ± 0.773.52 ± 0.76

Table 4. Severity distribution

StudentCriticalImportantMinor
Qwen2-VL-2B35.2%46.7%18.2%
Qwen2.5-VL-3B35.9%46.3%17.8%
Qwen2.5-VL-7B29.0%48.5%22.5%
  • On Novelty.
    • We answer this in our response to Reviewer MSKJ.
  • On RubiCap-7B-PixMoCap.
    • Thank you for reading so carefully and catching this. In Figure 5, the win rate of RubiCap-7B-PixMoCap vs. expert annotations should be 67.4% rather than 66%. We have corrected this.

Dear Reviewer fSL4,

Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.

Thank you for the rebuttal and clarifications, which addressed several of my questions. I will keep my weak accept rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their careful reading and constructive feedback. We are grateful that reviewers found our paper clearly written and easy to follow [MSKJ, eGAR], our rubric design transparent and interpretable [eGAR], and our framework an effective alternative to existing captioning metrics as verifiers, with only a one-time offline synthesis cost [fSL4]. All reviewers also noted that our experiments are extensive and comprehensive [fSL4, MSKJ, eGAR], spanning six evaluation dimensions, including a human study.

Below, we summarize our responses to the shared concerns.

  • Offline Rubric Synthesis. Our paper focuses on offline synthesis: we query the teacher committee only once, before training. Online re-synthesis adds latency to every training step and makes rubrics hard to audit. In contrast, offline rubrics can be reviewed and verified before training begins.
  • Novelty. Prior work on rubrics as rewards targets objective domains such as math, medicine, and science, where ground truth makes verifiers easy to build. To our knowledge, RubiCap is the first rubric-based RL framework for open-ended dense captioning. It turns a subjective, ground-truth-free task into auditable, sample-specific checks. Existing metrics, whether lexical metrics or VLM judges, underperform as rewards and are prone to reward hacking.
  • Rubric Quality. Human-written rubrics are costly, hard to scale, and still subjective. Instead, we use a teacher committee to approximate high-quality rubrics, and larger committees yield better results (Appendix J). Our results support the factual correctness of our rubrics: (i) annotators prefer RubiCap captions 70% of the time, giving them the highest Elo rating (Appendix K); (ii) RubiCap achieves the lowest hallucination rate in our blind-ranking experiment (Sec. 3.2); and (iii) RubiCap-trained models achieve higher factual accuracy on CaptionQA (Sec. 3.4).

Dear ACs and SACs,

Thank you for your time during the review process. So far, we have not received any responses from the reviewers, though we would be glad to answer their questions and address their concerns. As the discussion period is about to close, we would like to say a few words to shine our work and share MY belief.


We study the verification bottleneck in dense image captioning, an open-ended and subjective task that has no deterministic verifier for reinforcement learning.

We propose RubiCap, which distills the collective expertise of multiple teacher models into image-specific rubrics, turning caption quality into explicit, multi-dimensional reward signals. We have made a substantial effort to provide a wide range of experiments, which show that:

  1. Highest win rates on CapArena. RubiCap outperforms supervised distillation, prior RL methods, GPT-4V-augmented outputs, and even human-expert captions.
  2. Top-ranked in blind evaluation. In a blind ranking evaluation, captions from our 7B model receive the highest proportion of rank-1 assignments among all models, including 72B and 32B frontier models.
  3. Dense and informative captions. On CaptionQA, our captions carry dense-and-accurate information that our 7B model is comparable to 32B frontier models.
  4. Better pretraining utility. VLMs pretrained on captions from our compact RubiCap-3B outperform those trained on captions from proprietary models.
  5. Validated by humans. To verify our rubrics, generated captions, and evaluation results, we conducted a human study, in which our captions achieve the highest Elo scores among all baselines.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.