- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes RubiCap, a reinforcement learning framework for dense image captioning. It uses captions from a committee of visual models to identify the student's deficiencies and construct image-specific rubrics. A text-only judge checks generated captions against these criteria, and their weighted scores provide the GRPO reward. Experiments evaluate caption quality, word efficiency, retention of general capabilities, and the usefulness of generated captions for VLM pretraining.
Critical strengths
- Clear formulation: The paper is easy to follow, and the explicit criteria make the reward more interpretable than an overall quality score.
- Broad evaluation: The experiments cover multiple model sizes, component ablations, rubric-augmented SFT, human preferences, and downstream pretraining utility.
Critical weaknesses
- The reward has an important coverage gap: The rubrics are fixed offline and cover only the student's initial deficiencies (Section 2.2; Appendix E). New claims outside the checklist, or errors in previously correct content, may leave criterion scores unchanged. The text-only judge cannot independently verify these claims against the image. This gap needs direct evaluation.
- Teacher consensus is not sufficient factual validation: Shared teacher errors can enter the rubrics, while correct details outside the consensus may be omitted. The committee-size ablation and overall preference results do not directly establish rubric accuracy or coverage. An independent audit is needed to support the reliability of the reward.
Questions for authors
- If a caption satisfies the rubrics but adds a false claim outside them, does its reward decrease? Please test such cases and explain how newly introduced errors are constrained.
- Could the authors independently evaluate the factual accuracy and coverage of the synthesized rubrics, including shared teacher errors and content the initial student already described correctly?
2 comments
Dear Reviewer eGAR,
Thank you for your time and effort in reviewing our paper. As the author–reviewer discussion period is coming to a close, we would be happy to answer any remaining questions you may have. We appreciate your engagement.
We are grateful for your feedback and for recognizing that our paper is easy to follow and that our synthetic rubric design is transparent and interpretable. We also thank you for highlighting our wide range of analyses and empirical results, a strength other reviewers noted as well. Our study assesses caption quality across six different evaluation dimensions, including a human annotation study. We address your questions below.