acceptodds
Under review as a conference paper at ICLR 2027

DQ-RL: Fine-Grained Rewards from Diverse Queries for Dense Image Captioning

Abstract

Image captioning is a fundamental task in vision-language understanding and a core capability of large vision-language models. Existing efforts to improve this capability mainly rely on supervised fine-tuning of base models, often on domain-specific data, while recent work has begun to explore reinforcement learning (RL) for better generalization. However, applying RL to image captioning remains challenging: as an open-ended generation task, captioning lacks verifiable rewards, making reward design under the Reinforcement Learning with Verifiable Rewards (RLVR) paradigm particularly difficult. To address this challenge, we propose **Diverse-Query Reinforcement Learning** (**DQ-RL**). We design completeness queries (**CQs**) and faithfulness queries (**FQs**), perform unified sampling over the candidate query pool, evaluate sub-capabilities of generated captions, and use the resulting quantitative scores as rewards for optimization. Following this paradigm, we construct a k training dataset named **DQ-CAP-40k**. Extensive experiments show that DQ-RL, a 4B model, achieves an average improvement of \(1.4%\) over baseline models under the PRISM evaluation framework, and matches Qwen3.5-27B model at \(63.1%\). We also build **DQ-Bench**, an evaluation benchmark tailored for image captioning. On the three core metrics of DQ-Bench, namely Correct, Omit and Wrong, DQ-RL outperforms the baseline by , , and , respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, image caption

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mzBrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies reinforcement learning for detailed whole-image captioning. Completeness queries (CQs) are constructed offline and answered from the caption alone to assess information coverage. The current caption is also decomposed into faithfulness queries (FQs), which are checked against the image online. The two query types are sampled under a fixed budget. After filtering out full-score FQs, the remaining scores are aggregated into a reward used to train Qwen3.5-4B with GRPO. The paper introduces the DQ-CAP-40k training set and DQ-Bench evaluation set, and reports results on eight PRISM question-answering evaluations, field-level diagnostics, and component ablations.

Critical strengths
  1. The supervision addresses both omissions and factual errors. Hard CQs target content that the base model omits or misstates, while online FQs check factual claims in the current caption. These signals are complementary, and the fixed query budget helps control the cost of each evaluation. The design addresses a meaningful tension between coverage and accuracy in detailed captioning.
  2. Two complementary evaluations show improvements. Relative to Qwen3.5-4B, the average PRISM score increases from 61.7 to 63.1. On DQ-Bench, Correct rises from 85.5% to 94.1%, Omit falls from 9.4% to 2.1%, and Wrong falls from 5.1% to 3.8%. The final diagnostic judge differs from the training verifier. These results provide empirical support for better coverage and error control.
Critical weaknesses
  1. The reward can penalize a factual correction. Eqs. (7) and (8) remove full-score FQs from both the numerator and denominator. With the default α=0.75\alpha=0.75, consider six CQs, of which two are answered correctly, and seven FQs, of which six receive full scores and one receives 0.750.75. The reward is then R=(2+0.75×0.75)/7≈0.366R=(2+0.75\times0.75)/7\approx0.366. Correcting only the last FQ to obtain a full score, while holding the CQ outcomes and decomposition fixed, reduces the reward to R=2/6≈0.333R=2/6\approx0.333. This is a counterexample to the stated formula, not an observed training failure. The aggregate improvement in Table 4 does not rule out this defect in the objective. The implementation and its practical impact need to be checked.
  2. The contribution of the new design choices is not sufficiently isolated. CCCaption already combines completeness-oriented question answering, factual verification, and dynamic sampling. Table 1 uses results from that paper without matching the base model or training conditions. The ablations also do not separately test hard CQs and decomposition granularity, making their individual contributions difficult to assess.
  3. The evidence from DQ-Bench also needs further validation. Each field receives a single label, but the rules for fields containing multiple facts are not fully specified. Wrong is a field-level proportion, not a per-fact error rate. Table 5 examines changes in the base model's scores across judges rather than the stability of method rankings. Using Gemini consistently in the main evaluation is a reasonable control, but agreement between the field labels and human judgments has not been established. CAPability's human annotations of visual elements could provide an additional check
Questions for authors
  1. Reward implementation: Does Eq. (8) match the implementation? How often does the counterexample above arise in actual rollouts? How is the reward defined in the FQ-only setting when every FQ receives a full score? How are the null scores allowed in the appendix handled?
  2. Supervision reliability: Have you checked whether the decomposer omits erroneous details? How well do FQ verification scores agree with human judgments? Do the four rollouts for the same image share CQs to reduce the effect of query difficulty on within-group rankings?
  3. Controlled comparison: Is there a dual-reward control with the same base model, query pool, and budget? Can the available results distinguish the contributions of hard CQs, decomposition granularity, and random mixing?
  4. Evaluation validity: How is a field labeled when it contains correct facts, omissions, and errors together? Are human checks or cross-judge method rankings available? Was the configuration in Table 6 selected on data separate from the final test set?
2 comments
  1. Thank you for asking about the reward implementation details. We provide a detailed response regarding whether Eq. (8) matches the implementation, the empirical frequency of the non-monotonic counterexample, and the all-full-score case in the FQ-only setting in our response to Reviewer exDs, Comment 1. Briefly, the boundary-case description following Eq. (8) omitted the all-correct fallback used in the implementation. When all FQs receive full scores, full-score exclusion is disabled for that rollout, and all FQs are retained in both the numerator and denominator. Consequently, an all-correct rollout in the FQ-only setting receives reward α\alpha, and no 0/00/0 case occurs. We also quantify the frequency of the non-monotonic counterexample in actual rollouts and will correct the description and add the implementation details and statistics in the revision. A null score allowed by the appendix prompt indicates that the corresponding FQ cannot be reliably judged from the image; it is not treated as a numerical faithfulness score of zero. We therefore neither convert null to zero nor include it in reward aggregation. A rollout containing a null score or another invalid reward output is marked as invalid and excluded from subsequent training updates. Our training statistics show a valid reward-data rate of 99.68%, with only 0.32% invalid data. The invalid rate covers null scores together with other reward-computation errors. Therefore, 0.32% is the overall invalid reward rate and an upper bound on the incidence of null scores. We will explicitly document the handling of null scores and the resulting valid-data rate in the revision.

  2. For FQ verification, we selected 100100 FQs and manually rescored them using the same five-level criteria as the automated verifier, {0,0.25,0.50,0.75,1.00}\{0, 0.25, 0.50, 0.75, 1.00\}. The mean automatic and human scores are 0.7450 and 0.8175, respectively. The mean signed bias is −0.0725-0.0725, indicating that the automated verifier is slightly more conservative. The mean absolute error is only 0.0875, which is smaller than the score-bin interval of 0.250.25. Moreover, the exact score agreement is 88.00%, and Cohen's κ\kappa is 0.7477. These results demonstrate substantial agreement between the automated FQ verifier and human judgments, supporting its use as an automated reward judge during training. We will add this human-verification protocol and its results to the revision.

  3. Regarding the contribution of Hard CQ, we compared the model performance with and without Hard CQ, and the results are shown in the table. After introducing Hard CQ, Correct increases from 89.7% to 92.2%, an improvement of 2.5 percentage points, while Wrong decreases from 8.8% to 5.9%, and Omit remains basically unchanged. It can be seen that Hard CQ can bring an improvement in Correct. For the contribution of random mixing, we supplemented an ablation experiment with a fixed CQ/FQ ratio. The experimental results show that, under the same query budget and CQ/FQ configuration, random mixing compared with fixed-quota sampling raises Correct from 93.6% to 94.1% and reduces Wrong from 4.7% to 3.8%. This shows that random mixing can more fully utilize the supervision signals of the two types of queries, further improving the overall caption quality and factual reliability. The framework also allows adjusting λ\lambda according to application needs: increasing λ\lambda increases the sampling frequency of CQ, placing greater emphasis on completeness, while decreasing λ\lambda increases the sampling frequency of FQ, placing greater emphasis on factual accuracy. The parameter α\alpha independently adjusts the weight of FQ signals in the aggregated reward. Therefore, we emphasize that DQ-RL focuses on unified sampling . By contrast, CCCaption follows an idea similar to DAPO, performing online data filtering after each epoch, DQ-RL separately designs its reward and excludes full-score FQs to increase the discriminability of the resulting reward signals. Since CCCaption has not released the relevant code or weights, we cannot make a direct and fair comparison. | Settings | Correct↑ | Omit↓ | Wrong↓ | |---|---|---|---| | without Hard CQ | 89.7% | 1.5% | 8.8% | | Hard CQ | 92.2% | 1.9% | 5.9% | | without random | 93.6% | 1.7% | 4.7% | | random mixing | 94.1% | 2.1% | 3.8% |

  1. For human verification of the automated evaluation, we report the detailed results of a blinded study in our response to Reviewer oNKa, Comment 2. We randomly sampled 100100 images from the evaluation set and independently annotated all valid attribute-level records. The Cohen's κ\kappa values between the human reference labels and Gemini-3.1-Pro, Seed-2.0-Lite, and Qwen3.5-9B are 0.7730, 0.7195, and 0.1017, respectively. Gemini-3.1-Pro achieves the highest observed agreement and Cohen's κ\kappa, indicating substantial agreement between its field-level evaluations and human judgments and supporting its use as our primary evaluator.

    As a complementary validation independent of DQ-Bench, we additionally evaluate the models using the human-annotated visual elements provided by CAPability. The Qwen3.5-4B baseline obtains an average F1 score of 72.7, whereas DQ-RL achieves an average F1 score of 74.4, corresponding to an improvement of 1.7 points. This result shows that the improvement of DQ-RL over the same base model is not limited to the automated DQ-Bench evaluation and is also observed under an external evaluation based on human-annotated visual elements. We will add the human-verification protocol, agreement results, and complementary CAPability evaluation to the revision.

Reviewer_exDsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes DQ-RL for dense image captioning. It combines offline completeness queries with faithfulness queries extracted online from each generated caption. A sampler allocates a fixed query budget between the two types, and their scores form the GRPO reward. The paper also constructs a training dataset and DQ-Bench, and evaluates caption quality through field-level diagnostics and caption-mediated question answering.

Critical strengths
  1. Clear and practical formulation: Offline coverage queries and online verification of generated claims address complementary captioning errors. The fixed query budget also makes verification cost explicit.
  2. Useful diagnostic experiments: The paper reports public PRISM results alongside DQ-Bench and studies query routing, score scaling, full-score exclusion, and query budget.
Critical weaknesses
  1. The reward can decrease when factual correctness improves: In Equations (7)–(8), a fully correct FQ is removed from both the score sum and the denominator. When completeness scores are low, correcting a partially correct FQ can therefore lower the reward. The exclusion ablation does not resolve this inconsistency. The authors should clarify whether the implementation follows this definition.
  2. The empirical validation is incomplete: The contribution of unified sampling needs a matched comparison with separate dual-reward aggregation using the same queries, verifiers, and budget. DQ-Bench is also sensitive to the evaluator (Table 5), but the check covers only the base model; it does not establish whether the reported model rankings are stable across judges or agree with humans.
Questions for authors
  1. How does the reward avoid decreasing when an FQ is corrected to full correctness and then excluded? Please clarify the implemented formula and the all-correct-FQ case.
  2. Could the authors compare unified sampling with separate reward aggregation under matched conditions, and validate the main DQ-Bench comparisons using independent judges or human annotation?
1 comment
  1. Thank you for identifying this issue. After rechecking the implementation, we found that Eqs. (7)–(8) describe the implemented filtering operation, but the subsequent boundary-case description is incorrect. The implementation uses an all-correct fallback: if the initial computation gives Nfvalid=0N_f^{\mathrm{valid}}=0, meaning that all FQs receive a score of 1.01.0, we disable full-score exclusion for that rollout by resetting I(sif)=1\mathbb{I}(s_i^f)=1 for all ii and Nfvalid=NfN_f^{\mathrm{valid}}=N_f. All FQs are therefore retained in both the numerator and denominator. In the FQ-only setting, an all-correct rollout receives reward α\alpha, so no 0/00/0 case occurs. We will correct this description in the revision.

    We agree that, when other non-perfect FQs remain, increasing one FQ to 1.01.0 and subsequently excluding it does not guarantee that the mixed reward is strictly monotonic with respect to that FQ. The motivation for exclusion is that factual errors are sparse within a rollout. Retaining many full-score FQs dilutes the reward differences contributed by the few erroneous claims and makes them more susceptible to judge-score variability. Exclusion focuses the reward on unresolved factual errors, while the all-correct fallback appropriately rewards a fully correct FQ set.

    We specifically examined the early stage of training, where caption completeness is relatively low and this edge case is theoretically most likely to occur. In this early stage, strict reversals occur for about 2.872.87\\% of the rollout pairs with identical CQ scores, representing only 0.0960.096\\% of all within-group rollout pairs in the same stage. In the middle-to-late stage of training, this conditional rate further decreases to about 1.601.60\\%, or 0.0570.057\\% of all within-group pairs. Thus, the non-monotonic case exists mathematically and is more concentrated at the beginning of training, but it affects only a very small fraction of actual rollout comparisons even in this less favorable regime. We will correct the fallback description and clarify that the ablation demonstrates the overall empirical effect of exclusion rather than strict mathematical monotonicity.

  2. Regarding the contribution of unified sampling, we add an ablation with a fixed CQ/FQ ratio. Under the same query budget and the same CQ/FQ configuration, unified sampling improves Correct from 93.693.6\\% to 94.194.1\\% and reduces Wrong from 4.74.7\\% to 3.83.8\\% compared with fixed-quota sampling. This indicates that unified sampling exploits the supervision signals of both query types more fully, further improving overall caption quality and factual reliability.

    SettingsCorrect↑Omit↓Wrong↓
    unified sampling94.1%2.1%3.8%
    fixed CQ/FQ allocation93.6%1.7%4.7%
  3. Regarding the human verification of automatic evaluation, we have reported the blinded human verification results in detail in our response to Reviewer oNKa's comment 2 . We randomly sampled 100100 images from the evaluation set and independently annotated all valid attribute-level records therein. The Cohen's κ\kappa values between the human reference labels and Gemini-3.1-Pro, Seed-2.0-Lite, and Qwen3.5-9B are 0.7730, 0.7195, and 0.1017, respectively. Gemini-3.1-Pro achieves the highest observed agreement and Cohen's κ\kappa, indicating substantial agreement between its field-level evaluation and human judgments, and supporting our use of it as the primary evaluator.

Reviewer_oNKaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper improves dense image captioning in small vision-language models through fine-grained RL(reinforcement learning) rewards. It combines offline completeness queries targeting visual coverage and model weaknesses with online faithfulness queries obtained by decomposing captions into semantic statements. Unified sampling integrates these complementary signals. the authors introduce DQ-CAP-40k, containing 40000 training images with associated completeness queries, and DQ-Bench, comprising 2,000 images across four categories Experiments on prism and DQ-bench show improvements over baselines and competitive performance against larger models

Critical strengths
  1. The paper addresses an important problem in image captioning, where high-quality descriptions require both comprehensive coverage of visual content and faithfulness to the image.

  2. The paper is generally well written and clearly organized. The motivation and overall framework are presented coherently

Critical weaknesses
  1. The approach incurs additional computational cost and latency, potentially reducing overall training throughput. These overheads are important practical considerations for reinforcement learning.

  2. The reliability of the automated evaluation has not been sufficiently validated. In Table 5, the same base model receives Correct scores ranging from 85.6% to 96.4% depending on the evaluator, indicating substantial sensitivity to evaluator choice. I recommend incorporating blinded human evaluation, measuring agreement between automated and human judgments, and assessing the stability of method rankings across different evaluators.

Questions for authors

None

2 comments
  1. Training efficiency. Thank you for raising the practical issue of training efficiency. Under the same sampling budget, the policy models of both configurations are deployed on 1616 GPUs. The CQ-only baseline requires approximately 29.729.7 hours per epoch, whereas DQ-RL requires approximately 31.331.3 hours per epoch, an increase of approximately 1.61.6 hours (5.45.4\\%). To support online FQ generation and verification, DQ-RL uses two additional GPUs on the reward side and deploys the CQ and FQ reward services in parallel. Thus, at the cost of two additional reward-side GPUs, DQ-RL limits the increase in end-to-end training time to approximately 5.45.4\\%. We will add the per-epoch training time, hardware configuration, and parallel deployment details to clarify the trade-off between training speed and computational resources.

  2. Reliability of automated evaluation. Thank you for raising the concern about the reliability of the automated evaluation. We additionally conducted a blinded human verification study. We randomly sampled 100100 images from the evaluation set, and a human annotator independently labeled all valid attribute-level records using the same Correct/Wrong/Omit criteria as the automated evaluators. During annotation, the annotator was not shown either the automated labels or the identities of the evaluators. All three evaluators were compared with the human reference labels using the same images and attribute records.

    We measure agreement using observed agreement pop_o and unweighted Cohen's κ\kappa. Let MijM_{ij} denote the number of attribute records assigned to class ii by an automated evaluator and class jj by the human annotator, and let N=∑i,jMijN=\sum_{i,j}M_{ij} be the total number of valid pairs. We compute

    p_e=\sum_{i=1}^{3} \left(\frac{\sum_j M_{ij}}{N}\right) \left(\frac{\sum_j M_{ji}}{N}\right),\qquad \kappa=\frac{p_o-p_e}{1-p_e}.$$ | Evaluator | Agreement $p_o$ | Cohen's $\kappa$ | | :--- | :---: | :---: | | **Gemini-3.1-Pro** | **94.01%** | **0.7730** | | Seed-2.0-Lite | 93.66% | 0.7195 | | Qwen3.5-9B | 83.00% | 0.1017 | Gemini-3.1-Pro and Seed-2.0-Lite both exhibit substantial agreement with human judgments, with Gemini-3.1-Pro achieving the highest observed agreement and Cohen's $\kappa$. This supports our use of Gemini-3.1-Pro as the primary evaluator. In contrast, although Qwen3.5-9B achieves an observed agreement of $83.00\\%$, its low $\kappa$ indicates that much of this agreement is attributable to the imbalanced label distribution and should not be interpreted as reliable agreement across all categories. We will add the blinded human-verification protocol and these results to the revision.
Reviewer_oNKaOpenReview ↗

Response

Thank you for the detailed responses and additional experiments. These have addressed most of my concerns, and I will maintain my positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.