- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MoCo Rubric for rubric-based reinforcement learning in detailed image captioning. The method first uses shared-parameter, multi-role supervised fine-tuning to establish caption generation, rubric construction, and judging capabilities. It then applies reinforcement learning only to the caption policy, while updating the parameters shared by the Generator and Judge through an exponential moving average (EMA). Together with online rubrics constructed from current policy candidates, the framework aims to coordinate evaluation with the evolving caption policy and improve caption quality.
Critical strengths
- The paper presents a reasonable method design for coordinating caption generation and evaluation. Shared multi-role initialization provides a common foundation, online rubrics adapt to current candidates, and momentum updates allow the evaluation roles to gradually track policy changes. These components form a coherent framework for adaptive training feedback.
- Evaluation across caption quality and downstream tasks supports the method’s overall effectiveness. Pairwise comparisons, blind ranking, and dimension-specific scores assess caption quality, while caption-based question answering further examines the utility of the preserved visual information. These complementary evaluations provide broad empirical support for the reported improvements.
- Detailed ablations support the contributions of the main components and their combination. Comparisons cover shared versus separate initialization, offline versus online rubrics, and frozen, fully synchronized, and EMA-updated evaluation roles. Further comparisons of Generator and Judge updates provide evidence supporting joint momentum updates.
Critical weaknesses
- The motivation is not sufficiently established. The paper motivates its approach through inconsistent interpretations when separate models perform caption generation, rubric construction, and judging. However, a strong independent Judge can also provide complementary scrutiny of the generator’s outputs. Further justification is needed to explain why differences across roles constitute a problem and why greater consistency should improve task performance. Figure 1 illustrates an error missed by an overly broad rubric, but does not sufficiently establish whether the underlying issue is cross-role inconsistency rather than inadequate rubric quality or judging capability.
- The connection between the motivation and the observed gains lacks direct mechanistic analysis. The experiments support the effectiveness of the complete method, but do not clearly establish whether the gains arise from improved consistency. Section 3.4 analyzes parameter tracking and smoothing, without directly connecting these properties to changes in rubric quality or Judge accuracy. In particular, why parameter updates optimized for caption generation benefit the other two roles requires further explanation. The relationship among the motivation, method design, and performance improvements therefore remains unclear.
Questions for authors
-
Why is cross-role inconsistency a central problem that needs to be addressed? The introduction on page 2 motivates the method through semantic inconsistency across models and illustrates the issue with the undetected counting error in Figure 1. A common reason for using a strong independent Judge is to provide complementary scrutiny from a perspective different from that of the generator. Why should such differences harm training feedback, and why should shared initialization and coordinated evolution improve it? How can the failure in Figure 1 be distinguished from insufficiently accurate or comprehensive rubrics?
-
Could an additional evaluation step for tied rollout groups yield similar training benefits?The experimental analysis reports fewer rollout groups with identical rewards under the proposed method, allowing more groups to provide relative reward signals. If reducing tied groups is a major source of improvement, could effective training signals be recovered by keeping the Generator and Judge parameters fixed and adding an evaluation step only for tied groups—for example, reconstructing the rubric or applying finer-grained judgments? What additional benefits do joint EMA updates provide beyond this approach?
-
Why is the Judge configured without direct access to the image?Section 3.3 specifies that the Judge evaluates candidate captions using only the captions and rubrics, without directly accessing the image. Normally, a Judge considers the task input when assessing whether an output satisfies the task requirements. For image captioning, the original image is essential for determining factual accuracy and completeness. Why does the Judge obtain image information only indirectly through the Generator’s rubric, rather than inspecting the image directly? What motivates this input restriction?
-
Can rubrics constructed from a subset of candidates fairly evaluate the entire group?Section 3.3 randomly selects four of the sixteen candidate captions to construct a rubric, which is then applied to all sixteen candidates. The criteria may focus on issues exposed by the selected candidates while missing important quality differences in the remaining candidates. What motivates the choice of four candidates? How does the method ensure that the resulting criteria also apply to candidates excluded from rubric construction, without introducing subset-dependent bias into the within-group reward ranking?
7 comments
Response to Reviewer iUjg — Part 2/4
Q1.3 Why should shared initialization and coordinated evolution improve it?
A1.3 Shared Multi-Role SFT gives the Policy, Generator, and Judge a common parameter initialization while teaching their distinct tasks through role-specific supervision. This provides a compatible starting point for transferring Policy updates; it does not assume that the roles subsequently make identical judgments. As the Policy changes, we construct rubrics from its current candidates and gradually transfer its parameter updates to one momentum model used by both the Generator and Judge. The two rubric roles therefore use the same lagged checkpoint within each rollout group, while EMA limits abrupt changes from direct parameter copying. The original matched component ablation supports this design: with shared initialization and online rubrics, joint EMA increases the two-benchmark mean win rate from 66.61% with frozen rubric roles to 72.14%; updating only the Generator reaches 69.07%. We further test continued adaptation from the 300-step Policy checkpoint by comparing ten additional MoCo steps with rubric roles frozen at different earlier checkpoints (Table 3).
Table 3. Continued adaptation versus frozen rubric-role parameters.
| Parameter setting for the Rubric Generator and Rubric Judge | PixMo-Cap | DenseFusion | CapArena | CompreCap | DOCCI | Average win rate (%) |
|---|---|---|---|---|---|---|
| Shared starting checkpoint | 70.28 | 61.12 | 72.33 | 60.18 | 73.80 | 67.54 |
| SFT (shared) | 69.76 | 62.40 | 74.08 | 61.36 | 74.75 | 68.47 |
| Momentum checkpoint at step 100 | 69.35 | 64.60 | 70.90 | 60.82 | 73.15 | 67.77 |
| Momentum checkpoint at step 200 | 68.75 | 65.20 | 74.08 | 59.21 | 73.75 | 68.20 |
| Momentum checkpoint at step 300 | 72.18 | 63.20 | 74.25 | 60.64 | 74.35 | 68.92 |
| MoCo Rubric | 72.38 | 65.20 | 72.91 | 62.25 | 74.15 | 69.38 |
Continued MoCo updates reach 69.38%, compared with 68.92% for the strongest frozen setting in this short continuation experiment. Together, these results indicate the usefulness of a shared starting point and continued joint adaptation.
Q1.4 How can the failure in Figure 1 be distinguished from insufficiently accurate or comprehensive rubrics?
A1.4 We agree that Figure 1 does not distinguish cross-role interpretation differences from an insufficiently specific rubric. The original criterion checks whether a caption mentions a grid of cars; it does not require the correct number of columns. The Judge’s pass decision therefore follows the written criterion, while the column-count error goes undetected. We will revise the figure discussion to present this as a criterion-specification failure. Tables 1 and 2 address a separate question by holding rubric content fixed and testing criterion application, while Table 3 examines adaptation as the Policy evolves.
Response to Reviewer iUjg — Part 3/4
Q2. Could an additional evaluation step for tied rollout groups yield similar training benefits? The experimental analysis reports fewer rollout groups with identical rewards under the proposed method, allowing more groups to provide relative reward signals. If reducing tied groups is a major source of improvement, could effective training signals be recovered by keeping the Generator and Judge parameters fixed and adding an evaluation step only for tied groups—for example, reconstructing the rubric or applying finer-grained judgments? What additional benefits do joint EMA updates provide beyond this approach?
A2. Thank you for suggesting a targeted reevaluation of tied rollout groups. We agree that when all sixteen candidates receive the same reward, their within-group GRPO advantages vanish, and a more discriminative rubric could recover a training signal. To test this alternative, we keep RubiCap’s Generator and Judge fixed and update only its Policy: whenever a group is tied, we reconstruct its rubric once using the image, two reference captions, original rubric, and the same sixteen candidates, then rescore the group; no further retry is made if it remains tied.
As shown in Table 4, this retry improves RubiCap’s average win rate from 69.93% to 70.84%. It is therefore a useful intervention, although its mean remains below MoCo Rubric’s 72.83% in this comparison.
Table 4. Rubric retry with fixed Generator and Judge parameters.
| Method | PixMo-Cap | DenseFusion | CapArena | CompreCap | DOCCI | Average win rate (%) |
|---|---|---|---|---|---|---|
| RubiCap | 70.14 | 67.80 | 72.33 | 63.57 | 75.80 | 69.93 |
| RubiCap + rubric retry | 71.10 | 69.08 | 72.16 | 64.70 | 77.17 | 70.84 |
| MoCo Rubric | 73.94 | 70.34 | 78.00 | 63.38 | 78.51 | 72.83 |
The two procedures address different scopes of feedback. A retry acts only when a group is tied, whereas joint EMA updates the Generator and Judge throughout training, including for groups that already have nonzero reward variation. Table 1 shows why the latter groups can still matter: changing only the Judge under a fixed rubric reverses the signs of nonzero advantages for approximately 20% of candidates, although this diagnostic does not determine which Judge is correct. Separately, the matched component ablation in the original paper shows that joint EMA improves the two-benchmark mean from 66.61% with frozen rubric roles to 72.14% under the same shared initialization and online-rubric setting; updating only the Generator reaches 69.07%. Together, these results show that retrying ties can help, while continued, joint rubric-role adaptation has benefits in the evaluated setting that are broader than the tested tie-only procedure.
Q3. Why is the Judge configured without direct access to the image? Section 3.3 specifies that the Judge evaluates candidate captions using only the captions and rubrics, without directly accessing the image. Normally, a Judge considers the task input when assessing whether an output satisfies the task requirements. For image captioning, the original image is essential for determining factual accuracy and completeness. Why does the Judge obtain image information only indirectly through the Generator’s rubric, rather than inspecting the image directly? What motivates this input restriction?
A3. Thank you for emphasizing the importance of the image in assessing caption factuality and completeness. The image is used in our reward pipeline: the Policy sees it when generating captions, and the Generator examines it when constructing image-specific criteria from the image, reference captions, and current Policy candidates. We give the Judge the narrower task of checking each caption against those written criteria, so that rewards reflect one explicit rubric shared by the rollout group rather than an additional, implicit visual assessment by the Judge. This division follows RubiCap’s text-only judging setup (Huang et al., 2026, Appendix G) and the self-contained evaluation principle of Rubrics as Rewards (Gunjal et al., 2026, Sec. 3.1). We agree that this design cannot detect a visual error omitted from the rubric; we will make this limitation explicit when describing the Judge’s input.
Response to Reviewer iUjg — Part 1/4
Q1. Why is cross-role inconsistency a central problem that needs to be addressed? The introduction on page 2 motivates the method through semantic inconsistency across models and illustrates the issue with the undetected counting error in Figure 1. A common reason for using a strong independent Judge is to provide complementary scrutiny from a perspective different from that of the generator. Why should such differences harm training feedback, and why should shared initialization and coordinated evolution improve it? How can the failure in Figure 1 be distinguished from insufficiently accurate or comprehensive rubrics?
A1. Thank you for highlighting our motivation and the interpretation of Figure 1. We agree that an independent Judge can offer complementary scrutiny, and differences between models are not inherently harmful. Our term “semantic inconsistency” was too broad: it could imply alignment of internal model representations, whereas our concern is how rubric criteria are specified and applied to produce useful rewards as the Policy evolves. We therefore frame the motivation as rubric-feedback coordination across criterion specification, criterion application, and rubric-role adaptation. Figure 1 illustrates a criterion-specification failure; Tables 1 and 2 examine Judge-dependent feedback and its effect on learning, while Table 3 examines continued adaptation.
Q1.1 Why is cross-role inconsistency a central problem that needs to be addressed?
A1.1 The concern is not that different models must agree on every caption. Rather, the Judge’s application of a rubric determines the relative rewards used to update the Policy. Even when the Generator specifies the same criterion and pass/fail rule, different Judges may assign different binary decisions to a caption. This can change the caption’s reward relative to other rollouts and, in turn, which outputs GRPO reinforces. To measure how sensitive these signals are to Judge choice, we hold both candidate captions and rubrics fixed and replace only the Judge. The captions come from the Policy after Shared Multi-Role SFT and 300 RL steps, and the rubrics are produced by the shared SFT Generator. Table 1 reports differences relative to the shared SFT Judge.
Table 1. Judge-dependent changes in training feedback.
Replacing the Judge changes 8.61%–9.10% of criterion-level decisions and reverses the signs of nonzero GRPO advantages for 19.55%–20.95% of candidates. Thus, Judge choice changes relative training signals even when rubric content is identical. This comparison measures feedback sensitivity.
Q1.2 Why should differences in criterion interpretation and evaluation preferences between the Generator and Judge harm training feedback?
A1.2 Judge-dependent decisions can harm training if they rank lower-quality captions above better ones, because GRPO reinforces captions according to their relative rewards. We therefore test the training consequence rather than assuming that disagreement is harmful. In Table 2, all runs continue the same 300-step Policy checkpoint for 50 steps with the same training images and optimization settings. Each Generator’s rubrics are generated offline and reused unchanged across Judges; only the Policy is optimized.
Table 2. Policy performance under different Generator–Judge configurations.
With the shared SFT Generator’s rubrics fixed, the shared SFT Judge yields an average Policy win rate of 68.98%, compared with 67.83% for GPT-5.5 and 67.59% for Gemini 3.1 Pro. This shows that criterion application affects subsequent Policy performance without changing rubric content.