- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds on GROVE, a grounded video captioning model that writes a caption and predicts one box track per noun phrase. It adds two stages:
- Stage 1, a Multi-Instance Adapter. Each phrase query is expanded into K = 3 slots (query plus a learned offset). The slot tracks are trained with trajectory-level Hungarian matching and an orthogonality loss.
- Stage 2, self-alignment with RL. A frozen copy of the Stage-1 model scores sampled captions using an instance-presence and temporal-consistency reward. The captioner is trained with self-critical REINFORCE plus a KL anchor, without ground-truth captions or boxes.
Critical strengths
S1. The Multi-Instance Adapter gives a large, well-ablated grounding gain at low cost. On iGround validation, slots raise AP50 from 43.5 (the authors’ own K = 1 adapter) to 52.7. Trajectory-level matching, orthogonality, and slot offsets are each ablated (Table 4). Gains also hold on VidSTG and ActivityNet-Entities with dataset-specific fine-tuning (Table 3). This is the paper’s main substantive contribution.
S2. Both stages are ablated component by component, including reward terms, advantage clipping, and KL strength (Tables 4-5). That makes the contribution of each part assessable.
Critical weaknesses
W1. The “closed loop / grounding as verification” framing goes beyond what is built, and its closest precedents are not discussed.
- At inference, grounding remains a terminal prediction. C1 states that localization “feeds back to verify and refine generation instead of remaining a terminal prediction” (l.109–110), and the introduction faults GROVE because its grounding evidence “is still used only as a terminal prediction” (l.052). Yet at inference “this replay path is not used”: the policy generates the caption and its [DET] states go to the frozen grounding heads (Sec. 3.4, l.273). The feedback exists only as a training reward.
- Feeding grounding back into caption generation has direct precedents. Ma et al. (arXiv:1906.00283) used a cyclical decode–localize–reconstruct regimen that forces the network to ground individually decoded words, for image and video captioning. The submission cites it only within lists of prior work, without discussing its loop. Woodpecker (arXiv:2310.16045) verifies generated text with an open-set detector, which drives its gains on object existence and count, and corrects the text in a training-free, post-hoc manner; it is not cited. The delta (video, dense multi-slot tubes, and an RL reward from the model’s own frozen head) should be stated.
- Why consequential: the title contribution is a verification loop, but what is delivered is a training-time reward, and its relation to prior caption–grounding cycles and detector-based verification is not established.
W2. The evidence that the closed loop itself improves grounding and captions is limited.
- The gain is small. Stage 2 adds +1.2 AP50 and +3.3 CIDEr on validation (52.7 → 53.9; 93.8 → 97.1), and +1.6 / +4.4 on test (Table 1). No seeds or variance are reported.
- The gain depends on the KL anchor. Without it, Stage 2 falls below the Stage-1 model on every reported metric (Table 4): AP50 52.7 → 51.5, Recall 28.4 → 27.7, METEOR 19.9 → 18.7, CIDEr 93.8 → 91.4. With the anchor at λ_KL = 0.1–1.0, AP50 ranges from 52.9 to 53.9, i.e., +0.2 to +1.2 over Stage 1 (Table 5).
- Both reward terms are computed only from the verifier’s own predictions. The temporal term (Eq. 6) measures frame-to-frame stability of the predicted boxes and reaches its maximum for any box that stays fixed and visible. The presence term (Eq. 5) sums objectness over slots. The visual-reward variants (presence-only, temporal-only, single-slot, full) differ by at most 1.0 AP50 (52.9–53.9, Table 4). No statistics show how captions change under the reward, such as phrases per caption, caption length, or plural share.
- The hallucination metric conflates two things. It counts a phrase as hallucinated when it matches no annotated entity (Sec. 4.1), yet iGround annotators were asked to focus only on the objects that humans interact with rather than describing densely every object in the scene. Correct mentions of unannotated objects can therefore count as hallucinations.
- Why consequential: the paper’s central claim is that grounding feedback improves generation. The measured effect is small, comes from single runs, and depends on the KL anchor.
Questions for authors
Experiments that could help clarify more:
- Before and after Stage 2, please report noun phrases per caption, caption length, the share of plural phrases, and whether grounded entities are static or moving, to show the reward is not being exploited.
- Please report instance-level coverage (the fraction of ground-truth trajectories matched), and how often iGround phrases have more than one ground-truth trajectory, and of what kinds. iGround’s guidelines annotate many small objects, e.g. mushrooms in a pan, with a single bounding box labelled ‘mushrooms’, and use multiple boxes for each human hand when both are used, or each tool/container/appliance of the same type that the human is using. It is therefore unclear how often plural cases like Fig. 1’s “cookies” occur in the benchmark.
Clarifications:
- Table 2’s singular and non-singular AP50 (48.2 and 35.3; 49.2 and 36.5) are both below the overall validation AP50 of the same models (52.7; 53.9, Table 1). For GROVE, the overall (42.0) lies between its subsets (45.8 and 16.3). How are the subsets computed?
- The GROVE rows match GROVE’s published numbers. GROVE applies a threshold of 0.5 to the temporal objectness scores, while App. B decodes with conf = 0.3. Were the baselines evaluated under the same decoding settings?
6 comments
(2/4) W2 — Stage-2 reliability, the KL anchor, and faithfulness
2. Stage-2 reliability, the KL anchor, and faithfulness (W2)
To address the reviewer's concerns about the size and reliability of the Stage-2 gains, we added repeated runs, a KL-only control, reward diagnostics, and human evaluation. These show consistent improvements in grounding and caption faithfulness.
2.1 Repeated runs and gain size. We trained three Stage-1 checkpoints and continued each with Stage 2. The paired iGround test gains (mean ± standard deviation) are shown below:
| AP50 | Recall | METEOR | CIDEr |
|---|---|---|---|
| +1.6 ± 0.3 | +0.6 ± 0.2 | +0.7 ± 0.2 | +4.2 ± 0.5 |
The table summarizes positive mean gains across all metrics, and every metric also improves in every run. Stage 1 supplies most of the grounding gain; Stage 2 contributes more to caption quality. In the submitted run, test hallucination decreases from 21.6% to 19.2% (GROVE: 23.1%), accounting for 2.4 of the total 3.9-point reduction. Since grounding modules are frozen, Stage 2 affects localization through captions and their [DET] states.
2.2 Role of the KL anchor. Without KL, the submitted ablation declines from 52.7 to 51.5 AP50 and from 19.9 to 18.7 METEOR. This motivates retaining regularization. To separate the effect of visual feedback from KL-only updates, we added the following control using the same Stage-1 checkpoint and three Stage-2 seeds:
| iGround validation | AP50 ↑ | CIDEr ↑ | Obj. Halluc. ↓ |
|---|---|---|---|
| Stage-1 checkpoint | 52.7 | 93.8 | 21.4% |
| KL only, same updates | 52.5 ± 0.2 | 93.4 ± 0.3 | 21.3 ± 0.3% |
| Visual reward + KL | 53.7 ± 0.2 | 96.8 ± 0.4 | 18.8 ± 0.2% |
As shown above, KL-only updates remain near Stage 1. Adding visual reward under the same KL anchor improves AP50 by 1.2 and CIDEr by 3.4, while reducing automatic hallucination by 2.5 percentage points. Across the tested λ_KL values from 0.1 to 1.0, the submitted runs remain above Stage 1 in AP50 (52.9–53.9 versus 52.7) and CIDEr (94.2–97.1 versus 93.8). Appendix F also repeats the visual-reward ablations: AP50 differences are modest (at most 0.9), while hallucination ranges from 18.8% to 20.2%, compared with 21.8–22.4% for SCST.
2.3 Independent reward diagnostics. On held-out clips, the combined reward correlates with GT AP50, tube IoU, IDF1, and human faithfulness at Spearman ρ = 0.42, 0.48, 0.45, and 0.45. Phrase-level AUROC against human support labels is 0.77. The static-entity share changes from 38.4% to 38.9% (annotated: 37.6%) and mean box area from 0.142 to 0.140 (annotated: 0.138). The temporal term also remains positively correlated with GT tube quality on motion and occlusion subsets (Appendix E). Caption length, noun-phrase counts, and plural share are reported in our next response (Q1/Q2).
2.4 Human-verified faithfulness. Three blinded raters compare captions on 100 validation clips. They prefer Stage 2 on 55% of clips and Stage 1 on 26% (19% no preference). About 40% of automatically flagged phrases are judged visible, confirming the annotation-based metric's limitation. Direct human labels show the same direction of change in hallucination (13.8% → 11.3%), although this sample is small; mean faithfulness increases from 3.66 to 3.88.
Secs. 4.1, 4.2, and 4.4 now summarize this evidence. Appendices E and F provide the reward diagnostics, repeated runs, controls, and human evaluation.
(3/4) Q1 / Q2 — Caption statistics and multi-instance frequency
3. Caption statistics and multi-instance frequency (Q1 / Q2)
To address the reviewer's questions about caption changes and multi-instance frequency, we added caption statistics and an annotation breakdown. The aggregate caption statistics change little after Stage 2. Phrases with multiple linked trajectories are a minority and are most often hands or same-type tools and containers.
3.1 Caption and grounding statistics. The table below reports statistics on 500 validation clips. Plural phrases use the same head-noun POS rule (NNS/NNPS) as Table 2 in the revised manuscript. An entity is static when its box center moves less than 2% of the frame diagonal per sampled frame on average.
| Statistic | Multi-Instance Base | Full model | Annotated |
|---|---|---|---|
| Tokens per caption | 17.8 | 17.9 | – |
| Noun phrases per caption | 2.40 | 2.38 | – |
| Plural phrases | 14.6% (175/1,202) | 15.1% (180/1,192) | 14.2% (229/1,610) |
| Static grounded entities | 38.4% | 38.9% | 37.6% |
| Mean box area (fraction of frame) | 0.142 | 0.140 | 0.138 |
| Active slots per phrase and frame | 1.14 | 1.15 | – |
These results show little aggregate change in caption length, multiplicity, object motion, box size, or slot activation.
3.2 Instance frequency and linked trajectories. We count a phrase as multi-instance if some annotated frame contains at least two of its original boxes. In test, 241 of 403 plural phrases (59.8%) meet this criterion; the other 162 (40.2%) have at most one box per frame, including group-box annotations. Only 55 of 2,424 singular phrases (2.3%) are multi-instance. In training, 811 of 6,280 phrases (12.9%) have more than one linked trajectory. This is an upper bound on the multi-instance share because linking can fragment a track. The table below breaks down these training phrases by object type.
| Kind of training phrase with multiple linked trajectories | Phrases |
|---|---|
| Hands | 452 (55.7%) |
| Same-type tools, containers, or appliances in use | 176 (21.7%) |
| Food items boxed individually | 118 (14.5%) |
| Other objects | 65 (8.0%) |
The individually boxed food instances illustrated by “cookies” in Fig. 1 account for 118 / 6,280 training phrases (1.9%); hands and same-type tools are more common. Single-trajectory phrases also supervise extra slots through background targets. In sampled training clips, 1.3% of phrases have more than K linked trajectories; at most K are supervised, as described in Sec. 3.3 of the revised manuscript.
3.3 Recall by multiplicity. We group validation phrases by the largest number of original GT boxes in a frame and evaluate the Multi-Instance Base with annotated phrases as input. The table below reports instance recall and slot activation for each group.
| Max. GT boxes in one frame | Validation phrases | GT-instance [email protected] | Active slots per frame |
|---|---|---|---|
| 1 | 1,443 (89.6%) | 67.7% | 1.07 |
| 2 | 160 (9.9%) | 50.3% | 1.73 |
| 3 or more | 7 (0.4%) | 36.9% | 2.29 |
As shown above, slot activation rises with the number of visible instances, while recall is lower for phrases with greater multiplicity; the group with three or more instances contains only seven phrases. Our next response (Q2 / Clarifications) reports coverage with generated captions, counting unmentioned annotated entities as misses (Table 21 in the revised manuscript).
Appendix F (Table 16) reports caption statistics, and Appendix H (Tables 18–20) reports the annotation breakdown and multiplicity analysis. Appendix B specifies trajectory linking and its sensitivity.
(1/4) W1 — Training-time feedback and closest precedents
We thank the reviewer for the careful reading and the specific questions about prior work, evaluation, and iGround annotations.
Key additions.
Grounding feedback is a training reward in our method. At inference, the policy generates a caption once and then grounds it. We revised the framing to make this distinction explicit and added a dedicated discussion of the closest precedents.
1.1 Revised framing. The abstract and introduction motivate grounding as visual evidence for caption learning, and the first contribution identifies the self-rewarding framework built on multi-instance grounding. Sec. 3.4 of the revised manuscript states the single-pass inference procedure. Fig. 1 illustrates caption improvement through visual self-alignment, with feedback arrows denoting training-time policy updates; Fig. 3 compares GROVE with the full model. The revised title uses “Self-Rewarding” to identify the training-time reward source in the model's own grounding pathway.
Here, verification refers to entity support and temporal persistence estimated from multi-instance grounding. Appendix E directly evaluates these reward signals.
1.2 Closest precedents. Sec. 2 now discusses:
Our Stage 2 instead scores freely sampled video captions through the captioner's own frozen multi-instance grounding pathway. Its sequence-level presence and temporal rewards update the caption policy through policy gradients, with the verifier fixed and without caption or box targets in this stage.
1.3 Contribution of multi-instance evidence. The three-seed single-slot reward control uses the same frozen K = 3 verifier: AP50 / CIDEr / hallucination are 52.8 / 94.6 / 20.2%, versus 53.7 / 96.8 / 18.8% for the full reward (Appendix F, Table 15). Consistent with this, Stage 2 gives +0.4 AP50 / +0.7 CIDEr on single-instance GROVE versus +1.6 / +4.4 with the adapter; this comparison uses single runs with different Stage-1 checkpoints. Appendix H also reports the adapter's slot-design controls.
We revised the abstract, Sec. 1, the figure captions, Secs. 2 and 3.4, and the conclusion. Appendices E, F, and H provide the reward and slot-design evidence.