acceptodds
Under review as a conference paper at ICLR 2027

Grounding as Verification: Closing the Loop in Grounded Video Captioning

Abstract

Grounded video captioning systems generate descriptions with phrase-level localizations in the video. However, existing pipelines still flow one way, caption grounding: grounding reveals where the caption points, but never feeds back whether generated entities are visually supported. We push this view further by making grounding act as a verifier, forming a caption grounding loop in which localization evidence verifies and refines caption generation. For this purpose, we build the closed loop from two complementary directions. First, we observe that current grounding heads typically return only a single representative trajectory per phrase, regardless of whether the phrase refers to one or multiple entities. This provides only weak, presence-level verification: a plural phrase should be supported by multiple visual instances, yet the grounding head exposes only one. We therefore introduce a lightweight Multi-Instance Adapter that expands each phrase query into multiple persistent evidence slots, upgrading grounding into a structured multi-instance verifier. Second, we build annotation-free visual self-alignment with instance-presence and temporal-consistency rewards for KL-regularized policy optimization, improving captioning and grounding without ground-truth captions or extra box supervision during self-alignment. On iGround, the adapter improves AP50 from 40.0% to 51.7% over GROVE, and the full closed-loop model reaches 53.3% AP50 and 88.0 CIDEr. Results on VidSTG and ActivityNet-Entities further show effectiveness beyond the main benchmark. Together, these results support a verification-centered loop in which grounding provides feedback for better captions, and improved captions induce cleaner grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Grounded Video Captioning, Visual Grounding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kSHsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on GROVE, a grounded video captioning model that writes a caption and predicts one box track per noun phrase. It adds two stages:

  • Stage 1, a Multi-Instance Adapter. Each phrase query is expanded into K = 3 slots (query plus a learned offset). The slot tracks are trained with trajectory-level Hungarian matching and an orthogonality loss.
  • Stage 2, self-alignment with RL. A frozen copy of the Stage-1 model scores sampled captions using an instance-presence and temporal-consistency reward. The captioner is trained with self-critical REINFORCE plus a KL anchor, without ground-truth captions or boxes.
Critical strengths

S1. The Multi-Instance Adapter gives a large, well-ablated grounding gain at low cost. On iGround validation, slots raise AP50 from 43.5 (the authors’ own K = 1 adapter) to 52.7. Trajectory-level matching, orthogonality, and slot offsets are each ablated (Table 4). Gains also hold on VidSTG and ActivityNet-Entities with dataset-specific fine-tuning (Table 3). This is the paper’s main substantive contribution.

S2. Both stages are ablated component by component, including reward terms, advantage clipping, and KL strength (Tables 4-5). That makes the contribution of each part assessable.

Critical weaknesses

W1. The “closed loop / grounding as verification” framing goes beyond what is built, and its closest precedents are not discussed.

  • At inference, grounding remains a terminal prediction. C1 states that localization “feeds back to verify and refine generation instead of remaining a terminal prediction” (l.109–110), and the introduction faults GROVE because its grounding evidence “is still used only as a terminal prediction” (l.052). Yet at inference “this replay path is not used”: the policy generates the caption and its [DET] states go to the frozen grounding heads (Sec. 3.4, l.273). The feedback exists only as a training reward.
  • Feeding grounding back into caption generation has direct precedents. Ma et al. (arXiv:1906.00283) used a cyclical decode–localize–reconstruct regimen that forces the network to ground individually decoded words, for image and video captioning. The submission cites it only within lists of prior work, without discussing its loop. Woodpecker (arXiv:2310.16045) verifies generated text with an open-set detector, which drives its gains on object existence and count, and corrects the text in a training-free, post-hoc manner; it is not cited. The delta (video, dense multi-slot tubes, and an RL reward from the model’s own frozen head) should be stated.
  • Why consequential: the title contribution is a verification loop, but what is delivered is a training-time reward, and its relation to prior caption–grounding cycles and detector-based verification is not established.

W2. The evidence that the closed loop itself improves grounding and captions is limited.

  • The gain is small. Stage 2 adds +1.2 AP50 and +3.3 CIDEr on validation (52.7 → 53.9; 93.8 → 97.1), and +1.6 / +4.4 on test (Table 1). No seeds or variance are reported.
  • The gain depends on the KL anchor. Without it, Stage 2 falls below the Stage-1 model on every reported metric (Table 4): AP50 52.7 → 51.5, Recall 28.4 → 27.7, METEOR 19.9 → 18.7, CIDEr 93.8 → 91.4. With the anchor at λ_KL = 0.1–1.0, AP50 ranges from 52.9 to 53.9, i.e., +0.2 to +1.2 over Stage 1 (Table 5).
  • Both reward terms are computed only from the verifier’s own predictions. The temporal term (Eq. 6) measures frame-to-frame stability of the predicted boxes and reaches its maximum for any box that stays fixed and visible. The presence term (Eq. 5) sums objectness over slots. The visual-reward variants (presence-only, temporal-only, single-slot, full) differ by at most 1.0 AP50 (52.9–53.9, Table 4). No statistics show how captions change under the reward, such as phrases per caption, caption length, or plural share.
  • The hallucination metric conflates two things. It counts a phrase as hallucinated when it matches no annotated entity (Sec. 4.1), yet iGround annotators were asked to focus only on the objects that humans interact with rather than describing densely every object in the scene. Correct mentions of unannotated objects can therefore count as hallucinations.
  • Why consequential: the paper’s central claim is that grounding feedback improves generation. The measured effect is small, comes from single runs, and depends on the KL anchor.
Questions for authors

Experiments that could help clarify more:

  • Before and after Stage 2, please report noun phrases per caption, caption length, the share of plural phrases, and whether grounded entities are static or moving, to show the reward is not being exploited.
  • Please report instance-level coverage (the fraction of ground-truth trajectories matched), and how often iGround phrases have more than one ground-truth trajectory, and of what kinds. iGround’s guidelines annotate many small objects, e.g. mushrooms in a pan, with a single bounding box labelled ‘mushrooms’, and use multiple boxes for each human hand when both are used, or each tool/container/appliance of the same type that the human is using. It is therefore unclear how often plural cases like Fig. 1’s “cookies” occur in the benchmark.

Clarifications:

  • Table 2’s singular and non-singular AP50 (48.2 and 35.3; 49.2 and 36.5) are both below the overall validation AP50 of the same models (52.7; 53.9, Table 1). For GROVE, the overall (42.0) lies between its subsets (45.8 and 16.3). How are the subsets computed?
  • The GROVE rows match GROVE’s published numbers. GROVE applies a threshold of 0.5 to the temporal objectness scores, while App. B decodes with conf = 0.3. Were the baselines evaluated under the same decoding settings?
6 comments
AuthorsOpenReview ↗

(1/4) W1 — Training-time feedback and closest precedents

We thank the reviewer for the careful reading and the specific questions about prior work, evaluation, and iGround annotations.

Key additions.

  • Repeated runs: Stage 2 adds +1.6 ± 0.3 AP50 and +4.2 ± 0.5 CIDEr on test, with positive gains in every run.
  • KL-only control: Adding visual reward under the same KL anchor improves validation AP50 by 1.2 and reduces automatic hallucination by 2.5 percentage points.
  • Human evaluation: Stage-2 captions are preferred on 55% of clips versus 26% (p = 0.0017).

1. Training-time feedback and closest precedents (W1)

Grounding feedback is a training reward in our method. At inference, the policy generates a caption once and then grounds it. We revised the framing to make this distinction explicit and added a dedicated discussion of the closest precedents.

1.1 Revised framing. The abstract and introduction motivate grounding as visual evidence for caption learning, and the first contribution identifies the self-rewarding framework built on multi-instance grounding. Sec. 3.4 of the revised manuscript states the single-pass inference procedure. Fig. 1 illustrates caption improvement through visual self-alignment, with feedback arrows denoting training-time policy updates; Fig. 3 compares GROVE with the full model. The revised title uses “Self-Rewarding” to identify the training-time reward source in the model's own grounding pathway.

Here, verification refers to entity support and temporal persistence estimated from multi-instance grounding. Appendix E directly evaluates these reward signals.

1.2 Closest precedents. Sec. 2 now discusses:

  • Ma et al. (2020): a decode–localize–reconstruct cycle for image and video captioning, with ground-truth caption supervision but no localization supervision.
  • Woodpecker: training-free, post-hoc correction of image-LVLM outputs using an external open-set detector and a VQA model to gather visual evidence.
  • ViGoR: fine-tuning image LVLMs by rejection sampling, with fine-grained reward models learned from human evaluations and detector-based automatic rewards.
  • Self-Rewarding Language Models (Yuan et al., 2024): LLM-as-a-Judge rewards and iterative DPO that improve generation and reward evaluation together.

Our Stage 2 instead scores freely sampled video captions through the captioner's own frozen multi-instance grounding pathway. Its sequence-level presence and temporal rewards update the caption policy through policy gradients, with the verifier fixed and without caption or box targets in this stage.

1.3 Contribution of multi-instance evidence. The three-seed single-slot reward control uses the same frozen K = 3 verifier: AP50 / CIDEr / hallucination are 52.8 / 94.6 / 20.2%, versus 53.7 / 96.8 / 18.8% for the full reward (Appendix F, Table 15). Consistent with this, Stage 2 gives +0.4 AP50 / +0.7 CIDEr on single-instance GROVE versus +1.6 / +4.4 with the adapter; this comparison uses single runs with different Stage-1 checkpoints. Appendix H also reports the adapter's slot-design controls.

We revised the abstract, Sec. 1, the figure captions, Secs. 2 and 3.4, and the conclusion. Appendices E, F, and H provide the reward and slot-design evidence.

AuthorsOpenReview ↗

(2/4) W2 — Stage-2 reliability, the KL anchor, and faithfulness

2. Stage-2 reliability, the KL anchor, and faithfulness (W2)

To address the reviewer's concerns about the size and reliability of the Stage-2 gains, we added repeated runs, a KL-only control, reward diagnostics, and human evaluation. These show consistent improvements in grounding and caption faithfulness.

2.1 Repeated runs and gain size. We trained three Stage-1 checkpoints and continued each with Stage 2. The paired iGround test gains (mean ± standard deviation) are shown below:

AP50RecallMETEORCIDEr
+1.6 ± 0.3+0.6 ± 0.2+0.7 ± 0.2+4.2 ± 0.5

The table summarizes positive mean gains across all metrics, and every metric also improves in every run. Stage 1 supplies most of the grounding gain; Stage 2 contributes more to caption quality. In the submitted run, test hallucination decreases from 21.6% to 19.2% (GROVE: 23.1%), accounting for 2.4 of the total 3.9-point reduction. Since grounding modules are frozen, Stage 2 affects localization through captions and their [DET] states.

2.2 Role of the KL anchor. Without KL, the submitted ablation declines from 52.7 to 51.5 AP50 and from 19.9 to 18.7 METEOR. This motivates retaining regularization. To separate the effect of visual feedback from KL-only updates, we added the following control using the same Stage-1 checkpoint and three Stage-2 seeds:

iGround validationAP50 ↑CIDEr ↑Obj. Halluc. ↓
Stage-1 checkpoint52.793.821.4%
KL only, same updates52.5 ± 0.293.4 ± 0.321.3 ± 0.3%
Visual reward + KL53.7 ± 0.296.8 ± 0.418.8 ± 0.2%

As shown above, KL-only updates remain near Stage 1. Adding visual reward under the same KL anchor improves AP50 by 1.2 and CIDEr by 3.4, while reducing automatic hallucination by 2.5 percentage points. Across the tested λ_KL values from 0.1 to 1.0, the submitted runs remain above Stage 1 in AP50 (52.9–53.9 versus 52.7) and CIDEr (94.2–97.1 versus 93.8). Appendix F also repeats the visual-reward ablations: AP50 differences are modest (at most 0.9), while hallucination ranges from 18.8% to 20.2%, compared with 21.8–22.4% for SCST.

2.3 Independent reward diagnostics. On held-out clips, the combined reward correlates with GT AP50, tube IoU, IDF1, and human faithfulness at Spearman ρ = 0.42, 0.48, 0.45, and 0.45. Phrase-level AUROC against human support labels is 0.77. The static-entity share changes from 38.4% to 38.9% (annotated: 37.6%) and mean box area from 0.142 to 0.140 (annotated: 0.138). The temporal term also remains positively correlated with GT tube quality on motion and occlusion subsets (Appendix E). Caption length, noun-phrase counts, and plural share are reported in our next response (Q1/Q2).

2.4 Human-verified faithfulness. Three blinded raters compare captions on 100 validation clips. They prefer Stage 2 on 55% of clips and Stage 1 on 26% (19% no preference). About 40% of automatically flagged phrases are judged visible, confirming the annotation-based metric's limitation. Direct human labels show the same direction of change in hallucination (13.8% → 11.3%), although this sample is small; mean faithfulness increases from 3.66 to 3.88.

Secs. 4.1, 4.2, and 4.4 now summarize this evidence. Appendices E and F provide the reward diagnostics, repeated runs, controls, and human evaluation.

AuthorsOpenReview ↗

(3/4) Q1 / Q2 — Caption statistics and multi-instance frequency

3. Caption statistics and multi-instance frequency (Q1 / Q2)

To address the reviewer's questions about caption changes and multi-instance frequency, we added caption statistics and an annotation breakdown. The aggregate caption statistics change little after Stage 2. Phrases with multiple linked trajectories are a minority and are most often hands or same-type tools and containers.

3.1 Caption and grounding statistics. The table below reports statistics on 500 validation clips. Plural phrases use the same head-noun POS rule (NNS/NNPS) as Table 2 in the revised manuscript. An entity is static when its box center moves less than 2% of the frame diagonal per sampled frame on average.

StatisticMulti-Instance BaseFull modelAnnotated
Tokens per caption17.817.9–
Noun phrases per caption2.402.38–
Plural phrases14.6% (175/1,202)15.1% (180/1,192)14.2% (229/1,610)
Static grounded entities38.4%38.9%37.6%
Mean box area (fraction of frame)0.1420.1400.138
Active slots per phrase and frame1.141.15–

These results show little aggregate change in caption length, multiplicity, object motion, box size, or slot activation.

3.2 Instance frequency and linked trajectories. We count a phrase as multi-instance if some annotated frame contains at least two of its original boxes. In test, 241 of 403 plural phrases (59.8%) meet this criterion; the other 162 (40.2%) have at most one box per frame, including group-box annotations. Only 55 of 2,424 singular phrases (2.3%) are multi-instance. In training, 811 of 6,280 phrases (12.9%) have more than one linked trajectory. This is an upper bound on the multi-instance share because linking can fragment a track. The table below breaks down these training phrases by object type.

Kind of training phrase with multiple linked trajectoriesPhrases
Hands452 (55.7%)
Same-type tools, containers, or appliances in use176 (21.7%)
Food items boxed individually118 (14.5%)
Other objects65 (8.0%)

The individually boxed food instances illustrated by “cookies” in Fig. 1 account for 118 / 6,280 training phrases (1.9%); hands and same-type tools are more common. Single-trajectory phrases also supervise extra slots through background targets. In sampled training clips, 1.3% of phrases have more than K linked trajectories; at most K are supervised, as described in Sec. 3.3 of the revised manuscript.

3.3 Recall by multiplicity. We group validation phrases by the largest number of original GT boxes in a frame and evaluate the Multi-Instance Base with annotated phrases as input. The table below reports instance recall and slot activation for each group.

Max. GT boxes in one frameValidation phrasesGT-instance [email protected]Active slots per frame
11,443 (89.6%)67.7%1.07
2160 (9.9%)50.3%1.73
3 or more7 (0.4%)36.9%2.29

As shown above, slot activation rises with the number of visible instances, while recall is lower for phrases with greater multiplicity; the group with three or more instances contains only seven phrases. Our next response (Q2 / Clarifications) reports coverage with generated captions, counting unmentioned annotated entities as misses (Table 21 in the revised manuscript).

Appendix F (Table 16) reports caption statistics, and Appendix H (Tables 18–20) reports the annotation breakdown and multiplicity analysis. Appendix B specifies trajectory linking and its sensitivity.

Reviewer_mz9cOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies grounded video captioning and argues that existing systems use grounding only as a terminal output (caption → grounding), without feeding grounding evidence back to the caption generator. Built on GROVE, the method has two stages. Stage 1 adds a Multi-Instance Adapter that expands each [DET] phrase query into K = 3 slot queries through shared, orthogonally regularized residual embeddings, trained with trajectory-level Hungarian matching against ground-truth tubes. Stage 2 freezes a copy of the Stage-1 model as a verifier and replays sampled captions through it. It computes an instance-presence reward (objectness summed over slots) and a temporal-consistency reward (adjacent-frame self-IoU), and optimizes the generator with self-critical REINFORCE and a KL anchor, without caption or box annotations. On iGround test, AP50 goes from 40.0 to 51.7 after Stage 1 and to 53.3 after Stage 2, and CIDEr goes from 83.5 to 88.0. Additional results are reported on VidSTG and ActivityNet-Entities.

Critical strengths
  1. The Multi-Instance Adapter yields a large grounding gain, mostly on non-singular phrases (AP50 16.3 → 35.3, Table 2). The ablation shows that the gain is not from query capacity alone (naive duplicated queries 40.6 and K = 1 43.5, vs. 52.7).
  2. Reusing the model’s own grounding head as a reward requires no caption or box annotations in Stage 2, and it improves CIDEr by 4.4 on iGround test.
  3. The method and implementation are described clearly, including the trainable parameters and reward hyperparameters.
Critical weaknesses
  1. The “closed loop” amounts to a grounding-derived reward used only during training. At inference, the caption is generated once and then grounded (Sec. 3.4), which is the one-way flow the paper criticizes. The “items” → “cookies” example in Fig. 1 compares the outputs of two models rather than showing a refinement. Grounding-based verification of generated text has also been explored at inference (e.g., Woodpecker) and as a fine-grained training reward (e.g., ViGoR), and the adapter follows DETR-style set prediction over phrase queries. The reformulation claimed as the first contribution is therefore hard to view as a new problem setting.
  2. The reward does not implement the verification the paper motivates. The motivating failure is a plural phrase that looks grounded with only one instance. However, R_presence sums objectness over slots regardless of the phrase’s number, so it rewards activating more slots rather than detecting under-grounding. Averaging over phrases also rewards dropping hard-to-ground entities, and the larger temporal term favors static objects. The verifier is supervised on phrases from ground-truth captions, and whether its objectness drops for phrases without visual support, the premise of the loop, is not shown.
  3. The gains come mostly from supervised Stage 1, not from the loop. Stage 1, trained on ground-truth tubes, provides 11.7 of the 13.3 AP50 gain on iGround test. Stage 2 adds 1.6 AP50 and 4.4 CIDEr, and its reward variants differ by only 0.4 to 1.0 AP50 with no reported variance (Table 4). Stage 2 reuses the training videos on which the verifier was supervised, so its annotation-free property is never exercised. It is not applied on VidSTG and runs without the temporal term on ActivityNet-Entities, so the full loop is evaluated only on iGround.
  4. The evaluation does not separate visual faithfulness from alignment to the annotation vocabulary. Object hallucination counts generated noun phrases that match no annotated entity, and iGround Recall also requires phrase similarity to annotated phrases. Since the verifier is supervised on annotated entities, Stage 2 can improve both by steering captions toward the annotation vocabulary. No human evaluation or entity-count analysis is reported, and how the K slots enter AP50 and Recall is not specified.
Questions for authors

Q1. Is the verifier used at inference in any way (e.g., re-ranking or iterative refinement)? If not, how does the framework differ from RL with a grounding-based reward model?

Q2. How well does R_vis separate hallucinated from correct phrases on held-out videos (e.g., AUROC)?

Q3. How do the number of noun phrases per caption and the fraction of plural phrases change after Stage 2? Could you add a human evaluation of faithfulness?

Q4. How are the K slot outputs merged for iGround AP50 and Recall, and is GROVE evaluated with the same post-processing and fine-tuning budget?

Q5. Does Stage 2 help on unlabeled videos outside the Stage-1 training set?

Q6. Could you report multi-seed results for the Stage-2 ablations? The SCST baseline obtains lower CIDEr than the visual reward despite optimizing CIDEr directly; was it tuned separately?

Q7. The reward uses π_ref’s replayed [DET] states, while inference uses π_θ's. How far do π_θ's states drift, and does computing the reward from them change the results?

10 comments
AuthorsOpenReview ↗

(1/6) W1 / Q1 — Training-time feedback, prior work, and slot design

We thank the reviewer for the detailed assessment and the suggestions on framing, evaluation, and controls.

Key additions.

  • Repeated runs: Stage 2 adds +1.6 ± 0.3 AP50 and +4.2 ± 0.5 CIDEr on test, with positive gains in every run.
  • External videos: Stage 2 on unlabeled videos outside Stage 1 improves iGround test by +0.9 AP50 and +2.9 CIDEr.
  • Human evaluation: Stage-2 captions are preferred on 55% of clips versus 26% (p = 0.0017).

1. Training-time feedback, prior work, and slot design (W1 / Q1)

The reference verifier supplies the training reward. At inference, the trained policy generates a caption once and passes its own [DET] states directly to the grounding heads, without verifier replay, re-ranking, or iterative refinement.

1.1 Framing and inference. Stage 2, visual self-alignment, is policy-gradient RL with a grounding-derived reward. Our contribution lies in the multi-instance visual feedback: a frozen copy of the adapted captioner scores sampled captions using instance-presence and temporal-consistency rewards. This makes the stage self-rewarding. The same copy supplies the KL anchor, without an external parser, detector, or separately trained reward model. We revised the title to emphasize this training-time reward design.

As the reviewer notes, Fig. 1 compares separately generated captions, rather than showing inference-time refinement of one caption. The revised caption identifies the arrows as training-time policy updates. We have also revised the first contribution to emphasize multi-instance visual feedback for caption learning, rather than a new problem setting.

1.2 Relation to prior work. We added a dedicated comparison in Sec. 2 of the revised manuscript:

  • Ma et al. (2020) use a decode–localize–reconstruct cycle for image and video captioning, with ground-truth caption supervision but no localization supervision.
  • Woodpecker (Yin et al., 2024) corrects image-LVLM outputs after generation, without training, using an external open-set detector and a VQA model.
  • ViGoR (Yan et al., 2024) fine-tunes image LVLMs by rejection sampling, using fine-grained reward models learned from human evaluations and detector-based automatic rewards.
  • Self-Rewarding Language Models (Yuan et al., 2024) use LLM-as-a-Judge rewards and iterative DPO to improve both generation and reward evaluation.

Our Stage 2 instead uses policy-gradient optimization with a fixed multi-instance grounding verifier. Its presence and temporal-persistence rewards score sampled video captions without caption or box annotations.

1.3 Contribution of the multi-instance interface. With the same frozen K = 3 verifier and three Stage-2 seeds, retaining only the highest-objectness slot for reward computation gives 52.8 AP50 / 94.6 CIDEr / 20.2% hallucination, versus 53.7 / 96.8 / 18.8% for the full reward (Appendix F, Table 15). This directly tests the contribution of multi-instance feedback. Consistent with this, applying Stage 2 to single-instance GROVE gives +0.4 AP50 / +0.7 CIDEr on test, versus +1.6 / +4.4 with our adapter; this latter comparison uses single runs with different Stage-1 checkpoints.

The adapter builds on DETR-style set prediction. Its design uses phrase-anchored residual slots and matches each slot to a GT trajectory over the clip. Appendix H compares the following designs on iGround validation. Non-singular AP50 grounds the annotated captions, as in Table 2; AP50 follows the official protocol of Table 1:

Slot designAP50Non-singular AP50 (phrase-level)
Per-frame matching50.231.6
Independent per-slot projections51.933.9
Learned offsets with trajectory-level matching52.735.3

The table shows gains of 2.5 AP50 and 3.7 non-singular AP50 from trajectory-level matching. The AP50 gain averages 2.4 ± 0.3 across three runs. Residual offsets also outperform independent projections while using fewer slot-specific parameters.

The abstract presents the training-time visual feedback. Secs. 1 and 3.4 and Fig. 1 clarify its relation to inference; the conclusion describes the feedback as training-time. Sec. 2 discusses the precedents, Sec. 3.3 cites DETR, and Appendix H (Table 22) reports the slot-design comparison.

AuthorsOpenReview ↗

(2/6) W2 / Q2 — Entity support, multiplicity, and reward biases

2. Entity support, multiplicity, and reward biases (W2 / Q2)

To address the reviewer's concerns, we added reward diagnostics and caption statistics. These analyses test entity discrimination and multiplicity preferences, and examine slot activation, entity coverage, and temporal bias.

2.1 Slot activation and multiplicity. Stage 1 trains unmatched slots as background. With annotated phrases as input, one-instance phrases activate 1.07 slots per frame, and 9.1% ever activate more than one; two-instance phrases activate 1.73. Two active slots cover the same instance in 15.8% of cases.

Appendix E (Tables 7–8) compares captions with one-element edits, replaying both through each frozen verifier and pooling (wins + 0.5 × ties) / N over clips. For singular alternatives to supported plural descriptions, our presence term prefers the original in 77.6% of pairs, compared with 51.6% for the official GROVE checkpoint; the combined-reward results are 64.8% and 52.0%. Overstated plurals are not reliably rejected (54.0% combined; 46.3% presence alone), because Eq. (5) in the revised manuscript does not check number agreement.

2.2 Separating supported from hallucinated phrases. To test sensitivity to unsupported entities, we replace an annotated entity with an absent object. The presence term prefers the original caption in 82.2% of pairs (GROVE: 75.2%); the combined reward reaches 84.0% (GROVE: 76.7%; Appendix E, Table 7).

We further score the 1,202 noun phrases generated by the Multi-Instance Base on held-out validation clips. Labels come from the automatic hallucination metric and, separately, the 100-clip human-evaluation subset (240 phrases, 33 judged hallucinated). The table below reports AUROC under both labeling schemes.

Phrase scoreAUROC, automatic labels ↑AUROC, human labels ↑
GROVE (official checkpoint), combined0.640.69
Ours, presence term0.700.77
Ours, temporal term0.670.72
Ours, combined reward0.710.77

As shown above, with human labels, the combined reward achieves AUROC 0.77 versus 0.69 for GROVE (33 hallucinated phrases; indicative). The human evaluation also confirms that automatic labels incorrectly flag some visible, unannotated objects (Appendix F).

2.3 Caption content and temporal bias. Because the verifier is frozen, the policy changes the reward through its generated text. Active slots per phrase and frame change from 1.14 to 1.15. Using head-noun POS consistently, the plural share changes from 14.6% (175/1,202) to 15.1% (180/1,192), compared with 14.2% (229/1,610) in the annotations. Noun phrases per caption remain similar (2.40 → 2.38), while automatically flagged phrases decrease from 257 to 223 and annotated-entity coverage increases from 57.7% to 59.3%. The net change of −10 phrases consists of 34 fewer automatically flagged phrases and 24 more phrases matching annotated entities. These are mention counts; entity coverage counts distinct annotated phrases.

We also checked temporal-reward bias directly. The static-entity share changes from 38.4% to 38.9% (annotated: 37.6%), and mean box area from 0.142 to 0.140 (annotated: 0.138). The temporal term's correlations with GT tube IoU / IDF1 remain positive but weaken under motion and occlusion; for example, they are 0.43/0.46 overall and 0.25/0.28 on the occlusion subset. These diagnostics show little aggregate shift in the measured quantities.

Sec. 3.4 clarifies the reward's scope. Appendix E (Tables 7–13) reports the reward diagnostics, and Appendix F reports caption statistics and human evaluation.

AuthorsOpenReview ↗

(3/6) W3 / Q6 — Stage-2 gains, repeated runs, and tuned SCST

3. Stage-2 gains, repeated runs, and tuned SCST (W3 / Q6)

Stage 1 provides the main grounding gain, and Stage 2 further improves grounding and caption faithfulness. To address the reviewer's questions about reliability and SCST tuning, we now report independent runs, three-seed reward ablations, a KL-only control, and a separately tuned SCST baseline.

3.1 Roles of the two stages. Stage 1 contributes +11.7 of the +13.3 test AP50 improvement over GROVE. In the submitted run, Stage 2 reduces test hallucination from 21.6% to 19.2% (GROVE: 23.1%) and raises CIDEr from 83.6 to 88.0. Since grounding modules stay frozen, Stage 2 affects localization through generated captions and their [DET] states. On non-singular test phrases, with annotated captions as input, AP50 is 15.7 for GROVE, 33.6 after Stage 1, and 34.8 after Stage 2; this fixed-input change reflects updated [DET] states. The blinded human evaluation also favors Stage-2 captions (55% versus 26%; Appendix F).

3.2 Three independent runs. We trained three Multi-Instance Base checkpoints and continued each with Stage 2. Values below are mean ± standard deviation on iGround test; paired gains are computed against each run's own Stage-1 checkpoint.

ModelAP50RecallMETEORCIDEr
Multi-Instance Base51.6 ± 0.430.0 ± 0.321.5 ± 0.283.6 ± 0.6
Full model53.2 ± 0.630.6 ± 0.422.2 ± 0.387.8 ± 1.0
Paired Stage-2 gain+1.6 ± 0.3+0.6 ± 0.2+0.7 ± 0.2+4.2 ± 0.5

The table summarizes positive mean gains across all metrics, and every metric also improves in every run. The key Stage-1 ablations are also stable over three validation runs: trajectory-level matching adds +2.4 ± 0.3 AP50 over frame-level matching, and orthogonality adds +3.7 ± 0.6.

3.3 Reward ablations and the KL anchor. The following variants share one Stage-1 checkpoint and use three Stage-2 seeds on iGround validation. The single-slot reward retains only the slot with the highest mean objectness for each phrase in the same K = 3 verifier. The KL-only control sets the visual advantage to zero.

Stage-2 variantAP50 ↑CIDEr ↑Obj. Halluc. ↓
None (Stage-1 checkpoint)52.793.821.4%
KL only52.5 ± 0.293.4 ± 0.321.3 ± 0.3%
SCST (CIDEr), default Stage-2 settings52.0 ± 0.396.1 ± 0.421.8 ± 0.4%
SCST (CIDEr), tuned51.7 ± 0.497.3 ± 0.522.4 ± 0.5%
Single-slot visual reward52.8 ± 0.294.6 ± 0.420.2 ± 0.3%
Temporal term only52.9 ± 0.395.4 ± 0.519.8 ± 0.4%
Presence term only53.2 ± 0.396.2 ± 0.419.2 ± 0.3%
Full visual reward53.7 ± 0.296.8 ± 0.418.8 ± 0.2%

The KL-only control remains near Stage 1. Relative to this control, the full visual reward with the same KL anchor improves AP50 by 1.2 and reduces automatic hallucination by 2.5 percentage points. Differences among visual-reward variants are modest in AP50 (at most 0.9) but larger in hallucination.

3.4 Separately tuned SCST. The SCST baseline in Table 4 uses our default Stage-2 settings (lr 5e-7, λ_KL = 0.2), without separate tuning. The tuned variant searches lr ∈ {5e-7, 1e-6, 2e-6} and λ_KL ∈ {0.05, 0.2}, selecting settings and checkpoints by validation CIDEr to match its training objective. Our visual-reward runs use validation AP50, with CIDEr as a tie-breaker (Appendix A). With each method using its stated checkpoint-selection criterion, tuned SCST reaches comparable CIDEr (97.3 versus 96.8) but lower AP50 and higher hallucination. The three independent test runs in §3.2 separately establish the consistency of the Stage-2 gains.

Secs. 4.2 and 4.4 now summarize these findings. Appendix F (Tables 14–15) reports the repeated runs and controls. Our next response (W3/Q5) covers Stage 2 on unlabeled external videos and the scope of the other benchmarks.

Reviewer_h3sSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work extends GROVE with multiple grounding slots per noun phrase and uses the resulting grounding predictions to train the caption generator. The first stage uses supervised trajectory-level matching and an orthogonality loss to specialize the slots. The second stage freezes a reference verifier and updates the generator using instance-presence and temporal-consistency rewards, together with a KL penalty. The second stage consumes training videos without caption or box annotations. The main captioning and self-alignment evaluation is on iGround, with additional grounding experiments on VidSTG and ActivityNet-Entities.

Critical strengths

The multi-instance adapter solves a concrete limitation of the original interface. On iGround test, the supervised base raises AP50 from 40.0 to 51.7 while leaving captioning performance almost unchanged. The improvement is especially pronounced for non-singular phrases on validation: 16.3 to 35.3 AP50. The single-slot, duplicated-query, frame-level matching, and orthogonality ablations help explain where the gain comes from.

Self-alignment adds a further benefit beyond the supervised adapter. Test AP50 increases from 51.7 to 53.3 and CIDEr from 83.6 to 88.0. The validation entity-hallucination diagnostic decreases from 21.4% to 18.7%. The reward ablations and comparison with CIDEr-based SCST make a reasonable case for using grounding feedback rather than optimizing a text metric alone.

Critical weaknesses

My main concern is how much the reward actually verifies. Eq. (5) rewards objectness across slots without explicitly matching the number of instances to the phrase’s stated multiplicity. Eq. (6) rewards adjacent-frame box overlap, which can favor stationary objects or large boxes and penalize correctly tracked motion or occlusion. These are possible biases of the objective; the paper does not measure their effect.

The reported captioning gains are encouraging, but CIDEr and noun-phrase matching do not establish that the verifier detects incorrect counts, actions, or relations. A direct test with minimally different captions would help determine which claims the reward can verify. This matters for interpreting the closed-loop mechanism, even though the practical grounding improvements are clear.

Questions for authors
  1. On a small set of existing validation videos, does the verifier prefer the correct caption when alternatives differ only in instance count, an action, or an object relation? Such a test would clarify the reward’s capabilities beyond entity presence.

  2. Please specify the annotation versions used for Stage-1 training and evaluation, and how instance identities are associated across frames. The official GROVE distinguishes merged-instance training annotations from raw instance annotations and uses raw annotations for evaluation. How is this distinction handled in the reported comparisons?

4 comments
AuthorsOpenReview ↗

(1/3) W1 / W2 / Q1 — Reward discrimination and instance-count limitations

We thank the reviewer for recognizing the adapter's grounding gains and the additional benefit of self-alignment.

1. Reward discrimination and instance-count limitations (W1 / W2 / Q1)

On Q1: partly. The reward prefers the correct caption when an edit understates the count (77.6% presence, 64.8% combined), but not when it overstates it (46.3% presence, 54.0% combined). To address the reviewer's questions about count, actions, and relations, we added the minimal-pair evaluation below, including absent-entity substitutions.

On validation clips, we change one element of an annotated caption: replace an entity with an absent object; change a plural phrase to a singular alternative when multiple instances are visible; make the reverse edit when only one is visible; replace an action; or change a relation. An LLM (Qwen3.8-Max) generates the edits, and we retain those passing manual checking. Each frozen verifier scores both captions with the same reward. A win means that the original caption receives a higher reward than its edited alternative. Preference rate is (wins + 0.5 × ties) / N, pooled over clips; a tie is an absolute score difference below 10⁻⁶. Table 8 in the revised manuscript provides the win/tie/loss counts.

We compare our verifier with the official single-slot GROVE checkpoint, using the same combined-reward weights. The table below reports preference rates for both presence-only and combined rewards.

EditPairsGROVE, presenceGROVE, combinedOurs, presenceOurs, combined
Absent entity30075.2%76.7%82.2%84.0%
Fewer instances than visible12551.6%52.0%77.6%64.8%
More instances than visible30050.2%50.7%46.3%54.0%
Action30051.0%51.3%51.8%52.3%
Relation20050.0%50.5%50.5%51.0%

As shown in the table above, our combined reward reaches 84.0% on absent-entity edits, compared with 76.7% for GROVE. For plural-to-singular edits, our presence term prefers the original plural description in 77.6% of pairs, versus 51.6% for GROVE; the combined reward gives 64.8% versus 52.0%. The presence term favors plural wording in both directions, more strongly when multiple instances are visible; adding the temporal term reduces the plural-to-singular preference from 77.6% to 64.8%.

Preference rates remain near chance for edits that overstate multiplicity (54.0% combined; 46.3% presence), change actions (52.3% combined), or change relations (51.0% combined). The presence term sums slot objectness without explicitly checking number agreement.

We clarified the reward's scope in Sec. 3.4 and Appendix D, and added the minimal-pair results and counts in Appendix E (Tables 7–8).

AuthorsOpenReview ↗

(2/3) W1 — Temporal-reward bias under motion and occlusion

2. Temporal-reward bias under motion and occlusion (W1)

To address the reviewer's concern that temporal overlap may favor static entities or large boxes, we measured these statistics and tested how the temporal term relates to ground-truth tube quality under motion and occlusion. The statistics show little aggregate shift after Stage 2, while the correlations weaken in difficult cases.

2.1 Static entities and box size. On 500 validation clips, an entity is classified as static when its box center moves less than 2% of the frame diagonal per sampled frame on average. The table below compares these statistics before and after self-alignment with those of annotated tracks.

StatisticMulti-Instance BaseFull modelAnnotated tracks
Static grounded entities38.4%38.9%37.6%
Mean box area (fraction of frame)0.1420.1400.138

As shown above, the static share changes by 0.5 percentage points and mean box area decreases slightly. Both remain close to the annotated-track statistics, providing no indication of a substantial aggregate shift toward static entities or larger boxes.

2.2 Motion and occlusion. The correlation analyses below use model-generated captions from held-out validation clips, with reward and evaluation scores paired for the same caption. For each criterion, we take the most difficult quartile of the 500 clips (125 clips) and compute Spearman correlation between the temporal term and GT tube IoU / IDF1. Criteria use normalized GT-center displacement, background optical flow, GT box-area change, and visible-to-invisible switches of linked GT trajectories, respectively (Appendix E). The correlations for each subset are shown below.

SubsetAll clipsFast motionCamera motionScale changeOcclusion
ρ with tube IoU / IDF10.43 / 0.460.31 / 0.360.26 / 0.300.34 / 0.290.25 / 0.28

The table shows that correlations stay positive but are weaker under motion and occlusion. Eq. (6) in the revised manuscript assigns zero to a transition marked visible in only one frame, so genuine occlusion can lower the temporal reward even for an otherwise correct track.

2.3 Agreement with independent references. We also measure clip-level Spearman correlation with GT grounding on 500 validation clips and human faithfulness on 100 clips. The table below compares the reward terms.

RewardAP50Tube IoUIDF1Human faithfulness
Presence term0.360.340.290.33
Temporal term0.270.430.460.27
Combined reward0.420.480.450.45

These correlations are positive for all reward terms. Grouping clips by combined reward shows the same overall trend: higher-reward groups generally have better grounding quality (Table 13 in the revised manuscript).

Appendix E (Tables 10–13) reports these analyses. Sec. 3.4 summarizes the findings, and Appendix D describes the occlusion limitation.

AuthorsOpenReview ↗

(3/3) Q2 — Annotation versions and cross-frame instance identities

3. Annotation versions and cross-frame instance identities (Q2)

Stage 1 uses raw instance-level training annotations. All reported iGround comparisons use the same official raw instance-level evaluation annotations. To address the reviewer's questions about annotation versions and instance identities, we clarified these protocols and added a comparison under matched Stage-1 training settings.

3.1 Training and evaluation annotations. Stage 1 uses the raw annotations to preserve separately annotated instances; the official GROVE checkpoint uses the processed version, which merges same-type instance boxes within each frame. Its published validation and test results, which we report in Table 1 of the revised manuscript, are evaluated on raw annotations. Our main AP50 and Recall results use the same official evaluation code, all-frame protocol, and decoding threshold (0.5).

3.2 Instance identities. iGround provides per-frame boxes without instance identities. Before training-frame sampling, we link each phrase's boxes by Hungarian matching on box IoU (threshold 0.3, gaps of up to two annotated frames). Clip targets retain every trajectory visible in the sampled frames, including single-frame ones. Of 200 manually checked phrases with multiple linked trajectories, 94.5% had no identity switch and 85.0% had no fragmented trajectory. Frame-level counts use the original boxes, and AP50/Recall do not require the links. Retraining Stage 1 at linking thresholds of 0.2 or 0.5 changes validation AP50 by at most 0.5 points (Appendix B, Table 5).

3.3 Matched training controls. The following iGround validation controls share our raw annotations, HowToGround1M initialization, and Stage-1 recipe, except for the GROVE row, whose overall AP50 is published. Non-singular AP50 grounds the annotated captions, as in Table 2; overall AP50 follows the official protocol of Table 1.

ModelAP50Non-singular AP50
GROVE (merged-instance training annotations)42.016.3
Naive duplicated queries (K = 3)40.616.9
Single-slot adapter (K = 1)43.520.1
Our adapter (K = 3)52.735.3

In the matched comparisons above, the full adapter exceeds the single-slot control by 9.2 AP50 with the initialization, annotations, and training recipe held fixed. Naively duplicating queries also falls well below the full adapter. These controls support the contribution of specialized multi-instance slots beyond the change in annotation format.

Sec. 4.1 points to Appendix G for evaluation and training controls, and Appendix B specifies linking. Additional statistics on separately annotated instances and linked trajectories are reported in Appendix H.

We appreciate the reviewer's constructive feedback. The added diagnostics and protocol details clarify both the reward's capabilities and the source of the grounding gains. We welcome follow-up questions. If these additions address your concerns, we hope you will consider raising your score.

I believe the response has addressed some of my concerns. After reading all the reviews and the revised manuscript, I decided to maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.