acceptodds
Under review as a conference paper at ICLR 2027

PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summary

Abstract

Detailed video captioning has become an essential semantic interface for multimodal understanding and generation. While existing methods mainly target short videos, extending them to long videos remains challenging, primarily due to limited context windows of current video understanding models. A naive solution is to adopt segment-wise captioning followed by concatenation, yet this paradigm often suffers from inconsistent subject references and fragmented long-range narratives. To mitigate these issues, we introduce PIGEONS, an incremental long-video captioning model built upon a Persistent Identity Gallery and an Evolving Overarching Narrative Summary. Before caption generation, PIGEONS scans the entire video to construct an identity gallery that captures key subjects along with their visual references, thereby providing a stable referential anchor. The model then performs clip-by-clip captioning conditioned on the persistent gallery and an evolving overarching narrative summary, which compactly maintains and updates essential long-range contextual information after each captioning step. To enable these capabilities, we develop a three-stage training strategy that integrates supervised fine-tuning and reinforcement learning. Extensive experiments demonstrate that PIGEONS significantly improves long-video captioning performance, achieving superior accuracy, completeness, and subject referential consistency over existing approaches, while maintaining competitive performance on short-video captioning. PIGEONS will be open-sourced to facilitate future research.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
caption, long-video caption

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RSwiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

PIGEONS addresses inconsistent subject references and fragmented long-range context when segment-level video captions are concatenated. It first scans the entire video to construct an identity gallery containing subject identifiers and reference images. It then generates each segment's caption conditioned on the gallery, the preceding caption, and an evolving narrative summary. Training consists of three stages: gallery SFT, RL for gallery quality and construction efficiency, and caption SFT.

The paper evaluates captions on CapRiCorn-1K and UGC-VideoCap and measures caption-based question answering on WorldSense. This is an offline approach that processes the entire video before caption generation. The clearest empirical gain is in subject referential consistency.

Critical strengths

1. A substantial referential-consistency gap remains against a segment-wise captioning baseline.

In Table 4, AVoCaDO-Concat and PIGEONS achieve Accuracy scores of 49.8 and 50.7 and Coverage scores of 64.1 and 65.4, respectively. In contrast, Ref increases from 34.1 to 51.8, while average caption length decreases from 3,025.7 to 1,271.5. This is the most convincing comparison because it demonstrates a benefit beyond simply splitting the video into segments or generating longer descriptions.

2. Addressing reference errors through upstream gallery learning is well motivated.

Section 3.4 treats subject discovery and reference-image extraction as explicit learning targets and uses RL to improve gallery accuracy and construction efficiency. Improving the evidence supporting subject references, rather than only optimizing textual caption scores, is well aligned with the problem.

The substantial improvement in referential consistency over the segment-wise baseline is the main reason for my weak acceptance.

Critical weaknesses

Update after author responses (16 Nov 2026): The shared-gallery ablation, coherence-evaluation protocol, same-backbone comparison, and runtime/memory measurements address the concerns below. I consider these methodological questions resolved and have updated Soundness from 2 to 3.

1. The additional effects of the evolving summary and three-stage training are not sufficiently isolated.

The summary is a central component, but I could not find a comparison that removes only the summary while retaining the same gallery. Table 3 does not establish how much the summary improves long-range context over using the preceding caption alone.

Moreover, Cap-SFT-only achieves a Ref score of 49.4, compared with 51.8 for the full method. The finding that the full method performs better should be distinguished from the claim in Section 4.3 that all three stages are necessary. Separating the main benefit of gallery-conditioned captioning from the incremental benefit of the additional training stages would clarify the contribution.

2. The total processing cost relative to the main baseline is unclear.

The cost analysis in Appendix C.1 considers an interleaved approach that regenerates earlier captions when subjects are added to the gallery. The actual runtime and memory costs relative to AVoCaDO-Concat in Table 4 are less clear. Costs covering both gallery construction and caption generation are needed to assess the expense associated with the Ref improvement and the video scales at which the method is practical.

The ablation and cost questions concern the additional components' contributions and the method's practicality. A small incremental benefit from the summary or training stages would not by itself invalidate the main Ref improvement. I would reconsider my recommendation if clarification of the comparison conditions undermines that improvement.

Questions for authors

Q1. Using Table 3 and any existing ablation results or training records, please clarify the respective contributions of the evolving summary and the additional gallery-training stages. If available, a comparison that removes only the summary while keeping the gallery fixed would help separate these effects.

Q2. For videos already included in the evaluation, can you compare the total processing time and peak memory usage, from gallery construction through final caption generation, against AVoCaDO-Concat? Please specify the input resolution, segmentation, output settings, and caching configuration so that the quality–cost trade-off can be assessed.

6 comments
AuthorsOpenReview ↗

Response to the Reviewer RSwi [Concern 1 (Weakness 1 & Question 1)]

Concern 1: The additional effects of the evolving summary and three-stage training are not sufficiently isolated. Using Table 3 and any existing ablation results or training records, please clarify the respective contributions of the evolving summary and the additional gallery-training stages. If available, a comparison that removes only the summary while keeping the gallery fixed would help separate these effects.

Part 1: The summary is a central component, but I could not find a comparison that removes only the summary while retaining the same gallery. Table 3 does not establish how much the summary improves long-range context over using the preceding caption alone.

Part 2: Cap-SFT-only achieves a Ref score of 49.4, compared with 51.8 for the full method. The finding that the full method performs better should be distinguished from the claim in Section 4.3 that all three stages are necessary. Separating the main benefit of gallery-conditioned captioning from the incremental benefit of the additional training stages would clarify the contribution.


Response 1 (Part 1): Thank you for this constructive suggestion!

Using the same training configuration, we train the model by solely removing the evolving summary, the preceding caption, or both from the training data after the IG-RL stage. The performance of each setting on CapRiCorn-1K is shown in the table below:

ModelAccCovRef
PIGEONS50.765.451.8
Cap-SFT w/o evolving summary49.563.650.5
Cap-SFT w/o preceding caption50.464.951.3
Cap-SFT w/o summary & preceding caption49.163.350.4

The results show that removing the preceding caption alone has a relatively minor impact on benchmark metrics, as it primarily serves to improve local narrative continuity across segment boundaries. In contrast, when removing the evolving summary alone or removing both the evolving summary and the preceding caption simultaneously, both overall caption quality and subject referential consistency decrease slightly. This aligns with the expectation that the evolving summary provides global contextual information, and removing it weakens the connection between the current caption and prior context, thereby affecting the overall description quality.

It should be noted that the global and local coherence brought by the evolving summary and preceding caption is not prominently reflected in standard benchmark scores. Existing keypoint-based evaluation protocols focus on atomic facts and are largely insensitive to cross-clip narrative coherence, for which a dedicated quantitative metric is currently lacking in the community. Beyond Table 4, which quantitatively reflects better coherence through more concise yet higher-quality captions, we additionally employ Gemini-3.1-Pro as a judge model to rate both the global coherence of the captions and the local coherence at segment boundaries on a 1-to-5 scale. The results are presented in the table below:

ModelGlobal CoherenceLocal Coherence
PIGEONS3.834.43
Cap-SFT w/o evolving summary3.144.35
Cap-SFT w/o preceding caption3.622.94
Cap-SFT w/o summary & preceding caption2.562.62

These results clearly indicate that the evolving summary leads to better global coherence, whereas the preceding caption ensures superior local coherence. Additionally, the bottom-right corner of Figure 2 in the main paper qualitatively illustrates the improved caption readability and narrative flow enabled by such long-range coherence.


Response 1 (Part 2): We also acknowledge that our original wording describing the three-stage training as "necessary" was somewhat too aggressive. Indeed, performing SFT for gallery-conditioned captioning alone already yields satisfactory performance. However, our research goal is to explore how far the performance upper bound could be pushed, and the full three-stage recipe demonstrates headroom indeed exists, bringing additional improvements of +2.4 points. When computational or training resources are constrained, gallery-conditioned captioning alone remains a highly competitive and practical choice in terms of the cost-performance trade-off.

We will revise the wording in the final version to state that the three-stage training yields stronger performance, while avoiding the overly absolute claim that all three stages are "necessary."

AuthorsOpenReview ↗

Response to the Reviewer RSwi [Concern 2 (Weakness 2 & Question 2)]

Concern 2: The total processing cost relative to the main baseline is unclear. The cost analysis in Appendix C.1 considers an interleaved approach that regenerates earlier captions when subjects are added to the gallery. The actual runtime and memory costs relative to AVoCaDO-Concat in Table 4 are less clear. Costs covering both gallery construction and caption generation are needed to assess the expense associated with the Ref improvement and the video scales at which the method is practical.

Can you compare the total processing time and peak memory usage against AVoCaDO-Concat? Please specify the input resolution, segmentation, output settings, and caching configuration so that the quality–cost trade-off can be assessed.

Response 2: Thank you for this valuable suggestion!

We evaluate the stage-wise inference runtime and peak GPU memory usage of PIGEONS against AVoCaDO-Concat on CapRiCorn-1K, with both models deployed via vLLM across 4 NVIDIA H200 GPUs. Specifically, the maximum input resolution is set to 262,144262{,}144 (256×32×32256 \times 32 \times 32) pixels for PIGEONS and 200,704200{,}704 (256×28×28256 \times 28 \times 28) pixels for AVoCaDO-Concat. Both models adopt the exact same segmentation strategy: long videos are uniformly partitioned into segments with a maximum duration of two minutes, shorter videos are left intact, and every resulting clip is sampled at 2 FPS. For output generation, both models use greedy decoding with a maximum of 4,0964{,}096 output tokens and a 32 GB vLLM multimodal cache. Under these aligned configurations, the inference runtime and peak memory usage are directly comparable.

The table below reports the average per-video inference runtime (in seconds), broken down by stage and grouped by video duration. "Runtime Ratio" denotes the end-to-end runtime of PIGEONS relative to AVoCaDO-Concat, and "Ref Ratio" denotes the ratio of their subject referential consistency scores.

Video DurationLocal Media ProcessingGallery ConstructionIterative RefinementCaption GenerationTotalAVoCaDO-ConcatRuntime RatioRef RatioRef Improvement
0-2 min1.69.68.44.423.914.31.681.2939.7 -> 51.3 (+11.6)
2-4 min2.021.411.310.345.030.01.501.5234.4 -> 52.1 (+17.7)
4-6 min2.236.512.818.269.750.01.401.7531.7 -> 55.5 (+23.9)
6-8 min2.750.716.126.095.563.81.501.7228.9 -> 49.5 (+20.7)
8-10 min3.266.119.335.1123.781.21.521.7828.3 -> 50.5 (+22.2)
Avg2.230.512.415.560.640.21.511.5234.1 -> 51.8 (+17.7)

The results show that PIGEONS adds only 20.4 s of inference runtime per video on average (merely 4.8 s per minute of video) in exchange for a substantial 17.7-point improvement in Ref. Moreover, as video duration increases, the Runtime Ratio remains stable at around 1.5×1.5\times, while the Ref Ratio exhibits a clear upward trend and exceeds 1.7×1.7\times for videos longer than four minutes, indicating that PIGEONS scales favorably to long-video captioning.


The comparison of per-GPU peak memory usage (in GB) in vLLM is shown in the table below:

ModelModel WeightsNon-PyTorch OverheadPeak ActivationCUDA GraphsPeak KV Cache UsageTotal
PIGEONS14.930.320.860.1011.4127.62
AVoCaDO-Concat5.180.490.800.093.4610.02

The difference in peak memory usage is primarily attributable to backbone scale: PIGEONS adopts a 30B-A3B MoE backbone (48 layers) whereas AVoCaDO-Concat uses a 7B dense model (28 layers), resulting in larger model weights and per-token KV cache size.

To further evaluate our method under a comparable memory budget, we train PIGEONS by replacing the base model (Qwen3-Omni-Instruct) with Qwen2.5-Omni, matching the backbone of AVoCaDO:

Method (Base Model)AccCovRefTime Cost (s)Peak Memory (GB)
AVoCaDO-Concat (Qwen2.5-Omni)49.864.134.140.210.02
PIGEONS (Qwen2.5-Omni)50.064.544.352.912.75
PIGEONS (Qwen3-Omni)50.765.451.860.627.62

The results demonstrate that even with a smaller base model, our method consistently yields notable gains in subject referential consistency, while incurring only a modest overhead over AVoCaDO-Concat and reducing both the inference runtime and peak GPU memory compared to the original PIGEONS.


Overall, despite the moderate increase in inference overhead, PIGEONS substantially improves both subject referential consistency (+17.7 points on average) and information density (higher accuracy and coverage at nearly 0.4×0.4\times the caption length, as shown in Table 4), while demonstrating strong scalability for long-video captioning.

Reviewer_RSwiOpenReview ↗

Follow-up on the component ablations and inference costs

Thank you for the additional experiments and for revisiting the claim about the necessity of the three-stage training procedure.

The paper aims to address both inconsistent subject references and fragmented long-range narratives. The response strengthens the evidence for improved subject referencing, while the extent of the evolving summary's contribution to long-range narrative coherence remains less clearly established.

The new Cap-SFT ablations show that removing the summary reduces Coverage from 65.4 to 63.6 and Ref from 51.8 to 50.5. These results support a positive but modest incremental contribution on the existing benchmark metrics. The new judge-based coherence scores provide complementary evidence consistent with the intended roles of the summary and preceding caption: removing the summary has a larger effect on global coherence, whereas removing the preceding caption has a larger effect on local coherence. These findings are encouraging, but understanding how directly they measure the reduction of narrative fragmentation requires a clearer description of the coherence evaluation. The revised characterization of the full training recipe as an improvement over a strong gallery-conditioned Cap-SFT baseline is also appropriate.

The same-backbone comparison is particularly helpful. With Qwen2.5-Omni, PIGEONS improves Ref from 34.1 to 44.3, with average runtime increasing from 40.2 to 52.9 seconds and reported per-GPU peak memory from 10.02 to 12.75 GB. The larger backbone in the original configuration therefore cannot be the sole explanation for the improvement over AVoCaDO-Concat. The clearest gain in this comparison remains subject referencing, with much smaller changes in Accuracy and Coverage. The full Qwen3-Omni configuration provides further quality improvements at a substantially larger memory footprint, making the distinction between these configurations important when discussing practical benefits.

Taken together, the additional results make the referential-consistency benefit and its associated costs more convincing. The remaining limitation is that the evidence for this benefit is more direct than the evidence establishing how much the evolving summary alleviates long-range narrative fragmentation. Could you clarify two details that would help assess this second objective?

First, were identical precomputed identity galleries reused across the summary-ablation variants at evaluation, or did each trained variant construct its own gallery? If the galleries differ, the results remain informative as ablations of the complete trained configurations, but do not isolate summary conditioning with the gallery held constant. Is that the appropriate interpretation of these comparisons?

Second, could you provide the coherence-judge inputs, prompt, scoring rubric, and evaluated sample count? In particular, how does the rubric assess long-range narrative coherence separately from subject-reference consistency? This would help determine what the new scores establish about the summary's intended role in preserving narrative connections across clips.

Reviewer_cd3VOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PIGEONS addresses inconsistent subject references and fragmented narratives in segment-wise long-video captioning. It first scans the full video to construct a gallery of important subjects and representative crops. A second pass generates clip captions conditioned on this fixed gallery, the previous caption, and a structured narrative summary that is updated after each clip. The Qwen3-Omni backbone is trained using identity-gallery SFT, gallery-oriented reinforcement learning, and caption SFT. The main empirical result is improved subject referential consistency on CapRiCorn-1K, including against an AVoCaDO segment-and-concatenate baseline, while producing substantially shorter captions. The inference procedure is incremental within each pass but requires access to the complete video before caption generation.

Critical strengths
  • The method addresses a specific and important failure of segmented captioning. Stable visual references and persistent IDs offer a direct way to link recurring subjects despite clothing and scene changes. The candidate gallery also handles subjects whose narrative importance becomes apparent later (pp. 4-5).
  • The main referential-consistency gain remains under a similar segment-wise protocol. Against AVoCaDO-Concat, PIGEONS raises Ref from 34.1 to 51.8 while Acc/Cov change from 49.8/64.1 to 50.7/65.4 (Table 4, p. 9). The reported average caption length falls from 3025.7 to 1271.5, approximately 58%; the table does not specify the length unit. Complete input and generation-budget matching remains to be confirmed. Thus, the strongest result is improved referential consistency with a more compact output, rather than the very large overall-quality differences against unadapted models in Table 1.
  • The ablations support identity conditioning. In Table 3, switching from Cap-SFT without gallery data/conditioning to the gallery-conditioned Cap-SFT setting raises Ref from 28.1 to 49.4, with nearly unchanged Acc/Cov (rows 3-4, p. 8). This is consistent with the proposed gallery addressing identity continuity rather than merely improving generic caption quality.
  • The supplementary material is useful. It describes the 16K-video/37K-clip training corpus, teacher annotation and filtering, training hyperparameters, inference segmentation, and detailed prompts. Appendix C also acknowledges that gallery/summary growth limits context scalability and that duration bins differ in annotation granularity (pp. 14-24).
Critical weaknesses
  1. The summary component is not isolated. The evolving narrative summary is one of the two central contributions, but Table 3 varies gallery conditioning and training stages while retaining summary/history conditioning. Table 4 changes the model and training as well as the memory structure. The paper gives a concrete summary design and qualitative full-system examples, but there is no reported comparison that holds the gallery, training data, and backbone fixed while removing the summary or replacing it with a matched amount of prior-caption context. Consequently, the evidence supports the complete system and the gallery more clearly than it supports a distinct gain in long-range narrative coherence from the summary. Shorter captions alone do not identify that mechanism. An existing summary-specific ablation would be valuable; otherwise, the contribution claim should be narrower.

  2. The evaluation does not directly compare with the closest identity-aware captioning approaches. StoryTeller and RICE are discussed in Section 2.1 but not evaluated. StoryTeller's supplied cast information is a genuine protocol difference, so a direct unrestricted comparison would not necessarily be fair. RICE uses multi-frame visual attention within windows and feature-based textual association across windows. Section 2.1 should acknowledge the former while explaining how PIGEONS’ fixed global crop gallery differs; the statement about cross-window textual association is not itself incorrect. An input-matched comparison or a precise account of why adaptation is infeasible is needed to establish the practical advantage of learned gallery construction over previous identity-aware designs. The primary methods confirm these distinctions: StoryTeller, RICE.

  3. The evidence for the necessity of all three training stages is weaker than the wording suggests. Full PIGEONS improves on IG-SFT followed by Cap-SFT by 1.4 Acc, 1.6 Cov, and 1.9 Ref points (Table 3, rows 6 and 9), while Cap-SFT alone already reaches 49.4 Ref. These improvements may be useful, but no uncertainty estimates or direct held-out gallery-quality results are reported. Table 3 row 8 supports a downstream benefit from the combined format/efficiency reward, but the explanation that it prevents excessive review loops is qualitative; construction success rates, interaction counts, and gallery precision/recall are not shown. The experiment demonstrates a favorable recipe, but does not yet demonstrate that the full ordering is necessary or that the asserted gallery-quality/efficiency mechanism explains the gains.

Questions for authors
  1. Are there existing gallery-fixed runs that remove the evolving summary or replace it with a comparable token budget of caption history? If not, which evidence supports the summary-specific narrative-coherence claim beyond the complete-system comparison?
  2. Appendix A.3 specifies clips of at most two minutes, 2 FPS, and the maximum spatial resolution. Were these settings applied to every baseline in Tables 1 and 4, or were any captions reused or exceptions made? Please state generation budgets and the exact CapRiCorn judge/version; Table 2 already identifies the WorldSense judge as Gemini-2.5-Pro.
  3. From existing training/evaluation logs, can you provide gallery matching quality, construction completion rates, and review-round distributions before/after IG-RL and with/without the efficiency reward? Can paired video-level uncertainty be computed for the smaller Table 3 and Table 4 differences?
  4. How are P=0, Q=0, zero matched similarity, and zero within-group reward variance handled? What is the maximum number of review rounds, and is 1-0.1k clipped or permitted to become negative?
5 comments
AuthorsOpenReview ↗

Response to the Reviewer cd3V [Concern 1 (Weakness 1 & Question 1)]

Concern 1: The summary component is not isolated. The evolving narrative summary is one of the two central contributions, but Table 3 varies gallery conditioning and training stages while retaining summary conditioning. An existing summary-specific ablation would be valuable. Are there existing gallery-fixed runs that remove the evolving summary or replace it with a comparable token budget of caption history?

Response 1: Thank you for this constructive suggestion!

In our design (Section 3.3), the narrative context comprises two complementary components: the global evolving summary and the local preceding caption. Therefore, to isolate their individual contributions using the same training configuration, we train the model by solely removing the evolving summary, the preceding caption, or both from the training data after the IG-RL stage. The performance of each setting on CapRiCorn-1K is shown in the table below:

ModelAccCovRef
PIGEONS50.765.451.8
Cap-SFT w/o evolving summary49.563.650.5
Cap-SFT w/o preceding caption50.464.951.3
Cap-SFT w/o summary & preceding caption49.163.350.4

The results show that removing the preceding caption alone has a relatively minor impact on benchmark metrics, as it primarily serves to improve local narrative continuity across segment boundaries. In contrast, when removing the evolving summary alone or removing both the evolving summary and the preceding caption simultaneously, both overall caption quality and subject referential consistency decrease slightly. This aligns with the expectation that the evolving summary provides global contextual information, and removing it weakens the connection between the current caption and prior context, thereby affecting the overall description quality.


It should be noted that the global and local coherence brought by the evolving summary and preceding caption is not prominently reflected in standard benchmark scores. Existing keypoint-based evaluation protocols focus on atomic facts and are largely insensitive to cross-clip narrative coherence, for which a dedicated quantitative metric is currently lacking in the community. Beyond Table 4, which quantitatively reflects better coherence through more concise yet higher-quality captions, we additionally employ Gemini-3.1-Pro as a judge model to rate both the global coherence of the captions and the local coherence at segment boundaries on a 1-to-5 scale. The results are presented in the table below:

ModelGlobal CoherenceLocal Coherence
PIGEONS3.834.43
Cap-SFT w/o evolving summary3.144.35
Cap-SFT w/o preceding caption3.622.94
Cap-SFT w/o summary & preceding caption2.562.62

These results clearly indicate that the evolving summary leads to better global coherence, whereas the preceding caption ensures superior local coherence. Additionally, the bottom-right corner of Figure 2 in the main paper qualitatively illustrates the improved caption readability and narrative flow enabled by such long-range coherence.

AuthorsOpenReview ↗

Response to the Reviewer cd3V [Concern 2 (Weakness 2)]

Concern 2: The evaluation does not directly compare with the closest identity-aware captioning approaches. StoryTeller and RICE are discussed in Section 2.1 but not evaluated. StoryTeller's supplied cast information is a genuine protocol difference, so a direct unrestricted comparison would not necessarily be fair. RICE uses multi-frame visual attention within windows and feature-based textual association across windows.

Response 2: Thank you for your valuable suggestion!

While we have discussed the differences between PIGEONS and both StoryTeller and RICE in Section 2.1, we did not include them in our primary evaluation because neither of their models is publicly available.

Nevertheless, to provide a direct comparison, we make every effort to reproduce both approaches under the following settings. For StoryTeller, we follow its inference pipeline by taking the identity gallery generated by PIGEONS as the auxiliary cast information and replacing the unreleased StoryTeller model with Qwen3-Omni-Instruct. For RICE, since RICE-8B is not publicly available, we adopt the GPT-4o-based RICE inference pipeline described in the original paper. Their performance on CapRiCorn-1K is reported below:

MethodAccCovRef
StoryTeller25.736.58.4
RICE9.823.12.3
PIGEONS (Ours)50.765.451.8

These results demonstrate that, while StoryTeller leverages multiple specialist models with external cast information and RICE builds upon the proprietary GPT-4o, there remains a noticeable performance margin compared to our fully self-contained and soon-to-be open-sourced PIGEONS.

AuthorsOpenReview ↗

Response to the Reviewer cd3V [Concern 3 (Weakness 3 & Question 3)]

Concern 3: The evidence for the necessity of all three training stages is weaker than the wording suggests. Full PIGEONS improves on IG-SFT followed by Cap-SFT by 1.4 Acc, 1.6 Cov, and 1.9 Ref points, while Cap-SFT alone already reaches 49.4 Ref. These improvements may be useful, but no uncertainty estimates or direct held-out gallery-quality results are reported. Table 3 row 8 supports a downstream benefit from the efficiency reward, but the explanation is qualitative.

From existing training/evaluation logs, can you provide gallery matching quality, construction completion rates, and review-round distributions before/after IG-RL and with/without the efficiency reward? Can paired video-level uncertainty be computed for the smaller Table 3 and Table 4 differences?

Response 3: Thank you for your constructive suggestions!

In the table below, we report the gallery matching quality (measured by RaccR_{acc} in Eq. 7), construction completion rates, and the average number of review rounds across three settings: before IG-RL, and after IG-RL with and without ReffR_{eff}. Results are evaluated on both the training set and a held-out subset of 100 randomly sampled videos from CapRiCorn-1K, for which we manually annotated ground-truth identity galleries:

SettingGallery Quality (Train)Construction Completion Rate (Train)Review Rounds ↓\downarrow (Train)Gallery Quality (Test)Construction Completion Rate (Test)Review Rounds ↓\downarrow (Test)
Before IG-RL0.40250.1%4.450.21816%5.69
After IG-RL (w/ ReffR_{eff})0.69791.8%3.130.53268%3.78
After IG-RL (w/o ReffR_{eff})0.62083.2%3.640.44545%4.83

These results provide direct, quantitative evidence that the IG-RL stage and the efficiency-aware format reward ReffR_{eff} substantially enhance gallery construction quality, efficiency, and generalization. A higher-quality identity gallery in turn provides more reliable visual conditioning and improves training–inference consistency during caption generation, ultimately translating into stronger end-to-end captioning performance.


To assess the statistical reliability of the performance gains in Tables 3 and 4, we conduct paired video-level bootstrap resampling with 10,000 iterations on CapRiCorn-1K. We report the mean score differences in Acc, Cov, and Ref, along with their 95% confidence intervals (CIs), comparing full PIGEONS against key baselines:

Comparison (Full PIGEONS vs.)Δ\DeltaAcc95% CIΔ\DeltaCov95% CIΔ\DeltaRef95% CI
IG-SFT + Cap-SFT (Table 3, Row 6)1.4[0.83, 2.08]1.6[0.96, 2.33]1.9[1.42, 2.43]
Cap-SFT (Table 3, Row 4)1.7[1.12, 2.28]1.4[0.63, 2.21]2.4[1.83, 2.94]
AVoCaDO-Concat (Table 4)0.9[0.36, 1.53]1.3[0.50, 2.07]17.7[16.28, 18.97]

The results show that the 95% CIs for Δ\DeltaAcc, Δ\DeltaCov, and Δ\DeltaRef lie strictly above zero across all comparisons, confirming that the improvements achieved by the full three-stage training over the ablated variants and over AVoCaDO-Concat are statistically reliable.


We also acknowledge that our original wording describing the three-stage training as "necessary" was somewhat too aggressive. Indeed, performing SFT for gallery-conditioned captioning alone (Cap-SFT) or omitting the IG-RL stage (IG-SFT + Cap-SFT) already yields satisfactory performance. However, our research goal is to explore how far the performance upper bound could be pushed, and the full three-stage recipe demonstrates headroom indeed exists, bringing additional improvements of +2.4 and +1.9 Ref points over these two variants, respectively. When computational or training resources are constrained, Cap-SFT alone or IG-SFT + Cap-SFT remains a highly competitive and practical choice in terms of the cost-performance trade-off.

We will revise the wording in the final version to state that the three-stage training yields stronger performance, while avoiding the overly absolute claim that all three stages are "necessary."

Reviewer_bGTVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses two limitations of segment-wise long-video captioning: inconsistent subject references across clips and fragmented long-range narratives. The authors propose PIGEONS, a two-pass captioning framework built around a Persistent Identity Gallery and an Evolving Overarching Narrative Summary. Before caption generation, the model scans the entire video to construct a gallery of narratively important subjects, each associated with a stable identifier and a visual reference image. An auxiliary candidate gallery tracks potentially important subjects before their promotion to the main gallery. During the second pass, PIGEONS generates captions clip by clip, conditioned on the fixed identity gallery, the evolving narrative summary, and the caption of the preceding clip. The resulting clip-level captions are concatenated to form the final long-video caption. To support this framework, the authors introduce a three-stage training strategy comprising identity-gallery supervised fine-tuning, reinforcement learning for gallery accuracy and construction efficiency, and caption supervised fine-tuning. Experiments are conducted on CapRiCorn-1K, UGC-VideoCap, and WorldSense, covering videos of varying durations. The results show substantial improvements in subject referential consistency and long-video captioning performance, while maintaining competitive performance on short-video captioning. The paper also includes training-stage ablations and comparisons with segment-wise captioning followed by concatenation.

Critical strengths
  • A focused and relevant problem. The paper targets subject referential inconsistency, a concrete and important failure mode of segment-wise long-video captioning.

  • An intuitive and well-aligned solution. The persistent identity gallery provides stable visual anchors, while the evolving narrative summary and previous caption preserve long-range and local context. The design is simple, interpretable, and directly addresses the identified problems.

Critical weaknesses
  1. Limited methodological novelty. The overall approach seems somewhat incremental, as related ideas for long-video captioning appear to have been explored in earlier works such as ShareGPT4Video and GLaVE-Cap. The paper should more clearly differentiate PIGEONS from these approaches.

  2. Insufficient evidence of downstream video-understanding benefits. Video captioning should ultimately support video understanding. Although WorldSense provides an indirect caption-based QA evaluation, the experiments do not directly demonstrate that the generated captions improve downstream video-understanding tasks or agents.

Questions for authors

Questions for Authors

  1. Motivation compared with long-context VLMs. Given the rapid progress of long-context LLMs/VLMs, why should subject referential consistency in generated captions remain a central concern? If key frames and subtitles can be directly provided in context, under what conditions does PIGEONS offer advantages over direct video understanding in terms of accuracy, cost, or scalability?

  2. Data-generation cost. What are the wall-clock time and computational/monetary costs of captioning one video, including gallery construction, iterative refinement, and caption generation? How do these costs scale with video duration and the number of tracked subjects?

  3. Dataset scale and downstream utility. How many long-video-caption tokens were generated in this work, and how many could realistically be produced at scale? How does the resulting data compare with publicly available caption datasets on Hugging Face? Most importantly, does training a VLM on PIGEONS-generated captions improve long-video understanding more than training on existing open caption data?

  4. Controlled comparisons and component ablations. Can the authors provide comparisons using the same backbone, training data, segmentation strategy, and inference budget, together with separate ablations of the identity gallery, evolving summary, and previous-clip caption?

  5. Data decontamination. What decontamination was performed between the training videos and evaluation benchmarks, particularly given the use of TikTok-10M for training and the TikTok-based UGC-VideoCap benchmark for evaluation?

10 comments
AuthorsOpenReview ↗

Response to the Reviewer bGTV [Concern 1 (Weakness 1)]

Concern 1: Limited methodological novelty. The overall approach seems somewhat incremental, as related ideas for long-video captioning appear to have been explored in earlier works such as ShareGPT4Video and GLaVE-Cap. The paper should more clearly differentiate PIGEONS from these approaches.

Response 1: Thank you for highlighting these two insightful works!

Although both prior works [1,2] and PIGEONS study video captioning, our specific research problems and emphases are distinct. Specifically, ShareGPT4Video primarily focuses on capturing fine-grained visual details and temporal dynamics, while GLaVE-Cap leverages TrackFusion and CaptionBridge to further enhance descriptive granularity and strengthen the interaction between local and global captions. In contrast, PIGEONS specifically targets the challenges of inconsistent subject references and fragmented long-range narratives introduced by segment-wise captioning. To mitigate these issues, we train the model to autonomously construct a persistent identity gallery and maintain an evolving overarching narrative summary, which distinguishes PIGEONS from prior work in both problem formulation and methodology.

Furthermore, the two works also differ from PIGEONS in evaluation scope. They mainly validate their effectiveness on videos under 2 minutes, whereas PIGEONS is evaluated on long-form videos extending up to the 10-minute scale, and it outperforms the closed-source Gemini-3.1-Pro in this setting. Moreover, our strategy can theoretically scale to videos of arbitrary length.

We will cite these works and provide further discussion in the final version of the paper.


[1] ShareGPT4Video: Improving Video Understanding and Generation with Better Captions, NeurIPS 2024

[2] GLaVE-Cap: Global-Local Aligned Video Captioning with Vision Expert Integration, TCSVT 2026

AuthorsOpenReview ↗

Response to the Reviewer bGTV [Concern 2 (Weakness 2 & Question 3)]

Concern 2: Insufficient evidence of downstream video-understanding benefits. Video captioning should ultimately support video understanding. Although WorldSense provides an indirect caption-based QA evaluation, the experiments do not directly demonstrate that the generated captions improve downstream video-understanding tasks or agents.

How many long-video-caption tokens were generated, and how many could realistically be produced at scale? How does the resulting data compare with publicly available caption datasets on Hugging Face? Most importantly, does training a VLM on PIGEONS-generated captions improve long-video understanding more than training on existing open caption data?

Response 2: We sincerely thank the reviewer for this constructive suggestion and insightful questions!

To further verify the effectiveness of PIGEONS-generated captions in facilitating downstream video understanding, we randomly sample 10K videos each from FineVideo [3] and AVTime-50K [4] (20K videos in total). We then employ both AVoCaDO, the leading open-source baseline on CapRiCorn-1K, and our PIGEONS to generate captions for these 20K videos, which are subsequently used to perform SFT on Qwen2.5-Omni. The QA performance of the resulting models on Video-Holmes [5] and WorldSense [6] is presented in the table below:

ModelVideo-HolmesWorldSense
Qwen2.5-Omni43.745.4
+ trained with AVoCaDO-caption44.146.2
+ trained with PIGEONS-caption45.948.5

The results demonstrate that, compared to captions generated by AVoCaDO, training with PIGEONS-generated captions yields higher performance gains across both benchmarks, directly validating the effectiveness of PIGEONS in enhancing downstream video understanding.


Across the 20K sampled videos, PIGEONS generates approximately 17.2M caption tokens in total. Moreover, as PIGEONS will be open-sourced, it can be readily deployed in practice to synthesize caption data at an arbitrary scale.

Compared with existing video caption datasets publicly available on Hugging Face, which primarily consist of short videos within 2 minutes (e.g., ShareGPT4Video [1]: 27s, MiraData [7]: 72s, and GLaVE-1.2M [2]: 86s on average), PIGEONS enables the annotation of much longer videos, producing detailed captions with high overall quality and strong subject referential consistency.


[3] FineVideo, released by HuggingFace in 2024

[4] AVTime: Reinforcing Dense Audio-Visual Captions with Bidirectional Alignment, arXiv 2026

[5] Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?, ECCV 2026

[6] WorldSense: Evaluating Real-world Omnimodal Understanding for Multimodal LLMs, ICLR 2026

[7] MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captions, NeurIPS 2024

AuthorsOpenReview ↗

Response to the Reviewer bGTV [Concern 3 (Question 1)]

Concern 3: Motivation compared with long-context VLMs. Given the rapid progress of long-context LLMs/VLMs, why should subject referential consistency in generated captions remain a central concern? If key frames and subtitles can be directly provided in context, under what conditions does PIGEONS offer advantages over direct video understanding in terms of accuracy, cost, or scalability?

Response 3: Thank you for this insightful question!

Since long videos generally feature more complex storylines, which often involve multiple similar-looking subjects or appearance variations of the same subject over time (e.g., changes in clothing), merely scaling up the context window is insufficient for models to reliably resolve such challenging cases. Consequently, subject referential consistency remains a significant bottleneck even for long-context models.

As reported in Table 1 of our paper, even Gemini-3.1-Pro with a 1M context window achieves a subject referential consistency score of only 35.4 on 8–10 minute videos. Meanwhile, frontier open-source omni-modal models, Qwen3-Omni-Instruct and Qwen3-Omni-Captioner, attain average subject referential consistency scores of only 1.6 and 4.1, respectively, demonstrating that subject referential consistency remains an open challenge for long-context models.


To evaluate how the Qwen3-Omni series performs when explicit visual anchors are supplied in context, we provide the PIGEONS-generated identity galleries directly to both models to guide them in generating captions with more consistent subject references. The results across different settings on CapRiCorn-1K are presented below:

ModelAccCovRef
Qwen3-Omni-Instruct vanilla10.320.21.6
Qwen3-Omni-Instruct w/ identity gallery10.720.92.8
Qwen3-Omni-Captioner vanilla14.327.54.1
Qwen3-Omni-Captioner w/ identity gallery14.427.16.7
PIGEONS (Ours)50.765.451.8

These results show that while supplying an identity gallery to Qwen3-Omni brings slight improvements in subject referential consistency, the absolute scores remain relatively low. By contrast, PIGEONS, which undergoes dedicated training, delivers substantial performance gains. Moreover, PIGEONS will be open-sourced to allow the community to synthesize any number of captions with stronger subject referential consistency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.