- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
PIGEONS addresses inconsistent subject references and fragmented long-range context when segment-level video captions are concatenated. It first scans the entire video to construct an identity gallery containing subject identifiers and reference images. It then generates each segment's caption conditioned on the gallery, the preceding caption, and an evolving narrative summary. Training consists of three stages: gallery SFT, RL for gallery quality and construction efficiency, and caption SFT.
The paper evaluates captions on CapRiCorn-1K and UGC-VideoCap and measures caption-based question answering on WorldSense. This is an offline approach that processes the entire video before caption generation. The clearest empirical gain is in subject referential consistency.
Critical strengths
1. A substantial referential-consistency gap remains against a segment-wise captioning baseline.
In Table 4, AVoCaDO-Concat and PIGEONS achieve Accuracy scores of 49.8 and 50.7 and Coverage scores of 64.1 and 65.4, respectively. In contrast, Ref increases from 34.1 to 51.8, while average caption length decreases from 3,025.7 to 1,271.5. This is the most convincing comparison because it demonstrates a benefit beyond simply splitting the video into segments or generating longer descriptions.
2. Addressing reference errors through upstream gallery learning is well motivated.
Section 3.4 treats subject discovery and reference-image extraction as explicit learning targets and uses RL to improve gallery accuracy and construction efficiency. Improving the evidence supporting subject references, rather than only optimizing textual caption scores, is well aligned with the problem.
The substantial improvement in referential consistency over the segment-wise baseline is the main reason for my weak acceptance.
Critical weaknesses
Update after author responses (16 Nov 2026): The shared-gallery ablation, coherence-evaluation protocol, same-backbone comparison, and runtime/memory measurements address the concerns below. I consider these methodological questions resolved and have updated Soundness from 2 to 3.
1. The additional effects of the evolving summary and three-stage training are not sufficiently isolated.
The summary is a central component, but I could not find a comparison that removes only the summary while retaining the same gallery. Table 3 does not establish how much the summary improves long-range context over using the preceding caption alone.
Moreover, Cap-SFT-only achieves a Ref score of 49.4, compared with 51.8 for the full method. The finding that the full method performs better should be distinguished from the claim in Section 4.3 that all three stages are necessary. Separating the main benefit of gallery-conditioned captioning from the incremental benefit of the additional training stages would clarify the contribution.
2. The total processing cost relative to the main baseline is unclear.
The cost analysis in Appendix C.1 considers an interleaved approach that regenerates earlier captions when subjects are added to the gallery. The actual runtime and memory costs relative to AVoCaDO-Concat in Table 4 are less clear. Costs covering both gallery construction and caption generation are needed to assess the expense associated with the Ref improvement and the video scales at which the method is practical.
The ablation and cost questions concern the additional components' contributions and the method's practicality. A small incremental benefit from the summary or training stages would not by itself invalidate the main Ref improvement. I would reconsider my recommendation if clarification of the comparison conditions undermines that improvement.
Questions for authors
Q1. Using Table 3 and any existing ablation results or training records, please clarify the respective contributions of the evolving summary and the additional gallery-training stages. If available, a comparison that removes only the summary while keeping the gallery fixed would help separate these effects.
Q2. For videos already included in the evaluation, can you compare the total processing time and peak memory usage, from gallery construction through final caption generation, against AVoCaDO-Concat? Please specify the input resolution, segmentation, output settings, and caching configuration so that the quality–cost trade-off can be assessed.
6 comments
Response to the Reviewer RSwi [Concern 2 (Weakness 2 & Question 2)]
Concern 2: The total processing cost relative to the main baseline is unclear. The cost analysis in Appendix C.1 considers an interleaved approach that regenerates earlier captions when subjects are added to the gallery. The actual runtime and memory costs relative to AVoCaDO-Concat in Table 4 are less clear. Costs covering both gallery construction and caption generation are needed to assess the expense associated with the Ref improvement and the video scales at which the method is practical.
Can you compare the total processing time and peak memory usage against AVoCaDO-Concat? Please specify the input resolution, segmentation, output settings, and caching configuration so that the quality–cost trade-off can be assessed.
Response 2: Thank you for this valuable suggestion!
We evaluate the stage-wise inference runtime and peak GPU memory usage of PIGEONS against AVoCaDO-Concat on CapRiCorn-1K, with both models deployed via vLLM across 4 NVIDIA H200 GPUs. Specifically, the maximum input resolution is set to () pixels for PIGEONS and () pixels for AVoCaDO-Concat. Both models adopt the exact same segmentation strategy: long videos are uniformly partitioned into segments with a maximum duration of two minutes, shorter videos are left intact, and every resulting clip is sampled at 2 FPS. For output generation, both models use greedy decoding with a maximum of output tokens and a 32 GB vLLM multimodal cache. Under these aligned configurations, the inference runtime and peak memory usage are directly comparable.
The table below reports the average per-video inference runtime (in seconds), broken down by stage and grouped by video duration. "Runtime Ratio" denotes the end-to-end runtime of PIGEONS relative to AVoCaDO-Concat, and "Ref Ratio" denotes the ratio of their subject referential consistency scores.
| Video Duration | Local Media Processing | Gallery Construction | Iterative Refinement | Caption Generation | Total | AVoCaDO-Concat | Runtime Ratio | Ref Ratio | Ref Improvement |
|---|---|---|---|---|---|---|---|---|---|
| 0-2 min | 1.6 | 9.6 | 8.4 | 4.4 | 23.9 | 14.3 | 1.68 | 1.29 | 39.7 -> 51.3 (+11.6) |
| 2-4 min | 2.0 | 21.4 | 11.3 | 10.3 | 45.0 | 30.0 | 1.50 | 1.52 | 34.4 -> 52.1 (+17.7) |
| 4-6 min | 2.2 | 36.5 | 12.8 | 18.2 | 69.7 | 50.0 | 1.40 | 1.75 | 31.7 -> 55.5 (+23.9) |
| 6-8 min | 2.7 | 50.7 | 16.1 | 26.0 | 95.5 | 63.8 | 1.50 | 1.72 | 28.9 -> 49.5 (+20.7) |
| 8-10 min | 3.2 | 66.1 | 19.3 | 35.1 | 123.7 | 81.2 | 1.52 | 1.78 | 28.3 -> 50.5 (+22.2) |
| Avg | 2.2 | 30.5 | 12.4 | 15.5 | 60.6 | 40.2 | 1.51 | 1.52 | 34.1 -> 51.8 (+17.7) |
The results show that PIGEONS adds only 20.4 s of inference runtime per video on average (merely 4.8 s per minute of video) in exchange for a substantial 17.7-point improvement in Ref. Moreover, as video duration increases, the Runtime Ratio remains stable at around , while the Ref Ratio exhibits a clear upward trend and exceeds for videos longer than four minutes, indicating that PIGEONS scales favorably to long-video captioning.
The comparison of per-GPU peak memory usage (in GB) in vLLM is shown in the table below:
| Model | Model Weights | Non-PyTorch Overhead | Peak Activation | CUDA Graphs | Peak KV Cache Usage | Total |
|---|---|---|---|---|---|---|
| PIGEONS | 14.93 | 0.32 | 0.86 | 0.10 | 11.41 | 27.62 |
| AVoCaDO-Concat | 5.18 | 0.49 | 0.80 | 0.09 | 3.46 | 10.02 |
The difference in peak memory usage is primarily attributable to backbone scale: PIGEONS adopts a 30B-A3B MoE backbone (48 layers) whereas AVoCaDO-Concat uses a 7B dense model (28 layers), resulting in larger model weights and per-token KV cache size.
To further evaluate our method under a comparable memory budget, we train PIGEONS by replacing the base model (Qwen3-Omni-Instruct) with Qwen2.5-Omni, matching the backbone of AVoCaDO:
| Method (Base Model) | Acc | Cov | Ref | Time Cost (s) | Peak Memory (GB) |
|---|---|---|---|---|---|
| AVoCaDO-Concat (Qwen2.5-Omni) | 49.8 | 64.1 | 34.1 | 40.2 | 10.02 |
| PIGEONS (Qwen2.5-Omni) | 50.0 | 64.5 | 44.3 | 52.9 | 12.75 |
| PIGEONS (Qwen3-Omni) | 50.7 | 65.4 | 51.8 | 60.6 | 27.62 |
The results demonstrate that even with a smaller base model, our method consistently yields notable gains in subject referential consistency, while incurring only a modest overhead over AVoCaDO-Concat and reducing both the inference runtime and peak GPU memory compared to the original PIGEONS.
Overall, despite the moderate increase in inference overhead, PIGEONS substantially improves both subject referential consistency (+17.7 points on average) and information density (higher accuracy and coverage at nearly the caption length, as shown in Table 4), while demonstrating strong scalability for long-video captioning.
Follow-up on the component ablations and inference costs
Thank you for the additional experiments and for revisiting the claim about the necessity of the three-stage training procedure.
The paper aims to address both inconsistent subject references and fragmented long-range narratives. The response strengthens the evidence for improved subject referencing, while the extent of the evolving summary's contribution to long-range narrative coherence remains less clearly established.
The new Cap-SFT ablations show that removing the summary reduces Coverage from 65.4 to 63.6 and Ref from 51.8 to 50.5. These results support a positive but modest incremental contribution on the existing benchmark metrics. The new judge-based coherence scores provide complementary evidence consistent with the intended roles of the summary and preceding caption: removing the summary has a larger effect on global coherence, whereas removing the preceding caption has a larger effect on local coherence. These findings are encouraging, but understanding how directly they measure the reduction of narrative fragmentation requires a clearer description of the coherence evaluation. The revised characterization of the full training recipe as an improvement over a strong gallery-conditioned Cap-SFT baseline is also appropriate.
The same-backbone comparison is particularly helpful. With Qwen2.5-Omni, PIGEONS improves Ref from 34.1 to 44.3, with average runtime increasing from 40.2 to 52.9 seconds and reported per-GPU peak memory from 10.02 to 12.75 GB. The larger backbone in the original configuration therefore cannot be the sole explanation for the improvement over AVoCaDO-Concat. The clearest gain in this comparison remains subject referencing, with much smaller changes in Accuracy and Coverage. The full Qwen3-Omni configuration provides further quality improvements at a substantially larger memory footprint, making the distinction between these configurations important when discussing practical benefits.
Taken together, the additional results make the referential-consistency benefit and its associated costs more convincing. The remaining limitation is that the evidence for this benefit is more direct than the evidence establishing how much the evolving summary alleviates long-range narrative fragmentation. Could you clarify two details that would help assess this second objective?
First, were identical precomputed identity galleries reused across the summary-ablation variants at evaluation, or did each trained variant construct its own gallery? If the galleries differ, the results remain informative as ablations of the complete trained configurations, but do not isolate summary conditioning with the gallery held constant. Is that the appropriate interpretation of these comparisons?
Second, could you provide the coherence-judge inputs, prompt, scoring rubric, and evaluated sample count? In particular, how does the rubric assess long-range narrative coherence separately from subject-reference consistency? This would help determine what the new scores establish about the summary's intended role in preserving narrative connections across clips.
Response to the Reviewer RSwi [Concern 1 (Weakness 1 & Question 1)]
Concern 1: The additional effects of the evolving summary and three-stage training are not sufficiently isolated. Using Table 3 and any existing ablation results or training records, please clarify the respective contributions of the evolving summary and the additional gallery-training stages. If available, a comparison that removes only the summary while keeping the gallery fixed would help separate these effects.
Part 1: The summary is a central component, but I could not find a comparison that removes only the summary while retaining the same gallery. Table 3 does not establish how much the summary improves long-range context over using the preceding caption alone.
Part 2: Cap-SFT-only achieves a Ref score of 49.4, compared with 51.8 for the full method. The finding that the full method performs better should be distinguished from the claim in Section 4.3 that all three stages are necessary. Separating the main benefit of gallery-conditioned captioning from the incremental benefit of the additional training stages would clarify the contribution.
Response 1 (Part 1): Thank you for this constructive suggestion!
Using the same training configuration, we train the model by solely removing the evolving summary, the preceding caption, or both from the training data after the IG-RL stage. The performance of each setting on CapRiCorn-1K is shown in the table below:
The results show that removing the preceding caption alone has a relatively minor impact on benchmark metrics, as it primarily serves to improve local narrative continuity across segment boundaries. In contrast, when removing the evolving summary alone or removing both the evolving summary and the preceding caption simultaneously, both overall caption quality and subject referential consistency decrease slightly. This aligns with the expectation that the evolving summary provides global contextual information, and removing it weakens the connection between the current caption and prior context, thereby affecting the overall description quality.
It should be noted that the global and local coherence brought by the evolving summary and preceding caption is not prominently reflected in standard benchmark scores. Existing keypoint-based evaluation protocols focus on atomic facts and are largely insensitive to cross-clip narrative coherence, for which a dedicated quantitative metric is currently lacking in the community. Beyond Table 4, which quantitatively reflects better coherence through more concise yet higher-quality captions, we additionally employ Gemini-3.1-Pro as a judge model to rate both the global coherence of the captions and the local coherence at segment boundaries on a 1-to-5 scale. The results are presented in the table below:
These results clearly indicate that the evolving summary leads to better global coherence, whereas the preceding caption ensures superior local coherence. Additionally, the bottom-right corner of Figure 2 in the main paper qualitatively illustrates the improved caption readability and narrative flow enabled by such long-range coherence.
Response 1 (Part 2): We also acknowledge that our original wording describing the three-stage training as "necessary" was somewhat too aggressive. Indeed, performing SFT for gallery-conditioned captioning alone already yields satisfactory performance. However, our research goal is to explore how far the performance upper bound could be pushed, and the full three-stage recipe demonstrates headroom indeed exists, bringing additional improvements of +2.4 points. When computational or training resources are constrained, gallery-conditioned captioning alone remains a highly competitive and practical choice in terms of the cost-performance trade-off.
We will revise the wording in the final version to state that the three-stage training yields stronger performance, while avoiding the overly absolute claim that all three stages are "necessary."