- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OACap, an agent-based framework for detailed video captioning that uses the evolving caption to guide additional visual observations. Rather than generating a description through a single global pass, OACap identifies missing, ambiguous, or unsupported claims and selectively invokes tools such as spatial cropping, temporal sampling, OCR, and external retrieval to acquire additional evidence. The caption is iteratively revised based on the collected observations.
The paper also introduces FSCap-Bench, a benchmark containing 498 videos with human-verified Core and Support facts. Two evaluation metrics are proposed: the Core-Support Factuality Score (CSFS), which combines factual precision and importance-aware coverage, and the Cross-Modal Semantic Round-Trip Score (CSRS), which evaluates information preservation through a video-caption-video-caption reconstruction cycle.
Experiments demonstrate improvements in caption-based video question answering across multiple backbones. Controlled caption perturbations further show that CSFS responds to correct additions, hallucinations, and omissions in the expected direction.
Critical strengths
-
A useful formulation of detailed captioning as active evidence acquisition. The paper identifies a meaningful distinction between question-driven video understanding and open-ended detailed captioning. Unlike VideoQA, detailed captioning must determine which information is worth observing without receiving a specific question. OACap addresses this problem by treating the evolving caption as both the output and the control state of the observation process. The distinction between information gaps that can be resolved through revision and those requiring additional observation is conceptually useful. The framework also supports explicit evidence provenance, which could improve the interpretability and reliability of detailed video descriptions.
-
A complementary evaluation framework for factuality and semantic coverage. FSCap-Bench addresses limitations of conventional captioning metrics by evaluating atomic factual claims and distinguishing central video information from supplementary details. The controlled perturbation experiments provide encouraging evidence that CSFS responds appropriately to hallucinations and omissions. The caption-as-memory evaluation is also valuable because the downstream question-answering model receives only the generated caption, with the reader held fixed across comparisons. This helps assess whether the caption preserves reusable video information rather than merely producing a fluent description.
Critical weaknesses
-
The effectiveness of caption-guided observation is not isolated from additional inference computation. The main experiments compare OACap against single-pass captioning using matched model backbones. However, OACap performs additional observation, verification, and revision steps, potentially consuming substantially more inference computation than the baselines. The paper does not provide a controlled comparison against simpler alternatives using equivalent computational budgets, such as uniform dense frame sampling, repeated whole-video captioning, or multi-pass refinement without adaptive tool selection. This is a central limitation because the claimed contribution is not merely that additional observations improve captions, but that caption-guided selection of observations provides a better strategy. Without compute-matched comparisons, the experiments cannot determine how much of the improvement originates from the proposed observation policy rather than increased inference effort.
-
The benchmark validation does not fully establish the reliability or distinctiveness of the proposed evaluation metrics. Although the controlled perturbation experiments show that CSFS responds in the expected direction, these tests primarily establish sensitivity to predefined modifications. They do not establish whether the metric agrees with independent human judgments of caption quality across naturally generated descriptions. The CSRS evaluation introduces an additional limitation. The reported scores range approximately from 0.20 to 0.29, indicating substantial information loss through the reconstruction cycle. Moreover, CSRS depends on the text-to-video generator, the recaptioning model, the atomic-fact extractor, and the semantic matcher. As a result, it remains difficult to determine whether CSRS meaningfully measures the quality of the original caption or primarily reflects the limitations of the reconstruction pipeline. The evaluation also covers only 33 videos for CSRS, while FSCap-Bench itself is constructed through filters that explicitly favor cases requiring targeted observation. These design choices limit the conclusions that can be drawn about general caption quality.
Questions for authors
- Can the authors compare OACap against a compute-matched baseline that uses the same backbone and a comparable number of visual observations and model calls, but selects additional frames through uniform or non-adaptive sampling? A comparison with iterative caption refinement without targeted observation would also help isolate the contribution of the proposed observation policy.
2.How strongly do CSFS and CSRS correlate with independent human evaluations of factual accuracy, semantic completeness, and temporal consistency? In particular, can the authors demonstrate that CSRS distinguishes better captions from worse captions when the text-to-video generator and other reconstruction components are held fixed? Clarification of possible video overlap between FSCap-Bench and the downstream benchmarks would also strengthen the evaluation.
4 comments
Public Policy Isolation, Natural-Sample Human Validity, Canonical CSRS and Complementary RT-F1 – Part2
Canonical CSRS definition and the fixed-channel check. We thank the reviewer for identifying the equation/figure mismatch. It arose from an obsolete early definition that remained in Eq. 9, Appendix B.2, and Table 4 by oversight. The definition in Figures 1/4 is canonical and source-grounded: after , we atomize and evaluate its claims against the original video's human-verified Core/Support facts. Thus , with where , and are recovered-claim precision, weighted Core coverage and Support coverage, and the same and zero-denominator rules as CSFS are used. The earlier Eq. 9, , measures symmetric – agreement and corresponds to RT-F1. We have relabeled those legacy values as RT-F1; they are not CSRS. The source-grounded scores under the figure definition are the correct CSRS values and are retained.
For the fixed-channel question itself, RT-F1 remains a useful auxiliary diagnostic. We freeze LTX-2.5-60s generation, Qwen3.8-27B re-captioning, atomization, and Kimi matching. For degraded, single-A3B, single-27B, OACap, and gold captions, RT-F1 is 0.198/0.241/0.254/0.326/0.348, respectively. OACap−single-27B is +0.072 [0.061,0.083], with 84.8% pairwise wins. Swapping the generator, re-captioner, or matcher gives +0.057/+0.076/+0.072, respectively; the ordering is preserved (–0.98), although absolute levels shift. The RT-F1 study of 120 videos and 600 captions reuses the shared validation frame, and only the original 33-video exploration uses a separate sample. RT-F1 therefore complements, but is not numerically interchangeable with, source-grounded CSRS.
Natural-source validation and overlap. The shared 120-video validation frame comprises 40 post-filter FSCap videos and 80 disjoint prefilter videos sampled before F1–F4. For CSFS human validity, three blind judges rated 200 captions from the 40-video FSCap stratum across five tiers, and the natural-caption analysis uses the 120 captions from the three natural tiers. Rater-reliability, decomposition, and RT-F1 analyses use the full shared frame. Video-level overlap is 60/900 for Video-MME, 79/800 for ActivityNet-QA, and 228/5,705 for VCapsBench. Removing the 60 overlapping Video-MME videos leaves 840 videos and gives +4.89, compared with +4.87 from the unrounded overall scores. Duration scope is also explicit: FSCap-Bench is challenge-enriched rather than representative of YouTube's natural duration distribution (median 14.7s; 433/498 videos under two minutes). The direct LVChat protocols over 2–60 minutes provide a separate duration-generalization test.
The full serving-cost table appears in our response to Reviewer 74ut, and additional seed, annotation, and long-video details appear in our response to Reviewer 4Yx7. If these additions adequately address your concerns, we would be grateful if you would consider re-evaluating the paper's score.
The authors have adequately addressed the concerns raised in my review, and I have therefore increased my score. I hope these improvements will be properly reflected in the final version of the paper.
Thank You for the Helpful Feedback and Re-evaluation
Dear Reviewer FmHD, Thank you very much for your thorough review and for recognizing our revisions. We are grateful for your insightful feedback, which has significantly strengthened the experimental rigor and evaluation methodology of our paper. We will ensure all discussed improvements are fully incorporated into the final version.
Public Policy Isolation, Natural-Sample Human Validity, Canonical CSRS and Complementary RT-F1
Dear Reviewer FmHD,
Thank you for identifying the missing controls and validation. We address your two questions with: (1) compute-matched non-adaptive, refinement, and public-policy controls that isolate caption-guided observation; and (2) same-item human validation, a corrected source-grounded CSRS, fixed-channel RT-F1 checks, natural-output analyses, and overlap analyses.
Q1: Isolating policy from compute, adaptivity, and length. AdaFrame (Wu et al., CVPR 2019) and VideoAgent (Wang et al., ECCV 2024) retain their released selectors and visual/memory state; neither uses an evolving caption. Only the VLM/tool interface is adapted to Qwen3.8-27B. Two thresholds are frozen using 50 videos from the existing training split (12 validation trials), disjoint from FSCap and all test sets. The learned caption-free controller also uses training splits only.
Under approximately matched budgets, OACap exceeds VideoAgent and the learned caption-free controller by +5.39/+5.06 CSFS, respectively; VideoAgent with released defaults scores 80.72. A fixed caption-independent checklist scores 79.64 with 3.1 tools, 64 frames, and 30.2K controller tokens. Removing the revision/observation split gives 83.11 with 4.6 tools, compared with 87.24 for full OACap. These controls support the contribution of both caption-gap state and the split. Canonical CSRS independently evaluates source-grounded round-trip recovery. Concise OACap retains +18.62 over the 415-word baseline, while the 681-word single-pass degrades. For Qwen3.8-27B, P/Rcore/Rsup changes from 88.5/59.9/49.8→90.8/88.0/81.6, with +16.4 verified claims per video. The matched +5.06 contrast isolates the recovery of missing and supporting facts through caption-gap routing.
Complete-workflow reproductions. We additionally ran the released LLaVA-Video and VDC-Agent workflows as external references, rather than comparing only with simplified controls. These are complete-workflow references, not matched-backbone ablations:
On VCapsBench, OACap/Qwen3.8-27B (AR/IR/CR=70.24/21.36/89.32) improves AR by +6.13/+19.05 and CR by +14.37/+29.43 over LLaVA-Video/VDC-Agent, respectively. Its IR is higher by 6.89/6.83 points (21.36 versus 14.47/14.53), so we claim better answer recall and coverage, not uniform superiority on every metric.
Q2: Human validity and round-trip evaluation. Three blind judges rated the same 200 captions (40 stratified videos × five tiers) for factual, semantic, temporal, and overall quality (α=0.81). Every metric is evaluated on identical items:
CSFS obtains 91.0% pairwise agreement and partial ρ=0.76 after controlling for VDCScore, FactVC, and caption length. Adding CSFS raises R2 from 0.41 to 0.79 (p<10−12). When the analysis is restricted to the 120 naturally generated captions (single A3B, single 27B, and OACap, excluding synthetic and gold tiers), CSFS retains ρ=0.83, 88.3% pairwise accuracy, and partial ρ=0.69 given VDCScore. Fixed-channel RT-F1 gives ρ=0.66 and 76.7% pairwise accuracy. The correlation is therefore not driven by artificial extremes.