acceptodds
Under review as a conference paper at ICLR 2027

OACap: Observation-Adaptive Video Captioning via Caption-Centered Agentic Evidence Grounding

Abstract

Dense video captioning offers a rich textual interface to video understanding and generation. Recent advances in multimodal large language models have substantially improved fine-grained video perception and long-form text generation, laying a strong foundation for dense video captioning. However, existing methods still face two major challenges: achieving comprehensive coverage of fine-grained video content, and maintaining factual grounding and global semantic coherence as captions grow longer and more detailed. To this end, we propose Observation-Adaptive Video Captioner (OACap), an agentic model that learns to think with video. Rather than relying on a fixed perceptual context, OACap treats the evolving caption itself as the state of reasoning: it identifies missing, unsupported, or semantically inconsistent content, and adaptively acquires additional visual, temporal, or external evidence to verify and refine the description.

Primary area:
datasets and benchmarks
Keywords:
Video Dense Captioning, Evidence Grounding, Active Visual Perception, Long Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FmHDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OACap, an agent-based framework for detailed video captioning that uses the evolving caption to guide additional visual observations. Rather than generating a description through a single global pass, OACap identifies missing, ambiguous, or unsupported claims and selectively invokes tools such as spatial cropping, temporal sampling, OCR, and external retrieval to acquire additional evidence. The caption is iteratively revised based on the collected observations.

The paper also introduces FSCap-Bench, a benchmark containing 498 videos with human-verified Core and Support facts. Two evaluation metrics are proposed: the Core-Support Factuality Score (CSFS), which combines factual precision and importance-aware coverage, and the Cross-Modal Semantic Round-Trip Score (CSRS), which evaluates information preservation through a video-caption-video-caption reconstruction cycle.

Experiments demonstrate improvements in caption-based video question answering across multiple backbones. Controlled caption perturbations further show that CSFS responds to correct additions, hallucinations, and omissions in the expected direction.

Critical strengths
  1. A useful formulation of detailed captioning as active evidence acquisition. The paper identifies a meaningful distinction between question-driven video understanding and open-ended detailed captioning. Unlike VideoQA, detailed captioning must determine which information is worth observing without receiving a specific question. OACap addresses this problem by treating the evolving caption as both the output and the control state of the observation process. The distinction between information gaps that can be resolved through revision and those requiring additional observation is conceptually useful. The framework also supports explicit evidence provenance, which could improve the interpretability and reliability of detailed video descriptions.

  2. A complementary evaluation framework for factuality and semantic coverage. FSCap-Bench addresses limitations of conventional captioning metrics by evaluating atomic factual claims and distinguishing central video information from supplementary details. The controlled perturbation experiments provide encouraging evidence that CSFS responds appropriately to hallucinations and omissions. The caption-as-memory evaluation is also valuable because the downstream question-answering model receives only the generated caption, with the reader held fixed across comparisons. This helps assess whether the caption preserves reusable video information rather than merely producing a fluent description.

Critical weaknesses
  1. The effectiveness of caption-guided observation is not isolated from additional inference computation. The main experiments compare OACap against single-pass captioning using matched model backbones. However, OACap performs additional observation, verification, and revision steps, potentially consuming substantially more inference computation than the baselines. The paper does not provide a controlled comparison against simpler alternatives using equivalent computational budgets, such as uniform dense frame sampling, repeated whole-video captioning, or multi-pass refinement without adaptive tool selection. This is a central limitation because the claimed contribution is not merely that additional observations improve captions, but that caption-guided selection of observations provides a better strategy. Without compute-matched comparisons, the experiments cannot determine how much of the improvement originates from the proposed observation policy rather than increased inference effort.

  2. The benchmark validation does not fully establish the reliability or distinctiveness of the proposed evaluation metrics. Although the controlled perturbation experiments show that CSFS responds in the expected direction, these tests primarily establish sensitivity to predefined modifications. They do not establish whether the metric agrees with independent human judgments of caption quality across naturally generated descriptions. The CSRS evaluation introduces an additional limitation. The reported scores range approximately from 0.20 to 0.29, indicating substantial information loss through the reconstruction cycle. Moreover, CSRS depends on the text-to-video generator, the recaptioning model, the atomic-fact extractor, and the semantic matcher. As a result, it remains difficult to determine whether CSRS meaningfully measures the quality of the original caption or primarily reflects the limitations of the reconstruction pipeline. The evaluation also covers only 33 videos for CSRS, while FSCap-Bench itself is constructed through filters that explicitly favor cases requiring targeted observation. These design choices limit the conclusions that can be drawn about general caption quality.

Questions for authors
  1. Can the authors compare OACap against a compute-matched baseline that uses the same backbone and a comparable number of visual observations and model calls, but selects additional frames through uniform or non-adaptive sampling? A comparison with iterative caption refinement without targeted observation would also help isolate the contribution of the proposed observation policy.

2.How strongly do CSFS and CSRS correlate with independent human evaluations of factual accuracy, semantic completeness, and temporal consistency? In particular, can the authors demonstrate that CSRS distinguishes better captions from worse captions when the text-to-video generator and other reconstruction components are held fixed? Clarification of possible video overlap between FSCap-Bench and the downstream benchmarks would also strengthen the evaluation.

4 comments
AuthorsOpenReview ↗

Public Policy Isolation, Natural-Sample Human Validity, Canonical CSRS and Complementary RT-F1

Dear Reviewer FmHD,

Thank you for identifying the missing controls and validation. We address your two questions with: (1) compute-matched non-adaptive, refinement, and public-policy controls that isolate caption-guided observation; and (2) same-item human validation, a corrected source-grounded CSRS, fixed-channel RT-F1 checks, natural-output analyses, and overlap analyses.

Q1: Isolating policy from compute, adaptivity, and length. AdaFrame (Wu et al., CVPR 2019) and VideoAgent (Wang et al., ECCV 2024) retain their released selectors and visual/memory state; neither uses an evolving caption. Only the VLM/tool interface is adapted to Qwen3.8-27B. Two thresholds are frozen using 50 videos from the existing training split (12 validation trials), disjoint from FSCap and all test sets. The learned caption-free controller also uses training splits only.

MethodCallsToolsFramesController tokensWordsCSFS ↑
Single-pass1.00.03211.2K41566.50
Verbose single-pass1.00.03212.1K68165.92
Uniform dense sampling1.00.09631.5K46270.85
Repeated whole-video caption + merge4.00.09633.2K69072.11
Self-refine, no new evidence4.00.03228.4K53166.81
Fixed non-adaptive schedule4.13.164.030.4K64277.42
Public AdaFrame selector3.63.362.830.1K67981.40
Public VideoAgent3.53.263.230.0K67581.85
Learned caption-free controller3.53.163.529.9K67782.18
OACap concise3.43.164.029.1K42285.12
OACap full3.43.164.029.8K68587.24

Under approximately matched budgets, OACap exceeds VideoAgent and the learned caption-free controller by +5.39/+5.06 CSFS, respectively; VideoAgent with released defaults scores 80.72. A fixed caption-independent checklist scores 79.64 with 3.1 tools, 64 frames, and 30.2K controller tokens. Removing the revision/observation split gives 83.11 with 4.6 tools, compared with 87.24 for full OACap. These controls support the contribution of both caption-gap state and the split. Canonical CSRS independently evaluates source-grounded round-trip recovery. Concise OACap retains +18.62 over the 415-word baseline, while the 681-word single-pass degrades. For Qwen3.8-27B, P/Rcore/RsupP/R_{\mathrm{core}}/R_{\mathrm{sup}} changes from 88.5/59.9/49.8→90.8/88.0/81.6, with +16.4 verified claims per video. The matched +5.06 contrast isolates the recovery of missing and supporting facts through caption-gap routing.

Complete-workflow reproductions. We additionally ran the released LLaVA-Video and VDC-Agent workflows as external references, rather than comparing only with simplified controls. These are complete-workflow references, not matched-backbone ablations:

Complete workflowVideo-MME ↑VCapsBench AR ↑IR ↓CR ↑
LLaVA-Video (reproduced)60.41%64.1114.4774.95
VDC-Agent (reproduced)58.74%51.1914.5359.89

On VCapsBench, OACap/Qwen3.8-27B (AR/IR/CR=70.24/21.36/89.32) improves AR by +6.13/+19.05 and CR by +14.37/+29.43 over LLaVA-Video/VDC-Agent, respectively. Its IR is higher by 6.89/6.83 points (21.36 versus 14.47/14.53), so we claim better answer recall and coverage, not uniform superiority on every metric.

Q2: Human validity and round-trip evaluation. Three blind judges rated the same 200 captions (40 stratified videos × five tiers) for factual, semantic, temporal, and overall quality (α=0.81\alpha=0.81). Every metric is evaluated on identical items:

MetricFactual ρ\rhoSemantic ρ\rhoTemporal ρ\rhoOverall ρ\rho
VDCScore-Qwen0.590.640.550.62
RT-F1, fixed channel0.560.740.680.71
CSFS0.890.870.820.88

CSFS obtains 91.0% pairwise agreement and partial ρ=0.76\rho=0.76 after controlling for VDCScore, FactVC, and caption length. Adding CSFS raises R2R^2 from 0.41 to 0.79 (p<10−12p<10^{-12}). When the analysis is restricted to the 120 naturally generated captions (single A3B, single 27B, and OACap, excluding synthetic and gold tiers), CSFS retains ρ=0.83\rho=0.83, 88.3% pairwise accuracy, and partial ρ=0.69\rho=0.69 given VDCScore. Fixed-channel RT-F1 gives ρ=0.66\rho=0.66 and 76.7% pairwise accuracy. The correlation is therefore not driven by artificial extremes.

AuthorsOpenReview ↗

Public Policy Isolation, Natural-Sample Human Validity, Canonical CSRS and Complementary RT-F1 – Part2

Canonical CSRS definition and the fixed-channel check. We thank the reviewer for identifying the equation/figure mismatch. It arose from an obsolete early definition that remained in Eq. 9, Appendix B.2, and Table 4 by oversight. The definition in Figures 1/4 is canonical and source-grounded: after V→C→V′→C′V\rightarrow C\rightarrow V'\rightarrow C', we atomize C′C' and evaluate its claims against the original video's human-verified Core/Support facts. Thus CSRS(C∣V)=CSFS(C′∣V)CSRS(C\mid V)=CSFS(C'\mid V), with CSRS=2λP′Ccore′P′+Ccore′+(1−λ)S′,λ=0.8,CSRS=\frac{2\lambda P'C'_{\mathrm{core}}}{P'+C'_{\mathrm{core}}}+(1-\lambda)S',\qquad \lambda=0.8, where P′P', Ccore′C'_{\mathrm{core}} and S′S' are recovered-claim precision, weighted Core coverage and Support coverage, and the same λ\lambda and zero-denominator rules as CSFS are used. The earlier Eq. 9, 2∣M∣/(∣A∣+∣B∣)2|M|/(|A|+|B|), measures symmetric CC–C′C' agreement and corresponds to RT-F1. We have relabeled those legacy values as RT-F1; they are not CSRS. The source-grounded scores under the figure definition are the correct CSRS values and are retained.

For the fixed-channel question itself, RT-F1 remains a useful auxiliary diagnostic. We freeze LTX-2.5-60s generation, Qwen3.8-27B re-captioning, atomization, and Kimi matching. For degraded, single-A3B, single-27B, OACap, and gold captions, RT-F1 is 0.198/0.241/0.254/0.326/0.348, respectively. OACap−single-27B is +0.072 [0.061,0.083], with 84.8% pairwise wins. Swapping the generator, re-captioner, or matcher gives +0.057/+0.076/+0.072, respectively; the ordering is preserved (ρ=0.94\rho=0.94–0.98), although absolute levels shift. The RT-F1 study of 120 videos and 600 captions reuses the shared validation frame, and only the original 33-video exploration uses a separate sample. RT-F1 therefore complements, but is not numerically interchangeable with, source-grounded CSRS.

Natural-source validation and overlap. The shared 120-video validation frame comprises 40 post-filter FSCap videos and 80 disjoint prefilter videos sampled before F1–F4. For CSFS human validity, three blind judges rated 200 captions from the 40-video FSCap stratum across five tiers, and the natural-caption analysis uses the 120 captions from the three natural tiers. Rater-reliability, decomposition, and RT-F1 analyses use the full shared frame. Video-level overlap is 60/900 for Video-MME, 79/800 for ActivityNet-QA, and 228/5,705 for VCapsBench. Removing the 60 overlapping Video-MME videos leaves 840 videos and gives +4.89, compared with +4.87 from the unrounded overall scores. Duration scope is also explicit: FSCap-Bench is challenge-enriched rather than representative of YouTube's natural duration distribution (median 14.7s; 433/498 videos under two minutes). The direct LVChat protocols over 2–60 minutes provide a separate duration-generalization test.

The full serving-cost table appears in our response to Reviewer 74ut, and additional seed, annotation, and long-video details appear in our response to Reviewer 4Yx7. If these additions adequately address your concerns, we would be grateful if you would consider re-evaluating the paper's score.

The authors have adequately addressed the concerns raised in my review, and I have therefore increased my score. I hope these improvements will be properly reflected in the final version of the paper.

AuthorsOpenReview ↗

Thank You for the Helpful Feedback and Re-evaluation

Dear Reviewer FmHD, Thank you very much for your thorough review and for recognizing our revisions. We are grateful for your insightful feedback, which has significantly strengthened the experimental rigor and evaluation methodology of our paper. We will ensure all discussed improvements are fully incorporated into the final version.

Reviewer_4Yx7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a problem specific to detailed video captioning. Unlike VideoQA, open-ended captioning has no query that tells the model what evidence to gather, so the model has to decide for itself what to observe and how to merge those observations into a coherent global description. The authors propose OACap (Observation-Adaptive Video Captioner), which uses the evolving caption itself as the control signal. The current draft CnC_n is treated as a testable hypothesis, and coverage checks plus claim verification diagnose four kinds of gaps, Gn=Gnmiss∪Gnunsupported∪Gnconflict∪GntemporalG_n = G^{miss}_n \cup G^{unsupported}_n \cup G^{conflict}_n \cup G^{temporal}_n. The method then separates a revision gap (existing evidence is enough to fix it, so the caption is rewritten directly) from an observation gap (the evidence cannot decide between candidate descriptions, so one targeted tool call is issued: crop, dense temporal sampling, OCR, image retrieval, or web search). Returned evidence is normalized into tuples that carry a time interval, modality, source, confidence, and verification status, and these tuples drive a rewrite of the whole caption. The authors also introduce FSCap-Bench. Four filtering stages applied to five public benchmarks yield 498 videos and 996 verified challenges, with human-annotated, importance-stratified Core / Support facts. The benchmark comes with two metrics: CSFS, the harmonic mean of claim precision and importance-weighted Core coverage (λ=0.8), and CSRS, which measures semantic recoverability through a V→C→V′→C′V \to C \to V' \to C' round trip. The experiments cover caption-as-memory (a fixed text-only reader, with abstentions counted as errors), a validation of CSFS under four types of controlled perturbations, and a comparison against direct captioning with the same backbone.

Critical strengths
  1. The problem is real, and the framing differs in substance from existing agentic video work. Existing tool-augmented video methods usually trigger extra observation from an external query or a reasoning state. This paper instead uses the gaps in the evolving caption as the trigger. The design principle that "every tool call is tied to a possible caption change rather than a generic request for more detail" (§3.2) is concrete and testable, so it amounts to more than packaging.

  2. The key choices in the evaluation protocol are sound. The captioning stage never sees the downstream questions, the reader is fixed to one text-only model, and when the reader cannot answer it must abstain, with abstentions counted as errors. The authors also state that the reader's priors cannot be fully removed. This keeps "does the caption retain reusable information" apart from "did the observation stage peek at the questions", and it is the most rigorous piece of experimental design in the paper.

  3. CSFS targets a real pain point in caption evaluation. Taking the harmonic mean of importance-weighted Core coverage and precision deals head-on with short captions that score well on precision alone and long captions that pad unweighted coverage. The importance weights are assigned before candidate captions are scored, which is the right order.

  4. The authors check the direction in which the metric moves instead of reporting a single score. On 300 videos they build four types of controlled perturbations (+correct detail / +hallucination / Core deletion / Support deletion), 1500 captions in total, and compare CSFS with 10 existing metrics (PAC-S, EMScore, FactVC, BLEU-4, CIDEr-D, BERTScore, ARGUS, VDCScore, CapST, etc.). The video-level directional accuracy is 97.3% / 98.3% / 93.0% / 93.3%. This is the table with the strongest evidence in the paper.

  5. The authors are candid about the limits of their own metrics. They state that CSRS measures CC–C′C' consistency and not correctness with respect to VV (a hallucination repeated at both ends can still match), that the score mixes four sources of error (generator, re-captioner, atomizer, matcher), and that no human–machine agreement study was run for the judge. The pipeline also uses SHA-256 caching with hash checks to prevent mismatches across captioners, the F1–F4 funnel arithmetic is internally consistent (10,716 → 3,071 → 1,099 → 897 → 498), and retention rates are reported for every source.

Critical weaknesses

W1 (Decisive) The gains cannot be attributed: "smart observation" cannot be told apart from "observing a few more times", and the contribution of external retrieval is not isolated

OACap claims that what to observe is decided by the caption's gaps, so it should work better than indiscriminately observing a few more times, and its value should show up in efficiency. Yet the paper reports only quality and no cost at all:

  • Number of observations, iterations, and tool calls: not reported anywhere in the paper. The state tuple Sn=(En,Cn,Gn,Hn,Bn)S_n=(E_n,C_n,G_n,H_n,B_n) has a budget field and a stopping condition for when the budget is exhausted, but the paper never gives the budget's value, its unit, or how it is set.
  • The cost promise in §5 is not kept. §5 explicitly says "Observation cost is reported separately because equal caption length does not imply equal inference compute", yet neither the main text nor the appendix gives latency, token counts, wall-clock time, or any cost table.
  • There is no equal-compute comparison of any kind. There is no equal-compute uniform sampling and no tool-free multi-round self-refine. The authors say so themselves in the Limitations: "do not establish a quality–cost Pareto frontier against equal-compute uniform sampling or alternative multi-stage policies".
  • The contribution of external retrieval is not isolated either. The leakage safeguards in §3.3 are only stated, with no ablation that turns off web search and no attribution by source. The lead example itself uses a 2011 Emirates News Agency report to support the time and place of the "Dubai New Year's Eve fireworks", which puts outside news into the final description.

Readers therefore cannot rule out the explanation that the gains come from looking many more times and calling tools a few more times, and they cannot tell how much of the CSFS gain comes from external knowledge rather than from observing the video.

Requested from the authors: (1) the average number of observations, number of tool calls, total tokens, and latency per video; (2) comparisons with "equal-compute uniform frame sampling" and with "equal-compute tool-free multi-round self-refine"; (3) the exact budget setting and how sensitive the results are to it; (4) an ablation with web search turned off, and attribution by source.

W2 (Decisive) The evaluation framework has no human verification, so the conclusions of the main tables are not well supported

FSCap-Bench is one of the paper's two main contributions, and CSFS carries every conclusion in Table 3. Yet the key stages of this evaluation lack the most basic human quality assurance:

  • There is no human agreement study. The number of annotators, their qualifications, the adjudication protocol for the Core/Support split, and whether the verifier in the F4 filter is a person or a model are all left unstated, and the paper contains no κ or agreement-rate statistics at all. The authors acknowledge that they "validate directional sensitivity, but not human agreement or invariance to alternative atomic decompositions".
  • The scoring components are anonymous and unvalidated. The paper calls the CSFS judge only "a frozen judge", with no model name, version, or checkpoint, and the judge's prompt is not given. The matcher behind CSRS (Kimi-K3-SPOT) is mentioned only once, in the appendix, with no version or citation, and its own quality is not validated.
  • The video-level overlap audit was not carried out. The Limitations say: "Because FSCap-Bench draws from datasets also used in the downstream evaluation, claims of independent generalization require a video-level overlap audit". That audit was not done.
  • Statistical support is weak. The paper reports no random seeds, error bars, confidence intervals, or significance tests. Neither Table 1 nor Table 3 reports the number of evaluation samples. In Table 3 the three backbones gain +1.94 / +5.20 / +20.75 under the same pipeline. This is the largest single effect in the paper, yet there is no per-video distribution and no explanation.

When the right-or-wrong decisions behind a metric come from an anonymous model whose agreement with human judgment is unknown, the absolute values of CSFS cannot be used as comparable quantities.

Requested from the authors: (1) annotator information and agreement rates, plus a study of how CSFS correlates with human judgment; (2) the judge's model name, version, and prompt, and the matcher's version; (3) the video-level overlap audit; (4) error bars over multiple random seeds, sample sizes for Table 1 and Table 3, and the cause of the +20.75 gain.

W3 (Major) The problem is stated as general video captioning, but the evaluation covers only short videos

The paper frames its problem as general "detailed video captioning" and cites work on long-form and agentic long-video understanding (VideoAgent, Active Video Perception), yet the median video length in the benchmark is only 14.7 seconds, and 433/498 videos are under 2 minutes. On videos like these, "active observation" reduces to filling in close-range details instead of maintaining consistency over long spans. Cross-shot subject consistency, a core long-video problem, is not discussed anywhere in the paper: the evaluation dimensions are the factual correctness of individual claims and their temporal relations, and consistency of identity, appearance, or state across shots is not covered. The result is a mismatch between scope and claims: the paper positions its problem in the language of long-video understanding, but the evidence covers only short videos.

Requested from the authors: explain how OACap keeps descriptions consistent on genuinely long videos (for example, videos that span several scenes, or where a subject's appearance changes), or state explicitly that the paper's claims are limited to short videos.

Questions for authors
  1. On average, how many observations and how many tool calls does each video trigger, and what are the total tokens and the end-to-end latency? How is the budget set?
  2. Compared with equal-compute uniform frame sampling and with equal-compute tool-free multi-round self-refine, what does OACap's quality–cost curve look like? If the latter reaches a similar CSFS with the same compute, does the method's central claim still hold?
  3. Which model is the CSFS judge? Is there a human correlation study, even a small one (for example, 50–100 pairs)? How many annotators produced the Core/Support fact annotations, and what is their agreement rate?
  4. With web search turned off, how much do CSFS and the Table 1 accuracy each drop? How many scored credits come from retrieved evidence?
  5. In Table 3, the +20.75 gain for Qwen3.8-27B is several times the +1.94 / +5.20 of the other two backbones. What causes this? Is it a difference in configuration?
  6. How does OACap perform on long-video problems such as cross-shot subject consistency? If this was not evaluated, should the paper's claims be narrowed to short videos?
  7. What fraction of FSCap-Bench videos overlap at the video level with the Video-MME / ActivityNet-QA / VCapsBench evaluation sets? (The paper notes in the Limitations that this audit is needed, but it was not carried out.)
2 comments
AuthorsOpenReview ↗

Controls, Cost, Human Validation, and Long Videos

Dear Reviewer 4Yx7,

Thank you for the detailed concerns. We address your questions in three groups: (1) controller/serving costs, compute-matched controls, retrieval attribution, and the Qwen gain (Q1/Q2/Q4/Q5); (2) annotators, judges, human agreement, seeds, and overlap (Q3/Q7); and (3) direct cross-shot generalization on long videos (Q6).

Q1, Q2, Q4 & Q5: Cost, controlled attribution, and retrieval. AdaFrame and VideoAgent retain their released selector logic and visual/memory state; only the VLM/tool interface is adapted. All rows use Qwen3.8-27B. With released defaults and no test calibration, AdaFrame, VideoAgent, and AVP score 80.96/80.72/81.27, respectively, compared with 87.24 for frozen OACap (+6.28/+6.52/+5.97). For the matched rows, two thresholds are frozen on 50 videos from the existing training split, disjoint from FSCap and all test sets.

MethodToolsFramesController tokensTimeWordsCSFS ↑
Single-pass0.03211.2K14.2s41566.50
Uniform dense0.09631.5K36.8s46270.85
Self-refine (no new evidence)0.03228.4K32.1s53166.81
Fixed schedule3.164.030.4K37.5s64277.42
Public VideoAgent3.263.230.0K43.8s67581.85
Learned caption-free ctrl.3.163.529.9K36.8s67782.18
OACap concise3.164.029.1K39.5s42285.12
OACap w/o web+image retrieval2.96428.9K42.7s67886.78
OACap full3.164.029.8K44.9s68587.24

For Qwen3.8-27B, the gain is recall-driven: P/Rcore/RsupP/R_{\mathrm{core}}/R_{\mathrm{sup}} changes from 88.5/59.9/49.8→90.8/88.0/81.6, with +16.4 verified claims per video. The ordered controls give 66.50→70.85(+4.35)→77.42(+6.57)→82.18(+4.76)→87.24(+5.06). Their displayed endpoint difference is +20.74 (=87.24−66.50), whereas Table 3 reports +20.75 because it uses unrounded endpoints. These ordered matched-trace contrasts are not independent additive effects. The aligned contrasts are:

Backbone endpoint: single→full [95% CI]Fixed-trace componentCaption-gap componentInitial Core headroom
B1 (Table 3 order): +1.94 [1.41,2.47]+3.12+3.849.8%
B2 (Table 3 order): +5.20 [4.58,5.82]+4.08+4.3715.7%
Qwen3.8-27B:+20.75 [19.72,21.78]+6.57+5.0640.1%

The component increments use frozen matched traces and are not expected to sum to the end-to-end gains. For Kmax⁡={0,1,3,5,8}K_{\max}=\{0,1,3,5,8\}, CSFS is 66.50→76.41→84.89→87.24→87.64, and controller tokens are 11.2K→17.4K→25.1K→29.8K→33.5K.

End-to-end API cost is reported separately from controller calls. Across Qwen3.5-27B, Qwen3.5-122B-A10B, Qwen3.8-27B, and Kimi-K2.6, Direct averages 1.00 request, 21.51–83.06K input tokens, 0.70K output tokens, and ¥0.0250–0.0866 per video. OACap averages 37.07 requests, 232.77–589.80K input tokens, 46.79K output tokens, and ¥0.4667–0.8238. Normalized to 15 seconds, the Direct/OACap values are 0.19/6.94 requests, 4.03–15.55K/43.57–110.40K input tokens, 0.13K/8.76K output tokens, and ¥0.0047–0.0162/¥0.0874–0.1542. These serving figures include internal subrequests and retries under complete backbone-specific input accounting. They are therefore not directly comparable to the controller-visible token totals above. We explicitly acknowledge OACap's additional cost rather than presenting controller rounds as total serving cost.

Removing web and image retrieval decreases CSFS by 0.46 and Video-MME Short/Medium/Long by 0.22/0.25/0.12. The sources of scored credits are 97.8% video-native, 1.5% image, and 0.7% web. The default configuration uses 2.4 observation rounds, 3.1 tool calls, Nmax⁡=3N_{\max}=3, Kmax⁡=5K_{\max}=5, and frozen stopping; concise OACap controls caption length.

Q3 & Q7: Human validation, reproducibility, and overlap. Six trained researchers built FSCap-Bench: five independent annotators label each video, and one adjudicator resolves disagreements. All 498 items pass F4 without a final VLM verdict. Agreement statistics cover all 498 videos: Fleiss' κ=0.84\kappa=0.84 for validity and 0.79 for Core/Support, with mean pairwise temporal IoU=0.86. The shared 120-video validation frame comprises 40 post-filter FSCap videos and 80 disjoint prefilter videos. Three blind judges rated the 200 captions in its 40-video FSCap stratum (five tiers; α=0.81\alpha=0.81). CSFS obtains r/ρ/τ=0.89/0.88/0.73r/\rho/\tau=0.89/0.88/0.73 and 91.0% pairwise accuracy; 1,850 claims give 93.4% agreement (κ=0.86\kappa=0.86). These studies address agreement and correlation; judge versions, seeds, and overlap follow below.

AuthorsOpenReview ↗

Controls, Cost, Human Validation, and Long Videos – Part2

CSFS uses Qwen3.8-Max (qwen3.8-max-202607, temperature 0). The legacy symmetric round-trip diagnostic, RT-F1, uses the Qwen atomizer and Kimi-K3-SPOT matcher (kimi-k3-spot-202606). Canonical CSRS instead applies CSFS to C′C' against source-video gold facts. Appendix C documents the key-frame, evidence, segment-caption, verification, fusion, and summary prompts, together with runtime assembly. The evaluation release will include the blinded atomic-support, Core-coverage, harmonic-scoring, and abstention prompt. Independent atomization and human re-decomposition of all 600 captions in the shared frame give ρ=0.86\rho=0.86/89.7% pairwise. A three-person rater-reliability study on the same 600 captions gives ρ=0.84\rho=0.84, 88.0% pairwise accuracy, and partial ρ=0.70\rho=0.70 given VDCScore.

Table 1 evaluates 900 Video-MME videos (2,700 QAs), 800 ActivityNet-QA videos, and 5,705 VCapsBench videos. Table 3 uses 498 videos and 996 challenges per backbone. Every backbone in Tables 1 and 3 uses seeds 42/123/2026, with SD≤0.24 on Video-MME and ≤0.38 on CSFS. Video-level overlap with FSCap is 60/900 for Video-MME, 79/800 for ActivityNet-QA, and 228/5,705 for VCapsBench. Removing the 60 overlapping Video-MME videos leaves 840 videos and gives +4.89, compared with +4.87 from the unrounded overall scores; the displayed split macro-average rounds to +4.88.

Q6: Direct cross-shot generalization evidence. We evaluated 200 LVChat videos (120 at 2–10 minutes and 80 at 10–30 minutes) plus a disjoint set of 40 videos at 30–60 minutes. These are direct long-video evaluations, not CSRS round trips. We froze seeds 42/123/2026 and did not tune on these videos; failed cases remain in the evaluation. Blind annotators and an adjudicator constructed the reference facts. VideoAgent and OACap share a matched 64-frame, 3.1-tool, approximately 30K-controller-token protocol. The expanded protocols use 168/192 frames for VideoAgent/OACap, respectively, with 5.8 calls and 51.6K tokens.

For the 2–10, 10–30, and 30–60 minute groups, respectively, VideoAgent→OACap CSFS under the 64-frame protocol is 77.80→82.15/73.50→78.40/70.10±0.26→74.95±0.33, giving gains of +4.35/+4.90/+4.85. Single-pass→OACap CSECR is 7.2→2.4%/9.5→3.1%/13.8→4.6%. Under the expanded protocols, the gains are +4.80/+4.85/+4.90, and all confidence intervals exclude zero. LVChat therefore provides direct cross-shot generalization evidence over 2–60 minutes, complementing the main short/medium results. The 33-video RT-F1 exploration remains separate. Finally, FSCap-Bench is a challenge-enriched diagnostic subset rather than a representative sample of YouTube's duration distribution: its median duration is 14.7s, and 433/498 videos are under two minutes. We therefore base duration generalization on the direct 2–60-minute evaluations rather than FSCap alone.

Policy-isolation controls and the canonical CSRS definition also appear in our responses to Reviewers 74ut and FmHD. If these results adequately address your concerns, we would be grateful if you would consider re-evaluating the paper's score.

Reviewer_74utOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of when a model is writing a detailed video caption, how should it decide what to look at more closely? The authors propose a system, OACap that starts with an initial description, identifies missing or uncertain details, and uses tools such as cropping, denser frame sampling, OCR, and search to investigate them. It then updates the caption and repeats the process. The key idea is to distinguish details that can be recovered from existing observations from those that require another look at the video.

The authors evaluate captions as reusable memory using a separate, fixed language model answers questions using only the caption, and the captioner never sees those questions. They also introduce FSCap-Bench, with 498 videos, and two metrics for factual coverage and semantic recovery through video reconstruction. The results are promising across several backbones. My main concern is that the experiments establish the benefit of the overall pipeline, but do not yet show how much comes from caption-guided observation specifically, rather than additional computation, tool access, or longer descriptions. The proposed evaluation metrics also need stronger validation.

Critical strengths

The paper addresses an important problem - The core formulation that detailed captioning has no external question telling the model where to focus and using the current caption to identify what needs verification is a reasonable way to guide further observation, is sound. The distinction between revising the text and acquiring new evidence is particularly useful.

There are portions of evaluation which are good (other part see in weaknesses). The improvements are consistent across matched backbones and five OACap variants improve on their corresponding single-pass models across the Video-MME splits. Some gains are demonstrated as substantial. E.g. Qwen3.5-A3B improves from 62.00 to 73.44 on the medium split and from 57.33 to 66.95 on the long split.

The benchmark also makes a useful distinction between central facts and supporting details. Evaluating both correctness and coverage is appropriate for detailed captions. The controlled additions and deletions provide an initial check that CSFS responds to the intended changes. I also appreciate that the limitations section explicitly acknowledges several unresolved evaluation issues.

Critical weaknesses

Evaluation - The main comparison is against a single-pass captioner. OACap receives multiple opportunities to inspect the video, invoke specialized tools, and revise its output. This establishes that the expanded pipeline helps, but does not establish that its observation policy is better than simpler ways of spending the same budget. Additional baselines with uniform additional sampling, fixed segment-by-segment captioning followed by merging, and iterative revision using existing evidence will be helpful. Furthermore, caption length also needs to be reported and controlled separately: using the same output limit does not mean the resulting captions contain the same amount of text. Section 5 says observation cost is reported separately, but I could not find quantitative measurements of runtime, token consumption, or tool calls.

Novelty - Iterative caption refinement already appears in VDC-Agent, while Active Video Perception uses a plan–observe–reflect loop to gather targeted video evidence. OACap’s distinction is that the evolving caption supplies the observation objective, without a downstream question. That is a meaningful formulation, but the paper does not test whether it improves on a generic refinement or evidence-acquisition loop. An ablation removing the revision-versus-observation distinction would directly test a central contribution. Another useful comparison would retain the same tools and budget but replace caption-gap diagnosis with a fixed coverage checklist. The method also needs more operational detail. The equations name the policy, state, and update functions, but the prompts, gap-prioritization rules, observation budgets, and stopping thresholds are not specified sufficiently to reproduce the agent.

Questions for authors
  1. How much of the improvement comes from caption-guided observation itself? Have you compared OACap with uniform additional frame sampling, fixed segment-by-segment captioning, or iterative refinement under comparable compute budgets?

  2. Which definition of CSRS was used for Table 4? Figures 1 and 4 describe scoring the reconstructed caption against video-grounded facts, whereas Equation 9 measures agreement between the source and recovered captions. Please reconcile these definitions.

  3. Please address the novelty concerns in weaknesses.

2 comments
AuthorsOpenReview ↗

Adaptive Controls, Length, and Cost

Dear Reviewer 74ut,

Thank you for identifying the missing controls and the metric-definition ambiguity. We address your concerns in three parts: (1) adaptive and non-adaptive controls under comparable budgets, including length and cost; (2) same-item human validation and the canonical source-grounded CSRS definition; and (3) direct controls for novelty relative to generic refinement and plan–observe–reflect policies.

Q1: Caption-guided observation under comparable compute. AdaFrame (Wu et al., CVPR 2019) and VideoAgent (Wang et al., ECCV 2024) retain their released selector logic and visual/memory state; neither uses an evolving caption. All methods share one VLM/tool-interface adapter, and the selector logic itself is unchanged. For calibrated rows, we freeze two thresholds using 50 videos from the existing training split (12 validation trials), disjoint from FSCap and all test sets.

Token scope: Policy tables count controller inputs and returned observations, excluding internal stage requests and retries. Section 2 reports total serving input separately under each backbone's media/token accounting.

Non-tool controls.

MethodCallsToolsFramesController tokensLatencyWordsCSFS ↑
Single-pass1.00.03211.2K14.2s41566.50
Verbose single-pass1.00.03212.1K15.6s68165.92
Uniform dense sampling1.00.09631.5K36.8s46270.85
VDC-Agent-style self-refine, no new evidence4.00.03228.4K32.1s53166.81

Policy isolation in the shared harness (approximately matched tool/frame budgets). Budgets are approximately matched, but realized averages may differ. All quantities are test-set averages. Released selectors/controllers run in the shared Qwen3.8-27B harness with the same VLM and tools to isolate policy differences; these are not the native end-to-end costs of the original systems.

Policy / method (shared harness)CallsToolsFramesController tokensLatencyWordsCSFS ↑
Fixed non-adaptive schedule4.13.164.030.4K37.5s64277.42
AdaFrame selector (ported)3.63.362.830.1K38.4s67981.40
Active Video Perception-style plan–observe–reflect (ported)3.63.164.030.2K45.6s67782.03
VideoAgent policy (ported)3.53.263.230.0K43.8s67581.85
Learned caption-free controller3.53.163.529.9K36.8s67782.18
OACap concise3.43.164.029.1K39.5s42285.12
OACap full3.43.164.029.8K44.9s68587.24

Released defaults without calibration give AdaFrame, VideoAgent, and AVP 80.96/80.72/81.27, respectively, versus 87.24 for frozen OACap (+6.28/+6.52/+5.97). These gains do not depend on the 50-video calibration. Additional controls include segment-caption + merge at 71.32 (5.0 calls, no tools, 96 frames, 34.0K tokens, 41.5s, 692 words) and a fixed caption-independent checklist at 79.64 (3.5 calls, 3.1 tools, 64 frames, 30.2K tokens, 36.8s, 681 words). Removing the revision/observation split gives 83.11, with 4.2 calls, 4.6 tools, 84 frames, 35.8K tokens, 48.6s, and 695 words. Under approximately matched budgets, OACap exceeds VideoAgent, AVP, the checklist, and the learned controller by +5.39/+5.21/+7.60/+5.06, respectively. The verbose 681-word single-pass degrades, whereas 422-word OACap retains +18.62 over the 415-word baseline; length alone does not explain the gains.

Cost and reproducibility. OACap averages 2.4 observation rounds and 3.1 tool calls per video: 1.4 sampling, 0.9 cropping, 0.6 OCR, and 0.2 search. It uses Nmax⁡=3N_{\max}=3, Kmax⁡=5K_{\max}=5, and at most two parallel calls per round. Gap priorities are conflict (1.0), temporal (0.9), unsupported (0.8), and missing (0.6). The loop stops when no conflict or unsupported claim remains and at most one Core gap is open, when no new supported fact is found, or when the budget is exhausted. Sampling uses 8 fps and at most 32 frames per interval; cropping uses three 1024px keyframes. Controller counts exclude internal key-frame, extraction, fusion, verification, and retry requests. Full serving costs for Qwen3.5-27B, Qwen3.5-122B-A10B, Qwen3.8-27B, and Kimi-K2.6 are:

MethodRequests/videoInput tokens/videoOutput tokens/videoCost/video
Direct1.0021.51–83.06K0.70K¥0.0250–0.0866
OACap37.07232.77–589.80K46.79K¥0.4667–0.8238
AuthorsOpenReview ↗

Adaptive Controls, Length, and Cost – Part2

Per 15 seconds, Direct/OACap require 0.19/6.94 requests, 4.03–15.55K/43.57–110.40K input tokens, 0.13K/8.76K output tokens, and ¥0.0047–0.0162/¥0.0874–0.1542. OACap is substantially more expensive; we claim improved quality under an explicit observation budget, not API-cost parity. Expanded Appendix C specifies the stage prompts, expert suffixes, runtime media inputs, and assembly code, distinguishing current listings from historical runs.

Q2: Metric validation and canonical round-trip definition. The shared 120-video validation frame comprises 40 post-filter FSCap videos and 80 disjoint prefilter videos. Three blind judges rated the 200 captions in the 40-video FSCap stratum (five tiers; α=0.81\alpha=0.81). Against these ratings, CSFS obtains Pearson r=0.89r=0.89, Spearman ρ=0.88\rho=0.88, and 91.0% pairwise accuracy, compared with ρ=0.62\rho=0.62 for VDCScore and 0.34 for FactVC. Five independent annotators label each of the 498 FSCap videos, followed by adjudication; pre-adjudication validity agreement is Fleiss' κ=0.84\kappa=0.84. CSFS is therefore supported by same-item human validation, not only controlled perturbations.

Mechanism and canonical CSRS. For Qwen3.8-27B, the improvement is recall-driven: P/Rcore/RsupP/R_{\mathrm{core}}/R_{\mathrm{sup}} changes from 88.5/59.9/49.8→90.8/88.0/81.6. Verified claims increase by +16.4 per video, and the matched controls attribute +5.06 to caption-gap routing. The equation/figure mismatch arose from an obsolete early definition that remained in Eq. 9, Appendix B.2, and Table 4 by oversight. The definition in Figures 1/4 is canonical: for V→C→V′→C′V\rightarrow C\rightarrow V'\rightarrow C', CSRS(C∣V)=CSFS(C′∣V)=2λP′Ccore′P′+Ccore′+(1−λ)S′,λ=0.8.CSRS(C\mid V)=CSFS(C'\mid V)=\frac{2\lambda P'C'_{\mathrm{core}}}{P'+C'_{\mathrm{core}}}+(1-\lambda)S',\qquad \lambda=0.8.

Claims from C′C' are evaluated against the source video's human-verified Core/Support facts, yielding claim precision P′P', importance-weighted Core coverage Ccore′C'_{\mathrm{core}}, and Support coverage S′S'. The λ\lambda and zero-denominator rules follow CSFS. The obsolete symmetric 2∣M∣/(∣A∣+∣B∣)2|M|/(|A|+|B|) score measures only CC–C′C' agreement and corresponds to RT-F1. We have relabeled those legacy values as RT-F1; they are not CSRS. The source-grounded scores under the figure definition are the correct CSRS values and are retained. Human-validated CSFS on FSCap-Bench remains primary, while RT-F1 remains an auxiliary agreement diagnostic. The RT-F1 study of 120 videos and 600 captions reuses the shared validation frame; only the original 33-video exploration uses a separate sample.

Q3: Novelty relative to iterative refinement and adaptive perception. VDC-Agent-style self-refine reuses existing evidence, while AVP uses a generic plan–observe–reflect loop. OACap instead diagnoses missing or unsupported facts from the evolving caption and separates revision from new observation. The no-new-evidence, AVP, fixed-checklist, and no-split controls in Q1 test these distinctions directly.

Complete-workflow reproductions and additional cost, human-validation, and fixed-channel results are provided in our responses to Reviewers 4Yx7 and FmHD. If these additions adequately address your concerns, we would be grateful if you would consider re-evaluating the paper's score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.